Copy formatted org-mode text from Emacs to other applications

| categories: rtf, emacs | tags:

I do a lot of writing in org-mode and I thought it would be great if I could copy text from an org-file and paste it with formatting into other applications, e.g. Word, Gmail, etc…. Curiosity got the better of me and I wondered how this is done in other applications. It works by creating a Rich Text Format version of what you want to copy and then putting that on the clipboard. It isn't quite enough to just copy it, it needs to go in the clipboard as an RTF datatype. On Mac OSX I used pbcopy to make that happen.

Check out this video of this post in action: https://www.youtube.com/watch?v=irkmQnggVpE

One simple strategy to do this from org-mode is to generate HTML by export, and then convert it to RTF with a utility, e.g. textutil. For example like this.

(defun formatted-copy ()
  "Export region to HTML, and copy it to the clipboard."
  (interactive)
  (save-window-excursion
    (let* ((buf (org-export-to-buffer 'html "*Formatted Copy*" nil nil t t))
           (html (with-current-buffer buf (buffer-string))))
      (with-current-buffer buf
        (shell-command-on-region
         (point-min)
         (point-max)
         "textutil -stdin -format html -convert rtf -stdout | pbcopy"))
      (kill-buffer buf))))

(global-set-key (kbd "H-w") 'formatted-copy)

This works well for everything but equations and images. Citations leave a bit to be desired, but improving this is still a challenge.

Let us try this on some text. Some bold, italic, underline, struck and verbatim text to copy. Here are some example Formulas: H2O ionizes to form H+. We simply must have an equation: \(e^{i\pi} + 1 = 0\) 1. We should also have a citation kitchin-2015-examp and multiple citations kitchin-2016-autom-data,kitchin-2015-data-surfac-scien 2.

A code block:

import pycse.orgmode as org
import numpy as np
import matplotlib.pyplot as plt

x = np.linspace(0, 60, 500)
plt.figure(figsize=(4, 2))
plt.plot(np.exp(-0.1 * x) * np.cos(x),
         np.exp(-0.1 * x) * np.sin(x))
org.figure(plt.savefig('spiral.png'),
           caption='A spiral.',
           attributes=[['org', ':width 100']])
print('')
org.table([['H1', 'H2'], None, [1, 2], [2, 4]],
          caption='A simple table')
print('')
org.result(6 * 7)

Figure 1: A spiral.

Table 1: A simple table
H1 H2
1 2
2 4
42

In summary, this simple approach to generating RTF from exported HTML works really well for the simplest markups. To improve on getting figures in, getting cross-references, captions, proper references, etc… will require a more sophisticated export approach, and probably one that exports RTF directly. That is a big challenge for another day!

Bibliography

Footnotes:

1

There are probably some ways to get better images for equations. To get equation numbers and references to them will probably require a two pass build process.

2

This is another place where configuration will be required for bibliography style. Also, some checks to join neighboring footnotes.

Copyright (C) 2016 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.3.4

Discuss on Twitter

Using prefix args in ivy actions

| categories: ivy, emacs | tags:

Table of Contents

There is a brand new feature in ivy which allows you to pass prefix args to the action functions. This change has made it into MELPA by now, so you can try it out. Check out this 1 for an emacs -Q setup that I used for this post. This installs ivy and org-ref with some minimal setup.

The video for this post can be found here: https://www.youtube.com/watch?v=Y8HHLAE_-yA

In this post I will show how to use this new feature to create an ivy selection function that inserts a citation from a bibtex file, and with a prefix arg lets you choose the type of citation to insert.

org-ref provides a function that generates candidates for selection. Each candidate is a list where the car of the list is a display string, and the cdr is an a-list of properties. I have a lot of entries in here, so it is important to have a convenient selection tool.

(setq org-ref-bibtex-files '("references.bib"))
(length (orhc-bibtex-candidates))

Here is an example of the first entry in my bibliography. We will need to extract the key from that.

(elt (orhc-bibtex-candidates) 0)

Here is the key from that entry.

(cdr (assoc "=key=" (elt (orhc-bibtex-candidates) 0)))

By default we will insert that as kitchin-2015-examp but there are other types of citations we might use too like kitchin-2015-examp. org-ref provides a list of citation types we could insert. Here they are. This somewhat complicated code just wraps the string so it fits in the blog post nicely.

(with-temp-buffer 
  (insert (format "%s" org-ref-cite-types))
  (fill-region (point-min) (point-max))
  (buffer-string))

So, we are now prepared to make a simple ivy function to query our bibliography that has a default action to insert a standard citation, but we can use a prefix to change the citation type. The prefix arg is stored in the global variable ivy-current-prefix-arg which can be checked inside the action function. We can check for it in the action function and do something different if a prefix arg is used. Here is the function.

(defun cite ()
  (interactive)
  (ivy-read "select: " (orhc-bibtex-candidates)
            :action (lambda (entry) 
                      (let ((key (cdr (assoc "=key=" entry)))
                            (type (if ivy-current-prefix-arg
                                      (ivy-read "type: " org-ref-cite-types)
                                    "cite")))
                        (with-ivy-window
                          (insert (format "%s:%s" type key)))))))

To get the default action, we run M-x cite, enter our query, select an entry and press return. To get an alternate cite type, we run M-x cite, enter the query, select an entry, then type C-u return, which will prompt you for an alternate citation type, then insert your choice and the citation. Here are some examples. kitchin-2015-examp kitchin-2015-examp kitchin-2015-examp

In summary, these aren't functions you would want to use; they don't handle a lot of the nuances of multiple citations. They are just to illustrate in a pretty simple way how easy it is to use a prefix arg in an ivy action function now!

1 Bare bones setup

This will setup the bare bones emacs that I used for this post.

(setq package-user-dir (expand-file-name "sandbox"))

(setq package-archives
      '(("melpa" . "http://melpa.org/packages/")))

(require 'package)

;;; Code:

(package-initialize)

(unless (package-installed-p 'use-package)
  (package-refresh-contents)
  (package-install 'use-package))

(eval-when-compile
  (require 'use-package))

(setq use-package-always-ensure t)

(use-package ivy)

(use-package org-ref
 :init 
 (setq org-ref-default-bibliography '("~/Dropbox/bibliography/references.bib"))
 :config (require 'org-ref-helm-cite))

(global-visual-line-mode 1)
(setq org-confirm-babel-evaluate nil)
(load-theme 'leuven)

Copyright (C) 2016 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.3.4

Discuss on Twitter

Dynamic sorting with ivy

| categories: ivy, emacs | tags:

I have been exploring ivy a lot these days as a general purpose completion backend. One need I have is dynamic resorting of candidates. I illustrate how to achieve that here. A big thanks to Oleh Krehel (author of ivy) for a lot help today getting this working!

You may want to check out the video: https://www.youtube.com/watch?v=nFKfM3MOAd0

First, a typical ivy-read example. Below I have a set of contact data for some people, and have setup an ivy-read command that inserts the email in the current buffer by default, and a second action for the phone. What is missing that I would like to do is dynamically reorder the candidates, including sorting all the candidates, swapping candidates up and down to fine tune the order, and then finally applying an action to all the candidates.

(defun ct ()
  (interactive)
  (ivy-read "contact: " '(("Kno Body" "kb@true.you" "555-1212")
                          ("A. Person" "ap@some.come" "867-5304")
                          ("G. Willikers" "gw@not.me" "555-5555"))
            :action '(1
                      ("o" (lambda (x)
                             (with-ivy-window
                               (insert
                                (if (not (looking-back " ")) ", " "")
                                (elt x 0))))
                       "insert email")
                      ("p" (lambda (x)
                             (with-ivy-window
                               (insert
                                (if (not (looking-back " ")) ", " "")
                                (elt x 1))))
                       "insert phone"))))

So, first a set of functions to manipulate the candidates. We create a swap function, two functions to move candidates up and down, and two functions that sort the whole list of candidates in ascending and descending order. In each case, we just update the ivy collection with the new modified collection, we save the currently selected candidate, and then reset the state to update the candidates.

(defun swap (i j lst)
  "Swap index I and J in the list LST." 
  (let ((tempi (nth i lst)))
    (setf (nth i lst) (nth j lst))
    (setf (nth j lst) tempi))
  lst)

(defun ivy-move-up ()
  "Move ivy candidate up."
  (interactive)
  (setf (ivy-state-collection ivy-last)
        (swap ivy--index (1- ivy--index) (ivy-state-collection ivy-last)))
  (setf (ivy-state-preselect ivy-last) ivy--current)
  (ivy--reset-state ivy-last))

(defun ivy-move-down ()
  "Move ivy candidate down."
  (interactive)
  (setf (ivy-state-collection ivy-last)
        (swap ivy--index (1+ ivy--index) (ivy-state-collection ivy-last)))
  (setf (ivy-state-preselect ivy-last) ivy--current)
  (ivy--reset-state ivy-last))

(defun ivy-a-z ()
  "Sort ivy candidates from a-z."
  (interactive)
  (setf (ivy-state-collection ivy-last)
        (cl-sort (ivy-state-collection ivy-last)
                 (if (listp (car (ivy-state-collection ivy-last)))
                     (lambda (a b)
                       (string-lessp (car a) (car b)))
                   (lambda (a b)
                     (string-lessp a b)))))
  (setf (ivy-state-preselect ivy-last) ivy--current)
  (ivy--reset-state ivy-last))

(defun ivy-z-a ()
  "Sort ivy candidates from z-a."
  (interactive)
  (setf (ivy-state-collection ivy-last)
        (cl-sort (ivy-state-collection ivy-last)
                 (if (listp (car (ivy-state-collection ivy-last)))
                     (lambda (a b)
                       (string-greaterp (car a) (car b)))
                   (lambda (a b)
                     (string-greaterp a b)))))
  (setf (ivy-state-preselect ivy-last) ivy--current)
  (ivy--reset-state ivy-last))

Now, we make a keymap to bind these commands so they are convenient to use. I will use C-arrows for swapping, and M-arrows for sorting the whole list. I also add M-<return> which allows me to use a numeric prefix to apply an action to all the candidates. M-<return> applies the default action. M-1 M-<return> applies the first action, M-2 M-<return> the second action, etc…

This specific implementation assumes your candidates have a cdr.

(setq ivy-sort-keymap
      (let ((map (make-sparse-keymap)))
        (define-key map (kbd "C-<up>") 'ivy-move-up)
        (define-key map (kbd "C-<down>") 'ivy-move-down)

        ;; sort all keys
        (define-key map (kbd "M-<up>") 'ivy-a-z)
        (define-key map (kbd "M-<down>") 'ivy-z-a)

        ;; map over all all entries with nth action
        (define-key map (kbd "M-<return>")
          (lambda (arg)
            "Apply the numeric prefix ARGth action to every candidate."
            (interactive "P")
            ;; with no arg use default action
            (unless arg (setq arg (car (ivy-state-action ivy-last))))
            (ivy-beginning-of-buffer)
            (let ((func (elt (elt (ivy-state-action ivy-last) arg) 1)))
              (loop for i from 0 to (- ivy--length 1)
                    do 
                    (funcall func
                             (let ((cand (elt
                                          (ivy-state-collection ivy-last)
                                          ivy--index)))
                               (if (listp cand)
                                   (cdr cand)
                                 cand)))
                    (ivy-next-line)))
            (ivy-exit-with-action
             (lambda (x) nil))))
        map))

Ok, now we modify our ivy-read function to use the keymap.

(defun ctn ()
  (interactive)
  (ivy-read "contact: " '(("Kno Body" "kb@true.you" "555-1212")
                          ("A. Person" "ap@some.come" "867-5304")
                          ("G. Willikers" "gw@not.me" "555-5555"))
            :keymap ivy-sort-keymap
            :action '(1
                      ("o" (lambda (x)
                             (with-ivy-window
                               (insert
                                (if (not (looking-back " ")) ", " "")
                                (elt x 0))))
                       "insert email")
                      ("p" (lambda (x)
                             (with-ivy-window
                               (insert
                                (if (not (looking-back " ")) ", " "")
                                (elt x 1))))
                       "insert phone"))))

kb@true.you, gw@not.me, ap@some.come, 555-1212, 555-5555, 867-5304

Copyright (C) 2016 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.3.4

Discuss on Twitter

ob-hy.el - or better integration of hylang in org-mode

| categories: orgmode, hylang, emacs | tags:

The point of this post is to develop and test a more substantial integration of Hy into org-mode. We develop ob-hy.el here. This is based off of ob-clojure.el.

The next few blocks will get tangled to ob-hy.el. First, some variables.

(require 'ob)

(add-to-list 'org-structure-template-alist
             '("hy" "#+BEGIN_SRC hy\n?\n#+END_SRC" "<src lang=\"hy\">\n?\n</src>"))

(defvar org-babel-tangle-lang-exts)
(add-to-list 'org-babel-tangle-lang-exts '("hy" . "hy"))

(defvar org-babel-default-header-args:hy '())
(defvar org-babel-header-args:hy '((:results . "output")))
org-babel-header-args:hy

Next a function to expand the code body. This will allow us to pass vars in the header.

(defun org-babel-expand-body:hy (body params)
  "Expand BODY according to PARAMS, return the expanded body."
  (let* ((vars (mapcar #'cdr (org-babel-get-header params :var)))
         (result-params (cdr (assoc :result-params params)))
         (print-level nil)
         (print-length nil)
         (body (org-babel-trim
                (if (> (length vars) 0)
                    (concat "(let ["
                            (mapconcat
                             (lambda (var)
                               (format
                                "%S (quote %S)"
                                (car var)
                                (cdr var)))
                             vars "\n      ")
                            "]\n" body ")")
                  body))))
    (when (not (member "output" result-params))
      (setq body (format "(print (do  %s\n))" body)))
    body))
org-babel-expand-body:hy

And a function to execute the body. We still use a simple approach to write the code to a temp-file, execute it, capture the output, and delete the file. This limits things to

(defun org-babel-execute:hy (body params)
  "Execute a block of hy code with Babel."
  (let* ((temporary-file-directory ".")
         (tempfile (make-temp-file "hy-"))
         result
         (result-params (cdr (assoc :result-params params)))
         (body (org-babel-expand-body:hy body params)))

    (with-temp-file tempfile
      (insert body))

    (unwind-protect
        (progn
          (cond
           ((member "body" result-params)
            (setq result body))
           ((member "python" result-params)
            (setq result (shell-command-to-string
                          (format "hy2py %s" tempfile))))
           ((member "ast" result-params)
            (setq result (shell-command-to-string
                          (format "hy2py -a -np %s" tempfile))))
           (t
            (setq result (shell-command-to-string
                          (format "hy %s" tempfile)))))

          (org-babel-result-cond result-params
            result
            (condition-case nil (org-babel-script-escape result)
              (error result))))
      (delete-file tempfile))))

(provide 'ob-hy)
ob-hy

Now we tangle and load those blocks.

(org-babel-tangle)
(load-file "ob-hy.el")
t

Next, we do some tests. They are all simple tests.

1 Tests

1.1 Simple

(print "Hy world")
Hy world

We can see how this turns into Python:

(print "Hy world")
print(u'Hy world')

or the AST:

(print "Hy world")
Module(
    body=[Expr(value=Call(func=Name(id='print'), args=[Str(s=u'Hy world')], keywords=[], starargs=None, kwargs=None))])

Let's test :results value. It is not quite the value since we seem to get everything that is output from the script, but if you don't print stuff, it seems to get it right.

"test"
(+ 1 2 3)
6

1.2 vars in header

Here we test out adding variables to the header lines.

(print "Hy" data)
Hy world

Interesting, I am not sure where the space between them comes from. Let's check out the :results body option. It will show us the hy script that gets run.

(print "Hy" data)
(let [data (quote "world")]
(print "Hy" data))

Nothing obvious about the space there. We can test out passing block results in here.

(print data)
Hy  world

Here is the body of that:

(print data)
(let [data (quote "Hy world
")]
(print data))

2 Summary

It works well enough to make testing in org-mode pretty convenient. I can't think of anything else it "needs" right now, although communication with a repl might make it faster, and sessions are not supported at the moment. Saving that for another day ;)

Copyright (C) 2016 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.10

Discuss on Twitter

More on Hy and why I think it is a big deal

| categories: emacs, hylang, python | tags:

Yesterday I talked about hylang , a Lisp that basically compiles to and runs Python code. Today, I want to show a few reasons why this is a great idea, and an important one. Below I give a few examples of why the hylang approach is better (in my opinion of course) than Python with a few examples of things I have always wanted in Python but couldn't get.

1 Editing with hy-mode and lispy

There is a major mode for Hy: https://github.com/hylang/hy-mode also on MELPA. It gives us some syntax highlighting and better access to a REPL.

Let's load lispy (https://github.com/abo-abo/lispy ) for it so we also get amazing editing. I always wanted to use lispy style navigation and editing in Python, but the whitespace and indentation did not make it that easy. Problem solved with these. @abo-abo already added basic eval support for Hy to lispy since the post yesterday (https://github.com/abo-abo/lispy/commit/f7f71e38e241d92b6add05be6628ac987067b11c ); Thanks!

(add-hook 'hy-mode-hook
          (lambda ()
            (lispy-mode 1)))

2 Python with no whitespace, or commas in lists

You can still use indentation (it is good style), but this works!

(for [x [0 1 2 3 4 5]]
(if (> x 3) (print "too big")
(print x)))
0
1
2
3
too big
too big

This looks nicer.

(for [x [0 1 2 3 4 5]]
  (if (> x 3)
    (print "too big")
    (print x)))
0
1
2
3
too big
too big

This is a big deal too. Using Python in sessions in org-mode has always been a little complicated by the indentation and whitespace, especially with nested loops and functions. That problem is probably gone.

3 No confusion in expressions in statements

In Python you can do this:

a = 5
print(a)
print(a + 5)
5
10

But not this:

print(a=5)
print(a + 5)
  File "<stdin>", line 1
   print(a=5)
          ^
SyntaxError: invalid syntax

You can't put assignment statements and expression statements anywhere you want, they are only legal syntax in some places. For example, a=5 above actually looks like the print function has an argument of a that set to 5. Not true in Lisp; there are only expressions! So this works fine.

(print (setv a 5))
(print (+ a 5))
5
10

I just like this style of simple syntax.

4 Proper multiline lambda functions

Python syntax fundamentally limits you to one line lambdas. Not so for Hy. Let's use one in a filter to print even numbers. Here is an example with a two-liner but you could make them more complicated. In Python, you have to make a separate function for this. That isn't terrible, but if it is never used for anything else, it could be avoided.

(setv a [0 1 2 3 4 5 6 7 8])

(defn display [list filter]
  (for [x list] (if (filter x) (print x))))

(display a (lambda [x]
             (= (% x 2) 0)))
0
2
4
6
8

5 Macros and Extensible syntax

It is not easy to get real macro (code expansion) behavior in Python. Yes, there are decorators, and closures, and related things that get close to it. But there are not lisp-like macros.

Here is a (too) simple macro to allow for infix notation. It only works for two arguments, but could be extended for multiple arguments.

(defmacro infix [code]
  (quasiquote ((unquote (get code 1))
               (unquote (get code 0))
               (unquote (get code 2)))))

(print (infix (1 + 1)))
2

If we want new syntax we can get it!

(defreader $ [code]
  (quasiquote
   ((unquote (get code 1))
    (unquote (get code 0))
    (unquote (get code 2)))))

(print #$(1 + 1))
2

Why is this nice? Here is a math example that shows why you might want to change syntax.

5.1 Some math

See http://kitchingroup.cheme.cmu.edu/blog/2013/02/07/Solving-Bessel-s-Equation-numerically/ for the Python version of solving the Bessel equation numerically. Here we do it with hylang.

Why would we want infix notation? Here is a good reason. The prefix notation is not easy to read. Compare:

dzdx = 1.0 / x**2 * (-x * z - (x**2 - nu**2) * y)

to

(setv dzdx (* (/ 1.0 (** x 2)) (- (* (* -1 x) z) (* (- (** x 2) (** nu 2)) y))))

The infix notation is simpler to read. Still, the code below is not that hard to figure out, especially if there was a generalized infix notation that allowed (with parens for explicit operation precedence):

(setv dzdx (nfx (1.0 / x**2) * ((-x * z) - ((x**2 - nu**2) * y))))

So, here is the hylang equivalent to my previous Python version.

(import [numpy :as np])
(import [scipy.integrate [odeint]])
(import [scipy.special [jn]])
(import [matplotlib.pyplot :as plt])

(defn fbessel [Y x]
  "System of 1st order ODEs for the Bessel equation."
  (setv nu 0.0
        y (get Y 0)
        z (get Y 1))

  ;; define the derivatives
  (setv dydx z
        dzdx (* (/ 1.0 (** x 2)) (- (* (* -1 x) z) (* (- (** x 2) (** nu 2)) y))))
  ;; return derivatives
  [dydx dzdx])

(setv x0 1e-15
      y0 1.0
      z0 0.0
      Y0 [y0 z0])

(setv xspan (np.linspace 1e-15 10)
      sol (odeint fbessel Y0 xspan))

(plt.plot xspan (. sol [[Ellipsis 0]]) :label "Numerical solution")
(plt.plot xspan (jn 0 xspan) "r--" :label "Analytical solution")
(plt.legend :loc "best")

(plt.savefig "hy-ode.png")
2016-04-01 13:48:17.499 Python[12151:d13] CoreText performance note: Client called CTFontCreateWithName() using name "Lucida Grande" and got font with PostScript name "LucidaGrande". For best performance, only use PostScript names when calling this API.
2016-04-01 13:48:17.499 Python[12151:d13] CoreText performance note: Set a breakpoint on CTFontLogSuboptimalRequest to debug.
None

This looks really good to me, except for that prefix math. The array slice syntax is interesting. Not that obvious yet.

6 Interoperability with Python

http://docs.hylang.org/en/latest/tutorial.html#hy-python-interop

Write Hy code and use it in Python. Use Python code in Hy. Repeat. Sweet.

7 Integration of emacs and Hy

This isn't so beautiful but it illustrates a pretty awesome integration of Hy(python) into Emacs!

(defmacro hy (body)
  `(let* ((temporary-file-directory ".")
          (tempfile (make-temp-file "hy-")))
     (message (format "code: %S" ,body))
     (with-temp-file tempfile
       (mapc (lambda (form) (insert (format "%s" form))) ,body))
     (read (unwind-protect
               (shell-command-to-string
                (format "hy %s" tempfile))
             (delete-file tempfile)))))

(aref (hy '((import numpy)
            (setv a (numpy.array [1 2 3]))
            (setv b (numpy.array [1 2 3]))
            (print (* a b))))
      1)
4

This isn't perfect, and there are many ways it could break down. But if you are careful to make the output "read"able, you can literally embed Hy code in Emacs lisp and use the results, a total win for Science! I feel like it might need something like progn, but that would not change what this does dramatically.

8 Hypster and Hy Society.

http://notes.pault.ag/hy-survival-guide/ ROTFL. ironically of course ;)

And the @hylang Twitter account is run by Hy Society. Nice.

9 What do we still need?

  1. Experience. Hy seems relatively young compared to other Lisps. It isn't clear yet if this could work like Python does at scale in research. I sure look forward to finding out though!
  2. Proper infix notation for engineering math. I could live with no operator precedence if it led to a quicker solution for now. As long as something like (1.0 / x**2 * (-x * z - (x**2 - nu**2) * y)) is legal!
  3. A proper integration with org-mode and the REPL.
  4. Toolchains like emacs-lisp has. I just love those. Killer debugging, access to hyperlinked documentation, code navigation, … Maybe integration with something like SLIME or CIDER? Hyder?
  5. Use it in a proper big project to find out where the limitations are, maybe Hycse as a companion to Pycse (http://kitchingroup.cheme.cmu.edu/pycse/ )? or a rewrite of http://kitchingroup.cheme.cmu.edu/dft-book/ in Hy?

Overall, I am pretty excited about this project. The syntax is a bit reminiscent of Clojure, and Racket, the former by design. Lots of new ideas still seem to be percolating in, so there is likely good stuff to see in the future!

I haven't used it enough to see the warts yet, but already the top issues I had with Python are largely addressed, so I see this as a way to continue progress with all the benefits of Python.

Copyright (C) 2016 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.10

Discuss on Twitter
« Previous Page -- Next Page »