The Kitchin Research Group: emacs

Adding emacs command key-bindings and help functionality to org-mode

Posted November 22, 2015 at 10:08 AM | categories: orgmode, emacs | tags:

The documentation of functions in emacs allows you to put some light markup into function doc strings that will render as the key sequence required to run the command when you look up the help on the function. I would like to have something like that in org-mode. You can look up the key-binding to a command like this:

(substitute-command-keys "\\[org-agenda]")

C-c a

We are going to explore a way to recognize the syntax shown above, change its appearance to alert us that we are looking at an emacs command, add a tooltip, and make it clickable to open the documentation, and s (super) clickable to find the function code. Font lock is the tool we will use for this. Basically, we need a regular expression to match the syntax, and a function to find the next instance, and put some properties on the matched text.

I made a video (https://www.youtube.com/watch?v=VLUMW0sR4Vk ) showing what this post is all about. It isn't easy to see in the post ☺.

Here we use the `rx' library to build up a regular expression for this. It is a bit easier to document than a raw regexp. Since we are matching \ in the pattern, there are some obligatory escaping \ characters in there too. All we need is to integrate this into font-lock. We define a function that will move the point to the end of the next match, and put properties on the match. We will go ahead and make the text clickable so we can access documentation and code easily. The tooltip will show the key-binding to run the command.

(require 'rx)

(defvar elisp-symbol-keybinding-re
  (rx
   ;; opening \\[
   (eval "\\[")
   ;; one or more characters that are not ]
   (group (one-or-more (not (any "]"))))
   ;; The closing ]
   "]")
"Regexp for an elisp command keybinding syntax. \\[some-command]
Regexp group 1 matches `some-command'.")

(defun match-next-keybinding (&optional limit)
  "Move point to the end of the next expression matching
`elisp-symbol-keybinding-re', and put properties on the match
that shows the key sequence. Non-bound commands are not
fontified."
  (when (and (re-search-forward
              elisp-symbol-keybinding-re
              limit t)
             (fboundp (intern (match-string 1))))
    (let* ((beg (match-beginning 0))
           (end (match-end 0))
           (s (match-string 0))
           (command (match-string 1))
           (describe-func `(lambda ()
                    "Run `describe-function' on the command."
                    (interactive)
                    (describe-function (intern ,command))))
           (find-func `(lambda ()
                     "Run `find-function' on the command."
                     (interactive)
                     (find-function (intern ,command))))
           (map (make-sparse-keymap)))

      ;; this is what gets run when you click on it.
      (define-key map [mouse-1] describe-func)
      (define-key map [s-mouse-1] find-func)
      ;; Here we define the text properties
      (add-text-properties
       beg end
       `(local-map ,map
         mouse-face highlight
         help-echo ,(format
                     "%s\n\nClick for documentation.\ns-mouse-1 to find function."
                     (substitute-command-keys s))
         keybinding t)))))

Let's go ahead and make syntax for `some-command' too. This one seems simple enough we just write a regexp for it.

(defun match-next-emacs-command (&optional limit)
  "Move point to the end of the next expression matching
`this-syntax', and put a tooltip on the match
that shows the key sequence. Works on commands and variables."
  (when (and (re-search-forward
              "`\\([^']+\\)'"
              limit t)
             (or (boundp (intern (match-string 1)))
                 (fboundp (intern (match-string 1)))))
    (let* ((beg (match-beginning 0))
           (end (match-end 0))
           (s (match-string 0))
           (command (match-string 1))
           (describe-func
            `(lambda ()
               "Run `describe-function/variable' on the command."
               (interactive)
               (cond ((fboundp (intern ,command))
                      (describe-function (intern ,command)))
                     ((boundp (intern ,command))
                      (describe-variable (intern ,command))))))
           (find-func `(lambda ()
                     "Run `find-function' on the command."
                     (interactive)
                     (find-function (intern ,command))))
           (map (make-sparse-keymap)))

      ;; this is what gets run when you click on it.
      (define-key map [mouse-1] describe-func)
      (define-key map [s-mouse-1] find-func)
      ;; Here we define the text properties
      (add-text-properties
       beg end
       `(local-map ,map
         mouse-face highlight
         help-echo ,(format
                     "%s\n\nClick for documentation.%s"
                     (if (fboundp (intern command))
                         (substitute-command-keys (format "\\[%s]" command))
                       "Variable")
                     (if (fboundp (intern command))
                         "\ns-mouse-1 to find function."
                       ""))
         keybinding t)))))

Now we need a way to turn them on and off. We do that here with a minor mode.

(define-minor-mode emacs-keybinding-command-tooltip-mode
  "Fontify on emacs keybinding syntax. Adds a tooltip for
keybinding, and make the command clickable to get to the
documentation."
  :lighter " KB"
  (if emacs-keybinding-command-tooltip-mode
      ;; turn them on
      (font-lock-add-keywords
       nil
       '((match-next-keybinding 1 font-lock-constant-face)
         (match-next-emacs-command 1 font-lock-constant-face)))
    ;; turn them off
    (font-lock-remove-keywords
     nil
     '((match-next-keybinding 1 font-lock-constant-face)
       (match-next-emacs-command 1 font-lock-constant-face))))
  (font-lock-fontify-buffer))

Here we turn it on:

(emacs-keybinding-command-tooltip-mode -1)

Here are some sample uses. You can use \\[org-toggle-latex-overlays] to toggle latex overlays.

You can use \\[org-ref-helm-insert-cite-link] to insert citations.

That more or less does it! I don't know if this is the canonical way to do this, but it works nicely here. You can also use overlays, but I found them a little confusing because they are not editable, and you have to toggle the minor mode to see them. Here we have unobtrusive tooltips. One downside is these won't export in any fashion in org-mode since it is not part of the syntax. It might be a good idea to adjust `font-lock-extra-managed-props' for this

It works for this syntax too: `helm', which is also commonly used in doc strings. This should be pretty handy in org-mode documents about Emacs!

org-mode source

Org-mode version = 8.2.10

Discuss on Twitter

Insert org-entities into org-mode with helm

Posted November 21, 2015 at 11:37 AM | categories: helm, orgmode, emacs | tags:

org-mode has a lot of pre-defined entities (see http://kitchingroup.cheme.cmu.edu/blog/2013/10/03/Exporting-accented-characters-to-latex-from-org-mode/ ), otherwise known to me as non-ascii characters. I rarely remember what these are, and occasionally want to insert the LaTeX or HTML code, so here we build a helm command to show them to me, and allow me to select one for insertion. We generate the helm sources from org-entities below. It works pretty well!

(defun helm-insert-org-entity ()
  "Helm interface to insert an entity from `org-entities'.
F1 inserts utf-8 character
F2 inserts entity code
F3 inserts LaTeX code (does not wrap in math-mode)
F4 inserts HTML code"
  (interactive)
  (helm :sources (reverse
                  (let ((sources '())
                        toplevel
                        secondlevel)
                    (dolist (element (append
                                      '("* User" "** User entities")
                                      org-entities-user org-entities))
                      (when (and (stringp element)
                                 (s-starts-with? "* " element))
                        (setq toplevel element))
                      (when (and (stringp element)
                                 (s-starts-with? "** " element))
                        (setq secondlevel element)
                        (add-to-list
                         'sources
                         `((name . ,(concat
                                     toplevel
                                     (replace-regexp-in-string
                                      "\\*\\*" " - " secondlevel)))
                           (candidates . nil)
                           (action . (("insert utf-8 char" . (lambda (candidate)
                                                               (insert (nth 6 candidate))))
                                      ("insert org entity" . (lambda (candidate)
                                                           (insert (concat "\\" (car candidate)))))
                                      ("insert latex" . (lambda (candidate)
                                                          (insert (nth 1 candidate))))
                                      ("insert html" . (lambda (candidate)
                                                         (insert (nth 3 candidate)))))))))
                      (when (and element (listp element))
                        (setf (cdr (assoc 'candidates (car sources)))
                              (append
                               (cdr (assoc 'candidates (car sources)))
                               (list (cons
                                      (format "%10s %s" (nth 6 element) element)
                                      element))))))
                    sources))))

helm-insert-org-entity

Now I can write things like the particle was 60 Å in diameter at a temperature of 600°C, leading to an expansion coefficient of α=0.2 ± 0.01. It isn't quite as fast as knowing the keyboard shortcuts for those symbols, but a lot faster than looking them up then copy and pasting them. So far it seems like these export to HTML and LaTeX just fine, and they are more convenient and better looking than using the org-entities codes. This will make its way into jmax soon.

org-mode source

Org-mode version = 8.2.10

Discuss on Twitter

Asynchronously running python blocks in org-mode

Posted November 20, 2015 at 11:46 AM | categories: python, orgmode, emacs | tags:

Updated November 20, 2015 at 07:30 PM

If you run long Python blocks from org-mode, you might want to keep working while it runs. Currently Emacs gets blocked and you have to wait patiently. In this post we consider some ways to avoid this that run our code asynchronously, but still put results where they belong in the org-buffer.

This is a long post. You may want to see the video: https://www.youtube.com/watch?v=VDyoN8yipSE , or skip to the end where the best and final version is shown.

1 The async module

Here we consider an approach that uses https://github.com/jwiegley/emacs-async module. The idea is to tangle the Python block at point to a temp file, then asynchronously run it. We capture the output and put it back in the buffer. We use a uuid to find the place to put the results in org-mode format. Here is the code that implements this idea.

(require 'async)

(defun org-babel-async-execute ()
  "Run a python block at point asynchrously."
  (interactive)

  (let ((current-file (buffer-file-name))
        (uuid (org-id-uuid))
        (temporary-file-directory "./")
        (tempfile (make-temp-file "py-")))

    (org-babel-tangle '(4) tempfile)
    (org-babel-remove-result)
    (save-excursion
      (re-search-forward "#\\+END_SRC")
      (insert (format
               "\n\n#+RESULTS: %s\n: %s"
               (or (org-element-property :name (org-element-context))
                   "")
               uuid)))

    (async-start
     ;; what to start
     `(lambda ()
        ;; now we run the command then cleanup
        (prog1
            (shell-command-to-string (format "python %s" ,tempfile))
          (delete-file ,tempfile)))

     `(lambda (result)
        "Code that runs when the async function finishes."
        (save-window-excursion
          (save-excursion
            (save-restriction
              (with-current-buffer (find-file-noselect ,current-file)
                (goto-char (point-min))
                (re-search-forward ,uuid)
                (beginning-of-line)
                (kill-line)
                (insert (mapconcat
                         (lambda (x)
                           (format ": %s" x))
                         (butlast (s-split "\n" result))
                         "\n"))))))))))

org-babel-async-execute

Here is a block to test it on. We can run the block, and keep on working while the code runs. The results seem to get inserted correctly at the right point even if I am in another window or frame! We don't get easy access to continuous output of the command. This wouldn't work if we close Emacs, but who does that?

print 'hello world'
import time
time.sleep(5)

import os
print os.getcwd()
print time.asctime()

hello world
/Users/jkitchin/blogofile-jkitchin.github.com/_blog
Fri Nov 20 10:17:53 2015

There are some limitations to this approach. One of them is it assumes the src block is a stand-alone block that will run on its own. That is usually how I run mine, but I could see having other modules that should be tangled out of a file too. I think the script is being run in the current working directory, so it probably will find any local imports it needs.

You don't get any intermediate feedback on this process. It seems to be possible to do that with a different approach that puts some output in a new buffer, e.g. with start-process. But, you still need some clever code like the async model to know when to insert the results back into this buffer. We consider Emacs processes and sentinels next.

2 Emacs process approach with tangling

We can start a process in Emacs, and attach a sentinel function to it that runs after the process completes. Here is an example of that. We still tangle the src-block here.

(defun org-babel-async-execute ()
  (interactive)
  (let* ((current-file (buffer-file-name))
        (uuid (org-id-uuid))
        (temporary-file-directory "./")
        (tempfile (make-temp-file "py-"))
        (pbuffer (format "*%s*" uuid))
        process)

    (org-babel-tangle '(4) tempfile)
    (org-babel-remove-result)

    (save-excursion
      (re-search-forward "#\\+END_SRC")
      (insert (format
               "\n\n#+RESULTS: %s\n: %s"
               (or (org-element-property :name (org-element-context))
                   "")
               uuid)))

    (setq process (start-process
                   uuid
                   pbuffer
                   "python"
                   tempfile))

    (set-process-sentinel
     process
     `(lambda (process event)
        (when (string= "finished\n" event)
          (delete-file ,tempfile)
          (save-window-excursion
            (save-excursion
              (save-restriction
                (with-current-buffer (find-file-noselect ,current-file)
                  (goto-char (point-min))
                  (re-search-forward ,uuid)
                  (beginning-of-line)
                  (kill-line)
                  (insert (mapconcat
                           (lambda (x)
                             (format ": %s" x))
                           (split-string
                            (with-current-buffer ,pbuffer (buffer-string))
                            "\n")
                           "\n")))))))
        (kill-buffer ,pbuffer)))))

org-babel-async-execute

print 'hello world'
import time
time.sleep(10)

import os
print os.getcwd()
print time.asctime()

hello world
/Users/jkitchin/blogofile-jkitchin.github.com/_blog
Fri Nov 20 10:20:01 2015

That works well from what I can see. There are some limitations. I doubt this will work if you use variables in the src block header. Next we consider an approach that does not do the tangling, and that will show us code output as it goes.

3 Emacs process approach with no tangling

As an alternative to tangling to a file, here we just copy the code to a file and then run it. This allows us to use :var in the header to pass data in at run time. At the moment, this code only supports printed output from code blocks, not the value for :results.

(defun org-babel-async-execute:python ()
  "Execute the python src-block at point asynchronously.
:var headers are supported.
:results output is all that is supported for output.

A new window will pop up showing you the output as it appears,
and the output in that window will be put in the RESULTS section
of the code block."
  (interactive)
  (let* ((current-file (buffer-file-name))
         (uuid (org-id-uuid))
         (code (org-element-property :value (org-element-context)))
         (temporary-file-directory ".")
         (tempfile (make-temp-file "py-"))
         (pbuffer (format "*%s*" uuid))
         (varcmds (org-babel-variable-assignments:python
                   (nth 2 (org-babel-get-src-block-info))))
         process)

    ;; get rid of old results, and put a place-holder for the new results to
    ;; come.
    (org-babel-remove-result)

    (save-excursion
      (re-search-forward "#\\+END_SRC")
      (insert (format
               "\n\n#+RESULTS: %s\n: %s"
               (or (org-element-property :name (org-element-context))
                   "")
               uuid)))

    ;; open the results buffer to see the results in.
    (switch-to-buffer-other-window pbuffer)

    ;; Create temp file containing the code.
    (with-temp-file tempfile
      ;; if there are :var headers insert them.
      (dolist (cmd varcmds)
        (insert cmd)
        (insert "\n"))
      (insert code))

    ;; run the code
    (setq process (start-process
                   uuid
                   pbuffer
                   "python"
                   tempfile))

    ;; when the process is done, run this code to put the results in the
    ;; org-mode buffer.
    (set-process-sentinel
     process
     `(lambda (process event)
        (save-window-excursion
          (save-excursion
            (save-restriction
              (with-current-buffer (find-file-noselect ,current-file)
                (goto-char (point-min))
                (re-search-forward ,uuid)
                (beginning-of-line)
                (kill-line)
                (insert
                 (mapconcat
                  (lambda (x)
                    (format ": %s" x))
                  (butlast (split-string
                            (with-current-buffer
                                ,pbuffer
                              (buffer-string))
                            "\n"))
                  "\n"))))))
        ;; delete the results buffer then delete the tempfile.
        ;; finally, delete the process.
        (when (get-buffer ,pbuffer)
          (kill-buffer ,pbuffer)
          (delete-window))
        (delete-file ,tempfile)
        (delete-process process)))))

org-babel-async-execute:python

Let us try it out again.

print 'hello world'
import time
time.sleep(1)

for i in range(5):
    print i

    time.sleep(0.5)


import os
print os.getcwd()
print time.asctime()

print data

raise IOError('No file!')

hello world
0
1
2
3
4
/Users/jkitchin/blogofile-jkitchin.github.com/_blog
Fri Nov 20 19:30:16 2015
[1, 3]
Traceback (most recent call last):
  File "/Users/jkitchin/blogofile-jkitchin.github.com/_blog/py-84344aa1", line 18, in <module>
    raise IOError('No file!')
IOError: No file!

It works fine for this simple example. We get to see the output as the code executes, which is a pleasant change from the usual way of running python blocks. There is some support for some header arguments, notably the :var header. I don't use :results value in Python, so for now only output is supported. We even support Exceptions in the output finally!

Maybe some org-moder's out there can try this and run it through some more rigorous paces?

org-mode source

Org-mode version = 8.2.10

Discuss on Twitter

Functional and display math in technical documents

Posted November 19, 2015 at 06:07 AM | categories: orgmode, emacs | tags:

Updated November 19, 2015 at 02:43 PM

I have been thinking about a way to have functional and readable mathematics in technical documents. It has always bothered me that I have to write a LaTeX version of an equation, and then a separate implementation of the equation in code somewhere. At least twice in my life these separate representations have not agreed!

One solution might be if my functional code could be converted to LaTeX easily. I explore one simple approach to this here. It is somewhat inspired by this work here http://oremacs.com/2015/01/23/eltex/ on writing LaTeX in emacs-lisp, and from my work with org-mode in mixing narrative text, LaTeX and code.

The idea is to use emacs-lisp for the code, so it is functional, but provide an alternative output for the same code for a document conversion. In other words, we accept there is more than one version we need: a functional version for working, and a consumption version for presentation. We will generate the consumption version from the functional version.

I know emacs-lisp is not ideal for mathematics the way we are accustomed to seeing it, but it enables the idea I want to explore here so we will try it.

Here is the simplest example I could come up with for functional math. We can run it ourselves, and verify it is correct.

(+ 1 2 3)

Now, I can change the meaning of this code temporarily, so that it not only evaluates the form, but also represents the equation and result in LaTeX code. If this was incorporated into a preprocessor of the document, we could have a functional version representing our equations, in code form, and a presentation version generated from this version. The code that follows isn't how I would do this is in some production setting; it is only to show that you can temporarily change the meaning of "+". In a production setting, there would just be (+ 1 2 3) in the text, and a preprocessor would find all the sexps in the text, and replace them with the export format using code like this. At least, that is what I am imagining. It might be feasible to do this already with inline org-babel calls and an org-mode export filter, but I didn't try it here. So, here is the proof of concept code.

(cl-flet ((+ (lambda (&rest args)
               (format
                "$%s = %s$"
                (mapconcat #'number-to-string args " + ")
                (eval `(+ ,@args))))))
  (+ 1 2 3))

\(1 + 2 + 3 = 6\)

Here is an example that generates a fraction from a division.

(cl-flet ((/ (lambda (&rest args)
               (format
                "$\\frac{%s}{%s} = %s$"
                (car args)
                (mapconcat 'number-to-string (cdr args) " \\cdot ")
                (eval `(/ ,@args))))))
  (/ 1.0 2.0 3.0))

\(\frac{1.0}{2.0 \cdot 3.0} = 0.16666666666666666\)

As a proof of concept, this idea looks feasible, but this implementation has some limitations. Getting this to a complete workable approach would require a lot of work, basically creating transformation functions for many, many kinds of mathematical functions, and a lot of other kinds of logic. For example, (+ 1 2 (+ 3 4)) would not render correctly with the codes above. It isn't even clear what it should render to. I think you want 1 + 2 + (3 + 4) as the rendered output.

Anyway, it is an interesting idea, one that blurs the lines between code and mathematics. We are so used to the equation representation of mathematics, rather than the code representation that being able to go back and forth seems like a good idea, especially when one is derived from the other.

org-mode source

Org-mode version = 8.2.10

Discuss on Twitter

Clickable telephone numbers in mu4e messages

Posted November 04, 2015 at 12:45 PM | categories: emacs | tags:

We recently updated our university phone system to a VoIP system that uses Cisco Jabber. I am excited about that because finally I can make phone calls from Emacs with a little applescript automation! So, spoiler alert, this post mostly only applies to Macs, unless you know how to automate a Jabber client to make calls. How to make the telephone numbers clickable is general though, and could be used to do other things as well.

(defun cisco-call (phone-number)
  (interactive "sPhone number: ")
  (do-applescript
   (format "tell application \"Cisco Jabber\"
        activate
        tell application \"System Events\" to keystroke \"n\" using {shift down, command down}
        tell application \"System Events\" to keystroke \"%s\"
        tell application \"System Events\" to key code 36 #return
end tell" phone-number)))

cisco-call

I would like to go a step further, and make clickable phone numbers in my Emacs buffers. Let's take a look at some options.

1 org-mode phone link

This is a no-brainer approach. We can define an org-mode link that runs the cisco-call function.

(org-add-link-type
 "phone"
 (lambda (phone-number)
   (cisco-call phone-number)))

lambda

(phone-number)

(cisco-call phone-number)

This makes simple link that just calls the number in the path of the link.

2 Clickable text with button-lock

I have used the button-lock package very often to make clickable text. Here we use it to highlight phone numbers matching a regular expression that seems to match most US numbers. This seems to work great in org-mode buffers.

(require 'rx)

(defvar highlight-phone-numbers nil
 "Button for `highlight-phone-numbers'")

(defun highlight-phone-numbers ()
  "Make phone numbers of the following types clickable:
  (xxx) xxx-xxxx
  xxx.xxx.xxx
  xxxxxxxxxx
  xxx-xxx-xxxx"
  (interactive)
  (let ((inhibit-read-only t))
    (setq highlight-phone-numbers
          (button-lock-set-button
           (rx
            ;; optional () around area code
            (optional "(")
            (= 3 digit)
            (optional ")")
            ;; delimiters
            (or (optional "-")
                (optional ".")
                (optional " "))
            (= 3 digit)
            (or (optional "-")
                (optional ".")
                (optional " "))
            (= 4 digit))
           (lambda ()
             (interactive)
             (cisco-call (get-surrounding-text-with-property 'phone-number)))
           :face '((:background "Darkolivegreen2")
                   (:underline t))
           :help-echo "click to call"
           :keyboard-binding (kbd "RET")
           :additional-property 'phone-number))))

(add-hook 'text-mode 'highlight-phone-numbers)

highlight-phone-numbers

3 Phone numbers in mu4e messages

For some reason, the button-lock package doesn't seem to work in mu4e message buffers, Maybe it is because . The highlight-regexp package does work though, so for these special buffers we use a new approach. We will just put text properties where we want them, and use those properties to make the text clickable.

The messages are in read-only buffers, but we can inhibit that so we can modify the properties. All we need to do is create a little key map as a copy of the existing map, define some keys on it, then search through the buffer adding properties to every phone number we find. I wrote a function that does that, and put that function in a hook to run each time I open a message. Whammo, now I have clickable phone numbers in email! It works pretty well for me.

(defface mu4e-phone-face
  '((t (:foreground "SteelBlue4" :background "Darkolivegreen2" :underline t)))
  "Phone number directive face.")

(defun mu4e-highlight-phone-numbers ()
  "Make phone numbers clickable in mu4e-view buffers."
  (interactive)
  (let ((phone-regex (rx
                      ;; optional () around area code
                      (optional "(")
                      (= 3 digit)
                      (optional ")")
                      ;; delimiters
                      (or (optional "-")
                          (optional ".")
                          (optional " "))
                      (= 3 digit)
                      (or (optional "-")
                          (optional ".")
                          (optional " "))
                      (= 4 digit))))
    (save-excursion
      (let ((inhibit-read-only t))
        (goto-char (point-min))
        (while (re-search-forward phone-regex nil t)
          (let ((map (copy-keymap mu4e-view-mode-map))
                (start (match-beginning 0))
                (end (match-end 0)))

            ;; set file to be clickable to open the source
            (define-key map [mouse-1]
              `(lambda ()
                 (interactive)
                 (cisco-call ,(match-string 0))))

            ;; let letter c also make the call
            (define-key map "c"
               `(lambda ()
                 (interactive)
                 (cisco-call ,(match-string 0))))

            (set-text-properties
             start end
             `(local-map, map
                          face mu4e-phone-face
                          mouse-face highlight
                          help-echo "mouse-1: click to call"))))))))

(add-hook 'mu4e-view-mode-hook 'mu4e-highlight-phone-numbers)

4 Summary

That works pretty well for me overall. The phone number regex is not perfect, e.g. it makes any 10 digit number clickable, and it doesn't recognize international numbers. I am not sure I can call those through the Jabber client anyway. This is purely convenience for me to easily make calls from emails, or other kinds of documents I might read in Emacs.

I don't use phone calls very often, but an interesting thing might be to open a phone log in org-mode, or open the contact that has that phone number to log that you called them, and provide some notes for them. Alternatively, open a new capture for a phone log that could be refiled later.

org-mode source

Org-mode version = 8.2.10

Discuss on Twitter

« Previous Page -- Next Page »