The Kitchin Research Group

If you have a lot of src-blocks in your org-file, it might be nice to "tag" them and be able to jump around between them using tag expressions, or by the name of the block, language etc… Here we develop a way to do that and create a handy function to jump to blocks in the current buffer.

First, we look at how to "tag" a src-block. One way is to use a header like this:

#+header: :tags cool idiom two

These are not tags in the usual org-mode sense, they are just a space separated list of words we will later treat as tags. We can get the tags on a src-block with this function.

(defun src-block-tags (src-block)
  "Return tags for SRC-BLOCK (an org element)."
  (let* ((headers (-flatten
                   (mapcar 'org-babel-parse-header-arguments
                           (org-element-property :header src-block))))
         (tags (cdr (assoc :tags headers))))
    (when tags
      (split-string tags))))

src-block-tags

Now, we make a src-block with the tags "test" "one" and "idiom", and see how to tell if the block matches the tag expression "test+idiom".

(let* ((lexical-binding nil)
       (todo-only nil)
       (tags-list (src-block-tags (org-element-context)))
       (tag-expression "test+idiom"))
  (eval (cdr (org-make-tags-matcher tag-expression))))

t

It does, so we wrap that up into a function that tells us if a src-block matches some tag expression.

(defun src-block-match-tag-expression-p (src-block tag-expression)
  "Determine if SRC-BLOCK matches TAG-EXPRESSION."
  (let* ((lexical-binding nil)
         (todo-only nil)
         (tags-list (src-block-tags src-block)))
    (eval (cdr (org-make-tags-matcher tag-expression)))))

src-block-match-tag-expression-p

Here we test that on a block tagged "one three" on the expression "one-two" which means tagged one and not two.

(src-block-match-tag-expression-p (org-element-context) "one-two")

t

Those are the main pieces we need to jump around. We just need a selection tool with a list of filtered candidates. We get a list of src-block candidates to choose from in the next block as an example. Here we get blocks tagged one but not two. We can incorporate this into a selection backend like helm or ivy.

(org-element-map (org-element-parse-buffer) 'src-block
  (lambda (src-block)
    (when (src-block-match-tag-expression-p src-block "one-two")
      ;; Get a string and marker
      (cons
       (format "%15s|%15s|%s"
               (org-element-property :name src-block)
               (org-element-property :language src-block)
               (org-element-property :header src-block))
       (org-element-property :begin src-block)))))

(("    tag-matcher|     emacs-lisp|(:tags test one idiom)" . 1222)
 ("            nil|     emacs-lisp|(:tags one)" . 1641)
 ("            nil|     emacs-lisp|(:tags one three)" . 2120))

Now let us put that into ivy. We will ask for an expression to filter the blocks on, and then use ivy to narrow what is left, and the only action is to jump to the position of the selected block. You can start with a tag expression, or press enter to get all the tags. Then you can use ivy to further narrow by language, block name, or other tags.

(defun ivy-jump-to-src (tag-expression)
  (interactive "sTag expression: ")
  (ivy-read "Select: "
            (org-element-map (org-element-parse-buffer) 'src-block
              (lambda (src-block)
                (when (src-block-match-tag-expression-p src-block tag-expression)
                  ;; Get a string and marker
                  (cons
                   (format "%15s|%15s|%s"
                           (org-element-property :name src-block)
                           (org-element-property :language src-block)
                           (org-element-property :header src-block))
                   (org-element-property :begin src-block)))))
            :require-match t
            :action '(1
                      ("j" (lambda (pos) (interactive) (goto-char pos))))))

ivy-jump-to-src

For fun, here is a python block just for testing.

print(42)

42

That is it! It seems to work ok. There are some variations that might be preferrable, like putting the tags in as params in the src-block header to avoid needing a separate header line. It isn't clear how much I would use this, and it is slow if you have a lot of src blocks in a /large/org-file because of the parsing. (how large? I noticed a notable lag on my 22,800 line org-file this is in ;).

Copyright (C) 2016 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.10

In the last post we examined a molecule link to a src-block defining a molecule in some format. We blurred the distinction between program and data there. Here we re-separate them to try out some different ideas. We will use an org-mode special block to contain the "data" which is a molecular representation in some format. Then, we will use open-babel to convert the format to various other formats to explore using the data.

Here is a methane molecule (with 4 implicit hydrogens in the SMILES format). We put it in a named special block in org-mode, and even put a header on it to indicate the format and a display name!

We can use the SMILES representation block as input to a new command that converts it to the CML format, with coordinates. We use a simple shell command here and pass the contents of the molecule in as a variable. That is nice because in SMILES methane is represented by a single "C", and this CML is much more verbose.

echo $input | obabel -ismi -o cml --gen3d

<?xml version="1.0"?>
<molecule xmlns="http://www.xml-cml.org/schema">
 <atomArray>
  <atom id="a1" elementType="C" x3="1.047517" y3="-0.064442" z3="0.060284"/>
  <atom id="a2" elementType="H" x3="2.139937" y3="-0.064341" z3="0.059898"/>
  <atom id="a3" elementType="H" x3="0.683568" y3="-0.799429" z3="-0.661322"/>
  <atom id="a4" elementType="H" x3="0.683566" y3="0.927794" z3="-0.216100"/>
  <atom id="a5" elementType="H" x3="0.683669" y3="-0.321317" z3="1.056822"/>
 </atomArray>
 <bondArray>
  <bond atomRefs2="a1 a2" order="1"/>
  <bond atomRefs2="a1 a3" order="1"/>
  <bond atomRefs2="a1 a4" order="1"/>
  <bond atomRefs2="a1 a5" order="1"/>
 </bondArray>
</molecule>

We can also use the CML output as input to a command that generates an SVG image, again, passing the CML in via a variable in the header.

echo $cml | obabel -icml -o svg

With our previous molecule link we can refer to these in our text now as methane-smiles and methane-cml.

So far it all looks good. Let us do something new. We will use the SMILES representation to create an ase.atoms object in Python. First, we create an xyz format that ase can read. Rather than clutter up our document with the output, we silence it.

echo $input | obabel -ismi -o xyz --gen3d

Now, we can use the string generated in a Python file to generate a tempfile (or you could have saved the result above to a file and just read it in here). I was too lazy to make the file link to the image myself, so I setup a :file header and just print the result to stdout in this block. Although all we do here is create a new image, this demonstrates you can use data from a MOLECULE block and pass it into a Python script where other kinds of calculations might occur.

from ase.io import read, write

from tempfile import mkstemp
fd, fname = mkstemp(suffix=".xyz")
with open(fname, 'w') as f:
    f.write(xyz)

atoms = read(fname)
write('-', atoms, format="png")

The last point to discuss is discoverability. It would be helpful if we could use a program to "extract" molecular information about the molecules we use in our work. Here is a block that will map over the MOLECULE blocks and summarize what is found with a common format (SMILES again). We generate a table of clickable links to each molecule found in the documents. There is a small appendix in this document containing h2o and caffeine that will show in this table.

(defun mlc-to-smiles (blk)
  "Convert a molecule BLK to smiles format using openbabel."
  (let* ((headers (-flatten
                   (mapcar 'org-babel-parse-header-arguments
                           (org-element-property :header blk))))
         (format (cdr (assoc :format headers)))
         (content (buffer-substring-no-properties
                   (org-element-property :contents-begin blk)
                   (org-element-property :contents-end blk)))
         (tempfile (make-temp-file "obabel-")))
    (with-temp-file tempfile
      (insert content))

    ;; convert to smiles. This outputs a smiles string and the file it was
    ;; generated from. I don't know how to suppress the file, so we use awk to
    ;; just get the SMILEs strings. It is not pretty. I know.
    (prog1
        (s-trim (shell-command-to-string
                 (format  "obabel %s %s -osmi 2> /dev/null | awk '{print $1}'"
                          (format "-i%s" format) tempfile)))
      (delete-file tempfile))))


;; Generate the table of molecules
(append '(("Display name" "Name" "format" "SMILES representation"))
        '(hline)
        (org-element-map (org-element-parse-buffer) 'special-block
          (lambda (sb)
            (when (string= "MOLECULE" (org-element-property :type sb))
              (let ((headers (-flatten
                              (mapcar 'org-babel-parse-header-arguments
                                      (org-element-property :header sb)))))

                (list
                 (format "[[molecule:%s][%s]]" (org-element-property :name sb)
                         (cdr (assoc :display-name headers)))
                 (org-element-property :name sb)
                 (cdr (assoc :format headers))
                 (mlc-to-smiles sb)))))))

That seems pretty discoverable to me. We not only can discover the molecules in this post, but can pretty easily convert them to other formats (SMILES) in this case. Since we can run any code we want on them, we could just as well import them to a database, or do subsequent calculations on them.

The MOLECULE block is not standard, and I have only demonstrated here that it is suitable for this purpose. But, it looks like we could extend it and deal with a variety of formats. We can use headers to add metadata, format, etc… Some features I find missing are similar to those in code blocks where we can type C-c ' to edit them in special modes, and the nice syntax highlighting that often comes with that.

It might be helpful to make the export of MOLECULE blocks nicer looking and more functional. The default export, for example doesn't put an id attribute in the block. First, we rewrite an org-function to add the id attribute to the exported blocks so our molecule links will work.

(defun org-html-special-block (special-block contents info)
  "Transcode a SPECIAL-BLOCK element from Org to HTML.
CONTENTS holds the contents of the block.  INFO is a plist
holding contextual information."
  (let* ((block-type (downcase
                      (org-element-property :type special-block)))
         (contents (or contents ""))
         (html5-fancy (and (org-html-html5-p info)
                           (plist-get info :html-html5-fancy)
                           (member block-type org-html-html5-elements)))
         (attributes (org-export-read-attribute :attr_html special-block)))
    (unless html5-fancy
      (let ((class (plist-get attributes :class)))
        (setq attributes (plist-put attributes :class
                                    (if class (concat class " " block-type)
                                      block-type)))
        (when (org-element-property :name special-block)
          (setq attributes (plist-put
                            attributes :id
                            (org-element-property :name special-block))))))
    (setq attributes (org-html--make-attribute-string attributes))
    (when (not (equal attributes ""))
      (setq attributes (concat " " attributes)))
    (if html5-fancy
        (format "<%s%s>\n%s</%s>" block-type attributes
                contents block-type)
      (format "<div%s>\n%s\n</div>" attributes contents))))

org-html-special-block

It would be nice to add some additional information around the block, e.g. that it is a molecule, maybe some tooltip about the format, etc…, but we leave that to another day. These should probably be handled specially with a dedicated export function. You will note that MOLECULE blocks don't export too well, they should probably be wrapped in <pre> for HTML export. We will at least make them stand out with this bit of css magic.

#+HTML_HEAD_EXTRA:  <style>.molecule {background-color:LightSkyBlue;}</style>

Copyright (C) 2016 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.10

Here I am exploring some ideas on compact and functional representations of molecules in org-mode. We will use some functionality from OpenBabel (https://openbabel.org/docs/dev/index.html ) for conversion of formats.

One approach we could use is the SMILES representation. OpenBabel provides tools to convert SMILES to a visualization like this. Let's check out an old favorite: caffeine.

obabel -:"Cn1cnc2n(C)c(=O)n(C)c(=O)c12" -osvg

We can imagine the SMILES string is a program, and use an org-mode src block to contain it. It isn't quite a program, as it is more like data, but we can make the block executable if we define how to "execute" the block, and for that we will just have obabel generate the svg representation of the molecule. Here is our execute function. It simply generates the svg to stdout. We can use a :file header to capture it in a file.

(defun org-babel-execute:smiles (body params)
  (shell-command-to-string
   (format "obabel -:\"%s\" -osvg 2> /dev/null" body)))

org-babel-execute:smiles

You can find a smiles block in Appendix of molecules that was adapted from here .

Now, we need a link to refer to our molecule. We want the follow action to jump to our src block which should have a name. We will have it export as the name of the block linked to the molecule definition. This should work fine for definitions in the document. It is not robust to link to molecules in other org-files in the export. That would require those files to be exported too. For now we just define an HTML export.

(defun molecule-jump (name)
  (org-mark-ring-push)
  (org-open-link-from-string (format "[[%s]]" path)))

(defun molecule-export (path desc backend)
  (let ((name (save-window-excursion
                (molecule-jump path)
                (org-element-property :name (org-element-context)))))
    (cond
     ((eq 'html backend)
      (format "<a href=\"#%s\">%s</a>" name name)))))

(org-add-link-type
 "molecule"
 'molecule-jump
 'molecule-export)

Now we link to LSD and ethanol that allows us to navigate to the definition. We can also refer to a molecule in another file like ethanol. The links are clickable, and should jump to the molecule definition. On export to HTML they will be links to the definition.

Our link provides some limited functionality. We can provide more by making the follow action open a menu for example. Instead, we created a major mode here. It provides a function to convert smiles to CML. It is readily extensible to do other conversions.

One of the reasons we want to have molecules as "data" is so we can find them in our papers. Here is an example of that. We defined two molecules in the Appendix, and we find them here.

(org-element-map (org-element-parse-buffer)
    'src-block
  (lambda (src)
    (when (string= "smiles" (org-element-property :language src))
      (org-element-property :name src))))

There is still a lot to do to make this really functional. For example, we might want to use the molecules to write reactions. We might also want more advanced conversion or lookup functions, and more export options. It might be desirable to have tooltips on the links to see the molecules too. No doubt one might want to fine-tune the way the blocks run, so that options could be passed as header args. Maybe I will work on that another day.

CCN(CC)C(=O)[C@H]1CN(C)[C@@H]2Cc3c[nH]c4cccc(C2=C1)c34

CCO

(require 'easymenu)

(defun smiles-cml ()
  "Convert the smiles string in the buffer to CML."
  (interactive)
  (let ((smiles (buffer-string)))
    (switch-to-buffer (get-buffer-create "SMILES-CML"))
    (erase-buffer)
    (insert
     (shell-command-to-string
      (format "obabel -:\"%s\" -ocml 2> /dev/null"
              smiles)))
    (goto-char (point-min))
    (xml-mode)))

(defun smiles-names ()
  (interactive)
  (browse-url
   (format "http://cactus.nci.nih.gov/chemical/structure/%s/names"
           (buffer-string))))

(defvar smiles-mode-map
  nil
  "Keymap for smiles-mode.")

;; adapted from http://ergoemacs.org/emacs/elisp_menu_for_major_mode.html
(define-derived-mode smiles-mode fundamental-mode "smiles-mode"
  "Major mode for SMILES code."
  (setq buffer-invisibility-spec '(t)
        mode-name " ☺")

  (when (not smiles-mode-map)
    (setq smiles-mode-map (make-sparse-keymap)))
  (define-key smiles-mode-map (kbd "C-c C-c") 'smiles-cml)
  (define-key smiles-mode-map (kbd "C-c C-n") 'smiles-names)

  (define-key smiles-mode-map [menu-bar] (make-sparse-keymap))

  (let ((menuMap (make-sparse-keymap "SMILES")))
    (define-key smiles-mode-map [menu-bar smiles] (cons "SMILES" menuMap))

    (define-key menuMap [cml]
      '("CML" . smiles-cml))
    (define-key menuMap [names]
      '("Names" . smiles-names))))

smiles-mode

Copyright (C) 2016 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.10

It has always bothered me a little that I have to add a file image after code blocks in org-mode to see the results. That extra work… I also don't like having to explicitly print the figure in the code, since that is the extra work, just in a different place. Today I look into two approaches to this. First, we consider something like tooltips, and second just putting overlays of image files right on the file name. The plus side of this is no extra work. The downside is they won't export; that will still take the extra work, but you needed that for the caption anyway for now.

Here is a video illustrating the code in this post: https://www.youtube.com/watch?v=VuAnwCERM0U

Here is a test.

import matplotlib.pyplot as plt
plt.plot([0, 1, 2, 4, 16])
plt.savefig("test-fig.png")

(defvar image-tooltip-re (concat  "\\(?3:'\\|\"\\)\\(?1:.*\\."
                                  (regexp-opt '("png" "PNG" "JPG" "jpeg"
                                                "jpg" "JPEG" "eps" "EPS"))
                                  "\\)\\(?:\\3\\)")
  "Regexp to match image filenames in quotes")

(defun image-tooltip (window object position)
  (save-excursion
    (goto-char position)
    (let (beg end imgfile img s)
      (while (not (looking-at image-tooltip-re))
        (forward-char -1))
      (setq imgfile (match-string-no-properties 1))
      (when (file-exists-p imgfile)
        (setq img (create-image (expand-file-name imgfile)
                                'imagemagick nil :width 200))
        (propertize "Look in the minibuffer"
                    'display img)))))

(font-lock-add-keywords
 nil
 `((,image-tooltip-re
    0 '(face font-lock-keyword-face
             help-echo image-tooltip))))

(font-lock-fontify-buffer)

(defun next-image-overlay (&optional limit)
  (when (re-search-forward image-tooltip-re limit t)
    (setq beg (match-beginning 0)
          end (match-end 0)
          imgfile (match-string 1))
    (when (file-exists-p imgfile)
      (setq img (create-image (expand-file-name imgfile)
                              'imagemagick nil :width 300))
      (setq ov (make-overlay beg end))
      (overlay-put ov 'display img)
      (overlay-put ov 'face 'default)
      (overlay-put ov 'org-image-overlay t)
      (overlay-put ov 'modification-hooks
                   (list 'org-display-inline-remove-overlay)))))

(font-lock-add-keywords
 nil
 '((next-image-overlay (0  'font-lock-keyword-face t)))
 t)

import matplotlib.pyplot as plt
plt.plot([-0, 1, 2, 4, 16])
plt.savefig("test-fig.png")

(ov-clear 'org-image-overlay)

(font-lock-remove-keywords
 nil
 '((next-image-overlay (0  'font-lock-keyword-face t))))

(ov-clear 'org-image-overlay)

Copyright (C) 2016 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.10

I often want to rearrange org headlines. There are super convenient shortcuts for some things, like Alt-up/down to move up and down, and Alt-left/right to change levels, and shift variants of that when there are nested headlines. There is also refile for some use cases. The case these don't handle for me is when I have a headline at the bottom and I want to move it a distance. To cut down on key strokes, I usually kill the headline, jump to where I want, and paste it.

In lispy, there is a teleport command for this! I am going to adapt this here for reorganizing org-headlines. This version allows you to move headlines around on the visible area. You need another strategy for the candidates to jump to if you want to move a headline off the screen. Note though that ivy is really smart, you can have one file in a split view and it will jump to any headline in any window! I use ivy for the navigation here, but you could also use helm to select headlines, for example. This function puts your headline after the selected headline, unless you use a prefix arg, and then it goes before.

Check out the video to see this in action: https://www.youtube.com/watch?v=Vv-2888rpyc

Note: this code is a little more advanced than what is in the video; I added a numeric prefix to change the headline level.

(defun org-teleport (&optional arg)
  "Teleport the current heading to after a headline selected with avy.
With a prefix ARG move the headline to before the selected
headline. With a numeric prefix, set the headline level. If ARG
is positive, move after, and if negative, move before."
  (interactive "P")
  ;; Kill current headline
  (org-mark-subtree)
  (kill-region (region-beginning) (region-end))
  ;; Jump to a visible headline
  (avy-with avy-goto-line (avy--generic-jump "^\\*+" nil avy-style))
  (cond
   ;; Move before  and change headline level
   ((and (numberp arg) (> 0 arg))
    (save-excursion
      (yank))
    ;; arg is what we want, second is what we have
    ;; if n is positive, we need to demote (increase level)
    (let ((n (- (abs arg) (car (org-heading-components)))))
      (cl-loop for i from 1 to (abs n)
               do
               (if (> 0 n)
                   (org-promote-subtree)
                 (org-demote-subtree)))))
   ;; Move after and change level
   ((and (numberp arg) (< 0 arg))
    (org-mark-subtree)
    (goto-char (region-end))
    (when (eobp) (insert "\n"))
    (save-excursion
      (yank))
    ;; n is what we want and second is what we have
    ;; if n is positive, we need to demote
    (let ((n (- (abs arg) (car (org-heading-components)))))
      (cl-loop for i from 1 to (abs n)
               do
               (if (> 0 n) (org-promote-subtree)
                 (org-demote-subtree)))))

   ;; move to before selection
   ((equal arg '(4))
    (save-excursion
      (yank)))
   ;; move to after selection
   (t
    (org-mark-subtree)
    (goto-char (region-end))
    (when (eobp) (insert "\n"))
    (save-excursion
      (yank))))
  (outline-hide-leaves))

org-teleport

Now we add some new speed commands to help us out. I think we should be able to mark subtrees ("@" is bound to this, but I like "m" better) and kill them with a key stroke, in addition to teleporting them. Since we figured out the nice way to jump to a headline up there, we bind that to "q" which isn't used so far, and maps to a similar concept in lispy. The lowercase "t" is already bound to changing the TODO state, so we use capital "T" for the speed key to teleport. Note it is possible to "compose" the same effect by typing "k" to kill a headline, then "q" to jump somewhere else (or navigate where you want, and then "C-y" to paste it at the new location. Or, "T".

(add-to-list 'org-speed-commands-user (cons "m" 'org-mark-subtree))
(add-to-list 'org-speed-commands-user (cons "k" (lambda ()
                                                  (org-mark-subtree)
                                                  (kill-region
                                                   (region-beginning)
                                                   (region-end)))))
(add-to-list 'org-speed-commands-user
             (cons "q" (lambda ()
                         (avy-with avy-goto-line
                           (avy--generic-jump "^\\*+" nil avy-style)))))

(add-to-list 'org-speed-commands-user (cons "T" 'org-teleport))
"done"

done

Copyright (C) 2016 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.10