The Kitchin Research Group

Sometimes I define variables in the header of a code block and then use the code to analyze the data. In org-mode this is super, and you can read the file and easily see what is going on.

When you export the file, however, the information is lost, and in the exported result you cannot see what data went into a code block, or figure out where it is from.

Today we examine how to get that information into exported code. First, we setup a simple example that will do what need.

Now a code block that has a defined variable in the header that uses data from the table defined above.

return data

During export, org-mode does some interesting things to the document, including removing the headers from the code blocks, which makes it impossible to access them inside the export. The headers are apparently removed during org-babel-exp-process-buffer. It does not appear possible to advise this function because it processes the whole buffer at once, and we need to save data for each code block.

So, we will have to preprocess the buffer to get the parameters on each block, and then put the parameters in the export afterwards. For this, we can use a filter. We will preprocess the buffer to get names of tables, and parameters of src-blocks. (I suppose we could put this preprocessing in the advice function, but I tend to avoid advice when possible).

Here is how we can get a list of the table-names indicating their name or that they are results (results are enclosed in ()).

(org-element-map (org-element-parse-buffer) 'table
  (lambda (element)     
    (or (org-element-property :name element) (org-element-property :results element))))

Similarly, here is the list of parameters for each block.

(org-element-map (org-element-parse-buffer) 'src-block
  (lambda (element)     
    (org-element-property :parameters element)))

Now, we combine them with filters to modify the output. First, we preprocess to get each list, and then in the filter, we will pop off each value and insert the data. We will also get the language for each code block, and add that in the export. We use a filter because we are not modified the transcoded text, simply adding some new text in front of it.

(defun ox-mrkup-filter-table (text back-end info)
  (let ((tblname (pop tblnames)))
    (message "tblname is \"%s\"" tblname)
    ; pop does not remove nil from the list, so we do it here.
    (when (null tblname) (setq tblnames (cdr tblnames)))
    (cond
     ((listp tblname)  ; from results
      (concat (format "<br>Results: %s" (car tblname)) text))
     ((null tblname)   ; no name
      text)
     (t ; everything else
      (concat (format "<br>Table name: %s" tblname) text)))))

(defun ox-mrkup-filter-src-block (text back-end info)
  (let ((params (pop src-params))
        (lang (pop src-langs)))
    (when (null params) (setq src-params (cdr src-params)))
    (if params  
        (concat (format "<pre>Language = %s\nParameters = %s</pre>" lang params) text)
      text)))

;; preprocess to get table names, src parameters and languages.
(let ((tblnames (org-element-map (org-element-parse-buffer) 'table
                  (lambda (element)     
                    (or (org-element-property :name element)                    
                        (org-element-property :results element)))))

      (src-params (org-element-map (org-element-parse-buffer) 'src-block
                    (lambda (element)     
                      (org-element-property :parameters element))))

      (src-langs (org-element-map (org-element-parse-buffer) 'src-block
                    (lambda (element)     
                      (org-element-property :language element))))

      ;; register the filters
      (org-export-filter-table-functions '(ox-mrkup-filter-table))
      (org-export-filter-src-block-functions '(ox-mrkup-filter-src-block)))

  ;; and export the result
  (browse-url (org-export-to-file 'html "custom-src-table-export-3.html")))

#<process open custom-src-table-export-3.html>

Here is the resulting html file: custom-src-table-export-3.html which shows the new export behavior. It might not be too difficult to make links between the parameters and the tables, but it would require parsing the :parameters string. For now, this makes it easy enough to read in HTML where the data is coming from (assuming fluency in org-mode header arguments!).

Special thanks to Aaron Ecay, and Charles Berry on the org-mode mailing list for pointing me towards a solution.

Copyright (C) 2014 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.7c

I have found that the names of tables typically get lost when you export an org-file to another format like html or pdf. Since we may use named tables as data sources, it can become unclear in the exported file what has happened, or which table data came from. In this post, we examine how to include the name of a table in exported html. Here are two named tables tbl-1 and tbl-2 that will form the beginning of our effort.

Another table, so we have something to work with later.

Org-buffers get parsed into nested lists, with properties usually in plists. It will be convenient to get a list of the keys for an element, so we can tell what information we have on each element. Some code for this can be found here: http://www.emacswiki.org/emacs/mon-plist-utils.el . Rather than use that recursive approach, here we just loop through the plist and accumulate the keys.

(defun plist-get-keys (plist)
  (interactive)
  (let ((keys))
    (while (car plist)
      (add-to-list 'keys (car plist) t)
      (setq plist (cddr plist)))
    keys))

; example of use
(plist-get-keys '(:a 1 :b 3 :parent '(another plist)))

Now, when we parse a buffer for elements, we get a nested lisp data structure, and the best I can tell is we need the cadr of that list to get to the relevant plist of properties. So, here, we map over the tables, and see what properties are available.

(org-element-map
    (org-element-parse-buffer) 'table
  (lambda (element)  (plist-get-keys (cadr element))))

Depending on when you run the codeblock above (i.e. I ran it at different stages of development of this document, so some tables after this point are shown), you see different results; some of the tables are RESULTS from code blocks with no names, and two tables have a caption.

Let us now map over the tables and see if they have names. We add an unnamed table, and a named table, both with captions.

Here we get the names of the tables. Only three tables have names, and several are unnamed.

(org-element-map
    (org-element-parse-buffer) 'table
  (lambda (element)  (plist-get (cadr element) :name)))

If you think that is a little awkward, I agree. Here is probably a better way to get that information using features in org-mode..

(org-element-map
    (org-element-parse-buffer) 'table
  (lambda (element)  (org-element-property :name element)))

I had thought we could use a filter to add the name to each table. The issue with filtering is that we get the transcoded text directly, and no practical way to get back to the element it came from (at least none I could find). I have previously used filters (e.g. for changing links on export ) for something like this, but it involved parsing the document once, then exporting, and iterating through the results to change the output. I want to do something different here, and fix the issue on the export.

That requires us to derive a new backend for export, with our new function for formatting. This will give us access to the actual table element, and we can use the original transcoding function to get most of the table, and our own code to modify that before it is exported.

Basically, we just want to add an HTML anchor to the table with some text to indicate the table name. With the anchor we can then link to it elsewhere like this:

We just define a function that satisfies the transcoding function signature (element contents info), and if our element has a :name property, we will prepend it onto the usual table output for html. We will go ahead and code in some conditional code for different backends, although for now only handle the html backend.

(defun my-table-format (table contents info)
  (let ((tblname (org-element-property :name table)))    
    (cond
     ((eq (elt (plist-get info :back-end) 2) 'html)  
      (concat
       (when tblname
         (format "<br>TBLNAME: <a name=\"%s\"></a>%s<br>" tblname tblname))
       (org-html-table table contents info))))))

(org-export-define-derived-backend 'my-html 'html
  :translate-alist '((table . my-table-format)))


(browse-url (org-export-to-file 'my-html "custom-src-table-export.html"))

#<process open custom-src-table-export.html>

That seems to do it. You may need to see custom-src-table-export.html to see the newly annotated tables, since they probably do not show up in the blog post.

Copyright (C) 2014 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.7c

There are times where I want an org-file to be partly read-only. For example, there might be instructions that should not be modified. In this post we consider how to implement that. For now, we only want an org-section to be read-only, and we will designate those sections by a tag read_only. Then, the idea is that a hook function would be run when the org-file is loaded, and mark regions of text as read-only before the user can do anything.

In Emacs, you can mark a section of text, and set it to have a property of read-only. So, we can just map over the entries, and any heading that is tagged as read_only can be made read-only!

Here we set the first few characters of this buffer to be read-only.

(add-text-properties 1 8 '(read-only t))

t

Emacs is semi-serious about what read-only means. You cannot even change properties of read-only text, unless you set inhibit-read-only as a variable.

(let ((inhibit-read-only t))
 (remove-text-properties 1 8 '(read-only t)))

t

Now, we can map over the entries in this buffer, and set any heading tagged read_only to actually be that way like this.

(org-map-entries (lambda ()
                   (let* ((element (org-element-at-point))
                          (begin (org-element-property :begin element))
                          (end (org-element-property :end element)))
                     (add-text-properties begin end '(read-only t))))
                 "read_only")

To get this to work when org-mode is turned on, we just wrap it in a function, add the function to a hook, and a function to undo the read-only behavior. I found that if I use the end reported by org-element-at-point, it includes the first character of the next section, we take one away from the end to avoid that.

(defun org-mark-readonly ()
  (interactive)
  (org-map-entries
   (lambda ()
     (let* ((element (org-element-at-point))
            (begin (org-element-property :begin element))
            (end (org-element-property :end element)))
       (add-text-properties begin (- end 1) '(read-only t))))
   "read_only")
 (message "Made readonly!"))


(defun org-remove-readonly ()
  (interactive)
  (org-map-entries
   (lambda ()
     (let* ((element (org-element-at-point))
            (begin (org-element-property :begin element))
            (end (org-element-property :end element))
            (inhibit-read-only t))
         (remove-text-properties begin (- end 1) '(read-only t))))
     "read_only"))

(add-hook 'org-mode-hook 'org-mark-readonly)

That seem to be all there is. After executing the code above, when I open this file, the next section is read-only! I can use the other function to remove that if I need to edit it. Score one for Emacs + org-mode!

Copyright (C) 2014 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.7c

I recently added an org export to jasp which provides an org-mode representation of a Vasp calculation. It is still a work in progress, but the main ideas are developed. The idea is to represent a calculation as an org-mode headline with tabular data, properties and tags. This will make the representation human readable, but machine accessible. Here I examine how we could use it.

from jasp import *
from jasp.utils import get_jasp_dirs

for d in get_jasp_dirs('10'):
    with jasp(d) as calc:
        print calc.org(level=3)

for d in get_jasp_dirs('11'):
    with jasp(d) as calc:
        print calc.org(level=3)

print data.__class__

<type 'str'>

(cons '(label volume energy) 
      (cons 'hline
            (org-map-entries
             (lambda ()
               (list (nth 4 (org-heading-components))
                     (org-entry-get (point) "VOLUME")
                     (org-entry-get (point) "TOTAL_ENERGY"))) 
             ;; this next line specifies headlines tagged with 11
             ;; and that match the regular expression in {}
             "+11+{f_dash_.*}")))

import matplotlib.pyplot as plt
import numpy as np

print E

plt.plot(np.array(V), np.array(E), 'ko')
plt.xlabel('Volume $\AA^3$')
plt.ylabel('Energy (eV)')
plt.savefig('11-eos.png')

['-20.038179', '-19.912894', '-19.662755', '-20.275411', '-20.249632', '-20.202731', '-20.133749']

(save-excursion
  (org-open-link-from-string "[[#10/eos-exp/f+0]]")
  (org-entry-get (point) "STRESS"))

[-0.003191325893994518, -0.003191325893994518, -0.003191325893994518, -0.00042576435655845916, 0.00042576435655845916, 0.00042576435655845916]

import numpy as np
print np.abs(np.array(eval(stress))).max()

0.00319132589399

import numpy as np

print('cell = {}'.format(np.array(data)))
print('Volume = {}'.format(np.linalg.det(np.array(data))))

cell = [[-1.98408  0.      -1.98408]
 [ 0.      -1.98408  1.98408]
 [-3.53625  3.53625  3.53625]]
Volume = 41.7621235495

(save-excursion
  (org-open-link-from-string "[[#10/eos-exp/f+0]]")
  (org-entry-get (point) "VOLUME"))

41.7621235495

(let* ((unique-species '())
      (counts))
  ;; first get unique species
  (mapcar (lambda (s) (add-to-list 'unique-species s)) species)

  ;; now get counts
  (setq counts
        (mapcar (lambda (s)
                  (--count (string= it s) species)) unique-species))

  ;; Now put them together
  (mapconcat 'identity (cl-mapcar (lambda (s n)
                  (format "%s%s" s n)) unique-species counts) ""))

Ni1Al2

(let* ((unique-species '())
      (counts))
  ;; first get unique species
  (mapcar (lambda (s) (add-to-list 'unique-species s)) species)

  ;; now get counts
  (setq counts
        (mapcar (lambda (s)
                  (--count (string= it s) species)) unique-species))

  (mapconcat (lambda (x)
               (format "%s%s" (car x) (cdr x)))
             (cl-sort
              (cl-mapcar (lambda (a b) (cons a b)) unique-species counts)
              'string-lessp :key 'car)
             ""))

Al2Ni1

from collections import Counter
counts = Counter(species)
print ''.join(['{0}{1}'.format(x, counts[x]) for x in counts])

Ni1Al2

from collections import Counter
counts = Counter(species)
print ''.join(['{0}{1}'.format(x, counts[x]) for x in sorted(counts.keys())])

Al2Ni1

Copyright (C) 2014 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.7c

I have been creating homework problems in org-files, which I distribute to students. A recurring dilemma I have had is where do you put the solution? If you put it in a separate file, there is not an elegant way to keep them connected, so if you end up changing the problem, it is easy to forget to update the solution.

Recently I learned about org-crypt. This allows you to encrypt the content of a heading. All you have to do is tag a heading with :crypt:, and when you save the file it will encrypt the contents of the heading. In this example, we try that out. We will use symmetric encryption, which just uses a password. Here is the setup for your init.el file.

(require 'org-crypt)
(org-crypt-use-before-save-magic)
(setq org-tags-exclude-from-inheritance (quote ("crypt")))
;;  set to nil to use symmetric encryption.
(setq org-crypt-key nil)

Now, for the question. What is the Answer to the Ultimate Question of Life, The Universe, and Everything?

The solution is encrypted below. Note, all you have to do is tag the headline with crypt, and when you save the file, you will be prompted for a passphrase and to confirm the passphrase (I used the passphrase 1234). To decrypt the solution, run org-decrypt-entries. You may get a warning that autosave can cause leakage, and a request to disable it for this buffer. You can do that. Then, You will be prompted for a passphrase for symmetric encryption. Enter 1234, and you should be rewarded with the solution!

This could be a good way to keep solutions and problems together, as long as you can remember the passphrase! The passphrase would obviously have to be different for each problem (or one passphrase would open all solutions), and unguessable for this to be useful. Once again, Emacs can help you out. We just need to remember one secret passphrase, and then compute a cryptographic hash that we could use for the encrypting passphrase. We could then just concatenate our secret key onto some unchanging property of the problem that will not change, and generate a secret password. Like this for example.

(let ((my-secret-key "1234"))
  (format "This problem passphrase is %s." (secure-hash 'md5 (concat my-secret-key (org-get-heading t t)))))

This problem passphrase is 68ea5a0eefc31d34ee5d562891e4dcb7.

For this to work, you must keep your secret key absolutely secret, or someone could generate the passwords for each assignment. Also, you must make sure the property of the problem you use never changes, or you will not be able to generate a usable passphrase again! The property could be a filename, or problem label. All that really matters is that it not change, since the solution is stored in encrypted form. It might make sense to write the problem passphrase to a file, as long as the file is not accessible to students, and it is somehow always kept up to date.

Some other notes:

1. org-mode seems to save the passphrase somewhere, so when you save the file after decrypting it, the headline is automatically reencrypted with the same password.
2. If you remove the crypt tag, the file is saved in plain text
3. This seems to rely on GnuPG being installed on your computer.

Copyright (C) 2014 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.7c