The Kitchin Research Group

I use code blocks in org-mode a lot. I usually code in Python, and in Python I usually write code that prints output which I want to see. So I almost always want the code blocks to return the output, and not the value of the last function. I have set my default header args like this:

org-babel-default-header-args

However, I would prefer that emacs-lisp blocks use value for the results. I know I can get that by putting :results value in the code block header, but that annoys me. I learned today from http://orgmode.org/worg/org-contrib/babel/header-args.html that you can make language specific default headers!

This code in my init file sets emacs-lisp specific default headers:

(setq org-babel-default-header-args:emacs-lisp 
      (cons '(:results . "value")
            (assq-delete-all :results org-babel-default-header-args)))

That way I do not have type :results value at the top of every elisp block. Of course, if I want the output I could specify :results output in the block.

org-babel-default-header-args:emacs-lisp

Problem solved!

On a related note, I find I write so many blocks of python and elisp I added these templates:

;; add <p for python expansion
(add-to-list 'org-structure-template-alist
             '("p" "#+BEGIN_SRC python\n?\n#+END_SRC" "<src lang=\"python\">\n?\n</src>"))

;; add <el for emacs-lisp expansion
(add-to-list 'org-structure-template-alist
             '("el" "#+BEGIN_SRC emacs-lisp\n?\n#+END_SRC" "<src lang=\"emacs-lisp\">\n?\n</src>"))

I probably could have also coded the :results header into those too. They add a tidbit of convenience so I do not have to type python or emacs-lisp after expanding a source block with <s.

Copyright (C) 2014 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.5g

Today I will make a new org-mode link that lets me make links to files inside of Emacs packages. These files may be installed in different places on different systems (e.g. in the system directory, in ELPA directories, or in custom directories), so we need a way to construct paths to them. The application of this is eventually I hope to have some emacs packages of documentation, and I would like to have links between the packages that work no matter how they are installed.

I want a syntax that looks like pkg:rainbow-mode==rainbow-mode-pkg.el. We will have a function that parses that to get the package, and the path to the file in the package. Emacs has a function to find the path to the file that defines a library. I chose == because it seems unlikely that would be a string in a package or path.

(locate-library "rainbow-mode")

c:/Users/jkitchin/Dropbox/kitchingroup/jmax/elpa/rainbow-mode-0.9/rainbow-mode.elc

We can use that to construct the path to where we want. Say we want the file named "rainbow-mode-pkg.el"

(expand-file-name
 "rainbow-mode-pkg.el"
 (file-name-directory (locate-library "rainbow-mode")))

c:/Users/jkitchin/Dropbox/kitchingroup/jmax/elpa/rainbow-mode-0.9/rainbow-mode-pkg.el

In org-mode links, the link path gets passed to a function. We can split the string like this to get the package and relative path we are referring to.

(split-string "rainbow-mode==rainbow-mode-pkg.el" "==")

That is all of the pieces we need to construct the link function. Here it is.

(org-add-link-type 
 "pkg"
 (lambda (path)
   (let ((pkg) (relpath)
         (splitpath (split-string path "==")))
     (setq pkg (car splitpath))
     (setq relpath (nth 1 splitpath))
     (find-file (expand-file-name 
                 relpath 
                 (file-name-directory (locate-library pkg)))))))

pkg:rainbow-mode==rainbow-mode-pkg.el

This works too, but you have to use auctex-pkg as the package name.

pkg:auctex-pkg==doc/intro.texi

I think that is because locate-library looks for the file a library is defined in. That is not quite the same as the root directory of a package. It turns out to be a little more complicated to find that. Below is some code I hacked up looking at the package.el code. First let us examine some pieces.

This gives us information about an installed package.

(assq 'auctex package-alist)

(auctex . [(11 87 2) nil Integrated environment for *TeX*])

We can get the version of the package like this

(package-version-join (package-desc-vers (cdr (assq 'auctex package-alist))))

11.87.2

Ok, finally, we get the directory where it is installed like this:

(package--dir "auctex" "11.87.2")

c:/Users/jkitchin/Dropbox/kitchingroup/jmax/elpa/auctex-11.87.2

Note that in some places we use a package symbol, and in other places a string name.Putting that together, we have this block to get the install-dir of a package. If we have a package symbol we can get the path like this.

(let* ((pkg 'auctex)
       (pkg-name (symbol-name pkg)) ; convert symbol to string
       (desc (cdr (assq pkg package-alist)))
       (version (package-version-join (package-desc-vers desc)))
       (pkg-dir (package--dir pkg-name version)))
  pkg-dir)

c:/Users/jkitchin/Dropbox/kitchingroup/jmax/elpa/auctex-11.87.2

Usually, we will have a string though. We just have to make it a symbol with the intern function.

(setq pkg-name "auctex")
(setq pkg (intern pkg-name))
(setq desc (cdr (assq pkg package-alist)))

[(11 87 2) nil "Integrated environment for *TeX*"]

Now, we have all the pieces to get the path from a package name in a string:

(let* ((pkg-name "auctex")
       (pkg (intern pkg-name))
       (desc (cdr (assq pkg package-alist)))
       (version (package-version-join (package-desc-vers desc)))
       (pkg-dir (package--dir pkg-name version)))
  pkg-dir)

c:/Users/jkitchin/Dropbox/kitchingroup/jmax/elpa/auctex-11.87.2

Let us use that to rewrite the link, and address a few other limitations. We will use org-open-link-from-string so we can use org-link syntax in the path part of the link, e.g. to open a file at a line, or headline. Here is our new link.

(org-add-link-type 
 "pkg2"
 (lambda (path)
   (let ((pkg) (relpath) (pkg-dir) (link-string)
         (splitpath (split-string path "==")))
     (setq pkg-name (car splitpath))
     (setq relpath (nth 1 splitpath))
     (setq pkg-dir (let* ((pkg-symbol (intern pkg-name)) ;convert string to pkg                   
                          (desc (cdr (assq pkg-symbol package-alist)))
                          (version (package-version-join (package-desc-vers desc)))
                          (pkg-dir (package--dir pkg-name version)))
                     pkg-dir))
     (setq link-string (format "[[file:%s/%s]]" pkg-dir relpath))
     (message "link: %s" link-string)
     (org-open-link-from-string link-string))))

Now, we can do all of these: pkg2:auctex==doc/faq.texi pkg2:auctex==doc/faq.texi::should pkg2:auctex==doc/faq.texi::10

pkg2:auctex==doc/faq.texi::first place

Awesome!

Just for fun, I made a toy package called package1 in my elpa directory. That package has an org file in it. Now, I can test out the following links:

pkg2:package1==intro.org

pkg2:package1==intro.org::*Miscellaneous

pkg2:package1==intro.org::*subheading with words

pkg2:package1==intro.org::#install-section

pkg2:package1==intro.org::intro-target

They all work! That works for packages installed via the package manager. However, when I try this with my custom installed org-mode, it does not work. If I run (describe-package 'org) I see that org is a build in package, and that there is an alternate version available. It does not point to my org-installation.

pkg2:org==doc/library-of-babel.org

(princ (locate-library "org"))

c:/Users/jkitchin/Dropbox/kitchingroup/jmax/org-mode/lisp/org.elc

(princ (package-installed-p "org"))

nil

Obviously, we need to check if the package is installed via package.el, or if we should look somewhere else. Let us take a final stab at this. Let us review the challenge.

(print (locate-library "auctex"))
(print (locate-library "auctex-autoloads"))

nil

"c:/Users/jkitchin/Dropbox/kitchingroup/jmax/elpa/auctex-11.87.2/auctex-autoloads.el"

We may have to check for a package-autoloads. Ww can wrap that in an or macro, which will return the first non-nil result.

(let ((pkg-name "auctex"))
   (file-name-directory 
    (or (locate-library pkg-name)
        (locate-library (format "%s-autoloads" pkg-name)))))

c:/Users/jkitchin/Dropbox/kitchingroup/jmax/elpa/auctex-11.87.2/

Doing this on the org package shows that this points to a lisp directory.

(let ((pkg-name "org"))
   (file-name-directory 
    (or (locate-library pkg-name)
        (locate-library (format "%s-autoloads" pkg-name)))))

c:/Users/jkitchin/Dropbox/kitchingroup/jmax/org-mode/lisp/

So, let's try a final link function.

(org-add-link-type 
 "pkg3"
 (lambda (path)
   (let ((pkg-name) (relpath)(pkg-dir) (link-string)
         (splitpath (split-string path "==")))
     (setq pkg-name (car splitpath))
     (setq relpath (nth 1 splitpath))
     (setq pkg-dir (file-name-directory 
                    (or (locate-library pkg-name)
                        (locate-library (format "%s-autoloads" pkg-name)))))
(setq link-string (format "[[file:%s/%s]]" pkg-dir relpath))
     (message "link: %s" link-string)
     (org-open-link-from-string link-string))))

Now, we just have to make sure to use the right relative path. This link opens up an org-file in my installed version of org-mode:

pkg3:org==../doc/library-of-babel.org

I don't know if there is a more clever way to create these links. There are two parts to them: 1) the package, and 2) the relative path. The link syntax isn't that rich to do it without parsing the linkpath.

Copyright (C) 2014 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.5f

I am going to be assigning homeworks and quizzes in org-mode next semester, and it would be nice if there was a function that would save the file with the right name in the right place. The file will have some keywords at the top to identify the student, course and assignment. Those look like:

#+NAME: John Kitchin
#+ANDREWID: jkitchin
#+COURSE: 06-640
#+ASSIGNMENT: quiz1

I would like a function that saves the current buffer to a file like /some-path/ANDREWID-COURSE-ASSIGNMENT.org. That will be "turning the assignment in" so I can do some analysis of the files. I previously wrote about getting keywords from org-files here . We will build on that to accomplish this goal. Here is the function that seems to do what I want. For now it saves the file in the same directory, with the correct name.

(defun submit-assignment ()
  "Save current buffer to assignment. You must set #+ANDREWID, #+COURSE and #+ASSIGNMENT. There is no checking for this."
  (interactive)
  (save-buffer)
  (let ((org-keywords (org-element-map (org-element-parse-buffer 'element) 'keyword
                        (lambda (keyword) (cons (org-element-property :key keyword)
                                                (org-element-property :value keyword)))))
        (assignment-content (buffer-string)) ; store contents to write to file later
        (assignment-filename))
    ;; create the filename we will save the buffer in
    (setq assignment-filename (concat (cdr (assoc "ANDREWID" org-keywords))
                                      "-"
                                      (cdr (assoc "COURSE" org-keywords))
                                      "-"
                                      (cdr (assoc "ASSIGNMENT" org-keywords))
                                      ".org"))
    ;; now write out contents to filename
    (with-temp-file assignment-filename
      (insert assignment-content))
    (message "%s was submitted" assignment-filename)))

(submit-assignment)

jkitchin-06-640-quiz1.org was submitted

Now, we have a nice function that can be put as a link in the org-file:

elisp:submit-assignment

elisp:submit-assignment

This solution lacks some features I have to think about. For example, if the path where everyone submits to is common, everyone will be able to read the file, which is not desirable. It is also not feasible to make the submitted files read-only to the students this way (so they cannot modify the files after they turn them in). I could have the submit function do something like submit to box.com by email.

I am not sure how grading of these files would go. In the past, I have preferred PDF files (which you can export org to) because I could grade them using a tablet without modifying the actual content in the files. However it would not be simple at all to export an org-file to PDF with that data in a machine readable way. One point of having them in org-mode is that I could collect statistics from the headings of many files pretty easily. I have written code to annotate org-files in the past, which I might have to dig out.

There is a limitation of the submit function, and that is if the org-file references images or other files (I would expect many of my assignments would have plots in them). Those would not be copied to the right place. I handle those when I publish the blog, so it could be done. But, it would require creating a new directory, and writing all the files to it (including making sure the relative paths are handled correctly), or creating some kind of zip-archive that contains everything. That sounds like a project for another day!

Copyright (C) 2014 by John Kitchin. See the License for information about copying.

org-mode source

I have been exploring ways to facilitate learning and grading. One way I like is to provide quizzes inline with text Here are some examples of "interactive" quizzes with feedback to the user in org-mode. The principle idea is to use a headline for the question, and use properties to hold the answer. The answers are coded into links that run lisp code to check if the link text matches the correct answer. The properties are folded, so you have to willfully cheat to look at the answer. For getting instant feedback, this isn't such an issue. These aren't for formal assessment, but for knowledge checkpoints in active learning.

Blog alert: The links may only work in the actual org-file in Emacs, they will not function in html.

Here is an example in a subheading. These links play an amusing sound (crowd cheers for correct answer, a buzzer for a wrong answer). The sounds are stored locally with the file. This gives students immediate feedback on their answer. These might be incorporated into reading assignments.

elisp:(if (string= "4" (org-entry-get (point) "CORRECT-ANSWER"))  (play-sound-file "sounds/99636__tomlija__small-crowd-yelling-yeah.wav" 0.91 nil) (play-sound-file "sounds/fail-buzzer-01.wav" 0.91 nil))

elisp:(if (string= "6" (org-entry-get (point) "CORRECT-ANSWER"))  (play-sound-file "sounds/99636__tomlija__small-crowd-yelling-yeah.wav" 0.91 nil) (play-sound-file "sounds/fail-buzzer-01.wav" 0.91 nil))

elisp:(if (string= "-2" (org-entry-get (point) "CORRECT-ANSWER"))  (play-sound-file "sounds/99636__tomlija__small-crowd-yelling-yeah.wav" 0.91 nil) (play-sound-file "sounds/fail-buzzer-01.wav" 0.91 nil))

(org-add-link-type 
 "ans" 
 (lambda (path) 
   (let* ((correct-answer (org-entry-get (point) "CORRECT-ANSWER"))
          (ncorrect (org-entry-get (point) "NUM-CORRECT"))
          (num-correct (if ncorrect (string-to-number ncorrect) 0))
          (nincorrect (org-entry-get (point) "NUM-INCORRECT"))
          (num-incorrect (if nincorrect (string-to-number nincorrect) 0)))
     (if (string= path correct-answer) 
         (progn
          (org-entry-put (point) "NUM-CORRECT" (number-to-string (+ num-correct 1)))
          (message "correct"))
       (org-entry-put (point) "NUM-INCORRECT" (number-to-string (+ num-incorrect 1)))
       (message "incorrect")))))

ans:yes

ans:no

(org-add-link-type 
 "ans2" 
 (lambda (path) 
   (org-entry-put (point) "ANSWER" path)
   (let* ((nattempts (org-entry-get (point) "NUM-ATTEMPTS"))
          (num-attempts (if nattempts (string-to-number nattempts) 0)))
     (org-entry-put (point) "NUM-ATTEMPTS" (number-to-string (+ num-attempts 1))))
   (message "You selected %s for your answer" path)))

ans2:yes

ans2:no

ans2:yes

ans2:no

ans2:16

ans2:18

ans2:8

Copyright (C) 2014 by John Kitchin. See the License for information about copying.

org-mode source

I have found using tables in an org-file as data sources to code blocks very convenient for documenting work. A typical work flow might go like this:

1. Use a code block to generate some data in an org-table.
2. Use another code block to analyze the data.

For example, here is a code block that prints data in a table ¹:

import numpy as np

print '#+tblname: cos-data'
print '| x | cos(x)|'
print '|-'

for x in np.linspace(0, 2*np.pi, 10):
    print '|{0}|{1}|'.format(x, np.cos(x))

Now, we use that table in a code block to plot the data. We do this by using some header arguments to the code block:

#+BEGIN_SRC python :var data=cos-data

Then we can use the data variable inside the code block like this:

import numpy as np
import matplotlib.pyplot as plt

data = np.array(data) # data is a list coming in
x = data[:, 0]
y = data[:, 1]
plt.plot(x, y)
plt.xlabel('x')
plt.ylabel('cos(x)')
plt.savefig('images/cos-plot.png')

That is pretty awesome, but what if we have data in a table from another org-file? It turns out we can use it too. I have data for the sin(x) stored in a table called sin-data in sin.org , which I now want to use. We can access that table like this in a header arg:

#+BEGIN_SRC python :var data=sin.org:sin-data

And now use the data variable just like before!

import numpy as np
import matplotlib.pyplot as plt

data = np.array(data) # data is a list coming in
x = data[:, 0]
y = data[:, 1]
plt.plot(x, y)
plt.xlabel('x')
plt.ylabel('sin(x)')
plt.savefig('images/sin-plot.png')

This is a powerful capability, as it allows you to pull data from other files into your current analysis. For example, the supporting information files from some of our recent publications have org-files embedded in them with data stored in org-tables. You could use that data in your own analysis without having to type it in yourself. The only thing you need to do is make sure each table in a document is uniquely named.

Special thanks to Eric Schulte for pointing out the syntax for using external tables!

Copyright (C) 2014 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.5c