The Kitchin Research Group

I use code blocks in org-mode a lot. I usually code in Python, and in Python I usually write code that prints output which I want to see. So I almost always want the code blocks to return the output, and not the value of the last function. I have set my default header args like this:

org-babel-default-header-args

However, I would prefer that emacs-lisp blocks use value for the results. I know I can get that by putting :results value in the code block header, but that annoys me. I learned today from http://orgmode.org/worg/org-contrib/babel/header-args.html that you can make language specific default headers!

This code in my init file sets emacs-lisp specific default headers:

(setq org-babel-default-header-args:emacs-lisp 
      (cons '(:results . "value")
            (assq-delete-all :results org-babel-default-header-args)))

That way I do not have type :results value at the top of every elisp block. Of course, if I want the output I could specify :results output in the block.

org-babel-default-header-args:emacs-lisp

Problem solved!

On a related note, I find I write so many blocks of python and elisp I added these templates:

;; add <p for python expansion
(add-to-list 'org-structure-template-alist
             '("p" "#+BEGIN_SRC python\n?\n#+END_SRC" "<src lang=\"python\">\n?\n</src>"))

;; add <el for emacs-lisp expansion
(add-to-list 'org-structure-template-alist
             '("el" "#+BEGIN_SRC emacs-lisp\n?\n#+END_SRC" "<src lang=\"emacs-lisp\">\n?\n</src>"))

I probably could have also coded the :results header into those too. They add a tidbit of convenience so I do not have to type python or emacs-lisp after expanding a source block with <s.

Copyright (C) 2014 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.5g

In the last post I explored a way to expand a string template that was more readable than the usual format. Today I look at another approach where I use sexp expansions to accomplish the same thing. The idea is to embed lisp expressions and replace them by what they evaluate to.

In emacs-lisp, if we have a command in a string, we can "read" it, and then eval it.

Here we get the user-full-name:

(eval (read "user-full-name"))

John Kitchin

We can use this on variables too.

(setq some-variable "test")
(eval (read "some-variable"))

test

So, if we use a syntax to identify what to replace, we can substitute in the values. Let us try %() as the syntax.

(defun expand-template (s)
  "expand a template containing %() with the eval of its contents"
  (replace-regexp-in-string "%(\\([^)]+\\))"
                            (lambda (arg)
                              (format "%s" (eval (read (substring arg 2 -1))))) s))


(let ((key "kitchin-2014")
      (author "Kitchin, J. R.")
      (journal "HACS")
      (year "2014")
      (volume "1")
      (pages "1--10")
      (doi "10.1.1.109/hacs.1.10")
      (url "http://hacs.org/10.1.1.109/hacs.1.10")
      (pdf-dir "/home/jkitchin/pdfs")
      (template "
 :PROPERTIES:
  :Custom_ID: %(key)
  :AUTHOR: %(author
  :JOURNAL: %(journal)
  :YEAR: %(year)
  :VOLUME: %(volume)
  :PAGES: %(pages)
  :DOI: %(doi)
  :URL: %(url)
 :END:
[[cite:%(key)]] [[file:%(pdf-dir)/%(key).pdf][pdf]]\n\n"))

(expand-template template))

 :PROPERTIES:
  :Custom_ID: kitchin-2014
  :AUTHOR: Kitchin, J. R.
  :YEAR: 2014
  :VOLUME: 1
  :PAGES: 1--10
  :DOI: 10.1.1.109/hacs.1.10
  :URL: http://hacs.org/10.1.1.109/hacs.1.10
 :END:
[[cite:kitchin-2014]] [[file:/home/jkitchin/pdfs/kitchin-2014.pdf][pdf]]

That is pretty nice. I like it better than the plist expansion I used before. Presumably these variables would already be defined somewhere in your code.

I thought of trying this on a more complex expansion, and discovered a weakness in the regexp that finds the expansion values. It turns out to be simpler to use %{} as the delimiter than %(), because you may want nested parentheses. The regexp above does not correctly match sets of parentheses.

(defun expand-template (s)
  "expand a template containing %{} with the eval of its contents"
  (replace-regexp-in-string "%{\\([^}]+\\)}"
                            (lambda (arg)
                              (let ((sexp (substring arg 2 -1)))
                                (format "%s" (eval (read sexp))))) s))

(expand-template "2 * 2 = %{(* 2 2)}")

2 * 2 = 4

I am not sure this is a desirable way to make a template, with multiline code to be expanded, but at least this works!

(defun expand-template (s)
  "expand a template containing %{} with the eval of its contents"
  (replace-regexp-in-string "%{\\([^}]+\\)}"
                            (lambda (arg)
                              (let ((sexp (substring arg 2 -1)))
                                (format "%s" (eval (read sexp))))) s))

(expand-template "The result is %{(progn
  (if (> 4 3)
      'true
    'false))}")

The result is true

The regexp used in the expansion is not very robust. In particular if there is a } in the code, it will probably fail because the regexp does not match closing } correctly. Fixing that is beyond me right now!

Copyright (C) 2014 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.5g

At one point I had a string I wanted to fill in with a bunch of variables.

(insert (format"
 :PROPERTIES:
  :Custom_ID: %s
  :AUTHOR: %s
  :JOURNAL: %s
  :YEAR: %s
  :VOLUME: %s
  :PAGES: %s
  :DOI: %s
  :URL: %s
 :END:
[[cite:%s]] [[file:%s/%s.pdf][pdf]]\n\n"
key author journal year volume pages doi url key jorg-bib-pdf-directory key ))

I find that very difficult to use, because it is tedious to make sure all the variables are in the right order, and it is difficult to change later. In Python, you would be able to put named expansions in, e.g. {author} and then used named arguments. That does not exist as far as I know in emacs-lisp.

Below is an alternatme approach that uses concat to construct this string.

(let ((key "kitchin-2014")
      (author "Kitchin, J. R.")
      (journal "HACS")
      (year "2014")
      (volume "1")
      (pages "1--10")
      (doi "10.1.1.109/hacs.1.10")
      (url "http://hacs.org/10.1.1.109/hacs.1.10")
      (jorg-bib-pdf-directory "/home/jkitchin/pdfs"))

(concat "
 :PROPERTIES:
  :Custom_ID: " key "
  :AUTHOR: " author "
  :JOURNAL: " journal "
  :YEAR: " year "
  :VOLUME: " volume "
  :PAGES: " pages "
  :DOI: " doi "
  :URL: " url "
 :END:
[[cite:" key "]] [[file:" jorg-bib-pdf-directory "/" key ".pdf][pdf]]\n\n"))

 :PROPERTIES:
  :Custom_ID: kitchin-2014
  :AUTHOR: Kitchin, J. R.
  :JOURNAL: HACS
  :YEAR: 2014
  :VOLUME: 1
  :PAGES: 1--10
  :DOI: 10.1.1.109/hacs.1.10
  :URL: http://hacs.org/10.1.1.109/hacs.1.10
 :END:
[[cite:kitchin-2014]] [[file:/home/jkitchin/pdfs/kitchin-2014.pdf][pdf]]

That is kind of interesting. It is a little tedious to use all the quotes. It seems like there should be soemthing like named expansions. Let us write one of our own. We will use a regular expression to find {:keyword} and a plist. There is a regexp to match this, and then we can take the characters from position 1 to the second to last character as the keyword. That is not beautiful to me, but it works here. Then we just get the keyword from the plist. The keywords in a plist are symbols, and we will have strings. We have to use the intern function to convert them to symbols.

(defun expand-template (s plist)
  "expand a template containing {:keyword} with the definitions in plist"
  (replace-regexp-in-string "{\\(:[^}]+\\)}" 
                            (lambda (arg) 
                              (let ((keyword (intern (substring arg 1 -1))))
                                (format "%s" (plist-get plist keyword)))) s))

(let ((template "
 :PROPERTIES:
  :Custom_ID: {:key}
  :AUTHOR: {:author}
  :JOURNAL: {:journal}
  :YEAR: {:year}
  :VOLUME: {:volume}
  :PAGES: {:pages}
  :DOI: {:doi}
  :URL: {:url}
 :END:
[[cite:{:key}]] [[file:{:pdf-dir}/{:key}.pdf][pdf]]\n\n"))

(expand-template template
                 '(:key "kitchin-2014"
                        :author "Kitchin, J. R."
                        :journal "HACS"
                        :year 2014
                        :volume 1
                        :pages "1--10"
                        :doi "10.1.1.109/hacs.1.10"
                        :url "http://hacs.org/10.1.1.109/hacs.1.10"
                        :pdf-dir "/home/jkitchin/pdfs")))

 :PROPERTIES:
  :Custom_ID: kitchin-2014
  :AUTHOR: Kitchin, J. R.
  :JOURNAL: HACS
  :YEAR: 2014
  :VOLUME: 1
  :PAGES: 1--10
  :DOI: 10.1.1.109/hacs.1.10
  :URL: http://hacs.org/10.1.1.109/hacs.1.10
 :END:
[[cite:kitchin-2014]] [[file:/home/jkitchin/pdfs/kitchin-2014.pdf][pdf]]

That is pretty close to what I am used to from python! I am surprised there aren't other solutions for this around. I looked, and couldn't find them.

Copyright (C) 2014 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.5f

Today I will make a new org-mode link that lets me make links to files inside of Emacs packages. These files may be installed in different places on different systems (e.g. in the system directory, in ELPA directories, or in custom directories), so we need a way to construct paths to them. The application of this is eventually I hope to have some emacs packages of documentation, and I would like to have links between the packages that work no matter how they are installed.

I want a syntax that looks like pkg:rainbow-mode==rainbow-mode-pkg.el. We will have a function that parses that to get the package, and the path to the file in the package. Emacs has a function to find the path to the file that defines a library. I chose == because it seems unlikely that would be a string in a package or path.

(locate-library "rainbow-mode")

c:/Users/jkitchin/Dropbox/kitchingroup/jmax/elpa/rainbow-mode-0.9/rainbow-mode.elc

We can use that to construct the path to where we want. Say we want the file named "rainbow-mode-pkg.el"

(expand-file-name
 "rainbow-mode-pkg.el"
 (file-name-directory (locate-library "rainbow-mode")))

c:/Users/jkitchin/Dropbox/kitchingroup/jmax/elpa/rainbow-mode-0.9/rainbow-mode-pkg.el

In org-mode links, the link path gets passed to a function. We can split the string like this to get the package and relative path we are referring to.

(split-string "rainbow-mode==rainbow-mode-pkg.el" "==")

That is all of the pieces we need to construct the link function. Here it is.

(org-add-link-type 
 "pkg"
 (lambda (path)
   (let ((pkg) (relpath)
         (splitpath (split-string path "==")))
     (setq pkg (car splitpath))
     (setq relpath (nth 1 splitpath))
     (find-file (expand-file-name 
                 relpath 
                 (file-name-directory (locate-library pkg)))))))

pkg:rainbow-mode==rainbow-mode-pkg.el

This works too, but you have to use auctex-pkg as the package name.

pkg:auctex-pkg==doc/intro.texi

I think that is because locate-library looks for the file a library is defined in. That is not quite the same as the root directory of a package. It turns out to be a little more complicated to find that. Below is some code I hacked up looking at the package.el code. First let us examine some pieces.

This gives us information about an installed package.

(assq 'auctex package-alist)

(auctex . [(11 87 2) nil Integrated environment for *TeX*])

We can get the version of the package like this

(package-version-join (package-desc-vers (cdr (assq 'auctex package-alist))))

11.87.2

Ok, finally, we get the directory where it is installed like this:

(package--dir "auctex" "11.87.2")

c:/Users/jkitchin/Dropbox/kitchingroup/jmax/elpa/auctex-11.87.2

Note that in some places we use a package symbol, and in other places a string name.Putting that together, we have this block to get the install-dir of a package. If we have a package symbol we can get the path like this.

(let* ((pkg 'auctex)
       (pkg-name (symbol-name pkg)) ; convert symbol to string
       (desc (cdr (assq pkg package-alist)))
       (version (package-version-join (package-desc-vers desc)))
       (pkg-dir (package--dir pkg-name version)))
  pkg-dir)

c:/Users/jkitchin/Dropbox/kitchingroup/jmax/elpa/auctex-11.87.2

Usually, we will have a string though. We just have to make it a symbol with the intern function.

(setq pkg-name "auctex")
(setq pkg (intern pkg-name))
(setq desc (cdr (assq pkg package-alist)))

[(11 87 2) nil "Integrated environment for *TeX*"]

Now, we have all the pieces to get the path from a package name in a string:

(let* ((pkg-name "auctex")
       (pkg (intern pkg-name))
       (desc (cdr (assq pkg package-alist)))
       (version (package-version-join (package-desc-vers desc)))
       (pkg-dir (package--dir pkg-name version)))
  pkg-dir)

c:/Users/jkitchin/Dropbox/kitchingroup/jmax/elpa/auctex-11.87.2

Let us use that to rewrite the link, and address a few other limitations. We will use org-open-link-from-string so we can use org-link syntax in the path part of the link, e.g. to open a file at a line, or headline. Here is our new link.

(org-add-link-type 
 "pkg2"
 (lambda (path)
   (let ((pkg) (relpath) (pkg-dir) (link-string)
         (splitpath (split-string path "==")))
     (setq pkg-name (car splitpath))
     (setq relpath (nth 1 splitpath))
     (setq pkg-dir (let* ((pkg-symbol (intern pkg-name)) ;convert string to pkg                   
                          (desc (cdr (assq pkg-symbol package-alist)))
                          (version (package-version-join (package-desc-vers desc)))
                          (pkg-dir (package--dir pkg-name version)))
                     pkg-dir))
     (setq link-string (format "[[file:%s/%s]]" pkg-dir relpath))
     (message "link: %s" link-string)
     (org-open-link-from-string link-string))))

Now, we can do all of these: pkg2:auctex==doc/faq.texi pkg2:auctex==doc/faq.texi::should pkg2:auctex==doc/faq.texi::10

pkg2:auctex==doc/faq.texi::first place

Awesome!

Just for fun, I made a toy package called package1 in my elpa directory. That package has an org file in it. Now, I can test out the following links:

pkg2:package1==intro.org

pkg2:package1==intro.org::*Miscellaneous

pkg2:package1==intro.org::*subheading with words

pkg2:package1==intro.org::#install-section

pkg2:package1==intro.org::intro-target

They all work! That works for packages installed via the package manager. However, when I try this with my custom installed org-mode, it does not work. If I run (describe-package 'org) I see that org is a build in package, and that there is an alternate version available. It does not point to my org-installation.

pkg2:org==doc/library-of-babel.org

(princ (locate-library "org"))

c:/Users/jkitchin/Dropbox/kitchingroup/jmax/org-mode/lisp/org.elc

(princ (package-installed-p "org"))

nil

Obviously, we need to check if the package is installed via package.el, or if we should look somewhere else. Let us take a final stab at this. Let us review the challenge.

(print (locate-library "auctex"))
(print (locate-library "auctex-autoloads"))

nil

"c:/Users/jkitchin/Dropbox/kitchingroup/jmax/elpa/auctex-11.87.2/auctex-autoloads.el"

We may have to check for a package-autoloads. Ww can wrap that in an or macro, which will return the first non-nil result.

(let ((pkg-name "auctex"))
   (file-name-directory 
    (or (locate-library pkg-name)
        (locate-library (format "%s-autoloads" pkg-name)))))

c:/Users/jkitchin/Dropbox/kitchingroup/jmax/elpa/auctex-11.87.2/

Doing this on the org package shows that this points to a lisp directory.

(let ((pkg-name "org"))
   (file-name-directory 
    (or (locate-library pkg-name)
        (locate-library (format "%s-autoloads" pkg-name)))))

c:/Users/jkitchin/Dropbox/kitchingroup/jmax/org-mode/lisp/

So, let's try a final link function.

(org-add-link-type 
 "pkg3"
 (lambda (path)
   (let ((pkg-name) (relpath)(pkg-dir) (link-string)
         (splitpath (split-string path "==")))
     (setq pkg-name (car splitpath))
     (setq relpath (nth 1 splitpath))
     (setq pkg-dir (file-name-directory 
                    (or (locate-library pkg-name)
                        (locate-library (format "%s-autoloads" pkg-name)))))
(setq link-string (format "[[file:%s/%s]]" pkg-dir relpath))
     (message "link: %s" link-string)
     (org-open-link-from-string link-string))))

Now, we just have to make sure to use the right relative path. This link opens up an org-file in my installed version of org-mode:

pkg3:org==../doc/library-of-babel.org

I don't know if there is a more clever way to create these links. There are two parts to them: 1) the package, and 2) the relative path. The link syntax isn't that rich to do it without parsing the linkpath.

Copyright (C) 2014 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.5f

I have a need to send a lot of emails to users in my class. I have to send each student an email containing there userid and a password assigned to them. I have a list of these, so the strategy is to create a function that will email that information to one user, and then use mapcar to apply the function to each pair in a list. First, we work out a function that will send one email to one user.

(defun send-mail (userid password)
  "send email to userid@andrew.cmu.edu containing their password"
  (interactive)
  (mail)
  (mail-to)
  (insert (format "%s@andrew.cmu.edu" userid))
  (mail-subject)
  (insert "[06-640] account information")
  (mail-text)
  (insert (format "
An account has been created on gilgamesh.cheme.cmu.edu
userid: %s
password: %s" userid password))
  (mail-send-and-exit))

(send-mail "jkitchin" "trustme99")

That worked well. I ran the block and got the email.

Now, suppose I have this data:

We can pass that to a source block as a list of lists that will look like this:

 ((user1 trustme99) (user2 foolme99) (user3 blameme99))

Then, we can use a mapcar to process each element. Here I use a dummy function with two arguments. If I substitute the function above, each of these users would get an email.

(defun fun (a b)
  (princ (format "user: %s\npassword: %s\n" a b)))

(mapcar (lambda (x) (fun (car x) (cadr x))) data)

user: user1
password: trustme99
user: user2
password: foolme99
user: user3
password: blameme99

I am not sure that is the best way to get the first and second elements in the list element. It looks funny to me, but it works fine. the alternative is not much prettier:

(defun fun (a b)
  (princ (format "user: %s\npassword: %s\n" a b)))

(mapcar (lambda (x) (fun (nth 0 x) (nth 1 x))) data)

user: user1
password: trustme99
user: user2
password: foolme99
user: user3
password: blameme99

Copyright (C) 2014 by John Kitchin. See the License for information about copying.

org-mode source