The Kitchin Research Group

A new and improved Emacs gnuplot DSL

Fri, 05 May 2017 10:26:00 EDT

A significant limitation of the previous DSL I wrote is that all the plotting commands have to go in one macro. It would be nice to accumulate them in different forms, and when you want to run them all. A classic way to do that in Emacs lisp is to make a global variable, e.g. *gnuplot-cmds* and append commands to it. Then when you want to, run the commands.

A more modern approach is to use a closure to encapsulate the commands. Here is a "let over lambda" that defines a few functions that encapsulate an enclosed variable gnuplot-commands. We define one function to add commands to the list of commands, one to clear the commands, one to generate the gnuplot script as a string, and one to run the program. The enclosed variable gnuplot-commands is basically only accessible by these functions. It is encapsulated, similar to if we defined a class in Python then made an instance of it with an attribute that was accessible only be instance methods. On one hand, this "protects" the variable, and keeps it out of the global namespace. On the other hand, we lose the documentation that would have come with a defvar, and we have to define a function to access the contents of that variable.

(let ((gnuplot-commands '("set terminal qt")))

  (defun gnuplot-add-cmd (s)
    "Append the command S to gnuplot-cmds."
    (setq gnuplot-commands (append gnuplot-commands (list s))))

  (defun gnuplot-clear ()
    (setq gnuplot-commands '("set terminal qt")))

  (defun gnuplot-script ()
    (s-join "\n" gnuplot-commands)))

gnuplot-script

To run the commands, we define this function. It does not need to be in the closure because it only accesses the commands through functions we defined in the closure.

(defun gnuplot-show ()
    (let* ((temporary-file-directory ".")
           (cmdfile (make-temp-file "gnuplot-cmds-" nil ".gpl"))
           (shellcmd (format "gnuplot --persist -c \"%s\"" cmdfile))
           (cmds (gnuplot-script)))
      (with-temp-file cmdfile
        (insert cmds))
      (shell-command shellcmd)
      (delete-file cmdfile)
      cmds))

gnuplot-show

Last time I noted I had a new idea for the DSL syntax that would give us more flexibility to inject variables and code into the DSL. The idea is to use keywords, symbols that start with :, to indicate they should be replaced by the value of the non-keyword symbol in the environment, and for any form that starts with : to evaluate that form. So, (: - 5 4) would get replaced by 1. Here is the new macro for that.

(defmacro kargs (&rest args)
  "Convert symbols to strings, quote strings, and (expr) to what they evaluate to."
  `(s-join " " (list ,@(cl-mapcan
                        (lambda (s)
                          (list
                           (cond
                            ((keywordp s)
                             (format "%s"
                                     (symbol-value (intern (substring (symbol-name s) 1)))))
                            ((symbolp s)
                             (symbol-name s))
                            ((stringp s)
                             (format "\"%s\"" s))
                            ((and (listp s) (eq : (car s)))
                             `(with-output-to-string
                                (princ ,(cdr s))))
                            (t
                             (format "%s" s)))))
                        args))))

kargs

Now, our gnuplot macro is simpler, since all it does is add commands to the list. If the form is a string, we add it as is, if the form starts with (: stuff) we evaluate the cdr of the form, and otherwise, we pass the form contents to the kargs macro for processing.

(defmacro gnuplot (&rest forms)
  `(loop for s in (list ,@(mapcar (lambda (x)
                                    (cond
                                     ((stringp x)
                                      x)
                                     ((and (listp x) (eq : (car x)))
                                      `,(cdr x))
                                     (t
                                      `(kargs ,@x))))
                                  forms))
         do (gnuplot-add-cmd s)))

gnuplot

What did that gain us? First, we can break up a script so we can talk about it, maybe do some calculations, etc… Let's look at the example at http://gnuplot.sourceforge.net/demo/linkedaxes.html.

We can start with the basic settings.

(gnuplot-clear)

(gnuplot
 (set terminal png)
 (set output "linkedaxes.png")
 (set encoding utf8)
 (set key outside Left)
 (set bmargin 5)
 (set tmargin 6)
 (set style data lines)
 (set tics in)
 (set ticslevel 0.5)
 (set xlabel  "X-ray energy in eV")

 (set format y  \'%5.1fe\')
 (set title " Anomalous scattering factors ")
 (set xrange  [9000:14400])
 (set offset 0\,0\,1.0\,0)
 (set xtics nomirror)
 (set link x via 12398./x inverse 12398./x)

 (set x2label  "X-ray wavelength in Å")
 (set x2tics 0.1  format "%.1f Å" nomirror))

We need to download some data files. We can do that, and add another line to the gnuplot script. The escaping on the quotes and commas is especially tedious in this one ;) but, we don't need those pesky line-continuations here.

(shell-command "wget http://skuld.bmsc.washington.edu/scatter/data/Br.dat")
(shell-command "wget http://skuld.bmsc.washington.edu/scatter/data/Ta.dat")


(gnuplot
 (plot "Br.dat" volatile using 1:3 title \'Br f\"\'  lt 1 lw 3\, \'\' volatile using 1:2 title "Br f'"  lt 1 lw 1\,
       "Ta.dat" volatile using 1:3 title \'Ta f\"\' lt 2 lw 3\, \'\' volatile using 1:2 title \"Ta f\'\"  lt 2 lw 1))

(gnuplot-script)

set terminal qt
set terminal png
set output "linkedaxes.png"
set encoding utf8
set key outside Left
set bmargin 5
set tmargin 6
set style data lines
set tics in
set ticslevel 0.5
set xlabel "X-ray energy in eV"
set format y '%5.1fe'
set title " Anomalous scattering factors "
set xrange [9000:14400]
set offset 0,0,1.0,0
set xtics nomirror
set link x via 12398./x inverse 12398./x
set x2label "X-ray wavelength in Å"
set x2tics 0.1 format "%.1f Å" nomirror
plot "Br.dat" volatile using 1:3 title 'Br f"' lt 1 lw 3, '' volatile using 1:2 title "Br f'" lt 1 lw 1, "Ta.dat" volatile using 1:3 title 'Ta f"' lt 2 lw 3, '' volatile using 1:2 title "Ta f'" lt 2 lw 1

Finally, we can set the output to png, and run our program.

(gnuplot-show)

Looks good.

What about the fancy keyword formatting? Here is an example of that in action. :term gets replaced by the term variable, :png gets replaced by the filename, and :x is replaced by 4.

(gnuplot-clear)
(let ((x 4)
      (term "png")
      (png "\"polar.png\""))
  (gnuplot
   (set terminal :term)
   (set output :png)
   (set polar)
   (set dummy t)
   (plot sin\( :x *t\) \,cos\( :x *t\))
   (set offset 0\,0\,0\,0)))

(gnuplot-show)

set terminal qt
set terminal png
set output "polar.png"
set polar
set dummy t
plot sin( 4 *t) ,cos( 4 *t)
set offset 0,0,0,0

There are a few nuances I didn't expect. First, you have to escape the parentheses in this case because otherwise it looks like a form that will be ignored. Second, you have to quote the string to get quotes into the gnuplot script. Third, there has to be a space before and after the keywords for emacs to parse it correctly and do the substitution.

Let's look at one last example that uses the (: form). We reproduce a figure from http://gnuplot.sourceforge.net/demo/transparent_solids.html here.

(gnuplot-clear)
(gnuplot
 (set terminal pngcairo  background "#ffffff" enhanced font "arial,9" fontscale 1.0 size 512\, 384 )
 (set output "transparent-solids.png")
 ;; construct the title
 (set title (: format "\"%s\"" (concat "Interlocking Tori - PM3D surface" "with depth sorting and transparency")))

 ;; use lisp code to create a gnuplot command
 (: concat "unset" " " "border")

 (unset key)
 (set object 1 rect from screen 0\, 0\, 0 to screen 1\, 1\, 0 behind)
 (set object 1 rect fc  rgb \"gray\"  fillstyle solid 1.0  border -1)
 (set view 64\, 345\, 1.24375\, 0.995902)
 (set isosamples 50\, 20)
 (unset xtics)
 (unset ytics)
 (unset ztics)
 (set dummy u\,v)
 (set parametric)
 (set urange [ -pi : pi ])
 (set vrange [ -pi : pi ])

 (set style fill  transparent solid 0.30 border)
 (set pm3d depthorder)
 (set palette rgbformulae 8\, 9\, 7)
 (set pm3d interpolate 1\,1 flush begin noftriangles border lt black linewidth 0.500 dashtype solid corners2color mean)
 (set colorbox vertical origin screen 0.9\, 0.2\, 0 size screen 0.05\, 0.6\, 0 front  noinvert bdefault)

 (splot (: concat "cos(u)+.5*cos(u)*cos(v),sin(u)+.5*sin(u)*cos(v),.5*sin(v) with pm3d,"
           "1+cos(u)+.5*cos(u)*cos(v),.5*sin(v),sin(u)+.5*sin(u)*cos(v) with pm3d")))
(gnuplot-show)

set terminal qt
set terminal pngcairo background "#ffffff" enhanced font "arial,9" fontscale 1.0 size 512, 384
set output "transparent-solids.png"
set title "Interlocking Tori - PM3D surfacewith depth sorting and transparency"
unset border
unset key
set object 1 rect from screen 0, 0, 0 to screen 1, 1, 0 behind
set object 1 rect fc rgb "gray" fillstyle solid 1.0 border -1
set view 64, 345, 1.24375, 0.995902
set isosamples 50, 20
unset xtics
unset ytics
unset ztics
set dummy u,v
set parametric
set urange [-pi : pi]
set vrange [-pi : pi]
set style fill transparent solid 0.3 border
set pm3d depthorder
set palette rgbformulae 8, 9, 7
set pm3d interpolate 1,1 flush begin noftriangles border lt black linewidth 0.5 dashtype solid corners2color mean
set colorbox vertical origin screen 0.9, 0.2, 0 size screen 0.05, 0.6, 0 front noinvert bdefault
splot cos(u)+.5*cos(u)*cos(v),sin(u)+.5*sin(u)*cos(v),.5*sin(v) with pm3d,1+cos(u)+.5*cos(u)*cos(v),.5*sin(v),sin(u)+.5*sin(u)*cos(v) with pm3d

Overall this seems like an improvement to the DSL. I didn't invent the idea of reusing keywords this way out of the blue. In On Lisp, Paul graham uses "special" variable names in Chapter 18, where he shows how to use gensyms for special purposes, and also variables with special names like ?x. Even Emacs is using a variation of this idea. Check out this new let-alist macro:

(let-alist '((x . 5))
  (+ 1 .x))

There is a special variable inside the body that is a dot-name. The macro expands to provide a value for that symbol. I wonder if I should have tried to use an approach like this instead. Maybe another day. After I read and study the four defuns and single defmacro that make this possible!

You can see here what happens:

(macroexpand '(let-alist '((x . 5))
  (+ 1 .x)))

(let
    ((alist
      '((x . 5))))
  (let ((\.x (cdr (assq 'x alist))))
    (+ 1 \.x)))

The macro builds up an internal alist for the dot-names.

org-mode source

Org-mode version = 9.1.6

An emacs-lisp dsl for gnuplot

Thu, 04 May 2017 19:33:55 EDT

1. Embedding Python or gnuplot
2. An alternative approach using a DSL
3. Summary

Plotting is a pretty general feature we need in scientific work. In this post we examine a way we could get at least minimal plotting into Emacs-lisp with as lispy a syntax as reasonable.

1 Embedding Python or gnuplot

With org-mode we can fluidly integrate many languages in one document. That is not the goal here, where I want to integrate plotting into a program. You certainly could go this route to embed python programs in your lisp programs for plotting.

(defun python (code)
  (let* ((temporary-file-directory ".")
        (tmpfile (make-temp-file "py-" nil ".py")))
    (with-temp-file tmpfile
      (insert code))
    (shell-command-to-string (format "python %s" tmpfile))))

Here is that function in action.

(python "import matplotlib.pyplot as plt
import numpy as np
x = np.linspace(0, 1)
y = np.exp(x)
plt.plot(x, y, label='data')
plt.title('A Title')
plt.xlim([0, 1])
plt.ylim([1, 2.75])
plt.xlabel('x')
plt.ylabel('y')
plt.legend()
plt.savefig('figpy.png')")

And the corresponding figure:

This is irritating for a few reasons. One is it is annoying to write python programs in string form; you don't get much editor support for indentation or syntax highlighting, and you have to be careful with quotes. It is not that easy to switch that snippet to Python mode either. You are pretty limited in writing programs that expand and modify the code too. Basically you have to do that all by string manipulation.

Along these lines, you could imagine a gnuplot function. It ends up not being much better.

(defun gnuplot (cmds)
  (let* ((temporary-file-directory ".")
         (cmdfile (make-temp-file "gnuplot-cmds-" nil ".gpl"))
         (shellcmd (format "gnuplot --persist -c \"%s\"" cmdfile)))
    (with-temp-file cmdfile
      (insert cmds))
    (shell-command shellcmd)
    (delete-file cmdfile)))

You use this the same way.

(gnuplot "set title \"Simple Plots\" font \",20\"
set key left box
set samples 50
set style data points
set terminal png
set output \"gnuplot.png\"

plot [-10:10] sin(x),atan(x),cos(atan(x))")

It has the same limitations as our string-based Python solution. The benefit of them is the native command structure for Python or gnuplot is used, so anything they can do you can too.

2 An alternative approach using a DSL

As an alternative, we consider here a domain specific language (DSL) that maps onto gnuplot. Suppose we could do this instead.

(gnuplot
 (set terminal png)
 (set output "test.png")
 (set title "Simple Plots" font ",20")
 (set key left box)
 (set samples 50)
 (set style data points)

 (plot [-10:10] sin\(x\) \,atan\(x\) \,cos\(atan\(x\)\)))

Here is the figure from that code. The most annoying part of this is in the plot function we have to escape all the parentheses and commas, but otherwise it looks pretty lispy. The output of that program is the gnuplot commands that were generated for making the plot.

This retains a lot of benefits of programming in lisp. gnuplot has to be a macro though because we do not want to evaluate the s-expressions inside as lisp. For starters they just look lispy, I don't actually use them as lisp at all. Instead we transform them to the gnuplot code.

In the following code, I will develop the gnuplot macro. It has some sticky and tricky points, and it is not obvious it will support all the features of gnuplot, but I learned a lot doing it that I will share here.

Starting with a simple form inside the macro, I wanted to convert (set output "test.png") to "set output \"test.png\"". For this DSL, I want to treat every symbol in the form as if it should be turned into a string, anything that is a string should be quoted, and anything that is in parentheses (i.e. it passes listp) should be evaluated and converted to a string. Then all those strings should be joined by spaces. Here is a macro that does that (adapted from a solution at https://emacs.stackexchange.com/questions/32558/eval-some-arguments-in-a-macro/32570?noredirect=1#comment50186_32570).

There are a couple of corner cases that are handled here. If the arg is a string, we quote it. If the arg is not a symbol or string, then it is evaluated and converted to a string. Importantly, this is done in the run environment though, so we can inject variables into the gnuplot code.

(defmacro gargs (&rest args)
  "Convert symbols to strings, quote strings, and (expr) to what they evaluate to."
  `(s-join " " (list ,@(cl-mapcan
                        (lambda (s)
                          (list
                           (cond
                            ((symbolp s)
                             (symbol-name s))
                            ((stringp s)
                             (format "\"%s\"" s))
                            (t
                             `(with-output-to-string
                                (princ ,s))))))
                        args))))

Here are a few examples of how it works. The loop is just to get a vertical table in org-mode for the blog post.

(loop for s in
      (list (gargs set key title "before fit" size \, (+ 5 5))
            (gargs set title "red")
            (gargs set yrange [0:*])
            (gargs "5")
            (let ((x 6)) (gargs (identity x)))
            (gargs 'x)
            (gargs '(x))
            (gargs set label 1 "plot for [n=2:10] sin(x*n)/n" at graph .95\, graph .92 right))
      collect
      (list s))

A limitation of this is that we either have quote things like parentheses, commas, semi-colons and sometimes square brackets:

(gargs plot for [n=2:10] sin\(x*n\)/n notitle lw \(13-n\)/2)

Or we have to use the string form instead; we can always fall back to that.

(gargs "plot for [n=2:10] sin(x*n)/n notitle lw (13-n)/2")

The macro above will do the grunt work on each form in the gnuplot macro. Finally, for the gnuplot macro, I want to take all the forms, convert them to gnuplot commands, write them to a temporary file, and then run gnuplot on the file, and finally delete the temp file. I assume we start with a gui terminal so graphs pop up unless you change it in your macro body. Here is that macro. It returns the generated code so it easy to see if you got the right program.

(defmacro gnuplot (&rest forms)
  (let* ((temporary-file-directory ".")
         (cmdfile (make-temp-file "gnuplot-cmds-" nil ".gpl"))
         (shellcmd (format "gnuplot --persist -c \"%s\"" cmdfile))
         (cmd-string))
    `(let ((cmd-string (s-join "\n" (list ,@(mapcar (lambda (x)
                                                      (if (stringp x)
                                                          x
                                                        `(gargs ,@x)))
                                                    forms)))))
       (with-temp-file ,cmdfile
         (insert "set terminal qt\n")
         (insert cmd-string)
         (setq cmd-string (buffer-string)))
       (shell-command ,shellcmd)
       (delete-file ,cmdfile)
       cmd-string)))

Here is a figure adapted from http://gnuplot.sourceforge.net/demo/iterate.html. I use the string form for the last line to avoid escaping all the special characters.

(gnuplot
 (set terminal png)
 (set output "iteration.png")
 (set title "Iteration within plot command")
 (set xrange [0:3])
 (set label 1 "plot for [n=2:10] sin(x*n)/n" at graph .95\, graph .92 right)
 "plot for [n=2:10] sin(x*n)/n notitle lw (13-n)/2")

Here is the resulting figure.

That is overall pretty sweet. There is a little dissonance between the strings, escaped comma, etc.., and it is not terribly ideal for integrating with regular lisp code inside the macro yet. That seems to be a feature of my choice to use (expr) as the syntax to evaluate a form. It means you have to do some gymnastics to get some s-expressions into the graphs. For example below I use a couple of variables to inject values. To get a string I have to use format to add the extra quotes, and to get the number I have to use the identity function. I also used escaped characters in the last line to see the difference.

(let ((ts "Iteration and substitution")
      (x0 0)
      (xf 3)
      (g1 0.95)
      (g2 0.92))
  (gnuplot
   (set terminal png)
   (set output "iteration-2.png")
   (set title (format "\"%s\"" ts))
   ;; Note the escaped square brackets
   (set xrange \[ (identity x0) : (identity xf) \])
   (set label 1 "plot for [n=2:10] sin(x*n)/n" at graph (identity g1) \, graph (identity g2) right)
   ;; note here I escaped the parentheses!
   (plot for [n=2:10] sin\(x*n\)/n notitle lw \(13-n\)/2)))

3 Summary

For the simple plots here, my DSL worked ok. There is a tradeoff in the syntax I chose that has some consequences. We cannot use the values of symbols in this DSL without resorting to hackery like (identity sym). We also cannot use the infix notation for sin(x) without quoting it as "sin(x)" or escaping the parentheses, e.g. sin\(x\), likewise square brackets which lisp will read as a vector. Commas have to be escaped, which is probably an emacs-lisp issue. To address that would require a reader macro which emacs-lisp does not have great support for. I am calling this experiment done for now. I have another syntax idea to try out another day.

Here is a preview of what it might look like. It is basically the same but I reuse keywords to indicate that :x0 should be replaced by whatever x0 evaluates to, and (: - 1 0.05) should be evaluated. The special character escaping is still there of course, since that is a limitation of the emacs lisp reader I think. I might try using x0? and (? - 1 0.05) instead. That might be less confusing. I like that the keywords are syntax highlighted for free though, and you can't use them for anything else.

(let ((ts "Iteration and substitution")
      (x0 0)
      (xf 3)
      (g2 0.92))
  (gnuplot
   (set terminal png)
   (set output "iteration-2.png")
   (set title :ts)
   ;; Note the escaped square brackets
   (set xrange \[ :x0 : :xf \])
   (set label 1 "plot for [n=2:10] sin(x*n)/n" at graph (: - 1 0.05) \, graph :g2 right)
   ;; note here I escaped the parentheses!
   (plot for [n=2:10] sin(x*n)/n notitle lw (13-n)/2)))

This has the benefit of a little cleaner injection of variables and selective execution of parenthetical expressions, we will just ignore any that don't pass (= (car expr) :). That May not work for sin((: + 1 1) x) though, unless I escape the outer parentheses too.

org-mode source

Org-mode version = 9.0.5

Emulating Sparql queries in emacs-lisp with pattern matching

Sun, 30 Apr 2017 13:46:47 EDT

Sqarql is a query language for RDF triples. A triple is a data structure that consists of a (subject predicate object). Sparql lets you query the triples to extract data from them. I have been interested in using these to augment the SQL databases I generate from my org-files to be able to infer relationships between subjects and objects. For example, I could encode relationships into the contact database I use, and then infer new information that is not encoded explicitly. So far though I haven't found a good Sparql database that I can easily integrate into Emacs (or even play around with). I am reading On Lisp these days and chapters 18 and 19 talk about destructuring and pattern matching, and I realized these can be used to implement something like Sparql queries on simple lisp data structures. In this post I explore what it looks like and how to do it.

Let's consider a small database of triples that codify relationships between two people. For example, we can codify that Ann is Bob's mother with (Bob mother Ann). Here is our little database.

(setq triples '((Bob mother Ann)
                (Bill father Bob)
                (Lucy mother Jane)
                (Bob wife Jane)))

We can filter out facts from the database with a -filter. Here we filter out triples about Bob. Emacs has nice pattern matching in the pcase macro (see http://www.wilfred.me.uk/blog/2017/03/19/pattern-matching-in-emacs-lisp/ and http://newartisans.com/2016/01/pattern-matching-with-pcase/ for example). It turns out this is an amazing way to solve this problem. Here we look at triples with the pattern that they start with Bob.

(-filter (lambda (triple) (pcase triple (`(Bob ,_ ,_) t))) triples)

And here we get all the mothers.

(-filter (lambda (triple) (pcase triple (`(,_ mother ,_) t))) triples)

We can infer some facts about these people from the database by using some "rules". For example, there is not an entry that tells us directly who Bill's grandmother is. If we assume that the mother of a person's father is their grandmother, then we can infer Bill's grandmother is Ann. In this post, we examine how to write code that can find that answer. We will use pattern matching on pairs of triples to do it.

We can enumerate pairs of triples, and use pattern matching to find the pair of triples that meet the criteria we specify. The criteria we want to match is (in pseudo-sparql):

(Bill father ?dad) (?dad mother ?grandmother)

In other words, we want to find a triple that contains Bill as the subject, father as the predication, and then his father will be the object, and then find another triple that matches a predicate of mother with the subject equal to the father object we found in the first triple, and the object of the second triple will be Bill's grandmother. We enumerate pairs of triples for the comparison. Here is a way to do that. It is not a very efficient way to do it; it would be better to first filter out the triples that match (Bill father something) and then filter out the triples that match (anything mother any other thing) and then consider the pairs of those triples. I will save that for another day; efficiency is not the point today ;)

(loop for i below (length triples)
      append
      (loop
       for j below (length triples)
       if (not (= i j))
       collect
       (list (nth i triples) (nth j triples))))

You can see the pair that matches is the fourth one down (actually the first one matches too, but not exactly in the order of the pattern we specified). Next, we use pcase for the pattern matching. This amazing macro allows you to specify a pattern in terms of reusable variables so we can specify that the same value exists in multiple places. We will use this pattern (in pcase syntax):

`((Bill father ,dad) (,dad mother ,grandmother))

That means match a list that has the first element of (Bill father something) and store the value of something in the variable dad. The second element of the list must match (something mother another thing) and store the value of another thing in the variable grandmother. The two variables dad and grandmother are then available in the body of the pcase statement. Here is the code to loop over the triples and return the result when we find a match.

(catch 'result
  (loop for i below (length triples)
        do
        (loop
         for j below (length triples)
         if (not (= i j))
         collect
         (pcase (list (nth i triples) (nth j triples))
           (`((Bill father ,dad) (,dad mother ,grandmother))
            (throw 'result (format "Bill's dad is %s and his grandmother is %s" dad grandmother)))))))

Not bad. It would be worthwhile to encapsulate that into a macro perhaps, so you could just write something like this:

(select (dad grandmother) from triples where `((Bill father ,dad) (,dad mother ,grandmother)))

For fun I implemented a limited version of this below. It is fairly limited, and lightly tested. The biggest limitation is we hard-coded the search over pairs of triples. This version searches by brute force too, because I don't know how to build in filtering yet. It is another exercise for another day to remove these limitations. Here I just want to try out the macro with the syntactic sugar of "from" and "where" (which are totally ignored) as well at the backquoted query.

(defmacro select (&rest args)
  (let ((values (first args))
        (db (third args))
        (query (fifth args)))
    `(catch 'result
       (loop for i below (length ,db)
             do
             (loop
              for j below (length ,db)
              if (not (= i j))
              do
              (pcase (list (nth i triples) (nth j triples))
                (,query
                 (throw 'result (list ,@values)))))))))

Here is a fun way to write the query that finds the grandmother of the person named Bill with variable capture.

(select (person dad grandmother) from triples
        where `((,(and person (let Bill person)) father ,dad) (,dad mother ,grandmother)))

We can see the grandmother is Ann, as we found before.

Let's have a look at the macro expansion. Clearly our macro hides a lot of work from us!

(macroexpand '(select (person dad grandmother) from triples
        where `((,(and person (let Bill person)) father ,dad) (,dad mother ,grandmother))))

Bill

Bob

Ann

How about another example query. Who is Lucy's dad? The most direct query would be `(Lucy father ,dad), but a) that fact is not in the database, and b) our select macro won't search a single query anyway. So, let's examine how to find the answer by inference.

Let's assume that Lucy's dad is also the husband of her mother. Let's also assume that we can infer that if we know Jane is the wife of Bob, then Bob is the husband of Jane, and so we can infer from our database that Bob is Lucy's dad. This results in a query on a pair of triples that matches a pattern like:

(Lucy mother ?mom) (?dad wife ?mom)

Here is that query in our select macro.

(select (person mom dad) from triples
        where `((,(and person (let Lucy person)) mother ,mom) (,dad wife ,mom)))

Pretty cool! Clearly there is still a lot to do to make this practical. The implementation I used here wouldn't scale well with large numbers of triples, and its limited to a single kind of query. Chapters 18 and 19 in On Lisp address the query limitation (and they are not even limited to triples) and a different syntax style that is more Sparql like. When I get through them, I will probably add a new post on it. There are a lot of interesting problems to solve here including what to do if there are multiple matches, or inconsistent data? The Sparql select command allows you to group, order and limit the results which would be increasingly useful with larger triple stores. That would definitely add a lot of code to the macro!

org-mode source

Org-mode version = 9.0.5

Writing lisp code from Python

Mon, 30 May 2016 09:26:05 EDT

Some time ago I wrote about converting python data structures to lisp . I have expanded on that idea to writing lisp programs from Python! The newly expanded code that makes this possible can be found at https://github.com/jkitchin/pycse/blob/master/pycse/lisp.py .

Here are the simple data types known to pycse.lisp:

import pycse.lisp
import numpy as np

print("a string".lisp)
a = 5
b = 5.0
print(a.lisp)
print(b.lisp)
print([1, 2, 3].lisp)
print((1, 2, 3).lisp)
print({'a': 4}.lisp)
print(np.array([1, 2, 3]).lisp)
print(np.array([1.0, 2.0, 3.0]).lisp)

"a string"
5
5.0
(1 2 3)
(1 2 3)
(:a 4)
(1 2 3)
(1.0 2.0 3.0)

There are also some more complex types.

import pycse.lisp as pl

print(pl.Symbol('lambda'))
print(pl.Quote('lambda'))
print(pl.SharpQuote('lambda'))
print(pl.Cons("a", 5))
print(pl.Alist(["a", 2, "b", 5]))
print(pl.Vector([1, 2, 3]))

print(pl.Backquote([]))
print(pl.Comma([1, 2, 3]))
print(pl.Splice([1, 2, 3]))

lambda
'lambda
#'lambda
("a" . 5)
(("a" . 2) ("b" . 5))
[1 2 3]
`()
,(1 2 3)
,@(1 2 3)

You can nest these too.

import pycse.lisp as pl
print(pl.Quote(pl.Alist(["a", 2, "b", 5])))
print(pl.Backquote([pl.Symbol('+'), pl.Comma(pl.Symbol('b')), 5]))

'(("a" . 2) ("b" . 5))
`(+ ,b 5)

All that means we can use Python code to generate lisp programs. Here is an example where we make two sub-programs, and combine them into an overall program, then add one more subprogram to it. We wrap the results in an emacs-lisp block, then actually run the block!

import pycse.lisp as pl

p1 = [pl.Symbol('mapcar'),
      [pl.Symbol('lambda'),
       [pl.Symbol('x')],
       [pl.Symbol('*'),
        pl.Symbol('x'),
        pl.Symbol('x')]],
      pl.Quote([1, 2, 3, 4])]

p2 = [pl.Symbol('princ'), "Hello world"]

p = [pl.Symbol('list'), p1, p2]
p.append([pl.Symbol('+'), 5, 5])

print(p.lisp)

(list (mapcar (lambda (x) (* x x)) '(1 2 3 4)) (princ "Hello world") (+ 5 5))

(1 4 9 16)

Hello world

Wow, it worked! Here is another example of setting up a macro and then running it.

import pycse.lisp as pl
s = pl.Symbol
bq = pl.Backquote
c = pl.Comma

p1 = [s('defmacro'), s('f'), [s('x')],
      "A docstring",
      bq([s('*'), c(s('x')), 5])]


p2 = [s('f'), 5]

print(p1.lisp)

print(p2.lisp)

(defmacro f (x) "A docstring" `(* ,x 5))
(f 5)

I am not too sure where this will be super useful, but it is an interesting proof of concept. I haven't tested this much beyond the original post and this one. Let me know if you find issues with it.

org-mode source

Org-mode version = 8.2.10

OMG A Lisp that runs python

Wed, 30 Mar 2016 17:10:17 EDT

For a year now I have struggled with abandoning Python for Lisp. It's complicated, I have used Python for 15 years now, and have a lot of skill and knowledge in it. I have used emacs-lisp for about 5 years now, and have a far bit of skill with it too. They solve really different problems. Between the two, I find I like writing and editing elisp lots better than writing Python, except it lacks the scipy+numpy+matplotlib stack. I looked into Racket and Common Lisp, but they also don't really have that as nicely as Python does at the moment. It hit me earlier today that a Lisp that compiled to Python might be the right middle ground. I had seen this project Hy (http://docs.hylang.org/en/latest/quickstart.html ) earlier, but didn't connect the dots to this.

Let me do that here. First, an obligatory execute function to run org-mode code blocks.

(defun org-babel-execute:hy (body params)
  (let* ((temporary-file-directory ".")
         (tempfile (make-temp-file "hy-")))
    (with-temp-file tempfile
      (insert body))
    (unwind-protect
        (shell-command-to-string
         (format "hy %s" tempfile))
      (delete-file tempfile))))

org-babel-execute:hy

Now the basic Hello world example. It looks like lisp.

(print "Hy world")

Hy world

Now for a use that looks like Python:

(import numpy)
(setv a (numpy.array [1 2 3]))
(setv b (numpy.array [1 2 3]))
(print (numpy.dot a b))

WHAT!!!!

A simple plot? Surely it can't be so easy…

(import [matplotlib.pyplot :as plt])
(plt.plot [1 2 4 8])
(plt.xlabel "x")
(plt.ylabel "y")
(plt.savefig "hy-test.png")

2016-03-30 17:09:40.826 Python[94292:d13] CoreText performance note: Client called CTFontCreateWithName() using name "Lucida Grande" and got font with PostScript name "LucidaGrande". For best performance, only use PostScript names when calling this API.
2016-03-30 17:09:40.826 Python[94292:d13] CoreText performance note: Set a breakpoint on CTFontLogSuboptimalRequest to debug.

Wow. I am not sure what the warnings are, I seem to get them on my Mac for some reason. How about solving an equation?

(import [scipy.optimize [fsolve]])
(defn objective [x] (- 2 x))
(print (fsolve objective -1))

[ 2.]

     _.-^^---....,,--
 _--                  --_
<                        >)
|                         |
 \._                   _./
    ```--. . , ; .--'''
          | |   |
       .-=||  | |=-.
       `-=#$%&%$#=-'
          | ;  :|
 _____.,-#%&$@%#&#~,._____
       _---~~(~~-_.
     _{        )   )
   ,   ) -~~- ( ,-' )_
  (  `-,_..`., )-- '_,)
 ( ` _)  (  -~( -_ `,  }
 (_-  _  ~_-~~~~`,  ,' )  <---- My brain right now...
   `~ -^(    __;-,((()))
         ~~~~ {_ -_(())
                `\  }
                  { }

I may not be able to sleep tonight…

Ascii art courtesy of http://chris.com/ascii/index.php?art=people/body%20parts/brains and http://www.ascii-code.com/ascii-art/weapons/explosives.php .

org-mode source

Org-mode version = 8.2.10

A sexp version of a bibtex entry

Wed, 10 Jun 2015 08:54:00 EDT

Below you see a typical bibtex entry. Today we explore an alternate approach to represent the information (data) in that entry as s-expressions, i.e. as a lisp data structure. Why? because it seems like an interesting exploration!

@article{hallenbeck-2013-effec-o2,
  author =       "Hallenbeck, Alexander P. and Kitchin, John R.",
  title =        {Effects of \ce{O_2} and \ce{SO_2} on the capture capacity of a
                  primary-amine based polymeric \ce{CO_2} sorbent},
  keywords =     {RUA, orgmode},
  journal =      "Industrial \& Engineering Chemistry Research",
  pages =        "10788-10794",
  year =         2013,
  volume =       {52},
  number =       {31},
  doi =          "10.1021/ie400582a",
  url =          "http://pubs.acs.org/doi/abs/10.1021/ie400582a",
  eprint =       "http://pubs.acs.org/doi/pdf/10.1021/ie400582a",
}

Here is what that same data structure might look like as a sexp-based lisp data structure.

(article "hallenbeck-2013-effec-o2"
         (author "Hallenbeck, Alexander P. and Kitchin, John R.")
         (title "Effects of \ce{O_2} and \ce{SO_2} on the capture capacity of a primary-amine based polymeric \ce{CO_2} sorbent")
         (journal "Industrial \& Engineering Chemistry Research")
         (pages "10788-10794")
         (year 2013)
         (number 31)
         (doi "10.1021/ie400582a")
         (url "http://pubs.acs.org/doi/abs/10.1021/ie400582a")
         (eprint "http://pubs.acs.org/doi/pdf/10.1021/ie400582a"))

We can retrieve data from the sexp form pretty easily. Here we get the authors.

(let* ((art '(article "hallenbeck-2013-effec-o2"
                      (author "Hallenbeck, Alexander P. and Kitchin, John R.")
                      (title "Effects of \ce{O_2} and \ce{SO_2} on the capture capacity of a primary-amine based polymeric \ce{CO_2} sorbent")
                      (journal "Industrial \& Engineering Chemistry Research")
                      (pages "10788-10794")
                      (year 2013)
                      (number 31)
                      (doi "10.1021/ie400582a")
                      (url "http://pubs.acs.org/doi/abs/10.1021/ie400582a")
                      (eprint "http://pubs.acs.org/doi/pdf/10.1021/ie400582a")))
       (fields (cddr art)))
  (cadr (assoc 'author fields)))

Hallenbeck, Alexander P. and Kitchin, John R.

That is simple enough you might just write a little function to streamline it like this, and return a formatted string.

(defun get-article-field (article field)
  "Return value of FIELD in ARTICLE."
  (cadr (assoc field (cddr article))))

(let ((art '(article "hallenbeck-2013-effec-o2"
                     (author "Hallenbeck, Alexander P. and Kitchin, John R.")
                     (title "Effects of \ce{O_2} and \ce{SO_2} on the capture capacity of a primary-amine based polymeric \ce{CO_2} sorbent")
                     (journal "Industrial \& Engineering Chemistry Research")
                     (pages "10788-10794")
                     (year 2013)
                     (number 31)
                     (doi "10.1021/ie400582a")
                     (url "http://pubs.acs.org/doi/abs/10.1021/ie400582a")
                     (eprint "http://pubs.acs.org/doi/pdf/10.1021/ie400582a"))))
  (format "%s, doi:%s (%s)"
          (get-article-field art 'author)
          (get-article-field art 'doi)
          (get-article-field art 'year)))

Hallenbeck, Alexander P. and Kitchin, John R., doi:10.1021/ie400582a (2013)

You might be wondering, why is that even a little bit interesting? One reason is that it looks a little like what lisp returns after parsing an xml file. Another is, the data structure looks kind of like data, but it is also some code, if article was defined as a function! Let us consider what this might look like. I use a macro to define the field functions since in this case they all do the same thing, and these simply return a string with the field-name and value in curly brackets. We eval the macro to make sure it defines the function. I define an article function that wraps the fields in @bibtex-key{fields}, which defines a bibtex entry.

(defmacro make-field (field-name)
  "define a field that returns a string"
  `(defun ,(intern field-name) (content)
     (format "  %s = {%s}" ,field-name content)))

(loop for field in '("author" "title" "journal" "pages" "number" "doi" "url" "eprint" "year")
  do (eval `(make-field ,field)))

(defun article (bibtex-key &rest fields)
  (concat
   (format "@article{%s,\n" bibtex-key)
   (mapconcat (lambda (field) (eval field)) fields ",\n")
   "\n}\n"))

(article "hallenbeck-2013-effec-o2"
         (author "Hallenbeck, Alexander P. and Kitchin, John R.")
         (title "Effects of \ce{O_2} and \ce{SO_2} on the capture capacity of a primary-amine based polymeric \ce{CO_2} sorbent")
         (journal "Industrial \& Engineering Chemistry Research")
         (pages "10788-10794")
         (number 31)
         (year 2013)
         (doi "10.1021/ie400582a")
         (url "http://pubs.acs.org/doi/abs/10.1021/ie400582a")
         (eprint "http://pubs.acs.org/doi/pdf/10.1021/ie400582a"))

@article{hallenbeck-2013-effec-o2,
  author = {Hallenbeck, Alexander P. and Kitchin, John R.},
  title = {Effects of ce{O_2} and ce{SO_2} on the capture capacity of a primary-amine based polymeric ce{CO_2} sorbent},
  journal = {Industrial & Engineering Chemistry Research},
  pages = {10788-10794},
  number = {31},
  year = {2013},
  doi = {10.1021/ie400582a},
  url = {http://pubs.acs.org/doi/abs/10.1021/ie400582a},
  eprint = {http://pubs.acs.org/doi/pdf/10.1021/ie400582a}
}

Wow. We executed our data structure, and got a bibtex entry! That seems moderately interesting to me. Next is an example of taking the same data structure and rendering it as xml. This is some lispy wizardry, rather than use a macro to define functions, I temporarily define functions within a cl-flet macro, which I have to collect as a list of code. Then, I eval the list. This feels pretty odd, but seems like a lispy kind of thing to do.

(eval
 (list 'cl-flet
       (append (loop for field in '("author" "title" "journal" "pages"
                                      "number" "doi" "url" "eprint" "year")
                       collect (list (intern field)
                                     '(content)
                                     `(format "  <%s>%s</%s>" ,field content ,field)))
               '((article (bibtex-key &rest fields)
                          (concat
                           (format
                            "<article bibtex-key=\"%s\">\n" bibtex-key)
                           (mapconcat (lambda (field) (eval field)) fields "\n")
                           "\n</article>")))
               )
       ;; body of cl-flet
       '(article "hallenbeck-2013-effec-o2"
                (author "Hallenbeck, Alexander P. and Kitchin, John R.")
                (title "Effects of \ce{O_2} and \ce{SO_2} on the capture capacity of a primary-amine based polymeric \ce{CO_2} sorbent")
                (journal "Industrial \& Engineering Chemistry Research")
                (pages "10788-10794")
                (number 31)
                (year 2013)
                (doi "10.1021/ie400582a")
                (url "http://pubs.acs.org/doi/abs/10.1021/ie400582a")
                (eprint "http://pubs.acs.org/doi/pdf/10.1021/ie400582a"))))

<article bibtex-key="hallenbeck-2013-effec-o2">
  <author>Hallenbeck, Alexander P. and Kitchin, John R.</author>
  <title>Effects of ce{O_2} and ce{SO_2} on the capture capacity of a primary-amine based polymeric ce{CO_2} sorbent</title>
  <journal>Industrial & Engineering Chemistry Research</journal>
  <pages>10788-10794</pages>
  <number>31</number>
  <year>2013</year>
  <doi>10.1021/ie400582a</doi>
  <url>http://pubs.acs.org/doi/abs/10.1021/ie400582a</url>
  <eprint>http://pubs.acs.org/doi/pdf/10.1021/ie400582a</eprint>
</article>

Prefer json? No problem, just reformat the functions!

(eval
 (list 'cl-flet
       (append (loop for field in '("author" "title" "journal" "pages"
                                      "number" "doi" "url" "eprint" "year")
                       collect (list (intern field)
                                     '(content)
                                     `(format "   \"%s\": \"%s\"" ,field content)))
               '((article (bibtex-key &rest fields)
                          (concat
                           (format
                            "{\"article\":\n  {\"bibtex-key\": \"%s\",\n" bibtex-key)
                           (mapconcat (lambda (field) (eval field)) fields ",\n")
                           "}\n}"))))
       ;; body of cl-flet
       '(article "hallenbeck-2013-effec-o2"
                (author "Hallenbeck, Alexander P. and Kitchin, John R.")
                (title "Effects of \ce{O_2} and \ce{SO_2} on the capture capacity of a primary-amine based polymeric \ce{CO_2} sorbent")
                (journal "Industrial \& Engineering Chemistry Research")
                (pages "10788-10794")
                (number 31)
                (year 2013)
                (doi "10.1021/ie400582a")
                (url "http://pubs.acs.org/doi/abs/10.1021/ie400582a")
                (eprint "http://pubs.acs.org/doi/pdf/10.1021/ie400582a"))))

{"article":
  {"bibtex-key": "hallenbeck-2013-effec-o2",
   "author": "Hallenbeck, Alexander P. and Kitchin, John R.",
   "title": "Effects of ce{O_2} and ce{SO_2} on the capture capacity of a primary-amine based polymeric ce{CO_2} sorbent",
   "journal": "Industrial & Engineering Chemistry Research",
   "pages": "10788-10794",
   "number": "31",
   "year": "2013",
   "doi": "10.1021/ie400582a",
   "url": "http://pubs.acs.org/doi/abs/10.1021/ie400582a",
   "eprint": "http://pubs.acs.org/doi/pdf/10.1021/ie400582a"}
}

Is this useful? Great question. I don't plan to convert by bibtex files to sexp format anytime soon ;) The format I used above is just a simple one. It might be desirable to include individual authors instead of an author string, and maybe support attributes to establish an author order. An author structure might be more complex to include scientific ids like an orcid, alternative names, etc… Finally, the s-exp data structure is super easy to use in lisp, but other languages would have parse it into some native structure the way they parse json or xml. There is limited support for s-expressions in most other non-lispy languages.

I like the idea of data representation as code, and its conversion to some other kind of format. It is subtle here, but notice we never had to write a parser for the sexp notation. That already exists as the lisp interpreter. We did write code to use the data, and convert the data. The sexp notation is pretty easy to write, in contrast to the xml or json representations. Some interesting issues might be what to do with fields that are not defined, perhaps a macro would be used on the fly, or in the cl-flet definition. It is hard to imagine doing these things in another language than lisp!

org-mode source

Org-mode version = 8.2.10

Python data structures to lisp

Sat, 16 May 2015 10:47:59 EDT

I have an idea in mind that would use the output of python scripts in lisp functions. Xah Lee posted an idea for writing emacs commands in scripting languages . In this post I want to explore an extension of the idea, where a Python script will return output that can be read in Lisp, e.g. we can convert a Python list to a lisp list, or a dictionary to an a-list or p-list. I can already see that simple data structures will be "simple", and arbitrary data structures will offer a lot of challenges, e.g. nested lists or dictionaries…

If I could add some custom functions to the basic builtin types in Python, then I could use another approach to format python objects as lisp data types. This isn't recommended by Pythonistas, but I guess they don't want to use lisp as much as I do ;) I found this approach to modifying builtins:

http://stackoverflow.com/questions/2444680/how-do-i-add-my-own-custom-attributes-to-existing-built-in-python-types-like-a

We use that almost verbatim here to get what I want. This is a super low level way to add functions to the builtins. I add some simple formatting to floats, ints and strings. I add a more complex recursive formatting function to lists, tuples and dictionaries. A dictionary can be represented as an alist or plist. Both examples are shown, but I leave the alist version commented out. Finally, we add a lispify function to numpy arrays.

import ctypes as c

class PyObject_HEAD(c.Structure):
    _fields_ = [('HEAD', c.c_ubyte * (object.__basicsize__ -
                                      c.sizeof(c.c_void_p))),
                ('ob_type', c.c_void_p)]

_get_dict = c.pythonapi._PyObject_GetDictPtr
_get_dict.restype = c.POINTER(c.py_object)
_get_dict.argtypes = [c.py_object]

def get_dict(object):
    return _get_dict(object).contents.value

get_dict(str)['lisp'] = lambda s:'"{}"'.format(str(s))
get_dict(float)['lisp'] = lambda f:'{}'.format(str(f))
get_dict(int)['lisp'] = lambda f:'{}'.format(str(f))

import collections
import numpy as np

def lispify(L):
    "Convert a Python object L to a lisp representation."
    if (isinstance(L, str)
        or isinstance(L, float)
        or isinstance(L, int)):
        return L.lisp()
    elif (isinstance(L, list)
          or isinstance(L, tuple)
          or isinstance(L, np.ndarray)):
        s = []
        for element in L:
            s += [element.lisp()]
        return '(' + ' '.join(s) + ')'
    elif isinstance(L, dict):
        s = []
        for key in L:
            # alist format
            # s += ["({0} . {1})".format(key, L[key].lisp())]
            # plist
            s += [":{0} {1}".format(key, L[key].lisp())]
        return '(' + ' '.join(s) + ')'

get_dict(list)['lisp'] = lispify
get_dict(tuple)['lisp'] = lispify
get_dict(dict)['lisp'] = lispify
get_dict(np.ndarray)['lisp'] = lispify

Let us test these out.

from pylisp import *
a = 4.5
print int(a).lisp()
print a.lisp()
print "test".lisp()

print [1, 2, 3].lisp()
print (1, 2, 3).lisp()

print [[1, 3], (5, 6)].lisp()

print {"a": 5}.lisp()
print [[1, 3], (5, 6), {"a": 5, "b": "test"}].lisp()


A = np.array([1, 3, 4])
print A.lisp()
print ({"tree": [5, 6]}, ["a", 4, "list"], 5, 2.0 / 3.0).lisp()

4
4.5
"test"
(1 2 3)
(1 2 3)
((1 3) (5 6))
(:a 5)
((1 3) (5 6) (:a 5 :b "test"))
(1 3 4)
((:tree (5 6)) ("a" 4 "list") 5 0.666666666667)

Now, is that better than a single lisp function with a lot of conditionals to handle each type? I am not sure. This seems to work pretty well.

Here is how I imagine using this idea. We would have some emacs-lisp variables and use them to dynamically generate a python script. We run the python script, capturing the output, and read it back in as a lisp data structure. Here is a simple kind of example that generates a dictionary.

(let* ((elisp-var 6)
       (result)
      (script (format "
from pylisp import *
print {x: [2*y for y in range(x)] for x in range(1, %s)}.lisp()
" elisp-var)))

  ;; start a python process
  (run-python)
  (setq result (read (python-shell-send-string-no-output
   script)))
  (plist-get result :5))

(0 2 4 6 8)

That seems to work pretty well. One alternative idea to this is Pymacs , which I have written about before . This project isn't currently under active development, and I ran into some difficulties with it before.

Here we can solve the problem I previously posed and get the result back as an elisp float, and then reuse the result

(let* ((myvar 3)
       (script (format "from pylisp import *
from scipy.optimize import fsolve
def objective(x):
    return x - 5

ans, = fsolve(objective, %s)
print ans.lisp()" myvar)))
  (run-python)
  (setq result (read (python-shell-send-string-no-output
                       script)))
  (- 5 result))

0.0

Bottom line: we can write python code in lisp functions that are dynamically updated, execute them, and get lisp data structures back for simple data types. I think that could be useful in some applications, where it is easier to do parsing/analysis in Python, but you want to do something else that is easier in Lisp.

org-mode source

Org-mode version = 8.2.10

A prototype implementation of jasp in emacs-lisp

Wed, 24 Dec 2014 11:41:00 EST

1. The Atom class
2. The Atoms class
3. The Calculator class
4. Putting it all together to run calculations
5. Summary thoughts

I want to try implementing an interface to VASP in emacs-lisp. VASP is a program that uses density functional theory to calculate properties of materials. VASP was designed for the user to create several text-based input files that contain an atomic geometry, and calculation parameters. Then you run the VASP program, which reads those input files and generates output files. Finally, you parse out the results you want from the output files.

It is moderately tedious to do that, so we already have a very extensive Python interface (http://github.com/jkitchin/jasp ) that automates the input file creation and output file parsing, and with org-mode we have a pretty good literate research environment to document what we do. But, the Python/emacs environment is not as integrated as the emacs-lisp/emacs environment is, particularly when it comes to accessing documentation. So, I want to try out a lisp approach to see what this would look like, and if it has any benefits.

The bare bones implementation will have an Atom object, to store the type of atom and its coordinates, an Atoms object which will be a collection of atoms, and a unit cell, and a Calculator object that will store calculation parameters.

Then, we will try using it to see what advantages it might have. This will be a moderately long post.

1 The Atom class

This is my first serious effort to use the object system in emacs-lisp, so I do not claim it is optimal, or beautiful. It is functional though. We make a simple Atom class that holds the chemical symbol, and xyz coordinates. We do not consider having a magnetic moment at this point, and this class has no methods.

(defclass Atom ()
  ((symbol :initarg :symbol
           :documentation "A string for the chemical symbol")
   (x :initarg :x
      :documentation "The x coordinate")
   (y :initarg :y
      :documentation "The y coordinate")
   (z :initarg :z
      :documentation "The z coordinate"))
 "A class to represent an atom.")

(provide 'Atom)

Atom

Let us try it out. We make an Atom, then get the symbol using the oref function. I don't think we need a "Name" for the atom, so the second argument is nil.

(Atom nil :symbol "C" :x 0 :y 0 :z 0)

[object Atom nil "C" 0 0 0]

(let ((a1 (Atom nil :symbol "C" :x 0 :y 0  :z 0)))
  (oref a1 symbol))

It is not difficult to modify an Atom object. We use oset.

(let ((a1 (Atom nil :symbol "C" :x 0 :y 0  :z 0)))
  (oset a1 x 2.5)
  a1)

[object Atom nil "C" 2.5 0 0]

2 The Atoms class

We need at Atoms object, which will store a list of Atom objects, and a unit cell. I do not know how to make this class act like a list the way that the Atoms object in ase acts. We will have to access the list of atoms as a slot. We define one method here to get the ith atom from the object.

(defclass Atoms ()
  ((atoms :initarg :atoms
          :documentation "A list of `Atom' objects")
   (cell :initarg :cell
         :documentation "A 3-tuple of lengths of an orthorhombic unit cell"))
  "A class to represent an atoms object.")


(defmethod get-atom ((atoms Atoms) &rest args)
  "Get a list of atoms in ARGS.
Return atom if ARGS contains one element, a list of atom objects
otherwise."
  (cond
   ((= 1 (length args))
    (elt (oref atoms atoms) (car args)))
   (t
    (mapcar (lambda (i)
              (elt (oref atoms atoms) i))
            args))))

(provide 'Atoms)

get-atom

That seems to do what we need. It has some limitations:

We only allowed for orthorhombic unit cells
We did not enable any constraints or magnetic moments on the atoms.

Here is an example Atoms object. Note we use `, notation to ensure each Atom is created, since Atom is a constructor function that returns the object.

(Atoms nil
       :atoms `(,(Atom nil :symbol "C" :x 0 :y 0 :z 0)
                ,(Atom nil :symbol "O" :x 1.1 :y 0 :z 0))
       :cell '(8 9 10))

[object Atoms nil ([object Atom nil "C" 0 0 0] [object Atom nil "O" 1.1 0 0]) (8 9 10)]

We can drill into the object, e.g. to get the second atom:

(let ((A1 (Atoms nil
                 :atoms `(,(Atom nil :symbol "C" :x 0 :y 0 :z 0)
                          ,(Atom nil :symbol "O" :x 1.1 :y 0 :z 0))
                 :cell '(8 9 10))))
  (get-atom A1 1))

[object Atom nil "O" 1.1 0 0]

We can modify the atoms in place like this. Suppose we want to change the symbol of the first atom to "O". We use setf for this too.

(let ((A1 (Atoms nil
                 :atoms `(,(Atom nil :symbol "C" :x 0 :y 0 :z 0)
                          ,(Atom nil :symbol "O" :x 1.1 :y 0 :z 0))
                 :cell '(8 9 10))))
  (oset (get-atom A1 0) symbol "O")
  A1)

[object Atoms nil ([object Atom nil "O" 0 0 0] [object Atom nil "O" 1.1 0 0]) (8 9 10)]

The only think I do not like about this syntax is the need to get the list of atoms from the object. That is a little clunkier than the Python analog where the object is a list itself. That may be just my inexperience with emacs-lisp. Probably you can define some getter function to smooth this over.

This Atoms class lacks much of the functionality of the ase.Atoms class, but it is sufficient for this prototype.

3 The Calculator class

Next, we need our Calculator. This will store the parameters, and be responsible for creating the INCAR, POSCAR, KPOINTS, and POTCAR files, running a calculation, and getting data from the output. We also create a with-current-directory macro that will temporarily change the working directory since VASP uses the same filenames over and over, but in different directories.

(defmacro with-current-directory (directory &rest body)
  "Set the working directory temporarily set to DIRECTORY and run BODY.
DIRECTORY is expanded, and create it and its parents if needed."
  `(progn
     (unless (file-exists-p (file-name-as-directory
                             (expand-file-name ,directory)))
       (make-directory ,directory t))
     
     (let ((default-directory (file-name-as-directory
                                (expand-file-name ,directory)))) 
        ,@body)))


(defclass Jasp ()
  ((wd :initarg :wd
       :initform "."  ; default to the current working directory
       :documentation "Directory to run calculation in.")
   (encut :initarg :encut
          :documentation "Positive number in eV for planewave cutoff.
See URL `http://cms.mpi.univie.ac.at/vasp/vasp/ENCUT_tag.html'.")
   (nbands :initarg :nbands
           :documentation "Integer number of bands.
See URL `http://cms.mpi.univie.ac.at/vasp/vasp/NBANDS_tag.html'.")
   (kpts :initarg :kpts
         :initform (1 1 1)  ; default value
         :documentation "3-tuple for Monkhorst-Pack K-point grid.")
   (xc :initarg :xc
       :documentation "String of exchange correlation functional.")
   (atoms :initarg :atoms
          :documentation "An `Atoms' object."))
 "A class to represent a calculator that runs VASP.")


(defmethod view-atoms ((calc Jasp))
  "Open the ase-gui"
  (unless (and (file-exists-p "POSCAR")
               (file-exists-p "POTCAR"))
    (write-poscar calc)
    (write-potcar calc))
  (shell-command "ase-gui POSCAR"))


(defmethod write-poscar ((calc Jasp))
  "create a POSCAR file for CALC."
  (with-temp-file "POSCAR"
    (insert "Created by jasp.el\n")
    (insert "  1.0") ; unit cell scale factor
    (let* ((atoms (oref calc atoms))
           (cell (oref atoms cell)))
      (loop for v in cell
            for i below (length cell)     
            do
            (insert "\n")
            (loop for j below (length cell)
                  do
                  (if (equal i j)
                      (insert (format " %f " (float (elt cell i))))
                    (insert (format " %f " 0.0 ))))))
    ;; The next line is counts for each atom type. For each number in
    ;; this line, there will be a copy of the POTCAR in the POTCAR
    ;; file. In ase, we sort the atoms and group them so that there is
    ;; only one POTCAR per atom. We do not do that here yet. We will
    ;; have a POTCAR for each atom.
    (insert "\n")
    (loop for atom in (oref (oref calc atoms) atoms)
          do (insert (format "1 ")))
    
    ;; now we do the atoms
    (insert "\nCartesian\n")
    (loop for atom in (oref (oref calc atoms) atoms)
          do
          (insert
           (format "%f %f %f\n"
                   (oref atom x)
                   (oref atom y)
                   (oref atom z))))
    (buffer-string)))


(defmethod write-kpoints ((calc Jasp))
  "Create KPOINTS file for CALC. 
Limited to automatic generation, and no offset."
  (with-temp-file "KPOINTS"
    (insert "Automatic mesh
0
Monkhorst-Pack
")
    (dolist (k (oref calc kpts))
      (insert (format "%4d " k)))
    (insert "\n0.0 0.0 0.0")
    (buffer-string)))


(defmethod write-potcar ((calc Jasp))
  "Generate the POTCAR file for CALC.
No `Atom' grouping is done, there is one POTCAR for each atom."
  (with-temp-file "POTCAR"
    (let ((xc (oref calc xc))
          (atoms (oref calc atoms))
          (vasp_pp_path (getenv "VASP_PP_PATH")))
      (loop for atom in (oref atoms atoms)
            do
            (insert-file-contents
             (f-join
              vasp_pp_path
              (concat "potpaw_" xc)
              (oref atom symbol)
              "POTCAR"))))
    (buffer-string)))


(defmethod write-incar ((calc Jasp))
  "Generate the INCAR file for CALC."
  (with-temp-file "INCAR"
    (insert (format "ENCUT = %f\n" (oref calc encut)))
    (insert (format "NBANDS = %d\n" (oref calc nbands)))
    (buffer-string)))


(defmethod run ((calc Jasp))
  "Write out input files, and run VASP as a simple shell command"
  (write-poscar calc)
  (write-kpoints calc)
  (write-potcar calc)
  (write-incar calc)
  (shell-command "vasp"))


(defmethod update ((calc Jasp))
  "Run vasp if needed for CALC.
We just check for a properly ended OUTCAR."
  (with-current-directory
   (oref calc wd)
   (unless (and (file-exists-p "OUTCAR")
                (with-temp-buffer
                  (insert-file-contents "OUTCAR")
                  (re-search-forward
                  "                 Voluntary context switches:"
                  (point-max)
                  t)))
     (run calc))))


(defmethod get-potential-energy ((calc Jasp))
  "Get potential energy from CALC."
  (update calc)
  (with-current-directory
   (oref calc wd)
   (with-temp-buffer
     (insert-file-contents "OUTCAR")
     ;; go to last entry
     (while (re-search-forward
             "free  energy   TOTEN  =\\s-*\\([-]?[0-9]*\\.[0-9]*\\) eV"
             (point-max)
             t)
       nil)
     ;; return last match
     (string-to-number  (match-string 1)))))

(provide 'jasp)

get-potential-energy

This is worth some discussion. On one hand, the constructor is a bit more verbose than the implementation in Python. In Python we use a context handler in place of the macro here. On the other hand, that verbosity comes with detailed, accessible documentation for each argument. We only considered the simplest of input arguments. It might be trickier to include lists, and other types of input. But I think those can all be worked out like they were in ase. We only implemented the simplest job control logic, but that also can be worked out. The biggest challenge might be getting more complex data from the output. Nearly everything can be obtained from vasprun.xml also, in the event that parsing is to slow or difficult.

Now, let us test this out. We can make a calculator:

(setq calc (Jasp
            nil
            :xc "PBE"
            :encut 350
            :nbands 6
            :atoms (Atoms
                    nil
                    :atoms `(,(Atom nil :symbol "C" :x 0 :y 0 :z 0)
                             ,(Atom nil :symbol "O" :x 1.1 :y 0 :z 0))
                    :cell '(8 9 10))))

[object Jasp nil "." 350 6 (1 1 1) "PBE" [object Atoms nil ([object Atom nil "C" 0 0 0] [object Atom nil "O" 1.1 0 0]) (8 9 10)]]

We can call the class functions like this. Here we write out the corresponding POSCAR:

(write-poscar calc)

Created by jasp.el
  1.0
 8.000000  0.000000  0.000000 
 0.000000  9.000000  0.000000 
 0.000000  0.000000  10.000000 
1 1 
Cartesian
0.000000 0.000000 0.000000
1.100000 0.000000 0.000000

It looks a little backward if you have only seen Python, where this would be calc.write_poscar(). It is almost the same characters, just a different order (and no . in lisp)!

Here we get the KPOINTS file:

(write-kpoints calc)

Automatic mesh
0
Monkhorst-Pack
   1    1    1 
0.0 0.0 0.0

I cannot show the POTCAR file for licensing reasons, but it works.

(write-potcar calc)

and the INCAR file:

(write-incar calc)

ENCUT = 350.000000
NBANDS = 6

We run a calculation like this. This will run vasp directly (not through the queue system).

(run calc)

The returned 0 means the shell command finished correctly.

And we retrieve the potential energy like this:

(get-potential-energy calc)

-14.687906

Not bad. That is close to the result we got from a similar calculation here .

4 Putting it all together to run calculations

If we put this all together the way we might use it in practice, it looks like this.

(load-file "Atom.el")
(load-file "Atoms.el")
(load-file "Jasp.el")

(let* ((co (Atoms
            nil
            :atoms `(,(Atom nil :symbol "C" :x 0 :y 0 :z 0)
                     ,(Atom nil :symbol "O" :x 1.1 :y 0 :z 0))
            :cell '(8 9 10)))

       (calc (Jasp
              nil
              :xc "PBE"
              :nbands 6
              :encut 350
              :atoms co)))
  
  (get-potential-energy calc))

-14.687906

Compare this with this Python code which does approximately the same thing:

from ase import Atoms, Atom
from jasp import *

co = Atoms([Atom('C', [0,   0, 0]),
            Atom('O', [1.1, 0, 0])],
            cell=(6., 6., 6.))

with jasp('molecules/simple-co', #output dir
          xc='PBE',  # the exchange-correlation functional
          nbands=6,  # number of bands
          encut=350, # planewave cutoff
          atoms=co) as calc:
    print 'energy = {0} eV'.format(co.get_potential_energy())

They look pretty similar. One thing clearly missing from emacs-lisp that Python has is full support for numerics and plotting. Some of this could be addressed via Pymacs , but certainly not all of it. Some of it could also be handled using org-mode to enable data from emacs-lisp to go to other code blocks that can handle it.

Finally, for a little fun, we illustrate mapping over a range of bond lengths. There is more than one way to do this. For example, we could create a list of calculators, and then run over them. Here we create one calculator, and just change the x position in a loop. We use the more general setf approach instead of oset to see what it looks like.

(let* ((co (Atoms
            nil
            :atoms `(,(Atom nil :symbol "C" :x 0 :y 0 :z 0)
                     ,(Atom nil :symbol "O" :x 1.1 :y 0 :z 0))
            :cell '(8 9 10)))
       (calc (Jasp
              nil
              :wd nil
              :xc "PBE"
              :nbands 6
              :encut 350
              :atoms co)))
  (dolist (d '(1.05 1.1 1.15 1.2 1.25))
    ;; change working directory
    (setf (oref calc wd) (format "co-%s" d))
    ;; set x-coordinate on oxygen atom
    (setf (oref (elt (oref co atoms) 1) x) d)
    (print (format "d = %s\nEnergy = %s eV"
                   d
                   (get-potential-energy calc)))))

"d = 1.05
Energy = -14.195892 eV"

"d = 1.1
Energy = -14.698456 eV"

"d = 1.15
Energy = -14.814809 eV"

"d = 1.2
Energy = -14.660395 eV"

"d = 1.25
Energy = -14.319904 eV"

See http://kitchingroup.cheme.cmu.edu/dft-book/dft.html#sec-3-4-1 for how this was done in Python. It looks pretty similar to me.

5 Summary thoughts

We implemented a bare bones emacs-lisp calculator for VASP. The library automates creation of input files, running the calculation, and parsing the output.

It seems pretty feasible to implement a pretty complete interface to VASP in emacs-lisp. The main reasons to do this are:

Integrated access to documentation
Emacs editing of emacs-lisp code
Integration with org-mode

Even with Python editor that had access to documentation as deeply integrated as emacs has with emacs-lisp, it would still just be a Python editor, i.e. you probably could not use the editor to write org-mode, LaTeX, etc… It is time to recognize we need both scientific document creation and code editing capability in the same place! This kind of suggests a need to get a better Python environment going in Emacs, which deeper integration of the documentation. See this post for some progress in that area!

org-mode source

Org-mode version = 8.2.10

The Kitchin Research Group

A new and improved Emacs gnuplot DSL

An emacs-lisp dsl for gnuplot

Table of Contents

1 Embedding Python or gnuplot

2 An alternative approach using a DSL

3 Summary

Emulating Sparql queries in emacs-lisp with pattern matching

Writing lisp code from Python

OMG A Lisp that runs python

A sexp version of a bibtex entry

Python data structures to lisp

A prototype implementation of jasp in emacs-lisp

Table of Contents

1 The Atom class

2 The Atoms class

3 The Calculator class

4 Putting it all together to run calculations

5 Summary thoughts