The Kitchin Research Group

A partial symbolic numeric solver in emacs-lisp

Sun, 21 May 2017 11:33:15 EDT

1. The Newton solver in emacs-lisp

I have been exploring ways to use emacs-lisp to express scientific ideas. In this post, we explore a partial symbolic numeric solver in Emacs-lisp. This involves some syntactic developments to more clearly identify something we want to solve for and to then generate the code required to solve it.

In section The Newton solver you can find a simple implementation of a Newton solver in emacs-lisp. This function allows you to numerically solve equations that can be written in the form \(f(x) = 0\) for \(x\) given an initial guess. You write a function for \(f(x)\) and pass the function to the solver. This is a standard approach used in Python with fsolve, for example. Here is an example of solving a trivial problem: \(x - 4 = 0\) just to check that it works. We use a lambda function for \(f(x) = x - 4 = 0\). The answer is \(x=4\).

(newton-f (lambda (x) (- x 4)) 2)

That syntax is not too bad, but we have the whole lambda expression in there, and some repetition of what we want to solve for as an argument and in the function. It would be interesting if we could just have an expression that gets solved, e.g. (newton-f (- x? 4) 2) where x? indicates the thing to solve for.

We can do that! We can take an expression, flatten it and find the variable names that end with ?. We should check that there is only one, but for now we don't. Here is an example that does that. I use a nested expression here just to illustrate that the code finds the x? variable correctly.

(require 'dash)

(let ((body '((* (- x? 4) 1))))
  (loop for item in (-flatten body)
        if (and (symbolp item) (s-ends-with? "?" (symbol-name item)))
        collect item))

So, given an expression we can identify the unknown that should be the argument to a lambda function. So, we create a macro that takes that expression and constructs a function to solve it, then calls newton-f on it. The macro is syntactically useful here because we do not have to quote the expression. Here is that macro.

(defmacro solve (expression guess)
  `(newton-f
    (lambda ,(loop for item in (-flatten expression)
                   if (and (symbolp item) (s-ends-with? "?" (symbol-name item)))
                   collect item)
      ,expression)
    ,guess))

I call this a partial symbolic solver because we do some introspection symbolically to identify what to solve for, and then construct the code required to solve it. Here is that trivial example (x? - 4 = 0). It just shows we can have some nesting and it still works. I am not so thrilled with the initial guess, but this is an iterative solver, so you either need an initial guess, or a solution range.

(solve (* (- x? 4) 1) 3)

Here is what that expands into:

(macroexpand '(solve (* (- x? 4) 1) 3))

It expands into what we would have written in the first place. The benefit to us is less typing, and a simpler syntax. Both of those reduce the opportunity to make errors!

A more realistic problem might be: Reactant A flows into a continuously stirred tank reactor at a rate of \(F_{A0} = 1\) mol/min with a volumetric flow of \(v_0 = 1\) L/min.. The reactor achieves 50% conversion (\(X\)) of A to products. The reaction rate law is known to be \(-r_A = k C_A\) with \(k = 0.1\) 1/min. Estimate the volume of the reactor. If you have taken my class in reaction engineering, you know the following facts:

The exit molar flow is defined by \(F_A = F_{A0} (1 - X)\)
The exit concentration is \(C_A = F_A / v_0\)
The mole balance is defined by \(0 = F_{A0} - F_A + r_A V\)

That is all we need; we can solve for \(V\) from the last equation. This is simple enough you might do the algebra to get: \(V = \frac{F_{A0} - F_A}{-r_A}\) which can be simply evaluated. We use our solver here and compare it to the evaluation.

Here is the solver:

(let* ((Fa0 1)
       (X 0.5)
       (Fa (* Fa0 (- 1 X)))
       (k 0.1)
       (v0 1)
       (Ca (/ Fa v0))
       (r (* k Ca))
       (ra (* r -1)))
  (solve (+ Fa0 (* Fa -1) (* ra V?)) 2))

It is pretty hard to imagine doing something like this in Python! It would probably involve parsing a string.

Here is the evaluation from our algebra:

(let* ((Fa0 1)
       (X 0.5)
       (Fa (* Fa0 (- 1 X)))
       (k 0.1)
       (v0 1)
       (Ca (/ Fa v0))
       (r (* k Ca))
       (ra (* r -1)))
  (/ (- Fa0 Fa) (* -1 ra)))

Within the tolerance specified in newton-f, these are the same.

This is just the tip of the iceberg though. You may have noticed that none of the variables in the let* had any descriptions. Sure, you could put some comments after them, but those are not really part of the code.

Also, we had to define the variables in advance of the expression. That is a limitation of how computers work, they cannot evaluate undefined variables. It constrains how we can express the idea. What if we could instead specify the equation first, then the data? That way we are clear what we are trying to do at a higher level, and fill in the details later. Suppose we wanted a syntax like the block below instead. Here we emphasize the equation we are solving first, and then define the variables and quantities used in the equation, and finally the guess that we use to find the solution.

(solve1
 (eqn (+ Fa0 (* -1 Fa) (* ra V?)))
 (data ((k 0.1 "rate constant 1/min")
        (Ca0 1.0 "feed concentration")
        (v0 1 "volumetric flow L/min")
        (Fa0 (* v0 Ca0) "Inlet molar flow")
        (X 0.5 "Desired conversion")
        (Fa (* Fa0 (- 1 X)) "Exit molar flow")
        (Ca (/ Fa v0) "exit concentration")
        (ra (* -1 k Ca) "rate in the reactor")))
 (guess 8))

That is achievable with the solve1 macro below! It too has some limitations, mostly the order of the data block still has to be correct, e.g. you cannot use a variable before it is defined. It would take some serious macro-fu to sort these so that everything is defined in the right order! Still, it allows you to express an executable idea in the order we defined. The strings in this syntax for documenting the variables are ignored, but they could be used in the macro to print useful information or something else you could imagine. You could also make them mandatory to encourage documentation.

(defmacro solve1 (&rest body)
  (let ((expression (second (assoc 'eqn body)))
        (data (loop for d in (second (assoc 'data body))
                    collect (list (first d) (second d))))
        (guess (second (assoc 'guess body))))
    `(let* ,data
       (newton-f
        (lambda ,(loop for item in (-flatten expression)
                       if (and (symbolp item) (s-ends-with? "?" (symbol-name item)))
                       collect item)
          ,expression)
        ,guess))))

To summarize, lisp macros allow us to rewrite the syntax of code before it is evaluated. This gives us the opportunity to inspect it, and generate new code, e.g. functions with arguments based on the contents of expressions, to save us typing. It also allows us to define ideas in a near arbitrary order that make sense to us, and then rearrange them so they make sense to the computer. See, for example, this post for an example of changing how functions are defined.

This seems to be heading in the domain specific language direction. I think it would be very helpful in engineering problem solving to build up tools like this. They could start out simple for new students to use. They never need to see the macro parts of this, just to learn how to use them for problem solving. These beginner tools would be limited in what they could do to minimize how much lisp is required to be learned so students can focus on the problem solving. Eventually they might outgrow them, and in the process transition to having the full lisp language at their disposal for problem solving.

1 The Newton solver in emacs-lisp

This is an emacs-lisp implementation of Newton's method. It is a simple implementation for a single variable. The tolerance and step-size are hard-coded for this post because we focus on the partial symbolic solver, not the best solver methods.

;; See https://en.wikipedia.org/wiki/Newton%27s_method for the method

(defun newton-f (func x0)
  "Solve the equation FUNC(x)=0 using Newton's method.
X0 is an initial guess."
  (let* ((tolerance 1e-6)
         (x x0)
         (dx 1e-6)
         fx fpx)
    (while (> (abs (funcall func x)) tolerance)
      (setq fx (funcall func x)
            fpx (/ (- (funcall func (+ x dx)) (funcall func (- x dx))) (* 2 dx))
            x (- x (/ fx fpx))))
    x))

org-mode source

Org-mode version = 9.0.5

Collecting entries from files in a directory

Mon, 25 Aug 2014 21:42:05 EDT

I am running a class where students will be generating files that contain their answers. I want to quickly get a list of counts of all the answers. For example, six students will create a file called animal.dat in a directory called example/<studentid>, and that file will contain their favorite animal. I made some example files to test this idea out. Here are the contents.

cat example/*/animal.dat

dog
cat
dog
bird
dog
bird

You can see there are three dogs, two birds and a cat. I want code to do this counting, because in my real application there will be 58 of these files, and lots of times I need to aggregate them. Let us start with a simple example that counts the elements in a list.

(let ((animals '(dog cat dog bird dog bird))
      (counts '())
      place)
  (dolist (animal animals)
    (setq place (assoc animal counts))
    (message "place = %s" place)
    (if place
        (setf (cdr place) (+ 1 (cdr place)))
      (setq counts (cons `(,animal . 1) counts))))
  counts)

((bird . 2) (cat . 1) (dog . 3))

Let us turn that into a function.

(defun counts (list)
  (let ((counts '())
        place)
    (dolist (el list)   
      (setq place (assoc el  counts))
    (message "place = %s" place)
    (if place
        (setf (cdr place) (+ 1 (cdr place)))
      (setq counts (cons `(,el . 1) counts))))
    counts))

(counts '(dog cat dog bird dog bird))

((bird . 2) (cat . 1) (dog . 3))

Nice. Now we need a simple way to get that list. We need to glob the files to find them, then open them and read the value. Here is a way to get the files.

(f-entries "example"
           (lambda (f)
             (string= (file-name-nondirectory f) "animal.dat"))
           t)

/Users/jkitchin/blogofile-jkitchin.github.com/_blog/collect-entries/example/s1/animal.dat

/Users/jkitchin/blogofile-jkitchin.github.com/_blog/collect-entries/example/s2/animal.dat

/Users/jkitchin/blogofile-jkitchin.github.com/_blog/collect-entries/example/s3/animal.dat

/Users/jkitchin/blogofile-jkitchin.github.com/_blog/collect-entries/example/s4/animal.dat

/Users/jkitchin/blogofile-jkitchin.github.com/_blog/collect-entries/example/s5/animal.dat

/Users/jkitchin/blogofile-jkitchin.github.com/_blog/collect-entries/example/s6/animal.dat

Now we just need to run a mapcar over this list of files.

(mapcar
 (lambda (f)
   (with-temp-buffer
     (insert-file-contents f)
     (s-trim (buffer-string))))
 (f-entries "example"
            (lambda (f)
              (string= (file-name-nondirectory f) "animal.dat"))
            t))

dog

cat

dog

bird

dog

bird

Finally, putting this together, we have some code that maps over the files, and counts the entries.

(counts
 (mapcar
  (lambda (f)
    (with-temp-buffer
      (insert-file-contents f)
      (s-trim (buffer-string))))
  (f-entries "example"
             (lambda (f)
               (string= (file-name-nondirectory f) "animal.dat"))
             t)))

((bird . 2) (cat . 1) (dog . 3))

This will be helpful in dealing with 58 entries during my class!

org-mode source

Org-mode version = 8.2.7c

Autogenerating functions in emacs-lisp

Mon, 12 May 2014 18:05:45 EDT

I have a need to generate a lot of similar functions, and I do not want to cut and paste the code. I want to generate the functions with code. This seems to be what macros are for in emacs lisp.

As a prototype example, we will make functions that raise a number to a power. We want functions like power-3 and power-4 that raise numbers to the third and fourth powers. We will define functions like this for the numbers 0-9.

Here we define the macro. i do not want to get into the nitty gritty details of macro definitions here.

(defmacro make-power-n (n)
 `(defun ,(intern (format "power-%s" n)) (arg) (expt arg ,n)))

(make-power-n 4)

(power-4 4)

Now we use the macro and mapcar on it onto a list of numbers. We have to eval the macro in the mapcar lambda function.

(defmacro make-power-n (n)
 `(defun ,(intern (format "power-%s" n)) (arg) (expt arg ,n)))

(mapcar (lambda (x) (eval `(make-power-n ,x))) '(0 1 2 3 4 5 6 7 8 9))
 
;; example of a few functions
(list (power-0 3) (power-1 3) (power-2 3))

It works! We created 10 functions in a little bit of code.

org-mode source

Org-mode version = 8.2.6

Language specific default headers for code blocks in org-mode

Sun, 26 Jan 2014 12:06:12 EST

I use code blocks in org-mode a lot. I usually code in Python, and in Python I usually write code that prints output which I want to see. So I almost always want the code blocks to return the output, and not the value of the last function. I have set my default header args like this:

org-babel-default-header-args

(:exports . both)

(:results . replace output)

(:session . none)

(:cache . no)

(:noweb . no)

(:hlines . no)

(:tangle . no)

However, I would prefer that emacs-lisp blocks use value for the results. I know I can get that by putting :results value in the code block header, but that annoys me. I learned today from http://orgmode.org/worg/org-contrib/babel/header-args.html that you can make language specific default headers!

This code in my init file sets emacs-lisp specific default headers:

(setq org-babel-default-header-args:emacs-lisp 
      (cons '(:results . "value")
            (assq-delete-all :results org-babel-default-header-args)))

That way I do not have type :results value at the top of every elisp block. Of course, if I want the output I could specify :results output in the block.

org-babel-default-header-args:emacs-lisp

(:results . value)

(:exports . both)

(:session . none)

(:cache . no)

(:noweb . no)

(:hlines . no)

(:tangle . no)

Problem solved!

On a related note, I find I write so many blocks of python and elisp I added these templates:

;; add <p for python expansion
(add-to-list 'org-structure-template-alist
             '("p" "#+BEGIN_SRC python\n?\n#+END_SRC" "<src lang=\"python\">\n?\n</src>"))

;; add <el for emacs-lisp expansion
(add-to-list 'org-structure-template-alist
             '("el" "#+BEGIN_SRC emacs-lisp\n?\n#+END_SRC" "<src lang=\"emacs-lisp\">\n?\n</src>"))

I probably could have also coded the :results header into those too. They add a tidbit of convenience so I do not have to type python or emacs-lisp after expanding a source block with <s.

org-mode source

Org-mode version = 8.2.5g

Another alternative to string templates

Sun, 26 Jan 2014 09:56:45 EST

In the last post I explored a way to expand a string template that was more readable than the usual format. Today I look at another approach where I use sexp expansions to accomplish the same thing. The idea is to embed lisp expressions and replace them by what they evaluate to.

In emacs-lisp, if we have a command in a string, we can "read" it, and then eval it.

Here we get the user-full-name:

(eval (read "user-full-name"))

John Kitchin

We can use this on variables too.

(setq some-variable "test")
(eval (read "some-variable"))

test

So, if we use a syntax to identify what to replace, we can substitute in the values. Let us try %() as the syntax.

(defun expand-template (s)
  "expand a template containing %() with the eval of its contents"
  (replace-regexp-in-string "%(\\([^)]+\\))"
                            (lambda (arg)
                              (format "%s" (eval (read (substring arg 2 -1))))) s))


(let ((key "kitchin-2014")
      (author "Kitchin, J. R.")
      (journal "HACS")
      (year "2014")
      (volume "1")
      (pages "1--10")
      (doi "10.1.1.109/hacs.1.10")
      (url "http://hacs.org/10.1.1.109/hacs.1.10")
      (pdf-dir "/home/jkitchin/pdfs")
      (template "
 :PROPERTIES:
  :Custom_ID: %(key)
  :AUTHOR: %(author
  :JOURNAL: %(journal)
  :YEAR: %(year)
  :VOLUME: %(volume)
  :PAGES: %(pages)
  :DOI: %(doi)
  :URL: %(url)
 :END:
[[cite:%(key)]] [[file:%(pdf-dir)/%(key).pdf][pdf]]\n\n"))

(expand-template template))

 :PROPERTIES:
  :Custom_ID: kitchin-2014
  :AUTHOR: Kitchin, J. R.
  :YEAR: 2014
  :VOLUME: 1
  :PAGES: 1--10
  :DOI: 10.1.1.109/hacs.1.10
  :URL: http://hacs.org/10.1.1.109/hacs.1.10
 :END:
[[cite:kitchin-2014]] [[file:/home/jkitchin/pdfs/kitchin-2014.pdf][pdf]]

That is pretty nice. I like it better than the plist expansion I used before. Presumably these variables would already be defined somewhere in your code.

I thought of trying this on a more complex expansion, and discovered a weakness in the regexp that finds the expansion values. It turns out to be simpler to use %{} as the delimiter than %(), because you may want nested parentheses. The regexp above does not correctly match sets of parentheses.

(defun expand-template (s)
  "expand a template containing %{} with the eval of its contents"
  (replace-regexp-in-string "%{\\([^}]+\\)}"
                            (lambda (arg)
                              (let ((sexp (substring arg 2 -1)))
                                (format "%s" (eval (read sexp))))) s))

(expand-template "2 * 2 = %{(* 2 2)}")

2 * 2 = 4

I am not sure this is a desirable way to make a template, with multiline code to be expanded, but at least this works!

(defun expand-template (s)
  "expand a template containing %{} with the eval of its contents"
  (replace-regexp-in-string "%{\\([^}]+\\)}"
                            (lambda (arg)
                              (let ((sexp (substring arg 2 -1)))
                                (format "%s" (eval (read sexp))))) s))

(expand-template "The result is %{(progn
  (if (> 4 3)
      'true
    'false))}")

The result is true

The regexp used in the expansion is not very robust. In particular if there is a } in the code, it will probably fail because the regexp does not match closing } correctly. Fixing that is beyond me right now!

org-mode source

Org-mode version = 8.2.5g

Alternatives to long complex format statements in emacs-lisp

Mon, 20 Jan 2014 09:50:26 EST

At one point I had a string I wanted to fill in with a bunch of variables.

(insert (format"
 :PROPERTIES:
  :Custom_ID: %s
  :AUTHOR: %s
  :JOURNAL: %s
  :YEAR: %s
  :VOLUME: %s
  :PAGES: %s
  :DOI: %s
  :URL: %s
 :END:
[[cite:%s]] [[file:%s/%s.pdf][pdf]]\n\n"
key author journal year volume pages doi url key jorg-bib-pdf-directory key ))

I find that very difficult to use, because it is tedious to make sure all the variables are in the right order, and it is difficult to change later. In Python, you would be able to put named expansions in, e.g. {author} and then used named arguments. That does not exist as far as I know in emacs-lisp.

Below is an alternatme approach that uses concat to construct this string.

(let ((key "kitchin-2014")
      (author "Kitchin, J. R.")
      (journal "HACS")
      (year "2014")
      (volume "1")
      (pages "1--10")
      (doi "10.1.1.109/hacs.1.10")
      (url "http://hacs.org/10.1.1.109/hacs.1.10")
      (jorg-bib-pdf-directory "/home/jkitchin/pdfs"))

(concat "
 :PROPERTIES:
  :Custom_ID: " key "
  :AUTHOR: " author "
  :JOURNAL: " journal "
  :YEAR: " year "
  :VOLUME: " volume "
  :PAGES: " pages "
  :DOI: " doi "
  :URL: " url "
 :END:
[[cite:" key "]] [[file:" jorg-bib-pdf-directory "/" key ".pdf][pdf]]\n\n"))

 :PROPERTIES:
  :Custom_ID: kitchin-2014
  :AUTHOR: Kitchin, J. R.
  :JOURNAL: HACS
  :YEAR: 2014
  :VOLUME: 1
  :PAGES: 1--10
  :DOI: 10.1.1.109/hacs.1.10
  :URL: http://hacs.org/10.1.1.109/hacs.1.10
 :END:
[[cite:kitchin-2014]] [[file:/home/jkitchin/pdfs/kitchin-2014.pdf][pdf]]

That is kind of interesting. It is a little tedious to use all the quotes. It seems like there should be soemthing like named expansions. Let us write one of our own. We will use a regular expression to find {:keyword} and a plist. There is a regexp to match this, and then we can take the characters from position 1 to the second to last character as the keyword. That is not beautiful to me, but it works here. Then we just get the keyword from the plist. The keywords in a plist are symbols, and we will have strings. We have to use the intern function to convert them to symbols.

(defun expand-template (s plist)
  "expand a template containing {:keyword} with the definitions in plist"
  (replace-regexp-in-string "{\\(:[^}]+\\)}" 
                            (lambda (arg) 
                              (let ((keyword (intern (substring arg 1 -1))))
                                (format "%s" (plist-get plist keyword)))) s))

(let ((template "
 :PROPERTIES:
  :Custom_ID: {:key}
  :AUTHOR: {:author}
  :JOURNAL: {:journal}
  :YEAR: {:year}
  :VOLUME: {:volume}
  :PAGES: {:pages}
  :DOI: {:doi}
  :URL: {:url}
 :END:
[[cite:{:key}]] [[file:{:pdf-dir}/{:key}.pdf][pdf]]\n\n"))

(expand-template template
                 '(:key "kitchin-2014"
                        :author "Kitchin, J. R."
                        :journal "HACS"
                        :year 2014
                        :volume 1
                        :pages "1--10"
                        :doi "10.1.1.109/hacs.1.10"
                        :url "http://hacs.org/10.1.1.109/hacs.1.10"
                        :pdf-dir "/home/jkitchin/pdfs")))

 :PROPERTIES:
  :Custom_ID: kitchin-2014
  :AUTHOR: Kitchin, J. R.
  :JOURNAL: HACS
  :YEAR: 2014
  :VOLUME: 1
  :PAGES: 1--10
  :DOI: 10.1.1.109/hacs.1.10
  :URL: http://hacs.org/10.1.1.109/hacs.1.10
 :END:
[[cite:kitchin-2014]] [[file:/home/jkitchin/pdfs/kitchin-2014.pdf][pdf]]

That is pretty close to what I am used to from python! I am surprised there aren't other solutions for this around. I looked, and couldn't find them.

org-mode source

Org-mode version = 8.2.5f

Send email to a list of users

Mon, 13 Jan 2014 18:41:55 EST

I have a need to send a lot of emails to users in my class. I have to send each student an email containing there userid and a password assigned to them. I have a list of these, so the strategy is to create a function that will email that information to one user, and then use mapcar to apply the function to each pair in a list. First, we work out a function that will send one email to one user.

(defun send-mail (userid password)
  "send email to userid@andrew.cmu.edu containing their password"
  (interactive)
  (mail)
  (mail-to)
  (insert (format "%s@andrew.cmu.edu" userid))
  (mail-subject)
  (insert "[06-640] account information")
  (mail-text)
  (insert (format "
An account has been created on gilgamesh.cheme.cmu.edu
userid: %s
password: %s" userid password))
  (mail-send-and-exit))

(send-mail "jkitchin" "trustme99")

That worked well. I ran the block and got the email.

Now, suppose I have this data:

userid	password
user1	trustme99
user2	foolme99
user3	blameme99

We can pass that to a source block as a list of lists that will look like this:

 ((user1 trustme99) (user2 foolme99) (user3 blameme99))

Then, we can use a mapcar to process each element. Here I use a dummy function with two arguments. If I substitute the function above, each of these users would get an email.

(defun fun (a b)
  (princ (format "user: %s\npassword: %s\n" a b)))

(mapcar (lambda (x) (fun (car x) (cadr x))) data)

user: user1
password: trustme99
user: user2
password: foolme99
user: user3
password: blameme99

I am not sure that is the best way to get the first and second elements in the list element. It looks funny to me, but it works fine. the alternative is not much prettier:

(defun fun (a b)
  (princ (format "user: %s\npassword: %s\n" a b)))

(mapcar (lambda (x) (fun (nth 0 x) (nth 1 x))) data)

user: user1
password: trustme99
user: user2
password: foolme99
user: user3
password: blameme99

org-mode source

An improved index function for emacs-lisp

Sun, 22 Dec 2013 12:01:48 EST

I previously worked out an index function for a list of strings in emacs-lisp, but at the end I showed it would not work for arbitrary elements of a list. Here is an exercise to improve on that. The goal is a function that looks like this:

(index 1 '("a" 2 1 "b"))

that would return 2 in this case. Last time I used string=, which is why I could not find a number in the list. This time, we will use equal (see http://www.gnu.org/software/emacs/manual/html_node/elisp/Equality-Predicates.html ) which compares components of objects for equality. That should let us find arbitrary objects in a list.

Here is our improved function:

(defun index (object list)
  "return the index of object in list"
  (let ((counter 0)
        (found nil))
    (catch 'finished
      (dolist (listelement list counter)
        (if (equal object listelement)
            (progn
              (setq found t)
              (throw 'finished counter))
          ;; else increment counter
          (incf counter)))
    ;; if we found it return counter otherwise return nil
    (if found counter nil))))

Now, let us test some examples:

(index 1 '("a" 2 1 "b"))

No problem finding a number in a list.

(index "b" '("a" 2 1 "b"))

How about something more complicated, like a list in a list?

(index '(1 2) '("a" 2 1 (1 2) "b"))

That looks good.

(princ (index '(1 2) '("a" 2 1 (2 (1 2)) "b")))

nil

Note, we do not find the nested object. That is ok, the location of that object would require two indices, which this function is not designed for.

Here we consider an object of an a-list

(index '("nut" . "acorn") '(("nut" . "acorn") ("fruit" . "apple")))

I am not quite sure how you would use that, but it does illustrate the generality of the index function!

org-mode source

Summarizing org-files in a report

Tue, 30 Apr 2013 08:30:01 EDT

This is an example of using emacs-lisp to extract pieces of information about a bunch of org-files into a single report, as well as aggregating data from those files. The scenario where this would likely be useful is if you have a set of org-files that contain information, e.g. from a bunch of different calculations, or from documents turned in by different students, and you want to aggregate the results into a report.

In this example, I have a set of org-files in this directory that contain simulated homework assignments turned in. The files in this example all look something like this. Each heading corresponds to a problem, and there is a properties drawer for each heading that contains the grade.

We will create a navigation document that facilitates reviewing each of the files, as well as collecting the grades from the files. Here is what a typical file looks like:

#+PROPERTY: NAME Ellen Donnte
* 1a
  :PROPERTIES:
  :lettergrade: A
  :END:
* 1b
  :PROPERTIES:
  :lettergrade: R
  :END:
* 2
  :PROPERTIES:
  :lettergrade: A
  :END:
#+BEGIN_SRC emacs-lisp
(prin1 42)
#+END_SRC

#+RESULTS:
: 42

* 3
  :PROPERTIES:
  :lettergrade: C
  :END:

1 Creating a navigation document

In this section we write some code that creates text with a link to each file we need to review. This is something I imagine we would do after all the files have been turned in and collected. This buffer would facilitate navigating all the files, and checking them off. First we create checkboxes. All this does is create an easy to use navigation document that facilitates opening the files, grading them, and marking them as done.¹

(require 'find-lisp)

(dolist (fname (find-lisp-find-files "." "\\HW1.org$") nil)
  (princ (format "- [ ] [[file:%s][%s]]\n" fname (file-name-nondirectory fname))))

In the results above I have marked one entry as completed.

It might be preferrable to have links to places in the file, e.g. to problem 2.

(require 'find-lisp)

(dolist (fname (find-lisp-find-files "." "\\HW1.org$") nil)
  (princ (format "- [ ] [[file:%s::*2][%s - problem 2]]\n" fname (file-name-nondirectory fname))))

2 Aggregating properties

Our goal here is to use emacs-lisp to aggregate the letter grades from all the assignments into a table. This would be done after all the files have been reviewed. First, we write a function that gets the data we want. The function should take a filename, and return the letter grade for a problem, e.g. (get-letter-grade filename problem) -> lettergrade. Then, we will map that function onto a list of files.

(require 'find-lisp)

(defun get-letter-grade (filename problem-name)
  "Open filename, get the grade associated with the heading of problem-name."
  (with-temp-buffer
    (insert-file-contents filename)
    (let ((studentname nil)
           (lettergrade nil))
      (org-ctrl-c-ctrl-c) ; this is needed to read the NAME property!
      (setq studentname (org-entry-get (point) "NAME" t))
      (goto-char (point-min))
      (search-forward problem-name)
      (setq lettergrade (org-entry-get (point) "lettergrade"))
      
      (princ (format "|%s|%s|%s|\n" studentname problem-name lettergrade)))))

(princ "#+ATTR_HTML: :border 2 :rules all :frame border\n")
(princ "#+tblname: GRADES\n")
(princ "| Name | Problem | Grade |\n|-\n")

(dolist (problem-name '("1a" "1b" "2" "3") nil)
  (mapcar (lambda (fname) (get-letter-grade fname problem-name)) 
      (find-lisp-find-files "." "\\HW1.org$")))


Name	Problem	Grade
Slim Shady	1a	A
John Doe	1a	B
Jim Vicious	1a	B
Ellen Donnte	1a	A
Slim Shady	1b	B
John Doe	1b	A
Jim Vicious	1b	C
Ellen Donnte	1b	R
Slim Shady	2	R
John Doe	2	B
Jim Vicious	2	R
Ellen Donnte	2	A
Slim Shady	3	R
John Doe	3	B
Jim Vicious	3	D
Ellen Donnte	3	C

You could imagine some other kind of aggregating or analysis here too. Now that we have that table, we can use it in other analysis. Let us count the number of A's.

(save-excursion
  (goto-char (point-min))
  (search-forward-regexp "^#\\+tblname: GRADES")
  (next-line)
  (let ((A-COUNT 0)
        (letter-grade nil)
        ;; cddr is used to remove the first two rows of the table
        (data (cddr (org-table-to-lisp))))
    (dolist (entry data nil)
      (setq letter-grade (nth 2 entry))
      (if (equal  letter-grade "A")
          (incf A-COUNT)))
    (princ (format "%s A's counted" A-COUNT))))

4 A's counted

Since we are in org-mode, we can use the table directly! Let us do that and count the number of R's.

(let ((COUNT 0)
      (letter-grade nil))
    (dolist (entry (cddr data) nil)
      (setq letter-grade (nth 2 entry))
      (if (equal  letter-grade "R")
          (incf COUNT)))
    (princ (format "%s R's counted" COUNT))))

4 R's counted

3 Aggregating sections of org-files into one file

Another scenario that may be interesting is to collect all of the responses in a single document. This might be useful to show examples in class, or to review all the problems to see if there are common errors. Here we collect Problem 2.

(require 'find-lisp)

(generate-new-buffer "Problem 2")
(set-buffer "Problem 2")
(insert "#+TITLE: Summary of problem 2\n")

(dolist (fname (find-lisp-find-files "." "\\HW1.org$") nil)
  (save-excursion
    (goto-char (point-max))
    (org-mode)
    (with-temp-buffer 
      (insert-file-contents fname)
      (org-mode)
      (goto-char (point-min))
      (setq studentname (org-entry-get nil "NAME" t))
      (search-forward "* 2")
      (org-narrow-to-subtree)
      (forward-line) ; skip heading
      (setq text (buffer-substring (point) (point-max))))
    (insert (format "* 2 - %s\n" studentname))
    (insert text "\n")
          
    (search-backward "* 2")
    (org-entry-put nil "NAME" studentname)
    (org-entry-put nil "source" (format "[[%s][link]]" fname))
))

(switch-to-buffer "Problem 2")
(org-mode) ; switch to org-mode in that buffer

;; print the lines to see what we got
(dolist (line (split-string (buffer-string) "\n") nil) (princ (format ": %s\n" line)))

: #+TITLE: Summary of problem 2
: * 2 - Slim Shady
:   :PROPERTIES:
:   :lettergrade: R
:   :NAME:     Slim Shady
:   :source:   [[c:/Users/jkitchin/Dropbox/blogofile-jkitchin.github.com/_blog/org-report/Slim-Shady-HW1.org][link]]
:   :END:
: #+BEGIN_SRC python
: print 3
: 
: #+END_SRC
: 
: #+RESULTS:
: : 3
: 
: * 2 - John Doe
:   :PROPERTIES:
:   :lettergrade: B
:   :NAME:     John Doe
:   :source:   [[c:/Users/jkitchin/Dropbox/blogofile-jkitchin.github.com/_blog/org-report/John-Doe-HW1.org][link]]
:   :END:
: Here is my solution
: #+BEGIN_SRC python
: print 4
: #+END_SRC
: 
: #+RESULTS:
: : 4
: 
: * 2 - Jim Vicious
:   :PROPERTIES:
:   :lettergrade: R
:   :NAME:     Jim Vicious
:   :source:   [[c:/Users/jkitchin/Dropbox/blogofile-jkitchin.github.com/_blog/org-report/Jim-Vicious-HW1.org][link]]
:   :END:
: I could not figure this out
: * 2 - Ellen Donnte
:   :PROPERTIES:
:   :lettergrade: A
:   :NAME:     Ellen Donnte
:   :source:   [[c:/Users/jkitchin/Dropbox/blogofile-jkitchin.github.com/_blog/org-report/Ellen-Donnte-HW1.org][link]]
:   :END:
: #+BEGIN_SRC emacs-lisp
: (prin1 42)
: #+END_SRC
: 
: #+RESULTS:
: : 42

I am not super thrilled with this approach. It feels too much like hand-crafting a result, but it does show some possibilities!

org-mode source

An index function for strings in emacs-lisp

Tue, 05 Mar 2013 19:28:30 EST

I could not find an index function for strings in emacs-lisp. The position function seems to work for numbers, but not strings. Here is a version that works on strings.

(defun index (item list)
  "return index of item in list or nil"
  (let ((counter 0)
        (found nil))
    (dolist (listelement list counter)
      (if (string= item listelement)
        (progn 
          (setq found t)
          (return counter)) ; exit the loop
        ;; else increment counter
        (incf counter)))
    ;; if we found it return counter otherwise return nil
    (if found counter nil)))

index

Here are some example uses:

(index "test" '("a" "test" "y"))

(index "z" '("a" "b" "z"))

(index "testy" '("a" "test" "y"))

nil

This raises an error because we use string=.

(index 1 '("a" "test" "y" 1))

org-mode source

The Kitchin Research Group

A partial symbolic numeric solver in emacs-lisp

Table of Contents

1 The Newton solver in emacs-lisp

Collecting entries from files in a directory

Autogenerating functions in emacs-lisp

Language specific default headers for code blocks in org-mode

Another alternative to string templates

Alternatives to long complex format statements in emacs-lisp

Send email to a list of users

An improved index function for emacs-lisp

Summarizing org-files in a report

1 Creating a navigation document

2 Aggregating properties

3 Aggregating sections of org-files into one file

An index function for strings in emacs-lisp