Summarizing org-files in a report

Posted April 30, 2013 at 08:30 AM | categories: emacs-lisp, org-mode | tags:

Updated April 30, 2013 at 09:49 AM

This is an example of using emacs-lisp to extract pieces of information about a bunch of org-files into a single report, as well as aggregating data from those files. The scenario where this would likely be useful is if you have a set of org-files that contain information, e.g. from a bunch of different calculations, or from documents turned in by different students, and you want to aggregate the results into a report.

In this example, I have a set of org-files in this directory that contain simulated homework assignments turned in. The files in this example all look something like this. Each heading corresponds to a problem, and there is a properties drawer for each heading that contains the grade.

We will create a navigation document that facilitates reviewing each of the files, as well as collecting the grades from the files. Here is what a typical file looks like:

#+PROPERTY: NAME Ellen Donnte
* 1a
  :PROPERTIES:
  :lettergrade: A
  :END:
* 1b
  :PROPERTIES:
  :lettergrade: R
  :END:
* 2
  :PROPERTIES:
  :lettergrade: A
  :END:
#+BEGIN_SRC emacs-lisp
(prin1 42)
#+END_SRC

#+RESULTS:
: 42

* 3
  :PROPERTIES:
  :lettergrade: C
  :END:

1 Creating a navigation document

In this section we write some code that creates text with a link to each file we need to review. This is something I imagine we would do after all the files have been turned in and collected. This buffer would facilitate navigating all the files, and checking them off. First we create checkboxes. All this does is create an easy to use navigation document that facilitates opening the files, grading them, and marking them as done.¹

(require 'find-lisp)

(dolist (fname (find-lisp-find-files "." "\\HW1.org$") nil)
  (princ (format "- [ ] [[file:%s][%s]]\n" fname (file-name-nondirectory fname))))

In the results above I have marked one entry as completed.

It might be preferrable to have links to places in the file, e.g. to problem 2.

(require 'find-lisp)

(dolist (fname (find-lisp-find-files "." "\\HW1.org$") nil)
  (princ (format "- [ ] [[file:%s::*2][%s - problem 2]]\n" fname (file-name-nondirectory fname))))

2 Aggregating properties

Our goal here is to use emacs-lisp to aggregate the letter grades from all the assignments into a table. This would be done after all the files have been reviewed. First, we write a function that gets the data we want. The function should take a filename, and return the letter grade for a problem, e.g. (get-letter-grade filename problem) -> lettergrade. Then, we will map that function onto a list of files.

(require 'find-lisp)

(defun get-letter-grade (filename problem-name)
  "Open filename, get the grade associated with the heading of problem-name."
  (with-temp-buffer
    (insert-file-contents filename)
    (let ((studentname nil)
           (lettergrade nil))
      (org-ctrl-c-ctrl-c) ; this is needed to read the NAME property!
      (setq studentname (org-entry-get (point) "NAME" t))
      (goto-char (point-min))
      (search-forward problem-name)
      (setq lettergrade (org-entry-get (point) "lettergrade"))
      
      (princ (format "|%s|%s|%s|\n" studentname problem-name lettergrade)))))

(princ "#+ATTR_HTML: :border 2 :rules all :frame border\n")
(princ "#+tblname: GRADES\n")
(princ "| Name | Problem | Grade |\n|-\n")

(dolist (problem-name '("1a" "1b" "2" "3") nil)
  (mapcar (lambda (fname) (get-letter-grade fname problem-name)) 
      (find-lisp-find-files "." "\\HW1.org$")))


Name	Problem	Grade
Slim Shady	1a	A
John Doe	1a	B
Jim Vicious	1a	B
Ellen Donnte	1a	A
Slim Shady	1b	B
John Doe	1b	A
Jim Vicious	1b	C
Ellen Donnte	1b	R
Slim Shady	2	R
John Doe	2	B
Jim Vicious	2	R
Ellen Donnte	2	A
Slim Shady	3	R
John Doe	3	B
Jim Vicious	3	D
Ellen Donnte	3	C

You could imagine some other kind of aggregating or analysis here too. Now that we have that table, we can use it in other analysis. Let us count the number of A's.

(save-excursion
  (goto-char (point-min))
  (search-forward-regexp "^#\\+tblname: GRADES")
  (next-line)
  (let ((A-COUNT 0)
        (letter-grade nil)
        ;; cddr is used to remove the first two rows of the table
        (data (cddr (org-table-to-lisp))))
    (dolist (entry data nil)
      (setq letter-grade (nth 2 entry))
      (if (equal  letter-grade "A")
          (incf A-COUNT)))
    (princ (format "%s A's counted" A-COUNT))))

4 A's counted

Since we are in org-mode, we can use the table directly! Let us do that and count the number of R's.

(let ((COUNT 0)
      (letter-grade nil))
    (dolist (entry (cddr data) nil)
      (setq letter-grade (nth 2 entry))
      (if (equal  letter-grade "R")
          (incf COUNT)))
    (princ (format "%s R's counted" COUNT))))

4 R's counted

3 Aggregating sections of org-files into one file

Another scenario that may be interesting is to collect all of the responses in a single document. This might be useful to show examples in class, or to review all the problems to see if there are common errors. Here we collect Problem 2.

(require 'find-lisp)

(generate-new-buffer "Problem 2")
(set-buffer "Problem 2")
(insert "#+TITLE: Summary of problem 2\n")

(dolist (fname (find-lisp-find-files "." "\\HW1.org$") nil)
  (save-excursion
    (goto-char (point-max))
    (org-mode)
    (with-temp-buffer 
      (insert-file-contents fname)
      (org-mode)
      (goto-char (point-min))
      (setq studentname (org-entry-get nil "NAME" t))
      (search-forward "* 2")
      (org-narrow-to-subtree)
      (forward-line) ; skip heading
      (setq text (buffer-substring (point) (point-max))))
    (insert (format "* 2 - %s\n" studentname))
    (insert text "\n")
          
    (search-backward "* 2")
    (org-entry-put nil "NAME" studentname)
    (org-entry-put nil "source" (format "[[%s][link]]" fname))
))

(switch-to-buffer "Problem 2")
(org-mode) ; switch to org-mode in that buffer

;; print the lines to see what we got
(dolist (line (split-string (buffer-string) "\n") nil) (princ (format ": %s\n" line)))

: #+TITLE: Summary of problem 2
: * 2 - Slim Shady
:   :PROPERTIES:
:   :lettergrade: R
:   :NAME:     Slim Shady
:   :source:   [[c:/Users/jkitchin/Dropbox/blogofile-jkitchin.github.com/_blog/org-report/Slim-Shady-HW1.org][link]]
:   :END:
: #+BEGIN_SRC python
: print 3
: 
: #+END_SRC
: 
: #+RESULTS:
: : 3
: 
: * 2 - John Doe
:   :PROPERTIES:
:   :lettergrade: B
:   :NAME:     John Doe
:   :source:   [[c:/Users/jkitchin/Dropbox/blogofile-jkitchin.github.com/_blog/org-report/John-Doe-HW1.org][link]]
:   :END:
: Here is my solution
: #+BEGIN_SRC python
: print 4
: #+END_SRC
: 
: #+RESULTS:
: : 4
: 
: * 2 - Jim Vicious
:   :PROPERTIES:
:   :lettergrade: R
:   :NAME:     Jim Vicious
:   :source:   [[c:/Users/jkitchin/Dropbox/blogofile-jkitchin.github.com/_blog/org-report/Jim-Vicious-HW1.org][link]]
:   :END:
: I could not figure this out
: * 2 - Ellen Donnte
:   :PROPERTIES:
:   :lettergrade: A
:   :NAME:     Ellen Donnte
:   :source:   [[c:/Users/jkitchin/Dropbox/blogofile-jkitchin.github.com/_blog/org-report/Ellen-Donnte-HW1.org][link]]
:   :END:
: #+BEGIN_SRC emacs-lisp
: (prin1 42)
: #+END_SRC
: 
: #+RESULTS:
: : 42

I am not super thrilled with this approach. It feels too much like hand-crafting a result, but it does show some possibilities!

org-mode source

The Kitchin Research Group

Chemical Engineering at Carnegie Mellon University