The Kitchin Research Group

A few people have asked me about how I generate the posts in this blog, so I thought I would try to capture that process here. The blog itself is generated by blogofile ¹. Blogofile is a pure python, static blog generator. I will discuss the setup of that another time. Here I will focus on how the posts are written and published to the blog.

The posts in this blog were created using Emacs and org-mode ². Basically I create a heading in org-mode (any level will do, but I usually make them top headings, i.e. with one *). The post can contain all the markup in org-mode that can be exported to HTML, including equations, code, links, and images. I set org-properties of the heading for the categories and tags of the post.

Blogofile wants a post to be a file in a specific directory called _posts . The file must have a YAML heading, followed by the body of the html that will be the post. So, to create a blogofile post, we have to export the org-mode heading to a properly formatted post file, in the _posts directory of the blogofile root. org-mode has an excellent export system for generating html, but I had to write some customized emacs-lisp code to get exactly what I want. That code is available at https://github.com/jkitchin/jmax/blob/prelude/blogofile.el. I am sure this code is not written most elegantly, but it has worked for me for about 186 posts (counting this one). This file gets loaded in my init.el file. It is very likely you will have to run a fairly modern version of org-mode (at least 8+) to do this. You will also probably need to configure your org-mode like mine to get similar looking results. My current setup builds off of Prelude³. The contents of the personal directory can be found here https://github.com/jkitchin/jmax/tree/prelude. I keep a git version of org in there.

Basically the blogofile code does the following:

1. Grabs the content of the current org-mode heading
2. Constructs the YAML heading from information in properties of the heading, and the heading title
3. Exports the heading body into an HTML string. If your post has images in it, there is an additional step required. In blogofile.el, I temporarily redefined org-html–format-image so that it would copy the image to the images directory in the blogofile root, and return a url that would be correct when it is published to github. My blog uses Mathjax, so equations are automatically handled correctly.
4. Combine the YAML heading and HTML string and save it to a file in the _posts directory.
5. Enter a few new or update properties in the heading about when it was created and/or updated
6. I wanted the org-code for each post to be published on the blog so I also save the org-code to a directory in the blogofile root and add a link to it in the HTML file.

All of that action is written in a single command called bf-blogpost which I bind to the F10 key. So, after I am done with my blogpost, I press F10, and the post is created in my blogofile root directory. I find that pretty convenient.

To publish it to github, I have to do the following:

1. Change into the blogofile root directory, and run "blogofile build" which generates the html files from the new _post. I can locally serve the blog to check out the post if I want.
2. Copy the new files to a directory called _deploy, which is is the master branch of the repository at https://github.com/jkitchin/jkitchin.github.com, and only contains the published blog HTML
3. Commit the new changes to the repository and push them to github. A few minutes later, the new post is visible on the website, which is hosted by GitHUB (Thanks GitHUB!).

This all works well on my windows machines. I don't know why, but I cannot build the blog on a Linux machine. Maybe because of mixed dos/unix line endings, or something silly like that.

After spending some time getting this workflow to work, I find it pretty convenient. I love that I can write the posts in org-mode, because most of them are about using python or emacs-lisp, and I like to have the code and output together with my narrative text. I also like that the actual org-code is available as a link in each post. That way if I don't remember how I did something I can always go back to the source, or my students can learn how I used org-mode.

One thing I have not figured out yet is how to have uploads, e.g. if I use a datafile in a post, it would be nice to have the blogpost function automatically copy that file to the right place so that the link to it works in the blog. I haven't needed that too often, so it has not been a high priority.

Copyright (C) 2013 by John Kitchin. See the License for information about copying.

org-mode source

Continuing my work in using org-mode as a structured markup language to encode data in files, today we look at how to extract keywords and properties from org-files. This will enable me to store data in the org-files, and then retrieve it later from code. For example, you may want to store the email address, or a student-id in the file so you can connect some other property such as a grade to it.

First we consider the built-in org commands to access aspects of the file properties. The org-export-get-environment command returns a plist of properties of the file.

(org-export-get-environment)

(:author ("John Kitchin") :creator "Generated by Org mode 7.9.4 in Emacs 24.0.92.1." :date nil :description nil :email "johnrkitchin@gmail.com" :exclude-tags ("noexport") :headline-levels 3 :keywords nil :language "en" :preserve-breaks nil :section-numbers t :select-tags ("export") :time-stamp-file t :title "file-options" :with-archived-trees headline :with-author t :with-clocks nil :with-creator comment :with-drawers nil :with-email nil :with-emphasize t :with-entities t :with-fixed-width t :with-footnotes t :with-inlinetasks t :with-plannings nil :with-priority nil :with-special-strings t :with-sub-superscript t :with-toc t :with-tables t :with-tags not-in-toc :with-tasks t :with-timestamps t :with-todo-keywords t :input-file "c:/Users/jkitchin/Dropbox/blogofile-jkitchin.github.com/_blog/file-options.org" :footnote-definition-alist nil :id-alist nil :macro-modification-time "(eval (format-time-string \"$1\" '(20870 38242)))" :macro-input-file "file-options.org" :macro-date "(eval (format-time-string \"$1\"))" :macro-time "(eval (format-time-string \"$1\"))" :macro-property "(eval (org-entry-get nil \"$1\" 'selective))" :back-end nil :translate-alist nil)

We can access a particular property like this:

(plist-get (org-export-get-environment) ':email)

"johnrkitchin@gmail.com"

Most of those properties are set externally to this file. It may be useful to set your own options in a file, e.g. we may want to set a STUDENT-ID tag in our file.

#+STUDENT-ID: jkitchin

We want to be able to extract the value of that keyword in emacs-lisp code. Here is one org way to do that. This code parses the buffer to get a list of elements, then applies a lambda function that makes a (key . value) to all the keyword elements, finally returning an alist. The second function gets the value for a specific key.

; suggested by Nicolas Goaziou
(defun jk-org-kwds ()
  "parse the buffer and return a cons list of (property . value)
from lines like:
#+PROPERTY: value"
  (org-element-map (org-element-parse-buffer 'element) 'keyword
                   (lambda (keyword) (cons (org-element-property :key keyword)
                                           (org-element-property :value keyword)))))

(defun jk-org-kwd (KEYWORD)
  "get the value of a KEYWORD in the form of #+KEYWORD: value"
  (cdr (assoc KEYWORD (jk-org-kwds))))

(jk-org-kwd "STUDENT-ID")

jkitchin

That is serious org-fu there. Here is an alternative approach that uses searching and regular expressions. The idea is to use a case-insensitive regular expression search in the buffer to grab the value.

(defun jk-get-file-keyword (KEYWORD)
  "get the value from a line like this
#+KEYWORD: value
in a file."
  (interactive)
  (let ((case-fold-search t)
        (re (format "^#\\+%s:[ \t]+\\([^\t\n]+\\)" KEYWORD)))
    (if (not (save-excursion
               (or (re-search-forward re nil t)
                   (re-search-backward re nil t))))
        (error (format "No line containing #+%s: value found" KEYWORD)))
    (match-string 1)))

(jk-get-file-keyword "STUDENT-ID")

jkitchin

Both methods basically do the same thing, and enable you to store data in org-files that is easily retrievable.

Copyright (C) 2013 by John Kitchin. See the License for information about copying.

org-mode source

This is an example of using emacs-lisp to extract pieces of information about a bunch of org-files into a single report, as well as aggregating data from those files. The scenario where this would likely be useful is if you have a set of org-files that contain information, e.g. from a bunch of different calculations, or from documents turned in by different students, and you want to aggregate the results into a report.

In this example, I have a set of org-files in this directory that contain simulated homework assignments turned in. The files in this example all look something like this. Each heading corresponds to a problem, and there is a properties drawer for each heading that contains the grade.

We will create a navigation document that facilitates reviewing each of the files, as well as collecting the grades from the files. Here is what a typical file looks like:

#+PROPERTY: NAME Ellen Donnte
* 1a
  :PROPERTIES:
  :lettergrade: A
  :END:
* 1b
  :PROPERTIES:
  :lettergrade: R
  :END:
* 2
  :PROPERTIES:
  :lettergrade: A
  :END:
#+BEGIN_SRC emacs-lisp
(prin1 42)
#+END_SRC

#+RESULTS:
: 42

* 3
  :PROPERTIES:
  :lettergrade: C
  :END:

(require 'find-lisp)

(dolist (fname (find-lisp-find-files "." "\\HW1.org$") nil)
  (princ (format "- [ ] [[file:%s][%s]]\n" fname (file-name-nondirectory fname))))

(require 'find-lisp)

(dolist (fname (find-lisp-find-files "." "\\HW1.org$") nil)
  (princ (format "- [ ] [[file:%s::*2][%s - problem 2]]\n" fname (file-name-nondirectory fname))))

(require 'find-lisp)

(defun get-letter-grade (filename problem-name)
  "Open filename, get the grade associated with the heading of problem-name."
  (with-temp-buffer
    (insert-file-contents filename)
    (let ((studentname nil)
           (lettergrade nil))
      (org-ctrl-c-ctrl-c) ; this is needed to read the NAME property!
      (setq studentname (org-entry-get (point) "NAME" t))
      (goto-char (point-min))
      (search-forward problem-name)
      (setq lettergrade (org-entry-get (point) "lettergrade"))
      
      (princ (format "|%s|%s|%s|\n" studentname problem-name lettergrade)))))

(princ "#+ATTR_HTML: :border 2 :rules all :frame border\n")
(princ "#+tblname: GRADES\n")
(princ "| Name | Problem | Grade |\n|-\n")

(dolist (problem-name '("1a" "1b" "2" "3") nil)
  (mapcar (lambda (fname) (get-letter-grade fname problem-name)) 
      (find-lisp-find-files "." "\\HW1.org$")))

(save-excursion
  (goto-char (point-min))
  (search-forward-regexp "^#\\+tblname: GRADES")
  (next-line)
  (let ((A-COUNT 0)
        (letter-grade nil)
        ;; cddr is used to remove the first two rows of the table
        (data (cddr (org-table-to-lisp))))
    (dolist (entry data nil)
      (setq letter-grade (nth 2 entry))
      (if (equal  letter-grade "A")
          (incf A-COUNT)))
    (princ (format "%s A's counted" A-COUNT))))

4 A's counted

(let ((COUNT 0)
      (letter-grade nil))
    (dolist (entry (cddr data) nil)
      (setq letter-grade (nth 2 entry))
      (if (equal  letter-grade "R")
          (incf COUNT)))
    (princ (format "%s R's counted" COUNT))))

4 R's counted

(require 'find-lisp)

(generate-new-buffer "Problem 2")
(set-buffer "Problem 2")
(insert "#+TITLE: Summary of problem 2\n")

(dolist (fname (find-lisp-find-files "." "\\HW1.org$") nil)
  (save-excursion
    (goto-char (point-max))
    (org-mode)
    (with-temp-buffer 
      (insert-file-contents fname)
      (org-mode)
      (goto-char (point-min))
      (setq studentname (org-entry-get nil "NAME" t))
      (search-forward "* 2")
      (org-narrow-to-subtree)
      (forward-line) ; skip heading
      (setq text (buffer-substring (point) (point-max))))
    (insert (format "* 2 - %s\n" studentname))
    (insert text "\n")
          
    (search-backward "* 2")
    (org-entry-put nil "NAME" studentname)
    (org-entry-put nil "source" (format "[[%s][link]]" fname))
))

(switch-to-buffer "Problem 2")
(org-mode) ; switch to org-mode in that buffer

;; print the lines to see what we got
(dolist (line (split-string (buffer-string) "\n") nil) (princ (format ": %s\n" line)))

: #+TITLE: Summary of problem 2
: * 2 - Slim Shady
:   :PROPERTIES:
:   :lettergrade: R
:   :NAME:     Slim Shady
:   :source:   [[c:/Users/jkitchin/Dropbox/blogofile-jkitchin.github.com/_blog/org-report/Slim-Shady-HW1.org][link]]
:   :END:
: #+BEGIN_SRC python
: print 3
: 
: #+END_SRC
: 
: #+RESULTS:
: : 3
: 
: * 2 - John Doe
:   :PROPERTIES:
:   :lettergrade: B
:   :NAME:     John Doe
:   :source:   [[c:/Users/jkitchin/Dropbox/blogofile-jkitchin.github.com/_blog/org-report/John-Doe-HW1.org][link]]
:   :END:
: Here is my solution
: #+BEGIN_SRC python
: print 4
: #+END_SRC
: 
: #+RESULTS:
: : 4
: 
: * 2 - Jim Vicious
:   :PROPERTIES:
:   :lettergrade: R
:   :NAME:     Jim Vicious
:   :source:   [[c:/Users/jkitchin/Dropbox/blogofile-jkitchin.github.com/_blog/org-report/Jim-Vicious-HW1.org][link]]
:   :END:
: I could not figure this out
: * 2 - Ellen Donnte
:   :PROPERTIES:
:   :lettergrade: A
:   :NAME:     Ellen Donnte
:   :source:   [[c:/Users/jkitchin/Dropbox/blogofile-jkitchin.github.com/_blog/org-report/Ellen-Donnte-HW1.org][link]]
:   :END:
: #+BEGIN_SRC emacs-lisp
: (prin1 42)
: #+END_SRC
: 
: #+RESULTS:
: : 42

Copyright (C) 2013 by John Kitchin. See the License for information about copying.

org-mode source