The Kitchin Research Group

When you publish a Matlab m-file to HTML, Matlab includes the m-file source as an html comment in the output. They also provide a nice function called grabcode that will take a url, and open the source code in the editor. Today, we try a similar approach for org-mode.

This post is not totally self-contained. I have my own emacs-lisp module that converts org-mode to blogofile posts, and so far I have not made it broadly available. This is also a super exploratory idea, so I am just going to show the changes I need to make to my setup to get to the evaluation of the idea.

The idea is pretty simple, we just insert the current buffer string into an HTML comment. I just modify the bf-get-post-html function lightly to do that. This is a somewhat pathological example since there are html comments in the post! So, we will encode all the dashes to get around that.

(require 'browse-url)
(defun bf-get-post-html ()
  "Return a string containing the YAML header, the post html, my
copyright line, and a link to the org-source code."
  (interactive)
  (let ((org-source (buffer-string))
        (url-to-org (bf-get-url-to-org-source))
        (yaml (bf-get-YAML-heading))
        (body (bf-get-HTML)))

    (with-temp-buffer
      (insert yaml)
      (insert body)
      (insert
       (format "<p>Copyright (C) %s by John Kitchin. See the <a href=\"/copying.html\">License</a> for information about copying.<p>"
               (format-time-string "%Y")))
      (insert (format "<p><a href=\"%s\">org-mode source</a><p>"
                      url-to-org))
      (insert (format "<p>Org-mode version = %s</p>" (org-version)))
      ;; this is the only new code we need to add.
      (insert (format "
<!--
  ##### SOURCE BEGIN #####
%s
##### SOURCE END #####
-->" (browse-url-url-encode-chars org-source "[-]")))
      ;; return value
      (buffer-string))))

By itself, that has limited value to me. So, let's also create a grab-org-source function to get the embedded source and open it in a buffer. This might be a naive approach, we just use a regexp to find the source boundaries and open it in a new buffer. We have to unescape the dashes, which appear as %2D in the comments. Here is our function.

(defun grab-org-source (url)
  "Extract org-source from URL to a buffer named *grab-org-source*."
  (interactive "sURL: ")
  (switch-to-buffer (get-buffer-create "*grab-org-source*"))
  (erase-buffer)
  (org-mode)
  (insert
   (with-current-buffer
       (url-retrieve-synchronously url)
     (let (start)
       (re-search-forward
        "
<!--
  ##### SOURCE BEGIN #####
" nil t)
       (setq start (point))
       (re-search-forward "##### SOURCE END #####
-->" nil t)
       (buffer-substring start (match-beginning 0)))))
  (goto-char (point-min))
  (while (search-forward "%2D" nil t)
    (replace-match "-"))
  (goto-char (point-min)))

This concludes my basic proof of concept. I think there is a general escaping challenge in this approach, because it isn't clear if you can put really arbitrary stuff in an html comment, e.g. you cannot put –>! I am going to try incorporating this into my posts and see what other issues come up in the future.

Copyright (C) 2015 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.10

In a previous post I eliminated one annoying problem with sessions, which was getting rid of extraneous Python interpreter characters in the output. Another thing that has bothered me is when you close Emacs, or even the session buffer, the session is, of course, lost. That means when you reopen the file, you have to run each block in order to continue your work. There does not seem to be a selective way to do this in org. So, in this post, we consider a simple approach to automate that. We want a function that will run all the blocks in a current session that are above the current point.

The idea is we will go to the beginning of the buffer, find all blocks that match the language of the block we are in, and in the session, and execute them. We can tell if a block is in a session by looking at the :parameters property of the block. Interestingly, if a block is not in a session, then session will be "none", if it is in an unnamed session, session will be nil, and otherwise, session will be the session name.

Here is a function for testing if a block is in a session.

(defun src-block-in-session-p (&optional name)
  "Return if src-block is in a session of NAME.
NAME may be nil for unnamed sessions."
  (let* ((info (org-babel-get-src-block-info))
         (lang (nth 0 info))
         (body (nth 1 info))
         (params (nth 2 info))
         (session (cdr (assoc :session params))))

    (cond
     ;; unnamed session, both name and session are nil
     ((and (null session)
           (null name))
      t)
     ;; Matching name and session
     ((and
       (stringp name)
       (stringp session)
       (string= name session))
      t)
     ;; no match
     (t nil))))

Now, we need to get some information about the current point and block. We will want to run blocks that start before the current point, but not after. We will use org-element-map to find code blocks, and when the language and session of a code block matches the current block, and the block starts at a point earlier than the current point, then we will go to that block, and run it. Here is that code.

(defun org-babel-restart-session-to-point (&optional arg)
  "Restart session up to the src-block in the current point.
Goes to beginning of buffer and executes each code block with
`org-babel-execute-src-block' that has the same language and
session as the current block. ARG has same meaning as in
`org-babel-execute-src-block'."
  (interactive "P")
  (unless (org-in-src-block-p)
    (error "You must be in a src-block to run this command"))
  (let* ((current-point (point-marker))
         (info (org-babel-get-src-block-info))
         (lang (nth 0 info))
         (params (nth 2 info))
         (session (cdr (assoc :session params))))
    (save-excursion
      (goto-char (point-min))
      (while (re-search-forward org-babel-src-block-regexp nil t)
        ;; goto start of block
        (goto-char (match-beginning 0))
        (let* ((this-info (org-babel-get-src-block-info))
               (this-lang (nth 0 this-info))
               (this-params (nth 2 this-info))
               (this-session (cdr (assoc :session this-params))))
            (when
                (and
                 (< (point) (marker-position current-point))
                 (string= lang this-lang)
                 (src-block-in-session-p session))
              (org-babel-execute-src-block arg)))
        ;; move forward so we can find the next block
        (forward-line)))))

In the course of testing this, I found this function a little helpful to kill the current session so we start fresh.

(defun org-babel-kill-session ()
  "Kill session for current code block."
  (interactive)
  (unless (org-in-src-block-p)
    (error "You must be in a src-block to run this command"))
  (save-window-excursion
    (org-babel-switch-to-session)
    (kill-buffer)))

And also this one to remove all results in the buffer. This not at all selective, it removes results for session and non-session blocks.

(defun org-babel-remove-result-buffer ()
  "Remove results from every code block in buffer."
  (interactive)
  (save-excursion
    (goto-char (point-min))
    (while (re-search-forward org-babel-src-block-regexp nil t)
      (org-babel-remove-result))))

Ok, now for some testing. The rest of this post is pretty boring, just some blocks of mixed session and non-session to see if they get run. Skip to the 1.

def f(x):
    y = 4 * x
    return y

print(f(d))

16

Let us put a non-session block in this buffer for testing.

a = 5
print(a)

Now, some more named session blocks.

print f(5)

20

print 'ok'

ok

print 2

2

An unnamed session that should not get run in restarting the named test session.

print 886

print f(6)

24

Copyright (C) 2015 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.10

The org-ref code is finally all in emacs-lisp! This should make it much easier to install, and is another step closer to getting org-ref into MELPA. Previously, I had written the most significant code in org-mode source blocks that were intended to be tangled out. I found this was not really portable, because what gets tangled depends on your org-mode setup. I had to specifically set example blocks to not tangle, or org-ref would not work for other people, and if I forgot to set a block to tangle, it also would not work for others. That should not happen again now, since there is no more tangling.

There are some relatively new features in org-ref:

1. New colored org-ref links to differentiate them from other org-links. Citations are greenish, refs and labels are maroonish.
2. Context messages about links. With your cursor on a cite, ref or label link you will get a context message, e.g. a formatted citation, some context about the label a ref refers to, or a count of the labels in the mini-buffer.
3. There is now an org-ref menu in the Org menu.
4. There is a new org-ref-help function that opens an org-file of org-ref documentation.
5. Pretty thorough integration of helm throughout org-ref, and some integration of hydra.
6. A few utility libraries: doi-utils, isbn, wos, pubmed, arxiv, jmax-bibtex, sci-id, x2bib. Not all these are new, but if you didn't know about them, check them out.
7. Cask integration. This mostly provides access to testing and dependencies right now. org-ref is also now tested continuously at https://travis-ci.org/jkitchin/org-ref .

org-ref is basically feature complete I think (which is to say that once again, I do not have any big ideas for new features ;). There are some places where it could be refactored a little, e.g. there are some bibtex only functions in org-ref.el that really should go into jmax-bibtex.el (which also could be renamed). This is a very low priority though, because things are working fine as far as I can tell.

What does it need before going into MELPA? Probably some tests would be a good idea. On Travis, all that is really tested is that it loads with no errors. I would like to see some stability on my end, e.g. at least a week where no commits get made, and no errors are reported. And finally, I would like to make sure I have some time to handle issues that come up when a broader audience is trying it out.

My target date to get this in MELPA is June 1, 2015. Try out the new org-ref, and let me know how it goes!

Copyright (C) 2015 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.10

Using sessions for python in org-mode has always bugged me a little bit. Mostly the appearance of >>> and … in the output. For example:

print 8

>>> >>> >>> 8

Today on the org-mode mailing list someone suggested a patch that might fix that up. Hopefully that patch makes it into org-mode, but if you run off of ELPA like I do it will be some time before it appears in your working version.

In the meantime, inspired by my recent post on updating multiple results, here we add a new hook function that removes these annoying characters from a Python session results section. Here is my version of this code. "^: >>>$"

(defun org-babel-python-strip-session-chars ()
  "Remove >>> and ... from a Python session output."
  (when (and (string=
              "python"
              (org-element-property :language (org-element-at-point)))
             (string-match
              ":session"
              (org-element-property :parameters (org-element-at-point))))

    (save-excursion
      (when (org-babel-where-is-src-block-result)
        (goto-char (org-babel-where-is-src-block-result))
        (end-of-line 1)
        ;(while (looking-at "[\n\r\t\f ]") (forward-char 1))
        (while (re-search-forward
                "\\(>>> \\|\\.\\.\\. \\|: $\\|: >>>$\\)"
                (org-element-property :end (org-element-at-point))
                t)
          (replace-match "")
          ;; this enables us to get rid of blank lines and blank : >>>
          (beginning-of-line)
          (when (looking-at "^$")
            (kill-line)))))))

(add-hook 'org-babel-after-execute-hook 'org-babel-python-strip-session-chars)

import matplotlib.pyplot as plt
plt.plot([3, 4, 5])
plt.show()

def f(s):
    x = 2 * s
    # blank lines look like indentation errors
    return x

print f(4)

[<matplotlib.lines.Line2D object at 0x10955c290>]
8

print f(9)

18

Here we can make an inline figure.

plt.figure()
plt.plot([3, 4.5, 5])
plt.savefig('images/session-fig.png')
'images/session-fig.png'

Not bad. It seems to work! Maybe this will make sessions more usable for me.

[2015-03-12 Thu] New corner case, do not cause an error when results are silenced.

print 6

Testing getting rid of blank lines and empty : >>> lines.

a = 2
b = 3
c = 4
print
print 'a=      ', a
print 'b =     ', b
print 'a + b = ', a+b

a=       2
b =      3
a + b =  5

Copyright (C) 2015 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.10

There was a recent question on the org-mode mailing list about getting multiple named block results to update when a named code block is run. I suppose you might want to do this if you need to see the results in more than one place. org-mode (at the moment) only updates the first named block that it finds from the beginning of the buffer. Challenge accepted ;)

Here is a function that will update all the named RESULTS blocks. The idea is to make a hook function that runs after you run a block. The hook function will get the block name, and if there is one, find all the named results in the buffer and update them.

(defun update-results ()
  ;; get name of src block
  (let ((name (org-element-property :name (org-element-at-point)))
        (results)
        (begin))
    (when name
      (setq results
            (save-excursion
              (goto-char (org-babel-find-named-result name))
              (forward-line)
              (buffer-substring
               (point) (org-element-property :end (org-element-at-point)))))
      (save-excursion
        (goto-char (point-min))
        (while (setq begin (org-babel-find-named-result name (point)))
          (goto-char begin)
          (forward-line)
          (setf (buffer-substring
                 (point)
                 (org-element-property :end (org-element-at-point)))
                results))))))

(add-hook 'org-babel-after-execute-hook 'update-results)

Now let us test it out. Here is an unnamed block that should be ignored.

print 4

4

Here we have a named results block from a code block we will see later.

[0.0825119635983067, 0.12793443834890417, 0.5235765147357154]

Here is our named code block that just prints three random numbers.

import random

print [random.random() for i in range(3)]

[0.0825119635983067, 0.12793443834890417, 0.5235765147357154]

Swell, everytime I run the block, the named results get updated everywhere! It isn't tested more than this post, so I would spend some time trying out your use cases before doing anything mission critical. Your mileage might vary. For example, if you have a named block outside a narrowed region it is not clear to me it would update. In other words, there might be other corners where this doesn't update like you thing.

Copyright (C) 2015 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.10