The Kitchin Research Group

I am going to use Jupyter notebooks to teach from this semester. I really dislike preparing notebooks though. A browser is a really poor editor, and I really dislike Markdown. Notebooks do not seem to have any real structure in them, e.g. the collapsible outline that I am used to in org-mode, so for long notebooks, it is difficult to get a sense for the structure. I am anticipating spending up to 80 hours preparing notebooks this semester, so today I worked out some code to export org-mode to an ipython notebook!

This will let me use the power tools I am accustomed to for the creation of IPython notebooks for my students, and perhaps others who do not use org-mode.

Jupyter notebooks are just json files, so all we need to do is generate it from an org document. The basic strategy was to build up a lisp data structure that represents the notebook and then just convert that data structure to json. I split the document up into sequential markdown and code cells, and then encode those in the format required for the notebook (json).

So, here is an example of what can be easily written in org-mode, posted to this blog, and exported to an IPython notebook, all from one org-document.

Check out the notebook: exporting-orgmode-to-ipynb.ipynb .

def objective(x):
    return x**2 - 4

from scipy.optimize import fsolve

ans = fsolve(objective, 3)
print(ans)

[ 2.]

ans, = fsolve(objective, -3)
print(ans)

-2.0

ans = fsolve(objective, -3)

print(float(ans))
print(ans[0])

-2.0000000000000084
-2.0

import numpy as np

t = np.linspace(0, 2 * np.pi)

x = np.cos(t)
y = np.sin(t)

import matplotlib.pyplot as plt
plt.plot(x, y)
plt.axis('equal')
plt.xlabel('x')
plt.ylabel('y')
plt.savefig('circle.png')

(export-ipynb-buffer)

Copyright (C) 2017 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 9.0.3

I use org-mode extensively. I write scientific papers, keep notes on meetings, write letters of recommendation, notes on scientific articles, keep TODO lists in projects, help files for software, write lecture notes, students send me homework solutions in it, it is a contact database, … Some files are on Dropbox, Google Drive, Box, some in git repos, etc. The problem is that leads to org-files everywhere on my hard drive. At this point I have several thousand org-files that span about five years of work.

It is not that easy after a while to find them. Yes there are things like recent-files, bookmarks, counsel-find-file, helm-for-files, counsel/helm-locate, helm/counsel-grep/ag/pt, projectile for searching within a project, a slew of tools to search open buffers, there is recoll, etc… There are desktop search tools, and of course, good organization habits. Over a five year time span though, these change, and I have yet to find a solution to finding what I want. What about a file I made a year ago that is not in the current directory or this project, and not in my org-agenda-files list? How do I get a dynamic todo list across all these files? Or find all the files that cite a particular bibtex entry, or that were authored by a particular student?

Previously, I indexed org files with Swish-e to make it easy to search them, with an ability to search just headlines, or paragraphs, etc. The problem with that is the nightly indexing was slow since I basically had to regenerate the database each time due to limitations in Swish-e. Finally I have gotten around to the next iteration of this idea, which is a better database. In this post, I explore using sqlite to store headlines and links in org-files.

The idea is that anytime I open or save any org file, it will be added/updated in the database. The database will store the headlines and its properties and content, as well as the location and properties of all links and file keywords. That means I should be able to efficiently query all org files I have ever visited to find TODO headlines, tagged headlines, different types of links, etc. Here we try it out and see if it is useful.

(emacsql-flatten-sql [:select [name] :from main:sqlite_master :where (= type table)])

SELECT name FROM main.sqlite_master WHERE type = "table";

(emacsql org-db [:select [name] :from main:sqlite_master :where (= type table)])

(emacsql org-db [:pragma (funcall table_info files)])

(emacsql org-db [:pragma (funcall table_info headlines)])

du -hs ~/org-db/org-db.sqlite

(emacsql org-db [:select (funcall count) :from files])

(emacsql org-db [:select (funcall count) :from headlines])

(emacsql org-db [:select (funcall count) :from links])

(emacsql org-db [:select (funcall count) :from links :where (= type "cite")])

(emacsql org-db [:select :distinct [files:filename]
                 :from links :inner :join files :on (= links:filename-id files:rowid) 
                 :where (and (= type "cite") (like path "%kitchin-2015-examp%"))])

(ivy-read "Open: " (emacsql org-db [:select [files:filename links:begin]
                                    :from links :inner :join files :on (= links:filename-id files:rowid) 
                                    :where (and (= type "cite") (like path "%kitchin-2015-examp%"))])
          :action '(1 ("o"
                       (lambda (c)
                         (find-file (car c))
                         (goto-char (nth 1 c))
                         (org-show-entry)))))

/Users/jkitchin/Dropbox/CMU/manuscripts/2015/human-readable-data/manuscript.org

(emacsql org-db [:select (funcall count) :from tags])

(emacsql org-db [:select :distinct [files:filename headlines:title]
                 :from headlines :inner :join headline-tags :on (=  headlines:rowid headline-tags:headline-id)
                 :inner :join tags :on (= tags:rowid headline-tags:tag-id)
                 :inner :join files :on (= headlines:filename-id files:rowid)
                 :where (= tags:tag "electrolyte") :limit 5])

(emacsql org-db [:select [(funcall count)] :from
                 headlines :inner :join headline-properties :on (=  headlines:rowid headline-properties:headline-id)
                 :inner :join properties :on (= properties:rowid headline-properties:property-id)
                 :where (and (= properties:property "EMAIL") (not (null headline-properties:value)))])

(emacsql org-db [:select :distinct [headlines:title headline-properties:value] :from
             headlines :inner :join headline-properties :on (=  headlines:rowid headline-properties:headline-id)
             :inner :join properties :on (= properties:rowid headline-properties:property-id)
             :where (and (= properties:property "EMAIL") (like headline-properties:value "%jkitchin%"))])

(emacsql org-db [:select (funcall count) :from
             headlines :inner :join headline-properties :on (=  headlines:rowid headline-properties:headline-id)
             :inner :join properties :on (= properties:rowid headline-properties:property-id)
             :where (and (= properties:property "DEADLINE") (glob headline-properties:value "*2017*"))])

(emacsql org-db [:select :distinct [value] :from
                 file-keywords :inner :join keywords :on (= file-keywords:keyword-id keywords:rowid)
                 :where (and (> (funcall length value) 35) (= keywords:keyword "TITLE"))
                 :order :by value :desc
                 :limit 5])

(emacsql org-db [:select [(funcall count)] :from
                 file-keywords :inner :join keywords :on (= file-keywords:keyword-id keywords:rowid)
                 :where (and (= value "cmu-memo") (= keywords:keyword "LATEX_CLASS"))
                 :limit 5])

(org-db-query (and (= properties:property "DEADLINE") (glob headline-properties:value "*2017*")))

(org-db-query "DEADLINE={2017}")

Copyright (C) 2017 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 9.0.3

I have been using org-mode to make a contact database. A contact is basically just a headline with an EMAIL property, e.g. https://julien.danjou.info/projects/emacs-packages#org-contacts. I thought it would be nice to have an org-mode speed key so that if I was at the beginning of a contact headline, I could just press "e" to open an email buffer to that contact. This might generally be useful to have different speed keys that serve different purposes or are only defined on specific types of headlines.

Org-mode already had this feature in mind for speed keys. All you have to do is define the list of speed keys and their functions, provide a function that picks the right one, and add it to the org-speed-command-hook. Here is the code that makes this possible. This defines "c" to copy the email to the clipboard, "e" to email the contact, and "m" to copy a "name <email>" string to the clipboard, but only when you are on a headline with an EMAIL property. If there is not a contact specific speed key defined, then a user-defined speed key or a default key will be used if it is defined. In case I do not remember the keys, "?" will show them to me. It is small hack, but if you end up using the contact headlines for much, it might be really helpful as an alternative to M-x some-contacts-command.

(setq org-speed-commands-contacts
      '(("c" . (lambda ()
                 "Copy the email address to the clipboard."
                 (message (kill-new (org-entry-get (point) "EMAIL")))))
        ("e" . (lambda ()
                 "Send an email to the contact."
                 (let ((email (org-entry-get (point) "EMAIL")))
                   (compose-mail)
                   (message-goto-to)
                   (insert email)
                   (message-goto-subject)))) 
        ("m" . (lambda ()
                 "Copy \"name <email>\""
                 (message (kill-new
                           (format "%s <%s>"
                                   (nth 4 (org-heading-components))
                                   (org-entry-get (point) "EMAIL"))))))
        ("?" . (lambda ()
                 "Print contacts speed key help."
                 (with-output-to-temp-buffer "*Help*"
                   (princ "Contacts Speed commands\n===========================\n")
                   (mapc #'org-print-speed-command org-speed-commands-contacts)
                   (princ "\n")
                   (princ "User-defined Speed commands\n===========================\n")
                   (mapc #'org-print-speed-command org-speed-commands-user)
                   (princ "Built-in Speed commands\n=======================\n")
                   (mapc #'org-print-speed-command org-speed-commands-default))
                 (with-current-buffer "*Help*"
                   (setq truncate-lines t))))))

(defun org-speed-contacts (keys)
  (when (and (bolp) (looking-at org-outline-regexp)
             (not (null (org-entry-get (point) "EMAIL"))))
    (cdr (assoc keys org-speed-commands-contacts))))

(add-hook 'org-speed-command-hook 'org-speed-contacts)

Copyright (C) 2016 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 9.0

In this recent post I showed a way to use org-mode links to color text. The main advantage of that approach is it is explicit markup in the file, so it is persistent and exportable to html. The downside of that approach is you cannot use it in code, since the markup will break the code.

An alternative approach is to use overlays to color the text. This allows you to color the text, add annotations as tooltips and to provide a variety of highlighting colors. Overlays are not explicit markup in the file, so it is necessary to think of a way to save them so they can be restored later. We do this by using hook functions to store the overlays in a file-local variable on saving, and a file-local variable to restore the overlays when the file is opened. I bind the primary function `ov-highlighter/body' to a key, in my case hyper-h, which launches a hydra to access the commands.

You can find the code here: https://github.com/jkitchin/scimax/blob/org-9/ov-highlighter.el. Probably around mid-December it will get merged into the master branch.

Here is what this looks like in my buffer:

You may want to see the video:

1. blue green pink yellow custom
2. Put a comment here.
3. Markup a tpyo.
4. Get a list of the highlights in the buffer.

These highlights are pretty awesome. They work in code blocks, and comments. They also work in non-org files (only in Emacs of course).

a = 5
b = 6

print(a+b)#print the sum of a and b

11

Overall, this is pretty handy. You can highlight your own notes, provide feedback to others, etc. without changing the actual text in the document (well, except for the local variables at the end of the buffer, but these are usually in a "comment" that does not affect the document).

Here are few limitations though:

1. You can only edit/change the file in Emacs, and the hook functions have to enabled, or the overlay data will get corrupted. That means a merge conflict can ruin the overlays.
2. Anyone you share the file with needs to have the ov-highlighter library loaded too. Otherwise they will not see the highlights, and any edits will make the overlay data incorrect.
3. The highlights do not export from org-mode (although they do work with `htmlize-buffer'!).

(let* ((html-buffer (htmlize-buffer))
       (html (with-current-buffer html-buffer
               (buffer-string))))
  (with-temp-file "test.html"
    (insert html))
  (kill-buffer html-buffer))

(browse-url "test.html")

#<process open test.html>

Copyright (C) 2016 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 9.0

About a year ago I posted some code to run org-babel python blocks asynchronously. This year, my students asked for some enhancements related to debugging. Basically, they were frustrated by a few things when they got errors. First, they found it difficult to find the line number in the Traceback in the src block because there are no line numbers in the block, and it is annoying to do a special edit just for line numbers.

I thought about this, and figured out how to significantly improve the situation. The async python code in scimax now has the following features:

1. When you get a Traceback, it goes in the results, and each file listed in it is hyperlinked to the source file and line so it is easy to get to them.
2. The cursor jumps to the last line in the code block that is listed in the Traceback, and a beacon shines to show you the line
3. You can turn on temporary line numbers in the code block to see where the lines are in the block, and these disappear when you start typing. This is done in the variable `org-babel-async-python-show-line-numbers'.
4. You can control whether a buffer of the results shows or not via the variable `org-babel-async-python-show-results'.
5. When you run the block, you get a clickable link in the RESULTS section to kill the process.
6. You may also find the `autopep8' and `pylint' functions helpful.

The code for this is currently found here: https://github.com/jkitchin/scimax/blob/org-9/scimax-org-babel-python.el

Eventually, I will merge this into master, after I am sure about all the changes needed for org 9.0. That is not likely to happen until the semester ends, so I do not mess up my students who use scimax in class. So, sometime mid-December it will make into master.

To make async the default way to run a python block use this code, so that you can use C-c C-c to run them:

(require 'scimax-org-babel-python)
(add-to-list 'org-ctrl-c-ctrl-c-hook 'org-babel-async-execute:python)

As with the past few posts, this video will make it much more clear what the post is about:

Here is a prototypical example that shows how it works. While it runs you can view the progress if you click on the link to show the results.

import time

for i in range(5):
    print(i)
    time.sleep(2)

0 1 2 3 4 Traceback (most recent call last): File "Org SRC", line 5, in <module> time.sleep(2) KeyboardInterrupt

This block has a pretty obvious issue when we run it. The cursor jumps right to the problem!

print('This line is ok')
# 5 / 0
print('We will not see this')

This line is ok We will not see this

This block shows we can access any of the links in the Traceback. Here we have an error in calling a function that is raised in an external file.

import numpy as np
from scipy.integrate import odeint

Vspan = np.linspace(0, 2) # L

# dF/dV = F
def dFdV(F, V, v0):
    return F


print(odeint(dFdV, 1.0, Vspan))

Traceback (most recent call last): File "Org SRC", line 11, in <module> print(odeint(dFdV, 1.0, Vspan)) File "/Users/jkitchin/anaconda3/lib/python3.5/site-packages/scipy/integrate/odepack.py", line 215, in odeint ixpr, mxstep, mxhnil, mxordn, mxords) TypeError: dFdV() missing 1 required positional argument: 'v0'

Here we show how nice it is to be able to kill a process. This block will not end on its own.

while True:
    pass

Traceback (most recent call last): File "Org SRC", line 2, in <module> pass KeyboardInterrupt

a = 4
b = 5
c = a * b  # comment
# another comment


def f(x):
    return x
print(f(5))

import numpy as np

a = np.array(5, 5)

def f(x): return x

print(f(6))

Copyright (C) 2016 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 9.0