## Org-mode and ipython enhancements in scimax

| categories: | tags: | View Comments

We have made some improvements to using Ipython in org-mode in the past including:

Today I will talk about a few new features and improvements I have introduced to scimax for using org-mode and Ipython together.

The video for this post might be more obvious than the post:

## 1 Some convenience functions

There are a few nice shortcuts in the Jupyter notebook. Now we have some convenient commands in scimax to mimic those. My favorites are adding cells above or below the current cell. You can insert a new src block above the current one with (M-x org-babel-insert-block). You can use a prefix arg to insert it below the current block.

# code

# below

# some code


I am particularly fond of splitting a large block into two smaller blocks. Use (M-x org-babel-split-src-block) to do that and leave the point in the upper block. Use a prefix arg to leave the point in the lower block.

# lots of code in large block

# Even more code

# The end of the long block


You can execute all the blocks up to the current point with (M-x org-babel-execute-to-point).

## 2 ob-ipython-inspect works

In the original ob-ipython I found that ob-ipython-inspect did not work unless you were in special edit mode. That is too inconvenient. I modified a few functions to work directly from the org-buffer. I bind this to M-. in org-mode.

%matplotlib inline
import numpy as np

import matplotlib.pyplot as plt

# Compute areas and colors
N = 150
r = 2 * np.random.rand(N)
theta = 2 * np.pi * np.random.rand(N)
area = 200 * r**2
colors = theta

ax = plt.subplot(111, projection='polar')
c = ax.scatter(theta, r, c=colors, s=area, cmap='hsv', alpha=0.75)


<matplotlib.figure.Figure at 0x114ded710>

## 3 Getting selective output from Ipython

Out of the box Ipython returns a lot of results. This block, for example returns a plain text, image and latex result as output.

from sympy import *
# commenting out init_printing() results in no output
init_printing()

var('x y')
x**2 + y


2 x + y

We can select which one we want with a new header argument :ob-ipython-results. For this block you can give it the value of text/plain, text/latex or image/png.

var('x y')
x**2 + y


2 x + y

Or to get the image:

var('x y')
x**2 + y


This shows up with pandas too. This block creates a table of data and then shows the first 5 rows. Ipython returns both plain text and html here.

import pandas as pd
import numpy as np
import datetime as dt

def makeSim(nHosps, nPatients):
df = pd.DataFrame()
df['patientid'] = range(nPatients)
df['hospid'] = np.random.randint(0, nHosps, nPatients)
df['sex'] = np.random.randint(0, 2, nPatients)
df['age'] = np.random.normal(65,18, nPatients)
df['race'] = np.random.randint(0, 4, nPatients)
df['cptCode'] = np.random.randint(1, 100, nPatients)
df['rdm30d'] = np.random.uniform(0, 1, nPatients) < 0.1
df['mort30d'] = np.random.uniform(0, 1, nPatients) < 0.2
df['los'] = np.random.normal(8, 2, nPatients)
return df

discharges = makeSim(50, 10000)


patientid hospid sex age race cptCode rdm30d mort30d los 0 0 10 1 64.311947 0 8 False False 8.036793 1 1 6 0 82.951484 1 73 True False 7.996024 2 2 27 1 53.064501 3 95 False False 9.015144 3 3 37 0 64.799128 0 93 False False 10.099032 4 4 46 0 99.111394 2 25 False False 11.711427

patientid hospid sex age race cptCode rdm30d mort30d los
0 0 10 1 64.311947 0 8 False False 8.036793
1 1 6 0 82.951484 1 73 True False 7.996024
2 2 27 1 53.064501 3 95 False False 9.015144
3 3 37 0 64.799128 0 93 False False 10.099032
4 4 46 0 99.111394 2 25 False False 11.711427

We can use the header to select only the plain text output!

import pandas as pd
import numpy as np
import datetime as dt

def makeSim(nHosps, nPatients):
df = pd.DataFrame()
df['patientid'] = range(nPatients)
df['hospid'] = np.random.randint(0, nHosps, nPatients)
df['sex'] = np.random.randint(0, 2, nPatients)
df['age'] = np.random.normal(65,18, nPatients)
df['race'] = np.random.randint(0, 4, nPatients)
df['cptCode'] = np.random.randint(1, 100, nPatients)
df['rdm30d'] = np.random.uniform(0, 1, nPatients) < 0.1
df['mort30d'] = np.random.uniform(0, 1, nPatients) < 0.2
df['los'] = np.random.normal(8, 2, nPatients)
return df

discharges = makeSim(50, 10000)


patientid hospid sex age race cptCode rdm30d mort30d los 0 0 21 0 73.633836 1 38 False False 7.144019 1 1 16 1 67.518804 3 23 False False 3.340534 2 2 15 0 44.139033 0 8 False False 9.258706 3 3 29 1 45.510276 2 5 False False 10.590245 4 4 7 0 52.974924 2 4 False True 5.811064

## 4 Where was that error?

A somewhat annoying feature of running cells in org-mode is when there is an exception there has not been a good way to jump to the line that caused the error to edit it. The lines in the src block are not numbered, so in a large block it can be tedious to find the line. In scimax, when you get an exception it will number the lines in the src block, and when you press q in the exception traceback buffer it will jump to the line in the block where the error occurred.

print(1)
#raise Exception('Here')
print(2)


1 2

If you don't like the numbers add this to your init file:

(setq ob-ipython-number-on-exception nil)


## 5 Asynchronous Ipython

I have made a few improvements to the asynchronous workflow in Ipython. We now have a calculation queue, so you can use C-c C-c to execute several blocks in a row, and they will run asynchronously in the order you ran them. While they are running you can continue using Emacs, e.g. writing that paper, reading email, checking RSS feeds, tetris, … This also lets you run all the blocks up to the current point (M-x org-babel-execute-ipython-buffer-to-point-async) or the whole buffer (of Ipython) blocks asynchronously (M-x org-babel-execute-ipython-buffer-async).

To turn this on by default put this in your init file:

(setq org-babel-async-ipython t)


This requires all src blocks to have a name, and running the block will give it a name if you have not named the block. By default we use human-readable names. While the block is running, there will be a link indicating it is running. You can click on the link to cancel it. Running subsequent blocks will queue them to be run when the first block is done.

Here is an example:

import time
time.sleep(5)
a = 5
print('done')

print(3 * a)


15

Occasionally you will run into an issue. You can clear the queue with org-babel-async-ipython-clear-queue.

org-mode source

Org-mode version = 9.0.5

## A new org-mode exporter to Word for scimax

| categories: | tags: | View Comments

I am continuing to chip away to getting a reasonable export behavior for org-mode to MS Word. I have previously made some progress with Pandoc here and here, but those solutions never stuck with me. So here is another go. Here I leverage Pandoc again, but use a path through LaTeX to get citations without modifying the org-ref cite link syntax. The code for this can be found here: https://github.com/jkitchin/scimax/blob/master/ox-word.el. The gist is you use org-ref like you always do, and you specify the bibliography style for Pandoc like this:

You can download other csl files at https://www.zotero.org/styles. Then you can simply export the org-doc to a Word document with the key-binding C-c C-e w p.

Here is an example document to illustrate the exporter. I have written about data sharing in catalysis kitchin-2015-examp and surface science kitchin-2015-data-surfac-scien.

Here is an example source block.

%matplotlib inline
import matplotlib.pyplot as plt

plt.plot([1, 2, 3, 4, 5, 6])


See Ref. fig:line for example. These do not work. That might require additional pre-processing to replace them with numbers.

Here is the Word document that is generated: 2017-04-15.docx

As a penultimate result it might be ok. The references are reasonably formatted, but not compatible with Endnote, or other bibliography manager software. There are still some issues with Figure numbering and cross-references, but it is not too bad. The main benefit of this seems to be that one source generates HTML and the Word document.

# Bibliography

org-mode source

Org-mode version = 9.0.5

## Autoformatting ordinal numbers and fractions in orgmode

| categories: | tags: | View Comments

MS Word has a few things I like. One of them is the ability to autoformat things to make an ordinal number string like 1st to the superscripted version 1st while you type or a 1/2 to ½. I thought it would be pretty easy to implement that for org-mode. It turns out it was not so easy!

There does not appear to be a way to specify a regexp pattern as an abbreviation, or an abbrev that starts with a number. What we need for ordinal numbers is to recognize a sequence of numbers followed by "st", "nd", "rd" or "th" followed by a space or punctuation, and then superscript the letters. In case you didn't want the replacement to occur, you should be able to undo it and get back the original string. This addition was a little hard won, so I am sharing the lessons here.

The logic I used is to put a function in the post-self-insert-hook. The function only works in org-mode, when not in a codeblock and when looking back at a regexp that matches a pattern to be replaced. Getting it to undo was trickier than expected. Eventually I worked out that you put an undo boundary in place before the change, and then it seems like you can undo the changes. I created a minor mode so it is easy to toggle this on and off.

Here is the implementation:

(defcustom scimax-autoformat-ordinals t
"Determines if scimax autoformats ordinal numbers."
:group 'scimax)

(defun scimax-org-autoformat-ordinals ()
"Expand ordinal words to superscripted versions in org-mode.
1st to 1^{st}.
2nd to 2^{nd}
3rd to 3^{rd}
4th to 4^{th}"
(interactive)
(when (and scimax-autoformat-ordinals
(eq major-mode 'org-mode)
(not (org-in-src-block-p))
(looking-back "\$$?3:\\<\\(?1:[0-9]+\$$\$$?2:st\\|nd\\|rd\\|th\$$\\>\\)\$$?:[[:punct:]]\\|[[:space:]]\$$"
(line-beginning-position)))
(undo-boundary)
(save-excursion
(replace-match "\\1^{\\2}" nil nil nil 3))))

(defcustom scimax-autoformat-fractions t
"Determines if scimax autoformats fractions."
:group 'scimax)

(defun scimax-org-autoformat-fractions ()
"Expand fractions to take up space."
(interactive)
(when (and scimax-autoformat-fractions
(eq major-mode 'org-mode)
(not (org-in-src-block-p))
(looking-back "\$$?3:\\<\\(1/4\\|1/2\\|3/4\$$\\>\\)\$$?:[[:punct:]]\\|[[:space:]]\$$"
(line-beginning-position)))
(undo-boundary)
(save-excursion
(replace-match (cdr (assoc (match-string 3) '(("1/4" . "¼")
("1/2" . "½")
("3/4" . "¾"))))
nil nil nil 3))))

(defun scimax-org-autoformat ()
"Autoformat functions."
(scimax-org-autoformat-ordinals)
(scimax-org-autoformat-fractions))

(define-minor-mode scimax-autoformat-mode
"Toggle scimax-autoformat-mode'.  Converts 1st to 1^{st} as you type."
:init-value nil
:lighter (" om")
(if scimax-ordinal-mode
(remove-hook 'post-self-insert-hook #'scimax-org-autoformat 'local)))


This is now a feature in scimax. This marks the 500th blog post! That is ½ way to 1000. At the current rate of posting, it will be at least 5 years until I hit that!

org-mode source

Org-mode version = 9.0.5

## A better return in org-mode

| categories: | tags: | View Comments

Over on Stackoverflow someone wanted a better return in org-mode. They wanted return to add items in a list (instead of M-Ret). Someone posted a partial solution, and here I improve on it to add new items to lists, new headings after a heading, and new rows to tables. In each case, a double return on an empty item, headline or table row will delete that line, and terminate the list, headlines or table. You can still use M-Ret, and this function falls through to org-return like it did before. You can use a prefix arg to get a regular return if you want one (e.g. you want to press enter on a headline to push it down).

Here is the function. Give it a try. It is a small but helpful addition I think. I have not used it for long, so if you come across issues leave a comment!

(require 'org-inlinetask)

(defun scimax/org-return (&optional ignore)
A double return on an empty element deletes it.
Use a prefix arg to get regular RET. "
(interactive "P")
(if ignore
(org-return)
(cond

((eq 'line-break (car (org-element-context)))
(org-return-indent))

;; Open links like usual, unless point is at the end of a line.
;; and if at beginning of line, just press enter.
((or (and (eq 'link (car (org-element-context))) (not (eolp)))
(bolp))
(org-return))

;; Johansson!
(org-return))

;; checkboxes too
((org-at-item-checkbox-p)

;; lists end with two blank lines, so we need to make sure we are also not
;; at the beginning of a line to avoid a loop where a new entry gets
;; created with only one blank line.
((org-in-item-p)
(if (save-excursion (beginning-of-line) (org-element-property :contents-begin (org-element-context)))
(beginning-of-line)
(delete-region (line-beginning-position) (line-end-position))
(org-return)))

(if (not (string= "" (org-element-property :title (org-element-context))))
(progn (org-end-of-meta-data)
(outline-show-entry))
(beginning-of-line)
(setf (buffer-substring
(line-beginning-position) (line-end-position)) "")))

;; tables
((org-at-table-p)
(if (-any?
(lambda (x) (not (string= "" x)))
(nth
(- (org-table-current-dline) 1)
(org-table-to-lisp)))
(org-return)
;; empty row
(beginning-of-line)
(setf (buffer-substring
(line-beginning-position) (line-end-position)) "")
(org-return)))

;; fall-through case
(t
(org-return)))))

(define-key org-mode-map (kbd "RET")
'scimax/org-return)


Here are a few tests:

1. numbered item
2. second item
1. nested number
2. second number
• [ ] check 1
• [ ] check 2
• [ ] check 3

With some content

org-mode source

Org-mode version = 9.0.5

## Exporting org-mode to Jupyter notebooks

| categories: | tags: | View Comments

I am going to use Jupyter notebooks to teach from this semester. I really dislike preparing notebooks though. A browser is a really poor editor, and I really dislike Markdown. Notebooks do not seem to have any real structure in them, e.g. the collapsible outline that I am used to in org-mode, so for long notebooks, it is difficult to get a sense for the structure. I am anticipating spending up to 80 hours preparing notebooks this semester, so today I worked out some code to export org-mode to an ipython notebook!

This will let me use the power tools I am accustomed to for the creation of IPython notebooks for my students, and perhaps others who do not use org-mode.

Jupyter notebooks are just json files, so all we need to do is generate it from an org document. The basic strategy was to build up a lisp data structure that represents the notebook and then just convert that data structure to json. I split the document up into sequential markdown and code cells, and then encode those in the format required for the notebook (json).

So, here is an example of what can be easily written in org-mode, posted to this blog, and exported to an IPython notebook, all from one org-document.

Check out the notebook: exporting-orgmode-to-ipynb.ipynb .

## 1 Solve a nonlinear problem

Consider the equation $$x^2 = 4$$. Find a solution to it in Python using a nonlinear solver.

To do that, we need to define an objective function that will be equal to zero at the solution. Here is the function:

def objective(x):
return x**2 - 4


Next, we use fsolve with an initial guess. We get fsolve from scipy.optimize.

from scipy.optimize import fsolve

ans = fsolve(objective, 3)
print(ans)

[ 2.]


That should have been an obvious answer. The answer is in brackets because fsolve returns an array. In the next block we will unpack the solution into the answer using the comma operator. Also, we can see that using a different guess leads to a different answer. There are, of course, two answers: $$x = \pm 2$$

ans, = fsolve(objective, -3)
print(ans)

-2.0


Now you see we get a float answer!

Here are some other ways to get a float:

ans = fsolve(objective, -3)

print(float(ans))
print(ans[0])

-2.0000000000000084
-2.0


It is worth noting from the first result that fsolve is iterative and stops when it reaches zero within a tolerance. That is why it is not exactly -2.

## 2 Benefits of export to ipynb

1. I can use org-mode
2. And emacs
3. and ipynb for teaching.

The export supports org-markup: bold, italic, underlined, and ~~strike~~.

We can use tables:

Table 1: A table of squares.
x y
1 2
2 4
3 9
4 16

We can make plots.

import numpy as np

t = np.linspace(0, 2 * np.pi)

x = np.cos(t)
y = np.sin(t)

import matplotlib.pyplot as plt
plt.plot(x, y)
plt.axis('equal')
plt.xlabel('x')
plt.ylabel('y')
plt.savefig('circle.png')


Even include HTML: <font color="red">Pay special attention to the axis labels!</font>

## 3 Limitations

• Only supports iPython blocks
• Does not do inline images in results
• Will not support src-block variables
• Currently only supports vanilla output results

## 4 Summary

The code that does this is here: ox-ipynb.el . After I use it a while I will put it in scimax. There are some tricks in it to fix up some markdown export of latex fragments and links with no descriptions.

I just run this command in Emacs to get the notebook. Even it renders reasonably in the notebook.

(export-ipynb-buffer)
`

Overall, this looks extremely promising to develop lecture notes and assignments in org-mode, but export them to Ipython notebooks for the students.