The Kitchin Research Group

MS Word has a few things I like. One of them is the ability to autoformat things to make an ordinal number string like 1st to the superscripted version 1^st while you type or a 1/2 to ½. I thought it would be pretty easy to implement that for org-mode. It turns out it was not so easy!

There does not appear to be a way to specify a regexp pattern as an abbreviation, or an abbrev that starts with a number. What we need for ordinal numbers is to recognize a sequence of numbers followed by "st", "nd", "rd" or "th" followed by a space or punctuation, and then superscript the letters. In case you didn't want the replacement to occur, you should be able to undo it and get back the original string. This addition was a little hard won, so I am sharing the lessons here.

The logic I used is to put a function in the post-self-insert-hook. The function only works in org-mode, when not in a codeblock and when looking back at a regexp that matches a pattern to be replaced. Getting it to undo was trickier than expected. Eventually I worked out that you put an undo boundary in place before the change, and then it seems like you can undo the changes. I created a minor mode so it is easy to toggle this on and off.

Here is the implementation:

(defcustom scimax-autoformat-ordinals t
  "Determines if scimax autoformats ordinal numbers."
  :group 'scimax)

(defun scimax-org-autoformat-ordinals ()
  "Expand ordinal words to superscripted versions in org-mode.
1st to 1^{st}.
2nd to 2^{nd}
3rd to 3^{rd}
4th to 4^{th}"
  (interactive)
  (when (and scimax-autoformat-ordinals
             (eq major-mode 'org-mode)
             (not (org-in-src-block-p))
             (looking-back "\\(?3:\\<\\(?1:[0-9]+\\)\\(?2:st\\|nd\\|rd\\|th\\)\\>\\)\\(?:[[:punct:]]\\|[[:space:]]\\)"
                           (line-beginning-position)))
    (undo-boundary)
    (save-excursion
      (replace-match "\\1^{\\2}" nil nil nil 3))))


(defcustom scimax-autoformat-fractions t
  "Determines if scimax autoformats fractions."
  :group 'scimax)


(defun scimax-org-autoformat-fractions ()
  "Expand fractions to take up space."
  (interactive)
  (when (and scimax-autoformat-fractions
             (eq major-mode 'org-mode)
             (not (org-in-src-block-p))
             (looking-back "\\(?3:\\<\\(1/4\\|1/2\\|3/4\\)\\>\\)\\(?:[[:punct:]]\\|[[:space:]]\\)"
                           (line-beginning-position)))
    (undo-boundary)
    (save-excursion
      (replace-match (cdr (assoc (match-string 3) '(("1/4" . "¼")
                                                    ("1/2" . "½")
                                                    ("3/4" . "¾"))))
                     nil nil nil 3))))

(defun scimax-org-autoformat ()
  "Autoformat functions."
  (scimax-org-autoformat-ordinals)
  (scimax-org-autoformat-fractions))

(define-minor-mode scimax-autoformat-mode
  "Toggle `scimax-autoformat-mode'.  Converts 1st to 1^{st} as you type."
  :init-value nil
  :lighter (" om")
  (if scimax-ordinal-mode
      (add-hook 'post-self-insert-hook #'scimax-org-autoformat nil 'local)
    (remove-hook 'post-self-insert-hook #'scimax-org-autoformat 'local)))

This is now a feature in scimax. This marks the 500^th blog post! That is ½ way to 1000. At the current rate of posting, it will be at least 5 years until I hit that!

Copyright (C) 2017 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 9.0.5

Over on Stackoverflow someone wanted a better return in org-mode. They wanted return to add items in a list (instead of M-Ret). Someone posted a partial solution, and here I improve on it to add new items to lists, new headings after a heading, and new rows to tables. In each case, a double return on an empty item, headline or table row will delete that line, and terminate the list, headlines or table. You can still use M-Ret, and this function falls through to org-return like it did before. You can use a prefix arg to get a regular return if you want one (e.g. you want to press enter on a headline to push it down).

Here is the function. Give it a try. It is a small but helpful addition I think. I have not used it for long, so if you come across issues leave a comment!

(require 'org-inlinetask)

(defun scimax/org-return (&optional ignore)
  "Add new list item, heading or table row with RET.
A double return on an empty element deletes it.
Use a prefix arg to get regular RET. "
  (interactive "P")
  (if ignore
      (org-return)
    (cond

     ((eq 'line-break (car (org-element-context)))
      (org-return-indent))

     ;; Open links like usual, unless point is at the end of a line.
     ;; and if at beginning of line, just press enter.
     ((or (and (eq 'link (car (org-element-context))) (not (eolp)))
          (bolp))
      (org-return))

     ;; It doesn't make sense to add headings in inline tasks. Thanks Anders
     ;; Johansson!
     ((org-inlinetask-in-task-p)
      (org-return))

     ;; checkboxes too
     ((org-at-item-checkbox-p)
      (org-insert-todo-heading nil))

     ;; lists end with two blank lines, so we need to make sure we are also not
     ;; at the beginning of a line to avoid a loop where a new entry gets
     ;; created with only one blank line.
     ((org-in-item-p)
      (if (save-excursion (beginning-of-line) (org-element-property :contents-begin (org-element-context)))
          (org-insert-heading)
        (beginning-of-line)
        (delete-region (line-beginning-position) (line-end-position))
        (org-return)))

     ;; org-heading
     ((org-at-heading-p)
      (if (not (string= "" (org-element-property :title (org-element-context))))
          (progn (org-end-of-meta-data)
                 (org-insert-heading-respect-content)
                 (outline-show-entry))
        (beginning-of-line)
        (setf (buffer-substring
               (line-beginning-position) (line-end-position)) "")))

     ;; tables
     ((org-at-table-p)
      (if (-any?
           (lambda (x) (not (string= "" x)))
           (nth
            (- (org-table-current-dline) 1)
            (org-table-to-lisp)))
          (org-return)
        ;; empty row
        (beginning-of-line)
        (setf (buffer-substring
               (line-beginning-position) (line-end-position)) "")
        (org-return)))

     ;; fall-through case
     (t
      (org-return)))))


(define-key org-mode-map (kbd "RET")
  'scimax/org-return)

Here are a few tests:

1. numbered item
2. second item
   1. nested number
   2. second number

• unordered 1
• unordered 2
  • nested
  • nested 2
    • nested with link: http://emacs.stackexchange.com

• [ ] check 1
• [ ] check 2
• [ ] check 3

Copyright (C) 2017 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 9.0.5

I have been interested in functional text for a long time. With functional text you can read it, but also interact with it kitchin-2015-examp. Lately I have been thinking about how to use some features of object-oriented programming to functional text. The premise is to use an object hierarchy to encapsulate some knowledge, and provide functionality on the objects. We can use inheritance to customize some of this knowledge and functionality.

The example I will work out here is to provide functional text for chemical elements. The goal is to define some objects that represent elements, and construct font-lock rules from the objects to make the text functional in Emacs. Functional here means it stands out so we know there is something special about it, it has a tooltip to get some information (like what type of element it is, and its atomic mass), and it is clickable to get more functionality.

This post will make a lot more sense in this video: https://www.youtube.com/watch?v=IWxCj5cr8rY

First, we create a base class of an Element. I use an instance tracker for this to make book keeping easy later. The base class will have a name, synonyms for the name, and a default face to color it. We define a few methods to get an atomic mass and search google for the element. Finally, we provide a function to generate a tooltip, and a font-lock rule.

(defvar atomic-masses '(("Hydrogen" . 1.008)
                        ("Argon" . 39.948)
                        ("Sodium" . 22.989)
                        ("Palladium" . 106.42))
  "a-list of atomic masses.")

(defvar elements '() "List of known elements")
(setq elements '()) ;; this is to start over

(defclass element (eieio-instance-tracker)
  ((tracking-symbol :initform elements
                    :documentation "Variable that holds all class instances.")
   (name :initarg :name
         :documentation "The name of the element")
   (synonyms :initarg :synonyms :initform '()
             :documentation "List of regular expressions that match the element.")
   (face :initarg :face :initform 'font-lock-type-face
         :documentation "The face to use with font-lock."))
  "Base class for a chemical element.")

(defmethod element-atomic-mass ((x element))
  "Return atomic mass from `atomic-masses'."
  (cdr (assoc (oref x :name) atomic-masses)))

(defmethod element-help-echo ((x element))
  "A tooltip for the element.
It will look like class (inherited classes) mass=atomic-mass"
  (format "%s %s: mass=%s"
          (eieio-object-class x)
          (mapcar 'eieio-class-name (eieio-class-parents (eieio-object-class x)))
          (or (element-atomic-mass x) "unknown")))

(defmethod element-search ((x element))
  "Search google for the element"
  (google-this-string nil (oref x :name) t))

(defmethod element-font-lock-rule ((x element))
  "Return font-lock rule for the element."
  (let ((map (make-sparse-keymap)))
    (define-key map [mouse-1]
      (lambda ()
        "Construct the object and run `element-search' on it."
        (interactive)
        (element-search
         (eieio-instance-tracker-find
          (get-text-property (point) 'element-name)
          :name 'elements))))

    (list
     ;; Construct the pattern to match
     (rx-to-string `(: bow
                       (or  ,(oref x :name)
                            ,@(loop for sy in (oref x :synonyms)
                                    collect `(regexp ,sy)))
                       eow))
     0  ;; font-lock the whole match
     ;; These are the properties to put on the matches
     `(quote (face ,(oref x :face)
                   element-name ,(oref x :name)
                   local-map ,map
                   mouse-face 'highlight
                   help-echo ,(element-help-echo x))))))

Now, we can define some sub-classes. These are families of elements. For a metal, we change the face. For noble gases, we override the help-echo function, and for alkali metals we override the search function. The point is that we can customize the behavior for different classes.

(defclass metal (element)
  ((face :initform '(:foreground "orange" :underline t)))
  "Metal")

(defclass noble-gas (element)
  ()
  "A noble gas")

(defmethod element-help-echo ((x noble-gas))
  "I am not a common element.")

(defclass alkali (element metal)
  ()
  "Alkali metal")

(defmethod element-search ((x alkali))
  (let ((visible-bell t))
    (beep)
    (message "You clicked on an alkali metal: %s." (oref x :name))))

Now we can define some elements. These are all instances of each class. For some, we define synonyms, and alternate appearances. Note the synonyms are regular expressions.

(element :name "Hydrogen" :synonyms '("H" "[hH]ydrogen"))

(noble-gas :name "Argon" :synonyms '("Ar"))

(alkali :name "Sodium" :synonyms '("Na" "[nN]atrium"))
(alkali :name "Potassium" :synonyms '("K") :face '(:foreground "red"))

(metal :name "Palladium")

The instance tracker shows us the defined objects.

elements

(font-lock-add-keywords
 nil
 (mapcar 'element-font-lock-rule elements))

(font-lock-fontify-buffer)

Copyright (C) 2017 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 9.0.5

I have been thinking of better ways to write code that is more likely to have decent docstrings that are up to date, and maybe that enable automatic validation. One strategy is to keep documentation and code together, and by together I mean close together. The closer the better. I made some interesting progress in the last post, where I used a macro to let me put argument specific documentation in the same place that the argument is defined. Here I expand the idea to also provide argument default values, and validation code where the argument is defined inside the function, in addition to generating docstrings. This post is written in Emacs-lisp, mostly because I am more familiar with the macro language. The idea should apply to other lisps too.

Let's consider this prototypical, vanilla function definition, usage, and docstring.

(defun f1 (arg1 arg2)
  "Add two numbers."
  (+ arg1 arg2))

;; usage
(f1 3 4)

7

Here is what the help looks like from emacs.

(describe-function 'f1)

f1 is a Lisp function.

(f1 ARG1 ARG2)

For more information check the manuals.

Add two numbers.

It is clear I was lazy in writing the docstring; it does not even mention the arguments. There is also no validation of the arguments so if you pass a string and a number, you will get an error. There are no defaults either, so you have to provide both arguments. It seems like there could be significant room for improvement. Of course, I could bite the bullet and write a better function like this one:

(defun f1a (arg1 &optional arg2)
  "Add ARG1 and ARG2 together.
ARG1 and  ARG2 should both be numbers."
  (when (null arg2) (setq arg2 2))
  (unless (and (numberp arg1) (numberp arg2)) (error "arg1 and arg2 should both be numbers"))
  (+ arg1 arg2))

(list (f1a 3 4) (f1a 3))

Yes, I could do that, but it is tedious to do it all the time. And it still leaves something to be desired for me. The docstring does not say what the default value is for example, and that is hard-coded in the code, i.e. not introspectible until you look at the code. Next we consider an alternative way to write the function. Compare that to this function definition, usage and documentation. The function definition is a little more verbose. Providing documentation, defaults and validation code in any form would make it that way no matter what.

(defn f2 ((arg1 "A number" :validate numberp)
          (arg2 "A number" :validate numberp :default 2))
  "Add the arguments."
  (+ arg1 arg2))

;; usage
(list (f2 3 4) (f2 3))

(describe-function 'f2)

f2 is a Lisp function.

(f2 ARG1 &optional ARG2)

For more information check the manuals.

Add the arguments.
ARG1 : A number (valid = numberp)
ARG2 : A number (default = 2) (valid = numberp)

The documentation is built up from the information in the function definition, in a form that is mostly consistent with emacs-lisp documentation standards. defn is not a regular emacs-lisp function; it is a macro I developed to generate the function code. It turned out to be long, but the gist of it is that before defining the function I loop through the arguments and collect the docstrings, along with any information about default values and/or validation functions. Then I build up the list of arguments to put in the function. Then if any default values are set, I generate some code to set those values if they are not set in the function call, and finally a similar block of validation code. At the end, I construct the defun and return it. You can check out the code if you want here: https://github.com/jkitchin/scimax/blob/master/scimax-macros.el.

Let's take a look at what this code expands to.

(macroexpand-1
 '(defn f2 ((arg1 "A number" :validate numberp)
            (arg2 "A number" :validate numberp :default 2))
    "Add the arguments."
    (+ arg1 arg2)))

(defun f2
    (arg1 &optional arg2)
  "Add the arguments.\nARG1 : A number (valid = numberp)\nARG2 : A number (default = 2) (valid = numberp)\n"
  (progn
    (when
        (null arg2)
      (setq arg2 2)))
  (progn
    (unless
        (funcall 'numberp arg1)
      (error "In (%s %s) Expected %s to pass %S. Got %S" "f2" "(arg1 &optional arg2)" "arg1" 'numberp arg1))
    (unless
        (funcall 'numberp arg2)
      (error "In (%s %s) Expected %s to pass %S. Got %S" "f2" "(arg1 &optional arg2)" "arg2" 'numberp arg2)))
  (+ arg1 arg2))

You can see it expands to a regular defun, with a generated docstring, generated default settings code block, and generated validation code. Pretty nice.

Let's see what happens with a function that fails the validation. We should get an error. Here we capture the error so we can see it in the post.

(condition-case err
    (f2 "oak")
  (error
   (error-message-string err)))

In (f2 (arg1 &optional arg2)) Expected arg1 to pass numberp. Got "oak"

So we even get a useful error message when the wrong type of argument is provided. Compare that to the error message from the original version of this function. It tells us we got the wrong type, but not which argument.

(condition-case err
    (f1 "oak" 4)
  (error
   (error-message-string err)))

Wrong type argument: number-or-marker-p, "oak"

One last example to check out the &rest argument, with validation that every arg is a number.

(defn f4 ((rarg :rest
                :validate (lambda (x)
                            (-all-p 'identity (mapcar 'numberp x)))))
  "multiply all the arguments."
  (apply '* rarg))

(f4 1 2 3)

6

(condition-case err
    (f4 "oak" 4)
  (error
   (error-message-string err)))

In (f4 (&rest rarg)) Expected rarg to pass (lambda (x) (-all-p (quote identity) (mapcar (quote numberp) x))). Got ("oak" 4)

(describe-function 'f4)

f4 is a Lisp function.

(f4 &rest RARG)

For more information check the manuals.

multiply all the arguments.
RARG : No documentation

That looks ok too.

Copyright (C) 2017 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 9.0.7

I am going to use Jupyter notebooks to teach from this semester. I really dislike preparing notebooks though. A browser is a really poor editor, and I really dislike Markdown. Notebooks do not seem to have any real structure in them, e.g. the collapsible outline that I am used to in org-mode, so for long notebooks, it is difficult to get a sense for the structure. I am anticipating spending up to 80 hours preparing notebooks this semester, so today I worked out some code to export org-mode to an ipython notebook!

This will let me use the power tools I am accustomed to for the creation of IPython notebooks for my students, and perhaps others who do not use org-mode.

Jupyter notebooks are just json files, so all we need to do is generate it from an org document. The basic strategy was to build up a lisp data structure that represents the notebook and then just convert that data structure to json. I split the document up into sequential markdown and code cells, and then encode those in the format required for the notebook (json).

So, here is an example of what can be easily written in org-mode, posted to this blog, and exported to an IPython notebook, all from one org-document.

Check out the notebook: exporting-orgmode-to-ipynb.ipynb .

def objective(x):
    return x**2 - 4

from scipy.optimize import fsolve

ans = fsolve(objective, 3)
print(ans)

[ 2.]

ans, = fsolve(objective, -3)
print(ans)

-2.0

ans = fsolve(objective, -3)

print(float(ans))
print(ans[0])

-2.0000000000000084
-2.0

import numpy as np

t = np.linspace(0, 2 * np.pi)

x = np.cos(t)
y = np.sin(t)

import matplotlib.pyplot as plt
plt.plot(x, y)
plt.axis('equal')
plt.xlabel('x')
plt.ylabel('y')
plt.savefig('circle.png')

(export-ipynb-buffer)

Copyright (C) 2017 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 9.0.3