Literate programming in python with org-mode and noweb

| categories: python, org-mode | tags:

This post examines a different approach to literate programming with org-mode that uses noweb . I have adapted an example from http://home.fnal.gov/~neilsen/notebook/orgExamples/org-examples.html which has some pretty cool ideas in it.

The gist of using noweb is that in your source blocks you have labels like <<imports>>, that refer to other named code blocks that get substituted in place of the label. In the example below, we put labels for a code block of imports, for a function definition, a class definition, and a main function. This code block will get tangled to main.py . The noweb expansion happens at export, so here is the literal code block:

#+BEGIN_SRC python :noweb yes :tangle main.py
<<imports>>

<<some-func>>

<<class-dfn>>

<<main-func>>

if __name__ == '__main__':
    status = main()
    sys.exit(status)
#+END_SRC

You may want to just check out the org-mode source link at the bottom of the post to see all the details.

import sys
import numpy as np
import matplotlib.pyplot as plt

from argparse import ArgumentParser

def utility_func(arg=None):
    return 'you called a utility function with this arg: {0}'.format(arg)

class HelloWorld(object):
    def __init__(self, who):
        self.who = who

    def __call__(self):
        return 'Hello {0}'.format(self.who)

    def test(self):
        return True

def main():
    parser = ArgumentParser(description="Say hi")
    parser.add_argument("-w", "--who", 
                        type=str,
                        default="world",
                        help="Who to say hello to")
    args = parser.parse_args()
  
    who = args.who
  
    greeter = HelloWorld(who)
    greeter()

    print 'test func = ', greeter.test()
  
    print utility_func()
    print utility_func(5)

    return 0

if __name__ == '__main__':
    status = main()
    sys.exit(status)

1 imports

Now, we define a block that gives us the imports. We do not have to use any tangle headers here because noweb will put it in where it belongs.

import sys
import numpy as np
import matplotlib.pyplot as plt

from argparse import ArgumentParser

2 utility function

Now we define a function we will want imported from the main file.

def utility_func(arg=None):
    return 'you called a utility function with this arg: {0}'.format(arg)

3 class definition

Finally, let us define a class. Note we use noweb here too, and we get the indentation correct!

class HelloWorld(object):
    def __init__(self, who):
        self.who = who

    def __call__(self):
        return 'Hello {0}'.format(self.who)

    def test(self):
        return True

3.1 some class function

Now, let us make the some-other-func. This block is not indented, but with the noweb syntax above, it seems to get correctly indented. Amazing.

def test(self):
    return True

4 The main function

This is a typical function that could be used to make your module into a script, and is only run when the module is used as a script..

def main():
    parser = ArgumentParser(description="Say hi")
    parser.add_argument("-w", "--who", 
                        type=str,
                        default="world",
                        help="Who to say hello to")
    args = parser.parse_args()
  
    who = args.who
  
    greeter = HelloWorld(who)
    greeter()

    print 'test func = ', greeter.test()
  
    print utility_func()
    print utility_func(5)

    return 0

5 Tangle and run the code

This link will extract the code to main.py:

elisp:org-babel-tangle

We can run the code like this (linux):

python main.py --w John 2>&1
true
test func =  True
you called a utility function with this arg: None
you called a utility function with this arg: 5

or this (windows, which as no sh)

from main import *

main()
test func =  True
you called a utility function with this arg: None
you called a utility function with this arg: 5

6 Summary thoughts

The use of noweb syntax is pretty cool. I have not done anything serious with it, but it looks like you could pretty easily create a sophisticated python module this way that is documented in org-mode.

Copyright (C) 2014 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.5h

Discuss on Twitter

Make a list of org-files in all the subdirectories of the current working directory

| categories: recursive, org-mode, emacs | tags:

It would be helpful to get a listing of org-files in a directory tree in the form of clickable links. This would be useful, for example, to find all files associated with a project in a directory with a particular extension, or to do some action on all files that match a pattern. To do this, we will have to recursively walk through the directories and examine their contents.

Let us examine some of the commands we will need to use. One command is to get the contents of a directory. We will explore the contents of a directory called literate in my computer.

;; list contents of the directory
(let ((abspath nil)
      (match nil)
      (nosort t))
  (directory-files "literate" abspath match nosort))
makefile-main Makefile main.o main.f90 main literate.org hello.f90 circle.o circle.mod circle.f90 circle-area.png archive a.out .. .

Note the presence of . and ... Those stand for current directory and one directory up. We should remove those from the list. We can do that like this.

;; remove . and ..
(let ((abspath nil)
      (match nil)
      (nosort t))
  (remove "." 
          (remove ".." 
                  (directory-files "literate" abspath match nosort))))
makefile-main Makefile main.o main.f90 main literate.org hello.f90 circle.o circle.mod circle.f90 circle-area.png archive a.out

Next, we need to know if a given entry in the directory files is a file or a directory. Emacs-lisp has a few functions for that. We use absolute filenames here since the paths are relative to the "molecules" directory. Note we could use absolute paths in directory-files, but that makes it hard to remove "." and "..".

;; print types of files in the directory
(let ((root "literate")
      (abspath nil)
      (match nil)
      (nosort t))
  (mapcar (lambda (x)
            (cond
             ((file-directory-p (expand-file-name x root))
              (print (format "%s is a directory" x)))
             ((file-regular-p (expand-file-name x root))
              (print (format "%s is a regular file" x)))))
          (remove "." 
                  (remove ".." 
                          (directory-files root abspath match nosort)))))
"makefile-main is a regular file"

"Makefile is a regular file"

"main.o is a regular file"

"main.f90 is a regular file"

"main is a regular file"

"literate.org is a regular file"

"hello.f90 is a regular file"

"circle.o is a regular file"

"circle.mod is a regular file"

"circle.f90 is a regular file"

"circle-area.png is a regular file"

"archive is a directory"

"a.out is a regular file"

Now, we are at the crux of this problem. We can differentiate between files and directories. For each directory in this directory, we need to recurse into it, and list the contents. There is some code at http://turingmachine.org/bl/2013-05-29-recursively-listing-directories-in-elisp.html which does this, but I found that I had to modify the code to not list directories, and here I want to show a simpler recursive code.

(defun os-walk (root)
  "recursively walks through directories getting list of absolute paths of files"
  (let ((files '()) ; empty list to store results
        (current-list (directory-files root t)))
    ;;process current-list
    (while current-list
      (let ((fn (car current-list))) ; get next entry
        (cond 
         ;; regular files
         ((file-regular-p fn)
          (add-to-list 'files fn))
         ;; directories
         ((and
           (file-directory-p fn)
           ;; ignore . and ..
           (not (string-equal ".." (substring fn -2)))
           (not (string-equal "." (substring fn -1))))
          ;; we have to recurse into this directory
          (setq files (append files (os-walk fn))))
        )
      ;; cut list down by an element
      (setq current-list (cdr current-list)))
      )
    files))

(os-walk "literate")
c:/Users/jkitchin/Dropbox/blogofile-jkitchin.github.com/blog/literate/makefile-main c:/Users/jkitchin/Dropbox/blogofile-jkitchin.github.com/blog/literate/main.o c:/Users/jkitchin/Dropbox/blogofile-jkitchin.github.com/blog/literate/main.f90 c:/Users/jkitchin/Dropbox/blogofile-jkitchin.github.com/blog/literate/main c:/Users/jkitchin/Dropbox/blogofile-jkitchin.github.com/blog/literate/literate.org c:/Users/jkitchin/Dropbox/blogofile-jkitchin.github.com/blog/literate/hello.f90 c:/Users/jkitchin/Dropbox/blogofile-jkitchin.github.com/blog/literate/circle.o c:/Users/jkitchin/Dropbox/blogofile-jkitchin.github.com/blog/literate/circle.mod c:/Users/jkitchin/Dropbox/blogofile-jkitchin.github.com/blog/literate/circle.f90 c:/Users/jkitchin/Dropbox/blogofile-jkitchin.github.com/blog/literate/circle-area.png c:/Users/jkitchin/Dropbox/blogofile-jkitchin.github.com/blog/literate/a.out c:/Users/jkitchin/Dropbox/blogofile-jkitchin.github.com/blog/literate/Makefile c:/Users/jkitchin/Dropbox/blogofile-jkitchin.github.com/blog/literate/archive/empty-text-file.txt

Nice, that gives us a recursive listing of all the files in this directory tree. Let us take this a step further, and apply a function to that list to filter out a list of the org files. We will also create org-links out of these files.

(defun os-walk (root)
  (let ((files '()) ;empty list to store results
        (current-list (directory-files root t)))
    ;;process current-list
    (while current-list
      (let ((fn (car current-list))) ; get next entry
        (cond 
         ;; regular files
         ((file-regular-p fn)
          (add-to-list 'files fn))
         ;; directories
         ((and
           (file-directory-p fn)
           ;; ignore . and ..
           (not (string-equal ".." (substring fn -2)))
           (not (string-equal "." (substring fn -1))))
          ;; we have to recurse into this directory
          (setq files (append files (os-walk fn))))
        )
      ;; cut list down by an element
      (setq current-list (cdr current-list)))
      )
    files))

(require 'cl)

(mapcar 
 (lambda (x) (princ (format "[[%s][%s]]\n" x (file-relative-name x "."))))
 (remove-if-not 
  (lambda (x) (string= (file-name-extension x) "org"))
  (os-walk "literate")))

literate/literate.org

That is certainly functional. It might be nice to format the links a bit nicer to show their structure in a table of contents way, or to sort them in a nice order if there were many of these files.

Copyright (C) 2014 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.5h

Discuss on Twitter

Using yasnippet to get completion in ref links

| categories: org-mode | tags:

This post illustrates an alternative approach to completion in creating ref links compared to the approach shown here . In this approach we use a dynamic yasnippet to do the completion. We start with similar code that I used before to get a list of labels from the buffer. I used a slightly different regexp to recognize links in this version.

label:code-example

(defun get-labels ()
  (interactive)
  (save-excursion
    (goto-char (point-min))
    (let ((matches '()))
      (while (re-search-forward "label:\\([a-zA-z0-9:-]*\\)" (point-max) t)
        (add-to-list 'matches (match-string-no-properties 1) t))
      matches)))
get-labels

Let us see that in action:

(get-labels)
code-example \\ code:ref-snippet load-snippets

I think the \\ link is an artifact of the regexp in my get-labels code, and it would not appear in other examples.

Now, we are going to create a yasnippet that uses the list returned from get-labels to provide your choices. See http://capitaomorte.github.io/yasnippet/snippet-development.html#sec-3-8 for some details. We will tangle this code block into a local snippets directory.

label:code:ref-snippet

# -*- mode: snippet -*-
# --
ref:${1:$$(yas-choose-value (get-labels))} $0

Now we load the snippets directory.

label:load-snippets

(yas-load-directory "./snippets")

Finally, we can type ref, press tab to complete it, and then select the label you want from a list. Here are some examples:

ref:code-example

ref:code:ref-snippet

That also works! I cannot decide if I like this better than the Emacs completion. yasnippet gives a popup menu, which is not as easy to navigate as the Emacs completion mechanism. It also requires a working yasnippet, which has not made it into my regular work flows too often. I think I like the Emacs completion better (which actually goes through Icicles since I have that installed). I like it better because I do not have to leave the keyboard or use the arrow buttons to choose a label. However, I do need to bind that function to some key to use it, or type in the command name. It turns out I do not use ref links too often, so it is not too burdensome.

Copyright (C) 2014 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.5h

Discuss on Twitter

Storing label links in org-mode

| categories: org-mode | tags:

I am continuing to evolve how I can use org-mode. I have created a label link, which if clicked on checks to see that the label is unique in the buffer. It would be nice to be able to be on a

label:some-label
link, and to store it so we could create a
ref:some-label
later. That ref link is also clickable, and it jumps to the label it refers to, and provides a C-c & option to get back to the ref link. org-mode allows you to create org-PREFIX-store-link functions which store the link information. These functions must determine if they are responsible for storing the link and return nil if not. The first challenge is figuring out if the cursor is on a label link. Here is a function that does that.

This was a little challenging. The strategy to determine if the cursor is in a link is to search backward for a regular expression matching a label link. I found this was not sufficient, because it appeared to me that the matched string was only between the beginning of the label link and the point where the cursor was. So, after finding the beginning of the first label link before the cursor, then we search forward to find the whole link. Then we determine if the cursor is between the beginning and end of the match. If it is, then we are on a label link. Here is the code.

(defun in-label-link ()
  "return label if in a label link, or return nil

we store point, search forward to the first space, and backward to the previous space. then make sure label: is between them."
  (interactive)
  (let ((current-point (point)))
    (save-excursion
      (re-search-backward "label:\\([a-zA-z0-9:-]*\\)" (point-min) t)
      (re-search-forward "label:\\([a-zA-Z0-9-:]*\\)" (point-max) t)   
      (if (and (> current-point (match-beginning 0))
               (< current-point (match-end 0)))
          t
        nil))))

This code works for these kinds of links as far as I can tell. Interestingly, it only works when the cursor is to the right of label:. I am not sure if that is because of the regular expression or not.

label:plain-beginning

label:telabel

label:fig:test

label:bracket-in-line

Now, we create the code that stores the link. We only execute the code if we pass the function that checks if we are on a label link. If we are, then the label is stored in (match-string 1), and we create the link and store it. Finally, we add the function to org-store-link-functions so that it will be used when C-c l is pressed.

(defun org-label-store-link ()
  "store a link to a label. The output will be a ref to that label"
  ;; First we have to make sure we are on a label link. 
  (when (in-label-link)
    (org-store-link-props
     :type "ref"
     :link (concat "ref:" (match-string 1)))))

(add-hook 'org-store-link-functions 'org-label-store-link)

So, here is the evidence that it worked:

ref:plain-beginning

ref:telabel

ref:fig:test

ref:bracket-in-line

For each of these, I put the cursor on the labels, pressed C-c l, and then moved the cursor down here and pressed C-c C-l, and pressed enter and PRESTO! I had the reference that I wanted! That seems like a handy trick.

Copyright (C) 2014 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.5h

Discuss on Twitter

Using completion in ref links

| categories: org-mode | tags:

I came across this interesting post on using completion in links: http://draketo.de/light/english/free-software/custom-link-completion-org-mode-25-lines-emacs . I like the idea, but the type-flow for is not how I usually insert links. For the method there to work, you have to enter a link with C-c C-l, partially enter the link type, press enter, and then partially enter the description, which can be completed with tab. That is a lot of typing to me, compared to what I usually do which is type the link in directly. That habit does not work too well in large documents, and always has the possibility of a typo in the link, which then does not work or export correctly.

Here I explore how to make a

ref:label
link using a function that provides all the options available as labels. The idea is to write a function that generates a list of labels in the buffer, which you can make a link to. Let us try an interactive function with a list of arguments. We are first going to generate a list of labels from the buffer. We use this code to get a list of labels in the buffer. You will get to choose which label you want a link to, and the function will insert it for you. Here it is:

label:code-example

(defun get-labels ()
  (interactive)
  (save-excursion
    (goto-char (point-min))
    (let ((matches '()))
      (while (re-search-forward "label:\\(.*\\)" (point-max) t)
        (add-to-list 'matches (match-string-no-properties 1) t))
      matches)))

(defun org-insert-ref-link (&optional arg)
  (interactive (list (completing-read "label: " (get-labels))))
  (insert (format "ref:%s" arg)))
org-insert-ref-link

So, here you run the command with M-x org-insert-ref-link, press tab, and select the label you want to use. A link like this gets inserted in your buffer

ref:code-example
. This is pretty nice. It should reduce the number of ref link mistakes, and make it easier to find the labels in the whole buffer.

Copyright (C) 2014 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.5h

Discuss on Twitter
« Previous Page -- Next Page »