Getting geo-tagged information from photos for blogging

| categories: emacs, orgmode, geotag | tags: | View Comments

I am kind of late to this game, but recently I turned on location services for the camera on my phone. That means the location of the photo is stored in the photo, and we can use that to create urls to the photo location in a map for example. While traveling, I thought this would be a good application for org-mode to add functionality to documents with photos in them, e.g. to be able to click on them to see where they are from, or to automate creation of html pages with links to maps, etc. In this post I explore some ways to achieve those ideas. What I would like is a custom org link that shows me a thumbnail of the image, and which exports to show the image in an html file with a link to a pin on Google maps.

So, let's dig in. Imagemagick provides an identify command that can extract the information stored in the images. Here we consider just the GPS information. I some pictures on a recent vacation, and one is unimaginatively named IMG_1759.JPG. Let's see where it was taken.

identify -verbose IMG_1759.JPG | grep GPS
exif:GPSAltitude: 14426/387    
exif:GPSAltitudeRef: 0    
exif:GPSDateStamp: 2018:06:30    
exif:GPSDestBearing: 11767/80    
exif:GPSDestBearingRef: T    
exif:GPSImgDirection: 11767/80    
exif:GPSImgDirectionRef: T    
exif:GPSInfo: 1632    
exif:GPSLatitude: 22/1, 11/1, 614/100
exif:GPSLatitudeRef: N    
exif:GPSLongitude: 159/1, 40/1, 4512/100
exif:GPSLongitudeRef: W    
exif:GPSSpeed: 401/100    
exif:GPSSpeedRef: K    
exif:GPSTimeStamp: 3/1, 44/1, 3900/100

The interpretation here is that I took that photo at latitude 22° 11' 6.14" N, and longitude 159° 40' 45.12" W. Evidently I was moving at 4.01 in some unit; I can confirm that I was at least moving, I was on a ship when I took that picture, and it was moving.

According to this you can make a url to a Google maps pin in satellite picture mode that looks like this: http://maps.google.com/maps?q=22 11 6.14N,159 40 45.12W&t=k. It doesn't seem possible to set the zoom in this url (at least setting the zoom doesn't do anything, and I didn't feel like trying all the other variations that are reported to sometimes work). I guess that is ok for now, it adds some suspense that you have to zoom out to see where the image is in some cases.

We need a little function to take an image file and generate that link. We have to do some algebra on the latitude and longitude which are stored as integers with a division operator. I am going to pipe this through an old unix utility called bc mostly because it is simple, and I won't have to parse it much. bc is a little archaic, you have to set the scale first, which tells it how many decimal places to output. The degrees and minutes are integers, so we will have to deal with that later.

echo "scale=2; 614/100" | bc
6.14

Here is our function. I filter out the lines with GPS in them into an a-list. Then, I grab out the specific quantities I want and construct the url. There is a little hackery since it appears the degrees and minutes should be integers in the url formulation used here, so I convert them to numbers and then take the floor. The function is a little longer than I thought, but it isn't too bad I guess. It is a little repetitious, but not enough to justify refactoring.

(defun iphoto-map-url (fname)
  (let* ((gps-lines (-keep (lambda (line)
                             (when (s-contains? "GPS" line) (s-trim line)))
                           (process-lines "identify" "-verbose" fname)))
         (gps-alist (mapcar (lambda (s) (s-split ": " s t))  gps-lines))
         (latitude (mapcar
                    (lambda (s)
                      (s-trim (shell-command-to-string
                               (format "echo \"scale=2;%s\" | bc" s))))
                    (s-split "," (cadr (assoc "exif:GPSLatitude" gps-alist)))))
         (latitude-ref (cadr (assoc "exif:GPSLatitudeRef" gps-alist)))
         (longitude (mapcar
                     (lambda (s)
                       (s-trim
                        (shell-command-to-string
                         (format "echo \"scale=2;%s\" | bc" s))))
                     (s-split "," (cadr (assoc "exif:GPSLongitude" gps-alist)))))
         (longitude-ref (cadr (assoc "exif:GPSLongitudeRef" gps-alist))))
    (s-format "http://maps.google.com/maps?q=$0 $1 $2$3,$4 $5 $6$7&t=k"
              'elt
              (list
               (floor (string-to-number (nth 0 latitude)))
               (floor (string-to-number (nth 1 latitude)))
               (nth 2 latitude)
               latitude-ref
               (floor (string-to-number (nth 0 longitude)))
               (floor (string-to-number (nth 1 longitude)))
               (nth 2 longitude)
               longitude-ref))))
iphoto-map-url

Here is the function in action, making the url.

(iphoto-map-url "IMG_1759.JPG")
http://maps.google.com/maps?q=22 11 6.14N,159 40 45.12W&t=k

It is kind of slow, but that is because the identify shell command is kind of slow when you run it with the -verbose tag. Now, I would like the following things to happen when I publish it to html:

  1. I want the image wrapped in an img tag inside a figure environment.
  2. I want the image to by hyperlinked to its location in Google maps.

In the org file, I want a thumbnail overlay on it, so I can see the image while writing, and I want it to toggle like other images. I use an iPhone to take the photos, so we will call it an iphoto link.

Here is the html export function I will use. It is a little hacky that I hard code the width in at 300 pixels, but I didn't feel like figuring out how to get it from an #+attr_html line right now. It probably requires a filter function where you have access to the actual org-elements. I put the url to the image location in a figure caption here.

(defun iphoto-export (path desc backend)
  (cond
   ((eq 'html backend)
    (format "<figure>
<img src=\"%s\" width=\"300\">
%s
</figure>"
            path
            (format "<figcaption>%s <a href=\"%s\">map</a></figcaption>"
                    (or desc "")
                    (iphoto-map-url path))))))
iphoto-export

Ok, the last detail I want is to put an image overlay on my new link so I can see it. I want this to work with org-toggle-inline-images so I can turn the images on and off like regular image links with C-c C-x C-v. This function creates overlays as needed, and ties into the org-inline-image-overlays so they get deleted on toggling. We have to advise the display function to redraw these, which we clumsily do by restarting the org font-lock machinery which will redraw the thumbnails from the activate-func property of the links. I also hard code the thumbnail width in this function, when it could be moved out to a variable or attribute.

(defun iphoto-thumbnails (start end imgfile bracketp)
  (unless bracketp
    (when (and
           ;; it is an image
           (org-string-match-p (image-file-name-regexp) imgfile)
           ;; and it exists
           (f-exists? imgfile)
           ;; and there is no overlay here.
           (not (ov-at start)))
      (setq img (create-image (expand-file-name imgfile)
                              'imagemagick nil :width 300
                              :background "lightgray"))
      (setq ov (make-overlay start end))
      (overlay-put ov 'display img)
      (overlay-put ov 'face 'default)
      (overlay-put ov 'org-image-overlay t)
      (overlay-put ov 'modification-hooks
                   (list
                    `(lambda (&rest args)
                       (org-display-inline-remove-overlay ,ov t ,start ,end))))
      (push ov org-inline-image-overlays))))

(defun iphoto-redraw-thumbnails (&rest args)
  (org-restart-font-lock))

;; this redisplays these thumbnails on image toggling
(advice-add 'org-display-inline-images :after 'iphoto-redraw-thumbnails)

Next, we define the link with a follow, export, tooltip and activate-func (which puts the overlay on).

(org-link-set-parameters
 "iphoto"
 :follow (lambda (path) (browse-url (iphoto-map-url path)))
 :export 'iphoto-export
 :help-echo "Click me to see where this photo is on a map."
 :activate-func 'iphoto-thumbnails)

So finally, here is the mysterious image.

map

Now, in org-mode, I see the image in an overlay, and I can toggle it on and off. If I click on the image, it opens a browser to Google maps with a pin at the spot I took it. When I export it, it wraps the image in a <figure> tag, and puts a url in the caption to the map. If you click on it, and zoom out, you will see this is a picture of the Nāpali Coast on Kauai in Hawaii, and I was in fact out at sea when I took the picture. It was spectacular. Here is another one. This one is a little more obvious with the zoom. Here, I was on land. Since this link is bracketed, it does not show the overlay however in the org-file.

Another vacation picture, this time with a caption. map

Overall, this was easier than I expected. It might be faster to outsource reading the exif data to some dedicate library, perhaps in python that would return everything you want in an easy to parse json data structure. The speed of computing the url is only annoying when you export or click on the links though.

I didn't build in any error handling, e.g. if you do this on a photo with no GPS data it will probably not handle it gracefully. I also haven't tested this on any other images, e.g. south of the equator, from other cameras, etc. I assume this exif data is pretty standard, but it is a wild world out there… It would still be nice to find a way to get a string representing the nearest known location somehow, that would help the caption be more useful.

There is one little footnote to speak of, and that is I had to do a little hackery to get this to work with my blog machinery. You can see what it is in the org-source, I buried it in a noexport subheading, because it isn't that interesting in the grand scheme of things. It was just necessary because I export these org-files to blogofile, which then builds the html pages, instead of just exporting them. The images have to be copied to a source directory, and paths changed in the html to point to them. See, boring stuff. Otherwise, the code above should be fine for regular org and html files! Now, my vacation is over so it is time to get back to work.

Copyright (C) 2018 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 9.1.13

Read and Post Comments

Literate programming with python doctests

| categories: python, noweb, orgmode | tags: | View Comments

On the org-mode mailing list we had a nice discussion about using noweb and org-mode in literate programming. The results of that discussion were blogged about here. I thought of a different application of this for making doctests in Python functions. I have to confess I have never liked these because I have always thought they were a pain to write since you basically have to put code and results into a docstring. The ideas developed in the discussion above led me to think of a new way to write these that seems totally reasonable.

The idea is just to put noweb placeholders in the function docstring for the doctests. The placeholders will be expanded when you tangle the file, and they will get their contents from other src-blocks where you have written and run examples to test them.

This video might make the rest of this post easier to follow:

I will illustrate the idea using org-mode and the ob-ipython I have in scimax. The defaults of my ob-ipython setup are not useful for this example because it puts the execution count and mime types of output in the output. These are not observed in a REPL, and so we turn this off by setting these variables.

(setq ob-ipython-suppress-execution-count t
      ob-ipython-show-mime-types nil)

Now, we make an example function that takes a single argument and returns one divided by that argument. This block is runnable, and the function is then defined in the jupyter kernel. The docstring contains several noweb references to doctest blocks we define later. For now, they don't do anything. See The noweb doctest block section for the block that is used to expand these. This block also has a tangle header which indicates the file to tangle the results to. When I run this block, it is sent to a Jupyter kernel and saved in memory for use in subsequent blocks.

Here is the block with no noweb expansion. Note that this is easier to read in the original org source than it is to read in the published blog format.

def func(a):
    """A function to divide one by a.

    <<doctest("doctest-1")>>

    <<doctest("doctest-2")>>

    <<doctest("doctest-3")>>

    Returns: 1 / a.
    """
    return 1 / a

Now, we can write a series of named blocks that define various tests we might want to use as doctests. You can run these blocks here, and verify they are correct. Later, when we tangle the document, these will be incorporated into the tangled file in the docstring we defined above.

func(5) == 0.2
True

This next test will raise an Exception, and we just run it to make sure it does.

func(0)

ZeroDivisionErrorTraceback (most recent call last)
<ipython-input-6-ba0cd5a88f0a> in <module>()
----> 1 func(0)

<ipython-input-1-eafd354a3163> in func(a)
     18     Returns: 1 / a.
     19     """
---> 20     return 1 / a

ZeroDivisionError: division by zero

This is just a doctest with indentation to show how it is used.

for i in range(1, 4):
    print(func(i))
1.0
0.5
0.3333333333333333


That concludes the examples I want incorporated into the doctests. Each one of these blocks has a name, which is used as an argument to the noweb references in the function docstring.

1 Add a way to run the tests

This is a common idiom to enable easy running of the doctests. This will get tangled out to the file.

if __name__ == "__main__":
    import doctest
    doctest.testmod()

2 Tangle the file

So far, the Python code we have written only exists in the org-file, and in memory. Tangling is the extraction of the code into a code file.

We run this command, which extracts the code blocks marked for tangling, and expands the noweb references in them.

(org-babel-tangle)
test.py

Here is what we get:

def func(a):
    """A function to divide one by a.

    >>> func(5) == 0.2
    True

    >>> func(0)
    Traceback (most recent call last):
    ZeroDivisionError: division by zero

    >>> for i in range(1, 4):
    ...     print(func(i))
    1.0
    0.5
    0.3333333333333333


    Returns: 1 / a.
    """
    return 1 / a

if __name__ == "__main__":
    import doctest
    doctest.testmod()

That looks like a reasonable python file. You can see the doctest blocks have been inserted into the docstring, as desired. The proof of course is that we can run these doctests, and use the python module. We show that next.

3 Run the tests

Now, we can check if the tests pass in a fresh run (i.e. not using the version stored in the jupyter kernel.) The standard way to run the doctests is like this:

python test.py -v

Well, that's it! It worked fine. Now we have a python file we can import and reuse, with some doctests that show how it works. For example, here it is in a small Python script.

from test import func
print(func(3))
0.3333333333333333

There are surely some caveats to keep in mind here. This was just a simple proof of concept idea that isn't tested beyond this example. I don't know how many complexities would arise from more complex doctests. But, it seems like a good idea to continue pursuing if you like using doctests, and like using org-mode and interactive/literate programming techniques.

It is definitely an interesting way to use noweb to build up better code files in my opinion.

4 The noweb doctest block

These blocks are used in the noweb expansions. Each block takes a variable which is the name of a block. This block grabs the body of the named src block and formats it as if it was in a REPL.

We also grab the results of the named block and format it for the doctest. We use a heuristic to detect Tracebacks and modify the output to be consistent with it. In that case we assume the relevant Traceback is on the last line.

Admittedly, this does some fragile feeling things, like trimming whitespace here and there to remove blank lines, and quoting quotes (which was not actually used in this example), and removing the ": " pieces of ob-ipython results. Probably other ways of running the src-blocks would not be that suitable for this.

(org-babel-goto-named-src-block name)
(let* ((src (s-trim-right (org-element-property :value (org-element-context))))
       (src-lines (split-string src "\n"))
       body result)
  (setq body
        (s-trim-right
         (s-concat ">>> " (car src-lines) "\n"
                   (s-join "\n" (mapcar (lambda (s)
                                          (concat "... " s))
                                        (cdr src-lines))))))
  ;; now the results
  (org-babel-goto-named-result name)
  (let ((result (org-element-context)))
    (setq result
          (thread-last
              (buffer-substring (org-element-property :contents-begin result)
                                (org-element-property :contents-end result))
            (s-trim)
            ;; remove ": " from beginning of lines
            (replace-regexp-in-string "^: *" "")
            ;; quote quotes
            (replace-regexp-in-string "\\\"" "\\\\\"")))
    (when (string-match "Traceback" result)
      (setq result (format
                    "Traceback (most recent call last):\n%s"
                    (car (last (split-string result "\n"))))))
    (concat body "\n" result)))

Copyright (C) 2018 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 9.1.13

Read and Post Comments

Making it easier to extend the export of org-mode links with generic functions

| categories: emacs, orgmode | tags: | View Comments

I am a big fan of org-mode links. Lately, I have had a need to modify how some links are exported, e.g. defining new exports for different backends, or fine-tuning a particular backend. This can be difficult, depending on how the link was set up. Here is a typical setup I am used to using, where the different options for the backends are handled in a conditional statement in a single function. I will just use a link that just serves to illustrate the issues here. These links are just sytactic sugar for markup, they don't do anything else. We start with an example that just converts text to italic text for different backends like html or latex.

(defun italic-link-export (path desc backend)
  (cond
   ((eq 'html backend)
    (format "<em>%s</em>" path))
   ((eq 'latex backend)
    (format "\\textit{%s}" path))
   ;; fall-through case for everything else
   (t
    path)))

(org-link-set-parameters "italic" :export 'italic-link-export)
:export italic-link-export
(org-export-string-as "italic:text" 'html t)
<p>
<em>text</em></p>

(org-export-string-as "italic:text" 'latex t)
\textit{text}

This falls through though to the default case.

(require 'ox-md)
(org-export-string-as "italic:text" 'md t)

# Table of Contents



text


The point I want to make here is that this is not easy to extend as a user. You have to either modify the italic-link-export function, advise it, or monkey-patch it. None of these are especially nice.

I could define italic-link-export in a way that it retrieves the function to use from an alist or hash-table using the backend, but then you have to do two things to modify the behavior: define a backend specific function and register it in the lookup variable. It is also possible to just look up a function by a derived symbol, e.g. using fboundp, and then using funcall to execute it. This looks something like this:

;; a user definable function for exporting to latex
(defun italic-link-export-latex (path desc backend)
  (format "\\textit{%s}" path))

;; generic export function that looks up functions or defaults to
(defun italic-link-exporter (path desc backend)
  "Run `italic-link-export-BACKEND' if it exists, or return path."
  (let ((func (intern-soft (format "italic-link-export-%s" backend))))
    (if (fboundp func)
        (funcall func path desc backend)
      path)))

This has some indirection, but allows you to just define new functions to add new export backends, or replace single backend exports. It isn't bad, but there is room for improvement.

In this comment in org-ref, I saw a new opportunity to address this issue using generic functions in elisp! The idea is to define a generic function that handles the general export case, and then define additional functions for each specific backend based on the signature of the export function. I will switch to bold markup for this.

(cl-defgeneric bold-link-export (path desc backend)
 "Generic function to export a bold link."
 path)

;; this one runs when the backend is equal to html
(cl-defmethod bold-link-export ((path t) (desc t) (backend (eql html)))
 (format "<b>%s</b>" path))

;; this one runs when the backend is equal to latex
(cl-defmethod bold-link-export ((path t) (desc t) (backend (eql latex)))
 (format "\\textit{%s}" path))

(org-link-set-parameters "bold" :export 'bold-link-export)
:export bold-link-export

Here it is in action:

(org-export-string-as "some bold:text" 'html t)
<p>
some <b>text</b></p>

(org-export-string-as "some bold:text" 'latex t)

This uses the generic function.

(require 'ox-md)
(org-export-string-as "some bold:text" 'md t)

# Table of Contents



some text


The syntax for defining the generic function is pretty similar to a regular function. The specific methods are a little different since they have to provide the specific "signature" that triggers each method. Here we only differentiate on the type of the backend. It is nice these are all separate functions though. It makes it trivial to add new ones, and less intrusive to replace in my opinion.

Generic functions have many other potential applications to replace functions that use lots of conditions to control flow, with a fall-through option at the end. You can learn more about them here: https://www.gnu.org/software/emacs/manual/html_node/elisp/Generic-Functions.html. There is a lot more to them than I have illustrated here.

Copyright (C) 2018 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 9.1.13

Read and Post Comments

Adding keymaps to src blocks via org-font-lock-hook

| categories: emacs, orgmode | tags: | View Comments

Table of Contents

I had an idea to use custom keymaps in src-blocks. For example, you could then use lispy directly in your org-files without entering org-special-edit, or the elpy key-bindings in python blocks. There are other solutions I have seen, e.g. polymode, that claim to do this. You might guess that if they worked, I would not be writing this! There was some nice discussion about this idea on the org-mode mailing list, and Nicolas Goaziou pointed out this might be accomplished with the org-font-lock-hook.

You can check out the video here:

It was relatively easy to figure out how to do this. Keymaps can be added to regions during font-lock, so I just had to hook into the org-mode font lock system with a function to find the src blocks and add the keymap as a text-property. That took three steps:

  1. Define the keymaps to use. I use an a-list of (language . map) for this.
  2. Define the font-lock function. This will add the keymap properties to src-blocks.
  3. Define a minor mode to toggle this feature on and off.

Here is the definition of the keymaps. Generally I just copy the mode-map I want and then add some things to them. For example sometimes it is still a good idea to jump into the org-special-edit mode. For example, if you try to use a command in a Python block to send the buffer to the repl while in org-mode you are sure to get an error! You might also want to add the C-c C-e export command if you use that a lot. An alternative approach, of course, is to copy the org-map and add additional bindings to it. The choice is up to you.

(require 'lispy)
(require 'elpy)

(setq scimax-src-block-keymaps
      `(("ipython" . ,(let ((map (make-composed-keymap
                                  `(,elpy-mode-map ,python-mode-map ,pyvenv-mode-map)
                                  org-mode-map)))
                        ;; In org-mode I define RET so we f
                        (define-key map (kbd "<return>") 'newline)
                        (define-key map (kbd "C-c C-c") 'org-ctrl-c-ctrl-c)
                        map))
        ("python" . ,(let ((map (make-composed-keymap
                                 `(,elpy-mode-map ,python-mode-map ,pyvenv-mode-map)
                                 org-mode-map)))
                       ;; In org-mode I define RET so we f
                       (define-key map (kbd "<return>") 'newline)
                       (define-key map (kbd "C-c C-c") 'org-ctrl-c-ctrl-c)
                       map))
        ("emacs-lisp" . ,(let ((map (make-composed-keymap `(,lispy-mode-map
                                                            ,emacs-lisp-mode-map
                                                            ,outline-minor-mode-map)
                                                          org-mode-map)))
                           (define-key map (kbd "C-c C-c") 'org-ctrl-c-ctrl-c)
                           map))))

Next we define the function that will apply the keymap to each src block. The keymaps are only applied when they are defined in the variable above. This function is derived from org-fontify-meta-lines-and-blocks-1.

(defun scimax-add-keymap-to-src-blocks (limit)
  "Add keymaps to src-blocks defined in `scimax-src-block-keymaps'."
  (let ((case-fold-search t)
        lang)
    (while (re-search-forward org-babel-src-block-regexp limit t)
      (let ((lang (match-string 2))
            (beg (match-beginning 0))
            (end (match-end 0)))
        (if (assoc (org-no-properties lang) scimax-src-block-keymaps)
            (progn
              (add-text-properties
               beg end `(local-map ,(cdr (assoc
                                          (org-no-properties lang)
                                          scimax-src-block-keymaps))))
              (add-text-properties
               beg end `(cursor-sensor-functions
                         ((lambda (win prev-pos sym)
                            ;; This simulates a mouse click and makes a menu change
                            (org-mouse-down-mouse nil)))))))))))

Here we create an advice to trick any functions that need to know the major mode. We only apply the spoof if we are in org-mode and in a src block though. Otherwise we call the original function. So far lispy–eval is the only function I have needed it for. This might be a general strategy though to do other things like narrow to the src-block, or even go into special edit mode temporarily if there are commands that require it.

(defun scimax-spoof-mode (orig-func &rest args)
  "Advice function to spoof commands in org-mode src blocks.
It is for commands that depend on the major mode. One example is
`lispy--eval'."
  (if (org-in-src-block-p)
      (let ((major-mode (intern (format "%s-mode" (first (org-babel-get-src-block-info))))))
        (apply orig-func args))
    (apply orig-func args)))

We define a minor mode so we can toggle this on and off. Here we add the function to the org-font-lock-hook and advise the lispy–eval function. I had to add the font-lock-function to the end of the org-font-lock hook for some reason, and also add local-map as an extra-managed property so it would be removed when we toggle it off.

(define-minor-mode scimax-src-keymap-mode
  "Minor mode to add mode keymaps to src-blocks."
  :init-value nil
  (if scimax-src-keymap-mode
      (progn
        (add-hook 'org-font-lock-hook #'scimax-add-keymap-to-src-blocks t)
        (add-to-list 'font-lock-extra-managed-props 'local-map)
        (add-to-list 'font-lock-extra-managed-props 'cursor-sensor-functions)
        (advice-add 'lispy--eval :around 'scimax-spoof-mode)
        (cursor-sensor-mode +1))
    (remove-hook 'org-font-lock-hook #'scimax-add-keymap-to-src-blocks)
    (advice-remove 'lispy--eval 'scimax-spoof-mode)
    (cursor-sensor-mode -1))
  (font-lock-fontify-buffer))

(add-hook 'org-mode-hook (lambda ()
                           (scimax-src-keymap-mode +1)))

That is it! I am pretty sure this is a good idea. It helps a lot when you are writing a lot of short code blocks and near equal amounts of text (like in this blog post). It also helps write the code since many things like indentation, parentheses, etc. are automatically handled. That is what I used to go into special-edit mode all the time for!

I have not used this long enough to know if it causes any other surprises. If you try it and find any, leave a comment!

1 Update

It turns out you can have the best of all the worlds by combining keymaps. The make-composed-keymap creates a new keymap that combines a keymaps and falls through to a parent keymap. So here we use that to combine several keymaps, falling through to org-mode. The only subtlety I have come across is that I remapped <return> in orgmode to scimax/org-return, and not all modes define it, so I redefine it in some places to just be newline. Also to keep C-c C-c for executing the block, I add that back too.

I use a few maps here, and some of them seem to just add menus that are only active when your cursor is in the block. Pretty handy!

(setq scimax-src-block-keymaps
      `(("ipython" . ,(let ((map (make-composed-keymap
                                  `(,elpy-mode-map ,python-mode-map ,pyvenv-mode-map)
                                  org-mode-map)))
                        ;; In org-mode I define RET so we f
                        (define-key map (kbd "<return>") 'newline)
                        (define-key map (kbd "C-c C-c") 'org-ctrl-c-ctrl-c)
                        map))
        ("python" . ,(let ((map (make-composed-keymap
                                 `(,elpy-mode-map ,python-mode-map ,pyvenv-mode-map)
                                 org-mode-map)))
                       ;; In org-mode I define RET so we f
                       (define-key map (kbd "<return>") 'newline)
                       (define-key map (kbd "C-c C-c") 'org-ctrl-c-ctrl-c)
                       map))
        ("emacs-lisp" . ,(let ((map (make-composed-keymap `(,lispy-mode-map
                                                            ,emacs-lisp-mode-map
                                                            ,outline-minor-mode-map)
                                                          org-mode-map)))
                           (define-key map (kbd "C-c C-c") 'org-ctrl-c-ctrl-c)
                           map))))

2 Update #2

The previous version had some issues where it would only add a keymap to the first block. The code in this post now addresses that and uses cursor-sensor-functions to make sure we change key map on entering and leaving blocks. That might mean you need an emacs of at least version 25 to use this. I guess it will work with an earlier version, but the cursor-sensor-functions might get ignored. You might have to comment out the cursor-sensor-mode line

Thanks to those brave people alpha-testing this and helping refine the idea!

Copyright (C) 2017 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 9.0.7

Read and Post Comments

Org-mode and ipython enhancements in scimax

| categories: emacs, orgmode, ipython | tags: | View Comments

We have made some improvements to using Ipython in org-mode in the past including:

  1. Inline figures
  2. Export to Jupyter notebooks

Today I will talk about a few new features and improvements I have introduced to scimax for using org-mode and Ipython together.

The video for this post might be more obvious than the post:

1 Some convenience functions

There are a few nice shortcuts in the Jupyter notebook. Now we have some convenient commands in scimax to mimic those. My favorites are adding cells above or below the current cell. You can insert a new src block above the current one with (M-x org-babel-insert-block). You can use a prefix arg to insert it below the current block.

# code
# below
# some code

I am particularly fond of splitting a large block into two smaller blocks. Use (M-x org-babel-split-src-block) to do that and leave the point in the upper block. Use a prefix arg to leave the point in the lower block.

# lots of code in large block
# Even more code
# The end of the long block

You can execute all the blocks up to the current point with (M-x org-babel-execute-to-point).

2 ob-ipython-inspect works

In the original ob-ipython I found that ob-ipython-inspect did not work unless you were in special edit mode. That is too inconvenient. I modified a few functions to work directly from the org-buffer. I bind this to M-. in org-mode.

%matplotlib inline
import numpy as np

import matplotlib.pyplot as plt

# Compute areas and colors
N = 150
r = 2 * np.random.rand(N)
theta = 2 * np.pi * np.random.rand(N)
area = 200 * r**2
colors = theta

ax = plt.subplot(111, projection='polar')
c = ax.scatter(theta, r, c=colors, s=area, cmap='hsv', alpha=0.75)

<matplotlib.figure.Figure at 0x114ded710>

3 Getting selective output from Ipython

Out of the box Ipython returns a lot of results. This block, for example returns a plain text, image and latex result as output.

from sympy import *
# commenting out init_printing() results in no output
init_printing()

var('x y')
x**2 + y

2 x + y

We can select which one we want with a new header argument :ob-ipython-results. For this block you can give it the value of text/plain, text/latex or image/png.

var('x y')
x**2 + y

2 x + y

Or to get the image:

var('x y')
x**2 + y

This shows up with pandas too. This block creates a table of data and then shows the first 5 rows. Ipython returns both plain text and html here.

import pandas as pd
import numpy as np
import datetime as dt

def makeSim(nHosps, nPatients):
    df = pd.DataFrame()
    df['patientid'] = range(nPatients)
    df['hospid'] = np.random.randint(0, nHosps, nPatients)
    df['sex'] = np.random.randint(0, 2, nPatients)
    df['age'] = np.random.normal(65,18, nPatients)
    df['race'] = np.random.randint(0, 4, nPatients)
    df['cptCode'] = np.random.randint(1, 100, nPatients)
    df['rdm30d'] = np.random.uniform(0, 1, nPatients) < 0.1
    df['mort30d'] = np.random.uniform(0, 1, nPatients) < 0.2
    df['los'] = np.random.normal(8, 2, nPatients)
    return df

discharges = makeSim(50, 10000)
discharges.head()

patientid hospid sex age race cptCode rdm30d mort30d los 0 0 10 1 64.311947 0 8 False False 8.036793 1 1 6 0 82.951484 1 73 True False 7.996024 2 2 27 1 53.064501 3 95 False False 9.015144 3 3 37 0 64.799128 0 93 False False 10.099032 4 4 46 0 99.111394 2 25 False False 11.711427

patientid hospid sex age race cptCode rdm30d mort30d los
0 0 10 1 64.311947 0 8 False False 8.036793
1 1 6 0 82.951484 1 73 True False 7.996024
2 2 27 1 53.064501 3 95 False False 9.015144
3 3 37 0 64.799128 0 93 False False 10.099032
4 4 46 0 99.111394 2 25 False False 11.711427

We can use the header to select only the plain text output!

import pandas as pd
import numpy as np
import datetime as dt

def makeSim(nHosps, nPatients):
    df = pd.DataFrame()
    df['patientid'] = range(nPatients)
    df['hospid'] = np.random.randint(0, nHosps, nPatients)
    df['sex'] = np.random.randint(0, 2, nPatients)
    df['age'] = np.random.normal(65,18, nPatients)
    df['race'] = np.random.randint(0, 4, nPatients)
    df['cptCode'] = np.random.randint(1, 100, nPatients)
    df['rdm30d'] = np.random.uniform(0, 1, nPatients) < 0.1
    df['mort30d'] = np.random.uniform(0, 1, nPatients) < 0.2
    df['los'] = np.random.normal(8, 2, nPatients)
    return df

discharges = makeSim(50, 10000)
discharges.head()

patientid hospid sex age race cptCode rdm30d mort30d los 0 0 21 0 73.633836 1 38 False False 7.144019 1 1 16 1 67.518804 3 23 False False 3.340534 2 2 15 0 44.139033 0 8 False False 9.258706 3 3 29 1 45.510276 2 5 False False 10.590245 4 4 7 0 52.974924 2 4 False True 5.811064

4 Where was that error?

A somewhat annoying feature of running cells in org-mode is when there is an exception there has not been a good way to jump to the line that caused the error to edit it. The lines in the src block are not numbered, so in a large block it can be tedious to find the line. In scimax, when you get an exception it will number the lines in the src block, and when you press q in the exception traceback buffer it will jump to the line in the block where the error occurred.

print(1)
#raise Exception('Here')
print(2)

1 2

If you don't like the numbers add this to your init file:

(setq ob-ipython-number-on-exception nil)

5 Asynchronous Ipython

I have made a few improvements to the asynchronous workflow in Ipython. We now have a calculation queue, so you can use C-c C-c to execute several blocks in a row, and they will run asynchronously in the order you ran them. While they are running you can continue using Emacs, e.g. writing that paper, reading email, checking RSS feeds, tetris, … This also lets you run all the blocks up to the current point (M-x org-babel-execute-ipython-buffer-to-point-async) or the whole buffer (of Ipython) blocks asynchronously (M-x org-babel-execute-ipython-buffer-async).

To turn this on by default put this in your init file:

(setq org-babel-async-ipython t)

This requires all src blocks to have a name, and running the block will give it a name if you have not named the block. By default we use human-readable names. While the block is running, there will be a link indicating it is running. You can click on the link to cancel it. Running subsequent blocks will queue them to be run when the first block is done.

Here is an example:

import time
time.sleep(5)
a = 5
print('done')
print(3 * a)

15

Occasionally you will run into an issue. You can clear the queue with org-babel-async-ipython-clear-queue.

Copyright (C) 2017 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 9.0.5

Read and Post Comments

Next Page »