Asynchronously running python blocks in org-mode
Posted November 20, 2015 at 11:46 AM | categories: python, orgmode, emacs | tags:
Updated November 20, 2015 at 07:30 PM
If you run long Python blocks from org-mode, you might want to keep working while it runs. Currently Emacs gets blocked and you have to wait patiently. In this post we consider some ways to avoid this that run our code asynchronously, but still put results where they belong in the org-buffer.
This is a long post. You may want to see the video: https://www.youtube.com/watch?v=VDyoN8yipSE , or skip to the end where the best and final version is shown.
1 The async module
Here we consider an approach that uses https://github.com/jwiegley/emacs-async module. The idea is to tangle the Python block at point to a temp file, then asynchronously run it. We capture the output and put it back in the buffer. We use a uuid to find the place to put the results in org-mode format. Here is the code that implements this idea.
(require 'async) (defun org-babel-async-execute () "Run a python block at point asynchrously." (interactive) (let ((current-file (buffer-file-name)) (uuid (org-id-uuid)) (temporary-file-directory "./") (tempfile (make-temp-file "py-"))) (org-babel-tangle '(4) tempfile) (org-babel-remove-result) (save-excursion (re-search-forward "#\\+END_SRC") (insert (format "\n\n#+RESULTS: %s\n: %s" (or (org-element-property :name (org-element-context)) "") uuid))) (async-start ;; what to start `(lambda () ;; now we run the command then cleanup (prog1 (shell-command-to-string (format "python %s" ,tempfile)) (delete-file ,tempfile))) `(lambda (result) "Code that runs when the async function finishes." (save-window-excursion (save-excursion (save-restriction (with-current-buffer (find-file-noselect ,current-file) (goto-char (point-min)) (re-search-forward ,uuid) (beginning-of-line) (kill-line) (insert (mapconcat (lambda (x) (format ": %s" x)) (butlast (s-split "\n" result)) "\n"))))))))))
org-babel-async-execute
Here is a block to test it on. We can run the block, and keep on working while the code runs. The results seem to get inserted correctly at the right point even if I am in another window or frame! We don't get easy access to continuous output of the command. This wouldn't work if we close Emacs, but who does that?
print 'hello world' import time time.sleep(5) import os print os.getcwd() print time.asctime()
hello world /Users/jkitchin/blogofile-jkitchin.github.com/_blog Fri Nov 20 10:17:53 2015
There are some limitations to this approach. One of them is it assumes the src block is a stand-alone block that will run on its own. That is usually how I run mine, but I could see having other modules that should be tangled out of a file too. I think the script is being run in the current working directory, so it probably will find any local imports it needs.
You don't get any intermediate feedback on this process. It seems to be possible to do that with a different approach that puts some output in a new buffer, e.g. with start-process. But, you still need some clever code like the async model to know when to insert the results back into this buffer. We consider Emacs processes and sentinels next.
2 Emacs process approach with tangling
We can start a process in Emacs, and attach a sentinel function to it that runs after the process completes. Here is an example of that. We still tangle the src-block here.
(defun org-babel-async-execute () (interactive) (let* ((current-file (buffer-file-name)) (uuid (org-id-uuid)) (temporary-file-directory "./") (tempfile (make-temp-file "py-")) (pbuffer (format "*%s*" uuid)) process) (org-babel-tangle '(4) tempfile) (org-babel-remove-result) (save-excursion (re-search-forward "#\\+END_SRC") (insert (format "\n\n#+RESULTS: %s\n: %s" (or (org-element-property :name (org-element-context)) "") uuid))) (setq process (start-process uuid pbuffer "python" tempfile)) (set-process-sentinel process `(lambda (process event) (when (string= "finished\n" event) (delete-file ,tempfile) (save-window-excursion (save-excursion (save-restriction (with-current-buffer (find-file-noselect ,current-file) (goto-char (point-min)) (re-search-forward ,uuid) (beginning-of-line) (kill-line) (insert (mapconcat (lambda (x) (format ": %s" x)) (split-string (with-current-buffer ,pbuffer (buffer-string)) "\n") "\n"))))))) (kill-buffer ,pbuffer)))))
org-babel-async-execute
print 'hello world' import time time.sleep(10) import os print os.getcwd() print time.asctime()
hello world /Users/jkitchin/blogofile-jkitchin.github.com/_blog Fri Nov 20 10:20:01 2015
That works well from what I can see. There are some limitations. I doubt this will work if you use variables in the src block header. Next we consider an approach that does not do the tangling, and that will show us code output as it goes.
3 Emacs process approach with no tangling
As an alternative to tangling to a file, here we just copy the code to a file and then run it. This allows us to use :var in the header to pass data in at run time. At the moment, this code only supports printed output from code blocks, not the value for :results.
(defun org-babel-async-execute:python () "Execute the python src-block at point asynchronously. :var headers are supported. :results output is all that is supported for output. A new window will pop up showing you the output as it appears, and the output in that window will be put in the RESULTS section of the code block." (interactive) (let* ((current-file (buffer-file-name)) (uuid (org-id-uuid)) (code (org-element-property :value (org-element-context))) (temporary-file-directory ".") (tempfile (make-temp-file "py-")) (pbuffer (format "*%s*" uuid)) (varcmds (org-babel-variable-assignments:python (nth 2 (org-babel-get-src-block-info)))) process) ;; get rid of old results, and put a place-holder for the new results to ;; come. (org-babel-remove-result) (save-excursion (re-search-forward "#\\+END_SRC") (insert (format "\n\n#+RESULTS: %s\n: %s" (or (org-element-property :name (org-element-context)) "") uuid))) ;; open the results buffer to see the results in. (switch-to-buffer-other-window pbuffer) ;; Create temp file containing the code. (with-temp-file tempfile ;; if there are :var headers insert them. (dolist (cmd varcmds) (insert cmd) (insert "\n")) (insert code)) ;; run the code (setq process (start-process uuid pbuffer "python" tempfile)) ;; when the process is done, run this code to put the results in the ;; org-mode buffer. (set-process-sentinel process `(lambda (process event) (save-window-excursion (save-excursion (save-restriction (with-current-buffer (find-file-noselect ,current-file) (goto-char (point-min)) (re-search-forward ,uuid) (beginning-of-line) (kill-line) (insert (mapconcat (lambda (x) (format ": %s" x)) (butlast (split-string (with-current-buffer ,pbuffer (buffer-string)) "\n")) "\n")))))) ;; delete the results buffer then delete the tempfile. ;; finally, delete the process. (when (get-buffer ,pbuffer) (kill-buffer ,pbuffer) (delete-window)) (delete-file ,tempfile) (delete-process process)))))
org-babel-async-execute:python
Let us try it out again.
print 'hello world' import time time.sleep(1) for i in range(5): print i time.sleep(0.5) import os print os.getcwd() print time.asctime() print data raise IOError('No file!')
hello world
0
1
2
3
4
/Users/jkitchin/blogofile-jkitchin.github.com/_blog
Fri Nov 20 19:30:16 2015
[1, 3]
Traceback (most recent call last):
File "/Users/jkitchin/blogofile-jkitchin.github.com/_blog/py-84344aa1", line 18, in <module>
raise IOError('No file!')
IOError: No file!
It works fine for this simple example. We get to see the output as the code executes, which is a pleasant change from the usual way of running python blocks. There is some support for some header arguments, notably the :var header. I don't use :results value in Python, so for now only output is supported. We even support Exceptions in the output finally!
Maybe some org-moder's out there can try this and run it through some more rigorous paces?
Copyright (C) 2015 by John Kitchin. See the License for information about copying.
Org-mode version = 8.2.10