Adding GSL constants to Emacs in a dynamic module

| categories: emacs, dynamic-module | tags: | View Comments

The GNU Scientific Library defines a lot of physical constants. Since we are exploring how to make Emacs a more scientific environment to work in, it would be nice to import these constants to elisp. We do that through a dynamic module. This turned out to be tricky. I thought we could just use a funcall to defconst or defvar, but these are special forms and you cannot funcall them. @polizta on Stackoverflow pointed me to the path that led to success: You make a list like '(defconst sym val doc) and then eval it. That can be funcall'd, and it works nicely in the module below. It is a growing theme that it takes much hacking around to figure out how to do things like this.

The only other notable feature of this module is that I created a defconst function to make adding multiple constants less verbose. Here I only add two constants. There are about 408 constants defined in gsl_const_*.h, so brevity might be a good idea! Here is the module.

#include <gsl/gsl_const_mksa.h>
#include <string.h>
#include "emacs-module.h"

int plugin_is_GPL_compatible;

// I assume here that all values will be double. I can't think of any that would
// be ints
static void defconst (emacs_env *env, const char *name, double value, const char *doc)
{
  // These are functions we will call
  emacs_value eval = env->intern(env, "eval");  
  emacs_value list = env->intern(env, "list");

  // These will make up the list we will eventally eval
  emacs_value fdefconst = env->intern(env, "defconst");
  emacs_value sym = env->intern(env, name);
  emacs_value val = env->make_float(env, value);
  emacs_value sdoc = env->make_string(env, doc, strlen(doc));

  // make a list of (defconst sym val doc)
  emacs_value largs[] = {fdefconst, sym, val, sdoc};
  emacs_value qlist = env->funcall(env, list, 4, &largs);   

  // now eval the list of symbols
  emacs_value args[] = { qlist };  
  env->funcall(env, eval, 1, &args);
}

int emacs_module_init(struct emacs_runtime *ert)
{
  emacs_env *env = ert->get_environment(ert);

  defconst(env, "GSL-CONST-MKSA-SPEED-OF-LIGHT",
           GSL_CONST_MKSA_SPEED_OF_LIGHT,
           "Speed of light in vacuum (m/s).");
  
  defconst(env, "GSL-CONST-MKSA-PLANCKS-CONSTANT-H",
           GSL_CONST_MKSA_PLANCKS_CONSTANT_H,
           "Plank's constant, h");

  // This is what allows the shared library to provide a feature 
  emacs_value provide = env->intern(env, "provide");
  emacs_value provide_args[] = { env->intern(env, "gsl-constants") };
  env->funcall(env, provide, 1, provide_args);
  
  return 0;
}

Regular gcc will work to compile this module.

rm -f gsl-constants.so gsl-constants.o
gcc -Wall -I/usr/local/include -fPIC -c gsl-constants.c
gcc -shared -L/usr/local/include -lgsl -o gsl-constants.so gsl-constants.o

Here is in action.

(add-to-list 'load-path "/Users/jkitchin/vc/blogofile-jkitchin.github.com/_blog/dynamic-module/")
(require 'gsl-constants)
GSL-CONST-MKSA-SPEED-OF-LIGHT
299792458.0

We can see there is a docstring on that constant:

(describe-variable 'GSL-CONST-MKSA-SPEED-OF-LIGHT)
GSL-CONST-MKSA-SPEED-OF-LIGHT's value is 299792458.0


  This variable can be risky when used as a file-local variable.

Documentation:
Speed of light in vacuum (m/s).

For more information check the manuals.

It is worth thinking about what we accomplished here. The value of each constant in GSL is stored in a header file. The units are stored in a comment next to the value, and the documentation is in an html page somewhere. It is not easy to introspect that! Getting it all into an Emacs variable makes that more introspectable, and findable. That means while typing elisp code you will get completion on these variables. Check this out:

(apropos-variable "GSL-*")
(with-current-buffer "*Apropos*" (buffer-string))
Type RET on a type label to view its full documentation.

GSL-CONST-MKSA-PLANCKS-CONSTANT-H
  Variable: Plank's constant, h
GSL-CONST-MKSA-SPEED-OF-LIGHT
  Variable: Speed of light in vacuum (m/s).

It seems like it might be possible to partially automate creation of this module by parsing the gsl_const*.h files. There is no automating adding the doc strings though, I am pretty sure that will have to be done by hand ;(

Copyright (C) 2017 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 9.0.7

Read and Post Comments

Adding a GSL integration function to Emacs with a dynamic module

| categories: emacs, dynamic-module, integration | tags: | View Comments

Here we work out how to run this program: https://www.gnu.org/software/gsl/doc/html/integration.html#adaptive-integration-example in a dynamic module in emacs. The goal is to be able to evaluate \(\int_0^1 x^{-1/2} \log(x) dx\). According to the example page the answer is -4. We will define an integration function that takes at least a function and integration bounds as arguments, and several optional arguments to specify tolerances and limits. In other words we want to evaluate integrals of the form:

\(\int_a^b f(x; params) dx\)

I want that to happen in an elisp function with a signature like:

(gsl-integration-qags (lambda (x params) body) a b &optional params epsabs epsrel limit)

And that function will return a list containing (result error-estimate). Here is the C-code that makes this happen. It is more complex that the last example, and only compiles with gcc that allows nested functions. I don't know how to write this without that feature. This is more complex also because you have to create a workspace to do the integration inside the function that does the integration. The C-module also has extra code in it to allow for optional arguments.

#include <gsl/gsl_integration.h>
#include "emacs-module.h"

int plugin_is_GPL_compatible;

static emacs_value F_gsl_integrate (emacs_env *env, ptrdiff_t nargs, emacs_value args[], void *data)
{
  // nested function - only supported as an extension in gcc
  double f (double x, void *params) 
  {
    emacs_value fn = args[0];  // function we will integrate
    emacs_value x2[] = { env->make_float(env, x), params };
    emacs_value y = env->funcall(env, fn, 2, &x2);   
    
    return env->extract_float (env, y);
  }

  double a = env->extract_float (env, args[1]);
  double b = env->extract_float (env, args[2]);

  // default values for optional arguments
  double epsabs = 0.0;
  double epsrel = 1e-7;
  size_t limit = 1000;
  double result, error; 

  // Here is how I handle the optional arguments
  // (gsl-integrate func a b params epsabs epsrel limit)
  gsl_function F;
  F.function = &f;
  if (nargs >= 4) {F.params = args[3];}
  if (nargs >= 5 && env->is_not_nil(env, args[4])) {epsabs = env->extract_float(env, args[4]);}
  if (nargs >= 6 && env->is_not_nil(env, args[5])) {epsrel = env->extract_float(env, args[5]);}
  if (nargs >= 7 && env->is_not_nil(env, args[6])) {limit = env->extract_integer(env, args[6]);}

  gsl_integration_workspace * w = gsl_integration_workspace_alloc (limit);

  gsl_integration_qags (&F, // gsl_function pointer
                        a, // lower integration bound
                        b, // upper integration bound
                        epsabs, // absolute error tolerance
                        epsrel, // relative error tolerance
                        limit, // max number of subintervals for integration
                        w, // the workspace
                        // pointers to put results and error in
                        &result, &error);

  gsl_integration_workspace_free (w);
    
  // make a list of (result error) to return
  emacs_value Qlist = env->intern(env, "list");
  emacs_value Qresult = env->make_float (env, result);
  emacs_value Qerror = env->make_float (env, error);
  emacs_value list_args[] = { Qresult, Qerror };
  return env->funcall(env, Qlist, 2, list_args);
}

int emacs_module_init(struct emacs_runtime *ert)
{
  emacs_env *env = ert->get_environment(ert);
  
  // Here we create the function.
  emacs_value fset = env->intern(env, "fset");
  emacs_value args[2];
  args[0] = env->intern(env, "gsl-integration-qags"); // symbol to create for function
  // The function we set that symbol to.
  args[1] = env->make_function(env,
                               3, // min nargs
                               7, // max nargs
                               F_gsl_integrate,
                               "(gsl-integration-qags F A B &optional PARAMS EPSABS EPSREL LIMIT)\n" \
                               "Integrate F(x; params) from A to B.\n" \
                               "F is a function of a single variable and parameters.\n" \
                               "A is the lower bound of integration\n"  \
                               "B is the upper bound of integration.\n" \
                               "Optional parameters:\n"\
                               "PARAMS is a list of params to pass to F.\n" \
                               "EPSABS is a float (default 0.0) and is the absolute error tolerance.\n" \
                               "EPSREL is a float (default 1e-7) and is the relative error tolerance.\n" \
                               "LIMIT is the maximum number of subintervals for the integration (default 1000).\n" \
                               "Returns (list result error-estimate).\n" \
                               "See https://www.gnu.org/software/gsl/manual/html_node/QAGS-adaptive-integration-with-singularities.html.",
                               0);
  // This is basically (fset 'gsl-integration-qags (lambda func))
  env->funcall(env, fset, 2, args);
  
  // This is what allows the shared library to provide a feature 
  emacs_value provide = env->intern(env, "provide");
  emacs_value provide_args[] = { env->intern(env, "gsl-integration") };
  env->funcall(env, provide, 1, provide_args);
  
  return 0;
}

Building this was moderately tricky. It appears the first gcc on my path uses clang which does not support nested functions in C. I don't know enough C to figure out how to do this without a nested function though, since the function has to be defined at run-time based on the emacs env and args. gcc does support inline functions, so the code below uses a gcc that does compile it.

rm -f gsl-integration.so gsl-integration.o
/usr/local/Cellar/gcc/6.1.0/bin/gcc-6 -Wall -I/usr/local/include -fPIC -c gsl-integration.c
/usr/local/Cellar/gcc/6.1.0/bin/gcc-6  -shared -L/usr/local/include -lgsl -o gsl-integration.so gsl-integration.o

Now we add this directory to our path since it is not on it and require our new module.

(add-to-list 'load-path "/Users/jkitchin/vc/blogofile-jkitchin.github.com/_blog/dynamic-module/")
(require 'gsl-integration)
gsl-integration

Let us see our new function in action. We evaluate \(\int_0^1 x^{-1/2} \log(x) dx\). According to the example page the answer is -4. Here is an example where we ignore the parameters. You have to be careful; Emacs sometimes segfaults and crashes if you use an integer or float argument when it expects the other type.

(gsl-integration-qags (lambda (x params) (/ (log x) (sqrt x))) 0.0 1.0)
-4.000000000000085 1.354472090042691e-13

Here are some optional arguments.

(gsl-integration-qags (lambda (x params) (/ (log x) (sqrt x))) 0.0 1.0 nil nil 0.01)
-4.000000000000075 0.019526557540360034

Nice, with a larger epsrel argument we get a larger error. Note the arguments are positional, so we have to include them all just to set the epsrel argument. How about an easier example with parameters that we actually use. Here we integrate a constant, and set the value of the constant from the params arg. The integral should be the area of a rectangle of length 1 and width of the param we use.

(list
 (gsl-integration-qags (lambda (x params) (first params)) 0.0 1.0 '(1.0))
 (gsl-integration-qags (lambda (x params) (first params)) 0.0 1.0 '(0.5)))
1.0 1.1102230246251565e-14
0.5 5.551115123125783e-15

Wow! It actually works!!! That was harder won success than usual for me. I am claiming victory for now and leaving the following notes to future me:

  1. It would be nice to have optional keyword arguments. This would take some handling of the arguments beyond what I know how to do for now, unless it is possible to pull in something like plist-get the way we pull in fset, provide and list in this example.
  2. Error checking on types would be helpful. It is not good for Emacs to crash because 0 is not 0.0!
  3. In numpy there is often a feature to get full_output. Here, the workspace created in the function has more information available in a struct that might be helpful to have access to at times. It seems like it might be possible to get that here too.

Copyright (C) 2017 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 9.0.7

Read and Post Comments

Adding numerical methods to emacs with dynamic modules

| categories: emacs | tags: | View Comments

There is a relatively new feature in Emacs 25 that allows you to extend Emacs using compiled libraries (http://diobla.info/blog-archive/modules-tut.html). This could be very helpful in a few ways:

  1. To add functionality that exists in other libraries, e.g.
    1. libyaml
    2. libmemcached
    3. Embedding Ruby in Emacs
  2. Interface Emacs with hardware, e.g. a joystick, or ejecting a CD.
  3. To speed up slow elisp functions
    1. A c implementation of a fibonacci function is 150 times faster than an elisp version here.
    2. This json parser is up to 4 times faster than the json.el library for some operations.

I am interested in this in particular to bring numerical methods into Emacs. It is fair to ask why. Even I think the numpy/scipy/matplotlib Python stack is currently unparalleled in functionality for scientific programming. But I like writing elisp code so much more! So, we will take a look today at a simple example of integrating a function from the GNU Scientific Library into Emacs.

1 Using the GSL to calculate a Bessel function value

A canonical example of using the GSL is given at https://www.gnu.org/software/gsl/manual/html_node/An-Example-Program.html. Here it is for reference. It just calculates a value for a Bessel function. We save this program in a file called example.c.

#include <stdio.h>
#include <gsl/gsl_sf_bessel.h>

int
main (void)
{
  double x = 5.0;
  double y = gsl_sf_bessel_J0 (x);
  printf ("J0(%g) = %.18e\n", x, y);
  return 0;
}

We have to compile and run this program. Here are the commands to do that.

gcc -Wall -I/usr/local/include/ -c example.c
gcc -L/usr/local/include -lgsl example.o 
./a.out

That is a lot of code and steps to get one number.

What I would like to do instead is this:

(require 'gsl-sf-bessel)
(gsl-sf-bessel-J0 5)

So, enter the dynamic module!

2 A GSL dynamic module for a Bessel function

To create the dynamic module we need a small c file that wraps the GSL function and adds it to the Emacs environment. Here is the smallest example I could come up with for this one function. Basically, we create a function that takes the emacs environment and arguments, extract what we want from them, and use that to calculate what we need and return it to the environment. Then we register that function and define what the module provides in emacs_module_init.

#include <assert.h>
#include <gsl/gsl_sf_bessel.h>
#include "emacs-module.h"

int plugin_is_GPL_compatible;

static emacs_value
F_gsl_sf_bessel_J0 (emacs_env *env, ptrdiff_t nargs, emacs_value args[], void *data)
{
  assert (nargs == 1);
  double x = env->extract_float (env, args[0]);
  return env->make_float (env, gsl_sf_bessel_J0 (x));
}

int
emacs_module_init(struct emacs_runtime *ert)
{
        emacs_env *env = ert->get_environment(ert);

        emacs_value gsl_sf_bessel_J0_fn = env->make_function(env, 1, 1, F_gsl_sf_bessel_J0, "Regular cylindrical Bessel function of zeroth order, J_0(x)", NULL);

        emacs_value Qfset = env->intern(env, "fset");
        emacs_value Q_gsl_sf_bessel_J0 = env->intern(env, "gsl-sf-bessel-J0");
        emacs_value fset_args[] = { Q_gsl_sf_bessel_J0, gsl_sf_bessel_J0_fn };
        env->funcall(env, Qfset, 2, fset_args);

        emacs_value Qprovide = env->intern(env, "provide");
        emacs_value Q_gsl_sf_bessel = env->intern(env, "gsl-sf-bessel");
        emacs_value provide_args[] = { Q_gsl_sf_bessel };
        env->funcall(env, Qprovide, 1, provide_args);

        return 0;
}

Now we compile it into a shared library.

gcc -Wall -I/usr/local/include -fPIC -c gsl-sf-bessel.c
gcc -shared -L/usr/local/include -lgsl -o gsl-sf-bessel.so gsl-sf-bessel.o

That creates our shared library in gsl-sf-bessel.so.

ls *.so
(add-to-list 'load-path "/Users/jkitchin/vc/blogofile-jkitchin.github.com/_blog/dynamic-module/")
(require 'gsl-sf-bessel)
(gsl-sf-bessel-J0 5.0)
-0.17759677131433826

That is the same answer as we got before. Here is the documentation we defined. It could use some improvement, e.g. to note that the argument has to be a float, and that only one argument is allowed. I am not sure why the signature doesn't show a single argument.

(describe-function 'gsl-sf-bessel-J0)
gsl-sf-bessel-J0 is a Lisp function.

(gsl-sf-bessel-J0 &rest ARGS)

For more information check the manuals.

Regular cylindrical Bessel function of zeroth order, J_0(x)

I am not a very skilled C-programmer yet, so I don't know how hard it would be to make this function accept integers as well, or to vectorize it so you could have an arbitrary number of args to it and return a list.

3 Summary

Dynamic modules look promising to extend Emacs with. This example is about the simplest function from the GSL there is. There are many more (https://www.gnu.org/software/gsl/doc/html/index.html) functions that do linear algebra on arrays, integration or optimization of functions, interpolation of data, etc. I don't have a sense yet of how easy it will be to integrate these into a module.

It looks like you are not limited to writing these in C. There is an example of a plugin written in Rust here, and a framework to write them in Go. Maybe any language that can make a shared library with the required plugin_is_GPL_compatible symbol and emacs_module_init function would work. Those examples do not look significantly easier to write than the C versions though since I am not that fluent in those languages either.

There are some challenges to figure out in developing and using dynamic modules. Here are a few:

  1. The documentation on what is possible is not that great yet, so there is a lot of exploring to do. There are a fair number of examples out there though to learn from (https://github.com/emacs-pe/emacs-modules). The official example shows a lot of the functionality.
  2. I guess it will be tricky to distribute these. I don't know how easy it would be to build all the libraries for each platform for distribution on MELPA for example. I don't think there is a standard way to incorporate a compile step in elisp package installation. Also, you need an Emacs version of at least 25 with the dynamic module feature compiled in. It is not yet a default enabled option. The required emacs-module.h should be gotten from the emacs build, so people with binaries might not be able to build it anyway.
  3. Users will need the libraries the dynamic module uses. In this example, they will need libgsl.
  4. Once you require the module, it does not seem possible to modify it, rebuild it, and reload it. It appears you have to close Emacs and reload it. That is tedious.

It would be nice to have a more generic foreign function interface that would allow you to develop more on the elisp side. One effort in that direction is https://github.com/tromey/emacs-ffi. It looks like it might be a lot simpler to use than creating a dynamic module. Once it is installed, it looks like you can write elisp code to wrap the library functions. I will write about this on another day.

Copyright (C) 2017 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 9.0.7

Read and Post Comments

Adding keymaps to src blocks via org-font-lock-hook

| categories: orgmode, emacs | tags: | View Comments

Table of Contents

I had an idea to use custom keymaps in src-blocks. For example, you could then use lispy directly in your org-files without entering org-special-edit, or the elpy key-bindings in python blocks. There are other solutions I have seen, e.g. polymode, that claim to do this. You might guess that if they worked, I would not be writing this! There was some nice discussion about this idea on the org-mode mailing list, and Nicolas Goaziou pointed out this might be accomplished with the org-font-lock-hook.

You can check out the video here:

It was relatively easy to figure out how to do this. Keymaps can be added to regions during font-lock, so I just had to hook into the org-mode font lock system with a function to find the src blocks and add the keymap as a text-property. That took three steps:

  1. Define the keymaps to use. I use an a-list of (language . map) for this.
  2. Define the font-lock function. This will add the keymap properties to src-blocks.
  3. Define a minor mode to toggle this feature on and off.

Here is the definition of the keymaps. Generally I just copy the mode-map I want and then add some things to them. For example sometimes it is still a good idea to jump into the org-special-edit mode. For example, if you try to use a command in a Python block to send the buffer to the repl while in org-mode you are sure to get an error! You might also want to add the C-c C-e export command if you use that a lot. An alternative approach, of course, is to copy the org-map and add additional bindings to it. The choice is up to you.

(require 'lispy)
(require 'elpy)

(setq scimax-src-block-keymaps
      `(("ipython" . ,(let ((map (make-composed-keymap
                                  `(,elpy-mode-map ,python-mode-map ,pyvenv-mode-map)
                                  org-mode-map)))
                        ;; In org-mode I define RET so we f
                        (define-key map (kbd "<return>") 'newline)
                        (define-key map (kbd "C-c C-c") 'org-ctrl-c-ctrl-c)
                        map))
        ("python" . ,(let ((map (make-composed-keymap
                                 `(,elpy-mode-map ,python-mode-map ,pyvenv-mode-map)
                                 org-mode-map)))
                       ;; In org-mode I define RET so we f
                       (define-key map (kbd "<return>") 'newline)
                       (define-key map (kbd "C-c C-c") 'org-ctrl-c-ctrl-c)
                       map))
        ("emacs-lisp" . ,(let ((map (make-composed-keymap `(,lispy-mode-map
                                                            ,emacs-lisp-mode-map
                                                            ,outline-minor-mode-map)
                                                          org-mode-map)))
                           (define-key map (kbd "C-c C-c") 'org-ctrl-c-ctrl-c)
                           map))))

Next we define the function that will apply the keymap to each src block. The keymaps are only applied when they are defined in the variable above. This function is derived from org-fontify-meta-lines-and-blocks-1.

(defun scimax-add-keymap-to-src-blocks (limit)
  "Add keymaps to src-blocks defined in `scimax-src-block-keymaps'."
  (let ((case-fold-search t)
        lang)
    (while (re-search-forward org-babel-src-block-regexp limit t)
      (let ((lang (match-string 2))
            (beg (match-beginning 0))
            (end (match-end 0)))
        (if (assoc (org-no-properties lang) scimax-src-block-keymaps)
            (progn
              (add-text-properties
               beg end `(local-map ,(cdr (assoc
                                          (org-no-properties lang)
                                          scimax-src-block-keymaps))))
              (add-text-properties
               beg end `(cursor-sensor-functions
                         ((lambda (win prev-pos sym)
                            ;; This simulates a mouse click and makes a menu change
                            (org-mouse-down-mouse nil)))))))))))

Here we create an advice to trick any functions that need to know the major mode. We only apply the spoof if we are in org-mode and in a src block though. Otherwise we call the original function. So far lispy–eval is the only function I have needed it for. This might be a general strategy though to do other things like narrow to the src-block, or even go into special edit mode temporarily if there are commands that require it.

(defun scimax-spoof-mode (orig-func &rest args)
  "Advice function to spoof commands in org-mode src blocks.
It is for commands that depend on the major mode. One example is
`lispy--eval'."
  (if (org-in-src-block-p)
      (let ((major-mode (intern (format "%s-mode" (first (org-babel-get-src-block-info))))))
        (apply orig-func args))
    (apply orig-func args)))

We define a minor mode so we can toggle this on and off. Here we add the function to the org-font-lock-hook and advise the lispy–eval function. I had to add the font-lock-function to the end of the org-font-lock hook for some reason, and also add local-map as an extra-managed property so it would be removed when we toggle it off.

(define-minor-mode scimax-src-keymap-mode
  "Minor mode to add mode keymaps to src-blocks."
  :init-value nil
  (if scimax-src-keymap-mode
      (progn
        (add-hook 'org-font-lock-hook #'scimax-add-keymap-to-src-blocks t)
        (add-to-list 'font-lock-extra-managed-props 'local-map)
        (add-to-list 'font-lock-extra-managed-props 'cursor-sensor-functions)
        (advice-add 'lispy--eval :around 'scimax-spoof-mode)
        (cursor-sensor-mode +1))
    (remove-hook 'org-font-lock-hook #'scimax-add-keymap-to-src-blocks)
    (advice-remove 'lispy--eval 'scimax-spoof-mode)
    (cursor-sensor-mode -1))
  (font-lock-fontify-buffer))

(add-hook 'org-mode-hook (lambda ()
                           (scimax-src-keymap-mode +1)))

That is it! I am pretty sure this is a good idea. It helps a lot when you are writing a lot of short code blocks and near equal amounts of text (like in this blog post). It also helps write the code since many things like indentation, parentheses, etc. are automatically handled. That is what I used to go into special-edit mode all the time for!

I have not used this long enough to know if it causes any other surprises. If you try it and find any, leave a comment!

1 Update

It turns out you can have the best of all the worlds by combining keymaps. The make-composed-keymap creates a new keymap that combines a keymaps and falls through to a parent keymap. So here we use that to combine several keymaps, falling through to org-mode. The only subtlety I have come across is that I remapped <return> in orgmode to scimax/org-return, and not all modes define it, so I redefine it in some places to just be newline. Also to keep C-c C-c for executing the block, I add that back too.

I use a few maps here, and some of them seem to just add menus that are only active when your cursor is in the block. Pretty handy!

(setq scimax-src-block-keymaps
      `(("ipython" . ,(let ((map (make-composed-keymap
                                  `(,elpy-mode-map ,python-mode-map ,pyvenv-mode-map)
                                  org-mode-map)))
                        ;; In org-mode I define RET so we f
                        (define-key map (kbd "<return>") 'newline)
                        (define-key map (kbd "C-c C-c") 'org-ctrl-c-ctrl-c)
                        map))
        ("python" . ,(let ((map (make-composed-keymap
                                 `(,elpy-mode-map ,python-mode-map ,pyvenv-mode-map)
                                 org-mode-map)))
                       ;; In org-mode I define RET so we f
                       (define-key map (kbd "<return>") 'newline)
                       (define-key map (kbd "C-c C-c") 'org-ctrl-c-ctrl-c)
                       map))
        ("emacs-lisp" . ,(let ((map (make-composed-keymap `(,lispy-mode-map
                                                            ,emacs-lisp-mode-map
                                                            ,outline-minor-mode-map)
                                                          org-mode-map)))
                           (define-key map (kbd "C-c C-c") 'org-ctrl-c-ctrl-c)
                           map))))

2 Update #2

The previous version had some issues where it would only add a keymap to the first block. The code in this post now addresses that and uses cursor-sensor-functions to make sure we change key map on entering and leaving blocks. That might mean you need an emacs of at least version 25 to use this. I guess it will work with an earlier version, but the cursor-sensor-functions might get ignored. You might have to comment out the cursor-sensor-mode line

Thanks to those brave people alpha-testing this and helping refine the idea!

Copyright (C) 2017 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 9.0.7

Read and Post Comments

Org-mode and ipython enhancements in scimax

| categories: orgmode, emacs, ipython | tags: | View Comments

We have made some improvements to using Ipython in org-mode in the past including:

  1. Inline figures
  2. Export to Jupyter notebooks

Today I will talk about a few new features and improvements I have introduced to scimax for using org-mode and Ipython together.

The video for this post might be more obvious than the post:

1 Some convenience functions

There are a few nice shortcuts in the Jupyter notebook. Now we have some convenient commands in scimax to mimic those. My favorites are adding cells above or below the current cell. You can insert a new src block above the current one with (M-x org-babel-insert-block). You can use a prefix arg to insert it below the current block.

# code
# below
# some code

I am particularly fond of splitting a large block into two smaller blocks. Use (M-x org-babel-split-src-block) to do that and leave the point in the upper block. Use a prefix arg to leave the point in the lower block.

# lots of code in large block
# Even more code
# The end of the long block

You can execute all the blocks up to the current point with (M-x org-babel-execute-to-point).

2 ob-ipython-inspect works

In the original ob-ipython I found that ob-ipython-inspect did not work unless you were in special edit mode. That is too inconvenient. I modified a few functions to work directly from the org-buffer. I bind this to M-. in org-mode.

%matplotlib inline
import numpy as np

import matplotlib.pyplot as plt

# Compute areas and colors
N = 150
r = 2 * np.random.rand(N)
theta = 2 * np.pi * np.random.rand(N)
area = 200 * r**2
colors = theta

ax = plt.subplot(111, projection='polar')
c = ax.scatter(theta, r, c=colors, s=area, cmap='hsv', alpha=0.75)

<matplotlib.figure.Figure at 0x114ded710>

3 Getting selective output from Ipython

Out of the box Ipython returns a lot of results. This block, for example returns a plain text, image and latex result as output.

from sympy import *
# commenting out init_printing() results in no output
init_printing()

var('x y')
x**2 + y

2 x + y

We can select which one we want with a new header argument :ob-ipython-results. For this block you can give it the value of text/plain, text/latex or image/png.

var('x y')
x**2 + y

2 x + y

Or to get the image:

var('x y')
x**2 + y

This shows up with pandas too. This block creates a table of data and then shows the first 5 rows. Ipython returns both plain text and html here.

import pandas as pd
import numpy as np
import datetime as dt

def makeSim(nHosps, nPatients):
    df = pd.DataFrame()
    df['patientid'] = range(nPatients)
    df['hospid'] = np.random.randint(0, nHosps, nPatients)
    df['sex'] = np.random.randint(0, 2, nPatients)
    df['age'] = np.random.normal(65,18, nPatients)
    df['race'] = np.random.randint(0, 4, nPatients)
    df['cptCode'] = np.random.randint(1, 100, nPatients)
    df['rdm30d'] = np.random.uniform(0, 1, nPatients) < 0.1
    df['mort30d'] = np.random.uniform(0, 1, nPatients) < 0.2
    df['los'] = np.random.normal(8, 2, nPatients)
    return df

discharges = makeSim(50, 10000)
discharges.head()

patientid hospid sex age race cptCode rdm30d mort30d los 0 0 10 1 64.311947 0 8 False False 8.036793 1 1 6 0 82.951484 1 73 True False 7.996024 2 2 27 1 53.064501 3 95 False False 9.015144 3 3 37 0 64.799128 0 93 False False 10.099032 4 4 46 0 99.111394 2 25 False False 11.711427

patientid hospid sex age race cptCode rdm30d mort30d los
0 0 10 1 64.311947 0 8 False False 8.036793
1 1 6 0 82.951484 1 73 True False 7.996024
2 2 27 1 53.064501 3 95 False False 9.015144
3 3 37 0 64.799128 0 93 False False 10.099032
4 4 46 0 99.111394 2 25 False False 11.711427

We can use the header to select only the plain text output!

import pandas as pd
import numpy as np
import datetime as dt

def makeSim(nHosps, nPatients):
    df = pd.DataFrame()
    df['patientid'] = range(nPatients)
    df['hospid'] = np.random.randint(0, nHosps, nPatients)
    df['sex'] = np.random.randint(0, 2, nPatients)
    df['age'] = np.random.normal(65,18, nPatients)
    df['race'] = np.random.randint(0, 4, nPatients)
    df['cptCode'] = np.random.randint(1, 100, nPatients)
    df['rdm30d'] = np.random.uniform(0, 1, nPatients) < 0.1
    df['mort30d'] = np.random.uniform(0, 1, nPatients) < 0.2
    df['los'] = np.random.normal(8, 2, nPatients)
    return df

discharges = makeSim(50, 10000)
discharges.head()

patientid hospid sex age race cptCode rdm30d mort30d los 0 0 21 0 73.633836 1 38 False False 7.144019 1 1 16 1 67.518804 3 23 False False 3.340534 2 2 15 0 44.139033 0 8 False False 9.258706 3 3 29 1 45.510276 2 5 False False 10.590245 4 4 7 0 52.974924 2 4 False True 5.811064

4 Where was that error?

A somewhat annoying feature of running cells in org-mode is when there is an exception there has not been a good way to jump to the line that caused the error to edit it. The lines in the src block are not numbered, so in a large block it can be tedious to find the line. In scimax, when you get an exception it will number the lines in the src block, and when you press q in the exception traceback buffer it will jump to the line in the block where the error occurred.

print(1)
#raise Exception('Here')
print(2)

1 2

If you don't like the numbers add this to your init file:

(setq ob-ipython-number-on-exception nil)

5 Asynchronous Ipython

I have made a few improvements to the asynchronous workflow in Ipython. We now have a calculation queue, so you can use C-c C-c to execute several blocks in a row, and they will run asynchronously in the order you ran them. While they are running you can continue using Emacs, e.g. writing that paper, reading email, checking RSS feeds, tetris, … This also lets you run all the blocks up to the current point (M-x org-babel-execute-ipython-buffer-to-point-async) or the whole buffer (of Ipython) blocks asynchronously (M-x org-babel-execute-ipython-buffer-async).

To turn this on by default put this in your init file:

(setq org-babel-async-ipython t)

This requires all src blocks to have a name, and running the block will give it a name if you have not named the block. By default we use human-readable names. While the block is running, there will be a link indicating it is running. You can click on the link to cancel it. Running subsequent blocks will queue them to be run when the first block is done.

Here is an example:

import time
time.sleep(5)
a = 5
print('done')
print(3 * a)

15

Occasionally you will run into an issue. You can clear the queue with org-babel-async-ipython-clear-queue.

Copyright (C) 2017 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 9.0.5

Read and Post Comments

« Previous Page -- Next Page »