The Kitchin Research Group

I do a lot of scientific programming, and it is one of the reasons I have been learning to extend Emacs with dynamic modules. They have allowed me to add physical constants, numerical integration, root finding and linear algebra from established c-libraries to Emacs. Today I am taking a break from that and finally getting to another one of the reasons I started playing around with dynamic modules: zeromq. Zeromq is a messaging library that Jupyter uses to communicate with kernels. I thought we might get a smoother integration with Emacs and Jupyter if we could use zeromq directly to communicate between org-mode and the kernel. Currently we have to run a web server that does the communication for us via http requests. We won't solve the Jupyter problem today, but we will look at communicating with a Zeromq server from Emacs.

This might have lots of other useful applications. Suppose Emacs could communicate directly with other zeromq servers to retrieve data from, perhaps scientific data. It might even be possible for Emacs to run its own zeromq server, and other instances of Emacs could then communicate with it. Collaborative editing anyone?

Here we just implement the "Hello world" client example in the zeromq guide. The code for the server, a c-client, the mod-zmq library, a makefile and tests can be found at https://github.com/jkitchin/emacs-modules/tree/master/zeromq. All the server does is receive a string, and then send a response (in this case just the string "World") back to the client.

To run this, make sure to run the hwserver executable in a terminal. I wrapped the zeromq commands required to implement the client into a dynamic module. Since this example focuses on strings, the module returns strings to Emacs. I am not sure if that is always the right thing to do, as zeromq more generically uses bytes, but I will have to wait until I know more about zeromq to know if this is an issue.

This dynamic module uses a new feature that none of the previous posts used, and that is the user_ptr. These allow you to essentially return a reference pointer back to emacs that you can pass back to another function. That way they stay alive between function calls. For example, here we have to create a context and socket and pass these items to functions like zmq_send and zmq_recv.

The directory this library is in is not on my path, so we load it like this:

(add-to-list 'load-path (expand-file-name "."))
(require 'mod-zmq)

Here are the functions and their signatures that have been implemented so far. I only implemented the ones I needed for the client. The signatures may change in the future; this is just a proof of concept for now for the purpose of building the client.

(apropos-command "zmq*" t)
(with-current-buffer "*Apropos*" (buffer-string))

Type RET on a type label to view its full documentation.

zmq-close
  Function: (zmq-close SOCKET)
zmq-connect
  Function: (zmq-connect SOCKET ENDPOINT)
zmq-ctx-destroy
  Function: (zmq-ctx-destroy CONTEXT)
zmq-ctx-new
  Function: (zmq-ctx-new)
zmq-recv
  Function: (zmq-recv SOCKET LEN FLAGS)
zmq-send
  Function: (zmq-send SOCKET BUF FLAGS)
zmq-socket
  Function: (zmq-socket CONTEXT TYPE)

You can see the c code for the client here: hwclient.c . Here is a simple elisp version of the hwclient that basically does the same thing! The main difference is I added a while loop around the zmq-recv because sometimes it returns -1 and no result. So, here we loop until the return result is not -1. That seems to do the right thing.

(let* ((context (zmq-ctx-new))
       (socket (zmq-socket context ZMQ-REQ))
       (recv-ret -1)
       (result))

  (zmq-connect socket "tcp://localhost:5555")
  (zmq-send socket "Hello" 0)

  (while (= recv-ret -1)
    (setq result (zmq-recv socket 10 0)
          recv-ret (second result)))

  (print result)

  (zmq-close socket)
  (zmq-ctx-destroy context))

("World" 5)

Basically this creates the context, then the socket, and connects to it on port 5555 of the localhost where the server is running. Then we send the string "Hello". The server returns the string "World" and tells us it sent 5 bytes. Then we close the socket and destroy the context. There is a lot of code in the module to make this happen. A lot of it is converting args in emacs functions to things we can use in c, running a few lines of zmq commands, and then code to convert those results back to emacs values. Finally, there is code to register each function and define docstrings for them. I am not totally convinced this is the best way to do this, but it does work! An alternative might be emacs-ffi, which might enable most of this to be developed in just elisp.

Copyright (C) 2017 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 9.0.7