An Emacs zeromq library using an ffi

Thu, 13 Jul 2017 06:44:23 EDT

1. Summary thoughts
2. Modified ffi-define-function macro
3. The zeromq bindings

An alternative approach to writing your own dynamic module (which requires some proficiency in c) is to use a foreign function interface (ffi). There is one for emacs at https://github.com/tromey/emacs-ffi, and it is also a dynamic module itself that uses libffi. This lets you use elisp to create functions in Emacs that actually call functions in some other library installed on your system. Here, we use this module to recreate our zeromq bindings that I previously posted.

The emacs-ffi module works fine as it is, but I found it useful to redefine one of the main macros (define-ffi-function) with the following goals in mind:

Allow me to specify the argument names and docstrings for each arg that contain its type and a description of the arg.
Document what each function returns (type and description).
Combine those two things into an overall docstring on the function.

These are important to me because it allows Emacs to work at its fullest potential while writing elisp code, including having the right function signatures in eldoc, and easy access to documentation of each function. You can see the new definition here. For example, here is a docstring for zmq-send using that new macro:

zmq-send is a Lisp function.

(zmq-send *SOCKET *MSG LEN FLAGS)

For more information check the manuals.

send a message part on a socket.
http://api.zeromq.org/4-2:zmq-send

*SOCKET (:pointer) Pointer to a socket.
*MSG (:pointer) Pointer to a C-string to send
LEN (:size_t) Number of bytes to send
FLAGS (:int) 

Returns: Number of bytes sent or -1 on failure. (:int)

That has everything you need to know

(define-ffi-function zmq-send-ori "zmq_send" :int (:pointer :pointer :size_t :int) zmq)

zmq-send-ori

Compare that to this docstring from the original macro.

zmq-send-ori is a Lisp function.

(zmq-send-ori G251 G252 G253 G254)

For more information check the manuals.

You can see the zeromq function definitions in elisp here. Here is a list of the functions we have created:

Type RET on a type label to view its full documentation.

zmq
  Function: Returns a pointer to the libzmq library.
zmq-close
  Function: close ØMQ socket.
zmq-connect
  Function: create outgoing connection from socket.
zmq-ctx-destroy
  Function: terminate a ØMQ context.
zmq-ctx-new
  Function: create new ØMQ context.
zmq-recv
  Function: receive a message part from a socket.
zmq-send
  Function: send a message part on a socket.
zmq-socket
  Function: create ØMQ socket.

Now we can use these to create the client, this time in elisp. Just as in the last post, you need to run the hwserver in a terminal for this to work. Here is the client code.

(let* ((context (zmq-ctx-new))
       (socket (zmq-socket context ZMQ-REQ)))

  (with-ffi-string (endpoint "tcp://localhost:5555")
    (zmq-connect socket endpoint))

  (with-ffi-string (msg "Hi there")
    (zmq-send socket msg 5 0))

  (with-ffi-string (recv (make-string 10 ""))
    (let ((status -1))
      (cl-loop do (setq status (zmq-recv socket recv 10 0)) until (not (= -1 status)))) 
    (print (ffi-get-c-string recv)))

  (zmq-close socket)
  (zmq-ctx-destroy context))


"World     "

This client basically performs the same as the previously one we built. You can see we are mixing some programming styles here. For example, we have to create pointers to string variables in advance that the ffi will be writing to later like we would do in c. We use the with-ffi-string macro which frees the pointer when we are done with it. It basically just avoids me having to create, use, and destroy the pointers myself. So, there it is, a working elisp zeromq client!

1 Summary thoughts

For this example, I feel like the ffi approach here (with my modified function making macro) was much easier than what I previously did with a compiled c-library (although it benefited a lot from my recent work on the c-library). I really like working in elisp, which is a much greater strength of mine than programming in c. It is pretty clear, however, that you have to know how c works to use this, otherwise it isn't so obvious that some functions will return a status, and do something by side effect, e.g. put results in one of the arguments. The signatures of the ffi functions are basically limited by the signatures of the c-functions. If you want to change the signature in Emacs, you have to write wrapper functions to do that.

The macro I used here to create the functions creates really good (the kind I like anyway) docstrings when you use it fully. That isn't a totally new idea, I tried it out here before. In contrast, the original version not only didn't have a docstring, but every arg had a gensym (i.e. practically random) name! I think it would be very difficult to get the same level of documentation when writing c-code to make a module. In the c-code, there is a decoupling of the definition of a c-function (which always has the same signature) that gets data from the Emacs env, e.g. arguments, does stuff with them, and creates data to put back into the env, and the emacs_module_init function where you declare these functions to Emacs and tell it what to call the function in emacs, about how many arguments it takes, etc… The benefit of this is you define what the Emacs signature will look like, and then write the c-function that does the required work. The downside of this is the c-function and Emacs declarations are often far apart in the editor, and there is no easy way to auto-generate docstrings like I can with lisp macros. You would have to manually build them up yourself, and keep them synchronized. Also, I still have not figured out how to get emacs to show the right signature for c-generated functions.

The ffi approach still uses a dynamic module approach, so it still requires a modern Emacs with the module compiled and working. It still requires (in this case) the zeromq library to be installed on the system too. Once you have those, however, the elisp zeromq bindings by this approach is done completely in elisp!

It will be interesting in the coming weeks to see how this approach works with the GNU Scientific Library, particularly with arrays. Preliminary work shows that while the elisp ffi code is much shorter and easier to write than the corresponding c-code for some examples (e.g. a simple mathematical function), it is not as fast. So if performance is crucial, it may still pay off to write the c-code.

2 Modified ffi-define-function macro

Here are two macros I modified to add docstrings and named arguments too.

(defmacro define-ffi-library (symbol name)
  "Create a pointer named to the c library."
  (let ((library (cl-gensym))
        (docstring (format "Returns a pointer to the %s library." name)))
    (set library nil)
    `(defun ,symbol ()
       ,docstring
       (or ,library
           (setq ,library (ffi--dlopen ,name))))))

(defmacro define-ffi-function (name c-name return args library &optional docstring)
  "Create an Emacs function from a c-function.
NAME is a symbol for  the emacs function to create.
C-NAME is a string of the c-function to use.
RETURN is a type-keyword or (type-keyword docstring)
ARGS is a list of type-keyword or (type-keyword name &optional arg-docstring)
LIBRARY is a symbol usually defined by `define-ffi-library'
DOCSTRING is a string for the function to be created.

An overall docstring is created for the function from the arg and return docstrings.
"
  ;; Turn variable references into actual types; while keeping
  ;; keywords the same.
  (let* ((return-type (if (keywordp return)
                          return
                        (car return)))
         (return-docstring (format "Returns: %s (%s)"
                                   (if (listp return)
                                       (second return)
                                     "")
                                   return-type))
         (arg-types (vconcat (mapcar (lambda (arg)
                                       (if (keywordp arg)
                                           (symbol-value arg)
                                         ;; assume list (type-keyword name &optional doc)
                                         (symbol-value (car arg))))
                                     args)))
         (arg-names (mapcar (lambda (arg)
                              (if (keywordp arg)
                                  (cl-gensym)
                                ;; assume list (type-keyword name &optional doc)
                                (second arg)))
                            args))
         (arg-docstrings (mapcar (lambda (arg)
                                   (cond
                                    ((keywordp arg)
                                     "")
                                    ((and (listp arg) (= 3 (length arg)))
                                     (third arg))
                                    (t "")))
                                 args))
         ;; Combine all the arg docstrings into one string
         (arg-docstring (mapconcat 'identity
                                   (mapcar* (lambda (name type arg-doc)
                                              (format "%s (%s) %s"
                                                      (upcase (symbol-name name))
                                                      type
                                                      arg-doc))
                                            arg-names arg-types arg-docstrings)
                                   "\n"))
         (function (cl-gensym))
         (cif (ffi--prep-cif (symbol-value return-type) arg-types)))
    (set function nil)
    `(defun ,name (,@arg-names)
       ,(concat docstring "\n\n" arg-docstring "\n\n" return-docstring)
       (unless ,function
         (setq ,function (ffi--dlsym ,c-name (,library))))
       ;; FIXME do we even need a separate prep?
       (ffi--call ,cif ,function ,@arg-names))))

define-ffi-function

3 The zeromq bindings

These define the ffi functions we use in this post. I use a convention that pointer args start with a * so they look more like the c arguments. I also replace all _ with - so it looks more lispy, and the function names are easier to type.

(add-to-list 'load-path (expand-file-name "."))
(require 'ffi)

(define-ffi-library zmq "libzmq")


(define-ffi-function zmq-ctx-new "zmq_ctx_new"
  (:pointer "Pointer to a context")
  nil zmq
  "create new ØMQ context.
http://api.zeromq.org/4-2:zmq-ctx-new")


(define-ffi-function zmq-ctx-destroy "zmq_ctx_destroy"
  (:int "status")
  ((:pointer *context)) zmq
  "terminate a ØMQ context.
http://api.zeromq.org/4-2:zmq-ctx-destroy")


(define-ffi-function zmq-socket "zmq_socket"
  (:pointer "Pointer to a socket.")
  ((:pointer *context "Created by `zmq-ctx-new '.") (:int type)) zmq
  "create ØMQ socket.
http://api.zeromq.org/4-2:zmq-socket")


(define-ffi-function zmq-close "zmq_close"
  (:int "Status")
  ((:pointer *socket "Socket pointer created by `zmq-socket'")) zmq
  "close ØMQ socket.
http://api.zeromq.org/4-2:zmq-close")


(define-ffi-function zmq-connect "zmq_connect" 
  (:int "Status")
  ((:pointer *socket "Socket pointer created by `zmq-socket'")
   (:pointer *endpoint "Char pointer, e.g. (ffi-make-c-string \"tcp://localhost:5555\")"))
  zmq
  "create outgoing connection from socket.
http://api.zeromq.org/4-2:zmq-connect")


(define-ffi-function zmq-send "zmq_send"
  (:int "Number of bytes sent or -1 on failure.")
  ((:pointer *socket "Pointer to a socket.")
   (:pointer *msg "Pointer to a C-string to send")
   (:size_t len "Number of bytes to send")
   (:int flags)) 
  zmq
   "send a message part on a socket.
http://api.zeromq.org/4-2:zmq-send")


(define-ffi-function zmq-recv "zmq_recv"
  (:int "Number of bytes received or -1 on failure.")
  ((:pointer *socket)
   (:pointer *buf "Pointer to c-string to put result in.")
   (:size_t len "Length to truncate message at.")
   (:int flags)) 
  zmq
   "receive a message part from a socket.
http://api.zeromq.org/4-2:zmq-recv")


;; We cannot get these through a ffi because the are #define'd for the CPP and
;; invisible in the library. They only exist in the zmq.h file.
(defconst ZMQ-REQ 3
  "A socket of type ZMQ_REQ is used by a client to send requests
  to and receive replies from a service. This socket type allows
  only an alternating sequence of zmq_send(request) and
  subsequent zmq_recv(reply) calls. Each request sent is
  round-robined among all services, and each reply received is
  matched with the last issued request.")

org-mode source

Org-mode version = 9.0.7

The Kitchin Research Group

An Emacs zeromq library using an ffi

Table of Contents

1 Summary thoughts

2 Modified ffi-define-function macro

3 The zeromq bindings