Running scientific instruments in Emacs and recording the results

| categories: notebook, orgmode, emacs | tags:

Today we look at running a scientific instrument via http requests from Emacs and org-mode. We will use a Gamry Ref600 potentiostat because Gamry has very nicely provide a COM interface we can access via Python. This will be only a proof of concept to see what it is like. We will not consider any issues of security, etc…, only what is it like to do it.

The setup will look like this: we will run a flask web app that uses python to control the instrument via http requests. Why? Because I want to run the instrument from my Mac ;) and so far there are only Windows drivers for the instrument. So, we run the flask app on the Windows machine, and I run it from here on my Mac by sending requests. Flask takes care of converting requests to action using Python. You can see the Flask app here.

Let's see what is online:

curl jkitchin-win.cheme.cmu.edu:5000/pstats
(u'REF600-13089',)

We have one potentiostat online with serial number 13089. I have a dummy cell connected to it which has a little resistor on it. So we can run a cyclic voltammogram and it should be a straight line. We have to know a bit about what is returned. We will get a json file back, and it will have the data in it. The data will be a list of lists. The data we want is in columns 1 and 3 (python indexing). Obviously you need some prior knowledge of what data comes back to use this. That would come from reading some documentation.

import requests
import numpy as np
import matplotlib.pyplot as plt

resp = requests.get('http://jkitchin-win.cheme.cmu.edu:5000/cv?endv=0.25&startv=-0.25')

dj = resp.json()
data = np.array(dj['data'])

plt.plot(data[:, 1], data[:, 3])
plt.xlabel('Voltage (V)')
plt.ylabel('Current (A)')
plt.tight_layout()
plt.savefig('cv-1.png')

Well, there you have it. Possibly the first Gamry Ref600 to ever have been driven from a Mac ;) Let me be more explicit about that; I could also run this from Linux, an iPad, etc… You could do this in a browser, or in an IPython notebook, or in Matlab, among many other possibilities. You could write a script in perl, shell, ruby, emacs-lisp, or any other language that supports http requests.

I am not sure why the graph is not perfectly linear, maybe there is some capacitive charging that starts out. The resistance based on the current at 0.2V is about 2000 ohms, which is in good agreement with what is listed on the board the dummy cell is on.

1 Summary thoughts

There are a host of interesting issues one eventually has to consider here including security, but also error management and debugging. I hacked something like an http api here by running flask on the windows machine running the instrument. That is a layer of abstraction on an abstraction to start with. I think later instruments are likely to run these webservers themselves on small dedicated computers, e.g. via a Raspberry pi or Arduino chipset. It is not obvious how sophisticated you can make this with respect to triggering different instruments, etc…

In running this, my "notebook" was blocked while the experiment ran. It is possible to run things asynchronously, and sometimes that would make sense. In the example here, we have provided a very limited set of functions to "run" the potentiostat. It was only a proof of concept to get a sense for what it is like. In practice a fuller set of functions would be implemented. Another point to consider is how the data comes back from the potentiostat. We used json here because it is convenient, but we could just as well send files, and other sorts of data too.

This lays out the possibility to walk up to an instrument with an electronic notebook, setup and run the experiment, capture the results in the notebook and take it back to the office for analysis. Pretty cool.

2 Flask app

So, here is my flask app. We setup a few routes using get requests to do things like get a list of the potentiostats online, and to run a cyclic voltamogram. As a side note, after this post is over, I am turning off the app, so you won't be able to repeat the codes ;) This is not a beautiful, secure or error tolerant code. It works enough for a proof of concept of simple experiments.

from flask import Flask, request, jsonify
import time

app = Flask(__name__)

@app.route('/')
def hello_world():
    return 'Hello World!'

@app.route('/pstats')
def get_pstats():
    import win32com.client as client
    devices = client.Dispatch('GamryCOM.GamryDeviceList')
    result = str(devices.EnumSections())
    return result

@app.route('/close_pstat')
def close():
    import win32com.client as client
    devicelist = client.Dispatch('GamryCOM.GamryDeviceList')

    x = devicelist.EnumSections()[0]
    pstat = client.Dispatch('GamryCOM.GamryPstat')
    pstat.Init(x)

    pstat.Close()


def run_ramp(Sinit,  # start value
             Sfinal, # end value
             ScanRate=1,
             SampleRate=0.01,
             CtrlMode=1,  # GamryCOM.PstatMode
             fname=None):
    '''We assume the first device is the one you want.
    '''
    import win32com.client as client
    import numpy as np
    devicelist = client.Dispatch('GamryCOM.GamryDeviceList')

    x = devicelist.EnumSections()[0]

    pstat = client.Dispatch('GamryCOM.GamryPstat')
    pstat.Init(x)

    pstat.Open()

    dtaqcpiv=client.Dispatch('GamryCOM.GamryDtaqCpiv')
    dtaqcpiv.Init(pstat)

    sigramp=client.Dispatch('GamryCOM.GamrySignalRamp')
    sigramp.Init(pstat, Sinit, Sfinal, ScanRate, SampleRate, CtrlMode)

    pstat.SetSignal(sigramp)
    pstat.SetCell(1) # 1 == GamryCOM.CellOn

    try:
        dtaqcpiv.Run(True)
    except Exception as e:
        pstat.Close()
        raise

    # NOTE:  The comtypes example in this same directory illustrates the use of com
    # notification events.  The comtypes package is recommended as an alternative
    # to win32com.
    time.sleep(2) # just wait sufficiently long for the acquisition to complete.

    acquired_points = []
    count = 1
    while count > 0:
        count, points = dtaqcpiv.Cook(10)
        # The columns exposed by GamryDtaq.Cook vary by dtaq and are
        # documented in the Toolkit Reference Manual.
        acquired_points.extend(zip(*points))

    acquired_points = np.array(acquired_points)
    if fname is not None:
        np.savetxt(fname, acquired_points)

    pstat.Close()
    return jsonify({'data': acquired_points.tolist()})

@app.route('/cv')
def run_cv():
    result = str(request.values)
    startv = float(request.values.get('startv', -0.1))
    endv = float(request.values.get('endv', 0.1))
    scanrate = float(request.values.get('scanrate', 1.0))
    samplerate = float(request.values.get('samplerate', 0.01))

    data = run_ramp(startv, endv, scanrate, samplerate)
    return data


if __name__ == '__main__':
    app.run(host='jkitchin-win.cheme.cmu.edu', port=5000, debug=True)

Copyright (C) 2015 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.10

Discuss on Twitter

A sudo org-link and sh block

| categories: babel, orgmode, emacs | tags:

Shell blocks in org-mode are pretty useful, but they are a little limited in that it is not obvious how to run a sudo command in them.

So for example, this gives me a permission denied error.

ls /var/audit

One way to get around this is to create an org-mode link like this one:

;http://stackoverflow.com/questions/2472273/how-do-i-run-a-sudo-command-in-emacs
(org-add-link-type
 "sudo"
 (lambda (cmd)
   "Run CMD with sudo."
   (shell-command
    (concat "echo " (shell-quote-argument (read-passwd "Password? "))
            " | sudo -S " cmd))))

Now you can create a link like ls /var/audit, and when you click on it you will be prompted for a password, and then you will see a buffer containing the output. To get an actual sudo code block, you need a new org babel library. Here is an example of what it might look like. Tangle this file to generate the library. Note: This is a lightly modified version of ob-emacs-lisp.el, and I have not tested it very thoroughly.

;;; ob-sudo.el --- An org-mode source block to run shell commands as sudo

;;; Commentary:
;; Runs the block of code as a shell command with sudo.

;;; Code:

(defun org-babel-execute:sudo (body params)
  "Run BODY as a shell command using sudo."
  (let* ((passwd (shell-quote-argument (read-passwd "Password? ")))
         (result (shell-command-to-string
                  (concat "echo " passwd
                          " | sudo -S " body))))
    ;; this is verbatim from ob-emacs-lisp
    (org-babel-result-cond (cdr (assoc :result-params params))
      (let ((print-level nil)
            (print-length nil))
        (if (or (member "scalar" (cdr (assoc :result-params params)))
                (member "verbatim" (cdr (assoc :result-params params))))
            (format "%S" result)
          (format "%s" result)))
      (org-babel-reassemble-table
       result
       (org-babel-pick-name (cdr (assoc :colname-names params))
                            (cdr (assoc :colnames params)))
       (org-babel-pick-name (cdr (assoc :rowname-names params))
                            (cdr (assoc :rownames params)))))))

(provide 'ob-sudo)
;;; ob-sudo.el ends here

Let us add the current dir to our path so we can load it. If you use this a lot, you should put the library on your permanent path.

(add-to-list 'load-path (expand-file-name "."))

Now, add the sudo "language" to org-babel-load-languages.

(org-babel-do-load-languages
 'org-babel-load-languages
 '((emacs-lisp . t)
   (python . t)
   (sh . t)
   (matlab . t)
   (sqlite . t)
   (ruby . t)
   (perl . t)
   (org . t)
   (dot . t)
   (plantuml . t)
   (R . t)
   (sudo . t)))

And, here it is in action. Hopefully I am not giving away some important information here!

ls /var/audit
20141106003522.20141110021519
20141110021548.crash_recovery
20141112154126.crash_recovery
20141119201541.20141122145259
20141122145317.20141124214930
20141124215000.crash_recovery
20141126062011.20141202192451
20141202192507.crash_recovery
20141210133306.crash_recovery
20141225181819.20150106015256
20150106015325.20150111010018
20150111010121.crash_recovery
20150115195518.20150115200101
20150115200110.crash_recovery
20150123061227.20150215123411
20150215123454.crash_recovery
20150225004740.20150310201600
20150310201633.20150314214730
20150314214807.crash_recovery
20150323145600.20150329170647
20150329170721.crash_recovery
20150407215846.20150413000423
20150413000438.20150421122044
20150421122104.20150518122545
20150518122616.20150518124432
20150518124432.20150518124513
20150518124513.20150518125437
20150518125437.20150518125935
20150518125935.20150518132111
20150518132111.20150531202621
20150531202719.20150601123612
20150601123612.20150601124932
20150601124932.20150601125151
20150601125151.20150601125555
20150601125555.20150601131947
20150601131947.20150601132421
20150601132421.20150601133735
20150601133735.20150601140740
20150601140740.20150601154012
20150601154012.20150601155125
20150601155125.20150601155215
20150601155215.20150601160937
20150601160937.crash_recovery
20150613061543.20150614054541
20150614054541.20150625165357
20150625165432.20150625200623
20150625200623.20150628042242
20150628042242.20150628103628
20150628103628.20150630052100
20150630052100.20150701232519
20150702005345.20150710203212
20150710203226.not_terminated
current

Summary thoughts: I will reiterate again I have not tested this a lot, I was mostly interested in trying to make a new sh block with sudo support. Let me know if it has issues for you, and make sure you have backups of things it could mess up!

Copyright (C) 2015 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.10

Discuss on Twitter

Indexing headlines in org files with swish-e with laser-sharp results

| categories: orgmode, swishe, emacs | tags:

So far, it looks like swish-e is able to do some pretty focused searches on specific content types. However, the return results are not actually that sharp; in the way we have been using swish-e, it can only tell us the document path that matches, not where in the document the match is. To fix that, we need a new approach to what a "document" is, and a new approach to indexing. We will finally use the "-s prog" option in swish-e which runs an external program that prints stuff to stdout for swish-e to index. We will treat each headline in an org file as a "document" but rather than have the path to the file, we will put an org-mode link there that will take us right to the point of interest.

You can see this in action here: https://www.youtube.com/watch?v=bTwXtEb5Ng8

Basically, we need a program to output chunks like this for each headline in an org-file:

Path-Name: [[elisp:(progn (find-file "/Users/jkitchin/blogofile-jkitchin.github.com/_blog/ase-db.org") (goto-char 1))]]
Content-Length: 247
Document-Type: XML*

<headline><title>Using the ase database module</title><properties><FILE>/Users/jkitchin/blogofile-jkitchin.github.com/_blog/ase-db.org</FILE><BLOCKED></BLOCKED><categories>python, ase</categories><CATEGORY>ase-db</CATEGORY></properties></headline>

Then we need to tell swish-e to run the program and index its output. Here is the program to do that.

:;exec emacs -batch -l $0 "$@"
(require 'org)
(require 'xml)
(require 'cl)

(add-to-list 'load-path "~/Dropbox/kitchingroup/jmax/elpa/f-20140828.716")
(add-to-list 'load-path "~/Dropbox/kitchingroup/jmax/elpa/s-20140910.334")
(add-to-list 'load-path "~/Dropbox/kitchingroup/jmax/elpa/dash-20141201.2206")
(require 'f)

(defun print-tag (name attrs &optional closingp)
  "Print an xml tag with symbol NAME and ATTRS (a cons list of (attribute . value)).
if CLOSINGP print the closing tag instead."
  (format
   "<%s%s%s>"
   (if closingp "/" "")
   name
   (if (and attrs (not closingp))
       (concat
        " "
        (mapconcat
         (lambda (x)
           (format "%s=\"%s\""
                   (car x)
                   (xml-escape-string (cdr x))))
         attrs
         " "))
     "")))

(defmacro tag (name attributes &rest body)
  "macro to create an xml tag with NAME, ATTRIBUTES. BODY is executed in the tag."
  `(format "%s%s%s"
           (print-tag ,name ,attributes nil)
           (concat
            ,@body)
           (print-tag ,name nil t)))

(defun headline-xml (headline)
  "Return xml representation of an element HEADLINE."
  (let ((title (org-element-property :title headline))
        (properties (save-excursion
                      (goto-char
                       (org-element-property :begin headline))
                      (org-entry-properties))))
    (tag 'headline ()
         (tag 'title () (xml-escape-string (mapconcat 'identity title " ")))
         (when properties
           (tag 'properties ()
                (mapconcat
                 'identity
                 (loop for (p . v) in properties
                       collect (tag p () (xml-escape-string v)))
                 ""))))))

(defun headline-document (headline)
  "Return the headline \"document\" for swish-e to index."
  (let ((xml (replace-regexp-in-string
              "[^[:ascii:]]" ""
              (headline-xml headline))))
    (format "Path-Name: [[elisp:(progn (find-file \"%s\") (goto-char %s) (show-children))][link]]
Content-Length: %s
Document-Type: XML*

%s" (buffer-file-name)
(org-element-property :begin headline)
(length xml)
xml)))

(defun process-file (fname)
  "Print the `headline-document' for each headline in FNAME."
  (with-current-buffer (find-file-noselect fname)
    (mapconcat 'identity
               (org-element-map (org-element-parse-buffer)
                   'headline
                 (lambda (headline)
                   (princ (headline-document headline))))
               "")))

;; Here is the main work in the script.
(loop for dir in '("/Users/jkitchin/blogofile-jkitchin.github.com/_blog")
      do
      (loop for fname in (f-entries
                          dir
                          (lambda (x)
                            (string=  "org"  (file-name-extension x)))
                          t)
            do (ignore-errors
                 (princ (process-file fname)))))

Now we need a configuration file:

# Example configuration file

# where to save the index
IndexFile /Users/jkitchin/blogofile-jkitchin.github.com/_blog/index-org-headlines.swish-e

# index all tags for searching
UndefinedMetaTags auto
UndefinedXMLAttributes auto

And we run the indexer, I did this in an actual shell. For some reason, it was not possible to run here. The output is pretty useful though, as it tells you what MetaNames are searchable.

swish-e -c swish-org-headlines.conf -S prog -i ./swish-org-headlines.el
10:17 $ swish-e -c swish-org-headlines.conf -S prog -i ./swish-org-headlines.el
Indexing Data Source: "External-Program"
Indexing "./swish-org-headlines.el"
External Program found: ./swish-org-headlines.el
**Adding automatic MetaName 'headline' found in file '[[elisp:(progn (find-file "/Users/jkitchin/blogofile-jkitchin.github.com/_blog/writing-exams-in-orgmode.org") (goto-char 18) (show-children))][link]]'
**Adding automatic MetaName 'title' found in file '[[elisp:(progn (find-file "/Users/jkitchin/blogofile-jkitchin.github.com/_blog/writing-exams-in-orgmode.org") (goto-char 18) (show-children))][link]]'
**Adding automatic MetaName 'properties' found in file '[[elisp:(progn (find-file "/Users/jkitchin/blogofile-jkitchin.github.com/_blog/writing-exams-in-orgmode.org") (goto-char 18) (show-children))][link]]'
**Adding automatic MetaName 'file' found in file '[[elisp:(progn (find-file "/Users/jkitchin/blogofile-jkitchin.github.com/_blog/writing-exams-in-orgmode.org") (goto-char 18) (show-children))][link]]'
**Adding automatic MetaName 'blocked' found in file '[[elisp:(progn (find-file "/Users/jkitchin/blogofile-jkitchin.github.com/_blog/writing-exams-in-orgmode.org") (goto-char 18) (show-children))][link]]'
**Adding automatic MetaName 'categories' found in file '[[elisp:(progn (find-file "/Users/jkitchin/blogofile-jkitchin.github.com/_blog/writing-exams-in-orgmode.org") (goto-char 18) (show-children))][link]]'
**Adding automatic MetaName 'date' found in file '[[elisp:(progn (find-file "/Users/jkitchin/blogofile-jkitchin.github.com/_blog/writing-exams-in-orgmode.org") (goto-char 18) (show-children))][link]]'
**Adding automatic MetaName 'updated' found in file '[[elisp:(progn (find-file "/Users/jkitchin/blogofile-jkitchin.github.com/_blog/writing-exams-in-orgmode.org") (goto-char 18) (show-children))][link]]'
**Adding automatic MetaName 'category' found in file '[[elisp:(progn (find-file "/Users/jkitchin/blogofile-jkitchin.github.com/_blog/writing-exams-in-orgmode.org") (goto-char 18) (show-children))][link]]'
**Adding automatic MetaName 'points' found in file '[[elisp:(progn (find-file "/Users/jkitchin/blogofile-jkitchin.github.com/_blog/writing-exams-in-orgmode.org") (goto-char 1391) (show-children))][link]]'
**Adding automatic MetaName 'tags' found in file '[[elisp:(progn (find-file "/Users/jkitchin/blogofile-jkitchin.github.com/_blog/why-org-mode.org") (goto-char 25) (show-children))][link]]'
**Adding automatic MetaName 'alltags' found in file '[[elisp:(progn (find-file "/Users/jkitchin/blogofile-jkitchin.github.com/_blog/why-org-mode.org") (goto-char 25) (show-children))][link]]'
**Adding automatic MetaName 'todo' found in file '[[elisp:(progn (find-file "/Users/jkitchin/blogofile-jkitchin.github.com/_blog/why-org-mode.org") (goto-char 1733) (show-children))][link]]'
**Adding automatic MetaName 'closed' found in file '[[elisp:(progn (find-file "/Users/jkitchin/blogofile-jkitchin.github.com/_blog/why-org-mode.org") (goto-char 1733) (show-children))][link]]'
**Adding automatic MetaName 'timestamp_ia' found in file '[[elisp:(progn (find-file "/Users/jkitchin/blogofile-jkitchin.github.com/_blog/pdfsync.org") (goto-char 28) (show-children))][link]]'
**Adding automatic MetaName 'id' found in file '[[elisp:(progn (find-file "/Users/jkitchin/blogofile-jkitchin.github.com/_blog/org-to-docx-pandoc.org") (goto-char 5056) (show-children))][link]]'
**Adding automatic MetaName 'custom_id' found in file '[[elisp:(progn (find-file "/Users/jkitchin/blogofile-jkitchin.github.com/_blog/org-db.org") (goto-char 1311) (show-children))][link]]'
**Adding automatic MetaName 'calculation' found in file '[[elisp:(progn (find-file "/Users/jkitchin/blogofile-jkitchin.github.com/_blog/org-db.org") (goto-char 1311) (show-children))][link]]'
**Adding automatic MetaName 'volume' found in file '[[elisp:(progn (find-file "/Users/jkitchin/blogofile-jkitchin.github.com/_blog/org-db.org") (goto-char 1311) (show-children))][link]]'
**Adding automatic MetaName 'total_energy' found in file '[[elisp:(progn (find-file "/Users/jkitchin/blogofile-jkitchin.github.com/_blog/org-db.org") (goto-char 1311) (show-children))][link]]'
**Adding automatic MetaName 'stress' found in file '[[elisp:(progn (find-file "/Users/jkitchin/blogofile-jkitchin.github.com/_blog/org-db.org") (goto-char 1311) (show-children))][link]]'
**Adding automatic MetaName 'priority' found in file '[[elisp:(progn (find-file "/Users/jkitchin/blogofile-jkitchin.github.com/_blog/blog.org") (goto-char 15327) (show-children))][link]]'
**Adding automatic MetaName 'export_title' found in file '[[elisp:(progn (find-file "/Users/jkitchin/blogofile-jkitchin.github.com/_blog/blog.org") (goto-char 506769) (show-children))][link]]'
**Adding automatic MetaName 'export_author' found in file '[[elisp:(progn (find-file "/Users/jkitchin/blogofile-jkitchin.github.com/_blog/blog.org") (goto-char 506769) (show-children))][link]]'
**Adding automatic MetaName 'export_file_name' found in file '[[elisp:(progn (find-file "/Users/jkitchin/blogofile-jkitchin.github.com/_blog/blog.org") (goto-char 506769) (show-children))][link]]'
**Adding automatic MetaName 'export_date' found in file '[[elisp:(progn (find-file "/Users/jkitchin/blogofile-jkitchin.github.com/_blog/blog.org") (goto-char 506769) (show-children))][link]]'
**Adding automatic MetaName 'scheduled' found in file '[[elisp:(progn (find-file "/Users/jkitchin/blogofile-jkitchin.github.com/_blog/blog.org") (goto-char 516502) (show-children))][link]]'
**Adding automatic MetaName 'deadline' found in file '[[elisp:(progn (find-file "/Users/jkitchin/blogofile-jkitchin.github.com/_blog/blog.org") (goto-char 516502) (show-children))][link]]'
**Adding automatic MetaName 'votes' found in file '[[elisp:(progn (find-file "/Users/jkitchin/blogofile-jkitchin.github.com/_blog/blog.org") (goto-char 532031) (show-children))][link]]'
**Adding automatic MetaName 'timestamp' found in file '[[elisp:(progn (find-file "/Users/jkitchin/blogofile-jkitchin.github.com/_blog/blog.org") (goto-char 571125) (show-children))][link]]'
**Adding automatic MetaName 'clock' found in file '[[elisp:(progn (find-file "/Users/jkitchin/blogofile-jkitchin.github.com/_blog/blog-2014.org") (goto-char 21059) (show-children))][link]]'
**Adding automatic MetaName 'level' found in file '[[elisp:(progn (find-file "/Users/jkitchin/blogofile-jkitchin.github.com/_blog/blog-2014.org") (goto-char 46582) (show-children))][link]]'
**Adding automatic MetaName 'correct' found in file '[[elisp:(progn (find-file "/Users/jkitchin/blogofile-jkitchin.github.com/_blog/blog-2014.org") (goto-char 46582) (show-children))][link]]'
**Adding automatic MetaName 'permalink' found in file '[[elisp:(progn (find-file "/Users/jkitchin/blogofile-jkitchin.github.com/_blog/blog-2014.org") (goto-char 61814) (show-children))][link]]'
**Adding automatic MetaName 'hint' found in file '[[elisp:(progn (find-file "/Users/jkitchin/blogofile-jkitchin.github.com/_blog/blog-2014.org") (goto-char 340534) (show-children))][link]]'
**Adding automatic MetaName 'answer' found in file '[[elisp:(progn (find-file "/Users/jkitchin/blogofile-jkitchin.github.com/_blog/blog-2014.org") (goto-char 355206) (show-children))][link]]'
**Adding automatic MetaName 'correct-answer' found in file '[[elisp:(progn (find-file "/Users/jkitchin/blogofile-jkitchin.github.com/_blog/blog-2014.org") (goto-char 377210) (show-children))][link]]'
**Adding automatic MetaName 'post_filename' found in file '[[elisp:(progn (find-file "/Users/jkitchin/blogofile-jkitchin.github.com/_blog/blog-2014.org") (goto-char 415454) (show-children))][link]]'
**Adding automatic MetaName 'ordered' found in file '[[elisp:(progn (find-file "/Users/jkitchin/blogofile-jkitchin.github.com/_blog/blog-2014.org") (goto-char 423900) (show-children))][link]]'
**Adding automatic MetaName 'grade' found in file '[[elisp:(progn (find-file "/Users/jkitchin/blogofile-jkitchin.github.com/_blog/add-subheadings-to-headings.org") (goto-char 2822) (show-children))][link]]'
**Adding automatic MetaName ':export_file_name:' found in file '[[elisp:(progn (find-file "/Users/jkitchin/blogofile-jkitchin.github.com/_blog/add-properties-to-headings.org") (goto-char 2) (show-children))][link]]'
**Adding automatic MetaName 'firstname' found in file '[[elisp:(progn (find-file "/Users/jkitchin/blogofile-jkitchin.github.com/_blog/org-contacts/referee-contacts.org") (goto-char 155) (show-children))][link]]'
**Adding automatic MetaName 'lastname' found in file '[[elisp:(progn (find-file "/Users/jkitchin/blogofile-jkitchin.github.com/_blog/org-contacts/referee-contacts.org") (goto-char 155) (show-children))][link]]'
**Adding automatic MetaName 'email' found in file '[[elisp:(progn (find-file "/Users/jkitchin/blogofile-jkitchin.github.com/_blog/org-contacts/referee-contacts.org") (goto-char 155) (show-children))][link]]'
**Adding automatic MetaName 'affiliation' found in file '[[elisp:(progn (find-file "/Users/jkitchin/blogofile-jkitchin.github.com/_blog/org-contacts/referee-contacts.org") (goto-char 155) (show-children))][link]]'
**Adding automatic MetaName 'lettergrade' found in file '[[elisp:(progn (find-file "/Users/jkitchin/blogofile-jkitchin.github.com/_blog/org-report/Slim-Shady-HW1.org") (goto-char 29) (show-children))][link]]'
**Adding automatic MetaName 'difficulty' found in file '[[elisp:(progn (find-file "/Users/jkitchin/blogofile-jkitchin.github.com/_blog/problem-selection/problem-selection.org") (goto-char 1) (show-children))][link]]'
Removing very common words...
no words removed.
Writing main index...
Sorting words ...
Sorting 6,044 words alphabetically
Writing header ...
Writing index entries ...
  Writing word text: Complete
  Writing word hash: Complete
  Writing word data: Complete
6,044 unique words indexed.
4 properties sorted.
5,084 files indexed.  1,760,249 total bytes.  368,569 total words.
Elapsed time: 00:00:37 CPU time: 00:00:01
Indexing done!

Ok, now for the proof in the approach!

swish-e -f index-org-headlines.swish-e -w headline=generating

1000 link "separate-bib.org") (goto-char 1) (show-children))][link]]" 393 1000 link "blog-2014.org") (goto-char 158456) (show-children))][link]]" 229 1000 link "blog-2014.org") (goto-char 272383) (show-children))][link]]" 400 1000 link "blog-2014.org") (goto-char 158456) (show-children))][link]]" 229 1000 link "blog.org") (goto-char 448965) (show-children))][link]]" 389 1000 link "org-db.org") (goto-char 575) (show-children))][link]]" 204 1000 link "org-db.org") (goto-char 575) (show-children))][link]]" 204 1000 link "separate-bib.org") (goto-char 1) (show-children))][link]]" 393 1000 link "blog-2014.org") (goto-char 272383) (show-children))][link]]" 400 .

swish-e -f index-org-headlines.swish-e -w todo=TODO

1000 link "blog.org") (goto-char 16933) (show-children))][link]]" 342 1000 link "blog-2014.org") (goto-char 61231) (show-children))][link]]" 207 1000 link "blog-2014.org") (goto-char 60802) (show-children))][link]]" 274 1000 link "blog-2014.org") (goto-char 60289) (show-children))][link]]" 207 1000 link "blog-2014.org") (goto-char 61568) (show-children))][link]]" 246 1000 link "blog-2014.org") (goto-char 61231) (show-children))][link]]" 207 1000 link "blog-2014.org") (goto-char 60802) (show-children))][link]]" 274 1000 link "blog-2014.org") (goto-char 60289) (show-children))][link]]" 207 1000 link "blog.org") (goto-char 632875) (show-children))][link]]" 266 1000 link "blog.org") (goto-char 529123) (show-children))][link]]" 202 1000 link "blog.org") (goto-char 529087) (show-children))][link]]" 206 1000 link "blog.org") (goto-char 518108) (show-children))][link]]" 280 1000 link "blog.org") (goto-char 30559) (show-children))][link]]" 337 1000 link "blog-2014.org") (goto-char 61568) (show-children))][link]]" 246 .

1 Summary thoughts

This could be super useful for a lot of different elements: headlines, src-blocks, links, tables, paragraphs are the main ones that come to mind. You could have pretty focused searches that go straight to the matches!

Copyright (C) 2015 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.10

Discuss on Twitter

Integrating swish-e and Emacs

| categories: orgmode, emacs | tags:

swish-e is a software package that indexes files on your computer, and then allows you to search the index. Spotlight on my Mac is not working too well (sometimes not at all), and I want some more flexibility so today we try getting swish-e up and running and integrated with Emacs. I don't know that swish-e is the best tool for this available, but it has been on my radar a long time (probably since 2003 from this article ), and it was easy to setup and use.

I use homebrew, so installation was this simple:

brew install swish-e

To test things out, I will only index org-files. I have these all over the place, and they are not all in my org-mode agenda. So, finding them quickly would be awesome.

# Example configuration file

# Tell Swish-e what to directories to index
IndexDir /Users/jkitchin/Dropbox
IndexDir "/Users/jkitchin/Box Sync"
IndexDir /Users/jkitchin/blogofile-jkitchin.github.com

# where to save the index
IndexFile /Users/jkitchin/.swish-e/index.swish-e

# What to index
IndexOnly .org

# Tell Swish-e that .txt files are to use the text parser.
IndexContents TXT* .org

# Otherwise, use the HTML parser
DefaultContents HTML*

# Ask libxml2 to report any parsing errors and warnings or
# any UTF-8 to 8859-1 conversion errors
ParserWarnLevel 9

Now, we create our index.

swish-e -c ~/.swish-e/swish.conf
Indexing Data Source: "File-System"
Indexing "/Users/jkitchin/Dropbox"
Indexing "/Users/jkitchin/Box Sync"
Indexing "/Users/jkitchin/blogofile-jkitchin.github.com"
Removing very common words...
no words removed.
Writing main index...
Sorting words ...
Sorting 130,109 words alphabetically
Writing header ...
Writing index entries ...
  Writing word text: ...
  Writing word text:  10%
  Writing word text:  20%
  Writing word text:  30%
  Writing word text:  40%
  Writing word text:  50%
  Writing word text:  60%
  Writing word text:  70%
  Writing word text:  80%
  Writing word text:  90%
  Writing word text: 100%
  Writing word text: Complete
  Writing word hash: ...
  Writing word hash:  10%
  Writing word hash:  20%
  Writing word hash:  30%
  Writing word hash:  40%
  Writing word hash:  50%
  Writing word hash:  60%
  Writing word hash:  70%
  Writing word hash:  80%
  Writing word hash:  90%
  Writing word hash: 100%
  Writing word hash: Complete
  Writing word data: ...
  Writing word data:   9%
  Writing word data:  19%
  Writing word data:  29%
  Writing word data:  39%
  Writing word data:  49%
  Writing word data:  59%
  Writing word data:  69%
  Writing word data:  79%
  Writing word data:  89%
  Writing word data:  99%
  Writing word data: Complete
130,109 unique words indexed.
Sorting property: swishdocpath                            
Sorting property: swishtitle                              
Sorting property: swishdocsize                            
Sorting property: swishlastmodified                       
4 properties sorted.
3,208 files indexed.  54,104,974 total bytes.  8,038,594 total words.
Elapsed time: 00:00:16 CPU time: 00:00:13
Indexing done!

Now an example search. I have been looking into the Energy frontier research centers, and I want to find my notes on it. Here is a little query. I use a special output format to keep things simple for the parsing later, just the rank and path, separated by a tab.

swish-e -f ~/.swish-e/index.swish-e -x '%r\t%p\n' -w efrc
# SWISH format: 2.4.7
# Search words: efrc
# Removed stopwords:
# Number of hits: 2
# Search time: 0.000 seconds
# Run time: 0.008 seconds
1000	/Users/jkitchin/Dropbox/org-mode/journal.org
471	/Users/jkitchin/Dropbox/org-mode/proposals.org
.

Now, for the integration with Emacs. We just get that output in a string, split it, and get the parts we want. I think I will use helm to provide a selection buffer to these results. We need a list of cons cells (string . candidate). Then we write an interactive helm function. We provide two sources. One for the initial query, and another to start a new search, in case you don't find what you want.

(defun helm-swish-e-candidates (query)
  "Generate a list of cons cells (swish-e result . path)."
  (let* ((result (shell-command-to-string
                  (format "swish-e -f ~/.swish-e/index.swish-e -x \"%%r\t%%p\n\" -w %s"
                          (shell-quote-argument query))))
         (lines (s-split "\n" result t))
         (candidates '()))
    (loop for line in lines
          unless (or  (s-starts-with? "#" line)
                      (s-starts-with? "." line))
          collect (cons line (cdr (s-split "\t" line))))))


(defun helm-swish-e (query)
  "Run a swish-e query and provide helm selection buffer of the results."
  (interactive "sQuery: ")
  (helm :sources `(((name . ,(format "swish-e: %s" query))
                    (candidates . ,(helm-swish-e-candidates query))
                    (action . (("open" . (lambda (f)
                                           (find-file (car f)))))))
                   ((name . "New search")
                    (dummy)
                    (action . (("search" . (lambda (f)
                                             (helm-swish-e helm-pattern)))))))))
helm-swish-e

Now I can run M-x helm-swish-e and enter "efrc AND computing infrastructure" to find org files containing those words, then press enter to find the file. Nice and easy. I have not tested the query syntax very fully, but so far it is working fine!

Copyright (C) 2015 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.10

Discuss on Twitter

Clickable org-contacts in text files

| categories: orgmode, contacts | tags:

Continuing my adventures with clickable text (See clickable email addresses and clickable twitter handles ), here we consider how to get clickable names that are also in my org-contacts database. The goal is to have these names highlighted and clickable so that when I click on them I get a hydra menu of actions, e.g. to open the contact, email them, etc… We will again use button-lock to do the action. We will construct a fairly large regexp to match all the names in the org-contacts database. This turns out to be very easy using the regexp-opt function.

First, I formalize the code I used last time to get text around the point that has a text-property. We will use that to get the text that has been highlighted by button-lock.

(defun get-surrounding-text-with-property (property)
  "Return text surrounding point with the text-property PROPERTY."
  (let ((start) (end))
    (when (get-text-property (point) property)
      (save-excursion
        (while (get-text-property (point) property)
          (backward-char))
        (forward-char)
        (setq start (point))
        (while (get-text-property (point) property)
          (forward-char))
        (setq end (point)))
      (buffer-substring start end))))
get-surrounding-text-with-property

I want to use nicknames that are defined in my org-contacts database. We first try to return an assoc lookup, then the slower approach of looping through the entries to find a matching nickname.

(defun get-contact-from-name-or-nickname (name-or-nickname)
  "Return a contact from the org-contacts database for NAME-OR-NICKNAME."
  (or
   (assoc name-or-nickname (org-contacts-db))
   ;; no assoc, so now we have to find a nickname
   (catch 'contact
     (dolist (contact (org-contacts-db))
       (when (-contains? (s-split "," (or (cdr (assoc "NICKNAMES" (caddr contact))) " ")) name-or-nickname)
         (throw 'contact contact))))))
get-contact-from-name-or-nickname

Now, let us write a hydra function that will be our menu of actions. For some reason, when you click on a highlighted text the mouse moves to the end of the text, so in our hydra function we move back a char, and then get the info. Basically, we get the name, then get the contact, and extract what we need from there. Here we provide functionality to open a contact, email a contact or open the url of the contact (if it exists). I also want a conditional hydra, which doesn't seem to be an option yet, so we we roll our own here. Basically, we construct the code for a hydra, and only add a menu option to open the url if we find one in the contact. We will have to eval the code returned from this function to get the hydra body, and then call the body function in the action function for the highlighted text.

(defun conditional-hydra-actions ()
  "Construct code to create a hydra with conditional options."
  (let ((code  '(defhydra org-contacts (:color blue)
                  "Org contacts")))
    (setq code
          (append
           code
           '(("o" (progn
                    (backward-char)
                    (let* ((name (get-surrounding-text-with-property 'org-contact))
                           (contact (get-contact-from-name-or-nickname name))
                           (contact-marker (nth 1 contact)))
                      (switch-to-buffer (marker-buffer contact-marker))
                      (goto-char (marker-position contact-marker))
                      (show-subtree)))
              "Open contact"))))

    (setq code
          (append
           code '(("e" (progn
                         (backward-char)
                         (let* ((name (get-surrounding-text-with-property 'org-contact))
                                (contact (get-contact-from-name-or-nickname name))
                                (email (cdr (assoc "EMAIL" (caddr contact))))))
                         (mu4e~compose-mail email))
                   "Email contact"))))

    ;; conditional menu for opening a URL
    (let* ((name (get-surrounding-text-with-property 'org-contact))
           (contact (assoc name (org-contacts-db)))
           (url (cdr (assoc "URL" (caddr contact)))))
      (when url
        (setq code
              (append
               code '(("w" (progn
                             (backward-char)
                             (let* ((name (get-surrounding-text-with-property 'org-contact))
                                    (contact (get-contact-from-name-or-nickname name))
                                    (url (cdr (assoc "URL" (caddr contact)))))
                               (if url
                                   (browse-url url)
                                 (message "No url found."))))
                       "Open in browser"))))))
    code))
conditional-hydra-actions

I also want to have nicknames in this list, because sometimes I don't use the full names in my contact database. These are stored in a comma-separated property called NICKNAMES in entries that have them. A subtle point here is that it complicates looking up the contact in the database. Normally, I can get this by a simple assoc lookup. For the nicknames, that will fail, so we need a back up method. Now, the highlighting code. You can make the regexp by passing a list of strings to match to regexp-opt. We get our list of strings from:

(append
 (mapcar 'car (org-contacts-db))
 (let ((nicknames '()))
   (dolist (contact (org-contacts-db))
     (when (assoc "NICKNAMES" (caddr contact))
       (setq nicknames
             (append nicknames (s-split "," (cdr (assoc "NICKNAMES" (caddr contact))))))))
   nicknames))

I am not going to show them here to protect my contacts ;). Now, we create the function that highlights the contacts. and add it as a hook function to text-mode-hook.

(defun highlight-org-contacts ()
  (button-lock-set-button
   (regexp-opt
    (append
     (mapcar 'car (org-contacts-db))
     (let ((nicknames '()))
       (dolist (contact (org-contacts-db))
         (when (assoc "NICKNAMES" (caddr contact))
           (setq nicknames
                 (append
                  nicknames
                  (s-split "," (cdr (assoc "NICKNAMES" (caddr contact))))))))
       nicknames)))
   (lambda ()
     (interactive)
     (eval (conditional-hydra-actions))
     (org-contacts/body))
   :face '((:background "MistyRose1")
           (:underline t))
   :help-echo (format "An org contact")
   :keyboard-binding (kbd "RET")
   :additional-property 'org-contact))

(add-hook 'text-mode-hook 'highlight-org-contacts)

That does it. Now, whenever I open a text-based file, the names that are in my contacts are highlighted and actionable. This should be useful for meeting notes, etc…

Copyright (C) 2015 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.10

Discuss on Twitter
« Previous Page -- Next Page »