The Kitchin Research Group

One use for clickable text is in educational texts, or technical documents where you want easy access to glossaries for jargon or new words, or other context specific information. Here we consider some approaches to highlight words in an Emacs buffer that are defined in a glossary, to give them tooltips and make them clickable.

You may want to see the video of this in action, the blog post does not do it justice: http://www.youtube.com/watch?v=Ogavyl_QXiU

We assume we have a 1 in the current document that has the words we want to highlight as headlines. Here is a somewhat hacky way to get the list of keywords (hacky because we use cdr to get rid of the Glossary in the list). Our glossary only has two terms: INCAR and KPOINTS.

(save-excursion
    (org-open-link-from-string "[[*Glossary]]")
    (cdr  (org-map-entries (lambda ()
                             (nth 4 (org-heading-components)))
                           nil 'tree)))

We can use that list to make the regexp for button lock with regexp-opt like we did before. We illustrate two ideas here for the highlighted text. One is a dynamic tooltip, which we calculate on the fly and use to display the contents of the glossary heading when you mouse over the word or call local help from the keyboard (C-h .). Second, when you click on the word, you jump to the section in the glossary, and you can readily jump back with C-c & (Thanks org-mode!).

(defun highlight-glossary-words ()
  (button-lock-set-button
   (regexp-opt (save-excursion
                 (org-open-link-from-string "[[*Glossary]]")
                 (cdr  (org-map-entries
                        (lambda ()
                          (nth 4 (org-heading-components)))
                        nil 'tree))))
   (lambda ()
     "Jump to definition."
     (interactive)
     (let ((keyword (get-surrounding-text-with-property 'glossary)))
       (org-open-link-from-string (format "[[*%s]]" keyword))))
   :additional-property 'glossary
   :face '((:background "gray80") (:underline t))
   :help-echo (lambda (window object position)
                (save-excursion
                  (goto-char position)
                  (save-restriction
                    (org-open-link-from-string
                     (format "[[*%s]]" (get-surrounding-text-with-property 'glossary)))
                    (org-narrow-to-subtree)
                    (buffer-string))))))

(highlight-glossary-words)

That is pretty cool. You might want something a little smarter for the tooltip, e.g. just the first line of the headline, but this works fine for this little example. I noticed that flyspell seems to get the tooltip in KPOINTS, sometimes, when it thinks it is misspelled.

It might take some local variables to make this work only in this just a file, rather than in every file. Alternatively, you could define a function that opens the file and then applies this.

Copyright (C) 2015 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.10

swish-e is a software package that indexes files on your computer, and then allows you to search the index. Spotlight on my Mac is not working too well (sometimes not at all), and I want some more flexibility so today we try getting swish-e up and running and integrated with Emacs. I don't know that swish-e is the best tool for this available, but it has been on my radar a long time (probably since 2003 from this article ), and it was easy to setup and use.

I use homebrew, so installation was this simple:

brew install swish-e

To test things out, I will only index org-files. I have these all over the place, and they are not all in my org-mode agenda. So, finding them quickly would be awesome.

# Example configuration file

# Tell Swish-e what to directories to index
IndexDir /Users/jkitchin/Dropbox
IndexDir "/Users/jkitchin/Box Sync"
IndexDir /Users/jkitchin/blogofile-jkitchin.github.com

# where to save the index
IndexFile /Users/jkitchin/.swish-e/index.swish-e

# What to index
IndexOnly .org

# Tell Swish-e that .txt files are to use the text parser.
IndexContents TXT* .org

# Otherwise, use the HTML parser
DefaultContents HTML*

# Ask libxml2 to report any parsing errors and warnings or
# any UTF-8 to 8859-1 conversion errors
ParserWarnLevel 9

Now, we create our index.

swish-e -c ~/.swish-e/swish.conf

Indexing Data Source: "File-System"
Indexing "/Users/jkitchin/Dropbox"
Indexing "/Users/jkitchin/Box Sync"
Indexing "/Users/jkitchin/blogofile-jkitchin.github.com"
Removing very common words...
no words removed.
Writing main index...
Sorting words ...
Sorting 130,109 words alphabetically
Writing header ...
Writing index entries ...
  Writing word text: ...
  Writing word text:  10%
  Writing word text:  20%
  Writing word text:  30%
  Writing word text:  40%
  Writing word text:  50%
  Writing word text:  60%
  Writing word text:  70%
  Writing word text:  80%
  Writing word text:  90%
  Writing word text: 100%
  Writing word text: Complete
  Writing word hash: ...
  Writing word hash:  10%
  Writing word hash:  20%
  Writing word hash:  30%
  Writing word hash:  40%
  Writing word hash:  50%
  Writing word hash:  60%
  Writing word hash:  70%
  Writing word hash:  80%
  Writing word hash:  90%
  Writing word hash: 100%
  Writing word hash: Complete
  Writing word data: ...
  Writing word data:   9%
  Writing word data:  19%
  Writing word data:  29%
  Writing word data:  39%
  Writing word data:  49%
  Writing word data:  59%
  Writing word data:  69%
  Writing word data:  79%
  Writing word data:  89%
  Writing word data:  99%
  Writing word data: Complete
130,109 unique words indexed.
Sorting property: swishdocpath                            
Sorting property: swishtitle                              
Sorting property: swishdocsize                            
Sorting property: swishlastmodified                       
4 properties sorted.
3,208 files indexed.  54,104,974 total bytes.  8,038,594 total words.
Elapsed time: 00:00:16 CPU time: 00:00:13
Indexing done!

Now an example search. I have been looking into the Energy frontier research centers, and I want to find my notes on it. Here is a little query. I use a special output format to keep things simple for the parsing later, just the rank and path, separated by a tab.

swish-e -f ~/.swish-e/index.swish-e -x '%r\t%p\n' -w efrc

# SWISH format: 2.4.7
# Search words: efrc
# Removed stopwords:
# Number of hits: 2
# Search time: 0.000 seconds
# Run time: 0.008 seconds
1000	/Users/jkitchin/Dropbox/org-mode/journal.org
471	/Users/jkitchin/Dropbox/org-mode/proposals.org
.

Now, for the integration with Emacs. We just get that output in a string, split it, and get the parts we want. I think I will use helm to provide a selection buffer to these results. We need a list of cons cells (string . candidate). Then we write an interactive helm function. We provide two sources. One for the initial query, and another to start a new search, in case you don't find what you want.

(defun helm-swish-e-candidates (query)
  "Generate a list of cons cells (swish-e result . path)."
  (let* ((result (shell-command-to-string
                  (format "swish-e -f ~/.swish-e/index.swish-e -x \"%%r\t%%p\n\" -w %s"
                          (shell-quote-argument query))))
         (lines (s-split "\n" result t))
         (candidates '()))
    (loop for line in lines
          unless (or  (s-starts-with? "#" line)
                      (s-starts-with? "." line))
          collect (cons line (cdr (s-split "\t" line))))))


(defun helm-swish-e (query)
  "Run a swish-e query and provide helm selection buffer of the results."
  (interactive "sQuery: ")
  (helm :sources `(((name . ,(format "swish-e: %s" query))
                    (candidates . ,(helm-swish-e-candidates query))
                    (action . (("open" . (lambda (f)
                                           (find-file (car f)))))))
                   ((name . "New search")
                    (dummy)
                    (action . (("search" . (lambda (f)
                                             (helm-swish-e helm-pattern)))))))))

helm-swish-e

Now I can run M-x helm-swish-e and enter "efrc AND computing infrastructure" to find org files containing those words, then press enter to find the file. Nice and easy. I have not tested the query syntax very fully, but so far it is working fine!

Copyright (C) 2015 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.10

Usually the hydra menu is hard coded in the defhydra macro. Sometimes, you would like conditional options, that is, depending on some condition we get different options when we run the hydra and not when it was defined. This is an open issue in hydra. Here we explore a way to achieve that. The idea is to construct the code for the hydra, then eval it, and run the hydra. In this example we make the conditional menu depend on whether we are on an even or odd numbered line. I use the `' syntax for defining the list of code. ` is a variation of ' (quote) that enables you to use the , operator to evaluate that element while in data mode. So, here is our first idea:

(defun my-hydra-1 ()
  (interactive)
  (eval
   `(defhydra my-hydra-1 (:color blue) "My hydra"
      ,(if (evenp (line-number-at-pos))
           '("e" (message-box "Even line") "Even")
         '("o" (message-box "Odd line") "Odd"))
      ,(when t '("a" (message-box "always true") "always"))
      ;; This does not work. you must return a legitimate hydra menu item
      ;;      ,(when nil '("n" (message-box "never") "never"))
      ))
  (my-hydra-1/body))

(my-hydra-1)
(my-hydra-1)

my-hydra

As long as it is not expensive to compute the conditionals, this seems like an easy enough way to get conditional options in a hydra. One limitation of the previous approach is our menu conditionals must return a hydra menu, and not nil. Here is an alternative approach to writing the function that solves the issue of the nil return in the last function. Here we build up the code list using append. It might seem like a macro should be used here, but I have not figured out how to get the macro to run the conditionals at the run-time. Note, we cannot use funcall on the defhydra because that is a macro.

(defun my-hydra-2 ()
  (interactive)
  (let ((conditionals '((if (evenp (line-number-at-pos))
                            '("e" (message-box "Even second") "Even")
                          '("o" (message-box "Odd second") "Odd"))
                        (when t '("a" (message-box "always true") "always"))
                        (when nil '("n" (message-box "never") "never")))))
    (eval
     (append
      '(defhydra my-hydra-2 (:color blue) "My hydra")
      (loop for cond in conditionals
            with result = (eval cond)
            if (eval cond)
            collect (eval cond))))
    (my-hydra-2/body)))

(my-hydra-2)
(my-hydra-2)

That works too. Let us try another type of syntax where the conditional statements have a cons cell with a conditional statement, and a hydra menu option for when the statement is true. This is functionally similar to our first method, but has some advantages in brevity and less quoting. We add a conditional hint here too (at some expense of additional quoting).

(defun my-hydra-3 ()
  (interactive)
  (let ((conditionals
         `(((evenp (line-number-at-pos)) . ("e" (message-box "Even second") ,(format "Even: %s" (line-number-at-pos))))
           ((oddp (line-number-at-pos)) . ("o" (message-box "Odd second") ,(format "Odd: %s" (line-number-at-pos))))
           (t . ("a" (message-box "always true") "always"))
           (nil . ("n" (message-box "never") "never")))))
    (eval
     (append
      '(defhydra my-hydra-3 (:color blue) "My hydra")
      (loop for cond in conditionals
            if (eval (car  cond))
            collect (cdr cond))))
    (my-hydra-3/body)))

(my-hydra-3)
(my-hydra-3)

I cannot figure out how to abstract this much further. There is a little redundancy in names, e.g. in the defhydra and at the end, but it is not too bad, which would usually be handled by a macro. I tried some defmacros to try this, but I could not figure out how to get the conditionals to expand at the right times, which is at run time, and not at macro expansion time. I need a macro that generates a function that has the call to defhydra in it! Maybe next year ;)

Copyright (C) 2015 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.10

There are clickable mailto:jkitchin@andrew.cmu.edu links in org-mode, but standalone email addresses like jkitchin@cmu.edu are just ordinary text. Here, I want to explore using clickable text instead. I will use the button-lock package for this. I borrowed an email regexp from EmacsWiki: Regular Expression for this. The idea is to define a regular expression for an email address, and use it to make the emails clickable.

I am still not sure what the canonical way to get the value of the text that was highlighted. Here we use the :additional-property feature to set a property to t, and then use that property to get the characters that have a non-nil "email-address" property. It seems clunky, but it works. The main action is to compose an email in mu4e (my preferred email program in emacs). You could also put a call to helm here, or to a hydra for other options.

I make the email addresses stand out a little by giving them a gray background, and a tooltip so you can see why they are highlighted. I also bind RET so I don't have to use the mouse. Don't forget you can type C-h . to see the local help instead of mousing over it! Finally, we add a text-mode hook so this will get loaded when we open a text file (or one with a mode derived from text-mode like org-mode).

(defun highlight-email-addresses ()
  "Add button to email address. Clicking or RET will open a compose email window."
  (button-lock-set-button
   "\\w+\\(\\.\\w+\\)?@\\(\\w\\|\\.\\)+"
   (lambda ()
     (interactive)
     (let ((start) (end) (email-address))
       (while (get-text-property (point) 'email-address)
         (backward-char))
       (forward-char)
       (setq start (point))
       (while (get-text-property (point) 'email-address)
         (forward-char))
       (setq end (point))
       (setq email-address (buffer-substring start end))
       (mu4e~compose-mail email-address)))
     :face '((:background "gray80") (:underline t))
     :help-echo "click to send mu4e email"
     :keyboard-binding (kbd "RET")
     :additional-property 'email-address))

(add-hook 'text-mode-hook 'highlight-email-addresses)

That doesn't look too bad. Now, anytime I open an org-mode file with an email address in it, the address is highlighted in light gray, and underlined. I can click on it or put the cursor on it and press return and I get a compose email window open, with the email address pre-filled in! I am sure this will have some other applications.

Copyright (C) 2015 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.10

I have an idea in mind that would use the output of python scripts in lisp functions. Xah Lee posted an idea for writing emacs commands in scripting languages . In this post I want to explore an extension of the idea, where a Python script will return output that can be read in Lisp, e.g. we can convert a Python list to a lisp list, or a dictionary to an a-list or p-list. I can already see that simple data structures will be "simple", and arbitrary data structures will offer a lot of challenges, e.g. nested lists or dictionaries…

If I could add some custom functions to the basic builtin types in Python, then I could use another approach to format python objects as lisp data types. This isn't recommended by Pythonistas, but I guess they don't want to use lisp as much as I do ;) I found this approach to modifying builtins:

http://stackoverflow.com/questions/2444680/how-do-i-add-my-own-custom-attributes-to-existing-built-in-python-types-like-a

We use that almost verbatim here to get what I want. This is a super low level way to add functions to the builtins. I add some simple formatting to floats, ints and strings. I add a more complex recursive formatting function to lists, tuples and dictionaries. A dictionary can be represented as an alist or plist. Both examples are shown, but I leave the alist version commented out. Finally, we add a lispify function to numpy arrays.

import ctypes as c

class PyObject_HEAD(c.Structure):
    _fields_ = [('HEAD', c.c_ubyte * (object.__basicsize__ -
                                      c.sizeof(c.c_void_p))),
                ('ob_type', c.c_void_p)]

_get_dict = c.pythonapi._PyObject_GetDictPtr
_get_dict.restype = c.POINTER(c.py_object)
_get_dict.argtypes = [c.py_object]

def get_dict(object):
    return _get_dict(object).contents.value

get_dict(str)['lisp'] = lambda s:'"{}"'.format(str(s))
get_dict(float)['lisp'] = lambda f:'{}'.format(str(f))
get_dict(int)['lisp'] = lambda f:'{}'.format(str(f))

import collections
import numpy as np

def lispify(L):
    "Convert a Python object L to a lisp representation."
    if (isinstance(L, str)
        or isinstance(L, float)
        or isinstance(L, int)):
        return L.lisp()
    elif (isinstance(L, list)
          or isinstance(L, tuple)
          or isinstance(L, np.ndarray)):
        s = []
        for element in L:
            s += [element.lisp()]
        return '(' + ' '.join(s) + ')'
    elif isinstance(L, dict):
        s = []
        for key in L:
            # alist format
            # s += ["({0} . {1})".format(key, L[key].lisp())]
            # plist
            s += [":{0} {1}".format(key, L[key].lisp())]
        return '(' + ' '.join(s) + ')'

get_dict(list)['lisp'] = lispify
get_dict(tuple)['lisp'] = lispify
get_dict(dict)['lisp'] = lispify
get_dict(np.ndarray)['lisp'] = lispify

Let us test these out.

from pylisp import *
a = 4.5
print int(a).lisp()
print a.lisp()
print "test".lisp()

print [1, 2, 3].lisp()
print (1, 2, 3).lisp()

print [[1, 3], (5, 6)].lisp()

print {"a": 5}.lisp()
print [[1, 3], (5, 6), {"a": 5, "b": "test"}].lisp()


A = np.array([1, 3, 4])
print A.lisp()
print ({"tree": [5, 6]}, ["a", 4, "list"], 5, 2.0 / 3.0).lisp()

4
4.5
"test"
(1 2 3)
(1 2 3)
((1 3) (5 6))
(:a 5)
((1 3) (5 6) (:a 5 :b "test"))
(1 3 4)
((:tree (5 6)) ("a" 4 "list") 5 0.666666666667)

Now, is that better than a single lisp function with a lot of conditionals to handle each type? I am not sure. This seems to work pretty well.

Here is how I imagine using this idea. We would have some emacs-lisp variables and use them to dynamically generate a python script. We run the python script, capturing the output, and read it back in as a lisp data structure. Here is a simple kind of example that generates a dictionary.

(let* ((elisp-var 6)
       (result)
      (script (format "
from pylisp import *
print {x: [2*y for y in range(x)] for x in range(1, %s)}.lisp()
" elisp-var)))

  ;; start a python process
  (run-python)
  (setq result (read (python-shell-send-string-no-output
   script)))
  (plist-get result :5))

(0 2 4 6 8)

That seems to work pretty well. One alternative idea to this is Pymacs , which I have written about before . This project isn't currently under active development, and I ran into some difficulties with it before.

Here we can solve the problem I previously posed and get the result back as an elisp float, and then reuse the result

(let* ((myvar 3)
       (script (format "from pylisp import *
from scipy.optimize import fsolve
def objective(x):
    return x - 5

ans, = fsolve(objective, %s)
print ans.lisp()" myvar)))
  (run-python)
  (setq result (read (python-shell-send-string-no-output
                       script)))
  (- 5 result))

0.0

Bottom line: we can write python code in lisp functions that are dynamically updated, execute them, and get lisp data structures back for simple data types. I think that could be useful in some applications, where it is easier to do parsing/analysis in Python, but you want to do something else that is easier in Lisp.

Copyright (C) 2015 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.10