Calling remote code-blocks in org-mode

| categories: orgmode, interactive, plotting | tags:

Table of Contents

We often write code in supporting information files that generates figures or tables in a scientific manuscript. Today, we explore how to call those code blocks remotely but get the output in the file we call it from. We will write code in si.org that generates an interactive figure that is presented in this file. We will use data published in hallenbeck-2015-compar-co2. You can find the data we used in the SI for that paper, or more conveniently here .

So, we make a named code block in the si.org file called "figure-1". Then we call it like this:

#+call: si.org:figure-1() :wrap html

That executes the code block in the other file, and wraps the output in an HTML block in this file! I do not like my code blocks to execute when I export because they are usually expensive calculations, so I have to manually run the line with C-c C-c, but you can override that behavior with a local setting of org-export-babel-evaluate. So, without further delay, here is the result. Now we have a nice, neat blog post file, with code in an si.org file!

Copyright (C) 2016 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.10

Discuss on Twitter

Interactive figures in blog posts with mpld3

| categories: plotting, interactive, python | tags:

Continuing the exploration of interactive figures, today we consider the Python plotting library mpld3 . We will again use our own published data. We wrote this great paper on core level shifts (CLS) in Cu-Pd alloys boes-2015-core-cu. I want an interactive figure that shows the name of the calculation on each point as a tooltip. This data is all stored in the supporting information file, and you can see how we use it here. This figure shows how the core level shift of a Cu atom changes depending on the number of nearest neighbor Cu atoms. Just hover your mouse over a point to see the name and CLS for that point.

1 Data and code

You can check out our preprint at https://github.com/KitchinHUB/kitchingroup-51 . We are going to adapt the code to make Figure 6a in the manuscript interactive. The code needed a somewhat surprising amount of adaptation. Apparently the ase database interface has changed a lot since we write that paper, so the code here looks a bit different than what we published. The biggest difference is due to name-mangling so each key that started with a number now starts with _, and and periods are replaced by _ also. The rest of the script is nearly unchanged. At the end is the very small bit of mpld3 code that generates the figure for html. We will add tooltips onto datapoints to indicate what the name associated with each data point is. Here is the code.

import matplotlib.pyplot as plt
from ase.db import connect

# loads the ASE database and select certain keywords
db = connect('~/Desktop/cappa/kitchingroup-51/supporting-information/data.json')

keys = ['bcc', 'GS', '_54atom', 'ensam']

CLS, IMP, labels = [], [], []
for k in db.select(keys + ['_1cl']):
    name = k.keywords[-2]

    Cu0 = db.select('bcc,GS,_72atom,_0cl,_1_00Cu').next().energy
    Cu1 = db.select('bcc,GS,_72atom,_1cl,_1_00Cu').next().energy
    x0 = db.select(','.join(keys + [name, '_0cl'])).next().energy
    x1 = k.energy

    cls0 = x0 - Cu0
    cls1 = x1 - Cu1

    IMP.append(int(name[1]))
    CLS.append(cls1 - cls0)
    labels += ['{0} ({1}, {2})'.format(name, int(name[1]), cls1 - cls0)]

Cu0 = db.select(','.join(['bcc', 'GS', '_72atom',
                          '_0cl', '_1_00Cu'])).next().energy
Cu1 = db.select(','.join(['bcc', 'GS', '_72atom',
                          '_1cl', '_1_00Cu'])).next().energy

x0 = db.select(','.join(['bcc', 'GS', '_54atom',
                         '_0cl', '_1'])).next().energy
x1 = db.select(','.join(['bcc', 'GS', '_54atom',
                         '_1cl', '_1'])).next().energy

cls0 = x0 - Cu0
cls1 = x1 - Cu1

IMP.append(1)
CLS.append(cls1 - cls0)
labels += ['(1, {0})'.format(cls1 - cls0)]

Cu0 = db.select(','.join(['bcc', 'GS', '_72atom',
                          '_0cl', '_1_00Cu'])).next().energy
Cu1 = db.select(','.join(['bcc', 'GS', '_72atom',
                          '_1cl', '_1_00Cu'])).next().energy

x0 = db.select(','.join(['bcc', 'GS', '_54atom',
                         '_0cl', '_0'])).next().energy
x1 = db.select(','.join(['bcc', 'GS', '_54atom',
                         '_1cl', '_0'])).next().energy

cls0 = x0 - Cu0
cls1 = x1 - Cu1

IMP.append(0)
CLS.append(cls1 - cls0)
labels += ['(0, {0})'.format(cls1 - cls0)]

fig = plt.figure()

p = plt.scatter(IMP, CLS, c='g', marker='o', s=25)
ax1 = plt.gca()
ax1.set_ylim(-1.15, -0.6)
ax1.set_xlim(-0.1, 5.1)

ax1.set_xlabel('# Cu Nearest neighbors')
ax1.set_ylabel('Cu 2p(3/2) Core Level Shift (eV)')

ax1.set_title('Hover over a point to see the calculation name')

# Now the mpld3 stuff.
import mpld3
from mpld3 import plugins

tooltip = plugins.PointHTMLTooltip(p, labels, voffset=0, hoffset=10)
plugins.connect(fig, tooltip)

print mpld3.fig_to_html(fig)

I like this workflow pretty well. It seems less functional than plotly and Bokeh (e.g. it does not look like it you can export the data from the html here), but it is well integrated with Matplotlib, with my blogging style, and does not require a server, oran account. The code outputs html that is self-contained in the body of the html. The smooth integration with Matplotlib means I could have static images in org-mode, and dynamic images in HTML potentially. Overall, this is a nice tool for making interactive plots in blog posts.

Copyright (C) 2016 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.10

Discuss on Twitter

Interactive Bokeh plots in HTML

| categories: plotting, interactive, python | tags:

In our last post we examined the use of plotly to generate interactive plots in HTML. Today we expand the idea, and use Bokeh . One potential issue with plotly is the need for an account and API-key, some limitations on how many times a graph can be viewed per day (although I should aspire to have my graphs viewed 1000+ times a day!), and who knows what happens to the graphs if plotly ever goes out of business. While the static images we usually use have limited utility, at least they stick around.

So, today we look at Bokeh which allows you to embed some json data in your HTML, which is made interactive by your browser with more javascript magic. We get straight to the image here so you can see what this is all about. Briefly, this data shows trends (or lack of) in the adsorption energies of some atoms on the atop and fcc sites of several transition metals as a function of adsorbate coverage xu-2014-probin-cover. The code to do this is found here.

Using Bokeh does not integrate real smoothly with my blog workflow, which only generates the body of HTML posts. Bokeh needs some javascript injected into the header to work. To get around that, I show the plot in a frame here. You can see a full HTML version here: bokeh-plot.html .

This is somewhat similar to the plotly concept. The data is embedded in the html in this case, which is different. For very large plots, I actually had some trouble exporting the blog post (it was taking a long time to export and I killed it)! I suspect that is a limitation of the org-mode exporter though, because I could save the html files from Python and view them fine. I also noted that having all the javascript in the org-file make font-lock work very slow. It would be better to generate that only on export.

Note to make this work, we need these lines in our HTML header:

#+HTML_HEAD: <link rel="stylesheet" href="http://cdn.pydata.org/bokeh/release/bokeh-0.11.1.min.css" type="text/css" />
#+HTML_HEAD: <script type="text/javascript" src="http://cdn.pydata.org/bokeh/release/bokeh-0.11.1.min.js"></script>

Since we do not host those locally, if they ever disappear, our plots will not show ;(

1 The data and code

We will get the data from our paper on coverage dependent adsorption energies xu-2014-probin-cover. There are some data rich figures there that would benefit from some interactivity. You can get the data here: http://pubs.acs.org/doi/suppl/10.1021/jp508805h . Extract out the supporting-information.org and energies.json file to follow here. We will make Figure 2a in the SI document here, and make it interactive with hover tooltips.

import json

from collections import OrderedDict
from bokeh import mpl
from bokeh.plotting import *
from bokeh.models import HoverTool
from bokeh.embed import components

with open('/users/jkitchin/Desktop/energies.json', 'r') as f:
    data = json.load(f)


# color for metal
# letter symbol for adsorbate
colors = {'Cu':'Orange',
          'Ag':'Silver',
          'Au':'Yellow',
          'Pd':'Green',
          'Pt':'Red',
          'Rh':'Blue',
          'Ir':'Purple'}

all_ads = ['O', 'S']

TOOLS="crosshair,pan,wheel_zoom,box_zoom,reset,hover,previewsave"
p = figure(title="Correlation between atop and fcc sites", tools=TOOLS)

for metal in ['Rh', 'Pd', 'Cu', 'Ag']:
    for adsorbate in all_ads:
        E1, E2 = [], []
        for coverage in '0.25', '0.5', '0.75', '1.0':
            if (isinstance(data[metal][adsorbate]['ontop'][coverage], float) and
                isinstance(data[metal][adsorbate]['fcc'][coverage], float)):
                E1.append(data[metal][adsorbate]['ontop'][coverage])
                E2.append(data[metal][adsorbate]['fcc'][coverage])
        labels = ['{0}-{1} {2} ML'.format(metal, adsorbate, x)
                  for x in ['0.25', '0.5', '0.75', '1.0']]
        p.line('x', 'y', color=colors[metal],
               source=ColumnDataSource(data={'x': E1,
                                             'y': E2,
                                             'label': labels}))
        p.circle('x', 'y', color=colors[metal],
               source=ColumnDataSource(data={'x': E1,
                                             'y': E2,
                                             'label': labels}))


hover =p.select({'type': HoverTool})
hover.tooltips = OrderedDict([("(atop,fcc)", "(@x, @y)"),
                              ("label", "@label")])

p.xaxis.axis_label = 'Adsorption energy on the atop site'
p.yaxis.axis_label = 'Adsorption energy on the fcc site'

script, div = components(p)
script = '\n'.join(['#+HTML_HEAD_EXTRA: ' + line for line in script.split('\n')])

print '''{script}

#+BEGIN_HTML
<a name="figure"></a>
{div}
#+END_HTML
'''.format(script=script, div=div)

Copyright (C) 2016 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.10

Discuss on Twitter

Interactive plots in HTML with Plotly

| categories: plotting, interactive, python | tags:

Most of the plots in this blog are static. Today, I look at making them interactive. I will use https://plot.ly for this. I want to use some data from a paper we published on the relative stabilities of oxide polymorphs mehta-2015-ident-poten. We will make an interactive figure showing the relative stabilities of the RuO2 polymorphs. When you hover on a point, it will show you which polymorph the point refers to. Let's see the figure first here. If you think its interesting read on to see how we made it!

We get our data source here: http://pubs.acs.org/doi/suppl/10.1021/am4059149/suppl_file/am4059149_si_001.pdf .

Now, we extract the data files:

pdftk ~/Desktop/am4059149_si_001.pdf  unpack_files

That extracts a json file called supporting-information.json. We use it as suggested in the SI pdf to plot the equations of state for RuO2 for several polymorphs.

# coding=utf-8

import plotly.plotly as py
import plotly.graph_objs as go
import plotly.tools as tls
import numpy as np

import json
import matplotlib.pyplot as plt
from ase.utils.eos import EquationOfState
with open('supporting-information.json', 'rb') as f:
    d = json.loads(f.read())

BO2 = 'RuO2'
xc = 'PBE'

layout = go.Layout(title='Energy vs. Volume for RuO<sub>2</sub> polymorphs',
                   xaxis=dict(title='Volume (Å<sup>3</sup>)'),
                   yaxis=dict(title='Energy (eV)'))

traces = []

for polymorph in ['rutile','anatase','brookite','columbite','pyrite','fluorite']:

    # number of atoms in the unit cell - used to normalize
    natoms= len(d[BO2][polymorph][xc]['EOS']['calculations']
                [0]['atoms']['symbols'])
    volumes = [entry['data']['volume']*3./natoms for entry in
               d[BO2][polymorph][xc]['EOS']['calculations']]
    energies =  [entry['data']['total_energy']*3./natoms for entry in
                 d[BO2][polymorph][xc]['EOS']['calculations']]

    trace = go.Scatter(x=np.array(volumes),
                       y=np.array(energies),
                       mode='lines+markers',
                       name=polymorph,
                       text=polymorph)

    traces += [trace]

fig = go.Figure(data=traces, layout=layout)
plot_url = py.plot(fig, filename='ruo2-2')

print tls.get_embed(plot_url)

Pretty nice, now we should have an interactive plot in our browser with the data points labeled with tags, zooming, etc… That is nice for the blog. It isn't so nice for daily work, as there is no visual version of the plot in my org-file. Of course, I can visit the url to see the plot in my browser, it is just different from what I am used to. For everyone else, this is probably better. It looks like you can actually get the data from the web page, including some minimal analysis like regression, and save your view to an image! That could be pretty nice for some data sets.

1 Using Plotly yourself

First, go to https://plot.ly and sign up for an account. You will want to register your API key like this, which will save it in a file for your convenience. Then you can do things like I did above too.

import plotly.tools as tls
tls.set_credentials_file(username='jkitchin', api_key='xxxxxxx')

Copyright (C) 2016 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.10

Discuss on Twitter

Modern use of helm - sortable candidates

| categories: helm, emacs | tags:

helm continues to be my goto completion engine. I was perusing the source for helm-top, and noticed some cool new features, like sorting the candidates in the completion buffer! I also noticed that helm sources are preferably created with some new factory functions (as opposed to the a-lists I used to use). Here I explore some of these and illustrate how to make a sortable helm source.

First, we need a function to give us some candidates we will select from. I will use a function that returns a list of cons cells from a variable containing some data where each element in the data is a plist containing a number and key. I list strings as the number and key so we can see what sorting does later. The data is just a list of plists containing a "number" and a key that is a string. We will create a helm function with these as candidates, and an ability to sort them in ascending/descending order on either the number or key.

(setq h-data '((:num 1 :key "apple")
               (:num 9 :key "berry")
               (:num 2 :key "cactus")
               (:num 5 :key "dog")
               (:num 4 :key "frog")))

(defun h-candidates ()
  "Returns candidates for the helm source."
  (loop for cand in h-data
        collect (cons (format "%s %s"
                              (plist-get cand :num)
                              (plist-get cand :key))
                      cand)))

(print (h-candidates))
(("1 apple" :num 1 :key "apple") ("9 berry" :num 9 :key "berry") ("2 cactus" :num 2 :key "cactus") ("5 dog" :num 5 :key "dog") ("4 frog" :num 4 :key "frog"))

Now, provide sorting, we need to create a candidate transformer function. This function will take the current candidates and source, and return a new list of candidates, possibly sorted. We use a variable to store how to sort the candidates. We also need a way to trigger the sorting. We will bind M-<down> to a function that will set the sort function, and refresh helm. Here is a keymap definition we will use later.

(defvar h-map
  (let ((map (make-sparse-keymap)))
    (set-keymap-parent map helm-map)
    (define-key map (kbd "M-<down>")   'h-sort)
    map)
  "keymap for a helm source.")
h-map

Now, we define the sort variable, a function that sets the variable, refreshes the candidates, and finally resets the sort variable. A key point here is the sort functions must take two arguments, which will be two candidates, and each candidate is of the form (string . data). We want to sort on one of the elements in the data plists for this example.

(defvar h-sort-fn nil)

(defun h-sort ()
  (interactive)
  (let ((action (read-char "#decreasing (d) | #increasing (i) | a-z (a) | z-a (z: ")))
    (cond
     ((eq action ?d)
      (setq h-sort-fn (lambda (c1 c2) (> (plist-get (cdr c1) :num) (plist-get (cdr c2) :num)))))
     ((eq action ?i)
      (setq h-sort-fn (lambda (c1 c2) (< (plist-get (cdr c1) :num) (plist-get (cdr c2) :num)))))
     ((eq action ?a)
      (setq h-sort-fn (lambda (c1 c2) (string< (plist-get (cdr c1) :key) (plist-get (cdr c2) :key)))))
     ((eq action ?z)
      (setq h-sort-fn (lambda (c1 c2) (string> (plist-get (cdr c1) :key) (plist-get (cdr c2) :key)))))
     (t (setq h-sort-fn nil)))
     (helm-refresh)
     (setq h-sort-fn nil)))
h-sort

Next, we define a candidate transformer. This function takes the list of candidates and the source. Here, if we have defined a sort function, we use it to sort the candidates, and if not, return the candidates. A subtle point here is the use of -sort from dash.el, which does not modify the original list at all. The build in function sort does modify the candidate list somehow, and it does not work the way you want it to here. This function gets run as the helm pattern changes.

(defun h-candidate-transformer (candidates source)
  (if h-sort-fn
    (progn (message "Sorting with %s" h-sort-fn)
    (-sort h-sort-fn candidates))
  candidates))
h-candidate-transformer

Now, just for fun, we show that dynamically defined actions are possible. Here, we generate an action list that is different for even and odd numbers. These actions are pretty trivial, but give you an idea of what might be possible; custom, context specific actions.

;; Make dynamic actions based on the candidate selected
(defun h-action-transformer (actions candidate)
  "Candidate is the result selected."
  (if (evenp (plist-get candidate :num))
      '(("Even" . identity))
    '(("Odd" . identity))))
h-action-transformer

Finally, we are ready to create a helm source. We use the new factory function for creating the source with our keymap, candidates and transformer functions.

(setq h-source
      (helm-build-sync-source "number-selector"
        :keymap h-map
        :candidates #'h-candidates
        :filtered-candidate-transformer #'h-candidate-transformer
        :action-transformer #'h-action-transformer))

Now, you can run the helm source like this.

(helm :sources 'h-source)

You can sort the numbers in descending order by typing M-<down> and pressing d. To get ascending order, press i instead. To sort on the keys, type a sort from a to z, and press z to sort on z to a. If you press tab on a selection, you will see that the actions you get depend on whether the selection is an even or odd number! So, you can get some context specific actions depending on your selection. Pretty awesome.

Copyright (C) 2016 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.10

Discuss on Twitter
« Previous Page -- Next Page »