The Kitchin Research Group

Finding similar bibtex entries

Sat, 16 Sep 2017 10:00:47 EDT

A common task while writing scientific papers is citing previous research. I use org-ref extensively for that, and it makes it pretty easy to find similar references, e.g. that have common authors, or common keywords. It also lets me find similar articles in Web of Science or Scopus. Suppose that I have cited a particular paper, e.g. e boes-2016-neural-networ, and I want to find similar references to it that are already in my bibtex file, and similar by my definition. With org-ref I can easily search by keyword or author to find similar entries, but these are limited by what I search for, and they are not sorted. Today, I will explore the first step in a recommender system that calculates similarity, and provides a sorted list of candidates with the most relevant ones first.

The idea is to calculate some measure of similarity between the title of that reference, and the titles of other references in my bibtex file, and then sort them by similarity. This is the reference I want to find similar entries for:

Boes, J. R., Groenenboom, M. C., Keith, J. A., & Kitchin, J. R., Neural network and Reaxff comparison for Au properties, Int. J. Quantum Chem., 116(13), 979–987 (2016). https://doi.org/10.1002/qua.25115

The first thing we do is read in our bibtex file, and print a representative entry.

import bibtexparser
from bibtexparser.bparser import BibTexParser

with open('/Users/jkitchin/Dropbox/bibliography/references.bib') as bibtex_file:
    parser = BibTexParser()
    bib_database = bibtexparser.load(bibtex_file, parser=parser)
    entries = bib_database.entries

print(entries[10])

{'author': 'Jaan Aarik and Aleks Aidla and V{\\"a}ino Sammelselg and Teet\nUustare', 'title': 'Effect of Growth Conditions on Formation of \\ce{TiO_2}-{II}\nThin Films in Atomic Layer Deposition Process', 'journal': 'Journal of Crystal Growth', 'volume': '181', 'number': '3', 'pages': '259 - 264', 'year': '1997', 'doi': '10.1016/S0022-0248(97)00279-0', 'link': 'http://www.sciencedirect.com/science/article/pii/S0022024897002790', 'issn': '0022-0248', 'ENTRYTYPE': 'article', 'ID': 'aarik-1997-effec-growt'}

Each entry is a dictionary containing the fields and their values. For this exploration, I will only consider similarities between titles. The next step is we find which entry corresponds to the reference we want to find similarities to.

ids = [e['ID'] for e in entries]
i = ids.index('boes-2016-neural-networ')
print(entries[i])

{'author': 'Jacob R. Boes and Mitchell C. Groenenboom and John A. Keith\nand John R. Kitchin', 'title': 'Neural Network and {Reaxff} Comparison for {Au} Properties', 'journal': 'Int. J. Quantum Chem.', 'volume': '116', 'number': '13', 'pages': '979-987', 'year': '2016', 'doi': '10.1002/qua.25115', 'link': 'https://doi.org/10.1002/qua.25115', 'issn': '1097-461X', 'keyword': 'Kohn-Sham density functional theory, neural networks, reactive\nforce fields, potential energy surfaces, machine learning', 'ENTRYTYPE': 'article', 'ID': 'boes-2016-neural-networ'}

It is best if we make the entry we want to find similarities to the first one, so here we swap the first and i^th entries.

entries[0], entries[i] = entries[i], entries[0]

Now, we prepare the list of strings to get similarities for.

titles = [e.get('title', '') for e in entries]

We will use term frequency–inverse document frequency to get a vector that represents each title, and then use cosine similarity as a measure of similarity. Here is the place to note that I chose these, and could choose other ones too. Also, it is worth noting that in this measure of similarity I did not choose which keywords to measure similarity on.

The functionality for this is provided by sklearn. It has implemented functions for the algorithms above, and in just a few lines of code you get an array of tf-idf features to analyze. The array we get from our vectorizer contains normalized vectors, so we can get the cosine similarity just from a dot product of the vectors. The first row corresponds to the similarity of the first string to all the others. I want them sorted in descending order. The argsort function returns ascending order, so we use a trick to sort the negative of the similarity score which achieves that. There are certainly more advanced treatments of the text we could use by customizing the vectorizer, e.g. word stemming, but for now we neglect that.

from sklearn.feature_extraction.text import TfidfVectorizer

vectorizer = TfidfVectorizer(stop_words='english')
X = vectorizer.fit_transform(titles)

cosine_similarities = (X * X.T).A[0]

related_docs_indices = (-cosine_similarities).argsort()

print('The top 10 recommendations for {} are:\n'.format(S[0]))
for i, j in enumerate(related_docs_indices[1:11]):
    print('{i}. {ID}: {title}, {author}\n'.format(i=i + 1, **entries[j]))

The top 10 recommendations for Neural Network and {Reaxff} Comparison for {Au} Properties are:

behler-2010-neural: Neural network potential-energy surfaces for atomistic

simulations, J{\"o}rg Behler

boes-2017-neural-networ: Neural Network Predictions of Oxygen Interactions on a Dynamic

{Pd} Surface, Jacob R. Boes and John R. Kitchin

eshet-2010-ab: Ab Initio Quality Neural-Network Potential for Sodium, Hagai Eshet and Rustam Z. Khaliullin and Thomas D. K{\"u}hne

and J{\"o}rg Behler and Michele Parrinello

behler-2014-repres-poten: Representing Potential Energy Surfaces By High-Dimensional

Neural Network Potentials, J Behler

behler-2007-gener-neural: Generalized Neural-Network Representation of High-Dimensional

Potential-Energy Surfaces, J{\"o}rg Behler and Michele Parrinello

artrith-2012-high: High-Dimensional Neural Network Potentials for Metal Surfaces:

A Prototype Study for Copper, Nongnuch Artrith and J{\"o}rg Behler

behler-2015-const: Constructing High-Dimensional Neural Network Potentials: A

Tutorial Review, J{\"o}rg Behler

artrith-2011-high: High-Dimensional Neural-Network Potentials for Multicomponent

Systems: Applications To Zinc Oxide, Nongnuch Artrith and Tobias Morawietz and J{\"o}rg Behler

sosso-2012-neural-gete: Neural Network Interatomic Potential for the Phase Change

Material \ce{GeTe}, Gabriele C. Sosso and Giacomo Miceli and Sebastiano Caravati and J{\"o}rg Behler and Marco Bernasconi

lorenz-2006-descr: Descriptions of Surface Chemical Reactions Using a Neural

Network Representation of the Potential-Energy Surface, S{\"o}nke Lorenz and Matthias Scheffler and Axel Gross

It is evident that this is showing other references containing the words "neural network"! I guess that is a little disappointing, since these would just as easily been narrowed down in org-ref. On the other hand, they are sorted and grouped, which would not happen in org-ref. This is a comparison of pretty short strings (just the titles), so maybe this would be much more interesting if abstracts were also included. Including authors would give a different set as well (I tried it, and got a bunch of my own references!).

I don't think it would be very difficult to get this into an Emacs selection tool, e.g. helm/ivy. Check this out:

import pycse.lisp

related_docs_indices[1:6].lisp

'(1592 1650 299 1751 103)'

That is a result that can be read directly by lisp, so we could simply write the code above as a shell script that takes an argument, and returns a list of indices to sort the candidates on. The alternative is to implement this in elisp, perhaps via a dynamic module if there is already a good C library for this. My sense is the Python libraries are more advanced in functionality.

This could have a number of other applications. Given some reference content, you could imagine finding emails that are similar to it, finding RSS entries that are similar to it, finding org headlines that are related, similar files, or similarity with any other set of strings that can be gathered, e.g. from Crossref or some other search, etc. I predict there will be more on these topics in the future!

org-mode source

Org-mode version = 9.0.7

Automated bibtex entry tweeting

Thu, 25 Aug 2016 12:14:31 EDT

The goal in this post is to develop an elisp function that will tweet a bibtex entry. What I want is to be on a bibtex entry, and run a command that will generate a tweet and tweet it. Here is an example bibtex entry I will use in this post. Clearly, I couldn't simply tweet the entry, it is too long. What I want instead is to generate a picture of a formatted citation, to make a gist out of the bibtex entry so we can link to it, and then to provide links in the tweet to the doi, and the bibtex entry gist.

(with-current-buffer (find-file-noselect "~/Dropbox/bibliography/references.bib")
  (goto-char (point-min))
  (re-search-forward "kitchin-2016-autom-data,")
  (bibtex-copy-entry-as-kill)
  (with-temp-buffer
    (bibtex-yank 1)
    (buffer-string)))

First, we tackle making an image. Emacs has some capability to generate svg, and we can readily convert that to png for the tweet. Here we just go to the entry, and then generate a png. I build off the citation capability of org-ref to generate a pretty reasonably formatted entry. It isn't perfect; the volume is missing in the entry, so there is a blank space between two commas, but this is good enough for me. Note we need a png for twitter. It appears you cannot upload svg yet.

(let* ((entry (with-current-buffer (find-file-noselect "~/Dropbox/bibliography/references.bib")
                (goto-char (point-min))
                (re-search-forward "kitchin-2016-autom-data,")
                (bibtex-beginning-of-entry)
                (bibtex-parse-entry t)))
       (formatted-entry (orhc-formatted-citation entry))
       (lines (with-temp-buffer
                (insert formatted-entry)
                (fill-paragraph)
                (split-string  (buffer-string) "\n")))
       (svg (svg-create 500 (* 20 (length lines)))))

  (loop for i from 0
        for line in lines
        do
        (svg-text svg line
                  :font-size "12"
                  :stroke "black"
                  :x 0
                  :y (+ 15 (* i 15))
                  :stroke-width 0.3))

  (with-temp-file "authoring.svg"
    (svg-print svg)))

(shell-command "convert authoring.svg authoring.png")

It is easy enough to get the doi, and generate the url to it.

(let* ((entry (with-current-buffer (find-file-noselect "~/Dropbox/bibliography/references.bib")
                (goto-char (point-min))
                (re-search-forward "kitchin-2016-autom-data,")
                (bibtex-beginning-of-entry)
                (bibtex-parse-entry t))))
  (format "https://doi.org/%s" (cdr (assoc "doi" entry ))))

Next, we will put the entry as a gist on Github, so we can provide a link to it. I use the gist.el package, and here just do some trickery to put the entry in a temp-file named by the key so that the gist has a nice name. This returns the url to the gist, which we would want to incorporate into a tweet.

(with-current-buffer (find-file-noselect "~/Dropbox/bibliography/references.bib")
  (goto-char (point-min))
  (re-search-forward "kitchin-2016-autom-data,")
  (save-restriction
    (bibtex-narrow-to-entry)
    (bibtex-beginning-of-entry)
    (let* ((entry-string (buffer-string))
           (entry (bibtex-parse-entry))
           (key (cdr (assoc "=key=" entry)))
           (tfile (expand-file-name (format "%s.bib" key) temporary-file-directory)))
      (with-temp-file tfile
        (insert entry-string))
      (with-current-buffer (find-file-noselect tfile)
        (gist-buffer)))
    (with-temp-buffer
      (yank)
      (buffer-string))))

Ok, All the pieces are in place. The only piece left is creating the tweet, and tweeting it. I couldn't see an obvious way to do this with twittering mode, since I didn't see where to add an image. There is a Python library for this though, and it looks pretty easy to use. Here is an example usage.

from TwitterAPI import TwitterAPI
from twitter_secrets import CONSUMER_KEY, CONSUMER_SECRET, ACCESS_TOKEN_KEY, ACCESS_TOKEN_SECRET

api = TwitterAPI(CONSUMER_KEY, CONSUMER_SECRET, ACCESS_TOKEN_KEY, ACCESS_TOKEN_SECRET)
file = open('authoring.png', 'rb')
data = file.read()
r = api.request('statuses/update_with_media', {'status':'A test tweet using the TwitterAPI with an image.'}, {'media[]':data})
print(r.status_code)

200

It will be a tad hacky, but the script is so simple we can just make a template, and run it. We need to do these things: 1) make the image, 2) make the gist 3) format and send the tweet. Here is the elisp function to do that.

(defun tweet-bibtex ()
  "Tweet the bibtex entry at point."
  (interactive)
  (bibtex-beginning-of-entry)
  (bibtex-set-field "tweeted" (current-time-string))

  (let* ((entry-string (save-restriction
                         (bibtex-beginning-of-entry)
                         (bibtex-narrow-to-entry)
                         (buffer-substring-no-properties (point-min) (point-max))))
         (entry (bibtex-parse-entry t))
         (key (cdr (assoc "=key=" entry)))
         (doi (cdr (assoc "doi" entry)))
         (svg-file (expand-file-name (format "%s.svg" key) temporary-file-directory))
         (png-file (expand-file-name (format "%s.png" key) temporary-file-directory))
         (bib-file (expand-file-name (format "%s.bib" key) temporary-file-directory))
         (py-file (expand-file-name (format "%s.py" key) temporary-file-directory))
         (formatted-entry (orhc-formatted-citation entry))
         (lines (with-temp-buffer
                  (insert formatted-entry)
                  (fill-paragraph)
                  (split-string  (buffer-string) "\n")))
         (svg (svg-create 500 (* 20 (length lines))))
         (tweet (read-string "Tweet: "))
         gist-url
         full-tweet)

    ;; delete buffers and files
    (loop for buf in (list (concat key ".bib")
                           (concat key ".png")
                           (concat key ".svg")
                           (concat key ".py"))
          do
          (when (get-buffer buf)
            (kill-buffer (get-buffer buf))))

    ;; Step 1 make the image
    (loop for i from 0
          for line in lines
          do
          (svg-text svg line
                    :font-size "12"
                    :stroke "black"
                    :x 0
                    :y (+ 15 (* i 15))
                    :stroke-width 0.3))

    (with-temp-file svg-file
      (svg-print svg))

    (shell-command (format "convert %s %s" svg-file png-file))

    ;; Step 2, make the gist. Make a temp-file so the gist has a reasonable name
    (with-temp-file bib-file
      (insert entry-string))

    (let ((bib-buffer (find-file-noselect bib-file)))
      (with-current-buffer bib-buffer
        (gist-buffer))
      (kill-buffer bib-buffer))

    ;; get url off clipboard
    (setq gist-url (with-temp-buffer
                     (yank)
                     (buffer-string)))

    ;; Format and send the tweet:
    (setq full-tweet (format "#publication %s\nhttps://doi.org/%s\nbibtex: %s" tweet doi gist-url))

    (with-temp-file py-file
      (insert (format "from TwitterAPI import TwitterAPI
from twitter_secrets import CONSUMER_KEY, CONSUMER_SECRET, ACCESS_TOKEN_KEY, ACCESS_TOKEN_SECRET

api = TwitterAPI(CONSUMER_KEY, CONSUMER_SECRET, ACCESS_TOKEN_KEY, ACCESS_TOKEN_SECRET)
file = open('%s', 'rb')
data = file.read()
r = api.request('statuses/update_with_media', {'status':'''%s'''}, {'media[]':data})"
                      png-file
                      full-tweet)))

    (if (= 0 (shell-command (format "python %s" py-file)))
        (message "%s" full-tweet)
      (message "tweet failed ;("))))

;; Now, try it out.
(with-current-buffer (find-file-noselect "~/Dropbox/bibliography/references.bib")
  (goto-char (point-min))
  (re-search-forward "kitchin-2016-autom-data,")
  (tweet-bibtex))

You can see what this tweet looks like here:

#publication I tweeted this from a bibtex file.https://t.co/NGVlRGqKSJ
bibtex: https://t.co/0UEkvyBKAM pic.twitter.com/OpbAt1h3OP
— John Kitchin (@johnkitchin) August 25, 2016

That seems pretty reasonable. Now I only need to use it about 48,000 times to benefit from the time-savings M-x tweet-bibtex offers compared to manually making all those tweets ;)

org-mode source

Org-mode version = 8.3.4

A python version of the s-exp bibtex entry

Thu, 11 Jun 2015 10:02:33 EDT

In this post we explored representing a bibtex entry in lisp s-exp notation, and showed interesting things that enables. Here, I explore something similar in Python. The s-exp notation in Python is really more like tuples. It looks almost identical, except we need a lot of commas for the Python syntax. One significant difference in Python is we need to define the functions in advance because otherwise the function symbols are undefined. Similar to lisp, we can define the field functions at run-time in a loop. We have to use an eval statement, which some Pythonistas find distasteful, but it is not that different to me than what we did in lisp.

The syntax for "executing" the data structure is quite different than in lisp, because this data is not code in Python. Instead, we have to deconstruct the data, knowing that the function is the first object, and it takes the remaining arguments in the tuple.

Here is the proof of concept:

def article(bibtex_key, *args):
    "Return the bibtex formatted entry"
    return ',\n'.join(['@article{{{0}}}'.format(bibtex_key)] +[arg[0](arg[1]) for arg in args[0]] + ['}'])

fields = ("author", "title", "journal", "pages", "number", "doi", "url", "eprint", "year")

for f in fields:
    locals()[f] = eval ('lambda x: "  {0} = {{{1}}}".format("' + f + '", x)')

entry = (article, "hallenbeck-2013-effec-o2",
         (author, "Hallenbeck, Alexander P. and Kitchin, John R."),
         (title, "Effects of \ce{O_2} and \ce{SO_2} on the capture capacity of a primary-amine based polymeric \ce{CO_2} sorbent"),
         (journal, "Industrial \& Engineering Chemistry Research"),
         (pages, "10788-10794"),
         (year, 2013),
         (number, 31),
         (doi, "10.1021/ie400582a"),
         (url, "http://pubs.acs.org/doi/abs/10.1021/ie400582a"),
         (eprint, "http://pubs.acs.org/doi/pdf/10.1021/ie400582a"))


print entry[0](entry[1], entry[2:])

@article{hallenbeck-2013-effec-o2},
  author = {Hallenbeck, Alexander P. and Kitchin, John R.},
  title = {Effects of \ce{O_2} and \ce{SO_2} on the capture capacity of a primary-amine based polymeric \ce{CO_2} sorbent},
  journal = {Industrial \& Engineering Chemistry Research},
  pages = {10788-10794},
  year = {2013},
  number = {31},
  doi = {10.1021/ie400582a},
  url = {http://pubs.acs.org/doi/abs/10.1021/ie400582a},
  eprint = {http://pubs.acs.org/doi/pdf/10.1021/ie400582a},
}

We can still get specific fields out. Since we used a tuple here, it is not quite as nice as using a dictionary, but it is neither too bad, and it can be wrapped in a reasonably convenient function.

def article(bibtex_key, *args):
    "Return the bibtex formatted entry"
    return ',\n'.join(['@article{{{0}}}'.format(bibtex_key)] +[arg[0](arg[1]) for arg in args[0]] + ['}'])

fields = ("author", "title", "journal", "pages", "number", "doi", "url", "eprint", "year")

for f in fields:
    locals()[f] = eval ('lambda x: "  {0} = {{{1}}}".format("' + f + '", x)')

entry = (article, "hallenbeck-2013-effec-o2",
         (author, "Hallenbeck, Alexander P. and Kitchin, John R."),
         (title, "Effects of \ce{O_2} and \ce{SO_2} on the capture capacity of a primary-amine based polymeric \ce{CO_2} sorbent"),
         (journal, "Industrial \& Engineering Chemistry Research"),
         (pages, "10788-10794"),
         (year, 2013),
         (number, 31),
         (doi, "10.1021/ie400582a"),
         (url, "http://pubs.acs.org/doi/abs/10.1021/ie400582a"),
         (eprint, "http://pubs.acs.org/doi/pdf/10.1021/ie400582a"))


for field in entry[2:]:
    if field[0] == author:
        print field

def get_field(entry, field):
    for element in entry[2:]:
        if element[0] == field:
            return element[1]
    else:
        return None

print get_field(entry, title)
print get_field(entry, "bad")

(<function <lambda> at 0x1005975f0>, 'Hallenbeck, Alexander P. and Kitchin, John R.')
Effects of \ce{O_2} and \ce{SO_2} on the capture capacity of a primary-amine based polymeric \ce{CO_2} sorbent
None

So, it seems Python can do some things like lisp in treating functions like first-class objects that can be used as functions, or keys. I still like the lisp s-exp better, but this is an interesting idea for Python too.

org-mode source

Org-mode version = 8.2.10

A sexp version of a bibtex entry

Wed, 10 Jun 2015 08:54:00 EDT

Below you see a typical bibtex entry. Today we explore an alternate approach to represent the information (data) in that entry as s-expressions, i.e. as a lisp data structure. Why? because it seems like an interesting exploration!

@article{hallenbeck-2013-effec-o2,
  author =       "Hallenbeck, Alexander P. and Kitchin, John R.",
  title =        {Effects of \ce{O_2} and \ce{SO_2} on the capture capacity of a
                  primary-amine based polymeric \ce{CO_2} sorbent},
  keywords =     {RUA, orgmode},
  journal =      "Industrial \& Engineering Chemistry Research",
  pages =        "10788-10794",
  year =         2013,
  volume =       {52},
  number =       {31},
  doi =          "10.1021/ie400582a",
  url =          "http://pubs.acs.org/doi/abs/10.1021/ie400582a",
  eprint =       "http://pubs.acs.org/doi/pdf/10.1021/ie400582a",
}

Here is what that same data structure might look like as a sexp-based lisp data structure.

(article "hallenbeck-2013-effec-o2"
         (author "Hallenbeck, Alexander P. and Kitchin, John R.")
         (title "Effects of \ce{O_2} and \ce{SO_2} on the capture capacity of a primary-amine based polymeric \ce{CO_2} sorbent")
         (journal "Industrial \& Engineering Chemistry Research")
         (pages "10788-10794")
         (year 2013)
         (number 31)
         (doi "10.1021/ie400582a")
         (url "http://pubs.acs.org/doi/abs/10.1021/ie400582a")
         (eprint "http://pubs.acs.org/doi/pdf/10.1021/ie400582a"))

We can retrieve data from the sexp form pretty easily. Here we get the authors.

(let* ((art '(article "hallenbeck-2013-effec-o2"
                      (author "Hallenbeck, Alexander P. and Kitchin, John R.")
                      (title "Effects of \ce{O_2} and \ce{SO_2} on the capture capacity of a primary-amine based polymeric \ce{CO_2} sorbent")
                      (journal "Industrial \& Engineering Chemistry Research")
                      (pages "10788-10794")
                      (year 2013)
                      (number 31)
                      (doi "10.1021/ie400582a")
                      (url "http://pubs.acs.org/doi/abs/10.1021/ie400582a")
                      (eprint "http://pubs.acs.org/doi/pdf/10.1021/ie400582a")))
       (fields (cddr art)))
  (cadr (assoc 'author fields)))

Hallenbeck, Alexander P. and Kitchin, John R.

That is simple enough you might just write a little function to streamline it like this, and return a formatted string.

(defun get-article-field (article field)
  "Return value of FIELD in ARTICLE."
  (cadr (assoc field (cddr article))))

(let ((art '(article "hallenbeck-2013-effec-o2"
                     (author "Hallenbeck, Alexander P. and Kitchin, John R.")
                     (title "Effects of \ce{O_2} and \ce{SO_2} on the capture capacity of a primary-amine based polymeric \ce{CO_2} sorbent")
                     (journal "Industrial \& Engineering Chemistry Research")
                     (pages "10788-10794")
                     (year 2013)
                     (number 31)
                     (doi "10.1021/ie400582a")
                     (url "http://pubs.acs.org/doi/abs/10.1021/ie400582a")
                     (eprint "http://pubs.acs.org/doi/pdf/10.1021/ie400582a"))))
  (format "%s, doi:%s (%s)"
          (get-article-field art 'author)
          (get-article-field art 'doi)
          (get-article-field art 'year)))

Hallenbeck, Alexander P. and Kitchin, John R., doi:10.1021/ie400582a (2013)

You might be wondering, why is that even a little bit interesting? One reason is that it looks a little like what lisp returns after parsing an xml file. Another is, the data structure looks kind of like data, but it is also some code, if article was defined as a function! Let us consider what this might look like. I use a macro to define the field functions since in this case they all do the same thing, and these simply return a string with the field-name and value in curly brackets. We eval the macro to make sure it defines the function. I define an article function that wraps the fields in @bibtex-key{fields}, which defines a bibtex entry.

(defmacro make-field (field-name)
  "define a field that returns a string"
  `(defun ,(intern field-name) (content)
     (format "  %s = {%s}" ,field-name content)))

(loop for field in '("author" "title" "journal" "pages" "number" "doi" "url" "eprint" "year")
  do (eval `(make-field ,field)))

(defun article (bibtex-key &rest fields)
  (concat
   (format "@article{%s,\n" bibtex-key)
   (mapconcat (lambda (field) (eval field)) fields ",\n")
   "\n}\n"))

(article "hallenbeck-2013-effec-o2"
         (author "Hallenbeck, Alexander P. and Kitchin, John R.")
         (title "Effects of \ce{O_2} and \ce{SO_2} on the capture capacity of a primary-amine based polymeric \ce{CO_2} sorbent")
         (journal "Industrial \& Engineering Chemistry Research")
         (pages "10788-10794")
         (number 31)
         (year 2013)
         (doi "10.1021/ie400582a")
         (url "http://pubs.acs.org/doi/abs/10.1021/ie400582a")
         (eprint "http://pubs.acs.org/doi/pdf/10.1021/ie400582a"))

@article{hallenbeck-2013-effec-o2,
  author = {Hallenbeck, Alexander P. and Kitchin, John R.},
  title = {Effects of ce{O_2} and ce{SO_2} on the capture capacity of a primary-amine based polymeric ce{CO_2} sorbent},
  journal = {Industrial & Engineering Chemistry Research},
  pages = {10788-10794},
  number = {31},
  year = {2013},
  doi = {10.1021/ie400582a},
  url = {http://pubs.acs.org/doi/abs/10.1021/ie400582a},
  eprint = {http://pubs.acs.org/doi/pdf/10.1021/ie400582a}
}

Wow. We executed our data structure, and got a bibtex entry! That seems moderately interesting to me. Next is an example of taking the same data structure and rendering it as xml. This is some lispy wizardry, rather than use a macro to define functions, I temporarily define functions within a cl-flet macro, which I have to collect as a list of code. Then, I eval the list. This feels pretty odd, but seems like a lispy kind of thing to do.

(eval
 (list 'cl-flet
       (append (loop for field in '("author" "title" "journal" "pages"
                                      "number" "doi" "url" "eprint" "year")
                       collect (list (intern field)
                                     '(content)
                                     `(format "  <%s>%s</%s>" ,field content ,field)))
               '((article (bibtex-key &rest fields)
                          (concat
                           (format
                            "<article bibtex-key=\"%s\">\n" bibtex-key)
                           (mapconcat (lambda (field) (eval field)) fields "\n")
                           "\n</article>")))
               )
       ;; body of cl-flet
       '(article "hallenbeck-2013-effec-o2"
                (author "Hallenbeck, Alexander P. and Kitchin, John R.")
                (title "Effects of \ce{O_2} and \ce{SO_2} on the capture capacity of a primary-amine based polymeric \ce{CO_2} sorbent")
                (journal "Industrial \& Engineering Chemistry Research")
                (pages "10788-10794")
                (number 31)
                (year 2013)
                (doi "10.1021/ie400582a")
                (url "http://pubs.acs.org/doi/abs/10.1021/ie400582a")
                (eprint "http://pubs.acs.org/doi/pdf/10.1021/ie400582a"))))

<article bibtex-key="hallenbeck-2013-effec-o2">
  <author>Hallenbeck, Alexander P. and Kitchin, John R.</author>
  <title>Effects of ce{O_2} and ce{SO_2} on the capture capacity of a primary-amine based polymeric ce{CO_2} sorbent</title>
  <journal>Industrial & Engineering Chemistry Research</journal>
  <pages>10788-10794</pages>
  <number>31</number>
  <year>2013</year>
  <doi>10.1021/ie400582a</doi>
  <url>http://pubs.acs.org/doi/abs/10.1021/ie400582a</url>
  <eprint>http://pubs.acs.org/doi/pdf/10.1021/ie400582a</eprint>
</article>

Prefer json? No problem, just reformat the functions!

(eval
 (list 'cl-flet
       (append (loop for field in '("author" "title" "journal" "pages"
                                      "number" "doi" "url" "eprint" "year")
                       collect (list (intern field)
                                     '(content)
                                     `(format "   \"%s\": \"%s\"" ,field content)))
               '((article (bibtex-key &rest fields)
                          (concat
                           (format
                            "{\"article\":\n  {\"bibtex-key\": \"%s\",\n" bibtex-key)
                           (mapconcat (lambda (field) (eval field)) fields ",\n")
                           "}\n}"))))
       ;; body of cl-flet
       '(article "hallenbeck-2013-effec-o2"
                (author "Hallenbeck, Alexander P. and Kitchin, John R.")
                (title "Effects of \ce{O_2} and \ce{SO_2} on the capture capacity of a primary-amine based polymeric \ce{CO_2} sorbent")
                (journal "Industrial \& Engineering Chemistry Research")
                (pages "10788-10794")
                (number 31)
                (year 2013)
                (doi "10.1021/ie400582a")
                (url "http://pubs.acs.org/doi/abs/10.1021/ie400582a")
                (eprint "http://pubs.acs.org/doi/pdf/10.1021/ie400582a"))))

{"article":
  {"bibtex-key": "hallenbeck-2013-effec-o2",
   "author": "Hallenbeck, Alexander P. and Kitchin, John R.",
   "title": "Effects of ce{O_2} and ce{SO_2} on the capture capacity of a primary-amine based polymeric ce{CO_2} sorbent",
   "journal": "Industrial & Engineering Chemistry Research",
   "pages": "10788-10794",
   "number": "31",
   "year": "2013",
   "doi": "10.1021/ie400582a",
   "url": "http://pubs.acs.org/doi/abs/10.1021/ie400582a",
   "eprint": "http://pubs.acs.org/doi/pdf/10.1021/ie400582a"}
}

Is this useful? Great question. I don't plan to convert by bibtex files to sexp format anytime soon ;) The format I used above is just a simple one. It might be desirable to include individual authors instead of an author string, and maybe support attributes to establish an author order. An author structure might be more complex to include scientific ids like an orcid, alternative names, etc… Finally, the s-exp data structure is super easy to use in lisp, but other languages would have parse it into some native structure the way they parse json or xml. There is limited support for s-expressions in most other non-lispy languages.

I like the idea of data representation as code, and its conversion to some other kind of format. It is subtle here, but notice we never had to write a parser for the sexp notation. That already exists as the lisp interpreter. We did write code to use the data, and convert the data. The sexp notation is pretty easy to write, in contrast to the xml or json representations. Some interesting issues might be what to do with fields that are not defined, perhaps a macro would be used on the fly, or in the cl-flet definition. It is hard to imagine doing these things in another language than lisp!

org-mode source

Org-mode version = 8.2.10

Bibtex Entries from Arxiv.org

Mon, 09 Mar 2015 15:14:52 EDT

http://arxiv.org is an open-source physics preprint server where copies of scientific manuscripts can be found. For example, http://arxiv.org/abs/0801.1144 is a paper I wrote, and you can find the PDF for that paper here: http://arxiv.org/pdf/0801.1144v1 . Each entry at Arxiv has an arxiv number, and for this paper the number is "0801.1144". In this post, we explore some capabilities of the arxiv.el library which is part of org-ref (https://github.com/jkitchin/org-ref ).

To use this library, get the org-ref distribution, make sure it is on your path, and then require the library:

(require 'arxiv)

First, there is a new org-link: arxiv:0801.1144. This is a clickable link that simply opens arxiv.org at the URL for an arxiv number, and exports as a link to that entry in arxiv.

On the right hand side of the arxiv page, there is a link under References & Citations that takes you to a page where you can get a bibtex entry. The link for this entry is http://adsabs.harvard.edu/cgi-bin/bib_query?arXiv:0801.1144 . On that page, there is a link to a bibtex entry (http://adsabs.harvard.edu/cgi-bin/nph-bib_query?bibcode=2008PhRvB..77g5437K&data_type=BIBTEX&db_key=PHY&nocookieset=1 ). We can construct this link pretty easily, we just need the bibcode for that entry. arxiv.el provides a function for that.

(arxiv-get-bibliographic-code "0801.1144")

2008PhRvB..77g5437K

Next, once we have a url, we can get the text of the bibtex entry.

(arxiv-get-bibtex-entry "2008PhRvB..77g5437K")

@ARTICLE{2008PhRvB..77g5437K,
   author = {{Kitchin}, J.~R. and {Reuter}, K. and {Scheffler}, M.},
    title = "{Alloy surface segregation in reactive environments: First-principles atomistic thermodynamics study of Ag$_{3}$Pd(111) in oxygen atmospheres}",
  journal = {\prb},
archivePrefix = "arXiv",
   eprint = {0801.1144},
 primaryClass = "cond-mat.mtrl-sci",
 keywords = {Ab initio calculations of adsorbate structure and reactions, Density functional theory local density approximation gradient and other corrections, Oxidation},
     year = 2008,
    month = feb,
   volume = 77,
   number = 7,
      eid = {075437},
    pages = {075437},
      doi = {10.1103/PhysRevB.77.075437},
   adsurl = {http://adsabs.harvard.edu/abs/2008PhRvB..77g5437K},
  adsnote = {Provided by the SAO/NASA Astrophysics Data System}
}

Finally, arxiv.el wraps those to functions together into an interactive function arxiv-add-bibtex-entry which prompts you for an arxiv number, and then a bibtex file, and then adds the text above to your bibtex file. You can then clean the entry as you see fit. It is also possible to get the pdf for an arxiv entry via arxiv-get-pdf. This is an interactive function that will prompt you for an arxiv number and a pdf file name, and it will then get the pdf for you and open it. I have not integrated this with the bibtex entry function yet, but one would ideally clean the bibtex entry to get a uniform key, and then get the pdf and name it according to the key like we do in org-ref.

(arxiv-get-pdf "0801.1144" "0801.1144.pdf")

If you use words.el you will find a new function words-arxiv which allows you to search the selected text or word at point on arxiv.org.

I do not use arxiv.org a lot, so this is not super well tested on many articles in arxiv.org, but it has worked on the few examples I have tested so far.

org-mode source

Org-mode version = 8.2.10

Turn an ISBN to a bibtex entry

Sat, 31 Jan 2015 15:48:39 EST

Occasionally, I need a bibtex entry for a book. Books are often identified by an ISBN number. Similar to using Crossref to get metadata about a DOI, we can use a web service to get metadata about an ISBN. From that, we might be able to construct a bibtex entry.

Here is an example of what we can get for ISBN 9780309095211. It does not seem to matter if there are dashes in the ISBN or not.

(with-current-buffer
        (url-retrieve-synchronously
"http://xisbn.worldcat.org/webservices/xid/isbn/9780309095211?method=getMetadata&format=json&fl=*")
      (buffer-substring url-http-end-of-headers (point-max)))

{
 "stat":"ok",
 "list":[{
	"url":["http://www.worldcat.org/oclc/224969280?referer=xid"],
	"publisher":"National Academies Press",
	"form":["BC"],
	"lccn":["2006016786"],
	"lang":"eng",
	"city":"Washington, D.C.",
	"author":"Committee on the Guide to Recruiting and Advancing Women Scientists and Engineers in Academia, Committee on Women in Science and Engineering, Policy and Global Affairs, National Research Council of the National Academies.",
	"ed":"[Online-Ausg.]",
	"year":"2006",
	"isbn":["9780309095211"],
	"title":"To recruit and advance women students and faculty in science and engineering",
	"oclcnum":["224969280",
	 "70060944",
	 "756709329",
	 "804792476",
	 "817950524",
	 "833420290",
	 "836338922",
	 "704551455"]}]}

We get a nice json data string back. We can parst that to get an actual data structure.

(with-current-buffer
        (url-retrieve-synchronously
"http://xisbn.worldcat.org/webservices/xid/isbn/9780309095211?method=getMetadata&format=json&fl=*")
      (json-read-from-string
        (buffer-substring url-http-end-of-headers (point-max))))

((list .
       [((oclcnum .
                  ["224969280" "70060944" "756709329" "804792476" "817950524" "833420290" "836338922" "704551455"])
         (title . "To recruit and advance women students and faculty in science and engineering")
         (isbn .
               ["9780309095211"])
         (year . "2006")
         (ed . "[Online-Ausg.]")
         (author . "Committee on the Guide to Recruiting and Advancing Women Scientists and Engineers in Academia, Committee on Women in Science and Engineering, Policy and Global Affairs, National Research Council of the National Academies.")
         (city . "Washington, D.C.")
         (lang . "eng")
         (lccn .
               ["2006016786"])
         (form .
               ["BC"])
         (publisher . "National Academies Press")
         (url .
              ["http://www.worldcat.org/oclc/224969280?referer=xid"]))])
 (stat . "ok"))

Ok, so we should check that stat is ok, then build the bibtex entry. Accessing the metadata below seems pretty hacky; but it works, and I don't understand the deep nesting of results, and there seems to be a vector in there.

(let* ((results  (with-current-buffer
                    (url-retrieve-synchronously
                     "http://xisbn.worldcat.org/webservices/xid/isbn/9780309095211?method=getMetadata&format=json&fl=*")
                  (json-read-from-string
                   (buffer-substring url-http-end-of-headers (point-max)))))
       (status (cdr (nth 1 results)))
       (metadata (aref (cdar results) 0)))

  (unless (string= "ok" status)
    (error "Status is %s" status))

  (concat "@book{,\n"
          (mapconcat (lambda (x)
                       (format "  %s={%s}," (car x) (cdr x)))
                     metadata "\n")
          "}\n"))

@book{,
  oclcnum={[224969280 70060944 756709329 804792476 817950524 833420290 836338922 704551455]},
  title={To recruit and advance women students and faculty in science and engineering},
  isbn={[9780309095211]},
  year={2006},
  ed={[Online-Ausg.]},
  author={Committee on the Guide to Recruiting and Advancing Women Scientists and Engineers in Academia, Committee on Women in Science and Engineering, Policy and Global Affairs, National Research Council of the National Academies.},
  city={Washington, D.C.},
  lang={eng},
  lccn={[2006016786]},
  form={[BC]},
  publisher={National Academies Press},
  url={[http://www.worldcat.org/oclc/224969280?referer=xid]},}

That looks good to me. Let us finally wrap it into a function that will take an ISBN and bibtex file interactively, create a bibtex entry, and insert it if there is not an entry with a key like that already. If we have selected region, lI should note this code uses some functionality from my org-ref package (and when I am done here, I am adding it to the doi-utils package inside org-ref). This is a fancy function, built from the experience I have from writing doi-utils.

(defun isbn-to-bibtex (isbn bibfile)
  "Get bibtex entry for ISBN and insert it into BIBFILE unless an
entry with the generated key already exists in the file."
  (interactive
   (list
    (read-input
     "ISBN: "
     ;; now set initial input
     (cond
      ;; If region is active and it starts with a number, we use it
      ((and  (region-active-p)
             (s-match "^[0-9]" (buffer-substring (region-beginning) (region-end))))
       (buffer-substring (region-beginning) (region-end)))
      ;; if first entry in kill ring starts with a number assume it is an isbn
      ;; and use it as the guess
      ((if (s-match "^[0-9]" (car kill-ring))
           (car kill-ring)))
      ;; type or paste it in
      (t
       nil)))
    (ido-completing-read
     "Bibfile: "
     (append (f-entries "." (lambda (f) (f-ext? f "bib")))
             org-ref-default-bibliography))))

  (let* ((results (with-current-buffer
                      (url-retrieve-synchronously
                       (format
                        "http://xisbn.worldcat.org/webservices/xid/isbn/%s?method=getMetadata&format=json&fl=*"
                        isbn))
                    (json-read-from-string
                     (buffer-substring url-http-end-of-headers (point-max)))))
         (status (cdr (nth 1 results)))
         (metadata (aref (cdar results) 0))
         (new-entry)
         (new-key))

    ;; check if we got something
    (unless (string= "ok" status)
      (error "Status is %s" status))

    ;; construct an alphabetically sorted bibtex entry. I assume ISBN numbers go
    ;; with book entries.
    (setq new-entry
          (concat "\n@book{,\n"
                  (mapconcat
                   'identity
                   (loop for field in (-sort 'string-lessp (mapcar 'car metadata))
                         collect
                         (format "  %s={%s}," field (cdr (assoc field metadata))))
                   "\n")
                  "\n}\n"))

    ;; build entry in temp buffer to get the key so we can check for duplicates
    (setq new-entry (with-temp-buffer
                      (insert new-entry)
                      (org-ref-clean-bibtex-entry)
                      (setq new-key (bibtex-key-in-head))
                      (buffer-string)))
    (find-file bibfile)
    (goto-char (point-min))
    (when (search-forward new-key nil t)
      (beep)
      (setq new-key (read-input
                     (format  "%s already exists. Enter new key (C-g to cancel): " new-key)
                     new-key)))
    (goto-char (point-max))
    (insert new-entry)
    ;; set key. It is simplest to just replace it, even if it is the same.
    (bibtex-beginning-of-entry)
    (re-search-forward bibtex-entry-maybe-empty-head)
    (if (match-beginning bibtex-key-in-head)
        (delete-region (match-beginning bibtex-key-in-head)
                       (match-end bibtex-key-in-head)))
    (insert new-key)
    (bibtex-fill-entry)
    (save-buffer)))

isbn-to-bibtex

That is it, for the one ISBN I have tested it on, I get a nicely sorted bibtex entry in the file I select! Hopefully that means no more tedious bibtex entry entering for books! If you use org-ref, just update to the latest version and you should be able to use this function.

Now, back to that proposal I am writing that needs a lot of citations to books that are not in my bibtex file yet, but will be soon ;)

org-mode source

Org-mode version = 8.2.10

Generating your bibliography in another file

Thu, 30 Oct 2014 19:42:35 EDT

1. Getting the references in another file

It has been proposal season. This particular round of proposals had a requirement to print the references in a separate file from the proposal. Usually I just build a pdf from org-mode, and then manually separate the references. That is not very fun if you have to do it several times. Here we examine a way to avoid this issue by using a new nobibliography link from org-ref with the bibentry LaTeX package.

We wrote this paper mehta-2014-ident-poten and this one xu-2014-relat.

Bibliography

[mehta-2014-ident-poten] Prateek Mehta, Paul Salvador, John \& Kitchin, Identifying Potential \ce{BO2} Oxide Polymorphs for Epitaxial Growth Candidates, ACS Appl. Mater. Interfaces, 6(5), 3630-3639 (2014). link. doi.
[xu-2014-relat] Zhongnan Xu \& John Kitchin, Relating the Electronic Structure and Reactivity of the 3d Transition Metal Monoxide Surfaces, Catalysis Communications, 52(), 60-64 (2014). link. doi.

Here is the resulting pdf, with no references: separate-bib.pdf.

1 Getting the references in another file

Now, we need to get the reference file. We create a new file, in org-mode, mostly for the convenience of exporting that to a pdf. Here is the code that does that.

(let* ((base (file-name-sans-extension
              (file-name-nondirectory (buffer-file-name))))
       (bbl (concat base ".bbl"))
       (orgfile (concat base "-references.org"))
       (pdffile (concat base "-references.pdf")))
  (with-temp-file orgfile
    (insert
     (format "#+LATEX_CLASS: cmu-article
#+OPTIONS: toc:nil

#+BEGIN_LaTeX
\\input{%s}
#+END_LaTeX
" bbl)))

  (find-file orgfile)
  (org-latex-export-to-pdf)
  (org-open-file pdffile))

And, here is the reference file: separate-bib.pdf

I think this would be integrated into a noexport build section of a document that would generate the pdf and references.

org-mode source

Org-mode version = 8.2.7c

Navigating your bibtex file

Mon, 13 Oct 2014 10:22:27 EDT

You may be able to tell I am spending some time cleaning up bibtex files these days. One of the things I need to do is navigate around a bibtex file easily. There are some built-in navigation keys within an entry.

navigation	key strokes
next field	C-j
end of field	TAB
beginning of entry	C-M-a
end of entry	C-M-e

I am not aware of an easy way to navigate to the next or previous entry though. I would like something simple to do that. There is a regexp defined in bibtex "bibtex-entry-head", to search for the next or previous entry.

bibtex-entry-head

^[      ]*\(@[  ]*\(?:\(?:Article\|Book\(?:let\)?\|In\(?:Book\|Collection\|Proceedings\)\|M\(?:a\(?:nual\|stersThesis\)\|isc\)\|P\(?:\(?:hdThesi\|roceeding\)s\)\|TechReport\|Unpublished\)\)\)[        ]*[({][         
]*\([][[:alnum:].:;?!`'/*@+|()<>&_^$-]+\)

Here are two functions that do it. This was a little more subtle than I anticipated. The subtlety comes about if you are at the beginning of the entry, we need to move the cursor by a character, and then search forward because of the way re-search-forward works. I also wrote in an option for a prefix argument, so you can go forward or backward several entries.

(defun bibtex-next-entry (&optional n)
  "Jump to the beginning of the next bibtex entry. N is a prefix
argument. If it is numeric, jump that many entries
forward. Negative numbers do nothing."
  (interactive "P")
  ;; Note if we start at the beginning of an entry, nothing
  ;; happens. We need to move forward a char, and call again.
  (when (= (point) (save-excursion
                     (bibtex-beginning-of-entry)))
    (forward-char)
    (bibtex-next-entry))

  ;; search forward for an entry 
  (when 
      (re-search-forward bibtex-entry-head nil t (and (numberp n) n))
    ;; go to beginning of the entry
    (bibtex-beginning-of-entry)))


(defun bibtex-previous-entry (&optional n)
  "Jump to beginning of the previous bibtex entry. N is a prefix
argument. If it is numeric, jump that many entries back."
  (interactive "P")
  (bibtex-beginning-of-entry)
 (when 
     (re-search-backward bibtex-entry-head nil t (and (numberp n) n))
   (bibtex-beginning-of-entry)))

bibtex-previous-entry

That is pretty simple. Let us go ahead and bind these to M-n, and M-p, but only in bibtex-mode. Thanks to Xah Lee for this idea.

(defun jmax-bibtex-mode-keys ()
  "Modify keymaps used by `bibtex-mode'."
  (local-set-key (kbd "M-n") 'bibtex-next-entry)
  (local-set-key (kbd "M-p") 'bibtex-previous-entry))

;; add to bibtex-mode-hook
(add-hook 'bibtex-mode-hook 'jmax-bibtex-mode-keys)

jmax-bibtex-mode-keys

Now, C-n moves forward an entry, C-u 2 C-n moves you two entries, etc… and C-p moves you back an entry, while C-u 2 C-p moves you back two entries.

Finally, I sometimes want to jump to a field in an entry. Basically, I want a completion enabled function that lists the fields in the current entry, and then jumps to the selected field. Yes, you could simply do an incremental search forward or backward that is about as simple. But, then I would not get to remind myself how to do a completion command ;)

(defun jmax-bibtex-get-fields ()
  "Get a list of fields in a bibtex entry."
  (bibtex-beginning-of-entry)
  (remove "=type="
          (remove "=key="
                  (mapcar 'car (bibtex-parse-entry)))))

(defun jmax-bibtex-jump-to-field (field)
  "Jump to FIELD in the current bibtex entry"
  (interactive
   (list
    (ido-completing-read "Field: " (jmax-bibtex-get-fields))))
  (save-restriction
    (bibtex-narrow-to-entry)
    (bibtex-beginning-of-entry)
    (when
        ;; fields start with spaces, a field name, possibly more
        ;; spaces, then =
        (re-search-forward (format "^\\s-*%s\\s-*=" field) nil t))))

jmax-bibtex-jump-to-field

These functions live in https://github.com/jkitchin/jmax/blob/master/jmax-bibtex.el , which is the version we use on a regular basis.

org-mode source

Org-mode version = 8.2.7c

Title casing bibtex entry journal titles

Sun, 12 Oct 2014 09:23:17 EDT

I mostly love bibtex. You keep bibliographic entries in a central file, and you can cite them in your writing. Bibtex takes care of most of the formatting for you, but not always all of it. Lately, we have been writing some manuscripts for submission to ACS journals. They want the titles of journal articles included in the bibliography, preferrably in title-case, or in sentence case, but all the same format either way. Unfortunately, the achemso.bst bibtex format does not make this happen. You have to title-case or sentence case the titles themselves in your bibtex file. Well, at least we can get Emacs to do the heavy lifting on that for us.

First, the manual approach. Open your bibtex file, navigate to a title field, put your cursor on the first letter of the title, and press M-c until you get to the end of the title. That runs (capitalize-word). For a few titles, you might just do this. It does not take long.

For a lot of entries though, you might prefer some code to do it. Here we consider how to convert all article titles to Title case. The current code can be found at https://github.com/jkitchin/jmax/blob/master/jmax-bibtex.el .

First, we need to decide on some rules. We will capitalize every word in a title except for words like a, an, the, … unless they start the title. We do not want to change words with $, {} in them, or \, because these are either protected or LaTeX commands and we probably do not want to change them. The gist of our idea is to get the title, split it into words, capitalize each word that needs to be, join the words together, and then set the entry title to the new capitalized title.

We use functions from s.el , and doi-utils.org here.

(defvar jmax-lower-case-words
  '("a" "an" "on" "and" "for"
    "the" "of" "in")
  "List of words to keep lowercase")

(defun jmax-title-case-article (&optional key start end)
  "Convert a bibtex entry article title to title-case. The
arguments are optional, and are only there so you can use this
function with `bibtex-map-entries' to change all the title
entries in articles."
  (interactive)
  (bibtex-beginning-of-entry)

  (let* ((title (bibtex-autokey-get-field "title"))
         (words (split-string title))
         (lower-case-words '("a" "an" "on" "and" "for"
                             "the" "of" "in")))
    (when
        (string= "article" (downcase (cdr (assoc "=type=" (bibtex-parse-entry)))))
      (setq words (mapcar
                   (lambda (word)
                     (if (or
                          ;; match words containing {} or \ which are probably
                          ;; LaTeX or protected words
                          (string-match "\\$\\|{\\|}\\|\\\\" word)
                          ;; these words should not be capitalized, unless they
                          ;; are the first word
                          (-contains? lower-case-words (s-downcase word)))
                         word
                       (s-capitalize word)))
                   words))

      ;; Check if first word should be capitalized
      (when (-contains? jmax-lower-case-words (car words))
        (setf (car words) (s-capitalize (car words))))

      ;; this is defined in doi-utils
      (bibtex-set-field
       "title"
       (mapconcat 'identity words " "))
      (bibtex-fill-entry))))

jmax-title-case-article

Now, a single command converts this:

@article{campbell-2013-enthal-entrop,
  author =       {Charles T. Campbell and Jason R. V. Sellers},
  title =        {Enthalpies and entropies of adsorption on
                  well-defined oxide surfaces: experimental
                  measurements},
  journal =      CR,
  volume =       113,
  number =       6,
  pages =        {4106-4135},
  year =         2013,
  doi =          {10.1021/cr300329s},
  url =          {https://doi.org/10.1021/cr300329s},
  month =        6,
}

to this:

@article{campbell-2013-enthal-entrop,
  author =       {Charles T. Campbell and Jason R. V. Sellers},
  title =        {Enthalpies and Entropies of Adsorption on
                  Well-defined Oxide Surfaces: Experimental
                  Measurements},
  journal =      CR,
  volume =       113,
  number =       6,
  pages =        {4106-4135},
  year =         2013,
  doi =          {10.1021/cr300329s},
  url =          {https://doi.org/10.1021/cr300329s},
  month =        6,
}

We wrote the title case function so we can use it with bibtex-map-entries. That means we can fix every entry in a file by putting a comment at the top like this:

% (bibtex-map-entries 'jmax-title-case-article)  <- put cursor here. C-x C-e

The function is not perfect, and does not include every word that should not be capitalized. You will still want to review your entries, but hopefully this saves some typing in the end.

org-mode source

Org-mode version = 8.2.7c

Abbreviated journal names in bibtex

Sat, 11 Oct 2014 17:31:59 EDT

Some journals require abbreviated journal names in the bibliography, and some require full names. Unfortunately, it is not possible to have both in your bibtex file. Or is it…

It is possible to define a @string that is replaced in your bibtex file. If we have the definition of the @string in a separate file, we can specify its definition there, e.g. as an abbreviation, or as the full name. To make this useful, we need a simple way to add new journals, and to generate the definitions.

First, you can find accepted journal name abbreviations here: http://cassi.cas.org/search.jsp .

We are going to define a variable to hold the string definition, journal full name and an abbreviation. You can find our production version of what follows here: https://github.com/jkitchin/jmax/blob/master/jmax-bibtex.el

(defvar jmax-bibtex-abbreviations
  '(("ACAT" "ACS Catalysis" "ACS Catal.")
    ("AM" "Acta Materialia" "Acta Mater.")
    ("AMM" "Acta Metallurgica et Materialia" "Acta Metall. Mater.")
    ("AMiner" "American Mineralogist" "Am. Mineral.")
    ("AngC" "Angewandte Chemie-International Edition" "Angew. Chem. Int. Edit.")
    ("APLM" "APL Materials" "APL Mat.")
    ("ACBE" "Applied Catalysis B: Environmental" "Appl. Catal. B-Environ.")
    ("APL" "Applied Physics Letters" "Appl. Phys. Lett.")
    ("ASS" "Applied Surface Science" "Appl. Surf. Sci.")
    ("CL" "Catalysis Letters" "Catal. Lett.")
    ("CT" "Catalysis Today" "Catal. Today")
    ("CPL" "Chemical Physics Letters" "Chem. Phys. Lett")
    ("CR" "Chemical Reviews" "Chem. Rev.")
    ("CSR" "Chemical Society Reviews" "Chem. Soc. Rev.")
    ("CSR" "Chemical Society Reviews" "Chem. Soc. Rev.")
    ("CM" "Chemistry of Materials" "Chem. Mater.")
    ("CSA" "Colloids and Surfaces, A: Physicochemical and Engineering Aspects" "Colloids Surf., A")
    ("CPMS" "Computational Materials Science" "Comp. Mater. Sci.")
    ("CPC" "Computer Physics Communications" "Comput. Phys. Commun.")
    ("CGD" "Crystal Growth \\& Design" "Cryst. Growth Des.")
    ("CEC" "CrystEngComm" "CrystEngComm")
    ("ECST" "ECS Transactions" "ECS Trans.")
    ("EES" "Energy \\& Environmental Science" "Energy Environ. Sci.")
    ("HPR" "High Pressure Research" "High Pressure Res.")
    ("IC" "Inorganic Chemistry" "Inorg. Chem.")
    ("IECR" "Industrial \\& Engineering Chemistry Research" "Ind. Eng. Chem. Res.")
    ("JJAP" "Japanese Journal of Applied Physics" "Jpn. J. Appl. Phys.")
    ("JMatR" "Journal of  Materials Research" "J. Mater. Res.")
    ("JALC" "Journal of Alloys and Compounds" "J. Alloy Compd.")
    ("JAC" "Journal of Applied Crystallography" "J. Appl. Crystallogr.")
    ("JAP" "Journal of Applied Physics" "J. Appl. Phys.")
    ("JC" "Journal of Catalysis" "J. Catal.")
    ("JCP" "Journal of Chemical Physics" "J. Chem. Phys.")
    ("JCG" "Journal of Crystal Growth" "J. Crys. Growth")
    ("JMC" "Journal of Materials Chemistry" "J. Mater. Chem.")
    ("JMC" "Journal of Materials Chemistry" "J. Mater. Chem.")
    ("JMSL" "Journal of Materials Science Letters" "J. Mater. Sci. Lett.")
    ("JMS" "Journal of Membrane Science" "J. Memb. Sci.")
    ("JPE" "Journal of Phase Equilibria" "J. Phase Equilib.")
    ("JPCS" "Journal of Physics and Chemistry of Solids" "J. Phys. Chem. Solids")
    ("JPCM" "Journal of Physics: Condensed Matter" "J. Phys.: Condens. Matter")
    ("JSSC" "Journal of Solid State Chemistry" "J. Solid State Chem.")
    ("JACerS" "Journal of the American Ceramic Society" "J. Am. Ceram. Soc.")
    ("JACS" "Journal of the American Chemical Society" "J. Am. Chem. Soc.")
    ("JES" "Journal of The Electrochemical Society" "J. Electrochem. Soc.")
    ("JES" "Journal of The Electrochemical Society" "J. Electrochem. Soc.")
    ("JMS" "Journal of Membrane Science" "J. Memb. Sci.")
    ("JVST" "Journal of Vacuum Science \\& Technology A" "J. Vac. Sci. Technol. A")
    ("ML" "Materials Letters" "Mater. Lett.")
    ("MSE-BS" "Materials Science and Engineering B" "Mat. Sci. Eng. B-Solid")
    ("MOLSIM" "Molecular Simulation" "Mol. Sim.")
    ("Nature" "Nature" "Nature")
    ("NM" "Nature Materials" "Nat. Mater.")
    ("PML" "Philosophical Magazine Letters" "Phil. Mag. Lett.")
    ("PMA" "Philosophical Magazine A" "Phil. Mag. A")
    ("PA" "Physica A: Statistical Mechanics and its Applications" "Physica A")
    ("PB" "Physica B-Condensed Matter" "Physica B")
    ("PCCP" "Physical Chemistry Chemical Physics" "Phys. Chem. Chem. Phys.")
    ("PSSB" "physica status solidi (b)" "Phys. Status Solidi B")
    ("PRA" "Physical Review A" "Phys. Rev. A")
    ("PRB" "Physical Review B" "Phys. Rev. B")
    ("PRL" "Physical Review Letters" "Phys. Rev. Lett.")
    ("PCM" "Physics and Chemistry of Minerals" "Phys. Chem. Miner.")
    ("PSurfSci" "Progress in Surface Science" "Prog. Surf. Sci.")
    ("Science" "Science" "Science")
    ("SABC" "Sensors and Actuators B: Chemical" "Sensor. Actuat. B-Chem.")
    ("SS" "Surface Science" "Surf. Sci.")
    ("EPJB" "The European Physical Journal B" "Eur. Phys. J. B")
    ("JPC" "The Journal of Physical Chemistry" "J. Phys. Chem.")
    ("JPCB" "The Journal of Physical Chemistry  B" "J. Phys. Chem. B")
    ("JPCC" "The Journal of Physical Chemistry C" "J. Phys. Chem. C")
    ("JCP" "The Journal of Chemical Physics" "J. Chem. Phys.")
    ("TSF" "Thin Solid Films" "Thin Solid Films")
    ("TC" "Topics in Catalysis" "Top. Catal.")
    ("WR" "Water Research" "Water Res."))
  "List of (string journal-full-name journal-abbreviation)")

bibtex-abbreviations

This data structure will serve a few purposes.

We will generate the bib files that define the @string definitions
We will use it to modify bibtex files to use those strings.

First, here are some simple functions to generate the @string definitions.

(defun jmax-bibtex-generate-longtitles ()
  (interactive)
  (with-temp-file "longtitles.bib"
    (dolist (row bibtex-abbreviations)
      (insert (format "@string{%s=\"%s\"}\n"
                      (nth 0 row)
                      (nth 1 row))))))

(defun jmax-bibtex-generate-shorttitles ()
  (interactive)
  (with-temp-file "shorttitles.bib"
    (dolist (row bibtex-abbreviations)
      (insert (format "@string{%s=\"%s\"}\n"
                      (nth 0 row)
                      (nth 2 row))))))

jmax-bibtex-generate-shorttitles

(jmax-bibtex-generate-longtitles)
(jmax-bibtex-generate-shorttitles)

Here are the results of running that code: shorttitles.bib and longtitles.bib . This is the first step. We have the @strings defined. Now, we need to convert the names in a bibtex entry to use our string. We want to replace full names and abbreviated names with the @string.

(defun jmax-stringify-journal-name (&optional key start end)
  "replace journal name with a string. The strings are defined in `bibtex-abbreviations'."
  (interactive)
  (bibtex-beginning-of-entry)
  (when
      (string= "article"
               (downcase
                (cdr (assoc "=type=" (bibtex-parse-entry)))))
    (let* ((full-names (mapcar
                        (lambda (row)
                          (cons  (nth 1 row) (nth 0 row)))
                        bibtex-abbreviations))
           (abbrev-names (mapcar
                          (lambda (row)
                            (cons  (nth 2 row) (nth 0 row)))
                          bibtex-abbreviations))
           (journal (s-trim (bibtex-autokey-get-field "journal")))
           (bstring (or
                     (cdr (assoc journal full-names))
                     (cdr (assoc journal abbrev-names)))))
      (when bstring
        (bibtex-set-field "journal" bstring t)
        (bibtex-fill-entry)))))

jmax-stringify-journal-name

Now, with a single command, we can convert this:

@article{lizzit-2001-surfac-ru,
  author =       {S. Lizzit and A. Baraldi and A. Groso and K. Reuter
                  and M. Ganduglia-Pirovano and C. Stampfl and
                  M. Scheffler and M. Stichler and C. Keller and
                  W. Wurth and D. Menzel},
  title =        {Surface Core-level Shifts of Clean and
                  Oxygen-covered {R}u(0001)},
  journal =      {Physical Review B,
  volume =       63,
  number =       20,
  pages =        {nil},
  year =         2001,
  doi =          {10.1103/physrevb.63.205419},
  url =          {https://doi.org/10.1103/PhysRevB.63.205419},
  month =        5,
}

into this:

@article{lizzit-2001-surfac-ru,
  author =       {S. Lizzit and A. Baraldi and A. Groso and K. Reuter
                  and M. Ganduglia-Pirovano and C. Stampfl and
                  M. Scheffler and M. Stichler and C. Keller and
                  W. Wurth and D. Menzel},
  title =        {Surface Core-level Shifts of Clean and
                  Oxygen-covered {R}u(0001)},
  journal =      PRB,
  volume =       63,
  number =       20,
  pages =        {nil},
  year =         2001,
  doi =          {10.1103/physrevb.63.205419},
  url =          {https://doi.org/10.1103/PhysRevB.63.205419},
  month =        5,
}

If you have a lot of entries you want to modify, you can use bibtex-map-entries like this. Basically, put the elisp form in a comment, and then execute the elisp form

%% (bibtex-map-entries 'jmax-stringify-journal-name)  <- put cursor here. C-x C-e

This saves some effort. Over time, I will keep adding entries to the abbreviation table. As long as a standard journal name or abbreviation is in your bibtex file, this approach should work pretty well. After you replace the journal names with @string entries, you have to generate the string file, either shorttitles.bib or longtitles.bib, and in your LaTeX file, change your bibliography line to:

\bibliography{shorttitles,references}

The order is important. The @string definitions are in shorttitles.bib, and your bibtex entries in references.bib.

org-mode source

Org-mode version = 8.2.7c

The Kitchin Research Group

Finding similar bibtex entries

Automated bibtex entry tweeting

A python version of the s-exp bibtex entry

A sexp version of a bibtex entry

Bibtex Entries from Arxiv.org

Turn an ISBN to a bibtex entry

Generating your bibliography in another file

Table of Contents

Bibliography

1 Getting the references in another file

Navigating your bibtex file

Title casing bibtex entry journal titles

Abbreviated journal names in bibtex