Using org-ref to keep your bibtex files in order

| categories: emacs, bibtex | tags:

Maintaining an accurate, useful bibliography of references is critical for scientific writing. It is also not trivial. While it is easy to download and copy bibliographic entries to your database, these entries are often incomplete, not consistently formatted, and can contain invalid characters. org-ref provides several utility functions to help with this.

1 "cleaning" a bibtex entry

Consider this bibtex entry from http://pubs.acs.org/action/showCitFormats?doi=10.1021%2Fie500588j .

@article{doi:10.1021/ie500588j,
author = {Okada, Tomohiko and Ozono, Shoya and Okamoto, Masami and Takeda, Yohei and Minamisawa, Hikari M. and Haeiwa, Tetsuji and Sakai, Toshio and Mishima, Shozi},
title = {Magnetic Rattle-Type Core–Shell Particles Containing Iron Compounds with Acid Tolerance by Dense Silica},
journal = {Industrial & Engineering Chemistry Research},
volume = {0},
number = {0},
pages = {null},
year = {0},
doi = {10.1021/ie500588j},

URL = {http://pubs.acs.org/doi/abs/10.1021/ie500588j},
eprint = {http://pubs.acs.org/doi/pdf/10.1021/ie500588j}
}

On the surface it looks fine, but there are the following issues with it:

  1. The bibtex key is hard to remember. I like systematically named keys.
  2. There is a bare & in the journal title, which is not legal in LaTeX.
  3. There is no year entry, even though it is a 2014 entry. The pages, volume, and number are also problematic, but this is an ASAP article and the reference does not have those yet.
  4. It is hard to see, but the dash between core and shell is a non-ascii character, which can cause problems in LaTeX.
  5. The entry is not very nicely aligned or indented.

You can fix these problems by putting your cursor on the bibtex entry, and typing M-x org-ref-clean-bibtex-entry. This will fix the bibtex key to a standard form, align and indent the entry, escape the & so it is legal syntax, prompt you for a year, and show you the non-ascii characters so you can replace them. The resulting, nicely formatted entry is shown below.

@article{okada-2014-magnet-rattl,
  author =	 {Okada, Tomohiko and Ozono, Shoya and Okamoto, Masami
                  and Takeda, Yohei and Minamisawa, Hikari M. and
                  Haeiwa, Tetsuji and Sakai, Toshio and Mishima,
                  Shozi},
  title =	 {Magnetic Rattle-Type Core-Shell Particles Containing
                  Iron Compounds with Acid Tolerance by Dense Silica},
  journal =	 {Industrial \& Engineering Chemistry Research},
  volume =	 0,
  pages =	 {null},
  year =	 2014,
  doi =		 {10.1021/ie500588j},
  number =	 0,
  url =		 {http://pubs.acs.org/doi/abs/10.1021/ie500588j},
  eprint =	 {http://pubs.acs.org/doi/pdf/10.1021/ie500588j},
}

The key formatting comes from these definitions:

;; variables that control bibtex key format for auto-generation
;; I want firstauthor-year-title-words
;; this usually makes a legitimate filename to store pdfs under.
(setq bibtex-autokey-year-length 4
      bibtex-autokey-name-year-separator "-"
      bibtex-autokey-year-title-separator "-"
      bibtex-autokey-titleword-separator "-"
      bibtex-autokey-titlewords 2
      bibtex-autokey-titlewords-stretch 1
      bibtex-autokey-titleword-length 5)

You should develop a discipline to clean each entry as you add them, and before you cite them. It is a pain to change the key, and then find and change all the places you used that key before. Now that you have a systematic key, go ahead and download the pdf for the article, and save it in your pdf directory by that key name. Set the variable org-ref-pdf-directory to this directory, and later when you click on citations you will be able to open the pdf easily.

2 Validating your bibliography

elisp:bibtex-validate
will check your bibliography for valid syntax. This is a bibtex command.

org-bib.bib

3 Sorting your bibtex file

It is a good idea to keep your bibtex file sorted. This will facilitate finding duplicate entries, and will make it easier to find things. I usually add entries to the top of the file, and then clean them. Then run the command

elisp:bibtex-sort-buffer
. This will sort the entries for you. This is also a bibtex command.

org-bib.bib

4 Make a full bibliography pdf

A good way to check your bibliography for duplicates, spelling errors, and invalid formats is to make a pdf containing all the entries. Open your bibtex file, and run

elisp:org-ref-build-full-bibliography
. If all goes well, you will get a pdf of your bibliography that you can check for accuracy. If there are errors, you will have to fix them until the pdf is generated.

Try it out: org-bib.bib

5 Finding bad citation links

Sometimes you will get bad citation links in your document. Maybe there is no corresponding entry, maybe you typed in the wrong key, maybe you changed the key. Either way, you need to find them and fix them. Run the command

elisp:org-ref-find-bad-citations
to find them.
cite:test

6 Extracting citations entries

You will often work from your default bibliography for your own work. Eventually you will need to extract the entries cited so you can send them to someone. The command

elisp:org-ref-extract-bibtex-entries
will do that for you. If I have cited something
cite:calle-vallejo-2010-trend-stabil
.

7 Summary

You can see a screen cast of this post here: http://screencast.com/t/yZCOdO6kJ

8 References

9 Bibtex entries

#+BEGINSRC: text :tangle extract-bib7108tYg.bib @article{calle-vallejo-2010-trend-stabil, author = {Calle-Vallejo, F. and Martinez, J. I. and Garcia- Lastra, J. M. and Mogensen, M. and Rossmeisl, J.}, title = {Trends in Stability of Perovskite Oxides}, journal = "Angewandte Chemie-International Edition", volume = 49, number = 42, pages = {7699-7701}, year = 2010, doi = {10.1002/anie.201002301}, keyword = {density functional calculations heats of formation perovskites thermochemistry transition-metals catalysts ferroelectricity}, } #+ENDSRC

Copyright (C) 2014 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.6

Discuss on Twitter

Using org-ref for citations and references

| categories: emacs, org-mode | tags:

org-ref is an emacs-lisp module to handle bibliographic citations, and references to figures, tables and sections in org-mode. It was written first for use in org-mode, and for reasonable export to LaTeX. It does not work well for any other export (eg HTML) for now. The goal of org-ref is to make it easy to add citations, and that the citations be useful, clickable links. Below, I illustrate some of those features.

1 Installation

You can get the source here: https://github.com/jkitchin/jmax/blob/master/org-ref.org . This is an org-mode file, and you can load it with

(org-babel-load-file "org-ref.org")

That should be all you need for basic usage.

2 Basic usage for citations

org-ref was written with bibtex in mind. We will discuss customization to use biblatex later.

The first thing you should do is add a bibliography link to your document. This link is usually put at the end of the document, where you want the bibliography to be printed in a LaTeX export. Use the filename of the bibliography with its extension in the link, and you can use multiple files if they are separated by commas. The bibliography link is clickable, and it will open the bibliography that you clicked on. Depending on the LaTeX class you are exporting to, you may also need to set a bibliographystyle.

Out of the box, org-ref binds the key "C-c ]" to insert a citation. That key runs the command org-ref-insert-cite-link, which will prompt you for a regular expression to search your bibliographies for. The search is done using reftex, so you can mark several entries at once, and they will be added as a citation link. The actual kind of link used is dependent on the value of org-ref-default-citation-link, which defaults to cite.

If your cursor is on a citation link, or at the end of the link, you can run the command again to append new citations to the link. The new entries are separated by commas. The links are clickable, and when you click on them you will see a message in the minibuffer with the citation, and options to open the entry, open a pdf (if you have it), open the url (if the entry has one), or to open notes about the entry. You get this menu for the entry that you clicked on.

If you want a different type of citation, type C-u C-c ]. You will be prompted for a format, and you can choose a different type of link format. Most bibtex formats are supported, and some biblatex formats are supported.

3 Basic usage for references

The other use of org-ref is for references to labels in your document. You can put labels in figure and table captions, and then reference them in your document. The label link is clickable, and when you click on it there will be a message in the minibuffer telling you how many labels of that name were found (it should be one).

Table 1: An uninteresting table. ()
0 3
4 7

You can then refer to Table (tab-boring). The ref links are also clickable, and they take you to the spot where the label is defined. You can enter ref links with completion. Press C-c C-l, type ref, press enter, and then press tab. You will get a list of the labels defined in the buffer you can choose from. There are many things you can make a ref to including a tblname, a label link, an explicit \label{}, and an org-mode #+label: line. (tab-boring)

There is an eqref link that is used for equations. You must use LaTeX labels in the equation for this.

\[e^x = 4 \label{eq-exp} \]

You can see in (eq-exp) the problem to solve.

4 Miscellaneous links

There are two link types for generating a LateX list of tables and list of figures. These links are clickable, and they open a temporary buffer with a list of the tables or figures that you can click on. They export to \listoftables and \listoffigures

You can run these as commands, org-ref-list-of-tables and org-ref-list-of-figures if you do not want a list of those things in your exported document.

5 Customization

5.1 Default bibliography, pdf directory,

This is an optional configuration, but it is handy to define a default bibliography so that you can add citations to an org-file without defining a bibliography link. I store all pdfs of bibtex entries in the directory defined by org-ref-pdf-directory, and by the name of the bibtex entry label. This enables org-ref to open the pdf if it can find it. The notes file is optional, I create org-entries for each bibtex entry, which I have experimented with various ways of organizing them with tags, and in topical headings.

(setq org-ref-bibliography-notes "~/Dropbox/bibliography/notes.org"
      org-ref-default-bibliography '("~/Dropbox/bibliography/references.bib")
      org-ref-pdf-directory "~/Dropbox/bibliography/bibtex-pdfs/")

5.2 New key binding

The default key binding to insert a citation is C-c ]. I chose that because I do not like pressing shift to get ). However, this key binding usurps an org-mode agenda file command. To change this, set this variable

(setq org-ref-insert-cite-key "C-c )")
  • You may have to restart emacs to get C-c ] back.

5.3 Default link type

If you use another citation type alot, you may change the default link type. For example, you may prefer autocite links by default. Just set it like this:

(setq org-ref-default-citation-link "autocite")

5.4 New citation types

There are so many citation types between bibtex and biblatex. I did not try to add them all. You can add new citation links yourself in your init file. Here, we add a new cite link called citez, and assign a reftex menu key of z to it. This function automatically adds the new link to org-mode, with the citation menu functionality, creates the completion function, and adds the citation to the list of known types.

(org-ref-define-citation-link "citez" ?z)

(org-ref-define-citation-link "citeauthorfull" ?F) citeauthorfull:hautier-2012-accur

hautier-2012-accur

It is assumed that this will be exported as \citez[optional stuff]{label}. If you need more flexibility than that, you will have to define everything manually.

For example, the original cite link was defined like this.

(defun org-ref-cite-link-format (keyword desc format)
   (cond
    ((eq format 'html) (format "(<cite>%s</cite>)" path))
    ((eq format 'latex)
     (concat "\\cite" (when desc (format "[%s]" desc)) "{"
             (mapconcat (lambda (key) key) (org-ref-split-and-strip-string keyword) ",")
             "}"))))

(org-add-link-type
 "cite"
 'org-ref-cite-onclick-minibuffer-menu ;; clicking function
 'org-ref-cite-link-format) ;; formatting function

You should also add your new citation type to the list of org-ref-cite-types.

6 Summary

This covers most of the basic org-ref functionality. There are also several utility functions for interacting with org-buffers and bibtex files that will be described later.

See http://screencast.com/t/bxfafVydE for a screencast of using org-ref.

http://screencast.com/t/bxfafVydE

Copyright (C) 2014 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.7c

Discuss on Twitter

Autogenerating functions in emacs-lisp

| categories: emacs, emacs-lisp | tags:

I have a need to generate a lot of similar functions, and I do not want to cut and paste the code. I want to generate the functions with code. This seems to be what macros are for in emacs lisp.

As a prototype example, we will make functions that raise a number to a power. We want functions like power-3 and power-4 that raise numbers to the third and fourth powers. We will define functions like this for the numbers 0-9.

Here we define the macro. i do not want to get into the nitty gritty details of macro definitions here.

(defmacro make-power-n (n)
 `(defun ,(intern (format "power-%s" n)) (arg) (expt arg ,n)))

(make-power-n 4)

(power-4 4)
256

Now we use the macro and mapcar on it onto a list of numbers. We have to eval the macro in the mapcar lambda function.

(defmacro make-power-n (n)
 `(defun ,(intern (format "power-%s" n)) (arg) (expt arg ,n)))

(mapcar (lambda (x) (eval `(make-power-n ,x))) '(0 1 2 3 4 5 6 7 8 9))
 
;; example of a few functions
(list (power-0 3) (power-1 3) (power-2 3))
1 3 9

It works! We created 10 functions in a little bit of code.

Copyright (C) 2014 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.6

Discuss on Twitter

Make a list of org-files in all the subdirectories of the current working directory

| categories: emacs, org-mode, recursive | tags:

It would be helpful to get a listing of org-files in a directory tree in the form of clickable links. This would be useful, for example, to find all files associated with a project in a directory with a particular extension, or to do some action on all files that match a pattern. To do this, we will have to recursively walk through the directories and examine their contents.

Let us examine some of the commands we will need to use. One command is to get the contents of a directory. We will explore the contents of a directory called literate in my computer.

;; list contents of the directory
(let ((abspath nil)
      (match nil)
      (nosort t))
  (directory-files "literate" abspath match nosort))
makefile-main Makefile main.o main.f90 main literate.org hello.f90 circle.o circle.mod circle.f90 circle-area.png archive a.out .. .

Note the presence of . and ... Those stand for current directory and one directory up. We should remove those from the list. We can do that like this.

;; remove . and ..
(let ((abspath nil)
      (match nil)
      (nosort t))
  (remove "." 
          (remove ".." 
                  (directory-files "literate" abspath match nosort))))
makefile-main Makefile main.o main.f90 main literate.org hello.f90 circle.o circle.mod circle.f90 circle-area.png archive a.out

Next, we need to know if a given entry in the directory files is a file or a directory. Emacs-lisp has a few functions for that. We use absolute filenames here since the paths are relative to the "molecules" directory. Note we could use absolute paths in directory-files, but that makes it hard to remove "." and "..".

;; print types of files in the directory
(let ((root "literate")
      (abspath nil)
      (match nil)
      (nosort t))
  (mapcar (lambda (x)
            (cond
             ((file-directory-p (expand-file-name x root))
              (print (format "%s is a directory" x)))
             ((file-regular-p (expand-file-name x root))
              (print (format "%s is a regular file" x)))))
          (remove "." 
                  (remove ".." 
                          (directory-files root abspath match nosort)))))
"makefile-main is a regular file"

"Makefile is a regular file"

"main.o is a regular file"

"main.f90 is a regular file"

"main is a regular file"

"literate.org is a regular file"

"hello.f90 is a regular file"

"circle.o is a regular file"

"circle.mod is a regular file"

"circle.f90 is a regular file"

"circle-area.png is a regular file"

"archive is a directory"

"a.out is a regular file"

Now, we are at the crux of this problem. We can differentiate between files and directories. For each directory in this directory, we need to recurse into it, and list the contents. There is some code at http://turingmachine.org/bl/2013-05-29-recursively-listing-directories-in-elisp.html which does this, but I found that I had to modify the code to not list directories, and here I want to show a simpler recursive code.

(defun os-walk (root)
  "recursively walks through directories getting list of absolute paths of files"
  (let ((files '()) ; empty list to store results
        (current-list (directory-files root t)))
    ;;process current-list
    (while current-list
      (let ((fn (car current-list))) ; get next entry
        (cond 
         ;; regular files
         ((file-regular-p fn)
          (add-to-list 'files fn))
         ;; directories
         ((and
           (file-directory-p fn)
           ;; ignore . and ..
           (not (string-equal ".." (substring fn -2)))
           (not (string-equal "." (substring fn -1))))
          ;; we have to recurse into this directory
          (setq files (append files (os-walk fn))))
        )
      ;; cut list down by an element
      (setq current-list (cdr current-list)))
      )
    files))

(os-walk "literate")
c:/Users/jkitchin/Dropbox/blogofile-jkitchin.github.com/blog/literate/makefile-main c:/Users/jkitchin/Dropbox/blogofile-jkitchin.github.com/blog/literate/main.o c:/Users/jkitchin/Dropbox/blogofile-jkitchin.github.com/blog/literate/main.f90 c:/Users/jkitchin/Dropbox/blogofile-jkitchin.github.com/blog/literate/main c:/Users/jkitchin/Dropbox/blogofile-jkitchin.github.com/blog/literate/literate.org c:/Users/jkitchin/Dropbox/blogofile-jkitchin.github.com/blog/literate/hello.f90 c:/Users/jkitchin/Dropbox/blogofile-jkitchin.github.com/blog/literate/circle.o c:/Users/jkitchin/Dropbox/blogofile-jkitchin.github.com/blog/literate/circle.mod c:/Users/jkitchin/Dropbox/blogofile-jkitchin.github.com/blog/literate/circle.f90 c:/Users/jkitchin/Dropbox/blogofile-jkitchin.github.com/blog/literate/circle-area.png c:/Users/jkitchin/Dropbox/blogofile-jkitchin.github.com/blog/literate/a.out c:/Users/jkitchin/Dropbox/blogofile-jkitchin.github.com/blog/literate/Makefile c:/Users/jkitchin/Dropbox/blogofile-jkitchin.github.com/blog/literate/archive/empty-text-file.txt

Nice, that gives us a recursive listing of all the files in this directory tree. Let us take this a step further, and apply a function to that list to filter out a list of the org files. We will also create org-links out of these files.

(defun os-walk (root)
  (let ((files '()) ;empty list to store results
        (current-list (directory-files root t)))
    ;;process current-list
    (while current-list
      (let ((fn (car current-list))) ; get next entry
        (cond 
         ;; regular files
         ((file-regular-p fn)
          (add-to-list 'files fn))
         ;; directories
         ((and
           (file-directory-p fn)
           ;; ignore . and ..
           (not (string-equal ".." (substring fn -2)))
           (not (string-equal "." (substring fn -1))))
          ;; we have to recurse into this directory
          (setq files (append files (os-walk fn))))
        )
      ;; cut list down by an element
      (setq current-list (cdr current-list)))
      )
    files))

(require 'cl)

(mapcar 
 (lambda (x) (princ (format "[[%s][%s]]\n" x (file-relative-name x "."))))
 (remove-if-not 
  (lambda (x) (string= (file-name-extension x) "org"))
  (os-walk "literate")))

literate/literate.org

That is certainly functional. It might be nice to format the links a bit nicer to show their structure in a table of contents way, or to sort them in a nice order if there were many of these files.

Copyright (C) 2014 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.5h

Discuss on Twitter

yasnippets for jasp, ase and python

| categories: emacs, jasp, ase | tags:

In using [[http://github.com/jkitchin/jasp for calculations, I find there are lots of small python phrases I use over and over. Today I will examine using yasnippet to save time and keystrokes. yasnippet is a template expansion module, where you type a small set of characters, press Tab, and the characters "expand" to the full text. It is pretty sophisticated, and allows you to define "tab-stops" which you interactively fill in, and tab between like filling in a form.

All the snippets are defined in the

*Appendix
.

1 Tangle the snippets, and add them to yasnippet

Each snippet definition belongs in a file in a directory. The main directory is called "snippets". Since I anticipate using these snippets in org-mode, each snippet is defined in a directory within snippets called "org-mode". First, we make the directory here. I also want to use the snippets in python mode, so we also create a python-mode directory here. We do not have to duplicate the snippets. We can create a file called .yas-parents , with one line in it containing "org-mode".

mkdir -p snippets/org-mode
mkdir -p snippets/python-mode
echo "org-mode" > snippets/python-mode/.yas-parents

Each snippet is defined in a src block with a :tangle header. So, we can extract them all in one command here.

(org-babel-tangle)
snippets/org-mode/iase snippets/org-mode/imp snippets/org-mode/inp snippets/org-mode/ij snippets/org-mode/pl snippets/org-mode/pyl snippets/org-mode/pxl snippets/org-mode/pp snippets/org-mode/npa snippets/org-mode/awt snippets/org-mode/avw snippets/org-mode/agf snippets/org-mode/ape snippets/org-mode/atms snippets/org-mode/atm snippets/org-mode/cga snippets/org-mode/cc snippets/org-mode/wjn snippets/org-mode/wjl

We also need to add our new directory to yasnippets. This is done by adding the directory to the yas-snippet-dirs variable. You could add this to your init.el file to permanently add these snippets.

(add-to-list 'yas-snippet-dirs "c:/Users/jkitchin/Dropbox/blogofile-jkitchin.github.com/_blog/snippets")
c:/Users/jkitchin/Dropbox/blogofile-jkitchin.github.com/blog/snippets ~/.emacs.d/snippets c:/users/jkitchin/Dropbox/kitchingroup/jmax/elpa/yasnippet-20140106.1009/snippets

Finally, we reload all the snippet definitions, so our new definitions are ready to use.

(yas-reload-all)
[yas] Reloaded everything (snippets will load just-in-time)... (some errors, check *Messages*).

Alternatively, you might just load this directory.

(yas-load-directory "./snippets")

2 Using the snippets

Each of these snippets is for a python phrase, but I usually write my python blocks in org-mode. You would use these by typing the shortcut name, and then pressing tab. Below I show what each shortcut expands to.

wjl → with jasp('') as calc:

wjn → with jasp('',) as calc: calc.calculate(atoms)

cc → calc.calculate(atoms)

cga → atoms = calc.get_atoms()

atm → Atom('', )

atms → atoms = Atoms([], cell)=

ape → atoms.get_potential_energy()

agf → atoms.get_forces()

avw → from ase.visualize import view view(atoms)

awt → from ase.io import write write('.png', atoms, show_unit_cell=2)

npa → np.array()

pp → plt.plot(, )

pxl → plt.xlabel()

pyl → plt.ylabel()

pl → plt.legend()

ij → from jasp import *

inp → import numpy as np

imp → import matplotlib.pyplot as plt

iase → from ase import Atom, Atoms

What other snippets would be handy?

3 Appendix

3.1 jasp snippets

# -*- mode: snippet -*-
# --
with jasp('$1') as calc:
    $0
# -*- mode: snippet -*-
# --
with jasp('$1',$0) as calc:
    calc.calculate(atoms)
# -*- mode: snippet -*-
# --
calc.calculate(atoms)
# -*- mode: snippet -*-
# --
atoms = calc.get_atoms()

3.2 ase snippets

Template for an ase.Atom

# -*- mode: snippet -*-
# --
Atom('$1', $2)
# -*- mode: snippet -*-
# --
atoms = Atoms([$1], cell=$2)
# -*- mode: snippet -*-
# --
atoms.get_potential_energy()
# -*- mode: snippet -*-
# --
atoms.get_forces()
# -*- mode: snippet -*-
# --
from ase.visualize import view
view(${1:atoms})
# -*- mode: snippet -*-
# --
from ase.io import write
write('$1.png', ${2:atoms}, show_unit_cell=${3:2})

3.3 python snippets

# -*- mode: snippet -*-
# --
import numpy as np
# -*- mode: snippet -*-
# --
import matplotlib.pyplot as plt
# -*- mode: snippet -*-
# --
from ase import Atom, Atoms
# -*- mode: snippet -*-
# --
np.array($0)
# -*- mode: snippet -*-
# --
plt.plot($1, $2)
# -*- mode: snippet -*-
# --
plt.xlabel($1)
# -*- mode: snippet -*-
# --
plt.ylabel($1)
# -*- mode: snippet -*-
# --
plt.legend($1)
# -*- mode: snippet -*-
# --
from jasp import *

Copyright (C) 2014 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.5h

Discuss on Twitter
« Previous Page -- Next Page »