The Kitchin Research Group

10 years ago - org mode is awesome video

Thu, 04 Jul 2024 07:52:26 EDT

Ten years ago I posted a video on YouTube titled "org mode is awesome". This 18 minute video was a tour of features in org-mode that ranged included, outlining, task management, agendas, tables, code, exporting to different formats, and extendability. That video has been viewed 92.9K times!

A fair bit has changed since then, and a lot has stayed the same. org-mode is even more awesome! That video was made with what I called at the time jmax, and that has evolved into scimax today. I use scimax on a daily basis in my research, teaching and other work. It is as important in my work today as it was 10 years ago, and has survived watching other editors come and go.

I have had in mind to make an update of that video, but it has in my opinion really stood the test of time and is still highly relevant in its present form. The only thing that would really change is the background and font colors.

I have made over 200 other YouTube videos over that time, and many of them are on using Emacs and org-mode in a lot of different ways. These videos are organized in these playlists:

scimax-eln: Using scimax as an electronic lab notebook.
scimax: Videos on libraries I developed in scimax, including org-ref
org-mode: Videos exploring features in org-mode
pycse: Videos on using Python in Emacs to solve engineering and science problems

You can see how scimax has evolved, and continues to evolve over this time through these videos, and of course through the scimax repo at https://github.com/jkitchin/scimax. There are still great things coming for scimax, stay tuned!

org-mode source

Org-mode version = 9.7.5

Exporting numbered citations in html with unsorted numbered bibliography

Thu, 03 Dec 2015 15:53:08 EST

In this post we illustrated a simple export of org-ref citations to html. This was a simple export that simply replaced each citation with a hyperlink as defined in the export function for each type of link. Today we look at formatting in text citations with superscripted numbers, and having an unsorted (i.e. in order of citation) numbered bibliography. This will take one pass to get the citations and calculate replacements and the bibliography, and one pass to replace them and insert the bibliography.

This text is just some text with somewhat random citations in it for seeing it work. You might like my two data sharing articles ^1,2. We illustrate the use of org-mode in publishing computational work ^3,4,5, experimental ⁶and mixed computational and experimental work ^7,8. This example will correctly number multiple references to a citation, e.g. ¹and ².

This post is somewhat long, and the way I worked it out is at the end (Long appendix illustrating how we got the code to work). The short version is that we do some preprocessing to get the citations in the document, calculate replacement values for them and the bibliography, replace them in the org-buffer before the export in the backend (html) format we want, and then conclude with the export. This is proof of concept work.

The main issues you can see are:

Our formatting code is very rudimentary, and relies on reftex. It is not as good as bibtex, or presumably some citation processor. Major improvements would require abandoning the reftex approach to use something that builds up the bibliography entry, allows modification of author names, and accomodates missing information gracefully.
The bibliography contents reflect the contents of my bibtex file, which is LaTeX compatible. We could clean it up more, by either post-processing to remove some things like escaped &, or by breaking compatibility with LaTeX.
The intext citations could use some fine tuning on spaces, e.g. to remove trailing spaces after words, or to move superscripts to the right of punctuation, or to adjust spaces after some citations.
Changing the bibliography style for each entry amounts to changing a variable for the bibliography. We have to modify a function to change the intext citation style, e.g. to brackets, or (author year).
I stuck with only cite links here, and only articles and books. It would not get a citenum format correct, e.g. it should not be superscripted in this case, or a citeauthor format correct. That would require some code in the replacement section that knows how to replace different types of citations.

The org-ref-unsrt-html-processor function could be broken up more, and could take some parameters to fine-tune some of these things, and generalize some things like getting the citation elements for the buffer. Overall, I think this shows that citations in org-mode with org-ref are actually pretty flexible. It is not as good as bibtex/LaTeX, and won't be for an unforseeably long time unless someone really needs high quality citations in a format other than LaTeX. Note for LaTeX export, we don't have to do any preprocessing at all. If you wanted to try Word export, you might make a pandoc processor that replaces everything in pandoc citation syntax, and then use pandoc for the conversion. If you didn't care to use the bibtex database for anything else, you could just use backend specific markup to make it exactly right for your output. I did this in reference ⁸where you can see the chemical formulas are properly subscripted.

If you would like to see the bibtex file used for this you can get it here: numbered.bib

Bibliography

Kitchin, Examples of Effective Data Sharing in Scientific Publishing, ACS Catalysis, 5(6), 3894-3899 (2015). link. doi.
John Kitchin, Data Sharing in Surface Science, Surface Science , (0), - (2015). link. doi.
Zhongnan Xu & John R Kitchin, Tuning Oxide Activity Through Modification of the Crystal and Electronic Structure: From Strain To Potential Polymorphs, Phys. Chem. Chem. Phys., 17(), 28943-28949 (2015). link. doi.
Prateek Mehta, Paul Salvador & John Kitchin, Identifying Potential BO2 Oxide Polymorphs for Epitaxial Growth Candidates, ACS Appl. Mater. Interfaces, 6(5), 3630-3639 (2014). link. doi.
Curnan & Kitchin, Effects of Concentration, Crystal Structure, Magnetism, and Electronic Structure Method on First-Principles Oxygen Vacancy Formation Energy Trends in Perovskites, The Journal of Physical Chemistry C, 118(49), 28776-28790 (2014). link. doi.
Hallenbeck & Kitchin, Effects of O2 and SO2 on the Capture Capacity of a Primary-Amine Based Polymeric CO2 Sorbent, Industrial & Engineering Chemistry Research, 52(31), 10788-10794 (2013). link. doi.
Spencer Miller, Vladimir Pushkarev, Andrew, Gellman & John Kitchin, Simulating Temperature Programmed Desorption of Oxygen on Pt(111) Using DFT Derived Coverage Dependent Desorption Barriers, Topics in Catalysis, 57(1-4), 106-117 (2014). link. doi.
Jacob Boes, Gamze Gumuslu, James Miller, Andrew, Gellman & John Kitchin, Estimating Bulk-Composition-Dependent H₂ Adsorption Energies on Cu_xPd_1-x Alloy (111) Surfaces, ACS Catalysis, 5(), 1020-1026 (2015). link. doi.

1 The working code

Here is a function to process the org file prior parsing during the export process. This function goes into org-export-before-parsing-hook, and takes one argument, the backend. We simply replace all the citation links with formatted HTML snippets or blocks. If the snippets get longer than a line, it will break.

We use org-ref-reftex-format-citation to generate the bibliography, which uses reftex to format a string with escape characters in it.

(setq org-ref-bibliography-entry-format
      '(("article" . "<li><a name=\"\%k\"></a>%a, %t, <i>%j</i>, <b>%v(%n)</b>, %p (%y). <a href=\"%U\">link</a>. <a href=\"https://doi.org/%D\">doi</a>.</li>")
        ("book" . "<li><a name=\"\%k\"></a>%a, %t, %u (%y).</li>")))

(defun org-ref-unsrt-latex-processor () nil)
(defun org-ref-unsrt-html-processor ()
  "Citation processor function for the unsrt style with html output."
  (let (links
        unique-keys numbered-keys
        replacements
        bibliography-link
        bibliographystyle-link
        bibliography)
    ;; step 1 - get the citation links
    (setq links (loop for link in (org-element-map
                                      (org-element-parse-buffer) 'link 'identity)
                      if (-contains?
                          org-ref-cite-types
                          (org-element-property :type link))
                      collect link))

    ;; list of unique numbered keys. '((key number))
    (setq unique-keys (loop for i from 1
                            for key in (org-ref-get-bibtex-keys)
                            collect (list key (number-to-string i))))


    ;; (start end replacement-text)
    (setq replacements
          (loop for link in links
                collect
                (let ((path (org-element-property :path link)))
                  (loop for (key number) in unique-keys
                        do
                        (setq
                         path
                         (replace-regexp-in-string
                          key (format "<a href=\"#%s\">%s</a>" key number)
                          path)))
                  (list (org-element-property :begin link)
                        (org-element-property :end link)
                        (format "@@html:<sup>%s</sup>@@" path)))))

    ;; construct the bibliography string
    (setq bibliography
          (concat "#+begin_html
<h1>Bibliography</h1><ol>"
                  (mapconcat
                   'identity
                   (loop for (key number) in unique-keys
                         collect
                         (let* ((result (org-ref-get-bibtex-key-and-file key))
                                (bibfile (cdr result))
                                (entry (save-excursion
                                         (with-temp-buffer
                                           (insert-file-contents bibfile)
                                           (bibtex-set-dialect
                                            (parsebib-find-bibtex-dialect) t)
                                           (bibtex-search-entry key)
                                           (bibtex-parse-entry t)))))
                           ;; remove escaped & in the strings
                           (replace-regexp-in-string "\\\\&" "&"
                                           (org-ref-reftex-format-citation
                                            entry
                                            (cdr (assoc (cdr (assoc "=type=" entry))
                                                        org-ref-bibliography-entry-format))))))
                   "")
                  "</ol>
#+end_html"))

    ;; now, we need to replace each citation. We do that in reverse order so the
    ;; positions do not change.
    (loop for (start end replacement) in (reverse replacements)
          do
          (setf (buffer-substring start end) replacement))

    ;; Eliminate bibliography style links
    (loop for link in (org-element-map
                          (org-element-parse-buffer) 'link 'identity)
          if (string= "bibliographystyle"
                      (org-element-property :type link))
          do
          (setf (buffer-substring (org-element-property :begin link)
                                  (org-element-property :end link))
                ""))

    ;; replace the bibliography link with the bibliography text
    (setq bibliography-link (loop for link in (org-element-map
                                                  (org-element-parse-buffer) 'link 'identity)
                                  if (string= "bibliography"
                                              (org-element-property :type link))
                                  collect link))
    (if (> (length bibliography-link) 1)
        (error "Only one bibliography link allowed"))

    (setq bibliography-link (car bibliography-link))
    (setf (buffer-substring (org-element-property :begin bibliography-link)
                            (org-element-property :end bibliography-link))
          bibliography)))


(defun org-ref-citation-processor (backend)
  "Figure out what to call and call it"
  (let (bibliographystyle)
    (setq
     bibliographystyle
     (org-element-property
      :path (car
             (loop for link in
                   (org-element-map
                       (org-element-parse-buffer) 'link 'identity)
                   if (string= "bibliographystyle"
                               (org-element-property :type link))
                   collect link))))
    (funcall (intern (format "org-ref-%s-%s-processor" bibliographystyle backend)))))

(add-hook 'org-export-before-parsing-hook 'org-ref-citation-processor)

(browse-url (org-html-export-to-html))

#<process open ./blog.html>

2 Long appendix illustrating how we got the code to work

The first thing we need is a list of all the citation links, in the order cited. Here they are.

(mapcar
 (lambda (link) (org-element-property :path link))
 (loop for link in (org-element-map (org-element-parse-buffer) 'link 'identity)
       if (-contains? org-ref-cite-types (org-element-property :type link))
       collect link))

kitchin-2015-examp,kitchin-2015-data-surfac-scien

xu-2015-tunin-oxide,mehta-2014-ident-poten,curnan-2014-effec-concen

hallenbeck-2013-effec-o2

miller-2014-simul-temper,boes-2015-estim-bulk

kitchin-2015-examp

kitchin-2015-data-surfac-scien

boes-2015-estim-bulk

Now, we need to compute replacements for each citation link, and construct the bibliography. We will make a numbered, unsorted bibliography, and we want to replace each citation with the corresponding numbers, hyperlinked to the entry.

We start with a list of the keys in the order cited, and a number we will use for each one.

(loop for i from 1
      for key in (org-ref-get-bibtex-keys)
      collect (list key i))

kitchin-2015-examp	1
kitchin-2015-data-surfac-scien	2
xu-2015-tunin-oxide	3
mehta-2014-ident-poten	4
curnan-2014-effec-concen	5
hallenbeck-2013-effec-o2	6
miller-2014-simul-temper	7
boes-2015-estim-bulk	8

Now, we need to compute replacements for each cite link. This will be replacing each key with the number above. We will return a list of ((start end) . "replacement text") that we can use to replace each link. For fun, we make these superscripted html.

(let ((links (loop for link in (org-element-map (org-element-parse-buffer) 'link 'identity)
                   if (-contains? org-ref-cite-types (org-element-property :type link))
                   collect link))
      (replacements (loop for i from 1
                          for key in (org-ref-get-bibtex-keys)
                          collect (list key (number-to-string i)))))
  (loop for link in links
        collect (let ((path (org-element-property :path link)))
                  (dolist (repl replacements)
                    (setq path (replace-regexp-in-string (car repl) (nth 1 repl) path)))
                  (list (org-element-property :begin link)
                        (org-element-property :end link)
                        (format "<sup>%s</sup>" path)))))

950	1004	<sup>1,2</sup>
1073	1145	<sup>3,4,5</sup>
1160	1190	<sup>6</sup>
1236	1286	<sup>7,8</sup>
1364	1388	<sup>1</sup>
1392	1427	<sup>2</sup>
4091	4117	<sup>8</sup>

We also need to compute the bibliography for each key. We will use org-ref-reftex-format-citation to do this. For that we need the parsed bibtex entries, and a format string. org-ref provides most of this.

(setq org-ref-bibliography-entry-format
      '(("article" . "<li>%a, %t, <i>%j</i>, <b>%v(%n)</b>, %p (%y). <a href=\"%U\">link</a>. <a href=\"https://doi.org/%D\">doi</a>.</li>")
        ("book" . "<li>%a, %t, %u (%y).</li>")))

(concat "<h1>Bibliography</h1><br><ol>"
        (mapconcat
         'identity
         (loop for key in (org-ref-get-bibtex-keys)
               collect
               (let* ((result (org-ref-get-bibtex-key-and-file key))
                      (bibfile (cdr result))
                      (entry (save-excursion
                               (with-temp-buffer
                                 (insert-file-contents bibfile)
                                 (bibtex-set-dialect (parsebib-find-bibtex-dialect) t)
                                 (bibtex-search-entry key)
                                 (bibtex-parse-entry)))))
                 (org-ref-reftex-format-citation
                  entry
                  (cdr (assoc (cdr (assoc "=type=" entry))
                              org-ref-bibliography-entry-format)))))
         "")
        "</ol>")

Bibliography

Kitchin, Examples of Effective Data Sharing in Scientific Publishing, {ACS Catalysis}, 5(6), 3894-3899 (2015). link. doi.
"John Kitchin", Data Sharing in Surface Science, "Surface Science ", (0), - (2015). link. doi.
Zhongnan Xu \& John R Kitchin, Tuning Oxide Activity Through Modification of the Crystal and Electronic Structure: From Strain To Potential Polymorphs, {Phys. Chem. Chem. Phys.}, 17(), 28943-28949 (2015). link. doi.
Prateek Mehta, Paul Salvador \& John Kitchin, Identifying Potential BO2 Oxide Polymorphs for Epitaxial Growth Candidates, {ACS Appl. Mater. Interfaces}, 6(5), 3630-3639 (2014). link. doi.
Curnan \& Kitchin, Effects of Concentration, Crystal Structure, Magnetism, and Electronic Structure Method on First-Principles Oxygen Vacancy Formation Energy Trends in Perovskites, {The Journal of Physical Chemistry C}, 118(49), 28776-28790 (2014). link. doi.
"Hallenbeck \& Kitchin, Effects of O2 and SO2 on the Capture Capacity of a Primary-Amine Based Polymeric CO2 Sorbent, "Industrial \& Engineering Chemistry Research", 52(31), 10788-10794 (2013). link. doi.
Spencer Miller, Vladimir Pushkarev, Andrew, Gellman \& John Kitchin, Simulating Temperature Programmed Desorption of Oxygen on Pt(111) Using DFT Derived Coverage Dependent Desorption Barriers, {Topics in Catalysis}, 57(1-4), 106-117 (2014). link. doi.
Jacob Boes, Gamze Gumuslu, James Miller, Andrew, Gellman \& John Kitchin, Estimating Bulk-Composition-Dependent H₂ Adsorption Energies on Cu_xPd_1-x Alloy (111) Surfaces, {ACS Catalysis}, 5(), 1020-1026 (2015). link. doi.

org-mode source

Org-mode version = 8.2.10

Another parsing of links for citations with pre and post text.

Thu, 26 Jun 2014 20:16:43 EDT

Some LaTeX citations look like \cite[pretext][post text]{key}. Here I explore parsing a link like (pre text)(post text)key. Note you cannot use [] inside the link, as it breaks the link syntax. Also, these links must be wrapped in [[]] because of the parentheses and spaces in the parentheses. This is a very different approach than used here which used the description of the link to define the pre and post text. The disadvantage of that approach is that the key is hidden, whereas in this approach it is not; you can see the key and pre/post text.

The basic strategy will be to use a regexp to parse the link path. The regexp below is pretty hairy, but basically it looks for optional text in () and uses numbered groups to store what is found. Then, we use what we found to construct the LaTeX syntax. We redefine the function in org-ref that gets the key for clicking, and we redefine the cite format function. The result is that we retain the click functionality that shows us what the key refers to.

(defun org-ref-parse-key (s)
  "return pretext, posttext and bibtex key from a string like \"(pre text)(post text)bibtexkey\""
  (string-match "\\(?1:(\\(?2:[^)]*\\))\\)?\\(?3:(\\(?4:[^]]*\\))\\)?\\(?5:.*\\)" s)
  ;; return pretext postext key
  (list (match-string 2 s) (match-string 4 s) (match-string 5 s)))

(defun org-ref-get-bibtex-key-and-file (&optional key)
  "returns the bibtex key and file that it is in. If no key is provided, get one under point"
 (interactive)
 (let ((org-ref-bibliography-files (org-ref-find-bibliography))
       (file))
   (unless key
     ;; get the key
     (setq key (nth 2 (org-ref-parse-key (org-ref-get-bibtex-key-under-cursor)))))
   (setq file     (catch 'result
		    (loop for file in org-ref-bibliography-files do
			  (if (org-ref-key-in-file-p key (file-truename file)) 
			      (throw 'result file)))))
   (cons key file)))

(defun org-ref-format-cite (keyword desc format)
   (cond
    ((eq format 'latex)
     (let* ((results (org-ref-parse-key keyword))
	    (pretext (nth 0 results))
	    (posttext (nth 1 results))
	    (key (nth 2 results)))
       (concat "\\cite" 
	       (when pretext (format "[%s]" pretext))
	       (when posttext (format "[%s]" posttext))
	       (format "{%s}" key))))))

org-ref-format-cite

(org-ref-format-cite "(pre text)(post text)key" nil 'latex)

\cite[pre text][post text]{key}

(org-ref-format-cite "(pre text)key" nil 'latex)

\cite[pre text]{key}

(org-ref-format-cite "key" nil 'latex)

\cite{key}

It looks like they all work! Let us test the links: mehta-2014-ident-poten, (pre text)mehta-2014-ident-poten and (pre text)(post text)biskup-2014-insul-ferrom-films. a multiple citation mehta-2014-ident-poten,thompson-2014-co2-react,calle-vallejo-2013-number.

This seems to work from an export point of view. You can not mix multiple citations with this syntax, and I did not define the html export above. Otherwise, it looks like this might be a reasonable addition to org-ref.

org-mode source

Org-mode version = 8.2.6

Using org-files like el-files

Tue, 24 Jun 2014 21:32:58 EDT

I wrote some emacs-lisp code in org-mode, and load them with org-babel-load-file. I thought it would be nice if there was load path for org-files, similar to the one for lisp files. Here I document what it might look like.

We need a load path to search for the org-file.

(setq org-load-path '("~/Dropbox/kitchingroup/jmax/"))

~/Dropbox/kitchingroup/jmax/

Next, we need the function to do the loading. We need to find the org-file, and then load it.

(defun org-require (orgfile)
  "orgfile is a symbol to be loaded"
  (let ((org-file (concat (symbol-name orgfile) ".org"))
	(path))

  ;; find the org-file
  (catch 'result
    (loop for dir in org-load-path do
	  (when (file-exists-p
		 (setq path
		       (concat
			(directory-file-name dir)
			"/"
			org-file)))
	    (throw 'result path))))
  (org-babel-load-file path)))


(org-require 'org-ref)

Loaded ~/Dropbox/kitchingroup/jmax/org-ref.el

That looks pretty simple. You do need write access to the location where the org-file is though. Let us look at a version that copies the file to a temporary directory. For some reason, I am not able to use org-babel-load-file with this. But, it does look like I can tangle the file, and assuming (big assumption) that the file tangles to a regularly named .el file, this seems to work too.

(defun org-require (orgfile)
  "orgfile is a symbol to be loaded"
  (let ((org-file (concat (symbol-name orgfile) ".org"))
        (el-file (concat (symbol-name orgfile) ".el"))
	(path))

  ;; find the org-file
  (catch 'result
    (loop for dir in org-load-path do
	  (when (file-exists-p
		 (setq path
		       (concat
			(directory-file-name dir)
			"/"
			org-file)))
	    (throw 'result path))))
  (copy-file path temporary-file-directory t)

  (org-babel-tangle-file (concat temporary-file-directory (file-name-nondirectory path)))
  (load-file (concat temporary-file-directory el-file))
))

(org-require 'org-ref)

This actually seems pretty reasonable. I have not thought about complications but for simple cases, e.g. single org-file, it looks ok.

org-mode source

Org-mode version = 8.2.6

Better integration of org-mode and email

Sun, 08 Jun 2014 20:57:53 EDT

I like to email org-mode headings and content to people. It would be nice to have some records of when a heading was sent, and to whom. We store this information in a heading. It is pretty easy to write a simple function that emails a selected region.

(defun email-region (start end)
  "Send region as the body of an email."
  (interactive "r")
  (let ((content (buffer-substring start end)))
    (compose-mail)
    (message-goto-body)
    (insert content)
    (message-goto-to)))

that function is not glamorous, and you still have to fill in the email fields, and unless you use gnus and org-contacts, the only record keeping is through the email provider.

What I would like is to send a whole heading in an email. The headline should be the subject, and if there are TO, CC or BCC properties, those should be used. If there is no TO, then I want to grab the TO from the email after you enter it and store it as a property. You should be able to set OTHER-HEADERS as a property (this is just for fun. There is no practical reason for this yet). After you send the email, it should record in the heading when it was sent.

It turned out that is a relatively tall order. While it is easy to setup the email if you have everything in place, it is tricky to get the information on TO and the time sent after the email is sent. Past lispers had a lot of ideas to make this possible, and a day of digging got me to the answer. You can specify some "action" functions that get called at various times, e.g. after sending, and a return action when the compose window is done. Unfortunately, I could not figure out any way to do things except to communicate through some global variables.

So here is the code that lets me send org-headings, with the TO, CC, BCC properties, and that records when I sent the email after it is sent.

(defvar *email-heading-point* nil
  "global variable to store point in for returning")

(defvar *email-to-addresses* nil
  "global variable to store to address in email")

(defun email-heading-return ()
  "after returning from compose do this"
  (switch-to-buffer (marker-buffer  *email-heading-point*))
  (goto-char (marker-position  *email-heading-point*))
  (setq *email-heading-point* nil)
  (org-set-property "SENT-ON" (current-time-string))
  ;; reset this incase you added new ones
  (org-set-property "TO" *email-to-addresses*)
  )

(defun email-send-action ()
  "send action for compose-mail"
  (setq *email-to-addresses* (mail-fetch-field "To")))

(defun email-heading ()
  "Send the current org-mode heading as the body of an email, with headline as the subject.

use these properties
TO
OTHER-HEADERS is an alist specifying additional
header fields.  Elements look like (HEADER . VALUE) where both
HEADER and VALUE are strings.

save when it was sent as s SENT property. this is overwritten on
subsequent sends. could save them all in a logbook?
"
  (interactive)
  ; store location.
  (setq *email-heading-point* (set-marker (make-marker) (point)))
  (org-mark-subtree)
  (let ((content (buffer-substring (point) (mark)))
	(TO (org-entry-get (point) "TO" t))
	(CC (org-entry-get (point) "CC" t))
	(BCC (org-entry-get (point) "BCC" t))
	(SUBJECT (nth 4 (org-heading-components)))
	(OTHER-HEADERS (eval (org-entry-get (point) "OTHER-HEADERS")))
	(continue nil)
	(switch-function nil)
	(yank-action nil)
	(send-actions '((email-send-action . nil)))
	(return-action '(email-heading-return)))
    
    (compose-mail TO SUBJECT OTHER-HEADERS continue switch-function yank-action send-actions return-action)
    (message-goto-body)
    (insert content)
    (when CC
      (message-goto-cc)
      (insert CC))
    (when BCC
      (message-goto-bcc)
      (insert BCC))
    (if TO
	(message-goto-body)
      (message-goto-to))       
    ))

This works pretty well for me. Since I normally use this to send tasks to people, it keeps the task organized where I want it, and I can embed an org-id in the email so if the person replies to it telling me the task is done, I can easily navigate to the task to mark it off. Pretty handy.

org-mode source

Org-mode version = 8.2.6

Finding emails from tags from org-contacts database

Thu, 05 Jun 2014 14:42:21 EDT

Org-mode has a contacts manager called org-contacts. If you set it up, you can use it to insert email addresses using a tag in message-mode. Out of the box though, it only works on one tag. You cannot do something like +group-phd to get entries tagged group but not tagged phd. Here we develop a function to do that for us.

We could use the org-files and map the headings to do this, but org-contacts has already done this and has a database we can use instead. We get the database from org-contacts-filter. Here is the first entry.

(car (org-contacts-filter))

(Chris Jones #<marker at 1 in contacts.org> ((FILE . c:/Users/jkitchin/Dropbox/org-mode/contacts.org) (TAGS . :co2:) (ALLTAGS . :co2:) (BLOCKED . ) (COMPANY . Georgia Tech, Chemical Engineering) (EMAIL . Christopher.Jones@chbe.gatech.edu) (CATEGORY . contacts)))

It looks like we have (name marker (cons cells)) for each entry. We can get the tags associated with that entry like this.

We can get the tags for an entry with this code:

(let ((entry (car (org-contacts-filter))))
  (cdr (assoc "TAGS" (nth 2 entry))))

:co2:

We will use some code for org tags. Notably, from a tags expression, we can automatically generate code that tells us if we have a match. Here we generate the code to test for a match on "+co2-group".

(let ((todo-only nil))
  (cdr (org-make-tags-matcher "+co2-group")))

(and (progn (setq org-cached-props nil) (and (not (member group tags-list)) (member co2 tags-list))) t)

Note we will have to bind tags-list before we eval this.

So to use it, we need to split the tags from an org-contacts entry into a list of strings. It appears each entry just has the tag string, so we split the substring (skipping first and last characters) by ":" to get the list. We do that here, and test if a list of tags containing "co2" is matched by the expression "co2-junior".

(let* ((tags-list (split-string (substring ":co2:" 1 -1) ":"))
       (todo-only nil))
  (eval (cdr (org-make-tags-matcher "co2-junior"))))

It is. So, now we just need to loop through the database, and collect entries that match.

(defun insert-emails-from-tags (tag-expression)
  "insert emails from org-contacts that match the tags expression. For example:
group-phd will match entries tagged with group but not with phd."
  (interactive "sTags: ")
  (insert
   (mapconcat 'identity
	      (loop for contact in (org-contacts-filter)
		    for contact-name = (car contact)
		    for email = (org-contacts-strip-link (car (org-contacts-split-property
							       (or
								(cdr (assoc-string org-contacts-email-property
										   (caddr contact)))
								""))))
		    for tags = (cdr (assoc "TAGS" (nth 2 contact)))
		    for tags-list = (if tags
					(split-string (substring (cdr (assoc "TAGS" (nth 2 contact))) 1 -1) ":")
				      '())
		    if (let ((todo-only nil))
			 (eval (cdr (org-make-tags-matcher tag-expression))))
		    
		    collect (org-contacts-format-email contact-name email))
	      ",")))

This is not quite completion in message-mode, but it is good enough. You put your cursor in the To field, and run that command, enter the tag expression, and you will get your emails!

org-mode source

Org-mode version = 8.2.6

org-shift hooks for ordering citations

Wed, 21 May 2014 11:11:56 EDT

I wrote a function that sorts citations by year, but there might be a reason to order them some other way. Here we develop a method to use shift-arrow keys to do the ordering. We will need to write a function that gets the citations in a link, gets the key under point, and then swap with neighboring keys depending on the arrow pressed. It is trivial to get the key under point (org-ref-get-bibtex-key-under-cursor), and we saw before it is easy to get the keys in a link. Let us examine swapping elements of a list here. This is an old algorithm, we store the first value, replace it with the second value, and then set the second value.

(defun org-ref-swap-keys (i j keys)
 "swap the keys in a list with index i and j"
 (let ((tempi (nth i keys)))
   (setf (nth i keys) (nth j keys))
   (setf (nth j keys) tempi))
  keys)

(org-ref-swap-keys 2 3 '(1 2 3 4))

So, we need to get the keys in the link at point, the key at point, the index of the key at point, and then we can swap them, and reconstruct the link. Here is the function that does this, and that adds the hooks.

(defun org-ref-swap-citation-link (direction)
 "move citation at point in direction +1 is to the right, -1 to the left"
 (interactive)
 (let* ((object (org-element-context))	 
        (type (org-element-property :type object))
	(begin (org-element-property :begin object))
	(end (org-element-property :end object))
	(link-string (org-element-property :path object))
        (key (org-ref-get-bibtex-key-under-cursor))
	(keys (org-ref-split-and-strip-string link-string))
        (i (index key keys)) point) ;; defined in org-ref
   (if (> direction 0) ;; shift right
     (org-ref-swap-keys i (+ i 1) keys)
     (org-ref-swap-keys i (- i 1) keys))	
  (setq keys (mapconcat 'identity keys ","))
  ;; and replace the link with the sorted keys
  (cl--set-buffer-substring begin end (concat type ":" keys))
  ;; now go forward to key so we can move with the key
  (re-search-forward key) 
  (goto-char (match-beginning 0))))

(add-hook 'org-shiftright-hook (lambda () (org-ref-swap-citation-link 1)))
(add-hook 'org-shiftleft-hook (lambda () (org-ref-swap-citation-link -1)))

lambda

nil

(org-ref-swap-citation-link -1)

kanan-2008-in-situ,kanan-2009-cobal,lutterman-2009-self-healin,mcalpin-2010-epr-eviden,liu-2014-spect-studies!

That is it! Wow, not hard at all. Check out this video of the code in action: http://screencast.com/t/YmgA0fnZ1Ogl

org-mode source

Org-mode version = 8.2.6

Exporting citations with biblatex

Mon, 19 May 2014 07:29:18 EDT

1. Summary

This post illustrates that org-ref works with biblatex. We create a simple document and export it to pdf, and HTML for this post. We also explore how to modify the export behavior of a link. You should look at the org-source at the bottom to see the whole setup; it does not all export to either format.

We need a simple export type with no default packages to avoid the natbib packages I have setup in my default list. Here is the setup. Just run C-c C-c in the block to temporarily add this to your setup.

(add-to-list 'org-latex-classes
	     '("article-biblatex"
	       "\\documentclass{article}
 [NO-DEFAULT-PACKAGES]
 [PACKAGES]
 [EXTRA]"
	       ("\\section{%s}" . "\\section*{%s}")
	       ("\\subsection{%s}" . "\\subsection*a{%s}")
	       ("\\subsubsection{%s}" . "\\subsubsection*{%s}")
	       ("\\paragraph{%s}" . "\\paragraph*{%s}")
	       ("\\subparagraph{%s}" . "\\subparagraph*{%s}")))

Add some citations andriotis-2014-infor,armiento-2014-high,biskup-2014-insul-ferrom-films,chemelewski-2014-amorp-feooh,chen-2014-inter-effec and then a single citation chen-2014-inter-effec.

and a complicated latex \cite[pre text][post text]{chen-2014-inter-effec}. Note this one will export to LaTeX fine, but not to HTML.

I would like to create a citation link that exports that way. We will do it by using a parseable syntax in the description of a link. We will have to temporarily define a new format function to achieve this. Here it is, just for the autocite command.

(defun org-ref-format-autocite (keyword desc format)
  (when (eq format 'latex)
    (concat "\\autocite"
	    (cond
	     ((string-match "::" desc)
	      (format "[%s][%s]" (car (setq results (split-string desc "::"))) (cadr results)))
	     (desc (format "[%s]" desc)))
	    (format "{%s}" keyword))))

This is the syntax:

a citation with post text: [[autocite:armiento-2014-high][post text]]

a citation with pre and post text:  [[autocite:andriotis-2014-infor][pre text::post text]]

a citation with post text: armiento-2014-high (the post text is not rendered in html).

a citation with pre and post text: andriotis-2014-infor (the pre/post text is not rendered in html).

The links in org-mode are no longer that readable when they are collapsed as descriptive links, but they are not too bad as literal links.

Here is the file and exporting-with-biblatex.pdf . One of those links is for the pdf, and one is for the HTML file.

1 Summary

org-ref seems to work pretty well with biblatex now.

We use a printbibliography link here. This exports to the latex command, or an html bibliography.

Bibliography

[andriotis-2014-infor] Andriotis, Antonis N. and Mpourmpakis, Giannis and Broderick, Scott and Rajan, Krishna and Datta, Somnath and Sunkara, Mahendra and Menon, Madhu, "Informatics guided discovery of surface structure-chemistry relationships in catalytic nanoparticles", The Journal of Chemical Physics, 140:094705 (2014). link doi
[armiento-2014-high] Armiento, R. and Kozinsky, B. and Hautier, G. and Fornari, M. and Ceder, G., "High-throughput screening of perovskite alloys for piezoelectric performance and thermodynamic stability", Phys. Rev. B, 89:134103 (2014). link doi
[biskup-2014-insul-ferrom-films] Bi{\v{s}}kup, Neven and Salafranca, Juan and Mehta, Virat and Oxley, Mark P. and Suzuki, Yuri and Pennycook, Stephen J. and Pantelides, Sokrates T. and Varela, Maria, "Insulating Ferromagnetic \ce{LaCoO_{3-$\delta$}} Films: A Phase Induced by Ordering of Oxygen Vacancies", Phys. Rev. Lett., 112:087202 (2014). link doi
[chemelewski-2014-amorp-feooh] Chemelewski, William D. and Lee, Heung-Chan and Lin, Jung-Fu and Bard, Allen J. and Mullins, C. Buddie, "Amorphous \ce{FeOOH} Oxygen Evolution Reaction Catalyst for Photoelectrochemical Water Splitting", Journal of the American Chemical Society, 136:2843-2850 (2014). link doi
[chen-2014-inter-effec] Chen, Guangxu and Zhao, Yun and Fu, Gang and Duchesne, Paul N. and Gu, Lin and Zheng, Yanping and Weng, Xuefei and Chen, Mingshu and Zhang, Peng and Pao, Chih-Wen and Lee, Jyh-Fu and Zheng, Nanfeng, "Interfacial Effects in Iron-Nickel Hydroxide-Platinum Nanoparticles Enhance Catalytic Oxidation", Science, 344:495-499 (2014). link doi

org-mode source

Org-mode version = 8.2.6

Creating bibliographies in other formats with org-ref

Sun, 18 May 2014 10:11:18 EDT

org-ref automatically generates bibliographies in LaTeX export, and it does a reasonable job automatically generating HTML bibliographies (ox-bibtex probably does this better, but it relies on an external program, whereas this approach is all elisp). Here we illustrate how to generate other formats, e.g. plain text, or org-mode formatted.

org-ref provides a convenient function that generates a bibliography entry for a key formatted according to the variable org-ref-bibliography-entry-format. This variable is a string that uses the reftex percent escapes to create an entry. The default is setup for an HTML entry like this:

  "%a, %t, <i>%j</i>, <b>%v(%n)</b>, %p (%y). <a href=\"%U\">link</a>. <a href=\"https://doi.org/%D\">doi</a>."

We can redefine it temporarily to get other formats. Here is an example of getting an org-formatted entry with some italics and bold text.

(let ((org-ref-bibliography-entry-format "%a, %t, /%j/, *%v(%n)*, %p (%y). [[%U][link]]. [[https://doi.org/%D][doi]]."))
(org-ref-get-bibtex-entry-citation "andriotis-2014-infor"))

"Andriotis, Mpourmpakis, , Broderick, Rajan, Datta, Somnath, Sunkara \& Menon, Informatics guided discovery of surface structure-chemistry relationships in catalytic nanoparticles, The Journal of Chemical Physics, 140(9), 094705 (2014). link . doi .

Now, we put some citations of various types in for water splitting mccrory-2013-bench-heter, CO₂ capture alesi-2012-evaluat-primar, and microfluidic devices voicu-2014-microf-studies. We will convert these links to a bibliography shortly.

Next, we generate an org-formatted bibliography. We will create a bracketed label at the beginning of the entry, and the org-format after that. This is a functional enough bibliography to be useful I think, and it illustrates the ideas. We will do some light transforming to replace escaped & with regular & in the bibliography.

;; temorarily redefine the format
(let ((org-ref-bibliography-entry-format "%a, %t, /%j/, *%v(%n)*, %p (%y). [[%U][link]]. [[https://doi.org/%D][doi]]."))

  (mapconcat
   (lambda (key)
     (format "[%s] %s" key
	     (replace-regexp-in-string
	      "\\\\&"
	      "&" (org-ref-get-bibtex-entry-citation key))))
   (org-ref-get-bibtex-keys) "\n\n"))

[alesi-2012-evaluat-primar] Alesi & Kitchin, Evaluation of a Primary Amine-Functionalized Ion-Exchange Resin for \ce{CO_2} Capture, Industrial & Engineering Chemistry Research, 51(19), 6907-6915 (2012). link . doi .

[mccrory-2013-bench-heter] McCrory, Jung, Peters, Jonas & Jaramillo, Benchmarking Heterogeneous Electrocatalysts for the Oxygen Evolution Reaction, J. Am. Chem. Soc., 135(45), 16977–16987 (2013). link . doi .

[voicu-2014-microf-studies] Voicu, Abolhasani, Choueiri, Rachelle, Lestari, Seiler, , Menard, Greener, Guenther, Axel, Stephan & Kumacheva, Microfluidic Studies of \ce{CO_2} Sequestration by Frustrated {L}ewis Pairs, Journal of the American Chemical Society, 0(0), null (2014). [[][link]]. doi .

You can see some minor issues with the formatting, e.g. sometimes the link is empty, if there is no url in the bibtex entry. There is no easy way to fix that. The 0 and null values in the last entry are because that is an ASAP article, and that is what is in the bibtex entry. I do not try to expand the latex code, and do not plan to do that. I do not know why there appears to be a blank author in the last entry, or why the author full names are not used. Those are reftex issues and low priority to fix for me. They do not exist in the LaTeX export. The main point here is to get a reasonably useful bibliography that you can adapt as you want.

Bibliography

[alesi-2012-evaluat-primar] Alesi \& Kitchin, Evaluation of a Primary Amine-Functionalized Ion-Exchange Resin for \ce{CO_2} Capture, Industrial \& Engineering Chemistry Research, 51(19), 6907-6915 (2012). link. doi.
[mccrory-2013-bench-heter] McCrory, Jung, Peters, Jonas \& Jaramillo, Benchmarking Heterogeneous Electrocatalysts for the Oxygen Evolution Reaction, J. Am. Chem. Soc., 135(45), 16977--16987 (2013). link. doi.
[voicu-2014-microf-studies] Voicu, Abolhasani, Choueiri, Rachelle, Lestari, Seiler, , Menard, Greener, Guenther, Axel, Stephan \& Kumacheva, Microfluidic Studies of \ce{CO_2} Sequestration by Frustrated {L}ewis Pairs, Journal of the American Chemical Society, 0(0), null (2014). link. doi.

org-mode source

Org-mode version = 8.2.6

Exporting citations in html

Sat, 17 May 2014 14:41:43 EDT

Now that org-ref works well for LaTeX, I want to explore a simple approach to exporting citations with a bibliography in html. This will enable me to put citations in blogposts, like this one armiento-2014-high and these ones daza-2014-carbon-dioxid,mehta-2014-ident-poten,suntivich-2014-estim-hybrid. We should be able to have the same citation in more than one place like this armiento-2014-high, but only have one entry in the bibliography. The bibliography should be sorted if we ask for it alesi-2012-evaluat-primar. I am curious to see this book citation: day-1995-scien-englis.

The first thing we need is a list of bibtex keys cited in this buffer.

(defun org-ref-get-bibtex-keys ()
  "return a list of unique keys in the buffer"
  (interactive)
  (let ((keys '()))
    (org-element-map (org-element-parse-buffer) 'link
      (lambda (link)
        (let ((plist (nth 1 link)))
          (when (-contains? org-ref-cite-types (plist-get plist ':type))
            (dolist
                (key
                 (org-ref-split-and-strip-string (plist-get plist ':path)))
              (when (not (-contains? keys key))
                (setq keys (append keys (list key)))))))))
    keys))

(org-ref-get-bibtex-keys)

armiento-2014-high

daza-2014-carbon-dioxid

mehta-2014-ident-poten

suntivich-2014-estim-hybrid

alesi-2012-evaluat-primar

day-1995-scien-englis

Good. Now, we need to create an HTML string for the bibliography. For each key, we will create an unordered list of simple citations. Each citation will be a named anchor in html. Let us start with a function that takes a key, and generates the HTML for that entry.

(defun org-ref-get-bibtex-entry-html (key)
(interactive)

 (let ((org-ref-bibliography-files (org-ref-find-bibliography))
       (cb (current-buffer))
       (file) (entry))

   (setq file (catch 'result
                (loop for file in org-ref-bibliography-files do
                      (message "looking for %s in %s" key file)
                      (if (org-ref-key-in-file-p key (file-truename file))
                          (throw 'result file)
                        (message "%s not found in %s" key (file-truename file))))))
   (set-buffer (find-file-noselect file))
   (prog1
       (bibtex-search-entry key nil 0)
     (setq entry (org-ref-bib-citation))
     (set-buffer cb))

   (format "<li><a name=\"#%s\">[%s] %s<\\a><li>" key key entry)))

(org-ref-get-bibtex-entry-html "mehta-2014-ident-poten")

<li><a name="#mehta-2014-ident-poten">[mehta-2014-ident-poten] Mehta, Prateek and Salvador, Paul A. and Kitchin,  John R., "Identifying Potential \ce{BO_2} Oxide Polymorphs for  Epitaxial Growth Candidates", ACS Applied Materials \& Interfaces, 0:null (2014)<\a><li>

That looks excellent. Now we simply map that function over the list of keys.

(defun org-ref-get-html-bibliography ()
(interactive)
(let ((keys (org-ref-get-bibtex-keys)))
(when keys
(concat "<h1>Bibliography</h1>
<ul>"
(mapconcat (lambda (x) (org-ref-get-bibtex-entry-html x)) keys "\n")
"\n</ul>"))))

(org-ref-get-html-bibliography)

<h1>Bibliography</h1> <ul><li><a name="#armiento-2014-high">[armiento-2014-high] Armiento, R. and Kozinsky, B. and Hautier, G. and Fornari, M. and Ceder, G., "High-throughput screening of perovskite alloys for piezoelectric performance and thermodynamic stability", Phys. Rev. B, 89:134103 (2014)<\a><li> <li><a name="#daza-2014-carbon-dioxid">[daza-2014-carbon-dioxid] Daza, Yolanda A. and Kent, Ryan A. and Yung, Matthew M. and Kuhn, John N., "Carbon Dioxide Conversion by Reverse Water-Gas Shift Chemical Looping on Perovskite-Type Oxides", Industrial \& Engineering Chemistry Research, 53:5828-5837 (2014)<\a><li> <li><a name="#mehta-2014-ident-poten">[mehta-2014-ident-poten] Mehta, Prateek and Salvador, Paul A. and Kitchin, John R., "Identifying Potential \ce{BO_2} Oxide Polymorphs for Epitaxial Growth Candidates", ACS Applied Materials \& Interfaces, 0:null (2014)<\a><li> <li><a name="#suntivich-2014-estim-hybrid">[suntivich-2014-estim-hybrid] Suntivich, Jin and Hong, Wesley T. and Lee, Yueh-Lin and Rondinelli, James M. and Yang, Wanli and Goodenough, John B. and Dabrowski, Bogdan and Freeland, John W. and Shao-Horn, Yang, "Estimating Hybridization of Transition Metal and Oxygen States in Perovskites from O K-edge X-ray Absorption Spectroscopy", The Journal of Physical Chemistry C, 118:1856-1863 (2014)<\a><li> <li><a name="#alesi-2012-evaluat-primar">[alesi-2012-evaluat-primar] Alesi, W. Richard and Kitchin, John R., "Evaluation of a Primary Amine-Functionalized Ion-Exchange Resin for \ce{CO_2} Capture", Industrial \& Engineering Chemistry Research, 51:6907-6915 (2012)<\a><li> <li><a name="#day-1995-scien-englis">[day-1995-scien-englis] Robert A. Day, "Scientific English: A Guide for Scientists and Other Profesionals", , : (1995)<\a><li> </ul> <h1>Bibliography</h1> <ul><li><a name="#armiento-2014-high">[armiento-2014-high] Armiento, R. and Kozinsky, B. and Hautier, G. and Fornari, M. and Ceder, G., "High-throughput screening of perovskite alloys for piezoelectric performance and thermodynamic stability", Phys. Rev. B, 89:134103 (2014)<\a><li> <li><a name="#daza-2014-carbon-dioxid">[daza-2014-carbon-dioxid] Daza, Yolanda A. and Kent, Ryan A. and Yung, Matthew M. and Kuhn, John N., "Carbon Dioxide Conversion by Reverse Water-Gas Shift Chemical Looping on Perovskite-Type Oxides", Industrial \& Engineering Chemistry Research, 53:5828-5837 (2014)<\a><li> <li><a name="#mehta-2014-ident-poten">[mehta-2014-ident-poten] Mehta, Prateek and Salvador, Paul A. and Kitchin, John R., "Identifying Potential \ce{BO_2} Oxide Polymorphs for Epitaxial Growth Candidates", ACS Applied Materials \& Interfaces, 0:null (2014)<\a><li> <li><a name="#suntivich-2014-estim-hybrid">[suntivich-2014-estim-hybrid] Suntivich, Jin and Hong, Wesley T. and Lee, Yueh-Lin and Rondinelli, James M. and Yang, Wanli and Goodenough, John B. and Dabrowski, Bogdan and Freeland, John W. and Shao-Horn, Yang, "Estimating Hybridization of Transition Metal and Oxygen States in Perovskites from O K-edge X-ray Absorption Spectroscopy", The Journal of Physical Chemistry C, 118:1856-1863 (2014)<\a><li> </ul>

That is basically all we need. The citation links will export as hrefs to these named targets, so they should work fine. All we need to do is modify the blogofile code a bit to use this, and add those functions to org-ref, and we should get a bibliography in our blogposts.

Bibliography

[armiento-2014-high] Armiento, R. and Kozinsky, B. and Hautier, G. and Fornari, M. and Ceder, G., "High-throughput screening of perovskite alloys for piezoelectric performance and thermodynamic stability", Phys. Rev. B, 89:134103 (2014). link doi
[daza-2014-carbon-dioxid] Daza, Yolanda A. and Kent, Ryan A. and Yung, Matthew M. and Kuhn, John N., "Carbon Dioxide Conversion by Reverse Water-Gas Shift Chemical Looping on Perovskite-Type Oxides", Industrial \& Engineering Chemistry Research, 53:5828-5837 (2014). link doi
[mehta-2014-ident-poten] Mehta, Prateek and Salvador, Paul A. and Kitchin, John R., "Identifying Potential \ce{BO_2} Oxide Polymorphs for Epitaxial Growth Candidates", ACS Applied Materials \& Interfaces, 0:null (2014). link doi
[suntivich-2014-estim-hybrid] Suntivich, Jin and Hong, Wesley T. and Lee, Yueh-Lin and Rondinelli, James M. and Yang, Wanli and Goodenough, John B. and Dabrowski, Bogdan and Freeland, John W. and Shao-Horn, Yang, "Estimating Hybridization of Transition Metal and Oxygen States in Perovskites from O K-edge X-ray Absorption Spectroscopy", The Journal of Physical Chemistry C, 118:1856-1863 (2014). link doi
[alesi-2012-evaluat-primar] Alesi, W. Richard and Kitchin, John R., "Evaluation of a Primary Amine-Functionalized Ion-Exchange Resin for \ce{CO_2} Capture", Industrial \& Engineering Chemistry Research, 51:6907-6915 (2012). link doi
[day-1995-scien-englis] Robert A. Day, "Scientific English: A Guide for Scientists and Other Profesionals", , : (1995). link doi

org-mode source

Org-mode version = 8.2.6

The Kitchin Research Group

10 years ago - org mode is awesome video

Exporting numbered citations in html with unsorted numbered bibliography

Bibliography

1 The working code

2 Long appendix illustrating how we got the code to work

Bibliography

Another parsing of links for citations with pre and post text.

Using org-files like el-files

Better integration of org-mode and email

Finding emails from tags from org-contacts database

org-shift hooks for ordering citations

Exporting citations with biblatex

Table of Contents

1 Summary

Bibliography

Creating bibliographies in other formats with org-ref

Bibliography

Exporting citations in html

Bibliography