Finding missing citation entries in an org-file

| categories: org-mode, bibtex | tags:

Today we consider how to find citations in a document that have no corresponding entries in a bibtex file. There are a couple of pieces to this which we work out in stages below. First, we specify the bibtex file using a bibliography link defined in jorg-bib.el.

jorg-bib provides a function that gives us the relevant bibliography files found in this file.

(cite-find-bibliography)
bib1.bib bib2.bib

We can get a list of keys in these files

(let ((bibtex-files (cite-find-bibliography)))
(bibtex-global-key-alist))
(adams-1993-orien-imagin . t) (aarik-1997-effec-tio2 . t) (aruga-1985-struc-iron . t)

Now, here are some citations that we want to include in this document.

cite:aruga-1985-struc-iron,aarik-1997-effec-tio2

Here is a citation that is not in the bibtex file

cite:kitchin-2016-nobel-lecture

To find out if any of these are missing, we need a list of the citation keys in this document. We first get all the content from the cite links. We parse the buffer, and for each cite link, we get the path of the link, which contains our keys.

(let ((parsetree (org-element-parse-buffer)))
  (org-element-map parsetree 'link
    (lambda (link)       
      (let ((type (nth 0 link))
            (plist (nth 1 link))
            (content (nth 2 link)))
	(when (equal (plist-get plist ':type) "cite")
	  (plist-get plist ':path))))))
aruga-1985-struc-iron,aarik-1997-effec-tio2 kitchin-2016-nobel-lecture

That is almost what we need, but we need to separate the keys that are joined by commas. That function already exists in jorg-bib as cite-split-keys. We need to make a slight variation to get a list of all the entries, since the cite-split-keys returns a list of entries for each link. Here is on approach to that.

(let ((parsetree (org-element-parse-buffer))
      (results '()))
  (org-element-map parsetree 'link
    (lambda (link)       
      (let ((plist (nth 1 link)))
	(when (equal (plist-get plist ':type) "cite")
	  (setq results (append results (cite-split-keys (plist-get plist ':path))))))))
results)
aruga-1985-struc-iron aarik-1997-effec-tio2 kitchin-2016-nobel-lecture

Ok, now we just need to check each entry of that list against the list of entries in the bibtex files, and highlight any that are not good. We use an index function below to tell us if an element is in a list. This index function works for strings. We use the strange remove-if-not function, which requires something like triple negative logic to get the list of keys that are not in the bibtex files.

(require 'cl)

(defun index (substring list)
  "return the index of string in a list of strings"
  (let ((i 0)
	(found nil))
    (dolist (arg list i)
      (if (string-match substring arg)
	  (progn 
	    (setq found t)
	    (return i)))
      (setq i (+ i 1)))
    ;; return counter if found, otherwise return nil
    (if found i nil)))

;; generate the list of bibtex-keys and cited keys
(let* ((bibtex-files (cite-find-bibliography))
       (bibtex-keys (mapcar (lambda (x) (car x)) (bibtex-global-key-alist)))
       (parsetree (org-element-parse-buffer))
       (cited-keys))
  (org-element-map parsetree 'link
    (lambda (link)       
      (let ((plist (nth 1 link)))			     
	(when (equal (plist-get plist ':type) "cite")
	  (setq cited-keys (append cited-keys (cite-split-keys (plist-get plist ':path))))))))

(princ (remove-if-not (lambda (arg) (not (index arg bibtex-keys))) cited-keys))
)
(kitchin-2016-nobel-lecture)

The only improvement from here would be if this generated a temporary buffer with clickable links to find that bad entry! Let us take a different approach here, and print this to a temporary buffer of clickable links.

(require 'cl)

(defun index (substring list)
  "return the index of string in a list of strings"
  (let ((i 0)
	(found nil))
    (dolist (arg list i)
      (if (string-match substring arg)
	  (progn 
	    (setq found t)
	    (return i)))
      (setq i (+ i 1)))
    ;; return counter if found, otherwise return nil
    (if found i nil)))

;; generate the list of bibtex-keys and cited keys
(let* ((bibtex-files (cite-find-bibliography))
       (bibtex-keys (mapcar (lambda (x) (car x)) (bibtex-global-key-alist)))
       (bad-citations '()))

  (org-element-map (org-element-parse-buffer) 'link
    (lambda (link)       
      (let ((plist (nth 1 link)))			     
	(when (equal (plist-get plist ':type) "cite")
	  (dolist (key (cite-split-keys (plist-get plist ':path)) )
	    (when (not (index key bibtex-keys))
	      (setq bad-citations (append bad-citations
			    `(,(format "%s [[elisp:(progn (find-file \"%s\")(goto-char %s))][not found here]]\n"
		      key (buffer-file-name)(plist-get plist ':begin)))))
			    ))))))

(mapconcat 'identity bad-citations ""))

kitchin-2016-nobel-lecture

elisp:(progn (find-file "/home-research/jkitchin/Dropbox/blogofile-jkitchin.github.com/_blog/blog.org")(goto-char 1052))

That is likely to come in handy. I have put a variation of this code in jorb-bib, in the function called jorg-bib-find-bad-citations.

Copyright (C) 2014 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.6

Discuss on Twitter

A popup menu for citation links in org-mode

| categories: org-mode, bibtex | tags:

I have been exploring ways to get more information out of links in org-mode. I have considered popups , and right-clicking . Here I show how to get a popup menu on a citation link. The idea is that clicking or opening the ditation link should give you a menu. The menu should give you some context, e.g. if the bibtex key even exists. If it does, you should be able to get a quick view of the citation in the minibuffer. You should be able to open the entry in the bibtex file from the menu. If you have a pdf of the reference, you should have an option to open it. You should be able to open the url associated with the entry from the menu too.

Here is the function. We use https://github.com/auto-complete/popup-el , and some code from https://github.com/jkitchin/jmax/blob/master/jorg-bib.el .

(org-add-link-type
 "cite"
 ;; this function is run when you click on the link
 (lambda (link-string) 
   (let* ((menu-choice)
         ;; this is in jorg-bib.el
         (results (get-bibtex-key-and-file))
	 (key (car results))
	 (cb (current-buffer))
         (pdf-file (format (concat jorg-bib-pdf-directory "%s.pdf") key))
         (bibfile (cdr results)))
     (setq menu-choice
	   (popup-menu* 
	    (list (popup-make-item (if 
				       (progn
					 (let ((cb (current-buffer)) result)					
					   (find-file bibfile)
					   (setq result (bibtex-search-entry key))
					   (switch-to-buffer cb)
					   result))
				       "Simple citation"
				     "No key found")  :value "cite")
		  (popup-make-item (if
				       (progn
					 (let ((cb (current-buffer)) result)					  
					   (find-file bibfile)
					   (setq result (bibtex-search-entry key))
					   (switch-to-buffer cb)
					   result))
				       (format "Open %s in %s" key bibfile)
				     "No key found") :value "bib")
		  (popup-make-item 
		   ;; check if pdf exists.jorg-bib-pdf-directory is a user defined directory.
                   ;; pdfs are stored by bibtex key in that directory
		   (if (file-exists-p pdf-file)
		       (format "Open PDF for %s" key)
		     "No pdf found") :value "pdf")
		  (popup-make-item "Open URL" :value "web")
		  (popup-make-item "Open Notes" :value "notes")
		  )))

     (cond
      ;; goto entry in bibfile
      ((string= menu-choice "bib")       
       (find-file bibfile)
       (bibtex-search-entry key))

      ;; goto entry and try opening the url
      ((string= menu-choice "web")   
       (let ((cb (current-buffer)))
	 (save-excursion
	   (find-file bibfile)
	   (bibtex-search-entry key)
	   (bibtex-url))
	 (switch-to-buffer cb)))
       
      ;; goto entry and open notes, create notes entry if there is none
      ((string= menu-choice "notes")   
       (find-file bibfile)
       (bibtex-search-entry key)       
       (jorg-bib-open-bibtex-notes))

     ;; open the pdf file if it exists
     ((string= menu-choice "pdf")
      (when (file-exists-p pdf-file)
	  (org-open-file pdf-file)))

     ;; print citation to minibuffer
     ((string= menu-choice "cite")
      (let ((cb (current-buffer)))	
	(message "%s" (save-excursion (find-file bibfile)
				      (bibtex-search-entry key)  
				      (jorg-bib-citation)))
	(switch-to-buffer cb))))))
 ;; formatting
(lambda (keyword desc format)
   (cond
    ((eq format 'html) (format "(<cite>%s</cite>)" path))
    ((eq format 'latex)
     (concat "\\cite{"
	     (mapconcat (lambda (key) key) (cite-split-keys keyword) ",")
	     "}")))))

cite:daza-2014-carbon-dioxid,mehta-2014-ident-poten,test,ahuja-2001-high-ruo2

Here you can see an example of a menu where I have the PDF:

Here is an example menu of a key with no entry:

And, and entry with no PDF:

Here is the simple citation:

And a reference from the other bibliography:

Not bad! I will probably replace the cite link in jorg-bib with something like this.

Copyright (C) 2014 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.6

Discuss on Twitter

A better insert citation function for org-mode

| categories: org-mode, bibtex | tags:

I have setup a reftex citation format that inserts a cite link using reftex like this.

(eval-after-load 'reftex-vars
  '(progn
      (add-to-list 'reftex-cite-format-builtin
                   '(org "Org-mode citation"
                         ((?\C-m . "cite:%l"))))))

I mostly like this, but it does not let me add citations to an existing citation; doing that leads to the insertion of an additional cite within the citation, which is an error. One way to make this simple is to add another cite format which simple returns the selected keys. You would use this with the cursor at the end of the link, and it will just append the results.

(add-to-list 'reftex-cite-format-builtin
                   '(org "Org-mode citation"
                         ((?\C-m . "cite:%l")
			  (?a . ",%l"))))

That actually works nicely. I would like a better approach though, that involves less keywork. Ideally, a single function that does what I want, which is when on a link, append to it, and otherwise insert a new citation link. Today I will develop a function that fixes that problem.

(defun insert-cite-link ()
  (interactive)
  (let* ((object (org-element-context))
	 (link-string-beginning (org-element-property :begin object))
	 (link-string-end (org-element-property :end object))
	 (path (org-element-property :path object)))    
    (if (and (equal (org-element-type object) 'link) 
               (equal (org-element-property :type object) "cite"))
	(progn
	  (goto-char link-string-end)
	  (insert (concat "," (mapconcat 'identity (reftex-citation t ?a) ","))))
      (insert (concat "cite:" (mapconcat 'identity (reftex-citation t) ",")))
      )))

That function is it! Org-mode just got a lot better. That function only puts a cite link in, but since that is all I use 99.99+% of the time, it works fine for me!

Copyright (C) 2014 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.6

Discuss on Twitter

Putting link references to lines of code in a source block

| categories: org-mode | tags:

I keep forgetting about this interesting gem of a feature in org-mode code blocks. You can put references to specific lines of code outside the block! http://orgmode.org/manual/Literal-examples.html#Literal-examples

The following code block has some references in it that we can refer to later:

#+BEGIN_SRC emacs-lisp -n -r
(save-excursion (sc)
  (goto-char (point-min))) (jump)
#+END_SRC
1: (save-excursion
2:   (goto-char (point-min)))

In line

(sc)
we remember the current position.
(jump)
jumps to point-min.

To make this work with python we have to make a slight change to the reference format in the header.

#+BEGIN_SRC python -n -r -l "#(ref:%s)"
for i in range(5):                # (for)
    print i                       # (body)
#+END_SRC
1: for i in range(5):
2:     print i
0
1
2
3
4

In line

(for)
we initialize the loop, and in line
(body)
we run it.

Copyright (C) 2014 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.5h

Discuss on Twitter

Literate programming in python with org-mode and noweb

| categories: org-mode, python | tags:

This post examines a different approach to literate programming with org-mode that uses noweb . I have adapted an example from http://home.fnal.gov/~neilsen/notebook/orgExamples/org-examples.html which has some pretty cool ideas in it.

The gist of using noweb is that in your source blocks you have labels like <<imports>>, that refer to other named code blocks that get substituted in place of the label. In the example below, we put labels for a code block of imports, for a function definition, a class definition, and a main function. This code block will get tangled to main.py . The noweb expansion happens at export, so here is the literal code block:

#+BEGIN_SRC python :noweb yes :tangle main.py
<<imports>>

<<some-func>>

<<class-dfn>>

<<main-func>>

if __name__ == '__main__':
    status = main()
    sys.exit(status)
#+END_SRC

You may want to just check out the org-mode source link at the bottom of the post to see all the details.

import sys
import numpy as np
import matplotlib.pyplot as plt

from argparse import ArgumentParser

def utility_func(arg=None):
    return 'you called a utility function with this arg: {0}'.format(arg)

class HelloWorld(object):
    def __init__(self, who):
        self.who = who

    def __call__(self):
        return 'Hello {0}'.format(self.who)

    def test(self):
        return True

def main():
    parser = ArgumentParser(description="Say hi")
    parser.add_argument("-w", "--who", 
                        type=str,
                        default="world",
                        help="Who to say hello to")
    args = parser.parse_args()
  
    who = args.who
  
    greeter = HelloWorld(who)
    greeter()

    print 'test func = ', greeter.test()
  
    print utility_func()
    print utility_func(5)

    return 0

if __name__ == '__main__':
    status = main()
    sys.exit(status)

1 imports

Now, we define a block that gives us the imports. We do not have to use any tangle headers here because noweb will put it in where it belongs.

import sys
import numpy as np
import matplotlib.pyplot as plt

from argparse import ArgumentParser

2 utility function

Now we define a function we will want imported from the main file.

def utility_func(arg=None):
    return 'you called a utility function with this arg: {0}'.format(arg)

3 class definition

Finally, let us define a class. Note we use noweb here too, and we get the indentation correct!

class HelloWorld(object):
    def __init__(self, who):
        self.who = who

    def __call__(self):
        return 'Hello {0}'.format(self.who)

    def test(self):
        return True

3.1 some class function

Now, let us make the some-other-func. This block is not indented, but with the noweb syntax above, it seems to get correctly indented. Amazing.

def test(self):
    return True

4 The main function

This is a typical function that could be used to make your module into a script, and is only run when the module is used as a script..

def main():
    parser = ArgumentParser(description="Say hi")
    parser.add_argument("-w", "--who", 
                        type=str,
                        default="world",
                        help="Who to say hello to")
    args = parser.parse_args()
  
    who = args.who
  
    greeter = HelloWorld(who)
    greeter()

    print 'test func = ', greeter.test()
  
    print utility_func()
    print utility_func(5)

    return 0

5 Tangle and run the code

This link will extract the code to main.py:

elisp:org-babel-tangle

We can run the code like this (linux):

python main.py --w John 2>&1
true
test func =  True
you called a utility function with this arg: None
you called a utility function with this arg: 5

or this (windows, which as no sh)

from main import *

main()
test func =  True
you called a utility function with this arg: None
you called a utility function with this arg: 5

6 Summary thoughts

The use of noweb syntax is pretty cool. I have not done anything serious with it, but it looks like you could pretty easily create a sophisticated python module this way that is documented in org-mode.

Copyright (C) 2014 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.5h

Discuss on Twitter
« Previous Page -- Next Page »