Finding bibtex entries with no downloaded pdf

| categories: bibtex | tags:

We use bibtex for bibiliography management in our group. Almost every journal provides a utility to download bibtex entries, and you can pretty easily download bibtex entries from citeulike. It doesn't take too long though before you have a few hundred entries. You need some tools to interact with that database.

Bibtex-mode in Emacs provides some tools for working with your bibtex files. For example, you can (bibtex-validate) to check if entries are correct, and (bibtex-sort-buffer) to sort them by key.

I have a specific workflow to entering new entries. This is what I prefer to do:

  1. Go to journal, get bibtex entry, paste into bibtex file.
  2. delete the key that is used, if any
  3. type C-c C-c to autogenerate a key of my style
  4. Copy the key, download the pdf, and save the pdf as (format "%s.pdf" key) in my pdfs directory.
  5. Make an entry in a notes file for that reference. These entries are initially tagged as TODO to remind me to organize them.

Doing this has some payoffs; my org-mode cite links can open either the bibtex entry, or the pdf file directly from the org-file! The notes file is also an org-file, which I can organize as I see fit.

Sometimes I am lazy, and do not get all these steps done, especially the pdf download step. I like to have local copies of the pdf files so I can read them even if I am offline, and because I often annotate them using a tablet PC. It also makes it easy to send them to my students if I need to. Periodically, I like to go through my bibtex database to do some maintenance, download missing files, and notes entries etc… The problem is how do I know which entries have downloaded pdfs or note entries? It is not that difficult with a bit of elisp.

(find-file "~/Dropbox/bibliography/references.bib")
(bibtex-map-entries (lambda (bibtex-key start end) 
                      (let ((type  (cdr (car (bibtex-parse-entry)))))                        
                        (unless (file-exists-p 
                                 (format "~/Dropbox/bibliography/bibtex-pdfs/%s.pdf" bibtex-key))
                          (princ (format "%10s:  cite:%s has no pdf\n" type bibtex-key))))))
      Book:  cite:ambrose-2010-how-learn-works has no pdf
   article:  cite:gerken-2010-fluor-modul has no pdf
      Book:  cite:gray-1973-chemic-bonds has no pdf
   ARTICLE:  cite:kitchin-2003-tio2 has no pdf
   ARTICLE:  cite:kitchin-2012-prefac has no pdf
      Book:  cite:kittel-2005-introd-solid has no pdf
   ARTICLE:  cite:mccormick-2003-tio2-pd has no pdf
   ARTICLE:  cite:mhadeshwar-2004-nh3-ru has no pdf
      Misc:  cite:ni-website has no pdf
   ARTICLE:  cite:norskov-2006-respon has no pdf
      Book:  cite:reif-1965-fundam-statis has no pdf
   article:  cite:risch-2012-water-oxidat has no pdf
   ARTICLE:  cite:shultz-1995-prepar-and has no pdf
   ARTICLE:  cite:shultz-1997-prepar has no pdf
   ARTICLE:  cite:song-2002-h3pw1 has no pdf

Using that list, I can click on those links, which takes me to the entry in file. That entry probably has a url or doi that makes it easy to navigate to the journal page where I can download the pdf file. You could improve on the code above by filtering out only articles, for example.

Copyright (C) 2014 by John Kitchin. See the License for information about copying.

org-mode source

Discuss on Twitter

Finding bibtex entries with non-ascii characters

| categories: bibtex | tags:

I have found that some journals cannot handle bibtex entries with non-ascii characters in them. Unfortunately, when you paste bibtex entries into your reference file from the web, there are often non-ascii characters in them. Emacs usually shows those characters just fine, so it is difficult to find them. Here is a little recipe to go through each entry to find entries with non-ascii characters. These range from accented characters, greek letters, degree symbols, dashes, fancy quotes, etc… Since they are hard to see by eye, we can let Emacs find them for us, and then replace them with the corresponding ascii LaTeX commands.

I found a function to find non-ascii characters here: http://www.emacswiki.org/emacs/FindingNonAsciiCharacters . Now, we use a modified version of this on each entry in a bibtex file. If we find a character, we will print an org-mode link to make it easy to get right to the entry.

(defun contains-non-ascii-char-p ()
  "tests if buffer contains non-ascii character"
  (interactive)
  (let (point)
    (save-excursion
      (setq point
            (catch 'non-ascii
              (while (not (eobp))
                (or (eq (char-charset (following-char))
                        'ascii)
                    (throw 'non-ascii (point)))
                (forward-char 1)))))
    (if point
        (goto-char point)
      nil)))


(find-file "~/Dropbox/bibliography/references.bib")
(bibtex-map-entries (lambda (bibtex-key start end)                        
                      (save-restriction
                        ;; narrow so we only look at this entry. save-restriction will rewiden
                        (bibtex-narrow-to-entry)
                        (when (contains-non-ascii-char-p) (princ (format "cite:%s" bibtex-key)))))))
cite:suntivich-2011-perov-oxide

You can see I only had one reference in that file with a non-ascii character. I think it is best practice to replace these with pure LaTeX commands. See http://en.wikibooks.org/wiki/LaTeX/Special_Characters for a good reference on what commands are used for the accented characters.

Copyright (C) 2014 by John Kitchin. See the License for information about copying.

org-mode source

Discuss on Twitter

A function to save an org-file for submitting assignments in org-mode

| categories: org-mode | tags:

I am going to be assigning homeworks and quizzes in org-mode next semester, and it would be nice if there was a function that would save the file with the right name in the right place. The file will have some keywords at the top to identify the student, course and assignment. Those look like:

#+NAME: John Kitchin
#+ANDREWID: jkitchin
#+COURSE: 06-640
#+ASSIGNMENT: quiz1

I would like a function that saves the current buffer to a file like /some-path/ANDREWID-COURSE-ASSIGNMENT.org. That will be "turning the assignment in" so I can do some analysis of the files. I previously wrote about getting keywords from org-files here . We will build on that to accomplish this goal. Here is the function that seems to do what I want. For now it saves the file in the same directory, with the correct name.

(defun submit-assignment ()
  "Save current buffer to assignment. You must set #+ANDREWID, #+COURSE and #+ASSIGNMENT. There is no checking for this."
  (interactive)
  (save-buffer)
  (let ((org-keywords (org-element-map (org-element-parse-buffer 'element) 'keyword
                        (lambda (keyword) (cons (org-element-property :key keyword)
                                                (org-element-property :value keyword)))))
        (assignment-content (buffer-string)) ; store contents to write to file later
        (assignment-filename))
    ;; create the filename we will save the buffer in
    (setq assignment-filename (concat (cdr (assoc "ANDREWID" org-keywords))
                                      "-"
                                      (cdr (assoc "COURSE" org-keywords))
                                      "-"
                                      (cdr (assoc "ASSIGNMENT" org-keywords))
                                      ".org"))
    ;; now write out contents to filename
    (with-temp-file assignment-filename
      (insert assignment-content))
    (message "%s was submitted" assignment-filename)))

(submit-assignment)
jkitchin-06-640-quiz1.org was submitted

Now, we have a nice function that can be put as a link in the org-file:

elisp:submit-assignment
, or
elisp:submit-assignment
. The assignment will be copied to a location specified, and I can grade it, assess it, etc… from there. That function will have to be loaded by students, but since I provide them with lisp files that are loaded at startup that should not be an issue.

This solution lacks some features I have to think about. For example, if the path where everyone submits to is common, everyone will be able to read the file, which is not desirable. It is also not feasible to make the submitted files read-only to the students this way (so they cannot modify the files after they turn them in). I could have the submit function do something like submit to box.com by email.

I am not sure how grading of these files would go. In the past, I have preferred PDF files (which you can export org to) because I could grade them using a tablet without modifying the actual content in the files. However it would not be simple at all to export an org-file to PDF with that data in a machine readable way. One point of having them in org-mode is that I could collect statistics from the headings of many files pretty easily. I have written code to annotate org-files in the past, which I might have to dig out.

There is a limitation of the submit function, and that is if the org-file references images or other files (I would expect many of my assignments would have plots in them). Those would not be copied to the right place. I handle those when I publish the blog, so it could be done. But, it would require creating a new directory, and writing all the files to it (including making sure the relative paths are handled correctly), or creating some kind of zip-archive that contains everything. That sounds like a project for another day!

TODO rewrite the submit-assignment function to copy any images or files it links to into the submission directory
nil

Copyright (C) 2014 by John Kitchin. See the License for information about copying.

org-mode source

Discuss on Twitter

Interactive quizzes with feedback in org-mode

| categories: org-mode | tags:

I have been exploring ways to facilitate learning and grading. One way I like is to provide quizzes inline with text Here are some examples of "interactive" quizzes with feedback to the user in org-mode. The principle idea is to use a headline for the question, and use properties to hold the answer. The answers are coded into links that run lisp code to check if the link text matches the correct answer. The properties are folded, so you have to willfully cheat to look at the answer. For getting instant feedback, this isn't such an issue. These aren't for formal assessment, but for knowledge checkpoints in active learning.

Blog alert: The links may only work in the actual org-file in Emacs, they will not function in html.

Here is an example in a subheading. These links play an amusing sound (crowd cheers for correct answer, a buzzer for a wrong answer). The sounds are stored locally with the file. This gives students immediate feedback on their answer. These might be incorporated into reading assignments.

1 What is 2 + 4?

a)

elisp:(if (string= "4" (org-entry-get (point) "CORRECT-ANSWER"))  (play-sound-file "sounds/99636__tomlija__small-crowd-yelling-yeah.wav" 0.91 nil) (play-sound-file "sounds/fail-buzzer-01.wav" 0.91 nil))

b)

elisp:(if (string= "6" (org-entry-get (point) "CORRECT-ANSWER"))  (play-sound-file "sounds/99636__tomlija__small-crowd-yelling-yeah.wav" 0.91 nil) (play-sound-file "sounds/fail-buzzer-01.wav" 0.91 nil))

c)

elisp:(if (string= "-2" (org-entry-get (point) "CORRECT-ANSWER"))  (play-sound-file "sounds/99636__tomlija__small-crowd-yelling-yeah.wav" 0.91 nil) (play-sound-file "sounds/fail-buzzer-01.wav" 0.91 nil))

2 Lets make a custom link for this

Hand coding the links isn't very convenient. Let us write an org-link function that is more convenient. This new function will also record the number of correct and incorrect attempts. You could hard code some sound into them, but I did not do that here.

(org-add-link-type 
 "ans" 
 (lambda (path) 
   (let* ((correct-answer (org-entry-get (point) "CORRECT-ANSWER"))
          (ncorrect (org-entry-get (point) "NUM-CORRECT"))
          (num-correct (if ncorrect (string-to-number ncorrect) 0))
          (nincorrect (org-entry-get (point) "NUM-INCORRECT"))
          (num-incorrect (if nincorrect (string-to-number nincorrect) 0)))
     (if (string= path correct-answer) 
         (progn
          (org-entry-put (point) "NUM-CORRECT" (number-to-string (+ num-correct 1)))
          (message "correct"))
       (org-entry-put (point) "NUM-INCORRECT" (number-to-string (+ num-incorrect 1)))
       (message "incorrect")))))

That function was a bit trickier than I thought it would be. First, I used let* because the first time you click on a link the correct answer counters may not exist, so getting the property would return nil. If it is nil, then we should initialize to 0. If it is not nil, we need to convert the string that is returned to an integer so we can add to it later. Then, we need to convert that integer back to a string to store it as a property, otherwise it is stored as some strange control character.

Now, we can put our question in. question: Is 3 greater than 2?

Here are the new links:

ans:yes
ans:no

That function goes along way to making it pretty easy to write the answers. One limitation here is that you can only put one question per section. We address that later. It is still easy to cheat if you want. We could do a few things to make that harder. One idea would be to encrypt the answers, and compare the hash of the answers. Another is to just record what was clicked on, and then grade it later. In the next section we try that out.

3 Storing quiz answers and number of attempts

Here, we store the answer, and number of attempts. Students won't get feedback from this approach, but they cannot cheat either. They could just type answers in to the org-file, but this approach is like a Scantron multiple choice exam that could be automatically graded later. You could use something like this as an electronic clicker to quickly assess everyone's understanding of some concepts.

Here is our new link function.

(org-add-link-type 
 "ans2" 
 (lambda (path) 
   (org-entry-put (point) "ANSWER" path)
   (let* ((nattempts (org-entry-get (point) "NUM-ATTEMPTS"))
          (num-attempts (if nattempts (string-to-number nattempts) 0)))
     (org-entry-put (point) "NUM-ATTEMPTS" (number-to-string (+ num-attempts 1))))
   (message "You selected %s for your answer" path)))

And now a question: Is 3 greater than 2?

ans2:yes
ans2:no

This way works very well. It might also be a good idea to give each question a unique id (org-id-get-create) so you can collect specific results.

4 Using inline tasks instead of headings

If you were integrating these into a book, it would be inconvenient to use headings because they would mess up the outline. We can use inline tasks instead.

Is 3 greater than 2?

ans2:yes
ans2:no

This adds some additional lines, but does not change the heading level. We can put multiple questions in then.

What is 4*4?

ans2:16
ans2:18
ans2:8

Interesing, I noticed in this last example that a link path must have at least two characters to be automatically recognized by org-mode. I had to put double square brackets to make it a link!

5 Concluding thoughts

This seems like a pretty promising approach to getting structured input from students. You would have to invest some time training them to use emacs and org-mode, and not modify the file in ways they are not supposed to. You would also have to invest some time in writing analysis code to collect all the answers and analyze them somehow. I do not think that would be too hard though, and the payoff could be large. By giving students quick feedback they could have a better understanding of what they know, and might ask better questions. This could be a useful approach to active learning activities that students work on while reading.

Copyright (C) 2014 by John Kitchin. See the License for information about copying.

org-mode source

Discuss on Twitter

Python as alternative to Matlab for engineering calculations

| categories: python | tags:

For the past year I have been seriously exploring whether Python could be used as a practical alternative to Matlab in engineering calculations, particularly in chemical engineering undergraduate and graduate courses. Matlab is very well suited for these calculations, and I have used it extensively in teaching in the past. For example, there is my Matlab blog (http://matlab.cheme.cmu.edu ), and my cmu Matlab package that contains a very nice units package (https://github.com/jkitchin/matlab-cmu ). Matlab is widely used and recognized as a standard software package in engineering. My university has a site license for Matlab, so it doesn't cost me or my students anything to use. Matlab is easy to install, and has almost everything we need out of the box. So why try using something else then? Here are the main reasons:

  1. Not everyone has access to a "free" Matlab license, and Matlab may not be available to my students when they leave the University. Python offers a free, always available option to them.
  2. There are several recent Python distributions that are easy to install, and contain almost everything we need for engineering calculations.
  3. I have been increasingly integrating code into my lecture notes, and this is not easy with Matlab, but it is easy with Python.
  4. I use Python exclusively in my research, and although Matlab and Python are similar, they are different enough that switching between them is bothersome to me. I do not like teaching students to use tools I do not regularly use, and I believe I can provide them with more value by teaching with tools I have a high level of proficiency in.

1 The significance of an open-source alternative

Many people will be able to use Matlab or some other proprietary software that someone has paid for the license to use. Some people, however, will not have that option for a variety of reasons. Maybe the company they work for will not pay for the license, maybe they are unemployed, or entrepeneurs in a small startup that cannot afford it, maybe they are students at a University without a site license,… For these people Python is a viable option that is always available. That makes me happy.

2 Easy to install Python distributions

An important development in using Python as an alternative to Matlab is the development of many "one-click" installers. Ten years ago it took me about 2 weeks to download and build a Python environment suitable for scientific and engineering calculations. That has kept me from trying to use Python in teaching in the past. Today, I can download a package and install one in about 10 minutes! More importantly, so can my students.

My favorite distribution is the Enthought Canopy distribution (https://www.enthought.com/products/canopy/ ). This distribution comes with all the essential python modules, and an integrated editor with IPython. It is available for Windows, Macs and Linux. They offer free academic licenses.

Another good alternative is the Anaconda distribution (https://store.continuum.io/cshop/anaconda/ ) by Continuum Analytics. It is also available for Windows, Macs and Linux. I have not used this one, but it looks like it would be very good. Anaconda comes with the Spyder editor. They offer free academic licenses.

Python(x,y) is available for Windows (http://code.google.com/p/pythonxy/ ) and comes with the Spyder editor.

WinPython (http://winpython.sourceforge.net/ ) is also available for Windows, and comes with the Spyder editor.

The point here is that there are many options available now that make installing a Python distribution as easy as installing packages like Matlab. Enthought Canopy also provides a "desktop environment" similar to Matlab with an editor, documentation browser, package manager and console that is pretty easy to use.

3 Python + numpy/scipy/matplotlib does almost everything you need

Python by itself is not suitable for typical engineering calculations. You need the numerical, scientific and plotting libraries that provide that functionality. These are provided in numpy, scipy and matplotlib, which are included in the distributions described above.

Typical chemical engineering calculations involve one or more of the following kinds of math problems:

All of these are doable out of the box with the Python distributions discussed above. You can find many examples of using these, and more on my PYCSE blog (http://kitchingroup.cheme.cmu.edu ) and http://kitchingroup.cheme.cmu.edu/pycse/ . In short, almost every example I put in the Matlab blog has been done in Python. The only ones I did not do yet are some of the interactive graphics with the steam tables. I have not had time to work those out in detail.

I have found it convenient to augment theses with a package I wrote called pycse (https://github.com/jkitchin/pycse ).

  • an ode integrator with events similar to the one in Matlab
  • some numerical differentiation functions
  • linear and nonlinear regression with confidence intervals
  • some boundary value problem solvers
  • a publish function to convert python scripts to PDF via LaTeX

This package is still a work in progress. Notably, there is not a really good units package in Python that works as well as my Matlab units package does. Two that come close are quantities and pint . Both have some nuances that make them tricky for regular use, and both have some challenges in covering all the functions you might want to use them for.

4 Python from the educator perspective

Make that my perspective. I have developed an approach to using code in my lectures where I use the code to reinforce the structure of the problems, and to analyze the solutions that result. Doing that means I need to have code to show students, and the output, and sometimes to run the code to illustrate something. I also like these examples integrated into my lecture notes, so they have the right context around them.

I have found that Emacs+org-mode+python allows me to easily integrate notes, equations, images, code and output in one place, and then export it to a PDF which I can annotate in class. This ensures that the code and output stay synchronized, that the code is always right where it needs to be, in the right context, and that I can annotate actual code in class, and not pseudocode. This heavily influenced my decision to use Python because it leverages what I already know and want to do. In fact, using it makes me even better at what I already know and helps me learn more about it. That makes me happy!

Not everyone will be a content developer like this, but that is what I like to do. Python makes that process fun, and worth doing for me.

5 Final thoughts

In my opinion Python is and is becoming a more viable alternative to other packages like Matlab for scientific and engineering calculations. I have used it exclusively for about a year solving all kinds of engineering problems that I used to solve in Matlab.

Python is different, for sure. The main differences in my opinion are:

  • Python is less consistent in syntax than Matlab. For example, there are two ODE solvers in scipy with incompatible syntax. That is a result of the fact that you install a Python distribution made of packages written by many different people with different needs.
  • There is duplicated functionality between numpy and scipy.
  • Some functionality in scipy is provided by external "scikits" (http://scikits.appspot.com/ ).
  • Support for boundary value problems and partial differential equations is not as good in Python as it is in Matlab 1. At the undergraduate level, this is not a big deal. It is not like the Matlab functions are that easy to use!
  • Data regression in Python is not as complete as in Matlab.
  • indexing in Python starts at 0, and uses [], whereas in Matlab it starts at 1 and uses ()
  • You have to import most functions into Python. In contrast, Matlab has them all in one big namespace.

It is certainly doable to use Python for many scientific and engineering calculations. This past Fall I took the plunge, and taught a whole core course in chemical reaction engineering using Python! It was a Master's level course with 59 graduate students in it. I have also taught a graduate elective course in Molecular Simulation using Python. I still have some polishing to do before I would teach this to undergraduates, but I think it is definitely worth trying!

Footnotes:

1

It is true there are packages like FiPy (nil) for PDEs, and scikits for BVPS, (nil, nil). But these are not easily installed on all platforms, and typically require some developer experience in compiling.

Copyright (C) 2013 by John Kitchin. See the License for information about copying.

org-mode source

Discuss on Twitter
« Previous Page -- Next Page »