The Kitchin Research Group: orgmode

Find the most recent timestamp that is not in the future

Posted August 17, 2014 at 09:42 AM | categories: orgmode | tags:

In a previous post, we looked at a way to jump to a timestamp (post ). A limitation of that post was that our definition of "most recent" included timestamps in the future. Here we further constrain the definition to mean the most recent that is not in the future.

We will do that by getting the list of timestamps, then filtering out future ones.

Here is a list of time-stamps

[2014-08-17 Sun]
<2014-08-17 Sun 08:23>
<2014-08-17 Sun 09:23>
[2014-08-18 Mon]
[2014-08-17 Sun 13:00]
[2014-08-16 Sat 13:00]

Here is the code that filters out future timestamps. As I write this it is <2014-08-17 Sun 08:31>.

(let ((now (with-temp-buffer
             ;; sachac suggested this way to get a timestamp of "now"
             (org-insert-time-stamp (org-read-date t t ".") t)
             (buffer-string))))
  (remove-if (lambda (entry) (org-time> (car entry) now))
             (cl-sort
              (org-element-map (org-element-parse-buffer) 'timestamp
                (lambda (timestamp)
                  (cons (org-element-property :raw-value timestamp) (org-element-property :begin timestamp))))
              'org-time> :key 'car)))

((<2014-08-17 Sun 08:31> . 811) (<2014-08-17 Sun 08:23> . 613) ([2014-08-17 Sun] . 593) ([2014-08-16 Sat 13:00] . 712))

You can see that only past timestamps show up in the list, and they are sorted with the most recent timestamp. The remove-if function is in 'cl, which is loaded in my init files. You may have to require that if you don't load it in your init files.

org-mode source

Org-mode version = 8.2.7c

Discuss on Twitter

Jump to most recent timestamp in an org document

Posted August 14, 2014 at 11:45 AM | categories: orgmode | tags:

Timestamps serve many purposes in an org document: deadlines, scheduled time, record of time, etc… Depending on the document, you may have organized a set of tasks logically, but you are completing them in some other order. Here, we examine how to jump to the most recent timestamp.

Here is an example of an inactive (will not cause an agenda entry) timestamp [2014-08-07 Thu] and an active timestamp <2014-08-13 Wed>. You can also have times in the timestamps, e.g. <2014-08-14 Thu 13:00> or <2014-08-14 Thu 14:00>. There may be a timestamp that is in the future [2014-08-15 Fri].

We may have some headlines with a due date.

1 Headline with duedate

2 Analyzing the timestamps in this file

We can get a sorted list of the time-stamps like this.

(sort (org-element-map (org-element-parse-buffer) 'timestamp
	(lambda (timestamp)
	  (org-element-property :raw-value timestamp))) 'org-time>)

[2014-08-15 Fri]

<2014-08-14 Thu 14:00>

<2014-08-14 Thu 13:00>

<2014-08-13 Wed>

[2014-08-07 Thu]

Interestingly, the deadline timestamp does not appear in this list! I am not sure why it doesn't show up.

Unfortunately, having those time stamps in this buffer will complicate further analysis, because org-mode will parse them too. What we need to do next is figure out a way to sort with the positions, so we can jump to it.

Side bar: How to sort a cons list based on the first element of each cons cell? Here are two examples.

(cl-sort '((3 . "a") (2 . "b") (1 . "c")) '> :key 'car)

((3 . a) (2 . b) (1 . c))

(sort '((3 . "a") (2 . "b") (1 . "c"))
      (lambda (a b)
	(> (car a) (car b))))

((3 . a) (2 . b) (1 . c))

So, we just build up a sorted cons-list of timestamps and their beginning characters, then pop the top entry off and jump to it. Here is the code. Running this jumps directly to the most recent (which could be in the future) timestamp.

(let ((timestamps (cl-sort
		   (org-element-map (org-element-parse-buffer) 'timestamp
		     (lambda (timestamp)
		       `(,(org-element-property :raw-value timestamp) . ,(org-element-property :begin timestamp))))
		   'org-time> :key 'car)))
  (goto-char
   (cdr
    (pop timestamps))))

There are some limitations here:

Only free timestamps are shown, timestamps associated with deadlines and scheduling do not appear to be considered here. You would need to map over the headlines and check for deadlines to get these I think.
The code block above finds the timestamp with the highest time value, which may be in the future. It is an interesting exercise to think about how to find the most recent timestamp that is not in the future.

org-mode source

Org-mode version = 8.2.7c

Discuss on Twitter

Using org-mode outside of Emacs - sort of

Posted August 11, 2014 at 08:22 PM | categories: orgmode, emacs | tags:

1. Extracting tables from an org-file
2. Running code in an org-file

I recently posted about using Emacs for scripts (http://kitchingroup.cheme.cmu.edu/blog/2014/08/06/Writing-scripts-in-Emacs-lisp/ ). Someone was probably wondering, why would you do that, when you could use shell, python or perl? A good reason is to write scripts that can access data or code inside an org-file! This would allow you to leverage the extensive support for org-mode in Emacs, without a user necessarily even needing to use Emacs. Let us consider some examples.

1 Extracting tables from an org-file

If tables are named in org-mode, it is possible to extract the contents. Here is a table:

x	y
1	1
2	4
3	9

Another table might look like

a	b
1	1
2	8
3	27

It would be convenient to have a command-line utility that could extract the data from that table with a syntax like:

extract-org-table tblname orgfile --format lisp|csv|tab

Here is one way to do it:

;; org-table tblname orgfile lisp|csv|tab

(let ((tblname (pop command-line-args-left))
      (org-file (pop command-line-args-left))
      (format)
      (table)
      (content))
  (when command-line-args-left
    (setq format (pop command-line-args-left)))
  (find-file org-file)
  (setq table 
	(org-element-map (org-element-parse-buffer) 'table 
	  (lambda (element)
	    (when (string= tblname (org-element-property :name element))
	      element))
	  nil ;info
	  t )) ; first-match

  (unless table
    (error "no table found for %s" tblname))

  (when table
    (goto-char (org-element-property :contents-begin table))
    (let ((contents (org-table-to-lisp)))
      (if (string= format "lisp")
	  (print contents)
	;else      
	(dolist (row contents)
	  (unless (eq row 'hline)
	    (cond
	     ((string= format "csv")
	      (princ (mapconcat 'identity row ",")))
	     ((string= format "tab")
	      (princ (mapconcat 'identity row "\t")))
	     (t
	      (error "unsupported format: %s" format)))
	    (princ "\n")))))))

Let us try it out. org-babel-tangle

./extract-org-table data-2 org-outside-emacs.org lisp

(("a" "b") ("1" "1") ("2" "8") ("3" "27"))

./extract-org-table data-1 org-outside-emacs.org csv

x,y
1,1
2,4
3,9

./extract-org-table data-2 org-outside-emacs.org tab

That looks pretty reasonable, and you could even pipe the output to another classic unix command like cut to get a single column. Let us get the second column here.

./extract-org-table data-1 org-outside-emacs.org csv | cut -d , -f 2

That is starting to look like using data from an org-file, but outside of org. Of course, we are using org-mode, via emacs, but the point is a user might not have to know that, as long as a fairly recent Emacs and org-mode was installed on their system.

2 Running code in an org-file

It may be that there is code in an org-file that you might want to use, but for some reason choose not to cut and paste from the org-file to some script. Here is a simple code block:

import time
with open('results.dat', 'w') as f:
    f.write(time.asctime())

To call this externally we have to find the block and then run it.

;; org-run blockname org-file
;; run a code block in an org file
(let ((blockname (pop command-line-args-left))
      (org-file (pop command-line-args-left))
      (src))
  (find-file org-file)
  (setq src
	(org-element-map (org-element-parse-buffer) 'src-block
	  (lambda (element)
	    (when (string= blockname (org-element-property :name element))
	      element))
	  nil ;info
	  t )) ; first-match
  (when src
     (goto-char (org-element-property :begin src))
     ;; since we start with a fresh emacs, we have to configure some things.
     (org-babel-do-load-languages
      'org-babel-load-languages
      '((python . t)))
     (let ((org-confirm-babel-evaluate nil))
       (org-babel-execute-src-block))))

org-babel-tangle

./org-call.el python-block org-outside-emacs.org
cat results.dat

Mon Aug 11 20:17:01 2014

That demonstrates it is possible to call source blocks, but this is pretty limited in capability. You can only call a block; we did not capture any output from the block, only its side effects, e.g. it changed a file that we can examine. We have limited capability to set data into the block, other than through files. It might be possible to hack up something that runs org-babel-execute-src-block with constructed arguments that enables something like a var to be passed in. That is beyond today's post. When I get around to it, here is a reminder of how it might be possible to feed stdin to an emacs script: http://stackoverflow.com/questions/2879746/idomatic-batch-processing-of-text-in-emacs .

org-mode source

Org-mode version = 8.2.6

Discuss on Twitter

What we are using org-mode for

Posted August 08, 2014 at 01:59 PM | categories: orgmode | tags:

About four years ago I picked up Emacs again, because of org-mode. It started as an experiment to see if org-mode might be a solution to managing my growing list of obligations. Since then I have learned alot about org-mode. I was feeling reflective today, so here I describe a few of the things we are doing with it now, besides just organizing my obligations! I have found org-mode useful for many other things.

1 This blog

Every entry of this blog is written in org-mode and exported to an HTML file (https://github.com/jkitchin/jmax/blob/master/user/blogofile.el ) that is compiled by blogofile (http://www.blogofile.com/ ). Thanks to some emacs-lisp code, each post has the raw org-file linked to it so you can see exactly how the page was made.

2 Writing large technical documents/books

I used org-mode to write my book on modeling materials properties using density functional theory (http://kitchingroup.cheme.cmu.edu/dft-book ). This book has over 100 Python code examples that are integrated with the narrative text, including equations, figures and references.

I also used org-mode to write "Python computations in Science and Engineering" (http://kitchingroup.cheme.cmu.edu/pycse ), which shows that Python can be used extensively for solving scientific and engineering problems.

These "books" are not published in any traditional way; their value is in the org-format where the code is executable.

3 Writing scientific publications

The following papers from our group were written in org-mode. The supporting information files all contain embedded org-files with all of the code used to generate figures and analysis, as well as the data to make the papers reproducible and reusable. These papers include both experimental and computational work.

Spencer D. Miller, Vladimir V. Pushkarev, Andrew J. Gellman and John R. Kitchin, Simulating Temperature Programmed Desorption of Oxygen on Pt(111) Using DFT Derived Coverage Dependent Desorption Barriers, Topics In Catalysis, 57(1), 106-117 (2013). http://link.springer.com/article/10.1007%2Fs11244-013-0166-3
Alex Hallenbeck and John R. Kitchin, Effects of O₂ and SO₂ on the Capture Capacity of a Primary-Amine Based Polymeric CO2 Sorbent, I&ECR, 52 (31), 10788-10794 (2013). https://doi.org/10.1021/ie400582a .
Mehta, Prateek; Salvador, Paul; Kitchin, John, Identifying Potential BO₂ Oxide Polymorphs for Epitaxial Growth Candidates", ACS Applied Materials and Interfaces, 6(5), 3630-3639 (2014). http://pubs.acs.org/doi/full/10.1021/am4059149 .
Zhongnan Xu and John R Kitchin, Relating the Electronic Structure and Reactivity of the 3d Transition Metal Monoxide Surfaces, Catalysis Communications, 52, 60-64 (2014), https://doi.org/10.1016/j.catcom.2013.10.028 .

We have several more papers in progress that are being written in org-mode. We just export the LaTeX manuscript (https://github.com/jkitchin/jmax/blob/master/ox-manuscript.el ) and submit it. My students now use org-mode to produce reports for their PhD qualifying exams, MS project reports, and other internal reports I ask them to write. I have used it for writing internal memos and reports for the Department. I occasionally use it to write technical emails.

4 Writing software

I have used org-mode in three serious software development efforts. So far, all of them have been in emacs-lisp.

org-ref is an extension of org-mode that provides deeply integrated citation and referencing in org-mode documents using bibtex as the reference database. The software is written in emacs-lisp, in an org-file, and the source code is tangled out of it. See https://github.com/jkitchin/jmax/blob/master/org/org-ref.org . The code is about 1700 lines long.
doi-utils is a smaller package that allows you to download bibtex entries and pdfs from a DOI. See https://github.com/jkitchin/jmax/blob/master/org/doi-utils.org
org-show is an emacs-lisp package for using org-files to give presentations. See https://github.com/jkitchin/jmax/blob/master/org/org-show.org .

This literate programming approach to writing software is interesting. You can document via narrative text what the software does, and how it works, sometimes as you write the code. I have not used org-mode to write a large Python project yet. I am considering it.

5 Giving presentations

For some examples of using org-show, see: a. Org-mode is awesome https://www.youtube.com/watch?v=fgizHHd7nOo b. Using org-ref https://www.youtube.com/watch?v=JyvpSVl4_dg c. The org-show show https://www.youtube.com/watch?v=JZ8RK-R9O_g d. Scipy 2014: Python in teaching https://www.youtube.com/watch?v=IsSMs-4GlT8 e. Scipy 2013: reproducible research talk in Beamer generated from org-mode https://www.youtube.com/watch?v=1-dUkyn_fZA

Presenting from org-mode takes some adjustment, but it really lets you strip something down to what is important, with few distractions!

6 Teaching classes

The book on dft that I wrote has been used in a graduate course several times. The last time I taught it we actually used the org-file directly in class during lecture, and I had students turn in their assignments with org-mode. That was done on a computer cluster I administer, so everyone had the emacs/org-mode setup I created.

This fall I am teaching a graduate MS course in chemical reaction engineering. It will be taught from org-mode. All the notes are in org-mode, all the assignments are in org-mode. Emacs will communicate with a git-server I run that will provide authenticated delivery of course materials and assignments. Grading is all done through org-mode, with the grades stored directly in the assignment files. The source code behind this effort is located for now at https://github.com/jkitchin/jmax/tree/master/techela . There is no course website; students will run M-x techela, select their course, and the syllabus.org file will be updated and shown, with org-links to get assignments, open lecture notes, etc… The students will do all this on their own laptop, with a preinstalled version of https://github.com/jkitchin/jmax , Enthought Canopy Python, git, ssh and TeXLive.

I am needless to say, very excited to try it!

7 Managing obligations

And of course, I still use org-mode to manage my daily job responsibilities as a professor:

Running a research group of ~10 PhD and MS students, managing grants, paper deadlines,… I share org-files with my students through Dropbox to keep some activities coordinated.
Teaching classes
Department, college, university committee duties
Preparing for professional meetings
Conference organization
Keeping track of personal things
etc…

I just counted the lines of the files in my org-agenda:

wc -l ~/Dropbox/org-mode/*.org

    346 /c/Users/jkitchin/Dropbox/org-mode/abet.org
     28 /c/Users/jkitchin/Dropbox/org-mode/advising.org
    854 /c/Users/jkitchin/Dropbox/org-mode/cheme-undergrad-committee.org
    745 /c/Users/jkitchin/Dropbox/org-mode/committees.org
  11187 /c/Users/jkitchin/Dropbox/org-mode/contacts.org
    451 /c/Users/jkitchin/Dropbox/org-mode/department.org
      2 /c/Users/jkitchin/Dropbox/org-mode/email+gnus.org
     91 /c/Users/jkitchin/Dropbox/org-mode/email.org
    611 /c/Users/jkitchin/Dropbox/org-mode/gilgamesh.org
     91 /c/Users/jkitchin/Dropbox/org-mode/gnus.org
      6 /c/Users/jkitchin/Dropbox/org-mode/group.org
    749 /c/Users/jkitchin/Dropbox/org-mode/journal.org
    274 /c/Users/jkitchin/Dropbox/org-mode/lab.org
    352 /c/Users/jkitchin/Dropbox/org-mode/manuscripts.org
    117 /c/Users/jkitchin/Dropbox/org-mode/master.org
   1239 /c/Users/jkitchin/Dropbox/org-mode/meetings.org
    513 /c/Users/jkitchin/Dropbox/org-mode/notes.org
    118 /c/Users/jkitchin/Dropbox/org-mode/org-course.org
    182 /c/Users/jkitchin/Dropbox/org-mode/personal.org
    485 /c/Users/jkitchin/Dropbox/org-mode/prj-doe-early-career.org
    109 /c/Users/jkitchin/Dropbox/org-mode/prj-exxon.org
     25 /c/Users/jkitchin/Dropbox/org-mode/prj-gcep-project.org
     15 /c/Users/jkitchin/Dropbox/org-mode/prj-simonseed.org
     83 /c/Users/jkitchin/Dropbox/org-mode/prj-task-191.org
      8 /c/Users/jkitchin/Dropbox/org-mode/prj-task-203.org
    349 /c/Users/jkitchin/Dropbox/org-mode/prj-task-97-mat.org
    238 /c/Users/jkitchin/Dropbox/org-mode/proposals.org
    378 /c/Users/jkitchin/Dropbox/org-mode/research.org
      7 /c/Users/jkitchin/Dropbox/org-mode/researcher-ids.org
    110 /c/Users/jkitchin/Dropbox/org-mode/reviews.org
    619 /c/Users/jkitchin/Dropbox/org-mode/software.org
    629 /c/Users/jkitchin/Dropbox/org-mode/tasks.org
      2 /c/Users/jkitchin/Dropbox/org-mode/taxes.org
  21013 total

hmm… that contacts file is big. My agenda might speed up if I move it out. It is not so slow that it bothers me though. That happens when you dump gmail contacts to org-contacts. That is even after cleaning up all the obviously bad entries!

8 Summary

There is hardly an area in my work not touched by org-mode these days!

org-mode source

Org-mode version = 8.2.6

Discuss on Twitter

« Previous Page

The Kitchin Research Group

Chemical Engineering at Carnegie Mellon University

Find the most recent timestamp that is not in the future

Jump to most recent timestamp in an org document

1 Headline with duedate

2 Analyzing the timestamps in this file

Using org-mode outside of Emacs - sort of

Table of Contents

1 Extracting tables from an org-file

2 Running code in an org-file

What we are using org-mode for

1 This blog

2 Writing large technical documents/books

3 Writing scientific publications

4 Writing software

5 Giving presentations

6 Teaching classes

7 Managing obligations

8 Summary