The Kitchin Research Group

It would be helpful to get a listing of org-files in a directory tree in the form of clickable links. This would be useful, for example, to find all files associated with a project in a directory with a particular extension, or to do some action on all files that match a pattern. To do this, we will have to recursively walk through the directories and examine their contents.

Let us examine some of the commands we will need to use. One command is to get the contents of a directory. We will explore the contents of a directory called literate in my computer.

;; list contents of the directory
(let ((abspath nil)
      (match nil)
      (nosort t))
  (directory-files "literate" abspath match nosort))

Note the presence of . and ... Those stand for current directory and one directory up. We should remove those from the list. We can do that like this.

;; remove . and ..
(let ((abspath nil)
      (match nil)
      (nosort t))
  (remove "." 
          (remove ".." 
                  (directory-files "literate" abspath match nosort))))

Next, we need to know if a given entry in the directory files is a file or a directory. Emacs-lisp has a few functions for that. We use absolute filenames here since the paths are relative to the "molecules" directory. Note we could use absolute paths in directory-files, but that makes it hard to remove "." and "..".

;; print types of files in the directory
(let ((root "literate")
      (abspath nil)
      (match nil)
      (nosort t))
  (mapcar (lambda (x)
            (cond
             ((file-directory-p (expand-file-name x root))
              (print (format "%s is a directory" x)))
             ((file-regular-p (expand-file-name x root))
              (print (format "%s is a regular file" x)))))
          (remove "." 
                  (remove ".." 
                          (directory-files root abspath match nosort)))))

"makefile-main is a regular file"

"Makefile is a regular file"

"main.o is a regular file"

"main.f90 is a regular file"

"main is a regular file"

"literate.org is a regular file"

"hello.f90 is a regular file"

"circle.o is a regular file"

"circle.mod is a regular file"

"circle.f90 is a regular file"

"circle-area.png is a regular file"

"archive is a directory"

"a.out is a regular file"

Now, we are at the crux of this problem. We can differentiate between files and directories. For each directory in this directory, we need to recurse into it, and list the contents. There is some code at http://turingmachine.org/bl/2013-05-29-recursively-listing-directories-in-elisp.html which does this, but I found that I had to modify the code to not list directories, and here I want to show a simpler recursive code.

(defun os-walk (root)
  "recursively walks through directories getting list of absolute paths of files"
  (let ((files '()) ; empty list to store results
        (current-list (directory-files root t)))
    ;;process current-list
    (while current-list
      (let ((fn (car current-list))) ; get next entry
        (cond 
         ;; regular files
         ((file-regular-p fn)
          (add-to-list 'files fn))
         ;; directories
         ((and
           (file-directory-p fn)
           ;; ignore . and ..
           (not (string-equal ".." (substring fn -2)))
           (not (string-equal "." (substring fn -1))))
          ;; we have to recurse into this directory
          (setq files (append files (os-walk fn))))
        )
      ;; cut list down by an element
      (setq current-list (cdr current-list)))
      )
    files))

(os-walk "literate")

Nice, that gives us a recursive listing of all the files in this directory tree. Let us take this a step further, and apply a function to that list to filter out a list of the org files. We will also create org-links out of these files.

(defun os-walk (root)
  (let ((files '()) ;empty list to store results
        (current-list (directory-files root t)))
    ;;process current-list
    (while current-list
      (let ((fn (car current-list))) ; get next entry
        (cond 
         ;; regular files
         ((file-regular-p fn)
          (add-to-list 'files fn))
         ;; directories
         ((and
           (file-directory-p fn)
           ;; ignore . and ..
           (not (string-equal ".." (substring fn -2)))
           (not (string-equal "." (substring fn -1))))
          ;; we have to recurse into this directory
          (setq files (append files (os-walk fn))))
        )
      ;; cut list down by an element
      (setq current-list (cdr current-list)))
      )
    files))

(require 'cl)

(mapcar 
 (lambda (x) (princ (format "[[%s][%s]]\n" x (file-relative-name x "."))))
 (remove-if-not 
  (lambda (x) (string= (file-name-extension x) "org"))
  (os-walk "literate")))

literate/literate.org

That is certainly functional. It might be nice to format the links a bit nicer to show their structure in a table of contents way, or to sort them in a nice order if there were many of these files.

Copyright (C) 2014 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.5h

This post illustrates an alternative approach to completion in creating ref links compared to the approach shown here . In this approach we use a dynamic yasnippet to do the completion. We start with similar code that I used before to get a list of labels from the buffer. I used a slightly different regexp to recognize links in this version.

label:code-example

(defun get-labels ()
  (interactive)
  (save-excursion
    (goto-char (point-min))
    (let ((matches '()))
      (while (re-search-forward "label:\\([a-zA-z0-9:-]*\\)" (point-max) t)
        (add-to-list 'matches (match-string-no-properties 1) t))
      matches)))

get-labels

Let us see that in action:

(get-labels)

I think the \\ link is an artifact of the regexp in my get-labels code, and it would not appear in other examples.

Now, we are going to create a yasnippet that uses the list returned from get-labels to provide your choices. See http://capitaomorte.github.io/yasnippet/snippet-development.html#sec-3-8 for some details. We will tangle this code block into a local snippets directory.

label:code:ref-snippet

# -*- mode: snippet -*-
# --
ref:${1:$$(yas-choose-value (get-labels))} $0

Now we load the snippets directory.

label:load-snippets

(yas-load-directory "./snippets")

Finally, we can type ref, press tab to complete it, and then select the label you want from a list. Here are some examples:

ref:code-example

ref:code:ref-snippet

That also works! I cannot decide if I like this better than the Emacs completion. yasnippet gives a popup menu, which is not as easy to navigate as the Emacs completion mechanism. It also requires a working yasnippet, which has not made it into my regular work flows too often. I think I like the Emacs completion better (which actually goes through Icicles since I have that installed). I like it better because I do not have to leave the keyboard or use the arrow buttons to choose a label. However, I do need to bind that function to some key to use it, or type in the command name. It turns out I do not use ref links too often, so it is not too burdensome.

Copyright (C) 2014 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.5h

I am continuing to evolve how I can use org-mode. I have created a label link, which if clicked on checks to see that the label is unique in the buffer. It would be nice to be able to be on a

label:some-label

ref:some-label

This was a little challenging. The strategy to determine if the cursor is in a link is to search backward for a regular expression matching a label link. I found this was not sufficient, because it appeared to me that the matched string was only between the beginning of the label link and the point where the cursor was. So, after finding the beginning of the first label link before the cursor, then we search forward to find the whole link. Then we determine if the cursor is between the beginning and end of the match. If it is, then we are on a label link. Here is the code.

(defun in-label-link ()
  "return label if in a label link, or return nil

we store point, search forward to the first space, and backward to the previous space. then make sure label: is between them."
  (interactive)
  (let ((current-point (point)))
    (save-excursion
      (re-search-backward "label:\\([a-zA-z0-9:-]*\\)" (point-min) t)
      (re-search-forward "label:\\([a-zA-Z0-9-:]*\\)" (point-max) t)   
      (if (and (> current-point (match-beginning 0))
               (< current-point (match-end 0)))
          t
        nil))))

This code works for these kinds of links as far as I can tell. Interestingly, it only works when the cursor is to the right of label:. I am not sure if that is because of the regular expression or not.

label:plain-beginning

label:telabel

label:fig:test

label:bracket-in-line

Now, we create the code that stores the link. We only execute the code if we pass the function that checks if we are on a label link. If we are, then the label is stored in (match-string 1), and we create the link and store it. Finally, we add the function to org-store-link-functions so that it will be used when C-c l is pressed.

(defun org-label-store-link ()
  "store a link to a label. The output will be a ref to that label"
  ;; First we have to make sure we are on a label link. 
  (when (in-label-link)
    (org-store-link-props
     :type "ref"
     :link (concat "ref:" (match-string 1)))))

(add-hook 'org-store-link-functions 'org-label-store-link)

So, here is the evidence that it worked:

ref:plain-beginning

ref:telabel

ref:fig:test

ref:bracket-in-line

For each of these, I put the cursor on the labels, pressed C-c l, and then moved the cursor down here and pressed C-c C-l, and pressed enter and PRESTO! I had the reference that I wanted! That seems like a handy trick.

Copyright (C) 2014 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.5h

I came across this interesting post on using completion in links: http://draketo.de/light/english/free-software/custom-link-completion-org-mode-25-lines-emacs . I like the idea, but the type-flow for is not how I usually insert links. For the method there to work, you have to enter a link with C-c C-l, partially enter the link type, press enter, and then partially enter the description, which can be completed with tab. That is a lot of typing to me, compared to what I usually do which is type the link in directly. That habit does not work too well in large documents, and always has the possibility of a typo in the link, which then does not work or export correctly.

Here I explore how to make a

ref:label

label:code-example

(defun get-labels ()
  (interactive)
  (save-excursion
    (goto-char (point-min))
    (let ((matches '()))
      (while (re-search-forward "label:\\(.*\\)" (point-max) t)
        (add-to-list 'matches (match-string-no-properties 1) t))
      matches)))

(defun org-insert-ref-link (&optional arg)
  (interactive (list (completing-read "label: " (get-labels))))
  (insert (format "ref:%s" arg)))

org-insert-ref-link

So, here you run the command with M-x org-insert-ref-link, press tab, and select the label you want to use. A link like this gets inserted in your buffer

ref:code-example

Copyright (C) 2014 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.5h

This post explores a method to save an org-buffer to a zip archive, with all the referencing files. The challenge is that you may want to share the org-file with someone, but the links break if you send them the file, and it is not that trivial to find all the links and change them, and to copy the files to a place where the new links work.

The idea is to export the buffer to an org-file and process all the links to copy the files to a new directory, and change the links to point to these new files. For example, blog.pdf would be copied to the temporary directory, given a unique name, and then relinked. The text below includes some examples of the links that need to be modified.

A figure looks like:

Alternatively, we might use a to a file. We do not want to change urls, such as this one: http://kitchingroup.cheme.cmu.edu/blog/2013/09/28/Changing-links-to-files-so-they-work-in-a-blog/ . As in that example, we will create a list of all the links in the buffer, but only modify the links that are files. We can parse the buffer and get the links like this.

(let ((parsetree (org-element-parse-buffer))
      (counter 0))
  (org-element-map parsetree 'link
    (lambda (link) 
      (let ((type (nth 0 link))
            (plist (nth 1 link))
            (content (nth 2 link)))
        (princ (format "%s %s: %s %s\n" 
                       counter 
                       (plist-get plist ':type) 
                       (plist-get plist :path) 
                       content))
        (setq counter (+ counter 1))))))

0 file: ./blog.pdf nil
1 file: ./images/cos-plot.png nil
2 file: ./images/eos.png link
3 http: //kitchingroup.cheme.cmu.edu/blog/2013/09/28/Changing-links-to-files-so-they-work-in-a-blog/ nil

So, our overall strategy will be to create a new directory to store the new versions of the files in. Then, we will copy the files that links point to into that directory, and give them new unique names. We will rename the links to point to these new names. We do this because you may, for some reason have links to files with the same name but in different directories. We want to make sure we do not clobber the files by overwriting them. We use a simple method here, based on unique, temporary filenames. There are other ways to do it to, this way worked first. Finally, we will zip that new directory, and delete the new directory.

;; directory to save all exports in, using the current date
(setq org-archive (concat "org-archive-" (format-time-string "%Y-%m-%d" (current-time))))

;; delete directory and zip file if it exists
(when (file-exists-p (concat org-archive ".zip")) 
    (delete-file (concat org-archive ".zip") t))

(when (file-exists-p org-archive) 
    (delete-directory org-archive t))

;; make directory
(make-directory org-archive t)

;; get list of links, copy files and save names
(setq link-list (let ((parsetree (org-element-parse-buffer))
                     (counter 0))
                 (org-element-map parsetree 'link
                   (lambda (link) 
                     (let* ((type (nth 0 link))
                            (plist (nth 1 link))
                            (content (nth 2 link))
                            (path (plist-get plist :path))
                            (type (plist-get plist ':type))
                            (fname (car (last (split-string path "/"))))
                            (temporary-file-directory org-archive)
                            (new-file)
                            )     
                       (cond
                        ;; regular file with content
                        ((and (string= type "file")  content)
                         (setq new-file  (make-temp-file (file-name-sans-extension fname) nil 
                                                         (concat "." (file-name-extension fname))))
                         (with-temp-file new-file
                           (insert-file-contents path))
                         (format "[[./%s][%s]] " (file-name-nondirectory new-file) content))
                        ;; regular file with no content
                        ((and (string= type "file"))
                         (setq new-file  (make-temp-file (file-name-sans-extension fname) nil 
                                                         (concat "." (file-name-extension fname))))
                         (with-temp-file new-file
                           (insert-file-contents path))
                         (format "[[./%s]] " (file-name-nondirectory new-file)))
                        (t nil)))))))

;; save current buffer name
(setq current-name (buffer-name))

;; create filter for links and export org buffer
(let ((counter 0))
  (defun ox-mrkup-filter-link (text back-end info)
    (let ((link (nth counter link-list)))
      (if (not (string= link "nil")) (setq output   (format "%s" link))
        (setq output (format "%s" text)))
      (setq counter (+ counter 1))
      output))

  (let ((org-export-filter-link-functions '(ox-mrkup-filter-link)))
    (org-org-export-as-org)))

(switch-to-buffer "*Org ORG Export*")
(write-file (expand-file-name current-name org-archive))
(shell-command (concat "zip -R " org-archive ".zip  *"))
(rename-file (concat org-archive ".zip") (concat "../"org-archive ".zip"))
(kill-buffer)

(switch-to-buffer current-name)
(delete-directory org-archive t)  ;; get rid of temp-dir

This example works fine! The result is here: org-archive-2014-03-05.zip This code would ideally be put into a function, and cleaned up a little so there are not global variables being set here and there. A subsequent function might make it easy to attach this file to an email. That code might look something like this:

(mail)
(mail-to)
(insert "jkitchin@andrew.cmu.edu")
(mml-attach-file "./org-archive-2014-03-05.zip")

Copyright (C) 2014 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.5h