Collecting entries from files in a directory

| categories: emacs-lisp | tags: | View Comments

I am running a class where students will be generating files that contain their answers. I want to quickly get a list of counts of all the answers. For example, six students will create a file called animal.dat in a directory called example/<studentid>, and that file will contain their favorite animal. I made some example files to test this idea out. Here are the contents.

cat example/*/animal.dat
dog
cat
dog
bird
dog
bird

You can see there are three dogs, two birds and a cat. I want code to do this counting, because in my real application there will be 58 of these files, and lots of times I need to aggregate them. Let us start with a simple example that counts the elements in a list.

(let ((animals '(dog cat dog bird dog bird))
      (counts '())
      place)
  (dolist (animal animals)
    (setq place (assoc animal counts))
    (message "place = %s" place)
    (if place
        (setf (cdr place) (+ 1 (cdr place)))
      (setq counts (cons `(,animal . 1) counts))))
  counts)

((bird . 2) (cat . 1) (dog . 3))

Let us turn that into a function.

(defun counts (list)
  (let ((counts '())
        place)
    (dolist (el list)   
      (setq place (assoc el  counts))
    (message "place = %s" place)
    (if place
        (setf (cdr place) (+ 1 (cdr place)))
      (setq counts (cons `(,el . 1) counts))))
    counts))

(counts '(dog cat dog bird dog bird))

((bird . 2) (cat . 1) (dog . 3))

Nice. Now we need a simple way to get that list. We need to glob the files to find them, then open them and read the value. Here is a way to get the files.

(f-entries "example"
           (lambda (f)
             (string= (file-name-nondirectory f) "animal.dat"))
           t)
/Users/jkitchin/blogofile-jkitchin.github.com/blog/collect-entries/example/s1/animal.dat /Users/jkitchin/blogofile-jkitchin.github.com/blog/collect-entries/example/s2/animal.dat /Users/jkitchin/blogofile-jkitchin.github.com/blog/collect-entries/example/s3/animal.dat /Users/jkitchin/blogofile-jkitchin.github.com/blog/collect-entries/example/s4/animal.dat /Users/jkitchin/blogofile-jkitchin.github.com/blog/collect-entries/example/s5/animal.dat /Users/jkitchin/blogofile-jkitchin.github.com/blog/collect-entries/example/s6/animal.dat

Now we just need to run a mapcar over this list of files.

(mapcar
 (lambda (f)
   (with-temp-buffer
     (insert-file-contents f)
     (s-trim (buffer-string))))
 (f-entries "example"
            (lambda (f)
              (string= (file-name-nondirectory f) "animal.dat"))
            t))
dog cat dog bird dog bird

Finally, putting this together, we have some code that maps over the files, and counts the entries.

(counts
 (mapcar
  (lambda (f)
    (with-temp-buffer
      (insert-file-contents f)
      (s-trim (buffer-string))))
  (f-entries "example"
             (lambda (f)
               (string= (file-name-nondirectory f) "animal.dat"))
             t)))

((bird . 2) (cat . 1) (dog . 3))

This will be helpful in dealing with 58 entries during my class!

Copyright (C) 2014 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.7c

Read and Post Comments

Autogenerating functions in emacs-lisp

| categories: emacs, emacs-lisp | tags: | View Comments

I have a need to generate a lot of similar functions, and I do not want to cut and paste the code. I want to generate the functions with code. This seems to be what macros are for in emacs lisp.

As a prototype example, we will make functions that raise a number to a power. We want functions like power-3 and power-4 that raise numbers to the third and fourth powers. We will define functions like this for the numbers 0-9.

Here we define the macro. i do not want to get into the nitty gritty details of macro definitions here.

(defmacro make-power-n (n)
 `(defun ,(intern (format "power-%s" n)) (arg) (expt arg ,n)))

(make-power-n 4)

(power-4 4)
256

Now we use the macro and mapcar on it onto a list of numbers. We have to eval the macro in the mapcar lambda function.

(defmacro make-power-n (n)
 `(defun ,(intern (format "power-%s" n)) (arg) (expt arg ,n)))

(mapcar (lambda (x) (eval `(make-power-n ,x))) '(0 1 2 3 4 5 6 7 8 9))
 
;; example of a few functions
(list (power-0 3) (power-1 3) (power-2 3))
1 3 9

It works! We created 10 functions in a little bit of code.

Copyright (C) 2014 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.6

Read and Post Comments

Language specific default headers for code blocks in org-mode

| categories: emacs-lisp, org-mode | tags: | View Comments

I use code blocks in org-mode a lot. I usually code in Python, and in Python I usually write code that prints output which I want to see. So I almost always want the code blocks to return the output, and not the value of the last function. I have set my default header args like this:

org-babel-default-header-args
(:exports . both) (:results . replace output) (:session . none) (:cache . no) (:noweb . no) (:hlines . no) (:tangle . no)

However, I would prefer that emacs-lisp blocks use value for the results. I know I can get that by putting :results value in the code block header, but that annoys me. I learned today from http://orgmode.org/worg/org-contrib/babel/header-args.html that you can make language specific default headers!

This code in my init file sets emacs-lisp specific default headers:

(setq org-babel-default-header-args:emacs-lisp 
      (cons '(:results . "value")
            (assq-delete-all :results org-babel-default-header-args)))

That way I do not have type :results value at the top of every elisp block. Of course, if I want the output I could specify :results output in the block.

org-babel-default-header-args:emacs-lisp
(:results . value) (:exports . both) (:session . none) (:cache . no) (:noweb . no) (:hlines . no) (:tangle . no)

Problem solved!

On a related note, I find I write so many blocks of python and elisp I added these templates:

;; add <p for python expansion
(add-to-list 'org-structure-template-alist
             '("p" "#+BEGIN_SRC python\n?\n#+END_SRC" "<src lang=\"python\">\n?\n</src>"))

;; add <el for emacs-lisp expansion
(add-to-list 'org-structure-template-alist
             '("el" "#+BEGIN_SRC emacs-lisp\n?\n#+END_SRC" "<src lang=\"emacs-lisp\">\n?\n</src>"))

I probably could have also coded the :results header into those too. They add a tidbit of convenience so I do not have to type python or emacs-lisp after expanding a source block with <s.

Copyright (C) 2014 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.5g

Read and Post Comments

Another alternative to string templates

| categories: emacs-lisp | tags: | View Comments

In the last post I explored a way to expand a string template that was more readable than the usual format. Today I look at another approach where I use sexp expansions to accomplish the same thing. The idea is to embed lisp expressions and replace them by what they evaluate to.

In emacs-lisp, if we have a command in a string, we can "read" it, and then eval it.

Here we get the user-full-name:

(eval (read "user-full-name"))
John Kitchin

We can use this on variables too.

(setq some-variable "test")
(eval (read "some-variable"))
test

So, if we use a syntax to identify what to replace, we can substitute in the values. Let us try %() as the syntax.

(defun expand-template (s)
  "expand a template containing %() with the eval of its contents"
  (replace-regexp-in-string "%(\\([^)]+\\))"
                            (lambda (arg)
                              (format "%s" (eval (read (substring arg 2 -1))))) s))


(let ((key "kitchin-2014")
      (author "Kitchin, J. R.")
      (journal "HACS")
      (year "2014")
      (volume "1")
      (pages "1--10")
      (doi "10.1.1.109/hacs.1.10")
      (url "http://hacs.org/10.1.1.109/hacs.1.10")
      (pdf-dir "/home/jkitchin/pdfs")
      (template "
 :PROPERTIES:
  :Custom_ID: %(key)
  :AUTHOR: %(author
  :JOURNAL: %(journal)
  :YEAR: %(year)
  :VOLUME: %(volume)
  :PAGES: %(pages)
  :DOI: %(doi)
  :URL: %(url)
 :END:
[[cite:%(key)]] [[file:%(pdf-dir)/%(key).pdf][pdf]]\n\n"))

(expand-template template))
 :PROPERTIES:
  :Custom_ID: kitchin-2014
  :AUTHOR: Kitchin, J. R.
  :YEAR: 2014
  :VOLUME: 1
  :PAGES: 1--10
  :DOI: 10.1.1.109/hacs.1.10
  :URL: http://hacs.org/10.1.1.109/hacs.1.10
 :END:
[[cite:kitchin-2014]] [[file:/home/jkitchin/pdfs/kitchin-2014.pdf][pdf]]

That is pretty nice. I like it better than the plist expansion I used before. Presumably these variables would already be defined somewhere in your code.

I thought of trying this on a more complex expansion, and discovered a weakness in the regexp that finds the expansion values. It turns out to be simpler to use %{} as the delimiter than %(), because you may want nested parentheses. The regexp above does not correctly match sets of parentheses.

(defun expand-template (s)
  "expand a template containing %{} with the eval of its contents"
  (replace-regexp-in-string "%{\\([^}]+\\)}"
                            (lambda (arg)
                              (let ((sexp (substring arg 2 -1)))
                                (format "%s" (eval (read sexp))))) s))

(expand-template "2 * 2 = %{(* 2 2)}")
2 * 2 = 4

I am not sure this is a desirable way to make a template, with multiline code to be expanded, but at least this works!

(defun expand-template (s)
  "expand a template containing %{} with the eval of its contents"
  (replace-regexp-in-string "%{\\([^}]+\\)}"
                            (lambda (arg)
                              (let ((sexp (substring arg 2 -1)))
                                (format "%s" (eval (read sexp))))) s))

(expand-template "The result is %{(progn
  (if (> 4 3)
      'true
    'false))}")
The result is true

The regexp used in the expansion is not very robust. In particular if there is a } in the code, it will probably fail because the regexp does not match closing } correctly. Fixing that is beyond me right now!

Copyright (C) 2014 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.5g

Read and Post Comments

Alternatives to long complex format statements in emacs-lisp

| categories: emacs-lisp | tags: | View Comments

At one point I had a string I wanted to fill in with a bunch of variables.

(insert (format"
 :PROPERTIES:
  :Custom_ID: %s
  :AUTHOR: %s
  :JOURNAL: %s
  :YEAR: %s
  :VOLUME: %s
  :PAGES: %s
  :DOI: %s
  :URL: %s
 :END:
[[cite:%s]] [[file:%s/%s.pdf][pdf]]\n\n"
key author journal year volume pages doi url key jorg-bib-pdf-directory key ))

I find that very difficult to use, because it is tedious to make sure all the variables are in the right order, and it is difficult to change later. In Python, you would be able to put named expansions in, e.g. {author} and then used named arguments. That does not exist as far as I know in emacs-lisp.

Below is an alternatme approach that uses concat to construct this string.

(let ((key "kitchin-2014")
      (author "Kitchin, J. R.")
      (journal "HACS")
      (year "2014")
      (volume "1")
      (pages "1--10")
      (doi "10.1.1.109/hacs.1.10")
      (url "http://hacs.org/10.1.1.109/hacs.1.10")
      (jorg-bib-pdf-directory "/home/jkitchin/pdfs"))

(concat "
 :PROPERTIES:
  :Custom_ID: " key "
  :AUTHOR: " author "
  :JOURNAL: " journal "
  :YEAR: " year "
  :VOLUME: " volume "
  :PAGES: " pages "
  :DOI: " doi "
  :URL: " url "
 :END:
[[cite:" key "]] [[file:" jorg-bib-pdf-directory "/" key ".pdf][pdf]]\n\n"))
 :PROPERTIES:
  :Custom_ID: kitchin-2014
  :AUTHOR: Kitchin, J. R.
  :JOURNAL: HACS
  :YEAR: 2014
  :VOLUME: 1
  :PAGES: 1--10
  :DOI: 10.1.1.109/hacs.1.10
  :URL: http://hacs.org/10.1.1.109/hacs.1.10
 :END:
[[cite:kitchin-2014]] [[file:/home/jkitchin/pdfs/kitchin-2014.pdf][pdf]]

That is kind of interesting. It is a little tedious to use all the quotes. It seems like there should be soemthing like named expansions. Let us write one of our own. We will use a regular expression to find {:keyword} and a plist. There is a regexp to match this, and then we can take the characters from position 1 to the second to last character as the keyword. That is not beautiful to me, but it works here. Then we just get the keyword from the plist. The keywords in a plist are symbols, and we will have strings. We have to use the intern function to convert them to symbols.

(defun expand-template (s plist)
  "expand a template containing {:keyword} with the definitions in plist"
  (replace-regexp-in-string "{\\(:[^}]+\\)}" 
                            (lambda (arg) 
                              (let ((keyword (intern (substring arg 1 -1))))
                                (format "%s" (plist-get plist keyword)))) s))

(let ((template "
 :PROPERTIES:
  :Custom_ID: {:key}
  :AUTHOR: {:author}
  :JOURNAL: {:journal}
  :YEAR: {:year}
  :VOLUME: {:volume}
  :PAGES: {:pages}
  :DOI: {:doi}
  :URL: {:url}
 :END:
[[cite:{:key}]] [[file:{:pdf-dir}/{:key}.pdf][pdf]]\n\n"))

(expand-template template
                 '(:key "kitchin-2014"
                        :author "Kitchin, J. R."
                        :journal "HACS"
                        :year 2014
                        :volume 1
                        :pages "1--10"
                        :doi "10.1.1.109/hacs.1.10"
                        :url "http://hacs.org/10.1.1.109/hacs.1.10"
                        :pdf-dir "/home/jkitchin/pdfs")))
 :PROPERTIES:
  :Custom_ID: kitchin-2014
  :AUTHOR: Kitchin, J. R.
  :JOURNAL: HACS
  :YEAR: 2014
  :VOLUME: 1
  :PAGES: 1--10
  :DOI: 10.1.1.109/hacs.1.10
  :URL: http://hacs.org/10.1.1.109/hacs.1.10
 :END:
[[cite:kitchin-2014]] [[file:/home/jkitchin/pdfs/kitchin-2014.pdf][pdf]]

That is pretty close to what I am used to from python! I am surprised there aren't other solutions for this around. I looked, and couldn't find them.

Copyright (C) 2014 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.5f

Read and Post Comments

Next Page ยป