org-mode + Python + git in a graduate engineering course

| categories: orgmode, education | tags:

The Fall 2014 semester is over! I thought I would summarize here what we did. I taught a Master's course in Chemical Reaction Engineering for the second time. This time we had 58 students from around the world. What is remarkable about this course is that this time it was taught completely from Emacs. Every lecture was given from Emacs, every assignment completed in Emacs, turned in from Emacs, graded in Emacs, returned in Emacs.

Students came to class, opened Emacs, and ran a command that downloaded the latest syllabus and opened it. They opened the day's lecture notes from a link in the syllabus. I lectured from the notes in Emacs, editing them on the screen live, running Python code to illustrate engineering concepts. Students did exercises in class in Emacs, and ran an Emacs command to "turn it in" which pushed their work to a git server. Later, I ran an Emacs command to collect and grade the work, then return it. Students could run a menu command in Emacs to see their grade report.

Techela provided a menu of commands to turn in assignments, check grade reports, send me feedback, open the syllabus, etc… The notes were written in org-mode, and we used org-latex-fragments to see the equations. We used code-blocks to show Python examples of problem solving, every day in class. It was awesome!

The way this worked is that most of my students got laptops as part of the MS program they enrolled in. I had my jmax repo installed on those computers, along with git, TexLive and Canopy Python. jmax provided a starter-kit for emacs that gave it the functionality I use on a regular basis. In jmax, I created a package of code I call techela, which interfaces Emacs with git, and which provides commands to download and turn in assignments, and to control permissions on each repo. This enabled me to change repos to read-only after they were due, and to grant read access when I wanted students to see them. About 15% of the class had their own computer, and we had to help them get this software installed. This got done in the first week of class, thanks to the help of my teaching assistants.

I ran a gitolite server that served the course materials, and a repo for each assignment for each student. When students first ran techela, it created a set of ssh keys which were used to authenticate each student in the gitolite server. Techela automates creation of the repos, and the permissions on each repo. Grading of assignments was done in Emacs. Every assignment turned in was an org-file, and we stored grades in the assignments as file tags. Techela constructed a dynamic gradebook on demand, among other things.

Org-mode played a central role in the success of this course! It enabled the distribution of the notes in a readable, functional form. The programmable aspects of org-mode made it possible to create custom links for machine-gradable multiple choice questions, assignments, and solutions. It made it possible to use the assignments as a data storage source for grades, and later to pull the grades out for analysis.

Overall, the experience was just awesome. Techela has a few edges that need smoothed out, but I look forward to using it again this spring, this time on a course on Molecular Simulation!

Copyright (C) 2014 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.10

Discuss on Twitter

A database of quiz questions to generate quizzes and solutions

| categories: education | tags:

In this post I talked about a machine gradable quiz. One issue I have been trying to resolve is how to keep questions and solutions together. Currently, I have them separated, and that occasionally leads to them not being synchronized. One tricky part is if they are together, how do you create the quiz without answers in them? The answer is you have to "export" the quiz and selected problems, with selective content, and you want the elements of the question exported in some format, e.g. html or LaTeX.

I am going to explore representing a question as some data structure, not in org-mode for now. Then, we will use emacs-lisp to format the question the way we want it. The data we need about a question include the type of question, a unique id, the question, the choices, and the answer. For the choices, we might want some descriptive text for the solution. Here is one example of how we could represent a question as a struct in emacs-lisp. I do not claim here that this is the best or most complete way to represent it; it is simply a way that we could do it so we can discuss it later.

(defstruct question type id question choices)
question

The defstruct macro creates a constructor function "make-question" for us, and we create questions like this:

(setq q1 (make-question
          :type 'multiple-choice
          :id "id:1"
          :question "What is 2 + 2?"
          :choices '(('choice 0 "Not correct")
                     ('choice 2 "Not correct")
                     ('choice 4 nil 'answer)
                     ('choice "None of the above" "Not correct"))))
[cl-struct-question multiple-choice "id:1" "What is 2 + 2?" (((quote choice) 0 "Not correct") ((quote choice) 2 "Not correct") ((quote choice) 4 nil (quote answer)) ((quote choice) "None of the above" "Not correct"))]

The defstruct macro also creates accessor functions to get the information stored in the question. These go by "question-SLOTNAME" and they make it pretty easy to get the data. We can convert this structure to an org-mode formatted question by using a formatted string, substituting values from the question object where we want them. I will make the headline contain the question, create some properties, and then make an org-link for each choice. The mc link is something I defined in the previous post.

(format "** %s
  :PROPERTIES:
  :ID: %s
  :END:
%s" (question-question q1)
(question-id q1) 
(mapconcat 'identity (loop for i to (length (question-choices q1))
                           for choice in (question-choices q1)
                           collect
                           (format "[[mc:%s]] %s" i (elt choice 1)))
           "\n"))
** What is 2 + 2?
  :PROPERTIES:
  :ID: id:1
  :END:
[[mc:0]] 0
[[mc:1]] 2
[[mc:2]] 4
[[mc:3]] None of the above

To generate a quiz, we would just need to loop over several questions and insert the strings into a buffer then save it to a file.

To generate the key, we use pretty similar code, and additionally find the choice that is the answer, and print any hints with the choices.

(format "** %s
  :PROPERTIES:
  :ID: %s
  :CORRECT-ANSWER: %s
  :END:
%s" (question-question q1)
    (question-id q1)
    ;; here we loop through the choices looking for the answer 
    (loop for choice in (question-choices q1)
          until (-contains? choice '(quote answer))
          finally return (elt choice 1))
    (mapconcat 'identity (loop for i to (length (question-choices q1))
                               for choice in (question-choices q1)
                               collect
                               (format "[[mc:%s]] %s
%s" i (elt choice 1) (or (elt choice 2) "")))
               "\n"))
** What is 2 + 2?
  :PROPERTIES:
  :ID: id:1
  :CORRECT-ANSWER: 4
  :END:
mc:0 0
Not correct
mc:1 2
Not correct
mc:2 4

mc:3 None of the above
Not correct

It is a similar process to convert to LaTeX or HTML as well, just different syntax in the format statement.

The next question is how to make authoring the questions pretty easy, i.e. with a minimal amount of syntax that represents the question data and is easy to transform. The lisp data structure is not too bad, but might be less familiar to many. Next we consider how to represent this question in org-mode. For example the question would be in a headline, and the ID and answer stored as properties. The choices might be stored as sub-headings that are tagged as choices. Then, constructing the question would entail going to the question headline, getting the information you want, and formatting it like I did above. The next section represents a question in org-mode. I think it actually takes more typing to do this, but some of it reuses some org-machinery I am familiar with. In any case, it is possible to store all the same kind of information using combinations of tags and properties. The next section is the question (only really visible in org-mode, not in the html post), and the section after that is how to use it.

1 What is 2 + 2?   multiple_choice

1.1 0   choice

Not correct, This is the answer to 2 - 2.

1.2 2   choice

Not correct

1.3 4   choice answer

Correct.

1.4 None of the above   choice

This is not correct

2 Working with the org representation of a question

Now, we use the org-machinery to work with the question and represent it in different formats. Here we just get the choices for a question with a specific id. We jump to the question, find headings tagged choice, and collect them.

(save-excursion
  (org-open-link-from-string "id:99D480BD-9651-4987-9646-4BE90BA6CAEC")
  (save-restriction
    (org-narrow-to-subtree)
    (let ((choices '()))
      (org-map-entries
       (lambda ()
         (add-to-list
          'choices
          (org-heading-components)
          t))
       "choice")
      choices)))
3 3 nil nil 0 :choice:
3 3 nil nil 2 :choice:
3 3 nil nil 4 :choice:
3 3 nil nil None of the above :choice:

You could use the headline for each choice to construct a question. For example, here we output some HTML to represent checkboxes in an HTML form.

(save-excursion
  (org-open-link-from-string "id:99D480BD-9651-4987-9646-4BE90BA6CAEC")
  (let ((all-choices 
         (save-restriction
           (org-narrow-to-subtree)
           (let ((choices '()))
             (org-map-entries
              (lambda ()
                (add-to-list
                 'choices
                 (elt (org-heading-components) 4)
                 t))
              "choice")
             choices)))
        (question (elt (org-heading-components) 4))
        )
    (concat
     "<html><body>\n"
     "<h1>" question "</h1>\n"
     "<form action = \"\">"
     (mapconcat
      'identity
      (loop for choice in all-choices
            collect       
            (format "  <input type=\"checkbox\" name=\"choice\" value=\"%s\">%s<br>"
                    choice choice))
      "\n")
     "\n</form>\n"
     "</body></html>")))

What is 2 + 2?

0
2
4
None of the above

There is no action button to submit this answer, but you could add one if wanted. Clearly, you will have custom functions to export each format you want, with the information you want in it. The code for the org-mode representation is a little more verbose than the lisp data representation, but they accomplish basically the same thing. One thing that would be necessary to do is create some functions that take a list of question IDs and generate the quiz and solution in a buffer, and a helm interface for question selection would be pretty sweet.

I can see adding several other pieces of information about a problem, including categories or keywords, level of difficulty, how many points the problem is worth, last time it was assigned, relationship to course learning objectives, maybe information about who authored the problem, and a source if that is appropriate. These would allow you to consider machine grading, or correlating performance between problems, assessing how a class is doing on a problem, etc… Lots of potential here.

Copyright (C) 2014 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.7c

Discuss on Twitter

Machine-gradable quizzes in emacs+org-modex

| categories: education, org, emacs | tags:

In a previous post , we considered interactive quizzes in emacs and org-mode. Here we consider a variation of that idea with the aim of creating a machine-gradable quiz, i.e. a modern version of a scantron.

The idea is simple, we will let each question be represented by an org headline, and the choices will be clickable links that store the answer as a property in the headline. Each headline will have a unique id. The grading key will contain these ids and the correct answer, and a function will determine if the right answer was selected for each question.

Here is a simple link that will store the value of the link as a property. Note that the path must be at least two characters long to be recognized as a link, unless you wrap the link in double brackets. We will have the link insert a comment to indicate to the user what they chose. We do that because the PROPERTIES drawer is usually hidden, and it is not obvious it was changed.

(org-add-link-type 
 "mc" 
 (lambda (link)
   (save-restriction
     (org-narrow-to-subtree)
     (goto-char (point-max))
     (insert (concat
              (unless (bolp) "\n")
              (format "# you chose %s" link))))
       
   (org-entry-put (point) "ANSWER" link)))

Next, we add a subheading with some questions to test the link.

1 Some questions

1.1 question 1

What is 2 + 2?

a 1

b 2

c 4

1.2 question 2

What is 2 - 2?

a 0

b 2

c 4

2 Grading

We will store an alist of id and answer for each problem. To grade, we simple map over the alist, go to the section with the id, and compare the answers. When the answer is correct, we save a point, and when not, no point. We can use the org-mode machinery to jump to the problems and get the stored answer. We put some feedback at the end of the file to see what was right, and what was wrong.

(let* ((key '(("19C7BA30-A761-4C94-9F3B-E6010E263949" . "c")
              ("38FCCF3D-7FC5-49BF-BB77-486BBAA17CD9" . "a")))
       (MAX (length key))
       (points 0)
       (answer))
  
  (dolist (tup key)
    (save-excursion
      (org-open-link-from-string
       (format "id:%s" (car tup)))
      (setq answer (org-entry-get (point) "ANSWER"))
      (if (string= answer (cdr tup))
          (progn
            (setq points (+ 1 points))
            (goto-char (point-max))
            (insert (format "# id:%s: %s correct\n" (car tup) answer)))
        (goto-char (point-max))
        (insert (format "# id:%s: %s wrong (%s is correct)\n"
                        (car tup)
                        answer
                        (cdr tup))))))
  (goto-char (point-max))
  (insert (format
           "#+GRADE: %s" (/ (float points) (float MAX)))))

That works pretty well. I need to think about how to codify the key, since this would usually be stored in some file. We would also need to wrap the code block in a function that we could call easily. The org-id key is easy, but not very readable. It would make it easy to keep a database of these problems though.

Just for completeness, I want to save the key to a file, and use it. We simply write the alist in a file. Here are the contents, which are tangled to key.el. One alternative might be to have a solution copy of the quiz which has the answers in it, and we read the answers from the file.

(("19C7BA30-A761-4C94-9F3B-E6010E263949" . "c")
 ("38FCCF3D-7FC5-49BF-BB77-486BBAA17CD9" . "a"))

Now, we read it in like this. The rest of the code is basically the same.

(let* ((key (with-temp-buffer 
              (insert-file-contents "key.el")
              (read (current-buffer))))
       (MAX (length key))
       (points 0)
       (answer))
  
  (dolist (tup key)
    (save-excursion
      (org-open-link-from-string
       (format "id:%s" (car tup)))
      (setq answer (org-entry-get (point) "ANSWER"))
      (if (string= answer (cdr tup))
          (progn
            (setq points (+ 1 points))
            (goto-char (point-max))
            (insert (format "# id:%s: %s correct\n" (car tup) answer)))
        (goto-char (point-max))
        (insert (format "# id:%s: %s wrong (%s is correct)\n"
                        (car tup)
                        answer
                        (cdr tup))))))
  (goto-char (point-max))
  (insert (format
           "#+GRADE: %s" (/ (float points) (float MAX)))))

It is probably much easier to have a solution version of the quiz, and generate the key from it. For example, we can collect the ID and ANSWER from the problems in this file like this.

(let ((key '()))
  (org-map-entries
   (lambda ()
     (let ((id) (ans))
       (when (and
              (setq id (org-entry-get (point) "ID"))
              (setq ans (org-entry-get (point) "ANSWER")))
         (add-to-list 'key (cons id ans))))))
key)
(("38FCCF3D-7FC5-49BF-BB77-486BBAA17CD9" . "a")
 ("19C7BA30-A761-4C94-9F3B-E6010E263949" . "c"))

So, if we had a master solution file, we could read the key from there. That is the way to do this.

Copyright (C) 2014 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.7c

Discuss on Twitter