The Kitchin Research Group

A few times I have wanted to use a more natural search string like "A and pw and 350 and not kpt". The trouble is figuring out how to parse that string and turn it into search code. There may be nested logic, e.g. "(A xor B) and (pw and (200 or 300))". This means we have to recursively parse the sstring. Rather than invent this from scratch, we use pyparsing which is designed for that. There is some code in "Getting started with pyparsing" that provides an example on parsing search strings. I want to see how I can turn the parsed output into search code. Here, we parse the search string and generate something that looks like lisp code.

from pyparsing import *

class UnaryOperation(object):
    'takes one operand,e.g. not'
    def __init__(self, tokens):
        self.op, self.operands = tokens[0]

class BinaryOperation(object):
    'takes two or more operands, e.g. and, or'
    def __init__(self, tokens):
        self.op = tokens[0][1]
        self.operands = tokens[0][0::2]

class SearchAnd(BinaryOperation):
    def __repr__(self):
        return '(AND {0})'.format(' '.join(str(oper) for oper in self.operands))
        
class SearchOr(BinaryOperation):
    def __repr__(self):
        return '(OR {0})'.format(' '.join(str(oper) for oper in self.operands))

class SearchNot(UnaryOperation):
    def __repr__(self):
        return '(NOT {0})'.format(self.operands)

class SearchTerm(object):
    'represents a termthat is being searched. here just a word'                         
    def __init__(self, tokens):
        self.term = tokens[0]

    def __repr__(self):
        return self.term

# the grammar
and_ = CaselessLiteral("and")
or_ = CaselessLiteral("or")
not_ = CaselessLiteral("not")

searchTerm = Word(alphanums) | quotedString.setParseAction(removeQuotes)
searchTerm.setParseAction(SearchTerm)

searchExpr = operatorPrecedence( searchTerm,
                                 [(not_, 1, opAssoc.RIGHT, SearchNot),
                                  (and_, 2, opAssoc.LEFT, SearchAnd),
                                  (or_, 2, opAssoc.LEFT, SearchOr)])


print searchExpr.parseString('not kpt')[0]
print searchExpr.parseString('not (kpt and eos)')[0]
print searchExpr.parseString('wood and blue or red')[0]
print searchExpr.parseString('wood and blue and heavy or red')[0]

(NOT kpt)
(NOT (AND kpt eos))
(OR (AND wood blue) red)
(OR (AND wood blue heavy) red)

from pyparsing import *

class UnaryOperation(object):
    def __init__(self, tokens):
        self.op, self.operands = tokens[0]

class BinaryOperation(object):
    def __init__(self, tokens):
        self.op = tokens[0][1]
        self.operands = tokens[0][0::2]

class SearchAnd(BinaryOperation):
    def __repr__(self):
        return '(' + ' & '.join(['{}'.format(oper) for oper in self.operands]) + ')'
        
class SearchOr(BinaryOperation):
    def __repr__(self):
        return '(' + ' | '.join(['{}'.format(oper) for oper in self.operands]) +')'

class SearchXor(BinaryOperation):
    def __repr__(self):
        return '(' + ' ^ '.join(['{}'.format(oper) for oper in self.operands]) + ')'

class SearchNot(UnaryOperation):
    def __repr__(self):
        return 'TAGS[\'all\'] - {}'.format(self.operands)

class SearchTerm(object):
    def __init__(self, tokens):
        self.term = tokens[0]

    def __repr__(self):
        'instead of just the  term, we represent it as TAGS[term]'
        return 'TAGS[\'{0}\']'.format(self.term)

# the grammar
and_ = CaselessLiteral("and")
or_ = CaselessLiteral("or")
xor_ = CaselessLiteral("xor")
not_ = CaselessLiteral("not")

searchTerm = Word(alphanums) | quotedString.setParseAction(removeQuotes)
searchTerm.setParseAction(SearchTerm)

searchExpr = operatorPrecedence( searchTerm,
                                 [(not_, 1, opAssoc.RIGHT, SearchNot),
                                  (and_, 2, opAssoc.LEFT, SearchAnd),
                                  (xor_, 2, opAssoc.LEFT, SearchXor),
                                  (or_, 2, opAssoc.LEFT, SearchOr)])

print searchExpr.parseString('not kpt')[0]
print searchExpr.parseString('not (kpt and eos)')[0]
print searchExpr.parseString('kpt or not eos)')[0]
print searchExpr.parseString('wood and blue or red')[0]
print searchExpr.parseString('wood and blue xor red')[0]

# check it out on tags.
def search_tags(srch):
    'function to  search the TAGS  file'
    import pickle

    with open('TAGS.pkl', 'r') as f:
        TAGS = pickle.loads(f.read())
    
    s = searchExpr.parseString(srch)[0]
    return eval(str(s))
print search_tags('pw and A and not 300')

TAGS['all'] - TAGS['kpt']
TAGS['all'] - (TAGS['kpt'] & TAGS['eos'])
(TAGS['kpt'] | TAGS['all'] - TAGS['eos'])
((TAGS['wood'] & TAGS['blue']) | TAGS['red'])
((TAGS['wood'] & TAGS['blue']) ^ TAGS['red'])
set(['tags\\A\\pw\\350', 'tags\\A\\pw', 'tags\\A\\pw\\200', 'tags\\A\\pw\\400', 'tags\\A\\pw\\250'])

Copyright (C) 2014 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.5h

Today I explore searching for directories by using tags. We will create a TAGS table by first walking through the directories and tagging each directory with all of the relative path components. Then, we will use set algebra to identify specific directories.

First, let us make a directory setup to use. We will nest some calculations that might be typical. These will be nested directories that might contain planewave (pw) and k-points (kpts) convergence and equation of state (eos) directories, with the sets of calculations inside those. The idea then is that the directory components will form at least some of the tags.

import os

os.mkdir('tags')

for cmpd in ['A', 'B', 'C', 'D']:
    for c in ['pw', 'kpts', 'eos']:
        os.makedirs(os.path.join('tags', cmpd, c))

for cmpd in ['A', 'B', 'C', 'D']:
    for run in [1, 2, 3, 4, 5]:
        os.makedirs(os.path.join('tags', cmpd, 'eos', str(run)))

for cmpd in ['A', 'B', 'C', 'D']:
    for run in [200, 250, 300, 350, 400]:
        os.makedirs(os.path.join('tags', cmpd, 'pw', str(run)))

for cmpd in ['A', 'B', 'C', 'D']:
    for run in ['2x2x2', '4x4x4', '8x8x8']:
        os.makedirs(os.path.join('tags', cmpd, 'kpts', str(run)))

Let us just double check what this directory tree looks like for one compound A.

ls tags/A/*

tags/A/eos:
1
2
3
4
5

tags/A/kpts:
2x2x2
4x4x4
8x8x8

tags/A/pw:
200
250
300
350
400

Now, we will walk through the directories, and split the path components to create a TAGS structure. I will store these as sets within a dictionary. We will save the structure in a pickle file to reuse it later.

import os
import pickle

TAGS = {}
TAGS['all'] = set()

for root, dirs, files in os.walk('tags'):
    base, tail = os.path.split(root)
    TAGS['all'].add(root)
    while base:
        if tail in TAGS:
            TAGS[tail].add(root)
        else:
            TAGS[tail] = set([root])
        base, tail = os.path.split(base)

with open('TAGS.pkl', 'w') as f:
    f.write(pickle.dumps(TAGS))

print TAGS.keys()

['A', '1', 'all', 'B', '250', 'pw', '2x2x2', '300', 'C', '400', 'kpts', '8x8x8', 'eos', '3', '2', '5', '4', '350', '200', '4x4x4', 'D']

Now we have a lot of keys that tag each directory. Each tag is a set of directories, and we can do set algebra to get specific results. For example, we can find a result by appropriate differences, intersections and unions of the sets. An advantage of this approach is that order of the tags is not relevant (unlike the path, where each component must be in the right order).

import pickle

with open('TAGS.pkl', 'r') as f:
    TAGS = pickle.loads(f.read())

print TAGS['A'].intersection(TAGS['pw']).intersection(TAGS['300'])
print TAGS['300'].intersection(TAGS['pw']).intersection(TAGS['A'])

set(['tags\\A\\pw\\300'])
set(['tags\\A\\pw\\300'])

The syntax here is a tad heavy because of the chained dot notation operations. You can also use the logical operators like this:

import pickle

with open('TAGS.pkl', 'r') as f:
    TAGS = pickle.loads(f.read())

# 300 and pw and A
print TAGS['300'] & TAGS['pw'] & TAGS['A']

set(['tags\\A\\pw\\300'])

We can get a set of calculations, for example an equation of state like this:

import pickle

with open('TAGS.pkl', 'r') as f:
    TAGS = pickle.loads(f.read())

# 300 and pw and A
print TAGS['A'] & TAGS['eos']

set(['tags\\A\\eos', 'tags\\A\\eos\\5', 'tags\\A\\eos\\4', 'tags\\A\\eos\\1', 'tags\\A\\eos\\3', 'tags\\A\\eos\\2'])

Now, let us construct some more complex queries. With sets we use intersections for and and we construct unions of queries that are like an or. We examine different notations to see which one is better.

import pickle

with open('TAGS.pkl', 'r') as f:
    TAGS = pickle.loads(f.read())

# find calculation 3 in eos for B and C
print (TAGS['3']
       .intersection(TAGS['eos'])
       .intersection(TAGS['B'])
       .union
       (TAGS['3']
        .intersection(TAGS['eos'])
        .intersection(TAGS['C'])))

# this notation makes more sense to me. ^ = or
print TAGS['3'] & TAGS['eos'] & (TAGS['B'] ^ TAGS['C'])

set(['tags\\C\\eos\\3', 'tags\\B\\eos\\3'])
set(['tags\\C\\eos\\3', 'tags\\B\\eos\\3'])

You can see the two approaches give the same results. The logical operator syntax is more concise and (I think) more readable. Let us consider a query with "not". We can use a difference operator for that. We subtract all the paths with tag "B" from the set containing "4x4x4", which will give us paths tagged with "4x4x4" but not "B".

import pickle

with open('TAGS.pkl', 'r') as f:
    TAGS = pickle.loads(f.read())

# find calculation 4x4x4 but not B
print (TAGS['4x4x4'].difference(TAGS['B']))

# this makes more sense 4x4x4 but not B, 
# i.e. subtract paths tagged B from those tagged 4x4x4
print TAGS['4x4x4'] -  TAGS['B']

set(['tags\\D\\kpts\\4x4x4', 'tags\\A\\kpts\\4x4x4', 'tags\\C\\kpts\\4x4x4'])
set(['tags\\D\\kpts\\4x4x4', 'tags\\A\\kpts\\4x4x4', 'tags\\C\\kpts\\4x4x4'])

Note it is not so obvious how to get results not tagged with "A". We need to subtract the tagged calculations from some set.

import pickle

with open('TAGS.pkl', 'r') as f:
    TAGS = pickle.loads(f.read())

print TAGS['all'] - TAGS['A']  # not A

set(['tags\\D\\kpts', 'tags\\D\\pw\\200', 'tags\\D\\pw\\350', 'tags\\D\\pw\\250', 'tags\\D\\kpts\\8x8x8', 'tags\\C\\eos', 'tags\\D\\eos\\2', 'tags\\D\\eos\\3', 'tags\\D\\eos\\4', 'tags\\D\\eos\\5', 'tags\\B\\kpts\\2x2x2', 'tags\\C\\kpts\\4x4x4', 'tags\\C\\eos\\3', 'tags\\C\\eos\\2', 'tags\\C\\eos\\1', 'tags\\C\\kpts\\8x8x8', 'tags\\C\\eos\\5', 'tags\\C\\eos\\4', 'tags\\B\\kpts', 'tags\\C\\pw\\200', 'tags\\B\\eos\\2', 'tags\\B\\pw\\350', 'tags\\B\\eos\\1', 'tags\\B\\kpts\\8x8x8', 'tags\\C\\pw\\300', 'tags\\B\\eos\\4', 'tags\\B\\eos\\5', 'tags\\C\\kpts', 'tags\\D\\pw\\300', 'tags\\B\\kpts\\4x4x4', 'tags\\C\\kpts\\2x2x2', 'tags\\D\\kpts\\4x4x4', 'tags\\B\\pw\\250', 'tags', 'tags\\D\\pw\\400', 'tags\\D\\eos', 'tags\\C\\pw\\400', 'tags\\D\\kpts\\2x2x2', 'tags\\D\\pw', 'tags\\C\\pw\\250', 'tags\\C\\pw\\350', 'tags\\C\\pw', 'tags\\D\\eos\\1', 'tags\\B\\pw\\400', 'tags\\B\\pw', 'tags\\B\\eos', 'tags\\B\\pw\\300', 'tags\\B\\eos\\3', 'tags\\C', 'tags\\B', 'tags\\D', 'tags\\B\\pw\\200'])

We can also look at "or". This is done with the union function, which is the set of elements in either set. The logical operator is |.

import pickle

with open('TAGS.pkl', 'r') as f:
    TAGS = pickle.loads(f.read())

print TAGS['1'].union(TAGS['2'])
print TAGS['1'] | TAGS['2']        # 1 or 2

set(['tags\\B\\eos\\2', 'tags\\B\\eos\\1', 'tags\\A\\eos\\1', 'tags\\A\\eos\\2', 'tags\\D\\eos\\1', 'tags\\D\\eos\\2', 'tags\\C\\eos\\2', 'tags\\C\\eos\\1'])
set(['tags\\B\\eos\\2', 'tags\\B\\eos\\1', 'tags\\A\\eos\\1', 'tags\\A\\eos\\2', 'tags\\D\\eos\\1', 'tags\\D\\eos\\2', 'tags\\C\\eos\\2', 'tags\\C\\eos\\1'])

There is an xor operator too. xor is not the same as or, it means A xor B means "A or B but not both". The symmetric_difference function gives this behavior. The logical operator is ^.

import pickle

with open('TAGS.pkl', 'r') as f:
    TAGS = pickle.loads(f.read())

# I think this is like the xor, A or 2 but not both
print TAGS['A'].symmetric_difference(TAGS['2']), '\n'
print TAGS['A'] ^ TAGS['2']

set(['tags\\A\\pw\\300', 'tags\\A\\pw\\250', 'tags\\A\\eos', 'tags\\B\\eos\\2', 'tags\\A\\eos\\4', 'tags\\A\\kpts\\2x2x2', 'tags\\A\\pw\\400', 'tags\\A\\kpts\\4x4x4', 'tags\\A\\eos\\3', 'tags\\A\\kpts\\8x8x8', 'tags\\A\\pw\\350', 'tags\\A\\eos\\5', 'tags\\D\\eos\\2', 'tags\\A\\kpts', 'tags\\A', 'tags\\C\\eos\\2', 'tags\\A\\eos\\1', 'tags\\A\\pw', 'tags\\A\\pw\\200']) 

set(['tags\\A\\pw\\300', 'tags\\A\\pw\\250', 'tags\\A\\eos', 'tags\\B\\eos\\2', 'tags\\A\\eos\\4', 'tags\\A\\kpts\\2x2x2', 'tags\\A\\pw\\400', 'tags\\A\\kpts\\4x4x4', 'tags\\A\\eos\\3', 'tags\\A\\kpts\\8x8x8', 'tags\\A\\pw\\350', 'tags\\A\\eos\\5', 'tags\\D\\eos\\2', 'tags\\A\\kpts', 'tags\\A', 'tags\\C\\eos\\2', 'tags\\A\\eos\\1', 'tags\\A\\pw', 'tags\\A\\pw\\200'])

To summarize, this is one approach to using tags with the technical infrastructure of the set. The advantage is flexibility, that you do not need to know the full path to a result, provided you know the set of tags that refers to it. You can use the tags in any order.

There are many alternative approaches to implementing this idea. One could create a sqlite table and do SQL queries. You could also store lists in the dictionary, and use python code to find the matches. The syntax here is varied. Using the functional approach, the syntax gets heavy with all the dot notation. With the logical operators, the syntax is a little lighter.

Copyright (C) 2014 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.5h

This post examines a different approach to literate programming with org-mode that uses noweb . I have adapted an example from http://home.fnal.gov/~neilsen/notebook/orgExamples/org-examples.html which has some pretty cool ideas in it.

The gist of using noweb is that in your source blocks you have labels like <<imports>>, that refer to other named code blocks that get substituted in place of the label. In the example below, we put labels for a code block of imports, for a function definition, a class definition, and a main function. This code block will get tangled to main.py . The noweb expansion happens at export, so here is the literal code block:

#+BEGIN_SRC python :noweb yes :tangle main.py
<<imports>>

<<some-func>>

<<class-dfn>>

<<main-func>>

if __name__ == '__main__':
    status = main()
    sys.exit(status)
#+END_SRC

You may want to just check out the org-mode source link at the bottom of the post to see all the details.

import sys
import numpy as np
import matplotlib.pyplot as plt

from argparse import ArgumentParser

def utility_func(arg=None):
    return 'you called a utility function with this arg: {0}'.format(arg)

class HelloWorld(object):
    def __init__(self, who):
        self.who = who

    def __call__(self):
        return 'Hello {0}'.format(self.who)

    def test(self):
        return True

def main():
    parser = ArgumentParser(description="Say hi")
    parser.add_argument("-w", "--who", 
                        type=str,
                        default="world",
                        help="Who to say hello to")
    args = parser.parse_args()
  
    who = args.who
  
    greeter = HelloWorld(who)
    greeter()

    print 'test func = ', greeter.test()
  
    print utility_func()
    print utility_func(5)

    return 0

if __name__ == '__main__':
    status = main()
    sys.exit(status)

import sys
import numpy as np
import matplotlib.pyplot as plt

from argparse import ArgumentParser

def utility_func(arg=None):
    return 'you called a utility function with this arg: {0}'.format(arg)

class HelloWorld(object):
    def __init__(self, who):
        self.who = who

    def __call__(self):
        return 'Hello {0}'.format(self.who)

    def test(self):
        return True

def test(self):
    return True

def main():
    parser = ArgumentParser(description="Say hi")
    parser.add_argument("-w", "--who", 
                        type=str,
                        default="world",
                        help="Who to say hello to")
    args = parser.parse_args()
  
    who = args.who
  
    greeter = HelloWorld(who)
    greeter()

    print 'test func = ', greeter.test()
  
    print utility_func()
    print utility_func(5)

    return 0

elisp:org-babel-tangle

python main.py --w John 2>&1
true

test func =  True
you called a utility function with this arg: None
you called a utility function with this arg: 5

from main import *

main()

test func =  True
you called a utility function with this arg: None
you called a utility function with this arg: 5

Copyright (C) 2014 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.5h

Today someone asked about deleting multiple elements from a list (actually it was about deleting multiple atoms from an ase.Atoms object, but some principles here apply. I will address that actual question later.).

Deleting multiple items from a list is not directly possible in one command in Python. There are a few approaches to accomplishing something like it. Which one is best depends on your objective.

One problem is when you delete an item, the indices of every item after it also changes. One strategy then is to delete the elements in descending order, i.e. delete the largest indices first. That way, you do not change the indices of the smaller indices, so you can still delete them. We can sort them in reverse order like this:

a = [1, 2, 5, 6, 7]

ind2remove = [1, 3]

for i in sorted(ind2remove, reverse=True): 
    del a[i]

print a

[1, 5, 7]

An alternative approach is to make a new list that only has the elements you want using list comprehension. For example:

a = [1, 2, 5, 6, 7]

ind2remove = [1, 3]

a = [x for i,x in enumerate(a) if i not in ind2remove]

print a

[1, 5, 7]

With numpy arrays you can delete multiple elements like this:

import numpy as np

a = np.array([1, 2, 5, 6, 7])

ind2remove = [1, 3]

print np.delete(a, ind2remove)
print a

[1 5 7]
[1 2 5 6 7]

The delete command makes a new object; the original list is unchanged. Numpy arrays are technically immutable, so the only way to do this is to make a copy. Another way is to use a boolean mask that only selects the indices where the mask is True, and not where they are False.

import numpy as np

a = np.array([1, 2, 5, 6, 7])

ind2remove = [1, 3]

mask = np.ones(len(a), dtype=bool) 
mask[ind2remove] = False
print a[mask]
print a

[1 5 7]
[1 2 5 6 7]

There might be other ways to do this too. These examples are nearly indistinguishable for small lists. For very large lists (I guess 1000's of elements), you may find one method more efficient than the others.

Copyright (C) 2014 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.5h

I am exploring the possibility of using tags on python objects in conjunction with searches to find sets of objects. Here I want to explore some syntax and methods for doing that.

In org-mode there is a syntax like '+boss+urgent-project1' for and and not operators and 'A|B' for or operators. I think we need pyparsing to untangle this kind of syntax. See http://pyparsing.wikispaces.com/file/view/simpleBool.py for an example. Another alternative might be the natural language toolkit (nltk ). Before we dig into those, let us see some python ways of doing the logic.

Below we define some lists containing tags (strings). We

a = ['A', 'B', 'C']

b = ['A', 'B']

c = ['A', 'C']

d = [ 'B', 'C']

all_lists = [a, b, c, d]

# get functions with tags A and B
print 'A and B ',[x for x in all_lists if ('A' in x) & ('B' in x)]

# A not B
print 'A not B ',[x for x in all_lists if ('A' in x) & ('B' not in x)]

# B or C
print 'B or C ', [x for x in all_lists if ('B' in x) | ('C' in x)]

# B or C but not both
print 'B xor C ',[x for x in all_lists if ('B' in x) ^ ('C' in x)]

A and B  [['A', 'B', 'C'], ['A', 'B']]
A not B  [['A', 'C']]
B or C  [['A', 'B', 'C'], ['A', 'B'], ['A', 'C'], ['B', 'C']]
B xor C  [['A', 'B'], ['A', 'C']]

Those are not too bad. Somehow I would have to get pyparsing to generate that syntax. That will take a lot of studying. There are some other ways to do this too. Let us try that out with itertools.

a = ['A', 'B', 'C']

b = ['A', 'B']

c = ['A', 'C']

d = [ 'B', 'C']

all_lists = [a, b, c, d]

import itertools as it

# ifilter returns an iterator
print 'A and B ', list(it.ifilter(lambda x: ('A' in x) & ('B' in x), all_lists))

A and B  [['A', 'B', 'C'], ['A', 'B']]

I do not like this syntax better. The iterator is lazy, so we have to wrap it in a list to get the results. Eventually, I want to do something like these:

filter('A and B', all_lists)
A or B
A xor B
not A and B
not(A and B)

I think that calls for pyparsing. I think the syntax above is better (more readable) than this:

filter('A & B', all_lists)
A | B
A ^ B
~A & B
~(A & B)

It is not that obvious though how to get from that syntax to the code I illustrated above though.

Copyright (C) 2014 by John Kitchin. See the License for information about copying.

org-mode source

Org-mode version = 8.2.5h