| categories: | tags:

TL;DR:

Today I learned you can make a Word document from org-mode with Word comments in them. This could be useful when working with collaborators maybe. The gist is you use html for the comment, then export to markdown or html, then let pandoc convert those to docx. A comment in HTML looks like this:

<span class="comment-start" author="jkitchin">Comment text</span>The text being commented on <span class="comment-end"></span>


Let's wrap that in a link for convenience. I use a full display so it is easy to see the comment. I only export the comment for markdown and html export, for everything else we just use the path. We somewhat abuse the link syntax here by using the path for the text to comment on, and the description for the comment.

(org-link-set-parameters
"comment"
:export (lambda (path desc backend)
(if (member backend '(md html))
(format "<span class=\"comment-start\" author=\"%s\">%s</span>%s<span class=\"comment-end\"></span>"
(user-full-name)
desc
path)
;; ignore for other backends and just use path
path))
:display 'full
:face '(:foreground "orange"))


Now, we use it like this This is the commentThis is the text commented on.

In org-mode it looks like:

To get the Word doc, we need some code that first exports to Markdown, and then calls pandoc to convert that to docx. Here is my solution to that. Usually you would put this in a subsection tagged with :noexport: but I show it here to see it. Running this block generates the docx file and opens it. Here I also leverage org-ref to get some citations and cross-references.

(require 'org-ref-refproc)
(let* ((org-export-before-parsing-hook '(org-ref-cite-natmove ;; do this first
org-ref-csl-preprocess-buffer
org-ref-refproc))
(md (org-md-export-to-markdown))
(docx (concat (file-name-sans-extension md) ".docx")))
(shell-command (format "pandoc -s %s -o %s" md docx))
(org-open-file docx '(16)))


The result looks like this in MS Word:

That is pretty remarkable. There are some limitations in Markdown, e.g. I find the tables don't look good, not all equations are converted, some cross-references are off. Next we add some more org-features and try the export with HTML.

1. export features for test

Test cross-references, references, equations, etc…

Aliquam erat volutpat (Fig. fig-2). Nunc eleifend leo vitae magna. In id erat non orci commodo lobortis. Proin neque massa, cursus ut, gravida ut, lobortis eget, lacus. Sed diam. Praesent fermentum tempor tellus. Nullam tempus &yang-2022-evaluat-degree. Mauris ac felis vel velit tristique imperdiet. Donec at pede. Etiam vel neque nec dui dignissim bibendum. Vivamus id enim. Phasellus neque orci, porta a, aliquet quis in Table tab-1, semper a, massa. Phasellus purus (eq-1). Pellentesque tristique imperdiet tortor. Nam euismod tellus id erat &kolluru-2022-open-chall.

 x y 1 3 3 6

We have equations:

$$\label{org9973acf} y = mx + b$$
• bullet1
• nested bullet
• bullet2

some defintions:

emacs
greatest editor
1. item 1
2. item 2

One equation: $$e^{i\pi} - 1 = 0$$

A second equation:

$$e^{i\pi} - 1 = 0$$

3. Alternate build with HTML.

Here we consider For example, htmlalternate build approaches.

Run this to get the docx file. I find this superior; it has references, cross-references, equations, tables, figures, etc. Even a title.

(let* ((org-export-before-parsing-hook '(org-ref-csl-preprocess-buffer
org-ref-refproc))
(org-html-with-latex 'dvipng)
(f (org-html-export-to-html))
(docx (concat (file-name-sans-extension f) ".docx")))
(shell-command (format "pandoc -s %s -o %s" f docx))
(org-open-file docx '(16)))


org-mode source

Org-mode version = 9.5.5

Update on finding the minimum distance from a point to a curve

| categories: optimization | tags:

Almost 10 years ago I wrote about finding the minimum distance from a point to a curve using a constrained optimization. At that time, the way to do this used scipy.optimize.fmin_coblya. I learned today from a student, that sometimes this method fails! I reproduce the code here, updated for Python 3, some style updates, and to show it does indeed fail sometimes, notably when the point is "outside" the parabola.

import numpy as np
import matplotlib.pyplot as plt
from scipy.optimize import fmin_cobyla

def f(x):
return x**2

for P in np.array([[0.5, 2],
[2, 2],
[-1, 2],
[-2, 2],
[0, 0.5],
[0, -0.5]]):

def objective(X):
X = np.array(X)
return np.linalg.norm(X - P)

def c1(X):
x,y = X
return f(x) - y

X = fmin_cobyla(objective, x0=[P[0], f(P[0])], cons=[c1])

print(f'The minimum distance is {objective(X):1.2f}. Constraint satisfied = {c1(X) < 1e-6}')

# Verify the vector to this point is normal to the tangent of the curve
# position vector from curve to point
v1 = np.array(P) - np.array(X)
# position vector
v2 = np.array([1, 2.0 * X[0]])
print('dot(v1, v2) = ', np.dot(v1, v2))

x = np.linspace(-2, 2, 100)

plt.plot(x, f(x), 'r-', label='f(x)')
plt.plot(P[0], P[1], 'bo', label='point')
plt.plot([P[0], X[0]], [P[1], X[1]], 'b-', label='shortest distance')
plt.plot([X[0], X[0] + 1], [X[1], X[1] + 2.0 * X[0]], 'g-', label='tangent')
plt.axis('equal')
plt.xlabel('x')
plt.ylabel('y')


The minimum distance is 0.86. Constraint satisfied = True dot(v1, v2) = 0.0002913487659186309 The minimum distance is 0.00. Constraint satisfied = False dot(v1, v2) = 0.00021460906432962284 The minimum distance is 0.39. Constraint satisfied = True dot(v1, v2) = 0.00014271520451364372 The minimum distance is 0.00. Constraint satisfied = False dot(v1, v2) = -0.0004089466778209598 The minimum distance is 0.50. Constraint satisfied = True dot(v1, v2) = 1.9999998429305957e-12 The minimum distance is 0.00. Constraint satisfied = False dot(v1, v2) = 8.588744170160093e-06

So, sure enough, the optimizer is failing to find a solution that meets the constraint. It is strange it does not work on the outside. That is almost certainly an algorithm problem. Here we solve it nearly identically with the more modern scipy.optimize.minimize function, and it converges every time.

from scipy.optimize import minimize

for P in np.array([[0.5, 2],
[2, 2],
[-1, 2],
[-2, 2],
[0, 0.5],
[0, -0.5]]):

def objective(X):
X = np.array(X)
return np.linalg.norm(X - P)

def c1(X):
x,y = X
return f(x) - y

sol = minimize(objective, x0=[P[0], f(P[0])], constraints={'type': 'eq', 'fun': c1})
X = sol.x

print(f'The minimum distance is {objective(X):1.2f}. Constraint satisfied = {sol.status < 1e-6}')

# Verify the vector to this point is normal to the tangent of the curve
# position vector from curve to point
v1 = np.array(P) - np.array(X)
# position vector
v2 = np.array([1, 2.0 * X[0]])
print('dot(v1, v2) = ', np.dot(v1, v2))

x = np.linspace(-2, 2, 100)

plt.plot(x, f(x), 'r-', label='f(x)')
plt.plot(P[0], P[1], 'bo', label='point')
plt.plot([P[0], X[0]], [P[1], X[1]], 'b-', label='shortest distance')
plt.plot([X[0], X[0] + 1], [X[1], X[1] + 2.0 * X[0]], 'g-', label='tangent')
plt.axis('equal')
plt.xlabel('x')
plt.ylabel('y')


The minimum distance is 0.86. Constraint satisfied = True dot(v1, v2) = 1.0701251773603815e-08 The minimum distance is 0.55. Constraint satisfied = True dot(v1, v2) = -0.0005793028003104883 The minimum distance is 0.39. Constraint satisfied = True dot(v1, v2) = -1.869272921939391e-05 The minimum distance is 0.55. Constraint satisfied = True dot(v1, v2) = 0.0005792953298950909 The minimum distance is 0.50. Constraint satisfied = True dot(v1, v2) = 0.0 The minimum distance is 0.50. Constraint satisfied = True dot(v1, v2) = 0.0

There is no wisdom in fixing the first problem, here I just tried a newer optimization method. Out of the box with default settings it just worked. I did learn the answer is sensitive to the initial guess, so it could make sense to sample the function and find the point that is closest as the initial guess, but here the simple heuristic guess I used worked fine.

org-mode source

Org-mode version = 9.5.5

Caching expensive function calls so you don't have to rerun them

| categories: python | tags:

Check out the video at:

Nobody likes to run expensive jobs more than necessary, so cache solutions are often used where you save the results, and look them up later. There is functools.cache in Python, but it is memory only, and not persistent, so you start over in a new session.

For persistent cache, joblib (https://joblib.readthedocs.io/en/latest/) is a standard tool for this. Here is a simple example:

from joblib import Memory
location = '/Users/jkitchin/Dropbox/emacs/journal/2023/02/01/joblib_cache/joblib_cache'
memory = Memory(location, verbose=0)

@memory.cache
def fun(x=1.0):
print('If you see this, go get a coffee while it runs 🐌.')
return x**2

print(fun(2)) # Runs the function
print(fun(2)) # Looks up the cached value


If you see this, go get a coffee while it runs 🐌. 4 4

That works because joblib saves the results in a file in the location you specify.

Here is another example with another arg.

@memory.cache
def f2(x=1.0, a=3):
print(f'If you see this, go get a coffee while it runs. a={"🐌"*a}')
return a*x**2

(f2(2),       # Runs function
f2(2, a=3),  # calls cache
f2(2, 4))    # Runs another function because a changed


If you see this, go get a coffee while it runs. a=🐌🐌🐌 If you see this, go get a coffee while it runs. a=🐌🐌🐌🐌

 12 12 16

Here, we look up from the cache each time.

(f2(2), f2(2, a=3), f2(2, 4))

 12 12 16

1. where things start to go wrong

1.1. Global variables

First, we look at an uncached version of a function that uses a global variable.

a = 3

def f3(x=1.0):
return a*x**2

f3(2)


12

We can change a and see the change.

a=0
f3(2)


0

Now we look at a cached version.

a = 3
@memory.cache
def f4(x=1.0):
print('If you see this, go get a coffee while it runs 🐌.')
return a*x**2

(f4(2), f4(2), f4(2))


If you see this, go get a coffee while it runs 🐌.

 12 12 12

Changing the global variable does not change the cache though. uh oh. This is just wrong. The answers should clearly be 0. Incorrect cache invalidation strikes.

a = 0
(f4(2), f4(2), f4(2))

 12 12 12

1.2. running functions with mutable arguments

Using mutable arguments is a recipe for trouble and unanticipated problems, but it is easy to unintentionally do, and not always obvious, as I show here.

from ase.build import bulk
from ase.calculators.emt import EMT

atoms = bulk('Pd')
atoms.set_calculator(EMT())

@memory.cache
def f(atoms):
print('If you see this, go get a coffee.')
return atoms.get_potential_energy()

(f(atoms), f(atoms))



If you see this, go get a coffee. If you see this, go get a coffee.

 0.000342263 0.000342263

You can see this ran twice. The reason is that the atoms object was mutated by adding data onto it. Here is how I know:

import joblib
atoms = bulk('Pd')
atoms.set_calculator(EMT())
joblib.hash(atoms), atoms._calc.results

 ee2ed2eb9fdb4b3d6416803a33f43a22 nil

Here you can see that simply running the get energy function the hash changes because the results dictionary on the calculator changes. That means subsequent uses of the atoms object will have a different hash, and you cannot rely on that to look up the results. In this case the results should not change the output of the function, but since they are included in the hash, it incorrectly invalidates the hash.

atoms.get_potential_energy()
joblib.hash(atoms), atoms._calc.results

 d37ef0a5761f499060b4f55bdf644814 (energy : 0.0003422625372841992 energies : array ((0.00034226)) free_energy : 0.0003422625372841992 forces : array (((0 0 0))))

Suffice to say, this is non-obvious, but having seen it, not a surprise; mutable arguments are frequently a source of problems.

1.3. If you run the same function different ways, the cache is not reused

Some aspects of this are specific to org-mode and how scripts are run in it. Here we have to use an absolute path to make sure we use the right cache. That still doesn't solve the problem though as we will see.

from joblib import Memory
location = '/Users/jkitchin/Dropbox/emacs/journal/2023/02/01/joblib_cache/joblib_cache'
memory = Memory(location, verbose=0)

a = 3
@memory.cache
def f4(x=1.0):
print('If you see this, go get a coffee while it runs')
return a*x**2

print((f4(2), f4(2), f4(2)))


The issue is that joblib uses the file name it thinks the function is from in the path it saves the results. The filename is different

1.4. Fragile cache invalidation

joblib uses the function source in its hash. That means any change to the source, including the function name, renaming variables, whitespace, comments or docstring changes invalidates the hash even though they may have no change in the output. That is an overabundance of caution, but simple to implement.

@memory.cache
def f4(x=1.0):
# comment
print('If you see this, go get a coffee while it runs')
return a*x**2

print((f4(2), f4(2), f4(2)))


If you see this, go get a coffee while it runs (0, 0, 0)

2. Some partial solutions with pycse.hashcache

I wrote hashcache to solve some of these problems. It is actually built on top of joblib.

import pycse
pycse.__version__, pycse.__file__

 2.2.1 /Users/jkitchin/Dropbox/python/pycse/pycse/__init__.py
from pycse.hashcache import hashcache

hashcache.location = "/Users/jkitchin/Dropbox/emacs/journal/2023/02/01/cache"
hashcache.verbose = False

@hashcache
def h1(x):
print('This runs soo slow... Go get a coffee')
return x**2

h1(2), h1(2)

 4 4

2.1. No known problem with global variables

a = 3
@hashcache
def h4(x=1.0):
print('If you see this, go get a coffee while it runs')
return a*x**2

(h4(2), h4(2), h4(2))



If you see this, go get a coffee while it runs

 12 12 12
a=0
(h4(2), h4(2), h4(2))

 0 0 0

Whew!!! we got the right answers. hashcache does a better job detecting the external change.

2.2. hashcache and mutable arguments

hashcache does not solve the mutable argument problem, but, it does warn you it detected it.

from ase.build import bulk
from ase.calculators.emt import EMT

atoms = bulk('Pd')
atoms.set_calculator(EMT())

@hashcache
def h(atoms):
print('If you see this, go get a coffee.')
return atoms.get_potential_energy()

(h(atoms), h(atoms), h(atoms))

 0.000342263 0.000342263 0.000342263

2.3. Reuse the cache when you run different ways

hashcache uses the same cache at the function and function environment level, so it avoids reruns even from different places. It is a judgement call by you to say if this is the right thing to do.

print(h1(2), h1(2))


4 4

from pycse.hashcache import hashcache
hashcache.location = "/Users/jkitchin/Dropbox/emacs/journal/2023/02/01/cache"

@hashcache
def h1(x):
print('This runs soo slow... Go get a coffee')
return x**2

print(h1(2), h1(2))


2.4. Insensitivity to unimportant changes

Instead of hashing the source of the function, in hashcache I hash the bytecode instead. This is certainly less sensitive to unimportant changes like docstrings, comments or whitespace. I do use the function name in the hash, so even though that does not affect the output, I thought it might be confusing in the future.

Here, small changes like comments, docstrings, etc, don't affect the hash.

a = 3
@hashcache
def h4(x=1.0):
'doc string'
print('If you see this, go get a coffee while it runs')
return a*x**2

(h4(2), h4(2), h4(2))

 12 12 12

Probably not completely. It is almost certain I have not captured all the ways the cache should be invalidated, or when a new cache should be used. hashcache is for now, a proof of concept in understanding why this is a hard problem to solve. I prefer its behavior over the defaults in joblib so far though.

org-mode source

Org-mode version = 9.5.5

2022 in a nutshell

| categories: news | tags:

2022 was an interesting one. I still did all of my teaching remotely, but spent more time going in to the office for meetings, and have gotten back to professional travel for meetings.

1. Research group accomplishments

Minjie Liu and Yilin Yang both defended their PhDs and graduated in 2022. Luyang Liu, Ananya Srivastava and Karan Waghela all completed their MS degrees and graduated in 2022 also. Congratulations to all of them!

Maya Bhat and I participated in an iCorps workshop on a concept around design of experiments.

We welcomed seven PhD students from the Ulissi group into the group while he is on leave at Meta. We also welcomed two new first year PhD students who will begin new collaborative projects with Carl Laird and Andy Gellman. The group is suddenly quite large!

2. Publications

Our work this past year was divided in a few efforts. We had some work in method development, e.g. in uncertainty quantification, automatic differentiation, and vector search.

1. Yang, Y., Achar, S. K., & Kitchin, J. R. (2022). Evaluation of the degree of rate control via automatic differentiation. AIChE Journal, 68(6). http://dx.doi.org/10.1002/aic.17653
2. Yang, Y., Liu, M., & Kitchin, J. R. (2022). Neural network embeddings based similarity search method for atomistic systems. Digital Discovery. http://dx.doi.org/10.1039/d2dd00055e
3. Zhan, N., & Kitchin, J. R. (2022). Model-specific to model-general uncertainty for physical properties. Industrial & Engineering Chemistry Research, 1–04706. http://dx.doi.org/10.1021/acs.iecr.1c04706

This collaborative work on large catalyst models is was especially exciting. Stay tuned for many advances in this area in 2023.

1. Kolluru, A., Shuaibi, M., Palizhati, A., Shoghi, N., Das, A., Wood, B., Zitnick, C. L., … (2022). Open challenges in developing generalizable large-scale machine-learning models for catalyst discovery. ACS Catalysis, 12(14), 8572–8581. http://dx.doi.org/10.1021/acscatal.2c02291

We also wrote some collaborative papers on our work in high-throughput discovery of hydrogen evolution catalysts and segregation.

1. Broderick, K., Lopato, E., Wander, B., Bernhard, S., Kitchin, J., & Ulissi, Z. (2022). Identifying limitations in screening high-throughput photocatalytic bimetallic nanoparticles with machine-learned hydrogen adsorptions. Applied Catalysis B: Environmental, 121959. http://dx.doi.org/10.1016/j.apcatb.2022.121959
2. Bhat, M., Lopato, E., Simon, Z. C., Millstone, J. E., Bernhard, S., & Kitchin, J. R. (2022). Accelerated optimization of pure metal and ligand compositions for light-driven hydrogen production. Reaction Chemistry & Engineering. http://dx.doi.org/10.1039/d1re00441g
3. Yilin Yang, Zhitao Guo, Andrew Gellman and John Kitchin, Simulating segregation in a ternary Cu-Pd-Au alloy with density functional theory, machine learning and Monte Carlo simulations, J. Phys. Chem. C, 126, 4, 1800-1808. (2022). https://pubs.acs.org/doi/abs/10.1021/acs.jpcc.1c09647

It was another big year in citations for us (https://scholar.google.com/citations?user=jD_4h7sAAAAJ)!

3. Point Breeze Publishing, LLC

I started a publishing company this year Point Breeze Publishing, LLC. It is a way for me to sell booklets on using Python in Science and Engineering and to sustain the effort it takes to produce these. It has been a modest success so far, with about a dozen booklets that can help anyone get started from basic Python usage through data science and machine learning and design of experiments. For reading this, you can get 50% off all purchases with checkout code 2022-nutshell. Check it out, and leave a review if you get anything!

I am still working out what the next steps for this are. I have written most of the pycse content I had in mind now, 400+ pages of it. I would like to get these booklets in the hands of more students, and my stealthy advertising scheme on Twitter and YouTube has not made that happen yet. I have some ideas around molecular simulation, maybe a reboot of the DFT-book?, maybe something around scimax? Who knows, stay tuned!

4. Outlook for 2023

There will be lots of changes for the Kitchin Research Group in 2023. We had a massive growth at the end of 2022 as we welcomed many members of the Ulissi research group into our group while he is on leave at Meta for 2023. Last summer we had one PhD student and three MS students. Now we have 10 PhD students. That means we will start a lot of new research directions in large catalyst models and everything they enable. We have started several collaborations in the area of design of experiments, and look forward to seeing these grow. It should be exciting!

org-mode source

Org-mode version = 9.5.5

New publication - Identifying limitations in screening high-throughput photocatalytic bimetallic nanoparticles with machine-learned hydrogen adsorptions

| categories: | tags:

Hydrogen adsorption energies have long been used in screening to identify promising hydrogen evolution catalysts. In this work we combine a high-throughput experimental study of 5300 different bimetallic catalysts with a high-throughput computational screen of 16M adsorption energies to see how well adsorption energies work for this purpose. We developed a workflow to combine these data sets that accounts for surface stability and adsorption site distributions on alloy surfaces. We find that thermodynamically favorable adsorption energies are necessary to observe high activity, but they are not sufficient, and do not always lead to high activity.

@article{broderick-2022-ident-limit,
author = {Kirby Broderick and Eric Lopato and Brook Wander and Stefan Bernhard and John Kitchin and Zachary Ulissi},
title = {Identifying Limitations in Screening High-Throughput Photocatalytic Bimetallic Nanoparticles With Machine-Learned Hydrogen Adsorptions},
journal = {Applied Catalysis B: Environmental},
volume = {nil},
number = {nil},
pages = {121959},
year = {2022},
doi = {10.1016/j.apcatb.2022.121959},
url = {http://dx.doi.org/10.1016/j.apcatb.2022.121959},
DATE_ADDED = {Thu Sep 22 07:46:56 2022},
}


9