Beyond independent error assumptions in large GNN atomistic models
Janghoon Ock, Tian Tian, John Kitchin and Zachary Ulissi, “Beyond independent error assumptions in large GNN atomistic models”, The Journal of Chemical Physics 158(21) (2023). https://doi.org/10.1063/5.0151159
This summary was written by a language model from the paper and has not been checked line by line. Rely on the published paper for details.
Note: the extracted text available for this summary is the arXiv preprint (arXiv:2303.10797v1, March 2023) with its supporting information, not the final published version, so details may differ slightly from the journal article.
Question
Most practical uses of adsorption energies in catalysis are relative: finding the lowest-energy configuration on a surface, comparing catalysts along a volcano, or computing a reaction energy from two intermediates on one surface. DFT is known to be more reliable for such differences than for absolute values because its functional errors are correlated between similar structures and partly cancel on subtraction. Graph neural networks (GNNs) trained on the Open Catalyst 2020 (OC20) data are benchmarked by the mean absolute error of individual energies, and they are not trained on differences. The paper asks whether GNN errors are also correlated, so that they cancel in energy differences, or whether they behave as independent errors that add in quadrature and amplify the error of a difference (Ock et al. 2023).
Approach
The authors used the heuristic part of the OC20-Dense dataset: 995 adsorbate-catalyst combinations spanning 76 adsorbates and 850 bulk catalysts, with about 31 configurations each, for 31,081 systems and roughly 483 million system pairs. Each was relaxed with RPBE DFT and with five GNNs of successive generations (SchNet, DimeNet++, PaiNN, GemNet-OC and GemNet-OC-Large). Pairs were grouped by what they share: pairs differing only in the surface, pairs differing only in the adsorbate, pairs on the same surface with the same adsorbate but a different configuration (further restricted to pairs within 1.0, 0.5 or 0.2 eV of the surface minimum), and pairs sharing nothing. (The preprint’s bulleted definitions of the catalyst-swap and adsorbate-swap groups appear interchanged relative to its figure caption; the caption’s meaning is used here.) As a DFT reference, BEEF-vdW ensembles were computed on 1,529 of the RPBE-relaxed systems from 201 combinations. They defined a subgroup error cancellation ratio (SECR), one minus the ratio of the error standard deviation in a subgroup to that over all pairs, and a marginal correct sign ratio (MCSR), the fraction of pairs whose DFT difference is below 0.2 eV in magnitude that the GNN gives the correct sign. Finally, they compared atom embeddings of the adsorbate and its nearest surface neighbors between pairs using the RV coefficient.
Key findings
The chemically similar subgroups are a tiny fraction of all pairs: fewer than 7% share a catalyst or adsorbate, the adsorbate-sharing and configuration-sharing groups are 0.03% and 0.22%, and the mean DFT differences within the 0.5 and 0.2 eV configuration windows are only 0.14 and 0.05 eV, below the MAE of current GNNs. For BEEF-vdW, pairs sharing nothing had a negative SECR (about -3%), meaning amplified error, while the similar groups reached 42% to 77%; a supporting example showed an ensemble difference spread of 0.17 eV against 0.31 eV expected for independent errors. All GNNs except SchNet had positive SECR in every similar subgroup, which the authors present as the first demonstration that large GNN atomistic model errors are not independent. Cancellation was strongest for configuration pairs near the minimum, reaching 71% for GemNet-OC-Large at 0.5 and 0.2 eV, comparable to DFT. For GNNs, however, the error standard deviation of differences (0.96 eV for GemNet-OC over all pairs, against 0.68 eV for individual energies) fell below the individual-energy level only in the thresholded configuration groups (0.28 to 0.36 eV for the GemNet models), whereas DFT achieved this in all similar groups (0.10 to 0.25 eV versus 0.38 eV). MCSR at a 0.2 eV margin was 54% for SchNet and 64% overall for GemNet-OC, rising to 70% to 71% in the 0.5 eV configuration window. SECR did not improve steadily as MAE fell: GemNet-OC and GemNet-OC-Large showed less cancellation than DimeNet++ in several groups despite lower MAE. GemNet-OC-Large, same architecture as GemNet-OC but trained with extra molecular dynamics data, produced more similar adsorbate-neighbor embeddings for similar pairs and higher SECR, and those atoms accounted for 0.8 to 1.2 times the predicted adsorption energy in about 76% of cases.
Significance
The work reframes how GNN catalysis models should be judged: for screening and minimum-energy searches the relevant quantity is the error of differences between related systems, and error cancellation in those subgroups is proposed as a performance metric in its own right. It also suggests that training for correlated embeddings of similar structures could improve difference accuracy without changing the architecture.