Open Challenges in Developing Generalizable Large-Scale Machine-Learning Models for Catalyst Discovery
Adeesh Kolluru, Muhammed Shuaibi, Aini Palizhati, Nima Shoghi, Abhishek Das, Brandon Wood, C. Lawrence Zitnick, John R. Kitchin and Zachary W. Ulissi, “Open Challenges in Developing Generalizable Large-Scale Machine-Learning Models for Catalyst Discovery”, ACS Catalysis 12(14), 8572-8581 (2022). https://doi.org/10.1021/acscatal.2c02291
This summary was written by a language model from the paper and has not been checked line by line. Rely on the published paper for details.
Question
Machine-learned potentials for catalysis had mostly been trained on narrow datasets (for example CO and H on metals for CO2 reduction) of 100 to 50,000 examples, which interpolate well but generalize poorly. The Open Catalyst 2020 (OC20) dataset, with 1.28 million DFT relaxations (about 260 million single points) over 55 elements, 82 adsorbates and unary, binary and ternary materials, made a universal potential conceivable. This perspective asks what has been learned from the first wave of graph neural network (GNN) models on OC20, where they still fail, and which modeling and evaluation choices matter for a generalizable catalysis potential (Kolluru et al. 2022).
Approach
The authors review community progress on the three OC20 tasks: structure to energy and forces (S2EF), initial structure to relaxed energy (IS2RE) and initial structure to relaxed structure (IS2RS). They break down GemNet-dT validation errors by material class (intermetallics, metalloids, nonmetals, halides) and by adsorbate, compare gradient-derived and directly predicted forces for SpinConv and GemNet-dT, compare direct and relaxation-based routes to relaxed energies and structures, provide new baselines for direct IS2RS prediction (ForceNet, SpinConv, GemNet-dT, predicting displacements from the initial positions), evaluate relaxation-based models trained on a 2M subset versus the full 133M training set, and test whether OC20’s ab initio MD (38M) and rattled (17M) extra data help DimeNet++ and GemNet-OC.
Key findings
Errors are not uniform. Nonmetals perform markedly worse than metalloids despite similar training shares and contribute disproportionately to overall error, while intermetallics do best. Adsorbate size does not correlate with error, but bidentate adsorbates such as *CH*COH, *N*NO and *CH2*O, and adsorbates containing N and O, are among the worst.
On OC20, directly predicted forces beat energy-gradient forces. GemNet-dT force MAE was 0.021 eV/Å in-domain with direct forces versus 0.030 eV/Å with gradients, and direct models also gave lower energy errors. Gradients add a factor of 2 to 4 in memory and time and sometimes destabilize training (ForceNet, GemNet-OC). Direct forces sacrifice energy conservation, so they suit relaxations rather than MD, and the authors note the opposite trend on MD17 is unexplained.
For IS2RE, direct models trained on about 460k relaxed structures (best test in-domain MAE about 0.40 eV, Graphormer) approach relaxation-based models trained on roughly 300 times more data, but relaxation approaches generalize better out of distribution. GemNet-OC trained on all 133M S2EF points reached 0.35 eV in-domain and 0.34 eV on the out-of-domain-both split, with 16.06% of in-domain energies within threshold. Models trained on only 2M points remained competitive (GemNet-OC 0.41 eV), which the authors offer as an accessible route since none of 30 entries in the NeurIPS 2021 challenge used relaxations, likely because full training needed 128 to 512 GPUs.
For relaxed structures, direct predictions reached ADwT near 46 to 48% (versus 21.18% for simply returning the initial structure) but almost zero AFbT, meaning DFT forces on the predicted geometries were large; relaxation-based GemNet-dT reached 60.88% ADwT and 20.35% AFbT. Distance metrics at tight thresholds correlate better with force metrics, but below 0.04 Å the signal vanishes, and DwT cannot substitute for AFbT. The authors propose counting the DFT steps needed to finish an ML-relaxed structure as a more practical metric and call for cheap non-DFT metrics that track the DFT-based ones.
Adding MD data improved GemNet-OC force MAE (0.0179 to 0.0173 eV/Å) and IS2RS AFbT (35.27% to 38.05%), whereas it worsened DimeNet++ force MAE while improving its AFbT; rattled data helped IS2RS but slightly hurt force MAE.
The outlook lists open problems: whether one model or subset-specific models are better (splitting off nonmetals hurt, likely from less data), uncertainty quantification and active learning at OC20 scale, inference cost of ever larger models, combining datasets at different levels of DFT theory, the direct-versus-gradient puzzle, and physics-informed inputs such as charge densities or tight-binding features.
This summary is based on the arXiv v2 preprint text (June 2022) rather than the final published version.
Significance
The paper set an agenda for the OC20 community and for later universal catalysis potentials: it identified nonmetals and N- and O-containing adsorbates as weak spots that later work linked in part to surface reconstructions in the training data, argued that DFT-verified force metrics rather than distance metrics are needed to judge relaxed structures, and made a practical case for direct-force models in relaxation workflows. Later group studies of correlated errors (Ock et al. 2023), error imbalance (Abdelmaqsoud et al. 2024) and zero-shot transition state searches (Wander et al. 2025) build on these models and questions.