Investigating the error imbalance of large-scale machine learning potentials in catalysis
Kareem Abdelmaqsoud, Muhammed Shuaibi, Adeesh Kolluru, Raffaele Cheula and John R. Kitchin, “Investigating the error imbalance of large-scale machine learning potentials in catalysis”, Catalysis Science & Technology 14(20), 5899–5908 (2024). https://doi.org/10.1039/d4cy00615a
This summary was written by a language model from the paper and has not been checked line by line. Rely on the published paper for details.
Question
Machine learning potentials (MLPs) trained on the Open Catalyst 2020 (OC20) dataset, about 1.28 million DFT relaxations spanning 55 elements and 82 adsorbates, have seen their energy errors on the structure-to-energy-and-forces task level off near 0.2 eV over two years, despite increasingly elaborate architectures. The target is about 0.1 eV, roughly the intrinsic error of DFT. Earlier work had also found that errors are uneven across material classes, with nonmetals worst and intermetallics best. The paper asks whether this plateau and imbalance come from non-systematic errors in the training data, and tests two candidate sources: incomplete convergence of the DFT settings, which were fixed for all of OC20, and adsorbate-induced surface reconstruction, which corrupts the adsorption energy referencing scheme (Abdelmaqsoud et al. 2024).
Approach
For convergence, the authors took the one hundred nonmetallic systems with the highest ML errors and ran VASP convergence tests on the k-point multiplier (40 versus 80), plane-wave cutoff (350 to 550 eV) and smearing width (0.2 down to 0.01 eV). From these they chose tighter settings and recomputed single points for the OC20-200k training split and a 30k in-domain validation split, re-relaxing the reference slabs at the tighter settings. GemNet-OC models were then trained and evaluated at both the original and the tighter settings.
For reconstruction, they used the AdsorbML anomaly detector, which compares atom connectivity matrices (covalent radii scaled by 1.5) of the relaxed clean slab and of the relaxed adsorbate+slab with the adsorbate removed. Pre-trained GemNet-OC-Large, eSCN and EquiformerV2 models were evaluated on 30k samples from each in-domain and out-of-domain validation set with and without the flagged systems. They also retrained GemNet-OC from scratch on 156k structures with and without reconstructions, and compared a GemNet-OC model trained on raw total energies (OC20 plus OC22) against an adsorption energy model on the IS2RE validation sets, relaxing structures with ML to 0.03 eV/Å or 300 steps (Abdelmaqsoud et al. 2024).
Key findings
Forces were converged within 0.03 eV/Å and k-point errors were under 0.05 eV in total energy and 0.02 eV in adsorption energy. Plane-wave cutoff errors for the selected nonmetals reached about 2 eV at 350 eV, falling to a mean absolute value near 0.03 eV at 500 eV. Smearing errors reached about 0.15 eV. A smearing width of 0.1 eV left 2.5% of calculations electronically unconverged, against 6% at 0.05 eV, so the tighter settings were a multiplier of 40, a 500 eV cutoff and 0.1 eV smearing. On the 200k split, total energy differences exceeded 0.1 eV with nonmetals largest (about 0.15 eV mean absolute), but error cancellation brought adsorption energy differences to about 0.05 eV for all classes. Models trained at the tighter settings had essentially the same MAEs, and nonmetals and halides stayed worst, so convergence errors do not explain the imbalance.
About 22% of OC20 calculations showed surface reconstruction, more often for nonmetals and halides. Removing them from validation without retraining cut the average energy MAE from 0.248 to 0.160 eV (GemNet-OC), 0.245 to 0.156 eV (eSCN) and 0.226 to 0.137 eV (EquiformerV2), roughly 35%, with the largest drops for nonmetals and halides, while force errors were unchanged. Removing them from training as well only lowered the MAE from 0.31 to 0.29 eV, which the authors liken to models being robust to label noise when trained on large data. Total energy MAEs were not affected by reconstructions. The total energy model gave adsorption energy MAEs of 0.382 eV against 0.359 eV for the adsorption energy model, with large cancellation of errors between slab and adsorbate+slab predictions (about 50% on the out-of-domain catalyst sets) and a more even distribution across material classes (Abdelmaqsoud et al. 2024).
Significance
The work rules out DFT convergence as the source of the OC20 error plateau and points instead to an ill-posed target: when the adsorbate reconstructs the surface, the label includes a reconstruction energy that the clean-slab reference cannot capture. It argues for consistent referencing in catalysis datasets, for checking convergence on representative subsets rather than assuming cancellation, and for total energy models, which avoid the referencing problem, do not need a DFT-relaxed slab, and can predict clean slab energies. The authors flag the smearing scheme for semiconductors and insulators, and Hubbard U and magnetic initialization in OC22, as further possible sources of inconsistency (Abdelmaqsoud et al. 2024).