Uncertainty Quantification
Uncertainty quantification estimates how confident a model’s predictions are, separating uncertainty that comes from limited or noisy data from uncertainty in the model itself. In data-driven science it is used to decide where predictions can be trusted and where new data are most valuable.
Ensemble variance as a trust threshold for a surrogate optimizer (2021)
Yang, Jiménez-Negrón and Kitchin estimated uncertainty from 10 neural networks trained on the same data with different random initializations, relying on overparameterization to make them diverse, and treated a prediction as untrustworthy when its total-energy variance exceeded alpha times the maximum variance on the training set, with alpha between 2 and 3 working for all their systems (Yang et al. 2021). The idea is that variance comparable to the training data signals interpolation, while larger variance signals extrapolation and triggers a new DFT calculation. Summary: yang2021.
Model-specific versus model-general uncertainty (2022)
Zhan and Kitchin asked which sources of uncertainty different methods actually capture, using properties extracted from energy-volume equations of state for Pd and Au (Zhan 2022). Delta method intervals from nonlinear regression and Bayesian regression posteriors (variational inference and Hamiltonian Monte Carlo) agreed closely with each other, but intervals from different analytical EOS forms did not overlap for the bulk modulus, so both capture only parameter uncertainty within the chosen model, which the authors call model-specific uncertainty. A Gaussian process fitted jointly to the function and its derivatives gave wider distributions that covered nearly every EOS model’s prediction, an epistemic, model-general uncertainty that includes model selection. Summary: zhan2022a.
The delta method for trained least squares models (2022)
Zhan and Kitchin showed that the multiparameter delta method gives a post hoc prediction uncertainty for any model fitted by least squares, including neural networks, using only the gradient of the prediction and the Hessian of the loss with respect to the parameters, both obtained by automatic differentiation (Zhan 2021). For nearly singular neural network Hessians they proposed a practical recipe (diagonal shift, pseudoinverse, scaling to the training residuals, clipping to positive semidefinite). On a SingleNN potential for AuPd slabs, 98% of test energies fell inside 95% prediction intervals, while slabs at unseen lattice constants had standard errors about ten times larger (0.023 and 0.034 versus 0.002 eV/atom), so the measure flags extrapolation even where its intervals are no longer quantitatively calibrated. Summary: zhan2022.
Uncertainty of a predicted classification boundary (2023)
Because Hessian-based parameter uncertainties suited to regression do not apply to classifiers, Bhat and Kitchin measured boundary uncertainty in two ways: the 95% confidence band derived from a support vector classifier’s class membership probabilities (reported as area scaled by boundary length), and the density of samples within a radius of each boundary point (Bhat 2023). They note that with noise-free grid data the band reflects only model uncertainty, while with simulated distance-dependent misclassification noise it captures both model and data uncertainty, and they used sample density directly as an acquisition criterion. Summary: bhat2023a.
Beyond independent error propagation (2023)
Ock et al. (2023) showed that treating GNN prediction errors as independent overestimates the uncertainty of energy differences between similar adsorbate-catalyst systems, because errors are correlated and partly cancel, as BEEF-vdW ensemble differences also show for DFT (0.17 eV observed versus 0.31 eV for independent errors in one example). The degree of cancellation depends strongly on how chemically related the two systems are, which is relevant to propagating uncertainty into volcano plots and reaction energies. Summary: ock2023.
Calibrated latent distance uncertainty for relaxed energies (2024)
Musielewicz, Lan, Uyttendaele and Kitchin benchmarked ensembles, latent space distances, mean variance estimation heads and a sequence regressor for the uncertainty of EquiformerV2 relaxed adsorption energies on OC20, recalibrating each with a linear fit of binned RMSE against root mean variance and testing on out-of-domain data (Musielewicz et al. 2024). Because the relaxed-energy error distribution is far from Gaussian, they argued that distribution-free measures (parity R2 of the error-based calibration plot and a bootstrap confidence interval on Var(Z)) are more reliable than miscalibration area, negative log likelihood or rank correlation, which disagreed with one another. A per-atom nearest-neighbor distance in GemNet-OC’s latent space was best calibrated (parity R2 0.952, test CI(Var(Z)) [1.13, 1.79]), ahead of the best ensemble (0.905, [1.42, 2.57]). Summary: musielewicz2024.
Shallow ensembles for graph neural network potentials (2025)
Vinchurkar, Abdelmaqsoud and Kitchin tested Direct Propagation of Shallow Ensembles (DPOSE), in which only the last layer of a network is split into many heads trained with a Gaussian negative log-likelihood loss, as a cheap substitute for deep ensembles in a 64-head SchNet (Vinchurkar et al. 2025). The ensemble variance flagged inputs far from the training data, especially new elements (QM9 variance about 0.09 for CF4 versus 62 for CCl4; gold frames at 0.00022 versus silver at 0.61304 eV/atom) and molecules larger than those in training, but it stayed flat for strained intermetallic slabs whose energy errors grew, and it did not separate bulk from amorphous gold. The variance is therefore a useful but incomplete out-of-domain signal. Summary: vinchurkar2025.