Cross-Validation
Cross-validation estimates how well a fitted model predicts data it has not seen. Part of the data is held out, the model is fit to the rest, and the error on the held-out points is recorded; repeating this over many splits and averaging (often as a root-mean-square error) gives a cross-validation (CV) score. Because the score measures prediction rather than fit, it penalizes models with too many parameters that overfit the training data as well as models too simple to fit it, which makes it a standard criterion for choosing among candidate models or basis sets. In atomistic modeling it is used to select the terms of cluster expansions and to check machine-learned potentials fit to small DFT datasets.
CV scores for selecting cluster-expansion figures (2009)
Miller and Kitchin used exhaustive leave-one-, two- and three-out cross-validation to rank every complete figure set for cluster expansions of oxygen on Au(111) and Pt(111), discarding fits whose correlation matrices had condition numbers above 1000 and requiring at least 6000 valid prediction errors per figure set. They showed that figure sets with lower CV scores give narrower prediction-error distributions centered on zero, so the CV score doubles as an estimate of the expansion’s predictive uncertainty (about 20 meV rms for the best sets) (Miller 2009). Summary: miller2009a.
Bandwidth selection in kernel regression (2016)
Calfa and Kitchin chose the bandwidths of a categorical-data kernel regression model for crystal properties by least-squares leave-one-out cross-validation, minimizing the mean squared error of predictions in which each data point is left out of the weighted average (Calfa 2016). They used multistart local optimization (five starts, R np package) and note that this criterion can automatically discount irrelevant predictors. Fitting the bandwidths for one property (all five starts) took roughly 20 to 85 min of wall time on a 32-CPU machine for data sets of 746 and 1173 crystals. Summary: calfa2016.