SingleNN: Modified Behler-Parrinello Neural Network with Shared Weights for Atomistic Simulations with Transferability
M. Liu and J. R. Kitchin, “SingleNN: Modified Behler-Parrinello Neural Network with Shared Weights for Atomistic Simulations with Transferability”, The Journal of Physical Chemistry C 124(32), 17811-17818 (2020). https://doi.org/10.1021/acs.jpcc.0c04225
This summary was written by a language model from the paper and has not been checked line by line. Rely on the published paper for details.
Question
Behler-Parrinello neural networks (BPNNs) predict the total energy as a sum of atomic energies, each computed by a separate network for that atom’s element from symmetry-function fingerprints of its local environment. The authors identify two limitations of this scheme. The fingerprints are system specific, since the number of G2 and G4 functions grows with the number of elements, so fingerprints for different chemistries are not compatible. And because each element has its own network, nothing learned for one element can be reused for another. The paper asks whether a single network whose nonlinear layers are shared by all elements can match BPNN accuracy, and whether those shared layers act as a transferable transformation that lets a new element be fit by training only a linear output layer (Liu 2020).
Approach
The fingerprint problem is handled with weighted symmetry functions, following Gastegger and coworkers, in which the contributions of each neighbor (or neighbor pair) are multiplied by an element-dependent weight, so the fingerprint length depends only on the number of η, γ and ζ values chosen and not on the number of elements. Of three candidate weights (atomic number, covalent radius and Allen electronegativity), electronegativity worked best on the full data set and was used throughout. The SingleNN architecture feeds every atom’s fingerprint into one network with one output per element; an element mask selects the relevant output, so all atoms share the hidden-layer weights and only the final linear weights differ by element. The motivating hypothesis is that there is a common, possibly universal, nonlinear map of fingerprints into a latent space in which atomic energies of all elements are nearly linear (motivated by correlated energies across metals in similar configurations, as in miller2011).
The test data were an existing DFT benchmark set of pure Li, Mo (bcc), Cu, Ni (fcc), Si and Ge (group IV), 1580 structures in total and about 250 per element. Every model used 10 weighted G2 and 4 weighted G4 functions and two hidden layers of 20 nodes, giving 4446 parameters for the six-element BPNN against 846 for SingleNN. Training used L-BFGS in PyTorch on rescaled fingerprints, energies and forces, and each element’s data were split 64/16/20 into train, validation (early stopping) and test sets with five random seeds. Four models were compared: linear regression on the fingerprints, BPNN, SingleNN trained on all six elements, and transfer learning in which SingleNN was trained on five elements, its nonlinear layers frozen, and only the output weights fit to the held-out element.
Key findings
Linear regression already gave small test errors for Li, Ni and Cu (energy MAEs of 1.5 to 3.6 meV/atom, force MAEs of about 0.02 to 0.04 eV/Å) but errors roughly an order of magnitude larger for Si, Ge and Mo (11.5 to 19.1 meV/atom, 0.13 to 0.24 eV/Å), indicating that those energies are nonlinear in the fingerprints. BPNN improved every element, at least halving the errors for Si, Ge and Mo (for example Mo at 5.59 meV/atom and 0.162 eV/Å). SingleNN reached energy MAEs slightly larger than BPNN (Mo 8.25, Si 6.15, Ge 5.10 meV/atom) but slightly smaller force MAEs (for example Si 0.071 versus 0.113 eV/Å), so one shared transformation was roughly as good as six element-specific ones with about a fifth of the parameters.
Transfer learning was worse than full SingleNN for every element, as expected since the held-out element did not shape the latent space, but it was never worse than linear regression. For Cu and Ge it was consistently better across all five splits (Ge at 8.39 versus 12.60 meV/atom), which the authors take as evidence that the shared layers learned something useful from the other elements; for Li, Ni, Si and Mo the two were similar. Transfer learning had larger spread across seeds because it depends on weights learned from other data. Linear regression and transfer learning converged in under 10 L-BFGS epochs versus hundreds for BPNN and SingleNN, and remained at least twice as fast when compared at equal error. The authors caution that, like any nonlinear model, the network should not be trusted outside the fingerprint space of its training data.
Significance
SingleNN shows that sharing nonlinear weights across elements costs little accuracy in a Behler-Parrinello-style potential while sharply cutting parameter count, and that the shared layers can be reused as a fixed feature transformation so new elements are fit by a fast linear solve. The authors expect transferability to improve as more elements are added to the training of the shared layers, and they propose testing it next on multicomponent systems.