Transfer Learning

A research topic across the group’s papers.

Transfer learning reuses a model trained on one, usually much larger, data set or task as the starting point for a related one. Common variants freeze the learned feature layers and refit only a small output layer, or initialize all of a network’s weights from a pretrained checkpoint and then fine-tune on the target data. In atomistic machine learning it is used to cut the amount of new reference data and training time needed for new chemistries or properties, to reach higher accuracy with the same data, and to carry knowledge between chemical domains. See also Neural network potentials and Graph neural networks.

Freezing shared layers to fit a new element (2020)

Liu and Kitchin trained their shared-weight SingleNN potential on five of six pure elements (Li, Mo, Cu, Ni, Si, Ge), froze the nonlinear layers, and fit only the linear output weights on the held-out element. This was worse than training on all six but never worse than linear regression on the fingerprints, and it beat linear regression consistently across five data splits for Cu and Ge (Ge 8.39 versus 12.60 meV/atom). It converged in fewer than 10 L-BFGS epochs, versus hundreds for full training, though errors varied more between splits because they depend on what the shared layers learned from other elements (Liu 2020). Summary: liu2020.

From heterogeneous to homogeneous catalysis data (2023)

Garrison and coworkers initialized GemNet-T from public OC20 checkpoints (adsorbates on surfaces) and fine-tuned it, with a tenfold smaller learning rate, on energies of transition metal complexes in tmQMwB97MV. On the full data set the test MAE fell by 35.8% relative to training from scratch (4.85 versus 7.54 meV/atom, with energy-within-1-kcal/mol rising from 11.65 to 17.05%); on the neutral subset the MAE fell by 14.2% (3.26 versus 3.80 meV/atom). This simple approach indicates that heterogeneous catalysis data can transfer to molecular catalyst systems (Garrison et al. 2023). Summary: garrison2023.

Fine-tuning an Open Catalyst checkpoint for alloy slabs (2024)

Broderick et al. (2024) fine-tuned a GemNet-OC checkpoint pretrained on OC20/OC22 to a few thousand PBE or PBEsol plus D3 alloy slabs. Atom-swap energy differences improved from 0.08 to 0.05 eV error, PBE models saturated after about 200 slabs and PBEsol models after about 1000, pretraining on PBE data helped the PBEsol models slightly, and including chemically similar ternaries let the model predict AuCuPd accurately with no AuCuPd training data. Summary: broderick2024.

Online fine-tuning of Open Catalyst models on new material classes (2024)

Wang et al. (2024) fine-tuned Open Catalyst GemNet models one DFT frame at a time during structural relaxations of alcohols, metal oxides, zeolites and MOFs. The choice of pretrained checkpoint mattered most for oxides: the OC20-only model converged none of the oxide relaxations, whereas models trained on OC22 or on OC20 plus OC22 converged 90% and 70% of them. The benefit of transfer shrank as systems moved further from the pretraining data, as judged by a PCA map of GemNet atom embeddings, and long-range magnetic effects were identified as the main obstacle. Summary: wang2024.

Fine-tuning potentials on finite-difference displacements (2025)

Fine-tuning a pretrained EquiformerV2 on energies and forces of small-displacement single points around OC20 minima roughly halved its frequency error on minima (MAE about 29 cm-1) and improved Gibbs corrections, but degraded the lowest (imaginary) frequency of transition states, since the fine-tuning data contained no saddle points. Fine-tuning data must resemble the target task to help it (Wander et al. 2025). Summary: wander2025.

Papers on this topic