Reproducibility
Reproducibility is the ability of others to repeat a computational or experimental study and obtain the same result, and to build on it. For data-driven and AI-based work it depends on sharing far more than a narrative description: trained parameters, hyperparameters, data and preprocessing steps, code, and for language-model systems the model version and prompts. Related discussions appear in Agentic AI and in the summaries xin2025 and kitchin2025b.
Embedding data and code in publications (2015)
Kitchin argues that sharing becomes routine only when data, analysis and code are integrated into the authoring workflow itself (Kitchin 2015). He demonstrates attaching machine-readable data files (JSON, CSV, Excel) to Supporting Information PDFs and writing papers in org-mode so that tables are both human readable and usable by code, and shows how the JSON file from a DFT paper suffices to regenerate VASP inputs for an individual calculation. Where licenses forbid sharing some inputs (VASP pseudopotential files), he notes it is still usually possible to share enough for another expert to reproduce the work. Copying code into LaTeX does not guarantee it is what was run, whereas an executable document keeps code and results synchronized. See Data sharing, Literate programming and the summary kitchin2015.
Reproducibility and benchmarking in catalysis (2016)
In a multi-author Perspective on benchmarking, Bligaard et al. treat reproducibility as the precondition for fair catalyst comparison (Bligaard et al. 2016). They note that molecular catalysis reports often cannot be repeated because syntheses, reagents and protocols are incompletely documented, and they list reporting practices to address this: purity data for the material actually used, rate laws rather than single high-conversion points, selectivity at stated conversion, TON above 1, transparent active-site counting and appropriate controls. They also argue that journals, funders and institutions that reward novelty over openness make the problem worse, and point to the National Academies data-integrity report, the Nature reporting checklist and the Science TOP guidelines as signs of change. Summary: bligaard2016.
Embedding the exact data and code used (2017)
Because the files embedded by the automated export are the very data and code used to prepare the manuscript, the workflow removes errors from copying the wrong data in or from updating external files without updating the document (Kitchin et al. 2016). The manuscript source itself can also be embedded, so readers receive everything needed to regenerate the reported results. Summary: kitchin2017.
Reporting machine-learned models (2018)
Kitchin identifies reporting as perhaps the most significant challenge for ML in catalysis (Kitchin 2018). ML models can have hundreds of trained parameters plus many hyperparameters, are expressed as programs rather than equations, are fit to datasets of hundreds to tens of thousands of points, and depend on normalization and scaling pipelines. Narrative Methods sections, figures and current supporting information formats are not effective for conveying all of this, so he argues the community must develop and adopt new ways of sharing data and code. Summary: kitchin2018.
Scripted data retrieval from bibliographic databases (2019)
Rose and Kitchin argue that retrieving Scopus data by script with pybliometrics, rather than through manual downloads, makes bibliometric studies more reproducible, because data come from the same source by a documented procedure, and more exact, because updated data are easily pulled into the analysis (Rose 2019). Following a cited practitioner’s guide for social-science code and data, the automated process acts as the authoritative definition of the final data set and avoids idiosyncratic manual errors. The paper also notes an intrinsic limit: Scopus records are frequently added, deleted and corrected, so results drawn from such a database can never be perfectly reproducible. Summary: rose2019.
Executable notebooks for a computational alchemy tutorial (2021)
Griego, Kitchin and Keith paired their tutorial on alchemical perturbation DFT with an open-source Python package (phystone) and Binder-hosted Jupyter notebooks that regenerate the paper’s binding energy and barrier benchmark figures interactively from the VASP output files, and let readers apply the same workflow to their own reference calculations. They present this as a way to keep computational catalysis screening updatable and reproducible (Griego et al. 2020). Summary: griego2021.
Trust in self-driving laboratory data (2025)
Canty et al. argue that concerns about whether self-driving laboratory data can be trusted must be met by recording not only inputs and outputs but state and environmental metadata (such as temperature and relative humidity), by calibration, benchmarking and control studies built into workflows, and by reporting operational performance, degree of human involvement, resources used, waste and shortcomings (Canty et al. 2025). They note that few SDL technologies currently include repeatability or reproducibility metrics, that no standards or incentives for such studies exist, and recommend releasing code and data at publication and giving SDLs identifiers (analogous to ORCID) for provenance. Summary: canty2025.
Journal reproducibility policy for computational chemistry (2026)
A joint JCTC and JCIM editorial, with Kitchin among the editors, makes availability of the data and software needed to reproduce key results expected at submission for original research from May 1, 2026, with a required Data and Software Availability Statement (Amaro et al. 2026). It lists the materials to share as appropriate (code and scripts, representative inputs and outputs with a README, coordinates and custom parameters, key raw numbers, restart files, MD trajectories, ML models with hyperparameters and fixed train/validation/test splits, and new potential energy surfaces with their software), asks for exact software versions, dependencies and environments, and requires that confidential or licensed components not be indispensable to reproducing the main claims. It echoes Kitchin’s earlier arguments about reporting ML models (kitchin2018). Summary: amaro2026.
Reproducibility and longevity of published tools (2026)
The JCIM Application Note editorial treats sustainability, reproducibility and reuse as key evaluation criteria for digital resources (Nunes Alves et al. 2026). Reviewers must be given anonymous access to the resource and judge its functionality, reliability and documentation, and web services must be backed by an open-source repository so the community can restore them if hosting disappears. Summary: nunesalves2026.
Reproducibility for agentic catalysis: TRACE-AI (2026)
The TRACE-AI community guidelines argue that for AI agents running stochastic closed-loop campaigns, exact repetition of a trajectory is neither realistic nor meaningful, and reproducibility should instead mean end-to-end traceability plus statistical consistency, with independent runs under comparable constraints converging to similar regions of design space or performance distributions (Xin et al. 2026). The paper pairs this with an 18-item publication checklist spanning data management, model development, workflow integration and executable artifacts (for example, code that regenerates the ML-ready dataset, leakage-preventing splits, metrics with uncertainty rather than best-only results, logged trajectories, and documented execution environments). Summary: xin2026a.