Automating data sharing through authoring tools
John R. Kitchin, Ana E. Van Gulick and Lisa D. Zilinski, “Automating data sharing through authoring tools”, International Journal on Digital Libraries 18(2), 93-98 (2017; published online June 2016). https://doi.org/10.1007/s00799-016-0173-7
This summary was written by a language model from the paper and has not been checked line by line. Rely on the published paper for details.
Question
Funders, government policy and publishers increasingly expect researchers to share data, code and analysis pipelines, yet there is little guidance on what to share, how to share it, or how to link it to the publication. Data published as tables or figures in a PDF can usually be reused only by copying and pasting or digitizing, and supplementary files often lack metadata that would make them discoverable. The authors, a chemical engineer and two librarians at Carnegie Mellon, ask whether an authoring workflow can integrate data and code into a manuscript and then embed them automatically, in reusable form, into the published output (Kitchin et al. 2016).
Approach
The paper is a proof of concept built on Emacs and org-mode, a light-weight markup in which narrative, equations, figures, tables and executable code blocks share one document, and whose exporter parses the document into a data structure that can be translated to LaTeX, HTML and other formats. The key observation is that org tables and code blocks are “data” in the working document (a code block can read a named table directly), but this information is lost in a standard export. The authors therefore write a customized exporter in Emacs Lisp: derived HTML and LaTeX back ends that override only the handling of tables and source blocks. Domain-general examples illustrate the idea, and the authors note that MATLAB m-file publishing and IPython/Jupyter notebook conversion share the same working-to-consumption export idea and could implement it too.
Key findings
The running example computes the h-index of the first author from a table of citation counts for his 21 most cited papers (Scopus, as of May 20, 2015). An Emacs Lisp block reads the table from the document by name and loops to the answer, a Python block implements the same algorithm with the table passed in directly, and a third Python block plots the citation curve against a parity line; all give an h-index of 18. The authors emphasize that the working version of the document is functionally different from the consumption version, even though the reader of the PDF cannot see this.
During export, each table is written to a CSV file and each code block to a source file (named by its block name, or by an MD5 hash of its contents when unnamed). For LaTeX output these files are attached to the PDF with the attachfile package; for HTML output a link to the file and a base64 data URI are inserted. The names of all generated files are collected and, after the PDF is built, written into a custom PDF Info field called EmbeddedFiles using pdftk. The resulting manuscript PDF carries 16 embedded files, which can be listed with the Xpdf pdfdetach tool, while the Info field can be read with pdftk so that a program could find and analyze the embedded files. As a further demonstration of flexibility, the citation CSV was hidden inside a photograph of the Hunt Library using steganography (the steganopy Python package) and recovered with a short script; the authors stress that steganography is not the recommended mechanism, only evidence that the approach is general (XMP or other metadata could be used instead). The original manuscript source can also be embedded.
Because the embedded files are the very data and code used to prepare the document, the workflow removes errors from copying the wrong data or from updating external files without updating the paper, and once the framework exists it costs the author no extra effort. The full export code is given in the appendix.
Significance
The paper frames technical documents as existing in two versions, a functional working version that contains the data and a consumption version (PDF or HTML) derived from it, and shows that the derivation, including data embedding and machine-readable metadata, can be automated. It extends the group’s earlier manual practice of attaching data to Supporting Information (kitchin2015, kitchin2016a) toward a fully automated export step, and brings a library science perspective on discoverability: the authors note that making embedded data useful will require cooperation among publishers, researchers and search engines on tagging and indexing.