pybliometrics: Scriptable bibliometrics using a Python interface to Scopus
Michael E. Rose and John R. Kitchin, “pybliometrics: Scriptable bibliometrics using a Python interface to Scopus”, SoftwareX 10, 100263 (2019). https://doi.org/10.1016/j.softx.2019.100263
This summary was written by a language model from the paper and has not been checked line by line. Rely on the published paper for details.
Question
Scopus is one of the standard citation and abstract databases used in scientometrics, bibliometrics and the economics of science, but getting data out of it at scale is awkward. Manual downloads are capped (a search export is limited to 2000 records), are error prone and are hard to reproduce, while the few existing programmatic alternatives were written in PHP, bash, JavaScript or as repository plugins rather than in the language most data scientists already use. The paper asks whether a simple Python wrapper around the Scopus RESTful APIs can make Scopus data retrieval scriptable, cached and reproducible for researchers who do not know how REST interfaces or XML/JSON parsing work (Rose 2019).
Approach
This is a software publication describing the pybliometrics package (version 2.1, MIT license, distributed on PyPI, Python 3, built on the requests library). The package wraps eight Scopus APIs: the three search APIs (documents, authors, affiliations), the three retrieval APIs (abstracts, authors, affiliations) and two of the five metadata APIs (Citation Overview and Serial Title). Every class follows the same pattern: a search class takes a query string, while retrieval and metadata classes take a Scopus identifier for an author, document or affiliation; only the citation overview class needs an extra argument (the start year). Results are exposed as properties, mostly strings or lists of namedtuples, whose names follow the Scopus field names. Query results are cached to disk, a configuration file in a hidden .scopus folder in the user’s home directory holds the API credentials and cache paths (and is created interactively on first use), and a family of custom exceptions derived from those in requests allows error-specific handling. Some APIs require extra permissions from Elsevier, which the paper flags. The package is then demonstrated with four worked examples that combine it with the wider scientific Python stack (NetworkX, pandas, seaborn, scipy, nltk and scikit-learn).
Key findings
Downloading the records for every paper published in Science in 2010 returned 2260 documents, already more than the manual export allows; the first download took under 5 minutes depending on connection speed, while reuse from the cache took less than a second. A co-authorship network for the journal SoftwareX built with NetworkX contained 734 unique authors on 228 papers joined by 1636 edges, and because Scopus author profile IDs were used, no name disambiguation was needed. Using the restricted Citation Overview API, the authors collected yearly citation counts up to early 2018 for three papers cited in the Nobel decision for the economist Richard H. Thaler, and a two-sided Kolmogorov-Smirnov test (p < 2.9 x 10-16) rejected the hypothesis that the first two papers share the same citation distribution. Finally, abstracts of 226 SoftwareX papers (to June 2019) were analyzed with Latent Dirichlet Allocation assuming three topics; the topics were interpreted as technical terms describing the software, terms used to present it, and terms describing what it can be used for.
Significance
The authors argue that pybliometrics lowers the barrier to Scopus data to basic Python knowledge, and that scripted retrieval improves both reproducibility (data come from the same source by a documented procedure) and exactness (updated data can be pulled into an analysis easily). They also note a limit: Scopus itself is continually corrected, with records added and removed, so reproducibility of any database-derived result is inherently bounded. The package is aimed at science-of-science researchers and research evaluators, and integrates directly with visualization and machine learning tools in Python. Following a social-science guide to code and data that the paper cites, an automated retrieval script can serve as the authoritative definition of the final data set, a theme that connects to the group’s broader writing on shared, executable analyses (kitchin2015).