# Maneage: Customizable Template for Managing Data Lineage

Mohammad Akhlaghi1,2, Raúl Infante-Sainz1,2
1 Instituto de Astrofísica de Canarias, C/Vía Láctea, 38200 La Laguna,
Tenerife, ES.
2 Facultad de Física, Universidad de La Laguna, Avda. Astrofísico Fco. Sánchez
s/n, 38200, La Laguna, Tenerife, ES.
Corresponding author: Mohammad Akhlaghi
(mailto:mohammad<EMAIL_ADDRESS>
The era of big data has also ushered an era of big responsibility. Without it,
the integrity of the results will be a subject of perpetual debate. In this
paper, Maneage (management + lineage) is introduced as a low-level solution.
Maneage is a publishing and archival friendly data lineage management system
(in machine-actionable plain-text) for projects in the sciences or industry.
Its core principles include: stand-alone (e.g., not requiring anything beyond
a POSIX-compatible system, administrator privileges, or a network connection),
modular, straight-forward design, traceable input and output, temporal
lineage/provenance and free software (for scientific applications). A project
that uses Maneage will be able to publish the complete data lineage, making it
exactly reproducible (as a test on sufficiently conveying the data lineage).
The offered lineage/control isn’t limited downloading the raw input data and
processing them automatically, but also includes building the necessary data
analyze software with fixed versions and build configurations. Additionally,
Maneage also includes the final PDF report of the project, establishing direct
links between the data analysis and the narrative (with the precision of
sentence). Maneage enables incremental projects, where a new project can
branch off an existing one making only moderate changes and experimentation on
the methods. It can also be used on more ambitious projects once a
sufficiently large number of projects use it, for example automatic workflow
creation through machine learning tools, or automating data management plans.
As a demonstration, this paper is written using Maneage (snapshot ).

Keywords: Data Lineage, Data Provenance, Reproducibility, Scientific
Pipelines, Workflows

##  1 Introduction

The increasing volume and complexity of data analysis has been highly
productive, giving rise to a new branch of “Big Data” in many fields of the
sciences and industry. However, given its inherent complexity, the mere
results are barely useful alone. Questions such as these commonly follow any
such result: What inputs were used? What operations were done on those inputs?
How were the configurations or training data chosen? How did the quantitative
results get visualized into the final demonstration plots, figures or
narrative/qualitative interpretation? May there be a bias in the
visualization? See Figure 1 for a more detailed visual representation of such
questions for various stages of the workflow.

In data science and database management, this type of metadata are commonly
known as _data provenance_ or _data lineage_. Their definitions are elaborated
with other basic concepts in Section 2. Data lineage is being increasingly
demanded for integrity checking from both the scientific and industrial/legal
domains. Notable examples in each domain are respectively the “Reproducibility
crisis” in the sciences that was claimed by the Nature journal baker16, and
the General Data Protection Regulation (GDPR) by the European Parliament and
the California Consumer Privacy Act (CCPA), implemented in 2018 and 2020
respectively. The former argues that reproducibility (as a test on
sufficiently conveying the data lineage) is necessary for other scientists to
study, check and build-upon each other’s work. The latter requires the data
intensive industry to give individual users control over their data,
effectively requiring thorough management and knowledge of the data’s lineage.
Besides regulation and integrity checks, having a robust data governance
(management of data lineage) in a project can be very productive: it enables
easy debugging, experimentation on alternative methods, or optimization of the
workflow.

In the sciences, the results of a project’s analysis are published as
scientific papers which have also been the primary conveyor of the result’s
lineage: usually in narrative form, within the “Methods” section of the paper.
From our own experiences, this section is usually most discussed during peer
review and conference presentations, showing its importance. After all, a
result is defined as “scientific” based on its _method_ (the “scientific
method”), or lineage in data-science terminology. In the industry however,
data governance is usually kept as a trade secret and isn’t publicly published
or scrutinized. Therefore while the proposed approach introduced in this paper
(Maneage) is also useful in industrial contexts, the main practical focus
would be in the scientific front which has traditionally been more open to
publishing the methods and anonymous peer scrutiny.

figure-project-outline

Figure 1: Graph of a generic project’s workflow (connected through arrows),
highlighting the various issues/questions on each step. The green boxes with
sharp edges are inputs and the blue boxes with rounded corners are the
intermediate or final outputs. The red boxes with dashed edges highlight the
main questions on the respective stage. The orange box surrounding the
software download and build phases marks shows the various commonly recognized
solutions to the questions in it, for more see Appendix A.5.

The traditional format of a scientific paper has been very successful in
conveying the method with the result in the last centuries. However, the
complexity mentioned above has made it impossible to describe all the
analytical steps of a project to a sufficient level of detail. Citing this
difficulty, many authors suffice to describing the very high-level
generalities of their analysis, while even the most basic calculations (like
the mean of a distribution) can depend on the software implementation.

Due to the complexity of modern scientific analysis, a small deviation in the
final result can be due to many different steps, which may be significant.
Publishing the precise codes of the analysis is the only guarantee. For
example, smart18 describes how a 7-year old conflict in theoretical condensed
matter physics was only identified after the relative codes were shared.
Nature is already a black box which we are trying hard to unlock, or
understand. Not being able to experiment on the methods of other researchers
is an artificial and self-imposed black box, wrapped over the original, and
taking most of the energy of researchers.

miller06 found that a mistaken column flipping, leading to retraction of 5
papers in major journals, including Science. baggerly09 highlighted the
inadequate narrative description of the analysis and showed the prevalence of
simple errors in published results, ultimately calling their work “forensic
bioinformatics”. herndon14 and [a self-correction]horvath15 also reported
similar situations and ziemann16 concluded that one-fifth of papers with
supplementary Microsoft Excel gene lists contain erroneous gene name
conversions. Such integrity checks tests are a critical component of the
scientific method, but are only possible with access to the data and codes.

The completeness of a paper’s published metadata (or “Methods” section) can be
measured by a simple question: given the same input datasets (supposedly on a
third-party database like http://zenodo.orgzenodo.org), can another researcher
reproduce the exact same result automatically, without needing to contact the
authors? Several studies have attempted to answer this with different levels
of detail. For example allen18 found that roughly half of the papers in
astrophysics don’t even mention the names of any analysis software they have
used, while menke20 found that the fraction of papers explicitly mentioning
their tools/software has greatly improved in medical journals over the last
two decades.

ioannidis2009 attempted to reproduce 18 published results by two independent
groups but, only fully succeeded in 2 of them and partially in 6. chang15
attempted to reproduce 67 papers in well-regarded economic journals with data
and code: only 22 could be reproduced without contacting authors, and more
than half could not be replicated at all. stodden18 attempted to replicate the
results of 204 scientific papers published in the journal Science _after_ that
journal adopted a policy of publishing the data and code associated with the
papers. Even though the authors were contacted, the success rate was $26\%$.
Generally, this problem is unambiguously felt in the community: baker16
surveyed 1574 researchers and found that only $3\%$ did not see a
“reproducibility crisis”.

This is not a new problem in the sciences: in 2011, Elsevier conducted an
“Executable Paper Grand Challenge” gabriel11. The proposed solutions were
published in a special edition. Some of them are reviewed in Appendix B, but
most have not been continued since then. Before that, ioannidis05 proved that
“most claimed research findings are false”. In the 1990s, schwab2000,
buckheit1995, claerbout1992 describe this same problem very eloquently and
also provided some solutions that they used. While the situation has improved
since the early 1990s, these papers still resonate strongly with the
frustrations of today’s scientists. Even earlier, through his famous quartet,
anscombe73 qualitatively showed how distancing of researchers from the
intricacies of algorithms/methods can lead to misinterpretation of the
results. One of the earliest such efforts we found was roberts69 who discussed
conventions in FORTRAN programming and documentation to help in publishing
research codes.

From a practical point of view, for those who publish the data lineage, a
major problem is the fast evolving and diverse software technologies and
methodologies that are used by different teams in different epochs. zhao12
describe it as “workflow decay” and recommend preserving these auxiliary
resources. But in the case of software its not as straightforward as data: if
preserved in binary form, software can only be run on certain hardware and if
kept as source-code, their build dependencies and build configuration must
also be preserved. gronenschild12 specifically study the effect of software
version and environment and encourage researchers to not update their software
environment. However, this is not a practical solution because software
updates are necessary, at least to fix bugs in the same research software.
Generally, software is not a secular component of projects, where one software
can easily be swapped with another. Projects are built around specific
software technologies, and research in software methods and implementations is
itself a vibrant research topic in many domains dicosmo19.

add a short summary of the advantages of Maneage.

This paper introduces Maneage as a solution to these important issues. Section
2 defines the necessary concepts and terminology used in this paper leading to
a discussion of the necessary guiding principles in Section 3. Section 4
introduces the implementation of Maneage, going into lower-level details in
some cases. Finally, in Section 5, the future prospects of using systems like
this template are discussed. After the main body, Appendix A reviews the most
commonly used lower-level technologies used today. In light of the guiding
principles, in Appendix B a critical review of many workflow management
systems that have been introduced over the last three decades is given.
Finally, in Appendix D we acknowledge the various software (with a name and
version number) that were used for this project.

##  2 Definition of important terms

The concepts and terminologies of reproducibility and project/workflow
management and design are commonly used differently by different research
communities or different solution provides. As a consequence, before starting
with the technical details it is important to clarify the specific terms used
throughout this paper and its appendix.

###  2.1 Definition: input

Any computer file that may be usable in more than one project. The inputs of a
project include data, software source code, etc. (see hinsen16 on the
fundamental similarity of data and source code). Inputs may be encoded in
plain text (for example tables of comma-separated values, CSV, or processing
scripts), custom binary formats (for example JPEG images), or domain-specific
data formats [e.g., FITS in astronomy, see][]pence10.

Inputs may have initially been created/written (e.g., software source code) or
collected (e.g., data) for one specific project. However, they can, and most
often will, be used in other/later projects also. Following the principle of
modularity, it is therefore optimal to treat the inputs of any project as
independent entities, not mixing them with how they are managed (how software
is run on the data) within the project (see Section 2.3).

Inputs are nevertheless necessary for building and running any project. Some
inputs may already archived/published independently prior to the project’s
publication. In this case, they can easily be downloaded and used by
independent projects. Otherwise, they can be published with the project, but
as independent files, for example see
https://doi.org/10.5281/zenodo.3408481zenodo.3408481 akhlaghi19.

###  2.2 Definition: output

Any computer file that is published at the end of the project. The output(s)
can be datasets (terabyte-sized, small table(s) or image(s), a single number,
a true/false (Boolean) outcome), automatically generated software source code,
or any other file. The raw output files are commonly supplemented with a
paper/report that summarizes them in a human-friendly
readable/printable/narrative format. The report commonly includes highlights
of the input/output datasets (or intermediate datasets) as plots, figures,
tables or simple numbers blended into the text.

The outputs can either be published independently on data servers which assign
specific persistent identifiers (PIDs) to be cited in the final report or
published paper (in a journal for example). Alternatively, the datasets can be
published with the project source, for example
https://doi.org/10.5281/zenodo.1164774zenodo.1164774 [Sections 7.3 &
3.4]bacon17.

###  2.3 Definition: project

The most high-level series of operations that are done on input(s) to produce
the output(s). Because the project’s report is also defined as an output (see
above), besides the high-level analysis, the project’s source also includes
scripts/commands to produce plots, figures or tables.

With this definition, this concept of a “project” is similar to “workflow”.
However, it is important to emphasize that the project’s source code and
inputs are distinct entities. For example the project may be written in the
same programming language as one analysis step. Generally, the project source
is defined as the most high-level source file that is unique to that
individual project (its language is irrelevant). The project is thus only in
charge of managing the inputs and outputs of each analysis step (take the
outputs of one step, and feed them as inputs to the next), not to do analysis
by itself. A good project will follow the modularity principle: analysis
scripts should be well-defined as an independently managed software source.
For example modules in Python, packages in R, or libraries/programs in C/C++
that can be imported in higher-level project sources.

###  2.4 Definition: data provenance

Data provenance is a very generic term which points to slightly different
technical concepts in different fields like databases, storage systems and
scientific workflows. For example within a database, an SQL query from a
relational database connects a subset of the database entries to the output
(_why-_ provenance), their more detailed dependency (_how-_ provenance) and
the precise location of the input sources (_where-_ provenance), for more see
cheney09. In scientific workflows, provenance goes beyond a single database
and its datasets, but may includes many databases that aren’t directly linked,
the higher-level project specific analysis that is done on the data, and
linking of the analysis to the text of the paper, for example see bavoil05,
moreau08, malik13.

Here, we define provenance to be the common factor of the usages above: a
dataset’s provenance is the set of metadata (in any ontology, standard or
structure) that connect it to the components (other datasets or scripts) that
produced it. Data provenance thus provides a high-level view of the data’s
genealogy.

###  2.5 Definition: data lineage

Data lineage is commonly used interchangeably with Data provenance [for
example][among many others, just search “data lineage” in
scholar.google.com]cheney09. However, for clarity, in this paper we refer to
the term “Data lineage” as a low-level and fine-grained recording of the
data’s source, and operations that occur on it, down to the exact command that
produced each intermediate step. This _recording_ does not necessarily have to
be in a formal metadata model. But data lineage must be complete (see
completeness principle in Section 3.1), and allow extraction of data
provenance metadata, and thus higher-level operations like visualization of
the workflow.

###  2.6 Definition: reproducibility and replicability

These terms have been used in the literature with various meanings, sometimes
in a contradictory way. It is therefore necessary to clarify the precise usage
of this term in this paper. But before that, it is important to highlight that
in this paper we are only considering computational analysis. In other words,
analysis after data has been collected and stored as a file on a filesystem.
Therefore, many of the definitions reviewed in plesser18, that are about data
collection, are out of context here. We adopt the same definition of
leek17,fineberg19, among others:

  * •

Reproducibility: (same inputs $\rightarrow$ consistent result). Formally:
“obtaining consistent [not necessarily identical] results using the same input
data; computational steps, methods, and code; and conditions of analysis”
fineberg19. This is thus synonymous with “computational reproducibility”.

fineberg19 allow non-bitwise or non-identical numeric outputs within their
definition of reproducibility, but they also acknowledge that this flexibility
can lead to complexities: what is an acceptable non-identical reproduction?
Exactly reproducible outputs can be precisely and automatically verified
without statistical interpretations, even in a very complex analysis
(involving many CPU cores, and random operations), see Section 3.5. It also
requires no expertise, as claerbout1992 put it: “a clerk can do it”. Raul: I
don’t know if this is true… at least it needs a bit of training and an extra
time. Maybe remove last phrase? In this paper, unless otherwise mentioned, we
only consider bitwise/exact reproducibility.

  * •

Replicability: (different inputs $\rightarrow$ consistent result). Formally:
“obtaining consistent results across studies aimed at answering the same
scientific question, each of which has obtained its own data” fineberg19.

Generally, since replicability involves new data collection, it can be
expensive. For example the “Reproducibility Project: Cancer Biology”
initiative started in 2013 to replicate 50 high-impact papers in cancer
biology111https://elifesciences.org/collections/9b1e83d1/reproducibility-
project-cancer-biology. Even with a funding of at least $1.3 million, it later
shrunk to 18 projects kaiser18 due to very high costs. We also note that
replicability doesn’t have to be limited to different input data: using the
same data, but with different implementations of methods, is also a
replication attempt [also known as “in silico” experiments, see][]stevens03.

Raul: put white line to separate next paragraph from the previous list? Some
authors have defined these terms in the opposite manner. Examples include
hinsen15 and the policy guidelines of the Association of Computing
Machinery222https://www.acm.org/publications/policies/artifact-review-badging
(ACM, dated April 2018). ACM has itself adopted the 2008 definitions of
Vocabulaire international de métrologie (VIM).

Besides the two terms above, “repeatability” is also sometimes used in regards
to the concept discussed here and must be clarified. For example,
ioannidis2009 use “repeatability” to encompass both the terms above. However,
the ACM/VIM definition for repeatability is “a researcher can reliably repeat
her own computation”. Hence, in the ACM terminology, the only difference
between replicability and repeatability is the “team” that is conducting the
computation. In the context of this paper, inputs are precisely defined
(Section 2.1): files with specific/registered checksums (see Section 3.5).
Therefore our inputs are team-agnostic, allowing us to safely ignore
“repeatability” as defined by ACM/VIM.

##  3 Principles of the proposed solution

The core principle behind this solution is simple: science is defined by its
method, not its result. Statements that convey a “result” abound in all
aspects of human life (e.g., in fiction, religion and science). What
distinguishes one from the other is the “method” that the result was derived.
Science is the only class that attempts to be as objective as possible through
the “scientific method”. buckheit1995 nicely summarize this by pointing out
that modern scientific papers (narrative combined with plots, tables and
figures) are merely advertisements of a scholarship, the actual scholarship is
the scripts and software usage that went into doing the analysis.

This paper thus proposes a framework that is optimally designed for both
designing and executing a project, _as well as_ publication of the
(computational) methods along with the published paper/result. However, this
paper is not the first attempted solution to this fundamental problem. Various
solutions have been proposed since the early 1990s, see Appendix B for a
review. To better highlight the differences with those methods, and the
foundations of this method (which help in understanding certain implementation
choices), in the sub-sections below, the core principle above is expanded by
breaking it into logically independent sub-components.

It is important to note that based on the definition of a project (Section
2.3) and the first principle below (modularity, Section 3.2) this paper is
designed to be modular and thus agnostic to high-level choices. For example
the choice of hardware (e.g., high performance computing facility or a
personal computer), or high-level interfaces (for example a webpage or
specialized graphic user interface). The proposed solution in this paper is a
low-level skeleton that is designed to be easily adapted to any high-level,
project-specific, choice. For example, in terms of hardware choice, a large
simulation project simply cannot be run on smaller machines. However, when
such a project is managed in the proposed system, the complete project (see
Section 3.1) is published and readable by peers, who can be sure that what
they are reading, contains the full/exact environment and commands that
produced the result. In terms of interfaces, wrappers can be written over this
core skeleton for various other high-level cosmetics, for example a web
interface, a graphic user interface or plugins to text editors or notebooks
(see Appendix A.6).

###  3.1 Principle: Complete/Self-contained

A project should be self-contained, needing no particular features from the
host operating system (OS), and not affecting the host OS. At build-time (when
the project is building its necessary tools), the project shouldn’t need
anything beyond a minimal POSIX environment on the host which is available in
Unix-like operating system like GNU/Linux, BSD-based or macOS. At run-time
(when environment/software are built), it should not use or affect any host
operating system programs or libraries.

Generally, a project’s source should include the whole project: access to the
inputs (see Section 2), building necessary software (access to tarballs and
instructions on configuring, building and installing those software), doing
the analysis (run the software on the data) and creating the final narrative
report/paper in its final format. This principle has several important
consequences:

  * •

A complete project doesn’t need any privileged/root permissions for system-
wide installation, or environment preparations. Even when the user does have
root privileges, interfering with the host operating system for a project, may
lead to many conflicts with the host or other projects. This principle thus
allows a safe execution of the project, and will not cause any security
problems.

  * •

A complete project doesn’t need an internet connection to build itself or to
do its analysis and possibly make a report. Of course this only holds when the
analysis doesn’t inherently require internet, for example needing a live data
feed.

  * •

A complete project inherently includes the complete data lineage and
provenance: automatically enabling a full backtrace of the output datasets or
narrative, to raw inputs: data or software source code lines. This is very
important because many existing data provenance solutions require manual
tagging within the data workflow or connecting the data with the paper’s text
(Appendix B). Manual tagging can be highly subjective, prone to many errors,
and incomplete.

  * •

A complete project will not need any user interaction and can complete itself
automatically. This is because manual interaction is an incompleteness.
Interactivity is also an inherently irreproducible operation, exposing the
analysis to human error, and requiring expert knowledge.

The first two components are particularly important for high performance
computing (HPC) facilities: because of security reasons, HPC users commonly
don’t have privileged permissions or internet access.

A complete project as defined here is much less exposed to “workflow decay” as
defined by zhao12 (in particular under their missing execution environment
tests). As recommended by zhao12, a complete project automatically builds all
its necessary third-party tools, it doesn’t just assume their existence.
Ultimately, the executability of a project will decay once the host Linux
kernel inevitably evolves to such that the project’s fixed version of the GNU
C Library and GNU C Compiler can’t be built. This will happen on much longer
time scales than the high-level software mentioned in zhao12 and can be fixed
by changing the project’s (GNU) C library and (GNU) C Compiler to versions
that are build-able with the host kernel. These are very low-level components
and any possible change in the output should be minimal. Ultimately after
multiple decades, even that may not be possible, but even at that point,
thanks to the plain-text principle (Section 3.3), it can still be studied,
without necessarily executing it.

###  3.2 Principle: Modularity

A project should be compartmentalized or partitioned to independent modules or
components with well-defined inputs/outputs having no side-effects. In a
modular project, communication between the independent modules is explicit,
providing optimizations on multiple levels: 1) Execution: independent modules
can run in parallel, or modules that don’t need to be run (because their
dependencies haven’t changed) won’t be re-done. 2) Data lineage and data
provenance extraction (recording any dataset’s origins). 3) Citation: allowing
others to credit specific parts of a project. This principle doesn’t just
apply to the analysis, it also applies to the whole project, for example see
the definitions of “input”, “output” and “project” in Section 2.

Within the analysis phase, this principle can be summarized best with the Unix
philosophy, best described by mcilroy78 in the “Style” section. In particular
“Make each program do one thing well. To do a new job, build afresh rather
than complicate old programs by adding new ‘features”’. Independent parts of
the analysis can be maintained as independent software (for example shell,
Python, or R scripts, or programs written in C, C++ or FORTRAN, among others).
This core aspect of the Unix philosophy has been the cause of its continued
success (particularly through GNU and BSD) and development in the last half
century.

For the most high-level analysis/operations, the boundary between the
“analysis” and “project” can become blurry. It is thus inevitable that some
highly project-specific, and small, analysis steps are also kept within the
project and not maintained as a separate software package (that is built
before the project is run). This isn’t a problem, because inputs are defined
as files that are _usable_ by other projects (see Section 2.1). If necessary,
such highly project-specific software can later spin-off into a separate
software package later. One example of an existing system that doesn’t follow
this principle is Madagascar, it builds a large number of analysis programs as
part of the project (see Appendix B.3).

###  3.3 Principle: Plain text

A project’s primarily stored/archived format should be plain text with human-
readable encoding333Plain text format doesn’t include document container
formats like .odf or .doc, for software like LibreOffice or Microsoft Office.,
for example ASCII or Unicode (for the definition of a project, see Section
2.3). The reason behind this principle is that opening, reading, or editing
non-plain text (executable or binary) file formats needs specialized software.
Binary formats will complicate various aspects of the project: its usage,
archival, automatic parsing, or human readability. This is a critical
principle for long term preservation and portability: when the software to
read binary format has been depreciated or become obsolete and isn’t
installable on the running system, the project will not be readable/usable any
more.

A project that is solely in plain text format can be put under version control
as it evolves, with easy tracking of changed parts, using already available
and mature tools in software development: software source code is also in
plain text. After publication, independent modules of a plain-text project can
be used and cited through services like Software Heritage dicosmo18,dicosmo20,
enabling future projects to easily build on top of old ones, or cite specific
parts of a project.

Archiving a binary version of the project is like archiving a well cooked dish
itself, which will be inedible with changes in hardware (temperature,
humidity, and the natural world in general). But archiving the dish’s recipe
(which is also in plain text!): you can re-cook it any time. When the
environment is under perfect control (as in the proposed system), the
binary/executable, or re-cooked, output will be verifiably identical. One
illustrative example of the importance of source code is mentioned in smart18:
a seven-year old dispute between condensed matter scientists could only be
solved when they shared the plain text source of their respective projects.

This principle doesn’t conflict with having an executable or immediately-
runnable project444In their recommendation 4-1 on reproducibility, fineberg19
mention: “a detailed description of the study methods (ideally in executable
form)”.. Because it is trivial to build a text-based project within an
executable container or virtual machine. For more on containers, please see
Appendix A.1. To help contemporary researchers, this built/executable form of
the project can be published as an output in respective servers like
http://hub.docker.com (see Section 2.2).

Note that this principle applies to the whole project, not just the initial
phase. Therefore a project like Conda that currently includes a $+500$MB
binary blob in a plain-text shell script (see Appendix A.2.2) is not
acceptable for this principle. This failure also applies to projects that
build tools to read binary sources. In short, the full source of a project
should be in plain text.

###  3.4 Principle: Minimal complexity (i.e., maximal compatibility)

An important measure of the quality of a project is how much it avoids
complexity. In principle this is similar to Occam’s razor: “Never posit
pluralities without necessity” schaffer15, but extrapolated to project
management. In this context Occam’s razor can be interpreted like the
following cases: minimize the number of a project’s dependency software (there
are often multiple ways of doing something), avoid complex relations between
analysis steps (which is not unrelated to the principle of modularity in
Section 3.2), or avoid the programming language that is currently in vogue
because it is going to fall out of fashion soon and take the project down with
it, see Appendix A.7). This principle has several important consequences:

  * •

Easier learning curve. Scientists can’t adopt new tools and methods as fast as
software developers. They have to invest the majority of their time on their
own research domain. Because of this researchers usually continue their career
with the language/tools they learned when they started.

  * •

Future usage. Scientific projects require longevity: unlike software
engineering, there is no end-of-life in science (e.g., Aristotle’s work 2.5
millennia ago is still “science”). Scientific projects that depend too much on
an ever evolving, high-level software developing toolchain, will be harder to
archive, run, or even study for their immediate and future peers. One recent
example is the Popper software implementation: it was originally designed in
the HashiCorp configuration language (HCL) because it was the default for
organizing operations in GitHub. However, GitHub dropped HCL in October 2019,
for more see Appendix B.20.

  * •

Compatible and extensible. A project that has minimal complexity, can easily
adapt to any kind of data, programming language, host hardware or software and
etc. It can also be easily extended for new inputs and environments. For
example when a project management system is designed only to manage Python
functions (like CGAT-core, see Appendix A.5), it will be hard, inefficient and
buggy for managing an analysis step that is written in R and another written
in FORTRAN.

###  3.5 Principle: Verifiable inputs and outputs

The project should contain automatic verification checks on its inputs
(software source code and data) and outputs. When applied, expert knowledge
won’t be necessary to confirm the correct reproduction. It is just important
to emphasize that in practice, exact or bit-wise reproduction is very hard to
implement at the level of a file. This is because many specialized scientific
software commonly print the running date on their output files (which is very
useful in its own context).

For example in plain text tables, such meta-data are commonly printed as
commented lines (usually starting with #). Therefore when verifying such a
plain text table, the checksum which is used to validate the data, can be
recorded after removing all commented lines. Fortunately, the tools to operate
on specialized data formats also usually have ways to remove requested
metadata (like creation date), or ignore metadata altogether. For example the
FITS standard in astronomy pence10 defines a special DATASUM keyword which is
a checksum calculated only from the raw data, ignoring all metadata.

###  3.6 Principle: History and temporal provenance (version control)

No project is done in a single/first attempt. Projects evolve as they are
being completed. It is natural that earlier phases of a project are
redesigned/optimized only after later phases have been completed. This is
often seen in scientific papers, with statements like “we [first] tried method
[or parameter] XXXX, but YYYY is used here because it showed to have better
precision [or less bias, or etc]”. A project’s “history” is thus as
scientifically relevant as the final, or published, snapshot of the project.
All the outputs (datasets or narrative papers) need to contain the exact point
in the project’s history that produced them.

For a complete project (see Section 3.1) that is under version control (like
Git), this would be the unique commit checksum (for more on version control,
see Appendix A.4). This principle thus benefits from the plain-text principle
(Section 3.3). Note that with our definition of a project (Section 2.3),
“changes” in the project include changes in the software building or versions,
changes in the running environment, changes in the analysis, or changes in the
narrative. After publication, the project’s history can also be published on
services like Software Heritage dicosmo18, enabling precise citation and
archival of all stages of the project’s evolution.

Taking this principle to a higher level, newer projects are built upon the
shoulders of previous projects. A project management system should be able to
provide this temporal connection between projects. Quantifying how newer
projects relate to older projects (for example through Git branches) will
enable 1) scientists to simply use the relevant parts of an older project, 2)
quantify the connections of various projects, which is primarily of interest
for meta-research (research on research) or historical studies. In data
science, “provenance” is used to track the analysis and original datasets that
were used in producing a higher-level dataset. A system that uses this
principle will also provide “temporal provenance”, quantifying how a certain
project grew/evolved in the time dimension.

###  3.7 Principle: Free and open source software

Technically, as defined in Section 2.6, reproducibility is also possible with
a non-free and non-open-source software (a black box). This principle is thus
necessary to complement the definition of reproducibility. This is because
software freedom as an important pillar for the sciences as shown below:

  * •

Based on the completeness principle (Section 3.1), it is possible to trace the
output’s provenance back to the exact source code lines within an analysis
software. If the software’s source code isn’t available such important and
useful provenance information is lost.

  * •

A non-free software may not be runnable on a given hardware. Since free
software is modifiable, others can modify (or hire someone to modify) it and
make it runnable on their particular platform.

  * •

A non-free software cannot be distributed by the authors, making the whole
community reliant only on the proprietary owner’s server (even if the
proprietary software doesn’t ask for payments). A project that uses free
software can also release the necessary tarballs of the software it uses. For
example see https://doi.org/10.5281/zenodo.3408481zenodo.3408481 akhlaghi19 or
https://doi.org/10.5281/zenodo.3524937zenodo.3524937 infante20.

  * •

A core component of reproducibility is that anonymous peers should be able
confirm the result from the same datasets with minimal effort, and this
includes financial cost beyond hardware.

##  4 Implementation of Maneage

The proposed solution is an implementation of the principles discussed in
Section 3: it is complete and automatic (Section 3.1), modular (Section 3.2),
fully in plain text (Section 3.3), having minimal complexity (see Section
3.4), with automatically verifiable inputs & outputs (Section 3.5), preserving
temporal provenance, or project evolution (Section 3.6) and finally, it is
free software (Section 3.7).

In practice it is a collection of plain-text files, that are distributed in
pre-defined sub-directories by context, and are all under version-control
(currently with Git). In its raw form (before customizing for different
projects), it is a fully working skeleton of a project without much flesh:
containing all the low-level infrastructure, with just a small demonstrative
“delete-me” analysis. To start a new project, users will _clone_ 555In Git,
“clone”ing is the process of copying all the project’s file and their history
into the host system. the core skeleton, create their own Git branch, and
start customizing the core files (adding their high-level analysis steps,
scripts to generate figure and narrative) within their custom branch.

In this section we will review the current implementation of the reproducible
paper template. Generally, job orchestration is implemented in Make (a POSIX
software), this choice is elaborated in Section 4.1. We continue with a
general outline of the project’s file structure in Section 4.2. As described
there, we make a cosmetic distinction between “configuration” (or building of
necessary software) and execution (or running the software on the data), these
two phases are discussed in Sections 4.3 & LABEL:sec:projectmake.

###  4.1 Job orchestration with Make

When non-interactive, or batch, processing is needed (see Section 3.1), shell
scripts are usually the first solution that come to mind (see Appendix A.5.1).
However, the inherent complexity and non-linearity of progress in a scientific
project (where experimentation is key) makes it hard and inefficient to manage
the script(s) as the project evolves. For example, a script will start from
the top/start every time it is run. Therefore, even if $90\%$ of a research
project is done and only the newly added, final $10\%$ must be executed, a
script will always start from the start.

It is possible to manually ignore (by conditionals), or comment, parts of a
script to only do a special part. However, such conditionals/comments will
only add to the complexity and will discourage experimentation on an already
completed part of the project. This is also prone to very serious bugs in the
end (e.g., due to human error, some parts may be left-out or not up to date),
when re-running from scratch. Such bugs are very hard to notice during the
work and frustrating to find in the end. These problems motivated the creation
of Make in the early Unix operating system feldman79.

In the Make paradigm, process execution starts from the end: the final
_target_. Through the Make syntax, the user specifies the _prerequisite_(s) of
each target and a _recipe_ (a small shell script) to create the target from
the prerequisites (for more see Appendix A.5.2). With this lineage, Make is
thus able to build a dependency tree internally and find the rule(s) that need
to be executed on each run. This has many advantages:

  * •

Only executing necessary steps: in the scenario above, a researcher that has
just added the final $10\%$ of her research, will only have to run those extra
steps, without any modification to the previous parts. With Make, it is also
trivial to change the processing of any intermediate (already written) _rule_
(or step) in the middle of an already written analysis: the next time Make is
run, only rules that are affected by the changes/additions will be re-run, not
the whole analysis/project.

Most importantly, this enables full reproducibility from scratch with no
changes in the project code that was working during the research. This will
allow robust results and let scientists do what they do best: experiment, and
be critical to the methods/analysis without having to waste energy on the
added complexity of experimentation in scripts.

  * •

Parallel processing: Since the dependencies are clearly demarcated in Make, it
can identify independent steps and run them in parallel. This greatly speeds
up the processing, with no cost in terms of complexity.

  * •

Codifying data lineage and provenance: In many systems data provenance has to
be manually added. However, in Make, it is part of the design and no extra
manual step is necessary to fully track (or back-track) the series of steps
that generated the data.

Make has been a fixed component of POSIX (or Unix-like operating systems
including Unix, GNU/Linux, BSD, and macOS, among others) from very early days
of Unix almost 40 years ago. It is therefore, by far, the most reliable,
commonly used, well-known and well-maintained workflow manager today. Because
the core operating system components are built with it, Make is expected to
keep this unique position into the foreseeable future. Make is also well known
by many outside of the software developing communities. For example schwab2000
report how geophysics students have easily adopted it for the RED project
management tool used in their lab at that time (see Appendix B.1 for more on
RED). Because of its simplicity, we have also had very good feedback on using
Make from the early adopters of this system during the last year, in
particular graduate students and postdocs.

In summary Make satisfies all our principles (see Section 3), while avoiding
the well-known problems of using high-level languages for project management
like a generational gap and “dependency hell”, see Appendix A.7. For more on
Make and a discussion on some other job orchestration tools, see Appendices
A.5.2 and A.5 respectively.

###  4.2 General implementation structure

As described above, a project using this template is a combination of plain-
text files that are organized in various directories by context. Figure 2
shows this directory structure and some representative files in each
directory. The top-level source only has two main directories: tex/
(containing LaTeX files) and reproduce/ (containing all other parts of the
project) as well as several high-level files. Most of the top project
directory files are only intended for human readers (as narrative text, not
scripts or programming sources): COPYING is the project’s high-level copyright
license, README.md is a basic introduction to the specific project, and
README-hacking.md describes how to customize, or hack, the template for
creators of new projects.

In the top project directory, there are two non-narrative files: project
(which should have been under reproduce/) and paper.tex (which should have
been under tex/). The former is necessary in the top project directory because
it is the high-level user interface, with the ./project command. The latter is
necessary for many web-based automatic paper generating systems like arXiv,
journals, or systems like Overleaf.

figure-file-architecture

Figure 2:  Directory and file structure in a hypothetical project using this
solution. Files are shown with small, green boxes that have a suffix in their
names (for example format.mk or download.tex). Directories (containing
multiple files) are shown as large, brown boxes, where the name ends in a
slash (/). Directories with dashed lines and no files (just a description) are
symbolic links that are created after building the project, pointing to
commonly needed built directories. Symbolic links and their contents are not
considered part of the source and are not under version control. Files and
directories are shown within their parent directory. For example the full
address of format.mk from the top project directory is
reproduce/analysis/make/format.mk.

project is a simple executable POSIX-compliant shell script, that is just a
high-level wrapper script to call the project’s Makefiles. Recall that the
main job orchestrator in this system is Make, see Section 4.1 for why Make was
chosen. In the current implementation, the project’s execution consists of the
following two calls to the project script:

[language=bash] ./project configure # Build software from source (takes around
2 hours for full build). ./project make # Do the analysis (download data, run
software on data, build PDF).

The operations of both are managed by files under the top-level reproduce/
directory. When the first command is called, the contents of
reproduce/software are used, and the latter calls files under
reproduce/analysis. This highlights the _cosmetic_ distinction we have adopted
between the two main steps of a project: 1) building the project’s full
software environment and 2) doing the analysis (running the software).
Technically there is no difference between the two and they could easily be
merged under one directory. However, during a research project, researchers
commonly just need to focus on their analysis steps and will rarely need to
edit the software environment settings (maybe only once at the start of the
project). Therefore, having the files mixed under the same directory can be
confusing.

In summary, the same structure governs both aspects of a project: software
building and analysis. This is an important and unique feature in this
template. A researcher that has become familiar with Makefiles for
orchestrating their analysis, will also easily be able to modify the Makefiles
for the software that is built in their project, and feel free to customize
their project’s software also. Most other systems use third-party package
managers for their project’s software, thus discouraging project-specific
customization of software, for a full review of third party package managers,
see Appendix A.2.

###  4.3 Project configuration

A critical component of any project is the set of software used to do the
analysis. However, verifying an already built software environment (which is
critical to reproducing the research result) is a very hard. This has forced
most projects to move around the whole _built_ software environment (a black
box) as virtual machines or containers, see Appendix A.1. Because these black
boxes are almost impossible to reproduce themselves, they need to be archived,
even though they can take gigabytes of space. Package managers like Nix or GNU
Guix do provide a verifiable, i.e., reproducible, software building
environment, but because they aim to be generic package managers, they have
their own limitations on a project-specific level, see Appendix A.2.3.

Based on the principles of completeness and minimal complexity (Sections 3.1 &
3.4), a project that uses this solution, also contains the full instructions
to build its necessary software in the same language that the analysis is
orchestrated: Make. Project configuration (building software environment) is
managed by the files under reproduce/software. Project configuration involves
three high-level steps which are discussed in the subsections below: setting
the local directories (Section 4.3.1), checking a working C compiler (Section
4.3.2), and the software source code download, build and install (Section
4.3.3).

####  4.3.1 Setting local directories

All files built by the project (software or analysis) will be under a “build
directory” (orBDIR) on the host filesystem. No other location on the running
operating system will be affected by the project. Following the modularity
principle (Section 3.2), this directory should be separate from the source
directory. Therefore, at configuration time, the first thing to specify is the
build directory on the running system. The build directory can be specified in
two ways: 1) on the command-line with the –build-dir option, or 2) manually
giving the directory after running the configuration: it will stop with a
prompt and some explanation.

Two other local directories can optionally be specified by the project when
inputs are present locally (for the definition of inputs, see Section 2.1) and
don’t need to be downloaded: 1) software tarball directory and 2) input data
directory. The project just needs reading permissions on these directories:
when given, nothing will be written inside of them. The project will only look
into them for the necessary software tarballs and input data. If they are not
found, the project will attempt to download any necessary file from the
recoded URLs/PIDs within the project source. These directories are therefore
primarily tailored to scenarios where the project must run offline (based on
the completeness principle of Section 3.1).

After project configuration, a symbolic link is built the top project source
directory that points to the build directory. The symbolic link is a hidden
file named .build, see Figure 2. With this symbolic link, its always very easy
to access to built files, no matter where the build directory is actually
located on the filesystem.

####  4.3.2 Checking for a C compiler

This template builds all its necessary software internally to avoid dependency
issues with various software versions on different hosts. A working C compiler
is thus mandatory and the configure script will abort if a working C compiler
isn’t found. In particular, on GNU/Linux systems, the project builds its own
version of the GNU Compiler Collection (GCC), therefore a static C library is
necessary with the compiler. If not found, an informative error message will
be printed and the project will abort.

The custom version of GCC is configured to also build FORTRAN, C++,
objective-C and objective-C++ compilers. Python and R running environments are
themselves written in C, therefore they are also automatically built
afterwards if the project uses these languages. On macOS systems, we currently
don’t build a C compiler, but it is planned to do so in the future.

####  4.3.3 Verifying and building necessary software from source

All necessary software for the project, and their dependencies, are installed
from source. Researchers using the template only have to specify the most
high-level analysis software they need in
reproduce/software/config/installation/TARGETS.conf (see Figure 2). Based on
the completeness principle (Section 3.1), on GNU/Linux systems the dependency
tree is automatically traced down to the GNU C Library and GNU Compiler
Collection (GCC). Thus creating identical high-level analysis software on any
system. When the C library and compiler can’t be installed (for example on
macOS systems), the users are forced to rely on the host’s C compiler and
library, and this may hamper the exact reproducibility of the final result:
the project will abort if the final outputs have changed. Because the
project’s main output is currently a LaTeX-built PDF, the project also
contains an internal installation of TeXLive, providing all the necessary
tools to build the PDF, independent of the host operating system’s LaTeX
version and packages.

To build the software from source, the project needs access to its source
tarball or zip-file. If the tarballs are already present on the system, the
user can specify the respective directory at the start of project
configuration (Section 4.3.1). If not, the software tarballs will be
downloaded from pre-defined servers. Ultimately the origin of the tarballs is
irrelevant for this project, what matters is the tarball contents: checked
through the SHA-512 checksum [part of the SHA-2 algorithms, see][]romine15. If
the SHA-512 checksum of the tarball is different from the checksum stored for
it in the project’s source, the project will complain and abort. Because the
server is irrelevant, one planned taskadd task number is to allow users to
identify the most convenient server themselves, for example to improve
download speed.

Software tarball access, unpacking, building and installation is managed
through Makefiles, see Sections 4.1 & 4.2. The project’s software are
classified into two classes: 1) basic and 2) high-level. The former contains
meta-software: software needed to build other software, for example GNU Gzip,
GNU Tar, GNU Make, GNU Bash, GNU Coreutils, GNU SED, GNU Binutils, GNU
Compiler Collection (GCC) and etc666Note that almost all these GNU software
are also installable on non-GNU/Linux operating systems like BSD or macOS
also, exceptions include GNU Binutils.. The basic software are built with the
host operating system’s tools and are installed with any project. The high-
level software are those that are used directly in the science analysis and
can differ from project to project. However, because the basic software have
already been built by the project, the higher-level software are built with
them and independent of the host operating system’s tools.

Software building is managed by two top-level Makefiles that follow the same
classification. Both are under the reproduce/software/make/ directory (Figure
2): basic.mk and high-level.mk. Because basic.mk can’t assume anything about
the host, it is written to comply with POSIX Make and POSIX shell, which are
very limited compared to GNU Make and GNU Bash respectively. However, after it
is finished, a specific version of GNU Make (among other basic software), is
present, enabling us to assume the much advanced features of GNU tools in
high-level.mk.

The project’s software are installed under BDIR/software/installed. The .local
symbolic link in the top project source directory points to it for easy access
(see Figure 2). It contains the top-level POSIX filesystem hierarchy
subdirectories for the project including bin/, lib/, include/ among others.
For example the custom-built GNU Make executable is placed under
BDIR/software/installed/bin/make or alternatively .local/bin/make.

To orchestrate software building with Make, the building of each software
should be represented as a file. In the Makefiles that file should be used as
a _target_ , in the rule that builds the software, or _prerequisite_ , in the
rule(s) of software that depend on it. For more on Make, see Appendix A.5.2.
Initially we tried using the actual software’s built files (executable
programs, libraries or etc). However, given the variety of installed files,
using them as the software’s representative caused many complexities,
confusions and bugs. Therefore, in the current system, once a software is
built, a simple plain-text file is created under a sub-directory of
.local/version-info. The representative files for C/C++ programs or libraries
are placed under the proglib sub-directory. The Python or TeXLive
representative files are placed under the python and tex subdirectories
respectively. Make uses this file to refer to the software and arrange the
order of software execution. The contents of this plain-text file are the name
and possible citation to the software that are directly imported into the
final paper in the end. For more on software citation, see Section 4.3.4.

####  4.3.4 Software citation

Based on the completeness principle (Section 3.1), the project contains the
full list of installed software, their versions and their configuration
options. However, this information is buried deep into the project’s source. A
distilled fraction of this information must also be printed in the project’s
final report, blended into the narrative. Furthermore, when a published paper
is associated with the used software, it is important to cite that paper, the
citations help software authors gain more recognition and grants, encouraging
them to further develop it. This is particularly important in the case for
research software, where the researcher has invested significant time in
building the software, and requires official citation to justify continued
work on it.

One notable example that nicely highlights this issue is GNU Parallel tange18:
every time it is run, it prints the citation information before it starts.
This doesn’t cause any problem in automatic scripts, but can be annoying when
reading/debugging the outputs. Users can disable the notice, with the
–citation option and accept to cite its paper, or support its development
directly by paying $10000$ euros! This is justified by an uncomfortably true
statement777GNU Parallel’s FAQ on the need to cite software:
http://git.savannah.gnu.org/cgit/parallel.git/plain/doc/citation-notice-
faq.txt: “history has shown that researchers forget to [cite software] if they
are not reminded explicitly. … If you feel the benefit from using GNU Parallel
is too small to warrant a citation, then prove that by simply using another
tool”. In bug 905674888Debian bug on the citation notice of GNU Parallel:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=905674, the Debian
developers argued that because of this extra condition, GNU Parallel should
not be considered as free software, and they are using a patch to remove that
part of the code for its build under Debian-based operating systems. Most
other research software don’t resort to such drastic measures, however,
citation is important for them. Given the increasing number of software used
in scientific research, the only reliable solution is to automatically cite
the used software in the final paper.

As mentioned above in Section 4.3.3, a plain-text file is built automatically
at the end of a software’s successful build and installation. This file
contains the name, version and possible citation of that software. At the end
of the configuration phase, all these plain-text files are merged into one
LaTeX macro that can be imported directly into the final paper or report. In
this paper, this macro’s value is shown in Appendix D. The paragraph produced
by this macro won’t be too large, but it will greatly help in the recording of
the used software environment and will automatically cite the software where
necessary.

In the current version of this template it is assumed the published report of
a project is built by LaTeX. Therefore, every software that has an associated
paper, has a BibTeX file under the reproduce/software/bibtex directory. When
the software is built for the project (possibly as a dependency of another
software specified by the user), the BibTeX entry(s) are copied to the build
directory and the command to cite that BibTeX record is included in the LaTeX
macro with the name and version of the software, as shown in Appendix D.

For a review of the necessity and basic elements in software citation, see
katz14 and smith16. There are ongoing projects specifically tailored to
software citation, including CodeMeta (https://codemeta.github.io) and
Citation file format (CFF: https://citation-file-format.github.io). Both are
based on scheme.org, but are respectively implemented in the JSON-LD and YAML.
Another robust approach is provided by SoftwareHeritage dicosmo18. The feature
of the SoftwareHeritage is that a published paper isn’t necessary and it won’t
populate a research paper’s bibliography list. However, this also makes it
hard to count as academic credit. We are considering using these tools, and
export BibTeX entries when necessary.

###  4.4 High-level organization of analysis

Once a project is configured (Section 4.3), all the necessary software, with
precise versions and configurations, are built and ready to use. The analysis
phase of the project (running the software on the data) is also orchestrated
through Makefiles. For the unique advantages of using Make to manage a
research project, see Sections 4.1 & 4.2. In order to best follow the
principle of modularity (Section 3.2), the analysis is not done in one phase
or with a single Makefile. Here, the two high-level phases of the analysis are
reviewed. The organization of the lower-level analysis, in many modular
Makefiles, is discussed in Section 4.5.

After running ./project make, the analysis is done in two sequential phases:
1) preparation and 2) main analysis. The purpose of the preparation phase is
further elaborated in Section 4.4.2. Technically, these two phases are managed
by the two high-level Makefiles: top-prepare.mk and top-make.mk. Both are
under reproduce/analysis/make (see Figure 2) and both have an identical lower-
level analysis structure. But before that, in Section 4.4.1 the isolation of
the analysis environment from the host is discussed.

####  4.4.1 Isolated analysis environment

By default, the analysis part of the project is not exposed to any of the
host’s environment variables. This is accomplished through the ‘env -i’
command999Note that the project’s self-built env program is used, not the one
provided by the host operating system. Within the project, env is installed as
part of GNU Coreutils and -i is short for –ignore-environment., which will
remove the host environment. The project will define its own values for
standard environment variables to avoid using system or user defaults.
Combined with the fact that all the software were configured and compiled from
source for each project at configuration time (Section 4.3.3), this completely
isolates the analysis from the host operating system, creating an exactly
reproducible result on any machine that the project can be configured.

For example, the project builds is own fixed version of GNU Bash (a command-
line shell environment). It also has its own bashrc startup script101010The
project’s Bash startup script is under reproduce/software/bash/bashrc.sh, see
Figure 2., and the BASH_ENV environment variable is set to load this startup
script. Furthermore, the HOME environment variable is set to BDIR to avoid the
penetration of any existing Bash startup file of the user’s home directory
into the analysis.

####  4.4.2 Preparation phase

When ./project make is called, the first Makefile that is run is top-
prepare.mk. It is designed for any selection steps that may be necessary to
optimize top-make.mk, or to “prepare” for it. It is mainly useful when the
research targets are more focused than the raw input and may not be necessary
in many scenarios. Its role is described here with an example.

Let’s assume the raw input data (that the project received from a database)
has 5000 rows (potential targets for doing the analysis on). However, this
particular project only needs to work on 100 of them, not the full 5000. If
the full 5000 targets are given to top-make.mk, Make will need to create a
data lineage for all 5000 targets and project authors have to add checks in
many places to ignore those that aren’t necessary. This will add to the
project’s complexity and is prone to many bugs. Furthermore, if the filesystem
isn’t fast (for example a filesystem that exists over a network), checking all
the intermediate and final files over the full lineage can be slow.

In this scenario, the preparation phase finds the IDs of the 100 targets of
interest and saves them as a Make variable in a file under BDIR. Later, this
file is loaded into the analysis phase, precisely identifying the project’s
targets-of-interest. This selection phase can’t be done within top-make.mk
because the full data lineage (all input and output files) must be known to
Make before it starts to execute the necessary operations. It is possible to
for Make to call itself as another Makefile, but this practice is strongly
discouraged here because it makes the flow very hard to read. However, if the
project authors insist on calling Make within Make, it is certainly possible.

The “preparation” phase thus allows top-make.mk to optimally organize the
complex set of operations that must be run on each input and the dependencies
(possibly in parallel). It also greatly simplifies the coding for the project
authors. Ideally top-prepare.mk is only for the “preparation phase”. However,
projects can be complex and ultimately, the choice of which parts of an
analysis being a “preparation” can be highly subjective. Generally, the
internal design and concepts of top-prepare.mk are identical to top-make.mk.
Therefore in Section 4.5, where the lower-level management is discussed, we
will only focus on the latter to avoid confusion.

###  4.5 Low-level organization of analysis

A project consists of many steps, including data access (possibly by
downloading), running various steps of the analysis on the obtained data, and
creating the necessary plots, figures or tables for a published report, or
output datasets for a database. If all of these steps are organized in a
single Makefile, it will become very large, or long, and will be hard to
maintain, extend/grow, read, reuse, and cite. Generally, large files are a bad
practice because it is against the modularity principle (Section 3.2).

The proposed template is thus designed to encourage and facilitate modularity
by distributing the analysis in many Makefiles that contain contextually-
similar (or modular) analysis steps. In the rest of this paper these modular,
or lower-level, Makefiles will be called _subMakefiles_. The subMakefiles are
loaded into top-make.mk in a certain order and executed in one instance of
Make without recursion (see Section 4.5.1 below). In other words, this
modularity is just cosmetic for Make: Make “see”s all the subMakefiles as
parts of one file. However, this modularity plays a critical role for the
human reader/author of the project and is necessary in re-using or citing
parts of the analysis in other projects.

Within the project’s source, the subMakefiles are placed in
reproduce/analysis/make (with top-make.mk), see Figure 2. Therefore by design,
top-make.mk is very simple: it just defines the ultimate target (paper.pdf),
and the name and order of the subMakefiles that should be loaded into Make.

The precise organization of the analysis steps highly depends on each
individual project. However, many aspects of the project management are the
same, irrespective of the particular project, here we will focus on those.
Figure 3 is a general overview of the analysis phase in a hypothetical project
using this template. As described above and shown in Figure 3, top-make.mk
imports the various Makefiles under the reproduce/ directory that are in
charge of the different phases of the analysis. Each of the subMakefiles
builds intermediate targets, or outputs (files), which are shown there as blue
boxes. In the subsections below, the project’s analysis is described using
this graph. We’ll follow Make’s paradigm (see Section 4.1) of starting form
the ultimate target in Section 4.5.2, and tracing back its lineage all the way
up to the inputs and configuration files.

figure-data-lineage

Figure 3: Schematic representation of data lineage in a hypothetical
project/pipeline using Maneage. Each colored box is a file in the project and
the arrows show the dependencies between them. Green files/boxes are plain
text files that are under version control and in the source-directory. Blue
files/boxes are output files of various steps in the build-directory, located
within the Makefile (*.mk) that generates them. For example paper.pdf depends
on project.tex (in the build directory and generated automatically) and
paper.tex (in the source directory and written by hand). In turn, project.tex
depends on all the *.tex files at the bottom of the Makefiles above it. The
solid arrows and built boxes with full opacity are actually described in the
context of a demonstration project in this paper. The dashed arrows and lower
opacity built boxes, just shows how adding more elements to the lineage is
also easily possible, making this a scalable tool.

To avoid getting too abstract in the subsections below, where necessary, we’ll
do a basic analysis on the data of [data were published as supplementary
material on bioXriv]menke20 and try to replicate some of their results. Note
that because we are not using the same software, this isn’t a reproduction
(see Section 2.6). We can’t use the same software because they use Microsoft
Excel for the analysis which violates several of our principles: 1)
Completeness (as a graphic user interface program, it needs human interaction,
Section 3.1), 2) Minimal complexity (even free software alternatives like
LibreOffice involve many dependencies and are extremely hard to build, Section
3.4) and 3) Free software (Section 3.7).

####  4.5.1 Non-recursive Make

It is possible to call a new instance of Make within an existing Make
instance. This is also known as recursive
Make111111https://www.gnu.org/software/make/manual/html_node/Recursion.html.
Recursive Make is in fact used by many Make users, especially in the software
development communities. It is also possible within a project using the
proposed template.

However, recursive Make is discouraged in the template, and not used in it.
All the subMakefiles mentioned above are _included_
121212https://www.gnu.org/software/make/manual/html_node/Include.html into
top-make.mk, i.e., their contents are read into Make as it is parsing top-
make.mk. In Make’s view, this is identical to having one long file with all
the subMakefiles concatenated to each other. Make is only called once and
there is no recursion.

Furthermore, we have the convention that only top-make.mk (or top-prepare.mk)
can include subMakefiles. SubMakefiles should not include other subMakefiles.
The main reason behind this convention is the Minimal Complexity principle
(Section 3.4): a simple glance at top-make.mk, will immediately show _all_ the
project’s subMakefiles _and_ their loading order. When the names of the
subMakefiles are descriptive enough, this enables both the project authors,
and later, project readers to get a complete view of the various stages of the
project.

####  4.5.2 Ultimate target: the project’s paper or report (paper.pdf)

The ultimate purpose of a project is to report the data analysis result. In
scientific projects, this “report” is the published (or draft) paper. In the
industry, it is a quality-check and analysis of the final data product(s). In
both cases, the report contains many visualizations of the final data product
of the project, for example as a plot, figure, table, or numbers blended into
the narrative description.

In Figure 3 it is shown as paper.pdf, note that it is the only built file
(blue box) with no arrows leaving it. In other words, nothing depends on it:
highlighting its unique “ultimate target” position in the lineage. The
instructions to build paper.pdf are in paper.mk. The report’s source
(containing the main narrative, its typesetting as well as that of the figures
or tables) is paper.tex. To build the final report’s PDF, references.tex and
project.tex are also loaded into LaTeX. references.tex is part of the
project’s source and can contain the BibTeX entries for the bibliography of
the final report. In other words, it formalizes the connections of this
project with previous projects.

Another class of files that maybe loaded into LaTeX, but are not shown to
avoid complications in the figure, are the figure or plot data, or built
figures. For example in this paper, the demonstration figure shown in Section
4.5.7 is drawn directly within LaTeX (using its PGFPlots package). The project
only needed to build the plain-text table of numbers that were fed into
PGFPlots (tools-per-year.txt in Figure 3).

However, building some plots may not be possible with PGFPlots, or the authors
may prefer another tool to generate the visualization’s image file [for
example with Python’s Matplotlib, ][]matplotlib2007. For this scenario, the
actual image file of the visualization can be used in the lineage, for example
tools-per-year.pdf instead of tools-per-year.txt. See Section 4.8 on the
project publication for special considerations regarding these files.

####  4.5.3 Values within text (project.tex)

Figures, plots, tables and narrative aren’t the only analysis output that goes
into the paper. In many cases, quantitative values from the analysis are also
blended into the sentences of the report’s narration. For example this
sentence in the abstract of akhlaghi19: “… detect the outer wings of M51 down
to S/N of 0.25 …”. The reported signal-to-noise ratio (S/N) value “0.25”
depends on the analysis and is an output of the analysis just like paper’s
figures and plots. Manually typing the number in the LaTeX source is prone to
very important bugs: the author may forget to check it after a change in an
analysis (e.g., using a newer version of the software, or changing an analysis
parameter for another part of the paper). Given the evolution of a scientific
projects, this type of human error is very hard to avoid when such values are
manually written. Such values must also be automatically generated.

To automatically generate and blend them in the text, we use LaTeX macros. In
the quote above, the LaTeX source131313akhlaghi19 uses this template to be
reproducible, so its LaTeX source is available in multiple ways: 1) direct
download from arXiv:https://arxiv.org/abs/1909.112301909.11230, by clicking on
“other formats”, or 2) the Git or
https://doi.org/10.5281/zenodo.3408481zenodo.3408481 links is also available
on arXiv. looks like this: “detect the outer wings of M51 down to S/N of
$\demosfoptimizedsn$”. The LaTeX macro “\demosfoptimizedsn” is automatically
calculated and recorded during in the project and expands to the value “0.25”.
The automatically generated file project.tex stores all such inline output
macros. Furthermore, Figure 3 shows that it is a prerequisite of paper.pdf (as
well as the manually written LaTeX sources that are shown in green). Therefore
paper.pdf will not be built until this file is ready and up-to-date.

However, managing all the necessary LaTeX macros for a full project in one
file is against the modularity principle and can be frustrating and buggy. To
address this problem, all subMakefiles _must_ contain a fixed target with the
same base-name, but with a .tex suffix. For example in Figure 3, assume
out-1b.dat is a table and the mean of its third column must be reported in the
paper. Therefore in format.mk, a prerequisite of format.tex is out-1b.dat (as
shown by the arrow in Figure 3). The recipe of this rule will calculate the
mean of the column and put it in the LaTeX macro which is written in
format.tex. In a similar way, any other reported calculation from format.mk is
stored as a LaTeX macro in format.tex.

These LaTeX macro files thus form the core skeleton of the project: as shown
in Figure 3, the outward arrows of all built files of any subMakefile
ultimately leads to one of these LaTeX macro files. Note that _built_ files in
a subMakefile don’t have to be a prerequisite of its .tex file. They may point
to another Makefile’s LaTeX macro file. For example even though input1.dat is
a target in download.mk, it isn’t a prerequisite of download.tex, it is a
prerequisite of out-2a.dat (a target in demo-plot.mk). The lineage ultimate
ends in a LaTeX macro file in analysis3.tex.

####  4.5.4 Verification of outputs (verify.mk)

An important principle for this template is that outputs should be
automatically verified, see Section 3.5. However, simply confirming the
checksum of the final PDF, or figures and datasets is not generally possible:
as mentioned in Section 3.5, many tools that produce datasets or PDFs write
the creation date into the produced files. Therefore it is necessary to verify
the project’s outputs before the PDF is created. To facilitate output
verification, the project has a verify.mk Makefile, see Figure 3. It is the
only prerequisite of project.tex that was described in Section 4.5.2.
Verification is therefore the connection-point, or bottleneck, between the
analysis steps of the project and its final report.

Prior to publication, the project authors should add the MD5 checksums of all
theLaTeX macro files in the recipe of verify.tex. The necessary structure is
already there, so adding/changing the values is trivial. If any LaTeX macro is
different in future builds of the project, the project will abort with a
warning of the problematic file. When projects involve other outputs (for
example images, tables or datasets that will also be published), their
contents should also be validated. To do this, prerequisites should be added
to the verify.tex rule that automatically check the _contents_ of other
project outputs. Recall that many tools print the creation date automatically
when creating a file, so to verify a file, this kind of metadata must be
ignored. verify.tex contains some Make functions to facilitate checking with
some some file formats, others can be added easily.

####  4.5.5 Project initialization (initialize.mk)

The initialize.mk subMakefile is present in all projects and is the first
subMakefile that is loaded into top-make.mk (see Figure 3). Project authors
rarely need to modify/edit this file, it is part of the template’s low-level
infra-structure. Nevertheless, project authors are strongly encouraged to
study it and use all the useful variables and targets that it defines.
initialize.mk doesn’t contain any analysis or major processing steps, it just
initializes the system. For example it sets the necessary environment
variables, internal Make variables and defines generic rules like ./project
make clean (to clean/delete all built products, not software) or ./project
make dist (to package the project into a tarball for distribution) among
others.

It also adds one special LaTeX macro in initialize.tex: the current Git commit
that is generated every time the analysis is run. It is stored in
\projectversion macro and can be used anywhere within the final report. For
this PDF it has a value of . One good place to put it is in the end of the
abstract for any reader to be able to identify the exact point in history that
the report was created. It also uses the –dirty feature of Git’s –describe
output: if any version-controlled file is not already committed, the value to
this macro will have a -dirty suffix. If its in a prominent place (like the
abstract), it will always remind the author to commit their work.

####  4.5.6 Importing and validating inputs (download.mk)

The download.mk subMakefile is present in all Maneage projects and contains
the common steps for importing the input dataset(s) into the project. All
necessary input datasets to the project are imported through this subMakefile.
This helps in modularity and minimal complexity (Sections 3.2 & 3.4): to see
what external datasets were used in a project, this is the only necessary file
to manage/read. Also, a simple call to a downloader (for example wget) is not
usually enough. Irrespective of where the dataset is _used_ in the project’s
lineage, some operations are always necessary when importing datasets:

  * •

The file may already be present on the host, or the user may not have internet
connection. Hence it is necessary to check the given _input directory_ on the
host before attempting to download over the internet (see Section 4.3.1).

  * •

The network might temporarily fail, but connect with an automatic re-trial
(even today this isn’t uncommon). Crashing the whole analysis because of a
temporary network issue, requires human intervention and is against the
completeness principle (Section 3.1).

  * •

Before it can be used, the integrity of the imported file must be confirmed
with its stored checksum.

In some scenarios the generic download script may not be useful. For example
when the database takes queries and generates the dataset for downloading on
the fly. In such cases, users can add their own Make rules in this download.mk
to import the file. They can use its pre-defined structure to do the extra
steps like validating it. Note that in such cases the servers often encode the
creation date and version of their database system in the resulting file as
metadata. Even when the actual data is identical, this metadata (which is in
the same file) will differ based on the moment the query was done. Therefore a
simple checksum of the whole downloaded file can’t be used for validation in
such scenarios, see Section 3.5.

Figure 4:  Contents of the INPUTS.conf file for the demonstration dataset of
menke20. This file contains the basic, or minimal, metadata for retrieving the
required dataset(s) of a project: it can become arbitrarily long. Here,
M20DATA contains the name of this dataset within this project. MK20MD5
contains the MD5 checksum of the dataset, in order to check the validity and
integrity of the dataset before usage. MK20SIZE contains the size of the
dataset in human readable format. MK20URL is the URL which the dataset is
automatically downloaded from (only when its not already present on the host).
Note that the original URL (footnote 14) was too long to display properly
here.

Each external dataset has some basic information, including its expected name
on the local system (for offline access), the necessary checksum to validate
it (either the whole file or just its main “data”), and its URL/PID. In this
template, such information regarding a project’s input dataset(s) is in the
INPUTS.conf file. See Figures 2 & 3 for the position of INPUTS.conf in the
project’s file structure and data lineage respectively. For demonstration, in
this paper, we are using the datasets of menke20 which are stored in one .xlsx
file on bioXriv141414Full data URL: . Figure 4 shows the corresponding
INPUTS.conf, show the necessary information as Make variables for this
dataset.

Figure 5:  Simplified Make rule, showing how the downloaded data URL is
written into this paper (in Footnote 14). In Make, lines starting with a # are
ignored (thus used for human-readable comments, like the red line shown here).
The _target_ is placed before a colon (:) and its _prerequisite(s)_ is(are)
after the colon. Here, both the target and prerequisite can be seen in the
second line. The executable _recipe_ lines (shell commands to build the target
from the prerequisite), start with a TAB (shown here with a light gray
___TAB___). A Make recipe is an independent, or containerized, shell script.
In the recipe, $@ is an _automatic variable_ , expanding to the target file’s
name. For MK20URL, see Figure 4. The same URL is then passed to this paper
through the definition of the LaTeX variable \menketwentyurl that is written
in $(mtexdir)/download.tex. Later, when the paper’s PDF is being built, this
.tex file is loaded into it. $(mtexdir) is the directory hosting all the LaTeX
macro files for various stages of the analysis, see Section 4.5.3.

If menke20.xlsx exists on the _input_ directory, it will just be validated and
put it in the _build_ directory. Otherwise, it will be downloaded from the
given URL, validated, and put it in the build directory. Recall that the input
and build directories differ from system to system and are specified at
project configuration time, see Section 4.3.1. In the data lineage of Figure
3, the arrow from INPUTS.conf to menke20.xlsx symbolizes this step. Note that
in our notation, once an external dataset is imported, it is a _built_
product, it thus has a blue box in Figure 3.

It is sometimes necessary to report basic information about external datasets
in the report/paper. As described in Section 4.5.3, here this is done with
LaTeX macros to avoid human error. For example in Footnote 14, we gave the
full URL that this dataset was downloaded from. In the LaTeX source of that
footnote, this URL is stored as the \menketwentyurl macro which is created
with the simplified151515This Make rule is simplified by removing the
directory variable names to help in readability. Make rule below (it is
located at the bottom of download.mk).

In this rule, download.tex is the _target_ and menke20.xlsx is its
_prerequisite_. The _recipe_ to build the target from the prerequisite is the
echo shell command which writes the LaTeX macro definition as a simple string
(enclosed in double-quotes) into the download.tex. The target is built after
the prerequisite(s) are built, or when the prerequisites are newer than the
target (for more on Make, see Appendix A.5.2). Note that $(MK20URL) is a call
to the variable defined above in INPUTS.conf. Also recall that in Make, $@ is
an _automatic variable_ , which is expanded to the rule’s target name (in this
case, download.tex). Therefore if the dataset is re-imported (possibly with a
new URL), the URL in Footnote 14 will also be re-created automatically. In the
data lineage of Figure 3, the arrow from menke20.xlsx to download.tex
symbolizes this step.

####  4.5.7 The analysis

The analysis subMakefile(s) are loaded into top-make.mk after the
initialization and download steps (see Sections 4.5.6 and 4.5.5). However, the
analysis phase involves much more complexity. If done without modularity in
mind from the start, research project sources can become very long, thus
becoming hard to modify, debug, improve or read. Maneage is therefore designed
to encourage and facilitate splitting the analysis into multiple/modular
subMakefiles. For example in the data lineage graph of Figure 3, the analysis
is broken into three subMakefiles: format.mk, demo-plot.mk and analysis3.mk.

Theoretical discussion of this phase can be hard to follow, we will thus
describe a demonstration project on data from menke20. In Section 4.5.6, the
process of importing this dataset into the project was described. The first
issue is that menke20.xlsx must be converted to a simple plain-text table
which is generically usable by simple tools (see the principle of minimal
complexity in Section 3.4). For more on the problems with Microsoft Office and
this format, see Section 4.5. In format.mk (Figure 6), we thus convert it to a
simple white-space separated, plain-text table (menke20-table-3.txt) and do a
basic calculation on it.

Figure 6: Simplified contents of format.mk. Here, we want to convert the
downloaded XLSX dataset (Office Open XML Workbook format) to a simple plain-
text fixed-width-per-column table. For the position of this subMakefile in the
full project’s data lineage, see Figure 3. In particular, here the arrows of
that figure from menke20.xlsx to menke20-table-3.txt and from the latter to
format.tex are shown as the second and third Make rules. See Figure 5 and
Appendix A.5.2 for more on the Make notation and Section 4.5.7 for describing
the steps.

As shown in Figure 6, the first operation (or Make _rule_) is to define a
directory to keep the generated files. To keep the source and build-
directories separate, we thus define a1dir under the build-directory (BDIR,
see Section 4.3.1). We’ll then define all outputs/targets to be under this
directory. The second rule (which depends on the directory as a prerequisite),
then converts the Microsoft Excel spreadsheet file to a simple plain-text
format using the XLSX I/O program. But XLSX I/O only converts to CSV and we
don’t need all the columns here, so we further shorten and modify the table
(re-order columns and multiply them) using the AWK program (which is available
on any Unix-like operating system). In Figure 3 on the example data lineage,
this second rule is shown with the arrow from menke20.xlsx to
menke20-table-3.txt.

Finally, as described in Section 4.5.3, the last rule of a subMakefile should
be a LaTeX macro file (in Figure 6, this is the third rule). Ending each
analysis phase with a LaTeX macro is natural in many papers/reports. For
example, here, once the dataset is ready, we want to give the reader a general
view of the dataset size. We thus need to report the number of subjects
studied in menke20. Therefore in the LaTeX macro rule, we count them from the
simplified table of the second rule. In both cases, we write the sum as a
temporary shell variable v, which is respectively written into these two LaTeX
macros \menkenumpapers and \menkenumjournals. In the built PDF paper, they
expand to $\menkenumpapers$ (number of papers studied) and $\menkenumjournals$
(number of journals studied) respectively. This rule is shown schematically in
Figure 3 with the arrow from menke20-table-3.txt to format.tex.

To further demonstrate the concept, we’ll reproduce (with some enhancements)
Figure 1C of menke20 in Figure 7. Figure 7 also shows the number of papers
that were studied each year in the same plot (unlike the original plot). Its
horizontal axis also shows the full range of the data (starting from
$\menkefirstyear$) while the original Figure 1C in menke20 starts from 1997.
The reason menke20 decided to avoid earlier years was probably the small
number of papers before 1997. For example in , they had only studied papers.
Note that both the numbers of this sentence, and the first year of data
mentioned above, are actually LaTeX macros, see Figure 8).

figure-tools-per-year

Figure 7: Fraction of papers mentioning software tools (green line, left
vertical axis) to total number of papers studied in that year (light red bars,
right vertical axis in log-scale). Data from menke20. The subMakefile
archiving the executable lineage of figure’s data is shown in Figure 8 and
discussed in Section 4.5.7.

The operation of reproducing that figure is a contextually separate operation
from the operations that were described above in format.mk. Therefore we add a
new subMakefile to the project called demo-plot.mk, which is shown in Figure
8. As before, in the first rule, we make the directory to host the data
(a2dir). However, unlike before, this directory is placed under texdir which
is the directory hosting all LaTeX related files. This is because the plot of
Figure 7 is directly made within LaTeX, using its PGFPlots
package161616PGFPLots package of LaTeX: https://ctan.org/pkg/pgfplots. texdir
has some special features when using LaTeX, see Section
LABEL:sec:buildingpaper. PGFPlots uses the same graphics engine that is
building the paper, producing a high quality figure that blends nicely in the
paper.. Note that this is just our personal choice, other methods of
generating plots (for example with R, Gnuplot or Matplotlib) are also possible
within this system. As with the input data files of PGFPlots, it is just
necessary to put the files that are loaded into LaTeX under the $(BDIR)/tex
directory, see Section 4.8.

The plain-text table that is used to build Figure 7 is defined as the variable
a2mk20f1c of Figure 8 (just above the second rule). As shown in the second
rule, again we use GNU AWK to extract the necessary information from mk20tab3
(which was built in format.mk). mk20tab3 is thus the _prerequisite_ of
a2mk20f1c (along with a2dir). In Figure 3, this lineage is shown as the arrow
from menke20-table-3.txt (file name of mk20tab3) that points to tools-per-
year.txt (file name of a2mk20f1c).

As with all subMakefiles, demo-plot.mk finishes with the rule to build its
LaTeX macro file (demo-plot.tex) containing the values reported above. But
here, it doesn’t just depend on a2mk20f1c, it also depends on the menke-demo-
year.conf configuration file. This is also visible in the data lineage (Figure
3): two arrows point to demo-plot.tex, one from a configuration file, and one
from a built file. Configuration files are discussed in more detain in Section
4.5.8.

Figure 8: Contents of analysi2.mk subMakefile used to generate the data for
Figure 7.

In a similar manner many more subMakefiles can be added in more complex
analysis scenarios. This is shown with the lower opacity files and dashed
arrows of the data lineage in Figure 3. Generally, the files created within
one subMakefile don’t necessarily have to be a prerequisite of its LaTeX
macro. For example see demo-out.dat in Figure 3: it is managed in demo-
plot.mk, however, it isn’t a prerequisite of demo-plot.tex, it is a
prerequisite of out-3b.dat (which is managed in another-step.mk and is a
prerequisite of another-step.tex). Hence ultimately, through another file,
it’s descendants conclude in a LaTeX macro.

The high-level top-make.mk file is designed to simplify the addition of new
subMakefiles for the authors, and reading the source for readers (see Section
4.4). As mentioned before, this high-level Makefile just defines the ultimate
target (paper.pdf, see Section 4.5.2) and imports all the subMakefiles in the
specific order. For example Figure 9 shows this project’s top-make.mk. When
descriptive names are chosen for the subMakefiles, a simple glance over the
values to makesrc here provides a general understanding of the project without
needing to get into the technical details.

Figure 9:  General view of the High-level top-make.mk Makefile which manages
the project’s analysis that is in various subMakefiles. See Figures 2 & 3 for
its location in the project’s file structure and its data lineage, as well as
the subMakefiles it includes.

####  4.5.8 Configuration files

The analysis subMakefiles discussed above in Section 4.5.7 should only contain
the organization of an analysis, they should not contains any fixed numbers,
settings or parameters. Such elements should only be used as variables that
are defined elsewhere. In the data lineage plot of Figure 3, configuration
files are shown as the sharp-edged, green *.conf files in the top row.

The last recipe of Figure 8 is a good demonstration of their usage: in Section
4.5.7, we reported the number of papers studied by menke20 in . However, note
that in Figure 8, the year’s number is not written by hand in the subMakefile.
It is referenced through the menke-year-demo variable, which is defined in
menke-demo-year.conf, that is a prerequisite of the demo-plot.tex rule. This
is also visible in the data lineage of Figure 8.

All the configuration files of a project are placed under the
reproduce/analysis/config (see Figure 2) subdirectory, and are loaded into
top-make.mk before any of the subMakefiles, see Figure 9. The configuration
files greatly simplify project management from multiple perspectives as listed
below:

  * •

If an analysis parameter is used in multiple places within the project, simply
changing the value in the configuration file will change it everywhere in the
project. This is cortical in more complex projects and if not done like this
can lead to significant human error.

  * •

Configuration files enable the logical separation between the low-level
implementation and high-level running of a project. For example after writing
the project, the authors don’t need to remember where the number/parameter was
used, they can just modify the configuration file. Other co-authors, or
readers, of the project also benefit: they just need to know that there is a
unified place for high-level project settings, parameters, or numbers without
necessarily having to know the low-level implementation.

  * •

A configuration file will be a prerequisite to any rule that uses it’s value.
If the configuration file is updated (the value/parameter is changed), Make
will automatically detect the data lineage branch that is affected by it and
re-execute only that branch, without any human interference.

This is a great leap compared the current, mostly manual, project management
that many scientists employ. Manual management is prone to serious human error
factors: at the later phases of a project, scientists are least likely to
experiment on their project’s configurations. However, the later phases of a
project are precisely the times where the lower-level parts of the project are
complete and the authors can look at the bigger picture. This style of
managing project parameters therefore produces a much more healthy scientific
result where experimentation is very cheap during all phases of a project;
before its publication (by the authors) and after it (by the authors and
readers).

###  4.6 Projects as Git branches of Maneage

Maneage is fully composed of plain-text files distributed in a directory
structure (see Sections 3.3 & 4.2 and Figure 2). Therefore it can be
maintained under under version control systems like Git (for more on version
control, see Appendix A.4). Every commit in the version controlled history
contains _a complete_ snapshot of the data lineage, for more see the
completeness principle in Section 3.1.

Maneage is maintained by its developers in a central branch, which we’ll call
maneage hereafter. The maneage branch contains all the low-level
infrastructure, or skeleton, that is necessary for any project. This skeleton
is primarily the configuration features discussed in Section 4.3171717Recall
that project configuration files are located under reproduce/software in
Figure 2, and executed with the ./project configure command.. The maneage
branch only contains a minimal demonstration analysis in order to be
complete181818The names of all the files related to the demonstration of the
maneage branch have a delete-me prefix to highlight that they must be deleted
when starting a new project..

To start a new project, users simply clone it from its reference repository
and build their own Git branch over the most recent commit. This is
demonstrated in the first phase of Figure 10: the project started by branching
of commit 0c120cb in maneage. They can then start customizing Maneage for
their project and adding their high-level analysis in their own branch and
push it to their own Git repository. Manages contains a file called README-
hacking.md that has a complete checklist of steps to start a new project and
remove demonstration parts. This file is updated on the maneage branch and
will always be up-to-date with the low-level infrastructure.

figure-branching

Figure 10:  Projects start by branching off the main Maneage branch and
developing their high-level analysis over the common low-level infrastructure:
add flesh to a skeleton. The low-level infrastructure can always be updated
(keeping the added high-level analysis intact), with a simple merge between
branches. Two phases of a project’s evolution shown here: in phase 1, a co-
author has made two commits in parallel to the main project branch, which have
later been merged. In phase 2, the project has finished: note the identical
first project commit and the Maneage commits it branches from. The dashed
parts of Scenario 2 can be any arbitrary history after those shown in phase 1.
A second team now wants to build upon that published work in a derivate
branch, or project. The second team applies two commits and merges their
branch with Maneage to improve the skeleton and continue their research. The
Git commits are shown on their branches as colored ellipses, with their hash
printed in them. The commits are colored based on the team that is working on
that branch. The collaboration and paper icons are respectively made by
‘mynamepong’ and ‘iconixar’ and downloaded from www.flaticon.com.

After a project starts, Maneage will evolve. For example new features will be
added, low-level bugs will be fixed that are useful for any project. Because
all the changes in Maneage are committed on the maneage branch (that projects
also branch-off from) updating the project’s low-level infra-structure is as
easy as merging the maneage branch into the project’s branch. For example see
how Maneage’s 3c05235 commit has been merged into project’s branch trough
commit 2ed0c82 in Figure 10 (phase 1).

This doesn’t just apply to the pre-publication phase, when done in Maneage, a
project can be revived at any later date by other researchers as shown in
phase 2 of Figure 10. In that figure, a new team of researchers have decided
to experiment on the results of the published paper and have merged it with
the Maneage branch (commit a92b25a) to fix some possible portability problem
for their operating system that was fixed as a bug in Maneage after the
paper’s publication. Propagating bug fixes or improvements in the low-level
infrastructure to all projects using Maneage has been one of the reasons it
evolved so well over the last 5 years, see Section 4.9. As we started using it
in more projects by more users, many bugs were found and improvements made. We
would then implement it in Maneage and that fix would propagate to all other
projects automatically in their next merge.

Other scenarios include a third project that can easily merge various high-
level components from different projects into its own branch, thus adding a
temporal dimension to their data lineage. Modern version control systems
provide many more capabilities that can be exploited through Maneage in
project management, thanks to the shared branch it has with _all_ projects
that use it.

###  4.7 Multi-user collaboration on single build directory

Because the project’s source and build directories are separate, it is
possible for different users to share a build directory, while working on
their own separate project branches during a collaboration. Similar to the
parallel branch that is later merged in phase 1 of Figure 10.

To give all users privilege, Maneage assumes that they are in the same (POSIX)
user group of the system. All files built in the build directory are then
automatically assigned to this user group, with read-write permissions for all
group members (-rwxrwx—), through the sg and umask commands that are prepended
to the call to Make. The ./project script has a special –group option which
activates this mode in both configuration and analysis phases. It takes the
user group name as its argument and the built files will only be accessible by
the group members, even when the shared location is accessible by people
outside the project.

When multiple project members are contributing on a shared build directory,
they usually work on independent parts of the project which won’t cause any
conflict. When there is a conflict, a member can temporarily change the name
of the part’s top directory within their branch. For example if Alice is
working on the demo-plot.mk (Figure 8) in parallel with others, she can set
a2dir to be $(texdir)/tools-per-year-alice. Other project members can also
compare her results in this way and once it is merged into the master branch,
a2dir can be set to its original value.

The project already applies this strategy for part of the project that runs
LaTeX to build the final report. This is because project members will usually
also be editing their parts of the report/paper as they progress. To fix this,
when the project is configured and built with –group, each project member’s
user-name will be appended to the LaTeX build directory (which is under
$(BDIR)/tex).

However, human error is inevitable, so when the project takes long in some
phases, the user and group write-permission flags can be manually removed from
the respective subdirectories under $(BDIR) until the project is to be built
from scratch; maybe for a test prior to submission.

###  4.8 Publishing the project

Once the project is complete, publishing the project is the final step. In a
scientific scenario, it is public As discussed in the various steps before,
the source of the project (the software configuration, data lineage and
narrative text) is fully in plain text, greatly facilitating the publication
of the project.

####  4.8.1 Automatic creation of publication tarball

To facilitate the publication of the project source, Maneage has a special
dist target during the build process which is activated with the command
./project make dist. In this mode, Maneage will not do any analysis, it will
simply copy the full project’s source (on the given commit) into a temporary
directory and compress it into a .tar.gz file. If a Zip compression is
necessary, the dist-zip target can be called instead dist.

The dist tarball contains the project’s full data lineage and is enough to
reproduce the full project: it can build the software, download the data, run
the analysis, and build the final PDF. However, it doesn’t contain the Git
history, it is just a checkout of one commit. Instead of the history, it
contains all the necessary _built products_ that go into building the final
paper without the analysis: for example the used plots, figures, tables, and
project.tex, see Section 4.5.3. As a result, the tarball can be _also_ only
build the final report with a simple pdflatex paper command _without_ running
./project. When the project is distributed as a tarball (not as a Git
repository), building the report may be the main purpose, like the arXiv
distribution scenario discussed below, the data lineage (under the reproduce/
directory) is likely just a supplement.

####  4.8.2 What to publish, and where?

The project’s source, which is fully in hand-written plain-text, has a very
small volume, usually much less than one megabyte. However, the necessary
input files (see Section 2.1) and built datasets may be arbitrarily large,
from megabytes to petabytes or more. Therefore, there are various scenarios
for the publication of the project as described below:

  * •

Only source: Publishing the project source is very easy because it only
contains plain-text files with a very small volume: a commit will usually be
on the scale of $\times 100kB$. With the Git history, it will usually only be
on the scale of $\sim 5MB$.

    * –

Public Git repository: This is the simplest publication method. The project
will already be on a (private) Git repository prior to publication. In such
cases, the private configuration can be removed so it becomes public.

    * –

In journal or PDF-only preprint systems (e.g., bioRxiv): If the journal or
pre-print server allows publication of small supplement files to the paper,
the commit that produced the final paper can be submitted as a compressed
file, for example with the

    * –

arXiv: Besides simply uploading a PDF pre-print, on arXiv, it is also possible
to upload the LaTeX source of the paper. arXiv will run its own internal LaTeX
engine on the uploaded files and produce the PDF that is published. When the
project is published, arXiv also allows users to anonymously download the
LaTeX source tarball that the authors uploaded191919In the current arXiv user
interface, the tarball is available by clicking the “Other formats” link on
the paper’s main page, and then clicking “Download source”, it can be checked
with https://arxiv.org/abs/1909.11230 of akhlaghi19.. Therefore, simply
uploading the tarball from the ./project make dist command is sufficient for
arXiv, and will allow the full project data lineage to also be published there
with the LaTeX source. We done this in [arXiv:1909.11230]akhlaghi19 and
[arXiv:1911.01430]infante20. Since arXiv is mirrored in many institutes over
the planet, this is a robust way to preserve the reproducible lineage.

    * –

In output datasets: Many data storage formats support an internal structure
with the data file. One commonly used example today is the Hierarchical Data
Format (HDF), and in particular its HDF5 which can host a complex filesystem
in POSIX syntax. It is even used by some reproducible analysis solutions like
the Active papers project [for more, see Appendix B.10]hinsen11. Since the
volume of the project source is so insignificant compared to the output
datasets of most projects, the whole project source can be stored with each
published data file if the format supports it.

  * •

Source and data: The project inputs (including the software tarballs, or
possible datasets) may have a large volume. Publishing them with the source is
thus not always possible. However, based on the definition of inputs in
Section 2.1, they are usable in other projects: another project may use the
same data or software source code, in a different way. Therefore even when
published with the source, it is encouraged to publish them as separate files.

For example strategy was followed in
https://doi.org/10.5281/zenodo.3408481zenodo.3408481202020https://doi.org/10.5281/zenodo.3408481
which supplements akhlaghi19 which contains the following files.

    * –

Final PDF: for easy understanding of the project.

    * –

Git history: as the Git “bundle” of the project. This single file contains the
full Git history of the project until its publication date (only 4Mb), see
Section 4.6.

    * –

Project source tarball: output of ./project make dist, as explained above.

    * –

Tarballs of all necessary software: This is necessary in case the software
webpages is not accessible for any reason at a later date or the project must
be run with no internet access. This is only possible because of the free
software principle discussed in Section 3.7.

Note that akhlaghi19 used previously published datasets which are
automatically accessed when necessary. Also, that paper didn’t produce any
output datasets beyond the figures shown in the report, therefore the Zenodo
upload doesn’t contain any datasets. When a project involves data collection,
or added-value data products, they can also be uploaded with the files above.

####  4.8.3 Worries about getting scooped!

Publishing the project source with the paper can have many benefits for the
researcher and the larger community. For example if the source is published
with a pre-print, others my help the authors find bugs, or improvements to the
source that can affect the validity or precision of the result, or simply
optimize it so it does the same work in half the time for example.

However, one particular feedback raised by a minority of researchers is that
publishing the project’s reproducible data lineage immediately after
publication may hamper their ability to continue harvesting from all their
hard work. Because others can easily reproduce the work, others may take the
next follow-up project they originally intended to do. This is informally
known as getting scooped.

The level that this may happen is an interesting subject to be studied once
many papers become reproducible. But it is a valid concern that must be
addressed. Given the strong integrity checks in Maneage, we believe it has
features to address this problem in the following ways:

  1. 1.

This worry is essentially the 2nd phase of Figure 10. The commits of the other
team are built up on the commits of the original authors. It is therefore
perfectly clear (with the precision of a character!) how much of their result
is purely their own work (qualitatively or quantitatively). In this way,
Maneage can contribute to a new concept of authorship in scientific projects
and help to quantify newton’s famous “standing on the shoulders of giants”
quote. However, this is a long term goal and requires major changes to
academic value systems.

  2. 2.

The authors can be given a grace period where the journal, or some third
authority, keeps the source and publishes it a certain time after publication.
In fact, journals can create specific policies for such scenarios, for example
saying that all project sources will be available publicly, $N$ months/years
after publication while allowing authors to opt-out of it if they like, so the
source is published immediately with the paper. However, journals cannot
expect exclusive copyright to distribute the project source, in the same
manner they do with the final paper. As discussed in the free software
principle of Section 3.7, it is critical that the project source be free for
the community to use, modify and distribute.

This can also be done by the authors on servers like Zenodo, where you can get
the dataset’s final DOI first, and publish at a later date. Reproducibility is
indeed very important for the sciences, but the hard work that went into it
should also be acknowledged for the authors that would like to publish the
source at a later date.

###  4.9 Future of Maneage and its past

As with any software, the core architecture of Maneage will inevitably evolve
after the publication of this paper. The current version introduced here has
already experienced 5 years of evolution and several reincarnations. Its
primordial implementation was written for akhlaghi15. This paper described a
new detection algorithm in astronomical image processing. The detection
algorithm was developed as the paper was being written (initially a small
report!). An automated sequence of commands to build the figures, and update
the paper/report was a practical necessity as the algorithm was evolving. In
particular, it didn’t just reproduce figures, it also used LaTeX macros to
update numbers printed within the text. Finally, since the full analysis
pipeline was in plain-text and roughly 100kb (much less than a single figure),
it was uploaded to arXiv with the paper’s LaTeX source, under a reproduce/
directory, see https://arxiv.org/abs/1505.01664arXiv:1505.01664212121 To
download the LaTeX source of any arXiv paper, click on the “Other formats”
link, containing necessary instructions and links..

The system later evolved in bacon17, in particular the two sections of that
paper that were done by M. Akhlaghi (first author of this paper):
[http://doi.org/10.5281/zenodo.1163746zenodo.1163746]akhlaghi18a and
[http://doi.org/10.5281/zenodo.1164774zenodo.1164774]akhlaghi18b. With these
projects, the skeleton of the system was written as a more abstract “template”
that could be customized for separate projects. The template later matured by
including installation of all necessary software from source and used in
[https://doi.org/10.5281/zenodo.3408481zenodo.3408481]akhlaghi19 and
[https://doi.org/10.5281/zenodo.3524937zenodo.3524937]infante20. The short
historical review above highlights how this template was created by practicing
scientists, and has evolved and matured significantly.

We already have roughly 30 tasks that are left for the future and will affect
various high-level phases of the project as described here. However, the core
of the system has been used and become stable enough already and we don’t see
any major change in the core methodology in the near future. A list of the
notable changes after the publication of this paper will be kept in in the
project’s README-hacking.md file. Once the improvements become substantial,
new paper(s) will be written to complement or replace this one.

##  5 Discussion

  * •

Science is defined by its method, not its results. Just as papers are quality
checked for a reasonable English (which is not necessary for conveying the
final result), the necessities of modern science require a similar check on a
reasonable review of the computation, which is easiest to check when the
result is exactly reproducible.

  * •

Initiative such as https://software.ac.uk (UK) and http://urssi.us (USA) are
good attempts at improving the quality of research software.

  * •

Hiring software engineers is not the solution: the language of science has
changed. Could Galileo have created a telescope if he wasn’t familiar with
what a lens is? Science is not independent of its its tools.

  * •

The actual processing is archived in multiple places (with the paper on arXiv,
with the data on Zenodo, on a Git repository, in future versions of the
project).

  * •

As shown by the very common use of something like Conda, Software (even free
software) is mainly seen in executable form, but this is wrong: even if the
software source code is no longer compilable, it is still readable.

  * •

The software/workflow is not independent of the paper.

  * •

Cost of archiving is a critical issue (the NSF director mentions this during
the first Panel of the National Academies
meeting222222https://vimeo.com/367085708 (around minute 45:00)).

  * •

Meta-science (or “Science of science”, “economics of science”, “Research on
research”) and its importance.

  * •

Provenance tracking (like some tools in Appendix B) is built-in and doesn’t
need any manual tagging.

  * •

Important that a project starts by following good practice fineberg19, not an
extra step in the end.

  * •

It is possible to write graphic user interface wrappers like those in Appendix
B.

  * •

What is often discussed is “taking data and apply different methods to it”,
but an even more productive questions can be this: “take (exact) methods and
give different data to it”.

  * •

munafo19 discuss how collective action is necessary.

  * •

Research objects (Appendix B.16) can automatically be generated from the
Makefiles, we can also apply special commenting conventions, to be included as
annotations/descriptions in the research object metadata.

  * •

Provenance between projects: through Git, all projects based on this template
are automatically connected, but also through inputs/outputs, the lineage of a
project can be traced back to projects before it also.

  * •

gibney20: After code submission was encouraged by the Neural Information
Processing Systems (NeurIPS), the frac

  * •

When the data are confidential, perignon19 suggest to have a third party
familiar with coding to referee the code and give its approval. In this
system, because of automatic verification of inputs and outputs, no technical
knowledge is necessary for the verification.

  * •

miksa19b Machine-actionable data management plans (maDMPs) embedded in
workflows, allowing

  * •

miksa19a RDA recommendation on maDMPs.

  * •

FAIR Principles wilkinson16.

  * •

cheney09: “In both data warehouses and curated databases, tremendous (_and
often manual_) effort is usually expended in the construction”

  * •

https://arxiv.org/pdf/2001.11506.pdf

  * •

Apache NiFi for automated data flow.

  * •

https://arxiv.org/pdf/2003.04915.pdf: how data lineage can help machine
learning.

  * •

Interesting patent on “documenting data lineage”:
https://patentimages.storage.googleapis.com/c0/51/6e/1f3af366cd73b1/US10481961.pdf

  * •

Automated data lineage extractor: http://hdl.handle.net/20.500.11956/110206.

  * •

Caveat: Many low-level tools.

  * •

High-level tools can be written to exploit the low-level features.

##  6 Acknowledgments

The authors wish to thank Pedram Ashofteh Ardakani, Zahra Sharbaf and Surena
Fatemi for their useful suggestions and feedback on Maneage and this paper.
Work on the reproducible paper template has been funded by the Japanese
Ministry of Education, Culture, Sports, Science, and Technology (MEXT)
scholarship and its Grant-in-Aid for Scientific Research (21244012, 24253003),
the European Research Council (ERC) advanced grant 339659-MUSICOS, European
Union’s Horizon 2020 research and innovation programme under Marie Sklodowska-
Curie grant agreement No 721463 to the SUNDIAL ITN, and from the Spanish
Ministry of Economy and Competitiveness (MINECO) under grant number
AYA2016-76219-P. The reproducible paper template was also supported by
European Union’s Horizon 2020 (H2020) research and innovation programme via
the RDA EU 4.0 project (ref. GA no. 777388).

Appendices

##  Appendix A Survey of existing tools for various phases

Conducting a reproducible research project is a high-level process, which
involves using various lower-level tools. In this section, a survey of the
most commonly used lower-level tools for various aspects of a reproducible
project is presented with an introduction as relates to reproducibility and
the proposed template. In particular, we focus on the tools used within the
proposed template and also tools that are used by the existing reproducible
framework that is reviewed in Appendix B. Some existing solution to for
managing the different parts of a reproducible workflow are reviewed here.

###  A.1 Independent environment

There are three general ways of controlling the environment: 1) Virtual
machines, 2) Containers, 3) controlled build and environment. Below, a short
description of each solution is provided.

####  A.1.1 Virtual machines

Virtual machines (VMs) keep a copy of a full operating system that can be run
on other operating systems. This includes the lowest-level kernel which
connects to the hardware. VMs thus provide the ultimate control one can have
over the run-time environment of an analysis. However, the VM’s kernel does
not talk directly to the hardware that is doing the analysis, it talks to a
simulated hardware that is provided by the operating system’s kernel.
Therefore, a process that is run inside a virtual machine can be much slower
than one that is run on a native kernel. VMs are used by cloud providers,
enabling them to sell fully independent operating systems on their large
servers, to their customers (where the customer can have root access). But
because of all the overhead, they aren’t used often used for reproducing
individual processes.

####  A.1.2 Containers

Containers are higher-level constructs that don’t have their own kernel, they
talk directly with the host operating system kernel, but have their own
independent software for everything else. Therefore, they have much less
overhead in storage, and hardware/CPU access. Users often choose an operating
system for the container’s independent operating system (most commonly
GNU/Linux distributions which are free software).

Below we’ll review some of the most common container solutions: Docker and
Singularity.

  * •

Docker containers: Docker is one of the most popular tools today for keeping
an independent analysis environment. It is primarily driven by the need of
software developers: they need to be able to reproduce a bug on the “cloud”
(which is just a remote VM), where they have root access. A Docker container
is composed of independent Docker “images” that are built with Dockerfiles. It
is possible to precisely version/tag the images that are imported (to avoid
downloading the latest/different version in a future build). To have a
reproducible Docker image, it must be ensured that all the imported Docker
images check their dependency tags down to the initial image which contains
the C library.

Another important drawback of Docker for scientific applications is that it
runs as a daemon (a program that is always running in the background) with
root permissions. This is a major security flaw that discourages many high
performance computing (HPC) facilities from installing it.

  * •

Singularity: Singularity is a single-image container (unlike Docker which is
composed of modular/independent images). Although it needs root permissions to
be installed on the system (once), it doesn’t require root permissions every
time it is run. Its main program is also not a daemon, but a normal program
that can be stopped. These features make it much easier for HPC administrators
to install Docker. However, the fact that it requires root access for initial
install is still a hindrance for a random project: if its not present on the
HPC, the project can’t be run as a normal user.

  * •

Virtualenv: Discuss it later.

When the installed software within VMs or containers is precisely under
control, they are good solutions to reproducibly “running”/repeating an
analysis. However, because they store the already-built software environment,
they are not good for “studying” the analysis (how the environment was built).
Currently, the most common practice to install software within containers is
to use the package manager of the operating system within the image, usually a
minimal Debian-based GNU/Linux operating system. For example the
Dockerfile232323https://github.com/benmarwick/1989-excavation-report-
Madjedbebe/blob/master/Dockerfile in the reproducible scripts of clarkso15,
which uses sudo apt-get install r-cran-rjags -y to install the R interface to
the JAGS Bayesian statistics (rjags). However, the operating system package
managers aren’t static. Therefore the versions of the downloaded and used
tools within the Docker image will change depending when it was built. At the
time clarkso15 was published (June 2015), the apt command above would download
and install rjags 3-15, but today (January 2020), it will install rjags 4-10.
Such problems can be corrected with robust/reproducible package managers like
Nix or GNU Guix within the docker image (see Appendix A.2), but this is rarely
practiced today.

####  A.1.3 Package managers

The virtual machine and container solutions mentioned above, install software
in standard Unix locations (for example /usr/bin), but in their own
independent operating systems. But if software are built in, and used from, a
non-standard, project specific directory, we can have an independent build and
run-time environment without needing root access, or the extra layers of the
container or VM. This leads us to the final method of having an independent
environment: a controlled build of the software and its run-time environment.
Because this is highly intertwined with the way software are installed, we’ll
describe it in more detail in Section A.2 where package managers are reviewed.