# A Backend Platform for Supporting the Reproducibility of Computational
Experiments

Lázaro Costa1,2, Susana Barbosa2, Jácome Cunha1,2
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
( 1 Faculty of Engineering, University of Porto, Portugal
2 INESC TEC, Portugal)

###### Abstract

In recent years, the research community has raised serious questions about the
reproducibility of scientific work. In particular, since many studies include
some kind of computing work (for instance, executing an algorithm encoded
using some programming language), reproducibility is also a technological
challenge, not only in computer science, but in most research domains.

Replicability and computational reproducibility are not easy to achieve, not
only because researchers have diverse proficiency in computing technologies,
but also because of the variety of computational environments that can be
used. Indeed, it is challenging to recreate the same environment using the
same frameworks, code, programming languages, dependencies, and so on.

In this work, we propose an Integrated Development Environment allowing the
share, configuration, packaging and execution of an experiment by setting the
code and data used and defining the programming languages, code, dependencies,
databases, or commands to execute to achieve consistent results for each
experiment. After the initial creation and configuration, the experiment can
be executed any number of times, always producing exactly the same results.
Furthermore, it allows the execution of the experiment by using a different
associated dataset, and it can be possible to verify the reproducibility and
replicability of the results. Our platform aims to allow researchers in any
field to create a reproducibility package for their science that can be re-
executed on any other computer.

To evaluate our platform, we used it to reproduce 25 experiments extracted
from published papers. We have been able to successfully reproduce 20 (80%) of
these experiments achieving the results reported in such works with minimum
effort, thus showing that our approach is effective.

##  1 Introduction

Reproducibility is the ability to replicate the results of a study using the
original methods, materials, and data. It is paramount for science and is
widely considered to be a precondition of scientific advancement [6, 17].
Replicability is the capacity to obtain the same results using new methods,
materials, and data and/or consistent conditions with the original study [6,
17].

In recent years, many concerns have been raised related to the reproducibility
of today’s computational work. A significant amount of scientific research is
nowadays supported by some kind of computational experiment. Thus,
reproducible research is now more than ever a technological challenge in many
research domains.

Computational reproducibility should be easily achievable since it is possible
to specify detailed information about the computational environment being
executed. This should enable the reproduction of the computation environment
in the same way. But in practice, this does not happen (see section 3). The
complexity of today’s software makes the reproducibility process more
difficult [32].

Research in many fields (e.g., chemistry, climate change, biology) is
supported, in several studies, by computational experiments [32]. However,
many of these researchers are not experts in software development. Thus,
although they may be able to create the necessary code for their experiments
(e.g., scripts in Python or R), they may find it difficult to create
reproducible computing experiments. In fact, this is even the case for
computer science conferences (section 3).

To keep research reproducible, it is essential to investigate, perform,
analyze, and rerun the experiments associated with that research in order to
obtain corresponding results as before. For this to be possible, when working
with a computational experiment, it is necessary to rebuild the original
computing environment using the original code and its dependencies, the same
data, and the same script-based workflow [45, 5].

Describing all the dependencies used and which versions are used is not a
trivial task. Package management tools of each programming language (PL), such
as pip111https://pypi.org/project/pip/ for Python or
npm222https://www.npmjs.com for JavaScript, have integrated commands which can
build a file with the dependencies needed for an experiment. However, these
files do not have all the information to enable computational reproducibility.
Moreover, researchers without a computer science background may not know how
to use such tools [32]. There are tools to aid researchers in making their
computational experiments reproducible such as Open Lab [53], SciInc [62],
CDE-SP [48], Encapsulator [47], A Framework [43], CARE [33], VisTrails [24],
Binder [7], RenkuLab [37], Whole Tale [4], ReproZip [10, 12, 11], or RunMyCode
[54]. However, these approaches are limited to a set of PLs, do not allow the
use of more than one PL per project, do not allow the use of databases, or do
not allow researchers to specify which commands should be executed to achieve
the experiment results. The Association for Computing Machinery (ACM) has
motivated researchers to build a reproducibility package by creating a virtual
machine. However, again this is not easy for many researchers. In section 8 we
further discuss related work.

To assess the use of reproducibility tools, we defined the following research
question (RQ), which we answer in section 3:

RQ1: What is the adoption of reproducibility tools among researchers?

To answer this RQ we designed and distributed an online survey. Although the
aforementioned approaches allow the reproducibility of experiments, just one
of the 38 researchers answering our survey acknowledged to have used one of
these platforms to save and reproduce computational experiments.

Thus, with our work, we mainly seek to answer the following RQ (section 4):

RQ2: To which extent can we propose a platform that helps to make
computational experiments from different fields reproducible?

To answer this RQ, we propose an Integrated Development Environment (IDE) that
allows the share, creation, configuration, (re-) execution and packaging of
computational experiments, which also includes the validation of
reproducibility and replicability of the results. Researchers need to
associate a dataset to each experiment, however, it is possible to execute
using a different dataset. In this way, researchers can compare the results
achieved with the results of previous runs by using the associated data or
different data.

For now, we focus our work on the backend of the platform, not providing a
user interface (which we plan to design and build in future work). Our
platform is developed to support a variety of experiments, including
Artificial Intelligent (AI) experiments. For these experiments, we make
researchers aware of the importance of making the experience reproducible,
helping them associate the possible seed used with the project. In addition,
Furthermore, our approach can be used to create a capsule of the research
experiment containing the code and its dependencies, the data and databases,
the process, and the commands to achieve the experiment results. Our main goal
is to support the researchers in developing, executing and sharing the code
they produce for their experiments, allowing them to manage the data and code
associated with that experiment, without the need to manage dependencies,
compilers, and other variables. The proposed platform can handle many PLs,
databases, and other data files. It can also automatically infer information
from the experiment, such as PLs and dependencies, creating the proper
environment for the execution. The researcher can then easily export a
reproducibility package that can be executed on any computer just by double-
clicking a single file, making the experiment easily sharable and re-
executable.

To evaluate our platform, we selected a set of experiments included in already
published papers and tried to recreate them using our platform. In particular,
we seek to answer the following RQ (section 5):

RQ3: What kinds of experiments can supported?

To answer this RQ, we collected experiments from related work, from major
conferences, and some others from Zenodo so we could have a variety of
experiments. We collected 31 experiments from fields such as computer science,
medicine, or even climate change science. We were, in general, successful in
reproducing the experiments, achieving the same published results, and were
able to create a reproducibility package that anyone can re-execute just by
executing a single file.

Based on these RQs and our answers to them, the main contributions of this
work are:

  * •

a set of principles that a reproducibility platform should follow (section 2);

  * •

an initial overview of the perspectives of researchers about reproducibility
difficulties and tool usage (section 3);

  * •

the backend of a reproducibility platform tackling the reproducibility
principles and the researchers’ perspectives that can cope with experiments
from very different scientific fields (section 4);

  * •

a list of several reproducibility packages for existing scientific
publications (section 5);

Before we introduce our platform, in section 2 we discuss in more detail the
challenges of reproducing computational experiments, and we define a set of
principles a platform supporting computational reproducibility must follow to
be successful. This may help future researchers proposing other approaches for
reproducibility.

##  2 Challenges in Reproducibility

The digital transformation of science has made available new publication
channels for research data. These publications come from specialized data
repositories, large centralized or decentralized platform infrastructures, or
commercial application providers to traditional self-hosted web server
downloads. Some examples of these platforms are CKAN333https://ckan.org/,
DSpace444https://dspace.lyrasis.org/, EUDAT555https://www.eudat.eu/, GitHub
666https://github.com/, the Harvard
Dataverse777https://dataverse.harvard.edu/, Zenodo888https://zenodo.org/, and
derived systems.

Nowadays, these platforms provide ways to upload and publish research datasets
with an institutional, disciplinary focus, or general purpose. Such platforms
also need to fill the needs of different stakeholders in a variety of usage
scenarios [39, 36].

Although these platform help spread science, they are not enough to make it
reproducible. Researchers need to expose their results, but also other
artefacts, such as code and data, and explain how to use them to achieve the
published results. However, these platforms do not provide means to make the
code easily reproducible.

Based on previous work, we summarize a set of challenges a researcher needs to
take into consideration when intending to make an experiment easily
reproducible:

C1: Software

In the research process, the software has a high impact on the reproducibility
of experiments [14].

When someone tries to reproduce an experiment but uses the wrong software, the
wrong version, or even the wrong dependencies, the experiment may produce
different results from the original one. It is known as Code Rot [3]. The
management of the software to use is difficult as even newer versions of
software that claim to be compatible with the previous ones, may have
unintended differences that can affect reproducibility. To allow
reproducibility, all the information related to the software should be
identified to ensure a consistent behavior [32].

The term Dependency Hell refers to a common issue in software development
where multiple packages share a dependency, but depend on different and
incompatible versions of that dependency. As only one version of a dependency
is allowed in a project’s environment, resolving these conflicts and finding a
compatible solution can be challenging [48].

Another potential problem with reproducibility is when the libraries are
unavailable. To get these libraries, it is necessary to find them which
hinders reproducibility.

Software is the tool that helps the researcher in the reproducible process
(create, configure, run, encapsulate and re-run), however, it is not easy to
find easy-to-use and complete solutions [25, 64].

C2: Command

It is crucial to make available the commands to execute the code, along with
the file to execute and all the values of the parameters needed to reproduce
the experiment. Some researchers prefer not to use parameters on the command,
but this approach makes it difficult for those who intend to extend or explore
the research. So using parametrization is an important method for extension,
as it could give both the original researcher and other researchers the
ability to easily extend the approach, explore the parameter space, and
validate results [32].

However, using a single command with too many parameters can be very confusing
for other researchers to extend the experiment. To make the extension
exploration and reproducibility easier, it is better to simplify the command
and break it into smaller commands and smaller workflows. In the end, it is
necessary to explain what each command does and organise them in a general
workflow [59, 42, 32].

Furthermore, the expected result should be available for comparison, because
there is no guarantee that the same command applied by a different user on a
different machine will get the same result. This can happen due to the
differences in the dependencies installed, the software, the data used, and
the dependencies implicitly provided by the system [32].

C3: Data

Making available all the data is another important element to allow
reproducibility [41]. Most researchers use some kind of input as starting data
as well as the final generated results of their experiments. These inputs
sometimes are in remote locations, so security or privacy issues can occur
during the data download.

Some data files may contain private information, that is normally omitted from
the publication, but without it, the experiment is not reproducible. Receiving
this information as an input parameter can identify the need for the
information without publishing the delicate data, enhancing reproducibility
[32].

Furthermore, researchers intend to find and reuse existent data, but the
existing platforms and approaches do not include sufficient metadata (e.g.,
title, authors, version, PLs, software description, Uniform Resource
Identifier/DOI and execution requirements) to find and reuse data [55].

C4: Hardware

Enabling reproducibility using the same performance, scalability, and
efficiency is another challenge because the same hardware should be used
(e.g., the same computing power such as RAM, CPU, GPU, and storage). However,
in this work, we will not take into consideration experiments that may have
hardware requirements.

From the challenges C1, C2, and C3 we just discussed, we identify a set of
principles a reproducibility platform should follow. We also introduce
principles that are orthogonal to the challenges.

From C1

P1

Such a reproducibility platform should allow researchers to define the PLs
used in the experiment, the compilers, the dependencies, and the libraries, as
well as the correct version of each.

P2

It should allow saving all the dependencies needed to execute the experiment
together with the code.

From C2

P3

The platform should support the configuration of the commands for the
execution of the experiment.

P4

It should allow the validation of the computational reproducibility using the
results of previously performed experiments as a means of comparison.

From C3

P5

Such a platform should provide mechanisms that allow researchers to manage the
data of the experiment.

Orthogonal principals

P6

It should be able to deal with the diversity of PLs (and their dependencies)
used by researchers.

P7

It should support all types of experiments, from the execution of simple
scripts to the execution of more complex experiments (e.g., involving
databases or several PLs).

P8

The platform should support researchers with little knowledge of computing.

P9

It should enable the sharing of the experience easily.

P10

It should facilitate the re-execution of the experiment even for users outside
the research domain.

In this section, we have synthesized the challenges already identified in the
literature. However, we also want to have some more insights from the
researchers themselves. Thus, in the following section, we present a small
survey about the views of researchers on reproducibility.

##  3 Researchers’ Experience with Reproducibility

We performed a survey999Available at
https://docs.google.com/forms/d/e/1FAIpQLScUtalm-7X0BwjKbNtpam5WDywQ8wWVTzfO3sfE8K0yA0c1EQ/viewform.
in which the main goal was to gather some information related to the
familiarity of researchers with the concept, the challenges, requirements, and
the main difficulties of computational reproducibility. Furthermore, it is
important to know which platforms researchers use/know to reproduce
experiments. Note that this survey is not intended to be complete, but rather
to give us a more informed view about researchers’ perspectives on
reproducibility.

We made available this survey to all the authors of the 2022 edition of the
International Conference on Software Engineering since we used some of these
works in our evaluation (see section 5). From the 700 contacted authors, we
got 38 responses. Moreover, we spread the same survey on social media
platforms (Twitter, Facebook, and LinkedIn) through public posts, and also
distributed the survey through our contacts via email, for which we got 38
responses too.

Our sample is composed mostly of Ph.D. students (46%), followed by faculty
members (33%) and researchers (11%). They are from Portugal (46%), China
(16%), and the USA (13%). Furthermore, 90% of the respondents have more than
two years of research experience.

The “computational reproducibility” term is familiar to 84% of the surveyed,
and 79% tried to reproduce experiments performed by others. However, most of
them, 66%, had difficulties in doing so.

As we can see, only a few researchers did not find it difficult to reproduce
others’ works.

The main challenges in computational reproducibility found in this survey are
“Missing information to execute the experiment” (34%), “Data not available”
(25%), “Code not available” (24%), and “Installation not successful” (17%).

Only 11% of the surveyed know a reproducibility platform, namely Binder, while
all the others do not know any. Only 8% of the surveyed reproduce their
experiments using docker101010Docker is a platform designed to help developers
build, share, and run modern applications. Docker uses service virtualization
at the operating system level (https://www.docker.com/)., 8% of the surveyed
publish the code on Zenodo, and 5% on GitHub .

Finally, we asked what requirements were considered necessary to allow
computational reproducibility. The answers were “Make available the data”
(29%), “Make available the code” (28%), “Make available the description of all
the dependencies used and its versions” (24%), and “Make available the PL used
and its version” (18%).

To conclude, the answers to this survey show us that the term computational
reproducibility is a term well-known in the research community. However,
researchers usually have difficulties when trying to reproduce experiments
performed by others. Although with this small survey, we do not have a
complete view of the researchers’ perspectives on reproducibility, we can
conclude that, although there are already some proposals to aid in reproducing
computational experiments (see section 8), the researchers that answered this
survey still do not use them at scale, which also motivates our own work.

##  4 Design and Implementation of a Reproducibility Platform

Based on the principles defined in section 2 and the data collected from the
researchers described in section 3 in this section, we present our design and
implementation of a backend of a platform that gives support to the creation
and execution of computational experiments. Notice we focus our work on the
software environment and assume the hardware does not influence the outcome of
the experiments (although in some kinds of experiments that are actually
intended, we will not consider such cases). In this work, we do not provide a
user interface, which we plan to design and build in future work.

The methodology that researchers should follow to create a reproducibility
package using our platform can be formulated in the following chronological
order:

  1. 1.

Researchers create a new project, choose the type of the project and fill in
some fields (e.g., project name, description, etc.). For now, we only support
the execution of scripts with a database integrated and AI experiments.
However, the latter is necessary to make researchers aware of the
reproducibility of the experiment, so we force researchers to associate the
seeds used in the code with the project (Step 1 in fig. 1);

  2. 2.

They upload the experiment files (e.g., code and datasets) or reuse a pre-
existent project by providing its remote location. This functionality enables
the uploading of individual or zip files, which can contain all the needed
files of the experiment (Steps 2 and 3 in fig. 1).

  3. 3.

They can manage the files within the platform (e.g., create some missing
files/folders, edit and delete some unnecessary files/folders loaded by
mistake or no longer needed) (Step 4 in fig. 1);

  4. 4.

They configure the execution of the project and associate the dataset of that
execution (e.g., choose a specific version of the PL and configure the
databases) (Step 5 in fig. 1); This functionality can infer important
information such as the PLs, dependencies and their version, however, the
researcher must validate that information. In addition, it allows the
configuration of AI experiments. These experiments cannot be reproducible if
researchers do not define the seeds used. To try to solve this problem, we are
making researchers aware that they need to define the seeds used in the
project. Furthermore, the researcher can provide more information (e.g.,
commands to build the project, and the configurations of possible databases)
to generate a container specification and build a container image of the
experiment. In addition, researchers need to associate a dataset to each
experiment, allowing them to execute the experiment using a different
associated dataset. In this way, it is possible to verify the computational
reproducibility and replicability. After the platform concludes this step, our
approach will execute the experiment 2 times to verify the computational
reproducibility (Step 5 in fig. 1);

  5. 5.

Researchers execute the experiment using the dataset associated or a different
dataset. This functionality enables the reusing of container images, generated
in the previous step, to start a new execution. In addition, it allows adding
a pre-existing container image to add a database to the run. All the
executions which use the same container image reuse the same computational
environment to execute the experiment, however, if needed, the user can use
the same computational environment but using a different dataset. Furthermore,
it is possible to specify which commands one would like to run (Step 6 in fig.
1);

  6. 6.

Researchers get and visualize the results (Step 7 in fig. 1);

  7. 7.

Validate the computational reproducibility or the computational
replicability.Researchers can define the location of the output (e.g., console
output, a file, etc.), however, if it is not defined, we will compare all
files changed and the console output (Step 8 in fig. 1);

  8. 8.

Researchers build a reproducible package of the experiment. It allows building
a reproducible package, which contains all the information needed to execute
the experiment, that can be run on Windows, Mac OS or Unix operating systems.
It reuses the container image previously generated and adds all the necessary
information, including the commands executed when running the experiment (Step
9 in fig. 1).

fig. 1 presents an activity diagram formalizing this process.

Figure 1: The activity diagram of a researcher using our approach

###  4.1 Architecture

Our platform has three main components, namely “Data Management”,
“Computational Environment”, and “Code Execution”. Furthermore, we have a
database, which saves some required information. fig. 2 represents the
components’ diagram of this approach. In the following, we explain in detail
each one.

Figure 2: The components’ diagram of our platform

####  4.1.1 Data Management

This component is responsible for managing all the data of the platform, such
as adding, editing and deleting files and folders. The researcher can use this
component to upload all the necessary files on a project (e.g., code,
datasets, etc.), and the Data Management Component saves on the database all
the information of each file/folder.

This component complies with the principle P5 from section 2.

####  4.1.2 Computational Environment

The function of this component is to build a computing environment able to
deal with the diversity of development stacks (e.g., a variety of PLs,
dependencies and, if needed, databases integrated) used by researchers and
make that information available to the Code Execution Component (section
4.1.3) which executes the experiment.

The diversity of available computing environments is a challenge in general
software development and is being tackled by using containers [13, 49], a
technology that allows automatically building an environment based on a
specification. A specification file for a container, also known as a container
specification, is a document, like a template, that outlines the configuration
and requirements of a containerized application or service. It describes the
resources, environment, dependencies, and other necessary information for the
container to run consistently across different environments [13, 49]. One of
the most used containerization technologies is Docker111111www.docker.com
[49]. For instance, one may define a specification of an environment with
Python 3, PostgreSQL 15.1 and Apache 2.4. By using the same specification,
each member of a team works in the exact same environment as all the others.
We build on this same principle to allow researchers to reproduce their
experiments in a systematic way as the Computational Environment component is
based on containerization.

This component complies with the P1, P2, P6 and P7 principles from section 2.

####  4.1.3 Code Execution

This component is responsible for the execution of the experiment
code/scripts, validating the reproducibility or replicability of results and
building a reproducible package with all the information needed to re-execute
the experience in the same way.

This component complies with the P3, P4, P9 and P10 principles from section 2.

###  4.2 Implementation

In this section, we describe some relevant details of our implementation. We
implement our approach in Python121212https://www.python.org/ and
Node.js131313https://nodejs.org/en/ and use a Neo4j141414https://neo4j.com/
NoSql database, as well as Docker Hub.

The Data Management component is implemented in Node.js. We are using the
Express151515https://expressjs.com/ framework for communication. The
information of all the files and folders of an experiment are saved on a Neo4j
database.

The package responsible for the creation, configuration, execution and
packaging of experiments is implemented in Python. We are using the
Flask161616https://flask.palletsprojects.com/en/2.2.x/ framework for
communication and we are also using Docker Hub as the containerization
technology.

Our approach can build a variety of computing environments needed for
researchers to execute their experiments. Our approach already supports a
variety of PLs, namely C++, Perl, R, Java (maven), Python, Unix Shell and
Jupyter Notebook. We have implemented a heuristics-based algorithm that allows
the platform to infer the PLs of the project through file extensions.
Moreover, our approach can also infer the dependencies to be installed, by
constructing the project dependency graph.

The platform infers all the information needed to build the container
specification, as the technology is Docker, this file is called dockerfile.

The platform performs all the steps of fig. 1, including building a
reproducible package of the experiment. However, the execution of this
reproducible package, which contains all the information needed to execute the
experiment, is on the researcher’s Windows, Mac Os or Unix operating system.
The only requirement is to have Docker installed and running. This package can
then simply be clicked and the experiment is executed.

It is easy to add another PL. For that to be possible, it is only necessary to
do some updates to get a docker image of the wanted language from Docker
Hub171717Docker Hub is the world’s largest container image repository
containing open source projects that are made available in containers
(https://hub.docker.com/). and build a dockerfile with that information.
However, we do not expect this to be done by researchers, but instead by a
maintenance team of the tool.

Our approach supports databases, and it allows researchers to configure their
setup. The databases currently supported are Mongo, SQLite MySql and
PostgreSQL. As with PLs, it is easy to support another database technology.

##  5 Evaluation

We build an evaluation experiment to measure the capabilities of the platform.
The main goal of this experiment is to reproduce a variety of experiments,
from a simple script to experiments with integrated databases. We also
considered experiments from different scientific fields. For that to be
possible, we need to recreate the same environment using the same frameworks,
code, data sources, the same PLs, dependencies, and the same version of the
dependencies. After that, we execute the experiment and reproduce the results
claims on the publication.

###  5.1 Setup

We started by collecting experiments already published in scientific
conferences. To do so, we used two different computer science research
communities: software engineering through the conference IEEE/ACM
International Conference on Software Engineering (ICSE), and databases through
the conference International Conference on Very Large Databases (VLDB), three
very well-known scientific events in their respective area. We analysed the
first 30 articles of VLDB 2021181818http://vldb.org/pvldb/volumes/15/ and the
first 50 articles of ICSE
2022191919https://ieeexplore.ieee.org/xpl/conhome/9793835/proceeding?isnumber=9793541.
For each publication, we collected the link where the code of the publication
is available. However, many of the publications do not have this information:
for ICSE, only 32 papers have a link and for VDLB, only 19. From the ones with
code, we selected five publications by randomly generating five numbers for
each conference, each number representing one publication. For ICSE, we
generated five numbers between 1 and 32 and achieved [1,9,12,13,23]; for VDLB,
we generated five numbers between 1 and 19 and achieved [5,7,11,15,17].

Moreover, we tried to find computational experiments in other areas through
the Climate Change202020https://www.springer.com/journal/10584/ and Nature
Climate Change212121https://www.nature.com/nclimate/ journals, but the papers
would not include information about the experiment setup, without which we
could not reproduce the experiment. Although this was not an extensive study,
it is yet another evidence that reproducibility is not yet fully tackled by
many researchers. Regarding the query of medical, artificial intelligence and
climate experiments, we only considered the first 100 results of each one.
Once again, we randomly generated five numbers between 1 and 100 two times,
five representing the medical repositories, five representing the artificial
intelligence repositories and five representing the climate change
repositories. We achieved the numbers [25, 38, 44, 58, 69] for medical
repositories, the numbers [10, 24, 41, 64, 99] for artificial intelligence
repositories and the numbers [14, 21, 31, 59, 78] for climate change
repositories.

Furthermore, we have widened the range of experiences, and to do this, we have
made two searches in the Zenodo repository. We searched by
“Medical”222222https://zenodo.org/search?page=4&size=20&q=medical&type=software,
“Artificial
Intelligence”232323https://zenodo.org/search?page=1&size=20&q=Artificial%20Intelligence&type=software
and “Climate
Change”242424https://zenodo.org/search?page=1&size=20&q=climate%20change&type=software
expressions, and we only considered software repositories.

We chose computer science conferences and medical, artificial intelligence and
climate change experiments in the expectation that these papers would pose
more complex challenges and a variety of experiences from various domains.

In addition, we performed all the experiments available in all the papers
related to our approach (see section 8).

We then proceeded to collect all the information relevant to reproduce the
experiment, including the PL and version used, dependencies necessary to be
installed, if a database was necessary, and which commands were executed to
get the expected results. In table 1, we present the relevant information
related to the set of experiments chosen from VLDB (experiments E1–E5), from
ICSE (E6–E10), for climate change (E11–E15), for medical (E15–E20), for
artificial intelligence (E21–E25) and from related work (E26–E31) (the entries
in Repository link are clickable in the PDF version of the paper and lead to
the repository available in the paper). Furthermore, we added a column called
“Reproduced” in each table, which represents if we achieved reproducibility
using our approach (more details in section 6).

Table 1: Sample of reproducible experiments (NA stands for Not Applicable) Publication |  PLs |  Database |  Repository link |  Dependencies description |  How to execute |  Repro- duced |  Identifier
---|---|---|---|---|---|---|---
Adnan et al. (2022) |  Unix Shell/Python |  ✕ |  Link |  ✓ |  ✓ |  ✓ |  E1
Bai and Zhao (2022) |  Python |  ✕ |  Link |  ✕ |  ✓ |  ✓ |  E2
Chauhan et al. (2022) |  C++/Python/ Unix Shell |  ✕ |  Link |  ✕ |  ✓ |  ✓ |  E3
Sun et al. (2022) |  Python |  PostgreSQL |  Link |  ✕ |  ✓ |  ✓ |  E4
Zhou et al. (2022) |  C++ |  ✕ |  Link |  ✓ |  ✓ |  ✓ |  E5
Chen et al. (2022) |  Python |  ✕ |  Link |  ✓ |  ✓ |  ✓ |  E6
Kukucka et al. (2022) |  Java/Python/ Unix Shell |  ✕ |  Link |  ✓ |  ✓ |  ✓ |  E7
Nguyen and Grunske (2022) |  Python/Java |  ✕ |  Link |  ✓ |  ✓ |  ✕ |  E8
Xie et al. (2022) |  Python |  ✕ |  Link |  ✓ |  ✓ |  ✓ |  E9
Zhang et al. (2022) |  Java |  MySQL |  Link |  ✓ |  ✕ |  ✕ |  E10
Edwards (2022) |  R |  ✕ |  Link |  ✓ |  Package |  NA |  E11
Hemes (2023) |  R |  ✕ |  Link |  ✓ |  ✕ |  ✓ |  E12
Jehn (2021) |  Python |  ✕ |  Link |  ✕ |  ✕ |  ✓ |  E13
Lin (2023) |  Python |  ✕ |  Link |  ✕ |  ✓ |  ✓ |  E14
Qasmi and Ribes (2021) |  R |  ✕ |  Link |  ✓ |  Package |  NA |  E15
Fries (2021) |  Python |  ✕ |  Link |  ✕ |  ✓ |  ✓ |  E16
Han (2021) |  Python |  ✕ |  Link |  ✓ |  ✓ |  ✕ |  E17
Hoyt (2018) |  Python |  ✕ |  Link |  ✓ |  ✓ |  ✓ |  E18
Kailas et al. (2022) |  Python/Unix Shell |  ✕ |  Link |  ✓ |  ✓ |  ✓ |  E19
Trueto (2021) |  Python |  ✕ |  Link |  ✓ |  ✓ |  ✓ |  E20
Prezja (2023) |  Python |  ✕ |  Link |  ✓ |  Package |  NA |  E21
Santucci (2023) |  Python |  ✕ |  Link |  ✕ |  ✕ |  ✓ |  E22
Erickson (2019) |  Python/Unix Shell/ C++ |  ✕ |  Link |  |  ✓ |  ✕ |  E23
Coupette et al. (2022) |  Python /Jupyter Notebook |  ✕ |  Link |  ✓ |  ✕ |  ✓ |  E24
Pala and García-Cuesta (2023) |  Python |  ✕ |  Link |  ✓ |  ✕ |  ✓ |  E25
Tovar et al. (2023) |  Python/ C/Unix Shell |  ✕ |  Link |  ✓ |  Package |  NA |  E26
Meng et al. (2015) |  – |  – |  No info |  – |  – |  NA |  E27
Pham et al. (2015) |  – |  – |  No info |  – |  – |  NA |  E28
Collins et al. (2023) |  R |  ✕ |  Link |  ✓ |  ✓ |  ✓ |  E29
Essawy (2023) |  Python/Unix Shell |  ✕ |  Link |  ✕ |  ✕ |  ✕ |  E30
Hai (2023) |  Python |  SQlite |  Link |  ✕ |  ✕ |  ✓ |  E31

Our own experiment environment is a personal computer with a 6-core CPU, and
16GB RAM, running Ubuntu 20.04.

###  5.2 Execution

Our approach is the backend of the platform, not providing a user interface,
so we need to use its API on this execution. For each of the projects, we
started by creating and populating with the corresponding code the projects
(section 5.2.1). We then build the necessary environment (section 5.2.2) and
execute the experiments (section 5.2.3). Finally, we create a reproducibility
package (section 5.2.4).

####  5.2.1 Create a project and upload its files

During this task, we used the “Data Management Component” described in section
4.1.1. The first step is to create a project and fill in some important
fields, such as the project name and the project description. The response is
the unique identifier of the project created. During the following tasks, we
will need the returned unique identifier of the project.

The second step is to upload files to the project. We can upload a zip file or
use a remote location (Git or DOI). At the end of this task, a new project is
created in the platform.

####  5.2.2 Build the environment for each experiment

The main goal of this task is to recreate the same computational environment
of an experience. During this task, we used the “Computational Environment
Component” described in section 4.1.2. We will exemplify the building process
of two environments for two distinct experiments(E3 and E8).

The order in which the experiments are performed is irrelevant, and we could
choose another experiment, but we consider that the first experiment we want
to reproduce is the E3, and the second one is the E8. E3 uses C++, Python,
Jupyter Notebook, and Unix Shell as the PLs. Furthermore, this experiment has
only one Jupyter Notebook file. Python and Unix Shell files were not used in
this execution, so we do not consider these PLs. The documentation of E3
indicates that to build the experiment, it is necessary to run the commands
`g++ -O3 ./src/bbfs\_node.cpp -o out` and `g++ -O3 ./src/bbfs\_edge.cpp -o
out`. For this task, we make use of the “Build a docker image” functionality
of the “Computational Environment Component” (section 4.1.2). Our approach
infers the PLs (C++) and inserts the commands to build the project. Listing
LABEL:lst:requestData1 represents the request data after the user fills in all
the information.

⬇

1{"languages": ["C++"],

2"languagesVersion":{"C++":"gcc:8"},

3"commandsToAdd":[

4 "g++ -O3 ./src/bbfs_node.cpp -o out",

5 "g++ -O3 /src/bbfs_edge.cpp -o out"]}

Listing 1: Request data of E3

After receiving this request, the server generates a dockerfile, builds a
docker image and returns an identifier (ID) that represents that docker image.
In Listing LABEL:lst:dockerfile4, we can see the dockerfile generated for the
E3. The directory “files” contain all the files of the experiment.

⬇

1FROM ubuntu:20.04

2RUN apt update && apt upgrade -y

3RUN apt install -y gcc-8 make g++

4RUN update-alternatives \--install /usr/bin/gcc gcc /usr/bin/gcc-8 2000

5WORKDIR /files

6COPY ./files .

7RUN g++ -O3 ./src/bbfs_node.cpp -o out

8RUN g++ -O3 ./src/bbfs_edge.cpp -o out

Listing 2: Dockerfile of E3

E8 is a more complex experiment than E3 as it needs three PLs to execute
(Python, Java and Unix Shell). The authors of the experiment mentioned that
the prerequisites are “Java 8-11, Apache Maven and Python 3”. Furthermore, all
the Python dependencies that are necessary to install are in a file in the
repository. To conclude, it is necessary to execute the following commands `cd
RLCheck/jqf/`, `mvn package`, and `cd ../..`. Our approach needs to replace
`cd` command to `WORKDIR` in the generations of the dockerfile because the
`cd` command has no effect after the execution of that command, and `WORKDIR`
command changes the working directory for future commands. Thus, the request
type to generate a dockerfile is the same as E3, but the content of the
request data is different and is represented in Listing
LABEL:lst:requestData2.

⬇

1{"languages": ["python", "shell", "java"],

2"languagesVersion":{"java":"openjdk-11", "python":"python:3.8"},

3"commandsToAdd":["cd RLCheck/jqf/","mvn package","cd ../.."],

4"hasRequirementsFile": "true"}

Listing 3: Request data of E8

In Listing LABEL:lst:dockerfile8, we can see the dockerfile generated for E8.

⬇

1FROM ubuntu:20.04

2RUN apt update && apt upgrade -y

3RUN apt install -y python3.8 python3-pip

4RUN update-alternatives \--install /usr/local/bin/python python
/usr/bin/python3.8 2000

5RUN update-alternatives \--install /usr/bin/pip pip /usr/bin/pip3 2000

6RUN apt install -y openjdk-11-jdk openjdk-11-jre maven

7WORKDIR /files

8COPY ./files .

9RUN mvn package

10WORKDIR RLCheck/jqf/

11RUN mvn package

12WORKDIR ../..

13RUN pip install -r requirements.txt

Listing 4: Dockerfile of E8

####  5.2.3 Run the experiments

After we have a docker image created, we only need to execute the experiment.
For that to be possible, we need to insert the Id of the created docker image
in the previous task and which command we intend to run. To do so, we make use
of the “run an experiment” functionality of the “Code Execution Component”
(section 4.1.3).

Listing LABEL:lst:requestDataRunExperiment presents the request data to
execute E3.

⬇

1{"command": "./out freebase/edges.txt freebase/labels.txt 1234 5678 4 0.3
2000 0.8 1 10 0.01 2",

2"tagId": 100}

Listing 5: Request data for the execution of E3

####  5.2.4 Build the reproducible package

Finally, we created a reproducibility package for each of the experiments,
which we make publicly available252525https://zenodo.org/record/8050716.

We reuse the container image generated in the building of the computational
environment task and reuse all the information provided on the execution of
the experiment (commands) to encapsulate all that information in a folder
along with the scripts needed to run the project. We built a reproducible
package that can be run on Windows or Unix operating systems where the user
just needs to have Docker installed and running. By clicking in
“runExperiment” file, the experiment is executed automatically.

##  6 Discussion

We chose 31 different experiments from published work in two research
communities (ICSE and VLDB), in three research areas (Medical, Artificial
Intelligence and Climate Change) and experiments executed in related work
approaches.

Although 31 may not seem a significant number, making each one reproducible
was quite challenging due to the diversity of experiments considered,
encompassing different domains, and the variety of computational procedures
involved. During the configuration of the computational environment, we needed
to infer the information from the repository (e.g., PLs and dependencies used
) to try to fill in the parameters that were necessary to insert to build the
same computational environment. This information was only sometimes clear. Our
approach can infer the Python and R dependencies needed when the requirements
file is missing. However, deducing the dependencies for other PLs is very
difficult, so that information should be part of the project description.
Although our backend already includes such a feature, we must design an easy
way for researchers to have that information in the frontend.

We analyzed the articles on all the related work approaches. However, most
methods do not have a validation section (Open Lab [53], Encapsulator [47],
VisTrails [24], RenkuLab [37], Whole Tale [4], ReproZip [10, 12, 11],
RunMyCode [54] and Binder [7]). It was thus not possible to compare our
approach with those. An exception is CARE [33]. The authors selected seven
significant applications (e.g., VLC, Firefox). However, they did not reproduce
experiments but rather executed commands. Thus, there is no experiment to be
reproduced. Nevertheless, our platform supports the PLs of that commands.

The authors of E26 and E27 presented two case studies of complex, high-energy
physics applications and showed how their framework could be useful for
sandboxing, preserving, and distributing applications [43]. Although the
authors make available the case study link
262626https://curate.nd.edu/downloads/5712m615h8d, only one case study is
actually available. Nerveless, it is impossible to reproduce the experiment
because they only distribute a software package for enabling large-scale
distributed computing, and there is no expected result, so we cannot compare
our approach with this.

CDE-SP’s authors described a pipeline, identified by E28, they conducted to
determine the overall performance of CDE-SP [48]. However, they do not make
available the experiment code, so we cannot reproduce the same experiment.

SciInc’s authors selected three use cases for their evaluation (E29 [15], E30
[23], E31 [27]), and it is possible to access the repository [62]. We can
install and execute E29 and E31, but it is not possible to verify the
computational reproducibility because there is no information about the
commands executed and the expected results. E30 does not have information
about the experiment, but we tried to execute the “Main_Shell_Script.scr” file
in the root of the project. However, it was impossible to reproduce that file
since it tries to access information inside the `cmd` folder, but there is no
such folder in the repository, so not all the files from this repository have
been published.

The other three experiments we could not reproduce (E8, E10, E17 and E23) have
different causes. For E8, the problem was due to not finding a Java class
during the execution, E10 does not exchange information between the database
and the functions of the experiment, for E17, the necessary dependencies have
been installed, but the data needed to run the experiment is not available. We
asked permission to access the
CheXpert272727https://stanfordmlgroup.github.io/competitions/chexpert/ portal;
however, it was impossible to find the files inside the portal. For E23, there
is no information about how to install the project, so we tried to install the
project using ”python setup.py install“ command, however, during the
installation, some files were missing.

Nevertheless, from the 31 chosen experiments, four of them are packages, and
two do not have available the source code, so only 25 experiments are
available to be reproduced. We were able to successfully install the required
dependencies, execute, and create a reproducibility package for 20 of 25
(80%). Note that we were not even able to run E8, E10, E17, E23 and E30 on our
computer (without the platform), and thus we feel confident stating that this
was not an issue of our platform.

The execution of the experiments went smoothly. However, although the
repositories included the command lines to execute the code, they did not have
the expected results, so the verification of the reproducibility became more
difficult, but we managed to compare our results with the claims of
developments in the paper.

We are confident the challenges described in section 2 (C1, C2, C3) are well
addressed by our approach.

Software (C1) is the biggest problem when addressing reproducibility. Our
proposal allows users to build a complete container of the project, which will
make it available for reuse by other researchers. In this way, it is possible
to execute the same computational environment, even when some dependencies are
not publicly available anymore. This addresses principles P1 and P2.

Command (C2) implemented in our approach allows researchers to configure the
execution of experiments and execute the experiment with the same parameters.
The results achieved by previous executions can be used as a means of
comparison to validate the computational reproducibility. Principles P3 and P4
are tackled in this part of the work.

Data (C3) is handled in the platform by providing a mechanism to allow
researchers to manage the data of the experiment. This was also our principle
P5.

Our approach is able to deal with a diversity of PLs (and their dependencies)
used by researchers. Furthermore, it supports a variety of experiments, from
the execution of simple scripts to the execution of more complex experiments
(e.g., involving databases or several PLs). Thus, we can say principles P6 and
P7 are also addressed by our proposal.

Moreover, the reproducibility package that can be obtained from the platform
can be easily shared as it corresponds to a compressed file with just two
files. By executing one, with a simple double-click, the entire experiment is
executed. The only requirement is that docker, which is available both for
Unix and Windows systems, is installed and running. Thus, we have reduced the
problem of re-execution to having to install a single software tool, the most
used for container management. Thus, principles P9 and P10 are also addressed.

Our approach is the backend of a platform, not providing a user interface.
Thus, principle P8, which we consider a very important one, is not yet
supported by our work. Nevertheless, we plan to design and build such an
interface in future work [19].

##  7 Threats to Validity

The validation process has some threats divided into 4 sections [16, 60]
addressed next.

### Conclusion validity

##### Random heterogeneity of subjects

We tried to collect a variety of experiments, including experiments with only
one PL and experiments with more than one, and with and without the use of
databases. Moreover, we have selected experiments from different scientific
fields, including climate change and medicine. Thus, our subjects are
relatively heterogeneous and represent a sample of published scientific
experiments.

### Internal validity

##### Instrumentation.

The computational environment is only briefly described by the authors of the
experiments we used, so it is more difficult to try to infer what dependencies
and which versions were used in the execution of the original experiment.
Thus, there is the risk that we have executed the experiments in slightly
different conditions. Nevertheless, we were able to validate the results we
obtained with the ones in the papers, and thus this risk is quite low.

### Construct validity

##### Restricted generalizability across constructs.

Our approach improves computational reproducibility, however, for some
experiments it is necessary to have more storage and computational power.
Thus, we were not able to validate our approach with such demanding
requirements. Nevertheless, most of the experiments we analyzed during this
work did not have such restrictions and thus could potentially be executed
within our platform.

### External validity

##### Interaction of selection and treatment.

We collected a set of experiments from different computer science areas,
however, we do not have experiments from domains other than computer science.
Nevertheless, from experience, we know that at least in some areas it is not
common to report the computational setting and thus it was not possible to
find experiments from other domains. Moreover, the experiments we used are
quite complex (e.g., in terms of (the number of) PLs used, databases, etc.)
and thus we are confident that our platform can cope with experiments from
other domains.

##  8 Related Work

There have been some tools proposed to aid researchers in making their
computational experiments reproducible. We discuss them now based on their
capabilities.

###  8.1 Supported PLs

Researchers perform experiments using a variety of PLs, so it is essential
that the platform or the tool used can support such experiments. Some related
works such as Encapsulator [47], Binder [7], RenkuLab [37], Whole Tale [4],
Open Lab [53], VisTrails [24], RunMyCode [54] or the work by Haiyan [43]
support only a limited set of PLs and are not extendable, so it is impossible
to support new PLs. However, CDE-SP [48], CARE [33], SciInc [62], and ReproZip
[10, 12, 11] support an unlimited set of PLs.

###  8.2 Automatic installation of dependencies

We see in section 2, that one of the main challenges in computational
reproducibility is to install the required dependencies and library correctly.
In this way, it is crucial that the platform can infer and install all the
necessary dependencies and libraries to execute the code in the same way as
the owner of the experiment.

ReproZip [10, 12, 11], VisTrails [24], CDE-SP [48], CARE [33], SciInc [62],
and the work by Haiyan [43] can automatically track and identify all the
required dependencies, binary files, code, data, and files needed to execute
the code. Furthermore, Encapsulator [47] employs observed provenance capture
to gather detailed provenance information and expose the underlying mechanisms
of an analysis script. This process involves capturing fine-grained provenance
data.

Binder [7], RenkuLab [37] and Whole Tale [4] are limited approaches because
the researcher needs to install all the required dependencies in the platform.

###  8.3 Supported databases

In general, computational experiments do not have databases integrated.
However, complex experiments that involve databases are harder to reproduce,
because reproducible tools do not support these experiments. ReproZip [10, 12,
11] is the only approach that supports experiments that have integrated
databases.

###  8.4 Configure the execution of the experiments

Making available all the information to allow computational reproducibility is
essential to reproduce the same results; such information includes the values
of the parameters used in the execution of the experiment.

One way to easily make this information available is to be possible to share
the parameter spaces explored. Open Lab [53], VisTrails [24], Whole Tale [4],
ReproZip [10, 12, 11], RunMyCode [54] are approaches that make it possible
because the owner of the experiment can share the same parameter spaces
explored with other researchers and making the process of the configuration of
the experiment easier in the reproduction.

###  8.5 Reproducible package

Docker offers greater reliability and flexibility, as well as being
lightweight and capable of maintaining its own data and memory [3, 31]. By
utilizing container images, Docker enables faster start-up times and efficient
utilization of resources such as RAM and disk. This is due to the shared files
and layered file systems that are inherent to containerization. By using this
technique, it is possible to share a capsule with all the information needed
to execute the experiment in the same way without the need to use another
platform besides Docker.

Encapsulator [47], SciInc [62], ReproZip [10, 12, 11], CARE [33], VisTrails
[24], CDE-SP [48], and the work by Haiyan [43] are approaches that
automatically package everything required to execute the same experiment in
another machine without the use of a specific platform.

Binder [7], RenkuLab [37] and Whole Tale [4] create a Docker image of the
repository, but one needs the original platform to re-execute the code. Open
Lab [53], RunMyCode [54] are approaches that, such as the previous one, it is
necessary to re-execute the code on the platform, however, they do not use
virtualization.

###  8.6 Replicability of results

Replicability and reproducibility are two concepts that produce the same
result, but using different methods, code or data. In all the related work
approaches is not possible to mark or associate a dataset to each execution,
making the comprehension of the dataset used harder in future executions. In
this way, all the related work approaches do not validate the replicability of
results.

###  8.7 Validation of results

Validating the results stated in publications is hard work. Researchers have
to run the experiment, read the results section, and compare the results
obtained with the results stated in the publication. It was advantageous for
the researchers to enter what the expected result was in the run and in each
run this comparison was made in order to prove the reproducibility of results.
However, the approaches in the related papers do not support this
functionality.

###  8.8 Supported operating system

Researchers build their experiments in their favorite Operating Systems (OS),
so it should not be a limitation when we intend to reproduce an experiment.
Binder [7], RenkuLab [37], Whole Tale [4], Open Lab [53], VisTrails [24],
RunMyCode [54] and Encapsulator [47] are approaches that support all the
Operating Systems, so it is not a limitation. However, Encapsulator [47],
SciInc [62], CARE [33], ReproZip [10, 12, 11] and the work by Meng et al.
(2015) are approaches limited to the Linux environment. This might prove
problematic for many scientists who do not use Linux. Furthermore, CDE-SP [48]
only supports Linux and Mac OS environments.

We conclude that the main platforms and approaches for sharing and computing
research data are focused on specific domains and thus they are not able to
reproduce complex experiments (using several environments, databases, etc.) or
are not user-friendly platforms. In table 2 and table 3 we synthesize the
characteristics of the methods and tools that we have just described.

Table 2: Comparison of characteristics of reproducibility tools with Graphical User Interface (GUI) |  Binder |  Renku-Lab |  Whole Tale |  Repro- Zip |  Open Lab Me |  Vis- Trails |  RunMy- Code | Our
---|---|---|---|---|---|---|---|---
Unlimited PLs |  ✕ |  ✕ |  ✕ |  ✓ |  ✕ |  ✕ |  ✕ | ✓
Researcher-friendly platform |  ✓ |  ✕ |  ✓ |  ✕ |  ✓ |  ✓ |  ✓ | ✕
Data management |  ✓ |  ✓ |  ✓ |  |  ✓ |  ✓ |  ✓ | ✓
Supports databases |  ✕ |  ✕ |  ✕ |  ✓ |  ✕ |  ✕ |  ✕ | ✓
Supports experiments from different fields |  |  |  |  |  |  |  | ✓
Automatically infers dependencies |  ✕ |  ✕ |  ✕ |  ✓ |  ✕ |  ✓ |  ✕ |
Automatically infers PLs |  ✕ |  ✕ |  ✕ |  ✓ |  ✕ |  ✕ |  ✕ |
Automatically installs necessary dependencies |  |  |  |  ✓ |  ✕ |  ✓ |  ✕ | ✓
Configures execution of experiments |  ✕ |  ✕ |  ✓ |  ✓ |  ✓ |  ✓ |  ✓ | ✓
Creation of reproducibility package |  ✕ |  ✕ |  ✕ |  ✓ |  ✕ |  ✓ |  ✕ | ✓
Reproducibility package executed anywhere |  ✕ |  ✕ |  ✕ |  ✓ |  ✕ |  ✓ |  ✕ | ✓
Validation of results |  ✕ |  ✕ |  ✕ |  ✕ |  ✕ |  ✕ |  ✕ | ✓
Supported operating system |  All |  All |  All |  Linux |  All |  All |  All | All
Table 3: Comparison of characteristics of reproducibility tools without GUI (only command line interface) | CDE-SP | Encapsulator | [43] | CARE | SciInc | Our
---|---|---|---|---|---|---
Unlimited PLs | ✓ | ✕ | ✕ | ✓ | ✓ | ✓
Data management | ✕ | ✕ | ✕ | ✕ | ✕ | ✓
Supports experiments from different fields |  |  |  |  |  | ✓
Supports databases | ✕ | ✕ | ✕ | ✕ | ✕ | ✓
Automatically infers dependencies | ✓ | ✓ | ✓ | ✓ | ✓ |
Automatically infers PLs | ✓ | ✕ | ✕ | ✓ | ✓ |
Automatically installs dependencies | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
Configures execution of experiments | ✕ | ✕ | ✕ | ✕ | ✕ | ✓
Creation of reproducibility package | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
Reproducibility package executed anywhere | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
Validation of results | ✕ | ✕ | ✕ | ✕ | ✕ | ✓
Supported operating system | Linux/Mac OS | All | Linux | Linux | Linux | All

##  9 Conclusion

Our approach is designed to support researchers in ensuring their research is
reproducible and transparent during their study and in preparation for
publication. One of the biggest obstacles to achieve this is creating the same
computational environment, configuring it correctly, reproducing the
experiment, and verifying its computational reproducibility.

We propose a solution to address these challenges and assist researchers in
developing the experiment and preparing a reproducibility package to accompany
their paper. This will enable reviewers and others to rerun the experiments.
Our evaluation of the approach revealed that we successfully recreated and
reproduced 20 out of 25 experiments, demonstrating that our methodology and
backend platform can easily recreate, reproduce, and share experiments.

Currently, we are developing a frontend interface to enable researchers to
create reproducible packages with ease [19]. Moreover, we plan to further
study how to make the sharing of data among experiments easier for
researchers, but also for the community in general [18].

## References

  * Adnan et al. [2022] Muhammad Adnan, Yassaman Ebrahimzadeh Maboud, Divya Mahajan, and Prashant J. Nair.  Accelerating recommendation system training by leveraging popular choices.  _Proc. VLDB Endow._ , 15(1):127–140, jan 2022\.  ISSN 2150-8097.  doi: 10.14778/3485450.3485462.  URL https://doi.org/10.14778/3485450.3485462.
  * Bai and Zhao [2022] Jiyang Bai and Peixiang Zhao.  Tagsim: Type-aware graph similarity learning and computation.  _Proc. VLDB Endow._ , 15(2):335–347, feb 2022\.  ISSN 2150-8097.  doi: 10.14778/3489496.3489513.  URL https://doi.org/10.14778/3489496.3489513.
  * Boettiger [2015] Carl Boettiger.  An introduction to docker for reproducible research.  _SIGOPS Oper. Syst. Rev._ , 49(1):71–79, jan 2015.  ISSN 0163-5980.  doi: 10.1145/2723872.2723882.  URL https://doi.org/10.1145/2723872.2723882.
  * Brinckman et al. [2019] Adam Brinckman, Kyle Chard, Niall Gaffney, Mihael Hategan, Matthew B. Jones, Kacper Kowalik, Sivakumar Kulasekaran, Bertram Ludäscher, Bryce D. Mecum, Jarek Nabrzyski, Victoria Stodden, Ian J. Taylor, Matthew J. Turk, and Kandace Turner.  Computing environments for reproducibility: Capturing the “whole tale”.  _Future Generation Computer Systems_ , 94:854–867, 2019\.  ISSN 0167-739X.  doi: 10.1016/j.future.2017.12.029.  URL https://doi.org/10.1016/j.future.2017.12.029.
  * Brunsdon and Comber [2021] Chris Brunsdon and Alexis Comber.  Opening practice: supporting reproducibility and critical spatial data science.  _Journal of Geographical Systems_ , 23(4):477–496, oct 2021.  ISSN 14355949.  doi: 10.1007/s10109-020-00334-2.
  * Bush et al. [2020] Rosemary Bush, Andrea Dutton, Michael Evans, Rich Loft, and Gavin A. Schmidt.  Perspectives on Data Reproducibility and Replicability in Paleoclimate and Climate Science.  _Harvard Data Science Review_ , 2(4), dec 16 2020.  doi: 10.1162/99608f92.00cd8f85.  https://hdsr.mitpress.mit.edu/pub/dijwtzza.
  * Bussonnier et al. [2018] Matthias Bussonnier, Jessica Forde, Jeremy Freeman, Brian Granger, Tim Head, Chris Holdgraf, Kyle Kelley, Gladys Nalvarte, Andrew Osheroff, M Pacer, Yuvi Panda, Fernando Perez, Benjamin Ragan Kelley, and Carol Willing.  Binder 2.0 - Reproducible, interactive, sharable environments for science at scale.  In Fatih Akici, David Lippa, Dillon Niederhut, and M Pacer, editors, _Proceedings of the 17th Python in Science Conference_ , pages 113–120, Austin, Texas, 2018. SciPy.  doi: 10.25080/Majora-4af1f417-011.  URL https://doi.org/10.25080/Majora-4af1f417-011.
  * Chauhan et al. [2022] Komal Chauhan, Kartik Jain, Sayan Ranu, Srikanta Bedathur, and Amitabha Bagchi.  Answering regular path queries through exemplars, feb 2022.  ISSN 2150-8097.  URL https://doi.org/10.14778/3489496.3489510.
  * Chen et al. [2022] Zhuangbin Chen, Jinyang Liu, Yuxin Su, Hongyu Zhang, Xiao Ling, Yongqiang Yang, and Michael R. Lyu.  Adaptive performance anomaly detection for online service systems via pattern sketching.  In _Proceedings of the 44th International Conference on Software Engineering_ , ICSE ’22, page 61–72, New York, NY, USA, 2022. Association for Computing Machinery.  ISBN 9781450392211.  doi: 10.1145/3510003.3510085.  URL https://doi.org/10.1145/3510003.3510085.
  * Chirigati et al. [2013] Fernando Chirigati, Dennis Shasha, and Juliana Freire.  Packing experiments for sharing and publication.  In _Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data_ , SIGMOD ’13, page 977–980, New York, NY, USA, 2013. Association for Computing Machinery.  ISBN 9781450320375.  doi: 10.1145/2463676.2465269.  URL https://doi.org/10.1145/2463676.2465269.
  * Chirigati et al. [2016a] Fernando Chirigati, Rebecca Capone, Rémi Rampin, Juliana Freire, and Dennis Shasha.  A collaborative approach to computational reproducibility.  _Inf. Syst._ , 59(C):95–97, jul 2016a.  ISSN 0306-4379.  doi: 10.1016/j.is.2016.03.002.  URL https://doi.org/10.1016/j.is.2016.03.002.
  * Chirigati et al. [2016b] Fernando Chirigati, Rémi Rampin, Dennis Shasha, and Juliana Freire.  Reprozip: Computational reproducibility with ease.  In _Proceedings of the 2016 International Conference on Management of Data_ , SIGMOD ’16, page 2085–2088, New York, NY, USA, 2016b. Association for Computing Machinery.  ISBN 9781450335317.  doi: 10.1145/2882903.2899401.  URL https://doi.org/10.1145/2882903.2899401.
  * Clyburne-Sherin et al. [2019] April Clyburne-Sherin, Xu Fei, and Seth Green.  Computational reproducibility via containers in psychology.  _Meta-Psychology_ , 3, 11 2019.  doi: 10.15626/MP.2018.892.
  * Cohen-Boulakia et al. [2017] Sarah Cohen-Boulakia, Khalid Belhajjame, Olivier Collin, Jérôme Chopard, Christine Froidevaux, Alban Gaignard, Konrad Hinsen, Pierre Larmande, Yvan Le Bras, Frédéric Lemoine, Fabien Mareuil, Hervé Ménager, Christophe Pradal, and Christophe Blanchet.  Scientific workflows for computational reproducibility in the life sciences: Status, challenges and opportunities.  _Future Generation Computer Systems_ , 75:284–298, 2017\.  ISSN 0167-739X.  doi: 10.1016/j.future.2017.01.012.  URL https://doi.org/10.1016/j.future.2017.01.012.
  * Collins et al. [2023] Stephen Collins, Gavin Smart, Ben Albright, and David Crippin.  Food inspections evaluation.  urlhttps://chicago.github.io/food-inspections-evaluation/, May 2023\.
  * Cook and Campbell [1979] Thomas D Cook and D T Campbell.  _Quasi-Experimentation: Design and Analysis Issues for Field Settings_.  Houghton Mifflin, Boston, 1979.
  * Costa [2022a] Lázaro Costa.  A platform for the reproducibility of computational experiments.  In _2022 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)_ , pages 1–3, Rome, Italy, 2022a. IEEE.  doi: 10.1109/VL/HCC53370.2022.9833132.
  * Costa and da Silva [2019] Lázaro Costa and João Rocha da Silva.  Dendro: A fair, open-source data sharing platform.  In Antoine Doucet, Antoine Isaac, Koraljka Golub, Trond Aalberg, and Adam Jatowt, editors, _Digital Libraries for Open Knowledge_ , pages 384–387, Cham, 2019. Springer International Publishing.  ISBN 978-3-030-30760-8.  doi: 10.1007/978-3-030-30760-8˙39.  URL https://doi.org/10.1007/978-3-030-30760-8_39.
  * Costa [2022b] Lázaro Costa.  A platform for the reproducibility of computational experiments.  In _2022 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)_ , pages 1–3, 2022b.  doi: 10.1109/VL/HCC53370.2022.9833132.
  * Coupette et al. [2022] Corinna Coupette, Dirk Hartung, Janis Beckedorf, Maximilian Böther, and Daniel Martin Katz.  Law smells (code), April 2022.  URL https://doi.org/10.5281/zenodo.6468193.
  * Edwards [2022] Brandon PM Edwards.  napops: Point Count Offsets for Population Sizes of North American Landbirds, February 2022.  URL https://doi.org/10.5281/zenodo.5967578.
  * Erickson [2019] Adam Erickson.  adam-erickson/deepland: Deepland alpha release, December 2019.  URL https://doi.org/10.5281/zenodo.3568468.
  * Essawy [2023] Bakinam Essawy.  Vic_pre-processing_rules.  urlhttps://github.com/uva-hydroinformatics/VIC_Pre-Processing_Rules, May 2023\.
  * Freire and Silva [2012] Juliana Freire and Claudio T. Silva.  Making computations and publications reproducible with vistrails.  _Computing in Science & Engineering_, 14(4):18–25, 2012.  doi: 10.1109/MCSE.2012.76.
  * Freire et al. [2012] Juliana Freire, Philippe Bonnet, and Dennis Shasha.  Computational reproducibility: State-of-the-art, challenges, and database research opportunities.  In _Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data_ , SIGMOD ’12, page 593–596, New York, NY, USA, 2012. Association for Computing Machinery.  ISBN 9781450312479.  doi: 10.1145/2213836.2213908.  URL https://doi.org/10.1145/2213836.2213908.
  * Fries [2021] Jason Alan Fries.  som-shahlab/trove: Manuscript pre-release code, February 2021.  URL https://doi.org/10.5281/zenodo.4497214.
  * Hai [2023] Ton Hai.  Iqe.  urlhttps://bitbucket.org/TonHai/iqe/src/master/, May 2023.
  * Han [2021] Tianyu Han.  peterhan91/Medical-Robust-Training: Adversarial Training on Medical Data, June 2021.  URL https://doi.org/10.5281/zenodo.4926118.
  * Hemes [2023] Kyle Hemes.  Data: The magnitude and pace of photosynthetic recovery after wildfire in California ecosystems, April 2023.  URL https://doi.org/10.5281/zenodo.7651416.  Funding provided by: California Strategic Growth Council’s Climate Change Research Program* Crossref Funder Registry ID: Award Number: CCR20021.
  * Hoyt [2018] Charles Tapley Hoyt.  bio2bel/mesh v0.2.0, November 2018.  URL https://doi.org/10.5281/zenodo.1488650.
  * Huang and Wu [2018] Dijiang Huang and Huijun Wu.  Chapter 2 - virtualization.  In Dijiang Huang and Huijun Wu, editors, _Mobile Cloud Computing_ , pages 31–64. Morgan Kaufmann, 2018.  ISBN 978-0-12-809641-3.  doi: https://doi.org/10.1016/B978-0-12-809641-3.00003-X.  URL https://www.sciencedirect.com/science/article/pii/B978012809641300003X.
  * Ivie and Thain [2018] Peter Ivie and Douglas Thain.  Reproducibility in scientific computing.  _ACM Comput. Surv._ , 51(3), jul 2018.  ISSN 0360-0300.  doi: 10.1145/3186266.  URL https://doi.org/10.1145/3186266.
  * Janin et al. [2014] Yves Janin, Cédric Vincent, and Rémi Duraffort.  Care, the comprehensive archiver for reproducible execution.  In _Proceedings of the 1st ACM SIGPLAN Workshop on Reproducible Research Methodologies and New Publication Models in Computer Engineering_ , TRUST ’14, New York, NY, USA, 2014. Association for Computing Machinery.  ISBN 9781450329514.  doi: 10.1145/2618137.2618138.  URL https://doi.org/10.1145/2618137.2618138.
  * Jehn [2021] Florian Ulrich Jehn.  zutn/extreme-climate-change: Publication ready, March 2021.  URL https://doi.org/10.5281/zenodo.4588383.
  * Kailas et al. [2022] Prajwal Kailas, Shinichi Goto, Max Homilius, Calum A MacRae, and Rahul C Deo.  obi-ml-public/ehr_deidentification, June 2022.  URL https://doi.org/10.5281/zenodo.6617957.
  * Kim and Nah [2018] Youngseek Kim and Seungahn Nah.  Internet researchers’ data sharing behaviors.  _Online Information Review_ , 42(1):124–142, Jan 2018.  ISSN 1468-4527.  doi: 10.1108/OIR-10-2016-0313.  URL https://doi.org/10.1108/OIR-10-2016-0313.
  * Krieger et al. [2021] Louis Krieger, Remko Nijzink, Gitanjali Thakur, Chandrasekhar Ramakrishnan, Rok Roskar, and Stan Schymanski.  Repeatable and reproducible workflows using the renku open science platform.  In _EGU General Assembly 2021_ , pages EGU21–7655, Online, 19–30 Apr 2021, 2021.  doi: 10.5194/egusphere-egu21-7655.  URL https://doi.org/10.5194/egusphere-egu21-7655.
  * Kukucka et al. [2022] James Kukucka, Luís Pina, Paul Ammann, and Jonathan Bell.  Confetti: Amplifying concolic guidance for fuzzers.  In _2022 IEEE/ACM 44th International Conference on Software Engineering (ICSE)_ , pages 438–450, Pittsburgh, PA, USA, 2022. IEEE.  doi: 10.1145/3510003.3510628.
  * Langer et al. [2019] André Langer, Ellen Bilz, and Martin Gaedke.  Analysis of current RDM applications for the interdisciplinary publication of research data.  In Lucie-Aimée Kaffee, Kemele M. Endris, Maria-Esther Vidal, Marco Comerio, Mersedeh Sadeghi, David Chaves-Fraga, and Pieter Colpaert, editors, _Proceedings of the Workshop on Approaches for Making Data Interoperable (AMAR) Presented at the SEMANTiCS Conference 2019_ , volume 2447 of _CEUR Workshop Proceedings_ , Karlsruhe, Germany, September 2019\. CEUR-WS.org.  URL http://ceur-ws.org/Vol-2447/paper1.pdf.
  * Lin [2023] Shuqi Lin.  Shuqi-Lin/Dissolved-Oxygen-MLPrediction: Dissolved oxygen and hypolimnetic hypoxia predictions via machine learning models in various lakes, February 2023.  URL https://doi.org/10.5281/zenodo.7613549.
  * Liu and Salganik [2019] David M. Liu and Matthew J. Salganik.  Successes and struggles with computational reproducibility: Lessons from the fragile families challenge.  volume 5, page 2378023119849803, 2019.  doi: 10.1177/2378023119849803.  URL https://doi.org/10.1177/2378023119849803.
  * Liu et al. [2015] Ji Liu, Esther Pacitti, Patrick Valduriez, and Marta Mattoso.  A survey of data-intensive scientific workflow management.  _Journal of Grid Computing_ , 13(4):457–493, 2015\.  doi: 10.1007/s10723-015-9329-8.  URL https://doi.org/10.1007/s10723-015-9329-8.
  * Meng et al. [2015] Haiyan Meng, Rupa Kommineni, Quan Pham, Robert Gardner, Tanu Malik, and Douglas Thain.  An invariant framework for conducting reproducible computational science.  _Journal of Computational Science_ , 9:137–142, 2015.  ISSN 1877-7503.  doi: https://doi.org/10.1016/j.jocs.2015.04.012.  URL https://www.sciencedirect.com/science/article/pii/S1877750315000502.  Computational Science at the Gates of Nature.
  * Nguyen and Grunske [2022] Hoang Lam Nguyen and Lars Grunske.  Bedivfuzz: Integrating behavioral diversity into generator-based fuzzing.  In _2022 IEEE/ACM 44th International Conference on Software Engineering (ICSE)_ , pages 249–261, Pittsburgh, PA, USA, 2022. IEEE.  doi: 10.1145/3510003.3510182.
  * Nüst and Pebesma [2021] Daniel Nüst and Edzer Pebesma.  Practical Reproducibility in Geography and Geosciences.  _Annals of the American Association of Geographers_ , 111(5):1300–1310, 2021.  ISSN 24694460.  doi: 10.1080/24694452.2020.1806028.
  * Pala and García-Cuesta [2023] Riccardo Pala and Esteban García-Cuesta.  AXOM - Combination of Weak Learners eXplanations to Improve Random Forest eXplicability Robustness (Source Code), January 2023.  URL https://doi.org/10.5281/zenodo.7585845.  None.
  * Pasquier et al. [2018] Thomas Pasquier, Matthew K. Lau, Xueyuan Han, Elizabeth Fong, Barbara S. Lerner, Emery R. Boose, Mercè Crosas, Aaron M. Ellison, and Margo Seltzer.  Sharing and preserving computational analyses for posterity with encapsulator.  _Computing in Science & Engineering_, 20(4):111–124, 2018.  doi: 10.1109/MCSE.2018.042781334.
  * Pham et al. [2015] Quan Pham, Tanu Malik, and Ian Foster.  Auditing and maintaining provenance in software packages.  In Bertram Ludäscher and Beth Plale, editors, _Provenance and Annotation of Data and Processes_ , pages 97–109, Cham, 2015. Springer International Publishing.  ISBN 978-3-319-16462-5.  doi: https://doi.org/10.1007/978-3-319-16462-5˙8.
  * Potdar et al. [2020] Amit M Potdar, Narayan D G, Shivaraj Kengond, and Mohammed Moin Mulla.  Performance evaluation of docker container and virtual machine.  _Procedia Computer Science_ , 171:1419–1428, 2020.  ISSN 1877-0509.  doi: 10.1016/j.procs.2020.04.152.  URL https://doi.org/10.1016/j.procs.2020.04.152.  Third International Conference on Computing and Network Communications (CoCoNet’19).
  * Prezja [2023] Fabi Prezja.  Deep Fast Vision: Accelerated Deep Transfer Learning Vision Prototyping and Beyond, April 2023.  URL https://doi.org/10.5281/zenodo.7867100.
  * Qasmi and Ribes [2021] Saïd Qasmi and Aurélien Ribes.  Kriging for climate change package, August 2021.  URL https://doi.org/10.5281/zenodo.5233947.
  * Santucci [2023] Valentino Santucci.  valentinosantucci/hydrojump: Evoapps2023, February 2023.  URL https://doi.org/10.5281/zenodo.7595510.
  * Shevchenko [2022] Yury Shevchenko.  Open lab: A web application for running and sharing online experiments.  _Behavior Research Methods_ , 54(6):3118–3125, Dec 2022.  ISSN 1554-3528.  doi: 10.3758/s13428-021-01776-2.  URL https://doi.org/10.3758/s13428-021-01776-2.
  * Stodden et al. [2012] Victoria Stodden, Christophe Hurlin, and Christophe Pérignon.  Runmycode.org: A novel dissemination and collaboration platform for executing published computational results.  In _2012 IEEE 8th International Conference on E-Science_ , pages 1–8, Chicago, IL, USA, 2012. IEEE.  doi: 10.1109/eScience.2012.6404455.
  * Stodden et al. [2016] Victoria Stodden, Marcia McNutt, David H. Bailey, Ewa Deelman, Yolanda Gil, Brooks Hanson, Michael A. Heroux, John P.A. Ioannidis, and Michela Taufer.  Enhancing reproducibility for computational methods.  _Science_ , 354(6317):1240–1241, 2016.  doi: 10.1126/science.aah6168.  URL https://www.science.org/doi/abs/10.1126/science.aah6168.
  * Sun et al. [2022] Ji Sun, Jintao Zhang, Zhaoyan Sun, Guoliang Li, and Nan Tang.  Learned cardinality estimation: A design space exploration and a comparative evaluation, jan 2022.  ISSN 2150-8097.  URL https://doi.org/10.14778/3485450.3485459.
  * Tovar et al. [2023] Benjamin Tovar, Douglas Thain, Tim Shaffer, and Dinesh Rajan.  Cooperative computing tools (cctools).  urlhttps://github.com/cooperative-computing-lab/cctools, May 2023.
  * Trueto [2021] Trueto.  trueto/transformers_sklearn: transformers-sklearn V1.1, January 2021\.  URL https://doi.org/10.5281/zenodo.4453803.
  * van Hee [2002] Kees van Hee.  _Workflow Management: Models, Methods, and Systems_.  The MIT Press, 01 2002.  ISBN 9780262285414.  doi: 10.7551/mitpress/7301.001.0001.  URL https://doi.org/10.7551/mitpress/7301.001.0001.
  * Wohlin et al. [2012] Claes Wohlin, Per Runeson, Martin Höst, Magnus C. Ohlsson, Björn Regnell, and Anders Wesslén.  _Experimentation in software engineering_ , volume 9783642290442.  Springer-Verlag Berlin Heidelberg, 2012.  ISBN 978-364229044-2; 3642290434; 978-364229043-5.  doi: 10.1007/978-3-642-29044-2.  URL https://link.springer.com/book/10.1007/978-3-642-29044-2.
  * Xie et al. [2022] Huan Xie, Yan Lei, Meng Yan, Yue Yu, Xin Xia, and Xiaoguang Mao.  A universal data augmentation approach for fault localization.  In _Proceedings of the 44th International Conference on Software Engineering_ , ICSE ’22, page 48–60, New York, NY, USA, 2022. Association for Computing Machinery.  ISBN 9781450392211.  doi: 10.1145/3510003.3510136.  URL https://doi.org/10.1145/3510003.3510136.
  * Youngdahl et al. [2019] Andrew Youngdahl, Dai-Hai Ton-That, and Tanu Malik.  Sciinc: A container runtime for incremental recomputation.  In _2019 15th International Conference on eScience (eScience)_ , pages 291–300, San Diego, CA, USA, 2019. IEEE.  doi: 10.1109/eScience.2019.00040.
  * Zhang et al. [2022] Chen Zhang, Bihuan Chen, Xin Peng, and Wenyun Zhao.  Buildsheriff: Change-aware test failure triage for continuous integration builds.  In _2022 IEEE/ACM 44th International Conference on Software Engineering (ICSE)_ , pages 312–324, Pittsburgh, PA, USA, 2022. IEEE.  doi: 10.1145/3510003.3510132.
  * Zheng [2021] Zuduo Zheng.  Reasons, challenges, and some tools for doing reproducible transportation research.  _Communications in Transportation Research_ , 1:100004, 2021\.  ISSN 2772-4247.  doi: https://doi.org/10.1016/j.commtr.2021.100004.  URL https://www.sciencedirect.com/science/article/pii/S2772424721000044.
  * Zhou et al. [2022] Alexander Zhou, Yue Wang, and Lei Chen.  Butterfly counting on uncertain bipartite graphs.  _Proc. VLDB Endow._ , 15(2):211–223, feb 2022\.  ISSN 2150-8097.  doi: 10.14778/3489496.3489502.  URL https://doi.org/10.14778/3489496.3489502.