# Perspectives from India: Challenges and Opportunities for Computational
Tools to Enhance Confidence in Published Research

Tatiana Chakravorti<EMAIL_ADDRESS>1234-5678-9012 ,  Chuhao Wu
<EMAIL_ADDRESS>,  Sai Koneru<EMAIL_ADDRESS>and  Sarah Rajtmajer
<EMAIL_ADDRESS>Pennsylvania State UniversityState CollegePennsylvaniaUSA16803

(2024)

###### Abstract.

Over the past decade, a crisis of confidence in published scientific findings
has catalyzed widespread response from the research community, particularly in
the West. These responses have included policy discussions and changes to
existing practice as well as computational infrastructure to support and
evaluate research. Our work studies Indian researchers’ awareness,
perceptions, and challenges around research integrity. We explore
opportunities for Artificial Intelligence (AI)-powered tools to evaluate
reproducibility and replicability, centering cultural perspectives. We discuss
requirements for such tools, including signals within papers and metadata to
be included, and system hybridity (fully-AI vs. collaborative human-AI). We
draw upon 19 semi-structured interviews and 72 follow-up surveys with
researchers at universities throughout India. Our findings highlight the need
for computational tools to contextualize confidence in published research. In
particular, researchers prefer approaches that enable human-AI collaboration.
Additionally, our findings emphasize the shortcomings of current incentive
structures for publication, funding, and promotion.

Reproducibility, Research Challenges, Potential Incentives, Design
Requirements

††copyright: acmcopyright††journalyear: 2024††doi:
XXXXXXX.XXXXXXX††conference: ACM CHI Conference on Human Factors in Computing
Systems; May 11–16, 2024; Honolulu, HI††booktitle: CHI ’24: Proceedings of the
2024 CHI Conference on Human Factors in Computing Systems, May 11–16, 2024,
Honolulu, HI††price: 15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs: Human Centered
Computing Human Computer Interaction(HCI)

##  1\. Introduction

Questions about the reproducibility and replicability111Throughout this work,
we adopt definitions from (National Academies of Sciences, Engineering, and
Medicine et al., 2019; Nosek et al., 2021; Pineau et al., 2021). Namely,
_reproducibility_ refers to computational repeatability – obtaining consistent
computational results using the same data, methods, code, and conditions of
analysis; _replicability_ means obtaining consistent results on a new dataset
using similar methods. of published work have come to the foreground in
discussions about scientific process, incentives, and policies (Schooler,
2014; Maxwell et al., 2015; Loken and Gelman, 2017; Shrout and Rodgers, 2018;
National Academies of Sciences, Engineering, and Medicine et al., 2019). These
concerns first captured the attention of social and behavioral scientists, but
have since gained foothold across nearly all empirical scientific domains
(Baker, 2016b). In turn, the open science and science of science communities
have focused efforts on developing innovative methods to: improve research
practices from the outset; and, evaluate and contextualize prior work to
better understand what findings may be considered reliable and under what
circumstances (Gernsbacher, 2018; Schönbrodt and Mellor, 2018). Toward the
former, initiatives have focused on promoting transparency and openness (Nosek
et al., 2016), preregistration (Nosek et al., 2018), and statistical rigor
(Wasserstein and Lazar, 2016; McShane et al., 2019). Toward the latter,
computational tools are emerging, leveraging the state of the art in machine
learning, information retrieval, and natural language processing together with
a growing body of ground truth replication project outcomes for estimating
replicability of published findings (Yang et al., 2020; Pawel and Held, 2020;
Altmejd et al., 2019; Wu et al., 2021).

Thus far, however, discussion around replication has been centered in the
West. This is meaningful because changes to support scientific integrity are
deeply cultural and will require restructuring of institutional incentives,
e.g., rewarding truth over publishability (Nosek et al., 2012; Munafò et al.,
2020). To the best of our knowledge, little work has been done to understand
researchers’ perspectives on reproducibility and replicability in Asian
countries, and India in particular. Our work bridges this gap.

We conducted 19 semi-structured interviews with faculty members and PhD
scholars from universities across India representing diverse research areas.
We ask participants about their confidence in work published within their
fields, their concerns about reproducibility and replicability, and the
institutional and practical challenges they face. During our interviews, we
introduce a prototype AI-driven tool for estimating replicability of a given
finding. We use this prototype as a starting point to discuss the ways in
which computational tools can be designed and deployed to support researchers
in India. In addition to interviews, we survey 72 researchers across India
from similar domains. Survey results add cultural and statistical context to
our interview findings.

The following research questions drive this work:

  * •

RQ1: What is the current state of reproducibility and replicability in India
and to what extent do researchers in India integrate open science practices in
their own work?

  * •

RQ2: What are the experiences of researchers in India when they attempt to
replicate the work of their peers?

  * •

RQ3: What systematic incentives and changes are needed for broader integration
and promotion of reproducible research practices?

  * •

RQ4: What are the opportunities and limitations of AI-empowered tools to
support reproducible and replicable research workflows, according to our
participants? What features should these tools have?

Our findings contribute to ongoing global dialogue on scientific integrity and
highlight the importance of cultural context in understanding both challenges
and solutions to the so-called _replication crisis_. Centering researchers’
perspectives, we discuss opportunities for computational evaluation of
scientific integrity, ultimately suggesting hybrid human-AI solutions may best
serve researchers in India at this point in time.

##  2\. Related Work

Our work builds upon and contributes to the literature on scientific integrity
and open science, as well as technology-supported research processes.

###  2.1. The replication crisis

Recent years have witnessed growing concerns about the reproducibility,
replicability, and robustness of published research findings across a wide
swath of the scientific literature (Baker, 2016a). Large-scale replication
projects in psychology (Collaboration, 2015), economics (Camerer et al.,
2016), sociology (Camerer et al., 2018), biology (Errington et al., 2014),
physics (Feger et al., 2019a) and beyond have turned up disappointing results.
Various factors have been suggested as contributing to this crisis, including
selective reporting (Rowe, 2023), p-hacking (Head et al., 2015), poor
theoretical design, and unavailability of code and data (Miyakawa, 2020).

While each field has its own norms and conventions, some best practices to
support robust scientific progress have been proposed (Santana-Perez et al.,
2015; Stodden and Miguez, 2013). Key amongst these guidelines is thorough
documentation, including all details and environmental specifications required
to replicate computational experiments (Bánáti et al., 2015). Chard et al.
(2015) discuss the importance of data sharing and outline simple processes to
do so. Major challenges to reproducible and replicable research include lack
of well-aligned incentives and failure of peer review processes (Bajpai et
al., 2017; Niksirat et al., 2023; Samuel and König-Ries, 2022). These factors
collectively diminish researchers’ motivation to conduct quality checks,
prompting them to prioritize numbers of publications over research integrity.
Researchers have proposed that reproducibility and replicability should be
integrated into teaching and education (Karathanasis et al., 2022). They have
also voiced the need for changes to hiring and promotion to emphasize research
transparency (Schönbrodt, 2016; Long, 2017; Dougherty et al., 2019).

###  2.2. Transparency and open science

As noted, research transparency and opennness are foundational to
reproducibility and scientific integrity. This argument dates back to Merton’s
scientific norms (Anderson et al., 2010) and the notion of skepticism as
central to scientific endeavor. The practice of transparency includes
providing clear methodological descriptions and open access to all data, code,
and materials necessary to replicate and verify a reported finding (Patil et
al., 2016).

These practices, however, are still relatively rare (Bajpai et al., 2017).
Vandewalle et al. (2009) report results of a case study on IEEE-published
papers in signal processing. They find very few papers provide code and
reiterate the importance of materials sharing for reproducibility. Selective
reporting and insufficient description of methodology likewise obscure
transparency (Bajpai et al., 2017; Pineau et al., 2021). Open science
practices have the heralded as a key solution to mitigate the concerns around
reproducibility and replicability (McKiernan et al., 2016; Crüwell et al.,
2019). Practices like pre-registration, open peer review and transparency of
objectives have been suggested as a for research transparency (Beugelsdijk et
al., 2020). In 2013, the Centre for Open Science was established to support
the scientific research community and help to improve reproducibility and
transparency (Nosek et al., 2019; Foster and Deardorff, 2017). The movement
they began has spread to a number of countries, some of which have established
similar communities and organizations of their own (Sudlow et al., 2015; Ayris
et al., 2016; Burgelman et al., 2019).

###  2.3. Computational tools to support scientific integrity

Multiple tools and frameworks have been developed to promote open science and
reproducibility in scientific research practices (Feger et al., 2019b). One
example is the Open Science Framework (OSF), software to support researchers
at different stages of their research, allowing them to document idea
generation, study design, data collection, and reporting processes (Foster and
Deardorff, 2017). Another focus of development has been in the context of
computational tools to support reproducible code for data-intensive science
(Clyburne-Sherin et al., 2019), e.g., repository containerization (Ram, 2016).
These challenges are particularly common, e.g., in computational biology (Rule
et al., 2019; Papin et al., 2020) and AI. Ongoing empirical studies seek to
understand how researchers design experiments and to develop tools to support
them (Eiselmayer, 2020).

On the other hand, computational tools are also emerging to evaluate published
work according to its openness, reproducibility, and replicability. The
authors(Altmejd et al., 2019) assessed the accuracy of predicting experimental
result replication using black-box statistical models. Utilizing data from
four extensive replication projects conducted in experimental psychology and
economics, along with machine learning techniques, the authors developed
predictive models and examined the key variables influencing replicability
predictions. A model was introduced by Yang et al.(Yang et al., 2020) that
combines machine intelligence with human judgment to assess the likelihood of
replication for a given study. This model can serve as a valuable complement
to prediction market and survey replication methods. This model was trained
and validated on hundreds of manually replicated studies and tested on
independent datasets. Data from the four large-scale replication projects were
used to generate probabilistic forecasts of replication outcomes by the
researchers(Pawel and Held, 2020). These forecasts were subsequently assessed
for their discrimination, calibration, and sharpness. They proposed a model
that performs better than a naive model. Thanks to recent initiatives from
publication venues and funding organizations, open-access datasets and
software products are gaining recognition as significant research products.
The authors(Escamilla et al., 2023) have demonstrated that employing a hybrid
classifier proves effective in identifying URIs leading to open-access data
and software (OADS) even on less common hosting platforms.

Our interviews incorporate demonstration and discussion of one such tool. In
particular, we demonstrate for participants a prototype replication estimation
tool which takes as input a published paper and outputs a confidence score for
the paper’s primary finding as well as explanations for the system’s
assessment (Rajtmajer et al., 2022).

##  3\. Methods

We take an exploratory design approach using mixed-methods techniques
(Ivankova and Creswell, 2009; Pretorius et al., 2020; Kapania et al., 2022).
In particular, we conduct semi-structured interviews (Varanasi et al., 2022;
Thakkar et al., 2022) to understand the state of reproducibility and open
science in India from the perspectives of researchers in the social sciences
and engineering. We ask them about how they look for signals of credibility in
published work, personal and systemic factors that contribute to lack of
reproducibility and replicability, their vision for improvements to scientific
practice, and opportunities for technologies to support reproducible and
reliable research. We share with participants a prototype AI-driven
replicability assessment tool and solicit their feedback. We then survey a
wider population to validate key findings from our interviews. _IRB approval
for human subjects research was obtained prior to recruitment._

###  3.1. Participant recruitment

####  3.1.1. Interview recruitment

We recruited professors and PhD students with at least 1 year of research
experience in their domain. We recruited participants from Jadavpur University
and state of reproducibility in India (A)of Calcutta by sending recruitment
emails to all faculty and PhD students listed on departmental websites in the
social sciences and engineering. A total of 58 faculties from social science
and 33 faculties from engineering were sent the recruitment email. Both
universities hold a good rank in the NIRF ranking(Ranking, 2018). In addition,
we posted advertisements through social media and LinkedIn. Of our 19
participants, 13 were recruited through our direct emailing campaign and the
remaining 6 responded to our ad on social media. Participation was voluntary.
An overview of participants’ institutional affiliations and research areas are
provided in Table 1.222We do not provide participants’ ages in order to
mitigate risk of reindentification.

Table 1. Gender, rank, and affiliation of interview participants. ID | Gender | University | Profession | Research field
---|---|---|---|---
P1 | Female | University of Calcutta | Doctorate Candidate | Social Psychology
P2 | Male | Jadavpur University | Professor | Economics and International Trade
P3 | Male | Jadavpur University | Assistant Professor | Mechanical Engineering
P4 | Male | Jadavpur University | Assistant Professor | Electrical Engineering
P5 | Male | NIT Calicut | Assistant Professor | High Voltage Engineering
P6 | Male | IIT Bombay | Doctorate Candidate | Machine learning
P7 | Female | University of Calcutta | Doctorate Candidate | Psychology
P8 | Male | SRM University | Assistant Professor | Electronics Engineering
P9 | Female | National Instit. of Mental Health | Doctorate Candidate | Clinical Psychology
P10 | Male | University of Calcutta | Doctorate Candidate | Social Psychology
P11 | Male | Jadavpur University | Associate Professor | Dielectrics and Electrical Engineering
P12 | Male | Jadavpur University | Associate Professor | Electrical Engineering
P13 | Male | NIT Durgapur | Assistant Professor | Electrical Engineering
P14 | Male | BITS Pilani | Assistant Professor | Image Processing
P15 | Male | Jadavpur University | Professor | Power Engineering
P16 | Male | University of Calcutta | Professor | Labor Economics
P17 | Female | University of Calcutta | Doctorate Candidate | Social Psychology
P18 | Male | Jadavpur University | Assistant Professor | High Voltage Engineering
P19 | Male | Jadavpur University | Professor | Mechanical Engineering

####  3.1.2. Survey recruitment

Our survey was designed to validate primary findings from our interviews
across a wider community, in particular, given the multiculturalism in India
and the fact that most of our interview participants were from the eastern
part of the country. We selected 14 universities representing diversity of
location, selected from the top 100 universities in India based on the
National Institutional Ranking Framework (NIRF) (Ranking, 2018). We directly
emailed faculty listed on departmental webpages using the email addresses
provided in their University directory. Here as well, we selected faculty in
the social sciences and in engineering. In total, our survey was sent to 983
faculty members (338 social sciences, 645 engineering). Our email contained a
link to the survey, which was deployed as a Google Form. A one-time follow-up
email was sent out to all recipients approximately two weeks later.
Participation was voluntary. The survey was open from May end to August.

A total of 72 respondents completed the survey, amounting to a 7.32% response
rate (9.76% social sciences, 6.05% engineering). Table and Figure shown below
provide an overview of the number of surveys sent to the universities and the
diversity in the location. The university affiliation of each respondent was
not captured in order to preserve anonymity.

University SS Eng IIT Bombay 28 26 IIT Delhi 37 95 IIT Guwahati 30 26 IIT
Kanpur 11 62 IIT Kharagpur 28 61 IIT Madras 12 37 IIT Roorkee 27 48 IIT
Hyderabad 15 57 IIT Indore 18 37 IIT Bhubaneswar 18 34 Jadavpur University 16
25 University of Calcutta 42 8 Punjab University 23 69 BITS 33 60

(left) Locations of target universities across India. (right) Number of
surveys sent, by university and domain.
_IIT=Indian Institute of Technology; BITS=Birla Institute of Technology and
Science; SS=social science; Eng=engineering_

###  3.2. Data collection

####  3.2.1. Interview protocol

Semi-structured interviews (Sadek et al., 2023) were conducted virtually via
Zoom video conferencing between January 2023 and April 2023. Interviews lasted
between 25 and 40 minutes, depending on length of participant responses to
interview questions. All interviews were recorded and transcripts for analysis
were generated through transcription capabilities native to Zoom.

Our interview protocol included questions related to the state of
reproducibility and replicability in India and signals of credibility
researchers look for in published work. During the interview, we showed
participants a prototype AI-driven tool to estimate replicability of a given
finding. Specifically, the tool takes in a PDF and returns a score
representing the AI’s confidence that the primary claim of the paper would be
confirmed if the study were run again with new data. In addition to the score,
the tool provides some explanations for its assessment, including statistics
from the paper, metadata, and similar work in the literature. Details about
the AI underlying the tool’s functionality are out of scope of this study, but
a demonstration is publicly available (Rajtmajer et al., 2022). Figure
describes screen examples showing the interface of the prototype tool1

Participants were asked whether and how they might use this, or a similar,
system. They were asked what features (functional and design) computational
tools to evaluate replicability of findings should have and whether they would
trust it. Finally, we elicited participants’ perspectives about the
opportunity for hybrid human-AI systems in this space. _The complete interview
protocol is provided in the Appendix._

Figure 1. Screen captures showing the interface for the prototype replication
prediction tool.

####  3.2.2. Survey protocol

Our survey protocol included questions mirroring those asked during interviews
with the aim of understanding the robustness of interview findings at
universities across the country. Survey questions asked about participants’
perceptions of the state of the reproducibility in India, factors contributing
to lack of reproducibility and replicability of findings, and opportunities to
promote reproducible research practices in India. _The complete survey
protocol is provided in the Appendix._

###  3.3. Data analysis

####  3.3.1. Interview data analysis

Interview transcripts were studied using thematic analysis (Blandford et al.,
2016). This type of qualitative data analysis involves in-depth reading of
transcripts to identify patterns across the dataset and derive a set of themes
related to the research questions. In our study, patterns were identified
using initial codes representing participants’ stated opinions and beliefs.
These initial codes were grouped into higher-order themes. The first two
authors of the paper discussed similarities and differences among these themes
and representative examples. A final set of themes was thus inductively
determined.

####  3.3.2. Survey data analysis

Survey results were analyzed with descriptive statistics and exploratory data
visualizations. Open-ended questions (free text responses) were analyzed using
thematic analyses, as described above.

##  4\. Findings

Here, we summarize insights from our 19 interviews and 72 survey responses. We
report researchers’ perceptions of the state of reproducibility and
replicability in India, and challenges and opportunities for technologies to
support improved confidence in science. We highlight systemic and
institutional barriers to replicability, as reported by our participants.
Critically, all findings are situated within broader social and cultural
context, and offered as complimentary to Western perspectives.

###  4.1. State of reproducibility and replicability in India

####  4.1.1. Participant awareness

All participants were familiar with the concept of reproducibility, but only a
few (P7, P8, and P12) were aware of ongoing scientific dialogue around
challenges to reproducibility, i.e., the replication crisis. For example, P3
mentioned:

> I have heard about reproducibility, but not the crisis part. The majority of
> the researchers in my lab care about reproducibility. There are some who do
> not, but that percentage is rare. -P3

Conversely, roughly 80% of survey respondents reported having at least some
awareness of the replication crisis. Specifically, 44.44% of respondents
reported having heard a lot about the issue, 36.11% indicated having minimal
familiarity, while 19.44% stated they had no knowledge (see Figure 2).

During discussion, _every_ interview participant echoed the importance of
reproduction and replication. They saw these as fundamental requirements for
any scientific study and emphasized their role in upholding integrity and
transparency. Researchers felt that these concerns are not sufficiently
discussed. P5 commented that this topic is seldom openly discussed and should
receive more attention.

> I must congratulate you that it is a very wonderful topic that you are
> discussing. We don’t discuss this topic much in the open. Today everything
> is related to the output, the quantity of the paper, we have forgotten the
> quality of any outputs. So just in the number of publications it is coming.
> It is not a healthy practice at all. -P5

Researchers believed that the concepts of reproducibility and replicability
are contingent upon the type of research being done. Related to this, P7 held
the perspective that some empirical research is not reproducible by nature.

> In social justice there are more people who care about doing deep
> interviews, or clinical-type research, where you focus on case studies. So
> those are not reproducible by nature. -P7

On the other hand, P2 shared that their research involves incrementally
improving upon pre-existing work by introducing additional variables and
exploring either identical or new hypothesis. They mentioned that they do not
consider this reproduction or replication, but rather a means of validating
existing research.

####  4.1.2. Perceived peer awareness

When asked whether they believed their peers to be knowledgeable about issues
of reproducibility and replicability, interview participants expressed a sense
of uncertainty, mainly attributing their doubt to the scarcity of open
discussions on this subject. Nevertheless, they (P4, P10) believe that their
peers value reproducibility, although they may not be aware of ongoing efforts
to address this issue.

Survey findings were similar. 27.78% of survey respondents believed that their
peers had no awareness of the replication crisis, 45.83% believed their peers
to have limited understanding, and 26.38% believed their peers have robust
awareness about existing challenges to reproducibility and replicability (see
Figure 2).

Our survey also asked respondents whether they had ever tried to reproduce or
replication another researcher’s work (see Figure 3). We observe a significant
difference between responses from social scientists and engineers here.

Figure 2. State of reproducibility in India. (left) Participant awareness
about the reproducibility crisis; (right) Participants’ perceptions of peers’
awareness about the reproducibility crisis.

####  4.1.3. Adoption of open science practices

During interviews, P5, P15, P16, and P18 mentioned without prompting that they
try to adopt reproducible research practices in their own work and also in
their lab. They try to make their code and data available to other researchers
and for peer review processes.

> We do sometimes deposit the data set, even for very curated data that we
> create, even codes as well. In many cases, we deposit for public access, at
> least for journal access, so that the reviewers can look at it and make a
> decision on the basis of that. -P16

Nearly 90% of survey respondents reported that they openly share code and data
with comprehensive documentatiion, e.g., environment, hyperparameters, and
transparency of research methodologies.

> The code we build must be made available to the reviewers along with very
> good documentation about how to run it. The datasets used for
> experimentation are to be made available publicly. -Survey Participant

48.61% of survey respondents reported never having received any questions
related to reproducibility by reviewers of their work, 37.5% of respondents
reported rarely having been asked about reproducibility by reviewers, and only
8.33% reported being asked often. A follow-up question asked whether survey
respondents were ever asked by reviewers about data/code/material sharing
during the review process. Approximately 54.16% of respondents (63.64% of
respondents from the social sciences, 46.13% of respondents from engineering)
reported that they were never asked to share data/code/material. However, some
of the respondents mentioned that they always share their code and data
therefore preempting potential reviewer questions in that regard.

Figure 3. Responses to our survey question asking researchers whether they
have ever attempted to reproduce others’ research. (left) Engineering
researchers; (right) Social science researchers.

###  4.2. Challenges to reproducibility in India

Our interviews underscore the significant challenges researchers encounter due
to current practices and values in scholarly publishing. We explored these in
our survey to understand the challenges encountered by researchers while
attempting to reproduce or replicate others’ work. We asked researchers about
their experiences reproducing others’ work. 61.11% of respondents reported
having attempted to reproduce another researcher’s work. Notably, a
substantial portion of those who had attempted a reproduction were from
engineering backgrounds, a total of 89.74% from that group. Conversely, in
social sciences, this number was just 29% (see Figure 3). Subsequently, survey
respondents were asked to share the challenges they faced while replicating
others’ work.

####  4.2.1. Scholarly publishing practices

In terms of practices in scholarly publishing, both interview participants and
survey respondents reported challenges arising due to insufficient information
sharing. In computational research, this oftens manifests as lack of code and
data sharing, as P6 pointed out.

> There are some types of challenges that I face especially when the
> researchers have not given the code for their study. So in some other cases,
> the challenges that we generally face is that the authors have used some
> kind of dataset, and maybe they have not mentioned how to get that dataset
> or to generate that dataset. -P6

As highlighted by P13, a possible explanation for the hesitation to share
datasets could be due to the perception of an opportunity cost associated with
the potential utilization of the dataset for conducting additional
experiments, consequently contributing to future publications.

> The data that we have used for doing one work can be used for many other
> applications and maybe we can publish 3 more papers out of it. -P13

Participants also reported insufficient detail related to experiment design or
methodology specifications in publications. Researchers (P3, P4, P5, P6, P8,
and P15) noted encountering significant challenges stemming from missing
specifications where multiple essential parameters or experimental conditions
were often omitted in publications. They also reported difficulty reproducing
experiments with significant hardware requirements.

As shown in Figure 4, approximately 61% of survey respondents reported
unavailability data and approximately 38% reported the unavailability of code
as major concerns. The distribution of these responses was skewed, with 74.36%
respondents in engineering disciplines reporting that code sharing was a major
concern, as opposed to 27.27% of those surveyed from the social sciences.

Notably, 39% of respondents identified selective reporting as a significant
challenge to reproducibility. Low statistical power was also cited as a factor
contributing to lack of reproducibility, but it was only cited by 9% of the
social science researchers in the survey.

Figure 4. Survey responses enumerating challenges to reproducibility and
replicability.

####  4.2.2. Scholarly publishing values

Our participants frequently highlighted the demand for novel findings. As
highlighted by interviewees (P9, P10, and P18), researchers consistently face
pressure to publish novel findings, and since journals and conferences do not
readily publish replication attempts, conducting replication studies may be
perceived as less valuable or might even adverse effects on their career.

> I feel that there is a kind of negative stigma associated with people who do
> replication studies. They are like, you cannot bring something new to the
> field of research. Therefore even after hard work, no one values it. -P10

Participants reported pressure for high publication output. Considering that
adopting reproducible practices is a time-intensive endeavor and the emphasis
often lies on publishing greater number of articles rather than prioritizing
the quality of the work. As noted by P14, researchers with higher numbers of
publications tend to reap the rewards.

> In India the scenario is that if someone has 20 papers and some academic
> contacts, they will get more preference. There is no measure of how to
> quantify their quality. -P14

Our survey revealed that 44.44% of researchers believe there is substantial
pressure in India to publish a large number of papers for career advancement.
Furthermore, survey respondents mentioned that the authors often do not
respond to them when they had any questions related to their published
findings. Even when provided with the required information, they often found
themselves unable to reproduce reported results in the papers.

> I work in the area of applications of deep learning (DL) to IoT. Most of the
> time, I have observed that the performance results for the DL models are not
> really replicable. The reason could be the authors don’t share all the
> hyper-parameters for model training. However, even after trying with a range
> of hyper-parameter settings, we couldn’t replicate the results. This even
> happens for the A* conference papers. Hope with your findings and the
> corresponding publications, the researchers will start thinking about
> sharing the required information to reproduce the results. -Survey
> Participant

In many cases, the data and code are not openly available to the researchers,
as informed by one participant.

> Results are mixed. Some studies are easily reproducible as the code and data
> are available. Some other studies are difficult to replicate despite the
> availability of code and data because of inadequate documentation and
> (inadvertent) omissions of certain key parameters or assumptions. Many
> studies are not reproducible as the data/codes are not freely available and
> the methodologies are not described in enough detail in the publications.
> -Survey Participant

In the light of this, successful reproduction and replication efforts are
rare. Just 5.13% of our respondents from engineering and 6.06% from social
science reported successfully reproducing the work of others. Nevertheless,
participants were optimistic that there is gradual but slow improvement.  As
pointed out by P16, certain highly reputed journals have begun to adopt
guidelines that require authors to share data and code.

> The empirically weighted journals always value reproducibility and they want
> researchers to follow some specific guidelines for submitting the data so
> that other researchers can reproduce. In many cases, they deposited data for
> the journal access. -P16

Similarly, survey respondents shared a sense of optimism. Multiple survey
respondents pointed that in recent years, some researchers are making efforts
to enhance reproduciblity by making their work openly available to other
researchers. However, many researchers are yet to embrace this approach.

> Earlier years (before 2014, i.e., before the deep learning boom), it was
> tough to replicate. Many times code was not public and contacting the
> authors was fruitless. Also, the project pages (code readmes) were not well
> documented. So things were very tough at that time. Now the environment has
> changed. As others in the community make code public and make dedicated
> project pages (papers with code - a website has also come), it is easier to
> reproduce them. Many times, the reproduction conforms with the findings
> claimed in the papers. -Survey Participant

###  4.3. Incentives for reproducible research practices

Given the challenges to reproducibility, all interviewees agreed that changes
in funding models and publication practices might have the greatest influence
on promoting reproducible research. According to participants, since
reproducibility assessments demand significant time investments and funding
mechanisms often impose shorter timelines, rewarding reproductions and
replications is challenging. Additionally, the outcomes of such assessments
cannot be readily published which further contributes to a lack of motivation
and recognition for these endeavors. P12 asked:

> What will be my benefit as a researcher reproducing others? What would be my
> benefit that has to be clearly understood? That means, if we reproduce what
> else we can, apart from that, we can validate the research. But what is the
> utility of doing that? What will be my advantage as a researcher definitely,
> I will definitely try to publish some new things. -P12

In terms of publication models, participants highlighted scarcity of venues
that prioritize publishing reproducible research. P7 articulated this
sentiment:

> If there was a journal of reproducibility, then people will rush to do
> research to publish in the journal. Right? That is one thing that is hugely
> necessary to increase these practices. -P7

Another challenge mentioned in the interviews is that open-access journals are
not cost-free, which could serve as an incentive if they were not required to
cover the open access fees.

> In some of the journals we have to pay for open access. even for some
> transactions. If they can make the paper free of cost to connect. This will
> be a great inspiration for those who are dedicated. -P13

Researchers expressed a need for comprehensive structural changes, shifting
away from the current emphasis on publication quantity (P5, P6, P10) and
establishing of metrics akin to the h-index that can gauge contributions
towards open science and research reproducibility for individual researchers
(P6). A similar perspective was shared by a survey respondent as There is no
metric to value those researchers who are making their work open and
reproducible. Moreover, researchers suggested integration of scientific
integrity training, including best practices for reproducibility and
replicability into educational curricula (P13) and fundamental changes in
research culture. P15 shared:

> It’s not a reward. It’s kind of a practice. Probably you cannot reward for
> those things because it’s a practice, not a result. So it’s somewhat similar
> to tidying up your room every day. -P15

The interviews highlighted the need for substantial changes in incentives,
particularly in publication models, funding models, and career promotions.
Consequently, during the survey, researchers were presented with three
distinct options for the changes that could support reproducible research
practices. The participants were provided with the flexibility to choose more
than one option for this question. Most respondents opted for changes across
all three models. Notably, 81.94% of the respondents believe that significant
changes are necessary within publication models. The papers should be allowed
to be published only when they have sufficient information like codes, data,
and rigorous documentation. All these things needed to be verified by the
reviewers and that comes with time pressure and needs more reviewers clearly
mentioned by one of the participants in the survey.

> Many top computer vision or machine learning conferences now encourage code
> submission during paper submission. However, as a reviewer (due to time
> pressure) I am not willing to check the code by running it. So I think
> ultimately it comes down to reducing the load on the reviewers which in turn
> means training more reviewers to do the job right. -Survey Participant

Figure 5. Potential changes required for Reproducible Research Practices

In addition to the changes in different models, we have asked the surveyed
respondents about the need for adding coursework related to reproducible and
open science research practices within undergraduate curriculum. The
respondents can choose a single response from yes, no, maybe. It is notable
that a substantial 41.67% of respondents expressed uncertainty about this
while an equal percentage of professors endorsed the idea. Furthermore, the
support for the proposition is dependent on the research discipline with
greater interest among social science researchers in integration of the
practices into the curriculum compared to their counterparts in engineering.
Figure 5 provides the survey results for the potential changes required for
reproducible research practices.

###  4.4. Perceptions of a computational tool for estimating replicability

Here we explore the perceptions of the interviewed researchers concerning a
prototype AI-driven tool to estimate research replicability (Rajtmajer et al.,
2022). From our interviews, three themes emerged.

#### Utility

All 19 participants expressed that they had not encountered a tool capable of
predicting replicability and were excited to observe how it works. This became
evident as all the researchers willingly participated, driven by their desire
to learn about the study and share their perspectives, recognizing the
significant relevance:

> I feel that the study is very exciting and challenging in its own way, and I
> personally feel that if this method is introduced, then it will be much
> easier for the researchers to evaluate the research papers. -P1

The researchers perceived the tool as having a versatile utility, with some
asserting its usefulness even if it is accurate only half of the time (P6).
Additionally, certain researchers (P4, P13) proposed that the tool could be
particularly beneficial for early career researchers (P4), those lacking
access to research facilities (P13) who may lack the experience, and resources
to validate others’ work. Moreover, researchers noted that the tool can
expedite the literature review process (P5) and aid in the identification of
studies for further investigation (P6). Furthermore, there were suggestions
that the tool’s output can be integrated as metadata (P10). Lastly, P11 and
P12 mentioned that while the tool is very useful, specifically for
experimental research. In this context, P12 suggested that the tool should
possess characteristics specific to the domain:

> For engineering research, for example, we are looking into getting the
> research component part into two categories. One is the experimental part
> and another is the analyst. If you can provide us with the score for 2
> different parts, that will be easier for the whole analysis. Maybe the
> analysis part is quite reproducible, but not the experimental part. So if
> you can say we get different components. If you can add, I think that will
> be a good thing to do. -P12

#### Openness and explainability

Our analysis revealed a preference among participants (P2, P3, P10) for a tool
that adheres to open science and open-source practices. Specifically, P2
expressed the need for transparency in the training process. Additionally,
integration of explainable methods was a key point of discussion, with P2 and
P3 emphasizing the need for clear explanations regarding the features used and
score generation process. Furthermore, both P3 and P10 mentioned that instead
of a single score as an output, the tool should furnish an explanatory
rationale for its output.

> I think it is also necessary to point out the papers that are not
> reproducible, but it will be helpful if the tool can provide those papers
> having data missing. -P3

#### Trust

Participants had varying levels of trust in the protoype tool. The viewpoints
can be divided into two distinct aspects: confidence in the methods employed,
and robustness of its development. While some researchers expressed great
enthusiasm for the tool and its usefulness for the broader research community,
the majority struggled to place complete trust in an outcome generated by an
AI model. With the exception of P2, P12, and P16, the remaining participants
share a belief that they would place more trust in systems facilitating Human-
AI collaboration compared to fully autonomous AI systems. As an illustration,
P5, P6, P11, and P18 mentioned that they trust AI in general but prefer a
hybrid Human-AI system capable of integrating human feedback.

> It will be very helpful if human intelligence and AI can work together, and
> if you can create such kind of interface. That’s will be really awesome. -P5

The lack of trust in a fully AI system stems from awareness of their
limitations. Within this context, P12 added that AI systems heavily rely on
their training data and are not always correct:

> So the problem of all AI-based systems is that they have some inherent aired
> probability. That means we have to assume that they cannot be 100% accurate.
> And if the training data is not enough or the training is not accurate. They
> can provide drastically wrong results. That is the problem with these AI-
> based systems. But on average their performance is good. So, therefore, if I
> try this kind of system for so many databases, I believe that it will give
> you very good results. -P12

Similarly, a significant portion of the researchers conveyed in the
interviews, that they would not unconditionally trust the tool’s output,
instead they would assess reproducibility with their own expertise. P15, for
instance, mentioned his lack of trust in the software stating that he would
personally verify the results but would use the software just to obtain its
output.

> I wouldn’t rely on software totally. Rather, I would judge my intuition or
> use my intuition to check the veracity of the results. I mean compound to
> what I just said is that then does it mean that I will not be using it? No,
> I will use it to see the output. -P15

Also, researchers P3 and P18 stressed the importance of comprehensive testing
and validated the tool through user studies that engage other researchers to
establish its trustworthiness. In addition, the trust in the tool is
inherently intertwined with the research team responsible for its development.
This became evident when P5 mentioned a preference for assessing the research
group behind the tool’s implementation indicating that their trust in that
group influences their decision to utilize the tool for validating research
findings.

###  4.5. Signals of credibility

Experience was mentioned by many researchers as important for estimating
quality of research, but it comes with time and practice. We asked
participants to share some of the tangible signals they look for within a
paper or its metadata to estimate the reproducibility, replicability, and
general credibility of work in their area. Ultimately, these signals may serve
as input features to future algorithmic systems trained to estimate confidence
in published findings.

####  4.5.1. Journal and author reputation

Journal impact factor plays an outsized role in perceptions of credibility,
but our participants also noted that citation count can be misleading (P11,
P4).

> I think the journal impact factor plays an important role in judging the
> credibility of those works but I don’t look for citation count and I think
> it can mislead the researchers. -P11

> The impact factor of the journals plays a big role while doing the
> literature review but I don’t see that for citation count to choose a good
> paper in my field because it can mislead sometimes. There are some well-
> known journals in our field and we always look for those papers. -P4

According to P2, researchers always value journals that have high-impact
factors but that is not true across all domains because the calculation of
impact factors depends on many things. Also, P2 noticed that people always
cite papers that are more cited without much justification or knowledge about
it.

> Well, we always value journals that have high-impact factors. However, that
> is not always true for all the research domains because the calculation of
> the impact factor depends on many parameters. The citation count does not
> have a lot of effect on my decision. People prefer to cite the papers which
> are more cited but that is not a good practice I believe. -P2

On the other hand, P1 felt that journal impact factor and authors’ reputations
were both very important during the paper evaluation. She thinks highly
renowned author papers are more trustworthy.

> I’ll see the journals of high impact factors; when you go through those
> journals you will get some good ideas about how the paper is presented. You
> know Impact factor is the one by which people get trust. Obviously, you know
> some authors over there are highly rated because they have some seriously
> good insights that will help upcoming researchers like us. -P1

P15, a professor of Power Engineering mentioned he believes that reputable
journals have a high standard of peer review therefore people tend to believe
those results are reproducible.

####  4.5.2. Methodology and study design

The methodology and study design of a paper are important considerations when
estimating credibility of reported findings. P9, a graduate student in machine
learning commented he always goes through the methodology to get a clear
understanding of any paper.

> One important criterion would definitely be going through the methodology
> section of any paper during evaluation. That would be one thing if I am
> clear about how they have collected the data and made the design procedure,
> I will consider it. -P9

P4 also mentioned the importance of the methodology of any research work which
he considers one of the most significant features. P7, P10, and P14 mentioned
the importance of sample size and validation checks of the methods and results
of those reported outcomes. The methodology involves many hyper-parameters and
other environmental details that should be clearly mentioned. P11 who is
involved with hardware testing mentioned other factors like survey material,
and experimental setup which all are important for the credibility check of
any existing work in their domain according to him.

> I search papers based on my experience and see how logical is the overall
> paper in terms of the literature, survey, methodology, materials,
> experimental setup data, etc. -P11

####  4.5.3. Open Code and Data Availability

Open sharing of code and data plays an important role in replicating the
findings of a study. Given the nature of their research, the researchers from
engineering fields emphasized the importance of open code. For instance, P14,
a researcher in the area of image processing routinely seeks the source code
when evaluating any paper and attempts to execute it for validation purposes.

> While doing the paper evaluation, I searched for the source code. If the
> source code has already been given by the authors then I try to run those
> and evaluate the results. I also use the code with our new data and see how
> that performs -P14

P18 clearly mentioned the importance of data sharing. He thinks, if he doesn’t
share his collected data with others then it is not possible to reproduce the
work of the other researchers.

> Say, I’m doing primary survey-based research. Okay, then, I’ve got the data.
> I have to share the data with others with my publication and reported
> results. Then only others can reproduce my results, of course, otherwise,
> without the data, how can they reproduce my results? -P18

From the interviews, we observed some important features of credibility that
researchers look into for the validation of other works. We got a list of
features from the interviews that we incorporated into the survey questions.
They are open data, open code, study design, theoretical basis, sample size,
author reputation, and journal reputation, respectively. The participants were
allowed to select as many features as they thought were important. The highly
voted features are study design with 84.84% and theoretical basis with 87.88%
for the social science background. Whereas in engineering, 76.92% respondents
mentioned searching for open code and 71.79% looked for open data. Another
important observation from the survey is that 27.78% of the total respondents
looked for author reputation and 51.38% respondents voted for journal
reputation as a signal of credibility check. Figure 6 provides the survey
results for features required to validate findings from the interviews.

Figure 6. Features required to Validate Existing Findings

Enjoying the baseball game from the third-base seats. Ichiro Suzuki preparing
to bat.

###  4.6. Attitudes towards open science in India

As the scientific crisis is directly related to open science, it is required
to know the perception and acceptance of open science by Indian researchers.
Therefore we added this part during the survey, as we did not directly ask any
question about open science during the interview. From the survey responses,
we observed that 68.05% of total respondents heard about the open-science
movement whereas if you compare between two groups 81.81% of respondents from
the social science people knew about it whereas nearly 56.41% were from the
engineering background. Therefore, a scale of choices was given to them where
they had 6 choices, strongly support, mostly support, slightly support, no
opinion, slightly oppose, mostly oppose, and strongly oppose to choose
accordingly. Two participants from each background voted strongly opposed
which is nearly only 2.78% of the total responses. Although 25% of the total
respondents have no opinion of the Open Science Movement. The majority is
towards the support of open science.

####  4.6.1. Pre-registration

We asked survey participants about their experience with pre-registration, and
in particular, whether they have ever written one. 6.06% of respondents from
social science and 25.64% from engineering had pre-registered a study. Of the
two individuals who pre-registered studies in the social sciences, one was
eventually published and the other was not. From the engineering background,
10 people got pre-registered whereas 7 people were able to get a published
work from that but three were not accepted. Figure 7 provides the survey
results for the awareness of open science initiatives and prior experience
with pre-registration. Figure 8 provides the survey results showing the extent
of support for open science in India.

Figure 7. (left) Survey respondents’ awareness of open science initiatives;
(right) Survey respondents’ prior experience with pre-registration

Figure 8. Survey results showing the extent of support for open science
initiatives in India.

##  5\. Discussion

It is abundantly clear that better documentation, preservation, and open
sharing practices can benefit the research community (Falessi et al., 2006).
Open science practices have gained considerable significance in recent years,
with India actively participating in global initiatives (Scaria and Ray, 2018;
Fernandez, 2006). While various open science initiatives have popped up, our
findings indicate that their successful implementation remains a challenge in
India (Nazim et al., 2023). Given its scale, it is evident that the Indian
research community has the potential to significantly contribute to the
advancement of reproducibility and open science. Among researchers holding
academic positions in India, familiarity with the replication crisis and with
the principles of as open science appears limited.

Interestingly, when interviewed, qualitative researchers viewed that concerns
about reproducibility were not relevant to their research. However, Feger et
al. (2019c) argued the importance of replication in qualitative research such
as HCI. Our findings emphasize the need for greater emphasis on research
reproducibility and open science. While the government and institutions have
taken steps to make open science mandatory for all researchers, funding
agencies in India, for example, do not provide financial support for the
payment of processing charges for articles, despite mandates requiring
research to be deposited in open-access repositories (Nazim et al., 2023;
Hanief Bhat, 2014).

###  5.1. Education

Despite more than a decade of global attention to open science,
reproducibility and scientific integrity, important challenges still persist,
as we have highlighted. Our participants expressed enthusiasm that this study
would act as a voice highlighting their challenges and act as a catalyst to
improve scientific research.

####  5.1.1. Institutions dedicated to transparency and open science

Our findings motivate the establishment of institutions tasked with promoting
reproducibility, transparency, and openness within the academic ecosystem in
India. Our findings also suggest that there mere existence of such
institutions, without practical implementation at the grassroots level, will
not effectively address the problem. They must be deeply and meaningfully
embedded. Ideally, these institutions should also provide tangible assistance
to researchers navigating best practices, e.g., helping them share research
artifacts with proper documentation. They might also develop metrics to help
quantify researchers’ engagement with open science practices. These metrics
might offer a means by which to shift incentives and prioritize quality over
quantity of publication outputs.

####  5.1.2. Clear guidelines for transparency

We suggest that publications and funding institutions consider clear
guidelines for transparency, reproducibility, and openness. These criteria
should be rigorously assessed by reviewers as part of the peer review process.
It is important to acknowledge, however, that this may increase the workload
of reviewers who currently lack sufficient incentives for such tasks. In fact,
challenges to research reproducibility and challenges to peer review are
deeply interrelated, and solutions will necessarily be intertwines. One might
imagine a dedicated group responsible for verifying artifacts provided by
authors, but again, this raises systemic and organizational questions beyond
the scope of this work.

###  5.2. Well-aligned incentives

As evident from our findings, that there is notable interest among Indian
researchers, in improving reproducibility and prioritizing transparency and
openness. However, the lack of incentives discourages researchers from
investing significant effort in integrating open science practices and
strictly auditing the replicability of their work.

####  5.2.1. Funding and publication

The primary incentives in academic research are publications and funding.
Replication studies encounter challenges in finding suitable publication
venues, as the majority of journals and conferences prioritize novelty.

To address this issue, our findings suggest publication venues might dedicate
time and space to replication studies. This would provide researchers focused
on replication efforts with a platform for publication. It is worth noting
that some conferences have already taken steps in this direction by including
separate submission panels or organizing discussions focused on
reproducibility, as observed in Wilson et al. (2012).

Furthermore, the Indian government has demonstrated its commitment to
promoting open science. It is essential that this commitment extends to
addressing the replication crisis. Specifically, funding agencies such as the
Indian Council of Medical Research (ICMR) and the Department of Science and
Technology (DST) should actively encourage the adoption of reproducible
research practices.

####  5.2.2. Changes to hiring and promotion

Decisions about university hiring and promotion and India are signficantly
influenced by the quantity of a researcher’s publications and the reputation
of the journals in which they have published. In fact, this is appears to be
universal (Nosek et al., 2012; Hicks et al., 2015). These practices, that
prioritize quantity over quality, undermine research quality (Nelson et al.,
2012; Rahal et al., 2023). Specifically, researchers who prioritize
transparency and reproducibility may output fewer papers due to time intensive
nature of these practices. This could put them at a disadvantage in highly
competitive job markets or promotional tracks that prioritize quantity.

Our findings, therefore, suggest that universities in India might integrate
candidates’ contributions to open science and scientific integrity into their
hiring processes. It is imperative that Indian Universities begin to value and
recognize professors and scholars who actively promote open and transparent
research practices. Such recognition would encourage and incentivize their
efforts.

####  5.2.3. Metrics for reproducibility and transparency

Introducing a metric similar to h-index that can quantify a researcher’s
contributions to open science could serve as a potential first step in
reshaping the current publish or perish culture. Many interview participants
highlighted the need for such a metric and its absence in current discussions
around the topic. Similarly, they pointed out the lack of a metric to measure
research quality in general. Future work should focus on the design of the
metric, researcher attitudes, as well as potential opportunities and risks
associated with its implementation.

###  5.3. Well-designed technologies

Our findings underscore the potential value of tools that can provide context
for researchers about published findings, and aspirationally, estimate
research reproducibility, replicability, generalizability, and beyond. Our
participants enumerate a number of of essential features for training such
tools, including research design, theoretical basis, and sample size as highly
reliable indicators of reproducibility. We note that the social scientists and
engineers in our study differed somewhat in their perspectives, e.g., on the
importance of journal reputation and data sharing. These distinctions
highlight the need for future AI-driven technologies in this space to
recognize nuance across disciplines.

It is evident that researchers are more includes to trust a tool if its
training data, processes, and methodologies for generating the output score
are transparent and open. Additionally, researchers stressed the importance of
model explainability for building trust and fostering adoption. All
participants expressed a preference for hybrid systems that can integrate
inputs from both humans and AI, promoting human-AI collaboration.

##  6\. Limitations

Our work dives into the intricacies of scientific research culture in India,
shedding light on both the opportunities and challenges for improved
scientific practice. We note, however, that the insights and conclusions drawn
from this study are specific to India and may not be extrapolated to other
countries or research domains. We hope that this work will catalyze similar
studies focused on other countries, and broad recognition of the importance of
cultural context in discussion of these issues. We also note that researchers
in different domains place differing emphases on aspects of credibility most
pertinent to their respective fields, necessitating tailored approaches for
design and development of technologies to support reproducible and replicable
research moving forward.

##  7\. Conclusion

We study the perceptions, opportunities, and challenges related to
reproducible research as faced by Indian researchers in the social sciences
and engineering. Our work is a complement to similar studies and campaigns
focused primarily on researchers in the West. Our findings clearly indicate
that while there is general recognition of the value of reproduciblity, there
is insufficient motivation due to lack of tangible incentives for researchers
to meticulously document research processes and share them openly. Some
researchers choose to engage in these practices driven by their personal
ethics. We have suggested that there is a need for significant changes to
publication processes, funding models, recruitment and promotion. Our work
also acknowledges that the sheer volume of scientific output makes it
impossible to reproduce and replicate every published finding in the
literature. Accordingly, we explore researchers’ openness to AI-driven
technologies to support estimation of reproducibility and replicability. We
outline design requirements of these systems, in particular, finding that
hybrid human-AI systems may be viewed as most trustworthy for now. Finally, we
advocate for the integration of open science practices and into educational
pipelines. This approach aims to cultivate a culture of reproducibility and
research integrity within the academic community.

## References

  * (1)
  * Ram (2016) 2016\.  _ReproZip: Computational Reproducibility With Ease_.  https://doi.org/10.1145/2882903.2899401
  * Altmejd et al. (2019) Adam Altmejd, Anna Dreber, Eskil Forsell, Juergen Huber, Taisuke Imai, Magnus Johannesson, Michael Kirchler, Gideon Nave, and Colin Camerer. 2019.  Predicting the replicability of social science lab experiments.  _PloS one_ 14, 12 (2019), e0225826.
  * Anderson et al. (2010) Melissa S Anderson, Emily A Ronning, Raymond De Vries, and Brian C Martinson. 2010.  Extending the Mertonian norms: Scientists’ subscription to norms of research.  _The Journal of higher education_ 81, 3 (2010), 366–393.
  * Ayris et al. (2016) Paul Ayris, Jean-Yves Berthou, Rachel Bruce, Stefanie Lindstaedt, Anna Monreale, Barend Mons, Yasuhiro Murayama, Caj Södergård, Klaus Tochtermann, and Ross Wilkinson. 2016.  Realising the European open science cloud.  (2016).
  * Bajpai et al. (2017) Vaibhav Bajpai, Mirja Kühlewind, Jörg Ott, Jürgen Schönwälder, Anna Sperotto, and Brian Trammell. 2017.  Challenges with reproducibility. In _Proceedings of the Reproducibility Workshop_. 1–4.
  * Baker (2016a) Monya Baker. 2016a.  1,500 scientists lift the lid on reproducibility.  _Nature_ 533, 7604 (2016).
  * Baker (2016b) Monya Baker. 2016b.  Reproducibility crisis.  _Nature_ 533, 26 (2016), 353–66.
  * Bánáti et al. (2015) Anna Bánáti, Péter Kacsuk, and Miklós Kozlovszky. 2015.  Four level provenance support to achieve portable reproducibility of scientific workflows. In _2015 38th International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO)_. IEEE, 241–244.
  * Beugelsdijk et al. (2020) Sjoerd Beugelsdijk, Arjen van Witteloostuijn, and Klaus E Meyer. 2020.  A new approach to data access and research transparency (DART).  , 887–905 pages.
  * Blandford et al. (2016) Ann Blandford, Dominic Furniss, and Stephann Makri. 2016.  _Qualitative HCI research: Going behind the scenes_.  Morgan & Claypool Publishers.
  * Burgelman et al. (2019) Jean-Claude Burgelman, Corina Pascu, Katarzyna Szkuta, Rene Von Schomberg, Athanasios Karalopoulos, Konstantinos Repanas, and Michel Schouppe. 2019.  Open science, open data, and open scholarship: European policies to make science fit for the twenty-first century.  _Frontiers in big data_ 2 (2019), 43.
  * Camerer et al. (2016) Colin F Camerer, Anna Dreber, Eskil Forsell, Teck-Hua Ho, Jürgen Huber, Magnus Johannesson, Michael Kirchler, Johan Almenberg, Adam Altmejd, Taizan Chan, et al. 2016\.  Evaluating replicability of laboratory experiments in economics.  _Science_ 351, 6280 (2016), 1433–1436.
  * Camerer et al. (2018) Colin F Camerer, Anna Dreber, Felix Holzmeister, Teck-Hua Ho, Jürgen Huber, Magnus Johannesson, Michael Kirchler, Gideon Nave, Brian A Nosek, Thomas Pfeiffer, et al. 2018\.  Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015.  _Nature Human Behaviour_ 2, 9 (2018), 637–644.
  * Chard et al. (2015) Kyle Chard, Jim Pruyne, Ben Blaiszik, Rachana Ananthakrishnan, Steven Tuecke, and Ian Foster. 2015.  Globus data publication as a service: Lowering barriers to reproducible science. In _2015 IEEE 11th International Conference on e-Science_. IEEE, 401–410.
  * Clyburne-Sherin et al. (2019) April Clyburne-Sherin, Xu Fei, and Seth Ariel Green. 2019.  Computational reproducibility via containers in social psychology.  _Meta-Psychology_ 3 (2019).
  * Collaboration (2015) Open Science Collaboration. 2015.  Estimating the reproducibility of psychological science.  _Science_ 349, 6251 (2015), aac4716.  https://doi.org/10.1126/science.aac4716 arXiv:https://www.science.org/doi/pdf/10.1126/science.aac4716
  * Crüwell et al. (2019) Sophia Crüwell, Johnny van Doorn, Alexander Etz, Matthew C Makel, Hannah Moshontz, Jesse C Niebaum, Amy Orben, Sam Parsons, and Michael Schulte-Mecklenbeck. 2019.  Seven easy steps to open science.  _Zeitschrift für Psychologie_ (2019).
  * Dougherty et al. (2019) Michael R Dougherty, L Robert Slevc, and James A Grand. 2019.  Making research evaluation more transparent: Aligning research philosophy, institutional values, and reporting.  _Perspectives on Psychological Science_ 14, 3 (2019), 361–375.
  * Eiselmayer (2020) Alexander Eiselmayer. 2020.  Supporting the Design and Analysis of HCI Experiments. In _Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems_. 1–8.
  * Errington et al. (2014) Timothy M Errington, Elizabeth Iorns, William Gunn, Fraser Elisabeth Tan, Joelle Lomax, and Brian A Nosek. 2014.  An open investigation of the reproducibility of cancer biology research.  _Elife_ 3 (2014), e04333.
  * Escamilla et al. (2023) Emily Escamilla, Lamia Salsabil, Martin Klein, Jian Wu, Michele C Weigle, and Michael L Nelson. 2023.  It’s Not Just GitHub: Identifying Data and Software Sources Included in Publications.  _arXiv preprint arXiv:2307.14469_ (2023).
  * Falessi et al. (2006) Davide Falessi, Giovanni Cantone, and Martin Becker. 2006.  Documenting design decision rationale to improve individual and team design decision making: an experimental evaluation. In _Proceedings of the 2006 ACM/IEEE international symposium on Empirical software engineering_. 134–143.
  * Feger et al. (2019a) Sebastian S Feger, Sünje Dallmeier-Tiessen, Albrecht Schmidt, and Paweł W Woźniak. 2019a.  Designing for reproducibility: A qualitative study of challenges and opportunities in high energy physics. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_. 1–14.
  * Feger et al. (2019b) Sebastian S Feger, Sünje Dallmeier-Tiessen, Paweł W Woźniak, and Albrecht Schmidt. 2019b.  Gamification in science: A study of requirements in the context of reproducible research. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_. 1–14.
  * Feger et al. (2019c) Sebastian S Feger, Sünje Dallmeier-Tiessen, Paweł W Woźniak, and Albrecht Schmidt. 2019c.  The Role of HCI in Reproducible Science: Understanding, Supporting and Motivating Core Practices. In _Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems_. 1–6.
  * Fernandez (2006) Leila Fernandez. 2006.  Open access initiatives in India-an evaluation.  _Partnership: The Canadian Journal of Library and Information Practice and Research_ 1, 1 (2006).
  * Foster and Deardorff (2017) Erin D Foster and Ariel Deardorff. 2017.  Open science framework (OSF).  _Journal of the Medical Library Association: JMLA_ 105, 2 (2017), 203.
  * Gernsbacher (2018) Morton Ann Gernsbacher. 2018.  Rewarding research transparency.  _Trends in Cognitive Sciences_ 22, 11 (2018), 953–956.
  * Hanief Bhat (2014) Mohammad Hanief Bhat. 2014.  Exploring research data in Indian institutional repositories.  _Program_ 48, 2 (2014), 206–216.
  * Head et al. (2015) Megan L Head, Luke Holman, Rob Lanfear, Andrew T Kahn, and Michael D Jennions. 2015.  The extent and consequences of p-hacking in science.  _PLoS biology_ 13, 3 (2015), e1002106.
  * Hicks et al. (2015) Diana Hicks, Paul Wouters, Ludo Waltman, Sarah De Rijcke, and Ismael Rafols. 2015.  Bibliometrics: the Leiden Manifesto for research metrics.  _Nature_ 520, 7548 (2015), 429–431.
  * Ivankova and Creswell (2009) Nataliya V Ivankova and John W Creswell. 2009.  Mixed methods.  _Qualitative research in applied linguistics: A practical introduction_ 23 (2009), 135–161.
  * Kapania et al. (2022) Shivani Kapania, Oliver Siy, Gabe Clapper, Azhagu Meena SP, and Nithya Sambasivan. 2022.  ” Because AI is 100% right and safe”: User attitudes and sources of AI authority in India. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_. 1–18.
  * Karathanasis et al. (2022) Nestoras Karathanasis, Daniel Hwang, Vibol Heng, Rimal Abhimannyu, Phillip Slogoff-Sevilla, Gina Buchel, Victoria Frisbie, Peiyao Li, Dafni Kryoneriti, and Isidore Rigoutsos. 2022.  Reproducibility efforts as a teaching tool: A pilot study.  _PLOS Computational Biology_ 18, 11 (2022), e1010615.
  * Loken and Gelman (2017) Eric Loken and Andrew Gelman. 2017.  Measurement error and the replication crisis.  _Science_ 355, 6325 (2017), 584–585.
  * Long (2017) Christopher P Long. 2017.  Reshaping the tenure and promotion process so that it becomes a catalyst for innovative and invigorating scholarship.  _LSE Impact Blog_ (2017).
  * Maxwell et al. (2015) Scott E Maxwell, Michael Y Lau, and George S Howard. 2015.  Is psychology suffering from a replication crisis? What does “failure to replicate” really mean?  _American Psychologist_ 70, 6 (2015), 487.
  * McKiernan et al. (2016) Erin C McKiernan, Philip E Bourne, C Titus Brown, Stuart Buck, Amye Kenall, Jennifer Lin, Damon McDougall, Brian A Nosek, Karthik Ram, Courtney K Soderberg, et al. 2016\.  How open science helps researchers succeed.  _elife_ 5 (2016), e16800.
  * McShane et al. (2019) Blakeley B McShane, David Gal, Andrew Gelman, Christian Robert, and Jennifer L Tackett. 2019.  Abandon statistical significance.  _The American Statistician_ 73, sup1 (2019), 235–245.
  * Miyakawa (2020) Tsuyoshi Miyakawa. 2020.  No raw data, no science: another possible source of the reproducibility crisis.  , 6 pages.
  * Munafò et al. (2020) Marcus R Munafò, Christopher D Chambers, Alexandra M Collins, Laura Fortunato, and Malcolm R Macleod. 2020.  Research culture and reproducibility.  _Trends in Cognitive Sciences_ 24, 2 (2020), 91–93.
  * National Academies of Sciences, Engineering, and Medicine et al. (2019) National Academies of Sciences, Engineering, and Medicine et al. 2019\.  _Reproducibility and replicability in science_.  National Academies Press.
  * Nazim et al. (2023) Mohammad Nazim, Raj Kumar Bhardwaj, Anil Agrawal, and Afroz Bano. 2023.  Open access publishing in India: trends and policy perspectives.  _Global Knowledge, Memory and Communication_ 72, 4/5 (2023), 437–451.
  * Nelson et al. (2012) Leif D Nelson, Joseph P Simmons, and Uri Simonsohn. 2012.  Let’s publish fewer papers.  _Psychological Inquiry_ 23, 3 (2012), 291–293.
  * Niksirat et al. (2023) Kavous Salehzadeh Niksirat, Lahari Goswami, James Tyler, Alessandro Silacci, Sadiq Aliyu, Annika Aebli, Chat Wacharamanotham, Mauro Cherubini, et al. 2023\.  Changes in research ethics, openness, and transparency in empirical studies between chi 2017 and chi 2022.  (2023).
  * Nosek et al. (2016) Brian A Nosek, George Alter, George Christopher Banks, Denny Borsboom, Sara Bowman, Steven Breckler, Stuart Buck, Chris Chambers, Gilbert Chin, Garret Christensen, et al. 2016\.  Transparency and openness promotion (TOP) guidelines.  (2016).
  * Nosek et al. (2018) Brian A Nosek, Charles R Ebersole, Alexander C DeHaven, and David T Mellor. 2018.  The preregistration revolution.  _Proceedings of the National Academy of Sciences_ 115, 11 (2018), 2600–2606.
  * Nosek et al. (2021) Brian A Nosek, Tom E Hardwicke, Hannah Moshontz, Aurélien Allard, Katherine S Corker, Anna Dreber Almenberg, Fiona Fidler, Joseph Hilgard, Melissa Kline, Michèle B Nuijten, et al. 2021\.  Replicability, robustness, and reproducibility in psychological science.  (2021).
  * Nosek et al. (2019) Brian A Nosek, Lucy Ofiesh, Nicole Pfeiffer, Sara Bowman, David M Litherland, Abram Booth, Fitz Elliott, James Davis, Matthew Frazier, Dawn Pattison, et al. 2019\.  Center for Open Science: Proposal for NSF 19-537 Mid-scale Research Infrastructure.  (2019).
  * Nosek et al. (2012) Brian A Nosek, Jeffrey R Spies, and Matt Motyl. 2012.  Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability.  _Perspectives on Psychological Science_ 7, 6 (2012), 615–631.
  * Papin et al. (2020) Jason A Papin, Feilim Mac Gabhann, Herbert M Sauro, David Nickerson, and Anand Rampadarath. 2020.  Improving reproducibility in computational biology research.  , e1007881 pages.
  * Patil et al. (2016) Prasad Patil, Roger D Peng, and Jeffrey T Leek. 2016.  A statistical definition for reproducibility and replicability.  _BioRxiv_ (2016), 066803.
  * Pawel and Held (2020) Samuel Pawel and Leonhard Held. 2020.  Probabilistic forecasting of replication studies.  _PloS one_ 15, 4 (2020), e0231416.
  * Pineau et al. (2021) Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Larivière, Alina Beygelzimer, Florence d’Alché Buc, Emily Fox, and Hugo Larochelle. 2021.  Improving reproducibility in machine learning research (a report from the neurips 2019 reproducibility program).  _The Journal of Machine Learning Research_ 22, 1 (2021), 7459–7478.
  * Pretorius et al. (2020) Claudette Pretorius, Darragh McCashin, Naoise Kavanagh, and David Coyle. 2020.  Searching for mental health: a mixed-methods study of young people’s online help-seeking. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_. 1–13.
  * Rahal et al. (2023) Rima-Maria Rahal, Susann Fiedler, Adeyemi Adetula, Ronnie P-A Berntsson, Ulrich Dirnagl, Gordon B Feld, Christian J Fiebach, Samsad Afrin Himi, Aidan J Horner, Tina B Lonsdorf, et al. 2023\.  Quality research needs good working conditions.  _Nature Human Behaviour_ 7, 2 (2023), 164–167.
  * Rajtmajer et al. (2022) Sarah Rajtmajer, Christopher Griffin, Jian Wu, Robert Fraleigh, Laxmaan Balaji, Anna Squicciarini, Anthony Kwasnica, David Pennock, Michael McLaughlin, Timothy Fritton, et al. 2022\.  A synthetic prediction market for estimating confidence in published work. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 36. 13218–13220.
  * Ranking (2018) Home Ranking. 2018.  National institutional ranking framework.  _Methodology for Ranking of Academic Institutions in India (Ranking Metrics for Medical)_ (2018).
  * Rowe (2023) Anthony Rowe. 2023.  Recommendations to improve use and reporting of statistics in animal experiments.  _Laboratory Animals_ 57, 3 (2023), 224–235.
  * Rule et al. (2019) Adam Rule, Amanda Birmingham, Cristal Zuniga, Ilkay Altintas, Shih-Cheng Huang, Rob Knight, Niema Moshiri, Mai H Nguyen, Sara Brin Rosenthal, Fernando Pérez, et al. 2019\.  Ten simple rules for writing and sharing computational analyses in Jupyter Notebooks.  , e1007007 pages.
  * Sadek et al. (2023) Malak Sadek, Rafael A Calvo, and Celine Mougenot. 2023.  Trends, Challenges and Processes in Conversational Agent Design: Exploring Practitioners’ Views through Semi-Structured Interviews. In _Proceedings of the 5th International Conference on Conversational User Interfaces_. 1–10.
  * Samuel and König-Ries (2022) Sheeba Samuel and Birgitta König-Ries. 2022.  A collaborative semantic-based provenance management platform for reproducibility.  _PeerJ Computer Science_ 8 (2022), e921.
  * Santana-Perez et al. (2015) Idafen Santana-Perez, María S Pérez-Hernández, et al. 2015\.  Towards reproducibility in scientific workflows: An infrastructure-based approach.  _Scientific Programming_ 2015 (2015).
  * Scaria and Ray (2018) Arul George Scaria and Shreyashi Ray. 2018.  Open Science India Report.  (2018).
  * Schönbrodt (2016) FELIX Schönbrodt. 2016.  Changing hiring practices towards research transparency: The first open science statement in a professorship advertisement.  _Retrieved March_ (2016).
  * Schönbrodt and Mellor (2018) Felix Schönbrodt and David Mellor. 2018.  Academic job offers that mentioned open science. Open Science Framework.
  * Schooler (2014) Jonathan W Schooler. 2014.  Metascience could rescue the ‘replication crisis’.  _Nature_ 515, 7525 (2014), 9–9.
  * Shrout and Rodgers (2018) Patrick E Shrout and Joseph L Rodgers. 2018.  Psychology, science, and knowledge construction: Broadening perspectives from the replication crisis.  _Annual review of psychology_ 69 (2018), 487–510.
  * Stodden and Miguez (2013) Victoria Stodden and Sheila Miguez. 2013.  Best practices for computational science: Software infrastructure and environments for reproducible and extensible research.  _Available at SSRN 2322276_ (2013).
  * Sudlow et al. (2015) Cathie Sudlow, John Gallacher, Naomi Allen, Valerie Beral, Paul Burton, John Danesh, Paul Downey, Paul Elliott, Jane Green, Martin Landray, et al. 2015\.  UK biobank: an open access resource for identifying the causes of a wide range of complex diseases of middle and old age.  _PLoS medicine_ 12, 3 (2015), e1001779.
  * Thakkar et al. (2022) Divy Thakkar, Azra Ismail, Pratyush Kumar, Alex Hanna, Nithya Sambasivan, and Neha Kumar. 2022.  When is machine learning data good?: Valuing in public health datafication. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_. 1–16.
  * Vandewalle et al. (2009) Patrick Vandewalle, Jelena Kovacevic, and Martin Vetterli. 2009.  Reproducible research in signal processing.  _IEEE Signal Processing Magazine_ 26, 3 (2009), 37–47.
  * Varanasi et al. (2022) Rama Adithya Varanasi, Divya Siddarth, Vivek Seshadri, Kalika Bali, and Aditya Vashistha. 2022.  Feeling Proud, Feeling Embarrassed: Experiences of Low-income Women with Crowd Work. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_. 1–18.
  * Wasserstein and Lazar (2016) Ronald L Wasserstein and Nicole A Lazar. 2016.  The ASA statement on p-values: context, process, and purpose.  , 129–133 pages.
  * Wilson et al. (2012) Max Wilson, Wendy Mackay, Ed Chi, Michael Bernstein, and Jeffrey Nichols. 2012.  RepliCHI SIG: From a panel to a new submission venue for replication.  In _CHI’12 Extended Abstracts on Human Factors in Computing Systems_. 1185–1188.
  * Wu et al. (2021) Jian Wu, Rajal Nivargi, Sree Sai Teja Lanka, Arjun Manoj Menon, Sai Ajay Modukuri, Nishanth Nakshatri, Xin Wei, Zhuoer Wang, James Caverlee, Sarah M Rajtmajer, et al. 2021\.  Predicting the Reproducibility of Social and Behavioral Science Papers Using Supervised Learning Models.  _arXiv preprint arXiv:2104.04580_ (2021).
  * Yang et al. (2020) Yang Yang, Wu Youyou, and Brian Uzzi. 2020.  Estimating the deep replicability of scientific findings using human and artificial intelligence.  _Proceedings of the National Academy of Sciences_ 117, 20 (2020), 10762–10768.