**footnotetext: Equal contributors.

# Exploring the Role of Explainability in AI-Assisted Embryo Selection

Lucia Urcelay ∗ Barcelona Supercomputing Center  Daniel Hinjos ∗ Barcelona
Supercomputing Center  Pablo A. Martin-Torres ∗ Barcelona Supercomputing
Center  Marta Gonzalez ∗ Barcelona Supercomputing Center  Marta Mendez
Hospital Clínic de Barcelona  Salva Cívico  Hospital Clínic de Barcelona
Sergio Álvarez-Napagao  Barcelona Supercomputing Center  Universitat
Politécnica de Catalunya - Barcelona TECH  Dario Garcia-Gasulla  Barcelona
Supercomputing Center  Universitat Politécnica de Catalunya - Barcelona TECH

###### Abstract

In Vitro Fertilization is among the most widespread treatments for
infertility. One of its main challenges is the evaluation and selection of
embryo for implantation, a process with large inter- and intra-clinician
variability. Deep learning based methods are gaining attention, but their
opaque nature compromises their acceptance in the clinical context, where
transparency in the decision making is key. In this paper we analyze the
current work in the explainability of AI-assisted embryo analysis models,
identifying the limitations. We also discuss how these models could be
integrated in the clinical context as decision support systems, considering
the needs of clinicians and patients. Finally, we propose guidelines for the
sake of increasing interpretability and trustworthiness, pushing this
technology forward towards established clinical practice.

##  1 Introduction

Infertility is a common reproductive health problem that affects millions of
people worldwide, causing social, psychological, physical and economic
distress to the ones seeking to conceive [7]. In the coming years infertility
rates are projected to grow due to environmental and lifestyle factors [18,
37]. In vitro fertilization (IVF) technology is used to overcome infertility,
it involves the fertilization of an egg with sperm in the laboratory, followed
by the transfer of the resulting embryos into the patient’s uterus. The main
challenge of IVF is the selection of the embryos that will be either selected
for implantation, frozen (for later implantation) or discarded (if they
exhibit undesirable features). This selection is to be performed during the
early hours after embryo insemination, typically between three and five days
after. During this time, embryos are monitored in time-lapse imaging
incubators (TLI), facilitating uninterrupted embryo growth within stable
culture conditions. This technology offers a dynamic perspective on in vitro
embryonic development, augmenting the clinical effectiveness of IVF [29].

To assess quality, embryologists evaluate different morphological
characteristics depending on the embryo development phase. Early development
(days one to three) focus on cell number, symmetry and fragmentation rate,
while embryos reaching bastocyst stage (day five) are further assessed by
their expansion grade and the appearance of the inner cell mass (ICM) and the
Zona Pellucida (ZP), as well as to the trophectoderm cells (TE) [30]. These
morphological features represent the foundation of current development
assessment guidelines such as Gardner’s [15] or ASEBIR in Spain. These
approaches are limited by the subjective assessment of embryologists, which
causes inter and intra-observer variability. The success rates of IVF are
restricted due to these limitations. According to [1], the global life birth
rate via IVF in 2015 stood at 19.2% per oocyte retrieval with fresh embryo
transfer and 24.8% with frozen embryo transfer.

Artificial Intelligence (AI), and specially Deep Learning (DL), due to its
capacity for dealing with images, have recently been considered to assist in
the embryo assessment and selection process. AI has the potential to
facilitate and improve the process of embryo selection, increasing the
implantation success rates, and reducing the chances of multiple pregnancies.
AI can also mitigate inter and intra-observer variability, making results more
reproducible and comparable [35]. Finally, AI can help reduce the financial,
physical and emotional burden on patients, by optimising the treatment plan
and minimising the need for repeated cycles of IVF.

Whenever AI is considered for an application which directly affects human
life, trustworthiness must be assured. In this paper we review the current
work on AI-assisted embryo selection with special focus on Explainable
Artificial Intelligence (XAI), and how this is deployed. We analyze the
limitations of current work and propose approaches that would yield AI systems
that can provide better explanations for their recommendations. Moreover, we
discuss key aspects in the integration of these systems in a clinical context
where both clinicians and patients are involved in the decision-making
process.

##  2 Related Work

This work discusses the integration of explainability components into AI
solutions for embryo analysis in IVF, as well as the particularities of their
integration into a human-centric decision making process. Previous solutions
proposed in the bibliography are reviewed within the main discussion of this
work (see §3.1), as it contextualizes our contribution. In contrast, the
related work presented in this section comprehends previous analysis and
surveys conducted in the field, which have motivated this paper.

Among recent surveys on AI-assisted embryo selection in IVF, most restrict
themselves to the technicalities of the machine learning problem itself. That
is, what is the problem to solve, which data is used and how, and which are
the top-of-the-line performance metrics achieved by the models built. A few
discuss the biological aspects of the problem with particular depth [22],
while others focus on the variety of tasks one can tackle (_e.g._ , embryo
segmentation, quality grading, etc.) [20]. The most ambitious tasks, those of
predicting clinical pregnancy or fetal heartbeat from five days old fertilized
embryo images, are reviewed in [23], together with the ploidy status (number
of chromosomes) prediction task. [26] includes the annotation of data (_e.g._
, development phase, cell counting) in its review, while the particularities
of the time-lapse imaging are the focus of [28]. On top of embryos, [11] also
reviews AI applications for spermatozoa and oocyte analysis. [13] goes beyond
DL methods, and also consider Bayesian Networks and SVMs in their study.
Remarkably, very few works [2, 40, 34] study the ethical, socio-economic,
legal and cultural (ELSEC) aspects in AI-assisted embryo selection.

None of the previously discussed works includes a review on the XAI methods
used for embryo analysis, or discussion on practical and responsible
deployment. This work aims at filling that gap, considering that both these
issues need to be properly assessed before clinicians can integrate AI in
their day-to-day practice.

##  3 The Role of Explainable Artificial Intelligence in Embryo Analysis

Embryo analysis post-fertilization lasts for a maximum of 7 days. During that
time, clinicians assess the status and evolution of embryos, as these evolve
under controlled conditions. Through continuous visual monitoring and tracking
of their morphological features, experts eventually discard, freeze or select
embryos for transfer. Current embryo incubators include periodic microscopy,
with images being recorded at frequency in the order of minutes (_e.g._ , ten)
at several focal planes.

AI models trained on time-lapse embryo images have shown promise in embryo
analysis. Within the domain of images, explainable AI methods are often
feature scorers, representing their output as saliency maps on the input image
(see Figure 1). By nature, XAI methods for DL are approximations to the
model’s real behavior. It is worth noting that XAI is critical to ensure that
the decision-making process of an algorithm is as transparent and
understandable to clinicians and patients as possible. The lack of
explainability can lead to a lack of trust in the technology, which ultimately
hinders its adoption.

In this section we review the use of XAI on embryo images. Among the methods
found in the literature, we can differentiate between those applied after the
model is trained (post-hoc), and those where the explainability is part of the
model architecture and design (intrinsic). Post-hoc methods include model-
agnostic ones such as LIME [33] and KernelSHAP [27], that obtain the
explanations by perturbing the inputs and observing the changes caused on the
outputs; also, model-specific methods such as Grad-CAM [38] and Deep SHAP
[27], that use the parameters of neural network models to obtain the saliency
maps. Among intrinsic methods, we can find attention-based CNNs, where the
explainability is obtained from attention layers. A different approach is the
use of Speeded Up Robust Features (SURF) [6] to extract local visual features,
in conjunction with Gaussian Mixture Models (GMM) to obtain a Fisher vector
per image [21].

Figure 1: Saliency maps for Score-CAM and BR-NPA XAI methods from [17]. Ref. |  Task |  Architecture |  XAI Method(s) |  ClinicalEval
---|---|---|---|---
[10] |  Embryo Quality Grading |  DenseNet, ResNet |  Grad-CAM++ |  No
[43] |  Embryo Quality Grading |  LWMA-Net |  Grad-CAM |  No
[21] |  Embryo Quality Grading |  SURF and GMM, ResNet-101, EfficientNet-B1 |  SURF, Grad-CAM |  No
[4] |  Embryo Segmentation |  MASS-Net |  Grad-CAM |  No
[39] |  Development Stage Identification |  ResNet34, VGG16 |  Grad-CAM, SHAP, LIME |  Yes
[32] |  Embryo Quality Grading |  ResNet50, VGG-16 |  Grad-CAM |  No
[17] |  Development Stage Identification |  ResNet50, ABN |  B-CNN, ABN, InterByParts, Grad-CAM++, RISE, Score-CAM, Ablation-CAM, AM |  No
[12] |  Fetal Heart Pregnancy, Live Birth |  ResNet18, RandomForest |  Grad-CAM, SHAP |  No
[36] |  Live Birth |  ABN |  ABN |  Yes
[42] |  Embryo Quality Grading |  VGG-16 |  Grad-CAM |  No
[41] |  Embryo Quality Grading |  Xception, Inception-ResNET-v2 |  CAM |  No
Table 1: Summary table of studies that use XAI on embryo selection. The
Clinical Evaluation column indicates whether the resulting saliency maps have
been evaluated by experts or not.

###  3.1 Review on Current Work

Depending on the specific task, XAI is used and interpreted differently:

_Embryo Quality Grading:_ the classification of embryos (_e.g._ , day 5
blastocysts), into classes for selecting the best for implantation. This
challenge is the most addressed regarding XAI. Some studies apply Grad-CAM for
visualization [32, 42] and observe that key regions that the model relies on
seem to be consistent with clinical interpretation. [10] uses Grad-CAM++ for
visualization, while statistically correlating their model outputs with the
Gardner score. [41] directly use CAM saliency maps, and observe their model
focus on known features like cellular fragmentation, blastomeres or vacuoles.
A different approach compares Grad-CAM and SURF (computing a single Fisher
Vector per image), finding the former more prone to erroneously focus on
irrelevant areas [21].

_Embryo Development Stage Identification:_ consists on the automated labelling
of the embryo stage. Challenges include deformed cell shapes, poor visual
features or similarities between embryos at different stages. [39] compares
Grad-CAM and LIME classification activations, concluding that LIME
explanations seem less consistent with biologically important regions. Authors
also suggest SHAP could be useful to identify reasons for misclassification
between adjacent cleavage stages.

_Blastocyst Segmentation:_ four crucial morphological regions to segment: TE,
ZP, ICM and BL (see §1). A multiscaling architecture that outputs segmentation
masks for the four regions is introduced in [4], where authors apply Grad-CAM
to different layers to show the evolution of activations. A U-Net has also
been used to separate the background from the blastocyst [21], using an
ellipse on top of the segmentation mask to separate inner cell mass from
trophectoderm.

_Fetal Heart Pregnancy:_ is defined as the presence of fetal heartbeats in the
uterus. Of course, the embryo implantation process does not guarantee a
successful pregnancy, as it depends on many other factors (_e.g._ , age,
progesterone levels, _etc._). [12] uses Grad-CAM to detect morphological
indicators, and SHAP to account for relevant metadata that influences the
model’s decision, concluding that embryo images are the best predictor for
fetal heart beat, followed by age and pregnancy history.

_Live Birth:_ prediction from time-lapse imaging, [36] uses an Attention
Branch Network (ABN) [14]. Authors visualyze embryo features of relevance for
Live Birth by exploring the attention mechanism. Results indicated that no
common visual features could be associated to the predicted outcome of live or
non-live birth.

To the best of our knowledge, other very important challenges in this field,
such as Blastocyst Prediction or Ploidy Detection, are not addressed in the
context of explainability. This highlights the need for further research in
these tasks.

###  3.2 Limitations and Suggestions

A limited amount of research comprehends the use of AI-assisted IVF in
combination with XAI methods. A representative list can be found in Table 1.
In many cases XAI is only considered through the illustration and minor
discussion of a few saliency maps [4, 21, 41, 43]. Others take one more step
and analyze the map activations in order to correlate them to morphological
features [39, 42] or to the objective [12]. Few share the saliency maps with
expert embryologists for evaluation [39, 36]. Meanwhile, several studies
conclude that saliency maps should not be used as the sole source of
explainability in high risk medical domains [5, 16].

Considering the features that clinicians base their decisions on, and the
difficulty of interpreting saliency maps, we suggest the use of expert models.
Different models targeting different morphokinetical features, such as
fragmentation score, ICM and TE grades, ZP and Pronucleus characteristics, and
phase transition times. The ensemble of all these experts should be a white
box model (_e.g._ , GLM, Decision Tree) to gain an intermediate level of
explainability on how each feature contributes to the objective.

In the papers reviewed, the XAI method used for interpretability is rarely
baselined or justified. Two studies compare two explainability methods, [39]
(LIME vs Grad-CAM) and [21] (Bag-of-visual-words-based approach vs Grad-CAM).
Only one thorough comparative study was found [17], in which nine different
XAI methods, including five post-hoc ones, were evaluated using seven
faithfulness metrics. This is, as well, the only study in which explainability
methods are assessed through quality metrics such as the Increase in
Confidence or the Average Drop [9].

Another limitation in the field is privacy. None of the previous works makes
the code (except for [4]) or the data (except for [17]) publicly available,
making reproducibility impossible and replicability impractical. These flaws,
in a task with a high degree of subjectivity (inter and intra-observer
variability) induce a high risk of confirmation bias. This issue could be
mitigated through a quantitative benchmark measuring the model’s attention to
certain embryo regions of interest across a relatively big amount of samples.
That would require an additional segmentation model such as [4], which
isolates the TE, ZP, ICM, and BL regions. A good explanation from a good model
should overlap with the key segmented areas, and could be assessed with
metrics like IoU. This approach is limited to blastocyst images, and subject
to the accuracy of the segmentation model.

##  4 Integrating Explainable Artificial Intelligence in Decision Support
Systems

The performance of AI methods on top-of-the-line metrics (_e.g._ , precision,
recall, _etc._) is often the endpoint of most contributions in the embryo
assessment field. However, transforming a quantitatively performing model into
a successful practical deployment is no easy task, esp. when considering the
high-stakes involved in the context of IVF. The question on how to effectively
and safely introduce this technology into a clinical setting where the
different actors, including clinicians and patients, are involved, remains
unsolved. Next we aim at advancing towards a solution, filling a gap in
current literature.

###  4.1 Opaque Models Create Responsibility Gaps

The use of opaque AI models raises ethical and legal accountability concerns
[2], particularly when clinicians can not explain the decision-making process.
This creates a “responsibility gap”, and without established accountability
mechanisms it is challenging to determine who should be held responsible for
any potential harm. For example, in cases of sub-optimal embryo selection or
injury due to model recommendations, the decision-making process must be
explainable to patients seeking to understand what happened or, in more
extreme cases, seeking compensation for the damage caused. As of today,
distrust in AI applications in medicine also comes from doctors’ fear of legal
repercussions if something goes wrong due to unclear liability regimes [31].
If clinicians base their decision on these opaque AI models, the evaluation of
the decision-making process and, consequently, the determination of who is
responsible will be greatly hampered.

It is then essential that the explanations provided by the model are as clear
and understandable as possible by the clinician. For embryo image analysis
this motivates the use of saliency maps obtained through feature attribution
methods, which provide reliable (_i.e._ , true to the underlying model
behavior) but also accessible evidence. To further ground the produced
explanations in the context of clinical embryo selection, we suggest the
mapping of this evidence-based support to the established annotation and
evaluation guidelines and practices of the field, _e.g._ , through the
association of feature attributions with the already established morphokinetic
markers (such as a part of an image showing inappropriate cellular
fragmentation), so that evidence provided is given in a language appropriate
for the clinician and the problem. Still, due to the limitations of these XAI
methods, it is recommended to adhere to ethical guidelines in AI, such as the
one proposed by the EU [3]. By following this framework, other key
requirements such as data privacy and governance or human agency and oversight
can be accomplished.

###  4.2 Building Trust through Explanations

The successful integration of AI-assisted embryo selection systems in the
medical context heavily depends on the acceptance by the clinicians and,
consequently, their trust in them.

According to [31], there are several levels of trust which fall along a
spectrum, ranging from complete distrust to over-reliance on AI systems.
Studies have shown that over-reliance on the suggestions can cause clinicians
to take less initiative [25] and also to be more likely to accept incorrect
diagnosis [19]; this is known as _automation bias_ [24]. On the other side of
the spectrum lie clinicians which do not trust an algorithm that they do not
understand [8], a phenomenon known as _algorithmic aversion_. As a result, not
only training and education programs are necessary to ensure that clinicians
understand the capabilities and limitations of AI-assisted embryo selection
systems, but also making sure that the explanation or the model are not having
negative impacts on the clinician’s decision making process.

It is recommended to avoid offering the discretized outcome of models (_e.g._
, this embryo is good) in favour of more detailed information, such as outcome
probabilities (_e.g._ , this embryo is good with a probability of 55%). For
further bias avoidance, evidence may be presented only in the form of
explanations, with the outcome hidden (_e.g._ , this area shows relevant
visual features for determining if the embryo is good or not), so that the
clinician interprets and integrates the visual features unbiasedly w.r.t. to
the model’s final prediction. Unfortunately, this approach is not without
flaws either. It can induce _confirmation bias_ (_i.e._ , a clinician may only
review the evidence that supports its own hypothesis), while preventing the
methodology to exploit visual patterns not perceivable by humans, which may be
a limiting factor in embryo analysis. The proposed solution is to provide all
information available, outcome probabilities and saliency maps, and to extend
that information to all relevant cases (_e.g._ , an area shows relevant visual
features in favour of the embryo being good, an area shows features in favour
of it being bad).

###  4.3 Patient, Clinician, AI

IVF is a highly personal procedure, with a large impact in the physical and
mental health of individuals. Patients undergoing IVF have the right to know
how decisions are being made about their own healthcare, and to maintain their
decision-making power. To guarantee that, AI explanations should be also
accessible to them; they cannot be left behind by the complexity of
technology. Integrating explainability into AI-assisted embryo selection
systems can not only target clinicians, it must also empower patients. By
providing them with clear and easy to understand explanations of certain
decisions, patients can be more involved and better equipped for the decision-
making process regarding their own care. XAI results should be presented when
relevant, and under the supervision of the clinician for its adequate
interpretation and contextualization. Finally, it has to be mentioned that,
since the use of these systems directly involves the processing of their data,
the patient should be educated about how it will be processed and the use that
will be made of it, in addition to asking for their informed consent.

##  5 Conclusions

We present an analysis of the presence of XAI for embryo selection on the
literature. While there is a relative abundance of models for embryo selection
and many have shown promising results when trained on retrospective data, the
performance of these models in actual cases is yet unknown and their
integration in the medical decision process remains unclear. Due to the opaque
nature of these models, explainability is crucial to ensure their
applicability. However, although some studies tackle the issue of
explainability, the interpretations offered are often insufficient or do not
rely on medical experts for evaluation. We enumerate a set of guidelines and
suggestions that could help increase the interpretability and trustworthiness
of these systems, with the goal of advancing towards the successful and safe
integration of this technology in clinical practice.

## Acknowledgements

This work has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement no.957331 - knowlEdge.

## References

  * [1] G. Adamson et al.  “International Committee for Monitoring Assisted Reproductive Technology: world report on assisted reproductive technology, 2011”  In _Fertility and Sterility_ 110.6, 2018, pp. 1067–1080
  * [2] Michael Anis Mihdi Afnan et al.  “Ethical Implementation of Artificial Intelligence to Select Embryos in In Vitro Fertilization”, AIES ’21, 2021, pp. 316–326
  * [3] Pekka Ala-Pietilä et al.  “Ethics Guidelines for Trustworthy AI”  In _FUTURIUM - European Commission_ , 2018  URL: https://ec.europa.eu/futurium/en/ai-alliance-consultation
  * [4] Muhammad Arsalan et al.  “Human Blastocyst Components Detection Using Multiscale Aggregation Semantic Segmentation Network for Embryonic Analysis”  In _Biomedicines_ 10.7, 2022, pp. 1717
  * [5] Nishanth Arun et al.  “Assessing the Trustworthiness of Saliency Maps for Localizing Abnormalities in Medical Imaging”  In _Radiology: Artificial Intelligence_ 3.6, 2021, pp. e200267
  * [6] Herbert Bay, Tinne Tuytelaars and Luc Van Gool  “SURF: Speeded Up Robust Features”  In _Computer Vision – ECCV 2006_ Springer Berlin Heidelberg, 2006, pp. 404–417
  * [7] Jacky Boivin, Laura Bunting, John A Collins and Karl G Nygren  “International estimates of infertility prevalence and treatment-seeking: potential need and demand for infertility medical care”  In _Human reproduction_ 22.6, 2007, pp. 1506–1512
  * [8] Carrie J Cai et al.  “Hello AI: uncovering the onboarding needs of medical practitioners for human-AI collaborative decision-making”  In _Proceedings of the ACM on Human-computer Interaction_ 3.CSCW, 2019, pp. 1–24
  * [9] Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader and Vineeth N Balasubramanian  “Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks”  In _2018 IEEE winter conference on applications of computer vision (WACV)_ , 2018, pp. 839–847  IEEE
  * [10] Sonya M. Diakiw et al.  “An artificial intelligence model correlated with morphological and genetic features of blastocyst quality improves ranking of viable embryos”  In _Reproductive BioMedicine Online_ 45.6, 2022, pp. 1105–1117
  * [11] Irene Dimitriadis, Nikica Zaninovic, Alejandro Chavez Badiola and Charles L. Bormann  “Artificial intelligence in the embryology laboratory: a review”  In _Reproductive BioMedicine Online_ 44.3, 2022, pp. 435–448
  * [12] Noritoshi Enatsu et al.  “A novel system based on artificial intelligence for predicting blastocyst viability and visualizing the explanation”  In _Reproductive Medicine and Biology_ 21.1, 2022, pp. e12443
  * [13] Eleonora Inácio Fernandez et al.  “Artificial intelligence in the IVF laboratory: overview through the application of different types of algorithms for the classification of reproductive data”  In _Journal of Assisted Reproduction and Genetics_ 37.10, 2020, pp. 2359–2376
  * [14] Hiroshi Fukui, Tsubasa Hirakawa, Takayoshi Yamashita and Hironobu Fujiyoshi  “Attention Branch Network: Learning of Attention Mechanism for Visual Explanation”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019
  * [15] D.. Gardner and D. Sakkas  “Assessment of embryo viability: the ability to select a single embryo for transfer–a review”  In _Placenta_ 24 Suppl B, 2003, pp. S5–12
  * [16] Marzyeh Ghassemi, Luke Oakden-Rayner and Andrew L Beam  “The false hope of current approaches to explainable artificial intelligence in health care”  In _The Lancet Digital Health_ 3.11, 2021, pp. e745–e750
  * [17] Tristan Gomez, Thomas Fréour and Harold Mouchère  “Comparison of attention models and post-hoc explanation methods for embryo stage identification: a case study”  In _arXiv preprint arXiv:2205.06546_ , 2022
  * [18] A.. Gore et al.  “EDC-2: The Endocrine Society’s Second Scientific Statement on Endocrine-Disrupting Chemicals”  In _Endocrine Reviews_ 36.6, 2015, pp. E1–E150
  * [19] Yukinori Harada, Shinichi Katsukura, Ren Kawamura and Taro Shimizu  “Effects of a differential diagnosis list of artificial intelligence on differential diagnoses by physicians: an exploratory analysis of data from a randomized controlled study”  In _International Journal of Environmental Research and Public Health_ 18.11, 2021, pp. 5562
  * [20] Iza Sazanita Isa, Umi Kalsom Yusof and Murizah Mohd Zain  “Image Processing Approach for Grading IVF Blastocyst: A State-of-the-Art Review and Future Perspective of Deep Learning-Based Models”  In _Applied Sciences_ 13.2, 2023
  * [21] Athanasios Kallipolitis, Melina Tziomaka, Dimitris Papadopoulos and Ilias Maglogiannis  “Explainable computer vision analysis for embryo selection on blastocyst images”  In _2022 IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI)_ , 2022, pp. 1–4
  * [22] Jihyun Kim, Jaewang Lee and Jin Hyun Jun  “Non-invasive evaluation of embryo quality for the selection of transferable embryos in human in vitro fertilization-embryo transfer”  In _Clinical and Experimental Reproductive Medicine_ 49.4, 2022, pp. 225–238
  * [23] Evangelos Maziotis Konstantinos Sfakianoudis  “Reporting on the Value of Artificial Intelligence in Predicting the Optimal Embryo for Transfer: A Systematic Review including Data Synthesis”  In _Biomedicines_ 10.3, 2022, pp. 697
  * [24] John D Lee and Katrina A See  “Trust in automation: Designing for appropriate reliance”  In _Human factors_ 46.1, 2004, pp. 50–80
  * [25] Ariel Levy, Monica Agrawal, Arvind Satyanarayan and David Sontag  “Assessing the impact of automated suggestions on decision making: Domain experts mediate model errors but take less initiative”  In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ , 2021, pp. 1–13
  * [26] Claudio Michael Louis et al.  “Review of computer vision application in in vitro fertilization: the application of deep learning-based computer vision technology in the world of IVF”  In _Journal of Assisted Reproduction and Genetics_ 38.7, 2021, pp. 1627–1639
  * [27] Scott M. Lundberg and Su-In Lee  “A Unified Approach to Interpreting Model Predictions”  In _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , NIPS’17  Red Hook, NY, USA: Curran Associates Inc., 2017, pp. 4768–4777
  * [28] Kersti Lundin and Hannah Park  “Time-lapse technology for embryo culture and selection”  In _Upsala Journal of Medical Sciences_ 125.2, 2020, pp. 77–84
  * [29] Yasuyuki Mio and Kazuo Maeda  “Time-lapse cinematography of dynamic changes occurring during in vitro development of human embryos”  In _American journal of obstetrics and gynecology_ 199.6, 2008, pp. 660–e1
  * [30] Nahid Nasiri and Poopak Eftekhari-Yazdi  “An overview of the available methods for morphological scoring of pre-implantation embryos in in vitro fertilization”  In _Cell Journal (Yakhteh)_ 16.4, 2015, pp. 392
  * [31] Cecilia Panigutti, Andrea Beretta, Fosca Giannotti and Dino Pedreschi  “Understanding the impact of explanations on advice-taking: a user study for AI-based clinical Decision Support Systems”  In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ , 2022, pp. 1–9
  * [32] Elena Payá et al.  “Automatic characterization of human embryos at day 4 post-insemination from time-lapse imaging using supervised contrastive learning and inductive transfer learning techniques”  In _Computer Methods and Programs in Biomedicine_ 221, 2022
  * [33] Marco Tulio Ribeiro, Sameer Singh and Carlos Guestrin  “”Why Should I Trust You?”: Explaining the Predictions of Any Classifier”  In _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , KDD ’16  San Francisco, California, USA: Association for Computing Machinery, 2016, pp. 1135–1144  DOI: 10.1145/2939672.2939778
  * [34] Vasilija Rolfes et al.  “Artificial Intelligence in Reproductive Medicine – An Ethical Perspective”  In _Geburtshilfe und Frauenheilkunde_ 83.1, 2023, pp. 106–115
  * [35] Zev Rosenwaks  “Artificial intelligence in reproductive medicine: a fleeting concept or the wave of the future?”  In _Fertility and Sterility_ 114.5, 2020, pp. 905
  * [36] Yuki Sawada et al.  “Evaluation of artificial intelligence using time-lapse images of IVF embryos to predict live birth”  In _Reproductive Biomedicine Online_ 43.5, 2021, pp. 843–852
  * [37] Thalia R. Segal and Linda C. Giudice  “Before the beginning: environmental exposures and reproductive and obstetrical outcomes”  In _Fertility and Sterility_ 112.4, 2019, pp. 613–621
  * [38] Ramprasaath R. Selvaraju et al.  “Grad-CAM: Visual Explanations From Deep Networks via Gradient-Based Localization”  In _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , 2017
  * [39] Akriti Sharma et al.  “Explainable Artificial Intelligence for Human Embryo Cell Cleavage Stages Analysis”  In _Proceedings of the 3rd ACM Workshop on Intelligent Cross-Data Analysis and Retrieval_ , 2022, pp. 1–8
  * [40] Sivan Tamir  “Artificial intelligence in human reproduction: charting the ethical debate over AI in IVF”  In _AI and Ethics_ , 2022
  * [41] Prudhvi Thirumalaraju et al.  “Evaluation of deep convolutional neural networks in classifying human embryo images based on their morphological quality”  In _Heliyon_ 7.2 Elsevier, 2021
  * [42] Shanshan Wang et al.  “A Deep Learning Framework Design for Automatic Blastocyst Evaluation With Multifocal Images”  In _IEEE Access_ 9, 2021
  * [43] Chongwei Wu et al.  “LWMA-Net: Light-weighted morphology attention learning for human embryo grading”  In _Computers in Biology and Medicine_ 151, 2022


# A Mining Software Repository Extended Cookbook: Lessons learned from a
literature review

Daniel D. R. Barros<EMAIL_ADDRESS>Federal University of ABCAv.
dos Estados, 5001Santo AndréSão PauloBrazil09210-580 ,  Flávio Horita
<EMAIL_ADDRESS>Federal University of ABCAv. dos Estados, 5001Santo
AndréSão PauloBrazil09210-580 ,  Kanan C. Silva<EMAIL_ADDRESS>Federal University of ABCAv. dos Estados, 5001Santo AndréSão
PauloBrazil09210-580 and  Igor S. Wiese<EMAIL_ADDRESS>Federal University
of Technology - Parana (UTFPR)R. Rosalina Maria Ferreira, 1233Campo
MourãoParanáBrazil87301-899

###### Abstract.

The main purpose of Mining Software Repositories (MSR) is to discover the
latest enhancements and provide an insight into how to make improvements in a
software project. In light of it, this paper updates the MSR findings of the
original MSR Cookbook, by first conducting a systematic mapping study to
elicit and analyze the state-of-the-art, and then proposing an extended
version of the Cookbook. This extended Cookbook was built on four high-level
themes, which were derived from the analysis of a list of 112 selected
studies. Hence, it was used to consolidate the extended Cookbook as a
contribution to practice and research in the following areas by: 1) including
studies published in all available and relevant publication venues; 2)
including and updating recommendations in all four high-level themes, with an
increase of 84% in comments in this study when compared with the original MSR
Cookbook; 3) summarizing the tools employed for each high-level theme; and 4)
providing lessons learned for future studies. Thus, the extended Cookbook
examined in this work can support new research projects, as upgraded
recommendations and the lessons learned are available with the aid of samples
and tools.

††copyright: none

##  1\. Introduction

Software repositories such as version control systems contain a huge amount of
available data that have given rise to an extensive research endeavor in the
field of Mining Software Repositories (MSR) (Poncin et al., 2011). In light of
this, the challenging task of software engineering can be assisted by
extracting knowledge during the software development cycles (Xie et al.,
2009); in particular, using software repositories as the main information
source along with mining techniques to achieve improvements (Akbar and Kak,
2020; ao et al., 2020). Here, MSR contributes to discovering enhancements and
providing an insight into how to make improvements in a software project. MSR
research is already renowned in academia (Hemmati et al., 2013; Luzgin and
Kholod, 2020) and industry (Hassan, 2006; Tripathi et al., 2015) and has
become a key topic in important publication venues and research projects
(Guemes-Pena et al., 2018).

From the time when it began in 2004, the MSR field has been represented at the
MSR Conference (MSRConf) 111http://www.msrconf.org/, this is an important
event that seeks to advance MSR science and practices. Since then, MSR
research has been growing with the advance of technology and research studies
that have provided new techniques, mapping, trends, and a set of best
practices. Some mapping studies and surveys have also focused on MSR
(Chaturvedi et al., 2013; Farias et al., 2016). Among them, the original MSR
Cookbook (Hemmati et al., 2013) has been a seminal paper in which several
authors have surveyed MSRConf between 2004 and 2012 by formulating a set of
best practices and making a number of recommendations. Looking ahead, the
current expansion of the MSR field, suggests there are still many
opportunities to be addressed.

The challenges are twofold. First, owing to the growing interest in the
practice and research on MSR, the literature reviews tend to be out-of-date,
including the main reference-point discussed here, the original MSR Cookbook,
and the mapped works between 2004 and 2012. Second, the previous MSR
literature reviews (Farias et al., 2016; Chaturvedi et al., 2013; Cosentino et
al., 2017; Bavota, 2016) did not cover the increasing number of conferences
and workshops that are currently being provided by the MSR field.

In addressing these challenges, this study seeks to update the MSR research by
first carrying out a systematic mapping study to determine and analyze the
current state-of-the-art, and then proposing an extended Cookbook. This
extended Cookbook focuses on updating the recommendations of the four high-
level themes that can be found in the original MSR Cookbook, as well as
outlining the tools, experiences and achievements listed in the studies of the
lessons learned. When undertaking it, this paper set out with 237 studies from
the digital libraries available and, after applying the selection criteria,
112 papers were chosen. Out of these studies, 492 comments (notes on the
analyzed papers) were categorized in accordance with their subject-areas, and
summarized on the basis of the recommendations. As a result, there was an 84%
increase of comments between this study and the original MSR Cookbook, several
new recommendations were made and some were found to be no longer valid. As
well as confirming the interest of the community and the extent to which MSR
research had spread, the results also showed an increase in the number of
papers that apply mining techniques (e.g., collinearity, text mining,
classification, and prediction), together with recently published tools
designed to encourage new experiments and further analysis.

Within the context of the research conducted by Mendes et al. (Mendes et al.,
2020), this study seeks to make a contribution to the MSR field by updating
the state-of-the-art of the original MSR Cookbook. In addition, it explains
how this topic has evolved in recent years, or in other words, how the
findings have changed over time and the reasons why this study was conducted.
Finally, on the basis of an analysis of new data, this research investigates
further outcomes and expands the original MSR Cookbook.

##  2\. Related works

Hemmati et al. (Hemmati et al., 2013) have classified a list of guidelines
based on the period 2004 - 2013, which includes papers published at MSRConf.
These guidelines, called the original MSR Cookbook, were a compilation of
comments from papers that were categorized through an open coding approach
(Miles and Huberman, 1994). The compilation of the comments resulted in a list
of recommendations that formed the stages of a typical MSR process as follows:
i) to start with, data extraction and preprocessing; ii) the application of
mining techniques; iii) a performance analysis of data and iv) an
interpretation of the results; and finally v) data sharing.

Hence, one of the main research contributions of the original MSR Cookbook was
that it established four high-level themes inspired by the selected papers:
Theme 1: Data acquisition and preparation, Theme 2: Synthesis, Theme 3:
Analysis, and Theme 4: Sharing and replication. Theme 1 focuses on how and
what should be preprocessed and extracted from software artifacts, in
particular, those collected from software repositories. Theme 2 is concerned
with applying mining algorithms such as classification, collinearity, and text
mining, Theme 3 conducts a statistical analysis and makes an evaluation of
Theme 2. Finally, Theme 4 is concerned with sharing data and results. Figure 1
shows these themes and their features.

Figure 1. High-level Themes

There are some systematic mapping studies on MSR. Farias et al. (Farias et
al., 2016) conducted a study based on five editions of MSRConf between 2010
and 2015, and obtained evidence of software analysis, data sources, evaluation
methods, tools, and how the MSR area is improving. Chaturvedi et al.
(Chaturvedi et al., 2013) studied the tools employed in software repositories
by carrying out a systematic mapping from the MSRConf papers published between
2007 and 2012. Unlike previous studies, this work supplies the gap for
researching in new venues, including of course MSR as one of them, as the data
extraction considered all the digital libraries feasible and available.

There are also related works that investigated MSR by only concentrating on
one coding platform. Consentino et al. (Cosentino et al., 2017) mapped studies
in social coding platforms, but only focused on GitHub between 2009 and 2016.
Bavota (Bavota, 2016) conducted a study about mining unstructured data in
software repositories. However, this study has moved far from Consentino et
al. (Cosentino et al., 2017) and Bavota (Bavota, 2016), by not being
restricted to a coding platform or a particular technique.

By carrying out a systematic mapping study, this work upgraded the original
MSR Cookbook recommendations that are still valid and was able to make nine
new recommendations on the basis of the comments in the selected papers.
Moreover, it has mapped new tools that are employed in mining techniques, as
well as listing the lessons learned from the current and previous findings.

##  3\. Methodology

This section describes the research methodology employed in this work. A
systematic mapping study was first carried out based on the procedures defined
by Petersen et al. (Petersen et al., 2015). It describes the goals, search
strategy, data extraction and analysis. Having selected the primary studies,
it follows the procedural stages for building the extended Cookbook based on
the the original MSR Cookbook (Hemmati et al., 2013). We also examine the raw
data extracted, together with all the categorizations defined to encourage
reproducibility (Barros et al., 2021).

###  3.1. Goal

One of the key stages in a systematic mapping study is defining the goal of
the research which is undertaken in this study by adopting the goal definition
model of Basili et al. (V.R. Basili and Rombach, 1994) as shown in Figure 2.

Figure 2. Goal definition for the systematic mapping study

On the basis of the definitions shown in Figure 2, this study raises the
following research question (RQ): What high-level themes do the primary
studies address? The RQ must divide the selected studies into the themes
proposed by the original MSR Cookbook, which are those explained in Section 2.
Furthermore, it would be useful to track the evolutionary pattern of current
trends in research, as well as to identify any gaps and make further
recommendations to provide more extensive knowledge on the topic.

###  3.2. Search strategy

The search strategy comprises four steps: a) selecting the digital libraries
of primary studies; b) defining the selection criteria; c) defining the search
string; and d) establishing the search procedure.

Digital libraries were selected as long as they complied with the following
the conditions: (i) number of indexed and published studies, (ii) they matched
the subject with the main goals of this paper, (iii) availability of
publications, and (iv) the feasibility of finding studies that made proper use
of the advanced search. By conforming to the conditional requirements and the
work of Dybå et al. (Dyba et al., 2007), this study made use of the key
digital libraries for Software Engineering, which were as follows : ACM
Digital Library222https://dl.acm.org/, IEEE
Xplore333https://ieeexplore.ieee.org/, and Wiley Online
Library444https://onlinelibrary.wiley.com/. Springer
Link555https://link.springer.com/ and
ScienceDirect666https://www.sciencedirect.com/ digital libraries, both
recommended by Dybå et al., were not included as they do not allow an
”Abstract” search and advanced filtering. This meant they could not qualify as
they were unable to conform to the condition (iv).

The selection criteria were based on inclusion (IC) and exclusion constraints
(EC). A study is included if: (IC1) The study is related to Software
Repositories and uses Mining techniques. On the other hand, a study is
excluded if: (EC1) The study was published before 2013, as in the case of the
original MSR Cookbook which covered MSRConf studies between 2004 and 2012;
(EC2) The study is not written in English; (EC3) the study is a poster,
tutorial, dissertation, thesis or editorial; (EC4) The study is a previous
version of the same subject as that of this research; (EC5) The study is
secondary or tertiary (e.g., reviews, surveys); (EC6) The study does not deal
with Software Repositories; (EC7) The study is related to Software
Repositories but does not apply any Mining techniques; or (EC8) the study is
duplicated in one or more digital library.

A search string was created with the aim of mapping the MSR state-of-art and
returning studies in an assertive manner to categorize them accordingly.
During the pilot study, we discovered that recent MSR papers are likely to use
Artificial Intelligence or Machine Learning as devices to explain the mining
process in the papers. The search string included the terms ”Mining”,
”Learning”, and ”Artificial Intelligence” with ”Software Repositories” and
their synonyms. Owing to the fact that one of the most important conferences
(MSRConf) has common names in the search string, the abstract was the only
field included in the search within the databases chosen. As in the case of
the pilot study, the metadata that were found ignored the search constraints
and took all of the MSRConf papers into account when only filtering by ”title”
or ”keywords”. The search string chosen for this study was: (”Mining” AND
”Software Repositories”) OR (”Mining” AND ”Software Repository”) OR (”Mining”
AND ”Public Repositories”) OR (”Mining” AND ”Public Repository”) OR
(”Learning” AND ”Software Repositories”) OR (”Learning” AND ”Software
Repository”) OR (”Learning” AND ”Public Repositories”) OR (”Learning” AND
”Public Repository”) OR (”Artificial Intelligence” AND ”Software
Repositories”) OR (”Artificial Intelligence” AND ”Software Repository”) OR
(”Artificial Intelligence” AND ”Public Repositories”) OR (”Artificial
Intelligence” AND ”Public Repository”).

A search procedure was established to select the primary studies and then
answer the research question. This procedure followed three stages: (i)
selecting the studies in the digital libraries chosen; (ii) applying the
selection criteria, subject to the inclusion and exclusion constraints; and,
(iii) extracting and analyzing relevant information from the selected studies.
The search procedure is divided into the following phases: Phase 1 applies the
search string to the digital libraries chosen, following the selection
criteria without any exclusion. Phase 2 filters the raw dataset by means of
the basic qualitative exclusion criteria EC1/2/3/4/5 to get the first primary
dataset. Finally, Phase 3 applies the exclusion criteria EC6/7/8, (which are
regarded as the most accurate), and the results obtained from the final
dataset of the selected studies.

###  3.3. Data Extraction

The data extraction process collects each item of Extracted Information (EI)
from the digital libraries as follows: (EI1) the affiliation of the authors,
given by the author´s name, university and country; (EI2) the main details of
the study , including the title, publishing date, and DOI [Digital Object
Identifier]; (EI3) Type of material, whether the study is an article, a
research article or a tool; (EI4) Applied Mining Technique, whether the
applied technique is a Supervised Mining Technique, Unsupervised Mining
Technique, Evaluative Analysis of Mining, Machine Learning, Deep Learning, or
any other technique or heuristic chosen; (EI5) the theme addressed by the
study and categorized by this study.

###  3.4. Data Analysis

After the search procedure, there was an in-depth examination of the selected
papers to derive the recommendations per theme. Here, we followed the original
MSR Cookbook by interpreting a recommendation of a theme as representing a
consolidation of key features (called comments) found in the list of papers.
This involved first carrying out open coding and axial coding procedures
(Strauss and Corbin, 2007) to qualitatively analyze the topic that resulted in
comments, which were later reviewed and incorporated in the recommendations.
Owing to the number of extracted comments, we again followed the original MSR
Cookbook and decided that relevant recommendations would be those that were
found in at least two selected papers. Two researchers worked together, where
each one worked on a subset of the recommendations extracted from the selected
studies. The researchers discussed all classifications until they reached a
consensus about the results. Another two experienced researchers got the
subsets processed to review them and refine the mapping between the theme and
recommendations. In addition, we held a weekly meeting with the whole group of
co-authors to discuss and adjust results until we reached an agreement. This
study did not calculate the Inter-rater reliability (IRR) as the purpose of
mapping recommendations was to find categories (themes) rather than classify
the data according to emerged codes (McDonald et al., 2019).

##  4\. Results

This section first characterizes the results of this study, and later
introduces the recommendations of the themes.

###  4.1. Studies characterization

The data extraction was carried out between September and November 2020. With
regard to the procedures involved in the search and selection, in Phase 1 the
search returned 237 studies - the majority obtained by IEEE, followed by ACM
and Wiley. Phase 2 applied EC1/2/3/4/5 exclusion criteria with an outcome of
187 studies, and the percentage of returned studies remained about the same,
i.e., IEEE returned most of the studies, followed by ACM and Wiley. Lastly,
the remaining exclusion criteria (i.e., 6, 7, and 8) were applied in Phase 3
with a final list of 112 selected studies and, although there was less
difference between IEEE and ACM, IEEE was the data library with the most
studies, followed by ACM and Wiley. Figure 3 outlines the search and selection
procedure per Phase and shows the number of selected papers and comments per
the theme.

Figure 3. Searching and selecting procedure of studies

Apart from the details of the papers, obtained from the 112 selected studies,
the data extraction process enabled a considerable number of comments to be
created for each theme (492 in total) that later culminated in
recommendations.

With regard to the venues of the selected papers, the MSRConf was the
conference which had most published papers, followed by The International
Conference on Software Engineering (ICSE)777http://www.icse-conferences.org/,
The International Conference on Software Maintenance and Evolution
(ICSME)888https://conferences.computer.org/icsm/, and The Journal of Software:
Evolution and Process (JSEP)
999https://onlinelibrary.wiley.com/journal/20477481. This demonstrates that
our mapping study was able to identify papers from significant publication
venues in the area of software engineering. Figure 4 shows a graphical
representation of the distribution of papers among the publication venues.

Figure 4. Selected studies from each venue

###  4.2. Themes Recommendations

This section outlines the updated and new recommendations for each high-level
theme of the extended MSR Cookbook. The number of comments that were found in
the selected papers of this study is also shown as a means of demonstrating
the significance of the recommendation itself, as well as how it fits the
theme.

In the next sections, the following scheme is adopted to introduce the
recommendations and identify each of the themes. The first letter denotes its
theme – ”D” for Data Extraction and Modelling (Theme 1), ”S” for Synthesis
(Theme 2), ”A” for Analysis (Theme 3), and ”R” for Results (Theme 4). While
the second letter indicates whether the recommendation belongs to the original
MSR Cookbook by using the letter ”O” (Old) or is something new - through the
letter ”N” (New). These two letters are combined with a sequential number
which is used to order the recommendation within a theme (i.e., DO1 should be
understood as the first recommendation of Theme 1 and an updated version of
the original MSR Cookbook). The recommendations shaded with a gray background
and a solid line in a frame box were those found in the original MSR Cookbook,
while the ones shaded in green with a dashed line represent the new additions
made to this study. As this study is an extension, the original MSR Cookbook
recommendations that did not appear in this study will be only shown in the
Figure 6 as past references.

####  4.2.1. Theme 1: Data Extraction and Modeling

This theme was an important topic in past research studies (Hemmati et al.,
2013). However, recent study trends have been more concerned with applying
techniques to get relevant information, rather than extracting and preparing
data for usage. This can be explained by the technological maturity of the MSR
research tools available for extracting and preparing data for the next themes
(2-4) (Meurice and Cleve, 2014). In other words, our findings suggest that
there are already mature tools that are good enough to perform data extraction
and modeling, and hence there has been a decline in number of published papers
on Theme 1 for some time.

Source Control Management (SCM) is one of the key features of MSR. The domain
knowledge of SCM is essential for extraction and preparing data mining tools,
as well as the heuristics for scheduling data gathering (German et al., 2015),
and enhancing existing process capabilities (Gupta, 2014). This is established
in the following recommendation.

Source code mining can be either a straightforward or costly process since it
requires knowledge and time to extract relevant information, which thus
influences the choice of code extraction method. Regular expressions for
mining data in large software repositories can be, for instance, an
alternative to reducing data collection significantly (Awang Abu Bakar, 2014).
In contrast, tools and abstractions can hide the source of complexity by
turning a complex process of code mining into a simple query mechanism (Tiwari
et al., 2016).

The link between software repositories and external data sources
(Heterogeneous Data Systems (HDS)) represents a complex feature of MSR (Wilder
et al., 2016). Domain-Specific Languages (DSLs) can act as an interface to
access and connect HDS into the software repository mining (Wilder et al.,
2016), by reducing the complexity of MSR and providing new insights. However,
owing to domain complexity, the construction of DSL can be a hard process.
Some studies have addressed this issue by conducting a Feature-Oriented Domain
Analysis to overcome the challenges more easily (Huang et al., 2013). Thus,
this work makes the following remarkable recommendation in Theme 1.

There is a common sense view that the history of database systems helps our
understanding of mining approaches, including MSR. Software development has
become increasingly complex, by aggregating different sources of information
such as defect tracking systems. Hence, some studies have suggested ways to
incorporate additional information into the MSR process, such as adding issue
tracking for databases (Steinbeck, 2020) or even providing a dataset for bug
mining information (Lamkanfi et al., 2013). As the way this database is
evolving has the potential to contribute to MSR, this work also makes a
supplementary new recommendation on Theme 1.

####  4.2.2. Theme 2: Synthesis

This theme focuses on the application of mining techniques, such as clustering
and regression, and papers about it have been continuously published in recent
years. Our study does not show a significant reduction in publications in
Theme 2. Likewise, there remains a great interest in the practice and
research, which is encouraging new studies to be undertaken.

One way that is recommended to improve the chances of obtaining a decent
result in a mining algorithm, is to preprocess data properly before the mining
task. This is why Theme 1 comes before Theme 2. Hence, one useful
recommendation in this theme is to tune parameters and perform a sensitivity
analysis, as they can improve the modeling process. It is also possible to
find works doing tasks such as comparing text-based and dependence-based
approaches for determining the origins of bugs (Davies et al., 2014).

In MSR, a combination of classifiers (ensemble of classifiers) is likely to
exceed the accuracy of a single classifier by combining several of them. There
are works in the literature that adopt this approach and provide a taxonomy
for ensemble-based methods to overcome problems of an imbalanced
classification (Galar et al., 2012). The combination of results can be
visualized in some MSR tools (Barros et al., 2020), where the main indicators
(such as accuracy and prediction) can be compared, and thus allow better
decision-making. This is a key factor and our study makes a new recommendation
on this theme.

Artificial Intelligence (AI) is definitely a trend in MSR papers (Luzgin and
Kholod, 2020) and the performance standard of its solutions is related to how
good the data source is. Since it can be expected that there will be plenty of
data to work with, rich domain knowledge is essential for AI solutions in
terms of their performance and evolution. A common comment in papers that
adopt this approach is the relationship between rich domain knowledge and
success, such as data-driven applications (Misra et al., 2017) or a framework
for the retrieval of information (Parizy et al., 2014). Strategies to keep
software projects accurate and cohesive, such as improving software
maintenance (Gupta et al., 2017) or defect prediction (Abdalkareem et al.,
2020), are linked to rich domain knowledge as well.

The purpose of feature selection algorithms is to obtain the most suitable
subset from the source that fits the prediction model best, by removing
redundant and irrelevant data, to ensure the best possible prediction (Guyon
and Elisseeff, 2003). Feature selection outcomes show impressive figures for
prediction such as entropy and gain, and there are cases where the merging of
feature selection and other techniques proved to be successful, such as
learning-based approaches (Yue et al., 2018) or the longest common sequences
(LCS)(Neysiani and Babamir, 2019). However, applying feature selection
filtering to just the highly ranked features does not necessarily improve the
performance of the measures (Abozeed et al., 2020).

Despite the fact that when combined in a clever manner, feature selection can
definitely enhance the prediction models, when compared with recently
extracted features, historically-based features can make a significant
improvement. Learning-based (Yue et al., 2018; Abdalkareem et al., 2020) and
ranking (Ye, 2019) approaches are outperforming previous models that were not
based on historical data.

Deep Learning has achieved astonishing results in many areas of research,
including MSR, and progress is underway (Abozeed et al., 2020). In fact,
studies show that Deep Learning has been obtaining a higher degree of accuracy
than other techniques. For instance, the automatic categorization of software
projects through MSR (Nguyen and Nguyen, 2017), or even bug prediction models
(Abozeed et al., 2020) are relying on Deep Learning as the main tool to obtain
good results.

####  4.2.3. Theme 3: Analysis

In MSR, the analysis of results is definitely one of the most important
phases, since the conclusions are the main output. For this reason, Theme 3 is
still one of the subjects which has attracted the most attention from
researchers, and been the source of most of the comments found in this study.

Mining algorithms and processes have relied on automated tasks, where the
means of getting the best result can show wrong figures or biased results. In
order to address this challenge, manual verification of all outputs should be
presented in an analysis of MSR approaches. For instance, a manual analysis of
the relationship between refactoring and bugs should be done, as refactoring
always encourage bug-fixing activities (Penta et al., 2020). However, the task
of manually checking all of the outputs can be very demanding and become
impossible to achieve. Some automated sampling and verification strategies are
suggested, such as a machine learning-based model which includes efficient
preprocessing together with highly accurate prediction and a low level of
errors (Tariq et al., 2020).

The evolution of data mining techniques can be seen in recent works, and this
has led this research to conduct a new systematic mapping study as one of its
goals. Before applying sophisticated mining approaches, some studies have
found simple and efficient ways to check hypotheses about a set of data.
Others can answer simple questions through a correlation analysis before
taking a further step into mining techniques, such as determining whether bugs
foreshadow vulnerabilities (Camilo et al., 2015).

Precision and recall as the main measures of mining approaches have been
adopted by researchers and practitioners with a strong label of trust. For
this reason, the majority of papers include precision and recall measures to
ensure success in their strategies and experiments. Nonetheless, in certain
contexts, precision and recall may not be enough, and some studies use
F1-Score, Recall (Barros et al., 2020), or even techniques such as PCA and
Dynamic Weighting (Ali et al., 2013).

In a mining approach, statistical analyses are extremely important to
understand state-of-the-art metrics and to determine why some metrics are
relied on more than others. If conclusions are only drawn on the basis of
statistical analyses, this can blur a proper understanding of the data mining
system. Practical differences, based on empirical evidence such as the impact
of refactoring on the relationship between quality attributes and design
metrics, can lead to more assertive conclusions rather than simply providing a
complex statistical analysis (AlOmar et al., 2019).

Managing resources in software projects is one of the most challenging tasks.
Planning, scheduling, and allocating resources can be very tricky, especially
if the estimations are only based on random heuristics or guesses. However,
the prediction of software effort can really help in managing resources in
software projects, which has been demonstrated in studies such as those that
entail selecting the right predictors from huge software repositories in order
to ensure an accurate software effort estimation (Tariq et al., 2020).

####  4.2.4. Theme 4: Results

This theme corresponds to results where sharing data and outcomes permit a
further validation and replication of the studies. MSR studies generally do
not possess enough evidence to allow their individual results to be applied in
other studies, as the context or data sources might be very different. For
this reason, this theme, which is often ignored by most researchers, is very
important in the MSR field to keep the scheduling and replication of these
studies alive. Oddly enough, Theme 4 still has very few publications and
although there is an increase of Theme 4 papers in this study, they are far
from reaching their full potential.

The feasibility of Theme 4 depends on sharing studies data, tools, and
techniques to allow reproducibility. Historically, the replication of studies
in MSR has not been straightforward. Here, in recent years there have been a
number of publications that have enhanced the datasets with cleaning and
preprocessing text data (Lazar et al., 2014), along with frameworks for a
replication analysis of software projects (Ghezzi and Gall, 2013).

As long as replication studies are available, it should be a straightforward
task to compare mining techniques or assess findings in many projects. An
example of replication is the datasets available from studies, where bugs
extracted from tracking systems are mined and compared with the sources of
different systems (Lazar et al., 2014).

##  5\. Discussions

###  5.1. Implications for literature and practice

The research contributions made by this work include the following: 1) an
update of the MSR field, carried out by a systematic mapping study; 2) a wider
coverage of MSR studies; and 3) an extended Cookbook for updating
recommendations, making tools available, and providing lessons learned for
future research. Briefly, an update of MSR research is needed as it is an area
that is constantly growing. To address this, this work reviews the existing
state-of-the-art and draws attention to the challenges facing researchers and
practitioners. As the spread of MSR research has now become a reality, a wider
coverage of studies is absolutely indispensable. Here, rather than restricting
our review to the MSRConf as the works in the literature have done, this study
took into account all of the digital libraries available for mapping,
including several conferences, workshops, and new works in the MSR field.
Figure 4 summarizes the main publishing venues, where major international
conferences obtained most of their selected studies, which underlines the
importance and heterogeneity of this study. Owing to the heterogeneity of MSR
studies, grouping these studies is essential. This work extended and upgraded
the the original MSR Cookbook to divide the studies into four high-level
themes. This is designed to ensure that new research is conducted properly and
keeps the focus on MSR as a development field. The measures taken are
discussed in further detail in the next sections.

###  5.2. RQ analysis: In what high-level theme are the primary studies
addressed?

Theme 2 obtained the majority of papers, followed by Theme 3, Theme 1, and
finally Theme 4. In fact, the systematic mapping study revealed that the
papers on Theme 1 are declining and may reach an almost zero output over time,
which suggests that data extraction and modeling is a mature subject in the
MSR field. Another sign of MSR maturity is the increase in the number of Theme
4 papers, which are still under-represented relative to the others, but has
increased when compared with the original MSR Cookbook (Hemmati et al., 2013).

Figure 5 plots the evolving pattern of selected studies on the themes over
time. Even though a few papers were found, the number of studies in Theme 1 is
decreasing. In contrast, Theme 2 has been growing in recent years, especially
since 2016, which suggests that Theme 2 has become a trend in the MSR field
since then. This might be due to the fact that data mining applications have
space in the literature and, as the MSR field evolves, AI is becoming a
reality on MSR and might be the next growing subject in the years ahead. On
the opposite side, research studies with regard to Theme 3 have decreased in
the last few years, which suggests that an analysis of implemented techniques
is no longer common. Last but not least, the number of publications on Theme 4
is increasing.

Figure 5. Evolution of selected studies themes

####  5.2.1. Evolution of recommendations

With regard to the evolving pattern of recommendations, when this study is
compared with the original MSR Cookbook, Figure 6 shows how this has taken
place. The relative position of the original recommendations is kept in the
diagram to allow a graphical comparison to be made, even if they were not
found in either previous studies or in this work. When our findings were
compared with the original MSR Cookbook, there was a significant decrease in
recommendations in Theme 1 that can be explained by the fall in the number of
papers in recent years. Even though there remained previous recommendations
about SCM and code extraction, two new recommendations appeared which showed
how MSR had evolved. The first (DN1) suggested ways of how to perform MSR more
easily, while the second (DN2) focused on how historical data can lead to the
evolution. Theme 2 kept the recommendation of the original MSR Cookbook in the
area of tuning parameters and conducting a sensitivity analysis (SO2), and
there was a significant increase in this recommendation. The new
recommendations this study found in Theme 2 showed the combination of
techniques required to achieve better results, as well as those related to AI
techniques, i.e., a richer domain knowledge; feature selection; historically-
based features; and deep learning models. Theme 3 was given a new
recommendation, related to the prediction of software effort, and kept the
previous ones. Finally, the single previous recommendation of Theme 4 was kept
as well, with an increase in the number of its comments, although a new
recommendation about the replication of studies was also identified.

Figure 6. Evolution of themes recommendations

###  5.3. The Tools of MSR

MSR is also driven by the creation of tools to assist in carrying out tasks
and improving correlated processes. Spadini et al. (Spadini et al., 2018)
created the tool PyDriller, a Python framework that makes it easier to extract
data from mining Git repositories. The Boa (Dyer et al., 2013) tool provides
snapshots of Java projects in GitHub and SourceForge and is designed to reduce
the complexity of mining. Other tools are concerned with helping to integrate
software development with other areas, such as the gthbmining (Barros et al.,
2020) tool where the main goal is to discover DevOps trends in public
repositories. There are also papers focused on giving advice on how MSR should
be undertaken. Kalliamvakou et al. (Kalliamvakou et al., 2014) performed an
empirical study about public repositories and how contributors should interact
with them.

The tools obtained from the selected studies are categorized as well, to
strengthen the applicability of the extended Cookbook. Although categorizing
tools can be very difficult, the main purpose of this tool mapping is to help
the MSR community to make the best choice to stimulate new experiments and
theories, by allowing researchers more time to devote to making advances in
science, instead of creating new tools that have been designed and published
before. Table 1 displays the tools collected in the selected studies grouped
by themes and suggested categories, as well as the number of studies that were
found. Further details can be found at the replication package (Barros et al.,
2021) (sheet ”Tools”).

Table 1. Summary of Tools used in each Theme

###  5.4. Lessons learned

The results and mapping of the evolution of recommendations allowed this study
to state what lessons were learned from the existing literature. As our
findings showed that the themes and upgraded recommendations still coincided
with the four high-level themes of a typical MSR process, the lessons learned
were also grouped by themes. Finally, the findings from related works are
provided, as a means of guiding future extensions of this Cookbook.

Theme 1: Noise is still a challenge in data extraction and modeling. Noises
can be found not only in the source code (as pointed out by Recommendation DO1
(Guilardi et al., 2020)), but all kinds of software repositories (i.e. version
control systems, bug trackers, and communication channels). Thus, one of the
first steps of data extraction should be cleaning up data to prevent noises by
using heuristics and taking into account the results of a cost analysis
(Recommendation DO2 (Bakar, 2014)); The complexity of the procedure of MSR can
be reduced. The MSR process of data extraction and modeling can be made easier
by using APIs and tools or even drawing on the ideas available in the
literature and based on practice. There are tools based on this mapping study,
as well as ideas and shortcuts for addressing complexity properly
(Recommendation DN1 (Wilder et al., 2016)); The evolution of database
technology can help provide a software enhancement; and, external data sources
such as bug trackers and mail lists should be included in the data extraction
and modeling processes (recommendation DN2 (Lamkanfi et al., 2013)).

Theme 2: If you take care of the modeling process, you can achieve improved
performance measures. Recommendation SO2 suggested measures like tuning
parameters and performing sensitivity analyses to improve the modeling process
(Ramsauer et al., 2019). Unsurprisingly, Recommendation SN5 matched high-
performance deep learning models with efficient modeling processes (White et
al., 2015), and achieved better performance measures; A richer domain
knowledge is a key ingredient for success in MSR. Recommendation SN2 found
several samples where domain knowledge is the equivalent of MSR success (Misra
et al., 2017; Parizy et al., 2014). Moreover, domain knowledge helps feature
selection in several ways, e.g., the SN3 Recommendation demonstrated that only
relying on feature selection is not enough in all the cases that require
domain knowledge (Abozeed et al., 2020), and the SN4 Recommendation showed
that historical and well-known features outperform new data (Yue et al.,
2018); A combination of results should prevail when employing single
techniques. Recommendation SN1 showed that a combination of classifiers (e.g.,
ensembles) is more assertive than a single technique or any combination of two
techniques (Dit et al., 2014).

Theme 3: Before reaching any conclusions, manual checking and an understanding
of the practical differences between all the outputs are encouraged. Hence,
Recommendations AO1 (Dyer et al., 2013; Penta et al., 2020) and AO4 (AlOmar et
al., 2019) are still valid, and there was an increase of comments in both
papers; The context must be taken into account before selecting the
appropriate analysis indicators. Although the AO2 Recommendation showed that a
correlation analysis is still used (Camilo et al., 2015), the increase in the
number of AO3 Recommendations confirmed that context should be the driving-
force for selecting the appropriate analysis indicators (Ali et al., 2013).
Different classification strategies should be stimulated like Matthews
Correlation Coefficient (MCC) (Tantithamthavorn et al., 2019) or more tests
employed (e.g., the Scott-Knott test) to evaluate the statistical ranks of
classifiers (Ghotra et al., 2017); Resources management can be supported by
accurate software predictions. As suggested by Recommendation AN1 (Tariq et
al., 2020), resource planning, scheduling, and allocation can be supported by
accurate software predictions.

Theme 4: Sharing data and replicated studies can help make advances in
science. Sharing data, tools, and techniques definitely helps science make
progress as the RO1 Recommendation confirmed (Ghezzi and Gall, 2013).
Additionally, a replication of studies has helped science as well, as the RN1
recommendation demonstrated in its comparisons and assessment of advances in
data mining techniques (Lazar et al., 2014).

Related works: MSR mapping studies should include all the relevant venues.
Although previous MSR mapping studies made a significant contribution to the
literature, they only covered MSRConf papers (Hemmati et al., 2013; Farias et
al., 2016; Chaturvedi et al., 2013). In contrast this paper showed that other
relevant venues (i.e. ICSE, ICSME) have been publishing important papers and,
thus, should be included in future MSR mappings; Include as many MSR
techniques as you can while mapping. Instead of being studies based on a
single coding platform (Cosentino et al., 2017) or mapping studies with a few
techniques (Bavota, 2016), a good MSR mapping should include as many
techniques as possible so that it can provide recommendations and guidelines
to advance science.

##  6\. Threats to validity

Construction validity. This work attempted to select the best digital
libraries available for collecting research data, following the
recommendations made by Dybå et al. (Dyba et al., 2007). However, owing to the
constraints imposed by the search criteria, the digital libraries Springer
Link and ScienceDirect were excluded as, at the time of this research, they
did not provide an advanced search system based on the criteria defined in
Section 3.2. With a view to mitigating this threat, the search process was
carried out twice since there was a delay of two months between the findings.
Additionally, an exploratory search was conducted in the excluded databases
along with the analysis of the returned papers.

Internal validity. Although this paper adopted a systematic mapping strategy,
the inclusion and exclusion criteria were selected on the basis of the
authors’ own judgment. This means that some studies might have been selected
incorrectly. However, an attempt was made to attenuate this bias in the
systematic mapping protocol, by having an exhaustive discussion among the
researchers to ensure there was a common understanding. Moreover, this study
followed open coding and axial coding procedures, which meant that the data
extraction and findings were subject to a peer-review evaluation.

External validity. This paper performed a systematic mapping of MSR studies,
regardless of the venue. This implies that the study may have overlooked some
important studies that did not match the data extraction process but were only
published in MSR venues. Thus, it is not possible to generalize the
conclusions for all the MSR venues, such as MRSConf. However, the outcomes of
this study allowed us to gain an insight into the importance of this topic
from a wider range of relevant venues, e.g., ICSE, ESEM, and SANER.

##  7\. Conclusion

MSR has grown into one of the key areas of Software Engineering research and
is definitely a hot topic for new studies. The MSR protagonist has reflected
on important venues that are only devoted to dealing with MSR research, and
this has encouraged researchers and practitioners to enhance the quality and
extend the range of the field. Given the importance of MSR, this study decided
to update the research on MSR, first by carrying out a systematic mapping
study to examine and throw light on the current state-of-the-art, and then
proposing an extended Cookbook that upgraded the recommendations, tools
available, and number of lessons learned. The outcomes of 112 selected studies
confirmed that MSR is still growing, and showed there has been an increase of
84% in the comments in this study when compared with the number in the
original MSR Cookbook. This study investigated the applied techniques of the
selected studies, and found the most promising ones were Machine and Deep
Learning. Future lines of research should focus on keeping the Cookbook up–to-
date, and enhancing the mapping studies so that future trends can be
understood, as well as demonstrating how MSR studies can lead to new
recommendations and further discussion.

###### Acknowledgements.

This work is supported by PROPG and DIRPPG from UTFPR, as well as National
Council for Scientific and Technological Development (CNPq) (Grant no.
437937/2018-6).

## References

  * (1)
  * Abdalkareem et al. (2020) R. Abdalkareem, S. Mujahid, and E. Shihab. 2020\.  A Machine Learning Approach to Improve the Detection of CI Skip Commits.  _IEEE Transactions on Software Engineering_ (2020), 1–1.  https://doi.org/10.1109/TSE.2020.2967380
  * Abozeed et al. (2020) S. M. Abozeed, M. Y. ElNainay, S. A. Fouad, and M. S. Abougabal. 2020.  Software Bug Prediction Employing Feature Selection and Deep Learning. In _2019 International Conference on Advances in the Emerging Computing Technologies (AECT)_. IEEE.  https://doi.org/10.1109/aect47998.2020.9194215
  * Akbar and Kak (2020) S. A. Akbar and A. C. Kak. 2020.  A Large-Scale Comparative Evaluation of IR-Based Tools for Bug Localization. In _Proceedings of the 17th International Conference on Mining Software Repositories_ (Seoul, Republic of Korea) _(MSR ’20)_. Association for Computing Machinery, New York, NY, USA, 21–31.  https://doi.org/10.1145/3379597.3387474
  * Ali et al. (2013) N. Ali, Y. Guéhéneuc, and G. Antoniol. 2013\.  Trustrace: Mining Software Repositories to Improve the Accuracy of Requirement Traceability Links.  _IEEE Transactions on Software Engineering_ 39, 5 (2013), 725–741.  https://doi.org/10.1109/TSE.2012.71
  * AlOmar et al. (2019) E. A. AlOmar, M. W. Mkaouer, A. Ouni, and M. Kessentini. 2019.  On the Impact of Refactoring on the Relationship between Quality Attributes and Design Metrics. In _2019 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)_. 1–11.  https://doi.org/10.1109/ESEM.2019.8870177
  * ao et al. (2020) M. Paix ao, A. Uchôa, A. C. Bibiano, D. Oliveira, A. Garcia, J.Krinke, and E. Arvonio. 2020.  Behind the Intents: An In-Depth Empirical Study on Software Refactoring in Modern Code Review. In _Proceedings of the 17th International Conference on Mining Software Repositories_ (Seoul, Republic of Korea) _(MSR ’20)_. Association for Computing Machinery, New York, NY, USA, 125–136.  https://doi.org/10.1145/3379597.3387475
  * Awang Abu Bakar (2014) N. S. Awang Abu Bakar. 2014\.  Using regular expressions for mining data in large software repositories. In _The 5th International Conference on Information and Communication Technology for The Muslim World (ICT4M)_. 1–6.  https://doi.org/10.1109/ICT4M.2014.7020649
  * Bakar (2014) Normi Sham Awang Abu Bakar. 2014\.  Using regular expressions for mining data in large software repositories. In _The 5th International Conference on Information and Communication Technology for The Muslim World (ICT4M)_. IEEE.  https://doi.org/10.1109/ict4m.2014.7020649
  * Barros et al. (2021) D. Barros, K. Silva, F. Horita, and I. Wiese. 2021\.  _Extended Cookbook Data_.  https://doi.org/10.5281/zenodo.5104942
  * Barros et al. (2020) D. D. R. Barros, F. Horita, and D. G. Fantinato. 2020\.  Data mining tool to discover DevOps trends from public repositories: Predicting Release Candidates with gthbmining.rc. In _In Proceedings of the 34th Brazilian Symposium on Software Engineering (SBES ’20)._ (Natal). SBC, Porto Alegre, RS, Brasil.  https://doi.org/10.1145/3422392.3422501
  * Bavota (2016) G. Bavota. 2016\.  Mining Unstructured Data in Software Repositories: Current and Future Trends. In _2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER)_. IEEE.  https://doi.org/10.1109/saner.2016.47
  * Camilo et al. (2015) F. Camilo, A. Meneely, and M. Nagappan. 2015\.  Do Bugs Foreshadow Vulnerabilities? A Study of the Chromium Project. In _2015 IEEE/ACM 12th Working Conference on Mining Software Repositories_. 269–279.  https://doi.org/10.1109/MSR.2015.32
  * Chaturvedi et al. (2013) K.K. Chaturvedi, V.B. Sing, and P. Singh. 2013.  Tools in Mining Software Repositories. In _2013 13th International Conference on Computational Science and Its Applications_. IEEE.  https://doi.org/10.1109/iccsa.2013.22
  * Cosentino et al. (2017) V. Cosentino, J. L. Cánovas Izquierdo, and J. Cabot. 2017.  A Systematic Mapping Study of Software Development With GitHub.  _IEEE Access_ 5 (2017), 7173–7192.  https://doi.org/10.1109/ACCESS.2017.2682323
  * Davies et al. (2014) S. Davies, M. Roper, and M. Wood. 2014.  Comparing text-based and dependence-based approaches for determining the origins of bugs.  _Journal of Software: Evolution and Process_ 26, 1 (2014), 107–139.  https://doi.org/10.1002/smr.1619 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1002/smr.1619
  * Dit et al. (2014) B. Dit, M. Wagner, S. Wen, W. Wang, M. Linares-Vásquez, D. Poshyvanyk, and H. Kagdi. 2014.  ImpactMiner: A Tool for Change Impact Analysis. In _Companion Proceedings of the 36th International Conference on Software Engineering_. Association for Computing Machinery, New York, NY, USA, 540–543.  https://doi.org/10.1145/2591062.2591064
  * Dyba et al. (2007) T. Dyba, T. Dingsoyr, and G. K. Hanssen. 2007.  Applying Systematic Reviews to Diverse Study Types: An Experience Report. In _First International Symposium on Empirical Software Engineering and Measurement (ESEM 2007)_. 225–234.  https://doi.org/10.1109/ESEM.2007.59
  * Dyer et al. (2013) R. Dyer, H. A. Nguyen, H. Rajan, and T. N. Nguyen. 2013\.  Boa: A language and infrastructure for analyzing ultra-large-scale software repositories. In _Proceedings of the 35th International Conference on Software Engineering_ _(ICSE)_. 422–431.  https://doi.org/10.1109/ICSE.2013.6606588
  * Farias et al. (2016) M. A. F. Farias, R. Novais, M. C. Júnior, L. P. S. Carvalho, M. Mendonça, and R. O. Spínola. 2016.  A Systematic Mapping Study on Mining Software Repositories. In _Proceedings of the 31st Annual ACM Symposium on Applied Computing_ (Pisa, Italy) _(SAC ’16)_. Association for Computing Machinery, New York, NY, USA, 1472–1479.  https://doi.org/10.1145/2851613.2851786
  * Galar et al. (2012) M. Galar, A. Fernandez, E. Barrenechea, H. Bustince, and F. Herrera. 2012\.  A Review on Ensembles for the Class Imbalance Problem: Bagging-, Boosting-, and Hybrid-Based Approaches.  _IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews)_ 42, 4 (2012), 463–484.  https://doi.org/10.1109/TSMCC.2011.2161285
  * German et al. (2015) D. M. German, B. Adams, and A. E. Hassan. 2015\.  A Dataset of the Activity of the Git Super-repository of Linux in 2012. In _2015 IEEE/ACM 12th Working Conference on Mining Software Repositories_. 470–473.  https://doi.org/10.1109/MSR.2015.66
  * Ghezzi and Gall (2013) G. Ghezzi and H. C. Gall. 2013.  Replicating mining studies with SOFAS. In _2013 10th Working Conference on Mining Software Repositories (MSR)_. 363–372.  https://doi.org/10.1109/MSR.2013.6624050
  * Ghotra et al. (2017) B. Ghotra, S. McIntosh, and A. E. Hassan. 2017\.  A Large-Scale Study of the Impact of Feature Selection Techniques on Defect Classification Models. In _2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR)_. 146–157.  https://doi.org/10.1109/MSR.2017.18
  * Guemes-Pena et al. (2018) D. Guemes-Pena, C. Lopez-Nozal, R. Marticorena-Sanchez, and J. Maudes-Raedo. 2018.  Emerging topics in mining software repositories.  _Progress in Artificial Intelligence_ 7, 3 (2018), 237–247.  https://doi.org/10.1007/s13748-018-0147-7
  * Guilardi et al. (2020) Demetrio Guilardi, Jalves Nicácio, Bianca M. Napoleão, and Fabio Petrillo. 2020.  AndroidPropTracker. In _Proceedings of the IEEE/ACM 7th International Conference on Mobile Software Engineering and Systems_. ACM.
  * Gupta (2014) M. Gupta. 2014.  Nirikshan: Process Mining Software Repositories to Identify Inefficiencies, Imperfections, and Enhance Existing Process Capabilities. In _Companion Proceedings of the 36th International Conference on Software Engineering_ (Hyderabad, India) _(ICSE Companion 2014)_. Association for Computing Machinery, New York, NY, USA, 658–661.  https://doi.org/10.1145/2591062.2591080
  * Gupta et al. (2017) M. Gupta, A. Serebrenik, and P. Jalote. 2017.  Improving Software Maintenance Using Process Mining and Predictive Analytics. In _2017 IEEE International Conference on Software Maintenance and Evolution (ICSME)_. 681–686.  https://doi.org/10.1109/ICSME.2017.39
  * Guyon and Elisseeff (2003) Isabelle Guyon and André Elisseeff. 2003.  An introduction to variable and feature selection.  _Journal of machine learning research_ 3, Mar (2003), 1157–1182.
  * Hassan (2006) A. E. Hassan. 2006\.  Mining Software Repositories to Assist Developers and Support Managers. In _2006 22nd IEEE International Conference on Software Maintenance_. 339–342.  https://doi.org/10.1109/ICSM.2006.38
  * Hemmati et al. (2013) H. Hemmati, S. Nadi, O. Baysal, O. Kononenko, W. Wang, R. Holmes, and M. W. Godfrey. 2013.  The MSR Cookbook: Mining a decade of research. In _2013 10th Working Conference on Mining Software Repositories (MSR)_. 343–352.  https://doi.org/10.1109/MSR.2013.6624048
  * Huang et al. (2013) C. Huang, Y. Kamei, K. Yamashita, and N. Ubayashi. 2013\.  Using Alloy to Support Feature-Based DSL Construction for Mining Software Repositories. In _Proceedings of the 17th International Software Product Line Conference Co-Located Workshops_ (Tokyo, Japan) _(SPLC ’13 Workshops)_. Association for Computing Machinery, New York, NY, USA, 86–89.  https://doi.org/10.1145/2499777.2500714
  * Kalliamvakou et al. (2014) E. Kalliamvakou, G. Gousios, K. Blincoe, L. Singer, D. M. German, and D. Damian. 2014\.  The Promises and Perils of Mining GitHub. In _Proceedings of the 11th Working Conference on Mining Software Repositories_ _(MSR)_. 92––101.  https://doi.org/10.1145/2597073.2597074
  * Lamkanfi et al. (2013) A. Lamkanfi, J. Pérez, and S. Demeyer. 2013\.  The Eclipse and Mozilla defect tracking dataset: A genuine dataset for mining bug information. In _2013 10th Working Conference on Mining Software Repositories (MSR)_. 203–206.  https://doi.org/10.1109/MSR.2013.6624028
  * Lazar et al. (2014) A. Lazar, S. Ritchey, and B. Sharif. 2014.  Generating Duplicate Bug Datasets. In _Proceedings of the 11th Working Conference on Mining Software Repositories_ (Hyderabad, India) _(MSR 2014)_. Association for Computing Machinery, New York, NY, USA, 392–395.  https://doi.org/10.1145/2597073.2597128
  * Luzgin and Kholod (2020) V. A. Luzgin and I. I. Kholod. 2020.  Overview of Mining Software Repositories. In _2020 IEEE Conference of Russian Young Researchers in Electrical and Electronic Engineering (EIConRus)_. IEEE.  https://doi.org/10.1109/eiconrus49466.2020.9039225
  * McDonald et al. (2019) Nora McDonald, Sarita Schoenebeck, and Andrea Forte. 2019\.  Reliability and Inter-Rater Reliability in Qualitative Research: Norms and Guidelines for CSCW and HCI Practice.  _Proc. ACM Hum.-Comput. Interact._ 3, CSCW, Article 72 (Nov. 2019), 23 pages.  https://doi.org/10.1145/3359174
  * Mendes et al. (2020) E. Mendes, C. Wohlin, K. Felizardo, and M. Kalinowski. 2020\.  When to update systematic literature reviews in software engineering.  _Journal of Systems and Software_ 167 (Sept. 2020), 110607\.  https://doi.org/10.1016/j.jss.2020.110607
  * Meurice and Cleve (2014) L. Meurice and A. Cleve. 2014.  DAHLIA: A visual analyzer of database schema evolution. In _2014 Software Evolution Week - IEEE Conference on Software Maintenance, Reengineering, and Reverse Engineering (CSMR-WCRE)_. 464–468.  https://doi.org/10.1109/CSMR-WCRE.2014.6747219
  * Miles and Huberman (1994) M. B. Miles and A. M. Huberman. 1994.  _Qualitative data analysis: An expanded sourcebook_.  sage.
  * Misra et al. (2017) J. Misra, S. Sengupta, D. Rawat, M. Savagaonkar, and S. Podder. 2017.  Data-Driven Application Maintenance: Experience from the Trenches. In _2017 IEEE/ACM 4th International Workshop on Software Engineering Research and Industrial Practice (SER IP)_. 48–54.  https://doi.org/10.1109/SER-IP.2017..8
  * Neysiani and Babamir (2019) B. S. Neysiani and S. Morteza Babamir. 2019.  Improving Performance of Automatic Duplicate Bug Reports Detection using Longest Common Sequence : Introducing New Textual Features for Textual Similarity Detection. In _2019 5th Conference on Knowledge Based Engineering and Innovation (KBEI)_. 378–383.  https://doi.org/10.1109/KBEI.2019.8735038
  * Nguyen and Nguyen (2017) A. T. Nguyen and T. N. Nguyen. 2017.  Automatic Categorization with Deep Neural Network for Open-Source Java Projects. In _2017 IEEE/ACM 39th International Conference on Software Engineering Companion (ICSE-C)_. IEEE.  https://doi.org/10.1109/icse-c.2017.118
  * Parizy et al. (2014) M. Parizy, K. Takayama, and Y. Kanazawa. 2014.  Software Defect Prediction for LSI Designs. In _2014 IEEE International Conference on Software Maintenance and Evolution_. 565–568.  https://doi.org/10.1109/ICSME.2014.96
  * Penta et al. (2020) M. Di Penta, G. Bavota, and F. Zampetti. 2020.  On the Relationship between Refactoring Actions and Bugs: A Differentiated Replication. In _Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ (Virtual Event, USA) _(ESEC/FSE 2020)_. Association for Computing Machinery, New York, NY, USA, 556–567.  https://doi.org/10.1145/3368089.3409695
  * Petersen et al. (2015) K. Petersen, S. Vakkalanka, and L. Kuzniarz. 2015\.  Guidelines for conducting systematic mapping studies in software engineering: An update.  _Information and Software Technology_ 64 (2015), 1–18.  https://doi.org/10.1016/j.infsof.2015.03.007
  * Poncin et al. (2011) W. Poncin, A. Serebrenik, and M. v. d. Brand. 2011\.  Process Mining Software Repositories. In _2011 15th European Conference on Software Maintenance and Reengineering_. 5–14.  https://doi.org/10.1109/CSMR.2011.5
  * Ramsauer et al. (2019) R. Ramsauer, D. Lohmann, and W. Mauerer. 2019\.  The List is the Process: Reliable Pre-Integration Tracking of Commits on Mailing Lists. In _2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE)_. 807–818.  https://doi.org/10.1109/ICSE.2019.00088
  * Spadini et al. (2018) D. Spadini, M. F. Aniche, and A. Bacchelli. 2018.  PyDriller: Python Framework for Mining Software Repositories. In _Proceedings of the 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ _(ESEC/FSE)_. 908––911.  https://doi.org/10.1145/3236024.3264598
  * Steinbeck (2020) M. Steinbeck. 2020\.  Mining Version Control Systems and Issue Trackers with LibVCS4j. In _2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER)_. 647–651.  https://doi.org/10.1109/SANER48275.2020.9054841
  * Strauss and Corbin (2007) Anselm Strauss and Juliet M. Corbin. 2007.  _Basics of Qualitative Research : Techniques and Procedures for Developing Grounded Theory_ (3rd ed.).  SAGE Publications.
  * Tantithamthavorn et al. (2019) C. Tantithamthavorn, S. McIntosh, A. E. Hassan, and K. Matsumoto. 2019.  The Impact of Automated Parameter Optimization on Defect Prediction Models.  _IEEE Transactions on Software Engineering_ 45, 7 (2019), 683–711.  https://doi.org/10.1109/TSE.2018.2794977
  * Tariq et al. (2020) S. Tariq, M. Usman, and A.C.M. Fong. 2020.  Selecting best predictors from large software repositories for highly accurate software effort estimation.  _Journal of Software: Evolution and Process_ 32, 10 (2020), e2271.  https://doi.org/10.1002/smr.2271 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1002/smr.2271  e2271 JSME-19-0146.R3.
  * Tiwari et al. (2016) N. M. Tiwari, G. Upadhyaya, and H. Rajan. 2016\.  Candoia: A Platform and Ecosystem for Mining Software Repositories Tools. In _2016 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-C)_. 759–761.  https://doi.org/10.1145/2889160.2892662
  * Tripathi et al. (2015) A. Tripathi, S. Dabral, and A. Sureka. 2015\.  University-industry collaboration and open source software (OSS) dataset in mining software repositories (MSR) research. In _2015 IEEE 1st International Workshop on Software Analytics (SWAN)_. 39–40.  https://doi.org/10.1109/SWAN.2015.7070489
  * V.R. Basili and Rombach (1994) G. Caldiera V.R. Basili and H.D. Rombach. 1994.  The goal question metric approach.  _Encyclopedia of software engineering_ (1994), 528–532.
  * White et al. (2015) M. White, C. Vendome, M. Linares-Vasquez, and D. Poshyvanyk. 2015\.  Toward Deep Learning Software Repositories. In _2015 IEEE/ACM 12th Working Conference on Mining Software Repositories_. 334–345.  https://doi.org/10.1109/MSR.2015.38
  * Wilder et al. (2016) N. Wilder, J. M. Smith, and A. Mockus. 2016\.  Exploring a Framework for Identity and Attribute Linking across Heterogeneous Data Systems. In _2016 IEEE/ACM 2nd International Workshop on Big Data Software Engineering (BIGDSE)_. 19–25.  https://doi.org/10.1109/BIGDSE.2016.012
  * Xie et al. (2009) T. Xie, S. Thummalapenta, D. Lo, and C. Liu. 2009\.  Data Mining for Software Engineering.  _Computer_ 42, 8 (2009), 55–62.  https://doi.org/10.1109/MC.2009.256
  * Ye (2019) X. Ye. 2019.  Learning to Rank Reviewers for Pull Requests.  _IEEE Access_ 7 (2019), 85382–85391.  https://doi.org/10.1109/ACCESS.2019.2925560
  * Yue et al. (2018) R. Yue, Z. Gao, N. Meng, Y. Xiong, X. Wang, and J. D. Morgenthaler. 2018\.  Automatic Clone Recommendation for Refactoring Based on the Present and the Past. In _2018 IEEE International Conference on Software Maintenance and Evolution (ICSME)_. 115–126.  https://doi.org/10.1109/ICSME.2018.00021