# Temperature induced collapse of spin dimensionality in magnetic
metamaterials

Björn Erik Skovdal<EMAIL_ADDRESS>Nanny Strandqvist  Henry
Stopfel  Merlin Pohlit  Tobias Warnatz  Samuel D. Slöetjes  Vassilios Kapaklis
Björgvin Hjörvarsson  Department of Physics and Astronomy, Uppsala University,
Box 516, 751 20 Uppsala, Sweden

###### Abstract

Spin and spatial dimensionalities are universal concepts, essential for
describing both phase transitions and dynamics in magnetic materials. Lately,
these ideas have been adopted to describe magnetic properties of
metamaterials, as well as to replicate and explore ensembles of mesospins
belonging to different universality classes. Here, we take the next step by
investigating magnetic metamaterials not conforming to the conventional
framework of continuous phase transitions. Instead of a continuous decrease in
the moment with temperature, discrete steps are possible, resulting in a
binary transition in the interactions of the elements. The transition is
enabled by nucleation and annihilation of vortex cores, shifting topological
charges between the interior and the edges of the elements. Consequently, the
mesospins can be viewed as shifting their spin dimensionality, from 2 (XY-
like) to 0, (vortices) at the transition. The results provide insight into how
dynamics at different length-scales couple, which can lead to thermally driven
topological transitions in magnetic metamaterials.

††preprint: APS/123-QED

##  I INTRODUCTION

Magnetic metamaterials Heyderman and Stamps (2013) composed of mesospins as
building blocks Östman  _et al._ (2018a), offer the possibility of tailoring
magnetic interactions and dynamics in almost arbitrary ways. Previous
investigations have mainly been focused on the collective magnetic order,
dynamics and more exotic aspects such as frustration of artificial spin
systems Heyderman and Stamps (2013); Nisoli  _et al._ (2013, 2017);
Rougemaille and Canals (2019), while the internal magnetisation and dynamics
of the mesospins are less explored Gliga  _et al._ (2013, 2015); Slöetjes  _et
al._ (2021). For instance, extensive efforts have been made to mimic magnetic
systems of various spatial and spin dimensionalities, where the shape of the
elements have been used to enforce mesospins to be Ising- or XY-like Östman
_et al._ (2018a); Sendetskyi  _et al._ (2019); Ewerlin  _et al._ (2013);
Streubel  _et al._ (2018); Leo  _et al._ (2018); Arnalds  _et al._ (2016).
Even systems consisting of both Ising- and XY-mesospins have been fabricated
and investigated, which is a testament to the versatility of metamaterials
Arnalds  _et al._ (2014); Östman  _et al._ (2018b). The common denominator in
all these investigations, however, is that the spin dimensionality is treated
as a static property of the elements. Yet, the mesospins offer access to
continuous degrees of freedom and rich internal magnetic states that go well
beyond their atomic analogues. This attribute is the focal point of the
present study, as we turn our attention to arrays of mesospins that can
exhibit two distinct magnetisation states: collinear and vortex spin textures
Shinjo (2000); Kläui  _et al._ (2003). The mesospins can be thought of as
having variable spin dimensionality, where the interaction strength depends on
the inner magnetic textures. It is therefore possible to couple the changes in
spin dimensionality to changes in the collective properties of the mesospins.
The coupling of these internal and external degrees of freedom, in combination
with the exploration of the role of topological effects on the observed
transitions Mermin (1979); Tchernyshyov and Chern (2005), is the main
motivation behind the current work.

##  II MATERIALS AND METHODS

###  II.1 Sample manufacturing

Two sets of samples were prepared: one set for photo-emission electron
microscopy (PEEM) studies employing x-ray magnetic circular dichroism (PEEM-
XMCD) and the other set for magneto-optical Kerr effect (MOKE) measurements.
The samples were prepared depositing elemental Fe (99.95 at%) and Pd (99.95
at%) in an ultra-high vacuum (base pressure below $\sim 2\times 10^{-7}$ Pa)
DC magnetron sputtering system, operating using high purity Argon gas (99.995
%). The following sample structure was used: fused silica/Pd [40 nm]/Fe13Pd87
[10 nm]/Pd [2 nm] Östman  _et al._ (2014). After growth, electron beam
lithography (EBL) was used to pattern the Fe13Pd87 layers into circular
islands arranged on square lattices (see Fig. 1). The Ar+-ion milling process
following development of the EBL resist and subsequent mask deposition, was
stopped prior to penetration through the Pd seed layer, providing this way
electrical continuity across the whole sample surface. The PEEM-XMCD samples
with a disk diameter of $D=[75,150,350]$ nm were fabricated having two inter-
disk distances; one with the nearest neighbour distance set to G = D + 40 nm,
rendering the inter-disk coupling sufficiently weak to be ignored, and the
other with an edge-to-edge gap of G = 20 nm to promote interactions between
the magnetic elements Guslienko  _et al._ (2001). The MOKE samples were
patterned into disks with a diameter of D = $[250,350,450]$ nm with a small
edge-to-edge gap of G = 20 nm.

Figure 1: The magnetic metamaterial - square arrays of circular islands.
Scanning electron microscopy image of a representative structure with disks of
a diameter $D=150$ nm. (Left) Interacting disks with a gap of $G=20$ nm. The
inset shows the same structure at a higher magnification. (Right) Non-
interacting disks with $G=D+40$ nm. The major symmetry axes of the patterned
square lattices, [10] and [11] are also denoted.

###  II.2 PEEM-XMCD

To determine the magnetic state of the non-interacting and strongly coupled
disks, photoemission electron microscopy, employing x-ray magnetic circular
dichroism (PEEM-XMCD), was performed at the HERMES beamline at the SOLEIL
synchrotron Belkhou  _et al._ (2015), and the 11.0.1 beamline at the Advanced
Light Source Doran  _et al._ (2012). Prior to imaging, the samples were cooled
from room temperature to roughly 100 K, in absence of any external magnetic
field. The energy of the synchrotron beam was tuned to the Fe L3-edge (708.4
eV), and magnetic contrast was obtained from the asymmetry ratio of
intensities between right- and left-handed circularly polarised synchrotron
radiation.

###  II.3 Micromagnetic simulations and topological considerations

To understand the role of the interactions, we chose to explore the energies
of the vortex and collinear state for different disk sizes, using the MuMax3
micromagnetic simulation software Vansteenkiste  _et al._ (2014). The
simulations, used to mimic the experimental conditions, were performed using
one single disk, as well as square arrays of disks ($G=20$ nm), with periodic
boundary conditions. The saturation magnetisation, $M_{\text{sat}}=3.5\cdot
10^{5}$ A/M and exchange stiffness, $A_{\text{ex}}=3.36\cdot 10^{-12}$ J/m
were chosen based on previous work by Östman et al. Östman  _et al._ (2014)
and Ciuciulkaite et al. Ciuciulkaite  _et al._ (2019). The in-plane cell size
was set to 0.50(1)$l_{\text{ex}}$, where $l_{\text{ex}}$ is the exchange
length as defined by $M_{\text{sat}}$ and $A_{\text{ex}}$ Vansteenkiste  _et
al._ (2014).

###  II.4 Hysteresis protocols

The magnetisation data, displayed in Fig. 5, was collected using a MOKE system
in longitudinal configuration with the sample mounted on a cryostat, in a
temperature range of 80 K $<T<$ 400 K. The $p$-polarised incident laser beam
with $\lambda=$ 659 nm has a Gaussian profile with a spot diameter of roughly
2 mm. The number of islands contributing to the signal are therefore in the
order of $10^{7}$. The reflected laser beam was passed through an analyser
(extinction ratio of $10^{5}:1$) and then captured using a Si biased detector,
connected to a pre-amplifier and lock-in amplifier. In addition, the lock-in
amplifier was used to modulate the incident laser beam using a Faraday cell.
The sinusoidal external magnetic field, was applied along [10] of the square
lattice (see Fig. 1), had an amplitude of 40 mT and a frequency of 0.11 Hz
unless otherwise stated. The magnetisation data in all figures have been
binned (3:1) for aesthetic purposes.

##  III RESULTS AND DISCUSSION

We begin by discussing the size dependence of internal spin textures and
spontaneous magnetic order in circular islands. To this end, we fabricated
samples with nominal disk diameters $D=[75,150,350]$ nm, having edge-to-edge
gaps $G=[20,D+40]$ nm, on one and the same wafer. This approach allows us to
disregard any uncertainty related to $e.g.$ composition and thickness of the
islands. Representative parts of the samples with 150 nm islands are
illustrated in Fig. 1. The results of photo-emission electron microscopy on
these samples, with magnetic contrast obtained by using x-ray magnetic
circular dichroism (PEEM-XMCD), are displayed in Fig. 2. As inferred by the
black and white contrast, the disks with a diameter of 75 nm are all found to
exhibit a collinear component for both small and large distances between the
islands. Similarly, the 350 nm islands are found to be in a vortex state for
both the short and long distance between the islands, with no preferred sense
of rotation. When the disk diameter is 150 nm, a vortex state is obtained when
the distance between the islands is large, while a substantial amount of
mesospins have a collinear component, when the distance between the islands is
20 nm. Thus, for this particular diameter of islands, the interaction appears
to be strong enough to influence the inner magnetic states. To understand the
role of the interactions and the interplay between the involved length-scales,
we need to have a look at the energy landscape of the magnetic textures.

Figure 2: Diameter and interaction dependence of the disk magnetic texture.
PEEM-XMCD images of interacting islands (left column), and non-interacting
islands (right column), recorded at approximately 100 K after cooling from
room temperature in absence of external fields. Disks with a diameter of 350
nm have a preferred vortex texture for both interacting to non-interacting
disks, while disks with 75 nm diameter end up in the collinear state in both
cases. Disks with a diameter of 150 nm, display a stabilisation of the
collinear state when the distance between the islands is small, while
otherwise exhibiting vortex textures. The red circles indicate the disk sizes
and positions, obtained by overlapping the PEEM-X-ray absorption spectroscopy
images.

The magnetic energy of the metamaterial can be expressed as
$E=E_{\text{t}}+E_{\text{s}}+E_{\text{j}}$, where $E_{\text{t}}$ is the energy
cost of the magnetic texture arising from exchange interactions within the
islands, $E_{\text{s}}$ is the magnetostatic energy and $E_{\text{j}}$ is the
energy associated with the magnetostatic coupling between the islands. By
calculating the energy as a function of the vortex core displacement $r$ for a
disk with radius $R$, a single path – out of many – in the energy landscape
separating the collinear and the vortex state can be obtained. Ding et al.
used the same reasoning when calculating the energy barrier separating the two
states in a single Co dot analytically Ding  _et al._ (2005). Here, we chose
to do it numerically as the calculations can be generalised to include
interactions of elements in an arbitrary array. The results obtained from
calculations of interacting ($G=20$ nm) and non-interacting ($G=D+40$ nm)
disks with $D=[75,150,350]$ nm are shown in Fig. 3. When a vortex core is at
the centre of an island, its displacement is defined to be zero and the energy
is $E\approx E_{\text{t}}$ because $E_{\text{s}}\approx E_{\text{j}}\approx
0$. Shifting the core from the centre gives rise to a collinear component, and
consequently a stray field with a corresponding magnetostatic energy. For the
purpose of the calculation, the (virtual) vortex core can even be moved
outside the disk ($r>R$), corresponding to a $C$-state (see Fig. 3), with
varying degree of gradients in the magnetic texture. The energy maxima
obtained at $r\approx 0.9R$ can be viewed as activation barriers separating
the vortex and the collinear states. The energy difference between the
interacting and non-interacting islands increases with increasing $r$, as
illustrated in the figure.

In these simulations we use the energies obtained by MuMax3’s built in
functions for the vortex and the collinear state, without allowing for
relaxation of the magnetisation with respect to the total energy. Using this
approach, the intermediate states can be calculated without the systems
collapsing into either the vortex or the collinear state, enabling us to
estimate the height of the activation barrier. Qualitatively, the same results
can be obtained in relaxed systems by moving the core by means of an applied
magnetic field. This approach, however, does not provide any information on
the potential in the vicinity of the maximum of the activation barrier. In the
simulations with interacting islands, all vortex cores were displaced such
that the collinear component of each island was along [10], mimicking the
influence of an applied field along [10] as in the MOKE experiments. The
choice of either uniform or alternating vortex chirality had a negligible
impact on the outcome of the simulations.

Figure 3: The energy landscape of the transition between the vortex to the
collinear state. The landscape was obtained numerically by gradually moving
the vortex core outwards from the centre of the disks. Filled symbols
represent the energy values of interacting disks ($G$ = 20 nm), while empty
symbols for non-interacting disks. The shaded areas represent the
magnetostatic coupling $E_{\text{j}}$. The energies plotted are normalised to
the total energy of the vortex state $E_{\text{v}}$. The energy barrier
between the two states is tunable by choice of disk diameter and inter-island
interaction strength. The top part shows a schematic of the states for
different values of $r$, where the red dot indicates the position of the
vortex core, when $r>R$, a $C$-state is obtained. Figure 4: The boundary
between vortex and collinear states. The energy of the collinear state is
lowered (blue squares) by inter-island interactions. As a result, it is
possible to stabilise collinear states for a range of diameters which
otherwise favour formation of vortices (gray shaded area). The energies
plotted are normalised to the total energy of the vortex state $E_{\text{v}}$.

The size dependence in the energy of interacting and non interacting mesospins
is summarised in Fig. 4. For non-interacting islands the collinear state has
the lowest energy when $D\lesssim 140$ nm, while a vortex state is favoured
when the diameter is larger. With an inter-island distance of 20 nm, a
collinear state is favoured up to about $D\approx 190$ nm. Consequently the
critical size at which the vortex state is favoured is shifted to larger
diameters when interactions become prominent. Hence, a region of bistability
(marked by a grey shading in the figure) is obtained. We have thereby
rationalised the results displayed in Fig. 2, ${i.e.}$ why the 150 nm islands
form vortices in absence of interactions, while a significant amount of the
mesospins show a collinear component when they interact. When collinear, the
mesospins can be viewed as being two-dimensional Arnalds  _et al._ (2014) (XY-
rotor) and zero-dimensional Östman  _et al._ (2014) when in a vortex state. At
the same time, the magnetic texture can be defined in terms of topological
quasiparticles Zhang  _et al._ (2015); Donnelly  _et al._ (2020), as $e.g.$
illustrated in Fig. 6 and listed in Table 1 (Appendix A). The states displayed
in Fig. 6 can thereby be defined as having topological charge of +1,
determined by $1-g$, with $g$ being the genus of the structure Tchernyshyov
and Chern (2005). The boundary of the islands can host magnetic charges with
fractional winding numbers, $w$, which together with the winding number of the
bulk charge, $q$, must add up to 1 (see Fig. 6) Tchernyshyov and Chern (2005);
Slöetjes  _et al._ (2020). Consequently, the winding of the vortex can been
seen as being transferred to the edge of the island when the magnetisation
changes from a vortex to a collinear state ( $r>R$). The transition from a
vortex to a collinear state therefore involves no change in the total winding
number, while the magnetic cores and their polarity annihilate at the edges of
the islands.

While it is trivial to annihilate vortices by applying an external field, the
opposite is certainly not true. For this reason we chose to focus on the field
and temperature dependence of islands with a diameter of 250 nm and larger
($D=[250,350,450]$ nm), with gaps $G$ = 20 nm. Disks in this size range
spontaneously form vortices when cooled, while the application of an external
magnetic field results in a collinear state, allowing us to control the
magnetic texture of the mesospins. The stability of the $dressed$ collinear
state can thus be investigated by removing the external field while monitoring
the magnetisation of the samples. Representative magnetisation loops for the
450 nm islands, recorded at four different temperatures, are provided in the
top half of Fig. 5. The bottom part illustrates the temperature dependence of
the magnetisation.

Figure 5: Temperature dependence of the dimensionality transition in a
magnetic metamaterial. Magnetisation measurements with the applied field along
[10] (see Fig. 1) for the 450 nm islands having a gap of $G$ = 20 nm. (Top)
Representative hysteresis curves at 80, 220, 250 and 275 K. The curves have
been normalised for each individual temperature. The nucleation field crosses
$\mu_{0}H=0$ at approximately 220 K, indicating a stability of the collinear
state below this temperature, during the timescale of the measurement.
(Bottom) The magnetisation as a function of temperature at 0 and 40 mT as well
as the ratio $M_{\text{m}}$=$M_{\text{r}}$/$M_{\text{s}}$. The orange filled
circles mark the temperatures at which the hysteresis curves, shown above,
were taken. The size dependence of the transition temperature, $T_{\text{m}}$,
for island sizes $D=[250,350,450]$ nm is shown in the inset. The error bars of
the data in the inset are smaller than the data point symbols.

At $T$ = 275 K the hysteresis loops display a typical vortex nucleation
($H_{\text{n}}$) and annihilation ($H_{\text{a}}$) signature with zero
remanence Cowburn  _et al._ (1999), in line with both the PEEM-XMCD results as
well as the simulations. At temperatures below $\approx$ 250 K, the sample
exhibits clear remanence, consistent with the presence of a ferromagnetic
state. Ferromagnetic response requires alignment of mesospins with a net
moment, in stark contrast to the zero field PEEM-XMCD results. The reduction
of the remanent magnetisation ($M_{\text{r}}$) with temperature contains both
the decrease of the moment of the material (intrinsic material properties) as
well as changes in the texture and orientation of the mesospins. To
disentangle these contributions we need to identify their signatures. Changes
in the material-related magnetisation can be described by a power law up to
the ordering temperature of the materials ($T_{\mathrm{C}}$), while the
temperature dependence of the texture and orientation of the mesospins is
unknown. However, separation of the two contributions can be obtained by
identifying the difference in their field dependence. For instance, it is
sufficient to apply a relatively weak field to remove most of the magnetic
texture within a disk, while weak fields only marginally affect the thermally
induced excitations of the magnetisation in the material. This is seen in the
field dependence displayed in Fig. 5: at $T$ = 275 K, a field of approximately
6 mT is sufficient for obtaining a transition from a vortex to a collinear
state. Nevertheless, this field does little to alter the thermally induced
reduction of the magnetisation. We can therefore consider the magnetisation at
a field of 40 mT as predominantly representative for the material, $i.e.$ we
define this magnetisation as the saturation of the mesoscopic texture
($M_{\text{s}}$). Consequently, the temperature dependence of
$M_{\text{m}}$=$M_{\text{r}}$/$M_{\text{s}}$ can be thought of as
corresponding to the temperature dependence of the remanence of the mesospin
texture.

The results displayed in the bottom half of Fig. 5 (450 nm islands),
illustrate $M_{\text{r}}$, $M_{\text{s}}$ and
$M_{\text{m}}$=$M_{\text{r}}$/$M_{\text{s}}$, the inferred remanence of the
mesospins. A plateau with $M_{\text{m}}=3/4$ is observed, consistent with
constant magnetic texture and thereby a fixed topology of the mesospins, below
220K. At the same time, a substantial fraction of the moment (1/4) is
perpendicular and/or antiparallel with respect to the direction of the net
magnetisation. At 220 K there is an abrupt change in $M_{\text{m}}$, which
vanishes at 270 K. From 270 K and up to the intrinsic Curie temperature of the
material, the obtained hysteresis loops are consistent with the presence of
vortex states. Thus, we observe a sharp transition from a collective state of
interacting mesospins to non-interacting vortex states of the elements with
zero net magnetisation. To this observation we assign a collapse from a state
with a spin dimensionality of 2, to a state with zero spin dimensionality. The
temperature dependence of this transition can be determined by fitting an
error function to the data, from which the temperature $T_{\text{m}}$ where
$M_{\text{m}}(T_{\text{m}})=\frac{1}{2}M_{\text{m}}(T=0~{}\mathrm{K})$ (inset
of Fig. 5, bottom panel) is determined. An error function was chosen without
assigning any physical interpretation to it. A clear size dependence
($T_{\text{m}}\propto 1-1/D$) is observed. The results imply an increase in
stability of the collinear state with increasing disk size. Extrapolation to
infinite size yields a transition temperature approaching $T_{\text{C}}$ of
the continuous film, as expected. Extrapolation to zero in $T_{\text{m}}$
yields $\approx$ 100 nm island size, which corresponds to the lower limit for
the possibility to observe the transition. However, as illustrated in Fig. 4,
the collinear state is already favoured when $D\lesssim 190$ nm, rendering
this region unattainable. As the vortex state is energetically favoured above
this size, the observed transitions in 250, 350 and 450 nm elements are
kinetically limited and do not correspond to thermodynamic phase transitions
in a strict sense. This is further emphasised by the observed frequency
dependence of the transition temperature, $T_{\text{m}}$ (see Appendix B). The
dressed mesospins (collinear state) in this size range, are therefore in an
arrested metastable state with the boundaries of the mesospins being vital for
the creation and annihilation process of the vortex cores.

In conclusion, the temperature dependence of the topologically homeomorphic
transition is not only defined by the material properties and size of the
mesospins but also by their mutual interactions and internal degrees of
freedom. In particular, the distance between the elements alters the inter-
island interactions and thereby the internal magnetic states, while at the
same time, the internal magnetic states affect the interactions and thereby
the magnetic order. The interplay between these two length-scales leads to an
exotic transition, that is to say that it is not based on thermally induced
randomisation. Rather, the transition pertains to a thermally activated change
within the elements, resulting in a collapse of the effective interaction
strength and the mesospin dimensionality. The transition discussed here does
not have any trivial classical counterpart, calling for a new conceptual
framework involving mutual dependence of energy and length-scales Wilson
(1979). The results represent stumbling steps towards understanding emergent
properties and complexity that may extend beyond the immediate field of
physics Castellano  _et al._ (2009); Michaud and Szilva (2018).

## Data availability

The data that support the ﬁndings of this study are available from the authors
upon reasonable request.

## Acknowledgments

The authors would like to acknowledge the excellent user support provided to
them at the Advanced Light Source (Dr. Andreas Scholl, Dr. Rajesh Chopdekar)
and SOLEIL (Dr. Rahid Belkhou) synchrotrons, as well as Dr. Erik Östman for
helping out with data collection. This research used resources of the Advanced
Light Source, a U.S. DOE Office of Science User Facility under contract no.
DE-AC02-05CH11231. The excellent support and infrastructure of the MyFab
facility at the Ångström Laboratory of Uppsala University is also highly
appreciated. B.H. acknowledges financial support from the Swedish Research
Council and the Swedish Foundation for Strategic Research. V.K. acknowledges
support from the Knut and Alice Wallenberg Foundation project “Harnessing
light and spins through plasmons at the nanoscale” (2015.0060) and the Swedish
Research Council (Project No. 2019-03581).

##  Appendix A Mesospin classification

Figure 6: Magnetic texture and topology of a vortex (left) and a collinear
state (right). There is substantial texture in the magnetic state referred to
as collinear, however, the net transverse moment is zero. The interior
topological charge $q=+1$ (vortex) can be transferred to two topological
charges, with $w=+\frac{1}{2}$, at the edges (collinear) Tchernyshyov and
Chern (2005). The schematics next to the textures depict the changes in the
direction of the magnetisation with respect to the edge and the resulting
winding number.

Fig. 6 illustrates the vortex and the collinear states, as well as their
corresponding topological charges residing in the bulk and on the edges
respectively. A transition between the two states is topologically allowed for
elements of finite size, and can be driven by applying external fields or
thermally induced fluctuations Östman  _et al._ (2014). The amount of
topological charge that can be residing in an element is determined by the
genus $g$ (number of holes) of the structure, providing an additional knob for
the design of texture topology in magnetic metamaterials. This transition is
followed by a change in the effective spin dimensionality of the mesospins,
which is 0 for the vortex and 2 for the collinear state (see also Table 1).
Note that in the interior of the mesospins, non-integer winding numbers cannot
exist Tchernyshyov and Chern (2005); Zhang  _et al._ (2015). This is not the
case though for the edges where half-integer winding numbers are possible and
thus topological charges, $w$, which further correlate to stray field
emanating from or into the mesospin Tchernyshyov and Chern (2005). These
appear in pairs, resulting in integer numbers for the total charge on the
edge.

The spin dimensionality $d$ relates to the behaviour of the effective mesospin
moment, which for planar mesospins can be 0-, 1- or 2-dimensional Shinjo
(2000); Östman  _et al._ (2018a); Arnalds  _et al._ (2014), as illustrated in
Table 1. The sense of rotation $c$ describes the circulation (clock- and
anticlock-wise) of the magnetic texture and acquires non-zero values in the
vortex case (0-dimensional mesospin) Jenkins  _et al._ (2014). The polarity
$p$ relates to the direction of the out-of-plane component (up or down) of the
magnetisation for the core in a vortex. Finally, when considering the planar
topology, the bulk topological charge $q$, defines the magnetic texture of the
mesospins, being +1 for vortex states and 0 for Ising or XY-rotor states. The
blue coloured cells refer to mesospin properties and their associated
transitions, particularly studied in this work.

Table 1: Classification scheme for the planar mesospins.

##  Appendix B Kinetics

To obtain a better understanding on the interplay between kinetics and
thermodynamics, we explored the influence of the frequency of the applied
field on the obtained remanent magnetisation. The dwell time at zero field
provides a time window for a relaxation, which is proportional to the inverse
scanning frequency. The findings are illustrated in the lower panel of Fig. 7,
depicting a clear frequency dependence of the response from the 450 nm disks.
Increasing the scanning frequency, results in an increase of the transition
temperatures (inset in Fig. 7, lower panel), illustrating the importance of
the kinetic limitations on the transition Pohlit  _et al._ (2020); Slöetjes
_et al._ (2021). Consequently, we conclude that the ferromagnetic component
observed at low temperatures, corresponds to a metastable arrested state. The
frequency dependence can not be captured by a single exponential, highlighting
the non-Arrhenius behaviour of the homeomorphic transition.

Figure 7: Size and frequency dependence of the dimensionality transition.
(Top) Temperature dependence of the remanent versus the in-field (40 mT)
magnetization ratio, for all available disk diameters and for a gap $G$ = 20
nm. A linear scaling of the transition temperature $T_{\text{m}}$, determined
from the inflection points of the fitted curves versus the reciprocal of the
disk diameter $D$, can been seen in the inset. (Bottom) The frequency
dependence of the transition for a disk diameter $D$ = 450 nm and gap $G$ = 20
nm. A monotonic shift to higher temperature with increasing frequency is
observed, compatible with thermally activated relaxation processes of magnetic
metamaterials. The orange filled circles in each panel, mark the extracted
temperatures presented in the respective insets.

## References

  * Heyderman and Stamps (2013) L. J. Heyderman and R. L. Stamps, Journal of Physics: Condensed Matter 25, 363201 (2013).
  * Östman  _et al._ (2018a) E. Östman, U. B. Arnalds, V. Kapaklis, A. Taroni, and B. Hjörvarsson, Journal of Physics: Condensed Matter 30, 365301 (2018a).
  * Nisoli  _et al._ (2013) C. Nisoli, R. Moessner, and P. Schiffer, Rev. Mod. Phys. 85, 1473 (2013).
  * Nisoli  _et al._ (2017) C. Nisoli, V. Kapaklis, and P. Schiffer, Nature Physics 13, 200 (2017).
  * Rougemaille and Canals (2019) N. Rougemaille and B. Canals, Eur. Phys. J. B 92, 62 (2019).
  * Gliga  _et al._ (2013) S. Gliga, A. Kákay, R. Hertel, and O. G. Heinonen, Physical Review Letters 110, 117205 (2013).
  * Gliga  _et al._ (2015) S. Gliga, A. Kákay, L. J. Heyderman, R. Hertel, and O. G. Heinonen, Physical Review B 92, 060413 (2015).
  * Slöetjes  _et al._ (2021) S. D. Slöetjes, B. Hjörvarsson, and V. Kapaklis, Applied Physics Letters 118, 142407 (2021).
  * Sendetskyi  _et al._ (2019) O. Sendetskyi, V. Scagnoli, N. Leo, L. Anghinolfi, A. Alberca, J. Lüning, U. Staub, P. M. Derlet, and L. J. Heyderman, Physical Review B 99, 214430 (2019).
  * Ewerlin  _et al._ (2013) M. Ewerlin, D. Demirbas, F. Brüssing, O. Petracic, A. A. Ünal, S. Valencia, F. Kronast, and H. Zabel, Physical Review Letters 110, 177209 (2013).
  * Streubel  _et al._ (2018) R. Streubel, N. Kent, S. Dhuey, A. Scholl, S. Kevan, and P. Fischer, Nano Letters 18, 7428 (2018).
  * Leo  _et al._ (2018) N. Leo, S. Holenstein, D. Schildknecht, O. Sendetskyi, H. Luetkens, P. M. Derlet, V. Scagnoli, D. Lançon, J. R. L. Mardegan, T. Prokscha, A. Suter, Z. Salman, S. Lee, and L. J. Heyderman, Nature Communications 9, 2850 (2018).
  * Arnalds  _et al._ (2016) U. B. Arnalds, J. Chico, H. Stopfel, V. Kapaklis, O. Bärenbold, M. A. Verschuuren, U. Wolff, V. Neu, A. Bergman, and B. Hjörvarsson, New Journal of Physics 18, 023008 (2016).
  * Arnalds  _et al._ (2014) U. B. Arnalds, M. Ahlberg, M. S. Brewer, V. Kapaklis, E. T. Papaioannou, M. Karimipour, P. Korelis, A. Stein, S. Ólafsson, T. P. A. Hase, and B. Hjörvarsson, Applied Physics Letters 105, 042409 (2014).
  * Östman  _et al._ (2018b) E. Östman, H. Stopfel, I.-A. Chioar, U. B. Arnalds, A. Stein, V. Kapaklis, and B. Hjörvarsson, Nature Physics 14, 375 (2018b).
  * Shinjo (2000) T. Shinjo, Science 289, 930 (2000).
  * Kläui  _et al._ (2003) M. Kläui, C. A. F. Vaz, L. Lopez-Diaz, and J. A. C. Bland, Journal of Physics: Condensed Matter 15, R985 (2003).
  * Mermin (1979) N. D. Mermin, Reviews of Modern Physics 51, 591 (1979).
  * Tchernyshyov and Chern (2005) O. Tchernyshyov and G.-W. Chern, Physical Review Letters 95, 197204 (2005).
  * Östman  _et al._ (2014) E. Östman, U. B. Arnalds, E. Melander, V. Kapaklis, G. K. Pálsson, A. Y. Saw, M. A. Verschuuren, F. Kronast, E. T. Papaioannou, C. S. Fadley, and B. Hjörvarsson, New Journal of Physics 16, 053002 (2014).
  * Guslienko  _et al._ (2001) K. Y. Guslienko, V. Novosad, Y. Otani, H. Shima, and K. Fukamichi, Physical Review B 65, 024414 (2001).
  * Belkhou  _et al._ (2015) R. Belkhou, S. Stanescu, S. Swaraj, A. Besson, M. Ledoux, M. Hajlaoui, and D. Dalle, Journal of Synchrotron Radiation 22, 968 (2015).
  * Doran  _et al._ (2012) A. Doran, M. Church, T. Miller, G. Morrison, A. T. Young, and A. Scholl, Journal of Electron Spectroscopy and Related Phenomena 185, 340 (2012).
  * Vansteenkiste  _et al._ (2014) A. Vansteenkiste, J. Leliaert, M. Dvornik, M. Helsen, F. Garcia-Sanchez, and B. Van Waeyenberge, AIP Advances 4, 107133 (2014).
  * Ciuciulkaite  _et al._ (2019) A. Ciuciulkaite, E. Östman, R. Brucas, A. Kumar, M. A. Verschuuren, P. Svedlindh, B. Hjörvarsson, and V. Kapaklis, Physical Review B 99, 184415 (2019).
  * Ding  _et al._ (2005) H. F. Ding, A. K. Schmid, D. Li, K. Y. Guslienko, and S. D. Bader, Physical Review Letters 94, 157202 (2005).
  * Zhang  _et al._ (2015) S. Zhang, A. A. Baker, S. Komineas, and T. Hesjedal, Scientific Reports 5, 591 (2015).
  * Donnelly  _et al._ (2020) C. Donnelly, K. L. Metlov, V. Scagnoli, M. Guizar-Sicairos, M. Holler, N. S. Bingham, J. Raabe, L. J. Heyderman, N. R. Cooper, and S. Gliga, Nature Physics 21, 759–761 (2020).
  * Slöetjes  _et al._ (2020) S. D. Slöetjes, E. Folven, and J. K. Grepstad, Physical Review B 101, 014450 (2020).
  * Cowburn  _et al._ (1999) R. P. Cowburn, D. K. Koltsov, A. O. Adeyeye, M. E. Welland, and D. M. Tricker, Physical Review Letters 83, 1042 (1999).
  * Wilson (1979) K. G. Wilson, Scientific American 241, 158 (1979).
  * Castellano  _et al._ (2009) C. Castellano, S. Fortunato, and V. Loreto, Reviews of modern physics 81, 591 (2009).
  * Michaud and Szilva (2018) J. Michaud and A. Szilva, Physical Review E 97, 062313 (2018).
  * Jenkins  _et al._ (2014) A. S. Jenkins, E. Grimaldi, P. Bortolotti, R. Lebrun, H. Kubota, K. Yakushiji, A. Fukushima, G. de Loubens, O. Klein, S. Yuasa, and V. Cros, Applied Physics Letters 105, 172403 (2014).
  * Pohlit  _et al._ (2020) M. Pohlit, G. Muscas, I.-A. Chioar, H. Stopfel, A. Ciuciulkaite, E. Östman, S. D. Pappas, A. Stein, B. Hjörvarsson, P. E. Jönsson, and V. Kapaklis, Physical Review B 101, 134404 (2020).


XX XX 8 Month Generative AI in Evidence-Based Software Engineering: A White
Paper

# Generative AI in Evidence-Based Software Engineering: A White Paper

Matteo Esposito  University of Oulu, Oulu, Finland  Andrea Janes  Free
University of Bozen-Bolzano, Bozen-Bolzano, Italy  Davide Taibi  University of
Oulu, Oulu, Finland  Valentina Lenarduzzi  University of Oulu, Oulu, Finland

(2024)

###### Abstract

Context. In less than a year, practitioners and researchers witnessed a rapid
and wide implementation of Generative Artificial Intelligence. The daily
availability of new models proposed by practitioners and researchers has
enabled quick adoption. Textual-GAI’s capabilities enable researchers
worldwide to explore new generative scenarios, simplifying and hastening all
time-consuming text generation and analysis tasks. Motivation. The
exponentially growing number of publications in our field, with the increased
accessibility to information due to digital libraries, makes conducting
systematic literature reviews and mapping studies an effort and time-
insensitive task. Stemmed from this challenge, we investigated and envisioned
the role of GAIs in evidence-based software engineering. Future Directions.
Based on our current investigation, we will follow up the vision with the
creation and empirical validation of a comprehensive suite of models to
effectively support EBSE researchers.

††journal: IEEE Software

## Introduction

Systematic studies (SS), i.e., literature review and mapping studies, are
essential for collecting, organizing, and sharing knowledge within the
research community [1]. Such reviews analyze multiple studies to guide
informed research and practitioner-oriented decision-making in different
fields. While digital libraries and online tools offer some automation to
assist researchers in finding, organizing, and grabbing studies quickly,
existing tools do not fully support the entire SLR process, especially the
automation of study collection, application of criteria, and data synthesis
[2].

Natural language processing (NLP) research offers a state-of-the-art model
that can quickly analyze the amount of text documents and pull knowledgeable
information. Such a current trend materialized the concept of an AI that can
create or generate, i.e., generative AI (GAI), among the possibilities, “new”
text, art, and music based on the vast knowledge stored in the transformer
network as parameters.

Therefore, in response to the growing demand for improved research support
[2], and the tedious task of SS, we envision the role of GAI in evidence-based
software engineering (EBSE). Our paper presents a vision of the role that GAI
will eventually have in EBSE. Our goal is to inspire interest in integrating
generative AI with systematic studies in evidence-based software engineering
rather than attempting to present a comprehensive survey. Practitioners would
be interested in the evolution of the framework that will allow our vision to
come to fruition and the applicability of the presented GAI approaches to
industrial use cases kin to SS; researchers, on the other hand, will be
interested in the performance of our suite of models and the actual speedup.
More specifically, our vision describes the current use of AI in SS and
combines established systematic review methodologies with current technology,
such as GAI, to automate the entire review process. Our contributions are
three-folded: (i) We provide a collection of studies on the current state-of-
the-art leveraging GAIs,e.g., LLMs, in specific SS steps; (ii) We highlight
GAI usage gaps in the current research landscape and envision methods to fill
those gaps; (iii) We provide a sample empirical evaluation protocol for a
comprehensive suite of GAI models for EBSE.

## Background & Motivation

[1] were the first to introduce an evidence-based approach in SE, and the
guidelines they gave are these days the de-facto  standard. [3] investigated
the benefits the SE community can gain from this approach, like in medicine.
They tried to understand the evolution of systematic reviews in EBSE. [3]
conducted an SMS in which they discovered 52 papers that reported on the
importance of SLR in software engineering research. [2] study evidence-based
software engineering (EBSE) by focusing on the role of SLRs. Although SLRs are
at the core of EBSE, they are highly work-intensive. For the last ten years,
researchers have tried to automate the process of SLRs in software engineering
(SE) to decrease human effort and lessen the quality of SLRs. Finally, [2]
concludes that the SE community still needs to advance and develop more
automation strategies to reduce human effort further and make the SLR process
smoother.

Automated Tools. Currently, tools that include AI functionalities for part of
the SLR process, such as Data Extraction and Synthesis
exists111https://elicit.com222https://www.distillersr.com/. Nonetheless, the
current tools are proprietary. Contrary, our vision, is not directly related
to study nor focused on a systematic review, rather on presenting a structured
overview of the possibilities, though some non trivial, of GAI in EBSE.

Our work proposes a vision of the role of GAIs in trying to cover all SLR
activities, such as data extraction and synthesis. Hence, our approach uses a
unique set of methodologies and objectives for a much broader scope in the
case of SLR activity.

## Design

To our knowledge, our work classifies the latest literature on using specific
GAI models for specific SS tasks. We organized the SS tasks according to the
state-of-the-art guidelines [4]. We envision GAI to aid researchers in the
following five main SS tasks. We also focus on the topic of multivocal
literature review (MLR), aligning with the growing community interest, and
referring to the guidelines by [5].

Section Structure. The following sections and subsections adhere to the
guidelines provided by [4] as our structure for further discussion. We discuss
classical methodologies and how GAI supplements or automates these
methodologies.

### Research Identification

#### Create a search strategy

The first step of an SLR plays a crucial role in identifying relevant studies
related to the research topic or question under consideration. Researchers
develop the search string using specific strategies, including predefined
keywords and boolean operators, to look for relevant studies [4].

Classical Approach. One of the classical approaches to structure a search
string is to use PICO [6, 7].

PICO [7] is a framework borrowed from evidence-based practice, originating
from healthcare, to build a search string. PICO stands for: P:
Patient/Population/Problem; I: Intervention; C: Comparison; O: Outcome.

GAI Approach. According to recent research works [8], we suggest chat-based
LLM to support PICO string creation for software engineering studies as below.
Text-oriented LLMs can generate structured text following human specification
[9]. Therefore, they can generate PICO terms and recommend the search string
of online libraries [8]. The researchers can leverage this recent GAI
implementation to make canvas creation possible. So, we suggest using vision-
enabled LLM to depict the search string, such as a tree, to make it easy for
researchers to comprehend and modify the query accordingly. Furthermore, GAIs
can search automatically for papers or articles related to the user query over
several web-based libraries and combine them. This will save the researcher
effort and time.

#### Publication Bias

Publication bias is a peculiar type commonly associated with academic research
publications, where published research findings skew systematically toward
positive or statistically significant ones. Simultaneously, studies with
negative or non-significant findings are less likely to be published, or even
if published, they are published in a less visible way [4]. Classical
Approach. Multivocal Litterature Review. The MLR approach prevents publication
bias by bringing various perspectives, sources, and voices into the systematic
review process. It is appreciated that when the systematic review heavily
relies on the published academic literature, the conclusions could be heavily
biased since publication bias generally results in skewness towards positive
or statistically significant results. On the other hand, MLR aims to
incorporate sources and perspectives of all types, whether published or
unpublished studies and web sources, to get a more profound and objective
knowledge of the issue under study (a.k.a. gray literature).

GAI Approach. LLMs, like Google’s Gemini, Perplexity, or OpenAI GPT4o, are
capable of surfing the web to get fresh data relevant to the user’s questions.
Searching for grey literature also requires many searches on search engines
like Google and Yahoo, social media like YouTube and Yelp, or focused forums
such as StackOverflow.

On the other hand, gray literature is painfully non-tabulated and needs lots
of formatting. Prompt engineering, i.e., the design of the specific LLMs’
prompt, allows the researchers to tell the model how to orient and make sense
of the sheer amount of data, what to look for, and how to shape the output in
the easiest possible way. To the best of our knowledge, no prior study has
suggested the adoption of LLM in the conduct of an SLR. Therefore, we aim to
advance the state-of-the-art, adding to the empirical validation of our vision
a separate section surveying the grey literature to evaluate the LLM’s
performance.

#### Conducting the search

With the research string at hand, researchers must, without exception, adjust
the manually crafted search string for every digital library. These may be ACM
Digital Library (ACM DL), IEEEXplore, SpringerLink, ScienceDirect, and Scopus.
Classical Approach Searching through the different digital libraries is a
frustrating task. ACMDL, IEEEXplore, and Scopus have query languages, which
are primarily proprietary and have the same - or very similar - field names.
However, the implementation and field availability differ widely. More
specifically, the three major libraries do not share the usage of, e.g., the
wildcard characters ’*’ for matching part of the word. Moreover, not all allow
copy-pasting of a query string externally crafted.

GAI Approach In this step, we suggest two different GAI-based approaches,
given the availability of internet-enabled GAI (iGAI) [10]. When using an
iGAI, we see the role of, e.g., an LLM model for the role of search query
creation. We suggest fine-tuning an LLM to create a chatbot for each digital
library. We may fine-tune by sending pairs of messages such as search strings
as user input, a search query for each digital library, and an LLM reply.
Otherwise, an iGAI can create the search query and search by compiling and
executing a predetermined script that allows API access to the digital
libraries of interest. It is worth noticing that not all digital libraries
allow for API access or computational access for library scraping. We address
this acknowledged issue in Section Limitations. At the end of this step,
researchers have the collection of primary studies collected from all the
digital libraries of interest.

#### Documenting the Search

According to [4], conducting an SS should be transparent and replicable.
Therefore, different tools and conceptual frameworks assist researchers in
documenting the search protocol and findings. For instance, PRISMA, which
stands for Preferred Reporting Items for Systematic Reviews and Meta-Analyses,
is a set of guidelines (checklists) designed to help researchers improve the
reporting of systematic reviews and meta-analyses.

Classical Approach. The classical approach of documenting the search process
is quite labor-intensive, in which a PRISMA framework is prepared manually in
textual and graphical form. Researchers manually document each step of the
search process, including the selection of digital libraries, search strings,
inclusion/exclusion criteria, dates of the search, and the results of the
searches.

It is a manual process and is considerably time-consuming with a high
potential for errors, as it deals with many digital libraries and iterations
of search.

GAI Approach. GAI approach enables the automation of the PRISMA workflow. More
specifically, researchers can fine-tune an LLM designed according to the
PRISMA guidelines to generate the PRISMA workflow automatically.

### Study Selection

According to [4], the study selection step identifies and includes all
relevant studies that meet predefined inclusion criteria, guaranteeing that
the studies being synthesized are pertinent to the researchers’ investigation.
Researchers establish inclusion and exclusion criteria by considering previous
SS in the specific topic and based on the researchers’ own research
experience. According to [4], usual criteria include study language,
publication venue, types of participants, and research design choice.

Classical Approach. Researchers manually review each paper examining the
abstract and keywords or giving a quick read of the entire work. Based on the
researchers’ understandings, they exclude or include the studies for further
screening. Usually, SS encompass analyzing papers in the order of the
hundreds; therefore, it is evident that the preliminary screening time can be
computed at least in weeks, if not months. Moreover, more than one author must
invest the same amount of time reviewing the same article for eventual
disagreement. The extent of agreement among authors applying the
inclusion/exclusion criteria should be computed with the usual inter-rater
agreement (IRA) statistic such as Kendal’s $\tau$ or Cohen’s $\kappa$ .

GAI Approach. According to [11], LLM can be fine-tuned to apply user-defined
criteria to prompts. Nonetheless, to our knowledge, no prior studies have
investigated using LLMs to apply criteria to study selection in an SS.
Therefore, we can extend the [11] approach by applying inclusion/exclusion
criteria when evaluating papers. Moreover, we can expand further with
Retrieval-Augmented Generation (RAG). RAG feeds external knowledge to LLMs as
’context’ in the chat. Therefore, we can dynamically update the criteria,
leaving only the generic message/reply structure for fine-tuning.

### Study Quality Assessment

QA of an SS is also defined as explicit inclusion/exclusion criteria according
to [4]. Unfortunately, there is no silver bullet for QA checklists, mostly
tailored to researchers’ investigation topics. Classical Approach In the
classical approach, researchers usually follow the guidelines of [4] that
provide a list of QA checklists with designs to target for different study
designs. Researchers need to choose relevant QA checklists according to the
designs of the studies included in the review, like randomized controlled
trials, cohort studies, case-control studies, literature reviews, and mapping
studies. Each included study is manually evaluated against the relevant QA
checklist chosen to point out the methodological strength and related risk of
bias. This step is usually performed by several reviewers who independently
check the quality of the study. The studies may be given scores or put into
different categories based on the degree of compliance with QA criteria.
Higher scores indicate high quality or low risk of bias. The process is quite
time-consuming and subjective, and it relies on the interpretation and
judgment of humans.

GAI Approach GAI models can be instructed with QA criteria and guidelines so
that they can go ahead and perform an independent judgment on collected
studies. GAIs will be able to comb huge amounts of papers in a short time,
with the ability to respond instantly regarding methodological strengths,
biases, and possible concerns. Although GAIs are random in nature (see
Limitations) multiple GAIs allows remediating human subjectivity. Moreover,
models do not suffer fatigue, thus assuring consistency of application across
all studies. Notably, no prior researches have been conducted with respect to
the effectiveness of GAIs for QA assessments in SS.

### Data Extraction

According to [4], data extraction involves collecting and recording relevant
information from the selected studies.

Classical Approach. The classical data extraction method involves researchers
analyzing selected studies individually to identify and extract the requisite
data. Researchers design a standard form or template to document data
specifics, i.e., study characteristics, methodological approaches, sample
size, results, and other details. Then, this form is pilot-tested on a sub-set
of studies to ensure the form captures all data required and is user-friendly.
Then, researchers read each study, identify the relevant information, and
record this with the extraction form. More than one reviewer should perform
data extraction to validate the accuracy and consistency of the data
extracted. Hence, researchers must again calculate an IRA to assess biases and
misclassification. This is time-consuming, labor-intensive, and prone to human
error. Besides, the process can be overwhelming if the number of studies is
large.

GAI Approach LLM can be fine-tuned or instructed to answer in a specific
format and provide or ask to generate the data extraction form. More so, the
models can run scripts or create them on the fly to automate repetitive
activities, such as data extraction from tables, figures, and text.

Moreover, GAI systems can offer immediate feedback and validation and flag
possible inconsistencies or missing data in real-time. Finally, GAIs allow
researchers to process large-scale document collection within a fraction of
the time taken by manual methods [9]; thus, it is possible to review large-
scale literature exhaustively.

### Data Synthesis

According to [4], data synthesis involves extracting relevant information from
the extracted data. Therefore, this last step aims to grasp hidden patterns,
research gaps, and current trends in the research landscape. Thus, researchers
interpret the synthesized data to draw conclusions and implications based on
the context of the SS research questions.

Classical Approach. Classical data synthesis procedures include thematic
analysis that extracts common themes and concepts across studies. Or
statistical analysis. Depending on the nature of the data, statistical methods
may consist of meta-analysis or quantitative synthesis of the findings of
studies.

GAI Approach. Finally, GAI models can perform data analysis and synthesis
tasks on large-scale studies with ease. The models can process the collected
data much faster than using manual methods. Moreover, GAIs comprehend the
semantic context of the collected data and, therefore, can derive the meaning
of the text, deducing insightful aspects of the studies. Finally, an advanced
GAI model can read, analyze, and output images, tables, and other forms of
multimodal data to comprehensively synthesize the original studies’ findings.

## Roadmap

Table 1: Metrics and Measurements for Empirical Validation Step |  Metrics |  Measurement
---|---|---
Identification of Research |  Precision, Recall, F1-score |  Compare the relevance of studies retrieved by GAI-generated search strings against manually crafted ones
Publication Bias |  Diversity of sources, Comprehensiveness |  Evaluate diversity and comprehensiveness of sources retrieved by traditional vs. GAI-based methods
Conducting the Search |  Retrieval precision, Retrieval recall, Search time |  Assessing the efficiency and effectiveness of GAI-generated search queries compared to manual ones
Documenting the Search |  Completeness of documentation, Adherence to PRISMA guidelines |  Evaluate completeness and adherence of GAI-generated documentation to PRISMA guidelines
Study Selection |  Inter-rater agreement (e.g., Cohen’s kappa) |  Measure consistency between human reviewers and GAI models in selecting studies
Study Quality Assessment |  Agreement between human reviewers and GAI models (e.g., Cohen’s kappa) |  Assess agreement in evaluating study quality between human reviewers and GAI models
Data Extraction |  Precision, Recall, F1-score |  Compare accuracy and completeness of data extraction by human reviewers vs. GAI models
Data Synthesis |  Coherence of synthesized findings, Insightfulness of interpretations |  Compare coherence and insightfulness of synthesized findings from human reviewers vs. GAI models

This section sets up the road map for our idea to come to fruition. Our road
map consists of two steps: implementation of the framework and empirical
validation. Table1 shows the metrics we plan to use for measuring the
effectiveness of the GAI approaches of each SS step.

Feasibility. Our work can contribute to our field by highlighting the current
state-of-the-art GAIs-assisted SS tasks and spotting research gaps for future
research efforts. Therefore, our vision is based on previous and recent
studies assuring its feasibility.

Implementation. We intend to follow up on this vision by implementing a
comprehensive suite for autonomous SS. Our implementation will exploit the
contributions of the previous studies and new and original contributions as we
have presented in Section Design.

Empirical Validation. We will empirically validate the proof of concept for
our framework. Our validation will be based on established guidelines [12] and
will focus on performing an SLR comparing human results for each SLR step with
the GAI ones. More in detail, we foresee a validation protocol for the GAI
approach for SS.

Table 1 presents the metrics and the rationale to empirically validate each
phases.

## Limitations

This section presents the acknowledged limitations to our study and the
associated mitigation strategy.

### Scraping Digital Libraries

Internet-enabled GAIs can retrieve and scrape websites to retrieve up-to-date
information. Unfortunately, publisher agreements can limit scientific
publication scraping by limiting its access or denying it altogether. For
instance, the 2024-2026 ACM Digital Library Tiered-Band Open Access Model
Agreement Terms and Conditions
333https://libraries.acm.org/binaries/content/assets/libraries/dl/2024-2026-acm-
open-blank-agreement-template---consortium_may-2024.docx section IV, par. 6,
allows for computation access to authorized user. In this context,
computational access allows autonomous scraping of scientific publications in
the ACM Digital Library with no specific limitations. Nonetheless, GAI is
still technically a novel “issue” in the publication field. Therefore,
publishers can implement measures to limit access specifically to GAI. We can
alleviate this issue by providing a means for the GAI to accept personal or
institutional tokens. Nonetheless, the scientific community should deeply
reflect on the implications of having GAIs scraping scientific knowledge.
Unfortunately, the discussion on this topic is too broad to be comprehensively
addressed in a section of this vision paper. Nonetheless, we deemed it
essential to acknowledge it.

### Replicability of the findings

Replicating AI’s output is a major issue in AI-based SE research [9]. Also,
randomness is involved in more straightforward classification tasks.
Therefore, the outputs may vary on each use [9].

Replication packages are essential to reproduce the study analysis [6].
Nonetheless, if randomness is involved, obtaining the same authors’ results is
challenging.

We can mitigate this limitation when GAI platforms allow for custom setting
specific parameters to grant a consistent output, for instance, considering
OpenAI API References444https://platform.openai.com/docs/guides/text-
generation/how-should-i-set-the-temperature-parameter) two parameters to
reduce the randomness of the models and aid in consistency are provided,
namely “temperature” where lower values result in more consistent outputs and
“seed” allowing the control of the initial randomness.

Moreover, multiple GAI models should be employed to strengthen the
reliability. A single model should perform no single step if no prior cross-
evaluation occurs. Our empirical validation will also address this issue.

### Explainability of the GAI models

LLMs are increasingly employed in essential areas like healthcare, finance,
and policy. Therefore, we must ensure that domain experts can effectively
collaborate with these models.

We refer to [13] definition of explainable AI: “Given an audience, an
explainable Artificial Intelligence produces details or reasons to make its
functioning clear or easy to understand.”

Explainability serves as a means to connect human decision-makers with machine
learning models, helping to bridge the gap between the two [13].

According to [13], we define our audience as the researchers conducting SS.
Therefore, any insight and justification made by the explainable model should
be easy for them to follow. Transparency and applicability are essential to
allow researchers to trust the suggestions and outputs of GAIs used during the
entire SLR process [9, 4].

Explainability is hard to achieve due to GAI models’ complexity and black-box
nature. Thus, while on the one hand, GAI models are most likely to contribute
toward increasing the potential for improvements in various aspects of the SLR
process, simultaneously, on the other hand, their lack of explainability
remains one of the major bottlenecks for the development of trust and
collaboration with a community of researchers towards the reasoning behind
GAIs output. It is worth noticing that, due to the volatility of technology,
which advances month by month, GAI approaches that today’s works may not be
tomorrow, and vice versa; they can become obsolete in no time. Future studies
must build methods and techniques to enhance explainability in GAI models to
meet the demand from researchers conducting SS.

## Ethical Considerations

Our world is fast-changing. Until recently, AI used to categorize things, but
now it is serving humans imagination. Therefore, according to [14], SE
researchers and practitioners should not “look away”.

We need to ensure that GAI models support human researchers and not replace
them. They should improve human efficiency by making tedious tasks, such as
data extraction and search, quicker and less demanding while allowing
researchers more time for critical thinking on higher-level goals and research
objectives. Such collaboration would help preserve human oversight and
judgments,i.e., the embodiment of ethical considerations, critical thinking,
and domain expertise that guide the research process. Transparency and
explainability in GAI outputs ensure trust and integrity while conducting SS.

## Conclusions

In our study, we presented a brief overview of the possible roles of GAI in
EBSE to highlight their potential to make several SLR steps more time and
resource-efficient alongside a validation protocol.

## References

  * [1] Barbara A. Kitchenham and Tore Dybå  “Evidence-Based Software Engineering”  In _26th International Conference on Software Engineering (ICSE 2004), 23-28 May 2004, Edinburgh, United Kingdom_ IEEE Computer Society, 2004, pp. 273–281  DOI: 10.1109/ICSE.2004.1317449
  * [2] Katia R Felizardo and Jeffrey C Carver  “Automating systematic literature review”  In _Contemporary empirical methods in software engineering_ Springer, 2020, pp. 327–355
  * [3] Ronnie E.. Santos and Cleyton V.. Magalhães  “The use of systematic reviews in evidence based software engineering: a systematic mapping study”  In _2014 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM ’14, Torino, Italy, September 18-19, 2014_ ACM, 2014, pp. 53:1–53:4  DOI: 10.1145/2652524.2652553
  * [4] Barbara Kitchenham  “Guidelines for performing systematic literature reviews in software engineering”  Technical report, ver. 2.3 ebse technical report. ebse, 2007
  * [5] Vahid Garousi and Michael Felderer  “Guidelines for including grey literature and conducting multivocal literature reviews in software engineering”  In _Information and Software Technology_ 106, 2019, pp. 101–121  DOI: https://doi.org/10.1016/j.infsof.2018.09.006
  * [6] Matteo Esposito and Davide Falessi  “VALIDATE: A deep dive into vulnerability prediction datasets”  In _Information and Software Technology_ Elsevier, 2024, pp. 107448
  * [7] A Ralph Henderson  “Evidence-Based Medicine—How to Practice and Teach EBM. D. L. Sackett, W. S. Richardson, W. Rosenberg, and R. B. Haynes. New York: Churchill Livingstone, 1997, 250 pp. Paperback, 24.99. ISBN 0-443-05686-2.”  In _Clinical Chemistry_ 43.10, 1997, pp. 2014–2014  DOI: 10.1093/clinchem/43.10.2014
  * [8] Madhusudan Ghosh and Shrimon Mukherjee  “AlpaPICO: Extraction of PICO frames from clinical trial documents using LLMs”  In _Methods_ 226 Elsevier, 2024, pp. 78–88
  * [9] Matteo Esposito and Francesco Palagiano  “Leveraging Large Language Models for Preliminary Security Risk Analysis: A Mission-Critical Case Study”  In _Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering_ , EASE ’24  Salerno, Italy: Association for Computing Machinery, 2024, pp. 442–445  DOI: 10.1145/3661167.3661226
  * [10] Nikhil Sharma and Q Vera Liao  “Generative Echo Chamber? Effect of LLM-Powered Search Systems on Diverse Information Seeking”  In _Proceedings of the CHI Conference on Human Factors in Computing Systems_ , 2024, pp. 1–17
  * [11] Tae Soo Kim and Yoonjoo Lee  “EvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteria”  In _Proceedings of the CHI Conference on Human Factors in Computing Systems_ , CHI ’24  New York, USA: Association for Computing Machinery, 2024  DOI: 10.1145/3613904.3642216
  * [12] Per Runeson and Martin Höst  “Guidelines for conducting and reporting case study research in software engineering”  In _Empir. Softw. Eng._ 14.2, 2009, pp. 131–164
  * [13] Alejandro Barredo Arrieta and Natalia Díaz-Rodríguez  “Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI”  In _Information fusion_ 58 Elsevier, 2020, pp. 82–115
  * [14] Brittany Johnson and Tim Menzies  “Ethics: Why Software Engineers Can’t Afford to Look Away”  In _IEEE Software_ 41.1, 2024, pp. 142–144  DOI: 10.1109/MS.2023.3319768

Matteo Esposito is a post-doctoral researcher at the University of Oulu and
has served as the chairman of the ACM Rome Tor Vergata Student Chapter for
eight years. His primary research interest lies in Secure Software
Engineering, Large Language Models for SSE, and Quantum Software Engineering.
He earned his Ph.D., MSc, and BSc in Computer Science Engineering from the
University of Rome Tor Vergata, Italy, and is an ACM and IEEE Computer Society
member. Contact him at<EMAIL_ADDRESS>
Andrea Janes is an associate professor at the Free University of
Bozen/Bolzano. He was previously a senior lecturer and researcher at the FHV
Vorarlberg University of Applied Sciences in Dornbirn, Austria, and a
researcher at the Free University of Bozen/Bolzano, Italy. He holds a Master’s
degree in Business Informatics from the Vienna University of Technology and a
PhD in Computer Science (with honors) from the University of Klagenfurt,
Austria. He obtained the habilitation in Computer Science and Information
processing systems. He is particularly interested in Lean and Agile approaches
to software engineering, value-based software engineering, empirical software
engineering, software testing, and technology transfer. Contact him at
<EMAIL_ADDRESS>
Davide Taibi is a Full Professor at the University of Oulu, where he heads the
M3S Cloud research group. His research is mainly focused on Cloud and Edge
Software Architecture. In particular, my research group is working on
migration to cloud-native technologies; Good practices, patterns, and anti-
patterns for cloud-native technologies, including edge computing, serverless,
and microservices; Architectural Degradation and Architectural Technical Debt
- Quality of Open Source Software. Prediction of Open Source Adoption Risk,
Open Source Abandonment Prediction. He has been a member of the International
Software Engineering Network (ISERN) since 2018. Formerly, he worked at the
Free University of Bolzano, Technical University of Kaiserslautern, Germany,
Fraunhofer IESE - Kaiserslautern, Germany, and Università degli Studi
dell’Insubria, Italy. In 2011 he was one of the co-founders of
Opensoftengineering s.r.l., a spin-off company of the Università degli Studi
dell’Insubria. Contact him at<EMAIL_ADDRESS>
Valentina Lenarduzzi is an Associate Professor (Tenure Track) at the
University of Oulu, Finland, who specializes in data analysis in software
engineering, with a focus on software quality, maintenance, and evolution. She
holds a Ph.D. in Computer Science from the University of Insubria (Italy),
where she researched effort estimation and data analysis in software
engineering. Valentina has also held postdoctoral positions at LUT University
(Finland), Tampere University (Finland), and the Free University of Bozen-
Bolzano (Italy). She has contributed as a visiting researcher at institutions
such as the Technical University of Kaiserslautern and the Fraunhofer
Institute for Experimental Software Engineering IESE (Germany). Contact her at
<EMAIL_ADDRESS>