# Evidence of fractal structures in hadrons

Rafael P. Baptista  Instituto de Física - Universidade de São Paulo - Brazil
Lucas Q. Rocha  Instituto de Física - Universidade de São Paulo - Brazil  D.
P. Menezes  Departamento de Física, CFM-Universidade Federal de Santa
Catarina, Florianópolis, SC-CP. 476-CEP 88.040-900, Brazil  Luis A. Trevisan
Departamento de Matematica e Estatística - Universidade Estadual de Ponta
Grossa, Brazil  Constantino Tsallis  Centro Brasileiro de Pesquisas Físicas
and National Institute of Science and Technology of Complex Systems, Rua
Xavier Sigaud 150, Rio de Janeiro-RJ 22290-180, Brazil 5Santa Fe Institute,
1399 Hyde Park Road, Santa Fe, 87501 NM, USA 6Complexity Science Hub Vienna -
Josefstädter Strasse 39, 1080 Vienna, Austria  Airton Deppman  Instituto de
Física - Universidade de São Paulo - Brazil

(July 2023)

###### Abstract

This study focuses on the presence of (multi)fractal structures in confined
hadronic matter through the momentum distributions of mesons produced in
proton-proton collisions between 23 GeV and 63 GeV. The analysis demonstrates
that the $q$-exponential behaviour of the particle momentum distributions is
consistent with fractal characteristics, exhibiting fractal structures in
confined hadronic matter with features similar to those observed in the
deconfined quark-gluon plasma (QGP) regime. Furthermore, the systematic
analysis of meson production in hadronic collisions at energies below 1 TeV
suggests that specific fractal parameters are universal, independently of
confinement or deconfinement, while others may be influenced by the quark
content of the produced meson. These results pave the way for further research
exploring the implications of fractal structures on various physical
distributions and offer insights into the nature of the phase transition
between confined and deconfined regimes.

##  1 Introduction

The investigation of strongly interacting particles has always faced the
challenge of dealing with regimes where perturbative QCD (pQCD) cannot provide
accurate results. The structure of hadrons and the existence of quark-gluon
plasma (QGP) are notorious examples of systems where non-perturbative QCD
(npQCD) must be considered.

The large amount of data provided by high-energy colliders in the TeV energy
region has revealed several aspects of strong interactions in npQCD. One of
the most ubiquitous features is the q-exponential behaviour observed in
particle momentum distributions [1, 2, 3, 4], suggesting that nonextensive
statistical mechanics, based on the nonadditive entropy Sq, [5, 6] is the
appropriate framework to study the complex system formed during collisions.
The emergence of non-extensive statistics (or $q$-statistics for short) in
Yang-Mills theories has been demonstrated in [7], which uses self-energy
interactions and the renormalization group equation to show how thermofractal
structures can emerge in the interacting fields. These thermofractal
structures give rise to non-extensive statistics and $q$-exponential
distributions [8].

Assuming that the statistical mechanics grounded on Sq is the correct
framework to study the thermodynamics of QCD systems, it is natural to extend
it to describe high-energy processes as well. Hagedorn’s Self-Consistent
Thermodynamics [9] was generalized by incorporating non-extensive statistics
[10], resulting in a critical temperature, similar to the original theory, but
with a new formula for the hadron mass spectrum. The new formula provides a
better description compared to the original Hagedorn’s formula, accurately
describing the mass spectrum from the heaviest known hadrons down to the pion
mass [11]. This result suggests the existence of fractal structures in hadrons
beyond those already investigated in the deconfined regime. Could the confined
quark matter inside hadrons show fractal aspects? The present work offers a
systematic analysis of meson production in hadronic collisions at energies
below 1 TeV to address this question. It will be shown that the momentum
distributions of produced mesons resulting from collisions in the range of 23
GeV to 63 GeV exhibit many features expected in thermofractals.

Figure 1: Example of the simplest subprocess for meson production in $pp$
collision.

A subproduct of this work is the reconciliation of QCD with the early
theoretical approaches proposed in the 1960s by Chew, Frautshi [12] and
Hagedorn, among others. The self-consistent calculation, also known as the
bootstrap approach, for describing hadrons was an important line of research
whose prominence diminished with the emphasis given to quarks and QCD. The
understanding of fractal structures in Yang-Mills Theories unifies the two
approaches, clarifying and simplifying several aspects of Quantum
Chromodynamics.

The results obtained in the present study unveil new possibilities for
investigating the effects of fractal structures in hadronic processes.
Experimental data on the hadron structure are produced in HERA, JLab, and
NICA, among others, while the forthcoming Electron-Ion Colliders (EIC) offer
promising prospects for experimentally probing the intricate nature of the
hadronic structure [13, 14]. Consequently, the present work appears to hold
substantial relevance for the advancement of these colliders. Moreover, given
the presence of fractal structures in the Quark-Gluon Plasma (QGP), the
identification of similar structures in the hadronic phase may impose
constraints on the confinement-deconfinement phase transition. Thus, the
present findings should contribute significantly to the understanding of
fundamental characteristics associated with hadronic matter.

The non-extensive statistics has already been used in hadronic gas models [15,
16, 17]. The present approach differs from the previous works in the use of
the running-coupling and microscopic calculation of the subprocess involved in
hadron collisions.

##  2 Methods

For the investigation of the thermofractal structures in hadrons, this work
analyses the momentum distributions of mesons produced in $pp$ collisions. The
data is collected in the centre-of-mass frame at approximately zero rapidity,
where the colliding protons have momentum $P$ and total energy $E=\sqrt{s}$
[18]. In the relevant subprocess, two quarks interact and form a new meson,
while the other quarks are considered spectators. It is assumed that almost
all energy of the process is consumed in the production of the meson, so the
momentum of the remaining system is negligible. This assumption limits, to
some extent, the power of the present analysis, but it does not interfere with
the pursued objectives. In this configuration, if $\varepsilon$ is the
produced meson energy and $\varepsilon_{1}$, $\varepsilon_{2}$ are the
energies of the interacting quarks, then
$\varepsilon_{1}=\varepsilon_{2}=\varepsilon/2$.

Fig. 1 shows an example of the meson production process to assist the
discussion of some relevant aspects that will affect the analysis of the
experimental data. Since we are in the confined regime of the quark matter,
the meson production involves at least three vertices. Using the results of
the thermofractal theory for QCD, we have that each vertex involves a running
coupling given by

$g(\varepsilon)=G_{o}e_{q}(\varepsilon_{1},\lambda_{1};q)e_{q}(\varepsilon_{2},\lambda_{2};q)\,,$
(1)

where $G_{o}$ is a constant associated with the maximum value of the coupling,
the $q$-exponential distribution is given by

$e_{q}(\varepsilon,\lambda;q)=\left[1+(q-1)\frac{\varepsilon}{\lambda}\right]^{\frac{-1}{q-1}}\,,$
(2)

with $\varepsilon=\varepsilon_{1}+\varepsilon_{2}$, where $\varepsilon_{1}$
and $\varepsilon_{2}$ are the energy carried by the interaction partons in
each proton, represented by the indexes $1$ and $2$, $\lambda$ is a scale
parameter (effective temperature) and $q$ is the entropic index associated
with Sq.

For fractal structures, one expects to observe the characteristic
$q$-exponential distributions due to the form of the running coupling. Besides
the form of the distribution, it is expected that [7]

$\frac{1}{q-1}=\frac{11}{3}N_{c}-\frac{4}{3}\frac{N_{f}}{2}\,,$ (3)

where $N_{c}$ and $N_{f}$ are, respectively, the numbers of colours and
flavours. With $N_{c}=3$ and $N_{f}=6$, the formula above gives $q=8/7\simeq
1.14$, in agreement with high-energy experimental data analyses.

The work performs three scenarios of systematic analyses, as discussed below,
where the experimental data is fitted by three different models, with
increasing physical significance, where the momentum distribution is given by:

  1. 1.

A single $q$-exponential function, with

$\frac{d^{3}\sigma}{dp^{3}}=\sigma_{o}e_{q}(\varepsilon,\lambda;q)\,,$ (4)

where $q$, $\sigma_{0}$, and $\lambda$ are adjustable parameters.

  2. 2.

The product of three $q$-exponential functions with

$\frac{d^{3}\sigma}{dp^{3}}=\sigma_{o}e_{q}(\varepsilon_{1},\Lambda;\bar{\bar{q}})e_{q}(\varepsilon_{2},\Lambda;\bar{\bar{q}})e_{q}(\varepsilon,\lambda;q)\,,$
(5)

where $\varepsilon_{1}=\varepsilon_{2}=\varepsilon/2$,
$(\bar{\bar{q}}-1)^{-1}=[(q-1)/q]^{-1}$, and the parameters $\sigma_{o}$,
$\lambda$ and $\Lambda$ are adjusted. The parameter $\bar{\bar{q}}$ takes into
account that the parton momentum distributions in the hadron emerge in a
complex interaction of an equilibrated system [19]. The use of $\bar{\bar{q}}$
can be controversial. A thorough discussion about this point can be found in
Refs. [4, 3, 1, 20], but it can be advanced that results very similar to those
shown here can be obtained by using $\bar{\bar{q}}=q$.

  3. 3.

The product of three $q$-exponential functions, as in Scenario 2, but where
$\lambda$ is fixed at a value that will be found in Scenario 2 and the
parameters $\sigma_{o}$ and $\Lambda$ are adjusted;

Figure 2: Best fit results for adjustable parameters in Scenario 1.

The accumulated experience of fitting $q$-exponential distributions to high-
energy data in the deconfined regime has underlined the importance of
addressing parameter correlations to accurately extract pertinent physical
information [21, 11, 22]. This study incorporates the methodologies
established in those analyses.

##  3 Results

This section presents the results of each of the scenarios previously
delineated and gives limited and preliminary discussion for each scenario. A
global discussion will be presented in the next section.

###  3.1 Scenario 1

In the first scenario of the analysis, the best fits for the set of
experimental data are displayed in Fig. 2. The single $q$-exponential function
fitted the experimental data with the parameters presented in Table 1. The
overall agreement is good, with reduced chi-square in the range of 1 to 4. The
parameters $q$ and $\lambda$ reported in that table are plotted as a function
of the collision energy in Fig. 3. The worst fitting of the model to
experimental data happens for $K^{+}$ at 63 GeV, and the same relative result
will be observed in the other scenarios of the analysis. By a simple visual
inspection, it is possible to observe that data set for this case departs from
the general shape observed for all the other cases. This suggests that the
experimental analysis and data reduction could be revisited.

Figure 3: Behaviour of the parameters $q$ (left) and $\lambda$ (right) as a
function of the collision energy, according to the analysis in Scenario 1.

The values for both $\bar{q}$ and $\lambda$ appear in the range
$1.04<\bar{q}<1.10$ and $0.10<\lambda<0.15$, except for one set of data on
kaon production. The mean values are $q=1.069\pm 0.004$ and $\lambda=0.121\pm
0.003$. These results show that fractal structures may be present in the
hadron structure, but the values for both parameters are slightly different
from those found in the QGP multiparticle production, where $q=1.14$ and
$\lambda$ is close to the pion mass, $m_{\pi}=140$ MeV. Additional information
reporting other aspects of the analysis can be found in the Supplementary
Material.

The main difference between the process analyzed here and the multiparticle
production at higher energies is the deconfined regime for quarks and gluons
that are determinants of the minimum number of vertices in the process. While
in QGP the partonic production involves a single vertex, in the hadronic
collisions the confinement of quarks and gluons imposes a minimum of two
vertices for any interaction to keep quarks and gluons confined. In the meson
production studied in the present work, a minimum of three vertices are
necessary, as depicted in Fig 1: one related to the meson production, where
the partons produced in each of the interacting protons merge to form the new
particle, and the other two at the independent interaction in each of the
colliding protons, where the partons are originated.

The presence of the independent vertices makes all the differences in the
process. Roughly speaking, each vertex contributes with a q-exponential, so
the expected exponent in the function is twice that for the single vertex,
that is, for hadronic interactions, we expect a value $\bar{q}$ such that

$\frac{1}{\bar{q}-1}=\frac{2}{q-1}\,,$ (6)

and using the expected value $q=1.14$ it results that $\bar{q}=1.07$, in
agreement with the result of the analysis in Scenario 1 reported in Table 1.

The fitting of the q-exponential function faces some challenging aspects
associated with the correlations in the parameter space. As an example, Fig 4
presents the ellipses corresponding to a constant chi-square value. Most of
the ellipses have axes that are not parallel to the parameters axes,
reflecting the existence of the correlation. For comparison, the value
$q=1.07$ is indicated through a vertical line. Observe that the ellipses fall
around the expected theoretical value, except in the case of the data set for
$K^{+}$ at $63$ GeV.

Figure 4: Correlations between the parameters $\sigma_{o}$ and $\lambda$ in
Scenario 1. Ellipses with axes that are not parallel to the parameters axis
unveil the correlation between the parameters.

The considerations made above must be clarified in many aspects, but they are
indicative that the thermofractal structures manifest themselves in the
confined quark matter. The main trouble with the present analysis is the use
of a single form of q-exponential. If the explanation for the different value
of $q$ is the presence of independent vertices, why both vertices should
depend on $\varepsilon$ and not on $\varepsilon_{1}$ or $\varepsilon_{2}$.
Aside from this, it is not evident that the same data that was adjusted by a
single q-exponential function can be fitted as a product of q-exponential
functions equally well. This will be investigated in the following scenarios.

###  3.2 Scenario 2

Figure 5: Best fit results for adjustable parameters in Scenario 2.

To describe the subprocess depicted in Fig.1 using the thermofractal approach,
we need to use the running coupling for each of the vertices in the process,
which can be done by considering that in proton $P_{1}$ one pair parton-
antiparton is created, contributing with a term
$e_{q}(\varepsilon_{1},\lambda_{1},q)\cdot
e_{q}(\bar{\varepsilon}_{1},\lambda_{1},\bar{q})$ to the cross-section, and
the same happens for proton $P_{2}$, contributing with
$e_{q}(\varepsilon_{2},\lambda_{2},q)\cdot
e_{q}(\bar{\varepsilon}_{2},\lambda_{2},\bar{q})$. In the third subprocess,
two partons merge to form a meson, reducing the number of q-exponentials since
$e_{q}(\bar{\varepsilon}_{1},\lambda_{1},q)e_{q}(\bar{\varepsilon}_{2},\lambda_{2},q)=e_{q}(\varepsilon,\lambda,q)$.
The parameter $\lambda$ is associated with the energy scale involved in the
meson production, while the parameter $\Lambda$ is associated with the
colliding hadrons energy scale. Due to the energy-momentum conservation, the
constraint $\bar{\varepsilon}_{1(2)}=\varepsilon_{1(2)}$ must be satisfied.
The use of the index $\bar{q}$ for the vertices in the colliding protons is
due to the fact that the vertex occur in the complex environment of the hadron
structure, which is supposed to be in equilibrium [19].

With the considerations made above, the differential cross-section will be
proportional to the product of three q-exponential functions, i.e.,

$\frac{d^{3}\sigma}{dp^{3}}=\sigma_{o}e_{q}(\varepsilon_{1},\lambda_{1},\bar{\bar{q}})e_{q}(\varepsilon_{2},\lambda_{2},\bar{\bar{q}})e_{q}(\varepsilon,\lambda,q)\,.$
(7)

The parameters $\lambda_{1}$ and $\lambda_{2}$ are related to the internal
characteristics of the colliding protons. Due to the symmetries of the two
protons involved in the process, $\lambda_{1}=\lambda_{2}=\Lambda$, and the
expression above reduces to the one proposed for Scenario 2 of the analysis,
given in Eq. 5

The same sets of data were adjusted by the cross-section in Eq. 7 with
$q=1.14$ and the parameters $\sigma_{0}$, $\Lambda$ and $\lambda$ adjustable.
The results of the analysis for Scenario 2 are displayed in Fig. 5. The
formula can fit well the data, resulting in a fairly low chi-square for most
of the cases, as reported in Table 2. This is itself a promising result since
the physical role of each term in the cross-section can be clearly understood.
Reinforcing the physical significance of the results, the values for $\Lambda$
and $\lambda$ can be compared to those found in the higher energy processes.
The best-fit values for the scale parameters are displayed in Fig. 7.

For $\lambda$ and $\Lambda$ there is no evident dependence on the produced
meson mass. The values for $\lambda$ fall in the range $0.5~{}GeV$ and
$0.25~{}GeV$, while for $\Lambda$ the values vary on a broader range, from
$0.2~{}GeV$ to $1.4~{}GeV$. Both parameters present some dependence on the
collision energy, which results to be stronger for $\Lambda$ than for
$\lambda$.

Figure 6: Visual representation of the correlations in the parameter space
during the fitting process. The plot shows the correlations for $\sigma_{0}$
with $\lambda$ and $\Lambda$, as used in Scenario 2 of the analysis.

It is important to notice that, as happens in high-energy studies, the fitting
process for the hadron regime analysis performed here needs to address the
difficulty of correlation among the adjustable parameters [21, 11, 22]. The
correlations are illustrated in the $\lambda$ vs $\Lambda$ projection of the
parameter space shown in Fig. 6, where the typical ellipses already observed
in analyses of multiparticle production can be observed. The average value
$\langle\lambda\rangle=0.122\pm 0.007$ is a reasonable value for almost all
sets analyzed and is close to the average values obtained in Scenario 1,
whilst for $\Lambda$ the values are spread over a larger range around the
average $\langle\Lambda\rangle=0.55\pm 0.06~{}GeV$. Similar correlations exist
between other pairs of parameters, leading to a small variation of the chi-
square for the best fit when some of the parameters are fixed at reasonable
values, as observed in Tables 1 to 3. Additional information on the
correlations in the parameter space can be found in the Supplementary
Material. Therefore, to simplify the analysis, it is reasonable to assume that
$\lambda$ is constant. This modification corresponds to Scenario 3 of the
analysis.

The average $\langle\lambda\rangle$ is close to the pion mass, so the
parameter $\lambda$ will be fixed to the pion mass, $m_{\pi}=0.140~{}GeV$.
Assuming this value, this work departs from a purely statistical analysis, but
gains in the physical interpretation of the result.

At this point, it is important to stress that other possibilities of analysis
are present and will be left for future work.

Figure 7: Behaviour of the parameters $\lambda$ (left) and $\Lambda$ (right)
as a function of produced meson mass, according to the analysis in Scenario 2.

###  3.3 Scenario 3

Figure 8: Best fit results for adjustable parameters in Scenario 3. Figure 9:
Behaviour of the parameter $\Lambda$ as a function of collision energy for the
different meson species according to the analysis in Scenario 3. Straight
lines are fitted to the data for guiding the eyes.

The third and last scenario of the analysis represents the most reliable
description of the process presented in this work. While the number of
adjustable parameters was three in the two previous scenarios, the third
scenario has only two free parameters in the fitting. The other parameters
were fixed to values corresponding to well-known physical quantities, such as
the pion mass for $\lambda$ and the thermofractal QCD value for $q$, which has
been found in agreement with the value resulting from the experimental
analysis. With fewer adjustable parameters, the model is subjected to a
stronger test when compared with the experimental data, and the best-fit
values for the parameter tend to present more significant physical content.

The model can describe accurately all sets of experimental data, as shown in
the plots in Fig. 8. The best-fit values for $\Lambda$ are given in Table 3
and in the plots presented in Fig. 9. The parameter $\Lambda$ shows dependence
on the produced meson and on the collision energy. In the range analysed the
behaviour of the scale $\Lambda$ is approximately linear for all mesons
studied.

The study of correlations in the parameter spaces brings evidence of strong
correlations involving the adjustable parameters, which can cause some
difficulties in obtaining the real values for the adopted parameters in each
scenario of the analysis. In particular, Fig. 10 shows that the correlations
between $\sigma_{o}$ and $\Lambda$, the only free parameters in this scenario
of the analysis, roughly distribute along a power-law curve. This form is due
to the known relationship between the multiplicative factor of the
q-exponential function and the multiplicative factor in the argument of the
function, as demonstrated in Ref. [6] of the work, on page 35.

Figure 10: Correlations between the parameters $\sigma_{o}$ with $\Lambda$ in
Scenario 3.

##  4 Discussion and conclusion

The results obtained here provide at least some preliminary answers to the
questions posed in the Introduction. The findings show that fractal structures
may be present in the hadron structure, as the momentum distributions of the
produced mesons in $pp$ collisions can be accurately described by
q-exponential functions. The analysis progressively unveils the role of the
fractal structure in the subprocesses involved in the hadronic interactions.

In the first scenario of the analysis, a naive model of the fractal structure
was applied, and it was already possible to observe a good fit of the data
with values for the parameters $q$ and $\lambda$ that seemed to be consistent
with those observed in high-energy multiparticle production. Even with such a
naive approach, it was possible to find evidence of fractal structures with
characteristics similar to those found in the deconfined regime. Interpreting
the results from Scenario 1, it became evident that at least two vertices must
be involved in the confined regime to ensure the confinement of quarks and
gluons.

In Scenario 2, the subprocess involving partons from both colliding protons
was described more accurately than in the first scenario, introducing a new
parameter, $\Lambda$, while fixing $q=1.14$. The parameter $\Lambda$
represents the energy scale of the colliding protons, while the parameter
$\lambda$ represents the energy scale of the produced meson. The results
obtained in this scenario showed that the model for the momentum distribution
can describe the data, and the parameter $\lambda$ could be considered
independent of the collision energy and the produced meson species, although a
weak dependence on those variables cannot be completely ruled out. On the
other hand, variations in the parameter $\Lambda$ with energy and particle
mass were more evident.

In Scenario 3, a more realistic model was adopted, with only two adjustable
parameters, $\sigma_{o}$ and $\Lambda$, while keeping $q$ fixed to the same
value as before, i.e. q =1.14, and fixing $\lambda$ to the pion mass. Even
with these constraints, it was possible to describe the data, and the
behaviour of $\Lambda$ with the particle species and the collision energy was
analyzed.

The main aspects that can be inferred from the more realistic scenario are as
follows: the scale $\Lambda$ depends on the particle species and the collision
energy. While the energy dependence, for the range considered in the present
work, can be considered approximately linear for all particles analyzed, the
rate of increase of $\Lambda$ with the collision energy depends on the quark
composition of the created meson. Specifically, for pion and rho, which are
formed by $(u,\bar{d})$ valence quarks, $\Lambda$ increases at approximately
the same rate, whereas for the Kaon, formed by $(u,\bar{s})$ valence quarks,
the parameter increases at a faster rate. The different rates of increase in
$\Lambda$ cannot be attributed solely to the differences in meson masses,
since pion and rho have similar slopes with different masses, thereby it may
be associated with the different quantum numbers of the mesons.

Figure 11: Analysis of the residues in the fitting process for typical cases
using the Scenario 3 analysis. Additional results for all scenarios can be
found in the Supplementary Material.

Being more realistic, the third scenario allows the investigation, in the
hadron structure, of one surprising, and yet not understood, effect that is
observed in high-energy collisions is the presence of log-periodic
oscillations in the transverse momentum distributions. The oscillations appear
in the normalized residues after the fittings of the experimental data with
the Sq distributions. This work investigates the presence of the same kind of
oscillations in the hadronic case. Fig. 11 reports typical results of the
analysis, and the complete report of the residues can be found in the
Supplementary Material. Although the shorter range of momentum distribution
limits the analysis, the results obtained do not allow one to conclude that
the same oscillations appear in the confined regime. The absence of the log-
periodic oscillations in the hadronic regime can be indicative of the
connection between the oscillation and collective behaviour of the QGP,
possibly associated with the flow.

The conclusions one can infer from the study presented in this work are the
following: It is likely that the fractal structure is present in confined
hadronic matter with characteristics similar to those observed in higher
energy collisions, where the deconfinement of quarks and gluons is expected.
The significance of this finding lies in demonstrating the persistence of the
fractal structure through the confined/deconfined phase transition.

The energy scale associated with the produced meson is the pion mass, and it
is similar QGP freeze-out temperature. This is another evidence that the same
fractal structure is present in hadrons as well as in QGP.

The scale $\Lambda$ depends on the meson mass, on the meson quark content and
on the collision energy. The rate of the scale increase with energy seems to
be independent of the quark content, at least considering the limited
experimental data analyzed here.

This work presents a systematic analysis of meson production in proton-proton
($pp$) collisions within a specific energy range. The main objective is to
demonstrate that the thermofractal structures observed in multiparticle
production at higher energies are also present in the confined region. By
investigating fractal structures in hadrons, the research contributes
significantly to understanding the presence of fractal effects in the hadronic
processes, illuminating the physical meaning of the fractal parameters. It is
observed that certain fractal parameters, specifically $q$ and $\lambda$, are
universal and independent of whether the hadronic matter is in the confined or
deconfined regime, while $\sigma_{o}$ and $\Lambda$ may be influenced by the
quark structure of the produced meson.

The study opens up opportunities for further investigation in terms of
collision energy and particle species. Fractal characteristics, such as scale
invariance and self-similarity, may have significant effects on observables
like Parton Distribution Function (PDF) [23], Transverse Momentum Distribution
(TMD) [24, 25], Generalized Momentum Distribution (GPD) [26, 27] and other
distributions that can be measured experimentally. These effects will be
explored more thoroughly in future experiments using the Electron-Ion Collider
(EIC) [28, 29]. In Astrophysics, the hadronic composition and processes are
relevant in the study of massive objects [30, 31, 32] and can benefit from the
present results.

The z-scaling is a phenomenological approach to meson production that
evidences the scaling properties of the process[33, 34, 35]. The comparison
between the thermofractal approach and the z-scaling can further clarify the
underlying physics of the hadronic collisions.

Interestingly, the fact that the phase transition between confined and
deconfined regimes does not disrupt the fractal structure may offer
constraints on the nature of the phase transition itself.

In conclusion, the results presented in this work lay the groundwork for
future research aimed at understanding the intricate fractal nature of hadrons
and its implications for various physical distributions critical in high-
energy physics. The findings may lead to advancements in our comprehension of
the underlying structure of hadrons and aid in the interpretation of
experimental data from high-energy collisions.

##  5 Acknowledgements

The authors thank Prof. Otaviano A. M. Helene for an insightful discussion on
the correlations among parameters in the fitting process. A.D. is supported by
the Project INCT-FNA (Instituto Nacional de Ciência e Tecnologia - Física
Nuclear Aplicada) Proc. No. 464898/2014-5, and by the Conselho Nacional de
Desenvolvimento Científico e Tecnológico (CNPq-Brazil), grant 306093/2022-7.
C.T. is partially supported by CNPq and Faperj (Brazilian agencies).

$\pi^{+}$
---
$\sqrt{s}(GeV)$ | $\sigma_{0}(mb/GeV^{2}/c^{3})$ | $\lambda(GeV)$ | $\bar{q}$ | $\chi^{2}_{red}$
23 | 244(20) | 0.132(4) | 1.049(4) | 1.59
31 | 304(15) | 0.127(2) | 1.060(2) | 1.21
53 | 350(21) | 0.119(2) | 1.077(2) | 2.25
63 | 661(90) | 0.097(5) | 1.096(5) | 2.87
$K^{+}$
23 | 345(288) | 0.09(2) | 1.09(2) | 2.68
31 | 341(155) | 0.10(1) | 1.079(9) | 1.93
53 | 139(69) | 0.11(1) | 1.086(8) | 5.33
63 | 0.4(3)E6 | 0.02(3) | 1.17(2) | 2.25
$\rho^{+}$
23 | 593(466) | 0.13(2) | 1.04(1) | 3.81
31 | 813(263) | 0.115(9) | 1.056(5) | 1.09
53 | 1005(479) | 0.11(1) | 1.068(6) | 2.82
63 | 306(328) | 0.13(4) | 1.06(2) | 1.40
Table 1: Best-fit results for adjustable parameters in Scenario 1. $\pi^{+}$
---
$\sqrt{s}(GeV)$ | $\sigma_{0}(mb/GeV^{2}/c^{3})$ | $\lambda(GeV)$ | $\Lambda(GeV)$ | $\chi^{2}_{red}$
23 | 260(24) | 0.22(3) | 0.35(4) | 1.46
31 | 334(17) | 0.167(8) | 0.48(2) | 0.91
53 | 439(27) | 0.123(4) | 0.85(3) | 1.70
63 | 765(111) | 0.097(7) | 1.2(2) | 2.87
$K^{+}$
23 | 678(687) | 0.08(2) | 1.0(4) | 2.44
31 | 621(355) | 0.09(2) | 0.8(2) | 1.71
53 | 287(59) | 0.09(2) | 1.3(2) | 4.25
63 | 3(4)E3 | 0.05(2) | 0(2e12) | 2.50
$\rho^{+}$
23 | 956(1225) | 0.2(2) | 0.3(2) | 3.75
31 | 1274(618) | 0.13(3) | 0.43(6) | 1.00
53 | 2574(1876) | 0.09(2) | 0.66(9) | 2.61
63 | 431(656) | 0.2(1) | 0.5(3) | 1.39
Table 2: Best-fit results for adjustable parameters in Scenario 2. $\pi^{+}$
---
$\sqrt{s}(GeV)$ | $\sigma_{0}(mb/GeV^{2}/c^{3})$ | $\Lambda(GeV)$ | $\chi^{2}_{red}$
23 | 390(20) | 0.53(1) | 3.34
31 | 409(10) | 0.579(8) | 1.80
53 | 349(10) | 0.72(1) | 2.82
63 | 435(32) | 0.63(3) | 6.36
$K^{+}$
23 | 139(29) | 0.54(4) | 2.94
31 | 214(25) | 0.55(2) | 2.31
53 | 86(11) | 0.86(5) | 5.42
63 | 58(15) | 1.1(2) | 3.46
$\rho^{+}$
23 | 1479(254) | 0.37(2) | 3.51
31 | 1165(67) | 0.419(8) | 0.94
53 | 747(61) | 0.50(2) | 2.94
63 | 564(81) | 0.55(3) | 1.28
Table 3: Best-fit results for adjustable parameters in Scenario 3.

## References

  * [1] G. Biro, G. G. Barnafoldi, T. S. Biro, K. Urmossy, and A. Takacs, “Systematic Analysis of the Non-Extensive Statistical Approach in High Energy Particle Collisions-Experiment vs. Theory,” ENTROPY, vol. 19, p. 88, 2017\.
  * [2] M. D. Azmi and J. Cleymans, “Transverse momentum distributions in proton-proton collisions at LHC energies and Tsallis thermodynamics,” Journal of Physics G - Nuclear and Particles Physics, vol. 41, 2014.
  * [3] G. Wilk and Z. Wlodarczyk, “The imprints of nonextensive statistical mechanics in high-energy collisions,” Chaos, solitons & Fractals, vol. 13, pp. 581–594, 2002.
  * [4] J. Cleymans, G. I. Lykasov, A. S. Parvan, A. S. Sorin, O. V. Teryaev, and D. Worku, “Systematic properties of the Tsallis distribution: Energy dependence of parameters in high energy p-p collisions,” Physics Letters B, vol. 723, pp. 351–354, 2013.
  * [5] C. Tsallis, “Possible Generalization of the Boltzmann-Gibbs Statistics,” Journal of Statistical Physics, vol. 52, no. 1-2, pp. 479–487, 1988\.
  * [6] C. Tsallis, Introduction to the Nonextenssive Statistical Mechanics - Approaching a Complex World (Second Edition).  Cham: Springer, 2023.
  * [7] A. Deppman, E. Megias, and D. P. Menezes, “Fractals, nonextensive statistics, and QCD,” Physical Review D, vol. 101, 2020.
  * [8] A. Deppman, “Thermodynamics with fractal structure, Tsallis statistics, and hadrons,” Physical Review D, vol. 93, 2016.
  * [9] R. Hagedorn, “Statistical Thermodynamics of Strong Interactions at High Energies,” Nuovo Cimento, vol. 3, pp. 147+, 1965.
  * [10] A. Deppman, “Self-consistency in non-extensive thermodynamics of highly excited hadronic states,” Physica A - Statistical Mechanics and Its Applications, vol. 391, pp. 6380–6385, 2012.
  * [11] L. Marques, E. Andrade, II, and A. Deppman, “Nonextensivity of hadronic systems,” Physical Review D, vol. 87, 2013.
  * [12] G. Chew and S. Frautschi, “Principle of Equivalence for All Strongly Interacting Particles within S-Matrix Framework,” Physical Review Letters, vol. 7, pp. 394–&, 1961.
  * [13] O. Brüning, A. Seryi, and S. Verdú-Andrés, “Electron-Hadron colliders: EIC, LHeC and FCC-eh,” Front. Phys., vol. 10, 2022.
  * [14] LHeC Study Group, “A large hadron electron collider at CERN report on the physics and design concepts for machine and detector,” J. Phys. G Nucl. Part. Phys., vol. 39, p. 075001, 2012.
  * [15] J. Rożynek and G. Wilk, “Nonextensive nambu-jona-lasinio model of QCD matter,” The European Physical Journal A, vol. 52, Jan. 2016.
  * [16] P. H. G. Cardoso, T. N. da Silva, A. Deppman, and D. P. Menezes, “Quark matter revisited with non-extensive MIT bag model,” The European Physical Journal A, vol. 53, no. 10, 2017.
  * [17] E. Andrade, A. Deppman, E. Megias, D. P. Menezes, and T. N. da Silva, “Bag-type model with fractal structure,” Physical Review D, vol. 101, Mar. 2020.
  * [18] B. Alper, H. Böggild, P. Booth, F. Bulos, L. Carolll, G. Damgaard, G. V. Dardel, B. Duff, K. Hansen, F. Heymann, J. Jackson, G. Jarlskog, L. Jönsson, A. Klovning, L. Leistam, E. Lillethun, E. Lohse, G. Lynch, G. Manning, K. Potter, M. Prentice, P. Sharp, S. Sharrock, S. Ølgaard-Nielsen, D. Quarrie, and J. Weiss, “Production spectra of $\pi$$\pm$, k$\pm$, p$\pm$ at large angles in proton-proton collisions in the CERN intersecting storage rings,” Nuclear Physics B, vol. 100, no. 2, pp. 237–290, 1975.
  * [19] E. Megías, D. P. Menezes, and A. Deppman, “Non extensive thermodynamics for hadronic matter with finite chemical potentials,” Physica A: Statistical Mechanics and its Applications, vol. 421, pp. 15–24, Mar.
  * [20] L. Q. Rocha, E. Megías, L. A. Trevisan, K. K. Olimov, F. Liu, and A. Deppman, “Nonextensive statistics in high energy collisions,” Physics, vol. 4, pp. 659–671, June 2022.
  * [21] I. Sena and A. Deppman, “Systematic analysis of p(T)-distributions in p plus p collisions,” European Physical Journal A, vol. 49, 2013.
  * [22] T. Bhattacharyya, J. Cleymans, L. Marques, S. Mogliacci, and M. W. Paradza, “On the precise determination of the tsallis parameters in proton–proton collisions at lhc energies,” Journal of Physics G: Nuclear and Particle Physics, vol. 45, p. 055001, mar 2018.
  * [23] S. Dulat, T.-J. Hou, J. Gao, M. Guzzi, J. Huston, P. Nadolsky, J. Pumplin, C. Schmidt, D. Stump, and C.-P. Yuan, “New parton distribution functions from a global analysis of quantum chromodynamics,” Phys. Rev. D., vol. 93, 2016.
  * [24] M. G. Echevarria, I. Scimemi, and A. Vladimirov, “Unpolarized transverse momentum dependent parton distribution and fragmentation functions at next-to-next-to-leading order,” J. High Energy Phys., vol. 2016, 2016.
  * [25] M. Anselmino, M. Boglione, J. O. Gonzalez H., S. Melis, and A. Prokudin, “Unpolarised transverse momentum dependent distribution and fragmentation functions from SIDIS multiplicities,” J. High Energy Phys., vol. 2014, 2014.
  * [26] A. V. Belitsky and A. V. Radyushkin, “Unraveling hadron structure with generalized parton distributions,” Phys. Rep., vol. 418, no. 1-6, pp. 1–387, 2005.
  * [27] M. Diehl, “Introduction to GPDs and TMDs,” Eur. Phys. J. A, vol. 52, 2016.
  * [28] L. Frankfurt, M. Strikman, and C. Weiss, “Small-$\chi$ Physics: From HERA to LHC and beyond,” Annu. Rev. Nucl. Part. Sci., vol. 55, pp. 403–465, 2005.
  * [29] X. Chen, F.-K. Guo, C. D. Roberts, and R. Wang, “Selected science opportunities for the EicC,” Few-body Syst., vol. 61, 2020.
  * [30] Z.-H. Wang, Y. Sang, and X. Zhang, “Power-law distribution and scale-invariant structure from the first CHIME/FRB fast radio burst catalog,” Research in Astronomy and Astrophysics, vol. 23, p. 025002, 2023.
  * [31] M. Razeira, B. E. J. Bodmann, and C. A. Z. Vasconcellos, “Strange matter and strange stars with Tsallis statistics,” International Journal of Modern Physics D, vol. 16, pp. 365–372, 2007.
  * [32] G. Gervino, A. Lavagno, and D. Pigato, “Power-law quantum distributions in protoneutron stars,” Journal of Physics: Conference Series, vol. 442, p. 012065, June 2013.
  * [33] I. Zborovsky, Y. A. Panebratsev, M. V. Tokarev, and G. P. Skoro, “$Z$ scaling in hadron hadron collisions at high-energies,” Phys. Rev. D, vol. 54, pp. 5548–5557, 1996.
  * [34] I. Zborovsky, M. V. Tokarev, Y. A. Panebratsev, and G. P. Skoro, “$Z$ scaling in proton-nucleus collisions at high energies,” Phys. Rev. C, vol. 59, p. 2227, 1999.
  * [35] I. Zborovsky and M. Tokarev, “Generalized Z-scaling in proton-proton collisions at high energies,” Phys. Rev. D, vol. 75, p. 094008, 2007.


11institutetext: Aalto University, P.O. Box 15400, Espoo, Finland
11email<EMAIL_ADDRESS>11email<EMAIL_ADDRESS>11email<EMAIL_ADDRESS>
# The Viability of Continuous Experimentation in Early-Stage Software Startups

A Descriptive Multiple-Case Study

Vihtori Mäntylä  0000-0002-0448-9494 Bettina Lehtelä  0000-0002-2814-4386
Fabian Fagerholm  0000-0002-7298-3021

###### Abstract

Background: Continuous experimentation (CE) has been proposed as a data-driven
approach to software product development. Several challenges with this
approach have been described in large organisations, but its application in
smaller companies with early-stage products remains largely unexplored. Aims:
The goal of this study is to understand what factors could affect the adoption
of CE in early-stage software startups. Method: We present a descriptive
multiple-case study of five startups in Finland which differ in their
utilisation of experimentation. Results: We find that practices often
mentioned as prerequisites for CE, such as iterative development and
continuous integration and delivery, were used in the case companies. CE was
not widely recognised or used as described in the literature. Only one company
performed experiments and used experimental data systematically. Conclusions:
Our study indicates that small companies may be unlikely to adopt CE unless
1) at least some company employees have prior experience with the practice,
2) the company’s limited available resources are not exceeded by its adoption,
and  3) the practice solves a problem currently experienced by the company, or
the company perceives almost immediate benefit of adopting it.  We discuss
implications for advancing CE in early-stage startups and outline directions
for future research on the approach.

###### Keywords:

continuous experimentation, new product development, startup, continuous
software engineering, case study

##  1 Introduction

A mismatch between product features and customer needs is one of the most
common reasons for software startup failure [15]. While agile methods
emphasise customer value [6], building the right product appears to be a feat
that few startups achieve. Continuous experimentation (CE) is a software
engineering method where product development is driven by field experiments
with real users [29, 8, 9, 41, 2]. It strives to establish virtuous feedback
loops between business, development, and operations [11], and reportedly
improves product quality and business performance [7, 8], with promising
implications for startups.

Previous studies have found that startups tend to run experiments as part of
their product development process [21, 22]. However, the experiments are
usually neither planned nor run in the organised and systematic manner
characteristic of CE [21, 25]. Startups are often reluctant to incorporate
industry best practices, methods, and frameworks proposed by researchers into
their business and product development processes [15]. This raises the
question of whether CE, as envisioned in different frameworks and models in
the literature, fits the needs of small companies with early-stage products,
such as software startups.

This paper aims to elucidate possible reasons for why adopting CE in early-
stage startup (ESS) settings might be difficult. Rather than producing a
general theory of adoption of CE in startups, we develop propositions that can
be extended and validated in future studies, and that can point to novel
directions for research. Specifically, we address the following research
questions:

  1. RQ1:

Do ESS companies integrate experimentation into their software development
practices?

  2. RQ2:

How do ESS companies utilise user data in their software product development
process?

  3. RQ3:

What factors influence the adoption of CE in ESS companies?

We study these questions in the context of five startup companies. We
contribute  1) a description of the state of experimentation and overall mode
of operation in the case companies from the perspective of CE, providing
insight into what product development practices startups tend to adopt and
why; and  2) a set of factors that affect the adoption of CE in startup
companies, partly supporting earlier findings from startups (e.g., [21, 22,
13, 25, 26]) and larger companies (e.g., [41]), but with new and more detailed
factors relevant to adoption of CE.  The paper builds on results obtained in a
Master’s thesis [23].

##  2 Background

CE approaches software product development through experiments with real users
[29, 8, 9, 41, 2]. This includes collecting and analysing experimental data to
test product hypotheses, gaining insights for new feature ideas to be
evaluated in subsequent experiments. CE comes with various implications on
organisational structure and culture [19, 11, 41], product and software
development processes [29, 9], and software architecture, construction, and
deployment [9, 34].

###  2.1 Scientific and Online Controlled Experiments

Scientific experiments can provide causal descriptions of what consequences
varying a treatment has on a subject of interest [35]. Experimental setups
vary with the study they are used for, but they all share the same basic
elements:  1) a hypothesis to be tested, including evaluation criteria for its
acceptance or rejection;  2) sufficient control of independent variables;  3)
means for collecting data; and  4) experimental subject(s) under investigation
[35].

Online controlled experiments (OCE) [17] leverage the ease of access to users
through the Internet to deploy field experiments. These are often performed as
A/B tests where two different user groups receive a different treatment, e.g.,
the old versus new version of a feature. The treatments are usually assigned
randomly to the target groups and statistical analysis is used to determine
which of the tested implementations produced better results in terms of
improved metrics. OCEs can be arranged into an ongoing process to implement
CE.

###  2.2 Implementing Continuous Experimentation

CE can be seen as arising from advances in the use of experimentation in
product development [37], agile software development practices, and continuous
software engineering [11] practices such as continuous integration, continuous
delivery, and continuous deployment. Modern software engineering practices
like these are behind the experimentation processes in pioneering companies
such as Microsoft [18, 19], Netflix [1, 12], and others [10, 36, 38]. Applying
experimentation to product development reportedly gives increased customer
satisfaction, improved and quantified business goals, and a transformation to
a continuous development process [4]. Holmström Olsson et al. [29] position
experimentation as the last step of an organisational evolution ladder, where
agile software development, continuous integration, delivery, and deployment
are prerequisites.

Realising CE requires arranging experiments across time. A number of models
have been proposed to prescribe how CE can be implemented as a method. For
example, the HYPEX model describes a detailed experimentation process model
spanning the complete product development cycle [30]. A minimum viable feature
(MVF) is implemented and deployed, usage data is collected, and gap analysis
is applied to determine the next course of action. Experimental results are
fed back into the business strategy function, completing the feedback loop.

The RIGHT model describes an experimentation framework in detail [9]. It
features a continuous experimentation cycle encompassing business and product
vision, strategy, experiment design, and software construction activities. It
also describes an architectural model for the experimentation infrastructure
and related stakeholder roles. Furthermore, it covers experiment objects in
the form of minimum viable products (MVPs) and MVFs, which are used to rapidly
test product hypotheses. The RIGHT model assumes that CE capabilities can be
built little by little. Once multiple simultaneous experiments are running,
the full model becomes critical.

While other models are available, they share most of their essential
characteristics with HYPEX and RIGHT, and the latter is considered a reference
model in the literature [2].

###  2.3 Software Startups and Experimentation

Software startups are often characterised as being innovative and driven by
small teams, operating under high uncertainty and rapidly evolving, and being
challenged by a lack of resources, time pressure to deliver a product to
market, and only little working and operating history [31, 3, 27]. Startups
are thus distinctly different from traditional enterprises, in which new
product development is only one concern alongside concerns related to existing
products and customers.

Startups commonly fail [31], often by not achieving product-market fit [32,
20] – they create the wrong product [14]. To increase the chances of success,
the Lean Startup method proposes Build-Measure-Learn loops: testing hypotheses
about how users will react to software changes based on predefined metrics,
and using the results to make an MVP [32]. Many ideas from the Lean Startup
model can be found in continuous software engineering [11] and CE [30, 9].
Camuffo et al. [5] conducted a randomised study and concluded that the
scientific experimentation approach is beneficial for startups.

Research indicates that while experimentation in general is common in
startups, systematic experimentation is not widely adopted [22, 25]. Previous
studies addressing CE adoption (e.g., [41, 40, 22, 29]) acknowledge that it
requires high levels of skill and coordination, and advanced technological and
organisational capabilities. Auer et al. [2] identified six categories of
challenges:  1) cultural, organisational and managerial challenges,  2)
business challenges,  3) technical challenges,  4) statistical challenges,  5)
ethical challenges, and  6) domain specific challenges.  However, few studies
address CE specifically for software startups. Although some CE models allow
for gradual adoption (e.g., the RIGHT model [9]), knowledge about systematic
adoption strategies and application of the practice in software startups is
scarce. Research on CE often describes the benefits, but most studies rely on
examples from a small number of industry leaders whose methods are not
necessarily suitable for, e.g., smaller companies [34] or companies whose
products do not share the same characteristics of flexibility. Open questions
thus revolve around the circumstances under which CE is suitable for startups
and what characteristics of the product development and user data practices of
startups are compatible with CE.

##  3 Research Method

We conducted a descriptive multiple-case study [42] to investigate product
development practices in our case companies. Through semi-structured
interviews [28], we obtained insights into the product development practices,
utilisation of user data, and experimentation used in the companies, which we
analysed to develop propositions that address the research questions. We used
interviews because they provide relatively quick access to key information,
and because other data sources can be scarce in startup companies; for
example, extensive process documentation may not exist. Confidentiality
requirements prevent us from publishing the full set of collected data, but
excerpts are given in this paper. An online supplement gives details on the
study design and results [24].

###  3.1 Case Companies

We selected case companies (see Table 1) from two startup communities in the
capital region of Finland: _Maria 01_ (https://maria.io/), with over 175
startups residing in Kamppi, Helsinki; and _A Grid_ (https://agrid.fi/), with
around 150 startups located on the Aalto University campus in Otaniemi, Espoo.

For the company selection we used purposeful sampling: a convenience sample
guided by our research goals [28]. We sought to include companies at any stage
of the early product development life-cycle and with different kinds of
software-based products or services. Having a relaxed set of selection
criteria provided a heterogeneous set of cases and varied qualitative data.
All companies remain anonymous and we omit identifying information, such as
product details.

Table 1: Case companies and their context at the time of the study. ID | Location |  Business model |  Distribution channel |  Product stage |  Participant role
---|---|---|---|---|---
A | Maria 01 |  Free app, sell ads |  Mobile app |  V1.0 launch in progress |  Product owner (PO)
B | Maria 01 |  B2B (provisiona) |  API as a service |  Released |  COO
C | Maria 01 |  B2C (subscription) |  Side-loaded application |  Early access program live |  CTO
D | A Grid |  B2B (provisiona) |  SaaS |  Live for over a year |  CEO
E | A Grid |  B2C (subscription) |  Physical |  Live for over a year |  Lead software engineer
aProvision business model: the company receives a provision of sales from
their B2B customer.

###  3.2 Data Collection

Semi-structured interviews are suitable when the topic and expected data are
not well understood before starting the data collection [28]. We expected
startups to operate in varied ways that we could not fully anticipate. We
designed a semi-structured interview format with open-ended questions based on
the guide used by Lindgren & Münch [21]. We extended the question set with new
questions on tool and practice selection and added CE as a distinct topic.

As shown in Table 1, we interviewed one expert participant from each case
company, identified through discussion with company representatives, and based
on their broad understanding of both the technical and business aspects of
their company. In addition to the main interviews, follow-up data collection
with the same or additional participants was conducted to verify and fill in
some details. The interviews were performed in Finnish.

###  3.3 Data Analysis

The first author iteratively analysed the interview transcripts. Insights
obtained from one case company often provided a new perspective into data from
other companies. Multiple passes allowed building higher-order categories from
topics that appeared across case companies. The analysis was supported by data
review sessions with the two other authors. In these, each transcript was
walked through and discussed, with the first author explaining the interview
and preliminary insights obtained. Gradually, the sessions moved to
discussions about transcript coding and emerging insights from the data.
ATLAS.ti was used for the analysis task. Coding was performed in English
although the transcripts were in Finnish.

####  3.3.1 First-cycle Coding.

The analysis started without any predefined categories. Codes produced from
the first analysis iterations were long and descriptive, and multiple coding
passes were used to group similar codes under a single parent code. Higher-
level categories were built based on recurring themes.

####  3.3.2 Individual Case Analysis.

After coding and category-building, each company was analysed as an individual
case. Practices for software and product development and user data collection
and use were extracted for closer inspection. The information flow within each
company was represented graphically, giving a picture of the organisation and
the interactions between different stakeholders. The graphics are provided in
the supplementary material [24].

####  3.3.3 Cross-case Analysis and Second-cycle Coding.

Combined data from all five cases were also analysed. Tools and practices used
by the companies were analysed in tabular form. The coded interview data from
all five individual cases were re-coded into common themes that permeate the
whole data set. In this final phase, the analysis focused on uncovering
categories that could potentially explain the tool and practice choices made
by the companies.

##  4 Results

Systematic experimentation was rare in the case companies. Our results
question the attractiveness of CE for startups as multiple factors detract
from its use.

###  4.1 Case Company Descriptions

####  4.1.1 Company A.

Lack of previous experience in mobile application development was a major
bottleneck for data acquisition in Company A. Challenges involved
understanding what data was possible to get, what data was valuable, and which
tools were worth using. The application was instrumented to log some user
actions, and the team was actively seeking to improve their data collection.

A feedback form was used to collect user data, and app store reviews were a
regular feedback channel, with in-app prompts to request reviews. The product
owner (PO) reviewed the collected data weekly, but would have wanted to better
understand the real-world usage of the application, and have the capability to
chat with individual users to gain insights on how and why they used the app.

The PO was not familiar with CE, but mentioned iterative user testing –
resembling experimentation – for the application’s on-boarding experience.
Think-aloud user tests had been used with a small number of new users. Based
on this, design adjustments were made and subsequent user testing indicated an
improvement. The PO considered most of their development as some form of test.

> Well, almost everything we do is maybe more like tests. (Product owner,
> Company A)

However, it is not clear if these tests would qualify as planned experiments,
and whether hypotheses and measurable goals were set before starting
development.

####  4.1.2 Company B.

There was only restricted access to user data in this company, as their
service was used through a third-party UI. The company had to rely on the
limited data available from their own systems, as the business partners owning
the UI were not willing to share detailed user data. However, Company B
closely collaborated with current and potential customers for requirements
elicitation. Data from weekly customer meetings were used extensively from
early on, and the products were engineered to meet partner-defined functional
and non-functional requirements. Overall, the company appeared to base their
product development decisions on the data received through their business
partners and less on the scarcely available user data.

Although Company B would likely be technologically capable of running
continuous experimentation, the company’s B2B business model makes any
experimentation involving user data very difficult. The company claimed that
the pricing of their product was the only parameter they could adjust.

> We do not have resources at the moment to do anything repeatedly or as a
> process […] this interest rate is the only parameter we could control. (COO,
> Company B, on A/B testing)

Adjusting the pricing was also considered to be difficult, since any changes
would need to be coordinated with, and approved by, their business partners.
These obstacles to experimentation for companies operating in the B2B domain
are well recognised in earlier studies [41, 2].

####  4.1.3 Company C.

This company was in a transition phase where they had just launched their
application through an early access program (EAP). They used a form of
repeated experimentation when building the EAP version, working closely with
potential customers and industry experts to understand the requirements their
future customers would have for the product. After establishing the initial
set of requirements, they executed numerous Build-Measure-Learn iterations
where each new version of the product was subjected to a user test by an
expert user, including observation and an interview for feedback. The
collected user data was then converted into software development tasks.

> It has been this […] very tight develop and test type of work where we
> develop something and then go [try it out]. (CTO, Company C, on the
> development process)

However, there was no sign of predefined metrics or evaluation criteria which
would have indicated a systematic approach for experimentation. The company
had thought about possible future A/B testing and had recently implemented
feature flags, but mainly for licensing purposes. Instrumentation data was not
automatically sent to the company due to user privacy expectations and offline
capability requirements. The company expected that most feedback would be
received via email and discussing with the users. This was seen as good enough
for now and they recognised that some other way of handling user feedback
would need to be implemented once their customer base grows.

####  4.1.4 Company D.

This company used a well-defined process for product development planning,
following the Data-Insight-Belief-Bet (DIBB) framework [16]. The leadership
defined a set of objective key results (OKR) for the next quarter, to be
implemented by the product development teams. Each OKR contained a high-level
goal and the metrics to evaluate whether or not the goal was reached at the
end of the quarter. The focus was on meeting investor goals.

The development teams regularly built feature prototypes, mostly used
internally, but occasionally presented to end users or customer
representatives. An expert board reviewed the data from prototyping and
selected the best implementation based on qualitative feedback. The data was
used continuously as company D was not only reacting to negative feedback, but
also actively trying to learn from the data and even predicting future needs
based on historical data.

> Ten percent of the features [have come from when we analysed all feedback]
> and predicted what kind of things there would be coming. [We can say] we
> actually need to build this kind of feature because [customers] are going
> towards this kind of world. (CEO, Company D, on use of data)

The company employed numerous user data collection channels and used data
extensively in all stages of product development. Event-level quantitative
data was collected from different parts of the system and qualitative data was
collected through a chat embedded in the service. Research ops, a dedicated
business unit, analysed and refined all available data, and was responsible
for the data to be available to all members of the organisation.

####  4.1.5 Company E.

The last company offered a physical service; their software provided internal
support. There was no instrumentation for quantitative data collection due to
lack of interest in such feedback. To collect qualitative user data, the
company mainly observed office staff and read feedback from chat channels.
This internal feedback loop tied back to the development team’s Kanban board
where new input was entered as work items. A survey was used to collect
feedback from company field workers, but it was not entirely clear how the
data was used, suggesting a lack of systematic usage.

> We have sent our field workers these quick surveys just to see what is the
> general opinion about this app and such. We did not have any specific
> metrics like “rate this from 0 to 10” or anything. (Lead developer, Company
> E, on user surveys)

Feedback on the physical service was collected through a form and email, was
handled by customer service agents, and was generally not available to
everyone in the company. Overall, Company E did not systematically use data in
the internal software product development. We found indications that data was
seen as a way to detect problems rather than understanding users’ needs.

> […] often not receiving any feedback about anything indicates that things
> are probably quite ok as there is nothing to complain about. (Lead
> developer, Company E, on negative feedback)

The team had envisioned doing A/B testing once they have released customer-
facing applications. However, in its current state, the company lacked the
capabilities to effectively use experimentation since the software development
team was largely disconnected from the rest of the organisation.

###  4.2 Cross-case Analysis

All case companies had an incremental software development process and used
lean and agile practices such as continuous integration and delivery. They
also had several prerequisites for continuous deployment in place but none of
them actually deployed automatically to production.

Different forms of experimentation were used in four out of five companies.
Company A did user testing with different UI implementations and a MVP feature
implementation. A price change experiment was going on in Company B,
prototyping and expert user testing in a continuous cycle in Company C, and
extensive prototyping and user testing in Company D. Company C appeared to be
using systematic prototyping, but their data collection and data analysis
practices were not systematic: they simply let expert test users try the
product without planning what data to collect. Company D had advanced data
collection, storage, and refinement capabilities, used the data
systematically, and would most likely be capable of continuous
experimentation.

It appeared that the startups had set up their tools and practices based on
previous experience of founders or employees. There was isolated use of
practices such as user testing, prototyping, and even occasional A/B testing
in the companies, but those seemed to originate from individuals with previous
experiences with the techniques rather than the company actively adopting the
practices.

Allocating resources to execute the product roadmap in the most efficient way
was central to work planning in all case companies. Only Company B did not
mention funding as a resource limitation. All companies had roadmaps with far
more work than they could ever implement given their available schedule and
resources. This chronic lack of resources forced the companies to prioritise
work that would be beneficial in the next funding round, even at the expense
of feature development with real value to customers.

There was a tendency in the companies to avoid advanced practices until a
strong need emerged – a fiercely pragmatic stance towards method adoption. It
became apparent that even experiencing problems was not enough to adopt better
ways to operate. Even if the team would know a better way of doing their work,
the effort to set up a new system and the opportunity cost of being unable to
work on other, more important, tasks, could prevent change from taking place.
Change was seen as a waste of resources unless a perceived tangible benefit
was in sight within the time horizon that is currently relevant for the
company.

##  5 Discussion

The descriptive account given above allows us to construct a number of
propositions to address the research questions, which we now discuss.

###  5.1 CE and Software Development Methods (RQ1)

To address RQ1, we propose that software startups do not generally integrate
experimentation into their software development methods, but may use CE-
support-ing methods. Experimentation in a loose sense was present in our case
companies, but systematic, statistically controlled experimentation was
uncommon, in line with existing research showing a lack of systematic CE in
startups [21, 13, 25]. Product domains and business models that limit the
possibilities for experimentation further reduce integration. Maturing
startups may introduce more structure into their methods, and we then expect
systematic experimentation to be more frequent and deeply integrated into
methods and practices.

Several practices supporting CE were in place in the case companies. All
developed their products incrementally using agile or lean software
development practices. Only Company A built their mobile app manually. The
companies were similar in development practices, the single major difference
being automated testing, used by only two. Version control and backlog were
the only practices with an adoption rate above 35% in a prior study [14].
Compared to this, the case companies are fairly advanced in adopting key agile
practices. However, a closer look reveals rather selective adoption of agile
practices in three of our case companies, and some deviations were found in
all. Method selectiveness has been reported before, with a warning that
picking only some agile practices without supporting practices may lead to
adverse effects [14].

The prerequisites for experimentation thus appear at first glance to be in
place in the case companies, but they have picked methods selectively.
Experimentation does not appear to be integrated with the methods. We found
that only one company had a systematic approach to experimentation, which is
in line with findings from a study with ten case companies [21] reporting that
non-systematic experimentation was common, but systematic experimentation
among startup companies was rare.

###  5.2 Use of Data (RQ2)

For RQ2, we propose that  1) utilisation of user data in startups tends to
begin with ad hoc collection and interpretation of rich, qualitative data when
the user base is small and the startup is in its early stages,but that  2)
transitioning to more systematic use of data, which allows reliable
understanding of the effects of product decisions, requires deliberately
building specific technical capabilities and adopting an experimentation
framework.

Four of the five companies had good access to user data and established ways
for collecting qualitative feedback. Interviews, feedback forms, and email
were the most common channels, but social media, application store feedback,
and chat were also used. Company B’s B2B model practically prevented access to
qualitative end user data, and forced them to rely on data their business
partners were willing to share. These are known issues of the B2B context
[33].

Four of the companies had built instrumentation for collecting event-level
quantitative data, but only companies A and D had automated access to UI level
events; both demonstrated product improvement as a result of using this data.
Company B was limited to backend data, and Company C had implemented an opt-in
data collection mechanism primarily for debugging purposes.

Prior work suggests that CE adoption relies on initial awareness [13], and
progresses through several stages, each with many challenges to overcome [26].
However, we add the consideration that CE in the form often proposed in the
literature may be inadequate for startups as its costs may be too large. Our
findings show that, in line with previous research on organisational
development towards CE (e.g., [4, 29, 11]), the transition to systematic and
advanced use of data in startups requires considerable investment in
development of skills, procedures, data acquisition, and customer
collaboration in different forms. It requires a mode of functioning that is
not easily combined with the sparse resources of startups nor with the culture
of fast-moving innovation and a sharp focus on realising the founders’ product
vision. Adopting a systematic experimentation approach may face resistance
(c.f. [26]), pointing to the need for data acquisition and utilisation methods
that work for very early-stage startups, and can scale up as they grow and
mature.

###  5.3 The Appeal of Continuous Experimentation (RQ3)

The following three propositions concern conditions for adopting CE in
startups.

_P1. Adoption of a practice requires previous personal experience._ Only the
interviewee from Company D had heard about continuous experimentation, but was
unable to describe it in detail, which suggests that they had no prior
experience with it. Our findings suggest that the strongest adoption pathway
in this context is word of mouth and personal experience.

_P2. The practice must not require large resources to adopt or use._ Four out
of the five companies mentioned funding as critical to their their priorities.
The fear of running out of time and money may prevent startups from taking
risks in adopting practices, especially if large time and resource investments
are perceived to be needed. It is unclear what resources are required to adopt
CE. The case companies had already performed experiments, albeit in
unstructured ways. Becoming more systematic could be a matter of educating the
company employees. However, establishing an experimentation process and the
required technical infrastructure requires upfront work. Additionally, more
resources are needed to make variants for experiments than to build a single
version. These resource requirements may exceed what the companies believe
they can afford.

_P3. The practice must solve a serious enough problem within a reasonable
amount of time._ Whether CE solves a concrete enough problem and provides
enough perceived value is a question beyond personal experience and resource
requirements. From the company’s perspective, an issue must be serious enough
and timely, and the new practice must guide to a solution fast enough to keep
up with the speed dictated by investors. At least companies A and C, which had
not yet established a firm user base, were effectively looking for a product-
market fit and trying to test their ideas with early versions of their
application. These two companies seemed to be less inclined to do systematic
experimentation than they were to develop features from the roadmap and see
what users would say about the next version. Even though CE could have some
benefits, these companies may be more interested in increasing the speed of
development. Therefore, the companies would not recognise CE as a valuable
solution for their problem.

Earlier studies covering CE adoption (e.g., [29, 22, 9, 13, 39, 25, 26])
indicate that notable skill and coordination from the whole organisation is
required. They indicate that adopting CE is a journey that should be taken
gradually, starting with small-scale experimentation, and building increased
technical and organisational capabilities in each subsequent experimentation
round [22, 39]. Yaman et al. [39] propose having a CE champion with the
required expertise and mandate to facilitate adoption. This idea aligns well
with the identified need for prior experience, as well as the startup
companies’ suggested inability to see the value of systematic experimentation.

###  5.4 Implications for Practice

The propositions above suggest that CE is problematic for software startups.
On one hand, the possible benefits, especially the validation of value
hypotheses, are important for startups since they help avoid directions that
do not lead to a viable product. On the other hand, the considerable resources
required for full adoption, the many details and required rigour in currently
existing methods, and the potential mismatch with fast-paced startup culture
and the skills that startup employees currently have, mean that adopting CE
risks depleting precisely those scarce resources that startups need to build
their product.

As noted above, the current solution is to do a gradual, piecemeal adoption,
and to find a person or small team to spearhead the practice, using minimal
resources. However, we suggest that startups should consider CE without all
the prerequisites that are usually listed. Simple means of observing user
behaviour, such as sending back a single value from an app, and basic analyses
in a spreadsheet, can enable startups to focus on product questions rather
than the infrastructure. This turns the order of adoption around to provide
immediate value: instead of large up-front investments in CI, CD, and other
automation that must be constantly updated as the product changes, the focus
should be on developing the capability to dress product development questions
as simple experimental designs and finding the quickest and least resource-
intensive ways to execute them. This leverages the key assets that startups
have: knowledge about their product and customers, a sense of the product
vision, and an aptitude for fast-paced innovation, turned towards the practice
of experimentation itself.

###  5.5 Limitations

The credibility or internal validity [28] of this study is limited by the
participation of only one main interviewee per company. To counter this
threat, we sought to include participants who had the best possible knowledge
of the questions we wanted to ask. We also included a small number of
additional participants to provide missing details on, e.g., technical
matters. Credibility, as well as consistency [28] is also strengthened by our
use of researcher triangulation and the repeated data sessions carried out
during the analysis. Given the small size of the companies, we argue that our
study is credible and consistent in capturing the reality of each of the five
companies.

In terms of transferability or external validity [28], the variation in case
companies and the reporting of results and circumstances should help the
reader determine the extent to which our findings can be applied in specific
future cases. We have not sought to statistically validate the findings in a
larger sample. Rather, the aim of this study is to develop a descriptive
account of the case companies and develop propositions that could be used, for
example, to design a larger survey study to examine the prevalence of the
factors found here. We argue that the propositions obtained in this study
enable future studies to examine CE in startups in more detail than what was
possible based on existing studies to date, and that they can be used as
points of reflection for practitioners if they are considering to adopt CE in
a startup context.

##  6 Conclusions

We sought to understand the factors involved in adopting CE in software
startups. Through a descriptive multiple-case study in five Finnish startups,
we examined product development practices, and method choices, asking how
these companies collected and utilised user data in product development.

All companies used agile or lean software development practices and continuous
software development practices such as continuous integration and continuous
deployment. Most companies were able to collect qualitative user data. Two
were also doing automatic collection of user interface event level data from
their services. The companies used the collected data mostly for validating
that their services did not contain errors. Only one company appeared to be
systematically using the data for predicting future user requirements.

Continuous experimentation was not commonly known: only one participant had
heard about it and could not describe it in detail. Previous experience and
expected short-term value seem to be important factors when startups select CE
tools and practices. The companies struggled with limited resources, forcing
them to carefully prioritise work and foregoing the adoption of complex
methods.

A well-resourced company can afford dedicating an extra team to
experimentation without negatively affecting the development work. This is not
feasible for a startup with only a handful of developers, where the effort of
doing an experiment might require halting other development. Thus, it is
understandable that a startup company would prefer to simply continue
executing their roadmap.

More awareness of continuous experimentation could improve the adoption rate
of the practice among developers and entrepreneurs. Potential approaches
include teaching the practice in university curricula or to prepare
entrepreneurship training programs with CE-related material. The latter
approach has been tried with promising results [5] and may be worth pursuing
in further research.

However, the CE practice should also adapt to the requirements of different
kinds of companies. The research community should seek ways to make CE more
affordable for a larger variety of companies, to lower the adoption barrier on
both organisational and individual levels, and to make it more attractive and
easier to benefit from the practice’s advantages.

####  6.0.1 Acknowledgements

We express our gratitude to the study participants and participating companies
for their generous sharing of information.

## References

  * [1] Amatriain, X.: Beyond data: from user information to business value through personalized recommendations and consumer science. In: Proceedings of the 22nd ACM international conference on Information & Knowledge Management. pp. 2201–2208. ACM, San Francisco, California, USA (2013)
  * [2] Auer, F., Ros, R., Kaltenbrunner, L., Runeson, P., Felderer, M.: Controlled experimentation in continuous experimentation: Knowledge and challenges. Information and Software Technology 134, 106551 (2021)
  * [3] Berg, V., Birkeland, J., Nguyen-Duc, A., Pappas, I.O., Jaccheri, L.: Software startup engineering: A systematic mapping study. Journal of Systems and Software 144, 255–274 (2018)
  * [4] Bosch, J.: Building products as innovation experiment systems. In: van der Aalst, W., Mylopoulos, J., Rosemann, M., Shaw, M.J., Szyperski, C., Cusumano, M.A., Iyer, B., Venkatraman, N. (eds.) Software Business. ICSOB 2021. Lecture Notes in Business Information Processing, vol. 114, pp. 27–39. Springer Berlin Heidelberg, Berlin, Heidelberg (2012)
  * [5] Camuffo, A., Cordova, A., Gambardella, A., Spina, C.: A Scientific Approach to Entrepreneurial Decision Making: Evidence from a Randomized Control Trial. Management Science 66(2), 564–586 (2020)
  * [6] Dingsøyr, T., Lassenius, C.: Emerging themes in agile software development: Introduction to the special section on continuous value delivery. Information and Software Technology 77, 56–60 (2016)
  * [7] Fabijan, A., Dmitriev, P., Olsson, H.H., Bosch, J.: The Benefits of Controlled Experimentation at Scale. In: 43rd Euromicro Conference on Software Engineering and Advanced Applications (SEAA). pp. 18–26. IEEE, Vienna, Austria (2017)
  * [8] Fabijan, A., Dmitriev, P., Olsson, H.H., Bosch, J.: The evolution of continuous experimentation in software product development: From data to a data-driven organization at scale. In: IEEE/ACM 39th International Conference on Software Engineering (ICSE). pp. 770–780 (2017)
  * [9] Fagerholm, F., Sanchez Guinea, A., Mäenpää, H., Münch, J.: The right model for continuous experimentation. Journal of Systems and Software 123, 292–305 (2017)
  * [10] Feitelson, D.G., Frachtenberg, E., Beck, K.L.: Development and deployment at facebook. IEEE Internet Computing 17(4), 8–17 (2013)
  * [11] Fitzgerald, B., Stol, K.J.: Continuous software engineering: A roadmap and agenda. Journal of Systems and Software 123, 176–189 (2017)
  * [12] Gomez-Uribe, C.A., Hunt, N.: The netflix recommender system: Algorithms, business value, and innovation. ACM Trans. Manage. Inf. Syst. 6(4) (2016)
  * [13] Gutbrod, M., Münch, J., Tichy, M.: How do software startups approach experimentation? empirical results from a qualitative interview study. In: Felderer, M., Méndez Fernández, D., Turhan, B., Kalinowski, M., Sarro, F., Winkler, D. (eds.) Product-Focused Software Process Improvement. pp. 297–304. Springer International Publishing (2017)
  * [14] Klotins, E., Unterkalmsteiner, M., Chatzipetrou, P., Gorschek, T., Prikladnicki, R., Tripathi, N., Pompermaier, L.B.: Use of agile practices in start-up companies. e-Informatica Software Engineering Journal 15(1) (2021)
  * [15] Klotins, E., Unterkalmsteiner, M., Gorschek, T.: Software engineering in start-up companies: An analysis of 88 experience reports. Empirical Software Engineering 24(1), 68–102 (2019)
  * [16] Kniberg, H.: Spotify rhythm - how we get aligned (slides from my talk at agile sverige) (Jun 2016), https://blog.crisp.se/2016/06/08/henrikkniberg/spotify-rhythm, accessed: 2022-04-29
  * [17] Kohavi, R., Deng, A., Frasca, B., Walker, T., Xu, Y., Pohlmann, N.: Online controlled experiments at large scale. In: Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. pp. 1168–1176. KDD ’13, Association for Computing Machinery, New York, NY, USA (2013)
  * [18] Kohavi, R., Henne, R.M., Sommerfield, D.: Practical guide to controlled experiments on the web: listen to your customers not to the hippo. In: Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining. p. 959. ACM Press, San Jose, California, USA (2007)
  * [19] Kohavi, R., Crook, T., Longbotham, R., Frasca, B., Henne, R., Ferres, J.L., Melamed, T.: Online experimentation at microsoft. Data Mining Case Studies 11(2009), 39 (2009)
  * [20] Kotashev, K.: Startup failure rate: How many startups fail and why? (2022), https://www.failory.com/blog/startup-failure-rate, accessed: 2022-04-29
  * [21] Lindgren, E., Münch, J.: Software Development as an Experiment System: A Qualitative Survey on the State of the Practice. In: Lassenius, C., Dingsøyr, T., Paasivaara, M. (eds.) Agile Processes in Software Engineering and Extreme Programming, vol. 212, pp. 117–128. Springer International Publishing (2015)
  * [22] Lindgren, E., Münch, J.: Raising the odds of success: the current state of experimentation in product development. Information and Software Technology 77, 80–91 (2016)
  * [23] Mäntylä, V.: Continuous Experimentation in Finnish startups – A descriptive case study of A Grid and Maria 01 communities. Master’s thesis, Aalto University. School of Science (2022), http://urn.fi/URN:NBN:fi:aalto-202201301571
  * [24] Mäntylä, V., Lehtelä, B., Fagerholm, F.: Supplementary material for “the viability of continuous experimentation in early-stage software startups” (2022), https://docs.google.com/document/d/1Gx5dkKAZD-_0L5uNP1pZWRaiceoDZDlDPpLxnyTtShc/edit?usp=sharing, review version
  * [25] Melegati, J., Chanin, R., Wang, X., Sales, A., Prikladnicki, R.: Enablers and inhibitors of experimentation in early-stage software startups. In: Franch, X., Männistö, T., Martínez-Fernández, S. (eds.) Product-Focused Software Process Improvement. pp. 554–569. Springer International Publishing (2019)
  * [26] Melegati, J., Edison, H., Wang, X.: Xpro: A model to explain the limited adoption and implementation of experimentation in software startups. IEEE Transactions on Software Engineering 48(6), 1929–1946 (2022)
  * [27] Melegati, J., Guerra, E., Wang, X.: Understanding hypotheses engineering in software startups through a gray literature review. Information and Software Technology 133, 106465 (2021)
  * [28] Merriam, S.B., Tisdell, E.J.: Qualitative research: a guide to design and implementation. The Jossey-Bass higher and adult education series, John Wiley & Sons, San Francisco, CA, 4 edn. (2015)
  * [29] Olsson, H.H., Alahyari, H., Bosch, J.: Climbing the “stairway to heaven” – a mulitiple-case study exploring barriers in the transition from agile development towards continuous deployment of software. In: 38th Euromicro Conference on Software Engineering and Advanced Applications. pp. 392–399 (2012)
  * [30] Olsson, H.H., Bosch, J.: The HYPEX Model: From Opinions to Data-Driven Software Development. In: Bosch, J. (ed.) Continuous Software Engineering, pp. 155–164. Springer International Publishing, Cham (2014)
  * [31] Paternoster, N., Giardino, C., Unterkalmsteiner, M., Gorschek, T., Abrahamsson, P.: Software development in startup companies: A systematic mapping study. Information and Software Technology 56(10), 1200–1218 (2014)
  * [32] Ries, E.: The lean startup: how today’s entrepreneurs use continuous innovation to create radically successful businesses. Crown Business, New York, 1st ed edn. (2011)
  * [33] Rissanen, O., Münch, J.: Continuous Experimentation in the B2B Domain: A Case Study. In: 2015 IEEE/ACM 2nd International Workshop on Rapid Continuous Software Engineering. pp. 12–18. IEEE, Florence (2015)
  * [34] Schermann, G., Cito, J., Leitner, P.: Continuous experimentation: Challenges, implementation techniques, and current research. IEEE Software 35(2), 26–31 (2018)
  * [35] Shadish, W.R., Cook, T.D., Campbell, D.T.: Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin, Boston (2001)
  * [36] Steiber, A., Alänge, S.: A corporate system for continuous innovation: the case of Google Inc. European Journal of Innovation Management 16(2), 243–264 (2013)
  * [37] Thomke, S.H.: Managing experimentation in the design of new products. Management Science 44(6), 743–762 (1998)
  * [38] Wu, L., Grbovic, M.: How airbnb tells you will enjoy sunset sailing in barcelona? recommendation in a two-sided travel marketplace. In: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. pp. 2387–2396 (2020)
  * [39] Yaman, S.G.: Initiating the Transition towards Continuous Experimentation: Empirical Studies with Software Development Teams and Practitioners. Doctoral thesis, University of Helsinki (2019), http://urn.fi/URN:ISBN:978-951-51-5543-6
  * [40] Yaman, S.G., Fagerholm, F., Munezero, M., Münch, J., Aaltola, M., Palmu, C., Männistö, T.: Transitioning towards continuous experimentation in a large software product and service development organisation – a case study. In: Abrahamsson, P., Jedlitschka, A., Nguyen Duc, A., Felderer, M., Amasaki, S., Mikkonen, T. (eds.) Product-Focused Software Process Improvement. pp. 344–359. Springer International Publishing (2016)
  * [41] Yaman, S.G., Munezero, M., Münch, J., Fagerholm, F., Syd, O., Aaltola, M., Palmu, C., Männistö, T.: Introducing continuous experimentation in large software-intensive product and service organisations. Journal of Systems and Software 133, 195–211 (2017)
  * [42] Yin, R.K.: Case study research: design and methods. SAGE, Los Angeles, 5 edn. (2014)