11institutetext: Department of Physics & INFN, University of Rome ‘Tor
Vergata’, Via della Ricerca Scientifica 1, 00133 Rome, Italy

# Classifying Turbulent Environments via Machine Learning

Michele Buzzicotti 11 Fabio Bonaccorso 11

###### Abstract

The problem of classifying turbulent environments from partial observation is
key for some theoretical and applied fields, from engineering to earth
observation and astrophysics, e.g. to precondition searching of optimal
control policies in different turbulent backgrounds, to predict the
probability of rare events and/or to infer physical parameters labelling
different turbulent set-ups. To achieve such goal one can use different tools
depending on the system’s knowledge and on the quality and quantity of the
accessible data. In this context, we assume to work in a model-free setup
completely blind to all dynamical laws, but with a large quantity of (good
quality) data for training. As a prototype of complex flows with different
attractors, and different multi-scale statistical properties we selected 10
turbulent ‘ensembles’ by changing the rotation frequency of the frame of
reference of the 3d domain and we suppose to have access to a set of partial
observations limited to the instantaneous kinetic energy distribution in a 2d
plane, as it is often the case in geophysics and astrophysics. We compare
results obtained by a Machine Learning (ML) approach consisting of a state-of-
the-art Deep Convolutional Neural Network (DCNN) against Bayesian inference
which exploits the information on velocity and enstrophy moments. First, we
discuss the supremacy of the ML approach, presenting also results at changing
the number of training data and of the hyper-parameters. Second, we present an
ablation study on the input data aimed to perform a ranking on the importance
of the flow features used by the DCNN, helping to identify the main physical
contents used by the classifier. Finally, we discuss the main limitations of
such data-driven methods and potential interesting applications.

##  1 Introduction

Figure 1: 3D rendering of a turbulent flow under rotation at two different
values of rotation rate, namely $\Omega=4$ on the left side and $\Omega=10$ in
the right panel. In the first case the rotation frequency is small enough that
the flow develops a dynamics similar to the standard 3D non-rotating case,
with forward cascade of energy and small scales fluctuations. In the second
case instead rotation is large enough to produce an inverse energy transfer
that give rise to large-scales structures almost constant in the rotation
direction.

Extracting information about the statistical context from partial measurements
of a turbulent system is a fundamental problem in fluid dynamics. Unless the
data available are large and of very high quality, any linear regression
trying to statistically distinguish different turbulent set up from limited
data is destined to fail. This is evident if you need, e.g., to estimate the
turbulent intensity, given by a combination of the velocity root mean square,
viscosity and flow correlation length, out of sparse and temporal
measurements, and without any prior on the dissipative properties
corbetta2021deep . Similarly, many turbulent set-ups are subjected to
transitions at changing control parameters or boundary conditions, developing
macroscopically similar behaviour but different subtle multi-scale statistical
properties and/or transient phases alexakis2018cascades ; frisch1995turbulence
; Pope00 ; davidson2011voyage ; duraisamy2019turbulence . As a result,
classifying turbulent scenarios from spot measurements is a key problem to
reduce the complexity of predicting the flow evolution, to select optimal
policies, pre-trained in different set-ups, for optimal navigation
biferale2019zermelo ; novati2019controlled ; orzan2022optimizing ;
garnier2021review ; colabrese2017flow ; reddy2016learning ; reddy2018glider
and/or estimating the probability of extreme events frisch1995turbulence ;
scatamacchia2012extreme ; buzzicotti2021inertial ; biferale2016coherent ;
buaria2020self ; yeung2015extreme . Similarly in the context of data-driven
turbulence modelling of non-stationary and transient flows, an instantaneous
classification of the environment would be beneficial to perform an ad-hoc
fine-tuning of the closure model depending on the specific local turbulence
condition, e.g. high/low shear, stable/unstable conditions or strong/weak
updrafts, such as to cure the ‘curse-of-dimensionality’ and improve the
training performances maulik2019subgrid ; pathak2020using ; kochkov2021machine
; biferale2019self . As a prototype of complex flows with different dynamical
attractors, and different multi-scale statistical properties we selected 10
turbulent ‘ensembles’ by changing the rotation frequency of the frame of
reference of a 3d domain and we supposed to have access to a set of partial
observations limited to the instantaneous kinetic energy distribution in a 2d
plane, as it is often the case in geophysics and astrophysics
vallis2017atmospheric ; pedlosky1987geophysical ; akyildiz2002wireless ;
kalnay2003atmospheric ; buzzicotti_ocean_2021 ; carrassi2008data ;
brunton2016discovering ; bocquet2019data . Increasing rotation rate,
turbulence undergoes global and local changes, concerning the development of
quasi-2d columnar structures, extreme non-Gaussian small-scale anisotropic
fluctuations, long-range wave-wave correlations smith1999transfer ;
mininni2009helicity ; biferale2021rotating ; di2020phase , as a result, it can
be considered a paradigmatic test-bed where spatial and Fourier features have
different fingerprints in the statistical distributions.
Classification methods can be split in two different classes. The equation-
driven tools, where the data analysis is enhanced by the physical knowledge of
the underlying dynamical laws zou1992optimal ; ruiz2013estimating ;
di2018inferring , and the purely data-driven methods, based on the statistical
analysis of the available data. Some attempts to classify turbulence using
equation-driven tools have been developed in the recent years, using Fourier
spatio-temporal decomposition di2015spatio  or Nudging di2018inferring ;
di2020synchronization ; buzzicotti2020synchronizing . However, all these
techniques on top of the knowledge of the equation-of-motion require access to
important numerical resources to run digital twins of the observed system. In
this work we discuss the model-free approaches, where we are completely
agnostic of the system dynamical laws and we aim to implement purely data-
driven tools trained on a large quantity of good quality data.

Figure 2: Visualization of velocity magnitude on 2d planes extracted from a 3d
domain, in the stationary regime for the 10 different turbulent ‘ensembles’
used in the regression problem. The fields at $\Omega\leq 4$ are in the direct
cascade regime, while increasing the rotation rate we observe the transition
to the inverse energy cascade regime. (Bottom row panel ‘a’) Energy evolution
for the 10 different simulations for different $\Omega$, the dashed lines
split the data used for the training and for validation sets. (Bottom row
panel ‘b’) Energy spectra as a function of the wavenumbers for the same set of
simulations. The spectra are averaged on time in the stationary regime,
errorbars indicate the standard deviation and are generally inside the symbols
size.

In this paper, we attack this general inverse problem following recent
progresses in Machine Learning (ML) augmented fluid dynamics,
brenner2019perspective ; DuraisamyARFM2019 ; vinuesa2022enhancing ;
goodfellow2016deep ; brunton2019data ; brunton2020machine ;
biferale2019zermelo ; buzzicotti2021reconstruction ; brajard2019combining ;
borra2021using ; corbetta2021deep , by training a state-of-the-art Deep
Convolutional Neural Network (DCNN) in a supervised way,
krizhevsky2012imagenet ; he2016deep ; ronneberger2015u ; redmon2016you ,
thanks to a large dataset build up with the use of high performance Direct
Numerical Simulations (DNS) and accessible at the webpage http://smart-
turb.roma2.infn.it. We compare the non-linear ML regression against a Bayesian
baseline, based on the estimation of velocity and vorticity flow moments, in
order to have comparison with both large- and small-scale physics inputs.
Furthermore, we critically analyze the performance of the DCNN at changing the
amount of training data and the number of optimized hyper-parameters. Finally,
we try to open-the-black-box by performing an ablation study on the input data
aimed to rank the flow features used by the DCNN, and to help to identify the
main physical contents used by the classifier. Such ‘inverse-engineering’
approach, identifying key degrees-of-freedom for the classification can also
have important applications concerning active control to favour or contrast
flow transitions in other general set-ups. The paper is organized as follow.
In Sect. 2 we give details on the physics of turbulent flows and on the
regression problem we are focusing on. In Sect. 3 we describe the datasets
used. In Sect. 4 we describe the methodologies implemented, the DCNN and the
BI. Results are presented in Sect. 5. In Sect. 6 we present the ablation study
on the input data, and in Sect. 7 we draw our concluding remarks.

Figure 3: Visual representation of the DCNN input-output setup used in the
regression problem. The neural network is trained to map a plane of velocity
module extracted from 3d data to the rotation frequency of the reference
frame.

##  2 Physics Background

Before moving towards the details of the regression problem let us discuss the
physical background of turbulence under rotations. Rotating turbulence is
described by the Navier-Stokes equations (NSE) equipped with the Coriolis
force,

$\begin{cases}\partial_{t}\bm{u}+\bm{u}\cdot\nabla\bm{u}+2\Omega\hat{z}\times{\bm{u}}=-\nabla
p+\nu\Delta\bm{u}+\bm{f}\\\ \nabla\cdot\bm{u}=0,\end{cases}$ (1)

here $\nu$ is the kinematic viscosity, $2{\Omega\hat{z}}\times{\bm{u}}$ is the
Coriolis force produced by rotation, and $\Omega$ is the frequency of the
angular velocity around the rotation axis $\hat{z}$. The fluid density is
constant and absorbed into the definition of pressure $p$. $\bm{f}$,
represents an external forcing mechanism, see Sect. 3 for details on the type
of forcing used here. The relative importance of non-linear (advection) with
respect to linear terms (viscous dissipation) is given by the Reynolds number,
$Re=\varepsilon_{f}^{1/3}k_{f}^{-4/3}/\nu$, where $\varepsilon_{f}$ is the
rate of energy input at the forcing scale $L_{f}\sim 1/k_{f}$. Rotation
introduces a second non-dimensional control parameter, the Rossby number,
which represents the ratio between rotation frequency and the flow
characteristic time scale at the forcing wavenumber:

$Ro=\frac{(\varepsilon_{f}k_{f}^{2})^{1/3}}{\Omega}.$ (2)

While it is well known that increasing $Re$ leads to transitions to different
turbulent regime, the effects of varying $Ro$ are more subtle and the results
of contrasting flow features, including wave-wave interactions, large-scale
energy condensation in quasi-2d structures, small-scale anisotropic
intermittency biferale2016coherent . In the limit of large $Ro$ or small
$\Omega$, standard 3d homogeneous and isotropic turbulent dynamics is
observed, on the other hand, increasing $\Omega$, $Ro\leq 1$, the flow becomes
almost bidimensional deusebio2014dimensional ; marino2013inverse ;
marino2014large ; buzzicotti2018energy ; buzzicotti2018inverse ; di2020phase ,
moving with uniform velocity along the direction parallel to the rotation
axis, i.e. $\hat{z}$. Simultaneously, at high rotation rates, the flow
develops coherent/large-scales vortex like structures that live on the
horizontal plane, $(x,y)$, as can be seen in the renderings of Fig. 1. These
structures are fed by an inverse energy cascade that transports energy from
the injection scales towards the largest scales available in the physical
domain smith1999transfer ; seshasayanan2018condensates ; van2020critical .  In
Fig. 2 we present a visualization of our problem: classifying the turbulent
environments from one realization of the velocity module on 2d planes
extracted from 3d simulations at varying the reference rotation rate $\Omega$.
From the visual comparison of the 2d planes for different $\Omega$, (from
$\Omega=2$ to $\Omega=30$), and the results shown for the spectra and for the
temporal evolution of the total kinetic energy (panels ‘a’ and ‘b’) one can
immediately understand how it is not so difficult to classify the systems in
two rough classes, strong/weak rotations. On the other hand, it looks much
harder to distinguish different scenarios inside one of the two rough classes.
The aim of the paper is to find a tool capable to disentangle also the subtle
correlations that distinguish among all turbulent ensembles. We aim to solve
the inference problem with the new data analyses paradigms proposed in
computer vision such as a non-linear classifier based on DCNN
janai2020computer ; maron2020learning ; pidhorskyi2018generative ;
tan2020efficientdet ; chouhan2020applications ; cheng2021fashion ;
feng2019computer ; pathak2016context . We imagine to have only a partial
knowledge of the system, and in particular, for the sake of a possible
realistic implementation, we make the four following assumptions:

  * •

Time evolution is not accessible

  * •

We have access only to one scalar measurement, e.g. the velocity amplitude,
$|\bm{u}|$

  * •

We can observe the system on a 2d plane and not on the full 3d domain

As a baseline, we have repeated the same investigation using a Bayesian
Inference (BI) analysis, in order to compare the ‘supremacy’ of the black-box
DCNN with a tool based on correlations with physically meaningful observables,
i.e. moments of the velocity field and of its derivatives. Let us notice that
we have also attempted to perform the same classification, using the
unsupervised Gaussian Mixture Models (GMM) classification
rasmussen1999infinite ; reynolds2009gaussian ; HeGMM2011 . However we did not
find any relation between the unsupervised classification and the rotation
value, so it has not been included in the discussion with DCNN and BI.

Figure 4: Details of the Deep Convolutional Neural Network implemented in this
work.  The network starts with four convolutional layers that map the input
plane to 32 different 8x8 images of aggregated features. The second part of
the network is composed by two fully connected layers that give in output the
rotation prediction, $\Omega_{pred}$. All layers are followed by a ReLu
activation function, a dropout of $30\%$ and $20\%$ follows respectively the
two last fully connected layers.

##  3 Dataset

###  3.1 Numerical Simulations

To generate the database of turbulent flows on a rotating frame, we have
performed a set of Direct Numerical Simulations (DNS) of the NSE for
incompressible fluid in a triply periodic domain of size $L=2\pi$ with a
resolution of $N=256$ gird-points per spatial direction. We used a fully
dealiased parallel pseudospectral code, with time integration implemented as a
second-order Adams-Bashforth scheme and the viscous term implicitly
integrated. A linear friction term, $0.1\Delta^{-1}\bm{u}$, is added at wave-
numbers with module $|\bm{k}|\leq 2$ to reach a stationary state. At the same
time, viscous dissipation, $\nu\nabla^{2}\bm{u}$, is replaced by a
hyperviscous term, $\nu\nabla^{4}\bm{u}$, with $\nu=1.6\times 10^{-6}$, to
increase the inertial range of scales dominated by the non-linear energy
transfer. The forcing mechanism, $\bm{f}$, is a Gaussian process delta-
correlated in time, with support in wavenumber space centered at $k_{f}=4$.
The total energy time evolution for the 10 different simulations varying
$\Omega$ is presented in Fig. 2(a), while in panel (b) of the same figure we
show the energy spectra,
$E(k)=\frac{1}{2}\sum_{k\leq|\bm{k}|<k+1}|\bm{\hat{u}}(\bm{k})|^{2}$, averaged
over time in the stationary regime for the same set of 10 simulations. From
the energy spectra we can observe that while for $\Omega=2$ and $4$ (in the
direct cascade regime) there is a depletion of energy at wavenumbers smaller
than $|\bm{k}|<k_{f}$, for $\Omega$ values above the transition the split
cascade regime, also the small wavenumbers are filled in with energy driven by
the inverse cascade, namely from the forcing to the largest system’s scales.

###  3.2 Dataset extraction

Figure 5: (Panel a) Evolution of the Loss, $\mathcal{L}$, as a function of the
epochs, measured on both the training set (solid line full squares) and the
validation set (dashed line open circles), results are obtained with the DCNN
introduced in Fig. 4. (Panel d) Probability Density Function (PDF), measured
at different epochs, of the DCNN prediction normalized to the real value, for
the case of $\Omega_{true}=20$. The PDFs are measured on the whole validation
set. (Panels b and d) $\mathcal{L}$ and PDFs for the DCNN compared with the
performances of two different neural networks with the same architecture of
DCNN but with different weights. In particular, DCNN-f8 has 8 different
filters in each convolutional layer and a total of $145201$ parameters, while
DCNN-f64 has 64 filters in the convolutional layers and a total of $1403073$
trainable weights. (Panels c and f) Show the same results presented in panels
(a) and (d) but reducing the size of the training set by a factor 2 (red
lines) and by a factor 8 (blue lines) corresponding respectively to a total
number of $20$k and $5$k training planes for each $\Omega$.

The ML training and validation datasets, accessible at the webpage
http://smart-turb.roma2.infn.it, are extracted from the DNS described above,
following TurbRot , such as to match all assumption discussed in the
introduction (Sect. 1), namely;

  * •

In order to vary the Rossby number, we performed 10 numerical simulations each
with a different rotation frequency. Two simulations well below the transition
in the direct cascade regime, $\Omega=2,4$, three of them in the transition
region, $\Omega=5,6,7$, and the remaining five well inside the split cascade
regime with $\Omega=8,10,15,20,30$.

  * •

For each of the 10 simulations we have dumped a number of roughly $600$
snapshots of the full 3d velocity field with a temporal separation large
enough to decrease correlations between two successive data-points (see Fig.
2(a) for the total energy evolution).

  * •

For each configuration, $64$ horizontal planes $(x,y)$, (perpendicular to the
rotation axis), are selected at different $z$ and for each of them we computed
the module of the velocity field as;
$|\bm{u}|=\sqrt{u_{x}^{2}+u_{y}^{2}+u_{z}^{2}}$.

  * •

To increase the dataset, we shift each of the $64$ planes by choosing randomly
a new center of the plane and using periodic boundary conditions, such as to
obtain a total number of $50k$ planes for each $\Omega$ value.

The full dataset, in this way, is composed of half a million different
velocity module planes, out of which the $80\%$ of them are used as training
set while $20\%$ are used as validation set. To be sure that validation data
were never seen during training, none of the planes used in the validation set
is obtained from a random shift of a plane contained in the training set.
Instead, the two sets are built using the same protocol but from different
temporal realization of the flow splitting its time evolution in two separate
parts as also displayed by the vertical dashed lines in panel (a) of Fig. 2.

##  4 Methods

###  4.1 Deep Convolutional Neural Network (DCNN)

Fig. 3 schematizes the ML problem set-up. A plane of velocity module randomly
extracted from the dataset of 10 simulations is given in input to a DCNN with
the goal to infer the $\Omega$ value of the simulation from which the plane
has been generated. The training is performed in a supervised setting, having
pre-labelled all planes with their corresponding value, $\Omega_{true}$. As
solution of the regression problem, the DCNN gives in output a real number
corresponding to the predicted rotation value, $\Omega_{pred}$. In this way
the inferring task is not limited to a predefined set of classes and can be
generalized to estimate rotation values never observed during training. The
network is trained to minimize the following loss function;

$\mathcal{L}=\mathbb{E}\\{\left(\Omega_{true}-\Omega_{pred}\right)^{2}\\},$
(3)

which is the mean-squared-error between the target value $\Omega_{true}$ and
the predicted value $\Omega_{pred}$. $\mathbb{E}$ is the mean over a mini-
batch of 256 different planes. To implement the DCNN we used TensorFlow
abadi2016tensorflow  libraries with a Python interface, optimized to run on
GPUs.  Because of total dataset size, 245GB for the 500k different planes at a
resolution of $256^{2}$ grid points, the training had to be parallelized over
eight A100 Nvidia GPUs. As described in Fig. 4, the overall architecture is
composed of four convolutional layers that encode the input plane of velocity
module on $32$ different planes of size $8\times 8$. This first part of the
network is asked to aggregate the relevant features for the regression
problem. After, the results for the aggregated features are reshaped into a
single vector of $2048$ elements. The second half of the network, is composed
by two fully connected layers that transform the features vector into one real
number representing the network prediction, $\Omega_{pred}$. To reduce
overfitting the training of these two final fully connected layers is
performed using a dropout respectively of $30\%$ and $20\%$
srivastava2014dropout . All layers are followed by a non-linear ReLu
activation functions. Using a mini-batch of $256$ planes, and a training
dataset of $N_{tra}=400k$ planes, each epoch consists of $1563$ back-
propagation steps.  The backpropagation has been performed with an Adam
optimizer equipped with a Nesterov momentum dozat2016incorporating  using a
decreasing learning rate starting from $10^{-4}$ with a scheduler that
consists in a linear decrease of the learning rate up to reach $10^{-5}$ in 20
epochs and then to remain constant. To achieve a good training with the
learning protocol just introduced, we have normalized the input data to be in
the $[0;1]$ range, namely we rescaled the module intensity of each gird-point
by normalizing it to the maximum value calculated over the whole training set.
The same normalization constant has been used to rescale the validation set.
In the same way the training labels $\Omega_{true}$ have been rescaled by
their maximum value in the training. In Fig. 5 panel (a) we show the evolution
of the loss, $\mathcal{L}$, measured on the training and validation data as a
function of the number of epochs for the best model obtained in this work. The
loss evolution does not show evidence of overfitting and converges after
roughly $160$ epochs where the validation loss starts to saturate. In the
bottom row, panel (d) of Fig. 5, we show the Probability Density Function
(PDF) of predicting the correct rotation value during the different training
epochs measured over all planes of the validation set. Here $\Omega_{true}=20$
is used, but similar behaviours are observed for all the other $\Omega$. At
epoch $200$ the PDF becomes close to a delta-like distribution, suggesting
very good accuracy of the network, with errors only every few cases over one-
thousand, and within $30\%$ the correct value. In the central column, panels
(b,e) of Fig. 5, we compare the same results obtained varying the size of the
DCNN, precisely the number of its trainable weights keeping fixed the
architecture. Here the label DCNN-f8 and DCNN-f64 represent two neural
networks that have respectively $8$ and $64$ different filters in each
convolutional layer, instead of the $32$ as in the case of DCNN. The two
networks have a total number of parameters of roughly $145$k for the smaller
one and $1.4$M for the larger network. We can see that when the network is too
small the accuracy decreases on both the validation and the training sets
while if the network is too large there is some overfitting error on the
validation set. Also the PDFs is optimal for the intermediate network labeled
as DCNN. In the last column of Fig. 5, panels (c,f), we compare the DCNN
results trained on smaller datasets. Here we can appreciate the quick
deterioration of the prediction when reducing the quantity of the data used in
the training.

###  4.2 Bayesian Inference

The second statistical inference approach considered is the Bayesian Inference
(BI) aster2018parameter ; efron2013250 ; mackay1992bayesian . The idea is to
infer properties of the flow assuming the knowledge of some underlying
probability distributions. More specifically, the Bayes’ Theorem tells us that
the posterior probability, $P(\Omega_{i}|\mathcal{O})$, of observing
$\Omega_{i}$ given a measure of another generic observable, $\mathcal{O}$, can
be estimated as follow;

$P(\Omega_{i}|\mathcal{O})=\frac{P(\mathcal{O}|\Omega_{i})P(\Omega_{i})}{P(\mathcal{O})}=\frac{P(\mathcal{O}|\Omega_{i})P(\Omega_{i})}{\sum_{i}P(\mathcal{O}|\Omega_{i})P(\Omega_{i})}.$
(4)

$P(\Omega_{i})$ is the prior probability, in our case for all $\Omega_{i}$
equal to $0.1$ because the dataset is composed by the same number of planes
for each of the 10 $\Omega_{i}$. $P(\mathcal{O}|\Omega_{i})$, instead, is the
likelihood to get the measure $\mathcal{O}$ of a generic observable for a
fixed $\Omega_{i}$. Once known the posterior probability in Eq. (4), a first
Bayes estimation can be obtained as follows;

$\overline{\Omega}_{Bay}=\sum_{i=1}^{10}P(\Omega_{i}|\mathcal{O})\Omega_{i},$
(5)

which correspond to the mean value on the posterior probability. A second
possible estimation can be done as the most likely $\Omega_{i}$ given
$\mathcal{O}$, namely;

$\Omega^{*}_{Bay}=\max_{\Omega_{i}}P(\Omega_{i}|\mathcal{O})=\max_{\Omega_{i}}P(\mathcal{O}|\Omega_{i})P(\Omega_{i}),$
(6)

where in the last step the denominator of (4) is neglected because it does not
affect the maximization on $\Omega_{i}$.  For our problem, the two more
relevant physical observable are the total energy and the total squared
velocity gradients, representing large-scale (Gaussian) and small-scale (non-
Gaussian) features, respectively. Adding the information of flow time
evolution would be beneficial to have a measure of the waves propagation
inside the flow, but as discussed this is in general a much more complicated
measure and we are not assuming to rely on it. For this reason, in the sequel
we will proceed consider $\langle u^{2}\rangle$ and
$\langle(\partial_{x}u)^{2}\rangle$, where $u=|\bm{u}|$ is the module of the
horizontal velocity and $\langle\cdots\rangle$ stands the average over the
plane. For each of these two observable we have calculated the likelihood
$P(\mathcal{O}|\Omega_{i})$ from the $40k$ planes for each $\Omega_{i}$ of the
training dataset. Namely we have measured $P(\langle
u^{2}\rangle|\Omega_{i})$, $P(\langle(\partial_{x}u)^{2}\rangle|\Omega_{i})$
and the joint-likelihood, $P(\langle
u^{2}\rangle,\langle(\partial_{x}u)^{2}\rangle|\Omega_{i})$ on the training
set.

Figure 6: Joint-likelihood, $P(\langle
u^{2}\rangle,\langle(\partial_{x}u)^{2}\rangle|\Omega_{i})$, of the squared
velocity module and its derivative along the $x$ direction, measured for the
10 different $\Omega$. The joint-PDFs are measured from the training set, and
colours from dark-blue to red are proportional to the PDF value. The first
cluster on the left consists of planes in the direct cascade, as indicate by
the labels ($\Omega=2$ and $4$), the other clusters are composed by the
overlapping of the remaining $\Omega$ values.

Measuring the same observable on each plane of the validation dataset, and
knowing the likelihoods for each $\Omega_{i}$, we could estimate the posterior
probability $P(\Omega_{i}|\mathcal{O})$ (Eq. (4)) and than infer
$\Omega_{Bay}(\mathcal{O})$ (Eq. (5)). Fig. 6 shows the joint-likelihood of
the mean energy and mean gradients.  From this plot we can see that also
combining large and small scales observables does not allow to have a clean
statistical separation of the different $\Omega_{i}$ for all the 10
‘ensembles’ considered here. In the next section we present and discuss
results of $\Omega$ inference via DCNN and standard Bayesian approach.

Figure 7: Scatterplot of inferred vs real rotation value. The inference
estimation is obtained with the ML approach, $\Omega_{pred}$ (panel a), or
with BI, $\overline{\Omega}_{Bay}$, conditioning on the PDF of the velocity
square module (panel b), of the velocity gradients (panel c) or with the joint
PDF of velocity module and gradients (panel d). The diagonal lines indicate
the identity function, while the red full circles dots are the predictions for
each planes on the validation set, their size and color intensity is
proportional to the density of points in a given area. The mean values over
all predictions for all planes of each $\Omega_{true}$ are reported with the
open hexagon symbols. The vertical dashed lines separate the two rough classes
weak/strong rotation rates.

##  5 Results

Figure 8: PDF of rotation value predicted by the neural network,
$\Omega_{pred}$ (DCNN, solid lines full triangles), compared with the mean,
$\overline{\Omega}_{Bay}$ (red histograms), and most likely,
$\Omega^{*}_{Bay}$ (black histograms), Bayes inference using joint-likelihood
$P(\langle u^{2}\rangle,\langle(\partial_{x}u)^{2}\rangle|\Omega_{i})$, on the
validation set. For the sake of data presentation each of the PDF measured on
the 10 different $\Omega$ values are reported in 10 different panels (from ‘a’
to ‘l’). As indicated in the axis, the PDF are all normalized by their
corresponding $\Omega_{true}$ indicate in the labels of each panel.

A first quantitative comparison of the two techniques is presented in Fig. 7,
where in the four panels we report the scatterplot between the reference
rotation value, $\Omega_{true}$ and the corresponding prediction obtained via
the DCNN (panel a), and BI with the three different likelihoods, namely the
one based on the mean energy (panel b), on the mean square gradients (panel c)
and on the joint likelihood combining energy and gradients (panel d). The four
panels are all obtained with the same validation dataset of $10k$ planes for
each of the 10 $\Omega_{i}$ values.  The first important observation is that
the DCNN prediction is surprisingly close to the correct values not only on
average but also for each single sample. Indeed, from the top left panel of
Fig. 7 we can see that all points in the scatter plot fall well inside the
hexagon symbol, just few exceptions are observed for the cases of
$\Omega_{true}=8$ and $10$, and for $\Omega_{true}=2$ and $4$. Let us stress
that this result is far from being trivial and provides a first example where
a ML data-driven technique succeed in the inferring of an hidden parameter
from the analysis of real fully developed turbulent configuration. The reason
of this success is double, the intrinsic potential of DCNN and the good
quality of the dataset produced and implemented in the training of the
network. Indeed as already shown a dataset reduction quickly leads to a
deterioration of the training results, using a dataset up to eight times
smaller we have seen that the DCNN predictions does not change on average but
the points distribution around the correct values would be much larger, as
shown for the case of $\Omega=20$ in the PDFs of Fig. 5 panel (f). The second
important consideration can be drawn by the observation of the BI results.
Here, looking at panels (b-c-d) we can assess the ranking of different
features for the classification task. When using only the energy, BI is able
to distinguish among the two rough classes (below/above the energy transition)
only, while it is completely blind to differences among the intermediate
$\Omega$ values. In panel (c) we see that measuring small-scale features is
more useful for the classification, at least on average. Panel (d) shows that
combining small and large scales input improves the results, keeping the BI
close to the ML approach. Keeping adding different observable to condition the
BI approach is clearly doomed of fail, because of the difficulties to have a
large statistical sample to probe the conditional probability on a larger and
larger dimensional space.

Figure 9:  (Panel a) Normalized mean squared error (MSE) between the predicted
and true $\Omega$ values for each of the 10 classes considered. The average is
taken over all validation planes of each simulations, while the predictions
are obtained with DCNN (black squares), BI $\overline{\Omega}_{Bay}$ (green
triangles) and BI $\Omega^{*}_{Bay}$ (red circles). All BI estimations are
obtained using joint-likelihood $P(\langle
u^{2}\rangle,\langle(\partial_{x}u)^{2}\rangle|\Omega_{i})$. (Panel b) Mean
normalized distance between the mean predictions averaged over all planes in
each simulation and the selected value of $\Omega=8$, plotted as a function of
the simulation $\Omega$. Black squares solid line reports the DCNN
predictions, red circles dashed line reports the BI predictions.

The detailed comparison between DCNN and the best BI with joint likelihood on
velocity module and gradients is presented in Fig. 8, where 10 different
panels report the PDF of the DCNN and BI predictions, for all planes and all
$\Omega$ of the validation set. Each PDF is normalized to its reference
rotation value $\Omega_{true}$ such as $\Omega_{pred}/\Omega_{true}=1$
indicates the perfect prediction. Predictions in the small rotation regime
($\Omega=2,4$) are the more difficult because no large-scale coherent
structures are expected to appear in the flow and the resulting field is very
little affected by the presence of rotation, so the predictions are easily
confused among these two values. Comparing the two PDFs panels (a) and (b) we
notice that while Bayes peaks always at the center between the two values, the
DCNN is correctly peaked at the right value.  In general for all the analyzed
$\Omega$ (panels a-l) the DCNN accuracy is always very high and the larger
errors happen only for a few cases on the $10k$ planes of the validation set.
The last comparison between DCNN and BI is presented in Fig. 9. Here in panel
(a) there is a summary with the statistics of the different DCNN and BI
predictions. To do so we measure the Mean Squared Error (MSE) prediction
normalized to the target value,

$MSE=\frac{\langle(\Omega_{pred}-\Omega_{true})^{2}\rangle}{\Omega_{true}^{2}}.$
(7)

From this figure we can see that DCNN is superior to the BI in a consistent
way for all rotation rates. On the other hand, in panel (b), we focus on the
sensitivity of the two inferring tools to distinguish parameters used in
different simulations. In particular we aim to identify from the analysis of
all planes of each simulation the ones that belong to a specific rotation
class, that in this case was chosen to be $\Omega=8$. In Fig. 9(b) we measure
the mean normalized distance,

$\frac{\langle\Omega_{pred}-8\rangle}{8}.$ (8)

between the prediction $\Omega_{pred}$ and the value of interest, $\Omega=8$,
averaging over all planes from taken from the same simulation as a function of
the different $\Omega$ values used in the simulations. We can see that the
distance is sharply minimized when the planes come from the DNS with the
chosen rotation $\Omega=8$. In other words, Fig. 9(b) highlights the
sensitivity of the data-driven tools in classifying complex turbulent
conditions from ‘ensembles’ of different environments.

##  6 Occlusion

Here we present the ablation study on the input data aimed to perform a
ranking on the importance of the flow features used by the DCNN to get a
correct prediction, see lellep2022interpreted  where a similar approach has
been used on plane Couette flow to characterize the relevance of different
regions for the prediction of relaminarisation events. The idea consists on
keeping fixed the best DCNN trained model and to provide in input data where a
fraction of the information were occluded such as to evaluate the
deterioration of the inferring performance as a function of the different
features occluded. As illustrated in Fig. 10 the first type of ablation
performed is done in real space and consists in applying a filter (zero-mask)
over all high or low energy regions up to a given percentage of points in the
plane. In this way we aim to occlude a particular spatial input region,
characterized by its energy, in order to evaluate its relative importance for
the network prediction. The real space occlusion fields can be defined as;

$\displaystyle u^{<}_{o}(\bm{x})$
$\displaystyle=\begin{cases}u(\bm{x})&,\text{ if }|\bm{u}|^{2}<e_{h}\\\
0&,\text{ otherwise}\end{cases};$ (9) $\displaystyle u^{>}_{o}(\bm{x})$
$\displaystyle=\begin{cases}u(\bm{x})&,\text{ if }|\bm{u}|^{2}>e_{l}\\\
0&,\text{ otherwise}\end{cases},$ (10)

where $u_{o}^{<}$ is occluded field on the high energy regions, i.e. the mask
removes all points where the energy is larger than $e_{h}$. The same idea
holds for $u_{o}^{>}$, but now this field is masked on the low energy regions,
namely on all points with energy lower than $e_{l}$. The two thresholds
$(e_{h},e_{l})$, are chosen such that the percentage of points occluded is
equal in the two cases. In Fig. 10 there is a visualization for both the high
energy and low energy ablation for three different percentages of occlusion
and three different rotation values.

Figure 10: Examples of real space occlusions defined on the high and low
energy regions respectively for three different values of $\Omega$ and for
three different percentage of points occluded. The three rows from top to
bottom present visualizations for, $\Omega=20,7$ and $2$, with a percentage of
ablation of, $40\%,20\%$ and $2\%$, on both the high and the low energy
regions. Figure 11: MSE, as defined in Eq. (7), obtained for the predictions
on the occluded fields, varying the occlusion percentage for both masks
applied to the higher energy (green triangles) and to the lower energy regions
(red circles). Panels (a) and (b) present results for the weak rotation rates
($\Omega=2$ and $4$), while the remaining eight panels form (c) to (l) report
results for the strong rotation rates where the fields are dominated by the
intense vortical structures.

To evaluate the importance of the occluded regions in terms of the network
prediction, in Fig. 11 we measure the MSE, as defined in Eq. (7), obtained for
the predictions on the occluded fields, varying the percentage of occlusion
and comparing the ablation of high or low energy regions. Also in this case it
is useful to distinguish between the weak and the strong rotation rates. The
panels ‘a’ and ‘b’ show that for $\Omega=2$ the high energy regions are more
important, and their ablation deteriorates the prediction. On the other hand
for $\Omega=4$ the network focuses on the low energy regions which are found
to be more important by this ablation study. All the remaining panels from ‘c’
to ‘l’ correspond to the strong rotation rates, and support in a consistent
way that in the inverse cascade regime the intense vortical regions are more
important for the correct classification. It is interesting to observe that
for few cases, $\Omega=5,6$ and $20$, the occlusion of a small percentage of
the less energetic regions improved the DCNN prediction compared to the one
obtained on full input. The second type of occlusion, illustrated in Fig. 12,
is a Fourier-space ablation. Here we remove all the degrees-of-freedom (DOF)
contained in a specific circumference of radius $|\bm{k}|=k_{filt}$ from the
original plane in Fourier-space. Let us stress that the Fourier-space ablation
is non-local in real space. Hence, all points of the occluded inputs are
modified by the Fourier-space ablation. It is also important to notice that
when increasing the filter wavenumber, $k_{filt}$, the percentage DOF removed
from the system becomes larger.

Figure 12: Illustration of the effects of the Fourier type of occlusion
applied at three different wavenumbers namely $k_{filt}=1,\,2,4$, compared
with the original flow configuration presented in the first top visualization.

To quantify the relevance of a particular Fourier frequency $k_{filt}$ we have
evaluated the relative error between the DCNN predictions evaluated on the
occluded and on the full input, defined as;

$R(k_{filt})=\frac{(\text{DCNN}[u_{o}(\bm{x},k_{filt})]-\Omega_{true})^{2}}{(\text{DCNN}[u(\bm{x})]-\Omega_{true})^{2}},$
(11)

where we have indicated as DCNN$[u(\bm{x})]$ the network prediction and as
$u_{o}(\bm{x},k_{filt})$ the occluded field depending on the filter
wavenumber. $R(k_{filt})>1$ indicates high-relevance for the occluded features
because their removal deteriorates the prediction. On the contrary
$R(k_{filt})<1$ suggest low relevance for the DOF at $k_{filt}$ whose removal
improved the DCNN prediction. In Fig. 13 we report the results for the
relevance $R(k_{filt})$ averaged over all planes of the validation set. Panel
(a) shows results for a representative subset of the high rotation rates,
$\Omega=5,8$ and $30$, while panel (b) shows results for the two lower
rotation rates. The main plot in both panels shows the mean relevance $\langle
R(k_{filt})\rangle$, as a function of the filter wavenumbers, averaged over
all planes in the validation set of each $\Omega$. As already observed from
the real space occlusion, the role of the large scales depends on the rotation
rate. When the rotation is strong, $\Omega\geq 5$, the relevance of the large
and more energetic scales is preeminent. On the other hand, as expected, they
are not relevant for the weak rotation rates. Furthermore, with the Fourier
space occlusion we have access to the role of the smaller scales and we found
that the DCNN is sensitive to the occlusion on a wide range of scales up to
$k_{filt}\sim 40$. This supports the idea that the neural network has learned
to evaluate the complex multi-scale statistical properties to improve its
prediction. Only in the particular case of $\Omega=2$ we have not observed any
particular relevance at any scale. In the inset of both panels (a) and (b) of
Fig. 13 we report the $R(k_{filt})$ variance as a function of the filter
scale. Again the variance is evaluated over all planes of the validation set
of each $\Omega$. From the variance we can highlight the presence of an
exponential cutoff in the relevance of all wavenumbers larger than
$k_{filt}\sim 40$. This observation tells us that such scales are dominated by
viscous dissipation which is independent of the effect of rotation.

Figure 13: Relevance $R(k_{filt})$, see Eq. (11), resulting from the Fourier
space ablation as a function of the filter wavenumber $k_{filt}$. Panel (a)
shows results obtained at $\Omega=5,8$ and $30$, all in the inverse cascade
regime. Panel (b) focuses on the two weak rotation rates $\Omega=2,4$. In both
panels the main plot reports the mean relevance value ($\langle R\rangle$)
while the inset reports its variance $\sigma(R)$. The horizontal dashed lines
indicate the value $R(k_{filt})=1$, hence the value where the error on the
occluded input is equal to the error on the full field.

In conclusion we have shown an attempt to use ablation to perform a features
ranking on the flow properties more important for prediction of the DCNN. In
our case we have found that the large scales swirling structures generated by
rotation are generally more relevant for the correct classification of the
high rotation rates. On the other hand we have seen that there is a full range
of scales up to wavenumbers around $k_{filt}\sim 40$ that must be considered
to improve the inferring prediction.

##  7 Conclusions

We have studied the problem of inferring physical parameters of turbulent
flows with purely data-driven tools without assuming any knowledge on the
dynamical laws governing the systems. The case of study is 3d turbulence on a
rotating frame with the goal of inferring the rotation frequency from the
analysis of the flow velocity module on a 2d plane observed at a fixed time
from the domain top. This regression problem, without considering the temporal
dynamics of the flow, has not a clear solution even considering equation-
driven approaches, and it represents an open question for theoretical
physicists working on rotating flows. In this work we take advantage of the
new ML tools and of a big pre-labeled dataset from high quality DNS TurbRot
to tackle such problem. In particular we have trained a Deep Convolutional
Neural Network in a supervised way to obtain a ML model able to solve this
complex task. To compare the ‘supremacy’ of DCNN results and to guide the
reader towards a better understanding of the DCNN solution we have repeated
the same game using standard Bayesian Inference (BI). For the latter
estimation we have used two physically meaningful observables, moments of the
velocity and of its gradients. In this way we have shown how the combination
of more independent observables can allow to reach a better prediction,
however simply increasing the number of observables would lead to a failure
because of the difficulties to have a enough statistics to probe on the larger
and larger dimensional space of the conditional probability. Performing an
ablation study on the input data and fixing the best trained network we have
shown that the DCNN is sensitive to the removal of small fraction of the input
data, and it can be used to rank different flow features in terms of their
importance for the prediction. Performing an ablation in real space we have
shown that intense energy regions are relevant for a correct classification of
high rotation rates. At the same time with a Fourier space ablation we have
shown that DCNN is sensitive to changes in the input applied over a large
range of wavenumbers, indicating the capability of the network to identify not
only regions in space more relevant than other but also that it is measuring
non-trivial and multiscale observables to improve its prediction. As briefly
discussed in the introduction, an unsupervised classification of the same
dataset has been attempted using a GMM classification algorithm, however no
relation between the classes identified and the rotation value has been found.
Indeed GMM, as more in general k-means algorithms lee1999learning , are mainly
sensitive to the number of vortexes inside each plane or to their spatial
location and not to observables that are related to the rotation rate. For
this reason a detailed analysis using this tool has been omitted in the paper.
The classification of complex turbulent environments can be extended in
several ways and has many real world applications. One example, consists in
inferring of the intensification of extreme cyclones in the atmosphere
bhatia2019recent ; xu2021deep , where learning how to combine standard
information with more complex data as satellites images can lead to better
predictions of such phenomena. Another potential application is in self-
navigation problem, where autonomous vehicles need to know the on-time flow
conditions before to choose the best policy to optimize their navigation
biferale2019zermelo ; orzan2022optimizing ; novati2019controlled ;
reddy2016learning ; reddy2018glider . Also in the realm of turbulence
modelling it would be beneficial to perform an ad-hoc fine-tuning of the
closure model depending on the specific local turbulence condition to improve
the training performances maulik2019subgrid ; pathak2020using ;
kochkov2021machine . Our work show that DCNN can solve this kind of inferring
problems with unprecedented accuracy still under the limitation of having a
large quantity of good quality data.

## Acknowledgement

The authors thank Prof. Luca Biferale for inspiration and many useful
discussions. This work was supported by the European Research Council (ERC)
under the European Union’s Horizon 2020 research and innovation programme
(Grant Agreement No. 882340), and by the project Beyond Borders (CUP):
E84I19002270005 funded by the University of Rome Tor Vergata.

## Data Availability Statements

The datasets generated during and/or analysed during the current study are
available in the Smart-Turb repository, http://smart-turb.roma2.infn.it.

## Author contribution statement

All authors contributed equally to the paper.

## References

  * [1] Alessandro Corbetta, Vlado Menkovski, Roberto Benzi, and Federico Toschi.  Deep learning velocity signals allow quantifying turbulence intensity.  Science Advances, 7(12):eaba7281, 2021.
  * [2] Alexandros Alexakis and Luca Biferale.  Cascades and transitions in turbulent flows.  Physics Reports, 767:1–101, 2018.
  * [3] Uriel Frisch.  Turbulence: the legacy of AN Kolmogorov.  Cambridge University Press, 1995.
  * [4] Stephen B. Pope.  Turbulent Flows.  Cambridge University Press, 2000.
  * [5] Peter A Davidson, Yukio Kaneda, Keith Moffatt, and Katepalli R Sreenivasan.  A voyage through turbulence.  Cambridge University Press, 2011.
  * [6] Karthik Duraisamy, Gianluca Iaccarino, and Heng Xiao.  Turbulence modeling in the age of data.  Annual Review of Fluid Mechanics, 51:357–377, 2019.
  * [7] Luca Biferale, Fabio Bonaccorso, Michele Buzzicotti, Patricio Clark Di Leoni, and Kristian Gustavsson.  Zermelo’s problem: Optimal point-to-point navigation in 2d turbulent flows using reinforcement learning.  Chaos: An Interdisciplinary Journal of Nonlinear Science, 29(10):103138, 2019.
  * [8] Guido Novati, Lakshminarayanan Mahadevan, and Petros Koumoutsakos.  Controlled gliding and perching through deep-reinforcement-learning.  Physical Review Fluids, 4(9):093902, 2019.
  * [9] N Orzan, C Leone, A Mazzolini, J Oyero, and A Celani.  Optimizing airborne wind energy with reinforcement learning.  arXiv preprint arXiv:2203.14271, 2022.
  * [10] Paul Garnier, Jonathan Viquerat, Jean Rabault, Aurélien Larcher, Alexander Kuhnle, and Elie Hachem.  A review on deep reinforcement learning for fluid mechanics.  Computers & Fluids, 225:104973, 2021.
  * [11] Simona Colabrese, Kristian Gustavsson, Antonio Celani, and Luca Biferale.  Flow navigation by smart microswimmers via reinforcement learning.  Physical review letters, 118(15):158004, 2017.
  * [12] Gautam Reddy, Antonio Celani, Terrence J Sejnowski, and Massimo Vergassola.  Learning to soar in turbulent environments.  Proceedings of the National Academy of Sciences, 113(33):E4877–E4884, 2016.
  * [13] Gautam Reddy, Jerome Wong-Ng, Antonio Celani, Terrence J Sejnowski, and Massimo Vergassola.  Glider soaring via reinforcement learning in the field.  Nature, 562(7726):236–239, 2018.
  * [14] R Scatamacchia, L Biferale, and F Toschi.  Extreme events in the dispersions of two neighboring particles under the influence of fluid turbulence.  Physical review letters, 109(14):144501, 2012.
  * [15] Michele Buzzicotti and Guillaume Tauzin.  Inertial range statistics of the entropic lattice boltzmann method in three-dimensional turbulence.  Physical Review E, 104(1):015302, 2021.
  * [16] Luca Biferale, Fabio Bonaccorso, Irene M Mazzitelli, Michel AT van Hinsberg, Alessandra S Lanotte, Stefano Musacchio, Prasad Perlekar, and Federico Toschi.  Coherent structures and extreme events in rotating multiphase turbulent flows.  Physical Review X, 6(4):041036, 2016.
  * [17] Dhawal Buaria, Alain Pumir, and Eberhard Bodenschatz.  Self-attenuation of extreme events in navier–stokes turbulence.  Nature communications, 11(1):1–7, 2020.
  * [18] PK Yeung, XM Zhai, and Katepalli R Sreenivasan.  Extreme events in computational turbulence.  Proceedings of the National Academy of Sciences, 112(41):12633–12638, 2015.
  * [19] Romit Maulik, Omer San, Adil Rasheed, and Prakash Vedula.  Subgrid modelling for two-dimensional turbulence using neural networks.  Journal of Fluid Mechanics, 858:122–144, 2019.
  * [20] Jaideep Pathak, Mustafa Mustafa, Karthik Kashinath, Emmanuel Motheau, Thorsten Kurth, and Marcus Day.  Using machine learning to augment coarse-grid computational fluid dynamics simulations.  arXiv preprint arXiv:2010.00072, 2020.
  * [21] Dmitrii Kochkov, Jamie A Smith, Ayya Alieva, Qing Wang, Michael P Brenner, and Stephan Hoyer.  Machine learning–accelerated computational fluid dynamics.  Proceedings of the National Academy of Sciences, 118(21):e2101784118, 2021.
  * [22] Luca Biferale, Fabio Bonaccorso, Michele Buzzicotti, and Kartik P Iyer.  Self-similar subgrid-scale models for inertial range turbulence and accurate measurements of intermittency.  Physical review letters, 123(1):014503, 2019.
  * [23] Geoffrey K Vallis.  Atmospheric and oceanic fluid dynamics.  Cambridge University Press, 2017.
  * [24] Joseph Pedlosky et al.  Geophysical fluid dynamics, volume 710.  Springer, 1987.
  * [25] Ian F Akyildiz, Weilian Su, Yogesh Sankarasubramaniam, and Erdal Cayirci.  Wireless sensor networks: a survey.  Computer networks, 38(4):393–422, 2002.
  * [26] Eugenia Kalnay.  Atmospheric modeling, data assimilation and predictability.  Cambridge university press, 2003.
  * [27] Michele Buzzicotti, Benjamin A Storer, Stephen M Griffies, and Hussein Aluie.  A coarse-grained decomposition of surface geostrophic kinetic energy in the global ocean.  Earth and Space Science Open Archive, page 58, 2021.
  * [28] Alberto Carrassi, Michael Ghil, Anna Trevisan, and Francesco Uboldi.  Data assimilation as a nonlinear dynamical systems problem: Stability and convergence of the prediction-assimilation system.  Chaos: An Interdisciplinary Journal of Nonlinear Science, 18(2):023112, 2008.
  * [29] Steven L Brunton, Joshua L Proctor, and J Nathan Kutz.  Discovering governing equations from data by sparse identification of nonlinear dynamical systems.  Proceedings of the national academy of sciences, 113(15):3932–3937, 2016.
  * [30] Marc Bocquet, Julien Brajard, Alberto Carrassi, and Laurent Bertino.  Data assimilation as a learning tool to infer ordinary differential equation representations of dynamical models.  Nonlinear Processes in Geophysics, 26(3):143–162, 2019.
  * [31] Leslie M Smith and Fabian Waleffe.  Transfer of energy to two-dimensional large scales in forced, rotating three-dimensional turbulence.  Physics of fluids, 11(6):1608–1622, 1999.
  * [32] Pablo Daniel Mininni and A Pouquet.  Helicity cascades in rotating turbulence.  Physical Review E, 79(2):026304, 2009.
  * [33] Luca Biferale.  Rotating turbulence.  Journal of Turbulence, 22(4-5):232–241, 2021.
  * [34] P Clark Di Leoni, Alexandros Alexakis, L Biferale, and M Buzzicotti.  Phase transitions and flux-loop metastable states in rotating turbulence.  Physical Review Fluids, 5(10):104603, 2020.
  * [35] X Zou, IM Navon, and FX LeDimet.  An optimal nudging data assimilation scheme using parameter estimation.  Quarterly Journal of the Royal Meteorological Society, 118(508):1163–1186, 1992.
  * [36] Juan Jose Ruiz, Manuel Pulido, and Takemasa Miyoshi.  Estimating model parameters with ensemble-based data assimilation: A review.  Journal of the Meteorological Society of Japan. Ser. II, 91(2):79–99, 2013.
  * [37] Patricio Clark Di Leoni, Andrea Mazzino, and Luca Biferale.  Inferring flow parameters and turbulent configuration with physics-informed data assimilation and spectral nudging.  Physical Review Fluids, 3(10):104604, 2018.
  * [38] P Clark di Leoni, Pablo J Cobelli, and Pablo D Mininni.  The spatio-temporal spectrum of turbulent flows.  The European Physical Journal E, 38(12):1–10, 2015.
  * [39] Patricio Clark Di Leoni, Andrea Mazzino, and Luca Biferale.  Synchronization to big data: Nudging the navier-stokes equations for data assimilation of turbulent flows.  Physical Review X, 10(1):011023, 2020.
  * [40] Michele Buzzicotti and Patricio Clark Di Leoni.  Synchronizing subgrid scale models of turbulence to data.  Physics of Fluids, 32(12):125116, 2020.
  * [41] MP Brenner, JD Eldredge, and JB Freund.  Perspective on machine learning for advancing fluid mechanics.  Physical Review Fluids, 4(10):100501, 2019.
  * [42] Karthik Duraisamy, Gianluca Iaccarino, and Heng Xiao.  Turbulence modeling in the age of data.  Annual Review of Fluid Mechanics, 51(1):357–377, 2019.
  * [43] Ricardo Vinuesa and Steven L Brunton.  Enhancing computational fluid dynamics with machine learning.  Nature Computational Science, 2(6):358–366, 2022.
  * [44] Ian Goodfellow, Yoshua Bengio, and Aaron Courville.  Deep learning.  MIT press, 2016.
  * [45] Steven L Brunton and J Nathan Kutz.  Data-driven science and engineering: Machine learning, dynamical systems, and control.  Cambridge University Press, 2019.
  * [46] Steven L Brunton, Bernd R Noack, and Petros Koumoutsakos.  Machine learning for fluid mechanics.  Annual Review of Fluid Mechanics, 52:477–508, 2020.
  * [47] Michele Buzzicotti, Fabio Bonaccorso, P Clark Di Leoni, and Luca Biferale.  Reconstruction of turbulent data with deep generative models for semantic inpainting from turb-rot database.  Physical Review Fluids, 6(5):050503, 2021.
  * [48] Julien Brajard, Alberto Carrassi, Marc Bocquet, and Laurent Bertino.  Combining data assimilation and machine learning to emulate a dynamical model from sparse and noisy observations: a case study with the lorenz 96 model.  Geoscientific Model Development Discussions, pages 1–21, 2019.
  * [49] Francesco Borra and Marco Baldovin.  Using machine-learning modeling to understand macroscopic dynamics in a system of coupled maps.  Chaos: An Interdisciplinary Journal of Nonlinear Science, 31(2):023102, 2021.
  * [50] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  Advances in neural information processing systems, 25:1097–1105, 2012.
  * [51] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [52] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer, 2015.
  * [53] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi.  You only look once: Unified, real-time object detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 779–788, 2016.
  * [54] Enrico Deusebio, Guido Boffetta, Erik Lindborg, and Stefano Musacchio.  Dimensional transition in rotating turbulence.  Physical Review E, 90(2):023005, 2014.
  * [55] Raffaele Marino, Pablo Daniel Mininni, Duane Rosenberg, and Annick Pouquet.  Inverse cascades in rotating stratified turbulence: fast growth of large scales.  EPL (Europhysics Letters), 102(4):44006, 2013.
  * [56] Raffaele Marino, Pablo Daniel Mininni, Duane L Rosenberg, and Annick Pouquet.  Large-scale anisotropy in stably stratified rotating flows.  Physical Review E, 90(2):023018, 2014.
  * [57] Michele Buzzicotti, Hussein Aluie, Luca Biferale, and Moritz Linkmann.  Energy transfer in turbulence under rotation.  Physical Review Fluids, 3(3):034802, 2018.
  * [58] Michele Buzzicotti, Patricio Clark Di Leoni, and Luca Biferale.  On the inverse energy transfer in rotating turbulence.  The European Physical Journal E, 41(11):1–8, 2018.
  * [59] Kannabiran Seshasayanan and Alexandros Alexakis.  Condensates in rotating turbulent flows.  Journal of Fluid Mechanics, 841:434–462, 2018.
  * [60] Adrian van Kan and Alexandros Alexakis.  Critical transition in fast-rotating turbulence within highly elongated domains.  Journal of Fluid Mechanics, 899, 2020.
  * [61] Joel Janai, Fatma Güney, Aseem Behl, Andreas Geiger, et al.  Computer vision for autonomous vehicles: Problems, datasets and state of the art.  Foundations and Trends® in Computer Graphics and Vision, 12(1–3):1–308, 2020.
  * [62] Haggai Maron, Or Litany, Gal Chechik, and Ethan Fetaya.  On learning sets of symmetric elements.  In International Conference on Machine Learning, pages 6734–6744. PMLR, 2020.
  * [63] Stanislav Pidhorskyi, Ranya Almohsen, Donald A Adjeroh, and Gianfranco Doretto.  Generative probabilistic novelty detection with adversarial autoencoders.  arXiv preprint arXiv:1807.02588, 2018.
  * [64] Mingxing Tan, Ruoming Pang, and Quoc V Le.  Efficientdet: Scalable and efficient object detection.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10781–10790, 2020.
  * [65] Siddharth Singh Chouhan, Uday Pratap Singh, and Sanjeev Jain.  Applications of computer vision in plant pathology: a survey.  Archives of computational methods in engineering, 27(2):611–632, 2020.
  * [66] Wen-Huang Cheng, Sijie Song, Chieh-Yun Chen, Shintami Chusnul Hidayati, and Jiaying Liu.  Fashion meets computer vision: A survey.  ACM Computing Surveys (CSUR), 54(4):1–41, 2021.
  * [67] Xin Feng, Youni Jiang, Xuejiao Yang, Ming Du, and Xin Li.  Computer vision algorithms and hardware implementations: A survey.  Integration, 69:309–320, 2019.
  * [68] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros.  Context encoders: Feature learning by inpainting.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2536–2544, 2016.
  * [69] Carl Edward Rasmussen et al.  The infinite gaussian mixture model.  In NIPS, volume 12, pages 554–560. Citeseer, 1999.
  * [70] Douglas A Reynolds.  Gaussian mixture models.  Encyclopedia of biometrics, 741:659–663, 2009.
  * [71] Xiaofei He, Deng Cai, Yuanlong Shao, Hujun Bao, and Jiawei Han.  Laplacian regularized gaussian mixture model for data clustering.  IEEE Transactions on Knowledge and Data Engineering, 23(9):1406–1418, 2011.
  * [72] Luca Biferale, Fabio Bonaccorso, Michele Buzzicotti, and Patricio Clark di Leoni.  TURB-Rot. A large database of 3d and 2d snapshots from turbulent rotating flows. http://smart-turb.roma2.infn.it.  arXiv:2006.07469, 2020.
  * [73] Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al.  Tensorflow: A system for large-scale machine learning.  In 12th $\\{$USENIX$\\}$ symposium on operating systems design and implementation ($\\{$OSDI$\\}$ 16), pages 265–283, 2016.
  * [74] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  Dropout: a simple way to prevent neural networks from overfitting.  The journal of machine learning research, 15(1):1929–1958, 2014\.
  * [75] Timothy Dozat.  Incorporating nesterov momentum into adam.  2016\.
  * [76] Richard C Aster, Brian Borchers, and Clifford H Thurber.  Parameter estimation and inverse problems.  Elsevier, 2018.
  * [77] Bradley Efron.  A 250-year argument: Belief, behavior, and the bootstrap.  Bulletin of the American Mathematical Society, 50(1):129–146, 2013\.
  * [78] David JC MacKay.  Bayesian interpolation.  Neural computation, 4(3):415–447, 1992.
  * [79] Martin Lellep, Jonathan Prexl, Bruno Eckhardt, and Moritz Linkmann.  Interpreted machine learning in fluid dynamics: explaining relaminarisation events in wall-bounded shear flows.  Journal of Fluid Mechanics, 942, 2022.
  * [80] Daniel D Lee and H Sebastian Seung.  Learning the parts of objects by non-negative matrix factorization.  Nature, 401(6755):788–791, 1999.
  * [81] Kieran T Bhatia, Gabriel A Vecchi, Thomas R Knutson, Hiroyuki Murakami, James Kossin, Keith W Dixon, and Carolyn E Whitlock.  Recent increases in tropical cyclone intensification rates.  Nature communications, 10(1):1–9, 2019.
  * [82] Wenwei Xu, Karthik Balaguru, Andrew August, Nicholas Lalo, Nathan Hodas, Mark DeMaria, and David Judi.  Deep learning experiments for tropical cyclone intensity forecasts.  Weather and Forecasting, 36(4):1453–1470, 2021.


# Can a Chatbot Support Exploratory Software Testing? Preliminary Results

Rubens Copche
Grupo TCM
Assis, Brazil
<EMAIL_ADDRESS>
Yohan Duarte Pessanha
Federal University of Sao Carlos (UFSCar)
São Carlos, Brazil
<EMAIL_ADDRESS>
Vinicius Durelli
Federal University of Sao Joao Del Rei (UFSJ)
Sao Joao Del Rei, Brazil
<EMAIL_ADDRESS>
Marcelo Medeiros Eler
University of Sao Paulo (USP)
Sao Paulo, Brazil
<EMAIL_ADDRESS>
Andre Takeshi Endo
Federal University of Sao Carlos (UFSCar)
Sao Carlos, Brazil
<EMAIL_ADDRESS>

###### Abstract

Tests executed by human testers are still widespread in practice and fill the
gap left by limitations of automated approaches. Among the human-centered
approaches, exploratory testing is the de facto approach in agile teams.
Although it is focused on the expertise and creativity of the tester, the
activity of exploratory testing may benefit from support provided by an
automated agent that interacts with the human testers. This paper presents a
chatbot, called BotExpTest, designed to support testers while performing
exploratory tests of software applications. We implemented BotExpTest on top
of the instant messaging social platform Discord; this version includes
functionalities to report bugs and issues, time management of test sessions,
guidelines for app testing, and presentation of exploratory testing
strategies. To assess BotExpTest, we conducted a user study with six software
engineering professionals. They carried out two sessions performing
exploratory tests along with BotExpTest. Participants were capable of
revealing bugs and found the experience to interact with the chatbot positive.
Preliminary analyses indicate that chatbot-enabled exploratory testing may be
as effective as similar approaches and help testers to uncover different bugs.
Bots are shown to be valuable resources for Software Engineering, and
initiatives like BotExpTest may help to improve the effectiveness of testing
activities like exploratory testing.

_K_ eywords software testing $\cdot$ bots $\cdot$ exploratory tests

##  1 Introduction

Due to their practical use and broad applicability, a myriad of bots that vary
in complexity have been designed, developed, and deployed in widely varying
contexts. Over the last decade, technological advancements have enabled _bots_
to play an ever increasingly important role in many areas, particularly in
software development. This emerging technology has garnered the interest of
both software development researchers and practitioners, as bots can serve as
human assistants for a variety of software development-related tasks. In this
particular context, bots that provide support for specific aspects of software
development, such as keeping project-related dependencies up-to-date, are
referred to as _devbots_ [1].

Recent developments in machine learning algorithms and natural language
processing have led to the creation of bots that provide more user-friendly
experiences. Bots that harness natural language processing capabilities to
provide more intuitive and user-friendly experiences are commonly referred to
as _chatbots_. As their name implies, chatbots are software programs designed
to replicate human-like conversations or interactions with users [2, 3].

As mentioned, bots have been utilized to support various software engineering
tasks [4, 5, 6, 7, 8]. We set out to examine how chatbots can be leveraged to
assist testers throughout the testing process. Specifically, we posit that
chatbots are well-suited for providing assistance to testers throughout the
execution of Exploratory Testing (ET) tasks. ET is an approach to software
testing that entails carrying out a series of undocumented testing sessions to
uncover faults. ET leverages the skills and creativity of testers while they
explore the system under test (SUT), and the knowledge gained during ET
sessions is then used to further refine the exploration. Hence, ET is a goal-
focused, streamlined approach to testing that allows for flexibility in test
design and keeps testers engaged throughout the testing process [9, 10, 11,
12]. Owing to these benefits, ET has been gaining traction as a complement to
fully scripted testing strategies [13]: when combined with automated testing,
ET has the potential to increase test coverage and uncover edge cases. In
fact, there is evidence suggesting that ET can be equally or even more
effective than scripted testing in practical situations [14].

In practice, before ET sessions, testers engage with other testers and
developers to gather project-related information. However, due to the
complexity of most software projects, it becomes impractical to collect all
relevant information beforehand. As a result, interruptions that arise during
ET sessions for the purpose of gathering additional information can disrupt
the flow. One potential solution to overcome this issue is to employ a chatbot
that assists testers during ET sessions, providing guidance on the selection
of input data for achieving different levels of exploration. Furthermore, the
chatbot can encourage critical thinking and enable testers to make informed
decisions. To the best of our knowledge, this research is the first foray into
the potential of a chatbot in maximizing the effectiveness of ET.

This paper introduces BotExpTest, a chatbot designed to assist testers during
ET sessions. BotExpTest was built on top of the Discord platform and includes
features tailored to managing ET sessions and reporting bugs and issues.
Additionally, it incorporates features aimed at enhancing testers’ ability to
gain further insights that can be utilized to delve further into the
exploration of the SUT.111The development and evaluation of the current
version of BotExpTest took place prior to the release of ChatGPT and other
large language models (LLMs). However, in future work, we delve into the
potential integration of these advanced technologies. To evaluate how
BotExpTest performs “in the wild”, we conducted a user study with six
practitioners. The results from the user study would seem to indicate that
BotExpTest was able to help the participants to uncover several bugs.
Moreover, the participants expressed a positive opinion about the experience
and held an optimistic view regarding the potential future adoption of the
tool.

##  2 Related Work

_Chatbots_ are becoming increasingly popular in the software development
domain because they can be very versatile. In this context, _bots_ are
frequently classified based on their capacity of supporting different
activities such as code review, tests, bug fixing, verification and deployment
[4]. Storey et al. [15] surveyed developers and researchers to identify in
which situations they use _bots_ to support software engineering activities.
Here is what they found: to search and to share information, to extract and to
analyze data, to detect and to monitor events, to communicate in social media,
to connect stakeholders and developers, to provide feedback, and to recommend
individual or collaborative tasks associated with software development.

Many studies have proposed _bots_ to support software development activities.
_Performobot_ is a chatbot-based application that helps in planning, executing
and reporting the results of tasks related to load and performance testing
[8]. _Smart Advisor_ is an intelligence augmentation _bot_ that helps
developers with project specifics by employing domain and knowledge modeling
and in-process analytics to automatically provide important insights and
answer queries using a conversational and interactive user interface [6].
_Repairnator_ is a program repair _bot_ that creates software patches and
provides an explanation for each bug fixed using natural language as a human
collaborator would do [16]. _Tutorbot_ uses machine learning to retrieve
relevant content, guiding software engineers in their learning journey and
helping them keep pace with technology changes [17].

To the best of our knowledge, there is no specific chatbot devised to help
testers to conduct exploratory testing. In fact, a study conducted in Estonia
and Finland found out that only 25% of the software testing professionals
apply ET with some tool support. Mind mapping tools are the most frequently
used software, but testers also use text editors, spreadsheets and even actual
pen and paper, in addition to checklists and paper notes (e.g. post-its) [18].
In this context, Copche et al. [19] introduced a specific kind of mind map
called opportunity map (OM) as a way to improve the ET of mobile apps. The
authors conducted a study that compares OM-based ET with a traditional
session-based approach (baseline).

There are some tools to directly support activities involved in ET. For
instance, Leveau et al. [20] designed a tool called _Test Tracker_ to prevent
testers from running tests that have already been executed so they can run
more diversified test sessions to further explore the SUT.

There have been some attempts to integrate ET with automated approaches. For
instance, Shah et al. [21] proposed an hybrid approach that combines the
strengths of ET and scripted testing (ST). Broadly, they identified the
weaknesses of ST and proposed to use the strengths of ET as a solution.
Similarly, they identified the weaknesses of ET and proposed to use the
strengths of ST as a solution. When it comes to ST, for example, test case
design quality depends on the test designer skills. Considering the strengths
of ET, which includes the application of domain knowledge and the observation
of the system behavior for rapid feedback, the hybrid process allows the
testers to explore the SUT freely and to utilize their intuitions and
experience in identifying defects before writing test scripts.

##  3 BotExpTest

This section presents the design and main features of a chatbot we developed
to support ET. We set out by investigating existing work about ET, its core
practices and envisioned how a chatbot would help the tester to conduct more
effective ET sessions. To validate these ideas, we implemented BotExpTest.
Figure 1 shows the example of an interaction with BotExpTest: tester Beth
types ?commands and then BotExpTest shows all commands accepted.

Figure 1: BotExpTest replies its main commands.

###  3.1 Implementation

As instant messaging platforms are widely adopted and are today an essential
part of software projects, we opted to develop BotExpTest on top of them. For
this first release, we settled on using the Discord
platform222https://discord.com. Discord provides an open source platform with
highly configurable features for users and bots. BotExpTest is implemented as
a Node.js project; it has 22 classes and around 1.3K lines of JavaScript code.
It takes advantage of the Discord API to capture interactions from testers in
the chat, as well as to generate its own messages. To make the ET process
auditable, all messages exchanged between the chatbot and the testers are
recorded in a MongoDB database. BotExpTest is available as an open source
project at:

https://github.com/rcopche/BotExpTest

Figure 2 presents an overview of the BotExpTest architecture. The interaction
starts with the tester writing a message (command) to BotExpTest via Discord
(step 1). The message passes through the Discord Developer Portal, which is
then accessible by means of an API (steps 2-3). Up to this point, BotExpTest
interprets the message typed by the tester and reacts by sending a reply
(steps 4-5). Finally, the BotExpTest’s response is shown to the tester and new
interactions may occur. BotExpTest may also be the one that starts the
interaction.

Figure 2: Architecture of BotExpTest.

###  3.2 Main Features

During the first interaction between tester and BotExpTest, the first reaction
of the chatbot is to present itself, giving some pieces of information about
how to perform the next steps in the ET session. As a convention, testers
begin an interaction with BotExpTest using a message that starts with ’?’.
Adhering to this convention can be beneficial in scenarios where several
testers are communicating with each other in a chat, as it indicates when
testers intend to engage with the bot. For this version, the features
implemented by BotExpTest were elicited, prioritized and implemented; they are
described next.

Description of the test procedure: Using command ?manual, BotExpTest shows a
step-by-step description about how the test sessions are organized and should
be conducted, as well as the main features provided to tester by the chatbot.
This is illustrated in Figure 3.

Figure 3: BotExpTest replies to ?manual.

Charters: In ET, charters are used to organize the tests and represent the
goals that are supposed to be achieved in a test session. BotExpTest provides
an interface to set up the test charters and are available to testers by using
the message ?charter. Besides the charter name, app name, and the goals
description, it is also possible to attach images and other files related to
the charter.

Time management of testing sessions: In ET, testing sessions are conducted
within a limited time frame; usually, testers need to keep track of time. As
the tester is constantly interacting in the chat, BotExpTest reminds her about
the remaining time in the session, from time to time. To signal the start of a
session, the tester should type the command ?start. BotExpTest then asks the
time limit and starts to monitor the time elapsed in the session. During the
time range of the session, BotExpTest keeps track of all interactions that
occurred. Figure 4 shows how the time alerts are presented to the tester.

Figure 4: Time alerts and suggestions.

Bug and issue reporting: The identification of bugs and issues are the main
outcomes of a test session. To avoid using other tools for this task,
BotExpTest registers occurrences of bugs or issues. To do so, the tester types
command ?report; this task is exemplified in Figure 5. The tester interacts
with BotExpTest so that the charter, type (bug or issue), a detailed
description and potential attachments (e.g., screenshots of the bug) are
provided. The current version only stores the bug report, but it is possible
in future to integrate it with external tools like GitHub, Jira or Azure
DevOps.

Figure 5: Reporting a bug.

Curated knowledge about exploratory testing: The idea of this feature is for
BotExpTest to have a curated list of resources about the use of ET techniques.
In future, BotExpTest could be fine-tuned for specific projects so that
testers are better equipped to conduct exploratory tests. Figure 6 shows the
available resources after the tester types command ?help. By typing one of the
presented options, BotExpTest shows a detailed explanation about the concept
and how to apply it during the tests. For some options, the chatbot replies
with questions. We anticipate that this feature may help the testers to gain
more insights and execute more effective tests.

Currently, BotExpTest provides resources related to three main groups. Group
(i) brings well-known testing criteria that may help the tester to design
black-box tests. For example, classical criteria like equivalence partition
and boundary-value analysis are presented. Group (ii) is composed of
strategies for ET that are well-known and have been adopted in the literature
[22, 23]. For example, Bad Neighborhood Tour instructs the tester to revisit
buggy parts of software since bugs tend to cluster together. Finally, Group
(iii) contains guidelines for mobile app testing. For example, there are
several test scenarios related to specific characteristics of mobile apps like
network connections, geolocation, Bluetooth, camera, and UI events (scrolling,
swipe, etc).

Figure 6: Command ?help.

Active suggestions: During a test session, BotExpTest can actively start an
interaction with the tester. The chatbot can present some piece of information
obtained from the curated knowledge about exploratory testing. We believe that
actively interacting with the tester would increase the engagement with the
tests. For the current version, the decision about the timing and the
information provided is made randomly. In future, we expect that BotExpTest
could be evolved to make a more informed decision about the interaction needed
at a specific point of time.

##  4 User Study

This section describes an empirical study conducted with the aim to provide an
initial evaluation of BotExpTest. To this end, we posed the following research
questions (RQs):

  * •

RQ1: How is the interaction with BotExpTest during the exploratory testing?

  * •

RQ2: How do the participants perform with respect to the detection of bugs?

  * •

RQ3: How do the participants perceive BotExpTest?

To answer these questions, we conducted a user study with six participants
that were asked to use BotExpTest to support the ET of a mobile app named
Reminders333https://play.google.com/store/apps/details?id=com.chegal.alarm.
All participants work in the industry and have experience with software
development and testing. We adopted the app and related charters that were
openly available from Copche et al. [19]; the rationale here is to make some
analyses concerning similar approaches444We used the same app version,
charters and length of test sessions.. To collect the needed data and observe
the participants, we set up a computer with screen recording, and mirroring
the mobile device running the app under test. Each participant was invited to
use this computer in order to perform the tasks of the study. Figure 7
illustrates the testing environment used by the participants; the reminders
app is shown on the left, while the Discord UI (along with BotExpTest) is
presented on the right.

Figure 7: Setup used by the participants.

Initially, we provided the participants with detailed instructions about how
to perform the testing tasks. They were instructed to strictly follow the
charter, report any bug or issue identified, and the time management of the
session was supported by BotExpTest. All test sessions were recorded and the
participants could think aloud about the tasks being carried out. After the
introduction, the study was divided into two parts:

  * •

Training: this session lasted approximately 15 minutes and served as an
introduction to the usage of the chatbot. This allowed participants to become
acquainted with BotExpTest and Discord, allowing them to experiment with
possible interactions and explore the supported commands.

  * •

Test sessions: this iteration took approximately half an hour, in which two
test sessions with 15 minutes each occurred. Over the course of these two test
sessions, the chatbot-enabled ET took place.

All data was then retrieved and analyzed. To answer RQ1, we looked at the
interactions that occurred (messages exchanged) between the participant and
BotExpTest. We called Active Interactions the ones started by BotExpTest,
while Reactive Interactions are responses to the tester’s inquiries.

As for RQ2, we analyzed and cataloged the bugs reported by participants. In
particular, we cross-checked the bugs herein reported with the ones uncovered
in Copche et al. [19]. As an initial analysis, we intended to assess whether a
chatbot-enabled ET can detect a different set of bugs with respect to similar
approaches like baseline and OM (see Section 2).

We answered RQ3 with a Likert-scale survey that intends to understand the
perception of participants. The questions are divided into (i) how easy is to
interact and use BotExpTest, (ii) whether the user interface is adequate for
the proposed functionalities, and (iii) understanding the participants’
perception about the effectiveness of chatbot-enabled ET. There is also an
open question for comments and suggestions.

###  4.1 Analysis of Results

#### RQ1 - Interactions with BotExpTest.

Table 1 shows the number of active and reactive interactions of both the
chatbot and the participants; the values are also divided between training and
test sessions. Overall, BotExpTest produced 581 interactions (121 in training
and 460 in the test sessions), while the six participants had 496 interactions
(144 in training and 352 in the test sessions).

Table 1: Number of interactions (int.). TRAINING |  | TEST SESSIONS
---|---|---
BotExpTest |  | BotExpTest |
Reactive int. | 107 |  | Reactive int. | 340
Active int. | 14 |  | Active int. | 120
Total | 121 |  | Total | 460
Participants |  | Participants
Reactive int. | 37 |  | Reactive int. | 262
Active int. (accepted) | 100 |  | Active int. (accepted) | 90
Active int. (invalid) | 7 |  | Active int. (invalid) | 0
Total | 144 |  | Total | 352

During training, BotExpTest was proportionally more reactive (reactive/active
– 107/14) and the participants were more active (37/100); they also typed 7
invalid commands. The most typed commands were related to the chatbot usage
(?charter, ?commands, ?help, ?manual) and software testing resources. We
observed that the participants were more active in the interactions because
they were focused on figuring out how to use BotExpTest.

As for test sessions, BotExpTest was still more reactive (340/120) but
proportionally had more active interactions. This occurred due to the
reporting of bugs and issues (it asks actively for more pieces of
information). This fact also impacted the participants’ interactions: they
were more reactive (262/90) and did not type any invalid command. The most
typed commands were related to bug and issue reporting (e.g., ?report) and
management of the test sessions (e.g., ?start, ?charter). We also observed
that participants were focused on testing the app and this also limited the
interactions with the chatbot.

Answer to RQ1: We observed reasonable interactions between the participants
and BotExpTest. As the participants were exploring the chatbot in training,
they had more active interactions. On the other hand, the interactions were
more reactive and limited in the test sessions due to the time spent with
exploratory testing of the app itself and reporting bugs and issues.

#### RQ2 - Bugs.

The six participants reported 31 bugs. The most effective participant
uncovered nine bugs, while one of them reported three bugs. On average, the
participants detected 5.2 bugs (median: 5). The distribution of bugs detected
per participant can be seen in Figure 8, the boxplot/violin in the middle.

Figure 8 also shows the results for the baseline and OM approaches [19].
Observe that the average and median values of BotExpTest are slightly greater
than baseline (avg: 2.7, median: 3) and OM (avg: 4.3, median: 4). The range of
values is also smaller, so BotExpTest produced a more uniform performance of
participants. Due to the differences of samples and participants’ experience,
these results may not be generalized. One may argue that the bug detection
capability of BotExpTest is at least comparable to approaches without any
support (i.e., baseline) or that adopted some supporting artifact (i.e., OM).

Figure 8: Number of bugs detected by the participants.

Figure 9 shows a Venn diagram with the unique bugs detected by the approaches,
numbers between parentheses are bugs tracked to specific suggestions of the
approach. Out of 31 bugs reported by the participants using BotExpTest, 21
were unique since some reported the same bug. Eight bugs have been uncovered
in the Copche et al. study (1 by baseline, 3 by OM and 4 by both), but 13 yet-
unknown bugs were detected in this study. We were able to map three out of
these 13 bugs to specific insights provided by the chatbot.

Figure 9: Unique bugs.

Answer to RQ2: The participants were capable of uncovering an average of 5.2
bugs using BotExpTest. Their performance is comparable to similar approaches
evaluated in the literature. Furthermore, BotExpTest supported the
participants in detecting 13 previously unknown bugs.

#### RQ3 - Participants’ perception.

For the part (i) of questions, we asked about the easiness of finding
information, whether proper instructions are provided, and usability in
general. All participants agreed or strongly agreed that BotExpTest is easy to
use and interact. As for part (ii), the questions asked about specific
features like bug/issue reporting and time management of test sessions. Most
participants strongly agreed or agreed that the user interface for those
features is adequate. In particular, the bug/issue reporting was unanimously
well-evaluated (all strongly agreed).

For the last part of questions, the responses indicated that participants
would use a chatbot in similar tasks, and they perceived more organized test
sessions. They also thought that BotExpTest helped them to find more bugs. We
also asked if the chatbot helped them to understand new concepts of software
testing, and whether the suggestions made by BotExpTest were helpful; all
participants strongly agreed or agreed with those statements.

From the open question and our observations, we draw the following thoughts.
Participants believed that the chatbot helped to shorten the time spent with
process tasks, saving more time to test the app. BotExpTest worked as a rich
and centralized source of testing information; participants sometimes used the
message history to revisit decisions and bugs detected. One suggested that it
could support novice programmers testing their software, and another mentioned
that it could help teams without QAs. Finally, there were suggestions to add
support for other testing tasks, like managing test scripts, tracking the
status of test executions, and communication with stakeholders.

Answer to RQ3: The participants perceived BotExpTest as a valuable resource
while performing exploratory testing. Overall, the participants’ perceptions
were positive concerning the features, the ease of interaction, and testing
resources.

##  5 Concluding Remarks

This paper presents an initial effort on using chatbots to support exploratory
software testing. We implemented the first version of BotExpTest and evaluated
it with six software development professionals. The results gave evidence that
chatbot-enabled ET has potential to be as effective as similar approaches and
received positive feedback from the participants.

We recognize the limitations of this study, yet we intend to draw two main
future initiatives from the preliminary findings obtained. First, future
replications and extended controlled experiments are needed to better assess
the impact of chatbots in software testing. Then, BotExpTest could be evolved
with modern technologies so that it improves its testing support and
interaction capabilities. One direction is to include the ability of observing
the SUT (using e.g. monitoring or dynamic analyses as in [20]). This ability
would be used to feed the chatbot and provide more educated insights. On the
human-bot interaction side, LLMs (like ChatGPT, Bard) and other similar
technologies could be adopted to make the conversations more human-like, and
provide a broader (and personalized) access to software testing knowledge.

## Acknowledgments

Andre T. Endo is partially supported by grant #2023/00577-8, São Paulo
Research Foundation (FAPESP). Yohan Pessanha is supported by grant
#2022/13469-6, São Paulo Research Foundation (FAPESP) and Coordenação de
Aperfeiçoamento de Pessoal de Nível Superior (CAPES) grant
88887.801592/2023-00.

## References

  * [1] L. Erlenhov, F. G. de Oliveira Neto, R. Scandariato, and P. Leitner.  Current and future bots in software development.  In IEEE/ACM 1st International Workshop on Bots in Software Engineering (BotSE), pages 7–11. IEEE, 2019.
  * [2] W. Zadrozny, M. Budzikowska, J. Chai, N. Kambhatla, S. Levesque, and N. Nicolov.  Natural language dialogue for personalized interaction.  Communications of the ACM, 43(8):116–120, 2000.
  * [3] B. A. Shawar and E. Atwell.  Chatbots: are they really useful?  In Ldv forum, volume 22, pages 29–49, 2007.
  * [4] M. A. Storey and A. Zagalsky.  Disrupting developer productivity one bot at a time.  In The 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, pages 928–931, 2016.
  * [5] E. Paikari, J. Choi, S. Kim, S. Baek, M. Kim, S. Lee, C. Han, Y. Kim, K. Ahn, C. Cheong, et al.  A chatbot for conflict detection and resolution.  In 1st International Workshop on Bots in Software Engineering (BotSE), pages 29–33, 2019.
  * [6] V. S. Sharma, R. Mehra, V. Kaulgud, and S. Podder.  A smart advisor for software delivery-a bot framework for awareness, alerts and advice.  In The 1st International Workshop on Bots in Software Engineering (BotSE), pages 22–23, 2019.
  * [7] L. Erlenhov, F. G. De Oliveira Neto, and P. Leitner.  An empirical study of bots in software development: Characteristics and challenges from a practitioner’s perspective.  In Proceedings of the 28th ACM Symposium on the Foundations of Software Engineering, pages 445–455, 2020.
  * [8] D. Okanović, S. Beck, L. Merz, C. Zorn, L. Merino, A. van Hoorn, and F. Beck.  Can a chatbot support software engineers with load testing? approach and experiences.  In The ACM/SPEC International Conference on Performance Engineering, pages 120–129, 2020.
  * [9] J. Bach.  Exploratory testing explained, 2003.
  * [10] J. Lyndsay and N. Van Eeden.  Adventures in session-based testing.  Workroom Productions Ltd. May, 27, 2003.
  * [11] C. Kaner, H. Q. Nguyen, and J. L. Falk.  Testing computer software, 1993.
  * [12] Mariana Souza, Isabel K Villanes, Arilo Claudio Dias-Neto, and Andre T Endo.  On the exploratory testing of mobile apps.  In Proceedings of the IV Brazilian Symposium on Systematic and Automated Software Testing, pages 42–51, 2019.
  * [13] ISTQB.  Worldwide Software Testing Practices Report Message from the President Executive Summary Survey Questions and Analysis. 2018\.
  * [14] A. N. Ghazi, R. P. Garigapati, and K. Petersen.  Checklists to support test charter design in exploratory testing.  In International Conference on Agile Software Development, pages 251–258. Springer, 2017.
  * [15] M. A. Storey, A. Serebrenik, C. P. Rosé, T. Zimmermann, and J. D Herbsleb.  Botse: Bots in software engineering (dagstuhl seminar 19471).  In Dagstuhl Reports, volume 9. Schloss Dagstuhl-Leibniz-Zentrum für Informatik, 2020.
  * [16] M. Monperrus.  Explainable software bot contributions: Case study of automated bug fixes.  In 2019 IEEE/ACM 1st International Workshop on Bots in Software Engineering (BotSE), pages 12–15. IEEE, 2019.
  * [17] V. Subramanian, N. Ramachandra, and N. Dubash.  Tutorbot: contextual learning guide for software engineers.  In 2019 IEEE/ACM 1st International Workshop on Bots in Software Engineering (BotSE), pages 16–17, 2019.
  * [18] D. Pfahl, H. Yin, M. V Mäntylä, and J. Münch.  How is exploratory testing used? a state-of-the-practice survey.  In The 8th ACM/IEEE international symposium on empirical software engineering and measurement, pages 1–10, 2014.
  * [19] R. Copche, M. Souza, I. K. Villanes, V. Durelli, M. Eler, A. Dias-Neto, and A. T. Endo.  Exploratory testing of apps with opportunity maps.  In XX Brazilian Symposium on Software Quality, pages 1–10, 2021\.
  * [20] J. Leveau, X. Blanc, L. Réveillère, J. Falleri, and R. Rouvoy.  Fostering the diversity of exploratory testing in web applications.  In IEEE 13th International Conference on Software Testing, Validation and Verification (ICST), pages 164–174, 2020.
  * [21] S. M. A. Shah, C. Gencel, U. S. Alvi, and K. Petersen.  Towards a hybrid testing process unifying exploratory testing and scripted testing.  Journal of software: Evolution and Process, 26(2):220–250, 2014\.
  * [22] M. Micallef, C. Porter, and A. Borg.  Do exploratory testers need formal training? an investigation using HCI techniques.  In 9th International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pages 305–314, 2016.
  * [23] J. A. Whittaker.  Exploratory software testing: tips, tricks, tours, and techniques to guide test design.  Pearson Education, 2009.