# Chemistry and Radiative Transfer of Water in Cold, Dense Clouds

Eric Keto1, Jonathan Rawlings2, and Paola Caselli3
1Harvard-Smithsonian Center for Astrophysics, 160 Garden St, Cambridge, MA
02420, USA
2University College London, London, UK
3School of Physics and Astronomy, University of Leeds, Leeds LS2 9JT, UK
E-mail<EMAIL_ADDRESS>(EK<EMAIL_ADDRESS>(JR)
<EMAIL_ADDRESS>(PC)

(February 19, 2014)

###### Abstract

The Herschel Space Observatory’s recent detections of water vapor in the cold,
dense cloud L1544 allow a direct comparison between observations and chemical
models for oxygen species in conditions just before star formation. We explain
a chemical model for gas phase water, simplified for the limited number of
reactions or processes that are active in extreme cold ($<$ 15 K). In this
model, water is removed from the gas phase by freezing onto grains and by
photodissociation. Water is formed as ice on the surface of dust grains from O
and OH and released into the gas phase by photodesorption. The reactions are
fast enough with respect to the slow dynamical evolution of L1544 that the gas
phase water is in equilibrium for the local conditions thoughout the cloud. We
explain the paradoxical radiative transfer of the H2O ($1_{10}-1_{01}$) line.
Despite discouragingly high optical depth caused by the large Einstein A
coefficient, the subcritical excitation in the cold, rarefied H2 causes the
line brightness to scale linearly with column density. Thus the water line can
provide information on the chemical and dynamical processes in the darkest
region in the center of a cold, dense cloud. The inverse P-Cygni profile of
the observed water line generally indicates a contracting cloud. This profile
is reproduced with a dynamical model of slow contraction from unstable quasi-
static hydrodynamic equilibrium (an unstable Bonnor-Ebert sphere).

###### keywords:

Interstellar Medium (ISM), Nebulae: ISM Interstellar Medium (ISM), Nebulae:
abundances; Interstellar Medium (ISM), Nebulae, ISM: individual, L1544;
Interstellar Medium (ISM), Nebulae: molecules; Physical Data and Processes,
astrochemistry; Physical Data and Processes, radiative transfer;

##  1 Introduction

Observations of water vapor in the interstellar medium (ISM) by the Infrared
Space Observatory (van Dishoeck et al., 1999) and the Submillimeter Wave
Astronomy Satellite (SWAS) (Bergin et al., 2000) show general agreement with
chemical models for warm ($>300$ K) conditions in the ISM (Melnick et al.,
2000; Neufeld et al., 2000). However, in cold conditions, most of the water is
frozen onto dust grains (Viti et al., 2001; van Dishoeck, Herbst & Neufeld,
2013), and the production of water occurs mainly on the grain surfaces. In
order to test chemical models that include grain-surface chemistry we used the
Heterodyne Instrument for the Far-Infrared (HIFI) (de Graauw et al., 2010) on
the Herschel Space Observatory to observe the H2O ($1_{10}-1_{01}$) line in
the cold, dense cloud L1544 (Caselli et al., 2010, 2012). The first of these
two Herschel observations was made with the wide-band spectrometer (WBS) and
detected water vapor in absorption against the weak continuum radiation of
dust in the cloud. Follow-up observations with higher spectral resolution and
sensitivity, made with the high resolution spectrometer (HRS), confirmed the
absorption and detected a blue-shifted emission line that was predicted by
theoretical modeling (Caselli et al., 2010), but too narrow to be seen by the
WBS in the first observation.

With the better constraints provided by the second observation, we improved
the chemical and radiative transfer modeling in our previous papers. We
modified the radiative transfer code MOLLIE to calculate the line emission in
the approximation that the molecule is sub-critically excited. This assumes
that the collision rate is so slow that every excitation leads immediately to
a radiative de-excitation and the production of one photon which escapes the
cloud, possibly after many absorptions and re-emissions, before another
excitation. The emission behaves as if the line were optically thin with the
line brightness proportional to the column density. This approximation can be
correct even at very high optical depth as long as the excitation rate is slow
enough, C $<$ A/$\tau$, where C is the collision rate, A is the spontaneous
emission rate and $\tau$ the optical depth (Linke et al., 1977). Caselli et
al. (2012) presented the observations and the results of this modeling.

In this paper, we discuss in detail the theory behind the modeling. A
comparison of the spectral line observation with theory requires three models.
First, we require a hydrodynamical model to describe the density, velocity,
and temperature across the cloud. We use a model of slow contraction in quasi-
static unstable equilibrium that we developed in our previous research (Keto &
Field, 2005; Keto & Caselli, 2010). Second, we require a chemical model to
predict the molecular abundance across the varying conditions in the cloud.
Following the philosophy for simplified chemical networks in Keto & Caselli
(2008) or Bethell & Bergin (2009), we extract from a general chemical model
for photo-dissociation regions (Hollenbach et al., 2009) a subset of reactions
expected to be active in cold conditions, principally grain-surface reactions
as well as freeze-out and photodissociation. Third, we require a radiative
transfer model to generate a simulated molecular line. We modify our non-LTE
radiative transfer code MOLLIE to use the escape probability approximation.
This allows better control of the solution in extreme optical depth.

The three models are described in more detail in three sections below. The
relevant equations are included in the appendices.

##  2 The three models

###  2.1 The cold, dense clouds

Given their importance as the nurseries of star formation, the small ($<0.5$
pc), cold ($<15$ K), dense ($n>10^{3}$ cm-3) clouds in low-mass star ($<2$ M⊙)
forming regions such as Taurus are widely studied (Bergin & Tafalla, 2007; di
Francesco et al., 2007). Observations show a unique simplicity. They contain
no internal sources of heat, stars or protostars. Their internal turbulence is
subsonic, barely broadening their molecular line widths above thermal (Myers &
Benson, 1983). With most of their internal energy in simple thermal energy,
and the weak turbulence just a perturbation (Keto et al., 2006; Broderick &
Keto, 2010), the observed density structure approximates the solution of the
Lane-Emden equation for hydrostatic equilibrium (Lada et al., 2003; Kandori et
al., 2005). Correspondingly, most are nearly spherical with an average aspect
ratio of about 1.5 (Jijina, Myers & Adams, 1999). They are heated from the
outside both by cosmic rays and by the UV background of starlight and are
cooled from the inside by long wavelength molecular line and dust continuum
radiation (Evans et al., 2001). Because of their simplicity, we understand the
structure and dynamics of these small, cold, dense clouds better than any
other molecular clouds in the interstellar medium. They are therefore uniquely
useful as a laboratory for testing hypotheses of more complex phenomena such
as the chemistry of molecular gas.

###  2.2 Structure and dynamics

Our physical model for cold, dense clouds is computed with a spherical
Lagrangian hydrodynamic code with the gas temperature set by radiative
equilibrium between heating by external starlight and cosmic rays and cooling
by molecular line and dust radiation. The theory is discussed in Keto & Field
(2005) and Keto & Caselli (2008).

In our previous research (Keto & Caselli, 2010), we generated a dynamical
model for the particular case of L1544 by comparing observations and snapshots
in time out of a theoretical model for the contraction toward star formation.
We began the hydrodynamic evolution with a 10 M⊙ Bonnor-Ebert (BE) sphere with
a central density of $10^{4}$ cm-3 in unstable dynamical equilibrium and in
radiative equilibrium with an external UV field of one Habing flux. In the
early stages of contraction, the cloud evolves most rapidly in the center. As
long as the velocities remain subsonic, the evolving density profile closely
follows a sequence of spherical equilibria or BE spheres with increasing
central densities. We compared modeled CO and N2H+ spectra during the
contraction against those observed in L1544 and determined that the stage of
contraction that best matches the data has a central density of $1\times
10^{7}$ cm-3and a maximum inward velocity just about the sound speed (Keto &
Caselli, 2010). Figure 1 shows the density and velocity at this time along
with the H2O abundance and temperature.

In the present investigation we modify our numerical hydrodynamic code to
include cooling by atomic oxygen. This improves the accuracy of the calculated
gas temperature in the photodissociation region outside the molecular cloud.
The equations governing the cooling by the fine structure lines of atomic
oxygen are presented in the appendix.

###  2.3 Chemistry of H2O in cold conditions

The cold conditions in L1544 allow us to simplify the chemical model for gas
phase water. We include the four oxygen-bearing species most abundant in cold,
dark clouds, O, OH, H2O gas, and H2O ice. Even though all three gas phase
molecules may freeze onto the grains, we consider only one species of ice
because the formation of water from OH and the formation of OH from O are
rapid enough on the grain surface that most of the ice is in the form of H2O.
To provide a back reaction for the freeze-out of atomic oxygen and preserve
detailed balance, we arbitrarily assign a desorption rate for atomic O equal
to that of H2O even though the production of atomic O from H2O ice is not
indicated. Our simplified model is shown in figure 2. The resulting
abundances, calculated as equilibria between creation and destruction, are
shown in figures 3 and 4. Figure 3 shows the abundances near the
photodissociation region (PDR) boundary as a function of the visual
extinction, $A_{V}$. Figure 4 shows the abundances against the log of the
radius to emphasize the center.

Gas phase water is created by UV photodesorption of water ice which also
creates gas phase OH in a ratio H2O/OH = 2 (Hollenbach et al., 2009). In the
outer part of the cloud, the UV radiation derives from the background field of
external starlight. The inward attenuation of the UV flux is modeled from the
visual extinction as $\exp{(-1.8{\rm A_{V}})}$. In the interior where all the
external UV radiation has been attenuated, the only UV radiation is generated
by cosmic ray strikes on H2. In our previous paper (Caselli et al., 2012), we
set this secondary UV radiation to $1\times 10^{-3}$ times the Habing flux
(G0=1) (Hollenbach et al., 2009). In our current model, we use a lower level,
$1\times 10^{-4}$, that is more consistent with estimated rates (Shen et al.,
2004). The difference in abundance for the two rates is shown in figure 4.

H2O and OH are removed from the gas phase by UV photodissociation and by
freezing onto dust grains. To preserve detailed balance with the
photodissociation we include the back reactions, the gas phase production of
H2O, O + H2 $\rightarrow$ OH and OH + H2 $\rightarrow$ H2O, even though these
are not expected to be important in cold gas. Removal of gas phase water by
freeze-out is important in the interior where the higher gas density increases
the dust-gas collision rate, and hence the freeze-out rate.

We assume that the gas-phase ion-neutral reactions that lead to the production
of water are less important at cold temperatures ($<15$ K) than the reactions
that produce water on the surfaces of ice-coated dust grains. Thus, we do not
include gas-phase ion-neutral reactions in the model. This is valid if the
oxygen is quickly removed from the gas-phase by freeze-out and efficiently
converted into water ice on the grain surface.

By leaving out CO, we avoid coupling in the carbon chemistry. Although we
already have a simple model for the carbon chemistry (Keto & Caselli, 2008,
2010), we prefer to keep our oxygen model as simple as possible. This could
create an error of a factor of a few in the abundance of the oxygen species.
Carbon is one-third as abundant as oxygen, and in certain conditions CO is the
dominant carbon molecule. Therefore as much as one-third of the oxygen could
potentially be bound in CO. Ignoring O2 is less of a problem. Created
primarily by the reaction of OH with atomic oxygen, O2 tends to closely follow
the abundance of OH. Since the amount of oxygen in OH should be 1% or less
(figure 3), the abundance of O2 does not affect the abundances of the other
oxygen species, O, OH, and H2O.

Figures 3 and 4 compare the abundances from our simplified network with those
from the more complex network of Hollenbach et al. (2009) (courtesy of E.
Bergin) that includes gas-phase neutral-neutral and ion-neutral reactions. In
this calculation, we hold the cloud at the same time in its dynamical
evolution and allow the chemistry to evolve for 10 Myr from the assumed
starting conditions in which all species are atomic and neutral. Both models
generally agree. The gas-phase water peaks in a region near the boundary. Here
there is enough external UV to rapidly desorb the water from the ice, but not
so much as to dissociate all the molecules. Further inward, the abundance of
water falls as the gas density and the dust-gas collision rate (freeze-out
rate) both increase while the photodesorption rate decreases with the
attenuation of the UV radiation. At high Av, the water is desorbed only by
cosmic rays and the UV radiation they produce in collisions with H2. The
general agreement between the two models suggests that the simple model
includes the processes that are significant in the cold environment. The rate
equations for the processes selected for the simplified model are listed in
the appendix (§B).

Our simple model calculates equilibrium abundances. We can estimate the
equilibrium time scale from the combined rates for creation and destruction
(Caselli et al., 2002),

$t=\frac{t_{creation}t_{destruction}}{t_{creation}+t_{destruction}}$ (1)

where the time scales are the inverses of the rates. Figure 5 shows the
equilibrium time scales for each species as a function of radius. These may be
compared with the time for the hydrodynamic evolution. A cloud with a mass of
10 M⊙ and a central density of $2\times 10^{6}$ cm-3 has a free-fall time,
$t_{ff}=0.03$ Myr using the central density in the standard equation whereas
the sound crossing time is about 2 Myr (Keto & Caselli, 2010). Because the
chemical time scales are all shorter than the dynamical time scales the
chemistry reaches equilibrium before the conditions, density, temperature, and
UV flux change.

In this estimate of the time scale for chemical evolution, we are asking
whether the oxygen chemistry in the contracting molecular cloud can maintain
equilibrium as the cloud evolves dynamically. This is different from the
question of how long it would take for the chemistry to equilibrate if the gas
were held at molecular conditions but evolving from an atomic state.

$\begin{array}[]{cc}\includegraphics[width=234.87749pt]{StructureLVG}\end{array}$

Figure 1:  Model of a slowly contracting cloud in quasi-static unstable
equilibrium. The log of the density profile in cm-3 is shown in blue (dotted
line), the fractional abundance of H2O with respect to H2 is shown in green
(dashed line), the velocity as the black (solid) line, and the gas temperature
as the red (dot-dashed) line. The model spectrum is shown in figure 8.  Figure
2: Simplified model of the oxygen chemistry in a cold cloud. The model
includes 3 gas-phase species and H2O ice. The significant reactions at cold
temperatures (T $<300$ K) are the freeze-out of molecules colliding with dust
grains, cosmic ray and photodesorption of the ice, and photodissociation of
the gas phase molecules. Figure 3: Abundances of oxygen species as a function
of $A_{V}$ for the model of L1544 based on a slowly contracting Bonnor-Ebert
sphere. The figure emphasizes the variation of abundances in the PDR at the
edge. The figure compares the abundances for the physical conditions in
(figure 1) from two models: Hollenbach et al. (2009) (dashed lines) (courtesy
E. Bergin); and our simplified model (figure 2). Figure 4 shows abundances
from the same models but plotted against log radius to emphasize the
variations in the center.  Figure 4: Abundances of oxygen species, same as
figure 3, except plotted against the log of the radius rather than visual
extinction. This figure emphasizes the variations in abundance in the center.
The figure shows the H2O abundance calculated with our simplified model using
two values for the cosmic ray-induced UV photodesorption (equation 19). The
solid green line shows the abundance calculated with factor $\alpha=10^{-4}$.
The dotted line shows the abundance calculated with factor $\alpha=10^{-3}$.
The abundance calculated with the Hollenbach et al. (2009) model assumes
$\alpha=10^{-3}$ (dashed line).  Figure 5:  Time scales for chemical
equilibrium. From top to bottom, the three lines show the equilibration time
scales for H2O, OH, and O calculated from equation 1 and the reaction rates in
the appendix.

###  2.4 Radiative Transfer

We use our radiative transfer code MOLLIE (Keto, 1990; Keto & Rybicki, 2010)
to compute model H2O spectra to compare with the Herschel observation. Here we
encounter an interesting question. The large Einstein A coefficient of the H2O
($1_{10}-1_{01}$) line results in optical depths across the cloud of several
hundred to a thousand depending on excitation. High optical depths generally
result in radiative trapping and enhanced excitation of the line. In this
case, the line brightness could have a non-linear relationship to the column
density. For example, the line could be saturated. On the other hand, the
large Einstein A means that the critical density for collisional de-excitation
is quite high ($1\times 10^{8}$ cm-3) at the temperatures $<15$ K, higher than
the maximum density ($1\times 10^{7}$ cm-3) in our dynamical model of L1544.
This suggests that the line emission should be proportional to the column
density.

This question was addressed by Linke et al. (1977) who proposed a solution
using the escape probability approximation (Kalkofen, 1984). They assumed a
two level molecule, equal statistical weights in both levels, and the mean
radiation field, $\bar{J}$, set by the escape probability, $\beta$,

$\bar{J}=J_{0}\beta+(1-\beta)S$ (2)

where $J_{0}$ is the continuum from dust and the cosmic microwave background,
$S$ is the line source function, and

$\beta=(1-\exp{(-\tau)})/\tau.$ (3)

After a satisfying bout with three pages of elementary algebra and some
further minor approximations, they show that as long as $C<A/\tau$, the line
brightness is linearly dependent on the column density, no matter whether the
optical depth is low or high, provided that the line is not too bright.

To determine whether the water emission line brightness in L1544 has a non-
linear or linear dependence, we numerically solve the equations for the two-
level molecule with no approximations other than the escape probability and
plot the result. Figures 6 and 7 show the dependence of the antenna
temperature on the density for low and high densities respectively. Since the
column density, the optical depth, and the ratio C/A are all linearly
dependent on the density, any of these may be used on the abcissa. The latter
two are shown just above the axis. Figure 6 shows that the antenna temperature
of the water line emission is linearly dependent on the column density even at
high density or high optical depth. Figure 7 shows that the linear relation
breaks down when C/A is no longer small. The densities in both figures show
that the water line emission in L1544 is in the linear regime.

$\begin{array}[]{cc}\includegraphics[width=234.87749pt]{growth-
low4}\end{array}$

Figure 6:  The dependence of the observed antenna temperature of the H2O
($1_{10}-1_{01}$) line on the H2 number density (cm-3). Because the optical
depth and the ratio of the collision rate to spontaneous emission rate (C/A)
are both linearly dependent on the density, the abscissa can be labeled in
these units as well. Both are shown above the axis. The antenna temperature is
linearly dependent on the density or column density even at very high optical
depth as long as the ratio C/A is small.

$\begin{array}[]{cc}\includegraphics[width=234.87749pt]{growth-
high4}\end{array}$

Figure 7:  The dependence of the observed antenna temperature of the H2O line
($1_{10}-1_{01}$) on the number density. Same as figure 6 but at higher
densities where the ratio C/A is no longer small and the dependence of the
antenna temperature on the density is no longer linear.

For an intuitive explanation, suppose that a photon is absorbed on average
once per optical depth of one. A photon may be absorbed and another re-emitted
many times in escaping a cloud of high optical depth. The time scale for each
de-excitation is $A^{-1}$. Therefore, the time that it takes a photon to
escape the cloud is $\tau/A$. As long as this time is shorter than the
collisional excitation time ($1/C$), then on average, an emitted photon will
escape the cloud before another photon is created by the next collisional
excitation event and radiative de-excitation. In this case, the line remains
subcritically excited. The molecules are in the lower state almost all the
time. This is the same condition that would prevail if the cloud were
optically thin ($\bar{J}=0$ or $\beta=1$). On this basis, in our earlier paper
we determined the emissivity and opacity of the H2O line in L1544 by setting
$\bar{J}=0$ (Caselli et al., 2012). This approximation was earlier adopted in
analyzing water emission observed by the SWAS satellite (Snell et al., 2000)
where it is referred to as ”effectively optically thin”.

In this current paper, we seek an improved estimate of $\bar{J}>0$ and
$\beta<1$ by using the escape probability formalism as suggested by Linke et
al. (1977). We determine $\beta$ using the local velocity gradient as given by
our hydrodynamical model along with the local opacity using the Sobolev or
large velocity gradient (LVG) approximation (eqn. 3-40 Kalkofen, 1984). We use
the 6-ray approximation for the angle averaging. We allow for one free scaling
parameter on $\beta$ to match the modeled emission line brightness to the
observation. We scale the LVG opacity by 1/2. Because the opacity, column
density, and line brightness, are all linearly related, the scaling could be
considered to derive from any or any combination of these parameters. Given
all the uncertain parameters, for example the mean grain cross-section which
also affects the line brightness (appendix B), this factor of 2 is not
significant.

An alternative method to calculate the excitation is the accelerated
$\Lambda$-iteration algorithm (ALI). We do not know if this method is reliable
with the extremely high optical depth, several hundred to a thousand.
$\Lambda$-iteration generally converges, but whether it converges to the
correct solution cannot be determined from the algorithm itself (eqn. 6-33
Mihalas, 1978).

The excitation may be uncertain, but analysis with the escape probability
method allows us to understand the effect of the uncertainty. For example,
because we know that the dependence of the line brightness on the opacity or
optical depth is linear, we can say that any uncertainty in excitation results
in the same percentage uncertainty in the abundance of the chemical model, or
the pathlength of the structural model.

Once $\bar{J}$ is determined everywhere in the cloud, the equations of
statistical equilibrium are solved to determine the emissivity and opacity.
These are then used in the radiative transfer equation to produce the
simulated spectral line emission and absorption. This calculation is done in
MOLLIE in the same way as if $\bar{J}$ were determined by any other means, for
example, by $\Lambda$-iteration.

Both the emissivity and opacity depend on frequency through the Doppler
shifted line profile function (eqn. 2.14 Kalkofen, 1984) that varies as a
function of position in the cloud. We use a line profile function that is the
thermal width plus a microturbulent Gaussian broadening of 0.08 km s-1 derived
from our CO modeling (Keto & Caselli, 2010). By the approximation of complete
frequency redistribution (eqn. 10-39 Mihalas, 1978), both have the same
frequency dependence. This also implies that each photon emitted after an
absorption event has no memory of the frequency of the absorbed photon. It is
emitted with the frequency probability distribution described by the line
profile function Doppler shifted by the local velocity along the direction of
emission. We also assume complete redistribution in angle.

Figure 8 shows the modeled line profile against the observed profile. The VLSR
is assumed to be 7.16 kms-1, slightly different than 7.2 kms-1 used in Caselli
et al. (2012). The lower value is chosen here as the best fit to the H2O
observation.

The combination of blue-shifted emission and red-shifted absorption is the
inverse P-Cygni profile characteristic of contraction, with the emission and
absorption split by the inward gas motion in the front and rear of the cloud.
The absorption against the dust continuum is unambiguously from the front side
indicating contraction rather than expansion. This profile has also been seen
in other molecules in other low-mass cold, dense clouds, with the absorption
against the dust continuum (Di Francesco et al., 2001).

In L1544, because the inward velocities are below the sound speed, and the H2O
line width is just larger than thermal, the emission is shifted with respect
to the absorption by less than a line width. In the observations, what appears
to be a blue-shifted emission line is just the blue shoulder and wing of the
complete emission, most of which is brighter, redder and wider than the
observed emission.

Our model also shows weaker emission to the red of the absorption line. This
emission is from inward moving gas in the front side of the contracting zone.
Again most of the emission is absorbed by the envelope and only the blue
shoulder of the line is seen. The asymmetry between the red and blue emission
comes about because the absorbing envelope, which is on the front side of the
cloud, is closer in velocity to inward flowing gas (red) on the front side of
the contraction. This is the same effect that produces the blue asymmetric or
double-peaked line profiles seen in contraction in molecular lines without
such significant envelope absorption (Anglada et al., 1987). The model shows
more red emission than is seen in the observations. This red emission may be
absorbed by foreground gas that is not in the model. Figure 1 of Caselli et
al. (2012) shows additional red shifted absorption in H2O and red shifted
emission in CO, both centered around 9 kms-1. The blue wing of this red
shifted water line may be absorbing the red wing of the emission from the
dense cloud.

If L1544 were static, no inward contraction, the emission from the center
would be at the same frequency as the envelope. Because of the extremely high
optical depth, the absorption line is saturated and would absorb all the
emission. We would see only the absorption line. The depth of the absorption
line is set by brightness of the dust continuum which is weak (0.011 K) and
not by the optical depth of the line which is high (few hundred to a
thousand).

In the current radiative transfer calculation, we also use a slightly
different collisional excitation rate than before. The collisional rates for
ortho-H2O are different with ortho and para-H2. In our previous paper (Caselli
et al., 2012) we modeled the H2 ortho-to-para ratio as a lower limit 1:1 or
higher. Here we assume that almost all the hydrogen, 99.9%, is in the para
state. This is suggested by recent chemical models that require a low ortho-
to-para ratio to produce the high deuterium fraction observed in cold, dense
clouds. (e.g Kong et al., 2013; Sipilä, Caselli & Harju, 2013).

##  3 Interpretation

The shape of the line profile (figure 8) is unaffected by any uncertainty in
the excitation which scales the emission across the spectrum. The absorption
is saturated and does not scale with the excitation. Because of the very high
critical density for collisional de-excitation, we know that the line emission
is generated only in the densest gas ($>10^{6}$ cm-3) within a few thousand AU
of the center. Thus the observation of the inverse P-Cygni profile seen in H2O
confirms the model for quasi-hydrostatic contraction with the highest
velocities near the center (figure 1).

The chemical model requires external UV to create the gas phase water by
photodesorption. This confirms the physical model of L1544 as a molecular
cloud bounded by a photodissociation region. The UV flux necessarily creates a
higher temperature, up to about 100 K at the boundary by photoelectric
heating. This helps maintain the pressure balance at the boundary consistent
with the model of a BE sphere.

##  4 Uncertainties

The comparison of the simulated and observed spectral line involves three
models each with multiple parameters. Unavoidably the choice of parameters in
any one of the three models affects not only the choice of other parameters in
the other two models but also the interpretation. It would be a mistake to
focus on the uncertainties in any one of the models to the exclusion of the
others. For example, because of the linear relationship between the line
brightness, the optical depth, and the opacity, uncertainties in the
excitation, pathlength, and abundance, have equal effect on the spectrum. A
factor of two uncertainty in the excitation can be compensated by a factor of
two in the pathlength or a factor of two in the abundance of H2O. The
pathlength is unknown. On the plane of the sky, L1544 has an axial ratio of
2:1, but we are using a spherical model for the cloud. Our rates in the
chemical model involve estimation of the surface density of sites for
desorption and the covering fraction of water ice on the grains. The latter is
assumed to be one even though we know that CO and methane ice, not included in
the simple model, make up a significant fraction of the ice mantle. The
radiative excitation, parameterized as $\beta$ in the escape probability is
also uncertain because of the competing effects of high optical depth and
subcritical excitation.

On a linear plot, a factor of two difference in the brightness of the
simulated and observed spectral line looks to be a damning discrepancy.
However, there is at least this much uncertainty in each of the three models
and this does not significantly affect the conclusions of the study, namely
that the cloud can be modeled as a slowly contracting BE sphere bounded by a
photodissociation region with the gas phase water abundance set by grain
surface reactions.

In this paper, we concentrate on the observation of H2O, but there are also
other constraints that define the model. These are both observational and
theoretical. In an earlier paper, we showed how observations of CO and N2H+
define the physical model with the two spectral lines giving us information on
the outer and inner regions of the cloud respectively. In this regard, the
water emission gives us information in the central few thousand AU of the
cloud where the density approaches or exceeds the critical density for de-
excitation. This small volume of rapid inflow and high density does not much
affect the N2H+ spectrum which is generated in a much larger volume, and has
no affect at all on the CO spectrum. A successful model for L1544 has to
satisfy the constraints of all the data. On the theoretical side, there is an
infinite space of combinations of abundance, density, velocity, and
temperature that would form models that match the data. Only models that are
physically motivated are of interest. It may be tempting to change the
abundances, velocities, or densities arbitrarily, but this is unlikely to be a
useful exercise giving the infinite possibilities. A successful model for
L1544 has to be relevant to plausible theory.

There is a natural prejudice for more complex models that in principle contain
more details. The goal of our simplified models is to enhance our
understanding of the most significant phenomena. In our research on cold,
dense clouds, spanning a number of papers, we have developed simplified models
for the density and temperature structure, for the dynamics including
oscillations, for the CO chemistry, and in this paper simplified models for
H2O chemistry and radiative transfer. Each of these models isolates one or a
few key physical processes and shows how they generate the observables and
operate to control the evolution toward star formation.

$\begin{array}[]{cc}\includegraphics[width=234.87749pt]{SpectrumLVG}\end{array}$

Figure 8:  Observed spectrum of H2O (1${}_{10}-1_{01}$) (black lines with
crosses) compared with modeled spectrum (simple red line) for slow contraction
at the time that the central density reaches $1\times 10^{7}$ cm-3. The model
structure is shown in figure 1.

##  5 Conclusions

A simplified chemical model for cold oxygen chemistry primarily by grain
surface reactions is verified by comparing the simulated spectrum of the H2O
($1_{10}-1_{01}$) line against an observation of water vapor in L1544 made
with HIFI spectrometer on the Herschel Space Observatory.

This model reproduces the observed spectrum of H2O, and also approximates the
abundances calculated by a more complete model that includes gas-phase
neutral-neutral and ion-neutral reactions.

The gas phase water is released from ice grains by ultraviolet (UV)
photodesorption. The UV radiation derives from two sources: external starlight
and collisions of cosmic rays with molecular hydrogen. The latter may be
important deep inside the cloud where the visual extinction is high enough
($>50$ mag) to block out the external UV radiation.

Water is removed from the gas phase by photodissociation and freeze-out onto
grains. The former is important at the boundary where the UV from external
starlight is intense enough to create a photodissociation region. Here, atomic
oxygen replaces water as the most abundant oxygen species. In the center where
the external UV radiation is completely attenuated, freeze-out is the
significant loss mechanism.

Time dependent chemistry is not required to match the observations because the
time scale for the chemical processes is short compared to the dynamical time
scale.

The molecular cloud L1544 is bounded by a photodissociation region.

The water emission derives only from the central few thousand AU where the gas
density approaches the critical density for collisional de-excitation of the
water line. In the model of hydrostatic equilibrium, the gas density in the
center is rising with decreasing radius more steeply than the abundance of
water is decreasing by freeze-out. Thus the water spectrum provides unique
information on the dynamics in the very center.

The large Einstein A coefficient ($3\times 10^{-3}$ s-1) of the 557 GHz H2O
($1_{10}-1_{01}$) line results in extremely high optical depth, several
hundred to a thousand. However, the density ($<10^{7}$ cm-3) and temperature
($<15$ K) are low enough that the line is subcritically excited. The result is
that the line brightness under these conditions is directly proportional to
the column density.

##  6 Acknowledgements

The authors acknowledge Simon Bruderer, Fabien Daniel, Michiel Hogerheijde,
Joe Mottram, Floris van der Tak for interesting discussions on the radiative
transfer of water. PC acknowledges the financial support of the European
Research Council (ERC; project PALs 320620), of successive rolling grants
awarded by the UK Science and Technology Funding Council. JR acknowledges the
financial support of the Submillimeter Array Telescope.

## References

  * Anglada et al. (1987) Anglada G., Rodriguez L. F., Canto J., Estalella R., Lopez R., 1987, A&A, 186, 280
  * Bergin et al. (2000) Bergin E. A. et al., 2000, ApJ Lett, 539, L129
  * Bergin & Tafalla (2007) Bergin E. A., Tafalla M., 2007, ARAA, 45, 339
  * Bethell & Bergin (2009) Bethell T., Bergin E., 2009, Science, 326, 1675
  * Broderick & Keto (2010) Broderick A. E., Keto E., 2010, ApJ, 721, 493
  * Caselli et al. (2012) Caselli P. et al., 2012, ApJ Lett, 759, L37
  * Caselli et al. (2010) Caselli P. et al., 2010, A&A, 521, L29
  * Caselli et al. (2002) Caselli P., Walmsley C. M., Zucconi A., Tafalla M., Dore L., Myers P. C., 2002, ApJ, 565, 344
  * Conrath & Gierasch (1984) Conrath B. J., Gierasch P. J., 1984, Icarus, 57, 184
  * de Graauw et al. (2010) de Graauw T. et al., 2010, A&A, 518, L6
  * di Francesco et al. (2007) di Francesco J., Evans, II N. J., Caselli P., Myers P. C., Shirley Y., Aikawa Y., Tafalla M., 2007, Protostars and Planets V, 17
  * Di Francesco et al. (2001) Di Francesco J., Myers P. C., Wilner D. J., Ohashi N., Mardones D., 2001, ApJ, 562, 770
  * Evans et al. (2001) Evans, II N. J., Rawlings J. M. C., Shirley Y. L., Mundy L. G., 2001, ApJ, 557, 193
  * Fouchet, Lellouch & Feuchtgruber (2003) Fouchet T., Lellouch E., Feuchtgruber H., 2003, Icarus, 161, 127
  * Habing (1968) Habing H. J., 1968, Bulletin of the Astronomical Inst. of the Netherlands, 19, 421
  * Hollenbach et al. (2009) Hollenbach D., Kaufman M. J., Bergin E. A., Melnick G. J., 2009, ApJ, 690, 1497
  * Jijina, Myers & Adams (1999) Jijina J., Myers P. C., Adams F. C., 1999, ApJ Suppl, 125, 161
  * Kalkofen (1984) Kalkofen W., 1984, Methods in radiative transfer
  * Kandori et al. (2005) Kandori R. et al., 2005, AJ, 130, 2166
  * Keto et al. (2006) Keto E., Broderick A. E., Lada C. J., Narayan R., 2006, ApJ, 652, 1366
  * Keto & Caselli (2008) Keto E., Caselli P., 2008, ApJ, 683, 238
  * Keto & Caselli (2010) Keto E., Caselli P., 2010, MNRAS, 402, 1625
  * Keto & Field (2005) Keto E., Field G., 2005, ApJ, 635, 1151
  * Keto & Rybicki (2010) Keto E., Rybicki G., 2010, ApJ, 716, 1315
  * Keto (1990) Keto E. R., 1990, ApJ, 355, 190
  * Kong et al. (2013) Kong S., Caselli P., Tan J. C., Wakelam V., 2013, arxiv:1312.0971
  * Lada et al. (2003) Lada C. J., Bergin E. A., Alves J. F., Huard T. L., 2003, ApJ, 586, 286
  * Linke et al. (1977) Linke R. A., Goldsmith P. F., Wannier P. G., Wilson R. W., Penzias A. A., 1977, ApJ, 214, 50
  * Mathis, Rumpl & Nordsieck (1977) Mathis J. S., Rumpl W., Nordsieck K. H., 1977, ApJ, 217, 425
  * Melnick et al. (2000) Melnick G. J. et al., 2000, ApJ Lett, 539, L87
  * Mihalas (1978) Mihalas D., 1978, Stellar atmospheres /2nd edition/
  * Myers & Benson (1983) Myers P. C., Benson P. J., 1983, ApJ, 266, 309
  * Neufeld et al. (2000) Neufeld D. A. et al., 2000, ApJ Lett, 539, L107
  * Rawlings et al. (1992) Rawlings J. M. C., Hartquist T. W., Menten K. M., Williams D. A., 1992, MNRAS, 255, 471
  * Shen et al. (2004) Shen C. J., Greenberg J. M., Schutte W. A., van Dishoeck E. F., 2004, A&A, 415, 203
  * Sipilä, Caselli & Harju (2013) Sipilä O., Caselli P., Harju J., 2013, A&A, 554, A92
  * Snell et al. (2000) Snell R. L. et al., 2000, ApJ Lett, 539, L101
  * Tielens (2005) Tielens A. G. G. M., 2005, The Physics and Chemistry of the Interstellar Medium
  * Tielens & Hollenbach (1985) Tielens A. G. G. M., Hollenbach D., 1985, ApJ, 291, 722
  * van Dishoeck et al. (1999) van Dishoeck E. F. et al., 1999, in ESA Special Publication, Vol. 427, The Universe as Seen by ISO, Cox P., Kessler M., eds., p. 437
  * van Dishoeck, Herbst & Neufeld (2013) van Dishoeck E. F., Herbst E., Neufeld D. A., 2013, Chemical Reviews, 113, 9043
  * Viti et al. (2001) Viti S., Roueff E., Hartquist T. W., Pineau des Forêts G., Williams D. A., 2001, A&A, 370, 557

##  Appendix A Cooling by atomic oxygen fine structure lines

The fine structures lines of C+ and atomic O are the major coolants in the
diffuse ($n<1000$ cm-3), photodissociated gas around the molecular clouds. The
more important coolant at temperatures less than 100 K is C+. At higher
temperatures, oxygen becomes increasingly important in the energy balance. The
reason is that the 63.2 and 145.6 $\mu$m fine structure lines of atomic oxygen
have upper states 3P1 and 3P0 that are at 228 K and 326 K above ground,
considerably higher than the 92 K of the upper state, 2P3/2 of the 157.6
$\mu$m fine structure line of C+.

The cooling by atomic oxygen is simple to model because atomic oxygen is a
product of photodissociation and is therefore abundant only in gas with low Av
implying gas densities below the critical densities for collisional de-
excitation, 6400 and 3400 cm-3 for the 63.2 and 145.6 $\mu$m lines
respectively (table 2.7 of Tielens, 2005). At this density, we assume that the
optically thin approximation applies. In this case, every collisional
excitation to an upper state of the fine structure lines results in
spontaneous emission that escapes the cloud and cools the gas,

$\Lambda_{\rm O}=n({\rm O})n({\rm H_{2}})(E_{21}C_{21}+E_{20}C_{20})\ \ {\rm
ergs}\ {\rm cm}^{-3}\ {\rm s}^{-1}$ (4)

where the upward collision rates are,

$C_{21}=1.4\times 10^{-8}\frac{g_{1}}{g_{2}}C_{12}\exp{(-E_{21}/kT)}\sqrt{T}\
\ {\rm cm}^{3}{\rm s}^{-1}$ (5) $C_{20}=1.4\times
10^{-8}\frac{g_{0}}{g_{2}}C_{02}\exp{(-E_{20}/kT)}\sqrt{T}\ \ {\rm cm}^{3}{\rm
s}^{-1}.$ (6)

and the statistical weights are $g_{2}=5$, $g_{1}=3$, and $g_{0}=1$ and the
transition energies are $E_{12}/k=228$K and $E_{02}/k=326$K.

##  Appendix B Chemistry

###  B.1 Freeze-out

Molecules freeze onto dust grains, sticking when they collide. This process is
easily modeled. We follow Keto & Caselli (2008) to calculate the collision
timescale. The time scale for depletion onto dust may be estimated as
(Rawlings et al., 1992),

$\tau_{on}=(S_{0}R_{dg}n({\rm H_{2}})\sigma V_{T})^{-1}\ {\rm s}$ (7)

Here $S_{0}$ is the sticking coefficient, with $S_{0}=1$ meaning that the
colliding molecule always sticks to the dust in each collision; $R_{dg}$ is
the ratio of the number density of dust grains relative to molecular hydrogen;
$\sigma$ is the mean cross-section of the dust grains; and $V_{T}$ is the
relative velocity between the grains and the gas. If the grains have a power
law distribution of sizes with the number of grains of each size scaling as
the -3.5 power of their radii (Mathis, Rumpl & Nordsieck, 1977), then we can
estimate their mean cross-section as,

$\langle\sigma\rangle=\bigg{(}\int^{a_{2}}_{a_{1}}n(a)da\bigg{)}^{-1}\int^{a_{2}}_{a_{1}}n(a)\sigma(a)da,$
(8)

where $a_{1}$ and $a_{2}$ are the minimum and maximum grain sizes. If
$a_{1}=0.005$ $\mu$m and $a_{2}=0.3$ $\mu$m, then
$\langle\sigma\rangle=3.4\times 10^{-4}$ $\mu{\rm m}^{2}$. Similarly, the
ratio of the number densities of dust and gas may be estimated by computing
the mean mass of a dust grain and assuming the standard gas-to-dust mass ratio
of 100. If the density of the dust is 2 grams cm-3, then the ratio of number
densities is $R_{dg}=4\times 10^{-10}$.

Consistent with Keto & Caselli (2008), our model has a slightly lower value
for the grain cross-section, $1.4\times 10^{-21}$ cm2, than Hollenbach et al.
(2009), $\sigma_{h}=2\times 10^{-21}$ cm2. Both values are per hydrogen
nucleus (2H2 \+ H). Because the ice forms and desorbs off the grain surfaces,
larger values of the average cross-section result in fewer molecules in the
gas phase. The actual properties of grains in cold clouds are somewhat
uncertain.

The relative velocity due to thermal motion is,

$V_{T}=\bigg{(}{{8kT}\over{\pi\mu}}\bigg{)}^{1/2},$ (9)

where $T$ is the temperature and $\mu$ the molecular weight. The freeze-out
rate for species $i$ is,

$f_{i}=\tau_{on}^{-1}n(H_{2})\ \ {\rm cm}^{-3}\ {\rm s}^{-1}$ (10)

###  B.2 Gas-phase reactions

The neutral-neutral molecular and photodissociation reactions are from Tielens
& Hollenbach (1985). The reaction rate $k_{1}$ for ${\rm
O+H_{2}}\rightarrow{\rm OH+H}$ is,

$k_{1}=3.1\times 10^{-13}\ (T/300)^{2.7}\exp{(-3150/T)}\ \ {\rm cm}^{3}\ \
{\rm s}^{-1}$ (11)

The reaction $k_{2}$ for ${\rm OH+H_{2}}\rightarrow{\rm H_{2}O+H}$ is,

$k_{2}=2.0\times 10^{-12}(T/300)^{1.57}\exp{(-1736/T)}\ \ {\rm cm}^{3}\ \ {\rm
s}^{-1}$ (12)

OH + H has a rate,

$5.3\times 10^{-18}(T/300)^{-5.22}\exp{(-90/T)}.$ (13)

All three of these reactions have an activation barrier and are irrelevant at
temperatures below 300 K. The photodissociation rate for the destruction of OH
and the formation of O is,

$P_{1}=3.5\times 10^{-10}G_{0}\exp{(-1.7A_{V})}\ \ {\rm s}^{-1}$ (14)

and the rate for the destruction of H2O and formation of OH is,

$P_{2}=5.9\times 10^{-10}G_{0}\exp{(-1.7A_{V})}\\\ \ {\rm s}^{-1}.$ (15)

The unitless parameter $G_{0}=1$ corresponds to the average local interstellar
radiation field in the FUV band (Habing, 1968). $A_{V}$ is the visual
extinction.

###  B.3 Desorption

The desorption rates are from Hollenbach et al. (2009). The total desorption
rate includes thermal desorption, photodesorption, and desorption by cosmic
rays. We use equation 2 from Hollenbach et al. (2009) for the rate for thermal
desorption,

$D_{Th}=1.6\times
10^{11}\bigg{(}\frac{E_{i}}{k}\bigg{)}^{1/2}\bigg{(}\frac{m_{H}}{m_{i}}\bigg{)}^{1/2}\exp{\bigg{(}\frac{-E_{i}}{kT_{gr}}\bigg{)}}\
\ {\rm s}^{-1}\ \ {\rm molecule}^{-1}$ (16)

where $E_{i}/k$, the adsorption energy is 800, 1300, and 5770 K for O, OH, and
H2O respectively, and mi/mH is the weight of the species with respect to H.
The thermal desorption rate for water is negligible at the temperatures ($<15$
K) of cold, dense clouds. For the cosmic-ray desorption rate, we use equation
8 from Hollenbach et al. (2009). We include only the cosmic-ray desorption
rate for H2O,

$D_{CR}=4.4\times 10^{-17}{\rm molecule}^{-1}{\rm s}^{-1}.$ (17)

Both the thermal desorption rate and the cosmic ray desorption rate in units
of molecule-1 s-1 are multiplied by the number of molecules on the surface of
grains per molecule of H2 which is $N_{s}f_{s}A_{gr}R_{dg}$ where
$N_{s}=10^{15}$ cm-2 is the number of desorption sites per cm2 on the grain
surface (Hollenbach et al., 2009), $f_{s}=1$ is the fraction of the grain
surface covered by ice, the average surface area of a grain is 4 times the
grain cross-section, $A_{gr}=4\sigma=4\times 3.4\times 10^{-4}$ $\mu$m 2 (Keto
& Caselli, 2008), and the dust-to-gas ratio $R_{dg}=4\times 10^{-10}$ (Keto &
Caselli, 2008). The photodesorption rates are from equations 6 and 7
(Hollenbach et al., 2009),

$D_{UV}=G_{0}F_{0}Y_{i}f_{i}\ \exp{(-1.8A_{V})}\ {\rm s}^{-1}$ (18)

where $F_{0}=10^{8}$ is the number of UV photons per Habing flux, and
$Y_{i}=10^{-3}$ and $2\times 10^{-3}$ are the photodesorption yields per UV
photon per second for the production of OH and H2O respectively from table 1
of Hollenbach et al. (2009). We assume that all the ice is H2O and follow
Hollenbach et al. (2009) in assuming that the photodesorption of this water
ice results in twice as much OH as H2O in the gas phase.

The desorption of water ice does not result in the production of gas phase
oxygen, and we have no oxygen ice in our model. To provide a back reaction to
the freeze-out of atomic oxygen, we arbitrarily assign a desorption rate equal
to that of water. In regions of high extinction ($A_{V}>4$) this results in a
gas phase abundance of atomic oxygen that is approximately the same as
predicted by Hollenbach et al. (2009). This is $<0.001$ of the total oxygen
and has no effect on the other abundances. In the outer part of the cloud
where the UV flux is higher ($A_{V}<4$) most of the atomic oxygen derives from
photodissociation. Here the UV desorption off grains is insignificant.

Additional desorption is caused by the UV photons emitted by hydrogen
excitation by energetic electrons released in the ionization of hydrogen by
cosmic rays. We follow Shen et al. (2004) and scale this process as $10^{-4}$
of one Habing flux, $G_{0}=1$, so that,

$D_{CR\ UV}=\alpha G_{0}F_{0}Y_{i}f_{i}\ {s^{-1}}$ (19)

with $\alpha=10^{-4}$.

###  B.4 Equilibrium

In equilibrium, the rate equations in matrix form are,

$\begin{array}[]{lll}\begin{pmatrix}-(f_{O}+k_{1})&P_{1}&0&0\\\
k_{1}&-(f_{OH}+P_{1})&P_{2}&0\\\ 0&k_{2}&-(f_{H_{2}O}+P_{2})&D_{H_{2}O}\\\
f_{O}&f_{OH}&f_{H_{2}O}&-(D_{OH}+D_{H_{2}O})\\\ 1&1&1&1\\\
\end{pmatrix}\begin{pmatrix}O\\\ OH\\\ H_{2}O\\\ ICE\\\
\end{pmatrix}=\begin{pmatrix}0\\\ 0\\\ 0\\\ 0\\\ 1\\\
\end{pmatrix}\end{array}$

where the last row is the conservation equation for oxygen among all the
species. As written, this system is overdetermined, but can be solved by
dropping any one of the first 4 rows.

###  B.5 H2O ortho-para ratio

Since the ortho state of H2O is 24K above the para state, the O/P ratio in
thermal equilibrium is very small at lower temperatures (equation 41
Hollenbach et al., 2009). However, when the water molecule is formed, created
from OH on the grain surface for example, it is formed in the ratio of the
available quantum states, ortho:para 3:1. The ortho and para states of H2O
equilibrate by collisions with H or H2. If the chemical equilibrium time scale
is much shorter than the thermal equilibrium time scale, the O/P ratio will
not deviate much from 3:1. Observations generally show ratios close to 3:1
(van Dishoeck, Herbst & Neufeld, 2013).

We have not found previous research on the equilibration of H2O, but an
appreciation of the time scale can be estimated from previous research on the
equilibration of the ortho and para states of molecular hydrogen. The
dissociation energies of H-H and OH-H are not too different nor the
collisional cross-sections of the molecules. Conrath & Gierasch (1984) and
Fouchet, Lellouch & Feuchtgruber (2003) suggest three processes for the
equilibration of the ortho and para states of H2 are: (1) gas phase H
exchange, (2) gas phase paramagnetic conversion with H2, and (3) H exchange on
a surface. We assume that these same processes are applicable to the water.
The rates for these processes scale with the gas density through the collision
rate and scale as the inverse exponential of the temperature. Scaling the
rates for H2 from the conditions in the atmosphere of Jupiter to rarefied,
cold gas of the interstellar medium (10 K and $10^{6}$ cm-3) the time scales
for these processes are all $>1$ Gyr.

In contrast, the chemical time scale is very much shorter (figure 5)
throughout the cloud. In this model, water is dissociated in the gas phase by
photodissociation and also coming off the grain surfaces by photodesorption in
which gas phase OH is produced twice as often as gas phase H2O. The
equilibrium comparison between ortho-para equilibration and chemistry may not
be needed because the equilibration time scale exceeds the expected life times
of the cold, dense, clouds.


# Efficient Training of Large-scale Industrial Fault Diagnostic Models through
Federated Opportunistic Block Dropout

Yuanyuan Chen1, Zichen Chen1,2, Sheng Guo3, Yansong Zhao1, Zelei Liu1,
Pengcheng Wu1, Chengyi Yang3, Zengxiang Li3$\dagger$, Han Yu1$\dagger$

###### Abstract

Artificial intelligence (AI)-empowered industrial fault diagnostics is
important in ensuring the safe operation of industrial applications. Since
complex industrial systems often involve multiple industrial plants (possibly
belonging to different companies or subsidiaries) with sensitive data
collected and stored in a distributed manner, collaborative fault diagnostic
model training often needs to leverage federated learning (FL). As the scale
of the industrial fault diagnostic models are often large and communication
channels in such systems are often not exclusively used for FL model training,
existing deployed FL model training frameworks cannot train such models
efficiently across multiple institutions. In this paper, we report our
experience developing and deploying the Federated Opportunistic Block Dropout
(FedOBD) approach for industrial fault diagnostic model training. By
decomposing large-scale models into semantic blocks and enabling FL
participants to opportunistically upload selected important blocks in a
quantized manner, it significantly reduces the communication overhead while
maintaining model performance. Since its deployment in ENN Group in February
2022, FedOBD has served two coal chemical plants across two cities in China to
build industrial fault prediction models. It helped the company reduce the
training communication overhead by over 70% compared to its previous AI
Engine, while maintaining model performance at over 85% test F1 score. To our
knowledge, it is the first successfully deployed dropout-based FL approach.

## Introduction

In modern industries, machinery is becoming increasingly sophisticated and
facing highly demanding operational conditions. For example, rotating
machinery (e.g., turbines, fans and pumps) are key components that are widely
used in power generation and chemical plants. Slight performance
deterioration, if not addressed early, could lead to sudden breakdowns or even
serious accidents involving significant financial losses and/or human
casualty. As industries modernize towards the vision of Industry 4.0
(Ghobakhloo 2020), a wide variety of sensing devices are starting to be
deployed in industrial settings to help monitor important equipment. The data
collected by such devices make it possible to train intelligent fault
diagnostic models for system maintenance decision support.

Artificial intelligence (AI) technologies have increasingly been applied in
industrial fault diagnostic model training. The performance of such machine
learning-based solutions depends on having access to large amounts of high
quality data. However, data from a single factory might not be adequate to
train such models effectively. As data are often collected and owned by
different organizations in a given field, collaborative model training
(Warnat-Herresthal et al. 2021) has been recognized as a useful technique to
improve the quality of AI solutions in such situations.

As societies become increasingly aware of data privacy protection issues
(e.g., following the introduction of data privacy regulations such as the
General Data Protection Regulation (GDPR) (GDPR 2018)), federated learning
(FL) (Yang et al. 2019; Kairouz, McMahan et al. 2021) \- a privacy-preserving
collaborative machine learning paradigm - has emerged. It has been rapidly
gaining traction and has been applied in wide-ranging applications including
safety management (Liu et al. 2020), banking (Long et al. 2020) and smart
healthcare (Liu et al. 2022b).

In recent years, industrial fault detection applications powered by FL are
starting to emerge (Ma, Wen, and Wen 2021; Zhang et al. 2021; Geng et al.
2022; Wang et al. 2022). These applications generally build on top of the
popular Federated Averaging (FedAvg) FL model aggregation approach (McMahan et
al. 2017). Although they are useful for supporting privacy-preserving
collaborative model training, they are not optimized for training large-scale
deep neural network (DNN) models which are commonly required to build
effective industrial fault diagnostic models (Liu et al. 2022a). This is
exacerbated by constraints on bandwidth usage for FL model training imposed by
industries as the communication channel is often shared by multiple
applications, some of which are safety critical.

To address the aforementioned challenges facing FL-based industrial fault
diagnostics solutions, we propose the Federated Opportunistic Block Dropout
(FedOBD) approach (Chen et al. 2022). Its advantages are as follows:

  1. 1.

Training Large-scale DNNs Efficiently: FedOBD divides a DNN into semantic
blocks. Based on evaluating the importance of the blocks (instead of
determining individual parameter importance like in the cases of (Bouacida et
al. 2021; Gunesli et al. 2021)), it opportunistically discards unimportant
blocks in order to drastically reduce the size of the resulting model.
Combined with parameter quantization, FedOBD can significantly reduce the
communication overhead incurred during FL model training.

  2. 2.

Preserving Model Performance: As blocks that are most important to the
performance of the FL model are retained, FedOBD can preserve model
performance.

  3. 3.

Supporting Incentive Distribution: By storing the historical records of the
important blocks contributed to FL model training by each data owner, FedOBD
can provide useful information for contribution-based incentive allocation
(Cong et al. 2020; Yu et al. 2020).

Compared to existing efficient FL model training frameworks, FedOBD offers new
capabilities which can support more sophisticated use cases.

The FedOBD approach has been deployed through a collaboration between ENN
Group111https://www.enn.cn/ and the Trustworthy Federated Ubiquitous Learning
(TrustFUL) Lab222https://trustful.federated-learning.org/, Nanyang
Technological University (NTU), Singapore since February 2022. It is used to
replace the FedAvg model aggregation approach in the ENN FL model training
platform to support FL model training under server-based horizontal FL
settings (Yang et al. 2019). In such scenarios, FL participants’ datasets have
large overlaps in the feature space, but little overlap in the sample space.
It has helped ENN Group train intelligent industrial fault diagnostic models
involving two factories from two cities in China. Throughout the deployment
period, FedOBD has helped ENN Group reduce the training communication overhead
by over 70% compared to its previous implementation, while maintaining model
performance at over 85% test F1 score. To the best of our knowledge, it is the
first dropout-based federated learning method successfully deployed in
industrial settings.

With the help of FedOBD, the ENN Group federated learning subsystem has
avoided the problems of long delays of training/updating industrial fault
prediction models through FL, while maintaining comparable model performance.
This capability allows it to provide more rapid update of such models to its
enterprise customers and subsidiaries, thereby improving safe operation.

## Application Description

Figure 1: An overview of the ENN intelligent industrial fault diagnostics
platform. Figure 2: An overview of the ENN FL model training platform -
Client-Server FL Subsystem. Figure 3: An example user interface of the ENN
Client-Server FL Subsystem (configuring FL training). Figure 4: An example
user interface of the ENN Client-Server FL Subsystem (FL training process
visualization). Figure 5: An example user interface of the ENN Client-Server
FL Subsystem (model performance summary).

Founded in 1989, ENN Group’s business encompasses a diverse range of segments
within the natural gas and green energy industry including distribution,
trade, transportation and storage, production, and intelligent engineering,
with the aim of creating modern energy systems that improve people’s quality
of life. Leveraging its diverse industrial ecosystem, ENN has been building up
an industrial digital intelligence platform in order to empower the
stakeholders involved (including more than 25 million household customers and
over 200,000 enterprise customers in 20 provinces across China). In this
section, we provide detailed descriptions of the ENN FL model training
platform, which is part of its industrial digital intelligence platform.

The overall flow of this platform is shown in Figure 1. Enterprise customers
under the ENN Group (e.g., coal chemical plants) deploy sensing devices within
their factories to monitor equipment operation and collect data. These data
are stored locally within the data silo. Standard data preprocessing (e.g.,
Fourier transform, wavelet transform) is carried out locally to prepare the
data for analysis and model training. Nevertheless, as equipment faults do not
occur frequently, such data tend to be sparse and biased within each data
silo. The ENN Group offers its enterprise customers and subsidiaries from the
same industry sectors the option to join FL to collaboratively train fault
diagnostic models. Our focus is on its FL model training subsystem with a
client-server architecture. It consists of two components (Figure 2): 1) the
ENN FL Server, and 2) the ENN FL Client.

### ENN FL Server

The ENN FL Server hosts the FL model aggregation server and a set of utility
modules supporting the operations which system administrators need to perform.
It allows the system administrators to select from a range of FL model
training and aggregation approaches (e.g., FedAvg (McMahan et al. 2017),
FedOBD) incorporated into the system, select the base model to be trained via
FL, and configure the FL model training process. These operations can be
carried out through a dedicated set of user interfaces. Once the configuration
steps are completed, the information is sent to the FL model aggregation
server for execution via a local area network (LAN). As the LAN is a dedicated
communication channel for FL related operations, it does not place any
restrictions on bandwidth usage.

The FL model aggregation server is implemented following a modular design to
enable it to host alternative FL model aggregation approaches. It takes local
model updates from FL clients as inputs, and produces a global FL model as the
output, while making decisions on whether additional rounds of FL training are
required. Communication with the clients is managed by the Communication
Management module, which can accommodate special requirements (e.g., the need
for compressing the transmission via stochastic quantization (Alistarh et al.
2017)). This is because communications between the ENN FL Server and the ENN
FL Clients (which are deployed in different factories) take place over the ENN
wide area network (WAN), which is not dedicated to just FL model training.
Transmissions for other operational and business purposes also go through the
ENN WAN. Thus, this channel places a limit of up to 2 MB/sec to be used for FL
model training purposes, which severely restricts the speed of training large-
scale models through FL.

Figure 3 to Figure 5 illustrate the user interfaces (UIs) through which the
ENN FL model training platform visualizes the FL training process for the
system administrators. As it is designed for Chinese speaking users, we have
annotated regions in the UIs to highlight key design features. Figure 3 shows
the screen for the administrator to configure the FL training process for a
particular model by specifying important parameters (e.g., mode of federated
learning, the selected model training and aggregation approach). Once FL
training commences, the training activities are visualized in Figure 4 for the
administrators to monitor. In the example in Figure 4, one FL server and two
FL clients are involved. Activities performed by the server and each client
are listed in the corresponding box, making it easy to scroll back and forth
for an overview, and drill down into each record for more detailed
information. The overall training progress is illustrated in the FL training
flowchat on the left hand side panel. After training is concluded, a summary
of the performance of the resulting FL model as shown in Figure 5 is presented
to the administrators. Detailed FL model training activities and performance
evaluation results are stored by the platform to support review and auditing
in the future.

### ENN FL Client

Typically, each ENN FL Client is deployed in an industrial facility (e.g.,
factory, power plant, coal chemical plant). In the application of our focus,
industrial fault diagnostics, each facility deploys a set of sensors to
monitor the industrial equipment and collect the necessary data. The
Communication & Storage Management module aggregates the data and stores them
into the Local Database Server in the correct format. On top of this
infrastructure, the ENN Industrial Fault Diagnostic System carries out data
and signal processing, and performs fault prediction using a DNN-based model.
In order to leverage industrial fault data collected by different facilities,
this DNN-based fault prediction model is to be trained through FL.

Each ENN FL Client is incorporated with an FL Client Agent module. Similar to
the ENN FL Server, it is also implemented following the modular design
approach to enable it to host alternative FL model training and updating
approaches. The dedicated Communication Management module is also included to
accommodate special transmission requirements and to comply with the 2 MB/sec
transmission bandwidth usage limit placed on the ENN WAN for FL model
training.

## Use of AI Technology

Figure 6: The workflow of the FedOBD approach.

In this section, we describe the AI Engine of the ENN FL model training
subsystem, which is based on FedOBD. The workflow of the AI Engine is
illustrated in Figure 6. FedOBD (Chen et al. 2022) enables the FL server and
each FL client to determine the most important subset of parameters of a
large-scale deep neural network (DNN) model (which are organized into semantic
blocks) to be sent back and forth during FL model training, thereby reducing
the communication overhead incurred. Currently, it only supports horizontal
federated learning (HFL) in which data owners have large overlaps in the
feature space, but little overlap in the sample space (Yang et al. 2019).

In a client-server HFL system, there are in general $n$ clients who can
participate in FL model training. Each client $i$ has a local dataset
$D_{i}=\left\\{\left(\boldsymbol{x}_{j},\boldsymbol{y}_{j}\right)\right\\}_{j=1}^{M_{i}}$.
$\boldsymbol{x}_{j}$ denotes the $j$-th local training sample.
$\boldsymbol{y}_{j}$ denotes the corresponding ground truth label of
$\boldsymbol{x}_{j}$. $M_{i}$ denotes the total number of data samples in
$D_{i}$. The aim of HFL is to solve the following optimization problem under
the aforementioned setting:

$\min_{\boldsymbol{w}\in\boldsymbol{W}}\sum_{i=1}^{n}\frac{M_{i}}{M}\mathcal{L}_{i}(\boldsymbol{w};D_{i}).$
(1)

Here, $\boldsymbol{W}$ denotes the parameter space determined by a given
neural network. $M\coloneqq\Sigma_{i=1}^{n}M_{i}$ denotes the total number of
samples.
$\mathcal{L}_{i}(\boldsymbol{w};D_{i})\coloneqq\frac{1}{M_{i}}\sum_{j=1}^{M_{i}}\ell(\boldsymbol{w};\boldsymbol{x}_{j},\boldsymbol{y}_{j})$
denotes the local loss of a given client $i$.

### Opportunistic Block Dropout (OBD)

A DNN can be divided into semantic blocks consisting of consecutive layers.
Under the FedOBD FL model training approach, important semantic blocks in a
DNN are identified at the end of any given round of FL client local training
or FL server aggregation. Once this is done, semantic blocks are selected in
descending order of their importance until a dropout rate pre-specified by the
system administrators has been reached. Then, only these selected blocks of
the DNN, instead of the entire model, are transmitted to facilitate FL model
training. This design of FedOBD is different from existing dropout-based FL
training approaches including FedDropoutAvg approach (Gunesli et al. 2021) and
Adaptive Federated Dropout (AFD) approach (Bouacida et al. 2021). Both of
which randomly select individual model parameters to be dropped out without
organizing the model into semantic blocks first, making the resulting model
difficult to compress during transmission.

FedOBD can support popular NN architectures when decomposing the models. Layer
sequences such as $\langle$Convolution, Pooling, Normalization,
Activation$\rangle$ are commonly found in convolutional neural networks
(CNNs). Encoder layers are commonly found in Transformer based models. Other
NNs can include basic building blocks which can be used to divide a given
model into blocks. Other layers which cannot be grouped into commonly found
functional block patterns can be treated as singleton blocks. FedOBD uses the
Mean Block Difference (MBD) metric to measure block importance. It can be
computed as follows:

$\texttt{MBD}(\boldsymbol{b}_{r-1},\boldsymbol{b}_{r,i})\coloneqq\frac{\left\lVert\texttt{vector}(\boldsymbol{b}_{r-1})-\texttt{vector}(\boldsymbol{b}_{r,i})\right\rVert_{2}}{\texttt{NumberOfParameters}(\boldsymbol{b}_{r-1})}.$
(2)

$\boldsymbol{b}_{r-1}$ denotes the blocks of a previous model (e.g., the
received global FL model). $\boldsymbol{b}_{r,i}$ denotes the corresponding
blocks in an updated model (e.g., the local model trained by a client $i$ in
the current round). vector is an operator that concatenates parameters from
different layers of a block (if there are multiple layers involved) into a
single vector. The larger the MBD value of a block, the more important the
newer version of this block is.

Input : global model $\boldsymbol{w}_{r-1}$, local model
$\boldsymbol{w}_{r,i}$ in client $i$, the set of identified block structures
$\boldsymbol{B}$, dropout rate $\lambda\in[0,1]$.

Output : retained blocks.

1

2$\textnormal{important\\_blocks}\leftarrow\texttt{MaxHeap}()$;

3foreach  _$\boldsymbol{b}\in\boldsymbol{B}$_ do

4
$\textnormal{important\\_blocks}[\texttt{MBD}(\boldsymbol{b}_{r-1},\boldsymbol{b}_{r,i})]\leftarrow\boldsymbol{b}_{r,i}$;

5 end foreach

6$\textnormal{revised\\_model\\_size}\leftarrow 0$;

7 $\textnormal{retained\\_blocks}\leftarrow\texttt{List}()$;

8 while  _important_blocks_ do

9
$\boldsymbol{b}_{r,i}\leftarrow\textnormal{important\\_blocks}.\texttt{pop}()$;

10
$\textnormal{new\\_size}\leftarrow\textnormal{revised\\_model\\_size}+\lvert{}\texttt{vector}(\boldsymbol{b}_{r,i})\rvert$;

11 if  _
$\textnormal{new\\_size}>(1-\lambda)\lvert{}\texttt{vector}(\boldsymbol{w}_{r,i})\rvert$
_ then

12 continue __;

13

14 end if

15 $\textnormal{revised\\_model\\_size}\leftarrow\textnormal{new\\_size}$;

16 $\textnormal{retained\\_blocks}.\texttt{append}(\boldsymbol{b}_{r,i})$;

17 end while

return _retained_blocks_ ;

Algorithm 1 OBD

The Opportunistic Block Dropout (OBD) algorithm of FedOBD is shown in
Algorithm 1. Before a model is sent out either by the FL server or client,
each block is assigned an importance score. To achieve this goal, the sending
entity stores $\boldsymbol{w}_{r-1}$ which can be used to compare with the
current model $\boldsymbol{w}_{r,i}$ in a block by block fashion by following
Eq. (2).

Once the MBD values for all the blocks have been computed, FedOBD determines
which blocks to retain and which to be dropped (Lines 5-15) by ranking in
descending order of their MBD values (with the MaxHeap data structure), and
putting the blocks into the retained_blocks list one by one until the size of
the revised model reaches
$(1-\lambda)\lvert{}\texttt{vector}(\boldsymbol{w}_{r,i})\rvert$.
$\lambda\in[0,1]$ is the dropout rate (where 1 indicates the entire model is
dropped out, and 0 indicates no dropout). The retained blocks are then
quantized in preparation for transmission. Each retained block is stored in
the form of the differences between the corresponding parameter values in
block $\boldsymbol{b}_{r}$ and block $\boldsymbol{b}_{r-1}$.

### Overall Workflow of FedOBD

The overall workflow of the FedOBD approach is illustrated in Figure 6.

  1. 1.

Model Distribution: If it is the first time the initialized global FL model is
distributed to the FL clients, the entire model is sent out by the FL server.
Otherwise, the server performs OBD to determine the list of important blocks
of the model to be retained based on the given dropout rate, and only sends
out the quantized version of these retained blocks to the clients.

  2. 2.

Reconstruction of the Global Model: When the retained blocks from the server
in round $r$ is received by a client, it combines them with unchanged (i.e.,
not transmitted) blocks from the global model $\boldsymbol{w}_{r-1}$ which it
has received in the previous round to reconstruct $\boldsymbol{w}_{r}$. Local
training is then carried out based on this reconstructed global model.

Figure 7: Deployment of FedOBD into the ENN Client-Server FL Subsystem.

  3. 3.

2-Stage Local Model Training: FedOBD is implemented as a two-stage training
process. In the first stage, small local epochs are used. OBD selects
important blocks to be transmitted, and these blocks are further compressed
via quantization. In this way, FedOBD encourages frequent aggregation to
prevent overfitting in a communication efficient manner. As the global FL
model approaches convergence, FedOBD transits into the second stage in which
FL model training is switched to the training-aggregation mode. It consists of
a single round of training with more local epochs with FL model aggregation
being executed at the end of each epoch. In this way, the global FL model is
fine-tuned through more frequent aggregations in order to improve its
performance.

  4. 4.

Model Aggregation: When the retained blocks from a client $i$ in round $r$ is
received, the FL server combines them with unchanged (i.e., not transmitted)
blocks from the previous global model $\boldsymbol{w}_{r-1}$ to reconstruct
$\boldsymbol{w}_{r,i}$. The reconstructed local model updates for all clients
are then aggregated into a new global FL model $\boldsymbol{w}_{r}$ following
FedAvg (McMahan et al. 2017).

Steps 1 to 4 are repeated until model convergence. For more details about
FedOBD, please refer to (Chen et al. 2022).

## Application Development and Deployment

The FedOBD framework has been developed based on the PyTorch (Paszke et al.
2017) framework by teams from the Trustworthy Federated Ubiquitous Learning
(TrustFUL) Lab, Nanyang Technological University (NTU), Singapore, and the
Digital Research Institute, ENN Group, Beijing, China. Before deploying the AI
Engine, we have evaluated it against the classic FedAvg and four state-of-the-
art efficient FL model training approaches. They are:

  1. 1.

FedAvg (McMahan et al. 2017): It is a classic FL approach which does not
involve any compression or dropout operation.

  2. 2.

SignSGD (Bernstein et al. 2018): It is a distributed gradient compression
approach which only requires the signs of the gradients of client model
updates to be sent to the FL server. The server aggregates the gradients by
majority voting.

  3. 3.

FedPAQ (Reisizadeh et al. 2020): It is a stochastic quantization-based FL
approach designed to reduce communication overhead.

  4. 4.

Adaptive Federated Dropout (AFD) (Bouacida et al. 2021): It is an FL approach
that optimizes server-client communications and computation costs jointly.
Each client trains a selected subset of the global model parameters. We adopt
the Single-Model Adaptive Federated Dropout (SMAFD) variant for comparison.

  5. 5.

FedDropoutAvg (Gunesli et al. 2021): It is an FL approach that randomly drops
out a subset of model parameters as well as randomly drops out some clients
before performing FedAvg.

To compare the efficiency and performance of these approaches under different
FL settings, we designed FL scenarios involving 10 clients to perform image
classification on the CIFAR-10 datasets (Krizhevsky, Hinton et al. 2009) and
sentiment classification on the IMDB dataset (Maas et al. 2011). The original
test data are split uniformly to form separate validation and test datasets.
The local training and validation dataset of each client are sampled following
an i.i.d. setting. The FL server holds the test dataset. For the image
classification tasks under CIFAR-10, we use FL to train a DenseNet-40 (Huang
et al. 2017) base model which contains around 190,000 model parameters. For
the sentiment classification task under IMDB, we use FL to train a Transformer
based classification model consisting of 2 encoder layers followed by a linear
layer with around 17 million model parameters.

| CIFAR-10 | IMDB
---|---|---
| Communication | Test | Communication | Test
| Overhead (MB) | Accuracy | Overhead (MB) | Accuracy
FedAvg | 1,467.30 | 89.36% | 131,494.20 | 84.68%
SignSGD | 7,208.75 | 60.18% | 327,100.00 | 50.21%
FedPAQ | 463.54 | 89.22% | 41,541.05 | 82.94%
SMAFD | 522.23 | 17.38% | 63,365.16 | 63.94%
FedDropoutAvg | 512.20 | 87.53% | 43,992.33 | 84.17%
FedOBD | 101.59 | 90.17% | 12,899.03 | 84.98%

Table 1: Pre-deployment experiment results.

The results are summarized in Table 1. Communication overhead is computed as
the product between the average fraction of model size (in MBs) transmitted
per FL training step and the total number of FL training steps required. It
can be observed that FedOBD significantly outperforms all existing approaches
in terms of reducing communication overhead. In terms of test accuracy of the
resulting model, FedOBD significantly outperforms existing compression or
dropout-based efficient FL training approaches, achieving comparable
performance to FedAvg which does not engage in any model compression or
dropout. The results helped the design team make the decision to adopt FedOBD
as the FL approach for training large-scale industrial fault diagnostic models
in the ENN platform.

Figure 8: The architecture of the fault prediction deep neural network model.

FedOBD is deployed into the ENN Client-Server FL Subsystem as an alternative
FL training approach the system administrators can choose to use. It is added
into the user interface as a new option that can be selected during FL
training configuration (Figure 3). Once FedOBD is selected as the FL training
approach from the user interface, the model reconstruction method, the global
model aggregation method and the the block dropout and model distribution
method under FedOBD are loaded into the FL server. The intermediate model logs
module is also included in the FL server to store the global model from the
previous round to facilitate the evaluation of block importance by FedOBD, as
illustrated in Figure 7 (within the dashed rectangle A). At the same time, the
model reconstruction method, the 2-stage model training method and the the
block dropout and model uploading method under FedOBD are loaded into the FL
Client Agent. Similarly, the intermediate model logs module is included in the
FL Client Agent to store the global model from the previous round to
facilitate the evaluation of block importance by FedOBD, as illustrated in
Figure 7 (within the dashed rectangle B). In this way, FedOBD is incorporated
into the ENN FL platform.

## Application Use and Payoff

FedOBD has been deployed in ENN Group since February 2022 as part of its
intelligent industrial fault diagnostics platform. Since its deployment, the
company has switched from the original FedAvg based FL model training and
aggregation approach to FedOBD for part of its enterprise customers. So far,
FedOBD has been used to help two well-established coal chemical plants located
in two cities in China to train AI models for industrial fault
prediction.333At the request of our industry partners, the identities of these
factories are withheld.

Figure 8 illustrates the architecture of the DNN adopted by the ENN Group for
fault prediction in the current deployment cycle. The model consists of a
variety of vibration signal analysis methods to preprocess the original
vibration signals to improve the interpretability of the model. Among them,
the spectrum and the axis trajectory can show the frequency doubling component
related to the rotating speed in the signal, which is helpful for diagnosing
shaft faults. Envelope, continuous wavelet transform (CWT) and wavelet packet
transform (WPT) can extract high-frequency impact components from the signal,
which is helpful for diagnosing bearing and gearbox related faults. The
attention block is used to adjust the importance of the features from
different input blocks. At the end, two ResNet blocks are used to diagnose the
two major types of faults, respectively. The model contains 29 million model
parameters.

FedOBD is configured with the same hyperparameters as the previous FedAvg FL
model training approach in the system. Specifically, a local learning rate of
$10^{-5}$ is used in the ENN FL Clients. A quantization weight of $0.01$ is
used for quantizing uploaded and distributed models. Only a single local epoch
is used in FedAvg and the first stage of FedOBD. For the second training stage
of FedOBD, two local epochs are used.

ENN FL Training | Communication | Test
---|---|---
Approach | Overhead (MB) | F1 Score
Previous AI Engine | $368,407.60$ | $85.52\pm 0.83\%$
FedOBD AI Engine | $104,188.50$ | $85.02\pm 0.23\%$

Table 2: Deployment results.

Table 2 shows the average communication overhead of training the fault
prediction model until convergence for each round of model update, as well as
the average test F1 score of the resulting models thus far into the deployment
period under FedOBD. The results under the Previous AI Engine reflects the
same items but were collected during system operations in 2021 prior to
switching to FedOBD. It can be observed that FedOBD saves communication
overhead by $71.72\%$, while achieving comparable model performance. Due to
the 2 MB/sec bandwidth usage limit placed on FL model training related
communications by ENN Group, under the previous AI Engine, it took more than 2
days (around 52 hours) to train an updated version of the fault prediction
model involving the two factories. Under the FedOBD AI Engine, this time is
reduced to about half a day (around 14.5 hours).

As new data generated by the equipment monitoring sensing devices are
continually accumulated over time from and new equipment can be deployed in
the factories from time to time, it is necessary to frequently retrain the
fault prediction model via FL to keep it update to date. The deployment of
FedOBD has cut down the model training time by more than half while
maintaining model performance, thereby enabling timely retraining of the model
for enhanced safety and efficiency of operation.

## Maintenance

The AI Engine follows a modular design approach to achieve separation of
concerns. Thus far into the deployment period, although there have been
changes in personnel access rights and operating parameters in the system as
well as frequent retraining of the fault prediction model via FedOBD, such
changes have not necessitated any AI maintenance task for the AI Engine.

## Lessons Learned During Deployment

During the process of deploying the FedOBD approach, there are several lessons
worth sharing.

Firstly, the quality of training data is important to training effective
industrial fault prediction models. The data cleaning and preprocessing steps
by each participating data owner play an important role in model training. As
data preprocessing is still mainly performed based on human experience, it can
be expensive and prone to human errors. Since not all industry data owners
have an in-house data science team, it could be a challenge to obtain
preprocessed local data with consistently high quality. Thus, it could be
useful to adopt privacy-preserving data selection approaches such as (Li et
al. 2021) to automate this process.

Secondly, as industrial fault prediction is an important application with high
impact on operation safety and continuity, industry partners prefer some
degrees of model interpretability. Although there exist parameter dropout-
based FL approaches that can improve training efficiency (Bouacida et al.
2021; Gunesli et al. 2021), their lack of interpretability on parameter
dropout decisions hinders industry adoption. The block importance values
produced during the intermediate steps of FedOBD are helpful in providing the
decision-makers with much needed transparency to alleviate such concerns.

Last but not least, as time goes by, new monitoring data from the
participating factories will continue to be generated. Previously
unencountered fault types may emerge. In addition, the data distribution might
also change, resulting in concept drift (Lu et al. 2018). These factors can be
especially pronounced when new machines are incorporated into the factories.
Thus, the performance of previously trained FL models can deteriorate in the
face of these factors. Therefore, appropriate incremental training and
updating strategies of the FL model need to be put in place to ensure
successful deployment.

## Conclusions and Future Work

In this paper, we reported on our experience using a dropout-based technique
to enhance efficient collaborative training of large-scale deep models through
federated learning for industrial fault diagnostic models involving multiple
factories. We developed the FedOBD FL model training and aggregation approach,
which leverages a novel opportunistic importance-based semantic block dropout
method in combination with quantization-based FL model parameter compression
to drastically reduce communication overhead while preserving model
performance. Since its deployment in February 2022 in ENN Group, FedOBD has
helped two well-established coal chemical plants in two cities in China to
train machine learning models for fault diagnostics in order to support
predictive maintenance, and has made significant positive impact on ENN
Group’s operations.

In future, we will enhance the robustness of FedOBD against malicious FL
participants (Lyu et al. 2020). We will also explore how to link block
importance evaluation with FL client contribution evaluation to enhance
fairness (Shi, Yu, and Leung 2021), as well as personalizing the resulting
models (Tan et al. 2022) to FL participants. Eventually, we aim to incorporate
FedOBD into an opensource FL framework such as Federated AI Technology Enabler
(FATE) (Liu et al. 2021) and make it available to more developers, researchers
and practitioners.

## Acknowledgements

This research is supported, in part, by the National Research Foundation,
Singapore under its AI Singapore Programme (AISG Award No: AISG2-RP-2020-019);
the RIE 2020 Advanced Manufacturing and Engineering (AME) Programmatic Fund
(No. A20G8b0102), Singapore; Nanyang Assistant Professorship (NAP); and Future
Communications Research & Development Programme (FCP-NTU-RG-2021-014). Any
opinions, findings and conclusions or recommendations expressed in this
material are those of the author(s) and do not reflect the views of National
Research Foundation, Singapore.

## References

  * Alistarh et al. (2017) Alistarh, D.; Grubic, D.; Li, J.; Tomioka, R.; and Vojnovic, M. 2017.  QSGD: Communication-efficient SGD via gradient quantization and encoding.  _NeurIPS_ , 30: 1709–1720.
  * Bernstein et al. (2018) Bernstein, J.; Wang, Y.-X.; Azizzadenesheli, K.; and Anandkumar, A. 2018.  SignSGD: Compressed optimisation for non-convex problems.  In _ICML_ , 560–569.
  * Bouacida et al. (2021) Bouacida, N.; Hou, J.; Zang, H.; and Liu, X. 2021.  Adaptive Federated Dropout: Improving Communication Efficiency and Generalization for Federated Learning.  In _INFOCOM Workshops_ , 1–6.
  * Chen et al. (2022) Chen, Y.; Chen, Z.; Wu, P.; and Yu, H. 2022.  FedOBD: Opportunistic Block Dropout for Efficiently Training Large-scale Neural Networks through Federated Learning.  _arXiv preprint arXiv:2208.05174_.
  * Cong et al. (2020) Cong, M.; Yu, H.; Weng, X.; Qu, J.; Liu, Y.; and Yiu, S. M. 2020.  A VCG-based Fair Incentive Mechanism for Federated Learning.  _arXiv preprint arXiv:2008.06680_.
  * GDPR (2018) GDPR. 2018.  General data protection regulation.  https://gdpr-info.eu/.  Accessed: 2021-12-08.
  * Geng et al. (2022) Geng, D.; He, H.; Lan, X.; and Liu, C. 2022.  Bearing fault diagnosis based on improved federated learning algorithm.  _Computing_ , 104: 1–19.
  * Ghobakhloo (2020) Ghobakhloo, M. 2020.  Industry 4.0, digitization, and opportunities for sustainability.  _Journal of Cleaner Production_ , 252: doi:10.1016/j.jclepro.2019.119869.
  * Gunesli et al. (2021) Gunesli, G. N.; Bilal, M.; Raza, S. E. A.; and Rajpoot, N. M. 2021.  FedDropoutAvg: Generalizable federated learning for histopathology image classification.  _arXiv preprint arXiv:2111.13230_.
  * Huang et al. (2017) Huang, G.; Liu, Z.; Maaten, L. V. D.; and Weinberger, K. Q. 2017.  Densely connected convolutional networks.  In _CVPR_ , 4700–4708.
  * Kairouz, McMahan et al. (2021) Kairouz, P.; McMahan, H. B.; et al. 2021.  Advances and open problems in federated learning.  _Foundations and Trends in Machine Learning_ , 14(1-2): 1–210.
  * Krizhevsky, Hinton et al. (2009) Krizhevsky, A.; Hinton, G.; et al. 2009.  Learning multiple layers of features from tiny images.
  * Li et al. (2021) Li, A.; Zhang, L.; Wang, J.; Tan, J.; Han, F.; Qin, Y.; Freris, N. M.; and Li, X.-Y. 2021.  Efficient Federated-Learning Model Debugging.  In _ICDE_ , 372–383.
  * Liu et al. (2022a) Liu, C.; Cichon, A.; Królczyk, G.; and Li, Z. 2022a.  Technology development and commercial applications of industrial fault diagnosis system: a review.  _International Journal of Advanced Manufacturing Technology_ , 118: 3497–3529.
  * Liu et al. (2021) Liu, Y.; Fan, T.; Chen, T.; Xu, Q.; and Yang, Q. 2021.  FATE: An Industrial Grade Platform for Collaborative Learning With Data Protection.  _Journal of Machine Learning Research_ , 22(226): 1–6.
  * Liu et al. (2020) Liu, Y.; Huang, A.; Luo, Y.; Huang, H.; Liu, Y.; Chen, Y.; Feng, L.; Chen, T.; Yu, H.; and Yang, Q. 2020.  FedVision: An Online Visual Object Detection Platform Powered by Federated Learning.  In _IAAI_ , 13172–13179.
  * Liu et al. (2022b) Liu, Z.; Chen, Y.; Zhao, Y.; Yu, H.; Liu, Y.; Bao, R.; Jiang, J.; Nie, Z.; Xu, Q.; and Yang, Q. 2022b.  Contribution-Aware Federated Learning for Smart Healthcare.  In _IAAI_ , 12396–12404.
  * Long et al. (2020) Long, G.; Tan, Y.; Jiang, J.; and Zhang, C. 2020.  Federated Learning for Open Banking.  In Yang, Q.; Fan, L.; and Yu, H., eds., _Federated Learning: Privacy and Incentive_ , 240–254. Springer.
  * Lu et al. (2018) Lu, J.; Liu, A.; Dong, F.; Gu, F.; Gama, J.; and Zhang, G. 2018.  Learning under Concept Drift: A Review.  _IEEE Transactions on Knowledge and Data Engineering_ , 31(12): 2346–2363.
  * Lyu et al. (2020) Lyu, L.; Yu, H.; Ma, X.; Sun, L.; Zhao, J.; Yang, Q.; and Yu, P. S. 2020.  Privacy and Robustness in Federated Learning: Attacks and Defenses.  _arXiv preprint arXiv:2012.06337_.
  * Ma, Wen, and Wen (2021) Ma, X.; Wen, C.; and Wen, T. 2021.  An Asynchronous and Real-Time Update Paradigm of Federated Learning for Fault Diagnosis.  _IEEE Transactions on Industrial Informatics_ , 17(12): 8531–8540.
  * Maas et al. (2011) Maas, A. L.; Daly, R. E.; Pham, P. T.; Huang, D.; Ng, A. Y.; and Potts, C. 2011\.  Learning Word Vectors for Sentiment Analysis.  In _ACL_ , 142–150.
  * McMahan et al. (2017) McMahan, B.; Moore, E.; Ramage, D.; Hampson, S.; and y Arcas, B. A. 2017.  Communication-Efficient Learning of Deep Networks from Decentralized Data.  In _AISTATS_ , 1273–1282.
  * Paszke et al. (2017) Paszke, A.; Gross, S.; Chintala, S.; Chanan, G.; Yang, E.; DeVito, Z.; Lin, Z.; Desmaison, A.; Antiga, L.; and Lerer, A. 2017.  Automatic differentiation in PyTorch.  In _NIPS-W_.
  * Reisizadeh et al. (2020) Reisizadeh, A.; Mokhtari, A.; Hassani, H.; Jadbabaie, A.; and Pedarsani, R. 2020\.  FedPAQ: A communication-efficient federated learning method with periodic averaging and quantization.  In _AISTATS_ , 2021–2031.
  * Shi, Yu, and Leung (2021) Shi, Y.; Yu, H.; and Leung, C. 2021.  Towards Fairness-Aware Federated Learning.  _arXiv preprint arXiv:2111.01872_.
  * Tan et al. (2022) Tan, A. Z.; Yu, H.; Cui, L.; and Yang, Q. 2022.  Towards personalized federated learning.  _IEEE Transactions on Neural Networks and Learning Systems_ , doi:10.1109/TNNLS.2022.3160699.
  * Wang et al. (2022) Wang, X.; Garg, S.; Lin, H.; Hu, J.; Kaddoum, G.; and Piran, M. J. 2022.  Toward Accurate Anomaly Detection in Industrial Internet of Things Using Hierarchical Federated Learning.  _IEEE Internet of Things Journal_ , 9(10): 7110–7119.
  * Warnat-Herresthal et al. (2021) Warnat-Herresthal, S.; Schultze, H.; Shastry, K. L.; and et al. 2021.  Swarm Learning for decentralized and confidential clinical machine learning.  _Nature_ , 594: 265–270.
  * Yang et al. (2019) Yang, Q.; Liu, Y.; Cheng, Y.; Kang, Y.; Chen, T.; and Yu, H. 2019.  _Federated Learning_.  Morgan & Claypool Publishers.
  * Yu et al. (2020) Yu, H.; Liu, Z.; Liu, Y.; Chen, T.; Cong, M.; Weng, X.; Niyato, D.; and Yang, Q. 2020.  A Fairness-aware Incentive Scheme for Federated Learning.  In _AIES_ , 393–399.
  * Zhang et al. (2021) Zhang, W.; Li, X.; Ma, H.; Luo, Z.; and Lie, X. 2021.  Federated learning for machinery fault diagnosis with dynamic validation and self-supervision.  _Knowledge-Based Systems_ , 213: doi:10.1016/j.knosys.2020.106679.