# Non-adiabatic dynamics of the entanglement entropy in a symmetry-breaking
Haldane insulator

Junjun Xu  Institute of Theoretical Physics, University of Science and
Technology Beijing, Beijing 100083, China<EMAIL_ADDRESS>
###### Abstract

We study the non-adiabatic dynamics of a typical symmetry-protected
topological phase-the Haldane insulator phase with broken bond-centered
inversion. By continuously breaking the middle chain, we find the gap closes
at a critical point in the deep Haldane insulator regime with a change of
particle number partition of the left or right system. The adiabatic evolution
fails at this critical point and we show how to predict the dynamics of the
entanglement entropy near this point using a two-level model. These results
show that one can find a critical regime where the entanglement measurement is
relatively robust against perturbation that breaks the protecting symmetries
in the Haldane insulator. This is in contrast to the common belief that the
symmetry-protected topological phases are fragile without the protecting
symmetries.

##  1 Introduction

The robustness of symmetry-protected topological (SPT) phases to symmetry-
respecting perturbations makes them promising candidates for quantum computing
[1]. Recently, the topological characteristics of these phases have been
extended to out-of-equilibrium regime [2], and more recent studies show that
their topologies might be robust even when the initial state breaks the
protecting symmetries [3]. Thus it would be of both theoretical and technical
interests to understand to what extent these phases are robust during
dynamical evolution.

In this paper, we try to focus on this question: how robust the entanglement
measurement is if we not only break the protecting symmetries of the initial
state, but also the time-dependent Hamiltonian? Naively, one might imagine the
SPT state will adiabatically evolve to a trivial state which eliminates the
entanglement. However, we will show that one might find a critical point where
this adiabatic evolution fails, thus leads to relatively robust entanglement
measurement.

We consider the Haldane phase in a more controllable cold atoms system, which
was firstly considered by Dalla Torre et al., termed as the Haldane insulator
(HI), and the phase diagram has been well studied [4, 5]. This HI phase shows
non-trivial properties similar to the Haldane phase in spin-1
antiferromagnetic Heisenberg chains [6, 7] (the non-local string order, a two-
fold degeneracy of the entanglement spectrum, etc.), but with a revised
protecting bond-centered (BC) inversion symmetry [8, 9, 10, 11, 12]. Recent
theoretical progress has shed light on realizing this bosonic HI phase by
providing strong enough long-range interaction using Feshbach resonance [13].

In the following, we will consider the middle chain-breaking dynamics of the
HI phase with broken BC inversion. This dynamical process has been considered
by Pollmann et al. in the spin-1 chain respecting the BC inversion symmetry,
and they find a lower bound ($\log 2$) for the half-system von Neumann
entanglement entropy by adiabatically breaking the middle chain. This
entanglement measurement is due to the double degeneracy of the entanglement
spectrum, and is robust with symmetries even not protecting the edge modes and
string order [10]. In this work, we will focus on the entanglement entropy,
and show to what extent this measurement is robust. To our knowledge, this
problem has not been fully understood before.

This paper is organized as follows. First, we introduce the middle chain-
breaking experiment in an extended Bose-Hubbard system and consider its
adiabatic properties. We show the broken system has a vanishing energy gap in
the deep HI phase regime, as a result of the degeneracy of two trivial product
states appear in this phase. Then we consider the non-adiabatic dynamics. We
get the full numerical evolution of the entanglement entropy by integrating
the time-dependent Schrödinger equation. We find the entanglement entropy is
relatively robust in the deep HI regime. We further give an analytical
prediction to the entanglement entropy in this regime by mapping the system to
a two-level model. At last, we give our conclusions and mention the relevant
experiment measurements.

##  2 The middle-chain breaking and its adiabatic properties

We consider a chain-breaking experiment on the extended Bose-Hubbard model
$H=H_{0}+\gamma H^{\prime}$ with

$\displaystyle H_{0}=$ $\displaystyle-J\sum_{i\neq
L/2}(b^{\dagger}_{i}b_{i+1}+\mathrm{H.c.})+\frac{U}{2}\sum_{i}n_{i}(n_{i}-1)$
$\displaystyle+V\sum_{i\neq L/2}n_{i}n_{i+1},$ $\displaystyle H^{\prime}=$
$\displaystyle-J(b^{\dagger}_{L/2}b_{L/2+1}+\mathrm{H.c.})+Vn_{L/2}n_{L/2+1}$
(1)

describing the broken and linking Hamiltonian. We consider a chain length of
$L$. Here $J$ characterizes the nearest-neighbor hopping, and $U$, $V$ are the
on-site and nearest-neighbor interaction strength, with
$n_{i}=b_{i}^{\dagger}b_{i}$ the bosonic particle number operator at site $i$.
The parameter $\gamma\in[0,1]$ characterizes the strength of middle bond. At
$\gamma=1$, we recover the conventional extended Bose-Hubbard model. At
$\gamma=0$ this chain breaks into two length-$L/2$ subsystems.

In this work, we run an exact diagonalization calculation with chain length
$L=12$. This is a typical system size can be realized in current cold atoms
experiments [14]. To observe the HI phase, we constrain the maximum on-site
particle number to 2, the state space of which can then be mapped to an
effective spin-1 system. Experimentally, this constraint can be achieved by
appropriately include the Feshbach resonances in this system as recently
proposed in [13]. We consider a half-filling case with total particle number
$N=L$, which is equivalent to the total spin sector $S_{tot}^{z}=0$ in the
mapped spin system. We fix the edge particle number to be $2$ and $0$ in the
following calculation, which helps to break the ground state degeneracy in the
HI and density wave phases, and reducing the particle-hole excitations at the
edge. In experiment this can be done by reducing or increasing the local
potential at the left or right edge of the optical lattice. Such edge
configuration has also been considered in previous studies, and the HI is
found sandwiched between Mott insulator and density wave phases (for on-site
interaction $U/J=4.0$ the HI phase appears in the range $2.1\lesssim
V/J\lesssim 3.0$) [4, 5, 12].

Figure 1: Chain-breaking experiment of an open Haldane insulator. (a) An
Affleck-Kennedy-Lieb-Tasaki (AKLT) state description for middle chain-
breaking, where the black dots represent effective spin-1/2 particles. By
adiabatically breaking the middle bond, the AKLT state evolves to one of the
trivial states as the ground state. (b) The energy spectrum after breaking the
middle chain for different nearest-neighbor interaction strength $V$. The
lowest level crossing happens at around $V/J\approx 2.4$. (c) Ground state
particle number of the left-half system $N_{L}$ after breaking the middle
chain for different $V$. The particle number shifts at the critical point
$V/J\approx 2.4$, which is within the Haldane insulator regime (shaded region
in d and e). The energy gap after and before breaking the middle chain is
shown in (d) and (e). The results are obtained for $U/J=4$ with the chain
length $L=60/120$ based on DMRG calculations using ALPS package [15, 16, 17].

We note that even though the above choice of edge configuration stabilizes the
HI state, it in fact has already broken the BC inversion symmetry. Thus even
though the linking Hamiltonian $H^{\prime}$ conserves BC inversion, the system
loses its protecting symmetry all along with the time evolution. So by
breaking the middle chain, adiabatically the HI will evolve to a trivial
product state (here by “product state” we mean the state is composed of left
half-system state producting the right half-system state). However, since
there are two product states competing as illustrated in Fig. 1a, this
adiabatic evolution will not be valid if these two states are degenerate. We
show the energy spectrum for the lowest $40$ eigenstates in Fig. 1b. We find a
ground state level crossing around $V/J\approx 2.4$, which is due to the
competition of these two product states. This is consistent with the particle
number partition in the left-half system in Fig. 1c. Obviously, these two
product states are adiabatically connected to the Mott insulator and density
wave states respectively by changing the nearest-neighbor interaction $V/J$.
In Fig. 1d and e we show the energy gap after and before breaking the middle
chain. As the chain length increases, the level crossing extends to the full
HI regime as shown in Fig. 1d (shaded region), with the regime coincides with
Fig. 1e. These findings suggest that one can find regimes within the deep HI
regime where the adiabatic evolution fails, and thus protect the non-trivial
state even without the BC inversion symmetry. In the following, we will focus
on this regime and try to understand this non-adiabatic dynamics.

##  3 Non-adiabatic dynamics of the entanglement entropy

###  3.1 Full time evolution

We consider the full time evolution in this section, to show whether the
experiment is able to follow the above adiabatic processes. We consider the
dynamical breaking of middle chain which takes a linear form
$\gamma(t)=1-\Gamma t$, where $\Gamma$ characterizes the rate of this
breaking. The time-dependent Hamiltonian can be written as
$H(t)=H_{0}+(1-\Gamma t)H^{\prime}$. The wave function can be expanded using
the orthogonal eigenstates at $t=0$,

$|\psi(t)\rangle=\sum_{n}c_{n}(t)e^{-iE_{n}^{0}t}|\psi_{n}^{0}\rangle,$ (2)

where $|\psi_{n}^{0}\rangle$ and $E_{n}^{0}$ are the eigenvector and
eigenvalues of the system at $t=0$, i.e.,
$(H_{0}+H^{\prime})|\psi_{n}^{0}\rangle=E_{n}^{0}|\psi_{n}^{0}\rangle$. Under
Eq. (2), the Schrödinger equation becomes a set of coupled equations about the
coefficients $c_{n}(t)$,

$i\dot{c}_{n}(t)=-\Gamma t\sum_{m}\langle n|H^{\prime}|m\rangle
c_{m}(t)e^{-i(E_{m}^{0}-E_{n}^{0})t}.$ (3)

With the initial condition $c_{n}(0)=\delta_{n,0}$ the above equations give
the full time evolution of the system.

The more important and experimentally relevant quantity is the half-system von
Neumann entanglement entropy
$S_{L/2}=-\mathrm{Tr}\left[\rho_{l}\log\rho_{l}\right]$, where
$\rho_{l}=\mathrm{Tr}_{r}|\psi(t)\rangle\langle\psi(t)|$ is the reduced
density matrix of the left-half system with the trace over the right-half
system. We show the final-state entanglement entropy $S_{L/2}$ as a function
of inverse breaking rate $1/\Gamma$ and nearest-neighbor interaction $V$ in
Fig. 2a. We pick three typical values $V/J=1.4,2.4,3.4$ at breaking rate
$\Gamma/J=0.1$ and show the dynamics of $S_{L/2}$ in Fig. 2b. At the deep HI
regime around $V/J\approx 2.4$, we find the entanglement entropy is relatively
robust and evolves to a finite value.

Figure 2: (a) The final-state half entanglement entropy $S_{L/2}$ at
$t=1/\Gamma$ as a function of nearest-neighbour interaction $V$ and inverse
breaking rate $1/\Gamma$ for on-site interaction $U/J=4$ and chain length
$L=12$. (b) Full time evolution of $S_{L/2}$ for different nearest-neighbor
interaction $V/J=1.4,2.4,3.4$ at breaking rate $\Gamma/J=0.1$. The dark points
in (a) correspond to the data in (b) at $t=10/J$. The entanglement entropy is
relatively robust in the deep Haldane insulator regime at around $V/J\approx
2.4$.

###  3.2 A two-level prediction

The above full numerical calculations are time consuming and lack of physical
insights. Since there are two nearly degenerate states for the broken
Hamiltonian $H_{0}$ in the deep HI regime, we can describe the physics by a
two-level model. We label these two states as $|0\rangle$ and $|1\rangle$, and
the system can then be projected into these two subspaces as

$\displaystyle H_{eff}(t)=\left[\begin{array}[]{cc}\langle
0|H|0\rangle&\langle 0|H|1\rangle\\\ \langle 1|H|0\rangle&\langle
1|H|1\rangle\end{array}\right].$ (6)

We define $\langle 1|H|1\rangle-\langle 0|H|0\rangle=h-2\beta(1-\Gamma t)$,
$\langle 0|H|1\rangle=\alpha(1-\Gamma t)$, where $h=\langle
1|H_{0}|1\rangle-\langle 0|H_{0}|0\rangle$ is the gap of $H_{0}$ between these
two states. Here $\alpha=\langle 0|H^{\prime}|1\rangle$ and $\beta=(\langle
0|H^{\prime}|0\rangle-\langle 1|H^{\prime}|1\rangle)/2$ come from the hopping
and nearest-neighbor interaction terms respectively. In general, $\alpha$
could be a complex number, but we can always drop the complex phase factor
into state $|0\rangle$ or $|1\rangle$ and makes $\alpha$ real. Thus in the
following we choose $\alpha$ to be its modulus. The physics should not change
if we shift to a rotating frame by a unitary transformation $U=e^{i\langle
0|H_{0}|0\rangle t+i\left(\langle 0|H^{\prime}|0\rangle+\langle
1|H^{\prime}|1\rangle\right)\left(t-\Gamma t^{2}/2\right)/2}$, and the above
Hamiltonian becomes more symmetric as

$\displaystyle\tilde{H}_{eff}(t)=H_{eff}+i\dot{U}U^{\dagger}=\left[\begin{array}[]{cc}\beta(1-\Gamma
t)&\alpha(1-\Gamma t)\\\ \alpha(1-\Gamma t)&-\beta(1-\Gamma
t)+h\end{array}\right].$ (9)

Let’s first look at the asymptotic behavior at $h/J\to 0$ (which is true in
the thermodynamic limit of the HI phase and also for the critical point in our
finite system). Under this condition, the time evolution of the Hamiltonian
Eq. (9) is readily solved and the solution gives

$\displaystyle\psi_{+}(t)=\frac{e^{i\omega(t-\Gamma
t^{2}/2)}}{\sqrt{\alpha^{2}+(\beta-\omega)^{2}}}\left(\begin{array}[]{c}\beta-\omega\\\
\alpha\end{array}\right)$ (12)

for initial state as the ground state, and

$\displaystyle\psi_{-}(t)=\frac{e^{-i\omega(t-\Gamma
t^{2}/2)}}{\sqrt{\alpha^{2}+(\beta+\omega)^{2}}}\left(\begin{array}[]{c}\beta+\omega\\\
\alpha\end{array}\right)$ (15)

for initial state as the first excited state. Here
$\omega=\sqrt{\alpha^{2}+\beta^{2}}$. Thus it is obvious that besides a
dynamical phase, the probabilities of the system in these two states
$|0\rangle$ and $|1\rangle$ are constant with time. It means in the limit
$h/J\to 0$, the reduced density matrix
$\rho^{\pm}_{l}=\mathrm{Tr}_{r}|\psi_{\pm}(t)\rangle\langle\psi_{\pm}(t)|$
does not evolve with time and thus the half entanglement entropy of each state
$\psi_{\pm}(t)$ is a constant and robust to the middle-bond breaking.

For finite $h$, we consider the long-time nearly adiabatic limit $\Gamma/J\to
0$. In this case we have a rather slow breaking rate and the dynamics is
mainly determined by the details of time $t$ close to $1/\Gamma$. So the
physics is determined by the region $\beta(1-\Gamma t)\ll h$, and we have the
following Hamiltonian

$\displaystyle H_{ad}(t)=\left[\begin{array}[]{cc}0&\alpha(1-\Gamma t)\\\
\alpha(1-\Gamma t)&h\end{array}\right].$ (18)

To get the dynamics of $H_{ad}$, we try to expand the solution as a
superposition of orthogonal states Eq. (12) and (15) with $\beta=0$,

$\displaystyle\psi_{ad}(t)$
$\displaystyle=c_{+}(t)\psi_{+}(t)+c_{-}(t)\psi_{-}(t)$ (23)
$\displaystyle=c_{+}(t)\frac{e^{i\theta(t)}}{\sqrt{2}}\left(\begin{array}[]{c}-1\\\
1\end{array}\right)+c_{-}(t)\frac{e^{-i\theta(t)}}{\sqrt{2}}\left(\begin{array}[]{c}1\\\
1\end{array}\right),$

where $\theta(t)=\alpha(t-\Gamma t^{2}/2)$. The time-dependent Schrödinger
Equation then becomes

$\displaystyle i\dot{c}_{+}e^{i\theta}-i\dot{c}_{-}e^{-i\theta}$
$\displaystyle=0,$ (24) $\displaystyle
i\dot{c}_{+}e^{i\theta}+i\dot{c}_{-}e^{-i\theta}$
$\displaystyle=hc_{+}e^{i\theta}+hc_{-}e^{-i\theta},$ (25)

which can be reduced to the following second-order differential equation

$\ddot{c}_{+}+i\left[h+2\alpha(1-\Gamma t)\right]\dot{c}_{+}-\alpha h(1-\Gamma
t)c_{+}=0,$ (26)

with

$\displaystyle c_{-}=e^{2i\theta}\left(\frac{2i}{h}\dot{c}_{+}-c_{+}\right).$
(27)

The solution of Eq. (26) has an additional phase term. This can be seen by
setting $t\to\pm\infty$, and Eq. (26) becomes $2i\dot{c}_{+}-hc_{+}=0$ with
the solution $c_{+}(t\to\pm\infty)=e^{-iht/2}$. This phase can be removed by
defining

$\displaystyle c_{+}(t)=e^{-iht/2}\tilde{c}_{+}(t),$ (28)

and Eq. (26) can then be reduced to

$\ddot{\tilde{c}}_{+}+2i\alpha(1-\Gamma
t)\dot{\tilde{c}}_{+}+h^{2}\tilde{c}_{+}/4=0.$ (29)

By defining $z=-e^{i\pi/4}(1-\Gamma t)\sqrt{\alpha/\Gamma}$ and
$\nu=h^{2}/(\Gamma\alpha)$, the above equation is equivalent to the Hermite
differential equation

$\ddot{\tilde{c}}_{+}(z)-2z\cdot\dot{\tilde{c}}_{+}(z)-i\nu/4\cdot\tilde{c}_{+}(z)=0,$
(30)

the general solution of which is readily written as a linear combination of
Hermite polynomial $H_{\lambda}(z)$ and confluent hypergeometric function
$M(\lambda_{1},\lambda_{2},z)$ [20]

$\tilde{c}_{+}(z)=aH_{-i\nu/8}\left(z\right)+bM\left(\frac{i\nu}{16},\frac{1}{2},z^{2}\right),$
(31)

where $a$ and $b$ are constants to be determined by the initial condition.
Since we are interested in the nearly adiabatic side, we have the initial
condition

$\displaystyle\left|\tilde{c}_{+}\left(z\left(t\to-\infty\right)\right)\right|=1,|\tilde{c}_{-}\left(z\left(t\to-\infty\right)\right)|=0.$
(32)

Consider the asymptotic behaviors
$H_{\lambda}(z\left(t\to-\infty\right))=2^{\lambda}z^{\lambda}$,
$M(\lambda_{1},\lambda_{2},z\left(t\to-\infty\right))=\Gamma(\lambda_{2})(e^{z}z^{\lambda_{1}-\lambda_{2}}/\Gamma(\lambda_{1})+(-z)^{-\lambda_{1}}/\Gamma(\lambda_{2}-\lambda_{1}))$,
and plug these into the initial condition Eq. (32) we have the coefficients

$\displaystyle
a=-2^{\frac{i\nu}{8}}e^{-\frac{\pi\nu}{32}},b=\frac{e^{-\frac{3\pi\nu}{32}}}{\sqrt{\pi}}\left(1+e^{\frac{\pi\nu}{8}}\right)\Gamma\left(\frac{1}{2}-\frac{i\nu}{16}\right).$

Then from Eq. (27) and (28) we obtain the time-dependent solution of
Hamiltonian $H_{ad}$.

Since the two nearly degenerate states $|0\rangle$ and $|1\rangle$ at
$t=1/\Gamma$ are trivial product sates and have different particle number
partition for the left and right half systems, they contribute to the
entanglement entropy only by their probabilities. According to the above
solution, we arrive the final probability at the lowest energy state
$|0\rangle$

$\displaystyle P_{0}$
$\displaystyle=\left|c_{+}(1/\Gamma)e^{i\theta}-c_{-}(1/\Gamma)e^{-i\theta}\right|^{2}/2$
$\displaystyle=\frac{e^{-\frac{3\pi\nu}{16}}}{2\pi}\cdot\left|\frac{e^{\frac{\pi\nu}{16}}\pi}{\Gamma(\frac{1}{2}+\frac{i\nu}{16})}+\frac{4(-1)^{3/4}e^{\frac{\pi\nu}{16}}\pi}{\sqrt{\nu}\Gamma(\frac{i\nu}{16})}-\left(1+e^{\frac{\pi\nu}{8}}\right)\Gamma\left(\frac{1}{2}-\frac{i\nu}{16}\right)\right|^{2}.$

Then we have the half entanglement entropy $S_{L/2}=-P_{0}\log
P_{0}-(1-P_{0})\log(1-P_{0})$. In Fig. 3 we show $S_{L/2}$ as a function of
$\nu$ for different nearest-neighbor interaction strength $V$. Our analytical
prediction is shown as the dashed red line and coincides well with the full
numerical calculation at the deep HI regime around $V/J\approx 2.4$, except at
around $\nu=0$ where the long-time nearly adiabatic limit fails. The solid
square point labels the result for sufficient slow breaking rate
$\Gamma/J=0.1$ as in Fig. 2b. For larger $\nu$ one needs even slower breaking
rate. This indicates the entanglement measurement is relatively robust in this
deep HI regime.

Figure 3: The half entanglement entropy $S_{L/2}$ as a function of
$\nu=h^{2}/(\Gamma\alpha)$. The data points are from full numerical
calculations and the dashed red line is our two-level prediction. The dotted
black line labels $S_{L/2}=\log 2$. These results are for $U/J=4$ and chain
length $L=12$. For each $V$ we first calculate the gap $h$ and coupling
parameter $\alpha$, then choose different breaking rate $\Gamma$. For fixed
$V$, smaller $\nu$ corresponds to larger $\Gamma$. Around the critical point
$V/J\approx 2.4$ the non-adiabatic dynamics is well described by our two-level
prediction with the solid square point labels the result for $\Gamma/J=0.1$ as
in Fig. 2b.

##  4 Conclusions

We have studied the middle chain-breaking dynamics of the HI with broken BC
inversion symmetry. We find a critical point within the deep HI regime, where
the adiabatic evolution fails as a result of energy level crossing. We show
how to understand this non-adiabatic dynamics using a simple two-level model.
We give an analytical prediction to the entanglement entropy and find it is
relatively robust in this regime. We note that if we rotate the state space to
$\left(|0\rangle\pm|1\rangle\right)/\sqrt{2}$ in our two-level prediction, the
Hamiltonian near the adiabatic limit $H_{ad}(t)$ reminds us the one of a
Landau-Zener system, the infinite-long-time diabatic probability of which is
readily solved [18, 19]. In this paper we provide an alternative analytical
solution to the full time evolution. This result is quite general, and should
find applications in similar Landau-Zener-like systems. Our results suggest
the HI as an ideal system to study the non-equilibrium dynamics of SPT phases
with broken symmetries.

The entanglement entropy has been considered theoretically [21, 22], and is
realized recently in cold atoms experiments, where a combination of the
single-site-resolved microscope and many-body quantum interference is carried
out to directly measure the second-order Rényi entropy
$S_{2}=-\log\mathrm{Tr}(\rho_{A}^{2})$ [23, 24]. This Rényi entropy $S_{2}$
provides a lower bound for the von Neumann entropy we considered here and
shares similar behaviors. The generalization of our prediction to the second-
order Rényi entropy based on the two-level model is straightforward. We also
note that since the number partitions are distinct for the two lowest nearly
degenerate levels in the deep HI regime, one can directly get the von Neumann
entropy by measuring its number entanglement $S_{n}$ of half system [25].

This research is supported by Fundamental Research Funds for the Central
Universities (No. FRF-TP-19-013A3).

## References

## References

  * [1] Chiu C, Teo J C Y, Schnyder A P, and Ryu S 2016 Rev. Mod. Phys. 88, 035005
  * [2] McGinley M and Cooper N R 2018 Phys. Rev. Lett. 121, 090401
  * [3] Marks J A, Schüler M, and Devereaux T P 2021 Phys. Rev. Res. 3, 023137
  * [4] Dalla Torre E G, Berg E, and Altman E 2006 Phys. Rev. Lett. 97, 260401
  * [5] Rossini D and Fazio R 2012 New J. Phys. 14, 065012
  * [6] Haldane F D M 1983 Phys. Lett. A 93, 464; 1983 Phys. Rev. Lett. 50, 1153
  * [7] Affleck I 1989 J. Phys. Condens. Matter 1, 3047
  * [8] Berg E, Dalla Torre E G, Giamarchi T, and Altman E 2008 Phys. Rev. B 77, 245119
  * [9] Li H and Haldane F D M 2008 Phys. Rev. Lett. 101, 010504
  * [10] Pollmann F, Turner A M, Berg E, and Oshikawa M 2010 Phys. Rev. B 81, 064439
  * [11] Deng X and Santos L 2011 Phys. Rev. B 84, 085138
  * [12] Ejima S, Lange F, and Fehske H 2014 Phys. Rev. Lett 113, 020401
  * [13] Xu J, Gu Q, and Mueller E J 2018 Phys. Rev. Lett. 120, 085301
  * [14] Rispoli M, Lukin A, Schittko R, Kim S, Tai M E, Léonard J, and Greiner M 2019 Nature 573, 385
  * [15] White S R 1992 Phys. Rev. Lett. 69, 2863; 1993 Phys. Rev. B 48, 10345
  * [16] Schollwöck U 2005 Rev. Mod. Phys. 77, 259; 2011 Ann. Phys. (Amsterdam) 326, 96
  * [17] Bauer B et al., 2011 J. Stat. Mech. P05001; Dolfi M, Bauer B, Keller S, Kosenkov A, Ewart T, Kantian A, Giamarchi T, and Troyer M 2014 Comput. Phys. Commun. 185, 3430; http://alps.comp-phys.org.
  * [18] Landau L D 1932 Phys. Z. Sowjetunion 2, 46.
  * [19] Zener C 1932 Proc. R. Soc. London Ser. A 137, 696.
  * [20] Gradshteyn I S and Ryzhik I M 2014 Table of Integrals, Series, and Products, 8th ed., edited by Zwillinger D and Moll V (Elsevier, Amsterdam)
  * [21] Abanin D A and Demler E 2012 Phys. Rev. Lett. 109, 020504
  * [22] Daley A J, Pichler H, Schachenmayer J, and Zoller P 2012 Phys. Rev. Lett. 109, 020505
  * [23] Islam R, Ma R, Preiss P M, Tai M E, Lukin A, Rispoli M, and Greiner M 2015 Nature 528, 77
  * [24] Kaufman A M, Tai M E, Luking A, Rispoli M, Schittko R, Preiss P M, and Greiner M 2016 Science 353, 794
  * [25] Lukin A, Rispoli M, Schittko R, Tai M E, Kaufman A M, Choi S, Khemani V, Léonard J, and Greiner M 2019 Science 364, 256


# Improving Semi-Supervised Learning for Remaining Useful Lifetime Estimation
Through Self-Supervision

Tilman Krokotsch<EMAIL_ADDRESS>Mirko Knaak  Clemens Gühmann
Chair of Electronic Measurement and Diagnostic Technology, Technische
Universität Berlin  Thermodynamics & Power Systems, Power Train & Power
Engineering, IAV GmbH

###### Abstract

RUL estimation suffers from a severe data imbalance where data from machines
near their end of life is rare. Additionally, the data produced by a machine
can only be labeled after the machine failed. Semi-Supervised Learning (SSL)
can incorporate the unlabeled data produced by machines that did not yet fail.
Previous work on SSL evaluated their approaches under unrealistic conditions
where the data near failure was still available. Even so, only moderate
improvements were made. This paper proposes a novel SSL approach based on
self-supervised pre-training. The method can outperform two competing
approaches from the literature and a supervised baseline under realistic
conditions on the NASA Commercial Modular Aero-Propulsion System Simulation
dataset.

###### keywords:

Remaining Useful Lifetime, Predictive Maintenance, Semi-Supervised Learning,
Deep Learning

††journal: Reliability Engineering and System Safety

PDM

    Predictive Maintenance
RUL

    Remaining Useful Lifetime
DNN

    Deep Neural Network
MMD

    Maximum Mean Discrepancy
CNN

    Convolutional Neural Network
RNN

    Recurrent Neural Network
LSTM

    Long Short Term Memory Network
MLP

    Multi-Layer Perceptron
DA

    Domain Adaption
C-MAPSS

    Commercial Modular Aero-Propulsion System Simulation
RMSE

    Root Mean Squared Error
MSE

    Mean Squared Error
DANN

    Domain Adversarial Neural Network
HPC

    high-pressure compressor
UMAP

    Uniform Manifold Approximation and Projection
GRL

    Gradient Reversal Layer
ReLU

    Rectified Linear Unit
SSL

    Semi-Supervised Learning
AE

    Autoencoder
RBM

    Restricted Boltzmann Machine
IQR

    interquartile range

##  1 Introduction

Predictive Maintenance (PDM) is one of the core pillars of Industry 4.0 and
enables more cost-effective operation of machinery. While early approaches to
PDM focused on hand-crafted, physical models and heuristics, nowadays data-
driven methods are on the rise. Fueled by massive amounts of data provided by
an increasing number of sensors, data-driven PDM makes hand-crafting physical
models less necessary. Due to the advent of deep learning, data-driven models
can ingest even more data without the need for specialized feature
engineering. Nevertheless, PDM suffers from a server data imbalance as data
from healthy machines is far more ubiquitous than data from degraded or faulty
ones. This makes training effective data-driven models a challenging task.

Remaining Useful Lifetime (RUL) estimation, as a sub-field of PDM, is defined
as “the length from the current time to the end of the useful life”. [1] It
plays a vital role in effectively scheduling maintenance operations.
Unfortunately, labeling data for RUL estimation is only possible after a
machine fails, making it hard to acquire enough labeled data for conventional,
supervised approaches to work. On the other hand, large amounts of unlabeled
data are available from machines that did not yet fail. SSL can be a possible
solution for this problem.

Semi-Supervised Learning (SSL) uses labeled and unlabeled data in concert to
solve a learning problem [2]. It makes it possible to distill knowledge from
large amounts of unlabeled data, thus lowering the amount of labeled data
needed to achieve good performance. Specifically, SSL aims to learn a
conditional distribution $P(y|x)$ where $x\sim P(X)$ are the available
features (i.e. sensor readings) and $y\sim P(Y)$ are the labels (i.e. the
RUL). The learning algorithm has access to the set of labeled training data
$D_{L}=\\{(x_{1},y_{1}),\dots,(x_{i},y_{i})\\}$ and the mutually exclusive set
of unlabeled data $D_{U}=\\{x_{i+1},\dots,x_{i+j}\\}$.

Recent works, e.g. [3, 4], have shown promising results using different SSL
methods for RUL estimation. More specifically, they were able to modestly
reduce the test Root Mean Squared Error (RMSE) and RUL-Score on subsets of the
NASA Commercial Modular Aero-Propulsion System Simulation (C-MAPSS) dataset
using as few as 1% of the labeled data. Although these findings are leading in
the right direction, there are some shortcomings this paper wants to address.
First, the previous work only evaluates their SSL approaches on one of the
four subsets of the C-MAPSS dataset. As these subsets are relatively small
(compared to other deep learning data sets), at least all of the subsets
should be investigated to see if the performance improvements are universal.
Secondly, the time series of the unlabeled data were used as-is. In our
previous work [5], we have shown that unlabeled time series data for RUL
estimation should not contain time steps at the point of failure. If they do,
the whole time series could be labeled trivially, making the use of SSL
unnecessary. Therefore, the previous studies on SSL for RUL estimation may
produce overly optimistic results, as the approaches have access to more data
near failure than could be considered realistically. This _grade of
degradation_ of the unlabeled data, i.e. how far from failure the machines in
it are, is of significant importance for SSL.

As previous work produced only modest improvements, we propose a novel SSL
method based on self-supervised pre-training. Self-supervision has been proven
useful for pre-training deep neural networks on unlabeled data [6, 7, 8]. Our
pre-training task is to estimate the time between two time steps in a time
series as a proxy for the difference in RUL. Afterward, the pre-trained
network is fine-tuned on labeled data. We will conduct experiments on all four
subsets of C-MAPSS comparing our approach and two common SSL approaches, i.e.
Restricted Boltzmann Machines and Autoencoders, to a supervised baseline. This
will serve to answer our following research questions:

  * 1.

Do findings of previous SSL studies on RUL estimation hold when taking the
_grade of degradation_ of the unlabeled data into account?

  * 2.

Can self-supervised pre-training improve on other pre-training-based SSL
approaches for RUL estimation?

The remaining paper is structured as follows. First, we will lay out the
related work on RUL estimation with deep neural networks, Semi-Supervised
Learning and self-supervised learning in section 2. In section 3, we will
describe our network architecture, our semi-supervised learning approach, and
our self-supervised pre-training. Afterward, we explain our experimental
setup, including the data, performance metrics, and competing approaches, as
well as the evaluation procedure and hyperparameter selection. Section 5 is
concerned with the presentation and discussion of our results, while section 6
concludes this paper and gives an outlook on future work. The code to
reproduce the results of this paper is available on GitHub:
www.github.com/tilman151/self-supervised-ssl.

##  2 Related Work

This section gives an overview of the current state of the literature. First,
we will discuss RUL estimation with Deep Neural Networks. Second, we will
investigate Semi-Supervised Learning and self-supervised learning in the scope
of RUL estimation.

###  2.1 Remaining Useful Lifetime Estimation

RUL estimation is often treated as a regression problem. Recent work focuses
mainly on DNNs because they work on the raw data and do not require hand-
crafted features. Obviously, the early works were focuses on shallow networks
and Multi-Layer Perceptrons [9]. Later ones settled on Convolutional Neural
Networks and LSTM as network architectures.

Long Short Term Memory Networks are Recurrent Neural Networks and a natural
fit for the time series data seen in RUL applications. They process one time
step at a time with the help of an internal cell state derived from all
previously seen time steps. Zheng et al. [10] used LSTMs on three benchmark
datasets and found them working best compared to MLPs and CNNs. Wu et al. [11]
compared LSTMs against vanilla RNNs and GRUs. They declared LSTMs the winner,
too.

CNNs on the other hand seem better suited for image data than time series.
But, using 1d-convolution instead of 2d we can use it for RUL estimation, too.
In a comparison, Bai et al. [12] found CNNs equal to LSTMs in performance,
even though they are faster in inference and training. First attempts at RUL
estimation from Babu et al. [13] were still worse than LSTMs. They were still
using 2d-convolution and when Li et al. [14] switched to 1d, they were able to
surpass LSTMs altogether. Zhu et al. [15] combined features from multiple
hidden layers with their multiscale CNN which did better on their dataset than
traditional CNNs. Instead of using raw time series data, they transformed it
to the frequency domain and used that as the input of their network. Jiang et
al. [16] resorted to combining both network types and report better
performance.

Next to all networks were trained against Mean Squared Error (MSE) [10, 11,
13, 15, 16]. Only Li et al. [14] used RMSE.

###  2.2 Semi-Supervised Learning

SSL can be divided into several sub-fields [2]. The most common distinction is
the goal of the training process itself. While transductive methods are only
concerned with providing the labels for the unlabeled training data, inductive
methods yield a model that can be used on unseen data. We will focus on the
inductive methods, as for PDM applications we need to apply the trained model
on unseen data after training. Even though there are many diverse approaches
to SSL (e.g. S3VMs), we will narrow our perspective to the ones applicable to
DNNs.

Wrapper methods offer some of the oldest SSL approaches. They rely on
producing pseudo-labels for the unlabeled portion of the data and then using
it in conjunction with the labeled data to train supervised. Self-training is
one of the most basic methods and was published by Yarowsky et al. in 1995
[17]. First, a model trained only on the labeled data provides the pseudo-
labels for the unlabeled data. Afterward, a final model is trained on the
combined labeled and pseudo-labeled data.

Pre-training-based methods rely on unsupervised learning. The DNN or a part of
it is trained with an unsupervised learning method and taken as the
initialization for a supervised learning stage. The literature provides
examples for several unsupervised approaches, most commonly AEs, used for pre-
training. Cheng et al. [18] used deep autoencoders for semi-supervised machine
translation. Kingma et al. [19] used variational autoencoders for semi-
supervised image classification.

There are several deep learning methods for SSL that directly incorporate an
unsupervised component into their loss. Examples are Ladder Networks [20]
which incorporate an autoencoder reconstruction loss, or Pseudo-ensembles [21]
which use a consistency loss between the outputs of a parent network and
perturbed child networks for the unlabeled data. The survey of Van Engelen et
al. [2] gives an excellent overview of the previously mentioned methods.

There are a few papers on SSL for RUL estimation. Ellefsen et al. [3] used a
RBM for pre-training on the NASA C-MAPSS dataset. Yoon et al. [4] pre-trained
their network on the same dataset with a variational autoencoder. He et al.
[22] used ladder networks for RUL estimation of centrifugal pumps.
Unfortunately, the field of SSL suffers from a multitude of evaluation setups,
which makes comparing approaches difficult [23]. SSL for RUL estimation is not
excluded from this issue, as many papers report results only on parts of their
datasets, or omit critical information like the amount of available labeled
data. We aim to take these pitfalls into account in this work.

###  2.3 Self-Supervised Learning

Like many methods in deep learning, self-supervised learning first succeeded
in computer vision and spread to different fields from there. The aim was to
learn features that are beneficial for solving common tasks (image
classification, object detection, etc.) in the absence of labeled data. For
this, a so-called pre-text task is defined to train a neural network, which is
then used as a feature extractor for the real task. For example, Doersch et
al. [6] predicted the position of an image patch, cut from a larger image, in
relation to another patch from the same image. The trained network was then
able to perform object detection. Another example is Gidaris et al. [7], who
used predicting image rotation as a pre-text task for classification.
Approaches like these were able to produce state-of-the-art results in a semi-
supervised regime (large amount of unlabeled, small amount of labeled data)
but could not outperform supervised approaches trained on large-scale
datasets.

Self-supervised learning gained prominence as a pre-training technique in
natural language processing through Delvin et al. [8]. Their model, named
BERT, was pre-trained on the pre-text task of predicting missing words in
sentences sampled from a $3.3$M word dataset, including the whole of the
English Wikipedia. The model was then able to produce state-of-the-art results
on eleven benchmark tasks by training a single layer on top of the pre-trained
network.

Metric learning is another possible pre-text task that can be used supervised
and self-supervised. It aims to learn a distance or similarity metric between
pairs of data points. The recent work of Musgrave et al. [24] gives an
excellent overview of popular methods but shows that newer, more complex
approaches perform only as good as older, simple ones. This leads us to the
conclusion that even simple metric learning methods could be used as a self-
supervised pre-text task, too.

There is, to our knowledge, still a lack of work in self-supervised methods
for multivariate time series data as found in PDM applications. Recently,
Franceschi et al. [25] proposed a metric-learning-based pre-text task for time
series. They trained siamese networks [26, 27] to predict the similarity of
two time series snippets with a triplet loss. Their pre-trained network
outperformed dynamic time warping in an unsupervised regime, other deep
learning approaches in a semi-supervised regime and yield competitive results
when compared to supervised state-of-the-art approaches. This hints at self-
supervised learning as a promising direction for pre-training on time series
data.

##  3 Methods

In this section, we will describe the neural network we use for RUL
estimation, SSL via pre-training, and our novel self-supervised pre-training
technique.

(a) RUL Estimation Network

(b) Self-Supervised Siamese Network

Figure 1:  Overview of the networks used in this work: Depicted are the
feature extractor $f$, the regression head $g$, and the distance function $h$.
The supervised RUL estimation network takes a time frame $x_{i}$ as its input
and predicts the RUL value $\textsc{RUL}_{i}^{\prime}$. The siamese network
produces a pair of embeddings from the inputs $x_{i}$ and $x_{j}$. The
distance $h(f(x_{i}),f(x_{j}))$ is $r^{\prime}$, the predicted relative RUL.
The trainable parts of the networks are depicted in red.

###  3.1 Remaining Useful Lifetime Estimation Network

Figure 2: Architecture of the used Networks: The feature extractor $f$
consists of six $ConvBlocks$. These blocks are stacks of 1d-convolutions of
kernel size three, $m$ filters, and either $same$ or $valid$ convolution
padding with zeros, followed by Batch Normalization and a ReLU activation
function. The even layers use valid padding and the odd layers use same
padding. This makes it possible to use more $ConvBlocks$ with valid padding
only as same padding does not reduce the size of the input. In between the
$ConvBlocks$, we place 2d-dropout that drop whole time steps from their input.
The probability of dropping a time step is $p$. The last layer of the feature
extractor is a fully-connected layer reducing its input to a latent dimension
of 64. The regression head $g$ is a Batch Normalization layer followed by a
ReLU activation function and a fully-connected regression layer with a single
unit.

In general, DNNs for regression follow a common architecture, as seen figure
1(a). They consist of two networks: the feature extractor $f$ and a regression
head $g$. Networks for RUL estimation are no different, estimating the RUL
value of a sample $x$ as:

$\textsc{RUL}^{\prime}=g(f(x))$ (1)

As seen in section 2.1, the feature extractor can take the form of a CNN, LSTM
or any other network architecture. The regression head, on the other hand,
almost always takes the form of a simple linear layer ($ax+b$) or a MLP.

We expect the network input $x_{(i-w):i}^{(k)}$, or $x_{i}$ for short, to be a
time frame of size $w$ from a multivariate time series $k$ ending at time step
$i$. Therefore, we use a simple 1d-CNN feature extractor with a fully-
connected layer as the regression head. The feature extractor consists of
multiple 1d-CNN layers with Dropout, Batch Normalization [28], and a ReLU
activation function, followed by a single linear layer also with Batch
Normalization and ReLU. The complete architecture is depicted in figure 2. We
selected a CNN because they are faster to train than e.g. LSTMs and have less
trainable parameters than a MLP with similar performance. In this work, we
will focus on this feature extractor only, as we are mainly concerned with the
influence of pre-training it on unlabeled data. Furthermore, previous works
imply that differences in performance between different extractor
architectures for RUL estimation are marginal [14].

We train the network by mini-batch gradient descent with a RMSE loss:

$\mathcal{L}_{\textsc{RMSE}}=\sqrt{\frac{1}{|X|}\sum_{i=1}^{|X|}(\textsc{RUL}_{i}-\textsc{RUL}^{\prime}_{i})^{2}}$
(2)

where $X$ is a (mini-)batch of samples $\\{x_{1},...,x_{|X|}\\}$,
$\textsc{RUL}_{i}$ the RUL value associated with $x_{i}$ and
$\textsc{RUL}^{\prime}_{i}=g(f(x_{i}))$.

###  3.2 Semi-Supervised Learning through Pre-Training

SSL is a machine learning regime that includes labeled and unlabeled data into
the training process. In this paper, we focus on the two-stage approach of
combining unsupervised pre-training with supervised fine-tuning. First, the
feature extractor $f$ is trained using a method that does not require labels
on the features of the labeled and unlabeled data. Afterward, the pre-trained
feature extractor is used as the starting point for the conventional,
supervised training as described in the previous section. The regression head
$g$ is still initialized randomly.

This procedure aims for the feature extractor to learn useful features from
the unsupervised data that are beneficial to the supervised training. For this
to work, it is necessary to assume that the marginal data distribution $P(X)$
contains information about the conditional distribution $P(\textsc{RUL}|X)$ we
want to learn [2]. It follows that we have to assume that the labeled and
unlabeled data were generated by the same distribution $P$.

###  3.3 Self-Supervised Pre-Training

In this section, we lay out the intuitive motivation behind our proposed self-
supervised pre-training through analyzing the latent space of trained
networks. Afterward we derive a pre-text task from this intuition and devise a
training regime.

####  3.3.1 Motivation

To be able to design a suitable pre-text task, one has to understand the
supervised training process and its shortcomings. To gain an understanding of
the supervised training process, we will look at the features learned by the
feature extractor $f$ trained with different amounts of labeled data.
Unfortunately, high-dimensional data, such as the extracted features, is
impossible to understand for humans. Therefore, we project it into two
dimensions by using Uniform Manifold Approximation and Projection (UMAP) [29],
and plot the features in a scatter plot.

Figure 3(a) depicts the features extracted from the validation data of C-MAPSS
subset FD004 after training on all available labeled data. The network
achieves a validation MSE loss of $19.3$. We can observe a snake-like
structure in the features that broadens on the left side and narrows on the
right side. Though coloring the data points according to their associated RUL
value, we can see that the left end of the snake corresponds to high and the
right side to low RUL values. Furthermore, no discernible sub-structures are
apparent which shows that the network does not differentiate between time
frames from different time series and the same RUL. Because of the low
validation loss, we can conclude that this snake-like structure in the feature
space may be beneficial to RUL estimation.

Figure 3(b) depicts the features extracted by a feature extractor trained only
on three labeled time series, which corresponds to $2\%$ of the available
labeled data. The network achieves a validation loss of $31.8$. Again, we can
observe the snake-like structure but this time the high-RUL end is much more
feathered out. Additionally, the RUL values are not as clearly separated as in
the previous figure.

Comparing the two plots, we can theorize that the network fails in the low
data scenario because it cannot learn the tight snake-like structure. A
suitable pre-text task should result in a feature extractor that already
learned this structure which we can verify qualitatively with a plot like in
figure 3. Data points with similar RUL values should be near each other in the
feature space.

(a) All labeled time series

(b) Three labeled time series

Figure 3: Features produces by trained feature extractors: The depicted
features are UMAP [29] projections of the output of a feature extractor $f$.
The first plot shows an $f$ trained on all labeled data from FD004, while the
second one shows an $f$ trained on only three time series. Both were trained
for 10 epochs. The color indicates the RUL value associated with the
embedding. Both (a) and (b) were generated from the validation data of FD004.
The random seed for both training runs was fixed.

####  3.3.2 Definition of the Pre-Text Task

To derive a pre-text task from the intuition gained in the previous chapter,
we turn to the field of metric learning. Metric learning is concerned with
learning a distance or similarity metric between two data points of a dataset.
As we are aiming to learn self-supervised, we can use only information from
the unlabeled data to achieve this task. In our case, we want to learn a
latent space structure, where data points with similar RUL values are close
and points with a large difference in RUL are distant from each other. This
corresponds to predicting the _relative RUL_ between two data points. Given
two time series, $k$ and $l$, of data from machines run to failure, the
_relative RUL_ value $r$ is:

$r=\textsc{RUL}^{(k)}_{i}-\textsc{RUL}^{(l)}_{j}$ (3)

where $\textsc{RUL}^{(k)}_{i}$ is the RUL at time step $i$ in time series $k$.
If we declare $k=l$ and given the assumption of a linear degradation model of:

$\textsc{RUL}^{(k)}_{i}=|k|-i$ (4)

where $|k|$ is the length of time series $k$, we can rewrite equation 3 as:

$r=(|k|-i)-(|k|-j)$ (5)

It is trivial to see that $|k|$ is not needed at all to calculate $r$:

$\displaystyle r$ $\displaystyle=$ $\displaystyle j-i$ (6)

This makes it possible to calculate $r$ even on time series of unfailed
machines where $|k|$ is not known. Therefore, this pre-training target can be
used without any failure data and consequently without the true RUL values.

For a piece-wise linear degradation model as seen in the C-MAPSS dataset:

$\textsc{RUL}^{(k)}_{i}=\max{(\textsc{RUL}_{\textsc{max}},\;|k|-i)}$ (7)

the formula for $r$ is still an acceptable approximation, if we assume that
$i<j$ and $i-j\leq\textsc{RUL}_{\textsc{max}},\;\forall i,j$. We normalize $r$
to the range of $[0,1]$ by dividing it by $\textsc{RUL}_{\textsc{max}}$. The
final equation is then:

$r\approx\frac{j-i}{\textsc{RUL}_{\textsc{max}}},\quad\begin{aligned} i&<j\\\
i-j&\leq\textsc{RUL}_{\textsc{max}}\end{aligned}\quad\forall i,j$ (8)

####  3.3.3 Training Procedure

The goal of our pre-training is training a neural network on estimating the
target value $r$ from equation 8 from which we then extract the feature
extractor $f$ to be used to initialize the RUL estimation network before
supervised training. We realize this goal with siamese networks [27, 26].
Siamese networks take the form of the feature extractor $f(x)$ and a distance
function $h(a,b)$ operating on two samples, $x_{i}$ and $x_{j}$, so that:

$r^{\prime}=h(f(x_{i}),f(x_{j}))$ (9)

where $r^{\prime}$ is the predicted value of $r$. The function $h(a,b)$ is
defined as:

$h(a,b)=\left|\left|\frac{a}{||a||}-\frac{b}{||b||}\right|\right|^{2}$ (10)

where $||\cdot||$ is the euclidean norm. Dividing the embeddings by their norm
restricts the embedding space to a hyper ball, which often is found to be
useful [24]. Figure 1(b) depicts the schematic structure of the siamese
networks. We use MSE as a loss function with mini-batch gradient descent to
train the siamese networks:

$\mathcal{L}_{\textsc{MSE}}=\frac{1}{|X|}\sum_{k=1}^{|X|}{(r_{k}-r_{k}^{\prime})}$
(11)

For each (mini-)batch $X$, we sample $|X|$ time series from the training data,
uniformly sample pairs of time frames $x^{(k)}_{i}$ and $x^{(k)}_{j}$, and
calculate $r_{k}$ according to equation 8. Because the samples $x^{(k}_{i}$
and $x^{(k}_{j}$ are time frames from the same time series, a difference
between $i$ and $j$ that is much smaller than the length of the time frame
results in significant overlap between the samples. Therefore, we introduce a
minimum distance for sampling pairs, which is regarded as a hyperparameter. We
used the Adam optimizer [30] for training with the default values of $0.9$ and
$0.99$ for $\beta_{1}$ and $\beta_{2}$.

##  4 Experimental Design

This section describes the data set used for our experiments, which
performance metrics were used, and how the evaluation was conducted.

###  4.1 Data

We evaluate the effect of our pre-training technique on the publicly available
NASA C-MAPSS dataset. It contains four subsets (FD001 - FD004) of different
operating conditions and possible fault modes. Engines in FD001 and FD002
experience only high-pressure compressor (HPC) degradation, while the ones in
FD003 and FD004 can additionally experience fan degradation. Further, in FD001
and FD003 are engines run under only one operating condition, while engines
from FD002 and FD004 vary between six. Each subset is split into training and
test data by the dataset authors. The training data contains multivariate time
series data of aero engines up until the time step they failed. Each time
series can be considered as a different engine of the same type. The test
data’s time series stop at a random time step before failure, for which the
true RUL label is provided. Table 1 summarizes the details of the dataset. We
construct one validation set from each training set by taking $20\%$ of the
time series.

Dataset | FD001 | FD002 | FD003 | FD004
---|---|---|---|---
# Training Engines | 100 | 260 | 100 | 249
# Test Engines | 100 | 259 | 100 | 248
# Operation Conditions | 1 | 6 | 1 | 6
# Failure Modes | 1 | 1 | 2 | 2
Table 1: C-MAPSS Dataset

For pre-processing, we follow Li et al. [14] and select 14 of the 21 sensor
channels with the indices 2, 3, 4, 7, 8, 9, 11, 12, 13, 14, 15, 17, 20, and 21
as the input of the network. The features are scaled by channel, using a min-
max scaling calculated on each subsets’ training set. We then use a sliding
window with a step size of one to obtain frames of unified length in the time
dimension. We use a window size of 30, 20, 30, and 15 for the subsets 1-4.
These sizes were determined by the length shortest time series in each subsets
test set. The RUL labels are calculated by a piece-wise linear degradation
model with a maximum value $\textsc{RUL}_{\textsc{max}}$ of $125$.

###  4.2 Data Scenarios

As described in the introduction, unlabeled RUL data cannot contain the point
of failure as we could simply compute the labels otherwise. It follows that we
have to truncate the time series before failure when studying SSL, as the
model would have access to more failure data than normally available
otherwise. How early we truncate the time series represents the grade of
degradation of the engine that the time series was collected from. Previous
work on SSL for RUL estimation varied only the amount of labeled data
available, i.e. how many engines had already failed. Our experimental design
includes varying the grade of degradation for the unlabeled data, too.

Adapting the work of Krokotsch et al. [5], we impose data scenarios on each
subset. In our case, a data scenario is characterized by the number of failed
engines and grade of degradation of the unlabeled ones. Both factors are
interpreted as percentages. A data scenario of number of engines at $n\%$
would mean that only $n\%$ of the machines in the subset are used as labeled
data for training. The rest is assumed to have not yet failed and is used as
unlabeled data. This limits the amount of machine-to-machine variance that is
covered by the labeled data. A data scenario of grade of degradation at $n\%$
would mean that only the first $n\%$ of time steps of each unlabeled time
series is available during training. This effectively limits the amount of
available data near failure.

We use five different grades of degradation of $40\%$, $60\%$, $70\%$, $80\%$
and $90\%$ for our evaluation. A grades of degradation of $100\%$ would not
make sense, as the unlabeled machines would have failed already and could be
used as labeled data. The lower limit of percentages is chosen due to the
piece-wise linear degradation model. Using fewer than $40\%$ of the time steps
would mean that the unlabeled data contains next to no data with a RUL of less
than $\textsc{RUL}_{\textsc{max}}$. This means that the unlabeled data would
come from completely healthy machines and may add no benefit to training.

The number of failed engines is set at $2\%$, $10\%$, $20\%$, $40\%$, $100\%$.
Using $100\%$ of failed machines means that no unlabeled data is available and
pre-training is conducted on the labeled data only. We decided on these
percentages because preliminary experiments did not show any degradation in
performance for more than $40\%$ compared to using $100\%$ of the engines. The
lower limit of $2\%$ is chosen because it results in at least one failed
engine for each subset.

###  4.3 Performance Metrics

We employ two common performance metrics from the field of PDM. The first is
the RMSE, as described in equation (2) over the test set. The second is the
_RUL-Score_, first proposed in the PHM 2008 Data Challenge [31]. It is
calculated as follows:

$s_{i}=\begin{cases}e^{-\frac{\Delta\textsc{RUL}_{i}}{13}}-1&\text{for
}\Delta\textsc{RUL}_{i}<0\\\ e^{\frac{\Delta\textsc{RUL}_{i}}{10}}-1&\text{for
}\Delta\textsc{RUL}_{i}\geq 0\end{cases}$ (12)

where $\Delta\textsc{RUL}_{i}$ is the difference between predicted and true
RUL for sample $x_{i}$. This metric penalizes overestimating RUL more than
underestimating it as the former has a bigger impact on maintenance actions.
We report the sum over all samples in the test set following previous work in
the field.

We will discuss the results using both metrics. Although, we will focus on
RMSE as we view it as the more intuitive measure of performance.

###  4.4 Supervised Baseline

The baseline for our experiments is training the RUL estimation network in a
supervised fashion as described in section 3.1. The training will only
incorporate the available labeled data. We used the Adam optimizer [30] with
the default values for $0.9$ and $0.99$ for $\beta_{1}$ and $\beta_{2}$.

###  4.5 Competing Approaches

Aside from the baseline, we compare our pre-training approach to two other
methods found in the literature. The first approach is unsupervised pre-
training via a deep AE. We construct the AE so that our feature extractor
network is used as the encoder. The decoder is an exact mirror image of the
encoder with transposed convolutions replacing the convolutional layers. The
AE is then trained to reconstruct its input via mini-batch gradient descent
with a MSE loss. As for the baseline, we used the Adam optimizer.

The second approach is unsupervised pre-training via a RBM. The first layer of
the feature extractor is interpreted as a RBM with Gaussian visible units and
ReLU hidden units and trained until convergence. The other layers of the
feature extractor remain randomly initialized. Again, Adam was used as the
optimizer.

Even though [3, 4] reported results for RBMs and variational AEs, we choose to
reproduce the competing approaches ourselves. This has several reasons. First,
the mentioned papers evaluated their approaches only on FD001 or FD004 which
paints a limited picture of the approaches’ performance. Second, the mentioned
papers used a slightly different experimental setup, i.e. a different
$\textsc{RUL}_{max}$, making comparison difficult. Furthermore, the
aforementioned $\textsc{RUL}_{max}$ was optimized as a hyperparameter in one
of the papers. As this value controls the possible range of the performance
metrics, optimizing it results in improperly optimistic results.

###  4.6 Evaluation Procedure

The baseline, our approach and the competing SSL methods are evaluated on each
of the subsets of the C-MAPSS dataset subject to each of our selected data
scenarios. This results in 100 different experimental setups (4 subsets, 5
number of failed engines, 5 grades of degradation) for each SSL method. The
baseline includes only 20 setups, as it does not use unlabeled data which
makes varying the grade of degradation superfluous.

Each experimental setup is replicated ten times. For each replication, a new
split of labeled and unlabeled data is randomly samples according to the data
scenario. The splits are held constant across methods, i.e. the baseline
receives the same labeled data as the SSL methods given the same data
scenario. This makes comparison easier as much more replications than ten
would be needed for the low labeled data scenarios to receive statistically
stable performance estimates.

The pre-training stage for the SSL methods receives the labeled and unlabeled
portions of the data and trains for at least 100 epochs. The RBM is trained
for five epochs as it contains only one layer. Early stopping is used to
select the model with the lowest validation loss. For our method we monitor
the MSE loss for the relative RUL target and for the autoencoder the MSE
reconstruction loss.

The supervised training stage receives only the labeled portion of the data
and is trained for at least 200 epochs. The weights of the selected pre-
trained model are used to initialize the network in this stage for the SSL
methods. The baseline is initialized randomly. It should be noted that we
divide the output of the feature extractor by its norm if the pre-training was
self-supervised. Again, early stopping on the validation RMSE is used to
select the best model for which the performance metrics over the test set are
calculated. We report the performance for each subset/data scenario
combination separately and averaged over the ten replications.

###  4.7 Hyperparameter Selection

We began hyperparameter selection using the fixed network architecture shown
in figure 2 and conducted a random search in two steps. First, the
hyperparameters of the supervised stage (i.e. learning rate, dropout, and
batch size) were optimized for each subset of C-MAPSS. A network was trained
in a supervised fashion with all available labeled data as described in
section 4.6. After 100 trials, the hyperparameters of the network with the
best validation RMSE were selected. The hyperparameters in question can be
seen in table 2.

Hyperparameter | Search Space | Configuration for
---|---|---
FD001 | FD002 | FD003 | FD004
Learning Rate | $\operatorname{qlogu}(1^{-4},1^{-1},5^{-5})$ | $0.0056$ | $0.0903$ | $0.095$ | $0.06635$
Dropout | $\operatorname{qu}(0.0,0.5,0.1)$ | $0.4$ | $0.3$ | $0.2$ | $0.0$
Batch Size | [64, 128, 256, 512] | 128 | 512 | 64 | 64
Table 2: Hyperparameters of the Supervised Training Stage: The search space
$\operatorname{qlogu}(a,b,c)$ draws samples uniformly on a logarithmic scale
from the interval $[a,b]$ quantized to $c$. The search space
$\operatorname{qu}(a,b,c)$ draws samples uniformly from the interval $[a,b]$
quantized to $c$.

In a second step, the hyperparameters of the pre-training stages (i.e.
learning rate, dropout, batch size, and minimum pair distance) were optimized
similarly. The networks are trained without labels at $80\%$ grade of
degradation as described in section 4.6. For our method we selected
hyperparameters according to the validation MSE loss for the relative RUL
target and for the autoencoder according to the validation MSE reconstruction
loss. As stability was an issue, each hyperparameter configuration was trained
five times and the mean of these replications is used for selection. For the
RBM we adopted the hyperparameters from [3]. We set the learning rate for this
method to $10^{-4}$ by hand as it was not given in the paper.

It should be noted that all optimizations used no labels at all and can
therefore be conducted independently from the amount of labeled data. The
selected hyperparameters can be seen in table 3 and 4.

Hyperparameter | Search Space | Configuration for
---|---|---
FD001 | FD002 | FD003 | FD004
Learning Rate | $\operatorname{qlogu}(1^{-4},1^{-1},5^{-5})$ | $0.00015$ | $0.01155$ | $0.00615$ | $0.07455$
Dropout | $\operatorname{qu}(0.0,0.5,0.1)$ | $0.2$ | $0.4$ | $0.1$ | $0.1$
Batch Size | [64, 128, 256, 512] | 64 | 64 | 64 | 64
Minimum Distance | [1, 10, 15, 30] | 10 | 15 | 15 | 10
Table 3: Hyperparameters of the Self-Supervised Pre-Training: See table 2 for an explanation of the search spaces. Hyperparameter | Search Space | Configuration for
---|---|---
FD001 | FD002 | FD003 | FD004
Learning Rate | $\operatorname{qlogu}(1^{-4},1^{-1},5^{-5})$ | $0.0001$ | $0.0248$ | $0.015$ | $0.0006$
Dropout | $\operatorname{qu}(0.0,0.5,0.1)$ | $0.1$ | $0.4$ | $0.0$ | $0.0$
Batch Size | [64, 128, 256, 512] | 64 | 256 | 64 | 64
Minimum Distance | [1, 10, 15, 30] | 1 | 15 | 1 | 10
Table 4: Hyperparameters of the Autoencoder Pre-Training: See table 2 for an
explanation of the search spaces. The minimum distance was optimized for this
method, too, due to implementation reasons. It should not influence the
results significantly, though.

##  5 Results

First, we will describe the results of our experiments for each C-MAPSS
subset. Afterward, we will interpret the findings and set them into context.

###  5.1 Comparison of Approaches

Our experiments produced too many data points to present them all in detail.
We will therefore show only slices of our results as plots. The results
plotted against the percentage of labeled data are shown in figures 4 and 5 at
80% degradation. The results plotted against grade of degradation are shown in
figures 6 and 7 at 2% of the labeled data. The complete results are shown in
tables 5, 6, 7 and 8.

Overall we can conclude that a significant drop in baseline performance was
mostly observable for very low amounts of labeled data. In figure 4 we can see
that the performance of the baseline (blue) has a relatively small standard
deviation for 100%, 40%, and 20% of labeled data. For the subsets FD002 and
FD004, the mean and standard deviation increase only at 2% labeled data. For
FD001, the performance already drops at 40% labeled data and for FD003 at 10%.
One has to keep in mind, though, that FD002 and FD004 are more than twice as
large as FD001 and FD003, which means that they have a higher amount of
labeled data available at the same percentage.

Performance on FD001 shows next to no improvement through SSL. In figure 4 we
can see that the median RMSE performances is well inside of each others
interquartile ranges with the exception of 40% labeled data where AE pre-
training achieves much better performance. In one case, i.e. for 2% labeled
data, the performance of RBM pre-training was even worse than the baseline.
The RUL-Score metric in figure 5 paints a similar picture, even though pre-
trained models seem slightly better for 20% and 40% labeled data. For 2%
labeled data, the SSL approaches had a worse mean RUL-Score performance than
the baseline. These findings are stable for different grades of degradation,
too. Figures 6 and 7 show no discernible trend with respect to the grade of
degradation for any method. Overall, the AE seems to perform slightly better
than the other approaches.

Performance on FD002 clearly benefits from RBM and self-supervised pre-
training. The self-supervised pre-training, in particular, beats the baseline
and the other approaches most of the time, especially in low-labeled data
scenarios. On the other hand, there is an extreme drop in performance for our
method at 40% degradation. This is true for all percentages of labeled data
but most apparent for 2%. When looking at figure 7, only self-supervised pre-
training reliably beats the median performance of the baseline in terms of
RUL-Score for grades of degradation above 40%.

Performance gains on FD003 are better than on FD001 but still minor. For 2%
labeled data, the self-supervised pre-training beats the baseline and the
other methods for next to all grades of degradation in terms of mean RMSE and
RUL-score. Nevertheless, the IQRs of our method and the RBM are often highly
overlapping. As on FD001, we can see no trend concerning the grade of
degradation.

On FD004, we can see the benefits of our approach most clearly. While AE and
RBM pre-training bring next to no performance gains in terms of median RMSE,
self-supervised pre-training outperforms the baseline reliably, as seen in
figure 4 and 6. Nevertheless, we can observe a downward trend in performance
with respect to the grade of degradation for our method. The competing
approaches do not suffer from this. Figures 5 and 7 reveal devastating
performance losses for AE pre-training in terms of RUL-Score.

We can conclude that SSL can be beneficial for RUL estimation even under
realistic conditions with varying grades of degradation. However, no approach
was able to reliably beat the baseline on all subsets under all data
scenarios. A representative validation set and careful hyperparameter tuning
are needed to assure improved performance and detect negative outcomes. Self-
supervised pre-training seems to be a step in the right direction, as it often
outperforms the competing approaches. Nevertheless, its performance trends
downward on FD002 and FD004 with a falling grade of degradation.

Figure 4: RMSE at $\mathbf{80\%}$ degradation: The plots show the results on
the four subsets of the C-MAPSS dataset. We plot the RMSE test performance
against the percentage of labeled data. The remaining data were used as
unlabeled data with a grade of degradation of $80\%$. The box represents the
IQR and the middle line the median performance. The ends of the whiskers
depict the minimum and maximum performance that was not deemed an outlier.
Outliers are defined as more than 1.5 times IQR away from the lower or upper
end of the box. We can see that the performance degrades only slowly with less
labeled data. Significant drops in performance can be seen for $10\%$ labeled
data for FD003, 40% for FD001, and $2\%$ for the other subsets. Figure 5: RUL-
Score at $\mathbf{80\%}$ degradation: The plots show the results on the four
subsets of the C-MAPSS dataset. We plot the RUL Score test performance on a
logarithmic scale against the percentage of labeled data. The remaining data
was used as unlabeled data with a grade of degradation of $80\%$. See figure 4
for an explanation of the box plots. The results are similar to the RMSE in
figure 4 but show significantly worse scores for the AE pre-training on FD004.
Figure 6: RMSE at $\mathbf{2\%}$ of labeled data: The plots show the results
on the four subsets of the C-MAPSS dataset. We plot the RMSE test performance
against the grade of degradation when only $2\%$ of the labeled data was used.
The median performance of the baseline is depicted in blue. See figure 4 for
an explanation of the box plots. We can observe that the performance of the AE
and the RBM is relatively constant on all grades of degradation. The self-
supervised method’s performance degrades for lower degradation, which can be
seen best in FD004. Nevertheless, the self-supervised method shows much better
performance on higher grades of degradation in FD002 and FD004 than the
competing methods. On the remaining subsets, the performance of the self-
supervised method is comparable to the other approaches. Figure 7: RUL-Score
at $\mathbf{2\%}$ of labeled data: The plots show the results on the four
subsets of the C-MAPSS dataset. We plot the RUL Score test performance on a
logarithmic scale against the grade of degradation when only $2\%$ of the
labeled data was used. The median performance of the baseline is depicted in
blue. See figure 4 for an explanation of the box plots. The results are
similar to the RMSE in figure 4 but show significantly worse scores for the AE
pre-training on FD004. We can also observe that the median score of the self-
supervised approach is the lowest for all grades of degradation on FD002.
FD003 and FD004.

FD001 2% 10% 20% 40% 100% None $31.30\pm 6.16$ $25.45\pm 2.31$ $24.21\pm 1.31$
$21.94\pm 3.08$ $14.63\pm 0.81$ 40% AE $33.37\pm 9.53$ $24.71\pm 1.28$
$\mathbf{23.56\pm 3.12}$ $\mathbf{17.27\pm 1.47}$ $15.03\pm 0.61$ RBM
$33.49\pm 7.49$ $25.90\pm 1.93$ $25.09\pm 1.61$ $21.02\pm 3.17$ $15.67\pm
0.57$ Ours $32.35\pm 6.08$ $\mathbf{24.42\pm 1.58}$ $24.91\pm 1.87$ $21.38\pm
2.32$ $\mathbf{14.34\pm 0.81}$ 60% AE $32.17\pm 8.94$ $\mathbf{24.62\pm 1.75}$
$\mathbf{22.65\pm 2.46}$ $\mathbf{17.95\pm 1.92}$ $15.09\pm 0.52$ RBM
$32.64\pm 7.45$ $27.06\pm 1.79$ $24.78\pm 2.73$ $19.50\pm 2.93$ $15.68\pm
1.01$ Ours $\mathbf{29.88\pm 3.83}$ $25.81\pm 2.36$ $23.93\pm 1.72$ $22.41\pm
1.82$ $\mathbf{14.48\pm 0.87}$ 70% AE $34.93\pm 8.64$ $25.10\pm 2.23$
$\mathbf{21.70\pm 2.09}$ $\mathbf{17.71\pm 1.79}$ $15.02\pm 0.58$ RBM
$33.00\pm 7.40$ $26.00\pm 1.77$ $23.95\pm 1.55$ $19.21\pm 1.92$ $15.45\pm
0.71$ Ours $\mathbf{29.52\pm 5.26}$ $\mathbf{24.90\pm 1.47}$ $22.96\pm 2.38$
$18.51\pm 2.76$ $15.35\pm 0.78$ 80% AE $30.24\pm 7.16$ $25.32\pm 2.23$
$\mathbf{22.41\pm 3.07}$ $\mathbf{17.13\pm 1.31}$ $15.40\pm 0.62$ RBM
$33.58\pm 7.05$ $25.53\pm 2.36$ $24.43\pm 1.66$ $19.56\pm 4.02$ $15.55\pm
0.88$ Ours $\mathbf{28.50\pm 2.68}$ $\mathbf{25.13\pm 2.28}$ $23.64\pm 1.52$
$20.98\pm 2.52$ $\mathbf{14.33\pm 0.74}$ 90% AE $\mathbf{30.57\pm 7.00}$
$26.04\pm 1.92$ $\mathbf{21.84\pm 1.82}$ $\mathbf{17.26\pm 1.34}$ $14.89\pm
0.65$ RBM $31.86\pm 5.30$ $\mathbf{25.05\pm 2.17}$ $25.01\pm 2.53$ $19.44\pm
2.36$ $15.55\pm 0.47$ Ours $31.69\pm 6.62$ $25.20\pm 1.92$ $23.52\pm 2.24$
$18.74\pm 3.53$ $15.39\pm 0.78$ FD002 2% 10% 20% 40% 100% None $29.80\pm 6.38$
$23.87\pm 1.30$ $22.83\pm 1.03$ $20.73\pm 0.94$ $17.55\pm 0.62$ 40% AE
$33.86\pm 7.88$ $23.12\pm 1.11$ $21.91\pm 0.96$ $20.48\pm 0.90$
$\mathbf{17.19\pm 0.30}$ RBM $\mathbf{27.43\pm 4.30}$ $\mathbf{22.79\pm 0.70}$
$\mathbf{20.98\pm 0.65}$ $\mathbf{19.41\pm 0.92}$ $17.19\pm 0.51$ Ours
$42.04\pm 4.47$ $24.04\pm 0.84$ $22.09\pm 1.31$ $20.44\pm 0.80$ $18.34\pm
0.84$ 60% AE $30.95\pm 6.38$ $23.25\pm 0.92$ $22.26\pm 1.05$ $20.69\pm 0.91$
$17.78\pm 1.22$ RBM $26.50\pm 3.22$ $22.87\pm 0.90$ $\mathbf{21.36\pm 1.01}$
$\mathbf{19.42\pm 0.68}$ $\mathbf{17.54\pm 0.74}$ Ours $\mathbf{25.46\pm
3.79}$ $\mathbf{22.86\pm 1.17}$ $22.14\pm 1.35$ $20.45\pm 0.84$ $18.45\pm
0.78$ 70% AE $28.03\pm 4.85$ $23.84\pm 1.06$ $22.43\pm 1.83$ $20.49\pm 0.82$
$17.56\pm 1.22$ RBM $26.83\pm 3.91$ $22.85\pm 1.20$ $21.41\pm 1.19$ $19.33\pm
0.58$ $17.17\pm 0.46$ Ours $\mathbf{24.82\pm 4.00}$ $\mathbf{21.79\pm 1.42}$
$\mathbf{20.25\pm 0.77}$ $\mathbf{18.66\pm 0.69}$ $\mathbf{16.83\pm 0.44}$ 80%
AE $32.36\pm 7.24$ $23.29\pm 0.90$ $21.87\pm 0.90$ $19.92\pm 0.90$ $17.21\pm
0.58$ RBM $27.13\pm 4.05$ $22.84\pm 0.86$ $21.35\pm 0.66$ $\mathbf{19.13\pm
0.87}$ $\mathbf{17.10\pm 0.57}$ Ours $\mathbf{24.82\pm 3.43}$
$\mathbf{22.43\pm 1.11}$ $\mathbf{21.21\pm 1.26}$ $19.79\pm 0.65$ $18.23\pm
0.63$ 90% AE $31.55\pm 6.75$ $23.70\pm 1.07$ $22.30\pm 1.00$ $20.71\pm 1.36$
$17.55\pm 0.53$ RBM $26.98\pm 3.73$ $22.55\pm 1.12$ $20.99\pm 1.07$ $19.46\pm
0.70$ $17.28\pm 0.39$ Ours $\mathbf{24.99\pm 3.80}$ $\mathbf{21.48\pm 1.39}$
$\mathbf{20.26\pm 1.23}$ $\mathbf{18.82\pm 0.72}$ $\mathbf{16.80\pm 0.44}$

Table 5: RMSE results for FD001 and FD002: We report the mean and standard
deviation. The rows represent the _grade of degradation_ and the columns the
_percent of labeled data_ of the data scenario. The second column contains the
pre-training method, where _None_ is the baseline without any pre-training,
and _Ours_ is the self-supervised pre-training. The bold results mark the best
mean performance for each data scenario. If no result is bold, no approach was
able to beat the baseline.

FD003 2% 10% 20% 40% 100% None $42.33\pm 20.56$ $22.99\pm 4.02$ $16.55\pm
0.94$ $14.58\pm 1.06$ $13.88\pm 0.58$ 40% AE $35.08\pm 6.38$ $\mathbf{22.12\pm
5.31}$ $17.05\pm 1.63$ $14.75\pm 1.12$ $13.96\pm 0.59$ RBM $35.83\pm 7.00$
$22.98\pm 4.85$ $16.82\pm 1.13$ $15.19\pm 0.82$ $14.50\pm 0.50$ Ours
$\mathbf{31.82\pm 4.51}$ $22.98\pm 4.30$ $\mathbf{16.00\pm 0.86}$
$\mathbf{14.39\pm 0.85}$ $\mathbf{13.47\pm 0.18}$ 60% AE $34.00\pm 6.89$
$\mathbf{21.26\pm 4.86}$ $16.77\pm 1.62$ $15.09\pm 1.14$ $14.16\pm 0.49$ RBM
$\mathbf{31.63\pm 6.02}$ $22.63\pm 3.93$ $17.04\pm 1.89$ $15.33\pm 0.84$
$14.45\pm 0.42$ Ours $33.79\pm 4.74$ $24.80\pm 4.77$ $17.10\pm 1.19$
$\mathbf{14.52\pm 1.10}$ $\mathbf{13.09\pm 0.74}$ 70% AE $35.01\pm 5.99$
$\mathbf{22.00\pm 4.80}$ $\mathbf{16.36\pm 1.15}$ $15.39\pm 0.93$ $13.85\pm
0.42$ RBM $36.30\pm 7.52$ $22.36\pm 4.23$ $16.68\pm 1.35$ $15.51\pm 0.88$
$14.61\pm 0.45$ Ours $\mathbf{33.10\pm 4.34}$ $22.35\pm 4.82$ $16.73\pm 1.45$
$15.14\pm 1.18$ $\mathbf{13.64\pm 0.62}$ 80% AE $36.35\pm 5.06$
$\mathbf{21.24\pm 3.75}$ $\mathbf{16.49\pm 1.60}$ $14.82\pm 0.76$ $14.03\pm
0.50$ RBM $34.29\pm 6.83$ $22.87\pm 4.63$ $17.31\pm 0.86$ $15.52\pm 0.66$
$14.68\pm 0.56$ Ours $\mathbf{33.47\pm 5.57}$ $22.98\pm 5.18$ $16.54\pm 2.04$
$14.60\pm 0.91$ $\mathbf{13.05\pm 0.53}$ 90% AE $33.37\pm 5.36$
$\mathbf{20.14\pm 4.09}$ $17.10\pm 2.03$ $14.84\pm 0.88$ $14.17\pm 0.35$ RBM
$36.48\pm 7.74$ $23.47\pm 4.27$ $17.26\pm 1.35$ $15.06\pm 0.92$ $14.34\pm
0.57$ Ours $\mathbf{31.79\pm 3.53}$ $21.19\pm 4.13$ $\mathbf{16.34\pm 1.19}$
$14.78\pm 0.75$ $\mathbf{13.83\pm 0.49}$ FD004 2% 10% 20% 40% 100% None
$35.44\pm 2.22$ $27.00\pm 1.37$ $25.18\pm 1.13$ $23.51\pm 0.62$ $21.66\pm
0.90$ 40% AE $36.07\pm 4.39$ $28.53\pm 2.26$ $25.39\pm 1.36$ $\mathbf{23.20\pm
1.06}$ $21.62\pm 0.71$ RBM $35.99\pm 2.38$ $28.46\pm 2.20$ $25.74\pm 0.90$
$23.55\pm 0.62$ $21.48\pm 0.54$ Ours $36.41\pm 1.71$ $31.15\pm 3.66$ $25.44\pm
1.67$ $23.24\pm 1.84$ $\mathbf{21.11\pm 0.52}$ 60% AE $35.60\pm 3.68$
$28.34\pm 2.36$ $\mathbf{24.88\pm 0.97}$ $23.42\pm 0.97$ $21.30\pm 0.37$ RBM
$35.59\pm 3.69$ $27.61\pm 2.44$ $25.48\pm 1.22$ $23.22\pm 0.48$ $21.18\pm
0.62$ Ours $\mathbf{34.20\pm 4.08}$ $29.40\pm 2.77$ $25.31\pm 2.28$
$\mathbf{22.24\pm 1.21}$ $\mathbf{20.96\pm 0.67}$ 70% AE $35.72\pm 3.72$
$28.28\pm 1.82$ $25.34\pm 1.30$ $23.19\pm 0.71$ $21.26\pm 0.56$ RBM $35.41\pm
2.69$ $27.32\pm 2.30$ $25.40\pm 1.39$ $23.02\pm 0.73$ $21.08\pm 0.52$ Ours
$\mathbf{31.58\pm 1.57}$ $27.02\pm 2.07$ $\mathbf{23.88\pm 2.14}$
$\mathbf{22.48\pm 1.51}$ $\mathbf{20.95\pm 0.67}$ 80% AE $34.68\pm 2.34$
$28.16\pm 1.73$ $25.47\pm 1.79$ $22.74\pm 1.08$ $21.23\pm 0.62$ RBM $37.15\pm
4.56$ $27.95\pm 2.36$ $25.20\pm 1.08$ $23.43\pm 1.25$ $21.26\pm 0.89$ Ours
$\mathbf{29.23\pm 2.74}$ $\mathbf{25.84\pm 2.84}$ $\mathbf{24.26\pm 2.11}$
$\mathbf{21.46\pm 0.68}$ $\mathbf{20.95\pm 0.67}$ 90% AE $35.95\pm 2.96$
$27.57\pm 1.28$ $25.66\pm 1.14$ $23.43\pm 1.20$ $21.73\pm 0.59$ RBM $35.88\pm
2.70$ $27.84\pm 1.82$ $25.69\pm 1.31$ $23.73\pm 0.78$ $21.25\pm 0.44$ Ours
$\mathbf{31.07\pm 3.81}$ $\mathbf{24.20\pm 2.14}$ $\mathbf{24.17\pm 2.88}$
$\mathbf{21.82\pm 1.22}$ $\mathbf{20.90\pm 0.62}$

Table 6: RMSE results for FD003 and FD004: Please consult table 5 for further
information.

FD001 2% 10% 20% 40% 100% None $5.64e3\pm 3.25e3$ $1.49e4\pm 1.11e4$
$1.04e4\pm 7.48e3$ $5.28e3\pm 6.35e3$ $4.24e2\pm 1.15e2$ 40% AE $1.51e4\pm
1.28e4$ $\mathbf{4.57e3\pm 2.37e3}$ $\mathbf{7.39e3\pm 7.22e3}$
$\mathbf{1.03e3\pm 7.22e2}$ $4.79e2\pm 1.03e2$ RBM $1.62e4\pm 8.24e3$
$1.73e4\pm 1.83e4$ $9.79e3\pm 9.05e3$ $3.69e3\pm 4.48e3$ $5.32e2\pm 6.25e1$
Ours $2.04e4\pm 9.05e3$ $1.14e4\pm 8.40e3$ $1.57e4\pm 1.32e4$ $3.90e3\pm
2.89e3$ $\mathbf{3.97e2\pm 8.31e1}$ 60% AE $1.38e4\pm 1.28e4$
$\mathbf{9.80e3\pm 1.20e4}$ $\mathbf{4.76e3\pm 4.94e3}$ $\mathbf{9.85e2\pm
5.46e2}$ $4.79e2\pm 6.05e1$ RBM $1.76e4\pm 1.56e4$ $1.89e4\pm 2.34e4$
$1.55e4\pm 1.58e4$ $2.95e3\pm 4.13e3$ $5.14e2\pm 9.48e1$ Ours $1.80e4\pm
2.13e4$ $1.08e4\pm 5.36e3$ $8.08e3\pm 5.60e3$ $6.04e3\pm 5.29e3$
$\mathbf{4.04e2\pm 8.41e1}$ 70% AE $3.16e4\pm 3.61e4$ $1.42e4\pm 1.67e4$
$\mathbf{2.71e3\pm 2.45e3}$ $\mathbf{8.92e2\pm 4.70e2}$ $4.76e2\pm 7.56e1$ RBM
$1.49e4\pm 1.42e4$ $1.37e4\pm 1.51e4$ $5.71e3\pm 5.43e3$ $1.06e3\pm 5.36e2$
$4.91e2\pm 1.05e2$ Ours $9.03e3\pm 8.16e3$ $\mathbf{9.96e3\pm 8.28e3}$
$7.18e3\pm 8.90e3$ $1.45e3\pm 1.53e3$ $4.86e2\pm 9.98e1$ 80% AE $8.31e3\pm
9.62e3$ $\mathbf{1.32e4\pm 1.20e4}$ $\mathbf{4.74e3\pm 5.51e3}$
$\mathbf{9.11e2\pm 6.53e2}$ $5.22e2\pm 9.65e1$ RBM $3.00e4\pm 2.83e4$
$1.55e4\pm 2.14e4$ $6.26e3\pm 6.15e3$ $4.72e3\pm 8.42e3$ $5.13e2\pm 1.41e2$
Ours $1.13e4\pm 1.47e4$ $1.54e4\pm 1.46e4$ $6.48e3\pm 4.50e3$ $3.62e3\pm
2.74e3$ $\mathbf{3.98e2\pm 7.89e1}$ 90% AE $1.01e4\pm 5.01e3$ $1.25e4\pm
1.48e4$ $\mathbf{2.99e3\pm 2.53e3}$ $\mathbf{9.04e2\pm 3.92e2}$ $5.06e2\pm
1.74e2$ RBM $1.62e4\pm 1.09e4$ $1.37e4\pm 1.05e4$ $1.22e4\pm 1.37e4$
$1.27e3\pm 7.67e2$ $4.95e2\pm 6.30e1$ Ours $1.50e4\pm 1.18e4$
$\mathbf{1.06e4\pm 1.03e4}$ $7.26e3\pm 8.44e3$ $3.50e3\pm 6.38e3$ $4.89e2\pm
9.89e1$ FD002 2% 10% 20% 40% 100% None $6.95e4\pm 7.74e4$ $2.02e4\pm 1.70e4$
$9.21e3\pm 4.80e3$ $4.79e3\pm 1.95e3$ $1.99e3\pm 6.11e2$ 40% AE $1.25e5\pm
1.32e5$ $\mathbf{1.21e4\pm 5.90e3}$ $8.23e3\pm 4.71e3$ $5.80e3\pm 2.51e3$
$1.86e3\pm 5.80e2$ RBM $7.16e4\pm 7.74e4$ $1.35e4\pm 8.68e3$
$\mathbf{6.08e3\pm 2.18e3}$ $4.33e3\pm 2.30e3$ $\mathbf{1.69e3\pm 2.60e2}$
Ours $1.43e5\pm 8.00e4$ $1.45e4\pm 1.00e4$ $8.20e3\pm 3.92e3$
$\mathbf{4.12e3\pm 2.29e3}$ $2.71e3\pm 1.74e3$ 60% AE $1.01e5\pm 1.03e5$
$\mathbf{1.20e4\pm 5.64e3}$ $\mathbf{6.71e3\pm 2.70e3}$ $5.46e3\pm 2.16e3$
$2.08e3\pm 1.13e3$ RBM $3.92e4\pm 4.56e4$ $1.29e4\pm 7.54e3$ $8.26e3\pm
4.54e3$ $\mathbf{3.10e3\pm 8.06e2}$ $\mathbf{1.82e3\pm 3.69e2}$ Ours
$\mathbf{2.99e4\pm 2.72e4}$ $1.25e4\pm 1.04e4$ $7.69e3\pm 6.09e3$ $3.71e3\pm
9.67e2$ $2.78e3\pm 1.72e3$ 70% AE $7.06e4\pm 9.35e4$ $1.92e4\pm 1.71e4$
$1.02e4\pm 9.86e3$ $5.08e3\pm 1.72e3$ $2.59e3\pm 1.93e3$ RBM $6.14e4\pm
6.51e4$ $\mathbf{1.30e4\pm 4.50e3}$ $7.90e3\pm 5.90e3$ $3.57e3\pm 1.23e3$
$1.76e3\pm 2.73e2$ Ours $\mathbf{3.33e4\pm 4.39e4}$ $1.78e4\pm 2.29e4$
$\mathbf{4.74e3\pm 4.02e3}$ $\mathbf{2.76e3\pm 1.38e3}$ $\mathbf{1.67e3\pm
1.97e2}$ 80% AE $9.93e4\pm 1.01e5$ $1.56e4\pm 1.63e4$ $7.76e3\pm 4.89e3$
$4.58e3\pm 2.32e3$ $1.74e3\pm 2.47e2$ RBM $6.16e4\pm 6.11e4$
$\mathbf{1.02e4\pm 3.96e3}$ $8.27e3\pm 3.91e3$ $\mathbf{3.32e3\pm 1.34e3}$
$\mathbf{1.65e3\pm 2.47e2}$ Ours $\mathbf{3.55e4\pm 5.25e4}$ $1.04e4\pm
4.79e3$ $\mathbf{6.25e3\pm 4.44e3}$ $3.88e3\pm 2.41e3$ $2.13e3\pm 4.91e2$ 90%
AE $7.85e4\pm 7.03e4$ $1.48e4\pm 6.75e3$ $8.13e3\pm 3.31e3$ $5.43e3\pm 3.63e3$
$1.88e3\pm 3.15e2$ RBM $4.95e4\pm 5.63e4$ $1.03e4\pm 6.69e3$ $7.04e3\pm
5.64e3$ $3.44e3\pm 1.30e3$ $1.68e3\pm 2.25e2$ Ours $\mathbf{3.75e4\pm 5.15e4}$
$\mathbf{8.11e3\pm 4.40e3}$ $\mathbf{3.91e3\pm 2.10e3}$ $\mathbf{2.41e3\pm
6.83e2}$ $\mathbf{1.66e3\pm 2.01e2}$

Table 7: RUL-Score results for FD001 and FD002: We report the mean and
standard deviation. The rows represent the _grade of degradation_ and the
columns the _percent of labeled data_ of the data scenario. The second column
contains the pre-training method, where _None_ is the baseline without any
pre-training, and _Ours_ is the self-supervised pre-training. The bold results
mark the best mean performance for each data scenario. If no result is bold,
no approach was able to beat the baseline. Please note that the standard
deviation of these results can be misleading as RUL-Score is an exponential
measure.

FD003 2% 10% 20% 40% 100% None $1.18e8\pm 2.62e8$ $8.96e3\pm 1.20e4$
$1.31e3\pm 6.83e2$ $7.03e2\pm 3.82e2$ $7.48e2\pm 1.71e2$ 40% AE $2.93e4\pm
3.11e4$ $2.02e4\pm 5.66e4$ $1.40e3\pm 9.24e2$ $6.94e2\pm 3.24e2$ $7.13e2\pm
2.08e2$ RBM $9.06e4\pm 1.13e5$ $\mathbf{4.18e3\pm 3.15e3}$ $3.43e3\pm 7.44e3$
$9.74e2\pm 5.84e2$ $9.88e2\pm 3.10e2$ Ours $\mathbf{1.51e4\pm 1.31e4}$
$7.16e3\pm 9.81e3$ $\mathbf{1.02e3\pm 5.83e2}$ $\mathbf{6.20e2\pm 2.76e2}$
$\mathbf{4.95e2\pm 1.12e2}$ 60% AE $4.03e4\pm 6.81e4$ $\mathbf{7.16e3\pm
1.35e4}$ $1.39e3\pm 7.65e2$ $9.76e2\pm 3.91e2$ $7.10e2\pm 1.44e2$ RBM
$4.07e4\pm 6.87e4$ $9.77e3\pm 2.35e4$ $2.64e3\pm 4.86e3$ $1.03e3\pm 6.20e2$
$9.02e2\pm 3.54e2$ Ours $\mathbf{3.35e4\pm 4.47e4}$ $7.24e3\pm 7.05e3$
$1.71e3\pm 1.55e3$ $\mathbf{6.13e2\pm 2.42e2}$ $\mathbf{4.74e2\pm 1.71e2}$ 70%
AE $3.91e4\pm 3.60e4$ $7.12e3\pm 1.50e4$ $\mathbf{1.05e3\pm 4.12e2}$
$1.04e3\pm 6.46e2$ $6.78e2\pm 1.75e2$ RBM $6.09e4\pm 7.69e4$
$\mathbf{3.49e3\pm 3.45e3}$ $1.42e3\pm 6.43e2$ $1.01e3\pm 3.75e2$ $9.28e2\pm
2.79e2$ Ours $\mathbf{1.65e4\pm 1.88e4}$ $1.20e4\pm 3.14e4$ $1.16e3\pm 5.47e2$
$7.17e2\pm 2.83e2$ $\mathbf{5.25e2\pm 1.27e2}$ 80% AE $4.36e4\pm 3.58e4$
$\mathbf{4.11e3\pm 5.66e3}$ $1.21e3\pm 6.58e2$ $8.46e2\pm 3.09e2$ $6.14e2\pm
1.39e2$ RBM $5.96e4\pm 7.59e4$ $8.60e3\pm 1.82e4$ $1.47e3\pm 7.13e2$
$9.10e2\pm 2.92e2$ $1.01e3\pm 2.76e2$ Ours $\mathbf{4.20e4\pm 7.41e4}$
$6.77e3\pm 9.98e3$ $\mathbf{9.61e2\pm 3.88e2}$ $\mathbf{5.96e2\pm 2.08e2}$
$\mathbf{4.59e2\pm 1.39e2}$ 90% AE $3.22e4\pm 4.59e4$ $\mathbf{3.40e3\pm
5.39e3}$ $1.73e3\pm 1.23e3$ $8.26e2\pm 4.42e2$ $6.66e2\pm 8.99e1$ RBM
$8.56e4\pm 8.92e4$ $5.24e3\pm 6.90e3$ $1.97e3\pm 1.67e3$ $9.11e2\pm 5.46e2$
$9.92e2\pm 4.93e2$ Ours $\mathbf{1.31e4\pm 7.50e3}$ $3.44e3\pm 4.80e3$
$\mathbf{1.14e3\pm 5.58e2}$ $\mathbf{6.69e2\pm 2.35e2}$ $\mathbf{5.55e2\pm
9.86e1}$ FD004 2% 10% 20% 40% 100% None $1.63e5\pm 1.58e5$ $2.52e4\pm 2.33e4$
$1.69e4\pm 9.37e3$ $9.15e3\pm 3.29e3$ $6.21e3\pm 1.62e3$ 40% AE $1.44e6\pm
9.82e5$ $1.56e6\pm 8.75e5$ $2.08e6\pm 6.41e5$ $2.27e6\pm 1.34e6$ $1.77e6\pm
9.15e5$ RBM $1.60e5\pm 1.52e5$ $3.90e4\pm 3.43e4$ $1.69e4\pm 8.72e3$
$\mathbf{7.79e3\pm 2.62e3}$ $6.03e3\pm 2.25e3$ Ours $\mathbf{1.02e5\pm
4.04e4}$ $5.25e4\pm 7.18e4$ $\mathbf{1.65e4\pm 1.28e4}$ $1.12e4\pm 7.05e3$
$\mathbf{5.26e3\pm 1.71e3}$ 60% AE $1.10e6\pm 8.70e5$ $1.82e6\pm 8.91e5$
$1.71e6\pm 1.29e6$ $1.94e6\pm 1.00e6$ $1.42e6\pm 6.38e5$ RBM $1.41e5\pm
1.12e5$ $3.66e4\pm 3.26e4$ $\mathbf{1.29e4\pm 5.23e3}$ $8.53e3\pm 5.18e3$
$5.76e3\pm 1.30e3$ Ours $\mathbf{3.69e4\pm 1.60e4}$ $3.04e4\pm 1.76e4$
$1.66e4\pm 1.35e4$ $\mathbf{6.67e3\pm 3.77e3}$ $\mathbf{4.29e3\pm 9.92e2}$ 70%
AE $1.80e5\pm 2.11e5$ $3.86e4\pm 3.61e4$ $1.64e4\pm 7.75e3$ $1.08e4\pm 8.82e3$
$5.73e3\pm 1.43e3$ RBM $1.43e5\pm 1.39e5$ $3.68e4\pm 5.13e4$ $1.53e4\pm
9.11e3$ $9.53e3\pm 4.82e3$ $5.38e3\pm 1.94e3$ Ours $\mathbf{3.52e4\pm 1.87e4}$
$\mathbf{1.54e4\pm 6.76e3}$ $\mathbf{8.43e3\pm 4.92e3}$ $\mathbf{8.52e3\pm
5.06e3}$ $\mathbf{4.25e3\pm 1.01e3}$ 80% AE $1.46e6\pm 1.01e6$ $1.76e6\pm
1.09e6$ $1.37e6\pm 6.03e5$ $1.75e6\pm 6.24e5$ $1.71e6\pm 8.83e5$ RBM
$2.11e5\pm 2.24e5$ $3.41e4\pm 2.14e4$ $1.42e4\pm 8.29e3$ $9.48e3\pm 5.50e3$
$5.13e3\pm 1.09e3$ Ours $\mathbf{2.18e4\pm 2.11e4}$ $\mathbf{1.61e4\pm
1.51e4}$ $\mathbf{1.36e4\pm 1.10e4}$ $\mathbf{5.87e3\pm 2.13e3}$
$\mathbf{4.25e3\pm 1.01e3}$ 90% AE $1.58e5\pm 1.73e5$ $3.28e4\pm 2.23e4$
$1.59e4\pm 1.02e4$ $1.04e4\pm 5.96e3$ $5.74e3\pm 4.26e2$ RBM $1.46e5\pm
1.24e5$ $2.51e4\pm 2.21e4$ $1.61e4\pm 1.16e4$ $1.09e4\pm 5.88e3$ $5.60e3\pm
1.27e3$ Ours $\mathbf{5.05e4\pm 7.02e4}$ $\mathbf{1.14e4\pm 7.05e3}$
$\mathbf{1.17e4\pm 1.27e4}$ $\mathbf{5.52e3\pm 2.95e3}$ $\mathbf{4.04e3\pm
8.03e2}$

Table 8: RUL-Score results for FD003 and FD004: Please consult table 7 for
further information.

###  5.2 Discussion of Findings

Our results revealed several points worthy of further discussion. First of
all, there is the matter of minimal baseline performance drops when using as
few as 40% of the labeled data. A possible explanation for this is the fact
that the C-MAPSS dataset is the product of a simulation which may lead to
little variation between the individual time series. If the variation between
time series is small, the network needs less data to learn useful patterns
from it. The differences between the subsets may be explained by the varying
number of available time series per subset. Additional experiments where the
absolute number of labeled data is varied instead of a percentage can be used
to confirm this hypothesis.

The next point would be the discrepancies between RMSE and RUL-Score
performance best seen on FD004 for the AE pre-training. Even though the RMSE
performance is similar to the other approaches, the RUL-Score is much higher.
This phenomenon can be explained through the asymmetric nature of the RUL-
score, i.e. that late predictions incur a higher penalty than early ones. The
AE pre-trained models seem to make predominantly late predictions, even though
the absolute difference from the real RUL is similar to the other approaches.

Another interesting finding is, that RBM pre-training is competitive with AE
pre-training, even though it pre-trains only the first layer. Both methods use
a reconstruction-based pre-training task, which makes the comparison even more
interesting. An explaining factor could be that the bottleneck size of the
autoencoder was not optimized as a hyperparameter, although it has a
significant influence on the reconstruction capabilities of the AE. We did not
optimize it because changing the bottleneck size would change the number of
model parameters making the comparison with other methods difficult. A bigger
bottleneck may increase performance because of the increased parameter count
independently from pre-training. Additional experiments where the bottleneck
size is optimized for the autoencoder and then compared to the other
approaches on the same model architecture could prove this hypothesis.

The last point is the downward trend in performance for self-supervised pre-
training with respect to the grade of degradation. At least on FD002 and
FD004, we can observe that performance drops at lower grades of degradation
and even makes our approach worse than the baseline in some cases. This
problem may lie in the nature of the pre-text task we choose. It is based on
the assumption that differences in the features of two time frames are
correlated with the difference in RUL. If we take the piece-wise linear
degradation model of C-MAPSS at face value, there are no differences in RUL
above $\textsc{RUL}_{max}$. Consequently, there should not be a difference in
the features either. Our pre-text task on the other hand is only accurate for
a linear degradation model as we cannot take $\textsc{RUL}_{max}$ into account
on unlabeled data. Looking at the percentage of time frames above and below
$\textsc{RUL}_{max}$ for different grades of degradation, we can see that in
FD004 only 11% of the training data has a label below a $\textsc{RUL}_{max}$
of 125 at 40% grade of degradation. Coincidentally, the trend is most obvious
on this subset. The training data of FD001, FD002, and FD003 has 33%, 23%, and
16% labels below 125 at 40% grade of degradation. Less data with labels below
$\textsc{RUL}_{max}$ makes the approximation of our pre-text task less
accurate.

##  6 Conclusion & Future Work

We presented a study of three SSL approaches on the NASA C-MAPSS dataset under
improved, more realistic evaluation conditions. These conditions take into
account that realistic, unlabeled data does not contain features near the
point of failure. Concerning our first research question, our results show
that, contrary to previous studies, SSL does not always improve the accuracy
of RUL estimation. This underlines the importance of a representative
validation set to identify negative outcomes which may not always be available
in settings with few labeled time series.

In answering our second research question, we have shown that our SSL
approach, based on self-supervised pre-training, has superior performance
compared to the competing approaches under certain conditions. More work is
necessary to replicate these findings on other datasets. Nevertheless, our
approach was not able to beat the baseline performance reliably under all data
scenarios. Most notably, the performance dropped when the grade of degradation
was low.

Future work includes conducting the experiments outlined in the discussion
section to test the proposed explanations to the observed phenomena.
Additionally, advanced techniques from the field of metric learning, e.g. hard
sample mining or triplet loss, could be used to improve the self-supervised
pre-training. Theoretically, our approach could be used for Unsupervised
Domain Adaption (DA), too, as it shares many characteristics with SSL.
Unsupervised DA is of high interest for RUL estimation as labeled and
unlabeled data often do not share the same domain. Investigating the
effectiveness of our approach for general, non-linear degradation processes,
e.g. tool wear estimation, is another direction for future work.

## Acknowledgement

This work was funded by the Technische Universität Berlin with support of IAV
GmbH.

## References

  * [1] X. S. Si, W. Wang, C. H. Hu, D. H. Zhou, Remaining useful life estimation - A review on the statistical data driven approaches, European Journal of Operational Research 213 (2011).  doi:10.1016/j.ejor.2010.11.018.
  * [2] J. E. van Engelen, H. H. Hoos, A survey on semi-supervised learning, Machine Learning 109 (2020) 373–440.  doi:10.1007/s10994-019-05855-6.
  * [3] A. Listou Ellefsen, E. Bjørlykhaug, V. Æsøy, S. Ushakov, H. Zhang, Remaining useful life predictions for turbofan engine degradation using semi-supervised deep architecture, Reliability Engineering and System Safety 183 (2019) 240–251.  doi:10.1016/j.ress.2018.11.027.
  * [4] A. S. Yoon, T. Lee, Y. Lim, D. Jung, P. Kang, D. Kim, K. Park, Y. Choi, Semi-supervised learning with deep generative models for asset failure prediction, arXiv (2017).  arXiv:1709.00845.
  * [5] T. Krokotsch, M. Knaak, C. Gühmann, A novel evaluation framework for unsupervised domain adaption on remaining useful lifetime estimation, in: 2020 IEEE International Conference on Prognostics and Health Management (ICPHM), 2020.  doi:10.1109/ICPHM49022.2020.9187058.
  * [6] C. Doersch, A. Gupta, A. A. Efros, Unsupervised visual representation learning by context prediction, in: Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2015.
  * [7] S. Gidaris, P. Singh, N. Komodakis, Unsupervised representation learning by predicting image rotations, in: International Conference on Learning Representation, 2018.
  * [8] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, in: Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol. 1, 2019, pp. 4171–4186.  arXiv:1810.04805.
  * [9] N. Gebraeel, M. Lawley, R. Liu, V. Parmeshwaran, Residual life predictions from vibration-based degradation signals: A neural network approach, IEEE Transactions on Industrial Electronics 51 (2004) 694–700.  doi:10.1109/TIE.2004.824875.
  * [10] S. Zheng, K. Ristovski, A. Farahat, C. Gupta, Long Short-Term Memory Network for Remaining Useful Life estimation, in: IEEE International Conference on Prognostics and Health Management, IEEE, 2017, pp. 88–95.  doi:10.1109/ICPHM.2017.7998311.
  * [11] Y. Wu, M. Yuan, S. Dong, L. Lin, Y. Liu, Remaining useful life estimation of engineered systems using vanilla LSTM neural networks, Neurocomputing 275 (2018) 167–179.  doi:10.1016/j.neucom.2017.05.063.
  * [12] S. Bai, J. Z. Kolter, V. Koltun, An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling, arXiv (2018).  arXiv:1803.01271.
  * [13] G. Sateesh Babu, P. Zhao, X.-L. Li, Deep Convolutional Neural Network Based Regression Approach for Estimation of Remaining Useful Life, Database Systems for Advanced Applications 9642 (2016) 214–228.  doi:10.1007/978-3-319-32025-0.
  * [14] X. Li, Q. Ding, J. Q. Sun, Remaining useful life estimation in prognostics using deep convolution neural networks, Reliability Engineering and System Safety 172 (2018).  doi:10.1016/j.ress.2017.11.021.
  * [15] J. Zhu, N. Chen, W. Peng, Estimation of Bearing Remaining Useful Life Based on Multiscale Convolutional Neural Network, IEEE Transactions on Industrial Electronics 66 (2019).  doi:10.1109/TIE.2018.2844856.
  * [16] J.-R. Jiang, J.-E. Lee, Y.-M. Zeng, Time Series Multiple Channel Convolutional Neural Network with Attention-Based Long Short-Term Memory for Predicting Bearing Remaining Useful Life, Sensors 20 (166) (2020).
  * [17] D. Yarowsky, Unsupervised word sense disambiguation rivaling supervised methods, in: 33rd annual meeting of the association for computational linguistics, 1995, pp. 189–196.
  * [18] C. Cheng, B. Zhou, G. Ma, D. Wu, Y. Yuan, Wasserstein Distance based Deep Adversarial Transfer Learning for Intelligent Fault Diagnosis, arXiv (2019).  arXiv:1903.06753.
  * [19] D. P. Kingma, D. J. Rezende, S. Mohamed, M. Welling, Semi-supervised learning with deep generative models, in: Proceedings of the 27th International Conference on Neural Information Processing Systems-Volume 2, 2014, pp. 3581–3589.
  * [20] A. Rasmus, H. Valpola, M. Honkala, M. Berglund, T. Raiko, Semi-supervised learning with ladder networks, in: Proceedings of the 28th International Conference on Neural Information Processing Systems-Volume 2, 2015, pp. 3546–3554.
  * [21] P. Bachman, O. Alsharif, D. Precup, Learning with pseudo-ensembles, in: Proceedings of the 27th International Conference on Neural Information Processing Systems-Volume 2, 2014, pp. 3365–3373.
  * [22] R. He, Y. Dai, J. Lu, C. Mou, Developing ladder network for intelligent evaluation system: Case of remaining useful life prediction for centrifugal pumps, Reliability Engineering & System Safety 180 (2018) 385–393.  doi:https://doi.org/10.1016/j.ress.2018.08.010.
  * [23] A. Oliver, A. Odena, C. A. Raffel, E. D. Cubuk, I. Goodfellow, Realistic evaluation of deep semi-supervised learning algorithms, Advances in Neural Information Processing Systems 31 (2018) 3235–3246.
  * [24] K. Musgrave, S. Belongie, S.-N. Lim, A metric learning reality check, in: A. Vedaldi, H. Bischof, T. Brox, J.-M. Frahm (Eds.), Computer Vision – ECCV 2020, Springer International Publishing, Cham, 2020, pp. 681–699.
  * [25] J.-Y. Franceschi, A. Dieuleveut, M. Jaggi, Unsupervised scalable representation learning for multivariate time series, in: Advances in Neural Information Processing Systems, Vol. 32, Curran Associates, Inc., 2019, pp. 4650–4661.
  * [26] P. Baldi, Y. Chauvin, Neural networks for fingerprint recognition, neural computation 5 (3) (1993) 402–418.
  * [27] J. Bromley, J. W. Bentz, L. Bottou, I. Guyon, Y. LeCun, C. Moore, E. Säckinger, R. Shah, Signature verification using a “siamese” time delay neural network, International Journal of Pattern Recognition and Artificial Intelligence 7 (04) (1993) 669–688.
  * [28] S. Ioffe, C. Szegedy, Batch normalization: Accelerating deep network training by reducing internal covariate shift, in: F. Bach, D. Blei (Eds.), Proceedings of the 32nd International Conference on Machine Learning, Vol. 37 of Proceedings of Machine Learning Research, PMLR, Lille, France, 2015, pp. 448–456.
  * [29] L. McInnes, J. Healy, N. Saul, L. Großberger, Umap: Uniform manifold approximation and projection, Journal of Open Source Software 3 (29) (2018) 861\.
  * [30] D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, arXiv (2014).  arXiv:1412.6980.
  * [31] F. O. Heimes, Recurrent neural networks for remaining useful life estimation, in: International Conference on Prognostics and Health Management, IEEE, 2008\.

  *[SSL]: Semi-Supervised Learning
  *[PDM]: Predictive Maintenance
  *[RUL]: Remaining Useful Lifetime
  *[RMSE]: Root Mean Squared Error
  *[C-MAPSS]: Commercial Modular Aero-Propulsion System Simulation
  *[DNNs]: Deep Neural Network
  *[LSTM]: Long Short Term Memory Network
  *[LSTMs]: Long Short Term Memory Network
  *[MLPs]: Multi-Layer Perceptron
  *[CNNs]: Convolutional Neural Network
  *[RNNs]: Recurrent Neural Network
  *[CNN]: Convolutional Neural Network
  *[MSE]: Mean Squared Error
  *[DNN]: Deep Neural Network
  *[AEs]: Autoencoder
  *[RBM]: Restricted Boltzmann Machine
  *[ReLU]: Rectified Linear Unit
  *[MLP]: Multi-Layer Perceptron
  *[UMAP]: Uniform Manifold Approximation and Projection
  *[ RUL]: Remaining Useful Lifetime
  *[HPC]: high-pressure compressor
  *[AE]: Autoencoder
  *[RBMs]: Restricted Boltzmann Machine
  *[IQRs]: interquartile range
  *[IQR]: interquartile range
  *[DA]: Domain Adaption