# Temporal Aggregation for the Synthetic Control Method††thanks: Prepared for
2024 AEA P&P session “Treatment Effects: Theory and Implementation”

Liyang Sun, Eli Ben-Michael, and Avi Feller Sun: Department of Economics,
University College London and CEMFI, Email<EMAIL_ADDRESS>Ben-Michael:
Department of Statistics & Data Science and Heinz College of Information
Systems & Public Policy, Carnegie Mellon University. Feller: Goldman School of
Public Policy & Department of Statistics, University of California, Berkeley.
We thank Elizabeth Stuart for sharing data for the application. AF and LS
gratefully acknowledge support from the Institute of Education Sciences, U.S.
Department of Education, through Grant R305D200010. LS also acknowledges
support from Grant PID2022-143184NA-I00 funded by MCIN/AEI. See the associated
augsynth library for R.

###### Abstract

The synthetic control method (SCM) is a popular approach for estimating the
impact of a treatment on a single unit with panel data. Two challenges arise
with higher frequency data (e.g., monthly versus yearly): (1) achieving
excellent pre-treatment fit is typically more challenging; and (2) overfitting
to noise is more likely. Aggregating data over time can mitigate these
problems but can also destroy important signal. In this paper, we bound the
bias for SCM with disaggregated and aggregated outcomes and give conditions
under which aggregating tightens the bounds. We then propose finding weights
that balance both disaggregated and aggregated series.

##  1 Introduction

Empirical researchers often use the synthetic control method (SCM) to estimate
the impact of a treatment on a single unit in panel data settings (Abadie,
Diamond and Hainmueller, 2010). The “synthetic control” is a weighted average
of control units that balances the treated unit’s pre-treatment outcomes as
closely as possible.

Two challenges arise when using SCM with higher frequency data, such as when
the outcome is measured every month versus every year. First, because there
are more pre-treatment outcomes to balance, achieving excellent pre-treatment
fit is typically more challenging. Second, even when excellent pre-treatment
fit is possible, higher-frequency observations raise the possibility of bias
due to overfitting to noise. A recent review by Abadie and i Bastida (2022)
explicitly cautions about such bias from using disaggregated outcomes in SCM.
Instead, researchers can first aggregate the outcome series into lower-
frequency (e.g., annual) observations, and then estimate SCM weights that
minimize the imbalance in these aggregated pre-treatment outcomes. Doing so
mechanically improves pre-treatment fit as well.

In this paper, we propose a framework for temporal aggregation for SCM.
Adapting recent results from Sun, Ben-Michael and Feller (2023), we first
derive finite-sample bounds on the bias for SCM under a linear factor model
when using temporally disaggregated versus aggregated outcome series. With
these bounds, we then show that temporal aggregation only reduces bias to the
extent that doing so reduces noise without also overly reducing signal for the
underlying factors. The optimal trade-off between the two approaches depends
on unknown parameters. We argue, however, that finding synthetic control
weights that jointly balance both the disaggregated and aggregate series is a
promising compromise approach for practice.

Our setup builds on an expansive literature on SCM, especially recent papers
that modify SCM to mitigate bias both due to imperfect pre-treatment balance
(e.g., Ferman and Pinto, 2021; Ben-Michael, Feller and Rothstein, 2021) and
bias due to overfitting to noise (e.g., Kellogg et al., 2021). Most directly
relevant, Sun, Ben-Michael and Feller (2023) discuss using use multiple
outcomes to mitigate both sources of bias.

##  2 Setup

For each unit $i=1,\dots,N$ and at each lower-frequency time interval
$t=1,\dots,T$, we observe $K$ higher-frequency observations of the outcome
denoted as $Y_{itk}$ where $k=1,\dots,K$. For instance, in a monthly series
$t=1,\dots,T$ could represent years and $k=1,2,\dots,12$ the months within
each year. The choice of time interval $t$ reflects how we compute temporal
aggregates, such as aggregating monthly data into yearly averages. We maintain
fixed values for $N$ and $K$ throughout the analysis.

We denote the exposure to a binary treatment by $W_{i}\in\\{0,1\\}$. We
restrict our attention to the case where a single unit receives treatment, and
follow the convention that this is the first one, $W_{1}=1$. The remaining
$N_{0}\equiv N-1$ units are possible controls, often referred to as “donor
units.” To simplify notation, we limit to one post-treatment period,
$T=T_{0}+1$, though our results easily extend to larger $T$.

We denote the potential outcome under treatment $w$ with $Y_{itk}(w)$. We are
interested in the treatment effects for the treated unit during the $K$ higher
frequency observations during post-treatment period $T$:
$\tau_{Tk}=Y_{1Tk}(1)-Y_{1Tk}(0)$ for $k=1,\dots,K$. Since we directly observe
$Y_{1Tk}(1)=Y_{1Tk}$ for the treated unit, we focus on imputing the missing
counterfactual outcome under control, $Y_{1Tk}(0)$.

Throughout, we will focus on _de-meaned_ or _intercept-shifted_ weighting
estimators (Doudchenko and Imbens, 2017; Ferman and Pinto, 2021). We denote
$\bar{Y}_{i\cdot\cdot}\equiv\frac{1}{T_{0}K}\sum_{t=1}^{T_{0}}\sum_{j=1}^{K}Y_{itj}$
as the pre-treatment average for the outcome for unit $i$, and
$\dot{Y}_{itk}=Y_{itk}-\bar{Y}_{i\cdot\cdot}$ as the corresponding de-meaned
outcome. We consider estimators of the form:
$\widehat{Y}_{1Tk}(0)\equiv\bar{Y}_{1\cdot\cdot}+\sum_{i=2}^{N}\gamma_{i}\dot{Y}_{iTk},$
where $\gamma\in\mathcal{C}\subset\mathbb{R}^{N-1}$ is a set of weights. Our
paper centers on how to choose the weights $\gamma$ from a set
$\mathcal{C}=\\{\gamma\in\mathbb{R}^{N-1}\mid\|\gamma_{i}\|_{1}\leq
C,\sum_{i}\gamma_{i}=1\\}$ for a known $C$.

The first approach we consider is finding a synthetic control that has the
best pre-treatment fit on the (de-meaned) disaggregated high-frequency
outcomes $q^{\text{dis}}(\cdot)$. We refer to this set of weights as the
_disaggregated weights_ $\hat{\gamma}^{\text{dis}}$:

$\underset{\gamma\in\mathcal{C}}{\min}\frac{1}{T_{0}}\frac{1}{K}\sum_{k=1}^{K}\sum_{t=1}^{T_{0}}\left(\dot{Y}_{1tk}-\sum_{W_{i}=0}\gamma_{i}\dot{Y}_{itk}\right)^{2}.$

An alternative choice is to optimize the _aggregated objective_
$q^{\text{agg}}(\cdot)$, the pre-treatment fit for the temporally aggregated
outcomes via averaging. We refer to the set of weights that minimize this
objective as the _aggregated_ weights $\hat{\gamma}^{\text{agg}}$:

$\underset{\gamma\in\mathcal{C}}{\min}\frac{1}{T_{0}}\sum_{t=1}^{T_{0}}\left(\frac{1}{K}\sum_{k=1}^{K}\dot{Y}_{1tk}-\sum_{W_{i}=0}\gamma_{i}\dot{Y}_{itk}\right)^{2}.$

##  3 Bias bounds

To derive finite sample bias bounds for $Y_{1Tk}(0)-\widehat{Y}_{1Tk}(0)$ for
each $k=1,\dots,K$, we assume that the outcomes under control are generated as
$Y_{itk}(0)=\alpha_{i}+\beta_{tk}+L_{itk}+\varepsilon_{itk}$, with
$\sum_{t=1}^{T}\beta_{tk}=0$ for all $k$; see, for example, Athey et al.
(2021). After incorporating the additive two-way fixed effects, the model
component retains a term $L_{itk}$ with $\sum_{i=1}^{N}L_{itk}=0$ for all
$t,k$ and $\sum_{t=1}^{T}L_{itk}=0$ for all $i,k$. We assume the idiosyncratic
errors $\varepsilon_{itk}$ are mean zero sub-Gaussian random variables with
scale parameter $\sigma$, independent of the treatment status $W_{it}$. We
also assume independence across units and time, which is plausible if the
model components capture co-movement in the outcome.

Let the matrix $L\in\mathbb{R}^{N\times(TK)}$ contain $L_{itk}$ for the
treated unit and the remaining rows correspond to control units. Sun, Ben-
Michael and Feller (2023) show that under a low rank assumption $rank(L)<N-1$,
there exist oracle weights $\gamma^{\ast}$ over donor units that yield an
unbiased estimate for the control potential outcome:
$\mathbb{E}_{\varepsilon_{Tk}}\left[Y_{1Tk}(0)-\widehat{Y}_{1Tk}(0)\right]=L_{1Tk}-\sum_{i=2}^{N}\gamma^{\ast}_{i}L_{iTk}=0.$
Here the expectation is taken over the idiosyncratic errors in the respective
post-treatment periods.

The deterministic model component can be written as a linear factor model,
$L_{itk}=\boldsymbol{\phi}_{i}\cdot\boldsymbol{\mu}_{tk},$ with $r=rank(L)$,
where $\boldsymbol{\mu}_{tk}\in\mathbb{R}^{r}$ are latent time factors and
each unit has a vector of time invariant factor loadings
$\boldsymbol{\phi}_{i}\in\mathbb{R}^{r}$. Two important quantities for our
discussion are $\underline{\xi}^{\text{dis}}$ and
$\underline{\xi}^{\text{agg}}$, the smallest singular values of the variance-
covariance matrix of, respectively, the time factors $\mu_{tk}$ and the
averaged time factors $\bar{\mu}_{t}=\frac{1}{K}\sum_{k=1}^{K}\mu_{tk}$.
Following previous literature we assume that $\underline{\xi}^{\text{dis}}>0$,
which avoids issues of weak identification (Abadie, Diamond and Hainmueller,
2010).

For estimated weights $\hat{\gamma}$ based on pre-treatment fit, the bias for
the effect in period $Tk$ is due to inadequate balance in the model
components:
$Bias(\hat{\gamma})=L_{1Tk}-\sum_{i=2}^{N}\hat{\gamma}_{i}L_{iTk}.$ We can
decompose the bias from estimated weights $Bias(\hat{\gamma})$ into two terms
using the linear factor model:

$\displaystyle\sum_{t=1}^{T_{0}}\sum_{j=1}^{K}{\omega}_{tj}\left(\dot{Y}_{1tj}-\sum_{W_{i}=0}\hat{\gamma}_{i}\dot{Y}_{itj}\right)\
\ (\text{imbalance})$
$\displaystyle-\sum_{t=1}^{T_{0}}\sum_{j=1}^{K}{\omega}_{tj}\left(\dot{\varepsilon}_{1tj}-\sum_{W_{i}=0}\hat{\gamma}_{i}\dot{\varepsilon}_{itj}\right)\
\ (\text{overfitting})$

where $\omega_{tj}$ are transformations of the factor values that depend on
the estimator.

The first term is bias due to imperfect pre-treatment fit (or imbalance) in
the pre-treatment outcomes, $\dot{Y}_{itj}$. The second term is bias due to
overfitting to noise, also known as the approximation error. This term arises
because the optimization problems minimize imbalance in _observed_ pre-
treatment outcomes — noisy realizations of latent factors — rather than
minimizing imbalance in the latent factors themselves.

Theorem 1 in the appendix formally states high-probability bounds on the bias
terms, which we obtain using results from Sun, Ben-Michael and Feller (2023).
These bounds hold in finite samples and account for imperfect pre-treatment
fit. The leading terms in the bias due to imbalance and overfitting are:

$\displaystyle|Bias(\hat{\gamma}^{\text{dis}})|=$ $\displaystyle
O\left(\frac{1}{\underline{\xi}^{\text{dis}}}\right)+O\left(\frac{1}{\underline{\xi}^{\text{dis}}\sqrt{T_{0}K}}\right)$
$\displaystyle|Bias(\hat{\gamma}^{\text{agg}})|=$ $\displaystyle
O\left(\frac{1}{\underline{\xi}^{\text{agg}}\sqrt{K}}\right)+O\left(\frac{1}{\underline{\xi}^{\text{agg}}\sqrt{T_{0}K}}\right)$

where $\underline{\xi}^{\text{dis}}$ and $\underline{\xi}^{\text{agg}}$ are
the relevant smallest singular values for the disaggregated and aggregated
series.

The first term, bias due to imbalance, does not vanish for either approach.
Consistent with Ferman and Pinto (2021), bias remains because the SCM
objective does not converge to the objective minimized by the oracle weights
due to noise in the outcomes. Aggregation, however, reduces the level of noise
in the objective and therefore reduces the bias by a factor of $1/\sqrt{K}$.

The second term in the bias is the contribution of overfitting to noise. This
decreases in the total number of pre-treatment observations $T_{0}K$ for
_both_ the disaggregated and aggregated estimators. For the disaggregated
weights this follows prior results (e.g., Abadie, Diamond and Hainmueller,
2010). Aggregation reduces the noise in the objective, but this is
counteracted by a commensurate reduction in the number of pre-treatment
observations, netting out to the same risk of bias due to overfitting as with
the disaggregated weights.

In practice, the most important consideration in temporal aggregation is
whether doing so eliminates the signal for the underlying factors. In
particular, if (ignoring constant terms)
$\sqrt{K}\underline{\xi}^{\text{agg}}>\underline{\xi}^{\text{dis}}$,
aggregation will lead to tighter bias bounds. There are many scenarios where
we expect long-run variation to persist, especially after seasonally adjusting
the outcomes, and therefore believe this condition might hold. However, if
there is not substantial long-run variation and $\underline{\xi}^{\text{agg}}$
is very small, aggregating can leave little behind to learn about the latent
factor loadings, and possibly inflate the bias bound. Similar challenges arise
in time series model estimation, where aggregation can lead to biased
estimates for the true time series models (see Marcet, 1991, among others).

Rather than choose between these two extremes, we propose finding SCM weights
that control a linear combination of the two objectives, with weight $\nu$ on
the aggregated objective and weight $(1-\nu)$ on the disaggregated objective.
This creates an imbalance “frontier”, similar to the approach in Ben-Michael,
Feller and Rothstein (2022), where $\nu=0$ corresponds to the disaggregated
objective and $\nu=1$ to the aggregated objective. As formalized in Lemma 1 in
the appendix, if the SCM weights yield excellent pre-treatment fit on _both_
the disaggregated and aggregated outcomes, these weights will also achieve the
minimum of the two bounds. In general, finding the optimal $\nu^{\ast}$
involves model-derived parameters and is infeasible. As a heuristic, we
propose equal weight to the disaggregated and aggregated fits, $\nu=0.5$, and
then assessing sensitivity to this choice.

There are many directions for future work that incorporate recent innovations
in panel data methods, including first de-noising (e.g., Amjad, Shah and Shen,
2018) or seasonally adjusting the disaggregated outcome series. We could also
explore choosing an optimal level of temporal aggregation for a single SCM
objective. Finally, questions about temporal aggregation also arise in event
study and other panel data models, suggesting further avenues for fruitful
research.

##  4 Texas 2021 Abortion Restrictions

We revisit the Bell, Stuart and Gemmill (2023) study on the SB8, a 2021 Texas
law restricting abortion. Using monthly state-level live birth counts from
2016 to 2022, the authors construct a synthetic Texas to estimate SB8’s impact
on monthly births.

Figure 1: Imbalance “frontier”

Note: The pre-treatment fit for births aggregated to the yearly level (y-axis)
against the pre-treatment fit for births at the monthly level (x-axis) as we
vary the relative weight placed on balancing average yearly births from 0 (top
left corner) to 1 (bottom right corner). “Yearly + Monthly” denotes the fit
when equal weight is placed on both the aggregated and disaggregated series.

The original analysis uses SCM on monthly data ($\nu=0$). We explore
aggregating to yearly averages ($\nu=1$) and intermediate values of $\nu$ as
illustrated in Figure 1. Combining yearly and monthly births equally
($\nu=0.5$) achieves substantial balance on both fronts, reducing potential
bias. Appendix Figure 1 indicates a slightly larger estimated effect for post-
treatment monthly births using the combined approach ($\nu=0.5$) than the
original analysis. Appendix Figure 2 demonstrates stable estimates across a
wide range of $\nu$.

## References

  * (1)
  * Abadie, Diamond and Hainmueller (2010) Abadie, Alberto, Alexis Diamond, and Jens Hainmueller. 2010\. “Synthetic Control Methods for Comparative Case Studies: Estimating the Effect of California’s Tobacco Control Program.” Journal of the American Statistical Association, 105(490): 493–505.
  * Abadie and i Bastida (2022) Abadie, Alberto, and Jaume Vives i Bastida. 2022\. “Synthetic Controls in Action.” arXiv:2203.06279.
  * Amjad, Shah and Shen (2018) Amjad, Muhammad, Devavrat Shah, and Dennis Shen. 2018\. “Robust Synthetic Control.” Journal of Machine Learning Research, 19(22).
  * Athey et al. (2021) Athey, Susan, Mohsen Bayati, Nikolay Doudchenko, Guido Imbens, and Khashayar Khosravi. 2021\. “Matrix Completion Methods for Causal Panel Data Models.” Journal of the American Statistical Association, 116(536): 1716–1730.
  * Bell, Stuart and Gemmill (2023) Bell, Suzanne O., Elizabeth A. Stuart, and Alison Gemmill. 2023\. “Texas’ 2021 Ban on Abortion in Early Pregnancy and Changes in Live Births.” JAMA, 330(3): 281–282.
  * Ben-Michael, Feller and Rothstein (2021) Ben-Michael, Eli, Avi Feller, and Jesse Rothstein. 2021\. “The Augmented Synthetic Control Method.” Journal of the American Statistical Association, 116(536): 1789–1803.
  * Ben-Michael, Feller and Rothstein (2022) Ben-Michael, Eli, Avi Feller, and Jesse Rothstein. 2022\. “Synthetic controls with staggered adoption.” Journal of the Royal Statistical Society. Series B: Statistical Methodology, 84(2): 351–381.
  * Doudchenko and Imbens (2017) Doudchenko, Nikolay, and Guido W. Imbens. 2017\. “Difference-In-Differences and Synthetic Control Methods: A Synthesis.” arxiv 1610.07748.
  * Ferman and Pinto (2021) Ferman, Bruno, and Cristine Pinto. 2021\. “Synthetic Controls with Imperfect Pre-Treatment Fit.” Quantitative Economics.
  * Kellogg et al. (2021) Kellogg, Maxwell, Magne Mogstad, Guillaume A Pouliot, and Alexander Torgovitsky. 2021\. “Combining matching and synthetic control to tradeoff biases from extrapolation and interpolation.” Journal of the American statistical association, 116(536): 1804–1816.
  * Marcet (1991) Marcet, Albert. 1991\. “Temporal aggregation of economic time series.” In Rational expectations econometrics. , ed. Thomas J Sargent and Lars Peter Hansen, 237–282. Westview Press.
  * Sun, Ben-Michael and Feller (2023) Sun, Liyang, Eli Ben-Michael, and Avi Feller. 2023\. “Using Multiple Outcomes to Improve the Synthetic Control Method.” arXiv:2311.16260.

Online Appendix to the paper

Temporal Aggregation for the Synthetic Control Method

Liyang Sun, Eli Ben-Michael, and Avi Feller

######  Theorem 1.

In addition to assumptions stated above, suppose some oracle weights exist in
the set $\mathcal{C}$. Denote $\boldsymbol{\mu}_{tk}\in\mathbb{R}^{r}$ as the
time factors and assume that they are bounded above by $M$. Furthermore,
denoting $\sigma_{\text{min}}(A)$ as the smallest singular value of a matrix
$A$, assume that (i)
$\sigma_{min}\left(\frac{1}{T_{0}K}\sum_{tk}\mu_{tk}\mu_{tk}^{\prime}\right)\geq\underline{\xi}^{\text{dis}}>0$;
and (ii)
$\sigma_{min}\left(\frac{1}{T_{0}}\sum_{t}\left(\bar{\mu}_{t}\right)\left(\bar{\mu}_{t}\right)^{\prime}\right)\geq\underline{\xi}^{\text{agg}}>0$
where $\bar{\mu}_{t}=\frac{1}{K}\sum_{k=1}^{K}\mu_{tk}$. For any $\delta>0$,
let $\tilde{\sigma}=\left(2C\sqrt{\log
2N_{0}}+(1+C)\delta\right)(1+1/\sqrt{T_{0}K})\sigma$, with probability at
least
$1-8\exp\left(-\frac{\delta^{2}}{2}\right)-4\exp\left(-\frac{T_{0}K\delta^{2}}{2\sigma^{2}(1+C^{2})}\right)$,
the absolute bias satisfies the bound,

$\displaystyle\left|\text{Bias}(\hat{\gamma}^{\text{dis}})\right|$
$\displaystyle\leq\frac{rM^{2}}{\underline{\xi}^{\text{dis}}}\left(4(1+C)\sigma+2\delta+\frac{\tilde{\sigma}}{\sqrt{T_{0}K}}\right),$
(1) $\displaystyle\left|\text{Bias}(\hat{\gamma}^{\text{agg}})\right|$
$\displaystyle\leq\frac{rM^{2}}{\underline{\xi}^{\text{agg}}}\left(\frac{4(1+C)\sigma}{\sqrt{K}}+2\delta+\frac{\tilde{\sigma}}{\sqrt{T_{0}K}}\right).$
(2)

###### Proof.

Results follow directly from Theorem 1 of Sun, Ben-Michael and Feller (2023)
where $\hat{\gamma}^{\text{dis}}$ correspond to the concatenated weights, and
$\hat{\gamma}^{\text{agg}}$ correspond to the average weights in their
notation. ∎

######  Lemma 1.

Suppose there exists $\nu^{\ast}\in[0,1]$ such that for
$\hat{\gamma}^{\text{com}}\in\arg\min_{\gamma\in\mathcal{C}}\nu^{\ast}q^{\text{dis}}(\gamma)+(1-\nu^{\ast})q^{\text{agg}}(\gamma)$,
we have $q^{\text{dis}}(\gamma^{\text{com}})\leq
q^{\text{dis}}(\gamma^{\ast})$ and $q^{\text{agg}}(\gamma^{\text{com}})\leq
q^{\text{agg}}(\gamma^{\ast})$ almost surely. For any $\delta>0$, with
probability at least
$1-8\exp\left(-\frac{\delta^{2}}{2}\right)-4\exp\left(-\frac{T_{0}K\delta^{2}}{2\sigma^{2}(1+C^{2})}\right)$,
the absolute bias satisfies the bound,

$\displaystyle\left|\text{Bias}(\hat{\gamma}^{\text{com}})\right|\leq\min$
$\displaystyle\left\\{\frac{rM^{2}}{\underline{\xi}^{\text{dis}}}\left(4(1+C)\sigma+2\delta+\frac{\tilde{\sigma}}{\sqrt{T_{0}K}}\right),\frac{rM^{2}}{\underline{\xi}^{\text{agg}}}\left(\frac{4(1+C)\sigma}{\sqrt{K}}+2\delta+\frac{\tilde{\sigma}}{\sqrt{T_{0}K}}\right)\right\\}.$

###### Proof.

Since $q^{\text{dis}}(\gamma^{\text{com}})\leq q^{\text{dis}}(\gamma^{\ast})$
and $q^{\text{agg}}(\gamma^{\text{com}})\leq q^{\text{agg}}(\gamma^{\ast})$
almost surely, either of the two bias bounds stated in Theorem 1 is a valid
upper bound for the estimate based on the combined weights
$\hat{\gamma}^{\text{com}}$. We may therefore take the minimum of the two
bounds to bound $|Bias(\hat{\gamma}^{\text{com}})|$. ∎

Figure A1: Balance only in monthly vs in both monthly and yearly Figure A2:
Change in average estimated impact on monthly births as the relative weight on
monthly vs yearly births varies from 0 to 1.


# Failure Identification from Unstable Log Data using Deep Learning

Jasmin Bogatinovski1, Sasho Nedelkoski1, Li Wu1, Jorge Cardoso2, Odej Kao1
1Distributed and Operating Systems, Technical University Berlin, Germany
2Huawei Munich Research, Munich, Germany
{jasmin.bogatinovski<EMAIL_ADDRESS>
###### Abstract

The reliability of cloud platforms is of significant relevance because society
increasingly relies on complex software systems running on the cloud. To
improve it, cloud providers are automating various maintenance tasks, with
failure identification frequently being considered. The precondition for
automation is the availability of observability tools, with system logs
commonly being used. The focus of this paper is log-based failure
identification. This problem is challenging because of the instability of the
log data and the incompleteness of the explicit logging failure coverage
within the code. To address the two challenges, we present CLog as a method
for failure identification. The key idea presented herein based is on our
observation that by representing the log data as sequences of subprocesses
instead of sequences of log events, the effect of the unstable log data is
reduced. CLog introduces a novel subprocess extraction method that uses
context-aware neural network and clustering methods to extract meaningful
subprocesses. The direct modeling of log event contexts allows the
identification of failures with respect to the abrupt context changes,
addressing the challenge of insufficient logging failure coverage. Our
experimental results demonstrate that the learned subprocesses representations
reduce the instability in the input, allowing CLog to outperform the baselines
on the failure identification subproblems – 1) failure detection by 9-24% on
F1 score and 2) failure type identification by 7% on the macro averaged F1
score. Further analysis shows the existent negative correlation between the
instability in the input event sequences and the detection performance in a
model-agnostic manner.

###### Index Terms:

failure identification; system reliability; log data; cloud computing; deep
learning;

††publicationid: pubid:  ©2022 IEEE. Permission from IEEE must be obtained for
all uses, in any current or future media, including reprinting/republishing
this material for advertising or promotional purposes. This paper is accepted
at IEEE CCGrid 2022. For citations use references from the conference
proceedings.

##  I Introduction

Cloud systems are a mixture of complex multi-layered software and hardware.
They enable applications of ever-increasing heterogeneity and complexity
powering different technologies such as the Internet of Things, distributed
processing frameworks, databases, virtual reality, among others. The emergence
of complexity within the cloud relates to diverse maintenance challenges, with
an important challenge of being prone to failures [1]. The failures have a
significant impact on the performance affecting user experience and leading to
economic losses [2]. Therefore, accurate and timely failure identification is
crucial for enhancing the reliability of the cloud and its services.

Cloud providers are considering many approaches to address the problem of
failure identification, commonly by adopting various data-driven methods [3].
Their fundament resides in the available monitoring data, with log data (logs
for short) commonly being utilized. Other monitoring data, like key
performance indicator metrics (KPI, e.g., memory utilization, I/O bytes),
provide clues for detecting failures, however, they do not provide a verbose
description of the type of the failure [4] making the failure identification
incomplete. For example, a sharp increase in the curve of memory utilization
only indicates that the memory utilization increases, but it cannot tell why
it happens in isolation. In comparison, logs are textual data recording events
with different granularity, providing human-understandable clues for the
failure and its type. For example, from several repetitions of the two
consecutive log lines, “$l_{1}$: Interface changed state to up.” and ”$l_{2}$:
Interface changed state to down.”, operators can detect a failure in the
system, assign its type “Interface Flapping”, conclude that the interface is
flapping and obtain clues for potential root causes (bad cable connection a
common suspect in this example). A single log is composed of a static event
template describing the event (e.g., ”Interface changed state to
$\langle*\rangle$.”) and parameters (e.g., up) giving variable event
information.

The focus of this study is the problem of log-based failure identification.
Traditionally, it is addressed by manual analysis, like keyword search of
failure words (e.g., ”fail”) or log levels with great severity (e.g., ”error”)
[5]. Owning to the unprecedented development of the cloud systems, logs are
consistently generated in large volumes (several TB per day [3]), making the
task of manual log-based diagnosis cumbersome. Thereby, automatic approaches
for log-based failure identification are increasingly researched and adopted
[6, 7, 8, 9, 10].

Current approaches 1) identify failures from single log lines [10] or 2)
exploit groups/contexts of log events (i.e., co-occurring event
templates/events) in form of log sequences [11] (i.e., series of event
templates with external identifier) and count vectors [7]. Depending on the
assumed input, different challenges emerge. When considering groups of log
events, the challenge of unstable log sequences occurs [6]. Unstable log
sequences are sequences from the same type of workload execution having
slightly different sequential structures. Figure 1 depicts examples of
unstable sequences caused by different reasons. In the sequences of events
denoted with ”Event Duplication” and ”Missing Event”, the original sequence
$(E2,E5,E1,E4,E6)$ is modified by repeating a single event ”E4” or dropping
the event ”E1”, accordingly. The two sequences still represent normal system
behaviour but have slightly different structures. Such problems are common in
cloud systems where the log data is analyzed at a central place. The network
errors, limited throughput, or storage issues are referenced causes for events
repeating or dropping. There are other sources of instabilities (e.g., the
preprocessing of raw logs), altering the normal log sequences in a similar
way. Notably, the instability causes similar properties of the unstable normal
and failure sequences (e.g., shortened lengths or contexts differ in a single
event), making it harder to distinguish them from one another. For example,
the two sequences in Figure 1, labeled ”Misidentifying Event” and ”Failure”,
differ just within one event on the fourth position (”E7” and ”E3”). The first
arises due to an error in the log event preprocessing, while the second is
because the template describes a failure event. Therefore, the instability
inflicts a modeling challenge and increases the entropy in the data. From
modeling perspective, this requires accounting for the unequal importance of
the log events within the contexts, impairing the detection performance
otherwise [6].

Figure 1: Examples of unstable and failure log event sequences.

The methods using single log lines do not suffer from the problem of unstable
logs, predominantly due to incorporating information about the semantics of
the log events [8]. These approaches demonstrate strong performance [10]
however, they cannot detect failures that are not explicitly logged. For
example, in the aforenamed failure with type ”Interface Flapping” (with logs
$l_{1},l_{2}$), none of the two logs has a log level with greater severity
(i.e., ”error” or ”critical”), nor do they explicitly describe a failure. The
failure can be detected just within the context of several repetitions of the
specific pair of logs. Furthermore, these types of contextual failures occur
often. For example, for the release Pike (version 3.12.1) of a popular cloud
resource managing system OpenStack, there are more than 20% of failures not
explicitly logged within a single log line [12]. Acknowledging that developers
have an insufficient understating of the complexities of the running system
environment during development results in insufficient failure logging
coverage [3]. Conclusively, the failures that do not manifest in individual
lines make the failure identification possible only within the context of
other logs (e.g., presence/absence of frequently co-occurring logs).

Contributions. 1) To overcome the two challenges, in this paper as the main
contribution, we introduce CLog – a method for log-based failure
identification. The key idea of CLog is reducing the instability of the input
log event sequences by representing them as sequences of subprocesses (i.e.,
groups of similar contexts). Since subprocesses represent contexts (co-
occurring log event templates), their number is significantly smaller than the
event number used to represent the sequences. The two key benefits of the
change in the representation are that – a) by representing the log event
sequences by a smaller number of subprocesses, we directly reduce the entropy
in the input representation, reducing the effect of the unstable log
sequences; b) the modeling of contexts allows detecting failures in terms of
abrupt context changes, addressing the challenge of insufficient logging
failure coverage. The challenge that arises is the extraction of subprocesses.
2) To address it, we contribute a novel method for unsupervised subprocesses
extraction based on context-aware deep learning and clustering methods. 3) Our
experimental results on two datasets from OpenStack (with 172 failures)
demonstrate that CLog outperforms the baselines on the two failure
identification subproblems: failure detection (by 9-24% on F1 score) and
failure type identification (by 7% on macro average F1 score). By injecting
unstable event sequences, we show CLog’s robust performance dropping by just
6% under a severe ratio of unstable sequences. 4) Finally, we contribute by
open-sourcing the datasets and method for reproducibility purposes and
fostering the research on this practically relevant problem.

The remaining of the paper is structured as follows. Section II gives the key
observation for the approach, alongside the problem definition. Section III
describes the proposed methodology. Section IV discusses the experimental
results in response to four research questions. Section V discusses the
related work for the two sub-problems of failure detection and failure type
identification. Section VI concludes the paper and gives directions for future
work.

##  II Preliminary

###  II-A Problem Definition

In this paper, we address the problem of log-based failure identification [5].
We decompose the problem into two subproblems, i.e., (1) failure detection and
(2) failure type identification, defined in the following.

Failure Detection (FD). Let $L=\\{l_{1},l_{2}\dots l_{i}\dots l_{n}\\}$ be a
set of $n$ time-ordered logs from cloud services, and there exist an index set
$\mathbb{J}\in\mathbb{N}$ capturing dependency relation between the logs,
i.e., $s_{j}=(l_{ji}\in L|j\in\mathbb{J})$, where $l_{ji}$ denotes individual
log of the sequence $s_{j}$. Further, we assume that there exist a function
$p^{+}$ denoting the normality score of the sequence
$p^{+}(\phi(s_{j})):\mathbb{R}^{d}\mapsto\mathbb{R}$, where
$\phi:\mathbb{S}\mapsto\mathbb{R}^{d}$ is the representation function of
sequence $s_{j}$ into $d$-dimensional numerical vector space, and $\mathbb{S}$
is the available sequence set. The task of failure detection is defined as
finding the set
$A=\\{s_{j}\in\mathbb{S}|a_{1}<p^{+}(s_{j})||p^{+}(s_{j})>a_{2},j\in\mathbb{J}\\}$,
where $a_{1},a_{2}$ are constants such that $a_{1}<a_{2}$. Although the
individual logs $l_{i}$ in the sequence $s_{j}$ can describe normal events,
the overall sequence can denote a failure. The index set $\mathbb{J}$ in the
context of logs can represent task ID, process ID, or workload ID. It can be
given apriori (as considered here) or reconstructed by an additional
procedure. We assume that the majority of the log messages $l_{i}$ and the
sequences $s_{j}$ describe normal system behaviour.

Failure Type Identification (FTI). Given a set of detected failure sequences
$A$ and the set of failure types identifiers $T=\\{t_{1},t_{2}\dots t_{w}\\}$,
where $w$ denotes the number of unique failure type identifiers, the task of
failure type identification is finding a function $f(\phi(s_{i})):A\mapsto T$.

Failure Identification (FI). Given the sets $L,T,\mathbb{S}$, and
$\mathbb{J}$, the task of failure identification is finding the set
$\tilde{A}=\\{(s_{1},t_{1}),\dots(s_{i},t_{i})\dots(s_{|A|},t_{|A|})\\}$,
where the failure sequence $s_{i}$ is detected by estimating the normality
score function $\tilde{p}^{+}(\tilde{\phi}(s_{i}))$ and the thresholds
$\tilde{a_{1}}$ and $\tilde{a_{2}}$, while its type $t_{i}$ is identified by
the estimate of $\tilde{f}(\tilde{\phi}(s_{i}))$. The estimates of
$\tilde{\phi}(s_{i})$, $\tilde{p}^{+}(\tilde{\phi}(s_{i}))$ and
$\tilde{f}(\tilde{\phi}(s_{i}))$ further are used for representing, and
failure identification on novel sequences. CLog addresses the problem by
finding suitable representation for the sequences $\tilde{\phi}(s_{i})$
(Section III-B), which are used to find estimates for $\tilde{p}^{+}$ (Section
III-C1) and $\tilde{f}$ (Section III-C2).

Figure 2: Entropy (measuring instability) increases with more unique symbols.

###  II-B Key Observation

The key observation CLog relies on is reducing the entropy of the input
representation of log sequences by representing them as sequences of
subprocesses instead of log events. For example, a sequence of log events
$s_{i}=(E_{1},E_{2},E_{5},E_{3},E_{1})$ with a task ID $i$, where each of
$\\{E_{1},E_{2},E_{3},E_{5}\\}$ denotes log event template, can equivalently
be represented as sequence of two subprocesses/symbols $s_{i}=(S_{a},S_{b})$
such that $S_{a}=(E_{1},E_{2},E_{5})$, and $S_{b}=(E_{3},E_{1})$, while
$S_{a}$ and $S_{b}$ are referred to as subprocesses. Figure 2 depicts the
impact of changing the log event sequence representation into sequences of
subprocesses on data from OpenStack. It is seen that with increasing the
grouping window size, the number of log events in the subprocesses increases,
and the average entropy over all the sequences in the data also increases (the
triangles). The entropy is the highest when the whole sequence is represented
with individual log events (the diamond). Notably, when the sequences are
represented with subprocesses, the number of symbols used to represent the
sequences is smaller (the circles), and the average entropy is reduced. It
implies a reduction of the sensitiveness over the individual logs, effectively
reducing the effect of the instability in the log event sequences. An
important goal of CLog is to learn subprocesses by preserving their
characteristics, i.e., by learning context-aware event sequence groups.

##  III CLog: Method for Log-based Failure Identification

To address the problem of failure identification, we propose CLog. Figure 3
gives an overview of the method. It has three parts 1) log parsing, 2)
context-aware subprocesses extraction, and 3) failure identification. The log
parsing, as a general preprocessing procedure in log analysis [13], extracts
the event templates from the incoming raw log event sequences, transforming
the raw log sequences into sequences of log events. The event sequences are
processed by the context-aware subprocesses extraction part, converting them
into sequences of subprocesses. This part leverages our observation that by
representing the log sequences on a level of subprocesses, the entropy of the
representation sequence is smaller. This increases the stability of the input
and reduces the impact of the unstable logs. Finally, the processed log
sequences of subprocesses are given as input into the failure identification
part. The latter is composed of two modules (a) failure detector and (b)
failure type identification (FTI). The goal of the failure detector is to
detect the failure sequences of subprocesses. The FTI module further
identifies the failure types based on operators experience. CLog has two modes
of operation: offline and online. During the offline phase, the parameters of
the log parser, context-aware subprocesses part, and failure identification
parts are learned, and the models are induced and stored. In the online phase,
the stored models are loaded and used to identify failures. In the following,
we describe the internal mechanisms of the three parts of CLog in detail.

Figure 3: CLog overview.

###  III-A Log Parsing

The generated raw system logs are unstructured. Since we are interested in
modeling the sequences of events, as the first step, we extract the event
templates from the raw logs by applying automatic log parsing. Log parsing
decouples the log templates from log parameters (the variable part in a log),
directly extracting input in useable modeling format. We prefered automatic
log parsing because the alternatives (e.g., regular expression and ”grock”
patterns), although successful for parsing the templates, are system-specific,
requiring frequent updates, which makes them challenging for maintenance [13].

While there are many log parsers available, CLog utilizes a tree-based parser
Drain [14]. Zhu et al. [13] identified Drain among the most efficient in
comparison to 12 other parsers on ten benchmark datasets from diverse software
systems. The three main properties making it popular and widely adopted are
its correctness, efficiency and intuitive meaning of the hyperparameters
(making the tuning process undemanding). Once parsed, the events are organized
in sequences by CLog’s hyperparameter window size, corresponding to the
arriving time interval and task ID of the events, and are proceeded as output
towards the context-aware subprocess extraction part.

###  III-B Context-aware Subprocess Extraction

The context-aware subprocess extraction is the central part of the method. Its
goal is the extraction of subprocesses from the parsed log event sequences. By
representing an execution workload (with a task ID) on a higher-level
granularity, i.e., by subprocesses, we reduce the entropy in the input,
addressing the problem of unstable log event sequences. This part combines
context-aware neural network and clustering methods to learn explicit
relationships between the events within the event sequences preserving their
local properties. Figure 4 depicts the overall design of the context-aware
subprocess extraction part with a running example. Conceptually, it is
composed of three submodules – (1) preprocessing submodule that transforms the
input sequences into a format suitable for learning, (2) neural network
learning module which is combined with a batched kmeans method to learn
subprocesses in an unsupervised manner, (3) subprocess extraction module that
assigns a unique subprocess identifier to the input event sequence. In the
following, we describe the submodules.

Figure 4: The internal design of context-aware subprocess extraction part
(with a detailed explanation of a running example).

####  III-B1 Preprocessing Submodule

The goal of the preprocessing submodule is to preprocess the input log event
sequences in a unified format suitable for the neural network. It has two
components: padding and masking.

Padding. The padding component receives the sequences of log events as input,
with each event represented by a unique identifier (e.g., integer). We refer
to it as a token. The sequences of events in a given time interval are
different in length. However, the neural network requires a fixed-size
representation of the input. The padding component specifies a hyperparameter
$max\\_length$ and appends each of the shorter log sequences with a dedicated
token $[PD]$ up to $max\\_length$ to enforce fixed-size representation. The
longer sequences (having more than $max\\_length$ events) are truncated.
Notably, we add a dedicated token $[LSE]$ (Log Sequence Embedding) at the
beginning of each sequence. During learning, we enforce the sequence token
representations to propagate through the upper layers in the network via the
$[LSE]$ token. Thereby, $[LSE]$ attends over all the tokens from the sequence
and summarizes the relevant context during learning. The $[LSE]$ token serves
as a sequence vector representation later used to group contexts and identify
subprocesses. The output of the padding module is the prepended and padded
event sequence.

Masking. To learn context-aware groups, we consider a general self-supervised
learning task from natural language processing (NLP) research called Masked
Language Modeling (MLM) [15]. To apply the MLM task, the masking component is
processing the prepended and padded log event sequences in a suitable format.
More specifically, as input, it receives the prepended and padded log event
sequences and outputs a set of pairs of masked log event sequences and
original masked events. Masked log event sequences are sequences of log events
created by replacing all of the events from an original log sequence with a
special $[M]$ (masked) token. For example, for the input sequence
$(E_{2},E_{5},E_{3})$, one masked sequence is $(E_{2},[M],E_{3})$, with
$E_{5}$ being the original masked event. There are three masked event
sequences for this example. $[LSE]$ and $[PD]$ tokens are not affected by the
masking procedure. During training, a masked sequence is given as input to the
neural network, while the original masked token is used as the prediction
target. By predicting the mask from the co-occurring tokens, the method learns
the most important events from the surrounding context, extracting context-
aware representations. Note that by this procedure single input sequence is
multiplied several times. We keep track of the origin (the input event
sequence) of each masked sequence and use it to extract its corresponding
subprocess identifier (see Section III-B3).

####  III-B2 Neural Network Submodule

The neural network submodule learns context-aware groups of masked log
sequences. It implements a neural network following the design of a self-
attention encoder of the Transformer [15] architecture. The advantage given by
this architectural choice resides in its capability to learn the contextual
information between the input events. When learning the parameters of the
network, guided by a carefully designed cost function, the model learns local
relationships between the events based on their co-occurrence extracting
useful contextual features. The neural network submodule has four components:
vectorizer, encoder block, output layers and masked subprocess id assignment.
The vectorizer transforms the masked input event sequences of tokens into
numerical vector sequences. These vectors are called event embeddings and are
part of the training procedure. At the beginning of the training procedure,
the event vectors are randomly initialized and are updated during training.
This way, they learn contextual information about the events.

The encoder block is composed of a self-attention encoder layer. The self-
attention extracts co-occurring information by weighting the input vector
embeddings by their similarity to all the other embeddings in the given
context. Combining the self-attention with the MLM learning task modifies the
parameters of the network to learn the context of the original masked event
and extract sequential properties. The hyperparameters of the encoder are the
model size (denoted by $d$), the number of encoder layers, the number of
heads, and the dropout ratio (used to prevent overfitting). We reference the
reader to Devlin et al. [15] for the specific details on self-attention neural
networks. Particularly interesting is the embedding of the $[LSE]$ token.
Since $[LSE]$ serves as an embedding of the sequence, it learns contextual
properties of the masked input sequences. The output from the encoder is the
vector embedding of the $[LSE]$ token for each of the masked sequences,
proceeded towards the output layer.

The output layer is composed of two layers with nonlinear activation (RELU is
used). The purpose of it is to map the masked sequence embedding vector
$[LSE]$ of size $d$, to a vector with a size corresponding to the total number
of events/tokens $C$. The output of this layer is used to calculate the loss.
As an optimization loss function, we use categorical cross-entropy. Notably,
during the execution of a workload, some events occur only once (e.g.,
”notification of successful creation of a VM”) while others in greater
frequency (e.g., HTTP or RPC calls). When using the original loss formulation
on the MLM task, the less frequent events will be averaged out, resulting in
missing important information. To account for the imbalances of the
distribution of the events, we use weighted categorical cross-entropy given in
Eq. 1 as follows:

$J_{m}(\psi(\mathbf{s^{m}_{n,c}};\theta,\theta^{{}^{\prime}}),y^{m}_{n,c};\mathbf{w})=\frac{1}{|C|}\sum_{c=1}^{C}-w_{c}y^{m}_{n,c}log\frac{exp(\psi(\mathbf{s^{m}_{n,c}};\theta,\theta^{{}^{\prime}}))}{\sum_{i=1}^{C}exp(\psi(\mathbf{s^{m}_{n,i}};\theta,\theta^{{}^{\prime}}))}$
(1)

where $\psi$ denotes the function modeled by the neural network, $\theta$ and
$\theta{{}^{\prime}}$ are the parameters of the encoder, and the output layer
accordingly, $\mathbf{s^{m}_{n,c}}$ is a masked sequence obtained from the
$n$-th input sequence $\mathbf{s_{n}}$, $y^{m}_{n,c}$ is the original masked
event/token, $C$ denotes the total token numbers and $w_{c}$ represents the
weight of an individual token. The weights ($\mathbf{w}$ – a weights vector)
are assigned such that the less frequent events have weight values closer to
1, as opposed to the frequent ones that have values closer to 0. Therefore, we
optimize for preserving the correct predictions on the infrequent events,
addressing the challenge of the imbalance of the event frequency distribution.

Masked Subprocess ID assignment. The masked subprocess ID assignment receives
the vector embedding of the $[LSE]$ token as input. It applies the mini
batched kmeans algorithm [16] to group the embeddings of the masked event
sequences into a predetermined number of $\mathbf{k}$ subprocesses/centroids
identifiers. The mini batched kmeans algorithm is a commonly used method for
identifying similar instance groups in an unsupervised way. While the goal of
the encoder block is to learn context-aware representations, the mini batched
kmeans complements it by extracting similar context groups, enabling the
extraction of subprocesses. We used mini batch kmeans because it allows per
batch update of the network $(\theta$ and $\theta^{{}^{\prime}})$ and
clustering parameters (M) as opposed to the classical kmeans method. To group
the contexts, kmeans optimizes the loss given in Eq. 2 by altering between two
steps: 1) updating a centroid $m_{k}$ as the average of the embeddings
currently assigned to it, and 2) reassignment of the embeddings to the nearest
newly calculated centroid.

$J_{k}(\phi(\mathbf{s^{m}_{n}},\mathbf{r_{n}};\theta),\mathbf{M})=||\phi(\mathbf{s^{m}_{n}};\theta)-\mathbf{r_{n}}\mathbf{M}||_{2}$
(2)

where $\mathbf{M}\in\mathbb{R}^{kxd}$ represent the matrix of subprocess
context-group (interchangeably referred to as centroids), while
$\mathbf{r_{n}}$ is an indicator vector of discrete values (0’s and 1’s) with
just one element set to one, corresponding to the membership of the masked
sequence $\mathbf{s^{m}_{n}}$ to a certain centroid $m_{k}$. The number of
subprocesses identifiers $\mathbf{k}$ is a hyperparameter.

Finally, we add the two optimization losses as $J=J_{m}+\lambda J_{k}$ to
obtain the final loss subject to optimization. By combined optimization of the
two losses, the parameters of the context-aware subprocess extraction learn
local contexts and local-context groups based on their similarity. The role of
the hyperparameter $\lambda$ is to ensure learning of correct contexts and
correct context-embedding groups by trading off the impact of the two losses.
We further discuss the optimization procedure.

Optimization. The optimization is done in two phases: 1) pretraining and 2)
joint training. We first describe the pretraining phase. Since at the
beginning everything is initialized at random, we pre-train the neural network
parameters ($\theta$ and $\theta^{{}^{\prime}}$) by the weighted cross-entropy
loss (Eq. 1). That way, the model learns good initial parameters for the
encoder while extracting context-aware features for the masked sequences. The
pretraining is terminated after observing a lack of improvement in the loss on
five consecutive epochs. At the end of the pretraining, the $[LSE]$ vectors
are valid representations of the masked input sequences. Afterwards, the
subprocesses prototypes ($\mathbf{M}$) are initialized by kmeans using $[LSE]$
masked sequence embeddings of the training data.

Joint training (phase 2). The joint optimization function has a discrete
variable ($\mathbf{r_{n}}$), making the parameter updates non-trivial. To
address this issue, we calculate the gradients by alternating stochastic
gradient descent (ASGD) [16]. ASGD alters the updates of the network
parameters and centroids such that, when the network parameters are updated,
the centroids are fixed and vice versa. Therefore, the optimization problem
does not depend on the discrete variable, enabling the parameter updates. The
training of the network parameters and the centroids is done in batches. Eq. 3
is used for centroids update. At each batch, the centroids with newly assigned
embeddings are slightly updated based on their distance to the newly
calculated centroids. Additionally, some of the centroids are updated more
frequently than others making the loss convergence slower. Inspired by Yang et
al. [16], we resolve this issue by penalizing the updates with the term
$\frac{1}{c_{k}}$. $C_{k}$ counts the number of times a cluster is assigned an
embedding during an epoch. The larger the number of assigned embeddings, the
smaller is the centroid updated and vice versa. It normalizes the intensity of
the centroid update as a learning rate, different for each cluster.

$m_{k}\leftarrow
m_{k}-\frac{1}{c_{k}}(\phi(\mathbf{s^{m}_{n}};\theta)-m_{k})\mathbf{r_{n}}$
(3)

####  III-B3 Subbprocess Exctraction

The extraction of a subprocess identifier (ID) is done as follows. Given an
original input event sequence and the subprocess ID assignments of its masked
subsequences, we count the number of occurrences of the subprocesses IDs and
divide the counts by the length of the original input event sequence. The
subprocess ID with the highest score value is assigned as a subprocess ID for
the input event sequence. Intuitively, if the majority of the masked
subsequences are assigned with a single subprocess ID, the subprocess ID with
the maximal score value is the most relevant for the input event sequence.
Figure 4 depicts an example of extracting the subprocess $S_{1}$ for the
sequence $(E_{2},E_{5},E_{3})$.

###  III-C Failure Identification

The failure identification part is given sequences of subprocesses with the
same task ID as input. Figure 5 depicts the internal design. It is composed of
two subparts 1) failure detector and 2) failure type identifier. The failure
detector detects if the input sequence of subprocesses represents failure.
When failure is detected, the sequence proceeds towards the failure type
identification part, which identifies the type of failure based on prior
historical information. We describe the details in the following.

####  III-C1 Failure Detection

As a modeling choice for the failure detector, we considered Hidden Markov
Model (HMM) [17]. HMM, models the sequences of subprocesses by assuming that
the appearance of the next subprocess within the sequence depends only on the
current subprocess. The main advantages of HMM are that it directly handles
sequential data, does not require further preprocessing of the input, and is
fast for both learning and inference (with a reasonably high number of hidden
states). To produce normality score estimates for a sequence
$\tilde{p}^{+}(\mathbf{s_{i}})$, we used HMM probability scores
($t(\mathbf{s})$), calculated by marginalizing the probabilities over all the
subprocesses of the sequence and the hidden states of the fitted HMM
$t(\mathbf{s})=-\log\sum_{h}q(h)q(\mathbf{\mathbf{s}}|h)$, where $h$ denotes
the hidden states, and $q(\textbf{s}|h)$ denotes the likelihood of the
subprocess given the hidden state. The normality score estimates for a single
sequence $\mathbf{s_{i}}$ is given in Eq. 4, as follows:

$\tilde{p}^{+}(\mathbf{s_{i}})=(\frac{1}{|\mathbb{V}|}\sum_{s_{j}}^{|\mathbb{V}|}t(\mathbf{s_{j}})-t(\mathbf{s_{i}}))^{2}$
(4)

Figure 5: Internal architectural design of failure identification part.

where $\mathbb{V}$ is the set of validation normal sequences of subprocesses.
The normality score estimate $\tilde{p}^{+}(\mathbf{s_{i}})$ is a symmetric
positive function, given as the spread of the probability of the sequence
$\mathbf{s_{i}}$ under the HMM ($t(s_{i})$) from the mean score estimates of
the validation data. The parameters of the HMM are learned on the normal
training data, thereby, the failure detector models the normal system state.
We assume that the normal data is always obtainable from the periods of system
operation when there are no issue reports or log events with ”error” or
”critical” log levels. Any sequence with significantly different values for
the normality score estimate is detected as a failure. Using the symmetrical
property of the normality function, we estimate the thresholds as
$\tilde{a}_{1/2}=\mu\pm 3\sigma$, where $\mu$ and $\sigma$ are the mean value
and standard deviation of the validation score estimates calculated by
standard formulas. Thereby, the failure detector is fully unsupervised. The
number of hidden states is one hyperparameter of HMM.

####  III-C2 Failure Type Identification

Once the failure is detected, the failure sequence proceeds towards the
failure type identification module. This module leverages the redundancy
property of failures in cloud systems [5]. This property emerges for various
reasons, including temporary failure fixes by developers without addressing
the root cause, environmental issues (e.g., machine failure or network
disconnections), or running the same system in different environments.
Notably, the redundancy implies repetitive patterns in logs, allowing usage of
operational information to identify the failure type. The failure type
identification subpart has two components (1) feature extraction and (2)
failure type identification model.

The feature extraction processes the sequences of subprocesses in a format
suitable for the FTI learning method. Each sequence is represented by a count
vector that counts the number of occurrences of a subprocess within the
sequence. For example, for the sequence of subprocesses
$\mathbf{s}=(S_{1},S_{3},S_{1})$ and total of four subprocesses
($S_{1},S_{2},S_{3},S_{4}$), the count vector is given as
$CV(\mathbf{s})=(2,0,1,0)$. The absence/presence of certain subprocesses from
the sequence (e.g., lack of the subprocess with the event ”Failure to spawn an
instance.”) are distinctive features that discriminate among failure types.
Therefore, the count vector is a suitable sequence representation. We also
considered the normality score estimates from the failure detector as an
additional feature (pHMM).

The extracted features are used to fit the FTI model given by
$\tilde{f}(s_{i})$. FTI learns a multiclass classification model that
classifies the input sequences into several predefined types of failures. As
an adequate methods we considered several popular multiclass classification
methods, i.e., Random Forest (RF) [18], Decision Tree (DT) [19], Logistic
Regression (LR) [20] and AdaBoost [21]. They show good performance and do not
require extensive hyperparameter optimization [18]. The final output of CLog
is given as
$\tilde{A}=\\{(\mathbf{s_{j}},t_{i})|\mathbf{s_{j}}\in\mathbb{S},t_{i}\in\mathbb{T},\tilde{p}^{+}(\mathbf{s_{j}})<\tilde{a_{1}}||\tilde{p}^{+}(\mathbf{s_{j}})>\tilde{a_{2}},t_{i}=\tilde{f}(\mathbf{s_{j}}),j\in
J\\}$.

TABLE I: Dataset Statistics

Dataset name | Number of Tasks | No Failure | Assertion Failures | Number of Log Messages | Number of Unique Events | Number of Failure Events |  | Average number of events
---
fault-free task
OpenStack | 878 | 706 | 172 | 217534 | 518 | 167 | 1323
Syntetic | 500 | 421-476 | 24-79 | 167215 | 474 | 123 | 1309

##  IV Experimental Evaluation

In this section, we describe the experimental evaluation. We give details
about the experimental design, present and discuss the experimental results in
response to four research questions.

###  IV-A Experimental Design

####  IV-A1 OpenStack dataset

To evaluate CLog, we considered a large scale study of failures in OpenStack,
introduced in Cotroneo et al. [12]. To the best of our knowledge, it is the
most comprehensive publicly available dataset of log failure data from a cloud
system. Its strength is the wide range of covered failures following the most
common problem reports in the OpenStack bug repository. The faults are
generated by software fault-injection procedure, i.e., modifying the source
code of OpenStack and running a predefined workload under fault-injected and
fault-free (normal) conditions.

The considered fault types are grouped into four groups as of following: 1)
throw exception (method raises an exception in accordance to a predefined API
list), 2) wrong return value (method returns an incorrect value, e.g., return
null reference), 3) wrong parameter value (calling a method with an incorrect
value for a parameter), and 4) delay (method returns the result after a long
delay, e.g., caused by hardware failure – leading to triggering timeout
mechanisms or stall). As a running workload with a unique task ID, the authors
considered the creation of a new instance deployment. This workload configures
a new virtual infrastructure from scratch – it creates VM instances, volumes,
key pairs, and security groups, virtual network, assigns instance floating
IPs, reboots the instances, attaches the instances to volumes and deletes all
resources. Importantly, this comprehensive workload invokes the three key
services of OpenStack Nova, Cinder, and Neutron, causing diverse
manifestations of the faults as failures.

To generate ground truth labels for the failure state, assertion and API
checks are performed at the end of the workload runs. There are three failure
types: 1) failure instance, 2) failure SSH and 3) failure attaching volume.
While the authors provide information on a granularity of a workload with a
task ID, we further labeled the individual logs. More specifically, two human
annotators labeled more than 200000 logs to find the ones related to the
logged failure. The agreement between the annotators is 0.67 Cohen’s Kappa
score. TABLE I gives the detailed statistics of the used data.

Syntetic dataset: To evaluate the robustness of our method in dealing with
unstable log data, we have created a synthetic dataset. The data is created
similarly as in [6]. We start with the normal OpenStack dataset and apply the
following three operations to extract failure sequences, i.e., 1) random
removal of log events, 2) repetition of a randomly selected log event in the
sampled log sequence, and 3) random shuffling the order of several events. To
inject unstable log event sequences, we randomly sample 500 log sequences
(normal and failed), and in $b$-percentage in the sampled data, we inject the
aforenamed operations in random order.

####  IV-A2 Baselines

We compare the failure detection method against three unsupervised baselines
(two sequential-based DeepLog [11], HMM [17], and one count-based PCA [22]),
and two methods commonly used in practice by developers [5]. Those are ”Log
Level”, which uses the severity level of the log (i.e., failure exist if the
log level is one of ”error”, ”fatal”, or ”critical”) and ”Semantic” based on
the semantics of a log (i.e., a human identifies the failure as logged in a
single log line) [23]. Recent study [8] identifies DeepLog as having a state-
of-the-art performance among unsupervised methods. Additionally, we considered
a supervised automatic failure identification method LogRobust [6], that
requires labels for the severity level of the sequences. For failure type
identification, we compare against LogClass [4], which trains a multiclass
model on individual logs to identify failures types. We used task ID failure
type alongside the annotations of the single logs to construct a target label
and apply this method.

####  IV-A3 Experimental Setup

We conducted the experiments as follows. The hyperparameters of the log parser
Drain, i.e., the similarity threshold and depth, were set to 0.45 and 5 as
commonly used values for OpenStack logs [13]. For phase 1 the training was
performed for a maximal of 200 epochs, and phase 2 training for a maximal of
20 epochs. As an optimizer, we used SGD with a learning rate set to 0.0001.
For the encoder, the model size $d$ was set to 128, with two encoder layers
and four heads. To prevent overfitting, we set the dropout rate to 0.01.
Experimentally, we find that $\lambda$ with value 0.1 leads to robust results.
The optimized hyperparameters of CLog (performed on a separate validation set)
are the number of extracted subprocesses, the window size, and the number of
hidden states of the HMM. They were selected from the range values of the sets
$\\{10,20,30,40,50\\}$, $\\{60s,120s,180s,240s,300s\\}$ and $\\{2,4,8,16\\}$
accordingly. The $max\\_length$ was set to 32. The hyperparameters of the
considered FTI methods set are to their implementation defaults from the
sckit-learn library. The baselines for failure detection were trained
following a survey [2] of log-based failure detection from software systems.
LogClass was trained as in the original paper [4]. The failure detection
performance was evaluated on F1, precision and recall as common evaluation
metrics, with the failure being a positive label. The same performance scores
were used for FTI, with macro averaging over the three failure types. The
experiments were conducted on a Linux server with Intel Xeon(R) 2.40GHz CPU
and RTX 2080 GPU running with Python 3.6 and PyTorch 1.5.0.

###  IV-B Research Questions

TABLE II: Comparison of CLog against baselines on failure detection. Scores/Category | Unsupervised | Developer Practicies | Supervised
---|---|---|---
Methods: | CLog | HMM | PCA | DeepLog | Semantic | Log Level | LogRobust
F1 |  0.94±0.02 | 0.82±0.09 | 0.77±0.05 | 0.85±0.03 | 0.81±0.0 | 0.70±0.02 | 0.96±0.01
Precision |  0.97±0.03 | 0.8±0.11 | 0.82±0.07 | 0.78±0.02 | 1.0±0.0 | 0.74±0.02 | 0.94±0.02
Recall | 0.91±0.03 | 0.84±0.10 | 0.73±0.06 |  0.93±0.03 | 0.66±0.0 | 0.65±0.02 | 0.98±0.02

####  IV-B1 RQ1: How does CLog compare against baselines on the task of
failure detection?

We evaluate CLog detection performance against three unsupervised methods, two
commonly used developer practices and one supervised method. The training is
done on 60% randomly sampled normal sequences, while the thresholds (and other
hyperparameters) are selected on a random sample of 20% normal sequences. The
rest of the sequences are used to report the performance scores. The best
results for CLog are obtained for a total of 10 subprocesses (centroids), two
hidden states in the HMM and a window size of 180 seconds111The code and the
data are given in the GitHub repository of the project
https://github.com/context-aware-Failure-Identification/CLog.git. The
experiments are repeated ten times to reduce the assessment bias of the
results. We report the mean and standard deviation of the results.

Since CLog is an unsupervised method, we first discuss the results between
CLog and the unsupervised baselines. TABLE II shows the results. CLog
outperforms the unsupervised baselines by margins between 9-17% on the F1
score. Importantly, CLog and HMM both use HMM to model the sequences, but they
differ in the granularity of the input representation. Marginalizing over the
learning method suggest that changing the input representation of the log
event sequences with sequences of subprocesses is beneficial. Combining these
results with our observation (see Figure 2) demonstrates that reducing the
entropy by changing the input representation improves the detection
performance. CLog predominantly improves the precision over the sequence-based
methods (DeepLog and HMM), while having strong performance on recall. The
input of the sequential-based baselines has larger entropy which challenges
the discrimination against normal sequences, leading to many of them being
detected as failures, i.e., increasing the false positives. Comparing CLog
against the quantitative-based method (PCA) leads to good performance in
precision but reduced recall. The count vector representation and the limited
modeling power of PCA (as a linear model) are potential causes for the
incorrect detection of the true failures. Notably, from an economic
perspective, the observed improvements are significant because improving
failure detection by even 0.1% in F1 score can save hundreds of thousands of
dollars [9].

The improved performance of CLog against the two commonly used developer
practices, i.e., log level (by 24% on F1) and single line semantic-based
approach, is mainly due to the problem of insufficient logging failure
coverage. The semantic-based baseline is constructed based on the expert
inspection of the logs for failures, i.e., it detects all of the single-line
logged failures. However, as we observed when performing the manual log
analysis, and as shown in Cotroneto et al. [12], around 20% of the failures in
the dataset are not explicitly logged. In comparison, CLog can detect non-
logged failures because it models different contexts, i.e., its correlates
events co-occurring together. The violations of these contexts (e.g., an
expected log event is missing from the context) are informative in implicitly
detecting non-logged failures. The log level-based approach experiences the
lowest performance. Despite the problem of insufficient failure coverage, it
further suffers from the problem of wrong log level assignment [23]. A log may
be assigned a log level ”ERROR” but still describe a normal event. Therefore,
relying on the log level leads to reporting more failures than there are,
affecting the precision.

Finally, comparing CLog against the supervised baseline LogRobust, suggests
that CLog has a drop in performance by 2% on the F1 score while for others, it
exceeds 11%. However, LogRobust requires labeled log sequences to build a
model. Due to a large number of logs constantly being generated, the labeling
is often infeasible in practice, and it is the most common referenced critique
of the supervised methods [2]. Therefore, CLog has better practical properties
because of the high detection performance and unsupervised design.

TABLE III: Comparison of CLog against baselines on FTI.

Scores |  Multiclass method |  | CLog
---
(pHMM rep.)
| CLog
---
(CV rep.)
| CLog
---
(CV+pHMM
combined)
| LogClass
---
(TFILF)
F1 | RF | 0.74±0.0 | 0.86±0.01 | 0.86±0.01 | 0.84±0.05
DT | 0.74±0.0 | 0.78±0.04 | 0.78±0.03 | 0.84±0.08
LR | 0.72±0.0 | 0.86±0.0 |  0.87±0.0 | 0.8±0.1
AdaBoost | 0.69±0.0 | 0.86±0.02 |  0.87±0.02 | 0.62±0.12
Precision | RF | 0.74±0.0 | 0.86±0.01 | 0.86±0.01 | 0.83±0.05
DT | 0.74±0.0 | 0.78±0.04 | 0.78±0.03 | 0.82±0.07
LR | 0.71±0.0 | 0.85±0.0 |  0.87±0.0 | 0.77±0.08
AdaBoost | 0.71±0.0 | 0.86±0.02 | 0.86±0.02 | 0.59±0.13
Recall | RF | 0.75±0.0 | 0.87±0.01 | 0.87±0.01 | 0.86±0.06
DT | 0.75±0.0 | 0.81±0.04 | 0.81±0.03 | 0.85±0.07
LR | 0.73±0.0 | 0.88±0.0 |  0.89±0.0 | 0.88±0.11
AdaBoost | 0.73±0.0 | 0.87±0.02 | 0.88±0.02 | 0.74±0.14

####  IV-B2 RQ2: How effective is CLog for the problem of failure type
identification?

This RQ evaluates the capability of the FTI module of CLog to reuse the
historical information from the operator in detecting different types of
failures. Specifically, we evaluate three representations of the subprocess
sequences for CLog (1. probability score from the HMM (pHMM), 2. count vectors
(CV), and 3. combination of both) against LogClass [4] as a baseline. LogClass
uses single logs as input. We randomly sample 60% of the labeled failure
sequences/logs from the original dataset to train the multiclass model, while
the remaining 40% are used for evaluation. To reduce the bias due to the
sampling, we repeated the experiments 30 times. We report the average
performance scores and their standard deviation over the different methods,
evaluating the representations independent of the methods.

TABLE III enlists the results of the three different representations of CLog
and the baseline on the FTI subproblem. The analysis of the three
representations by CLog suggests that the combination (CV+pHMM) achieves the
best F1 score. Predominantly, the improvement originates from the count
vectors, seen by the better individual results in comparison with pHMM.
Finally, the combination (CV+pHMM) of CLog outperforms the baseline LogClass.
LogClass uses single logs to identify the type of failures. Therefore, if the
failure is not explicitly logged, LogClass cannot identify its type. On the
contrary, CLog considers the occurrence of the individual subprocesses and can
represent discriminative patterns among the types of failures improving the
performance.

Figure 6: Impact of the window size over the detection performance.

####  IV-B3 RQ3: How does the entropy of the sequences influence the failure
detection performance independent of the model?

With this question, we verify the impact of the unstable log event sequences
over the failure detection performance in a model agnostic manner. Following
our key observation given in Section II-B, we grouped the input events into
time-intervals of increasing window size and evaluated several models of CLog,
DeepLog, and PCA similarly as in RQ1. The best average performance of CLog
averaged over the window sizes (not individual as in RQ1) is obtained for 30
subprocesses.

Figure 6 shows the results. It can be observed that as the window size
increases, the detection performance decreases. Paring the average entropy
over the sequences for different window sizes (see Figure 2) with the
detection results reveals a negative correlation between the increased entropy
and failure detection performance. Owning to the greater modeling power CLog,
and DeepLog show a relatively lower drop in performance in the range 4-5%. In
comparison, PCA has a performance drop between 8-50%. Importantly, CLog
outperforms DeepLog because of the smaller instability in the input.

####  IV-B4 RQ4: How robust is CLog on the problem of failure detection?

We conduct experiments on the synthetic dataset. The failure detector was
trained on the original data as in RQ1. We randomly inject $b-$percentages
unstable sequences using the synthetic dataset generation procedure previously
described. TABLE IV shows the results. CLog’s detection method preservers high
detection performance even under a high ratio of unstable sequences. The
results demonstrate that the extracted subprocesses are sufficiently sensitive
to the local changes within the log sequences, making the performance robust.

TABLE IV: CLog Failure Detection Evaluation on Synthetic Data injection ratio | F1 | Precision | Recall
---|---|---|---
5% | 0.94 | 0.97 | 0.91
10% | 0.92 | 0.95 | 0.89
15% | 0.90 | 0.95 | 0.86
20% | 0.88 | 0.94 | 0.83

##  V Related Work

Failure Detection. There are plenty of works considering the problem of log-
based failure detection. Considering the assumption of whether log labels are
available or not, the methods are categorized into: unsupervised and
supervised. We first discuss the unsupervised methods. They are considered
practically useful because they do not require labels [2]. Predominantly,
these are one-class methods modeling the normal system state and reporting
failures when significant deviations occur. Yamanishi et al. [17] introduce an
unsupervised sequential method for failure detection that uses HMM on log
event sequences to model the normal state. The probability under the HMM is
used as a normality score. DeepLog [11] trains a neural network – LSTM on
sequences of log events on the auxiliary task of next event prediction. If the
output prediction of the auxiliary task is wrong, the method reports failure.
LogAnomaly [24] is similar to DeepLog, but further augments the input of
DeepLog with semantic and count-based features. Another popular method is PCA
[22]. It is a reconstruction-based method that constructs subspace from the
count vectors of the normal log event sequences and uses the reconstruction
error to detect failures. Compared to other methods that do not directly
address the problem of unstable sequences, CLog addresses it by representing
the sequences of log events with sequences of subprocesses. The semantic-based
methods use the semantics of the single logs to identify failures. Despite the
traditionally used approaches in industry, like keyword search (e.g., search
for ”error”) or log level search [5], another line of works learns properties
of failure logs. One such example is Logsy [10]. It combines labeled data from
other software systems and a hyperspherical loss when learning the
discriminative properties of the failures. The semantic baseline we considered
assumes that methods like Logsy perform ideally, thereby, we do not directly
compare with it. These methods are unable to detect failures that not
explicitly are logged. By modeling sequences, CLog detects contextual failures
when abrupt changes in the normality contexts of the co-occurring events
occur.

Supervised methods assume the availability of labels for logs from the target
system. LogRobust [8] uses a sequential representation of the input to train a
deep learning neural network – LSTM augmented with an attention mechanism to
learn failure sequences. Other methods, such as SVM, decision trees, logistic
regression and nearest neighbours, are also being considered [25]. Due to the
evolution of logs and their large volumes, the expensiveness of labeling is
referenced critique to supervised methods, questioning their usability [2].

Failure Type Identification. In one of the earliest works, Oliner et al. [26]
use keyword search of common words (e.g., ”interface failure”, ”error”, and
similar) to identify different types of failures within single logs of four
different supercomputer systems. Similarly, Meng et al. [27] use single log
lines represented as bag-of-words, alongside the Random Forest method to
classify different types of system logs. LogClass [4] introduces TF-ILF as a
novel representation method for individual logs and applies commonly used
multiclass classification methods to categorize the type of failure. A common
drawback of these approaches is the assumption of full failure coverage in
single logs. However, due to the problem of insufficient logging coverage
within the source code [3], this may not always be the case. Different from
others, we pair count vectors from the subprocesses of a given sequence with a
multiclass classifier to use the past information about similar failure types.

##  VI Conclusion

This paper addresses the problem of the automation of log-based failure
identification, which is a crucial maintenance task to enhance the reliability
in cloud systems. It introduces a novel method CLog, which decouples the
problem of failure identification into two subproblems 1) failure detection
and 2) failure type identification. We observe that by representing the input
log data as sequences of subprocesses instead of sequences of individual
events, the entropy in the input, caused by the unstable logs, is reduced.
CLog uses this observation and introduces a novel subprocess extraction
method, which jointly trains context-aware deep learning and clustering
methods to extract subprocesses. Our experiments demonstrate that the
extracted sequences of subprocesses are beneficial for improving the
performance of the two subproblems of 1) failure detection (by 9-24% over the
baselines) and 2) failure type identification (by 7% over the baseline).
Further, we show that CLog has robust performance, under high ratios of
injected unstable sequences, experiencing just a 6% performance drop. The key
observation presented herein opens new possibilities for how to most
efficiently extract meaningful subprocess with minimal information about the
sequences (e.g., discarding the sequence identifiers), which we aim to explore
next. These achievements can ultimately bridge the gap between automatic log-
based failure detection and root-cause analysis, further enhancing the
reliability of cloud systems.

## References

  * [1] P. Garraghan, R. Yang, Z. Wen, A. Romanovsky, J. Xu, R. Buyya, and R. Ranjan, “Emergent failures: Rethinking cloud reliability at scale,” _IEEE Cloud Computing_ , vol. 5, pp. 12–21, 2018.
  * [2] S. He, J. Zhu, P. He, and M. R. Lyu, “Experience report: System log analysis for anomaly detection,” in _Proc. of the 27th IEEE International Symposium on Software Reliability Engineering_ , 2016, pp. 207–218.
  * [3] S. He, P. He, Z. Chen, T. Yang, Y. Su, and M. R. Lyu, “A survey on automated log analysis for reliability engineering,” _ACM Comput. Surv._ , vol. 54, 2021.
  * [4] W. Meng, Y. Liu, S. Zhang, F. Zaiter, Y. Zhang, Y. Huang, Z. Yu, Y. Zhang, L. Song, M. Zhang, and D. Pei, “Logclass: Anomalous log identification and classification with partial labels,” _IEEE Trans. Netw._ , vol. 18, pp. 1870–1884, 2021.
  * [5] Q. Lin, H. Zhang, J.-G. Lou, Y. Zhang, and X. Chen, “Log clustering based problem identification for online service systems,” in _Proc. of the 38th International Conference on Software Engineering Companion_ , 2016, p. 102–111.
  * [6] X. Zhang _et al._ , “Robust log-based anomaly detection on unstable log data,” in _Proc. of the 27th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE)_ , 2019, p. 807–817.
  * [7] S. He, Q. Lin, J.-G. Lou, H. Zhang, M. R. Lyu, and D. Zhang, “Identifying impactful service system problems via log analysis,” in _Proc. of the 26th Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , 2018, p. 60–70.
  * [8] Z. Chen, J. Liu, W. Gu, Y. Su, and M. R. Lyu, “Experience report: Deep learning-based system log analysis for anomaly detection,” _CoRR_ , vol. 2107.05908, 2021.
  * [9] Y. Zhu, W. Meng, Y. Liu, S. Zhang, T. Han, S. Tao, and D. Pei, “Unilog: Deploy one model and specialize it for all log analysis tasks,” 2021.
  * [10] S. Nedelkoski, J. Bogatinovski, A. Acker, J. Cardoso, and O. Kao, “Self-attentive classification-based anomaly detection in unstructured logs,” _CoRR_ , vol. abs/2008.09340, 2020.
  * [11] M. Du, F. Li, G. Zheng, and V. Srikumar, “Deeplog: Anomaly detection and diagnosis from system logs through deep learning,” in _Proc. of the ACM SIGSAC Conference on Computer and Communications Security (CCS)_ , 2017, p. 1285–1298.
  * [12] D. Cotroneo, L. De Simone, P. Liguori, R. Natella, and N. Bidokhti, “How bad can a bug get? an empirical analysis of software failures in the openstack cloud computing platform,” in _Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_. New York, NY, USA: Association for Computing Machinery, 2019, pp. 200–211.
  * [13] J. Zhu, S. He, J. Liu, P. He, Q. Xie, Z. Zheng, and M. R. Lyu, “Tools and benchmarks for automated log parsing,” in _Proc. of the 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP)_ , 2019, p. 121–130.
  * [14] P. He, J. Zhu, Z. Zheng, and M. R. Lyu, “Drain: An online log parsing approach with fixed depth tree,” in _2017 IEEE International Conference on Web Services_. NY, USA: Curran Associates, 2017, pp. 33–40.
  * [15] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in _Proc. of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics_ , 2019, pp. 4171–4186.
  * [16] B. Yang, X. Fu, N. D. Sidiropoulos, and M. Hong, “Towards k-means-friendly spaces: Simultaneous deep learning and clustering,” in _Proc. of the 34th International Conference on Machine Learning_ , 2017.
  * [17] K. Yamanishi and Y. Maruyama, “Dynamic syslog mining for network failure monitoring,” in _Proc. of the 11nd SIGKDD International Conference on Knowledge Discovery and Data Mining_ , 2005, p. 499–508.
  * [18] L. Breiman, “Random forests,” _Mach. Learn._ , vol. 45, pp. 5–32, 2001.
  * [19] J. R. Quinlan, “Induction of decision trees,” _Mach. Learn._ , vol. 1, pp. 81–106, 1986.
  * [20] T. Hastie, R. Tibshirani, and J. Friedman, _The Elements of Statistical Learning_. New York, NY, USA: Springer New York Inc., 2001.
  * [21] Y. Freund and R. E. Schapire, “A short introduction to boosting,” in _Proc. of the 16 International Joint Conference on Artificial Intelligence_ , 1999, pp. 1401–1406.
  * [22] W. Xu, L. Huang, A. Fox, D. Patterson, and M. I. Jordan, “Detecting large-scale system problems by mining console logs,” in _Proc. of the 22nd Symposium on Operating Systems Principles_ , 2009, p. 117–132.
  * [23] H. Li, W. Shang, and A. E. Hassan, “Which log level should developers choose for a new logging statement?” _Empir. Softw. Eng._ , vol. 22, p. 1684–1716, 2017.
  * [24] W. Meng _et al._ , “Loganomaly: Unsupervised detection of sequential and quantitative anomalies in unstructured logs,” in _Proc. of the International Joint Conferences on Artificial Intelligence_ , 2019, pp. 4739–4745.
  * [25] J. Breier and J. Branišová, “Anomaly detection from log files using data mining techniques,” in _Information Science and Applications_. Berlin, Heidelberg: Springer Berlin Heidelberg, 2015, pp. 449–457.
  * [26] A. Oliner and J. Stearley, “What supercomputers say: A study of five system logs,” in _Proc. of the 37th Annual IEEE/IFIP International Conference on Dependable Systems and Networks_ , 2007, pp. 575–584.
  * [27] W. Meng, Y. Liu, S. Zhang, D. Pei, H. Dong, L. Song, and X. Luo, “Device-agnostic log anomaly classification with partial labels,” in _Proc of 26th International Symposium on Quality of Service_ , 2018, pp. 1–6.