# UniLog: Deploy One Model and Specialize it for All Log Analysis Tasks

Yichen Zhu3, Weibin Meng4, Ying Liu2, Shenglin Zhang5 Tao Han4, Shimin Tao4,
Dan Pei2<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>3University of Toronto  4Huawei  2Tsinghua University  5Nankai University

###### Abstract

Log analysis is vitally important for network reliability, and many log-based
tasks are derived to analysis logs, such as anomaly detection, failure
prediction, log compression, and log summarization. It is desired to have a
unified log analysis framework to simultaneously run all these log analysis
tasks on one model to achieve deployment convenience, superior task
performance, and low maintenance cost. However, due to severe challenges about
log data heterogeneity across devices, pioneer works design specialized
algorithms for each task. In this work, we formulate the log analysis as a
multi-task learning approach and propose to train a single model that can
perform various log analysis tasks. We name this unified log analysis approach
as UniLog. To effectively build an UniLog model, we propose a log data
pretrained transformer to utilize the enormous unlabeled log data, and a
corresponding multi-log-tasking finetune strategy for various log analysis
tasks. Extensive experiments across seven datasets on four log analysis tasks
demonstrate that UniLog achieves remarkable performance.

###### Index Terms:

Log Analysis, Pretrained Model, Transformer

##  I Introduction

Large-scale network services have been increasingly integrated into our daily
lives. Most of them are expected to be available on a 24 × 7 basis, and any
accidental downtime can lead to irreparable performance and significant
revenue loss. For instance, a downtime in Amazon in 2017 led to a loss of
around 150 million US dollars [1] [2]. Therefore, network service reliability
becomes increasingly important. These network services continuously generate
large amounts of logs (see Fig. 1 for examples), which describe a vast range
of events and record run-time information. The huge volume of logs makes it
impractical to manually inspect logs for key diagnostic information, even with
“search” or “grep” toolkit. Thus there are fast-growing demands to design
automatic log analysis frameworks.

Figure 1:  Examples of log analysis tasks. Log parsing is a pre-processing
step for automated log analysis.

Many automatic log analysis approaches have been proposed with the success of
machine learning and deep learning, as shown in Fig. 2. Existing log analysis
approaches can be classified into four tasks [1] (see Fig. 1), i.e., anomaly
detection aiming to proactively detecting anomalous service behaviors [3],
failure prediction which predicts service failure according to omen patterns
[4], log compression trying to save the storage space of massive logs [5], and
log summarization summarizing a series of logs [6].

Despite the tremendous success of the existing log analysis methods, these
algorithms only specialize in one type of log analysis task. Nevertheless,
real-life deployment constantly desired to do multiple log analysis tasks
simultaneously. Imagine a scenario: the maintenance system localizes anomalous
logs, and it warns the operators about a potential system failure that might
happen very soon. Meanwhile, the system summarizes the unreadable logs into
few words to let the operators examine the summarized logs and compress these
vast amounts of real-time logs into lossless log data for future usage. More
importantly, all these functionalities are performed via a single model, which
is easy to debug and upgrade.

Such a model is appealing yet hard to accomplish due to the following
challenges. First of all, log data are heterogeneous across domains (i.e.,
datasets, devices, time.) because different datasets and devices are always
filled with different rules to generate logs. Moreover, the style of generated
logs from the network system can change from time to time (i.e., due to system
upgrade), makes even more challenging to design a good model for log analysis
tasks across domains. Secondly, there are many types of log analysis tasks;
these tasks are not complemented from the perspective of algorithm design. For
example, anomaly detection needs to reconstruct the origin piece of the log.
In contrast, failure prediction needs to predict a specific time where
failures might happen. Therefore, current methods design task-specific models
for each log analysis task. Unifying these tasks into a single model is non-
trivial.

In this paper, we present a novel approach, named as UniLog, which unified
various log analysis tasks under a single framework, enables us to deploy a
single model, and specialize it for all log analysis tasks. The UniLog has the
following merits. First of all, it can generalize to log data from different
or even unseen domains. This is achieved by leveraging the Transformer [7], a
deep neural architecture that has been rapidly developed in recent years on
natural language processing (NLP) [8], computer vision [9], and graph data
[10]. We developed a large-scale unlabeled log data pretrain framework to mix
up the log data from different domains and use these data to pretrain the
Transformer neural networks. Such well designed pretrained Transformer can
perform log analysis across different devices and different time periods
effectively and consistently. Secondly, the UniLog is a framework to deploy
one model and specialize it for all log analysis tasks. It unifies different
log analysis tasks into a single framework. The UniLog would relieve the
operators from debugging and upgrading the model to the log analysis tasks
themselves. Last but not least, switching the UniLog model from one task to
another does not involve prior knowledge from the professionals, enabling
smooth and automatic workflow for maintaining the network log analysis system.

Extensive experiments are conducted to verify our proposed UniLog model. We
evaluate our method on four log analysis tasks, including log anomaly
detection, log failure prediction, log summarization, and log data
compression. Our model performs competitively or outperforms state-of-the-art
methods on seven public log datasets across these four log analysis datasets.
For instance, we outperform PreFix [4] by an average of 5.46% F1 score on
failure prediction tasks on Switch log dataset [4]. On log anomaly detection,
we achieve better F1 scores on both HDFS and BGL datasets than state-of-the-
arts. Furthermore, our experiments have shown that the UniLog can generalize
on unseen log data, which overcome log data domain adaption and concept drift
problem. We believe that UniLog offers an interesting and practical new
perspective on designing the model for log analysis tasks.

Figure 2: The number of log analysis papers published since 2003. We observed
that (1) automated log analysis has been continuously and actively
investigated in the past two decades, and (2) the number of publications
witnesses a sharp rise since 2016.

Figure 3: An example of existing log analysis frameworks. The top left is a
log compression framework. The top right is a log summarization framework. The
bottom left is a log failure predication framework. The bottom right are some
log anomaly detection algorithms. These methods are designed specifically for
a log analysis task. Our approach unified these log analysis tasks in to a
single model.

##  II Related Work

###  II-A Transformer

The Transformer is a type of deep neural network, which have been rapidly
developed since Vaswani et al. [7] and gradually dominate multiple field and
tasks, including natural language process (NLP) [8], computer vision [9, 11,
12] and graph [10]. BERT is the first pretrained model to pretrain the Bi-
Directional Transformer on large-scale unlabeled WikiText2 [8] and achieved
ground-breaking performance on multiple NLP tasks. Liu et al. [13] present to
add task-specific classification networks or different losses for each task.
The T5 [14] is another multi-tasking transformer. They convert the NLP tasks
to a text-to-text format, which enables to pretrain of the model. Their
paradigm presents a new pretrained dataset called C4 and achieves state-of-
the-art results in multiple NLP benchmarks. The Transformer has soon been
expanded to a huge model that contains billions of parameters [15] and
pretrained on enormous unlabeled data [16]. Some prior works adopt Transformer
or BERT for log anomaly detection. For example, the LogBERT [17] use
Transformer as a log feature extractor. Unlike previous works that only adopt
the Transformer as a feature extractor and perform anomaly detection on log
data, whereas, we train a single versatile Transformer model that is
pretrained on log data and can do multiple log analysis tasks.

###  II-B Log Analysis

This section briefly reviews prior works on four log analysis tasks, i.e.,
failure prediction, log anomaly detection, log compression, and log
summarization.

Log Failure prediction. Network reliability is vital for IT service
corporations, and any system failure could cause a significant loss. If the
occurrence of system failure can be predicted in advance, then the engineers
can take some proactive actions to prevent system collapse and avoid economic
loss. The log failure prediction is a supervised binary classification task.
Given a log sequence, we need to build a classifier to predict whether the
system will corrupt. Existing approaches utilize machine learning algorithms
such as XGBoost [4] to predict system failure.

Log Anomaly Detection. Detecting anomalies in logs is crucial for timely
identifying malfunctions of system. Log anomaly detection has been the most
active research field for log analysis. The log anomaly detection task is an
unsupervised binary task, where each log need to be labeled as normal or
abnormal. A conventional way to detect anomalous logs is to use reconstruction
method, where log are first encoder into the feature embedding space through a
mapping function, and then gradually recover the logs back from the embedding
space. By measure the distance with a designed distance metric (i.e., l2
distance) between test data point and normal log feature embedding, we are
able to detect the logs that cause system failure. If the calculated distance
is larger than a predefined threshold, we will consider it as a anomaly
sample. Existing works either use hand-crafted method, recurrent neural
network or transformer [18, 3, 17] to extract log feature embedding. Note that
failure prediction is a fundamentally different tasks than log anomaly
detection. The anomaly detection aim to locate the anomalous sample in the
existing data, whereas failure prediction need to predict the potential
anomalous time period that can have anomalous samples in the future.

Log Compression. Logs often need to be stored for a long time in practice
(i.e., a year) to analyze anomalous problems or root causes. However, storing
logs consumes a large amount of storage space and computing resources, which
incurs high operational costs. Log compression aims to compress massive log
data lossless with the minimum amount of storage. The query time also matters
since unzip all logs for a long period, and locating the target log is time-
consuming. The compression rate is usually to measure how well an algorithm
can compress a certain amount of data to minimize storage space. Log data can
be compressed by popular data compression techniques, such as bzip [19], 7zip
[20], or zip [21]. Logparse [22] is explicitly designed to do log data
compression, which has achieved a notable compression rate. This paper
integrates the log compression functionality into our framework, improving the
compression rate for log data.

Log Summarization. Logs record detailed run-time information of services.
After a failure is detected, diagnosed, or predicted, operators still have to
inspect the raw logs to gain a summarized view before taking action. However,
manual or rule-based log summarization has become inefficient and ineffective.
Log summarization aims to obtain the summarized information of necessary logs
for a given log stream. The output is a human-readable sentence. The log
summary task is similar to log compression. Nevertheless, the output of the
log compression task is not necessary to be readable, as long as its
information is lossless. In our case, it is necessary to keep the semantics
presented in the logs for the output summary to be readable. Meng et al. [6]
present a log summarization dataset and designed a log summary framework by
using the rule-based algorithm to produce a triplet for each log data. We
extend their work to the deep learning approach, which is more general and
practical for cross-service log data.

##  III Design of UniLog

Figure 4: The overview of UniLog. The (a) is the log preprocessing step to
prepare the pretrained data. The (b) represents the pretrain method we used in
UniLog. The (c) and (d) shows the modification we made in the Transformer
neural network architecture.

In this section, we first introduce two critical components in our UniLog
framework: log data pretrained Transformer and multi log analysis tasks
finetune strategy. We then discuss some modifications we made in the
Transformer neural network architectures.

###  III-A Log Data Pretrained Transformer

Pretrained model is prevalent and it dominates many fields in recent years,
for example, BERT [8] in NLP and ImageNet-pretrained backbone in computer
vision [23]. When training a neural network that can handle various log
analysis tasks, there are two significant challenges. First, log analysis
tasks vary, and the desired feature representation of log data is also
different for each sub task. Secondly, the log data are heterogeneous across
different network devices and different periods. The pretrained Transformer
can easily solve these two challenges via 1) the pretrained Transformer can
extract common log data feature representation from extensive unlabeled data,
and these feature representations can be a good initialization point to
further finetune to a task-specific feature representation by learning task-
specific training objective. 2) the Transformer architecture has been proven
that can be effectively scaled up and beneficial when the scale of data
increases [9], even when the data are noisy. For the log data, though the
meaningful semantic in the log data are scarce, our empirical study
demonstrates that it is effective to train the Transformer neural network on
large-scale log data.

The pretrain steps are simple. Similar to the natural language, the order
feature in the log sequence is important. For example, a system log cannot
print “change state to down” before the message of “change state to up”,
unless there are anomalous events. Because the pretrain stage do not involved
in labels, we choose a self-supervised training objective as the first step to
train a unsupervised transformer on log data, which enable the model to
understand the importance of log order. A common choice of the training
objective is to use BERT-style objective, which randomly sample 15% of tokens
in the input log sequence and then drop them. These dropped token are replaced
by a sentinel token and assigned a special token ID to it. The token ID is a
special token added to the vocabulary and does not correspond to any
wordpiece. The unsupervised training objective thus can be formulated as
predicting the dropped out tokens. The output should be the original log
sequence.

The natural language is normally considered as structural text: the contexts
between consecutive sentences are meaningful. Whereas the log data are
unstructured, typically there is no correlation between consecutive log
sequences. What matters to the log data is the consecutive words and sentences
from long range. Therefore, we proposed a novel pretrain objective based on
the particular property of log data. Specifically, we mask the tokens in the
same as we did in the BERT-style pretext task, except that we only mask the
consecutive words. We choose up to three consecutive tokens, where the token
with largest random probability is chosen. We then mask subsequent two or
three tokens. If the chosen token is the last token in the sequence, then we
mask the previous two tokens. The output is also the original log sequence. An
example of the pretrain objectives can be found in Table II. Secondly, we
increase the number of input token to increase the receptive field of log
data. As such, the Transformer is able to train with a large chunk of log
sequence, and potential learn the long range dependency between two log
sequences.

Log Data Preprocessing. The log preprocessing step is briefly shown in Figure
4(a). We define the word in a log is a token. Tokenization is a crucial step
to build a successful log analysis model. However, the structure of the log
sequence is quite different from the sentence in natural language. The top
part of Figure 1 shows few examples of log sequence. Basically, the log
sequence contains many meaningless separators and words. Therefore we first
use a set of special delimiters (.,:/;_=-”). These delimiters can primarily
separate the log. However, this is insufficient since many words in the logs
are not natural language texts, such as “LocalFaultAlarm_clear”. After using
the delimiter, we still get “LocalFaultAlarm”. We then use wordninja, a text
tokenizer based on English Wikipedia unigram frequencies, to separate
meaningful text in a log sequence. In the end, we can get a set of tokens
$\\{tok_{1},tok_{2},\dots,tok_{n}\\}$, where each token represents a word. For
instance, the word “LocalFaultAlarm_clear” is separated into “Local”, “Fault”,
“alarm”, “clear” by the first two steps. Additionally, we need to change the
capital word to lower case and set a word with a different status, i.e.,
change and changed, as the same token.

TABLE I: Description for the datasets. Datasets | Description | # of logs | # of anomalies/failures | A.D. | Comp. | Summ. | Fail.
---|---|---|---|---|---|---|---
HDFS | Hadoop distributed system log | 11,175,629 | 16,838 anomalies | ✓ | ✓ | ✓ | ✗
BGL | Blue Gene/L supercomputer log | 4,747,963 | 348,460 anomalies | ✓ | ✓ | ✓ | ✗
HPC | High performance cluster log | 433,489 | - | ✗ | ✓ | ✓ | ✗
Proxifier | Proxifier software log | 21,329 | - | ✗ | ✓ | ✓ | ✗
Hadoop | Hadoop MapReduce job | 394,308 | - | ✗ | ✓ | ✗ | ✗
ZooKeeper | ZooKeeper service | 74,380 | - | ✗ | ✓ | ✗ | ✗
Switch | Switch hardware failures | 29,174,680 | 2,204 failures | ✗ | ✗ | ✗ | ✓

Input and Output. In order to train a single model on the diverse set of log
analysis tasks, we reform the input and output format. We cast all of the
tasks we consider into a log-to-log format. Specifically, the model is fed by
log sequence and then returns some output logs. This framework provides a
consistent training objective to pretrain the UniLog on unlabeled log data and
then finetune to fit a specific task. In this way, the UniLog will focus on
extracting common feature representations in the log data. Then the model can
switch to learn task-specific features by finetuning. To specify in which task
the model should be finetuned or performed, we add a task-specific prefix to
the input log data sequence before feeding it to our model. For example, we
add “<anomaly>” before the log sequence if we desire to perform the log
anomaly task. The model will then give the output to determine whether a log
sequence is normal or anomalous. This holds the same for all tasks, thus
allowing the model to handle multiple tasks. The list of a prefix we used is
<anomaly>, <summarization>, <compression>, and <failure>, corresponding to log
anomaly detection, log summarization, log compression, and failure prediction.

###  III-B Log Analysis Task-Specific Finetune

Task-Specific Finetune Strategy. The pretrained log data-based Transformer
provides us the common feature representations that universally exist on log
data. In order to run specific log analysis tasks such as anomaly detection,
we need to finetune our model based on the task training objective so that the
model can learn task-specific feature representations. As the log analysis
contains multiple sub-tasks, the pretrained model needs to conduct multi-log-
tasking. However, perform multi-tasks on a single network can lead to
suboptimal results, especially on low-resource tasks [24] due to the co-
adaptation effect. UniLog aims to process a sequence of logs and then return
the output for every log analysis task. The previous method [25] gradually
unfreeze the parameters in the model to achieve this goal; however, it is
sensitive to the size of the training dataset and the total number of training
iterations [14]. We relax the multi-task finetune framework and introduce a
straightforward checkpoint-based method. Specifically, we first record the
checkpoint after the model finishes pretraining. Then we load this checkpoint
every time a new task needs to be finetuned. During inference, we load the
checkpoint for each finetuned task model and return the outputs. Therefore the
engineers can simply deploy UniLog, which is a single, unified model, and
perform multiple log analysis tasks at the same time. Since there is only one
model for deployment, the maintenance cost can be very cheap, and model
upgrading can be done under a single framework. The UniLog improves the system
reliability, enables the engineers to free their hands on debugging the
deployed model, and focuses on system maintenance itself.

Additionally, we introduce extra task-aware heads to finetune on the sub-
tasks. Each task-aware head is attached with two consecutive multilayer
perceptrons after the log feature embedding layers. A task-aware head is used
to disentangle the feature representations that are learned between different
log analysis tasks. For instance, there are four log analysis tasks in our
framework; then, we will have four task-aware heads, where each head
corresponding to a sub-task and return the task-specific results. We can
jointly optimize the task-aware head with our pretrained model during
finetuning to achieve better model performance.

For the log failure prediction, we use the softmax operation to return the
probability of a network system failure shortly. We use the cross-entropy loss
as our training objective. The log anomaly detection use l2 norm loss as the
pretrain objectives. This heuristic way is very effective since the pretrain
objective loss can represent how hard it is to reconstruct an input log
sequence. Similar to the other reconstruction-based anomaly detection method,
we set a predefined threshold as 1e-3 and assign the log sequence with
validation loss higher than this threshold as the anomalous sample. We use the
cross-entropy loss to train the log summarization task to output the
summarized log given a sequence of the log. The log compression is more
complex than the other tasks, thus we discuss it in a separate section.

Log Compression. Besides the transformer network that extracts the log feature
embedding, the log compression requires an entropy coding scheme. We choose
the arithmetic coding scheme [26] for simplicity. Specifically, given a log
sequence $s={s_{0},s_{1},\dots,s_{w}}$of length $w$ number of tokens, we first
use the predictor $\mathcal{T}$ to encode this stream of tokens and return the
probability $P(s)={P(s_{0}),P(s_{1}),\dots,P(s_{w})}$ for each token, where
$P(s)=P(s_{w}|s_{w-1},\dots,s_{0})$ is a token conditional probability depends
on the past tokens. In our context, the transformer we describe above can be
considered as a predictor. Then an entropy coding scheme is used to store the
token into a lossless code of bit-length $log_{2}P(s_{t}|s_{t-1},\dots,s_{0})$
based on the token probability given by the predictor. We iteratively compress
each token until all tokens are compressed. The decompression stage is similar
to the compression stage, except that the entropy coding scheme generates the
preceding tokens. Thus the transformer network does not involve in
decompression, making the decompression efficient.

###  III-C Details of the Transformer Architecture

The Transformer Model. After a log sequence is preprocessed into a sequence of
log tokens, it is mapped to a sequence of embeddings and then passed into the
encoder. The encoder is a stack of transformer blocks, and each block
comprises a multi-head self-attention layer [7] followed by a shallow feed-
forward network (FFN). We apply layer normalization to the input of each self-
attention layer and FFN. We also apply Dropout [27] within the FFN, on the
residual path, on the attention weights, and at the input and output of the
entire stack. The decoder structure is similar to the encoder, except there is
a standard attention layer after each self-attention layer. The self-attention
layers in the decoder use a form of causal self-attention. Unlike self-
attention in the encoder, which allows the model to attend to past outputs and
future outputs, causal self-attention only allows the model to attend to past
outputs. The decoder’s outputs are forward to a fully connected layer with
softmax, whose weights are shared with the input embedding matrix. We apply
relative position embedding and share this position embedding across all
layers in our model. More detailed explanations of the Transformer neural
network architectures can be found in [8]. In the next section, we introduce
two modification on Transformer architecture to improve the model performance
for log analysis tasks.

TABLE II: Examples of different pretrained method. The BERT-style objective includes a corruption term, where some token are replace by a random token, in our example, the word link in the original text is replace by word status. The original text denote that the model predicts the original input based on the reconstruction objective. Method | Input | Target
---|---|---
Prefix Language Modeling | The interface status changes | Physical link is up
BERT-style [8] | The <M>status <M>Physical status is up | original text
UniLog | The <M><M><M>Physical status is up | original text

Figure 5: The left is Post-LN and the right is Pre-LN structure which we used
in our paper.

Pre-LN Structure. Layer normalization (LN) [28] is an essential building block
of the transformer, and Post-LN architecture is a common choice. However,
since we utilize long-range sequences to learn the feature representation of
log data, the parameter gradients, which we used to update the model
parameters, can be extremely high and cause gradient overflow. Previous study
[29] on Pre-LN structure have shown to possess well-behaved properties on
gradients at initialization, thus alleviate the large gradient during
training. The comparison of Post-LN and Pre-LN structure are shown in Figure
5. In short, Post-LN places the LN on the main path so that the gradient of
the parameters near the output layer are large [29]. Whereas the Pre-LN place
on the residual path. An empirical study is performed in in the Fig. 5.

LogAct. Feed-forward network (FFN) is another fundamental component of the
Transformer. It is comprised of few stacks of multilayer perceptron followed
by non-linear activation. The non-linear activation controls how much
expressiveness [30] can a neural network represent. We introduce a novel non-
linear activation function, named as LogAct, to enhance the expressive power
of the Transformer, which is more suitable to deal with long-range,
unstructured log data for log analysis tasks.

Inspired by [31] and [32], our LogAct combines the GLU activation function
[31] and Swish activation function [32]. Specifically, the GLU unit can be
formulated as:

$GLU(x,W,V,b,c)=\sigma(xW+b)\otimes(xV+c)$ (1)

where $x$ is the input, $W$ and $V$ are weight matrix, $b$ and $c$ are bias.
The Swish activation function can be formulated as:

$Swish(x)=\frac{x}{1+e^{-\beta x}}$ (2)

where the $\beta$ is a learnable parameters. In our paper, we combine the
Swish function and GLU function as:

$LogAct(x,W,V,b,c)=\frac{(xW+b)}{1+e^{-\beta(xW+b)}}\otimes(xV+c)$ (3)

In our experiment, we replace very ReLU layer in the FFN with LogAct. We
conduct an ablation study on the effectiveness of LogAct in Table VII.

##  IV Experiments

TABLE III: Comparison of model performance for failure prediction task. Switch | Method | Precision | Recall | F1
---|---|---|---|---
M1 | HSMM | 32.27% | 95.3% | 48.21%
SKSVM | 8.25% | 76.09% | 14.89%
PreFix | 87.35% | 74.36% | 80.33%
UniLog | 84.25% | 79.60% | 81.86%
M2 | HSMM | 0.28% | 60.58% | 0.56%
SKSVM | 4.47% | 8.72% | 5.91%
PreFix | 59.79% | 58.59% | 59.19%
UniLog | 65.73% | 63.24% | 64.46%
M3 | HSMM | 26.32% | 11.11% | 15.63%
SKSVM | 0.79% | 91.91% | 1.58%
PreFix | 84.00% | 52.50% | 64.61%
UniLog | 87.90% | 64.58% | 74.46%

In this section, we evaluate our methods on four log analysis tasks: log
anomaly detection, log summarization, log compression, and log failure
prediction. We briefly summarized the datasets in Table I. Note that for some
tasks, the dataset may not have labels for evaluation; thus, we omit those
experiments (denotes as cross-mark in the table). The details baseline and
results will be introduced in the following sections for each task,
respectively. Through our experiment section, we sometime use abbreviation for
simplicity, i.e., ”A.D.” to denote anomaly detection, ”Comp.” to represent log
compression, and ”Summ.” for log summarization.

Figure 6: Anomaly detection results on HDFS dataset

Figure 7: Anomaly detection results on BGL dataset.

###  IV-A Setup

Implementation Details We follow the model design in Section 4. Specifically,
we use 3 blocks for both in encoder and decoder. Each block consists of a
multi-head self-attention layer, optional encoder-decoder attention layer, and
a feed-forward network. We use 4 heads for self-attention, 32 inner dimensions
for metrics in the attention layer, a hidden size of 128 for transformer
blocks, and an output dimension of 512 for FFN. We apply Dropout with 0.3
probability in the model for regularization. Unlike This result modern network
designed for NLP tasks, our model’s size is four numbers of magnitude smaller
since the scale of the public dataset for log analysis is much smaller than
those for NLP tasks. This enables us to achieve good performance for log
analysis tasks without the overwhelming number of parameters.

All experiments are done on NVIDIA V100 GPUs 32 GB memory with Python 3.5 and
Pytorch 1.3. We use NVIDIA Apex framework for distributed training. We use a
batch size of 128, and trained with AdamW [33]. We use 180 as the maximum
length of log sequence and pretrained with $2^{1}6=65,536$ step. We set the
initial learning rate as 5e-4 without learning rate warm-up stage and use
exponential learning rate decays. We fine-tuned all tasks for $2^{11}=2048$
steps to prevent over-fitting.

Datasets. We use the BGL [34], HDFS [35], HPC [36], ZooKeeper [37], Proxifier
[38], Hadoop [39] and Switch [4] to conduct our experiments. We use the
dataset split strategy from the prior works for fair comparison. For anomaly
detection, failure prediction, summarization and compression we follows the
Loganomaly [3], PreFix [4], LogSummary [6], and LogParse [5], respectively.
For the pretrained dataset, We utilize all log datasets mentioned above to
pretrain UniLog, except test data which we used for evaluation.

Evaluation Metrics. For anomaly detection, summarization, and failure
prediction, A method’s capability is usually assessed by four metrics that
have intuitive interpretation, i.e., precision, recall, and F1 score, and thus
we use these metrics to evaluate each method. For log compression, we use
compression rate $C=\frac{\textit{size of compressed logs}}{\textit{size of
original logs}}$, which indicates the model’s ability to store the same amount
of logs with the number of storage space.

TABLE IV: Comparison of model performance for log compression task. Method | HPC | HDFS | Zoo | Hadoop | Avg.
---|---|---|---|---|---
bzip | 5.7% | 9.3% | 3.0% | 5.6% | 6.1%
7zIp | 7.0% | 8.5% | 3.1% | 5.0% | 5.9%
zip | 8.6% | 11.4% | 4.8% | 8.0% | 8.2%
LogParse | 23.4% | 14.1% | 10.9% | 42.4% | 22.7%
LogParse + bzip | 3.4% | 3.2% | 2.4% | 3.6% | 3.2%
LogParse + zip | 4.4% | 7.8% | 2.8% | 5.0% | 5.0%
LogParse + 7zip | 3.7% | 6.3% | 2.8% | 3.4% | 4.1%
UniLog | 2.7% | 2.9% | 2.8% | 3.3% | 2.9%

###  IV-B Results.

Log Anomaly Detection. Log anomaly detection is the most widely studied topic
in the log analysis domain. In this section, we conduct experiments on two log
anomaly detection benchmark: HDFS and BGL to demonstrate the effectiveness of
our proposed framework. We compare UniLog to the following methods with 1) PCA
[35], PCA-based anomaly detection method vectorize each log sequence as an
event count vector and employed to find patterns between the dimensions of
event count vectors. 2) Invariant Mining [40] which was the ﬁrst applied to
log- based anomaly detection in [40] based on the assumption that program
invariants are the linear relationships that always hold during system
running. 3) LogCluster [39] that determine the state of a new log sequence by
computing its distance to representative vectors in knowledge base and report
the log sequence as normal or abnormal based on representation vector
distance. 4) DeepLog [39] is the first deep learning approach on log anomaly
detection. 5) LogAnomaly [3], the state-of-the-art method on log anomaly
detection, combines sequential patterns and quantitative patterns.

In Fig. 6 and Fig. 7 shows the experimental results for log anomaly detection
on HDFS and BGL dataset. Our proposed UniLog achieves the best performance on
both datasets. Specifically, on the HDFS dataset, UniLog achieved precision of
0.97, recall of 0.99, and f1 score of 0.98. The f1 score is 0.3% higher than
LogAnomaly. On the BGL dataset, our method achieve precision of 0.98, recall
of 1.00 and f1 score of 0.99. It further improve the f1 score 0.3% over
LogAnomaly. This improvement is non-trivial, since reliability on the system
is vital and an f1 score improve over 0.1% might save millions of dollars.

TABLE V: Comparison of model performance for log summarization task. Logs | Method | Precision | Recall | F1
---|---|---|---|---
BGL | TextRank | 0.893 | 0.238 | 0.347
LogSummary | 0.815 | 0.703 | 0.788
UniLog | 0.895 | 0.824 | 0.858
HDFS | TextRank | 0.893 | 0.238 | 0.347
LogSummary | 0.759 | 0.432 | 0.538
UniLog | 0.876 | 0.650 | 0.746
HPC | TextRank | 0.904 | 0.265 | 0.365
LogSummary | 0.819 | 0.911 | 0.840
UniLog | 0.835 | 0.911 | 0.871
Proxifier | TextRank | 0.663 | 0.050 | 0.093
LogSummary | 0.879 | 0.857 | 0.864
UniLog | 0.925 | 0.877 | 0.900
TABLE VI: Comparison of compression rate for Log data compression task with Post-LN versus Pre-LN structure. Task | Dataset | Method | Precision | Recall | F1
---|---|---|---|---|---
A.D. | BGL | Post-LN | 0.95 | 0.94 | 0.94
Pre-LN | 0.98 | 1.00 | 0.99
Summ. | BGL | Post-LN | 0.850 | 0.832 | 0.841
Pre-LN | 0.895 | 0.824 | 0.858

Log Summarization. After demonstrating the superiority of our method on log
anomaly detection, we evaluate it for log summarization that requires refining
the semantic information in the log sequence. We demonstrate the experimental
results for log data summarization. We choose TextRank [41], a classic keyword
extraction algorithm, and LogSummary [6] the state-of-the-art method design
specifically to summarize log data. Table V reports the experimental results;
it shows that the UniLog can significantly improve our log summarization
performance: UniLog is improving the F1 score by 0.070 on BGL data, 0.031 on
HPC data, and 0.036 on Proxifier over LogSummary. On hard task HDFS, the F1
score of UniLog is 0.208 higher than LogSummary, demonstrate that UniLog is
more effective than the classic model and LogSummary in a real-world
situation.

Log Failure Prediction. We now show UniLog can improve the performance on
failure prediction tasks. As shown in Table III, we perform experiment on
Switch log datasets to compare UniLog against a) SKSVM [42], a log-based
failure prediction system used in computer b) HSMM [43], a log-based failure
prediction in ISP devices, and c) Prefix [4], the state-of-the-art log failure
prediction algorithm for switch failure prediction. We can observe that,
though both SKSVM and HSMM are failure prediction algorithms, their
performance is relatively low. This is due to that these two methods are
designed specifically for computer and ISP service. As a result, they fail to
generalize on Switch log dataset. On the contrary, our UniLog achieves better
performance on all three switches, despite the fact that UniLog is not being
designed for switch failure prediction tasks. Particularly, on the M2 and M3
switch, the UniLog strongly outperforms these approaches notably, obtains a
+5.27% and +9.85% F1 score improvement. This provides another indication that
UniLog is able to capture generalizable knowledge in the cross-service log
data.

Log Compression. We also conduct experiments on log compression. We compare
our method to three popular data compression techniques that are widely used
in the computer system, bzip [19], 7zip [20], zip [21]. We also compare to
LogParse [22], a log parsing method that can adaptively classify log based on
the word. The LogParse can effectively combine with existing data compression
techniques to compress log data. We therefore compare our method to LogParse,
which is followed by bzip, 7zip, and zip algorithm. The experimental results
are shown in Table IV. Our method consistently improves the compression ratio
over all baseline methods. Specifically, UniLog increase 2.1% on average
compression rate on four datasets compare to the previous state-of-the-art,
which results in an excellent storage reduction. Besides, as the number of
samples in the target dataset increase, UniLog achieve more improvement on the
performance. For instance, on the largest dataset HDFS, where the number of
logs is over two orders of magnitudes larger than HPC and Hadoop, the UniLog
improves the compression rate relatively up to 3.4%.

TABLE VII: Comparison of model performance for Log data compression task with different activation layer. Task | Dataset | Method | Precision | Recall | F1
---|---|---|---|---|---
A.D. | BGL | ReLU | 0.97 | 0.99 | 0.98
LogAct | 0.98 | 1.00 | 0.99
Summ. | BGL | ReLU | 0.866 | 0.815 | 0.840
LogAct | 0.895 | 0.824 | 0.858
TABLE VIII: Comparison of model performance for Log data compression task with different pretrain objectives. Task | Dataset | Method | Precision | Recall | F1
---|---|---|---|---|---
A.D. | BGL | Prefix Language Model | 0.65 | 0.68 | 0.66
BERT-style | 0.95 | 0.94 | 0.94
UniLog | 0.98 | 1.00 | 0.99
Summ. | HDFS | Prefix Language Model | 0.325 | 0.739 | 0.451
BERT-style | 0.770 | 0.521 | 0.621
UniLog | 0.876 | 0.650 | 0.746

###  IV-C Ablation Study

TABLE IX: Comparison of model performance for Log data compression task with different pretrain objectives. Task | Dataset | Method | Precision | Recall | F1
---|---|---|---|---|---
A.D. | BGL | Prefix Language Model | 0.65 | 0.68 | 0.66
BERT-style | 0.95 | 0.94 | 0.94
UniLog | 0.98 | 1.00 | 0.99
Summ. | HDFS | Prefix Language Model | 0.325 | 0.739 | 0.451
BERT-style | 0.770 | 0.521 | 0.621
UniLog | 0.876 | 0.650 | 0.746
TABLE X: Comparison of model performance on log anomaly detection, log compression, and log summarization on HDFS data. Pretrain Dataset | Task | Precision | Recall | F1 | Comp. Rate
---|---|---|---|---|---
HDFS | A.D. | 0.94 | 0.96 | 0.95 | -
Comp. | - | - | - | 3.5%
Summ. | 0.807 | 0.545 | 0.651 | -
\+ BGL | A.D. | 0.90 | 0.93 | 0.91 | -
Comp. | - | - | - | 5.6%
Summ. | 0.652 | 0.380 | 0.480 | -
\+ (Proxifier | A.D. | 0.94 | 0.94 | 0.94 | -
\+ ZooKeeper | Comp. | - | - | - | 5.6%
\+ Hadoop + HPC) | Summ. | 0.812 | 0.597 | 0.688 | -
\+ Switch | A.D. | 0.97 | 0.99 | 0.98 | -
Comp. | - | - | - | 2.9%
Summ. | 0.876 | 0.650 | 0.746 | -
TABLE XI: Comparison of model performance for transfer learning to Hadoop. Method | Precision | Recall | F1
---|---|---|---
LogTransfer | 0.980 | 0.970 | 0.977
UniLog | 1.000 | 0.986 | 0.993

Pretrain Objectives. We conduct experiments to validate the effectiveness of
denoising objectives in our method. We compare with 1) Prefix Language
Modeling, where log sequences are feed into the network and ask to return the
next log sequence, 2) the BERT-style objective, where consecutive tokens are
masked. In Table IX, we present the experimental results on BGL anomaly
detection task and HDFS summarization task. The prefix language modeling is
only obtained an F1 score of 0.66, which is not suitable for the log analysis
task. On both tasks, the UniLog with our proposed pretrain objective achieve
significantly better results than both prefix language modeling and BERT-style
objective. On log summarization, which is a semantic essential task, our
pretrain objective outperform BERT-style objective by 0.125 on F1 score, shows
that our method is more suitable to extract representation in the unstructured
log data.

Pre-LN. We evaluate the importance of Pre-LN architecture in Table VI. The
experiments are conducted on BGL dataset with log anomaly detection and log
summarization. We can observe that the experimental results on both tasks are
improved. On the log summarization tasks, the F1 score of the model using
LogAct is 0.018% higher than the model using ReLU. The improvement on
precision is significant, which is over 0.031%.

LogAct. We verfy the effectiveness of our proposed LogAct layer. We evaluate
the model performance on the BGL dataset on log anomaly detection and log
summarization tasks, and the result is shown in Table VII. The precision,
recall, and F1 score on Summarization of LogAct is lifed by, respectively. On
the other hand, the On log anomaly detection, the LogAct increases on all
evaluation metrics by 1%. Because the performance on BGL dataset is already
high, thus these improvement is non-trivial. The results indicate the complex
feature representation that unstructured text may have and, therefore, can
benefit from more non-linearity.

Does Transformer Really Benefit from More Unstructured Log Data? The log data
is considered to be data-intensive yet lacks useful information. Hence we
study that if massive log data can really achieve better performance than
small-scale log data. We evaluate our assumption by progressively adding more
datasets into our pretrain model and then finetune on the task model. We
linearly reduce the total training iterations based on the number of log data
in the total pretrain dataset. The rest of the training strategy is kept the
same. We evaluate on HDFS to validate our assumption. We provide the results
on three tasks, log anomaly detection, log compression, and log summarization.
The Table X shows the experimental results. We can observe that when the HDFS
is the only pretrain dataset, the result on sub-tasks is relatively higher
than using HDFS + BGL + Proxifier + ZooKeeper. This is intuitive since
directly pretrain on the target data can certainly improve the performance on
finetuning. This phenomenon is utilized in the self-training [44]. When the
number of pretrain data is not large enough, the model’s eventual performance
on target data may be bias towards other datasets. However, the performance is
improved and eventually surpasses the performance on sole pretrained on HDFS.
Especially when we add the Switch dataset, the F1 score on anomaly detection
increases from 0.95 to 0.98, the compression rate increases from 3.5% to 2.9%,
and the F1 score on summarization is achieved 0.651 compared to 0.746. This
supports our assumption that the UniLog can indeed improve by training on more
log data, though the information entropy provided by the log data is small.

Does UniLog generalize on unseen dataset? As we have previously stated that,
UniLog can learn common features from massive log data via pretrain. In the
real world, the algorithm running on the server can often feed with unseen
data, which results in poor performance due to the data domain shift. Though
our method does not target to solve domain adaptation problems, we evaluate
the ability of UniLog to generalize on unseen data. We pretrain our model on
our pretrain dataset without the Hadoop dataset. Then, we evaluate our method
on the Hadoop dataset by finding anomalous samples in the Hadoop. We compare
our method with LogTransfer [45], a log-based domain adaptation method. The
LogTransfer is trained on the HDFS dataset and then evaluate on the Hadoop
dataset. Table XI shows that, with the pretrain-finetune paradigm, the
transfer learning ability of UniLog is better than LogTransfer, outperforms
the state-of-the-art method by 0.16 on the F1 score.

##  V Conclusion

This work aim to build a unified framework for log analysis tasks. We propose
UniLog, which enable the engineers to deploy one model that are specialized
for all log analysis tasks. Our experiments have shown that UniLog perform
competitively with, or outperform, SOTA methods on various log datasets across
four log analysis tasks.

## References

  * [1] Shilin He, Pinjia He, Zhuangbin Chen, Tianyi Yang, Yuxin Su, and Michael R Lyu.  A survey on automated log analysis for reliability engineering.  arXiv preprint arXiv:2009.07237, 2020.
  * [2] UpGuard.  The cost of downtime at the world’s biggest online retailer.  https://www.upguard.com/blog/the-cost-of-downtime-at-the-worlds-biggest-online-retailer, 2019\.
  * [3] Weibin Meng, Ying Liu, Yichen Zhu, Shenglin Zhang, et al.  Loganomaly: Unsupervised detection of sequential and quantitative anomalies in unstructured logs.  In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19., volume 7, pages 4739–4745, 2019.
  * [4] Shenglin Zhang, Ying Liu, Weibin Meng, Zhiling Luo, Jiahao Bu, Sen Yang, Peixian Liang, Dan Pei, Jun Xu, Yuzhi Zhang, et al.  Prefix: Switch failure prediction in datacenter networks.  Proceedings of the ACM on Measurement and Analysis of Computing Systems, 2(1):1–29, 2018.
  * [5] Jinyang Liu, Jieming Zhu, Shilin He, Pinjia He, Zibin Zheng, and Michael R Lyu.  Logzip: extracting hidden structures via iterative clustering for log compression.  In ASE, pages 863–873. IEEE, 2019.
  * [6] Weibin Meng, Federico Zaiter, Yuheng Huang, Ying Liu, Shenglin Zhang, Yuzhe Zhang, Yichen Zhu, Tianke Zhang, En Wang, Zuomin Ren, et al.  Summarizing unstructured logs in online services.  arXiv preprint arXiv:2012.08938, 2020.
  * [7] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, et al.  Attention is all you need.  In Advances in neural information processing systems, pages 5998–6008, 2017.
  * [8] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [10] Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie-Yan Liu.  Do transformers really perform bad for graph representation?  arXiv preprint arXiv:2106.05234, 2021.
  * [11] Yichen Zhu and Yi Wang.  Student customized knowledge distillation: Bridging the gap between student and teacher.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 5057–5066, October 2021.
  * [12] Yichen Zhu, Yuqin Zhu, Jie Du, Yi Wang, Zhicai Ou, Feifei Feng, and Jian Tang.  Make a long image short: Adaptive token length for vision transformers, 2021.
  * [13] Yang Liu.  Fine-tune bert for extractive summarization.  arXiv preprint arXiv:1903.10318, 2019.
  * [14] Colin Raffel, Noam Shazeer, Adam Roberts, et al.  Exploring the limits of transfer learning with a unified text-to-text transformer.  arXiv preprint arXiv:1910.10683, 2019.
  * [15] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  OpenAI blog, 1(8):9, 2019.
  * [16] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  arXiv preprint arXiv:2005.14165, 2020.
  * [17] Haixuan Guo, Shuhan Yuan, and Xintao Wu.  Logbert: Log anomaly detection via bert.  arXiv preprint arXiv:2103.04475, 2021.
  * [18] Min Du, Feifei Li, Guineng Zheng, and Vivek Srikumar.  Deeplog: Anomaly detection and diagnosis from system logs through deep learning.  In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pages 1285–1298, 2017.
  * [19] Julian Seward.  bzip.  www.b-zip2.org, 1996.
  * [20] Igor Pavlov.  7zip.  www.7-zip.org, 1999.
  * [21] Phil Katz.  zip.  infozip.sourceforge.net, 1989.
  * [22] Weibin Meng, Ying Liu, Federico Zaiter, Shenglin Zhang, Yihao Chen, Yuzhe Zhang, Yichen Zhu, En Wang, Ruizhi Zhang, Shimin Tao, et al.  Logparse: Making log parsing adaptive through word classification.  In 2020 29th International Conference on Computer Communications and Networks (ICCCN), pages 1–9. IEEE, 2020.
  * [23] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [24] Matthew E Peters, Sebastian Ruder, and Noah A Smith.  To tune or not to tune? adapting pretrained representations to diverse tasks.  arXiv preprint arXiv:1903.05987, 2019.
  * [25] Jeremy Howard and Sebastian Ruder.  Universal language model fine-tuning for text classification.  arXiv preprint arXiv:1801.06146, 2018.
  * [26] Ian H Witten, Radford M Neal, and John G Cleary.  Arithmetic coding for data compression.  Communications of the ACM, 30(6):520–540, 1987.
  * [27] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  Dropout: a simple way to prevent neural networks from overfitting.  The journal of machine learning research, 15(1):1929–1958, 2014\.
  * [28] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.  Layer normalization.  arXiv preprint arXiv:1607.06450, 2016.
  * [29] Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu.  On layer normalization in the transformer architecture.  In International Conference on Machine Learning, pages 10524–10533. PMLR, 2020.
  * [30] Joe Mellor, Jack Turner, Amos Storkey, and Elliot J Crowley.  Neural architecture search without training.  In International Conference on Machine Learning, pages 7588–7598. PMLR, 2021.
  * [31] Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier.  Language modeling with gated convolutional networks.  In International conference on machine learning, pages 933–941. PMLR, 2017.
  * [32] Prajit Ramachandran, Barret Zoph, and Quoc V Le.  Searching for activation functions.  arXiv preprint arXiv:1710.05941, 2017.
  * [33] Ilya Loshchilov and Frank Hutter.  Fixing weight decay regularization in adam.  2018\.
  * [34] Adam Oliner and Jon Stearley.  What supercomputers say: A study of five system logs.  In 37th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN’07), pages 575–584. IEEE, 2007.
  * [35] Wei Xu, Ling Huang, Armando Fox, David Patterson, and Michael Jordan.  Largescale system problem detection by mining console logs.  Proceedings of SOSP’09, 2009.
  * [36] Shilin He, Jieming Zhu, et al.  Experience report: System log analysis for anomaly detection.  In 2016 IEEE 27th International Symposium on Software Reliability Engineering (ISSRE), pages 207–218. IEEE, 2016.
  * [37] Shilin He, Jieming Zhu, Pinjia He, and Michael R Lyu.  Loghub: a large collection of system log datasets towards automated log analytics.  arXiv preprint arXiv:2008.06448, 2020.
  * [38] Jieming Zhu, Shilin He, Jinyang Liu, Pinjia He, Qi Xie, Zibin Zheng, and Michael R Lyu.  Tools and benchmarks for automated log parsing.  In ICSE-SEIP, pages 121–130. IEEE, 2019.
  * [39] Qingwei Lin, Hongyu Zhang, Jian-Guang Lou, et al.  Log clustering based problem identification for online service systems.  In ICSE, pages 102–111. ACM, 2016.
  * [40] Jian-Guang Lou, Qiang Fu, Shengqi Yang, Ye Xu, and Jiang Li.  Mining invariants from console logs for system problem detection.  In USENIX Annual Technical Conference, pages 23–25, 2010.
  * [41] Rada Mihalcea and Paul Tarau.  Textrank: Bringing order into text.  In Proceedings of the 2004 conference on empirical methods in natural language processing, pages 404–411, 2004.
  * [42] Errin W Fulp, Glenn A Fink, and Jereme N Haack.  Predicting computer system failures using support vector machines.  WASL, 8:5–5, 2008.
  * [43] Felix Salfner and Miroslaw Malek.  Using hidden semi-markov models for effective online failure prediction.  In 2007 26th IEEE International Symposium on Reliable Distributed Systems (SRDS 2007), pages 161–174. IEEE, 2007.
  * [44] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In International conference on machine learning, pages 1597–1607. PMLR, 2020.
  * [45] Rui Chen, Shenglin Zhang, Dongwen Li, Yuzhe Zhang, Fangrui Guo, Weibin Meng, Dan Pei, Yuzhi Zhang, Xu Chen, and Yuqing Liu.  Logtransfer: Cross-system log anomaly detection for software systems with transfer learning.  In 2020 IEEE 31st International Symposium on Software Reliability Engineering (ISSRE), pages 37–47. IEEE, 2020.