# Log Summarisation for Defect Evolution Analysis

Rares Dolga  0000-0002-1800-411X JPMorgan ChaseUK<EMAIL_ADDRESS>,
Ran Zmigrod  0009-0001-0169-1670 JPMorgan ChaseUK<EMAIL_ADDRESS>,
Rui Silva  0009-0003-6016-6432 JPMorgan ChaseUK<EMAIL_ADDRESS>,
Salwa Alamir  0009-0006-6650-7041 JPMorgan ChaseUK<EMAIL_ADDRESS>and  Sameena Shah  0009-0000-5960-5811 JPMorgan ChaseUK
<EMAIL_ADDRESS>
(2023; 2023-07-31; 2023-08-21)

###### Abstract.

Log analysis and monitoring are essential aspects in software maintenance and
identifying defects. In particular, the temporal nature and vast size of log
data leads to an interesting and important research question: How can logs be
summarised and monitored over time? While this has been a fundamental topic of
research in the software engineering community, work has typically focused on
heuristic-, syntax-, or static-based methods. In this work, we suggest an
online semantic-based clustering approach to error logs that dynamically
updates the log clusters to enable monitoring code error life-cycles. We also
introduce a novel metric to evaluate the performance of temporal log clusters.
We test our system and evaluation metric with an industrial dataset and find
that our solution outperforms similar systems. We hope that our work
encourages further temporal exploration in defect datasets.

Defect Detection, Log Analysis, Clustering, NLP

††copyright: acmlicensed††doi: 10.1145/3617572.3617881††journalyear:
2023††submissionid: fsews23sddmain-p59-p††isbn:
979-8-4007-0377-5/23/12††conference: Proceedings of the 1st International
Workshop on Software Defect Datasets; December 8, 2023; San Francisco, CA,
USA††booktitle: Proceedings of the 1st International Workshop on Software
Defect Datasets (SDD ’23), December 8, 2023, San Francisco, CA, USA††ccs:
Theory of computation Unsupervised learning and clustering††ccs: Software and
its engineering Software defect analysis

##  1\. Introduction

Log data is crucial in the analysis, maintenance, and fixing of errors in
software systems (Davies and Roper, 2013; Beyer et al., 2016). Logs are
verbose documents that detail a system’s state throughout its execution
lifecycle and as such, lead to vast amounts of data that make manual analysis
intractable in practice (Landauer et al., 2020a). Consequently, both research
and industry communities have focused on automatic log analysis tasks such as
failure diagnosis and prediction (Zhang et al., 2018), anomaly detection (Fu
et al., 2009a), log comprehension (Kiernan and Terzi, 2009), _inter alia_.

Log data for a system is temporal, i.e., logs stream sequentially as a system
is developed and tested. Therefore, one can monitor the existence and
persistence of code defects by monitoring their existence in logs (Li et al.,
2017). To do this, past research has focused on focused log comprehension
(Landauer et al., 2020b; Meng et al., 2019) and log evolution tracking. The
latter task has mainly been approached with heuristic-based systems (Jiang et
al., 2011; Shima, 2016) and unsupervised syntax-based systems (Hamooni et al.,
2016; He et al., 2017; Gurumdimma et al., 2015; Leichtnam et al., 2017). These
systems work well for small domains where data is structured, however, the
unstructured and large nature of modern log data limit the success of these
techniques for real world data.

Another approach to capture log evolution is to consider the semantic
representations of logs. Using semantic rather than syntactic meaning for logs
makes sense as they may vary across applications and developers (i.e., have
different free-form text). Semantic representations for logs have been
previously used for anomaly detection and defect prediction due to the free-
text nature of logs (Li et al., 2020; Meng et al., 2019). Moreover, past work
has also focused on static semantic log clustering using common approaches
(e.g., K-Means or Gaussian Mixture Models) (Grigorieva and Grin, 2021). While
dynamic clustering techniques exist (Cao et al., 2006), they lead to overly
erratic log evolution. Therefore, a gap exists in the software defect tracking
literature that we aim to explore.

In this paper, we introduce a novel online algorithm for clustering and
monitoring logs based on their semantic representation. Through collaboration
with a team of $61$ Software Reliability Engineers (SREs), who are responsible
for almost $500$ applications that jointly produce roughly $100,\\!000$ logs
per day, we extracted three key criteria for successful monitoring of log
evolution: identifying errors, providing meaningful representative messages,
and minimising disruption of log clusters. These criteria were used to
construct a novel evaluation metric that captures real world performance for
temporal log clustering. Experiments using a private industrial dataset
demonstrate that our online clustering method outperforms existing static and
dynamic clustering methods. Additionally, we demonstrate that using a richer
semantic representation further improves performance. We note that our work
could be used in generating more high level error log topics in both temporal
and non-temporal datasets. The contribution of this works are:

  1. (1)

We introduce a novel online semantic-based clustering algorithm for
classifying errors and monitoring defect evolution.

  2. (2)

We construct a performance metric for log evolution based on the experience of
real SREs.

  3. (3)

We demonstrate that our algorithm improves upon past clustering techniques on
a private dataset and provide baseline metrics on public log datasets.

(a) Time $t$

(b) Time $t+1$

(c) Time $t+2$

Figure 1. Example of Error Log Evolution. The top row provides a visual aid of
the clusters while the bottom row details the specific defects found at each
point in time.

##  2\. Related Work

Log mining in the context of software defects is a well-researched topic
(Landauer et al., 2020b) which mainly focuses on failure diagnosis (Zhang et
al., 2018), anomaly detection (Fu et al., 2009a) and log comprehension
(Kiernan and Terzi, 2009). Older research focuses on tracking log patterns
over time through heuristics like frequency, sentence length or n-grams to
group logs together (Jiang et al., 2011; Shima, 2016; Dai et al., 2022). More
recently, the FLAP system was introduced as an end-to-end framework for log
analysis (Li et al., 2017) based on syntactic log properties and an iterative
clustering process (Makanju et al., 2009). Other work exists that utilises
syntactic features to group logs (Fu et al., 2009b; Hamooni et al., 2016;
Vaarandi and Pihelgas, 2015; Tang et al., 2011) and track formed clusters (He
et al., 2017; Du and Li, 2019). The techniques presented in these works all
fail to extract a representative log for clustering which is a desirable
quality in terms of the explainability of the log error types. Similar work
has shown that it is possible to achieve syntactic representatives
(Jayathilake et al., 2017; Leichtnam et al., 2017; Gurumdimma et al., 2015).
Furthermore, past work has also utilised genetic algorithms to extract log
templates (Messaoudi et al., 2018).

Semantic representations of logs have been explored in anomaly detection and
defect prediction (Du et al., 2017; Li et al., 2020; Zhang et al., 2019; Meng
et al., 2019). Research also exists that examines log comprehension and
clustering using semantic representations (e.g., Word2Vec) (Grigorieva and
Grin, 2021). This work differs from the system proposed in this paper in two
fundamental ways. Firstly, it treats logs using a bag-of-words (BoW) approach
and so does not consider word order nor frequency. Therefore, it discards
important information that exists in more complex representations such as the
sentence embeddings of SBERT (Reimers and Gurevych, 2019). Secondly, it
applies the DBSCAN clustering algorithm which is a static algorithm and so it
is impossible to track log clusters over time. There are streaming clustering
techniques such as DenStream (Cao et al., 2006) which work with vector
representations, however, they do not provide representative extraction and
can have an erratic convergence.

##  3\. Tracking Error Log Evolution

This paper concerns the monitoring of error logs which are descriptive of
various defects encountered in a system’s execution. We consider a log to be
an individual message sent by the system during its execution (logs sent from
the same execution are related by an execution ID). In order to motivate and
devise an effective solution to our task, we introduce three key criteria for
successful monitoring of log evolution that were reached in collaboration with
over 60 SREs. We further motivate our criteria by a visual example in Figure
1.

#### Identifying Errors over Time

We wish to correctly discover patterns that represent a common defect or error
expressed by logs. We expect the types of errors to change over time and be
correctly captured by the algorithm. As can be seen in Figure 1, over time, we
no longer witness “User Interface Error“ defects while we begin to witness
“Connection Timeout“ defects.

#### Capturing Meaningful Representatives

It is important that we are able to describe each cluster by its common
defect. As such, a successful algorithm should provide a meaningful
representative to each cluster that an end-user is able to understand. A
meaningful representative is visually depicted to be near the centre of each
cluster in Figure 1. Additionally, each cluster has an understandable and
succinct representative error that aligns with common code defects as seen in
the tables of Figure 1.

#### Minimising Disruption

In order to effectively monitor the lifecycle of an error or defect, clusters
must remain active until their associated errors are fully resolved. As such,
we require clusters to evolve smoothly over time while maintaining high
quality clusters. There is minimal disruption illustrated in Figure 1 as
cluster 1 expands and eventually breaks off into cluster 4 (perhaps along with
cluster 2). The smooth evolution over time is needed to not introduce or
remove a defect cluster too quickly or erratically.

###  3.1. A Novel Online Clustering Algorithm

One of the contributions of this work is a novel online clustering algorithm
for error logs that is motivated by the above criteria. We draw some
inspiration from DenStream (Cao et al., 2006) who create dynamically changing
clusters. In particular, we adopt the same distance metric between logs and
cluster centroids. The intuition of our online clustering algorithm is
straight-forward: When new data points are received, we treat points either
one by one or in batches and merge them into the cluster with the smallest
cosine distance to the centroid of that cluster. We formulate our algorithm in
Alg. 1 using three hyperparameters:

  * $\theta$: The acceptance threshold of a log to a cluster. If the distance between a log and all cluster centroids is less than $\theta$, a new cluster is created.

  * $\alpha$: The rate of cluster centroid evolution. We update cluster centroids as a rolling average based on new logs that are merged into a cluster. Consequently, cluster centroids naturally change over time. To prevent this from occurring too quickly (and thus minimise disruption), we use $\alpha$ to slow change.

  * $\gamma$: The minimum cluster size before a centroid of a cluster may change. We use this hyperparameter to ensure outliers do not heavily impact smaller clusters that are more liable to centroid changes.

Data: List of batches

Hyperparameters: $\theta$; $\alpha$; $\gamma$

clusters = [];

for  _batch in data_ do

for  _p in batch_ do

min_dist = inf; c = null;

for  _clust in clusters_ do

dist = 1 - cosine_sim(clust.cen, p);

if  _dist $<$ min_dist_ then

min_dist = dist;

c = clust;

end if

end for

if  _min_dist $\leq$ $\theta$_ then

if  _c.len $\geq$ $\gamma$_ then

c.cen = (1-$\alpha$) $\cdot$ c.cen + $\alpha p$;

else

c.cen = $\frac{c.len}{c.len+1}\cdot c.cen$ \+ $\frac{1}{c.len+1}p$;

end if

c.len += 1

else

clust = new Clust(cen=p, len=1);

clusters.add(clust);

end if

end for

end for

Algorithm 1 Our online clustering algorithm, with updates coming as individual
points. Batched updates are more efficient.

For an end user to clearly interpret the clustering results, we extract an
error message from each cluster. For each cluster, we use cosine similarity to
select the sample which is closest to the centroid and use its log as the
cluster’s representative defect. Alternatively, the Levenshtein score can be
used to get the average message (Levenshtein, 1965). While this method may
give a better message as it is more specific to the log text, it is more
computationally expensive and so we stick to the cosine similarity
approach.111Let each cluster have $m$ logs, the average log length is $n$, and
the semantic representation dimensionality is $d$ where $d<<n$. Computing the
Levenshtein score has a time complexity of $O(m^{2}n^{2})$ as opposed to
$O(md)$ for the cosine similarity approach.

###  3.2. Performance Metrics

We next introduce a novel metric that formalises the three key criteria
identified at the start of this section. Our metric is devised of three
scores: Silhouette score ($S$), Representative Similarity ($R$) and Number of
Clusters Similarity ($C$). Each of these scores is computed over batches and
is in the range $[0,1]$.

The Silhouette score is a standard metric used for assessing unlabelled
clustering. It provides a measure of the similarity of each log to its own
cluster versus other clusters. Traditionally, the score is given in the range
$[-1,1]$, but we scale it so that it is comparable with our other scores.

(1) $S=\frac{1}{B}\sum_{b=1}^{B}\frac{Silhouette_{b}+1}{2}$

where $Silhouette_{b}$ is the Silhouette score for bath $b$.

The representative similarity score compares the representatives of each
cluster $c$ for batches $b$ and $b+1$ coming in at times $t$ and $t+1$
respectively. We use this score to evaluate the evolution of the
representative logs for each cluster. We use the cosine similarity as our
distance metric.

(2) $R=\frac{1}{B}\sum_{b=1}^{B}cosine\\_sim(repres_{b},repres_{b+1})$

where $repres_{b}$ is the semantic representations for batch $b$.

Lastly, the number of clusters similarity score measures how smoothly the
number of clusters evolves over time.

(3) $C=1-\frac{1}{B}\sum_{b=1}^{B}\frac{\left\lvert
nr\\_clust_{b}-nr\\_clust_{b-1}\right\rvert}{max(nr\\_clust_{b},nr\\_clust_{b-1})}$

where $nr\\_clust_{b}$ is the number of clusters in batch $b$.

Our log cluster evolution metric ($LCE$) is then comprised of a linear
combination of the three scores above, where all scores are scaled to be in
the range $[0,1]$ and $w_{S}+w_{R}+w_{C}=1$

(4) $LCE=w_{S}S+w_{R}R+w_{C}C$

##  4\. Experiments

In this section, we measure the performance of our online clustering algorithm
using different semantic representations.

###  4.1. Data

We primarily conduct experiments on a private industrial dataset from the SREs
we worked with. To do this, we collected log data from a database containing
all monitored applications. We then eliminated variability by removing
timestamps and URLs. The monitored applications generate around $100,\\!000$
logs per day, of which, about $1.5\%$ are related to errors and defects. We
extracted two months’ worth of data, resulting in a total of $57,\\!000$ error
logs. Each log is comprised of structured fields (e.g., system id, date, log
level) as well as unstructured, free-form text.

Since our dataset is private and cannot be shared, we also evaluate our method
on similar public datasets. We use Loghub (He et al., 2020), a collection of
logs from different systems which have a date element, a log level and free-
form text representing the meaning of the log. We note that this dataset is
significantly simpler than the private industry dataset due to generally
shorter and less varied log texts. We use a sample of $2,\\!000$ logs for each
of the Loghub systems: HDFS_2, Linux, Zookeeper and OpenStack. Note that each
dataset is much smaller than the private dataset and examines a single
framework rather than many applications. Throughout both the public and
private datasets, we consider a log to be stale after one month since its
creation.

#### Pre-processing and Semantic Representations

Logs are a challenging data type for text understanding models because they
tend to contain incomplete words, words with special characters (e.g.,
underscore), and system-specific terms. Consequently, we cleaned the data
using standard natural languages processing techniques, such as
tokenization(Webster and Kit, 1992), lemmatization(Khyani and B S, 2021), and
removal of English stop words given by the SpaCy library222https://spacy.io/.
Given the cleaned logs, we constructed semantic representations of the logs
using average Word2Vec333https://spacy.io/usage/spacy-101/#vectors-similarity
vectors as well as SBERT sentence embeddings (Reimers and Gurevych, 2019).

#### Temporal Splits

For the private dataset, we first split the two months’ data into a one-month
batch and the rest into five-day batches. The second split considers only one-
day batches. The reason for these two separate temporal settings is to show
results on stationary versus dynamic data. In the first split, due to the high
amount of data in the snapshot, the underlying distribution does not differ
from the next timestamps. However, for one-day splits, one batch is not a
meaningful sample of the entire distribution of logs. We use these datasets to
compare our model with other clustering algorithms. We use one day batches for
the public datasets.

###  4.2. Models and Algorithms

To benchmark our algorithm against the literature we picked two well-known
models which work with vector representations. Gaussian Mixture Model (GMM) is
an offline clustering algorithm which was selected because it allows clusters
of any shape to be formed. We adapt the algorithm to be online by re-
initialising each batch with the mean and covariance of the model trained on
the previous batch. We choose DenStream (Cao et al., 2006) as the second
algorithm because it trains in an online fashion and also allows clusters of
varying shapes. The optimal values for the hyper-parameters of all models were
selected based on a grid search conducted over past data.444For our algorithm,
the hyperparameters were $\theta=0.05$, $\alpha=0.1$, and $\gamma=100$. For
each algorithm we only use Word2Vec (embedding dimension of $300$) to compare
the algorithm’s performance. We further use SBERT (embedding dimension of
$756$) for our algorithm to show the improvement in using a more complex and
rich semantic representation. Non eof the models were fine-tuned on the data.

Table 1. Clustering evolution performance on private dataset.  Model | # Clusters | $\boldsymbol{R}$ | $\boldsymbol{S}$ | $\boldsymbol{C}$ | $\boldsymbol{LCE}$
---|---|---|---|---|---
| 1 Month snapshot + 5 day batches
GMMWord2Vec | 11 | 0.964 | 0.768 | 1 | 0.9
DenStreamWord2Vec | 48 | 0.92 | 0.771 | 0.803 | 0.83
OursWord2Vec | 198 | 0.999 | 0.865 | 0.96 | 0.94
OursSBERT | 27 | 0.999 | 0.97 | 0.96 | 0.97
| 1 day batches
GMMWord2Vec | 11 | 0.914 | 0.727 | 1 | 0.88
DenStreamWord2Vec | 48 | 0.92 | 0.771 | 0.862 | 0.85
OursWord2Vec | 205 | 0.999 | 0.879 | 0.999 | 0.95
OursSBERT | 28 | 0.999 | 0.972 | 0.998 | 0.98
Table 2. Clustering evolution performance on private and public datasets using OursSBERT and one day batches. Dataset | # Clusters | $\boldsymbol{R}$ | $\boldsymbol{S}$ | $\boldsymbol{C}$ | $\boldsymbol{LCE}$
---|---|---|---|---|---
Private Dataset | 28 | 0.999 | 0.972 | 0.998 | 0.980
HDFS_2 | 18 | 0.990 | 0.990 | 0.990 | 0.990
Linux | 31 | 0.990 | 0.860 | 0.980 | 0.940
Zookeper | 19 | 0.990 | 0.990 | 0.990 | 0.990
OpenStack | 14 | 0.990 | 0.790 | 0.990 | 0.920

###  4.3. Results

For all results, $LCE$ was computed with equal contribution weights. Table 1
shows the performance results across the different algorithms and semantic
representations on the private dataset. We observe that for both temporal
settings, our approach matches or outperforms both the GMM and DenStream
algorithms when using Word2Vec semantic representations across each individual
score as well as $LCE$. We note that GMM, the static clustering algorithm,
performs better when it has the initial one month snapshot whereas the dynamic
approaches perform consistently across both settings. This is expected as GMM
requires the initial mean and variance to attain better performance.

The biggest improvement between our approach DenStream is observed in the
cluster number similarity score. DenStream exhibits an erratic evolution which
we are able to mitigate in our approach using our evolution rate ($\alpha$)
and minimum cluster size hyperparameters ($\gamma$). The strong and consistent
performance of our approach across both temporal settings suggests that our
algorithm adapts easily to both stationary and non-stationary streams. The
robustness of our algorithm is further shown in Table 2 where we achieve good
results across public log datasets.555We further conducted a brief qualitative
analysis with the SREs we worked with. They provided positive feedback with
regards to the cluster evolution over time and pointed out certain clusters
matched specific errors they were aware of.

##  5\. Threats to Validity

We note a few threats to validity in this work. Firstly, we must be cautious
of overfitting to our data which would retract from the generalisibility of
the model. As clustering is an unsupervised problem (i.e., no labelled data),
we cannot use metrics such as ARI to evaluate our predictions. While the
Silhouette score and manual qualitative analysis can be done to demonstrated
good clusters, having access to a small labelled dataset would allow for more
robust evaluation. This is currently a gap in log-based software defect
datasets that we hope is addressed in future work. Indeed, one could use a
model as presented in this work to generate data.

##  6\. Conclusion

In this paper, we introduced a novel online semantic-based clustering
algorithm for error logs. Our algorithm is able to cluster log message
streams, track the reduction and emergence of defects that arise in code, and
provide a representative defect message for each cluster. The algorithm has
configurable cluster granularity and cluster evolution rate through
hyperparameters. Furthermore, we introduced a novel metric named $LCE$ that
can evaluate defect tracking performance based on industry specified criteria.
We demonstrated that our algorithm outperforms existing log clustering
approaches using a private dataset and showed strong performance across public
datasets. We hope that this work encourages more work in defect detection to
examine the temporal properties of logs and incorporate it into modern
systems.

## Disclaimer

This paper was prepared for informational purposes by the Artificial
Intelligence Research group of JPMorgan Chase & Co and its affiliates (“JP
Morgan”), and is not a product of the Research Department of JP Morgan. JP
Morgan makes no representation and warranty whatsoever and disclaims all
liability, for the completeness, accuracy or reliability of the information
contained herein. This document is not intended as investment research or
investment advice, or a recommendation, offer or solicitation for the purchase
or sale of any security, financial instrument, financial product or service,
or to be used in any way for evaluating the merits of participating in any
transaction, and shall not constitute a solicitation under any jurisdiction or
to any person, if such solicitation under such jurisdiction or to such person
would be unlawful.

## References

  * (1)
  * Beyer et al. (2016) Betsy Beyer, Chris Jones, Jennifer Petoff, and Niall Richard Murphy. 2016.  _Site Reliability Engineering: How Google Runs Production Systems_.  http://landing.google.com/sre/book.html
  * Cao et al. (2006) Feng Cao, Martin Ester, Weining Qian, and Aoying Zhou. 2006\.  Density-Based Clustering over an Evolving Data Stream with Noise. In _SDM_.
  * Dai et al. (2022) Hetong Dai, Heng Li, Che-Shao Chen, Weiyi Shang, and Tse-Hsun Chen. 2022.  Logram: Efficient Log Parsing Using $n$n-Gram Dictionaries.  _IEEE Transactions on Software Engineering_ 48, 3 (2022), 879–892.  https://doi.org/10.1109/TSE.2020.3007554
  * Davies and Roper (2013) Steven Davies and Marc Roper. 2013.  Bug localisation through diverse sources of information. In _2013 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW)_. 126–131.  https://doi.org/10.1109/ISSREW.2013.6688891
  * Du and Li (2019) Min Du and Feifei Li. 2019\.  Spell: Online Streaming Parsing of Large Unstructured System Logs.  _IEEE Transactions on Knowledge and Data Engineering_ 31, 11 (2019), 2213–2227.  https://doi.org/10.1109/TKDE.2018.2875442
  * Du et al. (2017) Min Du, Feifei Li, Guineng Zheng, and Vivek Srikumar. 2017\.  DeepLog: Anomaly Detection and Diagnosis from System Logs through Deep Learning. In _Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security_ (Dallas, Texas, USA) _(CCS ’17)_. Association for Computing Machinery, New York, NY, USA, 1285–1298.  https://doi.org/10.1145/3133956.3134015
  * Fu et al. (2009a) Qiang Fu, Jian-Guang Lou, Yi Wang, and Jiang Li. 2009a.  Execution Anomaly Detection in Distributed Systems through Unstructured Log Analysis. In _2009 Ninth IEEE International Conference on Data Mining_. 149–158.  https://doi.org/10.1109/ICDM.2009.60
  * Fu et al. (2009b) Qiang Fu, Jian-Guang Lou, Yi Wang, and Jiang Li. 2009b.  Execution Anomaly Detection in Distributed Systems through Unstructured Log Analysis.  _2009 Ninth IEEE International Conference on Data Mining_ (2009), 149–158.
  * Grigorieva and Grin (2021) Maria Grigorieva and Dmitry Grin. 2021.  Clustering error messages produced by distributed computing infrastructure during the processing of high energy physics data.  _International Journal of Modern Physics A_ 36 (04 2021), 2150070\.  https://doi.org/10.1142/S0217751X21500706
  * Gurumdimma et al. (2015) Nentawe Gurumdimma, Arshad Jhumka, Maria Liakata, Edward Chuah, and James Browne. 2015.  Towards Detecting Patterns in Failure Logs of Large-Scale Distributed Systems. In _2015 IEEE International Parallel and Distributed Processing Symposium Workshop_. 1052–1061.  https://doi.org/10.1109/IPDPSW.2015.109
  * Hamooni et al. (2016) Hossein Hamooni, Biplob Debnath, Jianwu Xu, Hui Zhang, Guofei Jiang, and Abdullah Mueen. 2016\.  LogMine: Fast Pattern Recognition for Log Analytics. In _Proceedings of the 25th ACM International on Conference on Information and Knowledge Management_ (Indianapolis, Indiana, USA) _(CIKM ’16)_. Association for Computing Machinery, New York, NY, USA, 1573–1582.  https://doi.org/10.1145/2983323.2983358
  * He et al. (2017) Pinjia He, Jieming Zhu, Zibin Zheng, and Michael R. Lyu. 2017\.  Drain: An Online Log Parsing Approach with Fixed Depth Tree. In _2017 IEEE International Conference on Web Services (ICWS)_. 33–40.  https://doi.org/10.1109/ICWS.2017.13
  * He et al. (2020) Shilin He, Jieming Zhu, Pinjia He, and Michael R. Lyu. 2020\.  Loghub: A Large Collection of System Log Datasets towards Automated Log Analytics.  https://doi.org/10.48550/ARXIV.2008.06448
  * Jayathilake et al. (2017) P.W.D.C. Jayathilake, N. R. Weeraddana, and H. K. E. P. Hettiarachchi. 2017.  Automatic detection of multi-line templates in software log files. In _2017 Seventeenth International Conference on Advances in ICT for Emerging Regions (ICTer)_. 1–8.  https://doi.org/10.1109/ICTER.2017.8257824
  * Jiang et al. (2011) Yexi Jiang, Chang-Shing Perng, and Tao Li. 2011.  Natural Event Summarization. In _Proceedings of the 20th ACM International Conference on Information and Knowledge Management_ (Glasgow, Scotland, UK) _(CIKM ’11)_. Association for Computing Machinery, New York, NY, USA, 765–774.  https://doi.org/10.1145/2063576.2063688
  * Khyani and B S (2021) Divya Khyani and Siddhartha B S. 2021.  An Interpretation of Lemmatization and Stemming in Natural Language Processing.  _Shanghai Ligong Daxue Xuebao/Journal of University of Shanghai for Science and Technology_ 22 (01 2021), 350–357.
  * Kiernan and Terzi (2009) Jerry Kiernan and Evimaria Terzi. 2009.  Constructing Comprehensive Summaries of Large Event Sequences.  _ACM Trans. Knowl. Discov. Data_ 3, 4, Article 21 (dec 2009), 31 pages.  https://doi.org/10.1145/1631162.1631169
  * Landauer et al. (2020a) Max Landauer, Florian Skopik, Markus Wurzenberger, and Andreas Rauber. 2020a.  System log clustering approaches for cyber security applications: A survey.  _Computers Security_ 92 (2020), 101739.  https://doi.org/10.1016/j.cose.2020.101739
  * Landauer et al. (2020b) Max Landauer, Florian Skopik, Markus Wurzenberger, and Andreas Rauber. 2020b.  System log clustering approaches for cyber security applications: A survey.  _Computers and Security_ 92 (2020), 101739.  https://doi.org/10.1016/j.cose.2020.101739
  * Leichtnam et al. (2017) Laetitia Leichtnam, Eric Totel, Nicolas Prigent, and Ludovic Me. 2017.  STARLORD: Linked security data exploration in a 3D graph. In _2017 IEEE Symposium on Visualization for Cyber Security (VizSec)_. 1–4.  https://doi.org/10.1109/VIZSEC.2017.8062203
  * Levenshtein (1965) Vladimir I. Levenshtein. 1965\.  Binary codes capable of correcting deletions, insertions, and reversals.  _Soviet physics. Doklady_ 10 (1965), 707–710.
  * Li et al. (2017) Tao Li, Yexi Jiang, Chunqiu Zeng, Bin Xia, Zheng Liu, Wubai Zhou, Xiaolong Zhu, Wentao Wang, Liang Zhang, Jun Wu, Li Xue, and Dewei Bao. 2017.  FLAP: An End-to-End Event Log Analysis Platform for System Management. In _Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ (Halifax, NS, Canada) _(KDD ’17)_. Association for Computing Machinery, New York, NY, USA, 1547–1556.  https://doi.org/10.1145/3097983.3098022
  * Li et al. (2020) Xiaoyun Li, Pengfei Chen, Linxiao Jing, Zilong He, and Guangba Yu. 2020.  SwissLog: Robust and Unified Deep Learning Based Log Anomaly Detection for Diverse Faults. In _2020 IEEE 31st International Symposium on Software Reliability Engineering (ISSRE)_. 92–103.  https://doi.org/10.1109/ISSRE5003.2020.00018
  * Makanju et al. (2009) Adetokunbo Makanju, Ayse Nur Zincir-Heywood, and Evangelos E. Milios. 2009.  Clustering event logs using iterative partitioning. In _Knowledge Discovery and Data Mining_.
  * Meng et al. (2019) Weibin Meng, Ying Liu, Yichen Zhu, Shenglin Zhang, Dan Pei, Yuqing Liu, Yihao Chen, Ruizhi Zhang, Shimin Tao, Pei Sun, and Rong Zhou. 2019.  LogAnomaly: Unsupervised Detection of Sequential and Quantitative Anomalies in Unstructured Logs. In _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19_. International Joint Conferences on Artificial Intelligence Organization, 4739–4745.  https://doi.org/10.24963/ijcai.2019/658
  * Messaoudi et al. (2018) Salma Messaoudi, Annibale Panichella, Domenico Bianculli, Lionel Briand, and Raimondas Sasnauskas. 2018\.  A Search-Based Approach for Accurate Identification of Log Message Formats. In _2018 IEEE/ACM 26th International Conference on Program Comprehension (ICPC)_. 167–16710.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks.  https://doi.org/10.48550/ARXIV.1908.10084
  * Shima (2016) Keiichi Shima. 2016\.  Length Matters: Clustering System Log Messages using Length of Words.  https://doi.org/10.48550/ARXIV.1611.03213
  * Tang et al. (2011) Liang Tang, Tao Li, and Chang-Shing Perng. 2011.  LogSig: Generating System Events from Raw Textual Logs. In _Proceedings of the 20th ACM International Conference on Information and Knowledge Management_ (Glasgow, Scotland, UK) _(CIKM ’11)_. Association for Computing Machinery, New York, NY, USA, 785–794.  https://doi.org/10.1145/2063576.2063690
  * Vaarandi and Pihelgas (2015) Risto Vaarandi and Mauno Pihelgas. 2015.  LogCluster - A data clustering and pattern mining algorithm for event logs. In _2015 11th International Conference on Network and Service Management (CNSM)_. 1–7.  https://doi.org/10.1109/CNSM.2015.7367331
  * Webster and Kit (1992) Jonathan Webster and Chunyu Kit. 1992.  Tokenization as the initial phase in NLP. 1106–1110.  https://doi.org/10.3115/992424.992434
  * Zhang et al. (2018) Shenglin Zhang, Ying Liu, Weibin Meng, Zhiling Luo, Jiahao Bu, Sen Yang, Peixian Liang, Dan Pei, Jun Xu, Yuzhi Zhang, Yu Chen, Hui Dong, Xianping Qu, and Lei Song. 2018.  PreFix: Switch Failure Prediction in Datacenter Networks.  _Abstracts of the 2018 ACM International Conference on Measurement and Modeling of Computer Systems_ (2018).
  * Zhang et al. (2019) Xu Zhang, Yong Xu, Qingwei Lin, Bo Qiao, Hongyu Zhang, Yingnong Dang, Chunyu Xie, Xinsheng Yang, Qian Cheng, Ze Li, Junjie Chen, Xiaoting He, Randolph Yao, Jian-Guang Lou, Murali Chintalapati, Furao Shen, and Dongmei Zhang. 2019.  Robust Log-Based Anomaly Detection on Unstable Log Data. In _Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ (Tallinn, Estonia) _(ESEC/FSE 2019)_. Association for Computing Machinery, New York, NY, USA, 807–817.  https://doi.org/10.1145/3338906.3338931