# Evaluating the Ability of LLMs to Solve Semantics-Aware Process Mining Tasks

Adrian Rebmann1, Fabian David Schmidt2, Goran Glavaš2, and Han van der Aa3
1Data and Web Science Group, University of Mannheim, Germany
<EMAIL_ADDRESS>2Center for Artificial Intelligence and Data Science,
University of Würzburg, Germany
{fabian.schmidt<EMAIL_ADDRESS>3Faculty of Computer Science,
University of Vienna, Austria
<EMAIL_ADDRESS>
###### Abstract

The process mining community has recently recognized the potential of large
language models (LLMs) for tackling various process mining tasks. Initial
studies report the capability of LLMs to support process analysis and even, to
some extent, that they are able to reason about how processes work. This
latter property suggests that LLMs could also be used to tackle process mining
tasks that benefit from an understanding of process behavior. Examples of such
tasks include (semantic) anomaly detection and next activity prediction, which
both involve considerations of the meaning of activities and their inter-
relations. In this paper, we investigate the capabilities of LLMs to tackle
such semantics-aware process mining tasks. Furthermore, whereas most works on
the intersection of LLMs and process mining only focus on testing these models
out of the box, we provide a more principled investigation of the utility of
LLMs for process mining, including their ability to obtain process mining
knowledge post-hoc by means of in-context learning and supervised fine-tuning.
Concretely, we define three process mining tasks that benefit from an
understanding of process semantics and provide extensive benchmarking datasets
for each of them. Our evaluation experiments reveal that (1) LLMs fail to
solve challenging process mining tasks out of the box and when provided only a
handful of in-context examples, (2) but they yield strong performance when
fine-tuned for these tasks, consistently surpassing smaller, encoder-based
language models.

###### Index Terms:

process mining, large language models, anomaly detection, next activity
prediction

##  I Introduction

Process mining involves analyzing event data from organizational processes to
gain actionable insights into the true manner in which they are executed.
Recently, the process mining community has taken notice of the potential of
large language models (LLMs) in tackling process mining tasks. Exploratory
studies have looked into the effectiveness of these generative models for this
purpose, yielding promising initial results. Among others, LLMs demonstrate
the ability to describe processes textually using event data, answer questions
about the event data, and assist in formulating queries to address specific
analysis questions [1, 2, 3]. This capability shows their potential to offer
valuable assistance to process analysts in manual data exploration and
analysis.

The greatest potential from an automated process analysis point-of-view lies
in LLMs’ impressive natural language understanding capabilities. These
capabilities could provide a strong foundation for supporting process mining
tasks, particularly those that benefit from a consideration of the meaning of
process steps and their relations. Examples include anomaly detection [4] and
next activity prediction [5]. Anomaly detection can improve performance by
identifying undesired process behavior based on the meaning of activities. For
instance, it can detect when a delivery is created despite the corresponding
purchase order being canceled. Next activity prediction can enhance
performance by narrowing down the set of potential next activities to those
that make semantic sense. For example, it can discard a _check request_
activity if the request has already been approved.

The capabilities of (smaller) language models for tackling process mining
tasks such as semantic anomaly detection have been shown [6]. However, an
investigation of the utility of LLMs for solving such semantic-aware process
mining tasks is still missing. This particularly relates to in-depth
evaluations of the capabilities of such models, e.g., with respect to their
ability to obtain process mining knowledge post-hoc by means of in-context
learning and supervised fine-tuning. A reason for this is the lack of properly
defined natural language processing (NLP) tasks that effectively conceptualize
the capability to perform semantics-aware process mining tasks. Consequently,
there is also a lack of benchmarking datasets similar to those available for
established NLP tasks such as Question Answering [7] and Text Summarization
[8], hindering structured evaluation experiments.

In this paper, we address these problems through the following contributions:

  * •

We define three tasks (Section III) that allow for assessing the capabilities
of (large) language models for semantics-aware process mining: (1) determining
whether an activity sequence represents a valid execution of a process, (2)
deciding whether the execution order of two activities is valid, and (3)
selecting the next process activity to be performed given an incomplete
activity sequence.

  * •

We provide a corpus of process behavior (Section IV) based on the largest
publicly available process model collection that can be used to establish
benchmarking datasets for semantic process analysis tasks. Based on this
corpus, we establish and provide benchmarking data sets that allow for
conducting quantitative evaluations of the performance of language models on
the three proposed tasks.

  * •

We assess the performance of open-source LLMs to solve the proposed tasks
based on the established benchmarking datasets in an experimental evaluation
(Section VI–Section VII). Therein, we compare LLMs in _in-context learning_
and _fine-tuning_ settings and discriminative encoder-based language models.

Our results indicate that the proposed tasks are too challenging for LLMs to
solve them out of the box and using in-context learning. However, if an LLM is
fine-tuned on a specific task it can achieve accurate results and consistently
outperforms discriminative encoder-based language models.

##  II Preliminaries

In this section, we define preliminaries on process models, event data, and
eventually-follows relations, which we require for the remainder of the paper.

Process Models. Let $\mathcal{A}$ be the universe of possible activities that
can be performed in organizational processes. We define a process model $M$ as
the set of executions that are allowed in a process. Each execution $\pi$ is
represented as an activity sequence $\pi=\langle a_{1},...,a_{n}\rangle$, with
$a_{i}\in\mathcal{A}$. We use $A_{M}\subseteq\mathcal{A}$, to denote the set
of activities that appear in the sequences of $M$.

Event Data. We adopt a simple event model, focusing on the control-flow of a
process. A trace $\sigma$ is a sequence that represents the events that have
been recorded for the execution of a single instance of an organizational
process. Such a trace consists of a finite sequence of activities
$\sigma=\langle a_{1},...,a_{n}\rangle$, with $a\in\mathcal{A}$. An event log
$L$ is a finite multi-set of traces. $A_{L}\subseteq\mathcal{A}$ denotes the
set of activities that appear in the traces of $L$.

Eventually-Follows Relations. We use the eventually-follows relation $\prec$
to capture ordering relations between pairs of activities (stemming either
from the traces of an event log or from activity sequences of a process
model). Given a trace $\sigma=\langle a_{1},...,a_{n}\rangle$, we use
$a_{i}\prec_{\sigma}a_{j}$ to denote that $a_{i}$ occurs (directly or
indirectly) before $a_{j}$ in the trace, with $1\leq i<j\leq n$. Accordingly,
$a_{i}\prec_{M}a_{j}$ holds if a model $M$ allows for an activity sequence
$\pi$ in which activity $a_{i}$ occurs before $a_{j}$. $\mathit{EF}_{M}$
denotes all eventually-follows relations of the activity sequences allowed by
$M$.

##  III Tasks

This section describes and defines three process mining tasks that benefit
from an understanding of process behavior. We design these tasks (1) such that
there is no access to historical event data and (2) to consider solely the
control-flow perspective. (1) allows us to assess whether a language model can
solve the tasks based purely on its encoded knowledge of how processes
generally work. (2) ensures that a language model determines its results
exclusively based on the meaning of the activities and their interrelations,
excluding other aspects like data attributes, which are often highly
organization-specific. Our tasks include two forms of _semantic anomaly
detection_ , as well as _semantic next activity prediction_.

###  III-A Semantic Anomaly Detection

Anomaly detection in process mining aims to identify outlying process behavior
in the traces of an event log [9]. Many approaches do this by identifying
statistical outliers [10], arguing that behavior is anomalous if it is
infrequent.

By contrast, semantic anomaly detection [4] focuses on the identification of
process behavior that does not make sense. Arguing that, just because behavior
is infrequent, does not make it anomalous, whereas just because something
happens regularly, does not mean that it is proper process behavior. For
instance, from a semantic point of view, an invoice should not be created if
the corresponding purchase order has been rejected and should therefore be
detected as anomalous behavior, independent of how frequently this happens.

Detecting anomalies based on process semantics requires a different approach
compared to frequency-based anomaly detection. Whereas frequency-based
detection can be performed by just using data in an event log (revealing
statistical outliers), semantic anomaly detection requires information about
how a process should (or should not) work in general. By definition, such
information needs to be obtained from outside of the event log, e.g., from
large knowledge bases or—as we do in this paper—from the knowledge encoded in
LLMs.

We define two specific tasks in this context, focusing on the trace and
activity-relation levels:

Trace-Level Semantic Anomaly Detection. Trace-level semantic anomaly detection
(T-SAD) is a binary classification problem in which a trace $\sigma\in L$
needs to classified as anomalous or not, according to its semantics. For
instance, given a trace $\sigma=\langle$_register application_ , _approve
application_ , _review application_ $\rangle$, the task is to classify that
$\sigma$ is anomalous. This is the case because an application should first be
reviewed and only then approved (or rejected). The challenge here is that
there is no specification available of the process at hand that can be used
for this. Rather that anomality needs to be inferred, requiring an
understanding of how processes generally work.

Activity-Level Semantic Anomaly Detection. Activity-level semantic anomaly
detection (A-SAD) is a more fine-granular task than T-SAD, focusing on pairs
of activities in a trace rather than on an entire trace at once. Specifically,
A-SAD focuses on classifying any eventually-follows relation
$a_{i}\prec_{\sigma}a_{j}$ of two activities $a_{i}$ and $a_{j}$ that appear
in a respective order in a trace $\sigma$ as being anomalous or not. For
instance, given the trace $\sigma=\langle$_create purchase order_ , _reject
purchase order_ , _create invoice_ $\rangle$, the eventually-follows relation
_reject purchase order_ $\prec_{\sigma}$ _create invoice_ should be classified
as anomalous, whereas the other pairwise relations, i.e., _create purchase
order_ $\prec_{\sigma}$ _reject purchase order_ and _create purchase order_
$\prec_{\sigma}$ _create invoice_ should be classified as valid.

###  III-B Semantic Next Activity Prediction

Next activity prediction, also known as _next event_ or _next step_
prediction, is a key predictive process monitoring task. Its objective is to
determine the subsequent activity in an ongoing process execution [5]. Various
approaches, predominantly based on supervised deep learning (e.g., [11, 12]),
have been developed to tackle this task.

As a semantics-aware counterpart for next activity prediction, we introduce
the semantic next activity prediction (S-NAP) task. For an incomplete trace
$\sigma$, which represents an ongoing process execution in which $k$
activities have been performed ($k\geq 1$), the task is to predict the next
activity $a_{k+1}$ in $\sigma$ based on a set of possible activities $A$. For
instance, given $\sigma=\langle$_create purchase order_ , _approve purchase
order_ $\rangle$ and $A=\\{$_create purchase order_ , _approve purchase order_
, _create invoice_ , _make payment_ $\\}$, the task is to predict $a_{k+1}$ as
_create invoice_. This is because, generally, an invoice should be created
before a payment is made.

Whereas approaches for (traditional) next activity prediction train a model on
historical traces from an event log $L$ to predict the next activity in
ongoing (i.e., unseen) executions of the process, S-NAP focuses on situations
where no such historical traces are available. As a result, the next activity
must be inferred by considering the semantics of the activities involved in a
process.

##  IV Datasets

This section details the creation and characteristics of the text corpus and
benchmarking datasets that we use to evaluate the ability of language models
to solve the proposed tasks. We make all datasets publicly available [13].

###  IV-A A Corpus of Process Behaviors

Language models require textual input. In order to assess their ability to
solve semantics-aware process mining tasks we, therefore, need a collection of
textual representations of process behavior, a so-called _corpus_. This corpus
then serves as a basis to create task-specific data that can be used for
training and evaluating language models on the proposed tasks.

Since no such corpus is readily available, we create one based on graphical
process models (i.e., process diagrams). To this end, we use sap-sam [14],
which is the largest publicly available collection of process diagrams to
date. In order to create a high-quality corpus, we select only English BPMN
diagrams from sap-sam that meet specific requirements. These ensure that the
corpus includes only unique and valid process behavior. In particular, we
require that a diagram can be transformed into a sound workflow net and that
no two diagrams have the same activity set. The former requirement mitigates
data quality issues in the sap-sam collection [14] and ensures that we can
properly generate activity sequences from the diagram. The latter makes sure
that we do not include duplicate behavior. Furthermore, we require a diagram
to contain at least two different activities to ensure that it actually
captures ordering relations between different activities.

We use the workflow net of each selected diagram to generate activity
sequences, capturing all executions allowed by the net. For loops, we ensure
that each loop is executed at most once, so that we capture relations
involving rework, yet, obtain a finite set of activity sequences. For each
net, this yields a process model $M$ according to the definition in Section
II, i.e., a set of activity sequences capturing its allowed behavior. We add
each such $M$ to the corpus.

We show the characteristics of the resulting corpus in Table I. As depicted
there, the complexity of the process models varies considerably. For instance,
the number of unique activities is 4.9 on average, whereas the maximum is 21
and the process models allow for 10.34 activity sequences on average, whereas
the maximum amount is 10,080.

TABLE I: Characteristics of the process behavior corpus. Characteristic | Total | Per process model
---|---|---
|  | Avg. | Min. | Max.
# Process models | 15,857 | – | – | –
# Unique activities | 49,108 | 4.70 | 2 | 21
# Unique activity sequences | 163,484 | 10.34 | 1 | 10,080

Based on this corpus, we can generate task-specific datasets for the
individual tasks.

###  IV-B Task-Specific Benchmarking Datasets

With a text corpus of process behaviors available, we can generate task-
specific benchmarking datasets that include task samples and a gold standard.
This gold standard enables objective, quantitative evaluation of the language
models on the tasks based on established evaluation measures. The
characteristics of the dataset are shown in Table II, whereas they are
established in the manner described in the following.

TABLE II: Characteristics of the task-specific benchmarking datasets. Task Dataset | Total | Valid | Anomalous
---|---|---|---
T-SAD | 291,251 | 150,301 | 140,950
A-SAD | 316,308 | 158,154 | 158,154
S-NAP | 1,289,081 | 1,289,081 | -

T-SAD. To establish the T-SAD dataset, we first create an event log $L$ for
each process model $M$ in the corpus such that each $\pi\in M$ becomes a trace
$\sigma\in L$. To make sure that there is a minimum number of traces per log,
we randomly duplicate traces in $L$ until a size of 100 is reached if $L$ does
not already contain at least 100 traces. Subsequently, for each trace
$\sigma\in L$, we make a decision regarding the insertion of noise, with a 50
percent probability. This noise insertion involves swapping two randomly
selected activities within the trace. After swapping, we check whether the
resulting sequence $\sigma^{\prime}$ is indeed anomalous, i.e.,
$\sigma^{\prime}\notin M$. If the sequence is found to still be valid, i.e.,
$\sigma^{\prime}\in M$, we continue iterating through potential swaps until we
obtain an anomalous sequence111We limit the number of retries to 10 per trace
to guarantee termination.. This ensures that the dataset contains (roughly)
the same amount of valid and anomalous traces, which is crucial for robust
model training and evaluation.

Each of the 291,251 records of the T-SAD dataset then consists of a trace
$\sigma$, the correct label of $\sigma$, i.e., $\mathit{Anomalous}$ if
$\sigma\notin M$ and $\mathit{Valid}$ otherwise, and the set of possible
activities in the process from which $\sigma$ originates as context
information.

A-SAD. We create the A-SAD dataset based on the set of eventually-follows
relations $\mathit{EF}_{M}$ for each process model $M$ in the corpus. The
relations in $\mathit{EF_{M}}$ represent all valid execution orders of
activities of $M$. Next to these, we create a set of anomalous relations
$\mathit{EF_{\not M}}$ , i.e., ones that are not in $\mathit{EF_{M}}$. To
provide a balanced dataset, we establish $\mathit{EF_{\not M}}$ by randomly
selecting relations that are not in $\mathit{EF_{M}}$, until we have an equal
number of valid and anomalous relations.

Each of the 316,308 records of the A-SAD dataset consists of an eventually-
follows relation $r$, the correct label of $r$, i.e., $\mathit{Anomalous}$ if
$r\notin EF_{M}$ and $\mathit{Valid}$ otherwise, and the set of all activities
$A_{M}$ in the process model from which the activities in $r$ originate as
context information.

S-NAP. For the S-NAP dataset, we first create an event log $L$ for each
process model $M$ in the corpus such that each $\pi\in M$ becomes a trace
$\sigma\in L$. Then, we generate all possible prefixes for each trace
$\sigma\in L$ and add them to $L$. This involves iteratively considering sub-
traces of increasing length $k$ from the first activity of a trace $\sigma$,
up to its full length, which ensures that every potential prefix is captured.

Each of the 1,289,081 records of the S-NAP dataset then consists of a
length-$k$ prefix ($\sigma_{k}$) of $\sigma$, the correct label of
$\sigma_{k}$, i.e., the activity at position $k+1$ in $\sigma$, and the set of
possible activities $A_{L}$ of the event log from which $\sigma$ originates as
context information.

##  V LLM-Based Process Mining

Neural language models, based on the Transformer architecture [15, 16] come in
two main flavors: (1) bidirectional language models, also commonly called
encoders, which are typically (pre-)trained via masked language modeling
objectives in which masked tokens are predicted from both left and right
context [16, 17] and (2) unidirectional language models, also known as
decoders, which are trained via autoregressive language modeling objectives
where the next token is predicted from the preceding context [18, 19]. LLMs
are very large instances of the latter category (with at least a billion
parameters) that are, following large-scale autoregressive language modeling,
typically additionally trained for instruction following, i.e., to provide
solutions to tasks given the natural language description of these tasks [20,
21]. Such instruction-tuning allows LLMs to generalize to new tasks through
textual task descriptions (since capturing meaning of language is what LLMs
excel at) and solve them successfully even when not provided with any task-
specific (training) examples.

Fine-Tuning LMs on Classification Tasks. Fine-tuning is the process of further
training a pretrained language model, in order to specialize it for a specific
(classification or regression) task. The advantage compared to training a
model from scratch is that the training data size for fine-tuning is
considerably smaller, thus reducing resources required to train a task-
specific model. We fine-tune language models for each of our semantics-aware
process mining tasks, all of which are classification tasks. We next briefly
describe the fine-tuning procedures for (a) discriminative classification with
an encoder LM and (b) generative classification with a decoder LM.

Figure 1: Illustration of discriminative classification with an encoder LM.

Discriminative Fine-Tuning of Encoder LMs. To fine-tune an encoder LM for
classification tasks, we extend the model’s base architecture (i.e., the
pretrained Transformer network) with an additional classification layer: the
parameters of the classifier are trained from scratch (i.e., randomly
initialized), whereas the encoder’s parameters are updated (i.e., fine-tuned).
Fine-tuning of an encoder LM for the T-SAD task is illustrated in Figure 1
using a record with a trace $\langle$_create invoice_ , _make payment_ ,
$\dots\rangle$ as input. The input is first split into a sequence of subword
tokens.222For more frequent words in the language, a token will commonly
correspond to the whole word; less frequent words, on the other hand, will
often be broken down into more frequent subtokens (e.g., “tokenization” may be
segmented into “token” and “ization”). The exact subword vocabulary is model
dependent, i.e., each LM comes with its own tokenizer. The actual input for
encoder LMs is commonly surrounded with synthetic sequence start ([CLS]) and
sequence end ([SEP]) tokens. The encoder (i.e., the Transformer network)
outputs one vector—a transformed/contextualized representation—for each token
in the input sequence, including the sequence start/end tokens. Let
$\mathbf{x}_{CLS}\in\mathbb{R}^{d}$ be the representation of the sequence
start token CLS (output of the encoder) with $d$ as the hidden size of the
encoder’s Transformer network; this vector $\mathbf{x}_{CLS}$ can be seen as a
latent semantic representation of the whole input text and is forwarded as
input to the classifier. The classifier, in turn, is a single-layer feed-
forward network:
$\hat{\textbf{y}}=\mathit{softmax}(\mathbf{W}_{cl}\cdot\mathbf{x}_{CLS}+\mathbf{b}_{cl})$;
$\mathbf{W}_{cl}\in\mathbb{R}^{c\times d}$ and
$\mathbf{b}_{cl}\in\mathbb{R}^{c}$ are the trainable parameters of the
classifier ($c$ is the number of classes in the classification task) and
$\mathit{softmax}$ is the function commonly used to convert real-valued
vectors into probability distributions—the final output $\hat{\textbf{y}}$ is
thus a probability distribution over the task’s classes. We train the model
(jointly update the parameters of both classifier and encoder in end-to-end
fashion) by minimizing the widely used cross-entropy loss, i.e., the negative
logarithm of the probability that the model predicted for the true class of
the input instance.

T-SAD and A-SAD are binary classification tasks (i.e., $c=2$) in which the
model predicts whether the traces and ordered activity pairs, respectively,
are _Valid_ or _Anomalous_. S-NAP is a multi-class classification task in
which the set of classes is defined with the activities in $A_{M}$ of the
process model $M$ from which the input record was created.

Generative Fine-Tuning of Decoder (L)LMs. Autoregressively trained decoder
LLMs cast classification tasks as language generation tasks. Concretely, each
class into which the preceding text is to be classified is assigned one token
from the vocabulary and the LLM’s language modeling head (a classifier over
the LLM’s vocabulary) is supposed to generate the token of the correct class.
For example, for the T-SAD task, we convert individual training instances into
prompts that couple (1) the set of process activities with (2) the concrete
trace (or activity sequence) that is to be judged as Valid or Anomalous. Then
we append the prompt that asks whether the sequence is anomalous, with the
token true assigned to the Anomalous sequences and token false to the Valid
sequences. The whole input for the decoder LLM for a single sequence is shown
below (the label token is underlined and in blue):

_Activities:{create order, approve order, reject order, create invoice, make
payment}_
_Activity sequence: [create order, reject order, create invoice, make
payment]_
_Anomalous: true_

We fine-tune a decoder LLM via constrained text generation: given the entire
preceding context (everything except the last token that indicates the class),
we predict the next token, but allow the language modeling head to only
predict the probabilities for the allowed class tokens (in the above example,
only true, false), as opposed to LLM pretraining in which the next token is
predicted over the entire vocabulary of the LLM. We illustrate constrained
generative fine-tuning of a decoder LM in Figure 2.

Figure 2: Illustration of constrained generative fine-tuning of a decoder LM.

The Transformer network of the decoder produces the output representation by
contextualizing all preceding tokens; the resulting vector is next compared
against the representations of the allowed class tokens (in the example, true,
false) to produce scores that are then converted into probabilities using
softmax. We minimize the negative log likelihood of the probability assigned
to the correct class token: as updating all LLM parameters is computationally
infeasible, we resort to parameter-efficient fine-tuning via low-rank
adaptation (LoRA) [22].333For brevity, we refer the reader to the original
work for details on LoRA.

Few-Shot In-Context Learning. In-context learning (ICL) aims to induce a model
to perform a task by providing a small set of input-label examples (so-called
“shots”) along with the task description; the query sample—the example (input)
for which the label is to be generated—is provided at the end of the prompt
[23]. In-context learning allows the LLM to understand the task (via the
description and a few labeled examples), without supervised fine-tuning (i.e.,
without any updates to LLM’s parameters).

Figure 3 displays the one-shot prompt for the S-NAP task: the description of
the task is followed by one labeled task instance—list of possible activities
and the prefix trace in question, along with the correct label—and then the
query instance for which the LLM is to generate the label.

Figure 3: One-shot in-context-learning prompt for the S-NAP task.

##  VI Experimental Setup

We first describe how we split the datasets for training, validation, and
testing, and then introduce the concrete LMs we use. We then provide details
on our ICL and fine-tuning setups. For reproducibility, we make all training
and evaluation scripts publicly
available.444https://github.com/a-rebmann/llms4pm

Dataset Portions. We split all datasets based on the process models from which
the samples originate using 70% of instances for training, 20% for validation,
and 10% for final performance evaluation. For these splits, we ensure that no
activity sequence of a model in the training sets appears in any of the
process models in validation and test portions. This ensures that no knowledge
of the process behavior is leaked from training to the validation and test
sets, allowing us to correctly assess the generalization abilities of the LMs.
Furthermore, we ensure a comparable complexity distribution of process models
across splits via stratified sampling based on the number of unique activities
in the models. Table III displays the sizes of all splits for all three of our
tasks.

TABLE III: Training, validation, and test split characteristics per task. Task | Total | Train | Validation | Test
---|---|---|---|---
T-SAD | 291,251 | 227,892 | 43,609 | 19,750
A-SAD | 316,308 | 229,402 | 56,154 | 30,752
S-NAP | 1,289,081 | 1,071,529 | 166,811 | 50,741

Large Language Models. We resort to two widely used open decoder-LLMs that
have demonstrated impressive performance on NLP benchmarks, namely: (1)
_Llama-3_ (_Llama_) LLM in its 8 billion parameter
version555https://huggingface.co/meta-llama/Meta-Llama-3-8B and (2)
_Mistral-2_ (_Mistral_) in its 7 billion parameter
version666https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2. We
evaluate both in few-shot ICL and fine-tuning setups.

Baselines. We use two baselines in our experiments: (1) We compare LLMs with
ICL (i.e., without task-specific fine-tuning) against a random classifier,
which assigns task’s classes to test instances with equal probability; (2) We
compare generatively fine-tuned LLMs against a discriminatively fine-tuned
encoder LM: specifically, we use _RobERTa_ [17] in its _large_
version,777https://huggingface.co/FacebookAI/roberta-large a strong and widely
used English-only bidirectional encoder LM.

Performance Metric. We measure performance using the macro F1-score, so that
classes equally contribute to the performance regardless of their size. Macro
F1 is the simple average of per-class F1 scores, with F1 of class $c$ being
the harmonic mean between $c$’s precision and recall.

In-Context Learning and Prompt Optimization. We experimented with several task
formulation prompts for each task and selected the optimal one based on
validation performance. We evaluated 6-shot, 10-shot, and 20-shot ICL for all
three tasks: for binary T-SAD and A-SAD tasks, we evenly balance positive and
negative instances; for S-NAP we sample one instance from randomly chosen 6,
10, or 20 process models, respectively. Different task description prompts led
to marginal performance differences. Somewhat surprisingly, prompts with fewer
shots produced better validation performance: we thus finally evaluate 6-shot
ICL on our test data.888See https://github.com/a-rebmann/llms4pm for the task
prompts we used.

Fine-Tuning. We fine-tune _Llama_ and _Mistral_ in batches of two instances
with gradient accumulation over 16 batches, resulting in an effective batch
size of 32. We fine-tune _RoBERTa_ also in batches of 32 instances. All models
are trained using the AdamW algorithm [24], with an initial learning rate of
1e-5. We fine-tune the LLMs for three epochs and _RoBERTa_ for ten epochs. We
run each combination of task and model three times using different random
seeds, corresponding to different random initialization of model parameters
and shuffling of training data in each run.

##  VII Results and Discussion

We report the performance of LLMs on the three tasks, followed by an in-depth
analysis of the models’ predictions and a discussion on training effort.

###  VII-A Main Results

Table IV shows the main results of our experiments per task for the random
baseline, the two LLMs (_Llama_ and _Mistral_) using the ICL approach, as well
as _RoBERTa_ and the LLMs using the fine-tuning approach (FT). We report mean
and ($\pm$) standard deviation variance over three runs for ICL and three
different random seeds for FT. We first discuss the much weaker ICL results
and then focus on fine-tuning performance.

TABLE IV: Main Results (macro $F_{1}$ scores). Approach |  | Task |
---|---|---|---
T-SAD | A-SAD | S-NAP
Random | 0.50 $\pm$ 0.000 | 0.50 $\pm$ 0.000 | 0.13 $\pm$ 0.000
ICL Mistral | 0.49 $\pm$ 0.022 | 0.44 $\pm$ 0.011 | 0.18 $\pm$ 0.018
ICL Llama | 0.51 $\pm$ 0.015 | 0.53 $\pm$ 0.021 | 0.32 $\pm$ 0.054
FT RoBERTa | 0.77 $\pm$ 0.006 | 0.85 $\pm$ 0.003 | 0.63 $\pm$ 0.048
FT Mistral |  0.79 $\pm$ 0.010 |  0.88 $\pm$ 0.002 | 0.68 $\pm$ 0.039
FT Llama |  0.79 $\pm$ 0.011 |  0.88 $\pm$ 0.000 |  0.69 $\pm$ 0.049

In-Context Learning Results. For ICL, we find that the performance of the LLMs
is at best marginally better (_Llama_) at worst (_Mistral_) slightly worse
than random performance for the two semantic anomaly detection tasks, T-SAD
and A-SAD. Specifically, _Llama_ achieves a macro $F_{1}$-score of 0.51 for
T-SAD and 0.53 for A-SAD, while _Mistral_ scores 0.44 resp. 0.49. These
results indicate that the LLMs have not effectively learned these tasks from
the few examples provided in the context. For the S-NAP task, ICL with LLMs
does outperform the random baseline, with _Llama_ exhibiting much stronger
performance (19-point gain over the random baseline) than _Mistral_ (only
5-point gain). The performance is nonetheless fairly poor in absolute terms
(mere 0.32 with Llama). These results suggest that these process mining tasks
drastically differ from the language processing tasks on which the LLM
instruction-tuning was carried out.

Fine-Tuning Results. Poor ICL performance shows that LLMs a priori know very
little about process semantics and thus need to be explicitly trained for our
tasks. The fine-tuned encoder-LM baseline, i.e., _RoBERTa_ , already achieves
drastically better performance than ICL with LLMs: for example, it obtains the
F1-score of 0.77 on the T-SAD task, which is an improvement of massive 26
points over the best ICL performance (0.51 by _Llama_). Fine-tuning the LLMs
yields even better performance, with _Llama_ and _Mistral_ achieving an
$F_{1}$-score of 0.79, a further 2-point improvement over _RoBERTa_ ’s
performance. The same trend holds for the other two tasks: on A-SAD, both
_Mistral_ and _Llama_ yield very strong performance $F_{1}$-scores of 0.88,
outperforming _RoBERTa_ by 3 points; on S-NAP, _Llama_ and _Mistral_ achieve
$F_{1}$-scores of 0.69 and 0.68, respectively (6- and 5-point respective gains
over _RoBERTa_).

These results show that decoder-based LLMs can effectively acquire the missing
process knowledge through explicit task-specific fine-tuning, yielding better
results than their (smaller) encoder-based counterparts such as _RoBERTa_. The
fine-tuned LLMs consistently outperform _RoBERTa_ on all three tasks, which
points to the benefits of much larger-scale pretraining to which they have
been comparatively exposed.

Task Comparison. The results also demonstrate considerable differences in
difficulty between the tasks. For A-SAD, the LLMs achieve an impressive
$F_{1}$-score of 0.88, while the maximum score for T-SAD is 0.79, and the best
model scores only 0.69 for S-NAP. This aligns with expectations. Solving T-SAD
requires the model to identify whether process behavior is valid within the
context of an entire trace, whereas A-SAD only necessitates assessing a single
behavioral relation. The S-NAP task is by far the most challenging of the
three and is simply unsolvable for many instances. For example, consider a
process that allows for the concurrent execution of activities. In such cases,
it is indeterminable—for both humans and automated approaches—which activity
occurs next in a trace based solely on a prefix, as there are multiple valid
options.

###  VII-B In-Depth Analysis

To determine if the models can handle certain types of processes better than
others, we conducted an in-depth analysis of the classification results
obtained from the fine-tuned language models. For brevity, we focus on T-SAD,
as its medium complexity best represents the three tasks.

We find that both the encoder and LLMs accurately detect anomalous traces
across a wide range of process domains. However, they appear to be
particularly effective at identifying anomalies for standard process types.
For example, in a claim-handling process, they correctly identify anomalies
such as $\langle$_Enter and verify claim_ , _Handle payment_ , _Assess claim_
$\rangle$, where the claim should be assessed before sending a payment. They
also correctly detect that the trace $\langle$_Confirm order_ , _Ship product_
, _Get shipment address_ , _Emit Invoice_ , _Receive Payment_ $\rangle$ is
anomalous given that the product is shipped before the address is determined
in this order-handling process.

For more specialized processes, we observe that LLMs often outperform RoBERTa.
For instance, _Llama_ correctly identifies the trace $\langle$_Arrival_ ,
_Treatment_ , _Triage_ , _Discharge_ , _Invoicing_ $\rangle$ of a hospital
process as anomalous, since _Triage_ should occur before _Treatment_. In
contrast, _RoBERTa_ fails to detect this anomaly. Conversely, _RoBERTa_
incorrectly flags the trace $\langle$_Disassemble system_ , _Refurbish
materials_ , _Clean and paint covers_ , _Mount materials_ , _Move to bay_ ,
_Calibrate_ , _Handover_ $\rangle$ of a refurbishing process as anomalous,
even though it is valid, whereas _Llama_ correctly classifies this trace as
valid.

Finally, there are instances where both the LLMs and _RoBERTa_ incorrectly
identify valid traces as anomalous. For example, both _Llama_ and _RoBERTa_
flag the trace $\langle$_Receive invoices of partners_ , _Handle payment of
customer_ , _Receive review_ , _Send payment to partners_ $\rangle$ as
anomalous, even though it is valid. According to the corresponding process
model in the corpus, a review can be received at any point during an execution
of the process, making this trace valid. However, this specificity might also
be challenging for a human to determine without further contextual
information.

###  VII-C Training Effort

Finally, we consider the effort required to train the language models on the
tasks. Since in-context learning does not require any training effort, as the
task-specific knowledge is provided at inference time, we focus on the
training effort of fine-tuning an LLM (_Llama_) versus an encoder (_RoBERTa_).

Table V shows the run times for fine-tuning the language models for the
different tasks per epoch, i.e., pass over all training samples.999Note that
the models were trained on the same type of GPU. As shown, the LLM requires
considerably more time for training than the encoder baseline. In particular,
training _Llama_ on the tasks takes up to 25 times longer than training
_RoBERTa_ per epoch on the same data. For instance, while fine-tuning _Llama_
for A-SAD takes 15 hours, _RoBERTa_ requires only around 40 minutes per epoch.
This difference can be attributed to the huge number of parameters that need
to be updated for the LLM during fine-tuning, even when using parameter-
efficient fine-tuning. However, it is worth stressing that the LLM requires
considerably fewer epochs to converge in terms of validation loss across
tasks. This indicates that it not only learns the tasks better (as shown in
the previous subsections), but also with fewer passes over the training data.

TABLE V: Average run times for fine-tuning (per epoch). Approach | Task
---|---
T-SAD | A-SAD | S-NAP
FT RoBERTa | 0.5h | 0.6h | 1.3h
FT Llama | 11.1h | 15.0h | 23.0h

##  VIII Related Work

Neural language models have been used for various process analysis tasks,
e.g., for annotating event logs with semantic information [25], detecting
anomalies [6], and constructing event logs based on textual records of process
steps [26].

With the success of LLMs, several studies have explored their efficacy in
handling process analysis tasks [3]. These tasks include transforming textual
process descriptions into formal process models [27, 28], generating textual
descriptions from process data such as models and event logs [1], and
identifying potential bottlenecks and undesired process behavior [1]. However,
much of the existing research relies on closed-source GPT models or
proprietary software like _ChatGPT_ , limiting structured and reproducible
evaluations [3].

The lack of proper evaluations of LLMs for process mining tasks has recently
been discussed in the process mining community [3, 29]. As a response, Berti
et al. proposed a benchmark101010https://github.com/fit-alessandro-berti/pm-
llm-benchmark for process mining analysis questions. It consists of 52 prompts
that are used to query various LLMs, whose answers are then rated by GPT-4o.
Although the benchmark provides interesting insights, using an LLM to rate the
results can yield biased outcomes. Such bias arises, e.g., from the tendency
of LLMs to recognize and favor their own output [30].

In contrast, we define process mining tasks that benefit from an understanding
of process behavior and evaluate LLMs using extensive task-specific
benchmarking datasets in both, in-context learning and fine-tuning settings.
Furthermore, our datasets provide gold standards, which allows for using
established evaluation measures for classification tasks, eliminating the need
for a proxy LLM to assess output quality.

##  IX Conclusion

In this paper, we investigated the capabilities of LLMs to solve semantics-
aware process mining tasks, i.e., tasks that benefit from an understanding of
process semantics. We defined three such tasks and provide an extensive
benchmarking dataset for each of them. Our evaluation experiments that use
them show that LLMs fail to solve these challenging process mining tasks out
of the box and in a few-shot in-context setting. However, our results
demonstrate that LLMs achieve accurate performance when fine-tuned for these
tasks, surpassing smaller, encoder-based language models.

In the future, we want to investigate the integration of state-of-the-art
process mining approaches with LLMs. Since we have shown that LLMs can solve
semantics-aware process mining tasks through encoded knowledge of process
semantics, integrating existing process mining approaches with LLMs may yield
performance improvements for classical process mining tasks they address. For
example, an existing next-activity prediction approach could be extended by an
LLM-based semantic check that rejects predictions that do not make sense,
thereby improving the overall prediction performance.

## Acknowledgment

We received support by the state of Baden-Württemberg through bwHPC.

Reproducibility. _Our training and evaluation scripts are available through
the project repository linked in Section VI. Our process behavior corpus and
benchmarking datasets are published separately_ [13].

## References

  * [1] A. Berti, D. Schuster, and W. M. van der Aalst, “Abstractions, scenarios, and prompt definitions for process mining with llms: A case study,” in _BPM_. Springer, 2023, pp. 427–439.
  * [2] U. Jessen, M. Sroka, and D. Fahland, “Chit-chat or deep talk: Prompt engineering for process mining,” _arXiv preprint arXiv:2307.09909_ , 2023\.
  * [3] B. Estrada-Torres, A. del Río-Ortega, and M. Resinas, “Mapping the landscape: Exploring large language model applications in business process management,” in _BPMDS_. Springer, 2024, pp. 22–31.
  * [4] H. van der Aa, A. Rebmann, and H. Leopold, “Natural language-based detection of semantic execution anomalies in event logs,” _Information Systems_ , vol. 102, p. 101824, 2021.
  * [5] D. A. Neu, J. Lahann, and P. Fettke, “A systematic literature review on state-of-the-art deep learning methods for process prediction,” _Artificial Intelligence Review_ , vol. 55, no. 2, pp. 801–827, 2022.
  * [6] J. Caspary, A. Rebmann, and H. van der Aa, “Does this make sense? machine learning-based detection of semantic anomalies in business processes,” in _BPM_. Springer, 2023, pp. 163–179.
  * [7] P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang, “Squad: 100,000+ questions for machine comprehension of text,” _arXiv preprint arXiv:1606.05250_ , 2016\.
  * [8] S. Narayan, S. B. Cohen, and M. Lapata, “Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization,” _arXiv preprint arXiv:1808.08745_ , 2018.
  * [9] W. M. P. van der Aalst and A. K. A. de Medeiros, “Process mining and security: Detecting anomalous process executions and checking process conformance,” _Electronic Notes in Theoretical Computer Science_ , vol. 121, pp. 3–21, 2005\.
  * [10] F. Bezerra and J. Wainer, “Algorithms for anomaly detection of traces in logs of process aware information systems,” _Information Systems_ , vol. 38, no. 1, pp. 33–44, 2013.
  * [11] J. Evermann, J.-R. Rehse, and P. Fettke, “Predicting process behaviour using deep learning,” _Dec. Support Systems_ , vol. 100, pp. 129–140, 2017.
  * [12] P. Pfeiffer, J. Lahann, and P. Fettke, “Multivariate business process representation learning utilizing gramian angular fields and convolutional neural networks,” in _BPM_. Springer, 2021, pp. 327–344.
  * [13] A. Rebmann, F. D. Schmidt, G. Glavaš, and H. van der Aa, “Process behavior corpus and benchmarking datasets,” May 2024. [Online]. Available: https://doi.org/10.5281/zenodo.11276246
  * [14] D. Sola, C. Warmuth, B. Schäfer, P. Badakhshan, J.-R. Rehse, and T. Kampik, “SAP Signavio academic models: A large process model dataset,” in _ICPM Workshops_. Springer, 2023, pp. 453–465.
  * [15] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [16] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” in _NAACL_. ACL, 2019, pp. 4171–4186.
  * [17] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” _arXiv preprint arXiv:1907.11692_ , 2019.
  * [18] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell _et al._ , “Language models are few-shot learners,” _Advances in neural information processing systems_ , vol. 33, pp. 1877–1901, 2020.
  * [19] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar _et al._ , “Llama: Open and efficient foundation language models,” _arXiv preprint arXiv:2302.13971_ , 2023.
  * [20] Y. Wang, S. Mishra, P. Alipoormolabashi, Y. Kordi, A. Mirzaei, A. Arunkumar, A. Ashok, A. S. Dhanasekaran, A. Naik, D. Stap _et al._ , “Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks,” in _EMNLP_ , 2022.
  * [21] S. Zhang, L. Dong, X. Li, S. Zhang, X. Sun, S. Wang, J. Li, R. Hu, T. Zhang, F. Wu _et al._ , “Instruction tuning for large language models: A survey,” _arXiv preprint arXiv:2308.10792_ , 2023.
  * [22] E. J. Hu, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen _et al._ , “Lora: Low-rank adaptation of large language models,” in _International Conference on Learning Representations_ , 2021.
  * [23] Q. Dong, L. Li, D. Dai, C. Zheng, Z. Wu, B. Chang, X. Sun, J. Xu, and Z. Sui, “A survey on in-context learning,” _arXiv preprint arXiv:2301.00234_ , 2022\.
  * [24] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in _International Conference on Learning Representations_ , 2018.
  * [25] A. Rebmann and H. van der Aa, “Enabling semantics-aware process mining through the automatic annotation of event logs,” _Inf. Syst._ , vol. 110, p. 102111, 2022.
  * [26] C. Kecht, A. Egger, W. Kratsch, and M. Röglinger, “Event log construction from customer service conversations using natural language inference,” in _ICPM_. IEEE, 2021, pp. 144–151.
  * [27] M. Grohs, L. Abb, N. Elsayed, and J.-R. Rehse, “Large language models can accomplish business process management tasks,” in _BPM_. Springer, 2023, pp. 453–465.
  * [28] H. Kourani, A. Berti, D. Schuster, and W. M. van der Aalst, “Process modeling with large language models,” _arXiv preprint arXiv:2403.07541_ , 2024.
  * [29] A. Berti, H. Kourani, H. Häfke, C.-Y. Li, and D. Schuster, “Evaluating large language models in process mining: Capabilities, benchmarks, and evaluation strategies,” in _BPMDS_. Springer, 2024, pp. 13–21.
  * [30] A. Panickssery, S. R. Bowman, and S. Feng, “Llm evaluators recognize and favor their own generations,” _arXiv preprint arXiv:2404.13076_ , 2024.


# A Dataset And Benchmark Of Underwater Object Detection for Robot Picking

###### Abstract

Underwater object detection for robot picking has attracted a lot of interest.
However, it is still an unsolved problem due to several challenges. We take
steps towards making it more realistic by addressing the following challenges.
Firstly, the currently available datasets basically lack the test set
annotations, causing researchers must compare their method with other _SOTAs_
on a self-divided test set (from the training set). Training other methods
lead to an increase in workload and different researchers divide different
datasets, resulting there is no unified benchmark to compare the performance
of different algorithms. Secondly, these datasets also have other
shortcomings, _e.g._ , too many similar images or incomplete labels. Towards
these challenges we introduce a dataset, Detecting Underwater Objects (DUO),
and a corresponding benchmark, based on the collection and re-annotation of
all relevant datasets. DUO contains a collection of diverse underwater images
with more rational annotations. The corresponding benchmark provides
indicators of both efficiency and accuracy of _SOTAs_ (under the MMDtection
framework) for academic research and industrial applications, where JETSON AGX
XAVIER is used to assess detector speed to simulate the robot-embedded
environment.

Index Terms—  Underwater object detection, robot picking, dataset, benchmark

##  1 Introduction

Underwater robot picking is to use the robot to automatically capture sea
creatures like holothurian, echinus, scallop, or starfish in an open-sea farm
where underwater object detection is the key technology for locating
creatures. Until now, the datasets used in this community are released by the
Underwater Robot Professional Contest (URPC111Underwater Robot Professional
Contest: http://en.cnurpc.org.) beginning from 2017, in which URPC2017 and
URPC2018 are most often used for research. Unfortunately, as the information
listed in Table 1, URPC series datasets do not provide the annotation file of
the test set and cannot be downloaded after the contest. Therefore,
researchers [1, 2] first have to divide the training data into two subsets,
including a new subset of training data and a new subset of testing data, and
then train their proposed method and other _SOTA_ methods. On the one hand,
training other methods results in a significant increase in workload. On the
other hand, different researchers divide different datasets in different ways,

Table 1: Information about all the collected datasets. * denotes the test set’s annotations are not available. _3_ in Class means three types of creatures are labeled, _i.e.,_ holothurian, echinus, and scallop. _4_ means four types of creatures are labeled (starfish added). Retention represents the proportion of images that retain after similar images have been removed. Dataset | Train | Test | Class | Retention | Year
---|---|---|---|---|---
URPC2017 | 17,655 | 985* | 3 | 15% | 2017
URPC2018 | 2,901 | 800* | 4 | 99% | 2018
URPC2019 | 4,757 | 1,029* | 4 | 86% | 2019
URPC2020ZJ | 5,543 | 2,000* | 4 | 82% | 2020
URPC2020DL | 6,575 | 2,400* | 4 | 80% | 2020
UDD | 1,827 | 400 | 3 | 84% | 2020
Fig. 1: Examples in DUO, which show a variety of scenarios in underwater
environments.

causing there is no unified benchmark to compare the performance of different
algorithms. In terms of the content of the dataset images, there are a large
number of similar or duplicate images in the URPC datasets. URPC2017 only
retains 15% images after removing similar images compared to other datasets.
Thus the detector trained on URPC2017 is easy to overfit and cannot reflect
the real performance. For other URPC datasets, the latter also includes images
from the former, _e.g._ , URPC2019 adds 2,000 new images compared to URPC2018;
compared with URPC2019, URPC2020ZJ adds 800 new images. The URPC2020DL adds
1,000 new images compared to the URPC2020ZJ. It is worth mentioning that the
annotation of all datasets is incomplete; some datasets lack the starfish
labels and it is easy to find error or missing labels. [3] pointed out that
although the CNN model has a strong fitting ability for any dataset, the
existence of dirty data will significantly weaken its robustness. Therefore, a
reasonable dataset (containing a small number of similar images as well as an
accurate annotation) and a corresponding recognized benchmark are urgently
needed to promote community development.

To address these issues, we introduce a dataset called Detecting Underwater
Objects (DUO) by collecting and re-annotating all the available underwater
datasets. It contains 7,782 underwater images after deleting overly similar
images and has a more accurate annotation with four types of classes (_i.e.,_
holothurian, echinus, scallop, and starfish). Besides, based on the
MMDetection222MMDetection is an open source object detection toolbox based on
PyTorch. https://github.com/open-mmlab/mmdetection [4] framework, we also
provide a _SOTA_ detector benchmark containing efficiency and accuracy
indicators, providing a reference for both academic research and industrial
applications. It is worth noting that JETSON AGX XAVIER333JETSON AGX XAVIER is
an embedded development board produced by NVIDIA which could be deployed in an
underwater robot. Please refer https://developer.nvidia.com/embedded/jetson-
agx-xavier-developer-kit for more information. was used to assess all the
detectors in the efficiency test in order to simulate robot-embedded
environment. DUO will be released in https://github.com/chongweiliu soon.

In summary, the contributions of this paper can be listed as follows.

$\bullet$ By collecting and re-annotating all relevant datasets, we introduce
a dataset called DUO with more reasonable annotations as well as a variety of
underwater scenes.

$\bullet$ We provide a corresponding benchmark of _SOTA_ detectors on DUO
including efficiency and accuracy indicators which could be a reference for
both academic research and industrial applications.

##  2 Background

In the year of 2017, underwater object detection for open-sea farming is first
proposed in the target recognition track of Underwater Robot Picking Contest
2017444From 2020, the name has been changed into Underwater Robot Professional
Contest which is also short for URPC. (URPC2017) which aims to promote the
development of theory, technology, and industry of the underwater agile robot
and fill the blank of the grabbing task of the underwater agile robot. The
competition sets up a target recognition track, a fixed-point grasping track,
and an autonomous grasping track. The target recognition track concentrates on
finding the high accuracy and efficiency algorithm which could be used in an
underwater robot for automatically grasping.

The datasets we used to generate the DUO are listed below. The detailed
information has been shown in Table 1.

URPC2017: It contains 17,655 images for training and 985 images for testing
and the resolution of all the images is 720$\times$405\. All the images are
taken from 6 videos at an interval of 10 frames. However, all the videos were
filmed in an artificial simulated environment and pictures from the same video
look almost identical.

URPC2018: It contains 2,901 images for training and 800 images for testing and
the resolutions of the images are 586$\times$480, 704$\times$576,
720$\times$405, and 1,920$\times$1,080. The test set’s annotations are not
available. Besides, some images were also collected from an artificial
underwater environment.

URPC2019: It contains 4,757 images for training and 1029 images for testing
and the highest resolution of the images is 3,840$\times$2,160 captured by a
GOPro camera. The test set’s annotations are also not available and it
contains images from the former contests.

URPC2020ZJ: From 2020, the URPC will be held twice a year. It was held first
in Zhanjiang, China, in April and then in Dalian, China, in August. URPC2020ZJ
means the dataset released in the first URPC2020 and URPC2020DL means the
dataset released in the second URPC2020. This dataset contains 5,543 images
for training and 2,000 images for testing and the highest resolution of the
images is 3,840$\times$2,160. The test set’s annotations are also not
available.

URPC2020DL: This dataset contains 6,575 images for training and 2,400 images
for testing and the highest resolution of the images is 3,840$\times$2,160.
The test set’s annotations are also not available.

UDD [5]: This dataset contains 1,827 images for training and 400 images for
testing and the highest resolution of the images is 3,840$\times$2,160. All
the images are captured by a diver and a robot in a real open-sea farm.

Fig. 2: The proportion distribution of the objects in DUO.

Fig. 3: (a) The distribution of instance sizes for DUO; (b) The number of
categories per image.

##  3 Proposed Dataset

###  3.1 Image Deduplicating

As we explained in Section 1, there are a large number of similar or repeated
images in the series of URPC datasets. Therefore, it is important to delete
duplicate or overly similar images and keep a variety of underwater scenarios
when we merge these datasets together. Here we employ the Perceptual Hash
algorithm (PHash) to remove those images. PHash has the special property that
the hash value is dependent on the image content, and it remains approximately
the same if the content is not significantly modified. Thus we can easily
distinguish different scenarios and delete duplicate images within one
scenario.

After deduplicating, we obtain 7,782 images (6,671 images for training; 1,111
for testing). The retention rate of the new dataset is 95%, which means that
there are only a few similar images in the new dataset. Figure 1 shows that
our dataset also retains various underwater scenes.

###  3.2 Image Re-annotation

Due to the small size of objects and the blur underwater environment, there
are always missing or wrong labels in the existing annotation files. In
addition, some test sets’ annotation files are not available and some datasets
do not have the starfish annotation. In order to address these issues, we
follow the next process which combines a CNN model and manual annotation to
re-annotate these images. Specifically, we first train a detector (_i.e.,_ GFL
[6]) with the originally labeled images. After that, the trained detector
predicts all the 7,782 images. We treat the prediction as the groundtruth and
use it to train the GFL again. We get the final GFL prediction called the
coarse annotation. Next, we use manual correction to get the final annotation
called the fine annotation. Notably, we adopt the COCO [7] annotation form as
the final format.

###  3.3 Dataset Statistics

The proportion of classes: The total number of objects is 74,515. Holothurian,
echinus, scallop, and starfish are 7,887, 50,156, 1,924, and 14,548,
respectively. Figure 2 shows the proportion of each creatures where echinus
accounts for 67.3% of the total. The whole data distribution shows an obvious
long-tail distribution because the different economic benefits of different
seafoods determine the different breed quantities.

The distribution of instance sizes: Figure 3(a) shows an instance size
distribution of DUO. _Percent of image size_ represents the ratio of object
area to image area, and _Percent of instance_ represents the ratio of the
corresponding number of objects to the total number of objects. Because of
these small creatures and high-resolution images, the vast majority of objects
occupy 0.3% to 1.5% of the image area.

The instance number per image: Figure 3(b) illustrates the number of
categories per image for DUO. _Number of instances_ represents the number of
objects one image has, and _Percentage of images_ represents the ratio of the
corresponding number of images to the total number of images. Most images
contain between 5 and 15 instances, with an average of 9.57 instances per
image.

Summary: In general, smaller objects are harder to detect. For PASCAL VOC [8]
or COCO [7], roughly 50% of all objects occupy no more than 10% of the image
itself, and others evenly occupy from 10% to 100%. In the aspect of instances
number per image, COCO contains 7.7 instances per image and VOC contains 3. In
comparison, DUO has 9.57 instances per image and most instances less than 1.5%
of the image size. Therefore, DUO contains almost exclusively massive small
instances and has the long-tail distribution at the same time, which means it
is promising to design a detector to deal with massive small objects and stay
high efficiency at the same time for underwater robot picking.

##  4 Benchmark

Because the aim of underwater object detection for robot picking is to find
the high accuracy and efficiency algorithm, we consider both the accuracy and
efficiency evaluations in the benchmark as shown in Table 2.

###  4.1 Evaluation Metrics

Here we adopt the standard COCO metrics (mean average precision, _i.e.,_ mAP)
for the accuracy evaluation and also provide the mAP of each class due to the
long-tail distribution.

AP – mAP at IoU=0.50:0.05:0.95.

AP50 – mAP at IoU=0.50.

AP75 – mAP at IoU=0.75.

APS – AP for small objects of area smaller than 322.

APM – AP for objects of area between 322 and 962.

APL – AP for large objects of area bigger than 962.

APHo – AP in holothurian.

APEc – AP in echinus.

APSc – AP in scallop.

APSt – AP in starfish.

For the efficiency evaluation, we provide three metrics:

Param. – The parameters of a detector.

FLOPs – Floating-point operations per second.

FPS – Frames per second.

Notably, FLOPs is calculated under the 512$\times$512 input image size and FPS
is tested on a JETSON AGX XAVIER under MODE$\\_$30W$\\_$ALL.

###  4.2 Standard Training Configuration

We follow a widely used open-source toolbox, _i.e.,_ MMDetection (V2.5.0) to
produce up our benchmark. During the training, the standard configurations are
as follows:

$\bullet$ We initialize the backbone models (_e.g.,_ ResNet50) with pre-
trained parameters on ImageNet [9].

$\bullet$ We resize each image into 512 $\times$ 512 pixels both in training
and testing. Each image is flipped horizontally with 0.5 probability during
training.

$\bullet$ We normalize RGB channels by subtracting 123.675, 116.28, 103.53 and
dividing by 58.395, 57.12, 57.375, respectively.

$\bullet$ SGD method is adopted to optimize the model. The initial learning
rate is set to be 0.005 in a single GTX 1080Ti with batchsize 4 and is
decreased by 0.1 at the 8th and 11th epoch, respectively. WarmUp [10] is also
employed in the first 500 iterations. Totally there are 12 training epochs.

$\bullet$ Testing time augmentation (_i.e.,_ flipping test or multi-scale
testing) is not employed.

###  4.3 Benchmark Analysis

Table 2 shows the benchmark for the _SOTA_ methods. Multi- and one- stage
detectors with three kinds of backbones (_i.e.,_ ResNet18, 50, 101) give a
comprehensive assessment on DUO. We also deploy all the methods to AGX to
assess efficiency.

In general, the multi-stage (Cascade R-CNN) detectors have high accuracy and
low efficiency, while the one-stage (RetinaNet) detectors have low accuracy
and high efficiency. However, due to recent studies [11] on the allocation of
more reasonable positive and negative samples in training, one-stage detectors
(ATSS or GFL) can achieve both high accuracy and high efficiency.

Table 2: Benchmark of _SOTA_ detectors (single-model and single-scale results) on DUO. FPS is measured on the same machine with a JETSON AGX XAVIER under the same MMDetection framework, using a batch size of 1 whenever possible. R: ResNet. Method | Backbone | Param. | FLOPs | FPS | AP | AP50 | AP75 | APS | APM | APL | APHo | APEc | APSc | APSt
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
_multi-stage:_ |  |  |  |  |  |  |  |  |  |  |  |  |  |
Faster R-CNN [12] | R-18 | 28.14M | 49.75G | 5.7 | 50.1 | 72.6 | 57.8 | 42.9 | 51.9 | 48.7 | 49.1 | 60.1 | 31.6 | 59.7
R-50 | 41.14M | 63.26G | 4.7 | 54.8 | 75.9 | 63.1 | 53.0 | 56.2 | 53.8 | 55.5 | 62.4 | 38.7 | 62.5
R-101 | 60.13M | 82.74G | 3.7 | 53.8 | 75.4 | 61.6 | 39.0 | 55.2 | 52.8 | 54.3 | 62.0 | 38.5 | 60.4
Cascade R-CNN [13] | R-18 | 55.93M | 77.54G | 3.4 | 52.7 | 73.4 | 60.3 | 49.0 | 54.7 | 50.9 | 51.4 | 62.3 | 34.9 | 62.3
R-50 | 68.94M | 91.06G | 3.0 | 55.6 | 75.5 | 63.8 | 44.9 | 57.4 | 54.4 | 56.8 | 63.6 | 38.7 | 63.5
R-101 | 87.93M | 110.53G | 2.6 | 56.0 | 76.1 | 63.6 | 51.2 | 57.5 | 54.7 | 56.2 | 63.9 | 41.3 | 62.6
Grid R-CNN [14] | R-18 | 51.24M | 163.15G | 3.9 | 51.9 | 72.1 | 59.2 | 40.4 | 54.2 | 50.1 | 50.7 | 61.8 | 33.3 | 61.9
R-50 | 64.24M | 176.67G | 3.4 | 55.9 | 75.8 | 64.3 | 40.9 | 57.5 | 54.8 | 56.7 | 62.9 | 39.5 | 64.4
R-101 | 83.24M | 196.14G | 2.8 | 55.6 | 75.6 | 62.9 | 45.6 | 57.1 | 54.5 | 55.5 | 62.9 | 41.0 | 62.9
RepPoints [15] | R-18 | 20.11M | 35.60G | 5.6 | 51.7 | 76.9 | 57.8 | 43.8 | 54.0 | 49.7 | 50.8 | 63.3 | 33.6 | 59.2
R-50 | 36.60M | 48.54G | 4.8 | 56.0 | 80.2 | 63.1 | 40.8 | 58.5 | 53.7 | 56.7 | 65.7 | 39.3 | 62.3
R-101 | 55.60M | 68.02G | 3.8 | 55.4 | 79.0 | 62.6 | 42.2 | 57.3 | 53.9 | 56.0 | 65.8 | 39.0 | 60.9
_one-stage:_ |  |  |  |  |  |  |  |  |  |  |  |  |  |
RetinaNet [16] | R-18 | 19.68M | 39.68G | 7.1 | 44.7 | 66.3 | 50.7 | 29.3 | 47.6 | 42.5 | 46.9 | 54.2 | 23.9 | 53.8
R-50 | 36.17M | 52.62G | 5.9 | 49.3 | 70.3 | 55.4 | 36.5 | 51.9 | 47.6 | 54.4 | 56.6 | 27.8 | 58.3
R-101 | 55.16M | 72.10G | 4.5 | 50.4 | 71.7 | 57.3 | 34.6 | 52.8 | 49.0 | 54.6 | 57.0 | 33.7 | 56.3
FreeAnchor [17] | R-18 | 19.68M | 39.68G | 6.8 | 49.0 | 71.9 | 55.3 | 38.6 | 51.7 | 46.7 | 47.2 | 62.8 | 28.6 | 57.6
R-50 | 36.17M | 52.62G | 5.8 | 54.4 | 76.6 | 62.5 | 38.1 | 55.7 | 53.4 | 55.3 | 65.2 | 35.3 | 61.8
R-101 | 55.16M | 72.10G | 4.4 | 54.6 | 76.9 | 62.9 | 36.5 | 56.5 | 52.9 | 54.0 | 65.1 | 38.4 | 60.7
FoveaBox [18] | R-18 | 21.20M | 44.75G | 6.7 | 51.6 | 74.9 | 57.4 | 40.0 | 53.6 | 49.8 | 51.0 | 61.9 | 34.6 | 59.1
R-50 | 37.69M | 57.69G | 5.5 | 55.3 | 77.8 | 62.3 | 44.7 | 57.4 | 53.4 | 57.9 | 64.2 | 36.4 | 62.8
R-101 | 56.68M | 77.16G | 4.2 | 54.7 | 77.3 | 62.3 | 37.7 | 57.1 | 52.4 | 55.3 | 63.6 | 38.9 | 60.8
PAA [19] | R-18 | 18.94M | 38.84G | 3.0 | 52.6 | 75.3 | 58.8 | 41.3 | 55.1 | 50.2 | 49.9 | 64.6 | 35.6 | 60.5
R-50 | 31.89M | 51.55G | 2.9 | 56.8 | 79.0 | 63.8 | 38.9 | 58.9 | 54.9 | 56.5 | 66.9 | 39.9 | 64.0
R-101 | 50.89M | 71.03G | 2.4 | 56.5 | 78.5 | 63.7 | 40.9 | 58.7 | 54.5 | 55.8 | 66.5 | 42.0 | 61.6
FSAF [20] | R-18 | 19.53M | 38.88G | 7.4 | 49.6 | 74.3 | 55.1 | 43.4 | 51.8 | 47.5 | 45.5 | 63.5 | 30.3 | 58.9
R-50 | 36.02M | 51.82G | 6.0 | 54.9 | 79.3 | 62.1 | 46.2 | 56.7 | 53.3 | 53.7 | 66.4 | 36.8 | 62.5
R-101 | 55.01M | 55.01G | 4.5 | 54.6 | 78.7 | 61.9 | 46.0 | 57.1 | 52.2 | 53.0 | 66.3 | 38.2 | 61.1
FCOS [21] | R-18 | 18.94M | 38.84G | 6.5 | 48.4 | 72.8 | 53.7 | 30.7 | 50.9 | 46.3 | 46.5 | 61.5 | 29.1 | 56.6
R-50 | 31.84M | 50.34G | 5.4 | 53.0 | 77.1 | 59.9 | 39.7 | 55.6 | 50.5 | 52.3 | 64.5 | 35.2 | 60.0
R-101 | 50.78M | 69.81G | 4.2 | 53.2 | 77.3 | 60.1 | 43.4 | 55.4 | 51.2 | 51.7 | 64.1 | 38.5 | 58.5
ATSS [11] | R-18 | 18.94M | 38.84G | 6.0 | 54.0 | 76.5 | 60.9 | 44.1 | 56.6 | 51.4 | 52.6 | 65.5 | 35.8 | 61.9
R-50 | 31.89M | 51.55G | 5.2 | 58.2 | 80.1 | 66.5 | 43.9 | 60.6 | 55.9 | 58.6 | 67.6 | 41.8 | 64.6
R-101 | 50.89M | 71.03G | 3.8 | 57.6 | 79.4 | 65.3 | 46.5 | 60.3 | 55.0 | 57.7 | 67.2 | 42.6 | 62.9
GFL [6] | R-18 | 19.09M | 39.63G | 6.3 | 54.4 | 75.5 | 61.9 | 35.0 | 57.1 | 51.8 | 51.8 | 66.9 | 36.5 | 62.5
R-50 | 32.04M | 52.35G | 5.5 | 58.6 | 79.3 | 66.7 | 46.5 | 61.6 | 55.6 | 58.6 | 69.1 | 41.3 | 65.3
R-101 | 51.03M | 71.82G | 4.1 | 58.3 | 79.3 | 65.5 | 45.1 | 60.5 | 56.3 | 57.0 | 69.1 | 43.0 | 64.0

Therefore, in terms of accuracy, the accuracy difference between the multi-
and the one- stage methods in AP is not obvious, and the APS of different
methods is always the lowest among the three size AP. For class AP, APSc lags
significantly behind the other three classes because it has the smallest
number of instances. In terms of efficiency, large parameters and FLOPs result
in low FPS on AGX, with a maximum FPS of 7.4, which is hardly deployable on
underwater robot. Finally, we also found that ResNet101 was not significantly
improved over ResNet50, which means that a very deep network may not be useful
for detecting small creatures in underwater scenarios.

Consequently, the design of high accuracy and high efficiency detector is
still the main direction in this field and there is still large space to
improve the performance. In order to achieve this goal, a shallow backbone
with strong multi-scale feature fusion ability can be proposed to extract the
discriminant features of small scale aquatic organisms; a specially designed
training strategy may overcome the DUO’s long-tail distribution, such as a
more reasonable positive/negative label sampling mechanism or a class-balanced
image allocation strategy within a training batch.

##  5 Conclusion

In this paper, we introduce a dataset (DUO) and a corresponding benchmark to
fill in the gaps in the community. DUO contains a variety of underwater scenes
and more reasonable annotations. Benchmark includes efficiency and accuracy
indicators to conduct a comprehensive evaluation of the _SOTA_ decoders. The
two contributions could serve as a reference for academic research and
industrial applications, as well as promote community development.

## References

  * [1] Long Chen, Zhihua Liu, Lei Tong, Zheheng Jiang, Shengke Wang, Junyu Dong, and Huiyu Zhou,  “Underwater object detection using Invert Multi-Class Adaboost with deep learning,”  arXiv, p. 2005.11552, 2020.
  * [2] Wei-Hong Lin, Jia-Xing Zhong, Shan Liu, Thomas Li, and Ge Li,  “ROIMIX: proposal-fusion among multiple images for underwater object detection,”  in ICASSP. 2020, pp. 2588–2592, IEEE.
  * [3] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals,  “Understanding deep learning requires rethinking generalization,”  in ICLR. 2017, OpenReview.net.
  * [4] Kai Chen and Jiaqi Wang et al,  “Mmdetection: Open mmlab detection toolbox and benchmark,”  arXiv, p. 1906.07155, 2019.
  * [5] Zhihui Wang, Chongwei Liu, Shijie Wang, Tao Tang, Yulong Tao, Caifei Yang, Haojie Li, Xing Liu, and Xin Fan,  “UDD: An Underwater Open-sea Farm Object Detection Dataset for Underwater Robot Picking,”  arXiv e-prints, p. arXiv:2003.01446, Mar. 2020.
  * [6] Xiang Li, Wenhai Wang, Lijun Wu, Shuo Chen, Xiaolin Hu, Jun Li, Jinhui Tang, and Jian Yang,  “Generalized focal loss: Learning qualified and distributed bounding boxes for dense object detection,”  arXiv, 2020.
  * [7] Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C Lawrence Zitnick, and Piotr Dollár,  “Microsoft COCO: Common Objects in Context,”  arXiv.org, May 2014.
  * [8] Mark Everingham, Andrew Zisserman, Christopher K. I. Williams, Luc Van Gool, Moray Allan, Christopher M. Bishop, Olivier Chapelle, Navneet Dalal, Thomas Deselaers, and Gyuri Dorkó,  “The 2005 pascal visual object classes challenge,”  Lecture Notes in Computer Science, vol. 111, no. 1, pp. 98–136, 2007.
  * [9] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li,  “Imagenet: A large-scale hierarchical image database,”  in CVPR, 2009.
  * [10] Hao Luo, Youzhi Gu, Xingyu Liao, Shenqi Lai, and Wei Jiang,  “Bag of Tricks and A Strong Baseline for Deep Person Re-identification,”  arXiv, Mar. 2019.
  * [11] Shifeng Zhang, Cheng Chi, Yongqiang Yao, Zhen Lei, and Stan Z. Li,  “Bridging the gap between anchor-based and anchor-free detection via adaptive training sample selection,”  in CVPR. 2020, pp. 9756–9765, IEEE.
  * [12] Shaoqing Ren, Kaiming He, Ross Girshick, and Sun Jian,  “Faster r-cnn: Towards real-time object detection with region proposal networks,”  in NIPS, 2015.
  * [13] Zhaowei Cai and Nuno Vasconcelos,  “Cascade r-cnn: High quality object detection and instance segmentation,”  T-PAMI, p. 1–1, 2019.
  * [14] Xin Lu, Buyu Li, Yuxin Yue, Quanquan Li, and Junjie Yan,  “Grid r-cnn,”  in CVPR, 2019.
  * [15] Ze Yang, Shaohui Liu, Han Hu, Liwei Wang, and Stephen Lin,  “Reppoints: Point set representation for object detection,”  in ICCV, 2019.
  * [16] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár,  “Focal loss for dense object detection,”  T-PAMI, vol. PP, no. 99, pp. 2999–3007, 2017.
  * [17] Xiaosong Zhang, Fang Wan, Chang Liu, Rongrong Ji, and Qixiang Ye,  “FreeAnchor: Learning to Match Anchors for Visual Object Detection,”  arXiv, 2019.
  * [18] Tao Kong, Fuchun Sun, Huaping Liu, Yuning Jiang, Lei Li, and Jianbo Shi,  “Foveabox: Beyound anchor-based object detection,”  IEEE TIP, 2020.
  * [19] Kang Kim and Hee Seok Lee,  “Probabilistic Anchor Assignment with IoU Prediction for Object Detection,”  ECCV, 2020.
  * [20] Chenchen Zhu, Yihui He, and Marios Savvides,  “Feature selective anchor-free module for single-shot object detection,”  in CVPR, 2019, pp. 840–849.
  * [21] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He,  “FCOS: fully convolutional one-stage object detection,”  CoRR, vol. abs/1904.01355, 2019.