# From Text Segmentation to Smart Chaptering:
A Novel Benchmark for Structuring Video Transcriptions

Fabian Retkowski1, Alexander Waibel1,2
1Karlsruhe Institute of Technology, Karlsruhe, Germany
2Carnegie Mellon University, Pittsburgh PA, USA
<EMAIL_ADDRESS>
###### Abstract

Text segmentation is a fundamental task in natural language processing, where
documents are split into contiguous sections. However, prior research in this
area has been constrained by limited datasets, which are either small in
scale, synthesized, or only contain well-structured documents. In this paper,
we address these limitations by introducing a novel benchmark YTSeg focusing
on spoken content that is inherently more unstructured and both topically and
structurally diverse. As part of this work, we introduce an efficient
hierarchical segmentation model MiniSeg, that outperforms state-of-the-art
baselines. Lastly, we expand the notion of text segmentation to a more
practical “smart chaptering” task that involves the segmentation of
unstructured content, the generation of meaningful segment titles, and a
potential real-time application of the models.

##  1 Introduction

Text segmentation, also occasionally referred to as document segmentation or
topic segmentation, is the task of delimiting the boundaries of topically (or
functionally) coherent segments of text, placing them in a hierarchical
structure, typically a linear one. Text segmentation has been shown to support
a number of applications and downstream tasks where long documents are
involved, such as information retrieval Prince and Labadié (2007); Shtekh et
al. (2018); Chivers et al. (2022) or text summarization Zechner and Waibel
(2000); Cho et al. (2022); Liu et al. (2022b); Retkowski (2023).

Despite its significance, the field currently lacks robust benchmarks, as it
is attested and evident in recent works Lukasik et al. (2020); Glavaš et al.
(2021). Most datasets like Choi Choi (2000) either suffer from their small
scale or are purely synthetic. In practice, Wiki-727K Koshorek et al. (2018)
is the only larger-scale available benchmark, consisting of more than 727,000
Wikipedia documents. However, Wikipedia documents may fall short of fully
representing the diversity and complexities of real-world text segmentation
challenges. These documents are well-structured, informative, and have a fixed
style (in accordance with Wikipedia’s Manual of Style). This uniformity may
not adequately reflect the unstructured and varied nature of text found in
other sources.

Structuring a document proves to be particularly valuable in two cases, both
of which frequently occur in spoken and conversational content: first, when
the content is inherently unstructured; and second, when the content, being
structured or semi-structured, lacks explicit or formal organization.
Consequently, text segmentation plays a crucial role in many recently rolled-
out AI-powered features in applications such as Discord, YouTube, Microsoft
Teams, and Zoom.

Based on this observation and the lack of available, large-scale benchmarks,
we have developed a novel benchmark centered around diverse spoken content. We
adopt a more holistic and practical approach than prior works, viewing text
segmentation as a valuable application that involves both the prediction of
segment boundaries as well as the generation of segment titles. Previous works
have not considered and evaluated the generation of section headings, which is
crucial for practical applications. We term this challenge smart chaptering to
describe the transformation of unstructured content into a high-level semantic
structure with meaningful headings, a critical process for improving document
comprehension and organization. This term reflects the additional capabilities
required for a system to offer practical utility. Correspondingly, we
introduce MiniSeg, a small-scale and state-of-the-art hierarchical
segmentation model focused on efficiency, thus viable for use in practical
settings. Finally, in addition to traditional offline settings, we also
evaluate our approach in online scenarios where real-time processing is
crucial, further expanding its practical applicability.

In summary, the contributions of our paper are:

  * •

The introduction of a novel larger-scale text segmentation benchmark YTSeg
which addresses an important limitation of this research area, the lack of
robust benchmarks, and gives researchers a chance to evaluate their models on
a benchmark other than Wiki-727K. In addition, it is the first available
benchmark around speech text segmentation.

  * •

We introduce MiniSeg, an efficient, hierarchical, state-of-the-art text
segmentation model that demonstrates the effectiveness of a number of
incremental methodological improvements compared to previous models.

  * •

We extend the (offline) text segmentation task by online segmentation and
title generation. For these tasks, we provide a strong set of baselines on our
benchmark.

##  2 YTSeg Dataset

As part of this work, we introduce a new benchmark, YTSeg, to evaluate text
segmentation systems on less structured and more diverse content than previous
benchmarks aimed for. The dataset consists of 19,299 English YouTube videos
with their transcripts and chapters. An example of a YouTube video organized
into chapters can be found in the screenshot provided in Figure A1. We
processed the data to adapt it for the text segmentation task, aligning the
sentences in the transcription with video chapters. In addition, we release
YTSeg[Titles] for the training and evaluation of generative models predicting
the chapter titles. By including video and audio data, the benchmark also
paves the way for multi-modal approaches. The dataset, along with the
instructions and scripts, is available online and released under a CC BY-NC-SA
4.0 license111https://huggingface.co/datasets/retkowski/ytseg.

Channel Name | # Videos
---|---
YaleCourses | 1015
The List | 984
Mashed | 904
Bestie | 577
Google Cloud Tech | 551
Linus Tech Tips | 461
Unveiled | 425
Flipping Physics | 390
Looper | 385
GeologyHub | 364
(a) YouTube Channels by Number of Videos

Channel Name | Length[h]
---|---
YaleCourses | 907.05
CS50 | 276.21
Rich Roll | 179.16
Andrew Huberman | 178.92
The List | 168.27
Mashed | 159.24
Tech With Tim | 131.37
Linus Tech Tips | 120.14
SAS Users | 106.12
David Bombal | 93.09
(b) YouTube Channels by Content Length

Table 1: YouTube Channels by Number of Videos and Content Length

###  2.1 Collection

For the dataset, we utilize yt-dlp to collect videos, their transcripts, and
their chapters. The dataset collection is limited to videos with closed
captions and chapters provided by the content creator. In both cases, YouTube
exposes whether the information is automatically generated.222While for closed
captions, this information can be accessed directly via yt-dlp, for chapters,
it is derived from raw data returned by yt-dlp’s low-level APIs. We note that
only a small subset of YouTube videos fall under this category, which
motivates the following procedure.

In the first step, we define a wide variety of seed keywords as listed in
Section E. Their purpose is to surface a higher-quality and diverse set of
videos that are more likely to have manually provided closed captions and
chapters. Then, we utilize the YouTube search and various search filters
(e.g., to filter videos without closed captions or to surface more recent
content or long-form videos). Based on the video search results, we select
corresponding channels to be crawled after reviewing a sample of the channel’s
videos for the audio language and the quality of its closed captions and
chapters.

###  2.2 Preprocessing

Following previous benchmarks for the text segmentation task, our dataset aims
to provide segment boundaries on a sentence level. For this, we sentence
tokenize the closed captions using the pre-trained Punkt tokenizer available
in the NLTK library Bird et al. (2009). We annotate the sentences with
respective timestamps based on the closed captions. It is important to note
that the closed captions do not always respect sentence boundaries, which may
necessitate potential splitting and joining of sentences while linearly
interpolating timestamps based on their character length. Unlike purely
textual datasets, the segment boundaries might not necessarily agree with
sentence boundaries (i.e., YouTube chapters can start or end in the middle of
a sentence). Thus, sentences spanning two chapters are assigned to the chapter
with the greater time-based overlap. We also observed instances where
sentences remain unassigned when the first chapter starts later or the last
chapter ends earlier than the first, respectively, the last caption. To
address this issue, we add an additional “Intro” or “Outro” chapter in these
cases.

We exclude all videos for which inconsistent timestamps cannot be fixed333In a
number of instances, we observed certain inconsistencies in the provided
timestamps fixable by simple rules., final sanity checks are not passed, or
our procedure returns an error. These errors can stem from various reasons,
such as empty captions, transcripts without punctuation, or a malformed VTT
format. This affects $4.70\%$ of the collected videos.

Finally, the data is split into stratified partitions for training,
validation, and testing based on the channel identifier (see Table 0(a)). As
part of this process, channels with only a single video form a separate group.

###  2.3 Data Statistics

The dataset comprises 19,299 videos from 393 channels, amounting to 6,533
content hours. The topics are wide-ranging, covering domains such as science,
lifestyle, politics, health, economy, and technology. The videos are from
various types of content formats, such as podcasts, lectures, news, corporate
events & promotional content, and, more broadly, videos from individual
content creators. Table 1(a) and the analysis depicted in Figure 1 offer
insights into the dataset’s diversity, while Table 1(b) shows that the content
hours are dominated by the long-form formats such as podcasts and lectures.
The dataset’s structural diversity is also evident in its data statistics, as
depicted in Table 2.3. In contrast to Wiki-727K, our benchmark exhibits a
greater number of segments per document and a higher number of sentences per
segment while simultaneously showing a wider variation.

Figure 1: UMAP McInnes et al. (2018) plot of YTSeg video titles, embedded
using Instructor Su et al. (2023). Category labels are assigned through zero-
shot classification with LLaMA 2 Touvron et al. (2023).

| YTSeg | Wiki-727K |
---|---|---|---
Document Length [Sent.] | 196.2 $\pm$ 267.2 | 57.6 $\pm$ 46.9 |
Video Length [min.] | 20.3 $\pm$ 25.3 | – |
1-3 Segment Length [Sent.] | 21.5 $\pm$ 34.2 | 13.6 $\pm$ 20.3 |
Segment Duration [min.] | 2.49 $\pm$ 2.98 | – |
Segments per Document | 9.12 $\pm$ 5.42 | 3.48 $\pm$ 2.23 |
1-3 Title Length [Words] | 4.03 $\pm$ 2.75 | 2.01 $\pm$ 1.49 |
Concentration Index1 | 9.50% | 24.96% |

  * 1

with $n=20$

Table 2: Data Statistics for YTSeg and Wiki-727K

###  2.4 Chapter Titles

Based on the same data partitioning outlined in Section 2.2, we prepared
another view on the same dataset, providing 173,195 pairs of sections and
chapter titles. We refer to this dataset view as YTSeg[Titles]. As a result of
the same data partitioning, all section title pairs of a particular video will
be assigned to the same data partition. The resulting data split can be found
in Table 0(b). We removed every pair for which the section title exceeds 75
characters to exclude atypically and excessively lengthy titles444Some of
these lengthy titles tend to be complete sentences or summaries, deviating
from our understanding of a title. We also note the computational advantages
of excluding them. accounting for 1.5% of the total number of titles. Despite
this, the title length is meaningfully longer and more diverse than in titles
in the Wiki-727K dataset (see Table 2.3). We also point out that the titles in
the Wiki-727K dataset are highly concentrated. The 20 most frequent titles in
the dataset account for 24.96% of the overall dataset, with the title
“History” alone constituting 7.53% of it. On the contrary, for the YTSeg
dataset, this concentration is notably lower, as the top 20 titles
collectively represent just 9.50% of the dataset. These titles predominantly
consist of functional segments like “Introduction” or “Conclusion”.

##  3 Methodology

In the following, we present the models designed for the text segmentation and
title generation task, which are used in our experiments and applied to the
newly introduced benchmark. We also elaborate on the modifications we have
made to adapt them for online implementation.

Figure 2: The hierarchical architecture of the segmentation model consists of
a sentence encoder and a document encoder returning the binary segment
boundaries. Figure 3: Our offline document encoder is a typical transformer
encoder with $N$ transformer layers, each of which applies a full attention
mask. Consequently, the encoder can attend to the whole document. In contrast,
our online document encoder has $N-M$ layers with causal attention masks that
only allow attention to past context, while the initial $M$ layers have
attention masks with limited right-side context, that, over these $M$ layers,
accumulate to a defined future context size $c$.

###  3.1 Offline Segmentation

The model used in our experiments (see the architecture outlined in Figure 2)
closely resembles the previous work of Koshorek et al. (2018) and Lukasik et
al. (2020). Both proposed a hierarchically structured network consisting of a
sentence encoder and a document encoder. The sentence encoder processes each
token within a sentence to generate a corresponding sentence representation.
Following this, the document encoder performs a sequence labeling task, where
its aim is to predict whether each sentence serves as a segment boundary. The
network is trained in a supervised fashion using a binary classification
objective.

In the following, we highlight the methodological differences between our
MiniSeg model and the hierarchical BERT architecture outlined in Lukasik et
al. (2020).

  * •

We utilize a pre-trained sentence transformer based on MiniLM Reimers and
Gurevych (2019); Wang et al. (2020) for the sentence encoder (33M parameters).
This network has specifically been trained on paraphrase data in a siamese
network structure to produce meaningful sentence representations.

  * •

For the document encoder, we use a randomly initialized RoFormer encoder with
12 layers, 8 attention heads, and 384-dimensional embeddings (26M parameters).
This Transformer variant uses rotary positional embeddings (RoPE) introduced
by Su et al. (2021).

  * •

Motivated by the class imbalance, we opted to use a weighted binary cross-
entropy term. In a thoughtful adjustment, we assigned double the weight to
segment boundaries with $w=[1,2]$. A similar reweighting has been performed by
Ghosh et al. (2022).

  * •

Instead of using the [CLS] token, we apply mean pooling on the output
embeddings to create fixed-sized sentence representations, as it has been
shown to outperform other pooling strategies Reimers and Gurevych (2019).

  * •

While training, we randomly sample a subset of gradients for the sentence-
encoding sub-network in each backward pass. We set this gradient sampling rate
to $0.5$, meaning half of the documents are backpropagated through the
sentence encoder. In our experiments, this has been shown to reduce the memory
requirement while having a regularizing effect and improving the final
performance.

###  3.2 Online Segmentation

Our proposed online segmentation model mirrors the offline segmentation model
in its architecture (see Section 3.1). The major difference is that we limit
the future context that the model can process, for which we use a different
masking strategy that we refer to as progressive context accumulation.
Starting from a purely causal model, we replace a subset of causal attention
masks with masks that allow attention to a controlled amount of future
context. The corresponding architecture and idea are illustrated in Figure 3.

Specifically, in the early $M$ layers of the document encoder, we employ
causal attention masks with an offset that is the limited right-side context
$\alpha_{i}$ where $i\in[1,M]$ is the index of the layer. These masks provide
each layer with selective access to a portion of additional future context,
and their sizes sum up to our predefined target future context size, denoted
as $c$. In the later $N-M$ layers, we transition back to causal attention
masks to prevent any additional future context from leaking into the
predictions.

This approach introduces a structural hyperparameter $\alpha$ defining the
partitioning distribution of the total future context size $c$ to be allocated
to each of the first $M$ layers. We note the relation between the introduced
hyperparameters: $M=|\alpha|$ and $c=\sum{\alpha_{i}}$.

###  3.3 Title Generation

We fine-tune a BART-large Lewis et al. (2020) model on our YTSeg[Titles]
dataset for section title generation. BART is a transformer encoder-decoder
model pre-trained on a denoising task.

  * •

For online title generation, we limit the amount of text we provide from a
given section to the model for generating titles. The term input span further
refers to the number of starting sentences from a section. This approach
enables titles to be prematurely generated in an online setting while
maintaining a defined latency (in terms of sentences).

  * •

Conditional title generation: To incorporate the context of the document’s
structure, we prepend previous section titles. This way, the model is
conditioned on both the content of the current section and the preceding
context, allowing the generation of more relevant and coherent titles. We note
that this approach requires generating titles sequentially, which affects
offline title generation. In contrast, for online title generation, titles are
always generated sequentially.

##  4 Experiments

###  4.1 Segmentation

|  | P (↑) | R (↑) | F1 (↑) | P$k$ (↓) | B1 (↑)
---|---|---|---|---|---|---
Wiki-727K | Bi-LSTM2,4 | 69.30 $\pm$ 0.10 | 49.50 $\pm$ 0.20 | 57.70 $\pm$ 0.10 | – | –
CS BERT3,4 | 69.10 $\pm$ 0.10 | 63.20 $\pm$ 0.20 | 66.00 $\pm$ 0.10 | – | –
Hier. BERT3,4 | 69.80 $\pm$ 0.10 | 63.50 $\pm$ 0.10 | 66.50 $\pm$ 0.10 | – | –
MiniSeg (Ours) | 68.57 $\pm$ 0.13 | 70.76 $\pm$ 0.13 | 69.65 $\pm$ 0.09 | 17.57 $\pm$ 0.06 | 59.81 $\pm$ 0.12
YTSeg | MiniSeg | 45.44 $\pm$ 0.83 | 41.48 $\pm$ 0.85 | 43.37 $\pm$ 0.60 | 28.73 $\pm$ 0.39 | 35.74 $\pm$ 0.68
Wiki-727K $\rightarrow$ YTSeg | MiniSeg | 48.30 $\pm$ 0.84 | 43.56 $\pm$ 0.84 | 45.81 $\pm$ 0.60 | 27.13 $\pm$ 0.43 | 37.89 $\pm$ 0.70
2-7 | MiniSeg ($c=0$) | 43.69 $\pm$ 0.79 | 37.49 $\pm$ 0.76 | 40.35 $\pm$ 0.55 | 29.81 $\pm$ 0.38 | 33.11 $\pm$ 0.72
| MiniSeg ($c=1$) | 45.05 $\pm$ 0.82 | 40.05 $\pm$ 0.80 | 42.41 $\pm$ 0.58 | 28.70 $\pm$ 0.40 | 34.72 $\pm$ 0.79
| MiniSeg ($c=3$) | 46.02 $\pm$ 0.88 | 41.45 $\pm$ 0.77 | 43.61 $\pm$ 0.58 | 28.08 $\pm$ 0.43 | 36.13 $\pm$ 0.76
| MiniSeg ($c=5$) | 46.24 $\pm$ 0.75 | 42.23 $\pm$ 0.91 | 44.15 $\pm$ 0.60 | 27.91 $\pm$ 0.80 | 36.62 $\pm$ 0.78
| MiniSeg ($c=8$) | 46.92 $\pm$ 0.80 | 41.89 $\pm$ 0.79 | 44.26 $\pm$ 0.57 | 27.68 $\pm$ 0.42 | 36.81 $\pm$ 0.77
| MiniSeg ($c=10$) | 45.99 $\pm$ 0.81 | 41.31 $\pm$ 0.89 | 43.52 $\pm$ 0.61 | 27.97 $\pm$ 0.39 | 36.35 $\pm$ 0.74
| MiniSeg ($c=20$) | 46.46 $\pm$ 0.89 | 42.34 $\pm$ 0.83 | 44.30 $\pm$ 0.61 | 27.95 $\pm$ 0.41 | 37.07 $\pm$ 0.76

  *   * 1

Boundary Similarity Fournier (2013)

  * 2

Koshorek et al. (2018)

  * 3

Lukasik et al. (2020)

  * 4

Results as reported by Lukasik et al. (2020).

Table 3: Results of our text segmentation models and baselines on Wiki-727K
and YTSeg. Standard deviations are estimated by bootstrapping the test set 100
times, similar as in Lukasik et al. (2020).

|  | P (↑) | R (↑) | F1 (↑)
---|---|---|---|---
Zero-Shot
1-5 | Wiki-727K |  01.58 $\pm$ 0.89 | 15.85 $\pm$ 3.61 |  02.87 $\pm$ 1.47
| YTSeg | 12.55 $\pm$ 4.21 |  08.30 $\pm$ 2.78 |  09.99 $\pm$ 2.41
|  Wiki-727K $\rightarrow$ YTSeg |  06.09 $\pm$ 1.92 | 18.07 $\pm$ 3.36 |  09.11 $\pm$ 2.19
Fine-Tuned on QMSum
1-5 | No Pre-Training | 21.62 $\pm$ 4.54 | 15.61 $\pm$ 3.58 | 18.13 $\pm$ 2.90
| Wiki-727K | 23.45 $\pm$ 5.42 | 11.88 $\pm$ 2.75 | 15.77 $\pm$ 2.71
| YTSeg | 31.09 $\pm$ 5.35 | 16.92 $\pm$ 3.20 | 21.92 $\pm$ 2.99
|  Wiki-727K $\rightarrow$ YTSeg | 25.21 $\pm$ 4.71 | 15.82 $\pm$ 3.15 | 19.44 $\pm$ 2.76

Table 4: Text segmentation results of MiniSeg on the QMSum dataset, both zero-
shot and fine-tuned.

|  | P (↑) | R (↑) | F1 (↑) | P$k$ (↓) | B (↑)
---|---|---|---|---|---|---
| MiniSeg | 45.44 $\pm$ 0.83 | 41.48 $\pm$ 0.85 | 43.37 $\pm$ 0.60 | 28.73 $\pm$ 0.39 | 35.74 $\pm$ 0.68
1-7 | w/o WBCE | 48.76 $\pm$ 0.95 | 31.66 $\pm$ 0.77 | 38.39 $\pm$ 0.64 | 30.53 $\pm$ 0.36 | 30.42 $\pm$ 0.71
| w/o RoPE1 | 42.13 $\pm$ 0.70 | 42.05 $\pm$ 0.81 | 42.09 $\pm$ 0.54 | 30.59 $\pm$ 0.47 | 33.75 $\pm$ 0.69
| w/o Pre-Training2 | 38.73 $\pm$ 0.81 | 28.62 $\pm$ 0.61 | 32.92 $\pm$ 0.50 | 33.52 $\pm$ 0.36 | 25.45 $\pm$ 0.59
| with [CLS] pooling | 45.76 $\pm$ 0.80 | 41.39 $\pm$ 0.85 | 43.47 $\pm$ 0.59 | 28.87 $\pm$ 0.39 | 35.54 $\pm$ 0.68
| w/o gradient sampling | 43.41 $\pm$ 0.83 | 40.38 $\pm$ 0.79 | 41.84 $\pm$ 0.57 | 29.94 $\pm$ 0.41 | 34.10 $\pm$ 0.72

  * 1

A standard transformer with sinusoidal positional encodings is used as the
document encoder.

  * 2

The weights of MiniLM, the sentence transformer, are initialized randomly.

Table 5: Results of ablated versions of MiniSeg on the YTSeg dataset.

We perform the following experiments to evaluate our benchmark and our
segmentation model:

  * •

First, we train our introduced MiniSeg model on the established benchmark
Wiki-727K and compare it against the baselines of Koshorek et al. (2018) and
Lukasik et al. (2020). For a fair comparison, we use the same setup as in
Koshorek et al. (2018) by predicting top-level sections of the document and
using the original preprocessing scripts.

  * •

We establish a benchmark for the YTSeg dataset employing our MiniSeg model.
This involves training the model on the dataset to set the baseline
performance.

  * •

In addition, we experiment with a two-stage training process where we first do
a task adaptational pre-training of the model on the Wiki-727K dataset and
then fine-tune it on our YTSeg benchmark.

  * •

We test and fine-tune our model on QMSum Zhong et al. (2021) to evaluate
whether our dataset and model can improve the segmentation of even more
unstructured content such as meetings. This dataset provides 232 segmented
meetings.

  * •

Finally, we train online segmentation models with different future context
sizes $c$. The corresponding partitioning $\alpha$ for each setting can be
found in Table A3.

We evaluate our segmentation models using a combination of standard binary
classification metrics, such as precision, recall, and F1 score, as well as
metrics specifically tailored for text segmentation tasks, including P$k$ as
introduced in the work of Beeferman et al. (1999) and Boundary Similarity, as
discussed in Fournier (2013). The results of the experiments are presented in
Table 4.1 and 4.1.

#### MiniSeg.

The experiments presented in Table 4.1 demonstrate that MiniSeg outperforms
the baselines, namely Bi-LSTM, cross-segmenter BERT, and hierarchical BERT, on
the established Wiki-727K benchmark, even though it is equipped with only 59
million parameters. Given this parameter count, it is meaningfully more
efficient compared to state-of-the-art baselines such as hierarchical BERT
(220 million parameters) and cross-segmenter BERT (336 million parameters).
Several factors contribute to the observed performance as demonstrated in an
ablation study shown in Table 4.1, with the strength of the pre-trained
sentence encoder emerging as one of the most crucial contributors. It is worth
noting that BERT Devlin et al. (2019), as used by Lukasik et al. (2020), was
not trained to represent sentences in particular and that Liu et al. (2019)
have described BERT as “significantly undertrained”. Additionally, our
approach relies on a weighted cross-entropy loss function, allowing us to
balance precision and recall. Importantly, while our model exhibits lower
precision compared to the baselines and ablated version, it excels in terms of
recall. Smaller incremental gains can be attributed to RoPE and the sampling
of gradients. Lastly, no noticeable effect is observed when replacing [CLS]
pooling with mean pooling.

#### Task Adaptation.

We find that the task-adaptational pre-training with Wiki-727K improves the
result on the YTSeg benchmark (see Table 4.1). This outcome contrasts with the
findings of Ghosh et al. (2022), who reported that such a pre-training step
has a negative or negligible effect when applied to semi-structured content.
While the domain is different (YouTube videos versus chat conversations), we
emphasize that the dataset in Ghosh et al. (2022) is synthetically constructed
and, as such, is qualitatively different from Wiki-727K and YTSeg presumably
contributing to the varying effectiveness of the pre-training.

#### Meeting Segmentation.

Our experiments on QMSum displayed in Table 4.1 reveal that the pre-training
with YTSeg improves the final performance of models on QMSum. Even in zero-
shot conditions, the beneficial effect of YTSeg becomes apparent. This
underscores the domain proximity of video content and meetings, both of which
are less structured and are spoken in nature. However, we note that due to its
small size, QMSum is not a robust benchmark, and the effect of Wiki-727K
remains inconclusive.

#### Online Segmentation.

In the results (see Table 4.1), we observe noticeable jumps in performance
when increasing the future context size. However, diminishing returns set in
after about three to five sentences of future context, especially when
considering the latency trade-off in online segmentation. This strongly
suggests that local context is more important. In fact, even a model without
future context at all scores solidly (with the performance only increasing
from 40.35 to 45.81 for a model without any future context and one with global
context).

###  4.2 Title Generation

|  | R1 (↑) | R2 (↑) | RL (↑) | BS1 (↑)
---|---|---|---|---|---
No Context | BART | 36.42 $\pm$ 0.36 | 17.03 $\pm$ 0.28 | 36.19 $\pm$ 0.35 | -4.21 $\pm$ 0.02
2-6 | BART ($s=1$) | 25.02 $\pm$ 0.31 | 10.98 $\pm$ 0.23 | 24.87 $\pm$ 0.30 | -4.97 $\pm$ 0.02
| BART ($s=3$) | 31.40 $\pm$ 0.31 | 14.32 $\pm$ 0.26 | 31.21 $\pm$ 0.32 | -4.51 $\pm$ 0.02
| BART ($s=5$) | 33.64 $\pm$ 0.31 | 15.61 $\pm$ 0.29 | 33.42 $\pm$ 0.35 | -4.37 $\pm$ 0.02
| BART ($s=10$) | 34.83 $\pm$ 0.33 | 16.18 $\pm$ 0.27 | 34.60 $\pm$ 0.37 | -4.30 $\pm$ 0.02
Previous Titles | BART | 42.79 $\pm$ 0.34 | 22.07 $\pm$ 0.30 | 42.45 $\pm$ 0.31 | -3.87 $\pm$ 0.02
2-6 | BART ($s=1$) | 27.66 $\pm$ 0.35 | 11.87 $\pm$ 0.26 | 27.47 $\pm$ 0.29 | -4.83 $\pm$ 0.02
| BART ($s=3$) | 36.33 $\pm$ 0.37 | 17.74 $\pm$ 0.31 | 36.08 $\pm$ 0.30 | -4.25 $\pm$ 0.02
| BART ($s=5$) | 36.02 $\pm$ 0.32 | 19.68 $\pm$ 0.32 | 35.74 $\pm$ 0.29 | -4.09 $\pm$ 0.02
| BART ($s=10$) | 41.52 $\pm$ 0.33 | 21.24 $\pm$ 0.30 | 41.21 $\pm$ 0.30 | -3.94 $\pm$ 0.03

  * 1

BARTScore Yuan et al. (2021)

Table 6: Results of the title generation models on the YTSeg[Titles] dataset.

Configuring Cloud Operations on Google Cloud – Google Cloud Tech (xIaaGef1QvI)
Intro Intro Introduction & agenda Intro How to know what’s going on in the
cloud How the operations components play together Operational use cases
Exploring the products of interest for the operations team Google Cloud
Operations Suite Site Reliability Engineering Google SRE SRE Practices
Customer success story: Krikey Customer success story: Krikey Customer Story -
Krikey Wrap-up Wrap up Wrap up & additional resources Generated (No Context)
Generated (Previous Titles) Reference

(a) An exemplary output showing duplicate section titles.

8 Email Etiquette Tips - How to Write Better Emails at Work – Harvard Business
Review (1XctnF7C74s) Intro Intro Why bother with email etiquette? Step 1: Have
a call to action if appropriate 1\. Have a call to action when appropriate
Include CTA in subject line Stick with one email thread for the same topic 2\.
Stick with one email thread for the same topic One email thread per topic Tip
#3 3\. Explain why you added in or took out recipients Manage recipients Tip
#5: summarize the sender’s main points 4\. Include your main point first
followed by context Summarize in your reply Tip #2: Include the context 5\.
summarize the sender’s main points in your reply Start with the main point 6.
Hyperlink Whatever Possible 6\. Hyperlink whatever possible Hyperlink whenever
possible Change Default Setting to reply instead of reply all 7\. Change
default setting to reply instead of reply all Change default setting to
"Reply" (not "Reply all") Change Undo Send Option to 30 Seconds 8\. Change
undo send option to 30 seconds Change undo send options Outro Outro Outro
Generated (No Context) Generated (Previous Titles) Reference

(b) An exemplary output showing inconsistent numbering and formatting. Figure
4: A comparison of chapter titles generated by the fine-tuned BART models,
both without any context and with previously generated titles, and the
reference titles on select examples from YTSeg’s validation dataset. A lack of
context, as highlighted, leads to repetitive titles, coherence gaps, and
variations in writing styles.

We conducted a series of experiments to evaluate the performance of our title
generation model, summarized as follows:

  * •

In our initial experiment, we conducted a comparative analysis of fine-tuning
the BART model under two distinct conditions. The first involved training the
model to generate titles solely based on the current section text, devoid of
any contextual information. In the second setting, the model was trained with
the added context of previous titles.

  * •

For the online setting, we assessed different scenarios where the model
receives a limited number of starting sentences $s\in[1,3,5,10]$. We conducted
this evaluation for both the context-less scenario and the scenario where
previous titles were incorporated as context.

The generation of section titles can be considered an extreme form of
summarization. As such, we evaluate our models using established metrics in
summarization: ROUGE Lin (2004), which measures the lexical overlap, and
BARTScore Yuan et al. (2021), an increasingly used metric for semantic
equivalence. The results of our experiments are shown in Table 4.2.

#### Importance of Context.

The results strongly underscore the difficulty in generating chapter titles
solely based on the content of the current section without additional
contextual information. An examination of the model’s outputs (see Figure 4)
reveals how the lack of context leads to degraded coherence between titles of
a document. Notably, the model has no knowledge about the placement of the
current section in the document, leading to the repetition of functional
titles such as “Intro”, which occur frequently in the dataset. Similarly, it
also cannot generate sequential numbering or uphold uniform stylistic elements
across the document’s titles. In contrast, supplying the model with previously
generated titles results in a meaningful increase in performance. This
approach provides the model with the past context of the document structure,
enabling stylistic continuity and a smoother flow between the titles.

#### Online Generation.

Expectedly, the model’s performance improves as the input span $s$ increases.
While diminishing returns are observable, they are less pronounced compared to
the segmentation models. It is worth pointing out that the BART model with
$s=3$ and which has access to the previous titles matches the performance of
the BART model that has no context at all, once again underscoring the
importance of context. Overall, considering both the segmentation models and
the title generation models, we see 3 to 5 sentences as a reasonable trade-off
between latency and performance for the future context size $c$ and the input
span $s$. We note that $c$ can, in principle, be independently chosen from $s$
for practical smart chaptering systems. The overall latency for the title
generation is dependent on the segmentation model, though, as segment
boundaries for the span $s$ need to be determined before the generation of the
title, as each sentence may belong to the next section.

##  5 Related Work

A number of benchmarks have been proposed for the text segmentation task.
However, the majority are either small in size or synthetically constructed,
often by concatenating documents or sections of documents Choi (2000); Chen et
al. (2009); Glavaš et al. (2016, 2021). Larger-scale benchmarks are scarce and
limited to a narrow type of documents such as Wikipedia articles Koshorek et
al. (2018) or specific domains like news Liu et al. (2022b)555The news text
segmentation dataset claimed to be made available by Liu et al. (2022b) is
presently inaccessible through the provided GitHub link, effectively yielding
Wiki-727K to be the only large-scale benchmark.. Equally, these datasets have
in common that they only encompass documents that are structured in nature. In
contrast, research on spoken, conversational, or generally unstructured or
semi-structured content is still in its infancy. Lv et al. (2021) and Cho et
al. (2022) separately proposed segmentation for lecture video transcripts.
Although these works are related to our task, they are only confined to a
single domain (lecture videos), while segments are artificially constructed
based on the presentation slides. In the context of conversational content,
Ghosh et al. (2022) constructed a dataset by joining excerpts of different
chat conversations, while Zhong et al. (2021) provided a small set of meeting
transcriptions segmented by topic shifts.

Regarding title generation, this study focuses on generating section headings
and chapter titles for video transcripts. While research exists in title
generation across various domains, including news headlines Gu et al. (2020);
Liu et al. (2020); Cai et al. (2023), product titles Yang et al. (2023); Zhu
et al. (2022), video titles Zeng et al. (2016); Yu et al. (2023),
StackOverflow posts Liu et al. (2022a); Zhang et al. (2022a, 2023), and pull
requests Irsan et al. (2022); Zhang et al. (2022b), our work addresses the
unique challenges of video content structuring. One distinctive aspect lies in
ensuring that all section headings of one document not only serve as
informative signposts but also maintain a coherent and seamless flow between
them. The closest work to ours has been conducted by Zhang et al. (2019),
whose proposed model generates hierarchical outlines for Wikipedia documents.

##  6 Conclusion

In this work, we present a novel benchmark for smart chaptering. The task aims
to segment unstructured content, in particular speech, conversations, and
transcriptions, in a linear sequence of chapters and provides each chapter
with a title. We think this benchmark is a valuable addition to the text
segmentation landscape as larger-scale, non-synthetic benchmarks are scarce,
and previous research focused primarily on well-structured, homogeneous
documents. As part of this, we propose an efficient and state-of-the-art
hierarchical segmentation model and a corresponding title-generating model,
both of which have also been architected to work online. By combining our
proposed segmentation and title generation models, various practical
applications are conceivable. For example, content creators, podcasters, and
educators could use it to structure their content for their audience. We see
our work also as a stepping stone to support even more unstructured content
and speech in a broader scope, such as meetings.

## Limitations

Our study is subject to several limitations. First, the benchmark only
provides English transcriptions which means it cannot assess text segmentation
algorithms in languages other than English or be utilized in multilingual or
cross-lingual contexts, an important area of research. Second, while the
benchmark is inherently multi-modal, our evaluations were conducted solely on
models trained on a single modality, which is the transcript, thus ignoring
potentially valuable contextual information. Third, we want to note that the
latency and real-timeliness of the online chaptering models depend on sentence
lengths as the models operate on a sentence-level granularity. This dependence
on sentence length restricts our ability to exert precise control over
latency. Lastly, our title generation model suffers from exposure bias since
it is trained using reference segmentations and prepending reference titles.
In practical systems, we rely on both generated segment boundaries and titles,
which can potentially lead to error propagation.

## Acknowledgements

This research is supported by the project "How is AI Changing Science?
Research in the Era of Learning Algorithms" (HiAICS), funded by the Volkswagen
Foundation. We also thank Jan Niehues for insightful discussions.

## References

  * Beeferman et al. (1999) Doug Beeferman, Adam Berger, and John Lafferty. 1999.  Statistical Models for Text Segmentation.  _Machine Learning_ , 34(1):177–210.
  * Bird et al. (2009) Steven Bird, Ewan Klein, and Edward Loper. 2009.  _Natural language processing with Python_ , 1st edition.  O’Reilly, Beijing ; Cambridge [Mass.].  OCLC: ocn301885973.
  * Cai et al. (2023) Pengshan Cai, Kaiqiang Song, Sangwoo Cho, Hongwei Wang, Xiaoyang Wang, Hong Yu, Fei Liu, and Dong Yu. 2023.  Generating User-Engaging News Headlines.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3265–3280, Toronto, Canada. Association for Computational Linguistics.
  * Chen et al. (2009) Harr Chen, S.R.K. Branavan, Regina Barzilay, and David R. Karger. 2009.  Global Models of Document Structure using Latent Permutations.  In _Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics_ , pages 371–379, Boulder, Colorado. Association for Computational Linguistics.
  * Chivers et al. (2022) Brian Chivers, Mason P. Jiang, Wonhee Lee, Amy Ng, Natalya I. Rapstine, and Alex Storer. 2022.  ANTS: A Framework for Retrieval of Text Segments in Unstructured Documents.  In _Proceedings of the Third Workshop on Deep Learning for Low-Resource Natural Language Processing_ , pages 38–47, Hybrid. Association for Computational Linguistics.
  * Cho et al. (2022) Sangwoo Cho, Kaiqiang Song, Xiaoyang Wang, Fei Liu, and Dong Yu. 2022.  Toward Unifying Text Segmentation and Long Document Summarization.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 106–118, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Choi (2000) Freddy Y. Y. Choi. 2000.  Advances in domain independent linear text segmentation.  In _1st Meeting of the North American Chapter of the Association for Computational Linguistics_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Fournier (2013) Chris Fournier. 2013.  Evaluating Text Segmentation using Boundary Edit Distance.  In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1702–1712, Sofia, Bulgaria. Association for Computational Linguistics.
  * Ghosh et al. (2022) Reshmi Ghosh, Harjeet Singh Kajal, Sharanya Kamath, Dhuri Shrivastava, Samyadeep Basu, and Soundararajan Srinivasan. 2022.  Topic Segmentation in the Wild: Towards Segmentation of Semi-structured & Unstructured Chats.  ArXiv:2211.14954 [cs].
  * Glavaš et al. (2021) Goran Glavaš, Ananya Ganesh, and Swapna Somasundaran. 2021.  Training and Domain Adaptation for Supervised Text Segmentation.  In _Proceedings of the 16th Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 110–116, Online. Association for Computational Linguistics.
  * Glavaš et al. (2016) Goran Glavaš, Federico Nanni, and Simone Paolo Ponzetto. 2016.  Unsupervised Text Segmentation Using Semantic Relatedness Graphs.  In _Proceedings of the Fifth Joint Conference on Lexical and Computational Semantics_ , pages 125–130, Berlin, Germany. Association for Computational Linguistics.
  * Gu et al. (2020) Xiaotao Gu, Yuning Mao, Jiawei Han, Jialu Liu, You Wu, Cong Yu, Daniel Finnie, Hongkun Yu, Jiaqi Zhai, and Nicholas Zukoski. 2020.  Generating Representative Headlines for News Stories.  In _Proceedings of The Web Conference 2020_ , WWW ’20, pages 1773–1784, New York, NY, USA. Association for Computing Machinery.
  * Irsan et al. (2022) Ivana Clairine Irsan, Ting Zhang, Ferdian Thung, David Lo, and Lingxiao Jiang. 2022.  AutoPRTitle: A Tool for Automatic Pull Request Title Generation.  pages 454–458. IEEE Computer Society.
  * Koshorek et al. (2018) Omri Koshorek, Adir Cohen, Noam Mor, Michael Rotman, and Jonathan Berant. 2018.  Text Segmentation as a Supervised Learning Task.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 469–473, New Orleans, Louisiana. Association for Computational Linguistics.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A Package for Automatic Evaluation of Summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Liu et al. (2020) Dayiheng Liu, Yeyun Gong, Yu Yan, Jie Fu, Bo Shao, Daxin Jiang, Jiancheng Lv, and Nan Duan. 2020.  Diverse, Controllable, and Keyphrase-Aware: A Corpus and Method for News Multi-Headline Generation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6241–6250, Online. Association for Computational Linguistics.
  * Liu et al. (2022a) Ke Liu, Guang Yang, Xiang Chen, and Chi Yu. 2022a.  SOTitle: A Transformer-based Post Title Generation Approach for Stack Overflow.  pages 577–588. IEEE Computer Society.  ISSN: 1534-5351.
  * Liu et al. (2022b) Yang Liu, Chenguang Zhu, and Michael Zeng. 2022b.  End-to-End Segmentation-based News Summarization.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 544–554, Dublin, Ireland. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A Robustly Optimized BERT Pretraining Approach.  ArXiv:1907.11692 [cs].
  * Lukasik et al. (2020) Michal Lukasik, Boris Dadachev, Kishore Papineni, and Gonçalo Simões. 2020.  Text Segmentation by Cross Segment Attention.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4707–4716, Online. Association for Computational Linguistics.
  * Lv et al. (2021) Tengchao Lv, Lei Cui, Momcilo Vasilijevic, and Furu Wei. 2021.  VT-SSum: A Benchmark Dataset for Video Transcript Segmentation and Summarization.  ArXiv:2106.05606 [cs].
  * McInnes et al. (2018) Leland McInnes, John Healy, Nathaniel Saul, and Lukas Großberger. 2018.  UMAP: Uniform Manifold Approximation and Projection.  _Journal of Open Source Software_ , 3(29):861.
  * Prince and Labadié (2007) Violaine Prince and Alexandre Labadié. 2007.  Text Segmentation Based on Document Understanding for Information Retrieval.  In _Natural Language Processing and Information Systems_ , Lecture Notes in Computer Science, pages 295–304, Berlin, Heidelberg. Springer.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.
  * Retkowski (2023) Fabian Retkowski. 2023.  The current state of summarization.  In Andreas Sudmann, Anna Echterhölter, Markus Ramsauer, Fabian Retkowski, Jens Schröter, and Alexander Waibel, editors, _Beyond Quantity. Research with Subsymbolic AI_ , 1 edition, pages 291–312. transcript Verlag, Bielefeld, Germany.
  * Shtekh et al. (2018) Gennady Shtekh, Polina Kazakova, Nikita Nikitinsky, and Nikolay Skachkov. 2018.  Applying Topic Segmentation to Document-Level Information Retrieval.  In _Proceedings of the 14th Central and Eastern European Software Engineering Conference Russia_ , CEE-SECR ’18, pages 1–6, New York, NY, USA. Association for Computing Machinery.
  * Su et al. (2023) Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. 2023.  One Embedder, Any Task: Instruction-Finetuned Text Embeddings.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 1102–1121, Toronto, Canada. Association for Computational Linguistics.
  * Su et al. (2021) Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. 2021.  RoFormer: Enhanced Transformer with Rotary Position Embedding.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023.  Llama 2: Open Foundation and Fine-Tuned Chat Models.  ArXiv:2307.09288 [cs].
  * Wang et al. (2020) Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020.  MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 5776–5788. Curran Associates, Inc.
  * Yang et al. (2023) Bang Yang, Fenglin Liu, Zheng Li, Qingyu Yin, Chenyu You, Bing Yin, and Yuexian Zou. 2023.  Multimodal Prompt Learning for Product Title Generation with Extremely Limited Labels.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 2652–2665, Toronto, Canada. Association for Computational Linguistics.
  * Yu et al. (2023) Yakun Yu, Jiuding Yang, Weidong Guo, Hui Liu, Yu Xu, and Di Niu. 2023.  TCR: Short Video Title Generation and Cover Selection with Attention Refinement.  In _Advances in Knowledge Discovery and Data Mining_ , Lecture Notes in Computer Science, pages 245–256, Cham. Springer Nature Switzerland.
  * Yuan et al. (2021) Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021.  BARTScore: Evaluating Generated Text as Text Generation.  In _Advances in Neural Information Processing Systems_ , volume 34, pages 27263–27277. Curran Associates, Inc.
  * Zechner and Waibel (2000) Klaus Zechner and Alex Waibel. 2000.  DIASUMM: Flexible Summarization of Spontaneous Dialogues in Unrestricted Domains.  In _COLING 2000 Volume 2: The 18th International Conference on Computational Linguistics_.
  * Zeng et al. (2016) Kuo-Hao Zeng, Tseng-Hung Chen, Juan Carlos Niebles, and Min Sun. 2016.  Title Generation for User Generated Videos.  In _Computer Vision – ECCV 2016_ , Lecture Notes in Computer Science, pages 609–625, Cham. Springer International Publishing.
  * Zhang et al. (2023) Fengji Zhang, Jin Liu, Yao Wan, Xiao Yu, Xiao Liu, and Jacky Keung. 2023.  Diverse title generation for Stack Overflow posts with multiple-sampling-enhanced transformer.  _Journal of Systems and Software_ , 200:111672.
  * Zhang et al. (2022a) Fengji Zhang, Xiao Yu, Jacky Keung, Fuyang Li, Zhiwen Xie, Zhen Yang, Caoyuan Ma, and Zhimin Zhang. 2022a.  Improving Stack Overflow question title generation with copying enhanced CodeBERT model and bi-modal information.  _Information and Software Technology_ , 148:106922.
  * Zhang et al. (2019) Ruqing Zhang, Jiafeng Guo, Yixing Fan, Yanyan Lan, and Xueqi Cheng. 2019.  Outline Generation: Understanding the Inherent Content Structure of Documents.  In _Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR’19, pages 745–754, New York, NY, USA. Association for Computing Machinery.
  * Zhang et al. (2022b) Ting Zhang, Ivana Clairine Irsan, Ferdian Thung, DongGyun Han, David Lo, and Lingxiao Jiang. 2022b.  Automatic Pull Request Title Generation.  In _2022 IEEE International Conference on Software Maintenance and Evolution (ICSME)_ , pages 71–81.  ISSN: 2576-3148.
  * Zhong et al. (2021) Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev. 2021.  QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5905–5921, Online. Association for Computational Linguistics.
  * Zhu et al. (2022) Wenya Zhu, Yinghua Zhang, Yu Zhang, Yuhang Zhou, Yinfu Feng, Yuxiang Wu, Qing Da, and Anxiang Zeng. 2022.  DHA: Product Title Generation with Discriminative Hierarchical Attention for E-commerce.  In _Advances in Knowledge Discovery and Data Mining_ , Lecture Notes in Computer Science, pages 275–287, Cham. Springer International Publishing.

##  Appendix A YouTube Chapters

Figure A1: A screenshot of a YouTube video featuring segments as chapters,
which form the basis of our new text segmentation benchmark YTSeg.

##  Appendix B Data Splits

Partition | # Examples
---|---
Training |  $16,404$ ($85\text{\,}\mathrm{\char 37\relax}$)
Validation |  $1,447$ ($7.5\text{\,}\mathrm{\char 37\relax}$)
Testing |  $1,448$ ($7.5\text{\,}\mathrm{\char 37\relax}$)
Total | $19,229$
(a) YTSeg data split

Partition | # Examples
---|---
Training |  $146,907$ ($84.8\text{\,}\mathrm{\char 37\relax}$)
Validation |  $13,206$ ($7.6\text{\,}\mathrm{\char 37\relax}$)
Testing |  $13,082$ ($7.6\text{\,}\mathrm{\char 37\relax}$)
Total | $173,195$
(b) YTSeg[Titles] data split

Table A1: Data splits for YTSeg and YTSeg[Titles]

##  Appendix C Hyperparameters

We manually tuned hyperparameters and provide the parameter sets responsible
for the results disclosed in this research. While training, we continuously
calculated the relevant test metrics (such as the F1 score for segmentation)
on the validation data and performed model selection based on this
information.

Hyperparameter | Value
---|---
Sentence Encoder | sentence-transformers/all-MiniLM-L6-v2
Loss Function | Weighted Binary Cross-Entropy
Cross-Entropy Weights | $[1,2]$
Learning Rate | $2.5\times 10^{-5}$
Batch Size | 115,000 Tokens
Epochs | 15
Learning Rate Schedule | Cosine
Optimizer | AdamW
Dropout Rate | 0.1
Gradient Sampling Rate | 0.5
Table A2: Hyperparameters for MiniSeg training on the YTSeg dataset

$\boldsymbol{c}$ | $\boldsymbol{M}$ | $\boldsymbol{\alpha}$
---|---|---
1 | 1 | $[1]$
3 | 2 | $[2,1]$
5 | 3 | $[2,2,1]$
8 | 4 | $[2,2,2,2]$
10 | 5 | $[2,2,2,2,2]$
20 | 7 | $[4,4,4,2,2,2,2]$
Table A3: Overview of the partitioning $\alpha$ and the number of future-
context-accumulating layers $M$ used in the corresponding online segmentation
models with future context size $c$.

Hyperparameter | Value
---|---
Base | facebook/bart-large
Learning Rate | $5\times 10^{-5}$
Batch Size | 10,000 Tokens
Epochs | 2
Learning Rate Schedule | Cosine
Optimizer | AdamW
Dropout Rate | 0.1
1-2 Decoding Strategy | Beam Sampling
Beam Size | $5$
Top $k$ | $50$
Top $p$ | $0.95$

Table A4: Hyperparameters for training and evaluating the title generation
model on YTSeg[Titles]

##  Appendix D Evaluation

We use the segeval666https://segeval.readthedocs.io/ package Fournier (2013)
for the computation of segmentation performance metrics, including $P_{k}$ and
Boundary Similarity. In both cases, we adhere to the default parameter
settings. For the evaluation of the title generation models, we rely on the
rouge-metric777https://github.com/li-plus/rouge-metric package that wraps and
reimplements the official ROUGE-1.5.5 Perl script Lin (2004). Lastly, for the
BARTScore, we utilize the official implementation and ParaBank2-trained BART
model888https://github.com/neulab/BARTScore provided by Yuan et al. (2021).

##  Appendix E Seed Keywords

  * •

lecture

  * •

podcast

  * •

meetup

  * •

theory

  * •

math

  * •

physics

  * •

chemistry

  * •

climate history

  * •

geometry

  * •

electrical engineering

  * •

media theory

  * •

fashion

  * •

tech

  * •

explained

  * •

"analysis of"

  * •

"introducing"

  * •

"simplified"

  * •

"explanation of"

  * •

"the art of"

  * •

"mechanics of"

  * •

"recent advances in"

  * •

"in a nutshell"

  * •

"the theory of"

  * •

"guide to"

  * •

"why do *"

  * •

"why does *"

  * •

"exploring *"

  * •

"* talk"

  * •

robotics

  * •

computer vision

  * •

virtual reality

  * •

insurance

  * •

dietary

  * •

azure

  * •

brain

  * •

linear algebra

  * •

oled

  * •

silicon

  * •

linux

  * •

deployment

  * •

nature

  * •

adobe

  * •

ui design

  * •

rna

  * •

pytorch

  * •

self driving cars

  * •

machine learning

  * •

data science


11institutetext: Department of Computer Science, University of Reading,
Reading, UK 22institutetext: Department of Meteorology, University of Reading,
Reading, UK
22email<EMAIL_ADDRESS>

# Transfer Learning for Instance Segmentation of Waste Bottles using Mask
R-CNN Algorithm

Punitha Jaikumar 11 Remy Vandaele 1122 Varun Ojha 11

###### Abstract

This paper111cite as: Jaikumar, P., Vandaele, R., Ojha, V. (2021). Transfer
Learning for Instance Segmentation of Waste Bottles Using Mask R-CNN
Algorithm. Intelligent Systems Design and Applications. ISDA 2020. Springer.
proposes a methodological approach with a transfer learning scheme for plastic
waste bottle detection and instance segmentation using the mask region
proposal convolutional neural network (Mask R-CNN). Plastic bottles constitute
one of the major pollutants posing a serious threat to the environment both in
oceans and on land. The automated identification and segregation of bottles
can facilitate plastic waste recycling. We prepare a custom-made dataset of
192 bottle images with pixel-by pixel-polygon annotation for the automatic
segmentation task. The proposed transfer learning scheme makes use of a Mask
R-CNN model pre-trained on Microsoft COCO dataset. We present a comprehensive
scheme for fine-tuning the base pre-trained Mask-RCNN model on our custom
dataset. Our final fine-tuned model has achieved 59.4 mean average precision
(mAP), that corresponds to the MS COCO metric. The results indicate promising
application of deep leaning for detecting waste bottles.

###### Keywords:

Convolutional neural networks Object detection Instance segmentation Deep
learning Transfer learning

##  1 Introduction

Over the years, the plastic bottle has evolved from being one of the most
useful materials available to a cause for the severe problem of pollution on
land and seas [11]. Plastic waste poses a serious threat to the environment.
The Ocean Conservancy annual report mentions that plastic bottles ranked among
the top 5 pollutants [16]. In 2019, Ocean Conservancy reportedly collected
1.75 million bottles from the beach cleanup initiative [16]. In order to
alleviate the problem, organizations around the world have taken various
measures to collect and segregate the pollutants for plastic waste recycling
and their eco-friendly disposal. Among those measures, drones and video
surveillance in urban waste management along with image processing systems can
be used to alert environmental agencies to take adequate measures.

In this work, we study the performance of a deep learning algorithm for
detecting the waste bottles from a set of images. We prepared a custom dataset
of 192 images for the training and validation of our deep learning bottle
detection model developed on mask region proposal convolutional neural network
(Mask R-CNN). Since object shapes and forms can vary, a dataset of 192 images
may not constitute an adequate training set. Therefore, we developed a
methodological framework using transfer learning [17] to exploit a model pre-
trained with over 5000 images ’minival’ subset of the Microsoft Common Object
in Context (COCO) dataset pertaining to ’val2014training’ images of 80 object
categories [14]. Further, the limitation of the small custom-made dataset is
addressed by applying data augmentation techniques during the fine-tuning
process to improve the model’s performance [20].

Our fine-tuned model was trained on the custom-made dataset and when evaluated
on instance segmentation offered a 59.4 mean average precision (mAP) following
the COCO evaluation metric, intersection over union (IoU) of threshold range
[0.5:0.05:0.95]. Also, the final fine-tuned Mask R-CNN model was able to
detect bottles from test images and videos. The trained models, the outputs
(images and videos), and the relevant Mask R-CNN open-source code (adapted
from [1]) is available at repository [10].

This paper is organized as follows: Sec. 2 refers to the relevant literature.
Sec. 3 describes the instance segmentation model training methods. The
obtained results are described in Sec. 4, followed by conclusions in Sec. 5.

##  2 Related Work

The object segmentation task has evolved over the years in the field of
computer vision. Image-based systems have revealed the potential to find an
automated solution for the environmental problems and research is initiated
for application in waste segregation for recycling purpose. The machine
learning techniques such as support vector machine, k-nearest neighbor,
decision tree, and logistic regression were used for plastic bottles waste
management related work [12, 13, 23]. The recent advancements in deep neural
networks have outperformed traditional machine learning models in the object
classification and detection tasks. Deep Learning in the field of computer
vision developed from image classification to object localization, object
segmentation, and to instance segmentation [7] (cf. Fig. 1). There are two
type of object detectors: two stage detectors-Region based R-CNN family and
single stage detectors: such as you look only once (YOLO) [18] and single shot
detector (SSD) [15].

Figure 1: Comparison of semantic segmentation, classification and
localization, object detection and instance segmentation. (Figure adapted from
[5])

Two-Stage Detectors The region proposed methods of object detection are from
the R-CNN family. The model pipeline detects objects in two stages: (1)
generating proposal of region of interest (RoI) and (2) classification of
objects in the proposed regions. The region proposal algorithms have evolved
in the following order [24]: R-CNN, Fast R-CNN, Faster R-CNN and Mask R-CNN.

Single-Stage Detectors The single-stage detectors take an approach of a simple
regression problem predicting the bounding box coordinates and the class
probabilities from images in one evaluation. The YOLO [18] and SSD [15]
families belong to this category of detector, and are designed for speed and
real-time use while they compromise on accuracy.

In [24], authors discussed the approaches adopted in the object segmentation
and compares the performance of various algorithms and its benchmark. The
training time versus accuracy will be the main practical trade-off for
detector selection when compared to other image segmentation and object
detection algorithms and methods [9]. Two-stage region proposed detectors
performed better over single stage detectors. In [6], the author concluded
that Faster R-CNN has significantly outperformed models when applied on the
dataset containing plastic bottle images of underwater debris. In [22],
authors used Faster R-CNN, single-shot multi-box detector (SSD), and YOLO
version 2 (YOLOv2) architectures; and compared their performances. They show
that the Faster R-CNN with the rotational region proposed network achieved
90.3% PASCAL VOC AP precision [4], followed by 90.1% achieved by SSD and 77.4%
achieved by YOLOv2.

Figure 2: Mask R-CNN

Mask R-CNN: This algorithm belongs to the two-stage detectors that uses
feature pyramid network (FPN), and region proposal network (RPN) for the
object segmentation. It is an extension of the Faster R-CNN [19] along with a
new pipeline for masking the detected objects for instance segmentation [8]
(cf. Fig. 2). In Mask R-CNN, the spatial layout of input object encoded by a
mask and predicted by using fully connected network (FCN) pixel to pixel
convolutions instead of the fully connected (FC) layers that will flatten into
vectors lacking spatial dimensions as in Faster R-CNN. The RoI-align (instead
of RoI-pooling) calculated with bi-linear interpolation to improve
segmentation accuracy. This technique had gained popularity as it improves
object instance segmentation precision [7, 9, 24]. In this work, for instance
segmentation of bottles at pixel-level, we opted a two-stage detector: Mask
R-CNN.

##  3 Bottle Segmentation using Mask R-CNN

The implementation of our bottle segmentation model started with the datasets
preparation. The Mask R-CNN model was built with Python 3, TensorFlow, Keras,
and OpenCV libraries [2] by adapting the code from open-source Mask R-CNN
implementation [1]. The annotated custom dataset along with pre-trained model
weights were fed into the training pipeline consisting of several stages.

Model configuration. The ResNet-50 and ResNet-101 were the feature backbone
network options for Mask R-CNN algorithm. The images were resized to dimension
$1024\times 1024$. The initial RoI parameter was 200 for training of the
classifier/mask heads. Later, for the incremental training, RoI parameter was
512 and the maximum ground truth instances for one image was updated from 100
to 512. Also, for incremental training, the detection maximum instance was
updated from 150 to 512 that facilitates the maximum detection of object
instances. The detection minimum confidence parameter values were 0.7 and 0.5
for tuning model’s performance. The parameter setup and training process
instructions are provided in [10].

Dataset description. The dataset consists of custom images downloaded from the
internet for this work. Though there are many state-of-the-art datasets such
as COCO and the PASCAL VOC that have general images for training and research
purpose, it was noted that the availability of images pertaining to the waste
bottle class was limited. To overcome this problem, a discrete set of images
were downloaded from the internet. The set contains single and multiple bottle
images of normal and deformed features possessing various backgrounds (cf.
Table 1). Detecting objects in multi-bottle images would be a challenge due to
occlusion and lighting conditions. This will be a challenge for the
generalization of the bottle instance segmentation for the model.

Data pre-processing. Image annotation of each object is a time-consuming and
tedious task. However, it is an essential initial step for an instance
segmentation task. It is necessary to define the pixel-wise ground truth for
the target objects for the models to perform segmentation and mask generation.
Unlike other object detectors, the Mask R-CNN model requires pixel-wise
annotation for training. There are many publicly available tools for data
annotation such as VIA-VGG[3] and LabelMe [21].

For this work, pixel-wise polygon annotation of the images was performed for
instance segmentation training as per the COCO dataset format. Custom dataset
images were of varied sizes and formats. VIA-VGG annotator tool was used for
the pixel-wise polygon graphical annotation. The tool generates the output in
json format for the annotated images. The segmented ground truth mask
represents a region-wise spatial position and axis of each target object. The
annotated dataset also includes two no-instance images and two partial-
annotation images. The images were resized to dimension $1024\times 1024\times
3$ and to preserve the aspect ratio, each image was padded with zero to match
the one-size training requirements (square format of same dimension). The
dimensions of the images range from small images of $150\times 255$ to
$2448\times 3264$ in the dataset. All the images stored in the system in two
folders with the training images and the pixel annotation and labeling
information as arrays for each image in the json file. These image files are
read one-by-one from the system drive for resizing. Video files input for
testing is pre-processed with OpenCV libraries in python to extract the images
frame-by-frame to be fed into the detection pipeline for segmentation. The
segmented output image frames were saved into the set directory path with the
OpenCV video.

Table 1: Custom dataset of bottles downloaded from Internet Type |  No. of Images |  Waste Bottle Specif Images |  Total Bottle Instance Count |  Bulk Bottle Instance Count |  Image Range-min pixel |  Image Range-max pixel |  Images Resized (RGB)
---|---|---|---|---|---|---|---
Train |  177 |  102 |  712 |  561 |  33750 |  7990272 |  $1024\times 1024\times 3$
Val |  15 |  3 |  113 |  104 |  535824 |  2250000 |  $1024\times 1024\times 3$

Transfer learning scheme. The custom dataset training was initialized with the
Mask R-CNN model pre-trained on MS-COCO dataset with 80 classes. The model
weights in the initial layers representing low-level features will be useful
in many classification tasks. While deeper layers learn the high-level
features, this can be altered and retrained according to the problem
definition. In the initial phase, the models were trained with backbone
ResNet-50 and ResNet-101 for the waste bottle instance segmentation using
stochastic gradient descent (SGD) and Adam optimizer. The deeper network of
ResNet-101 and SGD optimizer performed better comparatively and was opted for
further incremental training experiments on the custom dataset. Our transfer
learning scheme has the following stages for pre-trained model fine-tuning:

  * •

Stage 1: Head-layer training with all other layers frozen

  * •

Stage 2: Fine-tuning selected layer or all layers

  * •

Stage 3: Extended training (optional)

Table 2 shows the detailed experiment for the transfer learning scheme adopted
to train for bottle segmentation. Stage 1: Head layers trained with Data
Augmentation for 30 epochs initiated for ResNet101 Backbone. Stage 2: As part
of model fine-tuning, the stage 2 models trained with the learned weights of
models in the previous stage. The 4+ layers and ALL layers models trained for
30 epoch with selected data augmentation. Additionally, the 4+ layers models
M4 and M10 trained up to 100 epochs. And, as part of Stage 3, models trained
for an additional 20 epochs to assess whether it improves the performance.
With respect to the ALL layers training at stage 3, the 4+ layers trained
models M4 and M10 tuned up to 30 epochs initially and then up to 100 epochs.
Incrementally, in stage 3, models were trained up to 150 and 160 epochs for 4+
layers and ALL layers respectively. Each model after training were evaluated
by using COCO evaluation metric and compared against the model loss.

Model evaluation and optimization. Models performances were compared against
loss metrics and were evaluated by using COCO evaluation metric mAP [8, 14].
Image segmentation models being far too expensive for the cross-validation
method, Mask R-CNN hyper-parameter tuning was based on the configuration
parameters.

##  4 Results and Discussion

Table 3 shows the models mean average precision (mAP) results as per the MS
COCO evaluation metrics- AP50, AP75, AP95 and mAP[0.5:0.95:05]. The stage 2
model M11 performed well compared to all other models trained using the
transfer learning scheme. The 4+ layers with augmentation model in stage 2 at
the 100-th epoch achieved 53.0 mAP and at stage 3 tuning of 20 epochs shows
gradual improvement achieving 55.3 mAP for both ALL layers and 4+ layers.
Tuned model for ALL layers without augmentation performed better in stage 2,
achieving mAP for model M3 and in stage 3, achieving 56.2 and 56.4 mAP for
models M12 and M14 respectively. AP90 increased from 13.90 precision in stage
1 to 49.20 precision in stage 3 fine tuning.

Models trained without data augmentation at stage 2 and stage 3 performed
better comparatively. This could be due to the model learning with fine-tuned
feature with augmentation having less diversity in the training data as it was
already trained with augmentation in Head layer training. In this task, only
horizontal left-right flip was used, more varied augmentation if applied could
have shown improvement in the model performance at later stages of tuning.

Further, model behavior requires to be monitored at each incremental training
as the false-positives in-addition to true positives may affect the quality of
the output. The model’s performance did not improve much in later stages of
tuning, this could be attributed to an inadequate dataset. While training ALL
layers, it is important to decide the training based on the dataset as most of
the low-level features will be changed drastically; and it can impact the
output on extended training leading to over-fitting.

The model M11 achieved 59.4 mAP and 74.6 precision for AP50 at stage 2 fine
tuning for instance segmentation of bottle images including dense overlapped
instances of varied shapes and features compared to the study done with Faster
R-CNN models that shows detection of objects (including other class objects)
with no overlap in [6] with 60.6 mAP and in [22] with 86.4 precision for AP50
and 90.3 for rotational RPN. Figs. 3 shows the sample test image segmentation
for the model M11 and Fig. 4 shows the instance segmentation of dense
agglomerate of bottles. The number of instances segmented by the model
increases with the decrease in the detection minimum confidence level.

Table 2: Transfer learning(TL) scheme: incremental step by step fine-tuning.

Model Ref No. |  Starting Weightsa |  TL-Stage |  Training Layers |  Augmentation |  epochs |  Total epochs
---|---|---|---|---|---|---
M1 |  COCO∗ |  1 |  HEADS |  Y |  30 |  30
M2 |  M1 (30.h5) |  2 |  ALL |  Y |  30 |  60
M3 |  M1 (30.h5) |  2 |  ALL |  N |  30 |  60
M4 |  M1 (30.h5) |  2 |  4+ |  Y |  30 |  60
M5 |  M4 (30.h5) |  2 |  4+ |  Y |  70 |  130
M6 |  M5 (100.h5) |  3 |  ALL |  Y |  20 |  150
M7 |  M5 (100.h5) |  3 |  4+ |  Y |  20 |  150
M8 |  M4 (30.h5) |  3 |  ALL |  Y |  30 |  90
M9 |  M8 (30.h5) |  3 |  ALL |  Y |  70 |  160
M10 |  M1 (30.h5) |  2 |  4+ |  N |  30 |  60
M11 |  M10 (30.h5) |  2 |  4+ |  N |  70 |  130
M12 |  M11 (100.h5) |  3 |  ALL |  N |  20 |  150
M13 |  M11 (100.h5) |  3 |  4+ |  N |  20 |  150
M14 |  M10 (30.h5) |  3 |  ALL |  N |  30 |  90
M15 |  M14 (30.h5) |  3 |  ALL |  N |  70 |  160

∗COCO dataset-Pre-trained model. Fine-tuning with ResNet-101 as the backbone
architecture; SGD as the optimizer; steps as 1000; and learning rate as 0.001.
a Starting weights mentions the pre-trained model weights for incremental
training.

Table 3: Performance measure of the models shown in Table 2

Model Ref No. |  AP50 |  AP75 |  AP90 |  mAP50 Train |  mAPa Val
---|---|---|---|---|---
M1 |  75.60 |  72.89 |  13.90 |  96.60 |  57.00
M2 |  71.19 |  63.01 |  24.93 |  98.70 |  54.53
M3 |  70.29 |  62.69 |  35.77 |  98.96 |  55.36
M4 |  68.86 |  63.24 |  31.28 |  98.86 |  54.32
M5 |  66.66 |  61.46 |  37.18 |  99.08 |  53.02
M6 |  70.14 |  64.46 |  35.08 |  99.12 |  55.29
M7 |  69.93 |  64.22 |  35.04 |  99.11 |  54.87
M8 |  68.87 |  63.46 |  30.37 |  99.06 |  54.37
M9 |  68.62 |  63.36 |  30.90 |  98.91 |  54.16
M10 |  70.72 |  64.36 |  31.29 |  98.91 |  56.08
M11 |  74.59 |  65.31 |  41.24 |  99.13 |  59.36
M12 |  70.26 |  64.29 |  47.38 |  99.09 |  56.21
M13 |  70.61 |  66.50 |  49.20 |  99.11 |  57.83
M14 |  70.57 |  64.08 |  40.66 |  98.96 |  56.35
M15 |  70.40 |  63.00 |  40.77 |  99.11 |  55.54

amAP[0.5:0.95:0.05]-is mean of AP of IoU threshold range from 0.5 to 0.95 with
0.05 step size

Table 4: Final model - M11 fine-tuning Detection Minimum Confidence | 50% | 70% |  90%
---|---|---|---
mAP_val | 76.13 | 75.14 | 74.59
mAP_val [0.5:0.5:0.95]: (after tuning) | 59.80 | 59.52 | 59.36
mAP Increase (in %) |  $+0.44$% |  $+0.16$% | –

Figure 3: Test image segmentation results

(a) 90% Detection minimum_confidence

(b) 70% Detection minimum Confidence

(c) 50% Detection minimum Confidence

Figure 4: Dense instance segmentation

Table 4 shows the tuning model M11’s hyper-parameter detection minimum
confidence to 0.5 (50%) and 0.7 (70%). This tuning achieves a marginal
improvement in the mAP_val [0.5:0.05:0.95] of 0.44% and 0.16% over the
detection minimum confidence of 0.9 (90%). Overall, it is evident from the
results that with a well-planned fine-tuning approach, the model performance
can be leveraged to achieve better results with a limited dataset.

##  5 Conclusions

Mask R-CNN, one of the seminal architectures for instance segmentation studied
for its application in segmentation of plastic waste bottles. The challenges
such as the non-availability of a comprehensive dataset and resources for
training the model were addressed by adopting a transfer learning scheme and
data augmentation technique. The experiments conducted in different phases by
fine-tuning the pre-trained model with a varied parameter setting showed a
noticeable improvement in the performance. The models trained initially with
head layers on the pre-trained model and later fine-tuned with select layers
training with and without augmentation.

In the transfer learning scheme using Mask R-CNN, applied on waste bottle
instance segmentation of images, we observe that the initial head layers
training with ResNet-101 as backbone network with incremental fine-tuning
achieved an AP50 of 74.6 precision. Further, the model evaluated with mean
average precision (mAP) for instance segmentation of IoU threshold range
[0.5:0.95:0.5] had achieved a 59.4 mAP. Test of images and video data produced
a qualitatively noticeable good performance, in instance segmentation.

In our future work, we aim to improve training dataset availability using
generative adversarial networks and investigate the dense agglomerate of
bottle instances, where the model ignores the object segmentation because of
the non-max suppression and other configuration thresholds.

## References

  * [1] Abdulla, W.: Mask R-CNN for object detection and instance segmentation on keras and tensorflow. https://github.com/matterport/Mask_RCNN (2017), accessed 07-Oct-2020
  * [2] Chollet, F.: Keras: Deep learning library for theano and tensorflow. https://keras.io/ (2015), accessed 07-Oct-2020
  * [3] Dutta, A., Zisserman, A.: The VGG image annotator (VIA). CoRR abs/1904.10699 (2019), http://arxiv.org/abs/1904.10699
  * [4] Everingham, M., Eslami, S.A., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The pascal visual object classes challenge: A retrospective. International journal of computer vision 111(1), 98–136 (2015)
  * [5] Fei-Fei Li, Justin Johnson, S.Y.: Detection and segmentation, cs231n: Convolutional neural networks for visual recognition,stanford university. http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture11.pdf (2017), accessed 19-sept-2020
  * [6] Fulton, M., Hong, J., Islam, M.J., Sattar, J.: Robotic detection of marine litter using deep visual detection models. In: 2019 International Conference on Robotics and Automation (ICRA). pp. 5752–5758. IEEE (2019)
  * [7] Garcia-Garcia, A., Orts-Escolano, S., Oprea, S., Villena-Martinez, V., Martinez-Gonzalez, P., Garcia-Rodriguez, J.: A survey on deep learning techniques for image and video semantic segmentation. Applied Soft Computing 70, 41–65 (2018)
  * [8] He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask R-CNN. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 2961–2969 (2017)
  * [9] Huang, J., Rathod, V., Sun, C., Zhu, M., Korattikara, A., Fathi, A., Fischer, I., Wojna, Z., Song, Y., Guadarrama, S., et al.: Speed/accuracy trade-offs for modern convolutional object detectors. In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. pp. 7310–7311 (2017)
  * [10] Jaikumar, P.: Fine-tuned Mask R-CNN for plastic bottle instance segmentation. https://github.com/PJ1920/Mask-R-CNN (2020), accessed 09-Oct-2020
  * [11] Jambeck, J.R., Geyer, R., Wilcox, C., Siegler, T.R., Perryman, M., Andrady, A., Narayan, R., Law, K.L.: Plastic waste inputs from land into the ocean. Science 347(6223), 768–771 (2015)
  * [12] Kambam, L.R., Aarthi, R.: Classification of plastic bottles based on visual and physical features for waste management. In: 2019 IEEE International Conference on Electrical, Computer and Communication Technologies. pp. 1–6 (2019)
  * [13] Kokoulin, A.N., Tur, A.I., Yuzhakov, A.A.: Convolutional neural networks application in plastic waste recognition and sorting. In: 2018 IEEE Conference of Russian Young Researchers in Electrical and Electronic Engineering (EIConRus). pp. 1094–1098. IEEE (2018)
  * [14] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: European Conference on Computer Vision. pp. 740–755. Springer (2014)
  * [15] Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y., Berg, A.C.: Ssd: Single shot multibox detector. Lecture Notes in Computer Science p. 21–37 (2016). https://doi.org/10.1007/978-3-319-46448-0_2, http://dx.doi.org/10.1007/978-3-319-46448-0_2
  * [16] Oceanconservancy.org: The beach and beyond: Ocean conservancy annual report. https://oceanconservancy.org/trash-free-seas/international-coastal-cleanup/annual-data-release/ (2019), accessed 07-Oct-2020
  * [17] Pan, S.J., Yang, Q.: A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering 22(10), 1345–1359 (2009)
  * [18] Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: Unified, real-time object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 779–788 (2016)
  * [19] Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object detection with region proposal networks. In: Advances in neural information processing systems. pp. 91–99 (2015)
  * [20] Shorten, C., Khoshgoftaar, T.M.: A survey on image data augmentation for deep learning. Journal of Big Data 6(1), 60 (2019)
  * [21] Torralba, A., Russell, B.C., Yuen, J.: Labelme: Online image annotation and applications. Proceedings of the IEEE 98(8), 1467–1484 (2010)
  * [22] Wang, J., Guo, W., Pan, T., Yu, H., Duan, L., Yang, W.: Bottle detection in the wild using low-altitude unmanned aerial vehicles. In: 2018 21st International Conference on Information Fusion. pp. 439–444. IEEE (2018)
  * [23] Yang, M., Thung, G.: Classification of trash for recyclability status. CS229 Project Report 2016 (2016)
  * [24] Zhao, Z.Q., Zheng, P., Xu, S.t., Wu, X.: Object detection with deep learning: A review. IEEE Transactions on Neural Networks and Learning Systems 30(11), 3212–3232 (2019)