# End-to-end named entity recognition and relation extraction using pre-
trained language models

John M. Giorgi
University of Toronto
Department of Computer Science
The Donnelly Centre
Vector Institute
Toronto, ON M5G 1M1, Canada
<EMAIL_ADDRESS>
&Xindi Wang
University Health Network
Toronto, ON M5G 2N2, Canada
<EMAIL_ADDRESS>
&Nicola Sahar
Semantic Health
Toronto, ON M5H2R2, Canada
<EMAIL_ADDRESS>
&Won Young Shin
University of Toronto
Department of Electrical & Computer Engineering
Toronto, ON M5S, Canada
<EMAIL_ADDRESS>
&Gary D. Bader
University of Toronto
Department of Computer Science
The Donnelly Centre
Toronto, ON M5S 3E1, Canada
<EMAIL_ADDRESS>&Bo Wang
Peter Munk Cardiac Center, University Health Network
Vector Institute
Toronto, ON M5G 1M1, Canada
<EMAIL_ADDRESS>
###### Abstract

Named entity recognition (NER) and relation extraction (RE) are two important
tasks in information extraction and retrieval (IE & IR). Recent work has
demonstrated that it is beneficial to learn these tasks jointly, which avoids
the propagation of error inherent in pipeline-based systems and improves
performance. However, state-of-the-art joint models typically rely on external
natural language processing (NLP) tools, such as dependency parsers, limiting
their usefulness to domains (e.g. news) where those tools perform well. The
few neural, end-to-end models that have been proposed are trained almost
completely from scratch. In this paper, we propose a neural, end-to-end model
for jointly extracting entities and their relations which does not rely on
external NLP tools and which integrates a large, pre-trained language model.
Because the bulk of our model’s parameters are pre-trained and we eschew
recurrence for self-attention, our model is fast to train. On 5 datasets
across 3 domains, our model matches or exceeds state-of-the-art performance,
sometimes by a large margin.

##  1 Introduction

The extraction of named entities (named entity recognition, NER) and their
semantic relations (relation extraction, RE) are key tasks in information
extraction and retrieval (IE & IR). Given a sequence of text (usually a
sentence), the objective is to identify both the named entities and the
relations between them. This information is useful in a variety of NLP tasks
such as question answering, knowledge base population, and semantic search
(Jiang, 2012). In the biomedical domain, NER and RE facilitate large-scale
biomedical data analysis, such as network biology (Zhou et al., 2014), gene
prioritization (Aerts et al., 2006), drug repositioning (Wang & Zhang, 2013)
and the creation of curated databases (Li et al., 2015). In the clinical
domain, NER and RE can aid in disease and treatment prediction, readmission
prediction, de-identification, and patient cohort identification (Miotto et
al., 2017).

Most commonly, the tasks of NER and RE are approached as a pipeline, with NER
preceding RE. There are two main drawbacks to this approach: (1) Pipeline
systems are prone to error propagation between the NER and RE systems. (2) One
task is not able to exploit useful information from the other (e.g. the type
of relation identified by the RE system may be useful to the NER system for
determining the type of entities involved in the relation, and vice versa).
More recently, joint models that simultaneously learn to extract entities and
relations have been proposed, alleviating the aforementioned issues and
achieving state-of-the-art performance (Miwa & Sasaki, 2014; Miwa & Bansal,
2016; Gupta et al., 2016; Li et al., 2016; 2017; Zhang et al., 2017; Adel &
Schütze, 2017; Bekoulis et al., 2018a; b; Nguyen & Verspoor, 2019; Li et al.,
2019).

Many of the proposed joint models for entity and relation extraction rely
heavily on external natural language processing (NLP) tools such as dependency
parsers. For instance, Miwa & Bansal (2016) propose a recurrent neural network
(RNN)-based joint model that uses a bidirectional long-short term memory
network (BiLSTM) to model the entities and a tree-LSTM to model the relations
between entities; Li et al. (2017) propose a similar model for biomedical
text. The tree-LSTM uses dependency tree information extracted using an
external dependency parser to model relations between entities. The use of
these external NLP tools limits the effectiveness of a model to domains (e.g.
news) where those NLP tools perform well. As a remedy to this problem,
Bekoulis et al. (2018a) proposes a neural, end-to-end system that jointly
learns to extract entities and relations without relying on external NLP
tools. In Bekoulis et al. (2018b), they augment this model with adversarial
training. Nguyen & Verspoor (2019) propose a different, albeit similar end-to-
end neural model which makes use of deep biaffine attention (Dozat & Manning,
2016). Li et al. (2019) approach the problem with multi-turn question
answering, posing templated queries to a BERT-based QA model (Devlin et al.,
2018) whose answers constitute extracted entities and their relations and
achieve state-of-the-art results on three popular benchmark datasets.

While demonstrating strong performance, end-to-end systems like Bekoulis et
al. (2018a; b) and Nguyen & Verspoor (2019) suffer from two main drawbacks.
The first is that most of the models parameters are trained from scratch. For
large datasets, this can lead to long training times. For small datasets,
which are common in the biomedical and clinical domains where it is
particularly challenging to acquire labelled data, this can lead to poor
performance and/or overfitting. The second is that these systems typically
contain RNNs, which are sequential in nature and cannot be parallelized within
training examples. The multi-pass QA model proposed in Li et al. (2019)
alleviates these issues by incorporating a pre-trained language model, BERT
(Devlin et al., 2018), which eschews recurrence for self-attention. The main
limitation of their approach is that it relies on hand-crafted question
templates to achieve maximum performance. This may become a limiting factor
where domain expertise is required to craft such questions (e.g., for
biomedical or clinical corpora). Additionally, one has to create a question
template for each entity and relation type of interest.

In this study, we propose an end-to-end model for joint NER and RE which
addresses all of these issues. Similar to past work, our model can be viewed
as a mixture of a NER module and a RE module (Figure 1). Unlike most previous
works, we include a pre-trained, transformer-based language model,
specifically BERT (Devlin et al., 2018), which achieved state-of-the-art
performance across many NLP tasks. The weights of the BERT model are fine-
tuned during training, and the entire model is trained in an end-to-end
fashion.

Our main contributions are as follows: (1) Our solution is truly end-to-end,
relying on no hand-crafted features (e.g. templated questions) or external NLP
tools (e.g. dependency parsers). (2) Our model is fast to train (e.g. under 10
minutes on a single GPU for the CoNLL04 corpus), as most of its parameters are
pre-trained and we avoid recurrence. (3) We match or exceed state-of-the-art
performance for joint NER and RE on 5 datasets across 3 domains.

##  2 The Model

Figure 1 illustrates the architecture of our approach. Our model is composed
of an NER module and an RE module. The NER module is identical to the one
proposed by Devlin et al. (2018). For a given input sequence $s$ of $N$ word
tokens $w_{1}$, $w_{2}$, $\ldots$, $w_{N}$, the pre-trained BERTBASE model
first produces a sequence of vectors,
$\displaystyle{\bm{x}}_{1}^{(\text{NER})}$,
$\displaystyle{\bm{x}}_{2}^{(\text{NER})}$, $\ldots$,
$\displaystyle{\bm{x}}_{N}^{(\text{NER})}$ which are then fed to a feed-
forward neural network (FFNN) for classification.

Figure 1: Joint named entity recognition (NER) and relation extraction (RE)
model architecture.

$\displaystyle{\bm{s}}_{i}^{(\text{NER})}=\text{FFNN}_{\text{NER}}(\displaystyle{\bm{x}}_{i}^{(\text{NER})})$
(1)

The output size of this layer is the number of BIOES-based NER labels in the
training data, $|{C^{(\text{NER})}}|$. In the BIOES tag scheme, each token is
assigned a label, where the B- tag indicates the beginning of an entity span,
I- the inside, E- the end and S- is used for any single-token entity. All
other tokens are assigned the label O.

During training, a cross-entropy loss is computed for the NER objective,

$\mathcal{L}_{\text{NER}}=-\sum_{n=1}^{N}\log\Bigg{(}\frac{e^{\displaystyle{\bm{s}}_{n}^{(\text{NER})}}}{\sum_{c}^{C^{(\text{NER})}}e^{\displaystyle{\bm{s}}_{n,c}^{(\text{NER})}}}\Bigg{)}$
(2)

where ${\bm{s}}_{n}^{(\text{NER})}$ is the predicted score that token $n\in N$
belongs to the ground-truth entity class and ${\bm{s}}_{n,c}^{(\text{NER})}$
is the predicted score for token $n$ belonging to the entity class
$c\in{C^{(\text{NER})}}$.

In the RE module, the predicted entity labels are obtained by taking the
argmax of each score vector $\displaystyle{\bm{s}}_{1}^{(\text{NER})}$,
$\displaystyle{\bm{s}}_{2}^{(\text{NER})}$, $\ldots$,
$\displaystyle{\bm{s}}_{N}^{(\text{NER})}$. The predicted entity labels are
then embedded to produce a sequence of fixed-length, continuous vectors,
$\displaystyle{\bm{e}}_{1}^{(\text{NER})}$,
$\displaystyle{\bm{e}}_{2}^{(\text{NER})}$, $\ldots$,
$\displaystyle{\bm{e}}_{N}^{(\text{NER})}$ which are concatenated with the
hidden states from the final layer in the BERT model and learned jointly with
the rest of the models parameters.

$\displaystyle{\bm{x}}_{i}^{(\text{RE})}=\displaystyle{\bm{x}}_{i}^{(\text{NER})}\parallel\displaystyle{\bm{e}}_{i}^{(\text{NER})}$
(3)

Following Miwa & Bansal (2016) and Nguyen & Verspoor (2019), we incrementally
construct the set of relation candidates, $R$, using all possible combinations
of the last word tokens of predicted entities, i.e. words with E- or S-
labels. An entity pair is assigned to a negative relation class (NEG) when the
pair has no relation or when the predicted entities are not correct. Once
relation candidates are constructed, classification is performed with a deep
bilinear attention mechanism (Dozat & Manning, 2016), as proposed by Nguyen &
Verspoor (2019).

To encode directionality, the mechanism uses FFNNs to project each
$\displaystyle{\bm{x}}_{i}^{(\text{RE})}$ into head and tail vector
representations, corresponding to whether the ${i}^{\text{th}}$ word serves as
head or tail argument of the relation.

$\displaystyle\displaystyle{\bm{h}}_{i}^{(\text{head})}$
$\displaystyle=\text{FFNN}_{\text{head}}(\displaystyle{\bm{x}}_{i}^{(\text{RE})})$
(4) $\displaystyle\displaystyle{\bm{h}}_{i}^{(\text{tail})}$
$\displaystyle=\text{FFNN}_{\text{tail}}(\displaystyle{\bm{x}}_{i}^{(\text{RE})})$
(5)

These projections are then fed to a biaffine classifier,

$\displaystyle\displaystyle{\bm{s}}_{j,k}^{(\text{RE})}$
$\displaystyle=\text{Biaffine}(\displaystyle{\bm{h}}_{j}^{(\text{head})},\displaystyle{\bm{h}}_{k}^{(\text{tail})})$
(6)
$\displaystyle\text{Biaffine}(\displaystyle{\bm{x}}_{1},\displaystyle{\bm{x}}_{2})$
$\displaystyle=\displaystyle{\bm{x}}_{1}^{T}\displaystyle{\bm{\mathsfit{U}}}\displaystyle{\bm{x}}_{2}+\displaystyle{\bm{W}}(\displaystyle{\bm{x}}_{1}\parallel\displaystyle{\bm{x}}_{2})+\displaystyle{\bm{b}}$
(7)

where $\displaystyle{\bm{\mathsfit{U}}}$ is an $m\times|C^{(\text{RE})}|\times
m$ tensor, $\displaystyle{\bm{W}}$ is a $|C^{(\text{RE})}|\times(2*m)$ matrix,
and $\displaystyle{\bm{b}}$ is a bias vector. Here, $m$ is the size of the
output layers of $\text{FFNN}_{\text{head}}$ and $\text{FFNN}_{\text{tail}}$
and $C^{(\text{RE})}$ is the set of all relation classes (including NEG).
During training, a second cross-entropy loss is computed for the RE objective

$\mathcal{L}_{\text{RE}}=-\sum_{r=1}^{R}\log\Bigg{(}\frac{e^{\displaystyle{\bm{s}}_{r}^{(\text{RE})}}}{\sum_{c}^{C^{(\text{RE})}}e^{\displaystyle{\bm{s}}_{r,c}^{(\text{RE})}}}\Bigg{)}$
(8)

where ${\bm{s}}_{r}^{(\text{RE})}$ is the predicted score that relation
candidate $r\in R$ belongs to the ground-truth relation class and
${\bm{s}}_{r,c}^{(\text{RE})}$ is the predicted score for relation $r$
belonging to the relation class $c\in C^{(\text{RE})}$.

The model is trained in an end-to-end fashion to minimize the sum of the NER
and RE losses.

$\displaystyle\mathcal{L}=\mathcal{L}_{\text{NER}}+\mathcal{L}_{\text{RE}}$
(9)

###  2.1 Entity Pretraining

In Miwa & Bansal (2016), entity pre-training is proposed as a solution to the
problem of low-performance entity detection in the early stages of training.
It is implemented by delaying the training of the RE module by some number of
epochs, before training the entire model jointly.

Our implementation of entity pretraining is slightly different. Instead of
delaying training of the RE module by some number of epochs, we weight the
contribution of $\mathcal{L}_{\text{RE}}$ to the total loss during the first
epoch of training

$\displaystyle\mathcal{L}=\mathcal{L}_{\text{NER}}+\lambda\
\mathcal{L}_{\text{RE}}$ (10)

where $\lambda$ is increased linearly from 0 to 1 during the first epoch and
set to 1 for the remaining epochs. We chose this scheme because the NER module
quickly achieves good performance for all datasets (i.e. within one epoch). In
early experiments, we found this scheme to outperform a delay of a full epoch.

###  2.2 Implementation

We implemented our model in PyTorch (Paszke et al., 2017) using the BERTBASE
model from the PyTorch Transformers
library111https://github.com/huggingface/pytorch-transformers. Our model is
available at our GitHub repository222https://github.com/bowang-lab/joint-ner-
and-re. Furthermore, we use NVIDIAs automatic mixed precision (AMP) library
Apex333https://github.com/NVIDIA/apex to speed up training and reduce memory
usage without affecting task-specific performance.

##  3 Experimental Setup

###  3.1 Datasets and evaluation

To demonstrate the generalizability of our model, we evaluate it on 5 commonly
used benchmark corpora across 3 domains. All corpora are in English. Detailed
corpus statistics are presented in Table A.1 of the appendix.

####  3.1.1 ACE04/05

The Automatic Content Extraction (ACE04) corpus was introduced by Doddington
et al. (2004), and is commonly used to benchmark NER and RE methods. There are
7 entity types and 7 relation types. ACE05 builds on ACE04, splitting the
Physical relation into two classes (Physical and Part-Whole), removing the
Discourse relation class and merging Employment-Membership-Subsidiary and
Person-Organization-Affiliation into one class (Employment-Membership-
Subsidiary).

For ACE04, we follow Miwa & Bansal (2016) by removing the Discourse relation
and evaluating our model using 5-fold cross-validation on the bnews and nwire
subsets, where 10% of the data was held out within each fold as a validation
set. For ACE05, we use the same test split as Miwa & Bansal (2016). We use
5-fold cross-validation on the remaining data to choose the hyperparameters.
Once hyperparameters are chosen, we train on the combined data from all the
folds and evaluate on the test set. For both corpora, we report the micro-
averaged F1 score. We obtained the pre-processing scripts from Miwa & Bansal
(2016)444https://github.com/tticoin/LSTM-ER/tree/master/data.

####  3.1.2 CoNLL04

The CoNLL04 corpus was introduced in Roth & Yih (2004) and consists of
articles from the Wall Street Journal (WSJ) and Associated Press (AP). There
are 4 entity types and 5 relation types.

We use the same test set split as Miwa & Sasaki
(2014)555https://github.com/pgcool/TF-MTRNN/tree/master/data/CoNLL04. We use
5-fold cross-validation on the remaining data to choose hyperparameters. Once
hyperparameters are chosen, we train on the combined data from all folds and
evaluate on the test set, reporting the micro-averaged F1 score.

####  3.1.3 ADE

The adverse drug event corpus was introduced by Gurulingappa et al. (2012) to
serve as a benchmark for systems that aim to identify adverse drug events from
free-text. It consists of the abstracts of medical case reports retrieved from
PubMed666https://www.ncbi.nlm.nih.gov/pubmed. There are two entity types, Drug
and Adverse effect and one relation type, Adverse drug event.

Similar to previous work (Li et al., 2016; 2017; Bekoulis et al., 2018b), we
remove $\sim$130 relations with overlapping entities and evaluate our model
using 10-fold cross-validation, where 10% of the data within each fold was
used as a validation set, 10% as a test set and the remaining data is used as
a train set. We report the macro F1 score averaged across all folds.

####  3.1.4 i2b2

The 2010 i2b2/VA dataset was introduced by Uzuner et al. (2011) for the 2010
i2b2/Va Workshop on Natural Language Processing Challenges for Clinical
Records. The workshop contained an NER task focused on the extraction of 3
medical entity types (Problem, Treatment, Test) and an RE task for 8 relation
types.

In the official splits, the test set contains roughly twice as many examples
as the train set. To increase the number of training examples while
maintaining a rigorous evaluation, we elected to perform 5-fold cross-
validation on the combined data from both partitions. We used 10% of the data
within each fold as a validation set, 20% as a test set and the remaining data
was used as a train set. We report the micro F1 score averaged across all
folds.

To the best of our knowledge, we are the first to evaluate a joint NER and RE
model on the 2010 i2b2/VA dataset. Therefore, we decided to compare to scores
obtained by independent NER and RE systems. We note, however, that the scores
of independent RE systems are not directly comparable to the scores we report
in this paper. This is because RE is traditionally framed as a sentence-level
classification problem. During pre-processing, each example is permutated into
processed examples containing two blinded entities and labelled for one
relation class. E.g. the example: His PCP had recently started
ciprofloxacinTREATMENT for a UTIPROBLEM becomes His PCP had recently started
@TREATMENT$ for a @PROBLEM$, where the model is trained to predict the target
relation type, Treatment is administered for medical problem (TrAP).

This task is inherently easier than the joint setup, for two reasons: relation
predictions are made on ground-truth entities, as opposed to predicted
entities (which are noisy) and the model is only required to make one
classification decision per pre-processed sentence. In the joint setup, a
model must identify any number of relations (or the lack thereof) between all
unique pairs of predicted entities in a given input sentence. To control for
the first of these differences, we report scores from our model in two
settings, once when predicted entities are used as input to the RE module, and
once when ground-truth entities are used.

###  3.2 Hyperparameters

Besides batch size, learning rate and number of training epochs, we used the
same hyperparameters across all experiments (see Table A.2). Similar to Devlin
et al. (2018), learning rate and batch size were selected for each dataset
using a minimal grid search (see See Table A.3).

One hyperparameter selected by hand was the choice of the pre-trained weights
used to initialize the BERTBASE model. For general domain corpora, we found
the cased BERTBASE weights from Devlin et al. (2018) to work well. For
biomedical corpora, we used the weights from BioBERT (Lee et al., 2019), which
recently demonstrated state-of-the-art performance for biomedical NER, RE and
QA. Similarly, for clinical corpora we use the weights provided by Peng et al.
(2019), who pre-trained BERTBASE on PubMed abstracts and clinical notes from
MIMIC-III777https://mimic.physionet.org/.

##  4 Results

###  4.1 Jointly learning NER and RE

Table 1 shows our results in comparison to previously published results,
grouped by the domain of the evaluated corpus. We find that on every dataset
besides i2b2, our model improves NER performance, for an average improvement
of $\sim$2%. This improvement is particularly large on the ACE04 and ACE05
corpora (3.98% and 2.41% respectively). On i2b2, our joint model performs
within 0.29% of the best independent NER solution.

For relation extraction, we outperform previous methods on 2 datasets and come
within $\sim$2% on both ACE05 and CoNLL04. In two cases, our performance
improvement is substantial, with improvements of 4.59% and 10.25% on the ACE04
and ADE corpora respectively. For i2b2, our score is not directly comparable
to previous systems (as discussed in section 3.1.4) but will facilitate future
comparisons of joint NER and RE methods on this dataset. By comparing overall
performance, we find that our approach achieves new state-of-the-art
performance for 3 popular benchmark datasets (ACE04, ACE05, ADE) and comes
within 0.2% for CoNLL04.

Table 1: Comparison to previously published F1 scores for joint named entity
recognition (NER) and relation extraction (RE). Ours (gold): our model, when
gold entity labels are used as input to the RE module. Ours: full, end-to-end
model (see section 2). Bold: best scores. Subscripts denote standard deviation
across three runs. $\Delta$: difference to our overall score.

| Dataset | Method | Entity | Relation | Overall | $\Delta$
---|---|---|---|---|---|---
General | ACE04 | Miwa & Bansal (2016) | 81.80 | 48.40 | 65.10 | -5.69
|  | Bekoulis et al. (2018b) | 81.64 | 47.45 | 64.54 | -6.25
|  | Li et al. (2019) | 83.60 | 49.40 | 66.50 | -4.29
|  | Ours | 87.580.2 | 53.990.1 | 70.79 | –
| ACE05 | Miwa & Bansal (2016) | 83.40 | 55.60 | 69.50 | -3.42
|  | Zhang et al. (2017) | 83.50 | 57.50 | 70.50 | -2.42
|  | Li et al. (2019) | 84.80 | 60.20 | 72.50 | -0.42
|  | Ours | 87.210.3 | 58.630.0 | 72.92 | –
| CoNLL04 | Miwa & Sasaki (2014) | 80.70 | 61.00 | 70.85 | -7.30
|  | Bekoulis et al. (2018b) | 83.61 | 61.95 | 72.78 | -5.37
|  | Li et al. (2019) | 87.80 | 68.90 | 78.35 | 0.20
|  | Ours | 89.461.0 | 66.830.4 | 78.15 | –
Biomedical | ADE | Li et al. (2016) | 79.50 | 63.40 | 71.45 | -16.21
|  | Li et al. (2017) | 84.60 | 71.40 | 78.00 | -9.66
|  | Bekoulis et al. (2018b) | 86.73 | 75.52 | 81.13 | -6.53
|  | Ours | 89.560.1 | 85.770.4 | 87.66 | –
Clinical | i2b2* |  Si et al. (2019)** | 89.55 | – | – | –
|  | Peng et al. (2019) | – | 76.40 | – | –
|  | Ours (gold) | – | 72.030.1 | – | –
|  | Ours | 89.260.1 | 63.020.3 | 76.14 | –

  * *

To the best of our knowledge, there are no published joint NER and RE models
that evaluate on the i2b2 2010 corpus. We compare our model to the state-of-
the-art for each individual task (see section 3.1.4).

  * **

We compare to the scores achieved by their BERTBASE model.

Table 2: Ablation experiment results on the CoNLL04 corpus. Scores are reported as a micro-averaged F1 score on the validation set, averaged across three runs of 5-fold cross-validation. (a) Without entity pre-training (section 2.1). (b) Without entity embeddings (eq. 3). (c) Using a single FFNN in place of FFNNhead and FFNNtail (eq. 4 and 5) (d) Without FFNNhead and FFNNtail (e) Without the bilinear operation (eq. 7). Bold: best scores. Subscripts denote standard deviation across three runs. $\Delta$: difference to the full models score. Model | Entity | Relation | Overall | $\Delta$
---|---|---|---|---
Full model | 86.320.2 | 60.290.2 | 73.30 | –
(a) w/o Entity pre-training | 85.910.1 | 59.440.2 | 72.67 | -0.63
(b) w/o Entity embeddings | 86.070.1 | 59.520.2 | 72.80 | -0.51
(c) Single FFNN | 86.020.1 | 60.130.0 | 73.07 | -0.23
(d) w/o Head/Tail | 83.930.2 | 54.670.8 | 69.30 | -4.00
(e) w/o Bilinear | 86.350.1 | 59.600.0 | 72.98 | -0.33

###  4.2 Ablation Analysis

To determine which training strategies and components are responsible for our
models performance, we conduct an ablation analysis on the CoNLL04 corpus
(Table 2). We perform five different ablations: (a) Without entity pre-
training (see section 2.1), i.e. the loss function is given by equation 9. (b)
Without entity embeddings, i.e. equation 3 becomes
$\displaystyle{\bm{x}}_{i}^{(\text{RE})}=\displaystyle{\bm{x}}_{i}^{(\text{NER})}$.
(c) Replacing the two feed-forward neural networks, FFNNhead and FFNNtail with
a single FFNN (see equation 4 and 5). (d) Removing FFNNhead and FFNNtail
entirely. (e) Without the bilinear operation, i.e. equation 7 becomes a simple
linear transformation.

Removing FFNNhead and FFNNtail has, by far, the largest negative impact on
performance. Interestingly, however, replacing FFNNhead and FFNNtail with a
single FFNN has only a small negative impact. This suggests that while these
layers are very important for model performance, using distinct FFNNs for the
projection of head and tail entities (as opposed to the same FFNN) is
relatively much less important. The next most impactful ablation was entity
pre-training, suggesting that low-performance entity detection during the
early stages of training is detrimental to learning (see section 2.1).
Finally, we note that the importance of entity embeddings is surprising, as a
previous study has found that entity embeddings did not help performance on
the CoNLL04 corpus (Bekoulis et al., 2018a), although their architecture was
markedly different. We conclude that each of our ablated components is
necessary to achieve maximum performance.

Figure 2: Visualization of the attention weights from select layers and heads
of BERT after it was fine-tuned within our model on the CoNLL04 corpus. Darker
squares indicate larger attention weights. Attention weights are shown for the
input sentence: ”Ruby fatally shot Oswald two days after Kennedy was
assassinated.”. The CLS and SEP tokens have been removed. Four major patterns
are displayed: paying attention to the next word (first image from the left)
and previous word (second from the left), paying attention to the word itself
(third from the left) and the end of the sentence (fourth from the left).

###  4.3 Analysis of the word-level attention weights

One advantage of including a transformer-based language model is that we can
easily visualize the attention weights with respect to some input. This
visualization is useful, for example, in detecting model bias and locating
relevant attention heads (Vig, 2019). Previous works have used such
visualizations to demonstrate that specific attention heads mark syntactic
dependency relations and that lower layers tend to learn more about syntax
while higher layers tend to encode more semantics (Raganato & Tiedemann,
2018).

In Figure 2 we visualize the attention weights of select layers and attention
heads from an instance of BERT fine-tuned within our model on the CoNLL04
corpus. We display four patterns that are easily interpreted: paying attention
to the next and previous words, paying attention to the word itself, and
paying attention to the end of the sentence. These same patterns have been
found in pre-trained BERT models that have not been fine-tuned on a specific,
supervised task (Vig, 2019; Raganato & Tiedemann, 2018), and therefore, are
retained after our fine-tuning procedure.

To facilitate further analysis of our learned model, we make available Jupyter
and Google Colaboratory notebooks on our GitHub
repository888https://github.com/bowang-lab/joint-ner-and-re, where users can
use multiple views to explore the learned attention weights of our models. We
use the BertViz library (Vig, 2019) to render the interactive, HTML-based
views and to access the attention weights used to plot the heat maps.

##  5 Discussion and Conclusion

In this paper, we introduced an end-to-end model for entity and relation
extraction. Our key contributions are: (1) No reliance on any hand-crafted
features (e.g. templated questions) or external NLP tools (e.g. dependency
parsers). (2) Integration of a pre-trained, transformer-based language model.
(3) State-of-the-art performance on 5 datasets across 3 domains. Furthermore,
our model is inherently modular. One can easily initialize the language model
with pre-trained weights better suited for a domain of interest (e.g. BioBERT
for biomedical corpora) or swap BERT for a comparable language model (e.g.
XLNet (Yang et al., 2019)). Finally, because of (2), our model is fast to
train, converging in approximately 1 hour or less on a single GPU for all
datasets used in this study.

Our model out-performed previous state-of-the-art performance on ADE by the
largest margin (6.53%). While exciting, we believe this corpus was
particularly easy to learn. The majority of sentences ($\sim$68%) are
annotated for two entities (drug and adverse effect, and one relation (adverse
drug event). Ostensibly, a model should be able to exploit this pattern to get
near-perfect performance on the majority of sentences in the corpus. As a
test, we ran our model again, this time using ground-truth entities in the RE
module (as opposed to predicted entities) and found that the model very
quickly reached almost perfect performance for RE on the test set ($\sim$98%).
As such, high performance on the ADE corpus is not likely to transfer to real-
world scenarios involving the large-scale annotation of diverse biomedical
articles.

In our experiments, we consider only intra-sentence relations. However, the
multiple entities within a document generally exhibit complex, inter-sentence
relations. Our model is not currently capable of extracting such inter-
sentence relations and therefore our restriction to intra-sentence relations
will limit its usefulness for certain downstream tasks, such as knowledge base
creation. We also ignore the problem of nested entities, which are common in
biomedical corpora. In the future, we would like to extend our model to handle
both nested entities and inter-sentence relations. Finally, given that
multilingual, pre-trained weights for BERT exist, we would also expect our
model’s performance to hold across multiple languages. We leave this question
to future work.

## References

  * Adel & Schütze (2017) Heike Adel and Hinrich Schütze.  Global normalization of convolutional neural networks for joint entity and relation classification.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , Copenhagen, Denmark, September 2017. Association for Computational Linguistics.
  * Aerts et al. (2006) Stein Aerts, Diether Lambrechts, Sunit Maity, Peter Van Loo, Bert Coessens, Frederik De Smet, Leon-Charles Tranchevent, Bart De Moor, Peter Marynen, Bassem Hassan, Peter Carmeliet, and Yves Moreau.  Gene prioritization through genomic data fusion.  _Nature Biotechnology_ , 24(5):537–544, may 2006\.  doi: 10.1038/nbt1203.
  * Bekoulis et al. (2018a) Giannis Bekoulis, Johannes Deleu, Thomas Demeester, and Chris Develder.  Joint entity recognition and relation extraction as a multi-head selection problem.  _Expert Systems with Applications_ , 114:34–45, 2018a.
  * Bekoulis et al. (2018b) Giannis Bekoulis, Johannes Deleu, Thomas Demeester, and Chris Develder.  Adversarial training for multi-context joint entity and relation extraction.  _arXiv preprint arXiv:1808.06876_ , 2018b.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Doddington et al. (2004) George Doddington, Alexis Mitchell, Mark Przybocki, Lance Ramshaw, Stephanie Strassel, and Ralph Weischedel.  The automatic content extraction (ACE) program – tasks, data, and evaluation.  In _Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04)_ , Lisbon, Portugal, May 2004. European Language Resources Association (ELRA).  URL http://www.lrec-conf.org/proceedings/lrec2004/pdf/5.pdf.
  * Dozat & Manning (2016) Timothy Dozat and Christopher D Manning.  Deep biaffine attention for neural dependency parsing.  _arXiv preprint arXiv:1611.01734_ , 2016.
  * Gupta et al. (2016) Pankaj Gupta, Hinrich Schütze, and Bernt Andrassy.  Table filling multi-task recurrent neural network for joint entity and relation extraction.  In _Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers_ , pp. 2537–2547, 2016\.
  * Gurulingappa et al. (2012) Harsha Gurulingappa, Abdul Mateen Rajput, Angus Roberts, Juliane Fluck, Martin Hofmann-Apitius, and Luca Toldo.  Development of a benchmark corpus to support the automatic extraction of drug-related adverse effects from medical case reports.  _Journal of Biomedical Informatics_ , 45(5):885 – 892, 2012.  ISSN 1532-0464.  doi: https://doi.org/10.1016/j.jbi.2012.04.008.  URL http://www.sciencedirect.com/science/article/pii/S1532046412000615.  Text Mining and Natural Language Processing in Pharmacogenomics.
  * Jiang (2012) Jing Jiang.  _Information Extraction from Text_ , pp. 11–41.  Springer US, Boston, MA, 2012.  ISBN 978-1-4614-3223-4.  doi: 10.1007/978-1-4614-3223-4˙2.  URL https://doi.org/10.1007/978-1-4614-3223-4_2.
  * Lee et al. (2019) Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang.  Biobert: pre-trained biomedical language representation model for biomedical text mining.  _arXiv preprint arXiv:1901.08746_ , 2019.
  * Li et al. (2016) Fei Li, Yue Zhang, Meishan Zhang, and Donghong Ji.  Joint models for extracting adverse drug events from biomedical text.  In _IJCAI_ , volume 2016, pp. 2838–2844, 2016.
  * Li et al. (2017) Fei Li, Meishan Zhang, Guohong Fu, and Donghong Ji.  A neural joint model for entity and relation extraction from biomedical text.  _BMC bioinformatics_ , 18(1):198, 2017.
  * Li et al. (2015) Gang Li, Karen E. Ross, Cecilia N. Arighi, Yifan Peng, Cathy H. Wu, and K. Vijay-Shanker.  miRTex: A text mining system for miRNA-gene relation extraction.  _PLOS Computational Biology_ , 11(9):e1004391, sep 2015.  doi: 10.1371/journal.pcbi.1004391.
  * Li et al. (2019) Xiaoya Li, Fan Yin, Zijun Sun, Xiayu Li, Arianna Yuan, Duo Chai, Mingxin Zhou, and Jiwei Li.  Entity-relation extraction as multi-turn question answering.  _arXiv preprint arXiv:1905.05529_ , 2019.
  * Loshchilov & Hutter (2017) Ilya Loshchilov and Frank Hutter.  Fixing weight decay regularization in adam.  _CoRR_ , abs/1711.05101, 2017.  URL http://arxiv.org/abs/1711.05101.
  * Miotto et al. (2017) Riccardo Miotto, Fei Wang, Shuang Wang, Xiaoqian Jiang, and Joel T Dudley.  Deep learning for healthcare: review, opportunities and challenges.  _Briefings in bioinformatics_ , 19(6):1236–1246, 2017.
  * Miwa & Bansal (2016) Makoto Miwa and Mohit Bansal.  End-to-end relation extraction using lstms on sequences and tree structures.  _arXiv preprint arXiv:1601.00770_ , 2016.
  * Miwa & Sasaki (2014) Makoto Miwa and Yutaka Sasaki.  Modeling joint entity and relation extraction with table representation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pp. 1858–1869, 2014.
  * Nguyen & Verspoor (2019) Dat Quoc Nguyen and Karin Verspoor.  End-to-end neural relation extraction using deep biaffine attention.  In Leif Azzopardi, Benno Stein, Norbert Fuhr, Philipp Mayr, Claudia Hauff, and Djoerd Hiemstra (eds.), _Advances in Information Retrieval_ , pp. 729–738, Cham, 2019. Springer International Publishing.  ISBN 978-3-030-15712-8.
  * Pascanu et al. (2013) Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio.  On the difficulty of training recurrent neural networks.  In _International Conference on Machine Learning_ , pp. 1310–1318, 2013.
  * Paszke et al. (2017) Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer.  Automatic differentiation in PyTorch.  In _NIPS Autodiff Workshop_ , 2017.
  * Peng et al. (2019) Yifan Peng, Shankai Yan, and Zhiyong Lu.  Transfer learning in biomedical natural language processing: An evaluation of BERT and elmo on ten benchmarking datasets.  _CoRR_ , abs/1906.05474, 2019.  URL http://arxiv.org/abs/1906.05474.
  * Raganato & Tiedemann (2018) Alessandro Raganato and Jörg Tiedemann.  An analysis of encoder representations in transformer-based machine translation.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pp. 287–297, 2018.
  * Roth & Yih (2004) Dan Roth and Wen-tau Yih.  A linear programming formulation for global inference in natural language tasks.  In _Proceedings of the Eighth Conference on Computational Natural Language Learning (CoNLL-2004) at HLT-NAACL 2004_ , pp. 1–8, 2004.
  * Si et al. (2019) Yuqi Si, Jingqi Wang, Hua Xu, and Kirk Roberts.  Enhancing clinical concept extraction with contextual embedding.  _CoRR_ , abs/1902.08691, 2019.  URL http://arxiv.org/abs/1902.08691.
  * Uzuner et al. (2011) Özlem Uzuner, Brett R South, Shuying Shen, and Scott L DuVall.  2010 i2b2/va challenge on concepts, assertions, and relations in clinical text.  _Journal of the American Medical Informatics Association_ , 18(5):552–556, 2011.
  * Vig (2019) Jesse Vig.  A multiscale visualization of attention in the transformer model.  _CoRR_ , abs/1906.05714, 2019.  URL http://arxiv.org/abs/1906.05714.
  * Wang & Zhang (2013) Zhong-Yi Wang and Hong-Yu Zhang.  Rational drug repositioning by medical genetics.  _Nature Biotechnology_ , 31(12):1080–1082, dec 2013.  doi: 10.1038/nbt.2758.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le.  Xlnet: Generalized autoregressive pretraining for language understanding.  _arXiv preprint arXiv:1906.08237_ , 2019.
  * Zhang et al. (2017) Meishan Zhang, Yue Zhang, and Guohong Fu.  End-to-end neural relation extraction with global optimization.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pp. 1730–1740, 2017.
  * Zhou et al. (2014) XueZhong Zhou, Jörg Menche, Albert-László Barabási, and Amitabh Sharma.  Human symptoms–disease network.  _Nature Communications_ , 5, jun 2014.  doi: 10.1038/ncomms5212.

##  Appendix A Appendix

###  A.1 Corpus Statistics

Table A.1 lists detailed statistics for each corpus used in this study.

Table A.1: Detailed entity and relation counts for each corpus used in this study. |  Dataset |  Entity classes (count) |  Relation classes (count)
---|---|---|---
General |  ACE04 |  Person (12508), Organization (4405), Geographical Entities (4425), Location (614), Facility (688), Weapon (119), and Vehicle (209) |  Physical (1202), Person-Social (362), Employment-Membership-Subsidiary (1591), Agent-Artifact (212), Person-Organization-Affiliation (141), Geopolitical Entity-Affiliation (517)
|  ACE05 |  Person (20891), Organization (5627), Geographical Entities (7455), Location(1119), Facility (1461), Weapon (911), and Vehicle (919) |  Physical (1612), Part-Whole (1060), Person-Social (615), Agent-Artifact (703), Employment-Membership-Subsidiary (1922), Geopolitical Entity-Affiliation (730)
|  CoNLL04 |  Location (4765), Organization (2499), People (3918), Other (3011) |  Kill (268), Live in (521), Located in (406), OrgBased in (452), Work for (401)
Biomedical |  ADE |  Drug (4979), Adverse effect (5669) |  Adverse drug event (6682)
Clinical |  i2b2 |  Problem (19664), Test (13831), Treatment (14186) |  PIP (2203), TeCP (504), TeRP (3053), TrAP (2617), TrCP (526), TrIP (203), TrNAP (174), TrWP (133)

###  A.2 Hyperparameters and Model Details

Table A.2 lists hyperparameters and model details that were held constant
across all experiments. Table A.3 lists those that were specific to each
evaluated corpus.

Table A.2: Hyperparameter values and model details used across all experiments. Hyperparameter | Value |  Comment
---|---|---
Tagging scheme |  BIOES |  Single token entities are tagged with an S- tag, the beginning of an entity span with a B- tag, the last token of an entity span with an E- tag, and tokens inside an entity span with an I- tag.
Dropout rate |  0.1 |  Dropout rate applied to the output of all FFNNs and the attention heads of the BERT model.
Entity embeddings |  128 |  Output dimension of the entity embedding layer.
FFNNhead / FFNNtail |  512 |  Output dimension of the FFNNhead and FFNNtail layers
No. layers (NER module) |  1 |  Number of layers used in the FFNN of the NER module.
No. layers (RE module) |  2 |  Number of layers used in the FFNNs of the RE module.
Optimizer |  AdamW |  Adam with fixed weight decay regulatization (Loshchilov & Hutter, 2017).
Gradient normalization |  $\Gamma=1$ |  Rescales the gradient whenever the norm goes over some threshold $\Gamma$ (Pascanu et al., 2013).
Weight decay |  0.1 |  L2 weight decay.

Table A.3: Hyperparameter values specific to individual datasets. Similar to
Devlin et al. (2018), a minimal grid search was performed over the values 16,
32 for batch size and 2e-5, 3e-5, and 5e-5 for learning rate.

Dataset | No. Epochs | Batch size | Learning rate | Initial BERT weights
---|---|---|---|---
ACE04 | 15 | 16 | 2e-5 | BERT-Base (cased) (Devlin et al., 2018)
ACE05 | 15 | 32 | 3e-5 | BERT-Base (cased)
CoNLL04 | 10 | 16 | 3e-5 | BERT-Base (cased)
ADE | 7 | 16 | 2e-5 | BioBERT (cased) (Lee et al., 2019)
i2b2 | 12 | 16 | 2e-5 | NCBI-BERT (uncased) (Peng et al., 2019)


# Zero-guidance Segmentation Using Zero Segment Labels

Pitchaporn Rewatbowornwong  Nattanat Chatthee11footnotemark: 1 Ekapol
Chuangsuwanich  Supasorn Suwajanakorn
VISTEC, Thailand  Chulalongkorn University, Thailand
{pitchaporn.r_s18, nattanatc_pro<EMAIL_ADDRESS><EMAIL_ADDRESS>Equal contribution

###### Abstract

CLIP has enabled new and exciting joint vision-language applications, one of
which is open-vocabulary segmentation, which can locate any segment given an
arbitrary text query. In our research, we ask whether it is possible to
discover semantic segments without any user guidance in the form of text
queries or predefined classes, and label them using natural language
automatically? We propose a novel problem zero-guidance segmentation and the
first baseline that leverages two pre-trained generalist models, DINO and
CLIP, to solve this problem without any fine-tuning or segmentation dataset.
The general idea is to first segment an image into small over-segments, encode
them into CLIP’s visual-language space, translate them into text labels, and
merge semantically similar segments together. The key challenge, however, is
how to encode a visual segment into a segment-specific embedding that balances
global and local context information, both useful for recognition. Our main
contribution is a novel attention-masking technique that balances the two
contexts by analyzing the attention layers inside CLIP. We also introduce
several metrics for the evaluation of this new task. With CLIP’s innate
knowledge, our method can precisely locate the Mona Lisa painting among a
museum crowd (Figure 1). Project page: _https://zero-guide-seg.github.io/_.

##  1 Introduction

Figure 1: Zero-guidance Segmentation segments input images and generated text
labels for all segments without any guidance or prompting. Our method produces
these results using only pretrained networks with no fine-tuning or
annotations.

Semantic segmentation is a core computer vision problem that seeks to
partition an image into semantic regions. Traditionally, the semantic classes
of interest need to be predefined and are limited in number [23]. Earlier
methods thus cannot generalize beyond the training classes. With recent
advances in joint vision-language representation learning, e.g., CLIP [26],
newer methods [22, 19, 36] can successfully predict segments corresponding to
arbitrary text queries in a novel task called open-vocabulary segmentation.
These segmentation methods are guided by a text query, which describes what
already exists in the image and must be provided by the user. Another
meaningful milestone, however, is how we can segment an image _without_ user
input or guidance like text queries or predefined classes, and label such
segments automatically using natural language. Our work provides the first
baseline for this novel problem, referred to as _zero-guidance segmentation_
111We avoid calling this _zero-shot_ segmentation because it has been used in
the literature for setups that _require_ a text query but allow it to be
unseen during training, whereas our problem does not require any text query..

Our work is inspired by a recent research direction that solves segmentation
by leveraging CLIP [36, 39]; however, our key distinction is that we require
no segmentation datasets, no text query guidance, and no additional training
or fine-tuning. This problem is challenging partly because CLIP has been
trained with image captions that globally describe the scenes and provide no
spatially specific information for learning segmentation. Surprisingly, we
show that it is possible to distill the learned knowledge from two generalist
models: a self-supervised visual model, DINO[2], and a visual-language model,
CLIP [26], to solve zero-guidance segmentation without further training.

The overall idea is to first over-segment an image into small segment
candidates, then translate each segment into words, and finally join
semantically similar segments to form the output segments. In particular, we
identify segment candidates by clustering deep pixel-wise features from a DINO
that takes as input our image. Despite using no training labels, DINO has been
shown to produce class-discriminative features, allowing unsupervised
segmentation of the primary object in an image [2] or part co-segmentation
across images [1]. However, our main challenge lies in the next step, which
maps each segment into a meaningful representation that can be later
translated to words. We leverage CLIP’s joint space to model this intermediate
representation.

A naive way to project a segment to CLIP’s joint space is to input the segment
directly into CLIP’s image encoder, but this entirely ignores the surrounding
context needed for object recognition and disambiguation [38, 5, 4].
Alternatively, masking can be applied inside the attention layers, as done
with a transformer-based segmentation network [6]. However, when applied to
CLIP’s encoder, which was not trained for segmentation, these techniques
struggle to produce segment-specific embeddings due to the domination of
global context information. Evidence in [40, 37, 13] also suggests that a
transformer trained with image-level annotations, such as CLIP, may lose local
information in its tokens in later layers. We discovered similar issues:
masking in earlier layers removes global contexts and hurts recognition,
whereas masking in later layers fails to focus the embedding on a given
segment, resulting in all embeddings describing the same dominant object in
the image.

Another difficulty in balancing global and local contexts is that different
objects may require different degrees of context balancing. For small objects,
their CLIP embeddings can be dominated by global contexts, which describe
other prominent objects in the scene. This phenomenon matches the
characteristics of CLIP’s training captions, which often ignore unimportant
objects in the image. As a result, less prominent objects may require less of
the global contexts to highlight their semantics and local contexts.

To solve this, we introduce a novel attention-masking technique called _global
subtraction_ , which helps adjust the influence of global contexts in the
output embedding. The key idea is to first estimate the saliency or the
presence of a given segment in the global contexts by analyzing CLIP’s
attention values. Then, this saliency value will be used to determine how much
global contexts should be attenuated in the segment’s embedding. The resulting
embedding in CLIP’s joint vision-language space allows us to readily translate
it to text labels with an existing image-to-text generation algorithm [31].
And finally, we merge semantically similar segments with simple thresholding
by considering both their visual and text similarities.

To evaluate our algorithm that can output arbitrary text labels, we also
propose new evaluation metrics. Evaluating an algorithm under this setup is
not straightforward as the predicted labels may not necessarily match
predefined labels in a test set but can still be correct. This may result from
the use of synonyms, such as ‘cat’ vs. ‘feline’, or differences in label
granularity, such as ‘cat’ vs. ‘orange cat’ or ‘cat’s nose’ or ‘kitten’.
Generally, there is no single correct level of granularity, and each dataset
may arbitrarily adopt any level. To address this, we propose to first map the
predicted semantic labels to the existing ones in a given test set. After
that, we can use standard measurements, such as segment IoU, to evaluate the
results as if the algorithm performs segmentation with the predefined test
classes. We also introduce Segment Recall, which measures how often ground-
truth objects are discovered, and Text Generation Quality, which tests the
quality of our embedding technique given ground-truth oracle segmentation.

Our technique can automatically segment an image into meaningful segments as
shown in Figure 1 without any supervision or text guidance. There are still
performance gaps between our technique and other supervised methods or methods
fine-tuned on segmentation datasets—but none can specifically solve our
problem that lacks user guidance. Nonetheless, we provide a detailed analysis
on obstacles that lie ahead as well as ablation studies for the first approach
to this problem. In summary, our contributions are:

  * •

We introduce the first baseline to a novel problem, _zero-guidance
segmentation_ , which aims to segment and label an input image in natural
language without predefined classes or text query guidance. Our method does
not require a segmentation dataset or fine-tuning.

  * •

We propose a novel attention-masking technique to convert a segment into an
embedding in CLIP’s joint space by balancing global and local contexts.

  * •

We present evaluation metrics for the proposed setup.

##  2 Related Work

Open-vocabulary segmentation. This problem aims to predict segments in an
input image that correspond to a set of input texts not necessarily seen
during training. Prior solutions often involve a shared latent space between
the image and text domains. OpenSeg[11] uses datasets of images, captions, and
class-agnostic segmentation masks to train a mask proposal network before
matching the predicted masks with nouns in the captions using a shared latent
space. OVSeg [22] built a segmentation pipeline that fine-tunes CLIP on masked
images to make it more suitable for masked image classification. Xu et al.
[36] proposed a zero-shot segmentation baseline by matching CLIP’s embeddings
of masked images to text embeddings of classes. ZegFormer [8] performs class-
agnostic pixel grouping to create segments and uses CLIP to classify them.
Lseg [19] trains an image pixel encoder that encodes each pixel into an
embedding that is close to the corresponding text labels’ embeddings in CLIP’s
space. These methods show impressive results but still demand expensive
segmentation labels.

To avoid the use of segmentation datasets, GroupViT[35] proposes a new method
based on hierarchical vision transformers where the visual tokens represent
arbitrary regions instead of patches in a square grid. By using only image-
caption pairs, GroupVit can match each region from its visual tokens to input
text prompts. Zhou et al.[39] modifies CLIP for text-guided segmentation and
employs self-training to improve the results. In contrast, our work requires
neither additional training nor text prompts but can discover semantic
segments and label them automatically.

Attention masking in transformer. Masking self-attention is a common practice
in NLP to input a word sequence more efficiently [32]. In computer vision, few
explorations exist: Mask2Former [6] solves supervised segmentation by masking
self-attention layers of a transformer decoder, achieving state-of-the-art
results.Unlike Mask2Former, which is trained on specific segmentation
datasets, our method and the base models we used (CLIP and DINO-ViT) do not
have any explicit segmentation supervision. We found that using the masking
mechanism of Mask2Former yields noisy CLIP embeddings, which are often heavily
biased toward the foreground objects. This can be solved by our proposed
global subtraction technique.

Image segmentation with DINO. DINO [2] is a model that uses self-distillation
to learn rich features of an input image with no supervision and has been used
as a pre-trained network or representation extractor in many tasks [33, 34,
30, 14]. Caron et al. [2] demonstrated that DINO’s features effectively
capture object boundaries and scene layout [2], and Hamilton et al. [14]
further showed that these features can perform segmentation of not only
foreground objects but also other elements in the background, such as the sky.
Our method uses a simple DINO-based clustering, inspired by Amir et al. [1],
which requires no training and offers reasonable results. Note that our key
contribution in attention masking is orthogonal to this clustering choice.

Image-to-text generation with CLIP. The recent advent of CLIP leads to new
approaches in text-image tasks, including generating text from an input image.
ClipCap [24] trains a mapping network that joins CLIP with a pretrained
language model, GPT-2, and performs image captioning with faster training.
ZeroCap [31] performs zero-shot image caption by optimizing the value matrix
$V$ in each attention module in GPT-2 to guide the embedding of the output
text toward the target image’s embedding. The output texts display knowledge
learned from CLIP’s vast and diverse training set, such as names of
celebrities and pop culture references. This is a new ability unseen in older
image captioning methods. Note that our contribution is not directly in text
generation, rather we focus on inferring semantic segments and mapping them to
CLIP’s latent space.

##  3 Approach

Figure 2: Pipeline Overview. Our method first segments an input image by
clustering learned per-pixel features extracted from DINO. The input image is
then fed into CLIP’s image encoder. In this step, the produced segmentation
masks are used to modify CLIP’s attention to provide embeddings that are more
focused to each segment. The resulting embeddings are then used to optimize a
trained language model to generate texts closest to these embeddings. Lastly,
segments with similar embeddings and text outputs are merged together to form
more coherent segmentation results.

Given an input image, our goal is to partition this image into semantic
segments and label each segment using words in natural language. Our framework
consists of four stages: 1) we identify segment candidates based on clustering
deep per-pixel features of DINO-ViT [2], 2) we map each segment to an
embedding in the CLIP’s visual-language space using our proposed attention
masking technique, 3) we translate each CLIP embedding into words by
optimizing a generative language model with an existing technique, ZeroCap
[31], and 4) we merge segments with similar semantics.

###  3.1 Finding segment candidates with DINO

The goal of this step is to partition the input image into small over-
segments, which will be merged in the final step. To do this, we feed the
input image into DINO-ViT and use the “key” values from the last attention
layer as per-patch features, which have a total dimension of (#patch$\times
C$). Following [1], these features are then upsampled and refined with
DenseCRF [21] to the original input resolution, resulting in a pixel-wise
feature map ($H$$\times$$W$$\times$$C$). We provide more details in Appendix
B.

Given the pixel-wise feature map, we initially assign each pixel its own
cluster and perform agglomerative clustering by repeatedly merging any two
clusters with the smallest combined feature variance. We stop this process
when the target number of clusters $n=20$ is reached, and we additionally
merge clusters with similar feature vectors based on their cosine similarity,
also detailed in Appendix B. The output segments from this step may break
single objects into small parts, which lack semantic meanings by themselves.
However, our decision to oversegment first allows merging in the semantic
space of CLIP later on, which takes into account both vision and language
semantics and can be done with simple thresholding.

###  3.2 Transforming segment candidates into CLIP’s vision-language
embeddings

To map a given segment to CLIP’s vision-language space, our idea is to feed
the entire input image into CLIP’s image encoder while masking some of the
encoder’s attention layers with an alpha mask corresponding to the given
segment. One major consideration is which layers should the masking be applied
to properly balance global and local contexts. This turns out to be
challenging: masking in earlier layers destroys global contexts, whereas
masking in later layers eliminates local contexts. This finding agrees with
several studies [40, 37, 13] showing that vision transformers trained for
classification suffer from an “attention collapse”, where the attention in
deeper layers becomes near uniform and all tokens converge to the same value.
Another study [29] also suggests that CLIP may lack the ability to maintain
local information. In Appendix A, we show how CLIP’s attention maps become
less localized in later layers.

Masking in the middle layers also performs poorly because different objects
still require different degrees of context balancing depending on how salient
they are in the scene. For example, the embedding of a small, obscure object
in the background can be dominated by global contexts, which describe other
prominent objects in the scene. As a result, these small objects may require
more de-emphasis of the global contexts for their semantics to emerge.

Based on this observation, we propose a simple technique to estimate the
saliency of each segment and use it to modulate how much global information
should be removed or subtracted from individual tokens during attention
masking. We next explain how we apply masking to the attention module, and
then our _global subtraction_ technique.

####  3.2.1 Masking in self-attention module

Given a logit vector $x\in\mathbb{R}^{n}$ and a flattened mask
$M\in[0,1]^{n}$, we first define the masked softmax operator as:

$\mathrm{MaskedSoftmax}(x,M)=\frac{e^{x}\odot
M}{\sum_{j=1}^{n}\left(e^{x_{j}}\times M_{j}\right)},$ (1)

where $\odot$ denotes the element-wise multiplication. To mask a standard
attention module, we compute
$A_{i}^{\text{masked}}=\mathrm{MaskedSoftmax}\left(Q_{i}K^{T}/\sqrt{d_{k}},M\right)V$
for _every_ token $i$. In practice, when the mask size is larger than the
visual patch grid, we first downsample $M$ to the same size using area
interpolation. We also prepend one extra element to the flattened $M$ for the
global token, which is always set to one in our algorithm, and thus
$\text{\\#tokens}=\text{\\#patches}+1$.

Figure 3: Attention masking and global subtraction. To encode a segment into
CLIP’s space, we pass the input image into CLIP’s image encoder and mask self-
attention map in some layers with the segment’s mask. We apply this masking
inside masked softmax function while still computing normal softmax. Cosine
similarlity between masked and unmasked output is used to estimate the
saliency of the region. This similarity determines how much global context
needs to be reduced in global subtraction.

####  3.2.2 Global subtraction

To balance global and local contexts in our output embedding, we design a
proxy function that estimates the “saliency” of each segment or the segment’s
presence in the global contexts. This value will be used to determine how much
global contexts should be removed from the attention output. Note that our
saliency value is only defined with respect to the attention mechanism in CLIP
and is unrelated to other uses of “saliency” in the literature [16, 12].

We perform the following operations separately for each attention layer $l$
using its own saliency value $\mathcal{S}_{l}$. We compute $\mathcal{S}_{l}$
as the cosine similarity between the masked attention output
$A^{\text{masked}}$ and unmasked attention output
$A=\mathrm{Softmax}(QK^{T}/\sqrt{d_{k}})V$ at layer $l$, averaged over all
tokens (we omit the subscript $l$ from $A$ and $A^{\text{masked}}$ for
simplicity):

$\mathcal{S}_{l}=\frac{1}{\text{\\#tokens}}\sum_{i=1}^{\text{\\#tokens}}\mathrm{cossim}(A_{i},A_{i}^{\text{masked}}).$
(2)

The output of the attention layer $l$ is computed by subtracting the unmasked
attention output of the global token $A_{\text{0}}$ from the masked attention
$A^{\text{masked}}$:

$\displaystyle A^{\text{out}}$ $\displaystyle=A^{\text{masked}}-wA_{0},$ (3)
$\displaystyle\text{where }w$
$\displaystyle=\exp(-(\mathcal{S}_{l}+1)^{2}/2\sigma^{2}).$ (4)

This global subtraction weight $w$ is computed by applying a Gaussian function
with a standard deviation $\sigma$ to $(\mathcal{S}_{l}+1)$, making $w$
highest when $\mathcal{S}_{l}=-1$. In other words, when an object is _not_
salient, we remove _more_ global contexts from its attention output. We apply
these masking operations starting from attention layer 21, which is chosen
empirically, to the last layer 24. Finally, the embedding of our segment is
the output from our masked CLIP’s encoder, which additionally applies linear
projection to the global token value from the last attention layer.

###  3.3 Text generation from CLIP’s embbeding

To translate our segment embedding in CLIP’s joint space into words, we use an
existing image-to-text generation algorithm, ZeroCap [31]. This method uses a
pretrained language model GPT-2 [27] along with CLIP to optimize for a
sentence that describes an input image. This is done by optimizing specific
activations of GPT-2 (K and V matrices) to complete an initial prompt of
“Image of a …” and minimizing the difference between the output sentence and
the input image in CLIP’s joint space.

###  3.4 Merging segment candidates

In this step, we merge segments that are semantically similar or small
segments that may not be so meaningful by themselves from our
oversegmentation. We compute the similarity score between two segments using
the average of two measures: 1. the cosine similarity between their visual
embeddings (Section 3.2) and 2. the cosine similarity between their text
embeddings computed from CLIP’s text encoder. In our implementation, we also
reduce the number of merging combinations by limiting the pairing option. In
particular, we first continue the agglomorative clustering in the first step
(Section 3.1) until there is a single cluster representing the entire image.
By keeping track of the merging history, we obtain a binary tree where each
node represents a segment and each parent is the merged segment of its
children. We limit the pairing to only between siblings in this tree and
recursively merge segments up the tree when their similarity score is at least
$\tau_{\text{merge}}$.

##  4 New Evaluation Protocol

This section introduces new metrics to evaluate the accuracy of the output
segments and their corresponding text labels. To overcome the evaluation
challenges due to the use of synonyms or the difference in label granularity,
such as ‘car’ vs ‘wheel’, we first map the predicted labels to the predefined
ones in the test set (Section 4.1) and verify the reassignment using
thresholding or human evaluation (Section 4.2) before applying standard
metrics, such as IoU.

###  4.1 Label reassignment

Given a predicted segment $S_{i}$ and its predicted text $T_{i}$, our goal is
to relabel $S_{i}$ with $T^{*}_{i}$, which should be one of the test labels.
We describe two reassignment techniques based on text-to-text and segment-to-
text similarity.

Text-to-text similarity (TT). This technique relabels $S_{i}$ with the ground-
truth label that is closest to $T_{i}$ in the embedding space of Sentence-BERT
[28], a pre-trained text encoder widely used in NLP for computing text
similarity [10, 20, 7]. Formally, the new label $T^{*}_{i}$ is computed by

$\displaystyle T^{*}_{i}$ $\displaystyle=\operatorname*{\arg\,\max}_{t\in
T^{\text{gt}}}\left[\mathrm{cossim}^{\text{SBERT}}(T_{i},t)\right],$ (5)

Segment-to-text similarity (ST). This technique relabels $S_{i}$ with the
ground-truth label that is closest to $S_{i}$ in the CLIP’s joint image-text
space [15, 17]. That is,

$\displaystyle T^{*}_{i}$ $\displaystyle=\operatorname*{\arg\,\max}_{t\in
T^{\text{gt}}}\left[\mathrm{cossim}^{\text{CLIP}}(S_{i},t)\right],$ (6)

where $\text{cossim}^{\text{CLIP}}(s,t)$ uses CLIP’s image encoder for $s$ and
text encoder for $t$. Note that this relabeling is commonly used in open-
vocabulary settings [22, 11] and does not consider our predicted label $T_{i}$
during relabeling, but it is helpful for evaluating our pipeline _independent_
of the text generation which is borrowed from prior work (Section 3.3).

###  4.2 Reassignment verification

For evaluation, we need to verify that the reassigned label $T^{*}_{i}$ is
sufficiently close to the original label $T_{i}$ or its segment $S_{i}$. We
provide two kinds of verification. The first is based on simple thresholding
on the cosine similarity using $\tau_{\text{SBERT}}$ and $\tau_{\text{CLIP}}$
for TT and ST reassignments, respectively (Appendix C). The second involves
human judgement, in which we ask human evaluators to rate how well the
reassigned label describes its segment on a scale of 0-3, ranging from 0:
incorrect, 1: partially correct, 2: correct but too general/specific, 3:
correct. The full definitions are in Appendix E. Multiple thresholds will be
used to report our scores.

###  4.3 Metrics

Segmentation IoU evaluates the quality of the output segments in terms of
Intersection-over-Union (IoU) against the ground-truth segments in each test
image. Given a set of predicted segments with reassigned labels $T^{*}$,
segments with the same label $T^{*}$ are merged to form a single segment for
the label. Then, IoU for each image can be computed using a standard protocol
[9].

Segment Recall measures how many objects labeled in the ground truth are
discovered. This metric disregards any extra labels predicted by our method
that are not part of the ground truth labels. We consider each merged segment
of the same reassigned label a True Positive if its IoU against the
corresponding ground-truth segment is greater than $\tau_{\text{IoU}}$.
Segment Recall is the rate of True Positive over the number of grounding
segments.

Text Generation Quality measures the quality of text generation given an
oracle segmentation. That is, we feed each _ground-truth_ segment into our
model and compute the cosine similarity between our predicted label and the
ground-truth label. If the value is higher than $\tau_{\text{SBERT}}$, it is
considered a True Positive. The score is the True Positive rate over the
entire test set. This metric evaluates our attention-masking and text
generation components independent of the segment generation process (Section
3.1).

Figure 4: Qualitative results. Our method gives reasonable segments and output
free-language text labels representing all regions. The labels can describe
regions by different kinds of descriptions, such as object names, facial
expressions, locations, car models, or even animal breeds. Figure 5: Ablation
results. Comparison between the results from different segment encoding
methods. The crop-and-mask baseline often outputs text labels that is not
relevant to segments/input images. Our method without global subtraction
suffers from global leak and often mislabels non-salient objects. Without
semantic merging, text outputs look good, but it tends to over-segment.

##  5 Experiments

Datasets. We evaluate our results on two commonly used segmentation datasets:
Pascal Context [25] and Pascal VOC 2012 [9]. Pascal Context contains 5,000
validation images with segmentation ground truths of 459 object classes for
scene segmentation task. We use PC-59, the commonly used subset with 59 most
common objects, following [11, 35, 22], as well as the full PC-459. Pascal VOC
(PAS-20) is a segmentation dataset with 1,500 image-segment validation pairs
of 20 object classes. For both datasets, we report our results on the
validation splits, as the test splits are not publicly available, but the
validation splits were never used for hypertuning. For comparison with our own
variations and a crop-and-mask baseline, we test on the first 1,000 images of
Pascal Context dataset and full 1,500 image for Pascal VOC dataset (Section
5.3). We use the full datasets when compared to prior work (Section 5.4).

###  5.1 Zero-guidance segmentation results

We present our qualitative results in Figure 1, 4. Our method can discover
semantic segments and densely label them with diverse types of labels,
including names of objects, animal breeds, facial expressions, and places.
More results are in Appendix G. In Table 1, we report IoU and Recall scores
using different reassignment and verification techniques (Section 4.2),
computed on 1,000 randomly sampled images from PC-59. We observe that ST tends
to perform reassignment better than TT, as evident by its higher scores.
Reassigning words like ‘leg’ to the correct animal class in the ground-truth
set can be challenging when relying solely on text (TT), as it lacks any
additional context. But ST can access other visual information within the
segment, which better facilitates reassignment.

Table 1: Quantitative results on 1,000 random images from PAS-59’s validation
split. We use constants ($\tau_{\text{SBERT}}$ and $\tau_{\text{CLIP}}$) and
multiple human verification scores (h) for thresholding.

| Text-text reassign. | Segment-text reassign.
---|---|---
Threshold: |  | const. | h $\geq 1$ | const. | h $=3$ | h $\geq 2$ | h $\geq 1$
IoU |  | 11.2 | 11.0 | 19.3 | 14.2 | 20.9 | 22.7
Recall |  | 10.3 | 9.8 | 18.0 | 13.2 | 18.0 | 19.4

###  5.2 User study

We evaluate the quality of predicted labels using human evaluation. Each
segment and its predicted label were shown to three distinct human evaluators,
who were asked to rate how well the label describes the segment on a scale of
0-3, similar to the process in Section 4.2 except we show the predicted label
$T_{i}$ instead of the reassigned label $T_{i}^{*}$. Full details and the
score definitions are in Appendix E.

Table 2 shows that human evaluators found about 43% of our results to be
“correct” or “correct but too generic/specific” and 64% to be at least
“partially correct”. Most correct labels predicted by our method refer to
common objects, such as ‘table’ and ‘horse’, while incorrect labels are
usually adjectives, such as ‘small’, and ‘black’. An additional analysis can
be found in Appendix F.

Table 2: Distribution of human rating scores on the quality of the predicted
labels (0: incorrect, 1: partially correct, 2: correct but too
general/specific, 3: correct).

Human rating | 0 | 1 | 2 | 3
---|---|---|---|---
% of labels | 36.0 | 20.8 | 23.9 | 19.3

###  5.3 Ablation study

We compare our method with alternative attention-masking methods, which
include 1) cropping and masking the input image to fit the segment region
[36], 2) our method without global subtraction, and 3) our method without the
merging step. Note that we also apply the same merging in the crop-and-mask
baseline for a fair comparison. We show the results in Figure 5 and in Table
3.

In Figure 5, the crop-and-mask baseline often returns text labels that are
unrelated to the segments, like ‘video’ in column 2-4. Without global
subtraction, our method often fails to recognize objects in the background due
to the leak of global contexts. For example, the sky in column 1 is labeled
‘tower’, and almost everything in column 2 is labeled ‘room’. Our full
pipeline yields reasonable results, and can label ‘ladder’, ‘lamp’, ‘floor’,
and ‘reader’ correctly.

In Table 3, we report results based on ST reassignment and constant
thresholding. Our method outperforms all alternative masking techniques in
terms the IoU, Recall, and Text Generation Quality scores on PC-59. Global
subtraction also helps improve both $\text{IoU}_{\text{CLIP}}$ by 3.0-3.1
points. On PAS-20, our method achieves a slightly lower IoU than not using
global subtraction (1.1 lower). Upon inspection, we observe that the 20-class
PAS-20 tends to label only a few foreground objects while ignoring much of the
background (see Figure 8), and not using global subtraction may preserve the
embedding of these few objects better. This bias toward primary objects,
however, would not be beneficial if the goal is to discover _all_ semantic
objects in the image.

Table 3: Ablation study of our CLIP’s mask attention technique. IoU and Recall
are computed with ST and constant thresholding.

| PC-59 | PC-459 | PAS-20
---|---|---|---
Method |  IoU${}_{\text{c}}$ | Recall${}_{\text{c}}$ | TGQ | IoU${}_{\text{c}}$ | IoU${}_{\text{c}}$
Crop and Mask | 12.1 | 10.2 | 16.9 | 5.4 | 14.0
Ours w/o glob sub. | 14.5 | 15.0 | 11.8 | 7.2 | 21.2
Ours w/o merge | 16.4 | 11.8 | - | 10.2 | 18.3
Ours | 17.5 | 15.0 | 19.0 | 11.3 | 20.1

###  5.4 Comparison to zero-shot open-vocab baseline

As a reference, we provide a comparison with GroupVit [35], which solves a
related but different segmentation problem. GroupVit [35] requires a text
query from the test set to find each segment in the input image, and the
number of segments is fixed to 8 per image by its architecture. Table 4 shows
that GroupVit obtains a better IoU on PAS VOC with 20 classes; however, our
method is significantly closing the gap on the more challenging PAS
Context-59, especially with human-threshold IoU, despite using no text queries
to help find semantic segments. Figure 6 shows that our method can discover
more objects and provide more specific labeling, while GroupVit labels only a
few objects and does not label every part of the input image.

Table 4: Comparison to GroupVit [35], which solves a related but different
segmentation problem and requires input text queries. *denotes scores computed
on 1,000 random test images. IoU${}_{\text{c}}$ and IoU${}_{\text{h}}$ are IoU
with constant thresholding or human verification.

| PC-59 | PAS-20
---|---|---
Method | IoU${}_{\text{c}}$ | IoU${}_{\text{h }\geq 2}$ | IoU${}_{\text{h }\geq 1}$ | $\text{IoU}_{\text{c}}$
GroupVit [35] | 22.4 | - | - | 52.3
Ours | 19.6 | 20.9* | 22.7* | 20.1

Figure 6: Qualitative comparison to GroupVit [35]. Despite lower IoU score,
our method discovers more objects in input images, including objects not shown
in the dataset (hay, mirror). Our method can offer more specific labels, such
as ‘stool’.  Figure 7:  Label reassignment issue. Our predicted labels ‘roof’
and ‘pub’ are correct but are not matched to the ground-truth class ‘building’
during label reassignment.

##  6 Discussion and Analysis

Mismatched text labels during evaluation. Evaluation in our new setup is still
challenging, despite using label reassignment. For example, in Figure 7, our
algorithm breaks down the ‘building’ ground-truth segment into ‘roof’ and
‘pub’, which are correct. But ST reassignment assigns ‘pub’ to ‘sign’, which
is still technically correct but not counted toward our IoU score for
‘building’. Another problematic class is ‘person’ whose parts like ‘face’,
‘hair’, ‘shirt’ appear distinct in CLIP’s space and may not be mapped to
‘person’ (Figure 9). To overcome this challenge, we may need a new kind of
embedding space that understands the hierarchical nature of object parts.

Figure 8: Examples of Pascal VOC (PAS-20) segmentation dataset Figure 9:
Failure cases. 1) Classes like ‘person’ that has many visually distinct parts
are difficult to reassign labels correctly. 2) Background regions that share
boundaries with salient objects are still prone to global leak. 3) Semantic
merging may fail when the text outputs of the same object give different
descriptions.

Global context leakage. Some background segments that share significant
boundaries with primary objects can be mislabeled due to the influence of
global contexts as shown in Figure 9. Another problem that can cause context
leakage is from the low-resolution 24x24 image grid of CLIP visual tokens. As
we downsample our segment masks to fit this grid, we lose masking precision
and information can leak between neighboring segments.

Merge fail due to different labels of the same object. Our over-segment
outputs may use a wide variety of descriptions for the same object, such as
car model and car color. These segments may fail to merge into a single object
during the merging step (see Figure 9).

Conclusion. We have presented the first framework for _zero-guidance
segmentation_ , a novel problem that seeks to segment and label an image using
natural language automatically. We leverage two generalist models, DINO and
CLIP, and propose a technique to map a given segment to CLIP’s joint space by
masking CLIP’s attention, allowing zero-shot segmentation without the need for
any segmentation dataset or fine-tuning. We also introduce a new evaluation
protocol for this problem and will release our code.

## Acknowledgements

This research was supported by PTT public company limited and SCB public
company limited.

## References

  * [1] Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel.  Deep vit features as dense visual descriptors.  arXiv preprint arXiv:2112.05814, 2(3):4, 2021.
  * [2] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9650–9660, 2021.
  * [3] Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia.  Semantic textual similarity-multilingual and cross-lingual focused evaluation.  In Proceedings of the 2017 SEMVAL International Workshop on Semantic Evaluation (2017). https://doi. org/10.18653/v1/s17-2001, 2017.
  * [4] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan Yuille.  Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.  IEEE Transactions on Pattern Analysis and Machine Intelligence, PP, 06 2016.
  * [5] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam.  Rethinking atrous convolution for semantic image segmentation.  arXiv preprint arXiv:1706.05587, 2017.
  * [6] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar.  Masked-attention mask transformer for universal image segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1290–1299, 2022.
  * [7] Hyunjin Choi, Judong Kim, Seongho Joe, and Youngjune Gwon.  Evaluation of bert and albert sentence embedding performance on downstream nlp tasks.  In 2020 25th International conference on pattern recognition (ICPR), pages 5482–5487. IEEE, 2021.
  * [8] Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai.  Decoupling zero-shot semantic segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11583–11592, 2022.
  * [9] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes (voc) challenge.  International journal of computer vision, 88(2):303–338, 2010.
  * [10] Tianyu Gao, Xingcheng Yao, and Danqi Chen.  Simcse: Simple contrastive learning of sentence embeddings.  arXiv preprint arXiv:2104.08821, 2021.
  * [11] Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Yi Lin.  Open-vocabulary image segmentation.  arXiv preprint arXiv:2112.12143, 2021.
  * [12] Stas Goferman, Lihi Zelnik-Manor, and Ayellet Tal.  Context-aware saliency detection.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(10):1915–1926, 2012.
  * [13] Chengyue Gong, Dilin Wang, Meng Li, Vikas Chandra, and Qiang Liu.  Vision transformers with patch diversification.  arXiv preprint arXiv:2104.12753, 2021.
  * [14] Mark Hamilton, Zhoutong Zhang, Bharath Hariharan, Noah Snavely, and William T. Freeman.  Unsupervised semantic segmentation by distilling feature correspondences.  In International Conference on Learning Representations, 2022.
  * [15] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi.  CLIPScore: a reference-free evaluation metric for image captioning.  In EMNLP, 2021.
  * [16] Xiaodi Hou and Liqing Zhang.  Saliency detection: A spectral residual approach.  In 2007 IEEE Conference on Computer Vision and Pattern Recognition, pages 1–8, 2007.
  * [17] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani.  Imagic: Text-based real image editing with diffusion models.  arXiv preprint arXiv:2210.09276, 2022.
  * [18] Philipp Krähenbühl and Vladlen Koltun.  Efficient inference in fully connected crfs with gaussian edge potentials.  Advances in neural information processing systems, 24, 2011.
  * [19] Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and René Ranftl.  Language-driven semantic segmentation.  arXiv preprint arXiv:2201.03546, 2022.
  * [20] Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li.  On the sentence embeddings from pre-trained language models.  arXiv preprint arXiv:2011.05864, 2020.
  * [21] Ruiyu Li, Kaican Li, Yi-Chun Kuo, Michelle Shu, Xiaojuan Qi, Xiaoyong Shen, and Jiaya Jia.  Referring image segmentation via recurrent refinement networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5745–5753, 2018.
  * [22] Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu.  Open-vocabulary semantic segmentation with mask-adapted clip.  arXiv preprint arXiv:2210.04150, 2022.
  * [23] Shervin Minaee, Yuri Boykov, Fatih Porikli, Antonio Plaza, Nasser Kehtarnavaz, and Demetri Terzopoulos.  Image segmentation using deep learning: A survey.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(7):3523–3542, 2022.
  * [24] Ron Mokady, Amir Hertz, and Amit H Bermano.  Clipcap: Clip prefix for image captioning.  arXiv preprint arXiv:2111.09734, 2021.
  * [25] Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille.  The role of context for object detection and semantic segmentation in the wild.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 891–898, 2014.
  * [26] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International Conference on Machine Learning, pages 8748–8763. PMLR, 2021.
  * [27] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners.  2019\.
  * [28] Nils Reimers and Iryna Gurevych.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019\.
  * [29] Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, and Kurt Keutzer.  How much can clip benefit vision-and-language tasks?  arXiv preprint arXiv:2107.06383, 2021.
  * [30] Oriane Sim’eoni, Gilles Puy, Huy V. Vo, Simon Roburin, Spyros Gidaris, Andrei Bursuc, Patrick P’erez, Renaud Marlet, and Jean Ponce.  Localizing objects with self-supervised transformers and no labels.  In BMVC, 2021.
  * [31] Yoad Tewel, Yoav Shalev, Idan Schwartz, and Lior Wolf.  Zerocap: Zero-shot image-to-text generation for visual-semantic arithmetic.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17918–17928, 2022.
  * [32] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [33] Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman.  Generalized category discovery.  In IEEE Conference on Computer Vision and Pattern Recognition, 2022\.
  * [34] Yangtao Wang, Xi Shen, Shell Xu Hu, Yuan Yuan, James L. Crowley, and Dominique Vaufreydaz.  Self-supervised transformers for unsupervised object discovery using normalized cut.  In Conference on Computer Vision and Pattern Recognition, 2022.
  * [35] Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang.  Groupvit: Semantic segmentation emerges from text supervision.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18134–18144, 2022.
  * [36] Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai.  A simple baseline for zero-shot semantic segmentation with pre-trained vision-language model.  arXiv preprint arXiv:2112.14757, 2021.
  * [37] Fuzhao Xue, Jianghai Chen, Aixin Sun, Xiaozhe Ren, Zangwei Zheng, Xiaoxin He, Xin Jiang, and Yang You.  Deeper vs wider: A revisit of transformer configuration.  arXiv preprint arXiv:2205.10505, 2022.
  * [38] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia.  Pyramid scene parsing network.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2881–2890, 2017.
  * [39] Chong Zhou, Chen Change Loy, and Bo Dai.  Denseclip: Extract free dense labels from clip.  arXiv preprint arXiv:2112.01071, 2021.
  * [40] Daquan Zhou, Bingyi Kang, Xiaojie Jin, Linjie Yang, Xiaochen Lian, Zihang Jiang, Qibin Hou, and Jiashi Feng.  Deepvit: Towards deeper vision transformer.  arXiv preprint arXiv:2103.11886, 2021.
  * [41] Zhengxia Zou, Zhenwei Shi, Yuhong Guo, and Jieping Ye.  Object detection in 20 years: A survey.  arXiv preprint arXiv:1905.05055, 2019.

## Appendix: Zero-guidance Segmentation Using Zero Segment Labels

In this Appendix, we provide additional details and experiments:

  * •

Section A: CLIP’s self-attention visualization

  * •

Section B: Implementation details of our segment candidate finding method

  * •

Section C: Thresholds used in metrics

  * •

Section D: Details on hyperparameters tuning

  * •

Section E: User study

  * •

Section F: User study analysis

  * •

Section G: Additional results

  * •

Section I: Potential negative societal impacts

##  Appendix A CLIP’s Self-attention Visualization

Figure 10 visualizes the self-attention maps of CLIP’s image encoder across
different layers. The self-attention maps appear to be meaningful in the
earlier layers, i.e., the patch tokens mostly attend to regions that contain
semantically similar pixels, and the global token attends to regions with
prominent objects. However, the self-attention map appear more random and
uninterpretable in the later layers.

##  Appendix B Finding Segment Candidates with DINO: Implementation Details

We provide more implementation details for Section 3.1. We adopt DINO feature
extraction method from Amir et al. [1]. The method first feeds an input image
into DINO and extracts “key” values from the last attention layer as per-patch
features.

After extracting the features, we partition the image into segments by
clustering DINO’s features. We perform bottom-up clustering starting from each
patch. The merging is done recursively by combining two clusters with the
least combined variance. After this initial clustering, we end up with a
binary tree where the root is the cluster of all the patches. This binary tree
structure is used as a heuristic to perform divisive clustering. Each node in
the tree is represented by the average per-patch feature of its members. We
prune the siblings whose cosine similarity score is over $T_{Dino}=0.9$. This
yields a segmentation map with all leaf nodes of the binary tree as segments.
The two-stage clustering algorithm is chosen to lessen the computation
requirement since we start from a large number of patches ($111\times 111$
patches).

Following Amir et al, the segmentation map is then upsampled to input
resolution and refined using DenseCRF as described in [18]. The Unary Energy
is set as the normalized distance of each patch feature to all $k$ centroids,
and the pairwise connection is fully-connected. Pairwise edge potentials are
Gaussian kernels with location (pixel coordinates) as feature and Bilateral
kernels with location and RGB values as features. Our implementation can be
founded in the provided source code.

Figure 10: Visualization of self-attention in CLIP’s image encoder. Each row
shows the attention of the token of the pink patch across layers. The last row
shows global token’s attention.

##  Appendix C Thresholds Used in Metrics

S-BERT text-to-text similarity threshold ($\tau_{\text{SBERT}}$). We provide
Text-to-text IoU (IoU${}_{\text{tt}}$) scores with several
$\tau_{\text{SBERT}}$ threshold values in Figure 11 and Table 5. In the main
experiment, when referred to a constant threshold, we select
$\tau_{\text{SBERT}}=0.5$ as it represents an approximate minimum threshold
that human evaluators use to determine if two sentences share a common topic,
based on a user study [3, 10].

CLIP segment-to-text similarity threshold ($\tau_{\text{CLIP}}$). We provide
Segment-to-text IoU (IoU${}_{\text{st}}$) scores with several
$\tau_{\text{CLIP}}$ threshold values in Figure 12 and Table 6. Selecting the
threshold $\tau_{\text{CLIP}}$ is more challenging, since there is no
established consensus or user studies to rely on. Figure 13 shows histograms
of CLIP similarity scores between ground-truth image segments and their
corresponding ground-truth labels in Pascal Context and Pascal VOC datasets.
Given the distributions, we select $\tau_{\text{CLIP}}=0.1$ to be on the safe
side to report Segment-to-text IoU scores in the main experiment.

It is important to note that for our zero-guidance segmentation problem, the
thresholds $\tau_{\text{CLIP}}$ and $\tau_{\text{SBERT}}$ are used in the
label reassignment verification process (Section 4.2), which is part of the
evaluation not the segmentation algorithm itself. For a given algorithm,
varying the threshold values can result in distinct performance profiles,
e.g., a precision-recall curve, and several thresholds may be used together
for the purposes of evaluation and comparison, as is common practice in the
object detection literature [41].

IoU threshold ($\tau_{\text{IoU}}$). We use $\tau_{\text{IoU}}=0.5$, which is
commonly used in object detection tasks to determine if a predicted bounding
box is “correct” compared to the ground truth [41].

Figure 11: Text to text IoU and SBERT threshold  Figure 12: Segment to text
IoU and CLIP threshold  Figure 13: CLIP similarity score distribution between
the ground truth segment and the ground truth label Figure 14: Start masking
layer selection Figure 15: Global subtraction’s variance selection Figure 16:
Merging threshold selection

##  Appendix D Hyperparameters Tuning

We present how our hyperparameters, which are the layer to start attention-
masking, the global subtraction variance, and the merging threshold, are
tuned. Our tuning metrics are the Text-to-text IoU (IoU${}_{\text{tt}}$) and
Segment-to-text IoU (IoU${}_{\text{st}}$) with the constant thresholds. The
data used in this process are 100 randomly selected images from the Pascal
Context’s training split, which is never used for evaluation. Note here that
there is no training involved in our pipeline.

The first parameter is the layer where attention masking starts. We found that
masking from early layers erases all global context, resulting in poor results
as context can be crucial for recognizing objects. Masking only the last layer
also has poor results due to global leak. We found that masking attention of
the last four layers (21-24) gives the best scores (see Figure 14).

Another important hyperparameter is the variance ($\sigma^{2}$) in the
saliency estimation, which is used to determine the degree of global context
subtracted from a region (see Equation 4). The higher the variance, the more
global context is reduced. As seen in Figure 15, the optimal spot is at
$\sigma^{2}=2.5$.

The last parameter is the merging threshold $\tau_{merge}$ used to decide
which segment candidates to merge (Section 3.4). We found that
$\tau_{merge}=0.8$ returns the best scores on both IoU${}_{\text{tt}}$ and
IoU${}_{\text{st}}$ on the tuning set (see Figure 16).

Table 5: Text-to-text IoU with several SBERT thresholds

| IoU${}_{\text{tt}}$
---|---
$\tau_{\text{SBERT}}$ | 0.0 | 0.1 | 0.2 | 0.3 | 0.4 | 0.5 | 0.6 | 0.7 | 0.8 | 0.9 | 1.0
Pascal Context | 8.1 | 8.1 | 8.2 | 9.4 | 11.2 | 11.4 | 9.8 | 8.6 | 6.5 | 5.6 | 5.3
Pascal VOC | 11.2 | 11.2 | 11.6 | 16.0 | 23.9 | 27.3 | 24.2 | 21.5 | 15.3 | 12.0 | 11.2

Table 6: Segment-to-text IoU with several CLIP thresholds

| IoU${}_{\text{st}}$
---|---
$\tau_{\text{CLIP}}$ | 0.00 | 0.05 | 0.10 | 0.15 | 0.20 | 0.25 | 0.30 | 0.35
Pascal Context | 19.6 | 19.6 | 19.6 | 19.5 | 16.0 | 2.2 | 0.0 | 0.0
Pascal VOC | 20.1 | 20.1 | 20.1 | 20.8 | 24.8 | 4.2 | 0.0 | 0.0

##  Appendix E User Study Implementation Details

We conducted a user study using Amazon Mechanical Turk. Each evaluation task
contains a detailed instruction and 30 questions. We did not limit the number
of tasks per human evaluator. For each question, the evaluators were shown a
predicted segment, in the form of a highlighted region, overlaid on an input
image, and its predicted text label. The evaluators were then asked to rate
how well the label describes the segment on a scale of 0-3, defined in the
provided instruction as shown in Figure 21. There were a total of 23,076
questions, each evaluated by three different evaluators. The total number of
unique evaluators was 429, and the average number of questions answered by the
evaluators was about 155. We calculated the scores (Section 4.3) for each of
the three batches separately then reported the average. We include the full
instruction and a task example in Figure 21.

Figure 17: Examples of segmentation results that were evaluated by human
evaluators with scores ranging from 3 (correct) to 0 (incorrect). The full
definitions of the scores can be found in Figure 21.

##  Appendix F User Study Analysis

We provide examples of the scores given by human evaluators in Figure 17.
According to the result, most of our score-0 labels are single-word
adjectives, such as ‘small’ and ‘black’, or collective nouns, such as ‘group’
and ‘herd’. Another kind of score-0 labels is caused by biases toward
stereotypical appearances of objects, such as when a pet dog was mislabeled as
‘stray’ due to its shabby appearance in Figure 17 (row 4). Some of score-1
labels correspond to descriptions or abstract nouns that are related to their
segments but may not fully describe them, such as ‘reflection’, ‘dining’, and
‘sunny’, and some other labels describe specific but incorrect types of
objects, such as ‘uber’ or ‘military’. Most of our score-2 labels are nearly
accurate, but the segments may incompletely or excessively cover the referred
objects, such as ‘lush moss’, ‘jet engine’, and ‘few puppies’ in the second
row in Figure 17. Most of our score-3 labels accurately represent their
segments, such as ‘horse’ and ‘new cruiser’ shown in the first row, and they
can be descriptive even on background objects, such as ‘sandy beach’ and
‘crowd observing’, unlike labels from traditional segmentation methods.

##  Appendix G Additional Results

We present more qualitative results in this section. In Figure 18, we include
more results from the ablation experiment in Section 5.3. We show random
results of our method in Figure 19 for the Pascal Context dataset and Figure
20 for the Pascal VOC 2012 dataset.

Figure 18: Qualitative ablation analysis Figure 19: Randomly sampled results
from Pascal Context Figure 20: Randomly sampled results from Pascal VOC 2012

##  Appendix H Comparison to Supervised Baselines

Table 7 presents an IoU comparison with existing supervised open-vocabulary
baselines based on the numbers presented in [22] on three datasets, Pascal VOC
(PAS-20), Pascal Context with 59 most common classes (PC-59), and the full
version of Pascal Context with 459 classes (PC-459). Unlike our approach,
these methods require segmentation annotations (or pretrained segmentation
models) during training and text queries or labels to guide the segmentation.
Note that “open-vocabulary” means that the labels can be arbitrary and unseen
during training, but the labels still need to be provided to these methods to
specify which object to find or segment at inference time. Comparing scores
between such methods using the same predefined test labels is thus
straightforward and justified. Our method, on the contrary, predicts arbitrary
text labels at inference time and is not directly comparable using the same
standard benchmarks. Nonetheless, our proposed label reassignment procedure
can provide useful comparative analysis against these supervised baselines on
the same benchmarks. In Table 7, we report our IoU scores from the setups with
the best performance using segment-to-text IoU with a constant threshold
$\tau_{\text{CLIP}}=0.1$ and human score $\geq 1$. There is still a gap in
performance between our unsupervised method and these supervised baselines,
but our method performs only slightly worse on the more challenging PC-459.

Table 7: IOU scores comparison between supervised open-vocabulary segmentation baselines (trained with segmentation labels) and our unsupervised method. Method | PAS-20 | PC-59 | PC-459
---|---|---|---
Lseg [19] | 47.4 | - | -
SimBaseline [36] | 74.5 | - | -
ZegFormer [8] | 80.7 | - | -
OpenSeg [11] | - | 42.1 | 9.0
OVSeg [22] | 94.5 | 55.7 | 12.4
Ours - $\text{IoU}_{\text{c}}$ | 20.1 | 19.6 | 11.3
Ours - $\text{IoU}_{\text{h }\geq 1}$ | - | 22.7 | -

##  Appendix I Potential Negative Societal Impacts

Unlike traditional segmentation methods, our method outputs arbitrary text
labels and may describe people with incorrect assumptions or discriminatory
characteristics based on their stereotypical appearances, such as body shape,
clothes, nationality, and sexual orientation. For example, we found ‘Asian
woman’ or ‘homeless’ in some generated, which can be offensive in some
scenarios. Some characteristics, such as beauty and politics, are rather
subjective and challenging to filter without human intervention. Due to the
data-driven nature of the pre-trained models we use, our model would also be
biased toward the culture, preferences, and characteristics of the training
sets and may pose controversial issues.

Figure 21: User interface for the user study with a full instruction,
definitions, and examples of each score.