# Latent Concept-based Explanation of NLP Models

Xuemin Yu
Faculty of Computer Science,
Dalhousie University, Canada
<EMAIL_ADDRESS>
Fahim Dalvi
Qatar Computing Research
Institute, HBKU, Qatar
<EMAIL_ADDRESS>
Nadir Durrani
Qatar Computing Research
Institute, HBKU, Qatar
<EMAIL_ADDRESS>
Hassan Sajjad
Faculty of Computer Science,
Dalhousie University, Canada
<EMAIL_ADDRESS>

###### Abstract

Interpreting and understanding the predictions made by deep learning models
poses a formidable challenge due to their inherently opaque nature. Many
previous efforts aimed at explaining these predictions rely on input features,
specifically, the words within NLP models. However, such explanations are
often less informative due to the discrete nature of these words and their
lack of contextual verbosity. To address this limitation, we introduce the
Latent Concept Attribution method (LACOAT), which generates explanations for
predictions based on latent concepts. Our foundational intuition is that a
word can exhibit multiple facets, contingent upon the context in which it is
used. Therefore, given a word in context, the latent space derived from our
training process reflects a specific facet of that word. LACOAT functions by
mapping the representations of salient input words into the training latent
space, allowing it to provide predictions with context-based explanations
within this latent space.

Latent Concept-based Explanation of NLP Models

Xuemin Yu Faculty of Computer Science, Dalhousie University, Canada
<EMAIL_ADDRESS>Fahim Dalvi Qatar Computing Research Institute, HBKU, Qatar
<EMAIL_ADDRESS>

Nadir Durrani Qatar Computing Research Institute, HBKU, Qatar
<EMAIL_ADDRESS>Hassan Sajjad Faculty of Computer Science, Dalhousie
University, Canada<EMAIL_ADDRESS>

##  1 Introduction

The opaqueness of deep neural network (DNN) models is a major challenge to
ensuring a safe and trustworthy AI system. Extensive and diverse research
works have attempted to interpret and explain these models. One major line of
work strives to understand and explain the prediction of a neural network
model using the attribution of input words to prediction Sundararajan et al.
(2017a); Denil et al. (2014).

However, the explanation based solely on input words is less informative due
to the discrete nature of words and the lack of contextual verbosity. A word
consists of multifaceted aspects such as semantic, morphological, and
syntactic roles in a sentence. Consider the word “trump” in Figure 1. It has
several facets such as a verb, a verb with specific semantics, a named entity
and a named entity representing a certain aspect such as tower names, family
names, etc. We argue that given various contexts of a word in the training
data, the model learns these diverse facets during training. Given a test
instance, depending on the context a word appears, the model uses a particular
facet of the input words in making the prediction. The explanation based on
salient words alone does not reflect the facets of the word the model has used
in the prediction and results in a less informed explanation.

Figure 1: An example of various facets of word “trump”

Dalvi et al. (2022) showed that the latent space of DNNs represents the
multifaceted aspects of words learned during training. The clustering of
training data contextualized representations provides access to these
multifaceted concepts, hereafter referred to as _latent concepts_. Given an
input word in context at test time, we hypothesize that the alignment of its
contextualized representation to a latent concept represents the facet of the
word being used by the model for that particular input. We further hypothesize
that this latent concept serves as a correct and enriched explanation of the
input word. To this end, we propose the LAtent COncept ATtribution (LACOAT)
method that generates an explanation of a model’s prediction using the latent
concepts. LACOAT discovers latent concepts of every layer of the model by
clustering contextualized representations of words in the training corpus.
Given a test instance, it identifies the most salient input representations of
every layer with respect to the prediction and dynamically maps them to the
latent concepts of the training data. The shortlisted latent concepts serve as
an explanation of the prediction. Lastly, LACOAT integrates a plausibility
module that generates a human-friendly explanation of the latent concept-based
explanation.

LACOAT is a local explanation method that provides an explanation of a single
test instance. The reliance on the training data latent space makes the
explanation reliable and further reflects on the quality of learning of the
model and the training data. We perform qualitative and quantitative
evaluation of LACOAT using the part-of-speech (POS) tagging and sentiment
classification tasks across three pre-trained models. LACOAT generates an
enriched explanation that is useful in understanding the model’s reasoning for
a prediction. It also helps in understanding how the model has structured the
knowledge of a task. We also conduct human evaluation to measure the utility
of LACOAT with a human-in-the-loop.

##  2 Methodology

LACOAT consists of the following four modules:

  * •

The first module, ConceptDiscoverer, discovers latent concepts of a model
given a corpus.

  * •

PredictionAttributor, the second module, selects the most salient words (along
with their contextual representations) in a sentence with respect to the
model’s prediction.

  * •

Thirdly, ConceptMapper, maps the representations of the salient words to the
latent concepts discovered by ConceptDiscoverer and provides a latent concept-
based explanation.

  * •

PlausiFyer takes a latent concept explanation as input and generates a
plausible and human-understandable explanation of the prediction.

Consider a sentiment classification dataset and a sentiment classification
model as an example. LACOAT works as follows: ConceptDiscoverer takes the
training dataset and the model as input and outputs latent concepts of the
model. At test time, given an input sentence, PredictionAttributor identifies
the most salient input representations with respect to the prediction.
ConceptMapper maps these salient input representations to the training data
latent concepts and provides them as an explanation of the prediction.
PlausiFyer takes the test sentence and its concept-based explanation and
generates a human-friendly and insightful explanation of the prediction. In
the following we describe these modules in detail.

Consider $\mathbb{M}$ represents the DNN model being interpreted, with $L$
layers, each of size $H$. Let $\overrightarrow{z}_{w_{i}}$ be the _contextual
representation_ of a word $w_{i}$ in an input sentence
$\\{w_{1},w_{2},...,w_{i},....\\}$. The representation can belong to any
particular layer in the model, and LACOAT will generate explanations with
respect to that layer.

###  2.1 ConceptDiscoverer

The words are grouped in the high-dimensional space based on various latent
relations such as semantic, morphology and syntax Mikolov et al. (2013); Reif
et al. (2019). With the inclusion of context i.e. contextualized
representations, these groupings evolve into dynamically formed clusters
representing a unique facet of the words called _latent concept_ Dalvi et al.
(2022). Figure 1 shows a few examples of latent concepts that capture
different facets of the word "trump".

The goal of ConceptDiscoverer is to discover latent concepts given a model
$\mathbb{M}$ and a dataset $\mathbb{D}$. We follow an identical procedure to
Dalvi et al. (2022) to discover latent concepts. Specifically, for every word
$w_{i}$ in $\mathbb{D}$, we extract contextual representations
$\overrightarrow{z}_{w_{i}}$. We then cluster these representations using
agglomerative hierarchical clustering Gowda and Krishna (1978). Specifically,
the distance between any two representations is computed using the squared
Euclidean distance, and Ward’s minimum variance criterion is used to minimize
total within-cluster variance.

Each cluster represents a latent concept. Let
$\mathcal{C}={C_{1},C_{2},...,C_{K}}$ represents the set of latent concepts
extracted by ConceptDiscoverer, where each $C_{i}={w_{1},w_{2},...}$ is a set
of words in a particular context. For sequence classification tasks, we also
consider the [CLS] token (or a representative classification token) from each
sentence in the dataset as a “word” and discover the latent concepts. In this
case, a latent concept may consist of words only, [CLS] tokens only, or a mix
of both.

###  2.2 Salient Representations Extraction

Given an input instance $s$, the goal of PredictionAttributor is to extract
salient input representations with respect to the prediction $p$ from model
$\mathbb{M}$. Gradient-based methods have been effectively used to compute the
saliency of the input features for the given prediction, such as pure Gradient
Simonyan et al. (2014), Input x Gradient Shrikumar et al. (2017) and
Integrated Gradients (IG) Sundararajan et al. (2017b). In this work, we use IG
as our gradient-based method as it’s a well-established method from
literature. However, LACOAT is agnostic to the choice of the attribution
method, and any other method that identifies salient input representations can
be used while keeping the rest of the pipeline unchanged.

Formally, we first use IG to get attribution scores for every token in the
input $s$, and then select the top tokens that make up $50\%$ of the total
attribution mass (similar to top-P sampling).

In addition to the saliency based attribution method, we also experimented
with using the position of the output head as an indication of the most
salient contextual representation. A detailed description and results are
provided in Appendix A.

###  2.3 ConceptMapper

At test time, given an input sentence PredictionAttributor provides the
salient input representations. ConceptMapper maps each salient representation
to a latent concept $C_{i}$ of the training latent space. These latent
concepts highlight a particular facet of the salient representations that is
being used by the model and serve as an explanation of the prediction.

ConceptMapper uses a logistic regression classifier that maps a representation
$\overrightarrow{z}_{w_{i}}$ to one of the $K$ latent concepts. The model is
trained using the representations of words from $\mathbb{D}$ that are used by
ConceptDiscoverer as input features and the concept index (cluster id) as
their label. Hence, for a concept $C_{i}$ and a word $w_{j}\in C_{i}$, a
training instance of the classifier is the input
$x=\overrightarrow{z}_{w_{j}}$ and the output is $y=i$.

###  2.4 PlausiFyer

Interpreting latent concepts can be challenging due to the need for diverse
knowledge, including linguistic, task-specific, worldly, and geographical
expertise (as seen in Figure 1). PlausiFyer offers a user-friendly summary and
explanation of the latent concept and its relationship to the input instance
using a Large Language Model (LLM).

We use the following prompt for the sequence classification task:

    Do you find any common semantic, structural, lexi-
    cal and topical relation between these sentences
    with the main sentence? Give a more specific and
    concise summary about the most prominent relation
    among these sentences.

    main sentence: {sentence}
    {sentences}
    No talk, just go.

and the following prompt for the sequence labeling task:

    Do you find any common semantic, structural, lexi-
    cal and topical relation between the word highlig-
    hted in the sentence (enclosed in [[ ]]) and the
    following list of words? Give a more specific and
    concise summary about the most prominent relation
    among these words.

    Sentence: {sentence}
    List of words: {words}
    Answer concisely and to the point.

We did not provide the prediction or the gold label to LLM to avoid biasing
the explanation.

##  3 Experimental Setup

Figure 2: Sentiment task: Latent concepts of the most attributed words in
Layers 0, 6 and 12

##### Data

We use Parts-of-Speech (POS) Tagging and Sentiment Classification (Sentiment)
tasks for our experiments. The former is a sequence labeling task, where every
word in the input sentence is assigned a POS tag, while the latter classifies
sentences into two classes representing _Positive_ and _Negative_ sentiment.
We use the Penn TreeBank dataset Marcus et al. (1993) for POS and the ERASER
Movie Reviews dataset Pang and Lee (2004); Zaidan and Eisner (2008) for
Sentiment. The POS dataset consists of 36k, 1.8k and 1.9k splits for train,
dev and test respectively and 44 classes.

The ERASER movie review dataset consists of labeled paragraphs with human
annotations of the words and phrases. We filter sentences that have a
word/phrase labeled with sentiment and create a sentence-level sentiment
classification dataset. The dataset consists of 9.4k positive and 8.6k
negative instances. We randomly split it into 13k, 1.5k and 2.7k splits for
train, dev and test respectively.

##### Models

We fine-tune 12-layered pre-trained models; BERT-base-cased Devlin et al.
(2019), RoBERTa-base Liu et al. (2019) and XLM-Roberta Conneau et al. (2020)
using the training datasets of the tasks. We use _transformers_ Wolf et al.
(2020) with the default settings and hyperparameters. Task-wise performance of
the models is provided in Appendix Tables 4 and 5.

##### Module-specific hyperparameters

When extracting the activation and/or attribution of a word, we average the
respective value over the word’s subword units. We optimize the number of
clusters $K$ for each dataset as suggested by Dalvi et al. (2022). We use
$K=600$ (POS) and $K=400$ (Sentiment) for ConceptDiscoverer.

Since the number of words in $\mathbb{D}$ can be very high, and the clustering
algorithm is limited by the number of representations it can efficiently
cluster, we filter out words with frequencies less than 5 and randomly select
20 contextual occurrences of every word with the assumption that a word may
have a maximum of 20 facets. These settings are in line with Dalvi et al.
(2022). In the case of [CLS] tokens, we keep all of the instances.

We use a zero-vector as the baseline vector in PredictionAttributor’s IG,
using 500 approximation steps. For ConceptMapper, we use the cross-entropy
loss with L2 regularization and train the classifier with ‘lbfgs’ solver and
100 maximum iterations. To optimize the classifier and to evaluate its
performance, we split the dataset $\mathcal{D}$ into train ($90\%$) and test
($10\%$). ConceptMapper used in the LACOAT pipeline is trained using the full
dataset $\mathcal{D}$. Finally, for PlausiFyer, we use ChatGPT with a
temperature of 0 and a top_p value of 0.95.

(a) Sentiment: A positive labeled test instance correctly predicted by the
model.

(b) Sentiment: A negatively labeled test instance that is incorrectly
predicted as positive.

(c) POS: An adverb with semantics showing degree and intensity of an action

(d) POS: An incorrect prediction that can be detected from the latent concept

Figure 3: A few examples of LACOAT explanations for BERT using POS and
Sentiment tasks

##  4 Evaluation

We perform a qualitative evaluation, a human evaluation and a module-level
evaluation of LACOAT to measure its correctness and efficacy.

###  4.1 Qualitative Evaluation

In this section, we qualitatively evaluate the usefulness of the latent
concept-based explanation and the generated human-friendly explanation.

####  4.1.1 Evolution of Concepts

LACOAT generates the explanation for each layer with respect to a prediction.
The layer-wise explanation shows the evolution of concepts in making the
prediction. Figure 2 shows layers 0, 6 and 12’s latent concept of the most
attributed input token for RoBERTa fine-tuned on the sentiment task (see App
Fig 5 for other examples). We found that the initial layer latent concepts do
not always align with the sentiment of the input instance and may represent a
general language concept. For instance, Figure 2(a) shows the concept of
comparative and superlative adjectives of both positive and negative
sentiments and is not limited to representing the negative sentiment of the
most attributed word. In the middle layers, the latent concepts evolved into
concepts that align better with the sentiment of the input sentence as can be
seen in Figure 2(b). For instance, the latent concept of Figure 2(b) shows a
mix of adjectives and adverbs of negative sentiment, i.e. aligned with the
sentiment of the input sentence. In the sentiment task, the most attributed
word in the last layer is [CLS] which resulted in latent concepts consisting
of [CLS] representations of the most related sentences to the input. In such
cases, we randomly pick five [CLS] instances from the latent concept and show
their corresponding sentences in the figure (see Figure 2(c)). We found that
the last layer’s latent concepts are best aligned with the input instance and
its prediction and are the most informative explanation of the prediction. In
the rest of the paper, we focus our analysis on the explanations generated
using the last layer only and perform a human evaluation to evaluate their
efficacy and correctness.

####  4.1.2 Analyzing Last Layer Explanations

Figure 3 presents various examples of LACOAT for both POS tagging and
Sentiment tasks using BERT. The _sentence_ is the input sentence, _prediction_
is the output of the model and _true label_ is the gold label. The
_explanation_ is the final output of LACOAT. _Cluster X_ is the latent concept
aligned with the most salient word representation at the 12th layer and X is
the cluster ID. For sentiment, we randomly pick five [CLS] instances from the
latent concept and show their corresponding sentences in the figure.

##### Correct prediction with correct gold label

Figures 3(a) and 3(c) present a case of correct prediction with latent-concept
explanation and human-friendly explanation. The former are harder to interpret
especially in the case of sentence-level latent concepts as in Figure 3(a)
compared to latent concepts consisting of words (Figure 3(c)). However, in
both cases, PlausiFyer highlights additional information about the relation
between the latent concept and the input sentence. For example, it captures
that the adverbs in Figure 3(c) have common semantics of showing degree or
frequency. Similarly, it highlights that the reason of positive sentiment in
3(a) is due to praising different aspects of a film and its actors and
actresses.

##### Wrong prediction with correct gold label

Figures 3(b) and 3(d) show rather interesting scenarios where the predicted
label is wrong. In Figure 3(b), the input sentence has a negative sentiment
but the model predicted it as positive. The instances of latent concepts show
sentences with mixed sentiments such as “manages to charm” and “epitome of
twist endings” is positive, and “never quite lives up to its promise” is
negative. This provides the domain expert an evidence of a possible wrong
prediction. The PlausiFyer’s _explanation_ is even more helpful as it clearly
states that “there is no clear … relation between these sentences …".
Similarly, in the case of POS (Figures 3(d)) while the prediction is Noun, the
majority of words in the latent concepts are plural Nouns, giving evidence of
a possibly wrong prediction. In addition, the _explanation_ did not capture
any morphological relationship between the concept and the input word.

To study how the explanation would change if it is a correct prediction, we
employ TextAttack Morris et al. (2020) to create an adversarial example of the
sentence in Figure 3(b) that flips its prediction. The new sentence replaces
“laughing” with “kidding” which has a similar meaning but flipped the
prediction to a correct prediction. Figure 6 in the appendix shows the full
explanation of the augmented sentence. With the correct prediction, the latent
concept changed and the explanation clearly expresses a negative sentiment “…
all express negative opinions and criticisms …" compared to the explanation of
the wrongly predicted sentence.

##### Cross model analysis

LACOAT provides an opportunity to compare various models in terms of how they
learned and structured the knowledge of a task. Figure 4 compares the
explanation of RoBERTa (top) and XLMR (bottom) for identical inputs. Both
models predicted the correct label. However, their latent concept based
explanation is substantially different. RoBERTa’s explanation shows a large
and diverse concept where many words are related to finance and economics. The
XLMR’s latent concept is rather a small focused concept where the majority of
tokens are units of measurement. It is worth noting that both models are fine-
tuned on identical data.

Figure 4: Comparing explanation of RoBERTa (top) and XLMR (bottom)

####  4.1.3 Comparison to Input Feature based Explanation

In Figure 2, layer 0 serves as a direct comparison to the explanation provided
by the attribution methods. Here, IG highlights the input word “worst" in the
given example. While it informs the user of the most salient input token that
effects the prediction, it does not provide insights into the particular facet
the model has used in that particular example. Even if we compare the
explanation of other layers like layer 6, the latent concept of the attributed
word is more informative than the attributed word alone.

###  4.2 Human Evaluation

To evaluate the explanation generated by LACOAT empirically, we conduct a
human evaluation using four annotators across 50 test samples. Specifically,
given an explanation (e.g. Figure 3), three annotators are asked to answer the
following five questions:

  1. 1.

Regardless of the prediction, can you see any relation between the original
input and the concept used by the model? (Yes/No)

  2. 2.

Given the prediction, does the latent concept help you understand why the
model made that prediction? (Helps/Neutral/Hinders)

  3. 3.

Given the prediction, does the explanation help you understand why the model
made that prediction? (Helps/Neutral/Hinders)

  4. 4.

Does the explanation accurately describe the latent concept? (Yes/No)

  5. 5.

Is the explanation relevant to the task at hand? (Yes/No)

Q1 evaluates whether LACOAT attributes the correct concept to a given
prediction, while Q2 and Q3 measure the efficacy of LACOAT’s output in helping
a user understand the prediction. Q4 and Q5 evaluates the output of
PlausiFyer. They specifically separate out the cases where the explanation was
accurate but irrelevant to the task at hand.

Table 1 shows the consolidated labels by picking the majority label in case of
Yes/No questions and averaging the annotations in case of the rest. The
evaluation shows that the latent concept itself was not only relevant to the
task at hand, but also helped the user understand the model’s prediction. The
results for the helpfulness of the explanation text were mixed, with the
majority of the annotations stating that it did not help or hinder their
process. However upon inspection, we see that the explanation was mostly
helpful in all the cases where the model made the correct prediction, and not
helpful when the prediction was incorrect. Qualitatively analyzing the
explanation text for incorrect prediction shows that PlausiFyer mostly outputs
“There is no relationship between the sentences and the concepts”, which was
deemed as hindering by most of the annotators. While such an explanation may
serve as an indicator of a potential problem in the prediction, improving the
prompt may result in a response that is indicative of the issue with the
prediction. We leave this exploration for the future. Table 1 also shows the
agreement between the annotators using Fleiss’ Kappa. Since not all samples
were annotated by all annotators, we compute the average Fleiss’ kappa of each
annotator with the consolidated annotation. The agreement ranges from Fair to
Substantial across the five questions.

| Labels | Correct Samples | Incorrect Samples | All Samples
---|---|---|---|---
| Annotation | Fleiss $\kappa$
Q1 | Yes/No | 28 / 0 | 20 / 2 | 48 / 2 | 0.35
Q2 | Helps/Neutral/Hinders | 27 / 1 / 0 | 17 / 5 / 0 | 44 / 6 / 0 | 0.41
Q3 | Helps/Neutral/Hinders | 16 / 10 / 2 | 1 / 19 / 2 | 17 / 29 / 4 | 0.61
Q4 | Yes/No | 17 / 11 | 5 / 17 | 22 / 28 | 0.47
Q5 | Yes/No | 17 / 11 | 6 / 16 | 23 / 27 | 0.80

Table 1: Consolidated label distribution for Q1-Q5. Fleiss’ $\kappa$ scores
are computed by averaging each annotator with the consolidated annotation. The
consolidated labels and agreement scores are shown for all the samples, as
well as partitioned into those where the model prediction was
correct/incorrect.

###  4.3 Module Specific Evaluation

| POS | Sentiment
---|---|---
Layers | BERT | RoBERTa | BERT | RoBERTa
9 | 92.38 | 86.97 | 31.94 | 99.59
10 | 92.79 | 89.64 | 99.57 | 99.69
11 | 93.39 | 89.95 | 99.71 | 99.48
12 | 93.95 | 90.04 | 99.25 | 99.27
Table 2: Accuracy of PredictionAttributor in mapping a representation to the
correct latent concept. See Appendix Table 7, 8, 9 for complete results.

The correctness of LACOAT depends on the performance of each module it
comprised off. The ideal way to evaluate the efficacy of these modules is to
consider gold annotations. However, they are not available for any module. To
mitigate this limitation, we design various constrained scenarios where
certain assumptions can be made about the representations of the model. For
example, the POS model optimizes POS tags so it is highly probable that the
last layer representations form latent concepts that are a good representation
of POS tags as suggested by various previous works Kovaleva et al. (2019);
Durrani et al. (2022). One can assume that for ConceptDiscoverer, the last
layer latent concepts will form groupings of words based on specific tags and
for PredictionAttributor, the input word at the position of the predicted tag
should reside in a latent concept that is dominated by the words with the same
tag. In the following, we evaluate the correctness of these assumptions.

Latent Concept Annotation For the sake of evaluation, we annotated the latent
concepts automatically using the class labels of each task. Given a latent
concept, we annotate it with a certain class if more than 90% of the words in
the latent concept belong to that class. In the case of POS, the latent
concepts will be labeled with one of the 44 tags. For sentiment, the class
labels, Positive and Negative, are at sentence level. We tag a latent concept
as Positive/Negative if 90% of its tokens ([CLS] or words) belong to sentences
labeled as Positive/Negative in the training data. The latent concepts that do
not fulfill the criteria of 90% for any class are annotated as Mixed.

####  4.3.1 ConceptDiscoverer

ConceptDiscoverer identifies latent concepts by clustering the representation.
We questioned whether the discovered latent concepts are a true reflection of
the properties that a representation possesses. Using ConceptDiscoverer, we
form latent concepts of the last layer and automatically annotate them as
described above. We found 87%, 83% and 86% of the latent concepts of BERT,
RoBERTa and XLMR that perfectly map to a POS tag respectively. We further
analyzed other concepts where 90% of the words did not belong to a single tag.
We found them to be of compositional nature i.e. a concept consisting of
related semantics like a mix of adjectives and proper nouns about countries
such as Swedish and Sweden (Appendix Figure 9). For sentiment, we found 78%,
95% and 94% of the latent concepts of BERT, RoBERTa and XLMR to consist of
either Positive or Negative sentences. The high number of class-based clusters
of RoBERTa and XLMR show that at 12th layer, majority of their latent space is
separated based on these two classes (see Table 6 for detailed results).

Layers | 0 | 2 | 5 | 10 | 12
---|---|---|---|---|---
POS | Top 1 | 100 | 100 | 99.03 | 92.67 | 84.19
| Top 2 | 100 | 100 | 99.75 | 97.89 | 94.15
| Top 5 | 100 | 100 | 99.94 | 99.68 | 99.05
Sentiment | Top 1 | 100 | 100 | 97.19 | 83.09 | 68.24
| Top 2 | 100 | 100 | 99.63 | 92.67 | 83.24
| Top 5 | 100 | 100 | 99.94 | 97.75 | 94.24
Table 3:  BERT: Accuracy of ConceptMapper in mapping a representation to the
correct latent concept. See Table 10, 11 in the Appendix for results on all
layers.

####  4.3.2 PredictionAttributor

We questioned whether the salient input representation correctly represents
the latent space of the output. This specifically evaluates
PredictionAttributor. We calculate the number of times the representation of
the most salient word/[CLS] token maps to the latent concept of the identical
label as that of the prediction. We expect a high alignment at the top layers
for PredictionAttributor to be correct. We do not include ConceptMapper when
evaluating this and conduct the experiment using the training data only where
we already know the alignment of a salient representation and the latent
concept. Table 2 shows the results across the last four layers (See Appendix
Tables 7, 8, 9 for full results). For POS, we observed a successful match of
93.95%, 90.04% and 93.13% for BERT, RoBERTa and XLMR respectively. We observed
the mismatched cases and found them to be also of a compositional nature i.e.
latent concepts comprised of semantically related words (see Appendix Figure 9
for examples).

For sentiment, more than 99% of the time, the last layer’s salient
representation maps to the predicted class label, confirming the correctness
of PredictionAttributor. The performance drop for the lower layer is due to
the absence of class-based latent concepts in the lower layers i.e. concepts
that comprised more than 90% of the tokens belonging to sentences of one of
the classes.

####  4.3.3 ConceptMapper

Here, we evaluate the correctness of ConceptMapper in mapping a test
representation to the training data latent concepts. ConceptMapper trains
using representations and their cluster ids as labels. For every layer, we
randomly split this training data into 90% train and 10% test data. Here, the
test data serves as the gold standard annotation of latent concepts. We train
ConceptMapper using the training instances and measure the accuracy of the
test instances. Table 3 presents the accuracy of the POS and Sentiment tasks
using BERT (See Appendix Tables 10, 11 for results of other models). Observing
Top-1 accuracy, the performance of ConceptMapper starts high (100%) for lower
layers and drops to 84.19 and 68.24% for the last layer. We found that the
latent space becomes dense on the last layer. This is in line with Ethayarajh
(2019) who showed that the representations of higher layers are highly
anisotropic. This causes several similar concepts close in the space. If true,
the correct label should be among the top predictions of the mapper. We
empirically tested it by considering the top two and top five predictions of
the mapper, achieving a performance of up to 99.05% and 94.24% for the POS and
Sentiment tasks respectively.

##  5 Related work

The explainability methods can be approached by local explanations and global
explanations Madsen et al. (2023); Sundararajan et al. (2017a); Denil et al.
(2014); Selvaraju et al. (2020); Kapishnikov et al. (2021); Zhao and Aletras
(2023); Kim et al. (2018); Ghorbani et al. (2019); Jourdan et al. (2023); Zhao
et al. (2023); Ribeiro et al. (2016). Lyu et al. (2023) provides a
comprehensive survey on explainability methods in NLP. LACOAT is a local
explanation method providing post-hoc explanations given an input instance.
One of the most common ways for local explanations is to interpret the model
prediction based on the input features. However, the shortcoming of such an
explanation is the lack of contextual verbosity, which could not interpret the
multifaceted roles of the input features.

A number of works attempted to explain and interpret NLP models using human-
defined concepts Kim et al. (2018) and concepts extracted from hidden
representations Zhao et al. (2023); Ghorbani et al. (2019); Rajani et al.
(2020). Zhao et al. (2023); Kim et al. (2018) worked on the global explanation
based on a surrogate model. Zhao et al. (2023) trained the surrogate model
using on two optimization criteria i.e. auto encoding loss to stay faithful to
the original model distribution and impact of the latent concept to
prediction. Different from them, we provide local explanations and we ensure
the faithfulness of latent concepts to the model by extracting them directly
from the hidden representation without any supervised training. Rajani et al.
(2020) used k-nearest neighbors of the training data for low-confidence
predictions and showed them to be useful in revealing erroneous correlations
and misclassified instances and enhancing the performance of the finetuned
model. Dalvi et al. (2022) analyzed latent concepts of pretrained models in
terms of their ability to represent linguistic knowledge. Our
ConceptDiscoverer module is motivated from them. However, we propose a method
to explain a model’s prediction using training data latent concepts.

##  6 Conclusion

We presented LACOAT that provides a human-friendly explanation of a model’s
prediction. The qualitative evaluation and human evaluation showed that LACOAT
explanations are insightful in explaining a correct prediction, in
highlighting a wrong prediction and in comparing the explanations of models.
The reliance on training data latent space enables interpreting how knowledge
is structured in the network. Similarly, it enables the study of the evolution
of predictions across layers. LACOAT promises human-in-the-loop in the
decision-making process and is a step towards trust in AI.

##  7 Limitations

A few limitations of LACOAT are: 1) while hierarchical clustering is better
than nearest neighbor in discovering latent concepts as established by Dalvi
et al. (2022), it has computational limitations and it can not be easily
extended to a corpus of say 1M tokens. However, the assumptions that are taken
in the experimental setup e.g. considering the maximum 20 occurrences of a
word (supported by Dalvi et al. (2022)) work well in practice in terms of
limiting the number of tokens and covering all facets of a majority of the
words. Moreover, the majority of the real-world tasks have limited task-
specific data and LACOAT can effectively be applied in such cases.

2) LACOAT requires white-box access to the training data and the model for
explanation which may not be available for large language models such as Llama
and ChatGPT. For open-source models like Llama and Mistral, LACOAT can be
applied in their task-specific usage. For instance for sentiment
classification using Llama, the training data of sentiment can be used to
extract the latent space of Llama for the sentiment classification task.
However, the applicability of LACOAT for generative tasks is limited due to
the open vocabulary.

## References

  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451. Association for Computational Linguistics.
  * Dalvi et al. (2022) Fahim Dalvi, Abdul Rafae Khan, Firoj Alam, Nadir Durrani, Jia Xu, and Hassan Sajjad. 2022.  Discovering latent concepts learned in BERT.  In _International Conference on Learning Representations_.
  * Denil et al. (2014) Misha Denil, Alban Demiraj, and Nando de Freitas. 2014.  Extraction of salient sentences from labelled documents.  _CoRR_ , abs/1412.6815.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , NAACL-HLT ’19, pages 4171–4186, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
  * Durrani et al. (2022) Nadir Durrani, Hassan Sajjad, Fahim Dalvi, and Firoj Alam. 2022.  On the transformation of latent space in fine-tuned NLP models.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 1495–1516, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Ethayarajh (2019) Kawin Ethayarajh. 2019.  How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 55–65, Hong Kong, China. Association for Computational Linguistics.
  * Ghorbani et al. (2019) Amirata Ghorbani, James Wexler, James Zou, and Been Kim. 2019.  Towards Automatic Concept-based Explanations.  ArXiv:1902.03129 [cs, stat].
  * Gowda and Krishna (1978) K Chidananda Gowda and G Krishna. 1978.  Agglomerative clustering using the concept of mutual nearest neighbourhood.  _Pattern recognition_ , 10(2):105–112.
  * Jourdan et al. (2023) Fanny Jourdan, Agustin Picard, Thomas Fel, Laurent Risser, Jean Michel Loubes, and Nicholas Asher. 2023.  COCKATIEL: COntinuous Concept ranKed ATtribution with Interpretable ELements for explaining neural net classifiers on NLP tasks.  ArXiv:2305.06754 [cs, stat].
  * Kapishnikov et al. (2021) Andrei Kapishnikov, Subhashini Venugopalan, Besim Avci, Ben Wedin, Michael Terry, and Tolga Bolukbasi. 2021.  Guided Integrated Gradients: An Adaptive Path Method for Removing Noise.  ArXiv:2106.09788 [cs].
  * Kim et al. (2018) Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and Rory Sayres. 2018.  Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV).  ArXiv:1711.11279 [stat].
  * Kovaleva et al. (2019) Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. 2019.  Revealing the dark secrets of BERT.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4365–4374, Hong Kong, China. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A robustly optimized BERT pretraining approach.  _ArXiv:1907.11692_.
  * Lyu et al. (2023) Qing Lyu, Marianna Apidianaki, and Chris Callison-Burch. 2023.  Towards Faithful Model Explanation in NLP: A Survey.  ArXiv:2209.11326 [cs].
  * Madsen et al. (2023) Andreas Madsen, Siva Reddy, and Sarath Chandar. 2023.  Post-hoc Interpretability for Neural NLP: A Survey.  _ACM Computing Surveys_ , 55(8):1–42.  ArXiv:2108.04840 [cs].
  * Marcus et al. (1993) Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993.  Building a large annotated corpus of English: The Penn Treebank.  _Computational Linguistics_ , 19(2):313–330.
  * Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013.  Efficient estimation of word representations in vector space.  In _Proceedings of the ICLR Workshop_ , Scottsdale, AZ, USA.
  * Morris et al. (2020) John Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. 2020.  Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 119–126.
  * Pang and Lee (2004) Bo Pang and Lillian Lee. 2004.  A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts.  In _Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics_ , ACL ’04, page 271–es, USA. Association for Computational Linguistics.
  * Rajani et al. (2020) Nazneen Fatema Rajani, Ben Krause, Wengpeng Yin, Tong Niu, Richard Socher, and Caiming Xiong. 2020.  Explaining and improving model behavior with k nearest neighbor representations.
  * Reif et al. (2019) Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B Viegas, Andy Coenen, Adam Pearce, and Been Kim. 2019.  Visualizing and measuring the geometry of bert.  In _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc.
  * Ribeiro et al. (2016) Marco Ribeiro, Sameer Singh, and Carlos Guestrin. 2016.  “why should I trust you?”: Explaining the predictions of any classifier.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations_ , pages 97–101, San Diego, California. Association for Computational Linguistics.
  * Selvaraju et al. (2020) Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2020.  Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization.  _International Journal of Computer Vision_ , 128(2):336–359.  ArXiv:1610.02391 [cs].
  * Shrikumar et al. (2017) Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. 2017.  Learning important features through propagating activation differences.  In _Proceedings of the 34th International Conference on Machine Learning - Volume 70_ , ICML’17, page 3145–3153. JMLR.org.
  * Simonyan et al. (2014) Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2014.  Deep inside convolutional networks: Visualising image classification models and saliency maps.  In _Workshop at International Conference on Learning Representations_.
  * Sundararajan et al. (2017a) Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017a.  Axiomatic Attribution for Deep Networks.  ArXiv:1703.01365 [cs].
  * Sundararajan et al. (2017b) Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017b.  Axiomatic attribution for deep networks.  In _Proceedings of the 34th International Conference on Machine Learning_ , volume 70 of _Proceedings of Machine Learning Research_ , pages 3319–3328. PMLR.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Zaidan and Eisner (2008) Omar Zaidan and Jason Eisner. 2008.  Modeling annotators: A generative approach to learning from annotator rationales.  In _Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing_ , pages 31–40, Honolulu, Hawaii. Association for Computational Linguistics.
  * Zhao et al. (2023) Ruochen Zhao, Shafiq Joty, Yongjie Wang, and Tan Wang. 2023.  Explaining Language Models’ Predictions with High-Impact Concepts.  ArXiv:2305.02160 [cs].
  * Zhao and Aletras (2023) Zhixue Zhao and Nikolaos Aletras. 2023.  Incorporating Attribution Importance for Improving Faithfulness Metrics.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 4732–4745, Toronto, Canada. Association for Computational Linguistics.

##  Appendix A Salient Representation Extraction - Position Based Attribution
Method

This strategy uses the position of the output head as an indication of the
most salient contextual representation. For instance,

  * •

In the case of sequence classification, the representation of the [CLS] token,
$\overrightarrow{z}_{\texttt{[CLS]}}$ (or a model’s representative
classification token) will be considered as the most salient representation.

  * •

In the case of masked token prediction, the representation of the [MASK] token
($\overrightarrow{z}_{\texttt{MASK}}$) will be considered as the most salient
for making the prediction.

  * •

In the case of sequence labeling, the representation at the time step of the
prediction will be used. For example, in the case of POS tagging, for the
prediction of a tag of the word love in the sentence I [love] soccer, the
second time step’s representations ($\overrightarrow{z}_{w_{2}}$) will be
used.

##  Appendix B Finetuning Performance

We tuned several transformers BERT-base-cased, RoBERTa and XLM-RoBERTa. We
used standard splits for training, development and test data that we used to
carry out our analysis. The splits to preprocess the data are available
through git repository111https://github.com/nelson-liu/contextual-repr-
analysis. See Table 4 and Table 5 for statistics and classifier accuracy.

Task | Train | Dev | Test | Tags | BERT | RoBERTa | XLM-R
---|---|---|---|---|---|---|---
POS | 36557 | 1802 | 1963 | 48 | 96.81 | 96.70 | 96.75

Table 4: The fine-tuned performance of models, data statistics (number of
sentences) on training, development, and test sets used in the experiments,
and the number of tags to be predicted for the POS tagging task. Model: BERT,
RoBERTa, XLM-R

Task | Train | Dev | Test | Tags | BERT | RoBERTa | XLM-R
---|---|---|---|---|---|---|---
Sentiment | 13878 | 1516 | 2726 | 2 | 94.53 | 96.31 | 93.80

Table 5: The fine-tuned performance of models, data statistics (number of
sentences) on training, development, and test sets used in the experiments,
and the number of tags to be predicted for the sentiment classification task.
Model: BERT, RoBERTa, XLM-R

##  Appendix C Qualitative Evaluation - More Examples

###  C.1 Example for the Evolution of Concepts

5 presents the other example of latent concepts of the salient words in layer
0, 6, and 12. Similarly to the example shown in 2, the latent concept of this
example shows that the different forms of the verb “sit” and are not aligned
with its usage in the input instance. The concept in the middle layer aligns
better with the sentiment of the input sentence(Figure 5(b)). The most words
of layer 6’s latent concept match the sentiment of the input sentence. We also
randomly pick five [CLS] instances from the latent concept and show their
corresponding sentences in the figure (see Figure 5(c)). The concept of the
last layer is best aligned with the input sentence.

Figure 5: Sentiment task: Examples of the latent concepts of the most
attributed words in Layers 0, 6 and 12

###  C.2 Adversarial Example of the Sentence in Figure 3(b)

The augmented sentence has a similar meaning word “kidding” instead of
“laughing” (See Figure 6). The predicted label of the sentence becomes
Positive, which is matched to the gold label. The latent concept of the
“kidding” is more aligned with the sentence than the original one.

Figure 6: An augmented example for the test instance in Figures 3(b): The
augmented sentence replaced the “laughing” with “kidding” which has a similar
meaning. The label of the augmented sentence becomes positive, which is
matched with the gold label. The new predicted latent concept is more closely
aligned with the main sentence. The model may not learn the implicit meaning
of the “laughing stock” in the sentence.

###  C.3 Correct Predicted Label with Incorrect Gold Label

The automatic labeling of latent concepts based on the model’s class provides
an opportunity to analyze the wrong predictions of the model with respect to
the concept labels. We specifically observe the wrong predictions of test
instances. We discovered that many of the wrong prediction cases were not
caused by misclassification of the models but were due to the fact that the
gold label was annotated incorrectly. Figure 7 shows an example in which the
main sentence and the explanation sentence share the same sentiment. We can
see that the sentence provides critiques of the different aspects of the film.
But the gold label of this sentence is positive. We think the gold label for
this sentence is incorrect.

Figure 7: A correct prediction but incorrect gold label: The test instance
emphasizes the movie’s shortcomings and uses the word "especially" to
highlight the flaws. The explanation is rather long but it correctly
highlights that the sentences are about “critiques or opinions"

###  C.4 Incorrect Prediction in POS tagging Task

Figure 8 presents an incorrect prediction in the POS tagging task. The
prediction is aligned with a mixed concept that consists of nouns and
adjectives. According to the latent concept explanation, we know that the
model may not learn to distinguish the “noun” and “adjective”, which causes
the incorrect prediction.

Figure 8: An incorrect prediction (noun vs adjective) based on a latent concept made up of a mixture of nouns and adjectives: the “deputy” in this case is an adjective. The prediction aligns with a mixed cluster that contains both nouns and adjectives and the model may not learn to distinguish the “noun” and “adjective” in this case. The latent concept explanation is useful for the user to know that the model has used a mixed latent space for the prediction. The Explanation is rather wrong since it mentions that all these words are nouns.  | Sentiment
---|---
| BERT | RoBERTa | XLM-R
Layer | Neg | Pos | Mix | Neg | Pos | Mix | Neg | Pos | Mix
Layer 0 | 49 | 1 | 350 | 45 | 0 | 355 | 55 | 0 | 345
Layer 1 | 53 | 1 | 346 | 50 | 0 | 350 | 58 | 0 | 342
Layer 2 | 51 | 1 | 348 | 49 | 0 | 351 | 62 | 0 | 338
Layer 3 | 53 | 0 | 347 | 60 | 0 | 340 | 62 | 0 | 338
Layer 4 | 57 | 0 | 343 | 52 | 0 | 348 | 69 | 0 | 331
Layer 5 | 56 | 0 | 344 | 51 | 0 | 349 | 68 | 0 | 332
Layer 6 | 57 | 0 | 343 | 45 | 1 | 354 | 59 | 1 | 340
Layer 7 | 51 | 0 | 349 | 56 | 2 | 342 | 68 | 0 | 332
Layer 8 | 49 | 0 | 351 | 116 | 25 | 259 | 71 | 0 | 329
Layer 9 | 66 | 4 | 330 | 226 | 126 | 48 | 82 | 7 | 311
Layer 10 | 125 | 31 | 244 | 235 | 140 | 25 | 257 | 92 | 51
Layer 11 | 174 | 49 | 177 | 258 | 120 | 22 | 256 | 110 | 34
Layer 12 | 230 | 81 | 89 | 254 | 126 | 20 | 105 | 270 | 25
Table 6: Number of clusters for each polarity: “Neg” for negative Label, “Pos”
for positive, and “Mix” for mix label. The total number of clusters is 400.

##  Appendix D Module Specific Evaluation

###  D.1 ConceptDiscoverer \- Compositional Concept Examples

We found that the concepts are not always formed aligning to the output class.
Some concepts are formed by combining words from different classes. For
example in Figure 9(a), the concept is composed of nouns (specifically
countries) and adjectives that modify these country nouns. Similarly, Figure
9(b) describes a concept composed of different forms of verbs.

(a)

(b)

Figure 9: Compositional concepts: (a) A cluster representing countries (NNP)
and their adjectives (JJ), (b) Different form of verbs (Gerunds, Present and
Past participles).

###  D.2 ConceptDiscoverer \- Number of Clusters For Each Polarity in the
Sentiment Classification Task

Table 6 provides the number of clusters for each polarity in the sentiment
classification task. It shows that the majority of latent concepts are class-
based clusters at the last layer for the BERT, RoBERTa, and XLMR models.

###  D.3 ConceptMapper \- Position Based Attribution Method

We evaluate the module that is based on the position based attribution
approach as the number of times an input representation at the position of the
output head maps to the latent concept that is annotated with an identical
label as the output. For example, consider that the model predicts Proper Noun
(PN) for the input word "Trump". In order for the input representation of the
predicted label to be aligned with the latent concept, the representation of
the word "Trump" on at least the last layer should be in a cluster of words
whose label is PN.222We labeled concepts with a tag if 90% of the words in the
concept belongs to one class. Similarly for sentiment classification, we
expect the [CLS] representation on the last layer to map to a latent concept
that is dominated by the same class as the prediction.

For POS tagging task, the salient representation is identical for both the
position based and saliency based methods and results in the same performance
(Table 7). For the sentiment classification task, performance drops and even
reaches zero as in the case of XLMR. This may be due to the position-based
method which fails to find the right latent concept when the most attributed
word is different from the position of the output head (Table 8).

| POS
---|---
Layer | BERT | RoBERTa | XLM-R
Layer 0 | 16.81 | 14.29 | 17.66
Layer 1 | 17.79 | 16.49 | 18.89
Layer 2 | 21.16 | 20.18 | 20.71
Layer 3 | 22.79 | 20.13 | 31.03
Layer 4 | 29.70 | 24.65 | 40.51
Layer 5 | 46.74 | 29.26 | 60.31
Layer 6 | 73.19 | 42.38 | 77.32
Layer 7 | 84.52 | 57.46 | 85.78
Layer 8 | 90.68 | 82.84 | 89.41
Layer 9 | 92.38 | 86.97 | 91.97
Layer 10 | 92.79 | 89.64 | 92.64
Layer 11 | 93.39 | 89.95 | 92.59
Layer 12 | 93.95 | 90.04 | 93.13
Table 7: Position/Saliency-based method: accuracy of PredictionAttributor in mapping a representation to the correct latent concept in the POS tagging task. | Sentiment
---|---
Layer | BERT | RoBERTa | XLM-R
Layer 0 | 0 | 0 | 0
Layer 1 | 0 | 0 | 0
Layer 2 | 0 | 0 | 0
Layer 3 | 0 | 0 | 0
Layer 4 | 0 | 0 | 0
Layer 5 | 0 | 0 | 0
Layer 6 | 0 | 0 | 0
Layer 7 | 0 | 0 | 0
Layer 8 | 0 | 99.11 | 0
Layer 9 | 37.09 | 98.45 | 0
Layer 10 | 99.55 | 99.14 | 0
Layer 11 | 99.82 | 99.27 | 99.17
Layer 12 | 99.25 | 99.27 | 99.08
Table 8: Position-based method: accuracy of PredictionAttributor in mapping a representation to the correct latent concept in the Sentiment Classification task. The reason of zero values is that the position-based method fails to find the right latent concept when the most attributed word is different from the position of the output head. | Sentiment
---|---
Layer | BERT | RoBERTa | XLM-R
Layer 0 | 6.40 | 12.08 | 7.46
Layer 1 | 7.12 | 12.46 | 5.57
Layer 2 | 7.66 | 17.29 | 6.36
Layer 3 | 7.13 | 22.00 | 8.03
Layer 4 | 12.18 | 20.08 | 9.71
Layer 5 | 13.24 | 24.25 | 8.88
Layer 6 | 11.18 | 17.26 | 8.75
Layer 7 | 12.80 | 39.87 | 14.05
Layer 8 | 4.06 | 92.84 | 15.75
Layer 9 | 31.94 | 99.59 | 32.63
Layer 10 | 99.57 | 99.69 | 92.06
Layer 11 | 99.71 | 99.48 | 94.97
Layer 12 | 99.25 | 99.27 | 99.08
Table 9: Saliency-based method: accuracy of PredictionAttributor in mapping a
representation to the correct latent concept in the sentiment classification
task. The reason of very low values for the lower layers is mainly due to the
absence of class-based latent concepts in the lower layers i.e. concepts that
comprised more than 90% of the tokens belonging to sentences of one of the
classes.

###  D.4 ConceptMapper \- Accuracy of ConceptMapper for the Sentiment
Classification and POS Tagging task

We validate ConceptMapper by measuring the accuracy of the test instances for
both the sentiment classification and POS tagging tasks based on the BERT,
RoBERTa, and XLMR models. The top 1, 2, and 5 accuracy of ConceptMapper in
mapping a representation to the correct latent concept for each layer is shown
in Table 10 and Table 11. For all models, the performance of the top-5 is
above 99% for the POS tagging task and above 90% for the sentiment
classification task.

| POS
---|---
| BERT | RoBERTa | XLM-R
Layer | Top-1 | Top-2 | Top-5 | Top-1 | Top-2 | Top-5 | Top-1 | Top-2 | Top-5
Layer 0 | 100 | 100 | 100 | 99.91 | 99.95 | 99.98 | 99.99 | 100 | 100
Layer 1 | 100 | 100 | 100 | 99.92 | 99.94 | 99.98 | 100 | 100 | 100
Layer 2 | 100 | 100 | 100 | 99.76 | 99.92 | 99.98 | 99.72 | 99.98 | 100
Layer 3 | 99.85 | 99.98 | 100 | 99.38 | 99.85 | 99.98 | 98.25 | 99.60 | 99.98
Layer 4 | 99.72 | 99.92 | 99.97 | 98.67 | 99.58 | 99.87 | 97.72 | 99.60 | 99.98
Layer 5 | 99.03 | 99.75 | 99.94 | 97.69 | 99.15 | 99.73 | 97.05 | 99.23 | 99.91
Layer 6 | 97.76 | 99.34 | 99.83 | 96.52 | 98.71 | 99.59 | 95.8 | 98.95 | 99.76
Layer 7 | 96.51 | 98.91 | 99.68 | 94.72 | 98.11 | 99.57 | 93.92 | 98.31 | 99.80
Layer 8 | 95.27 | 98.52 | 99.79 | 92.56 | 97.55 | 99.52 | 94.20 | 98.52 | 99.80
Layer 9 | 94.54 | 98.25 | 99.70 | 92.24 | 97.48 | 99.55 | 92.79 | 97.82 | 99.73
Layer 10 | 92.67 | 97.89 | 99.68 | 91.61 | 97.19 | 99.55 | 92.03 | 97.66 | 99.60
Layer 11 | 90.86 | 97.34 | 99.64 | 90.72 | 96.77 | 99.58 | 90.40 | 97.28 | 99.67
Layer 12 | 84.19 | 94.15 | 99.05 | 86.88 | 95.13 | 99.15 | 85.07 | 94.57 | 99.08
Table 10: Top 1, 2, and 5 accuracy of ConceptMapper in mapping a representation to the correct latent concept for the POS tagging task. The top-5 performance reaches above 99% for all models demonstrating that the correct latent concept is among the top probable latent concepts of ConceptMapper. | Sentiment
---|---
| BERT | RoBERTa | XLM-R
Layer | Top-1 | Top-2 | Top-5 | Top-1 | Top-2 | Top-5 | Top-1 | Top-2 | Top-5
0 | 100 | 100 | 100 | 99.95 | 100 | 100 | 100 | 100 | 100
1 | 100 | 100 | 100 | 99.86 | 99.98 | 100 | 100 | 100 | 100
2 | 100 | 100 | 100 | 99.89 | 99.98 | 100 | 99.9 | 100 | 100
3 | 98.80 | 100 | 100 | 99.44 | 99.83 | 99.96 | 99.57 | 99.99 | 100
4 | 97.84 | 99.85 | 99.99 | 99.28 | 99.73 | 99.91 | 99.4 | 99.96 | 100
5 | 97.19 | 99.63 | 99.94 | 98.4 | 99.5 | 99.84 | 99.12 | 99.84 | 99.96
6 | 96.44 | 99.30 | 99.89 | 97.35 | 99.15 | 99.82 | 98.9 | 99.84 | 99.96
7 | 94.86 | 98.97 | 99.90 | 96.13 | 98.74 | 99.63 | 98.22 | 99.62 | 99.9
8 | 93.26 | 97.99 | 99.67 | 87.42 | 95.14 | 98.43 | 98.13 | 99.48 | 99.84
9 | 90.42 | 96.97 | 99.20 | 75.38 | 88.14 | 96.07 | 96.37 | 98.77 | 99.66
10 | 83.09 | 92.67 | 97.75 | 65.84 | 81.13 | 93.46 | 89.12 | 95.2 | 98.61
11 | 76.84 | 88.02 | 96.01 | 65.91 | 81.36 | 93.43 | 70.99 | 84.31 | 94.18
12 | 68.24 | 83.24 | 94.24 | 70.83 | 84.54 | 95.67 | 55.3 | 75.08 | 91.74
Table 11: Top 1, 2, and 5 accuracy of ConceptMapper in mapping a
representation to the correct latent concept for the sentiment classification
task. The top-5 performance reaches above 90% for all models demonstrating
that the correct latent concept is among the top probable latent concepts of
ConceptMapper.


# Exposing the Unseen: Exposure Time Emulation for Offline Benchmarking of
Vision Algorithms

Olivier Gamache1, Jean-Michel Fortin, Matěj Boxan, François Pomerleau,
Philippe Giguère1 *This research was supported by Fonds de Recherche du Québec
Nature et technologies (FRQNT) Team grant 254912 and Natural Sciences and
Engineering Research Council of Canada (NSERC) DRC Grant through the grant CRD
538321-18, in collaboration with FP Innovations and Resolute Forest
Products.1Northern Robotics Laboratory, Université Laval, Québec City, Québec,
Canada<EMAIL_ADDRESS>and<EMAIL_ADDRESS>
###### Abstract

Visual Odometry (VO) is one of the fundamental tasks in computer vision for
robotics. However, its performance is deeply affected by High Dynamic Range
(HDR) scenes, omnipresent outdoor. While new Automatic-Exposure (AE)
approaches to mitigate this have appeared, their comparison in a reproducible
manner is problematic. This stems from the fact that the behavior of AE
depends on the environment, and it affects the image acquisition process.
Consequently, AE has traditionally only been benchmarked in an online manner,
making the experiments non-reproducible. To solve this, we propose a new
methodology based on an emulator that can generate images at any exposure
time. It leverages BorealHDR, a unique multi-exposure stereo dataset collected
over $8.4\text{\,}\mathrm{km}$, on 50 trajectories with challenging
illumination conditions. Moreover, it contains pose ground truth for each
image and a global 3D map, based on lidar data. We show that using these
images acquired at different exposure times, we can emulate realistic images
keeping a Root-Mean-Square Error (RMSE) below $1.78\text{\,}\mathrm{\char
37\relax}$ compared to ground truth images. To demonstrate the practicality of
our approach for offline benchmarking, we compared three state-of-the-art AE
algorithms on key elements of Visual Simultaneous Localization And Mapping
(VSLAM) pipeline, against four baselines. Consequently, reproducible
evaluation of AE is now possible, speeding up the development of future
approaches. Our code and dataset are available online at this link:
https://github.com/norlab-ulaval/BorealHDR

##  I Introduction

Cameras can capture high-resolution details of a scene, at high frame rates,
and in a cost-effective manner. Because of this, they are used in many robotic
applications, ranging from object detection to localization. One such
application is VO, which is the task of predicting the displacement of a
camera between consecutive images. It is the basis of many vision-based
localization algorithms, such as VSLAM, and improving such a task is still an
active field of research [1, 2]. In outdoor settings, the quick variations in
illumination level and HDR scenes can severely compromise the performances of
VO algorithms [3]. In one second, a car leaving a tunnel can experience an
illumination variation over $120\text{\,}\mathrm{dB}$ [4], while a standard
12-bit channel camera has a theoretical dynamic range of around
$72\text{\,}\mathrm{dB}$. A boreal forest in winter is another example of such
HDR environments, where the sun reflected on the snow is highly contrasted
with the darkness of the trees. This contrast inevitably leads to saturated
pixels on both ends of the spectrum, as highlighted by blue and red colorized
hues in Figure 1, thus resulting in valuable information loss for VO
algorithms.

Figure 1:  Upper image: Overhead view of a trajectory from BorealHDR dataset,
taken in Montmorency Forest, Québec, Canada. The traveled trajectory is shown
in purple. Possible emulated exposure times $\Delta t_{e}$ along the
trajectory are depicted in orange. Lower image: Acquired brackets (6), with
one used to generate $\Delta t_{e}=$9\text{\,}\mathrm{ms}$$ from the upper
image. The bracket exposure times $\Delta t_{b}$ are $\\{1,2,4,8,16,32\\}$
$\mathrm{ms}$, increasing the dynamic range of our capture by 30 dB or 5
stops. Blue and red colorized pixels are over- and under-exposed,
respectively.

Accordingly, researchers developed AE approaches to modify the camera exposure
parameters, such as exposure time and gain, during operation to reduce the
impact of the dynamic range on VO [5, 6, 7].

Unfortunately, given the online nature of exposure control, comparing each
method is very challenging [8, 9], as one needs to exactly replicate test
conditions for each run, including environment lighting and camera poses. One
of the developed comparison methods for AE algorithms is based on installing
on an acquisition platform as many cameras as the number of benchmarked
methods. With the increasing number of tested approaches, the experimental
setup quickly becomes challenging and costly. Another way is to repeat
multiple times the exact same trajectory for each benchmarked AE. In an
uncontrolled environment, such as the one illustrated in Figure 1, the dynamic
illumination changes, and the length of the trajectory would make it
impossible to have a replicable benchmark.

In this work, we propose a novel method for an equitable comparison between AE
algorithms applied to robot localization, based on the bracketing technique
[10] and realistic image emulation. As illustrated by $\Delta t_{e}$ in Figure
1, our emulator framework allows generating images at any desired exposure
time for recorded trajectories. This is possible due to our multi-exposure
dataset, called BorealHDR, recorded in winter conditions at the Montmorency
Forest in Québec, Canada. It is composed of 50 trajectories, totaling more
than 4 $\mathrm{h}$ of recordings over 8.4 $\mathrm{km}$. It also includes 3D
lidar scans and inertial measurements. The novelty of our dataset and our
emulation technique allows us to benchmark, in an offline manner, AE
algorithms, feature extraction and VO pipelines, as opposed to single-exposure
datasets. In short, our contributions are the following:

  1. 1.

A new comparison method for AE algorithms based on an emulation framework that
allows to generate realistic images at any plausible exposure time;

  2. 2.

A multimodal dataset that provides stereo images at multiple exposures, and 3D
map ground truths, in kilometer-scale HDR environments; and

  3. 3.

First offline benchmarking of seven AE algorithms on VO-based experiments.

##  II Related Work

We first review the existing methodologies for comparison of AE algorithms.
Then, we go over available datasets for VO algorithms and their limitations in
the context of testing AE, highlighting the advantages of our dataset.
Finally, we discuss important state-of-the-art AE algorithms that will be used
for the first benchmarking on a large VO dataset.

###  II-A Methodologies for AE Algorithms Comparison

The nature of AE algorithms makes their comparison very challenging, as they
actively change camera parameters during execution. Here, we present state-of-
the-art AE benchmarking methods that address this issue, and categorize them
between static and moving cameras approaches.

Static Cameras – One way to evaluate AE methods is to fix a camera and acquire
multiple pictures at different exposure times. [11] used a surveillance camera
to collect images with 210 combinations of exposure parameters, six times a
day, to compare their new AE method with others. [5] developed an 18 real-
world static scenes dataset, where images at varying exposure times were
collected in each scene. [12] collected a dataset made of 1000 images with
multiple different exposures, covering indoor scenes and outdoor scenes. [8]
were the only ones using a stereo camera to collect a static scene dataset.
Multi-exposed static scene images allow comparing some proxies of VO such as
single image feature detection. However, complete trajectories, as proposed in
our multi-exposure dataset, are required to benchmark AE algorithms on
complete VO pipelines.

Moving Cameras – We mainly observe three trends using cameras in motion to
compare AE algorithms: simulation-based methods, multi-camera methods, and
multi-trajectory methods. An example of _simulation-based methods_ was used by
[5], which used the Multi-FoV synthetic dataset [13], to simulate multiple
versions of an image, but with different exposition levels. Similarly, [14]
trained a neural network to produce images with higher gradient information
using a synthetic dataset, where they simulated 12 different exposures.
Although these techniques reduce dataset development time, they face the
Sim2Real gap [15], in opposition to our approach, which relies only on real-
world images. _Multi-camera methods_ are the most widespread for AE algorithms
comparison, leveraging a hardware solution for effective data collection. In
this setup, two or three cameras are typically installed on a mobile platform
[5, 9, 6, 16]. This allows exact comparison since all AE methods are executed
simultaneously, thus facing the same conditions. [2] used four stereo cameras,
facilitating VO comparison, as the stereo depth estimation provides scaling
information. An important drawback for these multi-camera methods is that the
hardware complexity grows with the number of AE methods tested, rapidly
becoming impractical. On the contrary, our approach can be used to compare on
an unlimited number of AE methods, at a fixed acquisition cost. _Multi-
trajectory methods_ consist of repeating multiple times the exact same
trajectory using a different exposure control scheme at each iteration. It was
used by [7] to develop a gradient AE technique, which was tested using two
cameras installed on a motorized $0.16\text{\,}{\mathrm{m}}^{2}$ motion table.
This allowed for exact repetitions of the same trajectories multiple times,
with a ground truth precision near $0.1\text{\,}\mathrm{mm}$. To evaluate AE
methods in the presence of motion blur, [1] acquired images from three cameras
in motion on a motorized rail. While providing precise ground truth, the rail
approach severely restrains the total area covered by the trajectories. It
also limits the number of observable environments. A variation of the multi-
trajectory method was developed by [6]. They drove using stop-and-go maneuvers
on a single trajectory, where they stopped six times in total. At each stop,
they collected images at multiple exposure times. This technique restricts the
number of images that can be acquired, since any changes in the environment
would corrupt the benchmark.

In our case, we do not make any static assumption, and we are able to compare
AE methods using standard VO pipelines. Our dataset has the advantage of being
versatile and independent of the tested solutions, resulting in better
replicability, even if new algorithms are developed.

###  II-B Datasets for VO

Several datasets were collected aiming at improving VO against challenging
illumination conditions. The KITTI [17] and the Oxford [18] datasets both
acquired stereo cameras, lidars, Inertial Measurement Unit (IMU), and GPS
data, with Oxford offering different weathers, seasons, and illuminations. The
North Campus dataset [19] also acquired urban stereo images, changing between
indoors and outdoors, on a 15-month range, using a segway robotic platform.
The UMA-VI dataset [20] had for main purpose to acquire HDR images with a
large number of low-textured scenes. They used a handheld custom rig equipped
with cameras and IMU, but they only provided ground truth through loop closure
error. Closer to our dataset environment, TartanDrive [21] and FinnForest [22]
both acquired off-road data. The TartanDrive dataset contains seven
proprioceptive and exteroceptive modalities used for reinforcement learning
purposes, including stereo images. Although simpler, the FinnForest dataset is
also composed of stereo images in summer and winter, showing the same forest
landscapes under multiple conditions. From the papers described in Section
II-A, only [8] and [7] published their dataset. While the presented datasets
allowed great improvements of VO algorithms, ours is complementary by
providing the full dynamic range of scenes through exposure times cycling.
Combined with our emulation framework, we unlock the possibility to select the
exposure time _during playback_ , expanding the realm of camera parameters
algorithms evaluation.

###  II-C AE Algorithms

Most vision-based localization algorithms rely on image gradient information
to localize. For instance, [11] designed an image quality metric based on
gradient magnitude. Their exposure control scheme generates seven synthetic
versions of the latest acquired image, simulating different exposure levels,
to identify the next exposure value maximizing their metric. The AE algorithm
proposed by [5] sorts the gradient level of each pixel, and applies a weight
factor based on their percentile value. By combining their quality metric and
the Camera Response Function (CRF), they predict the best next exposure value.
[12] developed an image quality metric based on the gradient level and Shannon
entropy [23], used to detect saturation. To demonstrate the benchmarking
capabilities of our approach, we provide an implementation of the above
methods, which are often used for comparison [7, 2], since they cover main
aspects of image quality for localization algorithms. Other techniques exist,
such as [8], who takes into account the Signal-to-Noise Ratio (SNR), but we
did not implement it, since it is mostly based on camera gain.

##  III Theory

In this section, we detail the selected approaches for the development of our
system, namely the emulation technique, the AE implementations, and the lidar
ground truth. Details on how our system can be used for benchmarking are
presented in Section IV.

###  III-A Emulation Technique

From the real images acquired using the bracketing technique, we are able to
emulate an image at any other realistic exposure time. Our emulation method is
based on the image acquisition process, which maps the scene radiance $E$ to
the image pixel values $I(x)$, using the vignetting $V(x)$, the exposure time
$\Delta t$, and the CRF $f(\cdot)$. This process can be expressed using the
following equation [24]:

$I(x)=f\left(\Delta tV(x)E\right).$ (1)

From Equation 1, the relationship between two images $I_{\text{source}}$ and
$I_{\text{target}}$ and their respective exposure times $\Delta
t_{\text{source}}$ and $\Delta t_{\text{target}}$ can be defined as

$I_{\text{target}}=f\left(\frac{\Delta t_{\text{target}}}{\Delta
t_{\text{source}}}\cdot f^{-1}\left(I_{\text{source}}\right)\right),$ (2)

where $f^{-1}(\cdot)$ is the inverse CRF [25]. To estimate $f(\cdot)$ and
$f^{-1}(\cdot)$, we take multiple images of a static scene at several exposure
times, allowing to capture the whole dynamic range at fixed radiance,
following [26]. With $f^{-1}(\cdot)$ and Equation 2, it is thus possible to
emulate any targeted exposure times $I_{\text{target}}$ by using a known image
with an exposure time $\Delta t_{\text{source}}$. Note that, for the sake of
simplicity, an image taken from one exposure time in the bracketing cycle will
be called bracket.

Considering that the data is acquired while moving, the brackets are not taken
at the same time nor position. Therefore, we decided not to interpolate
between the brackets, since it would create artifacts in the resulting image.
Instead, we select the most appropriate $I_{\text{source}}$ from the available
brackets to emulate $I_{\text{target}}$. The selection process takes into
account the distance between $\Delta t_{\text{source}}$ and $\Delta
t_{\text{target}}$, and the amount of saturated pixels in $I_{\text{source}}$,
since these do not contain any usable information. Also, if we select a
bracket such as $\Delta t_{\text{target}}/\Delta t_{\text{source}}>1$, the
result will be an image with higher pixel values, which means that the noise
in the signal will also increase. Therefore, it is preferable to select
$I_{\text{source}}$ that has a higher exposure time compared to our desired
$I_{\text{target}}$ to maximize the SNR.

Based on these considerations, we developed a simple selection method named
HigherNoSat. The first step is to find the two closest brackets $\Delta
t_{\text{bl}}$ and $\Delta t_{\text{bh}}$, that bound $\Delta
t_{\text{target}}$, such that:

$\Delta t_{\text{bl}}<\Delta t_{\text{target}}<\Delta t_{\text{bh}}.$ (3)

Then, for each of the two brackets, we calculate the amount of image
saturation, which corresponds to the number of pixels with values of 0 and
4095, for our 12-bit channel images. If the saturation level of the higher
bracket is below $1\text{\,}\mathrm{\char 37\relax}$, we select it as the best
candidate, otherwise, we pick the lower one. Finally, if $\Delta
t_{\text{target}}$ is outside the range of available $\Delta
t_{\text{source}}$, we select the closest bracket.

###  III-B Implementation Details of Compared AE Algorithms

Based on Section II-C, we implement and benchmark three state-of-the-art AE
methods: $M_{\text{Shim}}$ [11], $M_{\text{Zhang}}$ [5], and $M_{\text{Kim}}$
[12]. These methods are not open-source, thus they were all implemented based
on our understanding of the papers. $M_{\text{Zhang}}$ suffers from
instabilities coming from the CRF estimation, which were partly resolved using
the description in [8]. The exposure control scheme of $M_{\text{Kim}}$ used a
Gaussian Process (GP), which sparsely sweep the camera exposure parameters
until convergence. In our implementation, we use a sliding window on the
trajectory to only consider the most recent images in the GP training. This
can cause steep changes in the desired exposure time, since the exploration
algorithm of the GP needs to cover a wide range of values to converge. We also
implemented four baseline methods, corresponding to the typical AE algorithms.
The first baseline algorithm is a fixed exposure time approach $M_{fixed}$.
The exposure time is selected once, at the beginning of each sequence, by
using a brightness target of $50\text{\,}\mathrm{\char 37\relax}$. The three
other baselines are variable exposure algorithms, seeking to keep the average
brightness target to $30\text{\,}\mathrm{\char 37\relax}$,
$50\text{\,}\mathrm{\char 37\relax}$, and $70\text{\,}\mathrm{\char 37\relax}$
of the 12-bit depth range. They are named respectively
$M_{$30\text{\,}\mathrm{\char 37\relax}$}$, $M_{$50\text{\,}\mathrm{\char
37\relax}$}$, $M_{$70\text{\,}\mathrm{\char 37\relax}$}$. These percentages
were chosen as to cover a wide range of exposure.

###  III-C Ground Truth Generation

To generate the ground truth trajectories, we employed a low-drifting variant
of the Iterative Closest Point (ICP) algorithm, as described by [27]. The
method registers deskewed point clouds at $10\text{\,}\mathrm{Hz}$ into a
simultaneously built dense 3D map of the environment. It relies on the on-
board lidar and is illumination-independent, with median error values around
$1\text{\,}\mathrm{\char 37\relax}$ of the total trajectory length. The ICP
and VO trajectories were time-synchronized using GPs.

##  IV Results

This section first describes our acquisition platform, before evaluating the
performances of the emulation method. Then, we elaborate our data collection
pipeline, and finally, conduct the first online AE benchmarking on large VO-
based trajectories.

###  IV-A Experimental Setup

The BorealHDR dataset was collected using a homemade water-resistant
acquisition backpack, depicted in Figure 2. The enclosure is a Pelican case
1510 containing a battery and a Jetson AGX Xavier Development kit embedded
computer. The camera sensors are two Basler a2A1920-51gcPRO, in a stereo-
calibrated configuration with a baseline of $18\text{\,}\mathrm{cm}$, and
hardware-triggered by an STM32 microcontroller. In addition, the platform is
equipped with a Velodyne VLP-16 lidar, an Xsens MTI-30 IMU, and an Emlid Reach
RS+ GPS receiver. The images are acquired at a rate of $r_{\text{real}}$ of 22
frames per second (FPS) in 12-bit color, using loss-less compression. The
image acquisition process cycles through six exposure time values, i.e.,
$\Delta T_{\text{bracket}}=\\{1,2,4,8,16,32\\}$ $\mathrm{ms}$, yielding an
effective emulation rate of $r_{\text{emul}}=$ 3.66 FPS. This number of
brackets is a compromise between the offline $r_{\text{emul}}$ and the
emulation error, detailed in Section IV-B. The small footprint of the
acquisition platform allows collecting data in narrow and hard-to-access
spaces for robotic vehicles, which was fundamental for our 8.4 $\mathrm{km}$
dataset in HDR environments.

Figure 2: Picture of the developed backpack for the dataset acquisition. Main
components are identified as follows: \raisebox{-.9pt} {1}⃝ Two Basler
a2A1920-51gcPRO cameras, \raisebox{-.9pt} {2}⃝ Xsens MTI-30 IMU,
\raisebox{-.9pt} {3}⃝ VLP16 3D lidar, \raisebox{-.9pt} {4}⃝ Emlid Reach RS+
GPS receiver, and \raisebox{-.9pt} {5}⃝ Ubiquiti UniFi UAP-AC-M wifi antenna.

###  IV-B Emulation

Figure 3: Validation of our emulation framework using 1000 ground truth images
captured with constant illumination, but varying exposure time. Left: RMSE
curves showing the emulation error if a single bracket was always selected, in
colors, compared to our bracket selection method HigherNoSat, in black. Red
symbols correspond to the emulated exposure times displayed on the right.
Right: Qualitative comparison of the distributions of five emulated exposure
times (columns) from six different bracket images (rows). The markers are
placed next to the selected bracket by HigherNoSat. Ground truth distributions
are overlaid in gray for each column.

To evaluate the performances of our image emulator, we collect five test
sequences of 1000 ground truth images with exposure times ranging from
$20\text{\,}\mathrm{\SIUnitSymbolMicro s}$ to $50\text{\,}\mathrm{ms}$, in
static scenes both indoor and outdoor. Then, we emulate the same images
following the method described in Section III-A, and calculate the RMSE
between $I_{\text{emul}}$ and $I_{\text{GT}}$ for each ground truth exposure.
To account for the camera’s intrinsic noise, we average the error between 25
consecutive images at the same exposure time, over the whole spectrum, and
subtract this noise from our results. The five test sequences serve to
validate our bracket selection method HigherNoSat, which does not have access
to ground truth images in real world settings. Overall, our tests show that
the emulation method maintains a median RMSE of $0.21\text{\,}\mathrm{\char
37\relax}$ and never surpasses $1.78\text{\,}\mathrm{\char 37\relax}$,
emphasizing its consistency and accuracy.

One of the test sequences is detailed in Figure 3, showing emulation
performances in controlled lab settings. On the left side, the RMSE curve from
HigherNoSat, in black, is compared to the error obtained by selecting a single
bracket as $I_{\text{source}}$ for the whole range, in colors. It shows that
our bracket selection maintains the error close to the lowest value of the six
colored curves. The right side plots present a qualitative evaluation of the
emulated images from the acquired brackets. Each column exposes the
distributions of pixel intensity for one target exposure time, while each row
emulates this image from a different bracket. Red markers represent which
exposure time in $\Delta T_{\text{bracket}}$ was chosen by our emulator via
HigherNoSat, for this image. We observe that our bracket selector consistently
picks the closest or second-closest emulated image to the ground truth
distribution, overlaid in gray.

###  IV-C Dataset Gathering

Our BorealHDR dataset comprises 50 sequences totaling around 8.4 $\mathrm{km}$
for 4 $\mathrm{h}$ of data. Most of them are loops, which is a common practice
for VSLAM datasets. The relatively low $r_{\text{emul}}$ implies that the data
should be collected at a low walking speed to avoid large displacements
between each acquisition cycle. Hence, our average pace is around
$2\text{\,}\mathrm{k}\mathrm{m}\mathrm{/}\mathrm{h}$, which increases the data
collection time but does not impact the users of our approach.
$333\text{\,}813\text{\,}\mathrm{i}$mages were gathered on two separate days
in April 2023 in the Montmorency boreal Forest near Québec City, Canada,
described in [28]. The multi-session acquisitions allowed to cover multiple
meteorological and illumination conditions. We concentrated the acquisitions
on HDR scenes by capturing trees and snow in the camera frame. A variety of
natural environments was also targeted during the acquisitions, such as 14
open spaces, 16 narrow tree corridors, 15 forest trails, and 5 larger forest
roads. From these, 20 sequences were acquired in bright sunlight, increasing
the HDR conditions. Using snowshoes to get to distant locations, we were able
to not only acquire images with different scene illuminations, but also
different 3D structures, allowing for a richer collection of test scenarios
for VSLAM algorithms.

###  IV-D Benchmark

The main contribution of our paper is to unlock _offline_ testing of active
vision methods that could only previously be tested in an _online_ manner. To
this effect, we conduct two series of tests to benchmark the methods explained
in Section III-B, namely _(1) feature tracking and uniformity_ , and _(2)
stereo VO_, both highlighting the impact of each AE algorithm in key aspects
of a VSLAM pipeline.

####  IV-D1 Feature Tracking and Uniformity

For feature-based VO algorithms, the performance is dependent on the quality
and quantity of detected keypoints. Therefore, we evaluate feature detection
for each implemented AE algorithms on our recorded dataset, over three
distinct tests. We selected the Scale-Invariant Feature Transform (SIFT) [29]
feature-detector, since it is the most accurate according to [30].

Figure 4: SIFT features analysis. (a) Uniformity of detected keypoints for an
image divided into a $20\times 20$ grid. (b) Number of match features between
a pair of images for all AE metrics. The gray shading in (a) and (b) is to
highlight the state-of-the-art AE methods. (c) Percentage of successful
trajectories based on the number of matches detected. If one image contains
less matches than $\tau_{\text{matches}}$, the sequence is marked as not
successful.

First, we estimate the uniformity of the detected keypoints, by dividing each
image into a $20\times 20$ grid and assessing its occupancy rate. A uniform
distribution of features is presumed to be a good proxy for VO algorithm
performance. The results are displayed in Figure 4a, which shows that most
implemented AE methods yield a similar grid coverage. We observe that
$M_{$70\text{\,}\mathrm{\char 37\relax}$}$ and $M_{\text{Kim}}$ provide
slightly reduced uniformity, with median values respectively
$1.2\text{\,}\mathrm{\char 37\relax}$ and $5.2\text{\,}\mathrm{\char
37\relax}$ under the average of the other combined methods. In Figure 4b we
show that we can also obtain the number of detected features that are matched
between consecutive images, again for all implemented AE methods.
$M_{\text{fixed}}$ is the one tracking the highest number of features between
two images, with a median of $810\text{\,}$ matches. This makes sense, since
exposure is calibrated when starting each sequence and the lack of exposure
change keeps a better flow between consecutive images. Now, looking at full
trajectories, we define a success criterion corresponding to a minimal number
of matches $\tau_{\text{matches}}$ between each consecutive image pairs. If,
for a given AE algorithm and trajectory, this criterion is always true, then
the sequence is considered successful. Accordingly, we evaluate the percentage
of successful trajectories with $min(\tau_{\text{matches}})=5$, the minimum
number of matches for motion estimation [31], and the results are shown in
Figure 4c. We observe that the overall robustness of the AE algorithms
decreases rapidly. For $\tau_{\text{matches}}=100$, which is largely under any
first quartile from Figure 4b, $M_{$30\text{\,}\mathrm{\char 37\relax}$}$
shows the poorest performance, completing $24\text{\,}\mathrm{\char 37\relax}$
of the dataset sequences successfully, while the best method,
$M_{\text{fixed}}$, achieves $56\text{\,}\mathrm{\char 37\relax}$ of success.
This clearly shows the challenge that represents our dataset for VO.

Many of the developed benchmarking techniques for AE algorithms would have
been able to conduct our first two experiments. However, the third one could
only have been done using methodologies based on moving camera, since we
evaluate the features on trajectories. More importantly, our approach enables
entirely reproducible testing, offline. Consequently, we can always add more
AE methods to the benchmarks, without having to collect new data.

####  IV-D2 Stereo Visual Odometry (VO)

Figure 5:  Example of a collected trajectory from our BorealHDRdataset. The
ground truth trajectory and one VO result are illustrated superimposed on the
generated lidar 3D map. Black color represents the ground, and other colors
highlight structures in the environment. Figure 6:  Averaged translation error
over all the trajectories as a function of the trajectory length, for the
implemented AE algorithms. The line styles depict three performance clusters,
going from the top with solid, dashed, and then solid again. The legend is
ordered based on performance.

We also investigate the impact of AE methods on VO, which is a key component
of VSLAM. We implemented a simple stereo VO pipeline based on _OpenCV_ [32]
using SIFT [29] for feature detection. The _evo_
111https://github.com/MichaelGrupp/evo library is used to calculate the
Relative Pose Error (RPE) [33] for each AE algorithm. Our ground truth
trajectories are generated using the lidar data and ICP, as explained in
Section III-C. An example using one trajectory from our dataset is illustrated
in Figure 5, along with the 3D map built with the lidar data. The RPE is
displayed in Figure 6, where we mainly observe three clusters. The low-tier
one contains $M_{\text{Kim}}$, which has a minimal error of
$8.3\text{\,}\mathrm{\char 37\relax}$, making it the least performing method.
This is caused by the GP’s exploration step, as explained in Section III-B.
The mid-tier cluster includes three methods having minimal translation errors
averaging $4.0\text{\,}\mathrm{\char 37\relax}$. It is composed of
$M_{\text{fixed}}$, $M_{$30\text{\,}\mathrm{\char 37\relax}$}$, and
$M_{$70\text{\,}\mathrm{\char 37\relax}$}$. As highlighted in Section IV-D1,
$M_{$30\text{\,}\mathrm{\char 37\relax}$}$ and $M_{$70\text{\,}\mathrm{\char
37\relax}$}$ produce more saturated pixels, decreasing the number of detected
features. Even though $M_{\text{fixed}}$ was among the best methods for
feature tracking, its unresponsiveness to illumination changes results in an
error of $4.1\text{\,}\mathrm{\char 37\relax}$ when applied to VO. Finally,
the top-tier results are obtained by $M_{\text{Shim}}$,
$M_{$50\text{\,}\mathrm{\char 37\relax}$}$, and $M_{\text{Zhang}}$, which have
respectively a minimal translation error of $3.0\text{\,}\mathrm{\char
37\relax}$, $2.9\text{\,}\mathrm{\char 37\relax}$, and
$2.6\text{\,}\mathrm{\char 37\relax}$. $M_{$50\text{\,}\mathrm{\char
37\relax}$}$ shows similar error compared to $M_{\text{Shim}}$ and
$M_{\text{Zhang}}$. We conjecture that this is caused by the snow floor
creating a textureless environment, reducing the overall gradient. All in all,
our new developed approach allows for the first comparison of AE methods
directly on a large VO dataset, in an offline manner.

##  V Conclusion And Future Work

In this work, we proposed an emulator framework based on a multi-exposure
dataset, which allows comparing AE algorithms in an offline and reproducible
manner. Our dataset, BorealHDR, contains 50 trajectories combining for
$8.4\text{\,}\mathrm{km}$, focusing on collecting images in HDR scenes taken
in a snowy boreal forest environment. In addition to the 12-bit color stereo
images, we also provide pose ground truth for each image and 3D map for each
sequence, based on lidar data. We have shown the versatility of our
methodology by benchmarking seven AE algorithms, of which three were from
recent works. We concluded that our emulation approach is an efficient
solution for offline benchmarking of active algorithms, such as AE, by making
experiments on multiple key elements affecting VSLAM pipelines. In the future,
we plan to extend our dataset to be multi-seasonal, and to contain more
drastic illumination changes and scenarios. We also plan to design novel AE
approaches that take into account the pose uncertainty estimates obtained by
the VSLAM pipeline, which were previously difficult to develop without a
flexible testing methodology.

## References

  * [1] Bin Han et al.  “Camera Attributes Control for Visual Odometry With Motion Blur Awareness”  In _IEEE/ASME Transactions on Mechatronics (TMECH)_ 28.4 Institute of ElectricalElectronics Engineers (IEEE), 2023, pp. 2225–2235  DOI: 10.1109/tmech.2023.3234316
  * [2] Yu Wang, Haoyao Chen, Shiwu Zhang and Wencan Lu  “Automated camera-exposure control for robust localization in varying illumination environments”  In _Autonomous Robots_ 46.4 Springer, 2022, pp. 515–534
  * [3] Georges Chahine and Cedric Pradalier  “Survey of Monocular SLAM Algorithms in Natural Environments”  In _Conference on Computer and Robot Vision (CRV)_ , 2018
  * [4] Justin Tomasi, Brandon Wagstaff, Steven L. Waslander and Jonathan Kelly  “Learned Camera Gain and Exposure Control for Improved Visual Feature Detection and Matching”  In _IEEE Robotics and Automation Letters (RA-L)_ 6.2 IEEE, 2021, pp. 2028–2035  DOI: 10.1109/LRA.2021.3058909
  * [5] Zichao Zhang, Christian Forster and Davide Scaramuzza  “Active exposure control for robust visual odometry in HDR environments”  In _IEEE International Conference on Robotics and Automation (ICRA)_ , 2017, pp. 3894–3901
  * [6] Joowan Kim, Younggun Cho and Ayoung Kim  “Proactive Camera Attribute Control Using Bayesian Optimization for Illumination-Resilient Visual Navigation”  In _IEEE Transactions on Robotics (T-RO)_ 36, 2020, pp. 1256–1271  DOI: 10.1109/TRO.2020.2985597
  * [7] Marc-André Bégin and Ian Hunter  “Auto-Exposure Algorithm for Enhanced Mobile Robot Localization in Challenging Light Conditions”  In _Sensors_ 22 MDPI, 2022, pp. 835  DOI: 10.3390/s22030835
  * [8] Ukcheol Shin et al.  “Camera Exposure Control for Robust Robot Vision with Noise-Aware Image Quality Assessment”  In _IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2019, pp. 1165–1172
  * [9] Inwook Shim et al.  “Gradient-based camera exposure control for outdoor mobile platforms”  In _IEEE Transactions on Circuits and Systems for Video Technology_ 29 Institute of ElectricalElectronics Engineers Inc., 2019, pp. 1569–1583  DOI: 10.1109/TCSVT.2018.2846292
  * [10] Mohit Gupta, Daisuke Iso and Shree K. Nayar  “Fibonacci Exposure Bracketing for High Dynamic Range Imaging”  In _IEEE International Conference on Computer Vision (ICCV)_ , 2013  DOI: 10.1109/iccv.2013.186
  * [11] Inwook Shim, Joon-Young Lee and In So Kweon  “Auto-adjusting camera exposure for outdoor robotics using gradient information”  In _IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2014, pp. 1011–1017
  * [12] Joowan Kim, Younggun Cho and Ayoung Kim  “Exposure Control Using Bayesian Optimization Based on Entropy Weighted Image Gradient”  In _IEEE International Conference on Robotics and Automation (ICRA)_ , 2018, pp. 857–864
  * [13] Zichao Zhang, Henri Rebecq, Christian Forster and Davide Scaramuzza  “Benefit of large field-of-view cameras for visual odometry”  In _IEEE International Conference on Robotics and Automation (ICRA)_ , 2016, pp. 801–808
  * [14] Ruben Gomez-Ojeda, Zichao Zhang, Javier Gonzalez-Jimenez and Davide Scaramuzza  “Learning-based image enhancement for visual odometry in challenging HDR environments”  In _IEEE International Conference on Robotics and Automation (ICRA)_ , 2018, pp. 805–811
  * [15] Sebastian Höfer et al.  “Sim2Real in robotics and automation: Applications and challenges”  In _IEEE transactions on automation science and engineering_ 18.2 IEEE, 2021, pp. 398–400
  * [16] Ishaan Mehta, Mingliang Tang and Timothy D. Barfoot  “Gradient-Based Auto-Exposure Control Applied to a Self-Driving Car”  In _Conference on Computer and Robot Vision (CRV)_ , 2020, pp. 166–173
  * [17] A. Geiger, P. Lenz and R. Urtasun  “Are we ready for autonomous driving? The KITTI vision benchmark suite”  In _IEEE Conference on Computer Vision and Pattern Recognition_ , 2012, pp. 3354–3361
  * [18] Dan Barnes et al.  “The Oxford Radar RobotCar Dataset: A Radar Extension to the Oxford RobotCar Dataset”  In _IEEE International Conference on Robotics and Automation (ICRA)_ , 2020, pp. 6433–6438  DOI: 10.1109/ICRA40945.2020.9196884
  * [19] Nicholas Carlevaris-Bianco and Ryan M. Eustice  “Learning visual feature descriptors for dynamic lighting conditions”  In _IEEE/RSJ International Conference on Intelligent Robots and Systems_ , 2014, pp. 2769–2776  DOI: 10.1109/IROS.2014.6942941
  * [20] David Zuñiga-Noël, Alberto Jaenal, Ruben Gomez-Ojeda and Javier Gonzalez-Jimenez  “The UMA-VI dataset: Visual–inertial odometry in low-textured and dynamic illumination environments”  In _The International Journal of Robotics Research_ 39 SAGE Publications Inc., 2020, pp. 1052–1060  DOI: 10.1177/0278364920938439
  * [21] Samuel Triest et al.  “TartanDrive: A Large-Scale Dataset for Learning Off-Road Dynamics Models”  In _IEEE International Conference on Robotics and Automation (ICRA)_ , 2022, pp. 2546–2552  DOI: 10.1109/ICRA46639.2022.9811648
  * [22] Ihtisham Ali et al.  “FinnForest dataset: A forest landscape for visual SLAM”  In _Robotics and Autonomous Systems_ 132 Elsevier B.V., 2020, pp. 103610  DOI: 10.1016/j.robot.2020.103610
  * [23] Claude Elwood Shannon  “A mathematical theory of communication”  In _The Bell system technical journal_ 27.3 Nokia Bell Labs, 1948, pp. 379–423
  * [24] Paul Bergmann, Rui Wang and Daniel Cremers  “Online Photometric Calibration of Auto Exposure Video for Realtime Visual Odometry and SLAM”  In _IEEE Robotics and Automation Letters_ 3 Institute of ElectricalElectronics Engineers Inc., 2018, pp. 627–634  DOI: 10.1109/LRA.2017.2777002
  * [25] M.D. Grossberg and S.K. Nayar  “Determining the camera response from images: What is knowable?”  In _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 25, 2003, pp. 1455–1467  DOI: 10.1109/TPAMI.2003.1240119
  * [26] J. Engel, V. Usenko and D. Cremers  “A Photometrically Calibrated Benchmark For Monocular Visual Odometry”  In _arXiv:1607.02555_ , 2016
  * [27] Vladimír Kubelka, Maxime Vaidis and François Pomerleau  “Gravity-constrained point cloud registration”  In _IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2022, pp. 4873–4879
  * [28] Dominic Baril et al.  “Kilometer-scale autonomous navigation in subarctic forests: challenges and lessons learned”  In _Field Robotics_ 2.1 Field Robotics Publication Society, 2022, pp. 1628–1660  DOI: 10.55417/fr.2022050
  * [29] David G Lowe  “Object recognition from local scale-invariant features”  In _IEEE International Conference on Computer Vision_ , 1999, pp. 1150–1157
  * [30] Shaharyar Ahmed Khan Tareen and Zahra Saleem  “A comparative analysis of sift, surf, kaze, akaze, orb, and brisk”  In _IEEE International conference on computing, mathematics and engineering technologies (iCoMET)_ , 2018, pp. 1–10
  * [31] David Nistér  “An efficient solution to the five-point relative pose problem”  In _IEEE transactions on pattern analysis and machine intelligence_ 26.6 IEEE, 2004, pp. 756–770
  * [32] G. Bradski  “The OpenCV Library”  In _Dr. Dobb’s Journal of Software Tools_ , 2000
  * [33] Jürgen Sturm et al.  “A benchmark for the evaluation of RGB-D SLAM systems”  In _IEEE/RSJ international conference on intelligent robots and systems_ , 2012, pp. 573–580

  *[VO]: Visual Odometry
  *[HDR]: High Dynamic Range
  *[AE]: Automatic-Exposure
  *[RMSE]: Root-Mean-Square Error
  *[VSLAM]: Visual Simultaneous Localization And Mapping
  *[IMU]: Inertial Measurement Unit
  *[CRF]: Camera Response Function
  *[SNR]: Signal-to-Noise Ratio
  *[GP]: Gaussian Process
  *[ICP]: Iterative Closest Point
  *[FPS]: frames per second
  *[ VO]: Visual Odometry
  *[SIFT]: Scale-Invariant Feature Transform
  *[RPE]: Relative Pose Error