# MemBERT: Injecting Unstructured Knowledge into BERT

Federico Ruggeri
DISI
University of Bologna
Bologna, Italy
<EMAIL_ADDRESS>
& Marco Lippi
DISMI
University of Modena and Reggio Emilia
Reggio Emilia, Italia
<EMAIL_ADDRESS>
& Paolo Torroni
DISI
University of Bologna
Bologna, Italy
<EMAIL_ADDRESS>

###### Abstract

Transformers changed modern NLP in many ways. However, they can hardly exploit
domain knowledge, and like other blackbox models, they lack interpretability.
Unfortunately, structured knowledge injection, in the long run, risks to
suffer from a knowledge acquisition bottleneck. We thus propose a memory
enhancement of transformer models that makes use of unstructured domain
knowledge expressed in plain natural language. An experimental evaluation
conducted on two challenging NLP tasks demonstrates that our approach yields
better performance and model interpretability than baseline transformer-based
architectures.

_Keywords_ Transformers $\cdot$ DistilBERT $\cdot$ Explainable AI $\cdot$
Memory-augmented architectures $\cdot$ Knowledge injection

##  1 Introduction

In recent years, deep learning models have produced stunning results in a wide
variety of domains. In natural language processing (NLP), for instance,
transformer-based architectures like BERT [1] set a new standard in many
tasks, such as question answering or sentiment analysis [2]. In spite of the
welcome leap in performance, however, a typical criticism transformer
architectures share with most deep learning models is that they act as black-
boxes, hence lacking interpretability. Sure, the attention mechanism [3] could
offer cues as to how to interpret the behaviour of such models. Nevertheless,
whether attention could be meaningfully used as an analysis tool is still a
matter of discussion [4, 5].

In principle, deep networks could be made more interpretable by integrating
domain knowledge. The idea would be to inject recognizable, readable knowledge
elements into the deep network, making it inspectable. The injected
information should hopefully improve the performance of the model too. This
approach is gaining momentum, leading to a variety of proposals [6]. Among
them, some interesting ones rely on data augmentation, model architectural
biases, regularization constraints, and retrofitting. However, each
architecture requires a specific knowledge representation formalism. For
instance, added objective constraints are introduced by viewing background
knowledge as a set of differentiable, and possibly learnable, rules [7, 8].
Unfortunately, the formalism also defines the limits of the representation,
introducing the need for a costly abstraction process. For instance, in the
legal domain, justifications given by experts may be valuable indicators of
the underlying decision process, and as such they should be good candidates
for injection. However, a structured conceptual mapping of the reasoning
behind such justifications would be extremely time-consuming and hampered by
referral actions, common sense motivations, and other well known issues in NLP
[9]. As an example in the domain of legal analytics [10], and more
specifically unfair clause detection [11], consider the following clause:

> We reserve the right to modify or terminate the service or your access to
> the service for any reason, without notice, at any time, and without
> liability to you.

This could be considered unfair for many reasons. For instance, it gives the
provider the power of unilaterally removing consumer content from the service,
as well as unilaterally terminating the services. In order to link each
identified unfairness category to a textual explanation interpretable by a
legal expert, one could define a list of legal rationales, i.e., principles
that justify the legal opinion about the clause’s unfairness, and link those
of them relevant to the above clause. These may be given the following textual
definition:

Unilateral Change:

The provider has the right for unilateral change of the contract, services,
goods, features for any reason at its full discretion, at any time.

Unilateral Termination:

the contract or access may be terminated for any reason, without cause or
leaves room for other reasons which are not specified.

However, if knowledge injection requires to put these definitions into a
formal language of some sort, a great number of additional questions arise,
like which requirements have to be satisfied in order to ensure a correct
usage of the knowledge provided. It is then evident that an architecture able
to exploit knowledge expressed in plain English would eliminate or reduce many
problems, and thus it would be extremely valuable.

In light of the above, we propose a method for injecting domain knowledge,
expressed in natural language, into transformer architectures. In particular,
the main contributions of the paper are the following:

  1. 1.

we propose an approach to inject domain knowledge within transformer
architectures, in the form of plain natural language sentences;

  2. 2.

we introduce smart sampling strategies that allow to scale up to large
memories, also improving the overall performance of the system;

  3. 3.

we evaluate our approach on two relevant application domains, that of
unfairness detection in consumer contracts, and that of claim detection in
argument mining, using the DistilBERT architecture [12].

The paper is structured as follows. Section 2 discusses the integration of
external knowledge and proposes a memory-augmented version of DistilBERT. We
introduce memory selection supervision regularization, also known as strong
supervision, and propose different attention-based sampling strategies to
address scalability. Section 3 describes the experimental setting of two
challenging scenarios, whereas Section 4 presents and discusses results and
in-depth memory interaction analysis. Section 5 discusses similar applications
of memory-based BERT architectures and the use of attention-based mechanisms
to make these models’ decisions interpretable with domain knowledge. Section 6
concludes.

##  2 Problem Formulation

In this paper, we consider the problem of integrating background knowledge
expressed in natural language within a transformer architecture, while
achieving an interpretable model. By doing so, we distinguish ourselves from
the majority of knowledge injection approaches, which assume an appropriate
knowledge representation. Such setup hides the critical bottleneck of
knowledge acquisition. Our case study can be informally labelled as learning
from textual descriptions, which resembles certain human learning processes.

###  2.1 Knowledge as an External Memory

Dealing with knowledge expressed in natural language requires ad-hoc
methodologies that can efficiently use it, while maintaining desired
properties like model interpretability. Past research on deep learning models
tackled the problem of handling external content by introducing external
memory blocks with which the model could interact in a differentiable way
[13]. This solution has allowed to easily handle complex reasoning tasks like
reading comprehension and question answering and has also opened a new path in
the context of meta-learning. On this basis, we view memory-augmented neural
models as a natural architecture for natural language knowledge integration.
As a major difference with such existing approaches, the knowledge we store in
our memory is not core information for the task (as, for example, in question
answering) but rather an auxiliary, external collection of pieces of
information that can be consulted to perform reasoning and to make the neural
model interpretable.

###  2.2 MemBERT: Memory-Augmented DistilBERT

Figure 1: MemBERT architecture. Each textual input is encoded via the pre-
trained DistilBERT layer. Subsequently, external knowledge is compared to the
given input to be classified. The relevant memory content is then used to
update the initial query before the final fully-connected layer for
classification. We use the term $\bar{M}$, with a slight abuse of notation, to
denote the employed knowledge, independently of a possible preliminary
sampling strategy.

The main building block of our architecture is a transformer-based model.
Among the many ones, DistilBERT [12] is a distilled version of BERT that
achieves competitive performance while limiting the overall computational
burden. It is thus particularly convenient for running experiments. Therefore,
from now on we will refer to DistilBERT specifically. However, this reference
to a specific transformer-based model should not mislead the reader, since our
method is applicable to any kind of neural model as a plug-and-play module.
DistilBERT is enhanced with a memory layer following the traditional memory-
augmented neural network (MANN) architecture [14, 15], whereby a neural
network is extended with an external memory block supporting reasoning. We
denote this model as memory-augmented BERT (memBERT). The DistilBERT layer is
applied to each input, as well as to each memory slot (see Figure 1). Like in
typical MANN architectures: (i) input texts are encoded via the pre-trained
DistilBERT layer and memories are loaded with relevant background information;
(ii) memory content is compared with the given text input via a memory-lookup
layer, producing a set of similarity scores; (iii) the memory extraction layer
converts similarity scores into attention scores $\\{a_{1},\dots,a_{M}\\}$ and
uses them to compute a single memory summary embedding vector via a weighted
sum of the memory slots; (iv) the memory summary embedding vector is
eventually used to update the input embedding vector (memory reasoning). For
instance, a simple form of input update consists in summing input and memory
summary embedding vectors together. Lastly, a fully-connected layer is used
for the classification task. In our architecture, we keep the original
DistilBERT pre-classifier layer and replace the last fully-connected layer
with a task-specific one.

As our case studies involve independent memory slots, we do not iterate such a
procedure multiple times. In the MANN jargon, this means setting the number of
hops to 1. Furthermore, motivated by integrated knowledge requirements
described in Section 2.1, we consider an MLP-based memory-lookup
implementation. More precisely, we resort to a single dense layer with number
of units in the $[32,512]$ range. For what concerns the requirement of model
interpretability, both our case studies have input examples that are
associated with multiple memory slots, i.e. target memory slots. In order to
obtain an interpretable model, the network has to correctly select at least
one target memory slot per example. For these reasons, we adopt sigmoid-based
attention scores rather than traditional softmax-based ones, being the memory
slots not mutually exclusive.

The last memory-related step in the model (the reasoning layer) is implemented
as a concatenation of input and memory summary embedding vectors.111All the
mentioned architectural choices, if not explicitly motivated, have been
selected after hyper-parameter tuning based on the achieved validation
performance. Being $N$ the number of training examples and $C$ the set of
classes, both DistilBERT and MemBERT are trained to minimize standard cross-
entropy loss:

$\mathcal{L}_{CE}=-\sum_{i=1}^{N}\sum_{c=1}^{C}p(y_{i}=c)\log
p_{\theta}(y_{i}=c)$ (1)

This setting is known in the literature as Weak Supervision (WS).

###  2.3 Guiding Memory Interaction with Strong Supervision

The concept of WS is important, since not always single memory slots can be
directly associated with individual training examples. This is a very general
setting. Yet, if background knowledge comes with the capability of naturally
linking each example with the associated memory content, it is then possible
to guide the model in the training phase by only focusing on specific target
memory slots. This procedure is formally known as Strong Supervision (SS) and
it has been widely explored in memory networks since their introduction [13].
SS can assume different formulations. In our experimental scenarios, it is
sufficient to enforce a preference for just one of the (possibly multiple)
memory slots associated with a given input example. Following [9], we limit
the memory-augmented neural model to a single reasoning iteration, i.e.,
memory hop, and we exploit SS in the form of a max-margin loss. More
precisely, we do not perform target sampling to ensure both the presence of
negative and positive examples, but we instead rely on batch examples to
formulate the constraint.

Formally, we define target memory slots as those that have been linked to the
input example during the annotation phase. We introduce a loss penalty term
that enforces target memory slots to have a higher similarity score with
respect to the input than to the remaining slots, up to a $\gamma$ margin:

$\mathcal{L}_{SS}=\frac{1}{N}\sum_{n=1}^{N}\frac{1}{|M^{n}_{+}||M^{n}_{-}|}\sum_{m_{+}\in
M^{n}_{+}}\sum_{m_{-}\in M^{n}_{-}}\left[\mathcal{L}(m_{+},m_{-})\right]\\\ $
(2)
$\mathcal{L}(m_{+},m_{-})=\max\Big{(}0,\gamma-\sigma(s(q^{n},m_{+}))+\sigma(s(q^{n},m_{-}))\Big{)}$
(3)

where $M^{n}_{+}$ is the set of target memory slots for a given input example
$q^{n}$, $M^{n}_{-}$ is the set of non-target memory slots for the $n$-th
input example, $(m_{+},m_{-})$ is a target/non-target memory slot pair for
$n$, and $\sigma$ is the (sigmoid) activation function. The hyper-parameter
$\gamma$ controls the intensity of the SS constraints, while trading-off
classification performance and model interpretability: large values of
$\gamma$ would basically turn the preference ranking induced by the hinge loss
into a classification loss. Overall, the loss function for MemBERT with SS is:

$\mathcal{L}=\mathcal{L}_{CE}+\mathcal{L}_{SS}$ (4)

###  2.4 Efficiently Handling Large Memories with Sampling

Dealing with complex deep learning models and large knowledge bases requires
an efficient retrieval of relevant content while maintaining scalable memory
usage. To this end, approximate solutions or sampling-based methods have been
explored, trading-off performance with efficiency [16]. However, such
solutions are mainly focused on simple comparison strategies like dot product
similarity that are easily executed in parallel and approximated. In our case,
we aim to define a more complex memory lookup phase. Therefore, we allow
neural-based similarities and strive to look for an efficient sampling
strategy to achieve scalability.

Memory sampling brings about both benefits and, possibly, issues. Reducing the
overall memory size enables large knowledge integration when dealing with
complex deep neural models like DistilBERT. Moreover, sampling eases the
knowledge integration process from the point of view of noise compensation. In
a similar fashion to curriculum learning [17], sampling reduces complexity by
considering a smaller memory, which, in turn, induces a limitation of spurious
lookup operations. This is also reflected at training time, where in each
batch the model sees a different memory, thus avoiding to always focus on
specific slots, which could possibly head to overfitting. Nonetheless,
sampling inherently introduces variance, therefore compatibility issues with
other strategies like strong supervision might arise as well. To compensate
for both problems, it is crucial to define smart sampling strategies that can
be conditioned on input examples. To this end, similarly to prioritized
experience replay (PER) [18], employed in reinforcement learning, we introduce
priority-based sampling strategies that progressively take into account the
importance of the added information with respect to both input examples and
task objectives. In particular, to efficiently deal with large memories, we
operate at batch level by sampling a shared knowledge base. Then, depending on
the given strategy, priority weights $p_{m_{i}}$ associated to each memory
slot $m_{i}$ are updated and available for the next batch step. More
precisely, we adopt the same priority definition of PER, where the temporal
difference (TD) error is replaced by a custom importance assignment function:

$p_{m_{i}}=(w_{m_{i}}+\epsilon)^{\alpha}$ (5)

where $w_{m_{i}}$ is the importance weight of the $i$-th memory slot $m_{i}$,
$\epsilon$ is a sufficiently small real value that avoids zero-based
exponentiation and $\alpha$ controls the priority degree. Intuitively,
$\alpha$ equal to 0 produces all priorities equal to 1, thus defining a
uniform sampling strategy. Lastly, sampling is applied both at training and at
inference time, when learnt priority weights are employed to sample the
memory. Algorithm 1 formally describes the training sampling procedure, while
Algorithm 2 describes the inference sampling procedure. Introduced strategy
variants differ on how priority weights are updated at each batch step.

1

Input: Data $D=\\{(\textbf{x}_{n},\textbf{y}_{n})\\}_{n=1}^{N}$,
Memory priority weights distribution $p_{|M|}$,
Model weights $\theta$

2 Initialize model weights $\theta$ = $\theta^{0}$

3 Initialize memory priority weights $p_{|M|}$ = $p^{0}_{|M|}$

4 repeat

5 Sample a minibatch $k$, $B^{k}=(\textbf{X},\textbf{Y})\subset\mathcal{D}$

6 Sample memory $\bar{M}$ using $p^{k-1}_{|M|}$

7 Compute model loss $\mathcal{L}(x|\bar{M})$ on minibatch $B$

8 Update model parameters $\theta^{k}$ using any optimizer

9 Compute memory importance weights $w^{k}_{|M|}$

10 Update memory priority weights $p^{k}_{|M|}=(w_{m_{i}}+\epsilon)^{\alpha}$

11 Normalize $p^{k}_{|M|}$ to get corresponding distribution

12until  _stopping criteria_

13Save memory priority weights $p_{|M|}$

Output: Trained model weights $\theta$, trained sampler priority weights
$p_{|M|}$

14

Algorithm 1 General Training Sampling Procedure

1

Input: Data $D=\\{(\textbf{x}_{n},\textbf{y}_{n})\\}_{n=1}^{N}$,
Learnt memory priority weights distribution $p_{|M|}$

2 repeat

3 Get next inference minibatch $k$, $B^{k}=(\textbf{X})\subset\mathcal{D}$

4 Sample memory $\bar{M}$ using $p_{|M|}$

5 Save model predictions $\tilde{\textbf{Y}}$

6until  _$B^{k}\in\varnothing$_

Output: Model predictions and sampled memory
$\\{(\tilde{\textbf{y}}_{n},\bar{\textbf{M}}_{n})\\}_{n=1}^{N}$

7

Algorithm 2 General Inference Sampling Procedure

####  2.4.1 Uniform Sampling

As a baseline strategy we consider uniform memory sampling. At each batch,
each memory slot has the same probability of being selected. Priorities are
kept fixed during training. Formally, being $|M|$ the memory size, we define
the probability of sampling each memory slot $m_{i}$ as
$p_{m_{i}}=\frac{1}{|M|}$.

####  2.4.2 Priority Sampling

Intuitively, uniformly sampling from memory immediately relieves us from the
problem of handling a large knowledge base, but it does not take into account
the importance of each slot. For instance, we would like to retrieve memory
slots depending on the characteristics of each input batch. In order to do
that, at training time, each individual input example should be able to modify
the underlying sampling distribution by giving more priority to those memory
slots that have been reputed useful. Thus, we introduce priority sampling
following the same perspective of PER [18]. In particular, we explore two
different formulations of priority that ground the notion of usefulness to a
specific architectural property of the model. More precisely, we consider the
memory attention layer as the major indicator of importance of each memory
slot with respect to the given input example, considering two variants: (i)
the first one relies exclusively on the attention value of each memory slot,
whereas (ii) the second formulation also takes into account the contribution
of each memory slot to the overall objective of training (i.e., the
classification loss).

##### Attention-Based Priority Sampling

In this formulation, the priority associated to each memory slot during
sampling is based only on the corresponding attention value computed by the
memory lookup layer. In particular, working with batches inherently requires a
consistent reduction operation that summarizes each memory slot importance at
batch level. To this end, we consider a masked average based reduction, which
considers the mean attention value of each memory slot for a certain class of
input examples. This strategy accounts for noise introduced by input examples
that are not interested by the background knowledge. For instance, in the case
of a binary classification task where memory contains valuable information for
the positive class only, we should limit sampling priority update to positive
examples, to avoid spurious memory selections.

Formally, the attention-based priority sampling defines each memory slot
importance as its summary attention value in the batch.

$w_{m_{i}}=\frac{\sum^{|B|}_{j}a_{j,i}}{\sum^{|B|}_{j}\mathbbm{1}_{y_{j}\in\textbf{Y}_{+}}}$
(6)

where $a_{i}$ is the attention value attributed to memory slot $i$, $|B|$ is
the batch size and $\textbf{Y}_{+}$ is the set of examples belonging to the
positive class.

Note that this strategy is purely similarity-based and it does not take into
account the contribution of each memory slot to the task-specific objective.
In other terms, it operates under the assumption that each knowledge
descriptive concept contained in the memory should have high semantic and
syntactic similarity with its associated input examples. Thus, it might not be
suitable in cases where input examples have high similarity with unrelated
memory slots due to their natural language formulation.

##### Loss Gain Priority Sampling

In order to render priority sampling aware of the usefulness of each memory
slot to the task itself, we consider an additional variant that directly
exploits the classification loss. In particular, we consider the loss
difference between the standard model architecture and the one augmented with
the memory layer. By doing so, we are able to estimate the informative gain
introduced by the external knowledge with respect to the task. The larger is
the difference, the larger is the impact of the memory. To associate a
specific priority value to each memory slot $m_{i}$, we weigh the computed
difference by the corresponding attention value $m_{i}$. We consider the same
reduction operation strategy adopted for attention-based priority sampling.
Formally, this loss gain priority sampling can be formulated as follows:

$w_{m_{i}}=\frac{\sum_{j}^{|B|}a_{j,i}\exp^{(\mathcal{L}_{CE}(x_{j})-\mathcal{L}_{CE}(x_{j}|\bar{M}))}}{\sum^{|B|}_{j}\mathbbm{1}_{y_{j}\in\textbf{Y}_{+}}}$
(7)

where $a_{i}$ is the attention value attributed to memory slot $i$, $\bar{M}$
is the sampled memory, $\mathcal{L}_{CE}(x)$ and $\mathcal{L}_{CE}(x|\bar{M})$
are the cross-entropy loss obtained without any interaction with the memory,
or with the introduction of the sampled memory $\bar{M}$, respectively. The
exponential guarantees a positive priority while awarding positive differences
with high values and down-weighting negative differences.

##  3 Experimental Setup

To test our approach, we consider two distinct classification scenarios. In
the first, we extend the work in [9] focusing on the legal domain for unfair
clause detection in online Terms of Service, by employing explanations given
by legal experts as domain knowledge. In the second scenario, we explore the
argument mining case study of claim detection by employing evidence
information as knowledge source, following the assumption that class
correlation might ease the classification task. To ensure reproducibility, we
made all our code and data available at the following repository:
https://bitbucket.org/hl2exe/membert/.

###  3.1 Knowledge as Class Descriptions: Unfairness Detection

Following [9], we formulate the problem of unfair clause detection in online
Terms of Service as a binary classification task, enriched by legal expertise
in the form of rationales, that is explanations of unfairness. It is a well-
known fact that online Terms of Service too often contain clauses that are
potentially unfair for the consumer, and NLP tools have been proven successful
in detecting them [11]. In our setting, each clause is the input text to be
classified, and the set of legal rationales defines the shared memory.

####  3.1.1 Data

The corpus originally published in [9] defines legal rationales for five
unfairness categories following a multi-label perspective (i.e., a clause can
be unfair according to more than a single category). These categories are:
arbitration on disputed arising from the contract (A); the provider’s right to
unilaterally modify the contract and/or the service (CH); the provider’s right
to unilaterally remove consumer content from the service, including in-app
purchases (CR); liability exclusions and limitations (LTD); the provider’s
right to unilaterally terminate the contract (TER). We randomly extracted 30
documents while ensuring that they contained at least one example per
unfairness category. Table 1 reports some statistics on the corpus, which we
name ToS-30, showing the number and percentage of unfair clauses per category,
as well as the amount of corresponding legal rationales defined by experts.
More technical details concerning the definition of both the unfairness
categories and the rationales can be found in [9].222Note that the corpus
labeling procedure distinguishes between potentially unfair and clearly unfair
clauses, but following [9] we merged these categories together.

Table 1: Corpus statistics for ToS-30. For each category of clause unfairness, we report the number and percentage of unfair clauses, and the number of rationales.  Type of clause | # unfair clauses | % unfair clauses | # rationales
---|---|---|---
Arbitration (A) | 45 | 0.8 | 8
Unilateral change (CH) | 89 | 1.7 | 7
Content removal (CR) | 58 | 1.1 | 17
Limitation of liability (LTD) | 161 | 3.0 | 18
Unilateral termination (TER) | 121 | 2.3 | 28

####  3.1.2 Setting

As in [9], we consider each unfairness category individually. In particular,
for a given unfairness category, we consider as positive examples those who
have been labeled as being unfair for the given category, while the remaining
sentences are viewed as not unfair. As competitors for our approach, we
consider the following neural baselines: (i) 2-layer CNN network; (ii) 1-layer
bi-LSTM network; (iii) MANN baseline as in [9]. Neural baselines use
512-dimensional embeddings, whereas the MANN baseline uses 256-dimensional
embeddings. All the embeddings are learnt from scratch. Additionally, models
are further regularized with $L^{2}$ penalty with $10^{-5}$ weight, dropout
rate in the $[0.5,0.7]$ range and are optimized with Adam optimizer with
$10^{-3}$ learning rate. For evaluation, we employ a 10-fold cross-validation
procedure. As a regularization criterion, all models are early stopped based
on the validation F1-score, with a patience equal to 10 for BERT models333We
initially trained for 3-4 epochs as a well known good practice, but we
immediately saw that more epochs were required to achieve significant
performance. and to 50 for other neural baselines.

###  3.2 Knowledge as Supporting Facts: Claim Detection

The second scenario of interest is claim detection in the context of argument
mining [19]. In particular, we formulate the problem as a binary
classification task where each input sentence can be either identified as
containing a claim or not. Here, background knowledge is expressed by the set
of labelled evidence that can potentially support a claim. Thus, we aim to
investigate whether evidence, which usually back up claims, bring some benefit
to the task of claim detection.

####  3.2.1 Data

We consider a portion of the IBM2015 argumentative corpus, built in the
context of the Debater project [20, 21]. The dataset consists of a collection
of Wikipedia pages, grouped into topics. The annotation procedure carried out
by IBM is context-dependent, which means that claims and evidence are
annotated with the assumption that the topic (context) is given. In our
scenario, we selected the four topics with the largest amount of claims and
evidence. Subsequently, we defined the set of evidence as the model memory.
Due to how argumentative texts were annotated, there are cases in which a
single sentence may contain both a claim and evidence or, more seldom, an
evidence spans through multiple sentences and incorporates a claim. Indeed,
these cases hinder the quality of selected data, but due to the low amount of
such spurious sentences, we do not consider further pre-processing steps and
use the corpus as is. Table 2 reports details about the selected subsets of
topics, focusing on their argumentative content. We remark that evidence in
the IBM2015 dataset are typically statements extracted from studies or facts
established by experts: it is thus a reasonable assumption to have a list of
such items available to support claim detection.

Table 2: Topics extracted from the IBM2015 corpus, with associated number of evidence and claims. For claims, we also report the percentage with respect to the overall corpus size, which corresponds to the frequency of the positive class. Topics | No. Evidence | No. Claim | Claim Ratio
---|---|---|---
1 | 130 | 113 | 4.2%
2 | 239 | 201 | 4.0%
3 | 578 | 288 | 3.8%
4 | 642 | 374 | 3.5%

####  3.2.2 Setting

As for unfairness detection, we consider sentence-level binary classification
task for claim detection. More precisely, each sentence can either be
identified as containing a claim or not. To progressively evaluate the
benefits of added knowledge, we consider incremental subsets of topics, from 1
up to 4. However, differently from the legal domain case study, the memory
content drastically increases as the number of topics gets larger, from 130
with a single topic up to 642 with four topics. Additionally, the number of
claims associated to each evidence is very low (range 1-5). Thus, a correct
usage of memory and of strong supervision is extremely challenging. Moreover,
such issue completely hinders the full knowledge scenario with MemBERT, i.e.,
the one where the model sees the whole memory content, due to memory issues.
Inevitably, in this scenario there is need to enforce the adoption of a
sampling strategy.

Given the low amount of samples, models are evaluated via a $k$-fold cross-
validation procedure (we chose $k$=4). As for unfairness detection, we
employed the multi-start technique with 3 repetitions, choosing the best one
according to the validation F1-score. At training time, models follow the same
regularization setup employed for unfair clause detection, by applying early
stopping on the validation F1-score. Patience is set to 10 for BERT models and
50 for other neural baselines. The MANN baseline uses 50-dimensional
embeddings, whereas all the other hyper-parameters are identical to the ToS-30
setting.

###  3.3 Memory Analysis

Knowledge integration evaluation cannot be reduced to task-specific metrics
only. Albeit by adding more knowledge one would expect increased
classification performance, there are other dimensions on which such
integration could bring benefit. For instance, natural language is inherently
straightforward to interpret. Thus, correctly linking such content to the
input would enrich the model with properties that compensate slight
fluctuations with respect to the achieved classification performance. Under
this critical perspective, we evaluate MemBERT to assess whether knowledge
integration is effective and to what extent.

As an initial step, we consider several memory-related metrics already
introduced in [9]. These metrics evaluate knowledge integration under several
perspectives, covering typical information retrieval analysis concerning top-K
memory ranking, as well as purely memory-oriented statistics like target
coverage. Such metrics will also be employed to analyze the behavior of
sampling in domains where memory size is intractable. For space issues, we
only select a limited amount of these metrics by picking the most informative
ones:

  * •

Memory Usage (U): percentage of examples for which memory is used.

  * •

Coverage (C): percentage of positive examples for which (at least one) memory
slot selection is correct.

  * •

Coverage Precision (CP): percentage of examples using memory, for which (at
least one) memory slot selection is correct.

  * •

Precision at K (P@K): percentage of positive examples for which at least one
correct memory target is retrieved in the first K positions.

  * •

Mean Reciprocal Rank (MRR): average of the reciprocal memory ranks.

A natural issue that arises when dealing with soft memory interactions is to
define an appropriate activation threshold $\delta$, i.e., the minimum value
to consider a memory slot has been used by the model. Indeed, even though with
SS regularization we explicitly enforce a specific activation threshold
$\gamma$, in the experimental setup the desired setting might not be feasible
at all. For instance, in the IBM2015 corpus, there are several factors that
might hinder such type of regularizations. More precisely, the large scale
knowledge base and the presence of evidence that are semantically close to non
target claims, prevents accentuated memory selections. To this end, in that
setting we explore memory-related metrics with variable activation thresholds,
so as to measure to what extent the model is capable to focus on certain
memory slots.

##  4 Results

We hereby describe the results obtained on the two scenarios, both considering
standard classification metrics, and providing an analysis of the memory
usage.

###  4.1 Unfairness Detection

For what concerns unfair clause detection, Table 3 summarizes the results
obtained on the ToS-30 corpus, by reporting the macro-F1 averaged on 10-fold
cross-validation. A first, clear result is that all BERT models maintain a
large performance gap over the other neural baselines. The introduced legal
rationales significantly aid memory-augmented models in categories for which
there are few positive examples like A and CR. On the other hand, on category
CH we can observe that memory-based approaches do not show an improvement over
their counterparts: MANNs perform worse than LSTMs, and MemBERT performs worse
than DistilBERT. This behavior might hide a problem in the nature of the
natural language rationales given for the CH category, which are evidently
less informative than those given for the other categories444Legal rationales
for CH category are very similar to each others, with minor distinctions that
might be hard to discriminate without supervised guidance.. The introduction
of SS regularization always leads to better performance for MemBERT models
that leverage full knowledge. In particular, it is particularly beneficial for
categories with increased memory size like CR, LTD and TER. Sampling-based
models manage to reach performance comparable with their full knowledge
counterparts. However, memory sampling attenuates the added contribution of SS
regularization, leading to mixed results for different strategies.

Table 3: Classification macro-F1 computed on 10-fold cross-validation for unfair examples on the ToS-30 dataset. WS and SS stay for weak and strong supervision, respectively. Sampling employs a 5-slot reduced size memory. We compare uniform (U), priority attention only (P-Att) and priority loss gain (P-LG) strategies. Priority sampling always considers negative example filtering (F) for correct priority update. Best results are in bold, second best results are underlined. | A | CH | CR | LTD | TER
---|---|---|---|---|---
No Knowledge |
CNN | 0.339 | 0.506 | 0.403 | 0.628 | 0.583
LSTM | 0.302 | 0.573 | 0.363 | 0.602 | 0.508
DistilBERT | 0.447 | 0.635 | 0.620 | 0.670 | 0.748
Full Knowledge |
MANN (WS) | 0.483 | 0.506 | 0.387 | 0.635 | 0.602
MANN (SS) | 0.465 | 0.516 | 0.414 | 0.605 | 0.660
MemBERT (WS) | 0.494 | 0.565 | 0.639 | 0.664 | 0.705
MemBERT (SS) | 0.504 | 0.609 | 0.670 | 0.686 | 0.737
Sampling |
MemBERT (WS) (U-5) | 0.514 | 0.556 | 0.609 | 0.678 | 0.702
MemBERT (WS) (P-5-Att-F) | 0.491 | 0.559 | 0.601 | 0.643 | 0.703
MemBERT (WS) (P-5-LG-F) | 0.475 | 0.574 | 0.660 | 0.678 | 0.716
MemBERT (SS) (U-5) | 0.503 | 0.580 | 0.617 | 0.652 | 0.702
MemBERT (SS) (P-5-Att-F) | 0.448 | 0.599 | 0.635 | 0.661 | 0.708
MemBERT (SS) (P-5-LG-F) | 0.490 | 0.536 | 0.625 | 0.656 | 0.706

Regarding the usage of memory, Table 4 reports statistics concerning the
predictions of our models on unfair examples. The results clearly show that SS
brings a crucial contribution in giving high importance to the correct memory
slots: in all the categories, one correct memory slot is ranked in the top 3
positions in over 55-80% of the cases. This behaviour is reflected also in the
MRR score which shows a clear advantage of SS over WS.

Table 4: Memory statistics concerning predictions on unfair examples only. Memory interaction is evaluated on ToS-30 test set. We report memory usage (U), the correct memory usage over unfair examples (C) and over examples for which memory is used (CP), along with a more fine-grained ranking version (P@1-3) and the mean reciprocal rank (MRR). Due to different memory usage, columns C–P@3 are not directly comparable. We set activation threshold $\delta$ to 0.5. Model | U | C | CP | P@1 | P@3 | MRR
---|---|---|---|---|---|---
Arbitration (A) |
MANN (WS) | 0.311 | 0.289 | 0.929 | 0.571 | 1.000 | 0.761
MANN (SS) | 0.689 | 0.644 | 0.935 | 0.903 | 0.968 | 0.861
MemBERT (WS) | 0.489 | 0.400 | 0.818 | 0.273 | 0.545 | 0.478
MemBERT (SS) | 0.956 | 0.911 | 0.953 | 0.767 | 0.837 | 0.848
Arbitration (CH) |
MANN (WS) | 0.169 | 0.090 | 0.533 | 0.000 | 0.067 | 0.299
MANN (SS) | 0.854 | 0.730 | 0.855 | 0.855 | 0.961 | 0.883
MemBERT (WS) | 0.404 | 0.382 | 0.944 | 0.250 | 0.750 | 0.522
MemBERT (SS) | 1.000 | 0.955 | 0.955 | 0.809 | 0.888 | 0.886
Arbitration (CR) |
MANN (WS) | 0.017 | 0.000 | 0.000 | 0.000 | 0.000 | 0.335
MANN (SS) | 0.672 | 0.414 | 0.615 | 0.282 | 0.872 | 0.612
MemBERT (WS) | 0.328 | 0.328 | 1.000 | 0.316 | 0.632 | 0.478
MemBERT (SS) | 1.000 | 0.948 | 0.948 | 0.431 | 0.879 | 0.681
Arbitration (LTD) |
MANN (WS) | 0.037 | 0.025 | 0.667 | 0.33 | 0.833 | 0.504
MANN (SS) | 0.814 | 0.534 | 0.656 | 0.313 | 0.573 | 0.501
MemBERT (WS) | 0.497 | 0.416 | 0.838 | 0.100 | 0.275 | 0.328
MemBERT (SS) | 1.000 | 0.919 | 0.919 | 0.224 | 0.565 | 0.474
Arbitration (TER) |
MANN (WS) | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.499
MANN (SS) | 1.000 | 0.471 | 0.471 | 0.438 | 0.537 | 0.536
MemBERT (WS) | 0.223 | 0.198 | 0.889 | 0.074 | 0.074 | 0.193
MemBERT (SS) | 1.000 | 0.851 | 0.851 | 0.438 | 0.579 | 0.567

###  4.2 Claim Detection

As for the claim detection scenario, Table 5 reports the achieved experimental
results on the four considered subsets of topics. As for unfair clause
detection, we evaluate knowledge-agnostic neural baselines as well as MANN
models. First of all, we note how MANNs achieved significant improvements over
CNNs and LSTMs, suggesting that the introduced knowledge is indeed beneficial
to the task itself. Similarly, MemBERT achieves higher F1-score with respect
to standard DistilBERT. Interestingly, even the uniform sampling strategy
manages to outperform other neural baselines, meaning that sampling also acts
as a noise compensation operator, thus, easing the process of knowledge
integration while increasing model robustness. This behaviour is accentuated
for MANNs, where the combination of SS and priority sampling largely
outperforms the version with full knowledge. More precisely, for 1 and 2
topics MANNs nearly match DistilBERT, yet with a drastically lower number of
parameters. These results corroborate the hypothesis that controlled and smart
knowledge integration can be particularly beneficial when limited training
data is available. Differently from unfairness detection, the large memory
size hinders selective memory lookup operations. Thus, metrics are computed
over incremental values of activation threshold $\delta$. In particular, we
mainly focus on the P@K metric, since it is a valuable indicator of the model
retrieval capabilities. Figures 2(a) reports values of P@K on the 1-Topic
case, along incremental value of $K$, with fixed $\delta=0.25$, whereas Figure
2(b) reports P@3 with incremental $\delta$ values. In both cases, we can
clearly see how loss gain priority sampling coupled with SS (P-10-LG-F) is the
setting that achieves the best performance.

(a)

(b)

Figure 2: Memory analysis on IBM2015, 1-Topic. Left: P@K for increasing $K$ values and $\delta=0.25$. Right: P@3 for increasing $\delta$ values. Metrics for sampling-based models are averaged across three distinct inferences on the test set. Table 5: Classification performance for dataset IBM2015. For each topics subset, we report the achieved macro F1-score. WS and SS stay for weak and strong supervision, respectively. Memory sampling is tested with a 10-slot reduced size memory. We report the uniform strategy (U), the priority attention only strategy (P-Att) and the priority loss gain strategy (P-LG). Priority sampling always considers negative example filtering (F) for correct priority update. Best results are in bold, second best results are underlined instead. | 1 Topic | 2 Topics | 3 Topics | 4 Topics
---|---|---|---|---
No Knowledge |  |  |  |
CNN | 0.196 | 0.283 | 0.287 | 0.268
LSTM | 0.194 | 0.344 | 0.278 | 0.272
DistilBERT | 0.317 | 0.431 | 0.405 | 0.451
Full Knowledge |  |  |  |
MANN (WS) | 0.252 | 0.380 | 0.325 | 0.336
MANN (SS) | 0.205 | 0.392 | 0.317 | 0.281
Sampling |  |  |  |
MANN (WS) (U-10) | 0.269 | 0.406 | 0.331 | 0.355
MANN (WS) (P-10-Att-F) | 0.251 | 0.402 | 0.322 | 0.358
MANN (WS) (P-10-LG-F) | 0.259 | 0.408 | 0.332 | 0.340
MANN (SS) (U-10) | 0.297 | 0.400 | 0.319 | 0.352
MANN (SS) (P-10-Att-F) | 0.264 | 0.423 | 0.332 | 0.348
MANN (SS) (P-10-LG-F) | 0.302 | 0.424 | 0.344 | 0.354
MemBERT (WS) (U-10) | 0.311 | 0.457 | 0.454 | 0.453
MemBERT (WS) (P-10-Att-F) | 0.275 | 0.449 | 0.422 | 0.434
MemBERT (WS) (P-10-LG-F) | 0.305 | 0.449 | 0.428 | 0.428
MemBERT (SS) (U-10) | 0.341 | 0.442 | 0.444 | 0.436
MemBERT (SS) (P-10-Att-F) | 0.354 | 0.424 | 0.421 | 0.423
MemBERT (SS) (P-10-LG-F) | 0.290 | 0.459 | 0.411 | 0.444

##  5 Related Work

A recent trend regarding combining memory-augmented architectures with
transformer architectures views the memory component as an auxiliary support
for efficient learning. More precisely, specific effort is dedicated to
solving the well-known problems of handling long-term dependencies, as well as
of processing long sequences. One common idea is to reduce the search space by
progressively storing intermediate input representations in the memory
component. Notable examples are Transformer-XL [22], Reformer [23], LongFormer
[24] and MemFormer [25]. More precisely, in these architectures the memory
component is directly inserted as an intermediate layer in the transformer
architecture, mainly addressing the multi-head attention step in order to
condition token selection.

In a similar fashion, but with different intent, transformers have been
augmented with an external memory block in order to efficiently deal with
data. This means handling complex input sequences like videos [26] by
overcoming the natural sequential information flow and learning global-level
corpus information. In the latter scenario, global and local attention
mechanisms have been combined by means of learnable parameter vectors [27, 28]
that account for word meanings in different context throughout the corpus
[29].

Another well-known use of the memory block is for external knowledge
integration. In this scenario, contributions are mainly distinguished by the
type of available knowledge and by the task of interest. Transformers have
been directly applied as advanced input encoding methods in traditional
memory-augmented architectures for information retrieval, dialogue systems
[30, 31] and question-answering [32]. Similarly, this combined model has been
used to tackle other well-known tasks like aspect-based sentiment analysis
[33]. In particular, in all these scenarios, the memory block is employed to
store textual passages that are specific of each individual dataset example,
either given as an additional input like question-answering [34] or
attentively extracted from a set of documents following an information-
retrieval phase [30]. Under the same perspective, knowledge bases have also
been encoded as external memory. Differently from our setup, in this scenario
each memory slot encodes a single word or entity tuples [35, 36] and the
underlying intent is to expand the contextual information of words to ease the
learning process. More precisely, knowledge as a richer complementary
vocabulary is just one of its possible formulations. Indeed, with knowledge in
the form of sentences in natural language we cover several scenarios of
different nature, spanning from additional contextual information to general
task-oriented constraints. For instance, the memory could contain textual
descriptions of the task itself and, thus, we could formulate the task of
learning from them while ignoring labels, thus resembling human learning.

Clearly, when dealing with transformer architectures, one major challenge is
interpretability. Many attempts at building interpretable NLP models are based
on the attention mechanism [3]. Although attention can be used to interpret
the behaviour of deep neural networks [37], whether it can also be considered
a proper _explanation_ tool is still an open debate, which hinges on the
definition of explanation itself. In fact, while attention provides a possible
interpretation of a network’s behaviour, it is not possible to guarantee
whether that interpretation is the right one [4, 5]. Therefore, we can
consider attention weights to be an explanation if we define them as a
_plausible_ , but not necessarily _faithful_ , reconstruction of the decision-
making process [38, 39]. It can be argued that attention is not _consistent_
with other explanation methods [4], but the same characteristic has been
observed in many other popular explanation tools [40]. Attention has also been
used as a means to integrate knowledge in models by specifically training it
to select relevant features [41, 42] or to model an auxiliary task [43, 44].
On a slightly different perspective, the “interpretability illusion” of BERT-
related models has been described as the phenomenon for which individual
neurons in BERT not always show a human-interpretable meaning [45]. More in
general, the use of background knowledge to make deep learning systems more
explainable and interpretable has been explored with knowledge graphs [46],
whereas another recent approach proposes to store commonsense knowledge in an
external symbolic memory that interacts with the neural model [47].

##  6 Conclusions

In this paper, we introduced the general problem of unstructured knowledge
integration within BERT models. In particular, we proposed a memory-augmented
version of DistilBERT that follows the traditional MANN architecture. We
explored two challenging experimental settings: (i) class descriptions for
unfair clause detection in the legal domain and (ii) evidence as support for
claim detection in the context of argumentation mining.

In spite of a rather straightforward formulation, promising results have been
achieved concerning classification performance and interpretability. Most
notably, when strong supervision is applied, memory-augmented models learn to
correctly select target memory slots with high precision and, in most cases,
reach higher performance. Memory sampling allows to scale up to large
knowledge bases. In particular, priority-based sampling strategies reach more
consistent performance from the viewpoint of both classification and
interpretability.

The underlying potential of unstructured knowledge integration is particularly
vast, since expert information in plain natural language is easy to acquire.
Many other domains could thus benefit from the proposed kind of approach.

In the future, we plan to explore several research directions. We will apply
this methodology to text generation, for example for argument generation.
Furthermore, we have always considered a fixed memory content, whereas,
certainly, knowledge integration might benefit from a dynamic formulation [8].
Correctly handling examples not covered by the integrated knowledge still
stands as an open problem. Lastly, more sophisticated memory interaction
mechanisms will be investigated to improve the performance of the approach.

## Acknowledgment

The authors would like to thank Andrea Galassi for fruitful discussions. Paolo
Torroni and Federico Ruggeri have been partially supported by the H2020
Project AI4EU (Grant Agreement No. 825619).

## References

  * [1] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv:1810.04805, 2018.
  * [2] Anton Chernyavskiy, Dmitry Ilvovsky, and Preslav Nakov.  Transformers:" the end of history" for nlp?  arXiv preprint arXiv:2105.00813, 2021.
  * [3] Andrea Galassi, Marco Lippi, and Paolo Torroni.  Attention in natural language processing.  IEEE Transactions on Neural Networks and Learning Systems, pages 1–18, 2020.
  * [4] Sarthak Jain and Byron C. Wallace.  Attention is not explanation.  In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proc. of NAACL-HLT 2019, pages 3543–3556. ACL, 2019.
  * [5] Sarah Wiegreffe and Yuval Pinter.  Attention is not not explanation.  In Proceedings of the 2019 EMNLP-IJCNLP Conference, pages 11–20. ACL, 2019.
  * [6] Laura von Rueden, Sebastian Mayer, Katharina Beckh, Bogdan Georgiev, Sven Giesselbach, Raoul Heese, Birgit Kirsch, Michal Walczak, Julius Pfrommer, Annika Pick, and et al.  Informed machine learning - a taxonomy and survey of integrating prior knowledge into learning systems.  IEEE Transactions on Knowledge and Data Engineering, page 1–1, 2021.
  * [7] Zhiting Hu, Xuezhe Ma, Zhengzhong Liu, Eduard Hovy, and Eric Xing.  Harnessing deep neural networks with logic rules, 2020.
  * [8] Zhiting Hu, Zichao Yang, Ruslan Salakhutdinov, and Eric Xing.  Deep neural networks with massive learned knowledge.  In Proc. of EMNLP 2016, pages 1670–1679. ACL, November 2016.
  * [9] Federico Ruggeri, Francesca Lagioia, Marco Lippi, and Paolo Torroni.  Detecting and explaining unfairness in consumer contracts through memory networks.  Art. Int. and Law, pages 1–34, 2021.
  * [10] Kevin D Ashley.  Artificial intelligence and legal analytics: new tools for law practice in the digital age.  Cambridge University Press, 2017.
  * [11] Marco Lippi, Przemysław Pałka, Giuseppe Contissa, Francesca Lagioia, Hans-Wolfgang Micklitz, Giovanni Sartor, and Paolo Torroni.  Claudette: an automated detector of potentially unfair clauses in online terms of service.  Artificial Intelligence and Law, 27(2):117–139, 2019.
  * [12] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.  arXiv preprint arXiv:1910.01108, 2019.
  * [13] Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus.  End-to-end memory networks.  In Advances in NeurIPS 2015 Conference, Montreal, Canada, pages 2440–2448, 2015.
  * [14] Jason Weston, Sumit Chopra, and Antoine Bordes.  Memory networks.  arXiv:1410.3916, 2014.
  * [15] Alex Graves, Greg Wayne, and Ivo Danihelka.  Neural Turing machines.  arXiv preprint arXiv:1410.5401, 2014.
  * [16] Sarath Chandar, Sungjin Ahn, Hugo Larochelle, Pascal Vincent, Gerald Tesauro, and Yoshua Bengio.  Hierarchical memory networks, 2016.
  * [17] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston.  Curriculum learning.  In Proc. of the 26th Int. Conf. on Machine Learning, ICML ’09, page 41–48. ACM, 2009.
  * [18] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver.  Prioritized experience replay.  In 4th Int. Conf. on Learning Representations, ICLR 2016, 2016\.
  * [19] Marco Lippi and Paolo Torroni.  Argumentation mining: State of the art and emerging trends.  ACM Transactions on Internet Technology (TOIT), 16(2):1–25, 2016\.
  * [20] Ruty Rinott, Lena Dankin, Carlos Alzate, Mitesh M Khapra, Ehud Aharoni, and Noam Slonim.  Show me your evidence-an automatic method for context dependent evidence detection.  In Proceedings of the 2015 EMNLP Conference, pages 440–450, 2015\.
  * [21] Noam Slonim, Yonatan Bilu, Carlos Alzate, Roy Bar-Haim, Ben Bogin, Francesca Bonin, Leshem Choshen, Edo Cohen-Karlik, Lena Dankin, Lilach Edelstein, et al.  An autonomous debating system.  Nature, 591(7850):379–384, 2021.
  * [22] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc V. Le, and Ruslan Salakhutdinov.  Transformer-xl: Attentive language models beyond a fixed-length context.  CoRR, abs/1901.02860, 2019.
  * [23] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya.  Reformer: The efficient transformer.  CoRR, abs/2001.04451, 2020.
  * [24] Iz Beltagy, Matthew E. Peters, and Arman Cohan.  Longformer: The long-document transformer.  CoRR, abs/2004.05150, 2020.
  * [25] Qingyang Wu, Zhenzhong Lan, Jing Gu, and Zhou Yu.  Memformer: The memory-augmented transformer.  CoRR, abs/2010.06891, 2020.
  * [26] Marcella Cornia, Matteo Stefanini, Lorenzo Baraldi, and Rita Cucchiara.  M${}^{\mbox{2}}$: Meshed-memory transformer for image captioning.  CoRR, abs/1912.08226, 2019.
  * [27] Sainbayar Sukhbaatar, Edouard Grave, Guillaume Lample, Hervé Jégou, and Armand Joulin.  Augmenting self-attention with persistent memory.  CoRR, abs/1907.01470, 2019.
  * [28] Mikhail S Burtsev, Yuri Kuratov, Anton Peganov, and Grigory V Sapunov.  Memory transformer.  arXiv:2006.11527, 2020.
  * [29] Qiyu Wu, Chen Xing, Yatao Li, Guolin Ke, Di He, and Tie-Yan Liu.  Taking notes on the fly helps BERT pre-training.  CoRR, abs/2008.01466, 2020.
  * [30] Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston.  Wizard of wikipedia: Knowledge-powered conversational agents.  arXiv:1811.01241, 2018.
  * [31] Yanjie Gou, Yinjie Lei, and Lingqiao Liu.  Contextualize knowledge bases with transformer for end-to-end task-oriented dialogue systems.  CoRR, abs/2010.05740, 2020.
  * [32] Tuan Manh Lai, Quan Hung Tran, Trung Bui, and Daisuke Kihara.  A gated self-attention memory network for answer selection.  CoRR, abs/1909.09696, 2019.
  * [33] Ming Jiang, Junlei Wu, Xiangrong Shi, and Min Zhang.  Transformer based memory network for sentiment analysis of web comments.  IEEE Access, 7:179942–179953, 2019.
  * [34] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang.  Squad: 100, 000+ questions for machine comprehension of text.  CoRR, abs/1606.05250, 2016.
  * [35] Haihong E., Wenjing Zhang, and Meina Song.  Kb-transformer: Incorporating knowledge into end-to-end task-oriented dialog systems.  In 15th SKG Conference, pages 44–48, 2019.
  * [36] Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis.  Generalization through memorization: Nearest neighbor language models.  CoRR, abs/1911.00172, 2019.
  * [37] Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi.  A survey of methods for explaining black box models.  ACM Comput. Surv., 51(5):93:1–93:42, August 2018.
  * [38] Cynthia Rudin.  Please stop explaining black box models for high stakes decisions.  Nature Machine Intelligence, 1:206–215, 2019.
  * [39] Mark O. Riedl.  Human-centered artificial intelligence and machine learning.  Human Behavior and Emerging Technologies, 1:33–36, 2019.
  * [40] Michael Neely, Stefan F. Schouten, Maurits J. R. Bleeker, and Ana Lucic.  Order in the court: Explainable AI methods prone to disagreement.  CoRR, abs/2105.03287, 2021.
  * [41] Tingting Qiao, Jianfeng Dong, and Duanqing Xu.  Exploring human-like attention supervision in visual question answering.  In AAAI, pages 7300–7307. AAAI Press, 2018.
  * [42] Yujia Bao, Shiyu Chang, Mo Yu, and Regina Barzilay.  Deriving machine attention from human rationales.  In EMNLP, pages 1903–1913. ACL, 2018.
  * [43] Emma Strubell, Patrick Verga, Daniel Andor, David Weiss, and Andrew McCallum.  Linguistically-informed self-attention for semantic role labeling.  In Proc. of the 2018 EMNLP Conference, pages 5027–5038. ACL, 2018\.
  * [44] Jianfei Yu, Luis Marujo, Jing Jiang, Pradeep Karuturi, and William Brendel.  Improving multi-label emotion classification via sentiment classification with dual attention transfer network.  In EMNLP, pages 1097–1102. ACL, 2018.
  * [45] Tolga Bolukbasi, Adam Pearce, Ann Yuan, Andy Coenen, Emily Reif, Fernanda Viégas, and Martin Wattenberg.  An interpretability illusion for bert.  arXiv preprint arXiv:2104.07143, 2021.
  * [46] Manas Gaur, Keyur Faldu, and Amit Sheth.  Semantics of the black-box: Can knowledge graphs help make deep learning systems more interpretable and explainable?  IEEE Internet Computing, 25(1):51–59, 2021.
  * [47] Pat Verga, Haitian Sun, Livio Baldini Soares, and William W Cohen.  Facts as experts: Adaptable and interpretable neural memory over symbolic knowledge.  arXiv preprint arXiv:2007.00849, 2020.