# Visualizing and Explaining Language Models

Adrian M.P. Braşoveanu
Modul University Vienna
and Modul Technology GmbH
Am Kahlenberg 1, 1190, Vienna, AT
<EMAIL_ADDRESS>
Răzvan Andonie
Central Washington University
400 E University Way,
Ellensburg, WA 98926, USA,
and Transilvania University
Bulevardul Eroilor 29, Brasov 500036, Romania
<EMAIL_ADDRESS>

###### Abstract

During the last decade, Natural Language Processing has become, after Computer
Vision, the second field of Artificial Intelligence that was massively changed
by the advent of Deep Learning. Regardless of the architecture, the language
models of the day need to be able to process or generate text, as well as
predict missing words, sentences or relations depending on the task. Due to
their black-box nature, such models are difficult to interpret and explain to
third parties. Visualization is often the bridge that language model designers
use to explain their work, as the coloring of the salient words and phrases,
clustering or neuron activations can be used to quickly understand the
underlying models. This paper showcases the techniques used in some of the
most popular Deep Learning for NLP visualizations, with a special focus on
interpretability and explainability.

_Keywords_ Language Models $\cdot$ Transformer $\cdot$ Explainable Language
Models $\cdot$ Visualization of Neural Networks

##  1 Introduction

Deep Learning (DL) models applied on texts need to cover the morphological,
syntactic, semantic and pragmatic layers. Crafting networks that operate on so
many levels is a challenging task due to the sparseness of the training data.
Such networks have been traditionally called Language Models (LMs) Ponte and
Croft (1998). The early iterations were based on statistical models Ponte and
Croft (1998), whereas the latest iterations use neural networks and
embeddings. Current LMs are trained on large corpora and are generally sparse.
Most current popular LMs are based on Transformer architectures Brown et al.
(2020).

The first implementation of a Transformer network Vaswani et al. (2017) proved
that it was possible to design networks that achieve good results for Natural
Language Processing (NLP) tasks with a set of multiple sequential attention
layers. A Transformer contains a series of self-attention layers that are
distributed through its various components. Self-attention is an attention
mechanism that computes a representation of a sequence from a set of different
positions of the same sequence. The Transformer model itself is simple and
consists from pairs of encoders and decoders. Encoders encapsulate layers of
self-attention coupled with feed-forward layers, whereas decoders encapsulate
self-attention layers followed by encoder-decoder attention and feed-forward
layers. The attention computation is done in parallel and the results are then
combined. The result is termed a multi-head attention, and it provides the
model with the ability to orchestrate information from different
representation subspaces (e.g., multiple weight matrices) at various positions
(e.g., different words in a sentence) Vaswani et al. (2017). Its outputs are
fed either to other encoders or into decoders, depending on the architecture.
There is no fixed number of encoders and decoders which can be included in
this architecture, but they will typically be paired (e.g., 10 encoders and 10
decoders). In newer architectures, encoders and decoders can also be used for
different tasks (e.g., encoder for Question Answering, and decoder for Text
Comprehension) Raghu and Schmidt (2020). While the model was initially
developed for machine translation tasks, it has been tested on multiple
domains and was demonstrated to work well.

During the last three years, hundreds of papers and LMs inspired by
Transformers were published, the best-known being BERT Devlin et al. (2019),
RoBERTa Liu et al. (2019), AlBERT Lan et al. (2020), XLNet Yang et al. (2019),
DistilBERT Sanh et al. (2019), and Reformer Kitaev et al. (2020). Some of the
most popular Transformer models are included in the Transformers library,
maintained by HuggingFace Wolf et al. (2019).

Many of these models are complex and include significant architectural
improvements compared to the early Transformer and BERT models. Explaining
their information processing flow and results is therefore difficult, and a
convenient and very actual approach is visualization. Our survey is focused on
visualization techniques used to explain LMs. We investigate two large tool
classes: (i) model-agnostic tools that can be used to explain BERT
predictions; and (ii) custom visualizations that are focused only on
explaining the inner workings of LMs based on neural networks. An early
version of this survey was published a year ago, but it was focused only on
visualizing Transformer networks Braşoveanu and Andonie (2020). We have since
extended the material to include new articles about Transformer
visualizations, other types of networks, as well as an extended section about
model-agnostics AI libraries that are focused on interpretability and
explainability for NLP.

In this survey, we look at the visualization of several types of LMs based on
DL networks, review the basic charts and patterns present in them and try to
understand the basic methodology that was used to produce these visual
representations. The rest of the paper is organized as follows: Section 2
presents the motivation and methodology of this survey. Section 3 showcases
the two classes of tools, whereas Section 4 discusses the various findings.
The paper concludes with some thoughts on the future of this class of
visualizations.

##  2 Background and Methodology

The need to quickly update NLP models in case of unforeseen events suggests
that developers will be well-served by explainable AI and visualization
libraries, especially since debugging Transformers is a complex task.
Visualizations are particularly important, as they help us debug the various
problems that such models exhibit and which can only be discovered through
large-scale analyses.

Traditional visualization libraries are based on the classic grammar of
graphics philosophy Wilkinson (2005) which is focused on the idea that
visualizations are compositional by design. They provide various visualization
primitives like circles or squares and a set of operations that can be applied
on top of these primitives to create more complex shapes or animations.
Unfortunately, such traditional visualization libraries like D3.js Bostock et
al. (2011), Vega Satyanarayan et al. (2017) or Tableau111www.tableau.com, do
not offer specific functions for visualizing feature spaces, neural network
layers or support for iterative design space exploration Park et al. (2018)
when designing AI models. What this means is that for AI tasks, a lot of the
functionality will have to be developed from scratch.

When visualizing more complex models like those built with Transformers, we
typically need to understand all the facets of the problem, from the data and
training procedure, to the input, network layers, weights or the outputs of
the neural network. The outputs are the core of explainability, as people will
not use the networks in commercial products if they can’t explain how the
outputs were obtained in the first place. What is also beneficial is to
highlight the paths that lead to certain outputs, as this illuminates the
features or parts of the models that may need to be changed to achieve the
desired results. This can sometimes be accomplished by using model-agnostic
tools specifically built for benchmarking or hyperparameter scoring, such as
Weights and Biases. We include such tools in our survey if examples of how to
use them for visualizing Transformers already exist, either in scientific
papers or other types of media posts (Medium posts, GitHub, etc.).

The second big class of visualizations discussed in this article is,
naturally, the class of visualizations specifically built around Transformers,
either for explaining it (like ExBERT Hoover et al. (2020)), or for explaining
certain model specific attributes (like embeddings or attention maps Vig
(2019a)).

We selected the libraries and visualizations presented here by reviewing the
standard Computer Science (CS) publication libraries (e.g., IEEE, ACM,
Elsevier, Springer, Wiley), but also online media posts (YouTube, Medium,
GitHub and arXiv). In this extremely dynamic research field, some articles
might be published on arXiv even up to a year before they are accepted for
publication in a traditional conference or journal, time in which they might
already garner hundreds of citations. The original BERT article Devlin et al.
(2019) and also one of the first articles that used visualization to explain
it Clark et al. (2019) were cited over a hundred times before being published
in conference proceedings.222Article Clark et al. (2019) has garnered 149
citations at the moment of the submission, before being published in a
conference or journal.

When testing new models, benchmarking and fine-tuning are the two operations
where we might spend the most time, as even if the scores are good, we might
want to try different hyperparameter settings (e.g., learning rate, number of
epochs, batch size, etc) Florea and Andonie (2019). A hyperparameter sweep (or
trial) is a central notion in both hyperparameter optimization and
benchmarking. It involves running one or multiple models with different values
for their hyperparameters. Since quite often the main goal behind running such
sweeps is improving existing models, but it is not necessarily related to
interpretability and explainability, we decided to include a minimal number of
such libraries here.

TensorBoard333https://www.tensorflow.org/tensorboard is a specialized
dashboard deployed with Google TensorFlow that covers the basic visualization
needs for ML experiments, from tracking, computing and visualizing metrics, to
model profiling and embeddings. It is not necessarily a good tool for creating
custom visualization or for explaining results, but it can be a good tool for
improving accuracy. It is sometimes also used with TensorFlow’s competing
libraries like PyTorch or FastAI. Neptune444https://neptune.ai/ is an open-
source ML benchmarking suite deployed for a variety of collaborative
benchmarking tasks, including notebook tracking.
Sacred555https://github.com/IDSIA/sacred and
Comet.ml666https://www.comet.ml/site/ are Neptune alternatives that provide
basic charting capabilities and dedicated dashboards. Weights and
Biases777https://www.wandb.com/ provides perhaps the largest sets of
visualization and customization capabilities. It comes packed with advanced
visualizations that include parallel coordinates Heinrich and Weiskopf (2015),
perhaps the best method to navigate hyperparameter sweeps. It is the easiest
and the most agile solution to integrate with production code or Jupyter
notebooks out of all the ones mentioned here. Ray Moritz et al. (2018), a
distributed benchmarking framework that contains its own fine-tuning engine
called Tune Liaw et al. (2018) is popular for optimizing Transformers.

Due to space limitations, we resume ourselves to discussing only the most
interesting visualizations, especially in the model-agnostic visualization
section, as otherwise this article could easily become an entire book.

##  3 Visualizing Language Models

Language models are difficult to train for a multitude of reasons, including
(but not limited to) cost, time or carbon footprint Strubell et al. (2019).
Most of the LMs need to be trained on GPUs or TPU pods for days or weeks. Due
to their generalization capabilities, such LMs can reliably estimate the
actions for which they have a reasonable number of examples in their training
datasets, whereas in cases with fewer examples they might overestimate the
predicted actions therefore inserting some bias Shwartz and Choi (2020).
Debugging or retraining such models therefore becomes a necessity, even if the
costs of such operations are still high. Visualization is just one of the
methods that can help us explain such large LMs, especially since it is often
combined with linguistics or statistics. Explaining the results in plain
English should be what we are aiming for when we build new LMs, but this may
sometimes require additional steps. An interpretation of the results, for
example, would generally depend on the target domain (e.g., medicine, law,
etc) Vellido (2020), as in some cases a complex reasoning process (e.g.,
compliance with local or international regulations) may need to be applied
before selecting the right words for an explanation. The visualization will
essentially highlight the intermediary steps (e.g., the components that lead
to a side effect for medication or a legal aspect in a certain jurisdiction)
required to create a basic interpretation, and therefore it is often kept
minimal and visualized features or processes are carefully selected. If it is
easy to navigate the various information pathways and understand the results
and their interpretations (e.g., where they may lead us), it may be safe to
call the respective visualizations explainable.

Using visualization to explain the AI processes is an expanding research
field. The main idea behind AI user interfaces should be to augment and expand
user capabilities, rather than replace intelligence Heer (2019). While not
necessarily needed to understand the next section, several recent surveys
about visualizations and DL can help provide additional context to the
interested readers. We particularly recommend the following: the introduction
on how Convolution Neural Networks "see" the world from Qin et al. (2018), the
discussion on visual interpretability from Zhang and Zhu (2018), and the
discussion on the importance of visualizing features from Nguyen et al.
(2019).

###  3.1 Model-Agnostic Explainable AI Tools

Explainable888Explainable points to the idea of describing or explaining in an
intuitive manner, via charts or tables, the prediction of an algorithm. AI
(XAI) is the key to enterprise adoption of the current wave of AI
technologies, from vision to NLP and symbolic computation. An early XAI survey
Hohman et al. (2019) describes methods through which visualizations can be
turned into explanations for the AI models and goes on to define the
terminology of the field. Early XAI libraries focused on visualizing ML
features, whereas recent libraries are focused on visualizing embeddings,
attention maps or various neural network layers Samek et al. (2019).

Traditionally, the first step towards transparency was to describe the
contribution of each feature to the final result Guyon and Elisseeff (2003).
This often lead to partial explanations of the results, as in reality if the
models themselves were black-box, knowing the name of the features was not in
itself enough. Output is definitely the most important part that we would like
to explain, but not the only one. To create full explanations, we need to be
able to explain the entire process from its input, to its various
transformations (e.g., layers), training process and output. Explanations also
need to be able to reflect state changes. For example, when computing Shapley
values feature contributions are combined, and then a score that signifies the
feature importance within that set of features is generated. If features are
added or removed from this bundle, the Shapley value for a particular feature
will change accordingly. This dynamic nature of the explainability is rarely
explained, but it is one of the reason why visualizations in particular are a
good fit for creating explanations in the first place.

Some early model-agnostic XAI libraries that were applied to NLP and
Transformers visualizations include LIME Ribeiro et al. (2016) and SHapley
Additive exPlanations (SHAP) Lundberg and Lee (2017). The later was introduced
to unify multiple explanation methods into a single model for interpreting
predictions. Both SHAP and LIME can be used with classical ML libraries like
scikit-learn and XGBoost, as well as with modern DL libraries like PyTorch or
Keras / TensorFlow. SHAP provides visualizations for summary and dependency
plots. Another XAI alternative to SHAP and LIME, ELI5 Fan et al. (2019), is
currently routinely used for explaining BERT predictions, and was found to be
more secure in case of adversarial network attacks Slack et al. (2020).

The visualizations created with LIME and SHAP are typically restrained to
classic charts (e.g., line, bars or word clouds. The summary plots or
interaction charts Lundberg and Lee (2017) from SHAP are relatively easy to
understand, whereas the more complex force plot charts like feature impact
Lundberg et al. (2018) are not necessarily easy to use as they require a
certain learning time. While the feature impact chart simply plots the
expected feature impact with red (features with positive contribution to the
prediction result) or blue (features with a likely negative contribution to
the result) colors and should in theory be an easy-to-understand chart, there
are no direct (e.g., in chart via a legend) explanations on how to interpret
the start or end values, or what do the indicators placed on top of various
components mean in some cases. The interpretation of such force plot charts is
generally missing and people need to read additional documentation to
understand the results. This is far from ideal, as, in our opinion,
visualizations need to be self-explanatory.

It can be argued that explainability libraries like SHAP or LIME tend to focus
on highlighting correlations or statistical effects rather than features, and
are, therefore, less reliable than interpretable models which only showcase a
list of features or algorithms that contributed to the results. We consider
auditing to encompass both sides of the problem, as interpretability and
explainability are the key towards understanding and clearing such LMs for
deployment in real-world products. Keeping this in mind, we think that
focusing on neural network visualization for NLP can only help in this
process, as visualizations can help process, select and highlight the most
important features included in such models.

Both SHAP and LIME were proven to be easily fooled with adversarial attacks
Slack et al. (2020). The idea of deploying biased classifiers for tasks like
credit rating, recommendation or search ranking sounds a bit counterintuitive
because a single classifier should not be able to do much harm. However, given
the fact that such large models typically end up being ensembles, one single
classifier can actually lead to severe damage including wrong predictions,
different sets of biases and ultimately even different outputs than the ones
typically expected from the respective LMs. Not being able to correctly audit
such models (e.g., investigate their output and the features that have
contributed the most to it) can lead to problems with clients and regulatory
agencies.

The list of attacks that can be perpetrated using LMs is extended every month,
and therefore models may need to be periodically tested to assess their
suitability for certain tasks. Some of the most robust attacks include:
creating token sequences that act like universal adversarial triggers on
specific target predictions when concatenated to any input from a dataset
Wallace et al. (2019a); training data extraction attacks in which text
sequences like public information or code are extracted from the training
corpora and used to attack the trained language model Carlini et al. (2020);
spelling attacks in which random spellings for well-known words are generated
through modifying the gradients during training Sun et al. (2020); hotflip
attacks in which a gradient-based embeddings swap is performed to change
classification results Ebrahimi et al. (2018) or even textfooler attacks Jin
et al. (2020) in which multiple attributes like embeddings, part-of-speech
matches or cosine similarities are used to perform a counter-fitted embeddings
swap to fool untargeted classifiers and entailment relations. Possibilities to
reuse part of the code of these attacks to create new attacks also exists if
frameworks like TextAttack Morris et al. (2020) or OpenAttack Zeng et al.
(2020) are used. Sometimes these kinds of adversarial attacks can also be used
to improve results, as demonstrated by improving aspect-based sentiment
analysis tasks by creating artificial sentences Karimi et al. (2020) or by
using various BERT attribution patterns (e.g., pruned self-attention heads
from a certain task) as adversaries Hao et al. (2020). To understand these
attacks, visualization can be a useful tool. For example, Chen Chen et al.
(2020) showcases three types of attacks perpetrated at character, word and
sentence level using visualizations built around various metrics like accuracy
and successful attack ratios. The Interpret framework Wallace et al. (2019b)
uses saliency maps color highlights to showcase defenses against hotflip and
untargeted classification attacks. A later publication then shows how almost
all interpretations built with Interpret can be manipulated through gradient
attacks (e.g., using some large gradients for irrelevant words) Wang et al.
(2020). However, most of the visualization efforts have been focused on
explaining the various neural network activations from Transformer networks,
rather than on the various attacks that can fool Transformers, therefore the
visualization of such attacks is a relatively nascent area.

An alternative approach to such explainability libraries that can easily pick
up wrong signals could be to simply use models designed specifically to be
interpretable. The Neural Additive Models (NAMs) combines the features of
classic DNNs with the interpretability approach advocated by Generalized
Additive Models (GAM) Agarwal et al. (2020). However, since such models are
quite recent, their applicability to NLP has not yet been fully explored.

Interpretability and explainability are often used with interchangeable
meaning. It has to be noted that quite often, interpretability has a domain-
specific component Vellido (2020), whereas explainability is a more general
term. Explainability is the term preferred by Information Visualization
designers and researchers, whereas interpretability is generally the term that
is preferred by ML researchers, statisticians and mathematicians.

Many other explainable AI libraries use Shapley values for computing feature
importance. However, in many cases we were only barely able to discover
mentions of their usage for NLP (e.g.,
DeepExplain999https://github.com/marcoancona/DeepExplain), and therefore we
decided not to include them in this survey.

###  3.2 Visualizing Recurrent Neural Networks for NLP

When visualizing LMs, it is best to start with the language resources used for
their creation, from corpora to embeddings. To uncover biases in such large
models we need to study gender differences, disciplines, languages, cultural
context or regional and diachronic variations. A good method to include such
information and compare resources for language variation is showcased in
Fanhauser’s work Fankhauser et al. (2014). It uses a grids, heatmaps and word
clouds to provide quick access to large amounts of data about English
dialects. Another work uses scatter plots to visualize how and why large
corpora differ Kessler (2017). Similar methods have later been used for
visualizing large-scale embeddings.

Karpathy Karpathy et al. (2015) proposed a method through which to visually
interpret the results of Recurrent Neural Networks (RNNs), Long Short-Term
Meories (LSTMs) and Gated Recurrent Units (GRUs). He suggests that using
interpretable activations can help navigate longer texts, whereas saturation
plots would help showcase the gated units statistics.

Table 1: Articles focused on explaining RNN LMs through visualizations. Topic | Visualization Subject | Chart Type
---|---|---
a) Special topics |  |
predicting next word Luo et al. (2019) |  syntactic heights head attentions |  parallel lines heatmaps
emergence of units Lakretz et al. (2019) |  activations of cells and gates connectivity |  line charts connectivity charts
b) Hidden states |  |
visualizations of representations Li et al. (2016) |  modification and negation clause composition first-derivative saliency |  t-SNE heatmap saliency bars saliency grids
hidden states semantics Sawatzky et al. (2019) |  Predictive Semantic Encodings performance metrics |  PSE charts Bar charts
activation of RNNs Karpathy et al. (2015) |  cells with interpretable activation |  saturation plots bar charts overlap charts
hidden states LSTMVis Strobelt et al. (2018) |  phrase selections hidden state patterns |  hidden pathways tables with cell activation PCA
hidden states RNNVis Ming et al. (2017) |  navigation sentence hidden state word |  control panel glyph-based chart state clusters word clusters
hidden states ActiVis Kahng et al. (2018) |  navigation neuron activation instance selection |  model overview heatmaps
c) Graph Convolution |  |
inductive text classification Zhang et al. (2020) |  attention performance metrics |  attention map line charts
unsupervised domain adaptation Wu et al. (2020) |  embeddings performance metrics |  PCA line charts
GCN with label propagation Wang and Leskovec (2020) |  node embeddings performance |  graphs line charts

Around the same time, an LM visualization survey Li et al. (2016) notes that
classic Computer Vision visualization was focused on inverting
representations, back-propagation and generating images from sketches, all
techniques that work well for images. In NLP, however, it is important to
focus on important keywords, composition and dimensional locality; especially
since many of the words will depend on the context Li et al. (2016). Important
models will be able to capture this kind of information, and therefore it
should be present in visualizations. The early saliency heatmaps clearly
showcased these aspects for LSTM and Bi-Directional LSTMs Strobelt et al.
(2018). By later adapting the ideas of first order saliency from Computer
Vision, researchers were able to highlight intensification and negation, as
well as differences between two sequences at various at consecutive time
steps. Their saliency heatmap for SEQ2SEQ auto-encoders also works well for
predicting corresponding tokens at each time step Luo et al. (2019).

The main characteristic that connects these papers, as it can easily be
observed in Table 1, regardless of the number of visualizations included in
them, is the fact that they are focused on a interpretability and
explainability of NLP models through visualization. We have analyzed the
following characteristics:

  * •

Topic \- the main topic of the paper (e.g., attention, representation,
information probing) followed by the papers in which this topic is addressed;

  * •

Visualization Subject \- since visualizations included in these papers were
focused on a large set of subjects from Transformer components (e.g.,
attention heads), to correlation between tasks (e.g., via Pearson correlation
charts) or performance (e.g., accuracy or other metrics represented via line
charts), we have decided to extract all these in a separate column to
understand what kind of charts we might be interested in creating when
exploring a certain topic.

  * •

Chart Type \- includes the various types of visual metaphors used for
rendering the chosen subjects. Most of the chart types are classic (e.g.,
line, bar chart, t-SNE), very few being rebranded (e.g., attention maps are
heatmaps) or actually new (e.g., comparative attention graphs). The chart
names need to give us a clear idea of what they represent.

A large class of visualizations is dedicated to the activation of neurons and
the representation of hidden states, as it can easily be seen from Table 1.

LSTMVis Strobelt et al. (2018) uses a large grid of sentences for matching the
various state patterns that are then expanded through additional views in the
same interface. The key innovation of visualizing hidden state changes keeps
the focus on the right keywords, whereas the match views help enlighten
particular cases. Many other visualizations similar to LSTMVis (e.g., RNNVis
Ming et al. (2017) or ActiVis Kahng et al. (2018)) follow the same template: a
control panel is used for selecting the phrase or sentence, a middle view is
focused on the word clusters or neuron activations, whereas the last view is
typically a matrix view with highlighted cells which showcases the important
words that are featured in the activated pathways. Such integrated views offer
us holistic views of what these models can accomplish, except for the fact
that they are rarely focused also on the corpora that was used for training.
In time, the visualization designers started to include this information as
well, as we will observe in the next section.

We considered that Graph Convolutional Networks are also worth exploring, but
since there are entire libraries dedicated to this task which are not
necessarily model-agnostic (e.g., PyTorch Geometric Fey and Lenssen (2019)) we
have limited ourselves to including some papers that offer some classic
visualizations that are typically included in such libraries (e.g., node
embeddings, performance).

###  3.3 Visualizing Transformers for NLP

No other types of neural networks have led to such an increased demand for
custom visualizations since the days of the Kohonen’s Self-Organizing Maps
Kohonen (1995) or Manbelbrot’s fractals Mandelbrot (1977) like the
Transformers. When selecting Transformer visualization papers, we decided to
focus on the most important topics related to interpretability and
explanation. We have therefore eliminated papers that used only classic charts
(e.g., bars, lines, pies). We decided to focus on the works that tried to
visualize as many aspects of Transformer models as possible, from attention
maps, to structural or informational probing, neural network layers, and
multilingualism.

Many Transformer visualizations are focused on attention. While the attention
mechanism is indeed important for the Transformer architecture, and it
improves results for NLP tasks, they are not necessarily easy to interpret if
sophisticated encoders are used Jain and Wallace (2019). Due to this fact, it
is often not easy to test various explanations by simply modifying the weights
and verifying if the outputs are also changed as a result of this. Alternative
theories suggest that simply looking at information flows through such models
is not enough, and that attention should only be used as explanation if
certain conditions are met, e.g., if the weight distributions found via
adversarial training do not perform well Wiegreffe and Pinter (2019). Since
attention is central to Transformer models, many visualizations are rightly
focused on this topic. The fact that such visualizations capture the dynamic
nature of the output is not necessarily sufficient to consider them
explanations. A good explanation needs to highlight the reasoning chain that
lead to the particular output. This is the main reason why we have mainly
looked for those visualizations that focus on multiple aspects of the network
in this work.

Table 2: Articles focused on explaining Transformer topics through visualizations. Topic | Visualization Subject | Chart Type
---|---|---
a) Attention |  |
attention explanation Jain and Wallace (2019) Wiegreffe and Pinter (2019) |  feature importance correlation permutation adversarial attention performance metrics |  Kendall rank statistics scatterplots adversarial charts multiple line charts
attention flow Abnar and Zuidema (2020) DeRose et al. (2021) |  raw attention raw attention map comparative attention flows |  attention graph heatmap comparative attention graphs
multi-head self-attention Voita et al. (2019a) |  layers attention for rare words dependency scores active heads |  importance charts heatmaps bar charts line charts
b) Hidden states |  |
intermediate layers Song et al. (2020) |  intermediate layers clustering |  PCA
information interactions interpretation Hao et al. (2020) |  scoring attention information flow for tokens evaluation accuracy attention heads correlation |  heatmap attribution graphs line charts Pearson correlation charts
causal mediation analysis Vig et al. (2020a) |  indirect effects effects comparison attention |  heatmaps line chart attention heads
evolution of representations Voita et al. (2019b) |  token changes and influences distances between layers token occurences |  line charts line charts t-SNE clustering
c) Probing |  |
structural probes Tenney et al. (2019) |  summary statistics layer-wise performance predictions probing |  bar chart bar distribution chart multiple bar charts
multilingual probes Dufter and Schütze (2020) Eger et al. (2020) |  probing task positional embeddings performance metrics stability of training size |  PCA cosine similarity matrices line charts bar charts
information theoretic probing of classifiers Voita and Titov (2020a) |  coding components performance metrics |  bar charts line charts
psycholinguistics tests Gauthier et al. (2020) |  comparing predictions performance metrics test content |  tables distribution charts line charts

The recent success of Transformers helped power many NLP tasks to the top of
the leaderboards. BERT visualizations have focused on explaining these great
results through visualizations, therefore highlighting: (i) the role of
embeddings and relational dependencies within the Transformer learning
processes Reif et al. (2019); (ii) the role of attention during pre-training
or training (e.g., Su et al. (2020) or Vig (2019a)) or (iii) the importance of
various linguistic phenomena encoded in its language model like direct
objects, noun modifiers, or possessive pronouns. Clark et al. (2019).

Current XAI methods for Transformer models have further developed and
supported the idea that understanding the linguistic information which is
encoded in the resulting models is key towards understanding the good
performances in NLP tasks. Probing tasks Conneau et al. (2018) are simple
classification problems focused on linguistic features designed to help
explore embeddings and LMs. For example, by using structural probing Hewitt
and Manning (2019), structured perceptron parsers Maudslay et al. (2020)) or
visualization (e.g., as demonstrated through BERT embeddings and attention
layers visualizations like those from Clark et al. (2019) and Vig (2019a)),
one should be able to understand what kind of linguistic information is
encoded into a Transformer model, but also what has changed since previous
runs.

We have discovered two large classes of Transformer visualizations:

  * •

Focused \- visualizations centered on a single subject like attention. The
papers themselves might present multiple visualizations, but these
visualizations are not single tools.

  * •

Holistic \- visualizations or systems which seek to explain the entire
Transformer model or lifecycle.

####  3.3.1 Focused Transformer Visualizations

The most important papers dedicated to focused visualizations are summarized
in Table 2. We used the same conventions in this table like the ones applied
in Table 1.

We can clearly distinguish several large topics in this group of focused
papers: the relation between attention and model outputs (e.g., especially in
Jain and Wallace (2019), Wiegreffe and Pinter (2019), Abnar and Zuidema
(2020), Voita et al. (2019a)), the analysis of captured linguistic information
via probing (e.g., in Voita et al. (2019b),Tenney et al. (2019)), the
interpretation of information interaction (e.g., in Hao et al. (2020), Voita
et al. (2019b)), and multilingualism (e.g., in Tenney et al. (2019) or Eger et
al. (2020)). In fact, if we look close to Table 1 we can distinguish 3 large
classes of subjects: a) attention; b) hidden states and c) structural or
information probing. Papers that work on similar topics also tend to use the
same kind of visual metaphors. This sometimes happens due to replication of a
previous study (e.g., Wiegreffe and Pinter (2019) replicates the experiments
from Jain and Wallace (2019) to prove that attention weights do not explain
everything), whereas in other cases this happens because there is no need for
more complicated visual metaphors (e.g., line charts are used in more than
half of the papers to represent performance). Besides the widespread use of
the heatmaps that represent attention maps, one chart type that deserves to be
highlighted in this category is the attention graph Hao et al. (2020) which
tracks the information flow between the input tokens for a given prediction.

One of the key methods used for explaining the output of LMs is called probing
Ettinger et al. (2016), used to analyze the linguistic information encoded in
a fixed length vector. Recent iterations like structural probes Hewitt and
Manning (2019) evaluate if syntax trees are embedded in a network’s word
representation space. Identifying linear transformations is evidence for
entire syntax trees embedded in the LM’s vector geometry. This method works
well for limited cases in which distances between words are known. The critics
of probing argue that differences in probing accuracy between the various
classifiers essentially render them unusable as they fail to distinguish
between different representations, e.g., two LMs can end up having different
linguistic representations even if based on the same initial BERT model. In
such scenarios, one cannot compare the accuracy of the classifiers used to
predict their labels. To counteract for such cases, a recent information-
theoretic probing with minimum description length method was proposed Voita
and Titov (2020b). The basic idea is that instead of predicting labels, the
probe transmits data (a description) which is then evaluated based on the
returned description’s length. Such probes can be implemented on top of the
classic structural probes, and are fairly stable. According to Pilault Pilault
et al. (2020), classic structural probes may not be enough for complex tasks
like summarization simply because it seems that increasing the number of
random encoders provide significant performance boosts. This suggests that the
information-theoretic probes with minimum description length may be the better
probes for this task, however this remains to be demonstrated by future
experiments, as Voita’s model was published after Pilault’s article.

####  3.3.2 Holistic Transformer Visualizations

While in Section 3.2 several groups tried to expand their visualizations of
hidden states to encompass the entirety of the model, they have rarely
included the corpora provenance or an easy method to navigate it. Due to this
aspect, in the case of Transformers, since such visualizations often included
the whole lifecycle (e.g., including the corpora with known provenance), we
decided to present them in a separate subsection.

Some of the most interesting tools or papers included in the category of
holistic visualizations are compared in Table 3. These visualization systems
typically integrate most of the components of a Transformer and provide
detailed summaries of them. We have examined two large classes of attributes:

  * •

Components represents the various components of the neural networks: from
corpus, to embeddings, positional heads, attention maps or outputs.

  * •

Summary includes the various types of views that offer us information about
the state of a neuron or a layer, as well as overviews, statistics or details
about the various errors encountered. Statistics might include different types
of information: from correlations between layers or neurons to statistical
analyses of the results. The errors column represents any error analysis
method through which we can highlight where a particular error comes from
(e.g., corpus, training procedure, layer, etc). While it can be argued that
neuron or layer views should be included in the components section, the way
these views are currently implemented suggests they are rather summaries, as
neurons or layers can have different states.

We have decided against including chart types in Table 3, as each
visualization suite or paper included some novel visualization types besides
attention maps (heatmaps), parallel coordinates or line and bar charts.

Table 3: Comparison of holistic Transformer visualizations. Legend: Corpus
(C); Embeddings (E); Heads (H); Attention Maps (A); outputs (out); errors
(err); neuron (neu); layers (lay); overview (ove); statistics(sta)

.  Article Components Summary C E H A out err neu lay ove sta BertViz Vig
(2019b) ✓ ✓ ✓ ✓ ✓ ✓ ✓ Clark Clark et al. (2019) ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ VisBERT van
Aken et al. (2020) ✓ ✓ ✓ ✓ ✓ ✓ ✓ ExBERT Hoover et al. (2020) ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
AttViz Skrlj et al. (2020) ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ Vector Norms Kobayashi et al.
(2020) ✓ ✓ ✓ ✓ ✓ ✓ Dictionary Learning Yun et al. (2021) ✓ ✓ ✓ ✓ ✓ ✓

In our view, none of the examined visualization systems has yet managed to
examine all the facets of the Transformers. This is perhaps because this area
is relatively new and there is no consensus on what needs to be visualized.
While it is quite obvious that individual neurons or attention maps
(regardless of if they are averaged or not) are useful, and it is best to
visualize them, the same cannot be said about the training corpora today, as
only a few systems considered this aspect (e.g., Skrlj et al. (2020) and
Hoover et al. (2020)). This is not really ideal, as lots of errors might
simply come from a bad corpora, but researchers might simply not be aware of
them Brasoveanu et al. (2018). Errors themselves are only seriously discussed
in a single publication Clark et al. (2019). ExBERT Hoover et al. (2020) and
AttViz Skrlj et al. (2020) deserve a special mention here, as they combine
different views on the corpus, embeddings and attention maps to provide a
holistic image of a Transformer model.

A study that looks at the similarity and stability of neural representations
in LMs and brains van der Heijden et al. (2020) shows that combining
predictive modelling with Representation Similarity Analysis (RSA) techniques
can yield promising results. This article deserves a special mention as it can
be included in both focused and holistic visualizations. Their visualizations
are rather basic in terms of design, but they contain lots of insights, for
example one of the tables they produced showcases the RSA results for various
layers of multiple models like BERT, Elmo and others. These kinds of analyses
are rather new, and we hope they will become more common in the next years, as
they might help us clarify which LMs are more similar to the human brains.

####  3.3.3 Vision Transformers

Transformers have been applied in numerous fields due to their flexibility.
Perhaps the most important application is the unification of vision and NLP.
In the last two years, this has seen explosive growth. It would be impossible
to cover all the various models in this article, and therefore we have
selected only a few interesting topics related to this expansion. The one that
comes to mind first is Visual Question Answering (VQA) Gan et al. (2020), as
this is perhaps the task that multimodal researchers have been trying to solve
for decades. Vision Transformers offer an elegant, but expensive solution.
Table 4 provides a short summary of interesting papers in this research
direction.

Table 4: Articles focused on explaining Language and Vision (LAV) models through visualizations. Domain | Topic | Chart Type
---|---|---
language and vision Cao et al. (2020) Han et al. (2021) |  modality importance layer-level importance |  attention heatmaps line charts feature maps
biomedical Li et al. (2020) | attention maps | head attention maps
VQA Gan et al. (2020) Kim et al. (2021) |  multimodal alignment training curves |  visual alignment charts line charts

##  4 Discussion

Multilingual models based on Transformer architectures are error-prone since
they are trained on large collections of texts. Due to this training process,
the extraction of semantics from language data often captures implicit biases
hidden in languages themselves or contextual biases triggered by adaptation to
various niches Zhong et al. (2019).

Besides the obvious questions of interpretability (e.g., which features are
most important for the prediction, what are best-performing models for
creating ensembles?) and explainability (e.g., what linguistic information is
actually encoded in the model and why do random encoders perform better for
summarization?), another important question is security (e.g., are these
models stable enough / do they obtain the same result in any circumstance?).
Through adversarial training (Wallace, 2019) it is possible to remove some
procedural biases from model-agnostic explainability libraries like LIME or
SHAP, as well as well-known attack vectors (e.g., trigger words that might
lead to a different language model output) Slack et al. (2020). It can be
argued that probing tasks are really only good for simple tasks like NER or
POS and for verifying if certain structures were encoded in a LM, but not for
more complex tasks like summarization or word-level polysemy disambiguation
Yun et al. (2021). Dictionary activation is a method that uses visualization
heavily to peek into the activation of words and their senses. Such a method
helps uncover both the various layers on which certain word senses are
actually learned, and the contexts that might trigger these new senses.

There are several available options for understanding the inner workings, as
well as the results produced by Transformer networks. Each of them have their
advantages or shortcomings, briefly discusses in the following.

Model-agnostic tools like the XAI libraries or the hyperparameter optimization
and benchmarking tools can be used with a variety of networks. Due to this,
model-agnosticism the visualization skills learned while debugging a certain
network (e.g., a Convolutional Neural Network) will be easily transferred to
debugging and optimizing other networks (e.g., Recurrent Neural Networks). By
building a transferable set of skills, users might be more reluctant to try
model-specific approaches, like those from the second category discussed in
this paper. Some of these model-agnostic tools might be more susceptible to
various adversarial attacks Slack et al. (2019), whereas some other tools
might not provide us with sufficiently advanced visualizations to match our
needs. If the users are already comfortable with some of these options, then
they might well be their Swiss-Army knife for any scenario, whereas if they
will need specific visualization scenarios (e.g., visualize a specific
attention map), it is possible that they will eventually use the Transformer
focused visualizations.

While visualization of RNNs started quite early in the DL era, it can be seen
as a precursor to the Transformer visualization. Quite often, many of the
topics that were important during this ear have remained important during the
Transformer era (e.g., hidden states)

Some of the most useful tools discovered during this exploration include:
visualization of attention maps (e.g., Clark et al. (2019)) and embeddings Vig
(2019a), hidden states visualization (e.g., Strobelt et al. (2018)), parallel
coordinates plots Clark et al. (2019), and the inclusion of corpus views from
ExBERT Hoover et al. (2020).

Current generation of pre-trained LMs based on Transformers Wolf et al. (2019)
was shown to be relatively good at picking up syntactic cues like noun
modifiers, possessive pronouns, prepositions or co-referents Clark et al.
(2019) and semantic cues like entities and relations Han et al. (2019), but
has not performed well at capturing different perspectives Chen et al. (2019),
global context Wadden et al. (2019) or relation extraction Gao et al. (2019).
This may be because biases can be already included in the embeddings and later
propagated to the downstream tasks Gonen and Goldberg (2019).

The MIT Computation Psycholinguistics Laboratory created two useful tools for
exploring LMs: the LM Zoo and the SyntaxGym Gauthier et al. (2020). The first
allows users to install some classic LMs, whereas the SyntaxGym allows users
to run pyscholinguistics tests and generate useful visualizations based on
their results.

The two large classes of Transformer visualizations we examined (focused on
explaining Transformer topics or holistic) are proof that the field is
extremely dynamic. While many of the articles focused on explaining
Transformer topics like attention or information probing tend to use classic
statistical chart types (e.g., bar charts, line charts, PCA, or Pearson
correlation charts), we do not consider this a bad thing as we are still in
the exploration phase of this technology. Some of these articles also showcase
new charts like attention graphs or attention maps.

The second class of visualizations includes tools like BertViz Vig (2019a),
AttViz Skrlj et al. (2020), VisBERT van Aken et al. (2020) or ExBERT Hoover et
al. (2020), that aim to visualize the entire lifecycle of a neural network
from corpora and inputs to the model outputs mainly through following the
information flow through the various components. They also offer detailed
statistics for neurons or network layers. Since most of the models included in
this category are rather new, it is expected that this class will expand in
the next years.

One important thing to note about visualization methods is that they can
easily be imported into other domains. The averaged attention heatmaps used by
Vig in his causal mediation analysis for NLP Vig et al. (2020a), for example,
were later reused for protein analysis in biology Vig et al. (2020b).
Similarly, attention maps Vig (2019a) developed for BERT models are now used
in a wide variety of disciplines, from vision and speech to biology or
genetics.

The end goal of future visualization frameworks should be to visualize the
entire lifecycle of the Transformer models, from inputs and data sources
(e.g., training corpora), to embeddings or attention maps, and finally
outputs. In the end, errors observed when creating such models can come from a
variety of sources: from the text corpora, from some random network layer or
even from some external Knowledge Graph that might feed some data into the
model. Tracking such errors would be costly without visualizations.

##  5 Conclusion

While the current visualizations aspire to be model-agnostic, we think the
directions opened by the various Transformer or RNN visualizations are worthy
of expanding upon. In fact, since this is a ubiquitous architecture that has
also branched from NLP into areas like semantic video processing, natural
language understanding (e.g., speech, translation) and generation (e.g., text
generation, music generation), the next generation XAI libraries will probably
be built upon it.

Going beyond current visualizations that are model-agnostic, future frameworks
will have to provide visualization components that focus on the important
Transformer components like corpora, embeddings, attention heads or additional
neural network layers that might be problem-specific. By focusing on the
common components from larger architectures, it should be possible to enhance
reusability. Other important features that should be included in future
frameworks are the ability to summarize the model’s state (e.g., through
averaged attention heatmaps or similar visualization mechanisms) at various
levels (e.g., neurons, layers, inputs and outputs), as well as the possibility
to compare multiple settings for one or multiple models. It is important to
note that most of the current visualizations seem to be designed to showcase
methods and properties that were already known to belong to neural networks.
It would be interesting to design visualizations that allow us to explore
neural networks, that help us discover new properties. Such interactive
exploration tools would significantly expand the role of visualization from
communication to knowledge discovery.

One interesting direction is the automated development of model specific
visualizations, as more complex neural networks might also include many
specific components that cannot always be included into more general model
agnostic frameworks.

## References

  * Ponte and Croft [1998] Jay M. Ponte and W. Bruce Croft.  A language modeling approach to information retrieval.  In W. Bruce Croft, Alistair Moffat, C. J. van Rijsbergen, Ross Wilkinson, and Justin Zobel, editors, _SIGIR ’98: Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, August 24-28 1998, Melbourne, Australia_ , pages 275–281. ACM, 1998.  ISBN 1-58113-015-5.  doi:10.1145/290941.291008.  URL https://doi.org/10.1145/290941.291008.
  * Brown et al. [2020] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_ , 2020.  URL https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb496741-8bfb8ac142f64a-Abstract.html.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , pages 5998–6008, 2017.  URL https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html.
  * Raghu and Schmidt [2020] Maithra Raghu and Eric Schmidt.  A survey of deep learning for scientific discovery.  _CoRR_ , abs/2003.11755, 2020.  URL https://arxiv.org/abs/2003.11755.
  * Devlin et al. [2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: pre-training of deep bidirectional transformers for language understanding.  In Jill Burstein, Christy Doran, and Thamar Solorio, editors, _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , pages 4171–4186. Association for Computational Linguistics, 2019\.  doi:10.18653/v1/n19-1423.  URL https://doi.org/10.18653/v1/n19-1423.
  * Liu et al. [2019] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized BERT pretraining approach.  _CoRR_ , abs/1907.11692, 2019.  URL http://arxiv.org/abs/1907.11692.
  * Lan et al. [2020] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut.  ALBERT: A lite BERT for self-supervised learning of language representations.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net, 2020.  URL https://openreview.net/forum?id=H1eA7AEtvS.
  * Yang et al. [2019] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le.  Xlnet: Generalized autoregressive pretraining for language understanding.  In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett, editors, _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , pages 5754–5764, 2019.  URL https://proceedings.neurips.cc/paper/2019/hash/dc6a7e655d7e5840e6-6733e9ee67cc69-Abstract.html.
  * Sanh et al. [2019] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf.  Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter.  _CoRR_ , abs/1910.01108, 2019.  URL http://arxiv.org/abs/1910.01108.
  * Kitaev et al. [2020] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya.  Reformer: The efficient transformer.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net, 2020.  URL https://openreview.net/forum?id=rkgNKkHtvB.
  * Wolf et al. [2019] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew.  Huggingface transformers: State-of-the-art natural language processing.  _CoRR_ , abs/1910.03771, 2019.  URL http://arxiv.org/abs/1910.03771.
  * Braşoveanu and Andonie [2020] Adrian MP Braşoveanu and Răzvan Andonie.  Visualizing transformers for nlp: a brief survey.  In _2020 24th International Conference Information Visualisation (IV)_ , pages 270–279. IEEE, 2020.  ISBN 978-1-7281-9134-8.  doi:10.1109/IV51561.2020.
  * Wilkinson [2005] Leland Wilkinson.  _The Grammar of Graphics, Second Edition_.  Statistics and computing. Springer, 2005.  ISBN 978-0-387-24544-7.
  * Bostock et al. [2011] Michael Bostock, Vadim Ogievetsky, and Jeffrey Heer.  D3 data-driven documents.  _IEEE Trans. Vis. Comput. Graph._ , 17(12):2301–2309, 2011.  doi:10.1109/TVCG.2011.185.  URL https://doi.org/10.1109/TVCG.2011.185.
  * Satyanarayan et al. [2017] Arvind Satyanarayan, Dominik Moritz, Kanit Wongsuphasawat, and Jeffrey Heer.  Vega-lite: A grammar of interactive graphics.  _IEEE Trans. Vis. Comput. Graph._ , 23(1):341–350, 2017.  doi:10.1109/TVCG.2016.2599030.  URL https://doi.org/10.1109/TVCG.2016.2599030.
  * Park et al. [2018] Deokgun Park, Seungyeon Kim, Jurim Lee, Jaegul Choo, Nicholas Diakopoulos, and Niklas Elmqvist.  Conceptvector: Text visual analytics via interactive lexicon building using word embedding.  _IEEE Trans. Vis. Comput. Graph._ , 24(1):361–370, 2018.  doi:10.1109/TVCG.2017.2744478.  URL https://doi.org/10.1109/TVCG.2017.2744478.
  * Hoover et al. [2020] Benjamin Hoover, Hendrik Strobelt, and Sebastian Gehrmann.  ex bert: A visual analysis tool to explore learned representations in transformer models.  In Asli Celikyilmaz and Tsung-Hsien Wen, editors, _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, ACL 2020, Online, July 5-10, 2020_ , pages 187–196. Association for Computational Linguistics, 2020.  doi:10.18653/v1/2020.acl-demos.22.  URL https://doi.org/10.18653/v1/2020.acl-demos.22.
  * Vig [2019a] Jesse Vig.  A multiscale visualization of attention in the transformer model.  In Marta R. Costa-jussà and Enrique Alfonseca, editors, _Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28 - August 2, 2019, Volume 3: System Demonstrations_ , pages 37–42. Association for Computational Linguistics, 2019a.  ISBN 978-1-950737-49-9.  doi:10.18653/v1/p19-3007.  URL https://doi.org/10.18653/v1/p19-3007.
  * Clark et al. [2019] Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning.  What does BERT look at? an analysis of bert’s attention.  _CoRR_ , abs/1906.04341, 2019.  URL http://arxiv.org/abs/1906.04341.
  * Florea and Andonie [2019] Adrian Florea and Razvan Andonie.  Weighted random search for hyperparameter optimization.  _Int. J. Comput. Commun. Control_ , 14(2):154–169, 2019.  doi:10.15837/ijccc.2019.2.3514.  URL https://doi.org/10.15837/ijccc.2019.2.3514.
  * Heinrich and Weiskopf [2015] Julian Heinrich and Daniel Weiskopf.  Parallel coordinates for multidimensional data visualization: Basic concepts.  _Comput. Sci. Eng._ , 17(3):70–76, 2015.  doi:10.1109/MCSE.2015.55.  URL https://doi.org/10.1109/MCSE.2015.55.
  * Moritz et al. [2018] Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I. Jordan, and Ion Stoica.  Ray: A distributed framework for emerging AI applications.  In Andrea C. Arpaci-Dusseau and Geoff Voelker, editors, _13th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2018, Carlsbad, CA, USA, October 8-10, 2018_ , pages 561–577. USENIX Association, 2018.  URL https://www.usenix.org/conference/osdi18/presentation/nishihara.
  * Liaw et al. [2018] Richard Liaw, Eric Liang, Robert Nishihara, Philipp Moritz, Joseph E. Gonzalez, and Ion Stoica.  Tune: A research platform for distributed model selection and training.  _CoRR_ , abs/1807.05118, 2018.  URL http://arxiv.org/abs/1807.05118.
  * Strubell et al. [2019] Emma Strubell, Ananya Ganesh, and Andrew McCallum.  Energy and policy considerations for deep learning in NLP.  In Anna Korhonen, David R. Traum, and Lluis Marquez, editors, _Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers_ , pages 3645–3650. Association for Computational Linguistics, 2019\.  doi:10.18653/v1/p19-1355.  URL https://doi.org/10.18653/v1/p19-1355.
  * Shwartz and Choi [2020] Vered Shwartz and Yejin Choi.  Do neural language models overcome reporting bias?  In Donia Scott, Núria Bel, and Chengqing Zong, editors, _Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020_ , pages 6863–6870. International Committee on Computational Linguistics, 2020.  doi:10.18653/v1/2020.coling-main.605.  URL https://doi.org/10.18653/v1/2020.coling-main.605.
  * Vellido [2020] Alfredo Vellido.  The importance of interpretability and visualization in machine learning for applications in medicine and health care.  _Neural Comput. Appl._ , 32(24):18069–18083, 2020\.  doi:10.1007/s00521-019-04051-w.  URL https://doi.org/10.1007/s00521-019-04051-w.
  * Heer [2019] Jeffrey Heer.  Agency plus automation: Designing artificial intelligence into interactive systems.  _Proc. Natl. Acad. Sci. USA_ , 116(6):1844–1850, 2019.  doi:10.1073/pnas.1807184115.  URL https://doi.org/10.1073/pnas.1807184115.
  * Qin et al. [2018] Zhuwei Qin, Fuxun Yu, Chenchen Liu, and Xiang Chen.  How convolutional neural networks see the world - A survey of convolutional neural network visualization methods.  _Math. Found. Comput._ , 1(2):149–180, 2018.  doi:10.3934/mfc.2018008.  URL https://doi.org/10.3934/mfc.2018008.
  * Zhang and Zhu [2018] Quanshi Zhang and Song-Chun Zhu.  Visual interpretability for deep learning: a survey.  _Frontiers Inf. Technol. Electron. Eng._ , 19(1):27–39, 2018.  doi:10.1631/FITEE.1700808.  URL https://doi.org/10.1631/FITEE.1700808.
  * Nguyen et al. [2019] Anh Nguyen, Jason Yosinski, and Jeff Clune.  Understanding neural networks via feature visualization: A survey.  In Samek et al. [2019], pages 55–76.  ISBN 978-3-030-28953-9.  doi:10.1007/978-3-030-28954-6_4.  URL https://doi.org/10.1007/978-3-030-28954-6_4.
  * Hohman et al. [2019] Fred Hohman, Minsuk Kahng, Robert Pienta, and Duen Horng Chau.  Visual analytics in deep learning: An interrogative survey for the next frontiers.  _IEEE Trans. Vis. Comput. Graph._ , 25(8):2674–2693, 2019.  doi:10.1109/TVCG.2018.2843369.  URL https://doi.org/10.1109/TVCG.2018.2843369.
  * Samek et al. [2019] Wojciech Samek, Grégoire Montavon, Andrea Vedaldi, Lars Kai Hansen, and Klaus-Robert Müller, editors.  _Explainable AI: Interpreting, Explaining and Visualizing Deep Learning_ , volume 11700 of _Lecture Notes in Computer Science_.  Springer, 2019.  ISBN 978-3-030-28953-9.  doi:10.1007/978-3-030-28954-6.  URL https://doi.org/10.1007/978-3-030-28954-6.
  * Guyon and Elisseeff [2003] Isabelle Guyon and André Elisseeff.  An introduction to variable and feature selection.  _J. Mach. Learn. Res._ , 3:1157–1182, 2003.  URL http://jmlr.org/papers/v3/guyon03a.html.
  * Ribeiro et al. [2016] Marco Túlio Ribeiro, Sameer Singh, and Carlos Guestrin.  "why should I trust you?": Explaining the predictions of any classifier.  In Balaji Krishnapuram, Mohak Shah, Alexander J. Smola, Charu C. Aggarwal, Dou Shen, and Rajeev Rastogi, editors, _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016_ , pages 1135–1144. ACM, 2016\.  ISBN 978-1-4503-4232-2.  doi:10.1145/2939672.2939778.  URL https://doi.org/10.1145/2939672.2939778.
  * Lundberg and Lee [2017] Scott M. Lundberg and Su-In Lee.  A unified approach to interpreting model predictions.  In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , pages 4765–4774, 2017.  URL https://proceedings.neurips.cc/paper/2017/hash/8a20a8621978632d76c43dfd28b67767-Abstract.html.
  * Fan et al. [2019] Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli.  ELI5: long form question answering.  In Anna Korhonen, David R. Traum, and Lluis Marquez, editors, _Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers_ , pages 3558–3567. Association for Computational Linguistics, 2019\.  doi:10.18653/v1/p19-1346.  URL https://doi.org/10.18653/v1/p19-1346.
  * Slack et al. [2020] Dylan Slack, Sophie Hilgard, Emily Jia, Sameer Singh, and Himabindu Lakkaraju.  Fooling LIME and SHAP: adversarial attacks on post hoc explanation methods.  In Annette N. Markham, Julia Powles, Toby Walsh, and Anne L. Washington, editors, _AIES ’20: AAAI/ACM Conference on AI, Ethics, and Society, New York, NY, USA, February 7-8, 2020_ , pages 180–186. ACM, 2020\.  ISBN 978-1-4503-7110-0.  doi:10.1145/3375627.3375830.  URL https://doi.org/10.1145/3375627.3375830.
  * Lundberg et al. [2018] Scott M Lundberg, Bala Nair, Monica S Vavilala, Mayumi Horibe, Michael J Eisses, Trevor Adams, David E Liston, Daniel King-Wai Low, Shu-Fang Newman, Jerry Kim, et al.  Explainable machine-learning predictions for the prevention of hypoxaemia during surgery.  _Nature biomedical engineering_ , 2(10):749–760, 2018.
  * Wallace et al. [2019a] Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh.  Universal adversarial triggers for attacking and analyzing NLP.  In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_ , pages 2153–2162. Association for Computational Linguistics, 2019a.  doi:10.18653/v1/D19-1221.  URL https://doi.org/10.18653/v1/D19-1221.
  * Carlini et al. [2020] Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom B. Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel.  Extracting training data from large language models.  _CoRR_ , abs/2012.07805, 2020.  URL https://arxiv.org/abs/2012.07805.
  * Sun et al. [2020] Lichao Sun, Kazuma Hashimoto, Wenpeng Yin, Akari Asai, Jia Li, Philip S. Yu, and Caiming Xiong.  Adv-bert: BERT is not robust on misspellings! generating nature adversarial samples on BERT.  _CoRR_ , abs/ 2003.04985, 2020.  URL https://arxiv.org/abs/2003.04985.
  * Ebrahimi et al. [2018] Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou.  Hotflip: White-box adversarial examples for text classification.  In Iryna Gurevych and Yusuke Miyao, editors, _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 2: Short Papers_ , pages 31–36. Association for Computational Linguistics, 2018.  ISBN 978-1-948087-34-6.  doi:10.18653/v1/P18-2006.  URL https://www.aclweb.org/anthology/P18-2006/.
  * Jin et al. [2020] Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits.  Is BERT really robust? A strong baseline for natural language attack on text classification and entailment.  In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020_ , pages 8018–8025. AAAI Press, 2020.  URL https://aaai.org/ojs/index.php/AAAI/article/view/6311.
  * Morris et al. [2020] John X. Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi.  Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in NLP.  In Qun Liu and David Schlangen, editors, _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, EMNLP 2020 - Demos, Online, November 16-20, 2020_ , pages 119–126. Association for Computational Linguistics, 2020.  ISBN 978-1-952148-62-0.  doi:10.18653/v1/2020.emnlp-demos.16.  URL https://doi.org/10.18653/v1/2020.emnlp-demos.16.
  * Zeng et al. [2020] Guoyang Zeng, Fanchao Qi, Qianrui Zhou, Tingji Zhang, Bairu Hou, Yuan Zang, Zhiyuan Liu, and Maosong Sun.  Openattack: An open-source textual adversarial attack toolkit.  _CoRR_ , abs/2009.09191, 2020.  URL https://arxiv.org/abs/2009.09191.
  * Karimi et al. [2020] Akbar Karimi, Leonardo Rossi, and Andrea Prati.  Adversarial training for aspect-based sentiment analysis with BERT.  In _25th International Conference on Pattern Recognition, ICPR 2020, Virtual Event / Milan, Italy, January 10-15, 2021_ , pages 8797–8803. IEEE, 2020.  ISBN 978-1-7281-8808-9.  doi:10.1109/ICPR48806.2021.9412167.  URL https://doi.org/10.1109/ICPR48806.2021.9412167.
  * Hao et al. [2020] Yaru Hao, Li Dong, Furu Wei, and Ke Xu.  Self-attention attribution: Interpreting information interactions inside transformer.  _CoRR_ , abs/2004.11207, 2020.  URL https://arxiv.org/abs/2004.11207.
  * Chen et al. [2020] Xiaoyi Chen, Ahmed Salem, Michael Backes, Shiqing Ma, and Yang Zhang.  Badnl: Backdoor attacks against NLP models.  _CoRR_ , abs/2006.01043, 2020.  URL https://arxiv.org/abs/2006.01043.
  * Wallace et al. [2019b] Eric Wallace, Jens Tuyls, Junlin Wang, Sanjay Subramanian, Matt Gardner, and Sameer Singh.  Allennlp interpret: A framework for explaining predictions of NLP models.  In Sebastian Padó and Ruihong Huang, editors, _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019 - System Demonstrations_ , pages 7–12. Association for Computational Linguistics, 2019b.  doi:10.18653/v1/D19-3002.  URL https://doi.org/10.18653/v1/D19-3002.
  * Wang et al. [2020] Junlin Wang, Jens Tuyls, Eric Wallace, and Sameer Singh.  Gradient-based analysis of NLP models is manipulable.  In Trevor Cohn, Yulan He, and Yang Liu, editors, _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, EMNLP 2020, Online Event, 16-20 November 2020_ , pages 247–258. Association for Computational Linguistics, 2020.  ISBN 978-1-952148-90-3.  doi:10.18653/v1/2020.findings-emnlp.24.  URL https://doi.org/10.18653/v1/2020.findings-emnlp.24.
  * Agarwal et al. [2020] Rishabh Agarwal, Nicholas Frosst, Xuezhou Zhang, Rich Caruana, and Geoffrey E. Hinton.  Neural additive models: Interpretable machine learning with neural nets.  _CoRR_ , abs/2004.13912, 2020.  URL https://arxiv.org/abs/2004.13912.
  * Fankhauser et al. [2014] Peter Fankhauser, Jörg Knappen, and Elke Teich.  Exploring and visualizing variation in language resources.  In Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Hrafn Loftsson, Bente Maegaard, Joseph Mariani, Asunción Moreno, Jan Odijk, and Stelios Piperidis, editors, _Proceedings of the Ninth International Conference on Language Resources and Evaluation, LREC 2014, Reykjavik, Iceland, May 26-31, 2014_ , pages 4125–4128. European Language Resources Association (ELRA), 2014.  URL http://www.lrec-conf.org/proceedings/lrec2014/summaries/185.html.
  * Kessler [2017] Jason S. Kessler.  Scattertext: a browser-based tool for visualizing how corpora differ.  In Mohit Bansal and Heng Ji, editors, _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, System Demonstrations_ , pages 85–90. Association for Computational Linguistics, 2017.  doi:10.18653/v1/P17-4015.  URL https://doi.org/10.18653/v1/P17-4015.
  * Karpathy et al. [2015] Andrej Karpathy, Justin Johnson, and Fei-Fei Li.  Visualizing and understanding recurrent networks.  _CoRR_ , abs/1506.02078, 2015.  URL http://arxiv.org/abs/1506.02078.
  * Luo et al. [2019] Hongyin Luo, Lan Jiang, Yonatan Belinkov, and Jim Glass.  Improving neural language models by segmenting, attending, and predicting the future.  In Anna Korhonen, David R. Traum, and Lluis Marquez, editors, _Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers_ , pages 1483–1493. Association for Computational Linguistics, 2019\.  doi:10.18653/v1/p19-1144.  URL https://doi.org/10.18653/v1/p19-1144.
  * Lakretz et al. [2019] Yair Lakretz, Germán Kruszewski, Theo Desbordes, Dieuwke Hupkes, Stanislas Dehaene, and Marco Baroni.  The emergence of number and syntax units in LSTM language models.  In Jill Burstein, Christy Doran, and Thamar Solorio, editors, _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , pages 11–20. Association for Computational Linguistics, 2019\.  doi:10.18653/v1/n19-1002.  URL https://doi.org/10.18653/v1/n19-1002.
  * Li et al. [2016] Jiwei Li, Xinlei Chen, Eduard H. Hovy, and Dan Jurafsky.  Visualizing and understanding neural models in NLP.  In Kevin Knight, Ani Nenkova, and Owen Rambow, editors, _NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016_ , pages 681–691. The Association for Computational Linguistics, 2016.  doi:10.18653/v1/n16-1082.  URL https://doi.org/10.18653/v1/n16-1082.
  * Sawatzky et al. [2019] Lindsey Sawatzky, Steven Bergner, and Fred Popowich.  Visualizing RNN states with predictive semantic encodings.  In _30th IEEE Visualization Conference, IEEE VIS 2019 - Short Papers, Vancouver, BC, Canada, October 20-25, 2019_ , pages 156–160. IEEE, 2019.  ISBN 978-1-7281-4941-7.  doi:10.1109/VISUAL.2019.8933744.  URL https://doi.org/10.1109/VISUAL.2019.8933744.
  * Strobelt et al. [2018] Hendrik Strobelt, Sebastian Gehrmann, Hanspeter Pfister, and Alexander M. Rush.  Lstmvis: A tool for visual analysis of hidden state dynamics in recurrent neural networks.  _IEEE Trans. Vis. Comput. Graph._ , 24(1):667–676, 2018.  doi:10.1109/TVCG.2017.2744158.  URL https://doi.org/10.1109/TVCG.2017.2744158.
  * Ming et al. [2017] Yao Ming, Shaozu Cao, Ruixiang Zhang, Zhen Li, Yuanzhe Chen, Yangqiu Song, and Huamin Qu.  Understanding hidden memories of recurrent neural networks.  In Brian D. Fisher, Shixia Liu, and Tobias Schreck, editors, _12th IEEE Conference on Visual Analytics Science and Technology, IEEE VAST 2017, Phoenix, AZ, USA, October 3-6, 2017_ , pages 13–24. IEEE Computer Society, 2017.  doi:10.1109/VAST.2017.8585721.  URL https://doi.org/10.1109/VAST.2017.8585721.
  * Kahng et al. [2018] Minsuk Kahng, Pierre Y. Andrews, Aditya Kalro, and Duen Horng (Polo) Chau.  Activis: Visual exploration of industry-scale deep neural network models.  _IEEE Trans. Vis. Comput. Graph._ , 24(1):88–97, 2018.  doi:10.1109/TVCG.2017.2744718.  URL https://doi.org/10.1109/TVCG.2017.2744718.
  * Zhang et al. [2020] Yufeng Zhang, Xueli Yu, Zeyu Cui, Shu Wu, Zhongzhen Wen, and Liang Wang.  Every document owns its structure: Inductive text classification via graph neural networks.  In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault, editors, _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , pages 334–339. Association for Computational Linguistics, 2020.  doi:10.18653/v1/2020.acl-main.31.  URL https://doi.org/10.18653/v1/2020.acl-main.31.
  * Wu et al. [2020] Man Wu, Shirui Pan, Chuan Zhou, Xiaojun Chang, and Xingquan Zhu.  Unsupervised domain adaptive graph convolutional networks.  In Yennun Huang, Irwin King, Tie-Yan Liu, and Maarten van Steen, editors, _WWW ’20: The Web Conference 2020, Taipei, Taiwan, April 20-24, 2020_ , pages 1457–1467. ACM / IW3C2, 2020.  doi:10.1145/3366423.3380219.  URL https://doi.org/10.1145/3366423.3380219.
  * Wang and Leskovec [2020] Hongwei Wang and Jure Leskovec.  Unifying graph convolutional neural networks and label propagation.  _CoRR_ , abs/2002.06755, 2020.  URL https://arxiv.org/abs/2002.06755.
  * Fey and Lenssen [2019] Matthias Fey and Jan Eric Lenssen.  Fast graph representation learning with pytorch geometric.  _CoRR_ , abs/1903.02428, 2019.  URL http://arxiv.org/abs/1903.02428.
  * Kohonen [1995] Teuvo Kohonen.  _Self-Organizing Maps_ , volume 30 of _Springer Series in Information Sciences_.  Springer, 1995.  ISBN 978-3-642-97612-4.  doi:10.1007/978-3-642-97610-0.  URL https://doi.org/10.1007/978-3-642-97610-0.
  * Mandelbrot [1977] Benoit Mandelbrot.  _Fractal Geometry of Nature_.  W. H. Freeman, 1977.
  * Jain and Wallace [2019] Sarthak Jain and Byron C. Wallace.  Attention is not explanation.  In Jill Burstein, Christy Doran, and Thamar Solorio, editors, _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , pages 3543–3556. Association for Computational Linguistics, 2019\.  doi:10.18653/v1/n19-1357.  URL https://doi.org/10.18653/v1/n19-1357.
  * Wiegreffe and Pinter [2019] Sarah Wiegreffe and Yuval Pinter.  Attention is not not explanation.  In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_ , pages 11–20. Association for Computational Linguistics, 2019.  doi:10.18653/v1/D19-1002.  URL https://doi.org/10.18653/v1/D19-1002.
  * Abnar and Zuidema [2020] Samira Abnar and Willem H. Zuidema.  Quantifying attention flow in transformers.  In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault, editors, _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , pages 4190–4197. Association for Computational Linguistics, 2020.  ISBN 978-1-952148-25-5.  URL https://www.aclweb.org/anthology/2020.acl-main.385/.
  * DeRose et al. [2021] Joseph F. DeRose, Jiayao Wang, and Matthew Berger.  Attention flows: Analyzing and comparing attention mechanisms in language models.  _IEEE Trans. Vis. Comput. Graph._ , 27(2):1160–1170, 2021.  doi:10.1109/TVCG.2020.3028976.  URL https://doi.org/10.1109/TVCG.2020.3028976.
  * Voita et al. [2019a] Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov.  Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned.  In Anna Korhonen, David R. Traum, and Lluis Marquez, editors, _Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers_ , pages 5797–5808. Association for Computational Linguistics, 2019a.  doi:10.18653/v1/p19-1580.  URL https://doi.org/10.18653/v1/p19-1580.
  * Song et al. [2020] Youwei Song, Jiahai Wang, Zhiwei Liang, Zhiyue Liu, and Tao Jiang.  Utilizing BERT intermediate layers for aspect based sentiment analysis and natural language inference.  _CoRR_ , abs/2002.04815, 2020.  URL https://arxiv.org/abs/2002.04815.
  * Vig et al. [2020a] Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart M. Shieber.  Causal mediation analysis for interpreting neural NLP: the case of gender bias.  _CoRR_ , abs/2004.12265, 2020a.  URL https://arxiv.org/abs/2004.12265.
  * Voita et al. [2019b] Elena Voita, Rico Sennrich, and Ivan Titov.  The bottom-up evolution of representations in the transformer: A study with machine translation and language modeling objectives.  In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_ , pages 4395–4405. Association for Computational Linguistics, 2019b.  doi:10.18653/v1/D19-1448.  URL https://doi.org/10.18653/v1/D19-1448.
  * Tenney et al. [2019] Ian Tenney, Dipanjan Das, and Ellie Pavlick.  BERT rediscovers the classical NLP pipeline.  In Anna Korhonen, David R. Traum, and Lluis Marquez, editors, _Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers_ , pages 4593–4601. Association for Computational Linguistics, 2019\.  doi:10.18653/v1/p19-1452.  URL https://doi.org/10.18653/v1/p19-1452.
  * Dufter and Schütze [2020] Philipp Dufter and Hinrich Schütze.  Identifying necessary elements for bert’s multilinguality.  _CoRR_ , abs/2005.00396, 2020.  URL https://arxiv.org/abs/2005.00396.
  * Eger et al. [2020] Steffen Eger, Johannes Daxenberger, and Iryna Gurevych.  How to probe sentence embeddings in low-resource languages: On structural design choices for probing task evaluation.  In Raquel Fernández and Tal Linzen, editors, _Proceedings of the 24th Conference on Computational Natural Language Learning, CoNLL 2020, Online, November 19-20, 2020_ , pages 108–118. Association for Computational Linguistics, 2020.  doi:10.18653/v1/2020.conll-1.8.  URL https://doi.org/10.18653/v1/2020.conll-1.8.
  * Voita and Titov [2020a] Elena Voita and Ivan Titov.  Information-theoretic probing with minimum description length.  _CoRR_ , abs/2003.12298, 2020a.  URL https://arxiv.org/abs/2003.12298.
  * Gauthier et al. [2020] Jon Gauthier, Jennifer Hu, Ethan Wilcox, Peng Qian, and Roger Levy.  Syntaxgym: An online platform for targeted evaluation of language models.  In Asli Çelikyilmaz and Tsung-Hsien Wen, editors, _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, ACL 2020, Online, July 5-10, 2020_ , pages 70–76. Association for Computational Linguistics, 2020.  ISBN 978-1-952148-04-0.  doi:10.18653/v1/2020.acl-demos.10.  URL https://doi.org/10.18653/v1/2020.acl-demos.10.
  * Reif et al. [2019] Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B. Viégas, Andy Coenen, Adam Pearce, and Been Kim.  Visualizing and measuring the geometry of BERT.  In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett, editors, _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , pages 8592–8600, 2019.  URL https://proceedings.neurips.cc/paper/2019/hash/159c1ffe5b61b41b3c4d8f4c2150f6c4-Abstract.html.
  * Su et al. [2020] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai.  VL-BERT: pre-training of generic visual-linguistic representations.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net, 2020.  URL https://openreview.net/forum?id=SygXPaEYvH.
  * Conneau et al. [2018] Alexis Conneau, Germán Kruszewski, Guillaume Lample, Loïc Barrault, and Marco Baroni.  What you can cram into a single \&!#* vector: Probing sentence embeddings for linguistic properties.  In Iryna Gurevych and Yusuke Miyao, editors, _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers_ , pages 2126–2136. Association for Computational Linguistics, 2018.  ISBN 978-1-948087-32-2.  doi:10.18653/v1/P18-1198.  URL https://www.aclweb.org/anthology/P18-1198/.
  * Hewitt and Manning [2019] John Hewitt and Christopher D. Manning.  A structural probe for finding syntax in word representations.  In Jill Burstein, Christy Doran, and Thamar Solorio, editors, _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , pages 4129–4138. Association for Computational Linguistics, 2019\.  doi:10.18653/v1/n19-1419.  URL https://doi.org/10.18653/v1/n19-1419.
  * Maudslay et al. [2020] Rowan Hall Maudslay, Josef Valvoda, Tiago Pimentel, Adina Williams, and Ryan Cotterell.  A tale of a probe and a parser.  _CoRR_ , abs/2005.01641, 2020.  URL https://arxiv.org/abs/2005.01641.
  * Ettinger et al. [2016] Allyson Ettinger, Ahmed Elgohary, and Philip Resnik.  Probing for semantic evidence of composition by means of simple classification tasks.  In _Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP, RepEval@ACL 2016, Berlin, Germany, August 2016_ , pages 134–139. Association for Computational Linguistics, 2016.  doi:10.18653/v1/W16-2524.  URL https://doi.org/10.18653/v1/W16-2524.
  * Voita and Titov [2020b] Elena Voita and Ivan Titov.  Information-theoretic probing with minimum description length.  In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020_ , pages 183–196. Association for Computational Linguistics, 2020b.  doi:10.18653/v1/2020.emnlp-main.14.  URL https://doi.org/10.18653/v1/2020.emnlp-main.14.
  * Pilault et al. [2020] Jonathan Pilault, Jaehong Park, and Christopher J. Pal.  On the impressive performance of randomly weighted encoders in summarization tasks.  _CoRR_ , abs/2002.09084, 2020.  URL https://arxiv.org/abs/2002.09084.
  * Vig [2019b] Jesse Vig.  Visualizing attention in transformer-based language representation models.  _CoRR_ , abs/1904.02679, 2019b.  URL http://arxiv.org/abs/1904.02679.
  * van Aken et al. [2020] Betty van Aken, Benjamin Winter, Alexander Löser, and Felix A. Gers.  Visbert: Hidden-state visualizations for transformers.  In Amal El Fallah Seghrouchni, Gita Sukthankar, Tie-Yan Liu, and Maarten van Steen, editors, _Companion of The 2020 Web Conference 2020, Taipei, Taiwan, April 20-24, 2020_ , pages 207–211. ACM / IW3C2, 2020.  ISBN 978-1-4503-7024-0.  doi:10.1145/3366424.3383542.  URL https://doi.org/10.1145/3366424.3383542.
  * Skrlj et al. [2020] Blaz Skrlj, Nika Erzen, Shane Sheehan, Saturnino Luz, Marko Robnik-Sikonja, and Senja Pollak.  Attviz: Online exploration of self-attention for transparent neural language modeling.  _CoRR_ , abs/2005.05716, 2020.  URL https://arxiv.org/abs/2005.05716.
  * Kobayashi et al. [2020] Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui.  Attention module is not only a weight: Analyzing transformers with vector norms.  _CoRR_ , abs/2004.10102, 2020.  URL https://arxiv.org/abs/2004.10102.
  * Yun et al. [2021] Zeyu Yun, Yubei Chen, Bruno A. Olshausen, and Yann LeCun.  Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factors.  _CoRR_ , abs/2103.15949, 2021.  URL https://arxiv.org/abs/2103.15949.
  * Brasoveanu et al. [2018] Adrian Brasoveanu, Giuseppe Rizzo, Philipp Kuntschik, Albert Weichselbraun, and Lyndon J. B. Nixon.  Framing named entity linking error types.  In Nicoletta Calzolari, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Kôiti Hasida, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asunción Moreno, Jan Odijk, Stelios Piperidis, and Takenobu Tokunaga, editors, _Proceedings of the Eleventh International Conference on Language Resources and Evaluation, LREC 2018, Miyazaki, Japan, May 7-12, 2018_. European Language Resources Association (ELRA), 2018.  URL http://www.lrec-conf.org/proceedings/lrec2018/summaries/612.html.
  * van der Heijden et al. [2020] Niels van der Heijden, Samira Abnar, and Ekaterina Shutova.  A comparison of architectures and pretraining methods for contextualized multilingual word embeddings.  In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020_ , pages 9090–9097. AAAI Press, 2020.  URL https://aaai.org/ojs/index.php/AAAI/article/view/6443.
  * Gan et al. [2020] Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu.  Large-scale adversarial training for vision-and-language representation learning.  In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_ , 2020.  URL https://proceedings.neurips.cc/paper/2020/hash/49562478de4c54fafd4ec46fdb297de5-Abstract.html.
  * Cao et al. [2020] Jize Cao, Zhe Gan, Yu Cheng, Licheng Yu, Yen-Chun Chen, and Jingjing Liu.  Behind the scene: Revealing the secrets of pre-trained vision-and-language models.  In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, _Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VI_ , volume 12351 of _Lecture Notes in Computer Science_ , pages 565–580. Springer, 2020\.  ISBN 978-3-030-58538-9.  doi:10.1007/978-3-030-58539-6_34.  URL https://doi.org/10.1007/978-3-030-58539-6_34.
  * Han et al. [2021] Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang.  Transformer in transformer.  _CoRR_ , abs/2103. 00112, 2021.  URL https://arxiv.org/abs/2103.00112.
  * Li et al. [2020] Yikuan Li, Hanyin Wang, and Yuan Luo.  A comparison of pre-trained vision-and-language models for multimodal representation learning across medical images and reports.  In Taesung Park, Young-Rae Cho, Xiaohua Hu, Illhoi Yoo, Hyun Goo Woo, Jianxin Wang, Julio C. Facelli, Seungyoon Nam, and Mingon Kang, editors, _IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2020, Virtual Event, South Korea, December 16-19, 2020_ , pages 1999–2004. IEEE, 2020.  ISBN 978-1-7281-6215-7.  doi:10.1109/BIBM49941.2020.9313289.  URL https://doi.org/10.1109/BIBM49941.2020.9313289.
  * Kim et al. [2021] Wonjae Kim, Bokyung Son, and Ildoo Kim.  Vilt: Vision - and - language transformer without convolution or region supervision.  In Marina Meila and Tong Zhang, editors, _Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 5583–5594. PMLR, 2021.  URL http://proceedings.mlr.press/v139/kim21k.html.
  * Zhong et al. [2019] Ming Zhong, Danqing Wang, Pengfei Liu, Xipeng Qiu, and Xuanjing Huang.  A closer look at data bias in neural extractive summarization models.  _CoRR_ , abs/1909.13705, 2019.  URL http://arxiv.org/abs/1909.13705.
  * Slack et al. [2019] Dylan Slack, Sophie Hilgard, Emily Jia, Sameer Singh, and Himabindu Lakkaraju.  How can we fool LIME and shap? adversarial attacks on post hoc explanation methods.  _CoRR_ , abs/1911.02508, 2019.  URL http://arxiv.org/abs/1911.02508.
  * Han et al. [2019] Xu Han, Tianyu Gao, Yuan Yao, Deming Ye, Zhiyuan Liu, and Maosong Sun.  Opennre: An open and extensible toolkit for neural relation extraction.  In Sebastian Padó and Ruihong Huang, editors, _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019 - System Demonstrations_ , pages 169–174. Association for Computational Linguistics, 2019\.  doi:10.18653/v1/D19-3029.  URL https://doi.org/10.18653/v1/D19-3029.
  * Chen et al. [2019] Sihao Chen, Daniel Khashabi, Wenpeng Yin, Chris Callison-Burch, and Dan Roth.  Seeing things from a different angle: Discovering diverse perspectives about claims.  In Jill Burstein, Christy Doran, and Thamar Solorio, editors, _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , pages 542–557. Association for Computational Linguistics, 2019\.  doi:10.18653/v1/n19-1053.  URL https://doi.org/10.18653/v1/n19-1053.
  * Wadden et al. [2019] David Wadden, Ulme Wennberg, Yi Luan, and Hannaneh Hajishirzi.  Entity, relation, and event extraction with contextualized span representations.  In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_ , pages 5783–5788. Association for Computational Linguistics, 2019.  doi:10.18653/v1/D19-1585.  URL https://doi.org/10.18653/v1/D19-1585.
  * Gao et al. [2019] Tianyu Gao, Xu Han, Hao Zhu, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou.  Fewrel 2.0: Towards more challenging few-shot relation classification.  In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_ , pages 6249–6254. Association for Computational Linguistics, 2019.  doi:10.18653/v1/D19-1649.  URL https://doi.org/10.18653/v1/D19-1649.
  * Gonen and Goldberg [2019] Hila Gonen and Yoav Goldberg.  Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them.  In Amittai Axelrod, Diyi Yang, Rossana Cunha, Samira Shaikh, and Zeerak Waseem, editors, _Proceedings of the 2019 Workshop on Widening NLP@ACL 2019, Florence, Italy, July 28, 2019_ , pages 60–63. Association for Computational Linguistics, 2019.  ISBN 978-1-950737-42-0.  URL https://www.aclweb.org/anthology/W19-3621/.
  * Vig et al. [2020b] Jesse Vig, Ali Madani, Lav R. Varshney, Caiming Xiong, Richard Socher, and Nazneen Fatema Rajani.  Bertology meets biology: Interpreting attention in protein language models.  _CoRR_ , abs/2006.15222, 2020b.  URL https://arxiv.org/abs/2006.15222.