0 Research Application Authors are with Harvard University. Viégas and
Wattenberg are also with Google, but this work was done at Harvard. Emails:
{catherineyeh, yidachen, aoyuwu, fernanda<EMAIL_ADDRESS><EMAIL_ADDRESS>
# AttentionViz: A Global View of Transformer Attention

Catherine Yeh  Yida Chen  Aoyu Wu  Cynthia Chen  Fernanda Viégas  and Martin
Wattenberg

###### Abstract

Transformer models are revolutionizing machine learning, but their inner
workings remain mysterious. In this work, we present a new visualization
technique designed to help researchers understand the self-attention mechanism
in transformers that allows these models to learn rich, contextual
relationships between elements of a sequence. The main idea behind our method
is to visualize a joint embedding of the query and key vectors used by
transformer models to compute attention. Unlike previous attention
visualization techniques, our approach enables the analysis of global patterns
across multiple input sequences. We create an interactive visualization tool,
AttentionViz (demo: http://attentionviz.com), based on these joint query-key
embeddings, and use it to study attention mechanisms in both language and
vision transformers. We demonstrate the utility of our approach in improving
model understanding and offering new insights about query-key interactions
through several application scenarios and expert feedback.

###### keywords:

Transformer, Attention, NLP, Computer Vision, Visual Analytics

AttentionViz, our interactive visualization tool, allows users to explore
transformer self-attention at scale by creating a joint embedding space for
queries and keys. (a) In language transformers, these visualizations reveal
striking visual traces that can be linked to attention patterns. Each point in
the scatterplot represents the query or key version of a word, as denoted by
point color. Users can explore individual attention heads (left) or zoom out
for a “global” view of attention (right). (b) Our visualizations also divulge
interesting insights in vision transformers, such as attention heads that
group image patches by hue and brightness. Border color denotes query
embeddings of a patch (green) or key embeddings (pink). (c) Sample input
sentences (from [23]) and (d) images (synthetic dataset) are provided for
reference.

Introduction

The transformer neural network architecture [52] is having a major impact on
fields ranging from natural language processing (NLP) [13, 42] to computer
vision [14]. Indeed, transformers are now deployed in large, real-world
systems used by hundreds of millions of people (e.g., Stable Diffusion,
ChatGPT, Microsoft Copilot). However, the mechanisms behind this success
remain somewhat mysterious, especially as new capabilities continue to emerge
with increasing model complexities and sizes [11, 60]. A deeper understanding
of transformer models could help us build more reliable systems, troubleshoot
problems, and suggest avenues for improvement.

In this work, we describe a new visualization technique aimed at better
comprehending how transformers operate. (We include a brief introduction to
transformers in Sec. 1.) The target of our analysis is the characteristic
transformer self-attention mechanism, which allows these models to learn and
use a rich set of relationships between input elements. Although attention
patterns have been intensively studied, previous techniques generally
visualize information related to just a single input sequence (e.g., one
sentence or image) at a time. Typical approaches create bipartite graph [53,
51] or heatmap [30, 20] representations of attention weights for a given input
sequence.

Our method offers a higher-level perspective, in which we can view the self-
attention patterns of many input sequences at once. One inspiration for this
approach is the success of tools such as the Activation Atlas [5], which
allows a researcher to “zoom out” to see an overview of a neural network, then
drill down for details. In our case, we seek to build a kind of “attention
atlas” that can provide researchers with a rich and detailed view of how a
transformer’s various attention heads operate. The primary new technique is
visualizing a joint embedding of the query and key vectors used by
transformers, which creates a visual signature for an individual attention
head.

To illustrate our technique, we implement AttentionViz, an interactive
visualization tool that allows users to explore attention in both language and
vision transformers. AttentionViz affords exploration through multiple levels
of detail (Fig. AttentionViz: A Global View of Transformer Attention),
providing both a global view to see all attention heads at once and the
ability to zoom in on details in a single attention head or input sequence.

We demonstrate the utility of our technique through several application
scenarios with AttentionViz and domain expert interviews. For concreteness, we
focus on what the visualization can reveal about a few widely-used
transformers: BERT [13], GPT-2 [41], and ViT [14]. We find several
identifiable “visual traces” linked to attention patterns in BERT, detect
novel hue/frequency behavior in ViT’s visual attention mechanism, and uncover
potentially anomalous behavior in GPT-2. User feedback also supports the wider
applicability of our approach in visualizing other embeddings at scale.

To summarize, the contributions of this work include:

  * •

A visualization technique for exploring attention trends in transformer models
based on joint query-key embeddings.

  * •

AttentionViz, an interactive tool that applies our technique for studying
self-attention in vision and language transformers at multiple scales.

  * •

Application scenarios and expert feedback showing how AttentionViz can reveal
insights about transformer attention patterns.

##  1 Background on Transformer Models

The transformer, introduced in [52], is a neural network architecture designed
to operate on sequential input. A full description of transformers is beyond
the scope of this paper, but a few concepts are critical for understanding our
work. First, a transformer receives as input a set of vectors (often called
embeddings). Embeddings can represent a variety of input types. In text-based
transformers, they correspond to words or pieces of words; in vision
transformers, they encode patches of pixels.

The network iteratively transforms these vectors via a series of attention
layers, each of which moves information between pairs of embeddings. The name
“attention” suggests that not all embeddings will be equally related; certain
pairs will interact more strongly–i.e., pay more “attention” to each other.
Attention layers determine which pairs should interact, and what information
should flow between them.

For example, in a transformer operating on the words of the sentence, “The
brown capybara is sleeping now,” one might expect high attention (and
information flow) between embeddings for “capybara” and “is,” but not between
“brown” and “now.” The self-attention mechanism, which is our focus in this
paper, allows transformers to learn and use a rich set of relationships
between elements of a sequence, yielding significant performance improvements
across various NLP and computer vision tasks [13, 41, 14].

There may be different reasons for embedding pairs to attend to each other.
For instance, in our example sentence,“brown” and “capybara” are linked by an
adjective-noun relation, while “capybara” and “is” form a subject-verb
relation. To allow for several relation types, transformer attention layers
consist of multiple attention heads, each of which can represent a different
pattern of attention and information flow.

Each attention head computes its own attention pattern using a bilinear form
computed from a query weight matrix $W_{Q}$ and key weight matrix $W_{K}$.
Concretely, for two embedding vectors $x$ and $y$, attention is related to the
scaled inner product of a query vector, $W_{Q}x$, and a key vector, $W_{K}y$.
Letting $d$ be the dimension of $W_{K}y$, we have:

$f(x,y)=\frac{1}{\sqrt{d}}\langle W_{Q}x,W_{K}y\rangle$

Given embedding vectors $\\{x_{1},x_{2},\ldots,x_{n}\\}$, we compute the self-
attention between $x_{i}$ and the other vectors using the softmax function:

$attn(x_{i},x_{j})=\mathrm{softmax}_{j}(f(x_{i},x),\ldots,f(x_{i},x_{n}))=e^{f(x_{i},x_{j})}/\sum_{k}e^{f(x_{i},x_{k})}$

Critically, this formula shows that the greater the dot product between the
query and key vectors, the higher the final attention value will be, a fact
that we rely upon in our joint embedding visualization.

There is much more to the transformer architecture than we cover here. In
particular, we have only described the attention weighting between pairs of
embeddings, and not the specific information that flows between them. (As
discussed later, this is an area ripe for further investigation.) One last
technical point is worth mentioning, however, since it will help interpret
images later in the paper. The initial embeddings given to a transformer
typically incorporate a vector representation of their ordering (for a 1D
sequence) or spatial configuration (for a grid, as in vision transformers).
For sequences, these position vectors are defined using trigonometric
functions, and are located on a helix-like curve in high-dimensional space
(see [52]).

###  1.1 Models Studied in this Paper

We study three transformer models: BERT (language), GPT-2 (language), and ViT
(vision). Each has been an important object of study in the machine learning
community, and the three span a range of transformer architectures and
applications. BERT, or Bidirectional Encoder Representations from Transformers
[13], is a multi-layer transformer encoder. As a bidirectional model, BERT can
attend to tokens (i.e., input elements) in either direction. GPT-2, or
Generative Pre-trained Transformer 2 [42], is a multi-layer transformer
decoder. GPT-2 is a unidirectional model, meaning it only attends to previous
tokens. ViT, or Vision Transformer [14], employs a self-attention-based
transformer architecture by splitting images into “patches” and treating them
like tokens in a sentence. Similar to BERT, ViT is a multi-layer,
bidirectional transformer encoder. In this work, we look at ViT performance on
16x16 (ViT-16) and 32x32 (ViT-32) patch sizes.

##  2 Related Work

Many researchers have attempted to investigate the inner workings of
transformers. [8, 34] seek to understand the performance improvement from
transformer-based language models by exploring learned linguistic
representations, and [49] observed that BERT recapitulates classic steps in
natural language analysis, from part-of-speech tagging to relation
classification. Another popular approach is mechanistic interpretability,
i.e., reverse engineering transformer models (e.g., [16, 37, 15]). Attention,
the backbone of transformers, has also been studied intensively. For example,
attention appears to relate to syntactic structures in NLP systems [9, 57] and
gestalt-like grouping in vision transformers [33]. Researchers have compared
ViT’s visual attention mechanism with convolutional filters as well, finding
that attention is more robust against image occlusion, corruption, and high-
frequency noise [35, 40]. In our discussion of related work, we focus on
visual approaches for studying transformer attention.

###  2.1 Visualizing Attention in a Single Input Sequence

Attention patterns naturally lend themselves to visualization, in both
language and vision transformers [39, 12, 21, 31, 4]. These visualizations are
largely focused on visualizing attention weights between query and key tokens
in a single input sequence using bipartite graphs (e.g., [53, 51, 30, 48]) or
heatmaps (e.g., [22, 30, 20, 43, 25, 1, 10]).

A few visualizations have been proposed that allow comparison across multiple
models or layers. For instance, Attention Flows [12] supports users in
comparing attention within and across layers of BERT, as well as among
attention heads given a single sentence. Dodrio [59] uses a grid view, applied
to single inputs, that enables direct comparison of attention heads. Another
system, VisQA [21], visualizes attention at different heads for visual
question-answering tasks by showing heatmaps of language self-attention,
vision self-attention, and language-vision cross-attention. Even in these
model-comparison systems, however, an analyst must look at different inputs,
one at a time, to identify and verify patterns for a given attention head.

###  2.2 Beyond Single Inputs: Visualizing Embeddings and Activation
Maximization

It is natural to seek patterns that hold across multiple inputs. One technique
that has proved effective toward this goal is visualizing collections of
embedding vectors from multiple input sequences [18, 47, 58, 3, 19, 46]. For
example, [43] visualized BERT embeddings for the same word used in many
different contexts, and found clusters that corresponded to word senses. In an
exploration of syntax processing, [8] visualized embeddings from a
multilingual BERT model and once again found meaningful clusters that helped
with interpretation. LMFingerprints [45] uses a tree-based radial layout to
compare embedding vectors across different language models.

A second technique, used for vision transformers in [17, 61], aims to find
images which maximize activations of particular units. Applied to embedding
vectors, this technique produces clearly interpretable results. The authors
note, however, that when applied to query and key vectors the technique does
not seem to produce useful results.

###  2.3 Gaps in the Literature

We note three gaps in the existing literature which motivate our work.

First, visualizing embedding vectors has been shown to be an effective
technique for analyzing patterns across multiple inputs, but we know of no
systematic attempt to visualize query and key embeddings in transformer
models. [6] also argues that the intermediate artifacts of self-attention,
such as queries and keys, are underexplored. These observations motivate our
joint query-key embedding technique.

Second, although visualization techniques have been proposed to compare
multiple embeddings (e.g., [26, 2, 3]), these methods are often limited to a
few embeddings and cannot address our needs of comparing embeddings at
different transformer heads and layers. Thus, we design a global matrix view
to visualize query-key embeddings at scale.

Finally, bipartite graph representations have proven helpful in analyzing NLP-
based transformers, but we have not seen them applied to vision tasks. We
explore this direction by creating bipartite-style visualizations to study
image attention patterns in ViT.

##  3 Goals & Tasks

The overarching aim of this work is to design a novel visualization technique
that allows exploration of global attention trends in transformer models. To
collect some initial feedback about this idea and learn more about user needs,
we talked with 5 machine learning (ML) researchers (4 Ph.D. students and 1
professor) interested in model interpretability. During these individual
interviews, we asked experts to describe their current practices and
challenges when working with transformers and how attention visualizations
could aid in their research objectives. We will refer to these experts as
E1-5.

###  3.1 Goals

Ultimately, our conversations with experts yielded three main goals:

G1 - Understand how self-attention informs model behavior. Overall, all 5
experts wanted to better understand the behavior of different attention heads
and what transformer models are learning through their characteristic self-
attention mechanism. Thus, they expressed the desire to be able to quickly and
easily explore attention patterns. E2 explained that “attention is still
pretty closed-box and there’s lots of mysteries,” so gaining a deeper
understanding of transformer attention patterns could provide insights into
“why large language models fail at reasoning tasks and math,” for example.

G2 - Compare & contrast attention heads. E5 mentioned that visualizing
differences in attention heads could help with hypothesis generation, which is
the first step in their research process: “Visualization can help formulate
hypotheses to test and get an intuitive sense of what transformers are doing.”
Additionally, 3 experts (E1, E2, E5) noted that attention head comparison
would be useful for model pruning and editing purposes. That is, if two
attention heads appear to behave similarly, perhaps one could be removed
without significantly impacting model performance. In the words of E1,
comparing heads might allow us to “find parts of the model that are actually
useful.”

G3 - Identify attention anomalies. Four researchers (E2-5) wanted to identify
irregularities and potential behavioral issues with transformers through
attention pattern exploration. This information could then be used for model
debugging purposes. For instance, E4 said “visualizing attention could help
you notice when the model is looking at the wrong thing, even if the result is
correct.” E3 agreed, reiterating the importance of debugging especially in the
context of model training: “Training often fails and dies, but it’s hard to
understand why it fails or produces unexpected behavior.”

###  3.2 Tasks

Given these goals, we developed the following set of design tasks:

T1 - Visualize attention heads at scale. To help users quickly explore model
behavior [G1] and easily compare & contrast attention patterns [G2], our tool
simultaneously visualizes self-attention heads across transformer layers.

T2 - Explore query-key interactions. E1 and E4 expressed the desire to better
understand query-key pairing information toward improving their understanding
of transformer self-attention. Thus, our tool further supports attention
pattern comparison [G2] and anomaly detection [G3] through visualizing query-
key interactions.

T3 - Probe attention at multiple levels. Our tool allows for local and global
comparisons of attention [G2] by providing visualizations at the
sentence/image, head, and model levels. The flexibility of switching between
multiple views in a single interface also facilitates knowledge discovery [G1]
and helps users identify model irregularities [G3].

T4 - Customize model and data inputs. AttentionViz is easily extendable to new
transformers and datasets, affording quick visual comparison [G2] and
synthesis of attention patterns [G1] across different models and modalities
(language & vision).

##  4 Query/Key Embeddings & Design of AttentionViz

To address these goals and tasks, we build a tool called AttentionViz. The
primary technique used by our tool is a visualization of the joint embedding
of query and key vectors for each attention head. In this section, we first
describe the motivation and mathematics that underlie this technique, then
discuss the design of the full application.

Figure 1: Creating a joint query-key embedding space for a single attention
head. In the NLP case, given an input sentence, we first transform each token
into its corresponding query and key vector. Then, we use t-SNE/UMAP/PCA to
project these $1\times d$ vectors into 2D/3D scatterplot coordinates. For the
BERT, GPT-2, and ViT models used, $d=64$.

###  4.1 Visualizing Query/Key Embeddings

The technique behind AttentionViz is relatively simple, although as we
describe below, it requires two mathematical tricks to be effective. Recall
that each transformer attention head transforms input embeddings into query
vectors and key vectors by applying matrices $W_{Q}$ and $W_{K}$, respectively
(Sec. 1). These matrices project the original vector embeddings to a lower
dimensional space, essentially selecting a particular type of information from
the higher-dimensional vector embeddings. Therefore, by inspecting the query
and key vectors, one might hope to learn what information is selected by
$W_{Q}$ and $W_{K}$.

A central observation is that the relative positions of query and key vectors
can offer clues about how attention will be distributed, since attention
coefficients depend on the dot product between queries and keys. To see why,
consider a hypothetical situation where query and key vectors always have the
same norm. Then, closer distances would directly relate to higher attention
coefficients. In practice, query and key vectors vary in norm, so the
relationship between dot product and distance is not precise. However, as
described in the following sections, we can arrange for this relation to be
surprisingly close.

Fig. 1 illustrates our technique with a synthetic example of a single
attention head in a language transformer. To create the joint embedding, we
first obtain the query and key vector representation of each token in a given
sentence (Sec. 1). Then, we use one of three dimensionality-reduction methods
to project these high-dimensional vectors onto a shared, lower-dimensional
subspace: t-SNE [50], UMAP [32], or PCA [24]. The output from these
dimensionality-reduction algorithms is a 2D/3D scatterplot, where each point
represents a single query or key token. The same process can be used to create
joint embeddings for ViT attention heads, where each token is an image patch.
By default, we visualize queries in green and keys in pink. However, there are
multiple color encodings users can choose from (see Sec. 4.2), and other
palettes can be easily substituted into the system.

Our joint embeddings allow users to explore the fine-grained interactions
between queries and keys, and the shape of these plots can often serve as
visual indicators of the underlying self-attention patterns (see Sec. 6). Each
dimensionality-reduction technique produces different patterns for a given
dataset, offering unique insights and use cases.

Figure 2: Left: original queries and keys in joint embedding space. Right:
Increased overlap after translating keys to align query and key centroids.

####  4.1.1 Vector Normalization

While designing AttentionViz, we noticed two “free parameters,” which can be
varied without losing any information. Tuning these parameters creates a
closer relationship between embedding distance and attention weights, and
greatly improves the readability of the visualization. These normalizations
are applied prior to dimensionality reduction (Fig. 1).

Key Translation: Query and key vectors are sometimes well separated in our
visualizations (Fig. 2, left). This separation makes it difficult to directly
compare query and key embeddings. However, a simple mathematical trick allows
us to move these embeddings closer together, without affecting the self-
attention computation for any given input sequence. In particular, note that
the softmax function is translation invariant: i.e., for any constant $a$, we
have
$\mathrm{softmax}_{j}(x_{1}+a,x_{2}+a,\ldots)=\mathrm{softmax}_{j}(x_{1},x_{2},\ldots)$.
Now, consider a query vector $x$ and key vectors $y_{1},\ldots,y_{n}$. For any
vector $v$, we have:

$\begin{split}\mathrm{attention}_{j}(x)&=\mathrm{softmax}_{j}(\langle
x,y_{1}\rangle,\langle x,y_{2}\rangle,\ldots)\\\
&=\mathrm{softmax}_{j}(\langle x,y_{1}\rangle+\langle x,v\rangle,\langle
x,y_{2}\rangle+\langle x,v\rangle,\ldots)\\\ &=\mathrm{softmax}_{j}(\langle
x,y_{1}+v\rangle,\langle x,y_{2}+v\rangle,\ldots)\end{split}$

where the second step follows by translation invariance. This implies that
without changing attention patterns in any given input, we can translate all
key vectors such that the query and key distributions for each attention head
have identical centroids. This makes it much easier to compare queries and
keys (Fig. 2, right).

Scaling Queries and Keys: In some transformers, such as GPT-2, we observed
cases where the average query norm was very different from the average key
norm. This difference makes it hard to interpret key-query relations in a
joint embedding. Mathematically, it indicates a poor relationship between dot
product and distance; visually, it means queries might be a tiny cluster,
surrounded by a loose cloud of keys.

Luckily, scale is another “free parameter” of the system. Self-attention
levels depend only on dot products of query and key vectors, so if we scale
all query vectors by a factor of $c\neq 0$, and all key vectors by a factor of
$c^{-1}$, the attention values are unchanged. This allows high-attention
query-key pairs to be closer together in our joint visualizations as depicted
in Fig. 3a. (A subtle point: on its own, scaling leaves cosine distance
unchanged; however, in combination with translation normalization it has a
nontrivial effect.)

To determine the optimal value of $c$, we can define a weighted correlation
metric that places heavier weight on query-key pairs with smaller distances,
since we care most about nearby queries and keys in the joint visualization.
We can thus choose a scale factor $c$ such that the weighted correlation
between query-key dot products and distances is maximized. This scaling method
allows for the distances in the joint embedding space to most accurately
represent the actual attention values between queries and keys.

Figure 3: (a) Ideal distance-attention relationship, where query-key pairs
with higher dot products are closer in the joint embedding space. (b) Example
attention head with a strong, negative correlation (-0.983) between query-key
distance and dot product in BERT.

####  4.1.2 Distance as a Proxy for Attention

As explained above, ideally, if a query-key pair has a large, positive dot
product (corresponding to a high final self-attention value), they should be
placed closer together in the embedding space, and vice versa (Fig. 3a). Thus,
we expect distance to be inversely correlated with attention in our joint
query-key embeddings. We study this potential link by computing the Spearman
rank correlation between cosine distance and dot product for each attention
head in BERT, GPT-2, and ViT. We also experimented with using Euclidean
distance as our distance metric when creating t-SNE and UMAP projections of
queries and keys, but this generally led to weaker distance-dot product
correlations.

Across multiple datasets and models, the relationship between distance and
attention holds fairly well. For example, with Wiki-Auto data [23], the mean
correlation between query-key distances and dot products is -0.938 for BERT
and -0.792 for GPT. An example result from BERT is shown in Fig. 3b. On the
set of COCO images used [28], the mean correlation is -0.873 for ViT-32 and
-0.884 for ViT-16.

###  4.2 Color Encodings

To visualize different properties of queries and keys, AttentionViz offers
various color encodings. The default option colors points by token type, i.e.,
query or key. For vision transformers, users can color by image patch row or
column to visualize positional patterns (Fig. 9). Since images encode their
own color information, we also allow users to view the original patches
without additional styling elements (Fig. 7).

For language transformers, we support two positional color schemes: normalized
and discrete. To compute normalized position, we divide each token’s position
in a sentence by the sentence length to produce a continuous color scale.
Lighter hues denote tokens closer to the beginning of the sentence (Fig. 4b).
Our discrete position encoding takes each token’s position and applies the
modulo operator to get its remainder when divided by 5. Thus, the $1^{st}$ and
$6^{th}$ tokens receive the same color, the $2^{nd}$ and $7^{th}$ tokens
receive the same color, etc. We use the same five colors to encode queries and
keys at different positions, using darker hues for the former. Although this
scheme introduces ambiguity for sentences with length > 5, we find our
discrete coloring helpful in seeing relationships based on small offsets in
position (e.g., queries paying attention to keys one step away as in Fig. 10,
left). Ambiguities are also easily resolved by hovering over tokens, and in
our explorations, we did not see patterns with offsets greater than 2 or 3.
Users can color by query/key norm as well (Fig. 11a).

###  4.3 Views

AttentionViz provides three main interactive views for attention exploration:
Matrix View, Single View, and Sentence/Image View.

Figure 4: Connecting form to function in BERT. (a) In Matrix View, there are
several spiral-shaped plots in layer 3. (b) By zooming into one such head (L3
H9) using Single View, we can see positional attention patterns by using a
light-to-dark color scheme that encodes position in the input sequence. (c)
These patterns can be confirmed by exploring sentence-level visualizations.
Figure 5: Exploring attention patterns with global search. (a) Heads with
fewer clusters of search results often demonstrate more semantic behavior,
while heads with dispersed results focus more on token position. (b) Zooming
into L2 H6, a head with one main result cluster, we indeed see a large group
of semantically related query and key tokens.

####  4.3.1 Matrix View

The initial view in AttentionViz is Matrix View, which uses small multiples to
visualize all the attention heads in a transformer at once (Fig. 4a), directly
addressing [T1] and [T3]. Each row corresponds to a model layer, moving from
earlier layers at the top of the interface to later layers at the bottom. With
this “global” perspective, users can more easily scan for patterns across
different transformer layers and heads, compared with single-plot (e.g., [53])
or instance-level visualizations (e.g., [3, 47]). All the models used in this
work had the same architecture: 12 layers x 12 heads per layer = 144 attention
heads in total, but our system scales to other dimensions.

In Matrix View, users can view the joint query-key embeddings created with
t-SNE, UMAP, or PCA. They can also switch between model types (i.e., BERT,
GPT-2, ViT-16/32) or datasets [T4], explore different color schemes, and view
the resultant plots in 2D or 3D. Matrix View supports a global search feature
(Fig. 5a), which helps highlight patterns in token locations across different
heads and offers another way to analyze attention at scale (see Sec. 6).

####  4.3.2 Single View

Users can click on any plot in Matrix View to zoom into Single View (Fig. 4b),
which affords exploration of a single attention head in closer detail [T3].
Like Matrix View, users can switch between colorings, dimensions, projection
modes, datasets, and models in Single View [T4]; all graphical changes sync
between views to facilitate comparison. The user can click on a point to
highlight all tokens in the corresponding input sequence, spotlighting the
relevant queries and keys in our joint embedding space. Users also have the
option to project attention lines onto the scatterplots, which connect query
and key tokens (Fig. 4c). We only show the top 2 attention weights for each
token to enhance readability. Our attention lines feature supports [T2] and
offers a new way to visualize attention patterns in transformers at the head
level.

In Single View, users can also search for tokens and use our labelling feature
to uncover semantic patterns in the data, similar to [47]. For instance, in
Fig. 5b, search reveals that query/key tokens with similar meanings are placed
together in the joint embedding for this BERT head, indicating strong
attention between them (Sec. 4.1.2).

Figure 6: ViT Image View. (a) Original image. (b) Transparency attention
heatmap. We highlight the selected token (query) with a green border. (c)
Overlaid attention arrows. (d) Global attention flow. The square icon means an
image patch has the strongest attention connection with the <CLS> token, which
is not in the original image.

####  4.3.3 Sentence/Image View

Sentence/Image View allows for exploration of the fine-grained attention
patterns within a single sentence or image [T2, T3]. Both views are
synchronized with Single View, matching the attention lines overlaid on each
query/key scatterplot for a smooth user experience.

Sentence View. When using BERT or GPT-2, users can click on a point in Single
View to open Sentence View in the left sidebar, which displays a BertViz-
inspired visualization of sentence-level attention with the clicked token
highlighted [53] (Fig. 4c). We also considered using heatmap visualizations
(e.g., [39]), but it seemed that the bipartite graph approach would offer
greater readability and ease of pattern exploration for longer sentences. The
opacity of the lines connecting query tokens in the left column and key tokens
in the right column signifies their corresponding attention strength. Hovering
on a token highlights token-specific attention lines. To reduce the noise from
classification tokens and separators in BERT, or the first token in GPT-2
(Sec. 6), users can hide the attention lines from these special tokens. Other
query and key tokens can also be toggled on/off, and all attention lines will
be re-normalized accordingly. Users have the option of viewing the aggregate
attention pattern for each attention head as well, to offer another layer of
comparison (Fig. 10a).

Image View. For image-based input in ViT, when users click on an image patch,
the side panel displays its corresponding original image and highlights the
clicked token with a colored border (Fig. 6a). Users also see an image
overlaid with an attention heatmap, where the transparency indicates the
attention weight between the clicked image patch and other regions of the
image (Fig. 6b).

Beyond visualizing the attention of a single token, Image View allows users to
explore the overall attention pattern within an image by showing arrowed
attention lines between different image patches. We provide users with two
options when visualizing the attention arrows. The first option overlays
arrows on the top of original image patches, with each arrow representing the
strongest attention connection between a starting image patch and destination
patch (Fig. 6c). This creates a simplified bipartite attention graph for users
to characterize the most important patterns within a specific head. The second
option shows all strong attention connections (i.e., $attn(x_{i},x_{j})>0.1$)
beside the original image, offering a more comprehensive view of attention
(Fig. 6d). In this visualization, both opacity and line thickness are used to
encode the strength of attention connections. We also tried visualizing all
weights between queries and keys to more closely mirror [53], but this often
produced overcrowded, inscrutable results.

##  5 System Implementation

To process model inputs and compute attention information, we use the Hugging
Face Transformers library and PyTorch. We use pre-trained implementations of
BERT, GPT-2 (small), and ViT-16/32 with model weights from Google and OpenAI.
For each NLP dataset, we randomly sample 200 sentences ($\sim$10k tokens per
attention head, including both queries and keys). Due to the increased
computational size of image attention data, we display 10 images per head
(1000 tokens) for ViT-32 and 4 images per head (1576 tokens) for ViT-16.
Larger datasets can be inputted into AttentionViz at the cost of increased
system latency. Currently, it takes $\sim$6 seconds to load each NLP dataset,
and $\sim$10 seconds to load ViT-16 data. After extracting query and key
vector embeddings for each attention head, we generate the corresponding 2D/3D
t-SNE, UMAP, and PCA coordinates (Sec. 4.1). To produce semantic labels (e.g.,
“dog” or “background”) for image patches in ViT, we use the DeepLabv3
segmentation model [7]. In total, it takes $\sim$3 hours to preprocess each
BERT/GPT-2 dataset on a NVIDIA A100 GPU; for ViT-32, it takes $\sim$30
minutes.

Our final AttentionViz prototype consists of a Python/Flask backend that
communicates with a frontend written in Vue and Typescript. The demo system is
available at: http://attentionviz.com. Due to the large size of the data and
browser memory constraints, we load pre-computed attention/projection
information via JSON files through the backend. For ViT, the backend also
performs image manipulation (e.g., patch highlighting and transparency
adjustments) to display in the frontend. We use Deck.gl to visualize the
resultant query-key joint embeddings. AttentionViz is highly extensible and
model-agnostic, allowing users to add new transformers and datasets to the
system.

Figure 7: In ViT-32 Matrix View, we find two interesting visual attention
heads: one head orders the black-and-white image tokens according to
brightness, while the other aligns the colorful patches based on hue.
Attention patterns shown in Image View confirm attention flow between patches
with the same luminance.

##  6 Findings & Evaluation

We illustrate the utility of AttentionViz with three application scenarios, as
well as feedback from domain experts. Our scenarios target the goals in Sec.
3, and show how AttentionViz can offer insights about global self-attention
trends in vision and language transformers.

Data. For BERT/GPT-2, we experimented with various NLP datasets but focus on
two for our application scenarios. We use Wiki-Auto [23] as a baseline to
sample general input sentences and SuperGLUE AXb [56] to explore task-specific
attention patterns for textual entailment. For ViT, we sample images from
ImageNet [44] and Microsoft COCO [28], as well as synthetic image data.

User Interviews. We invited E2 and E3 for a second round of interviews, and
included two new experts, E6 (interpretability researcher) and E7 (vision
science Ph.D. student). As in Sec. 3, all experts were interviewed
individually. We first gave experts a quick demo of our tool and shared some
of our own findings, asking them to share any thoughts or insights (Sec.
6.1-6.3). Then, we asked for more general feedback about the main strengths,
weaknesses, and novelties of AttentionViz (Sec. 6.4). We also asked experts
about possible extensions or applications of this technique for visualizing
embeddings at scale.

Figure 8: Left: Dataset of spatial patterns with different frequencies and
angles. Center: In Single View, we observe that one attention head of ViT-32
arranges image tokens based on the frequencies and angles of their spatial
pattern. Right: The attention heatmap in Image View further confirms these
findings – spatial patterns with similar angles pay greater attention to each
other.

###  6.1 Goal: Understanding Machine Visual Attention

AttentionViz can be especially helpful in uncovering new insights about
attention in vision transformers due to the inherently visual nature of image
patch data [G1].

Hue/brightness specializations in visual attention. We were curious if any
visual attention heads specialize in either color-based and brightness-based
patterns. To test this, we provided the pre-trained ViT-32 model with
synthetic color and brightness gradient images (Fig. 7), loading the resultant
query and key tokens into AttentionViz.

Browsing global PCA patterns in Matrix View, we identified two attention heads
that resemble color and colorless vision. One head appears to align black-and-
white image tokens based on brightness, and the other aligns colorful patches
based on hue. Our dataset contains color and brightness gradient images in all
orientations, and we see similar patches cluster together in the joint
embedding space regardless of their position in the original images. The
attention heatmap in Image View confirms these findings; tokens pay the most
attention to other tokens with the same color or brightness. E7 was intrigued
by these results, having previously studied the color latent space of
convolutional neural networks (CNNs), and expressed interest in using our tool
to further explore the differences between CNN and ViT behavior.

Frequency filtering and angle detection. Frequencies and angles are low-level
characteristics of image data. To investigate if the vision transformer has an
attention head that associates visual patterns based on these features, we
created images of sinusoidal signals with varying frequencies and
orientations, processing them using our pretrained ViT-32 model. Examining the
resultant query and key embeddings in Matrix View, we identified an attention
head that separates image tokens based on their spatial pattern’s frequencies
(x-axis) and angles (y-axis) (Fig. 8). With Image View, we observed that
tokens in the images of concentric circles are paying attention to other
tokens with similar curvatures, further confirming that this attention head
associates visual patterns based on their angles.

E7 said this result was interesting, but not too surprising given our
hue/brightness findings, and was more curious about heads that do not exhibit
this “attend to similar patches” behavior. One experiment they proposed was to
study attention modulation, e.g., if the same image patch (e.g., vertical
stripes) occurs in different contexts in two images (e.g., zebra vs.
umbrella), do we see unique attention patterns?

Figure 9: Coloring image patches by row highlights positional attention
patterns in ViT-32. In Layer 1, tokens in the same row and adjacent columns
form small clusters. Image View reveals a look at left pattern. In Layer 4,
large clusters of tokens form based on row positions. Using the arrowed lines,
we see a wider, bidirectional attention flow. Figure 10: Other visual traces
of attention. Left: Heads with small “clumps” often have even tighter
positional patterns than spirals. Our discrete position encoding, which colors
each token based on its position modulo 5, highlights a “next-token” attention
trend. Right: Layered bands of queries and keys only appear with SuperGLUE AXb
data [56], indicating strong attention to text start, end, and midpoint.

Increasing attention distance across model layers. As noted in [14], self-
attention attends more broadly across images in deeper layers of vision
transformers. We confirmed this finding using our interactive, joint
visualizations of query and key tokens in AttentionViz. With Matrix View, we
colored patches by image “row” and “column” to find four attention heads in
layers 1 and 2 of ViT-32 that group tokens with their nearest spatial
neighbors: on their left, right, top, and bottom. In layers 3 and 4, we saw
similar positional attention patterns, but image tokens pay attention to all
the patches in the same row or column, beyond their nearest neighbors (Fig.
9). This suggests that unlike CNNs, which process images using a square
filter, the self-attention mechanism in transformers often processes images
row by row and column by column, analogous to an elongated filter.

###  6.2 Goal: Finding Global Attention Traces

To understand how self-attention patterns vary across different heads in
language transformers [G2], we used AttentionViz to explore BERT.

Positional attention signatures. With TSNE, we observed several attention
heads with unique shapes, e.g., the spiral-shaped plots in layer 3 (Fig. 4a).
For example, coloring layer 3 head 9 by normalized position in Single View
reveals that token position increases as we move from the outside to the
inside of the spiral (Fig. 4b). We used Sentence View to examine this pattern
more closely (Fig. 4c), confirming that there is a positional, “next-token”
attention pattern. This “spiral” also reflects the initial ordering vector
given to transformers (Sec. 1).

We then noticed other identifiable “traces” in Matrix View, finding that plots
with small “clumps” also encode positional patterns (Fig. 10, left), which we
verified with our discrete position coloring. The difference between “spirals”
and “clumps” appears to be whether tokens attend selectively to others one
position away, versus at several different possible positions (Fig. 4c).
Similarly, we learned that in heads with high query-key overlap, tokens
typically attend to themselves and other instances of the same token,
exhibiting a “look at self” pattern. Zooming into these heads, we see clear
semantic clusters of nearby query-key pairs as shown in Fig. 5b, further
supporting this observation.

[29] shows that earlier transformer layers have the most information about
linear word order, aligning with our findings and previous work such as [9,
53]. During our interviews, E2, E6, and E7 immediately noticed these
interesting geometries, particularly spirals, and were curious about how much
of the observed structure is purely due to position. This inspired several
follow-up experiment ideas from experts, e.g., manipulating or removing the
positional embeddings in transformer models and seeing how our query-key
visualizations change.

Task-specific traces. After visualizing multiple datasets with AttentionViz,
we found that the shapes of joint embeddings are highly consistent across
different NLP tasks. However, we did see one visual trace that only arises in
some later layers of BERT with the SuperGLUE AXb data (Fig. 10, right).
Clicking on one such head (layer 8 head 9) and coloring by position, we
observed a query-key “sandwich,” where keys and queries at the beginning of
the text are stacked on top, followed by queries and keys at the end of the
text in reverse order.

Sentence View reveals that the start, middle, and end of the text receive the
most attention. The overall plot shape and attention pattern suggests that
these heads can identify a text’s “midpoint” and differentiate between
sentences, mirroring how in entailment tasks, two sentences are compared to
see if they have similar meanings. Queries also mostly attend to keys in the
same sentence. [25, 54] shows how syntactic and task-specific information is
most prominent in mid-to-later model layers, perhaps explaining the uniqueness
of this trace.

Global search patterns. The aggregate search feature in Matrix View can also
be used to quickly scan for and compare attention trends across heads [G2]. We
found that patterns in the search results reflect the previously identified
visual attention traces (Fig. 5a). For example, heads that are spiral-shaped
or have small clumps of queries/keys have more dispersed search results,
indicative of their underlying positional attention patterns. On the other
hand, heads with the “look at self” attention pattern only have one cluster of
search results, emphasizing the strong interaction between queries and keys of
the same token.

Even if a joint query-key embedding does not have a distinctive shape, we see
that if there are only a few search result clusters, the head may display more
semantic behavior; otherwise, there is likely a positional attention pattern.
[49] notes that semantic information is spread across BERT’s layers, which we
confirmed with AttentionViz. All of our experts were particularly excited by
this feature of our tool and its ability to facilitate attention pattern
comparisons.

###  6.3 Goal: Identifying Anomalies and Unexpected Behavior

Through interacting with the joint query-key embeddings in AttentionViz, we
discovered some irregular model behaviors [G3].

Figure 11: Anomalies in GPT-2. (a) In early model layers, we witness a
significant disparity between query-key norms for many attention heads (e.g.,
L1 H8 prior to norm scaling). (b) Example of the prevalent “attend to first”
pattern in later layers. Sentence View reveals latent attention behavior after
hiding the first token. Figure 12: Identifying a “look at self” attention head
in ViT-32. (a) Single View shows queries and keys are sparsely distributed in
the joint embedding space. (b) Zooming in, query and key vectors of the same
image token are tightly overlapped. (c) Image View reveals tokens pay most
attention to themselves. (d) Comparing the learned parameters of query and key
projection layers confirms that they learn redundant projections.

Norm disparities and null attention. While exploring GPT-2 in Matrix View, we
observed that in early model layers, some query and key clusters were well-
separated, even after key translation (Sec. 4.1.1). By coloring by norm (as
measured before the norm scaling step), we saw that in many heads, there is a
significant disparity between the norms of query and key vectors (Fig. 11a).
When query norms are small (i.e., light green), key norms tend to be large
(i.e., dark pink), and vice versa. Computing the average norm difference
between queries and keys in GPT-2 vs. BERT, we found that in the former, the
mean query norm - key norm = -4.59 across attention heads, while in the
latter, the mean difference is only 0.41. None of our experts could explain
this finding: “It doesn’t really make sense why queries and keys would have
such different norms” (E6). Interestingly, a paper published after we made
this observation [11] points to out-of-control query and key norms as a cause
of serious training instability, indicating that this phenomenon may be worth
studying further. This observation inspired our scaling approach from Sec.
4.1.1 as well.

We also noticed that in many GPT-2 heads, most attention is directed to the
first token (Fig. 11b), especially in later layers. [54] briefly mentions that
the first token is treated as a null position for attention-receiving in GPT-2
“when the linguistic property captured by the attention head doesn’t appear in
the input text.” However, this phenomenon remains underexplored, proposing
another open interpretability question to consider. E2 and E6 both noticed
this anomalous behavior on their own with our tool, and all of our experts
were surprised by this finding. [55] shows that pruning the majority of
attention heads in transformers may not significantly impact model
performance, which perhaps can be partially attributed to this dominant null
attention pattern. Regardless, AttentionViz allows users to filter out
attention paid to the first token, uncovering hidden query-key interactions.

“Look at self” attention heads. AttentionViz can also reveal surprising
attention patterns in vision transformers. In Matrix View, we identified
several heads in early layers of ViT-32 with very diffused key-query clusters
(Fig. 12a). Looking at one such attention head (layer 0 head 8), we discovered
that the query and key embeddings of the same token form a small but dense
cluster, with each query-key pair well-separated from the others (Fig. 12b).
From the transparency heatmap in Image View, we see that the patch is solely
attending to itself (Fig. 12c). Switching to the arrowed attention lines, we
discover that the overall attention pattern for this image is “look at self,”
where no information is flowing between image tokens in this head.

After identifying this irregular attention pattern, we checked the learned
parameters of the query and key matrices with a correlation test. We found a
strong similarity score (linear correlation $=\textbf{0.94}$), indicating that
the query and key layers in this ViT head are indeed learning redundant
projections (Fig. 12d). E3 noted that this knowledge could inform model
pruning experiments, and AttentionViz could similarly be used to detect
potential training failures or other irregularities.

###  6.4 Takeaways from User Feedback

Merits of Matrix View. Several experts found the “global” perspective provided
by Matrix View to be the most novel and valuable part of AttentionViz. As E6
said, “It’s great for quick comparison and frees you from tuning
hyperparameters when you want to visualize multiple embeddings at once.” E7
also mentioned that Matrix View is useful because “for smaller visualizations,
I can just code up something myself, but it’s a lot harder at scale and with
more data.” These comments suggest that this idea of visualizing and comparing
embeddings at scale may be beneficial in other ML settings as well.

Applications for joint query-key embeddings. Experts proposed various use
cases and extensions for our visualization technique, evidencing its wider
applicability. For example, E2 suggested visualizing patterns in untrained or
corrupt transformers, and both E3 and E7 wanted to visualize changes in
attention during training for their own models, aligning with our original
goals (Sec. 3). E2 and E6 also suggested adapting our tool to help with causal
tracing, explaining that “it might be useful to track attention flow
throughout the model for hypothesis testing.” Similarly, E3 expressed interest
in looking into “how two attention patterns connect in different heads,” which
could certainly be applied to visualizing induction head pairs. E2 noted that
adding a way to “quantify similarity between two heads” could be useful, while
E6 proposed “measuring or visualizing randomness in heads” for model pruning
purposes.

Embedding projections – to trust or not to trust? E3 highlighted the
challenges of using projection methods. While they appreciated the striking
geometric patterns (e.g., spirals) we found, E3 expressed some skepticism
about interpreting these visualizations due to the distortion from techniques
such as t-SNE and UMAP: “How do I know if I can trust what I see?” This
emphasizes the importance of tying visual insights to actionable
interventions, perhaps through augmenting our tool to support hypothesis
testing in addition to exploration.

Flexibility-usability tradeoff. E2 indicated that AttentionViz “feels very
usable and customizable,” contrasting with “existing visualization tools that
are too overwhelming to learn and use.” However, some experts like E6 were
still worried that “showing all the features and heads might be overwhelming…
Is there a way to summarize the information? Or focus more on a specific
task?” E7 added, “I wonder if there’s a quicker, more digestible way to label
heads,” suggesting an approach closer to feature visualization [36]. We
designed AttentionViz to be a flexible tool (e.g., allowing attention analysis
in different transformers and at different granularities), but it seems that
the flexibility-usability tradeoff [27] of our design could still be improved.

Additional interaction modes. Some experts suggested additional interaction
modes, e.g., on-the-fly inference (E3) or further dimensionality reductions on
circled clusters of queries and keys to reveal additional information and
perform fine-grained analyses (E2). E7 stressed the importance of allowing
users to directly upload new datasets to the system: “The tool could be even
more powerful… people are going to want to explore more with it like adding
their own images.”

##  7 Conclusions & Future Work

In this work, we introduce a new technique for visualizing transformer self-
attention based on a joint embedding space for queries and keys. Applying our
technique, we create AttentionViz (demo: http://attentionviz.com), an
interactive visualization tool, and use it to gain insights about attention in
both language and vision transformers. For instance, we discover novel
hue/frequency behavior in ViT, and striking query-key norm disparities in
GPT-2. Although our approach is tailored to self-attention, it can be
generalized to other attention mechanisms (e.g., cross attention) with proper
modifications to vector normalization. Similarly, we focus on semantic
patterns, but AttentionViz could be used to study syntactic features as well,
e.g., by using NLP libraries to add metadata such as part of speech.

Expert feedback also points to several avenues for future work, such as
finding ways to manage the complexity of multiple embedding visualizations and
focus users on features of interest. Our system is currently limited by data
pre-computation times and memory requirements, but we plan to improve the
scalability of AttentionViz and allow users to add new inputs on the fly. A
sufficiently large random sample (e.g., a few thousand tokens) is enough to
surface large-scale, global phenomena, but larger datasets and models may
reveal additional semantic insights, as suggested by [38]. Trying other data
sampling approaches might be fruitful as well (e.g., to reduce biases from
Zipf’s law [62]).

Another natural direction for future research is exploring how to incorporate
information from value vectors in each attention head [52]. These value
vectors are an essential part of the attention mechanism, though it is not
clear how to visualize them in the context of queries and keys. Finding the
right visualization approach might shed more light on how attention heads
function. Finally, although AttentionViz is an exploratory tool, adapting it
for hypothesis testing and/or causal tracing might provide support for
practical model debugging.

###### Acknowledgements.

We would like to thank all the participants in our user interviews for their
time and invaluable insights. We also thank the anonymous reviewers for
helping us improve this paper with their thorough and constructive comments.
Finally, we are grateful for the thoughtful feedback and support provided by
the members of the Harvard Insight + Interaction Lab throughout this project.

## References

  * [1] E. Aflalo, M. Du, S.-Y. Tseng, Y. Liu, C. Wu, N. Duan, and V. Lal.  Vl-interpret: An interactive visualization tool for interpreting vision-language transformers.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21406–21415. IEEE, New Orleans, 2022. doi: 10 . 1109/CVPR52688 . 2022 . 02072
  * [2] D. L. Arendt, N. Nur, Z. Huang, G. Fair, and W. Dou.  Parallel embeddings: a visualization technique for contrasting learned representations.  In Proceedings of the 25th International Conference on Intelligent User Interfaces, pp. 259–274. ACM, Cagliari, 2020. doi: 10 . 1145/3377325 . 3377514
  * [3] A. Boggust, B. Carter, and A. Satyanarayan.  Embedding comparator: Visualizing differences in global structure and local neighborhoods via small multiples.  In 27th International Conference on Intelligent User Interfaces, pp. 746–766. ACM, Helsinki, 2022. doi: 10 . 1145/3490099 . 3511122
  * [4] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin.  Emerging properties in self-supervised vision transformers.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9650–9660. IEEE, Montreal, 2021. doi: 10 . 1109/ICCV48922 . 2021 . 00951
  * [5] S. Carter, Z. Armstrong, L. Schubert, I. Johnson, and C. Olah.  Activation atlas.  Distill, 4(3), 2019. doi: 10 . 23915/distill . 00015
  * [6] H. Chefer, S. Gur, and L. Wolf.  Transformer interpretability beyond attention visualization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 782–791. IEEE, Online, 2021. doi: 10 . 1109/CVPR46437 . 2021 . 00084
  * [7] L.-C. Chen, G. Papandreou, F. Schroff, and H. Adam.  Rethinking atrous convolution for semantic image segmentation.  arXiv Preprint, 2017. doi: 10 . 48550/arXiv . 1706 . 05587
  * [8] E. A. Chi, J. Hewitt, and C. D. Manning.  Finding universal grammatical relations in multilingual BERT.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5564–5577. ACL, Online, 2020. doi: 10 . 18653/v1/2020 . acl-main . 493
  * [9] K. Clark, U. Khandelwal, O. Levy, and C. D. Manning.  What does BERT look at? an analysis of BERT’s attention.  In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 276–286. ACL, Florence, 2019. doi: 10 . 18653/v1/W19-4828
  * [10] J.-B. Cordonnier, A. Loukas, and M. Jaggi.  On the relationship between self-attention and convolutional layers.  In Eighth International Conference on Learning Representations. Addis Ababa, 2020. doi: 10 . 48550/arXiv . 1911 . 03584
  * [11] M. Dehghani, J. Djolonga, B. Mustafa, P. Padlewski, J. Heek, J. Gilmer, A. Steiner, M. Caron, R. Geirhos, I. Alabdulmohsin, et al.  Scaling vision transformers to 22 billion parameters.  arXiv Preprint, 2023. doi: 10 . 48550/arXiv . 2302 . 05442
  * [12] J. F. DeRose, J. Wang, and M. Berger.  Attention flows: Analyzing and comparing attention mechanisms in language models.  IEEE Transactions on Visualization and Computer Graphics, 27(2):1160–1170, 2021. doi: 10 . 1109/TVCG . 2020 . 3028976
  * [13] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv Preprint, 2018. doi: 10 . 48550/arXiv . 1810 . 04805
  * [14] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In Ninth International Conference on Learning Representations. Online, 2021. doi: 10 . 48550/arXiv . 2010 . 11929
  * [15] N. Elhage, T. Hume, C. Olsson, N. Schiefer, T. Henighan, S. Kravec, Z. Hatfield-Dodds, R. Lasenby, D. Drain, C. Chen, et al.  Toy models of superposition.  arXiv Preprint, 2022. doi: 10 . 48550/arXiv . 2209 . 10652
  * [16] N. Elhage, N. Nanda, C. Olsson, T. Henighan, N. Joseph, B. Mann, A. Askell, Y. Bai, A. Chen, T. Conerly, et al.  A mathematical framework for transformer circuits.  Transformer Circuits Thread, 2021.  https://transformer-circuits.pub/2021/framework/index.html.
  * [17] A. Ghiasi, H. Kazemi, E. Borgnia, S. Reich, M. Shu, M. Goldblum, A. G. Wilson, and T. Goldstein.  What do vision transformers learn? a visual exploration.  arXiv Preprint, 2022. doi: 10 . 48550/arXiv . 2212 . 06727
  * [18] F. Hohman, M. Kahng, R. Pienta, and D. H. Chau.  Visual analytics in deep learning: An interrogative survey for the next frontiers.  IEEE Transactions on Visualization and Computer Graphics, 25(8):2674–2693, 2018. doi: 10 . 1109/TVCG . 2018 . 2843369
  * [19] F. Hohman, H. Park, C. Robinson, and D. H. P. Chau.  Summit: Scaling deep learning interpretability by visualizing activation and attribution summarizations.  IEEE Transactions on Visualization and Computer Graphics, 26(1):1096–1106, 2019. doi: 10 . 1109/TVCG . 2019 . 2934659
  * [20] B. Hoover, H. Strobelt, and S. Gehrmann.  exBERT: A visual analysis tool to explore learned representations in Transformer models.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 187–196. ACL, Online, 2020\. doi: 10 . 18653/v1/2020 . acl-demos . 22
  * [21] T. Jaunet, C. Kervadec, R. Vuillemot, G. Antipov, M. Baccouche, and C. Wolf.  Visqa: X-raying vision and language reasoning in transformers.  IEEE Transactions on Visualization and Computer Graphics, 28(1):976–986, 2022. doi: 10 . 1109/TVCG . 2021 . 3114683
  * [22] X. Ji, Y. Tu, W. He, J. Wang, H.-W. Shen, and P.-Y. Yen.  Usevis: Visual analytics of attention-based neural embedding in information retrieval.  Visual Informatics, 5(2):1–12, 2021. doi: 10 . 1016/j . visinf . 2021 . 03 . 003
  * [23] C. Jiang, M. Maddela, W. Lan, Y. Zhong, and W. Xu.  Neural CRF model for sentence alignment in text simplification.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7943–7960. ACL, Online, 2020. doi: 10 . 18653/v1/2020 . acl-main . 709
  * [24] I. T. Jolliffe.  Principal Components in Regression Analysis, pp. 129–155.  Springer, New York, 1986. doi: 10 . 1007/978-1-4757-1904-8_8
  * [25] O. Kovaleva, A. Romanov, A. Rogers, and A. Rumshisky.  Revealing the dark secrets of BERT.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pp. 4365–4374. ACL, Hong Kong, 2019. doi: 10 . 18653/v1/D19-1445
  * [26] Q. Li, K. S. Njotoprawiro, H. Haleem, Q. Chen, C. Yi, and X. Ma.  Embeddingvis: A visual analytics approach to comparative network embedding inspection.  In 2018 IEEE Conference on Visual Analytics Science and Technology, pp. 48–59. IEEE, Berlin, 2018. doi: 10 . 1109/VAST . 2018 . 8802454
  * [27] W. Lidwell, K. Holden, and J. Butler.  Universal principles of design, revised and updated: 125 ways to enhance usability, influence perception, increase appeal, make better design decisions, and teach through design.  Rockport Pub, Beverly, 2010.
  * [28] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick.  Microsoft coco: Common objects in context.  In European Conference on Computer Vision, pp. 740–755. Springer, Zurich, 2014. doi: 10 . 1007/978-3-319-10602-1_48
  * [29] Y. Lin, Y. C. Tan, and R. Frank.  Open sesame: Getting inside BERT’s linguistic knowledge.  In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 241–253. ACL, Florence, 2019. doi: 10 . 18653/v1/W19-4825
  * [30] S. Liu, T. Li, Z. Li, V. Srikumar, V. Pascucci, and P.-T. Bremer.  Visual interrogation of attention-based models for natural language inference and machine comprehension.  In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 36–41. ACL, Brussels, 2018. doi: 10 . 18653/v1/D18-2007
  * [31] J. Ma, Y. Bai, B. Zhong, W. Zhang, T. Yao, and T. Mei.  Visualizing and understanding patch interactions in vision transformer.  IEEE Transactions on Neural Networks and Learning Systems, pp. 1–10, 2023. doi: 10 . 1109/TNNLS . 2023 . 3270479
  * [32] L. McInnes, J. Healy, and J. Melville.  Umap: Uniform manifold approximation and projection for dimension reduction.  arXiv Preprint, 2018. doi: 10 . 48550/arXiv . 1802 . 03426
  * [33] P. Mehrani and J. K. Tsotsos.  Self-attention in vision transformers performs perceptual grouping, not attention.  arXiv Preprint, 2023. doi: 10 . 48550/arXiv . 2303 . 01542
  * [34] A. Miaschi, D. Brunato, F. Dell’Orletta, and G. Venturi.  Linguistic profiling of a neural language model.  In Proceedings of the 28th International Conference on Computational Linguistics, pp. 745–756. International Committee on Computational Linguistics, Barcelona, 2020. doi: 10 . 18653/v1/2020 . coling-main . 65
  * [35] M. M. Naseer, K. Ranasinghe, S. H. Khan, M. Hayat, F. Shahbaz Khan, and M.-H. Yang.  Intriguing properties of vision transformers.  In Advances in Neural Information Processing Systems, vol. 34, pp. 23296–23308. Curran Associates, Inc., Online, 2021. doi: 10 . 48550/arXiv . 2105 . 10497
  * [36] C. Olah, A. Mordvintsev, and L. Schubert.  Feature visualization.  Distill, 2(11), 2017. doi: 10 . 23915/distill . 00007
  * [37] C. Olsson, N. Elhage, N. Nanda, N. Joseph, N. DasSarma, T. Henighan, B. Mann, A. Askell, Y. Bai, A. Chen, et al.  In-context learning and induction heads.  arXiv Preprint, 2022. doi: 10 . 48550/arXiv . 2209 . 11895
  * [38] M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, et al.  Dinov2: Learning robust visual features without supervision.  arXiv Preprint, 2023. doi: 10 . 48550/arXiv . 2304 . 07193
  * [39] C. Park, I. Na, Y. Jo, S. Shin, J. Yoo, B. C. Kwon, J. Zhao, H. Noh, Y. Lee, and J. Choo.  Sanvis: Visual analytics for understanding self-attention networks.  In 2019 IEEE Visualization Conference, pp. 146–150. IEEE, Vancouver, 2019. doi: 10 . 1109/VISUAL . 2019 . 8933677
  * [40] N. Park and S. Kim.  How do vision transformers work?  In Tenth International Conference on Learning Representations. Online, 2022. doi: 10 . 48550/arXiv . 2202 . 06709
  * [41] A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, et al.  Improving language understanding by generative pre-training.  OpenAI Blog, 2018.  https://openai.com/research/language-unsupervised.
  * [42] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al.  Language models are unsupervised multitask learners.  OpenAI Blog, 2019.  https://openai.com/research/better-language-models.
  * [43] E. Reif, A. Yuan, M. Wattenberg, F. B. Viegas, A. Coenen, A. Pearce, and B. Kim.  Visualizing and measuring the geometry of bert.  In Advances in Neural Information Processing Systems, vol. 32. Curran Associates, Inc., Vancouver, 2019. doi: 10 . 5555/3454287 . 3455058
  * [44] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al.  Imagenet large scale visual recognition challenge.  International Journal of Computer Vision, 115:211–252, 2015. doi: 10 . 1007/s11263-015-0816-y
  * [45] R. Sevastjanova, E. Cakmak, S. Ravfogel, R. Cotterell, and M. El-Assady.  Visual comparison of language model adaptation.  IEEE Transactions on Visualization and Computer Graphics, 29(1):1178–1188, 2022. doi: 10 . 1109/TVCG . 2022 . 3209458
  * [46] V. Sivaraman, Y. Wu, and A. Perer.  Emblaze: Illuminating machine learning representations through interactive comparison of embedding spaces.  In 27th International Conference on Intelligent User Interfaces, p. 418–432. ACM, Helsinki, 2022. doi: 10 . 1145/3490099 . 3511137
  * [47] D. Smilkov, N. Thorat, C. Nicholson, E. Reif, F. B. Viégas, and M. Wattenberg.  Embedding projector: Interactive visualization and interpretation of embeddings.  arXiv Preprint, 2016. doi: 10 . 48550/arXiv . 1611 . 05469
  * [48] H. Strobelt, S. Gehrmann, M. Behrisch, A. Perer, H. Pfister, and A. M. Rush.  Seq2seq-vis: A visual debugging tool for sequence-to-sequence models.  IEEE Transactions on Visualization and Computer Graphics, 25(1):353–363, 2019. doi: 10 . 1109/TVCG . 2018 . 2865044
  * [49] I. Tenney, D. Das, and E. Pavlick.  BERT rediscovers the classical NLP pipeline.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4593–4601. ACL, Florence, 2019. doi: 10 . 18653/v1/P19-1452
  * [50] L. van der Maaten and G. Hinton.  Visualizing data using t-sne.  Journal of Machine Learning Research, 9(86):2579–2605, 2008.  http://jmlr.org/papers/v9/vandermaaten08a.html.
  * [51] A. Vaswani, S. Bengio, E. Brevdo, F. Chollet, A. N. Gomez, S. Gouws, L. Jones, Ł. Kaiser, N. Kalchbrenner, N. Parmar, et al.  Tensor2tensor for neural machine translation.  arXiv Preprint arXiv:1803.07416, 2018.
  * [52] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin.  Attention is all you need.  In Advances in Neural Information Processing Systems, vol. 30. Curran Associates, Inc., Long Beach, 2017. doi: 10 . 48550/arXiv . 1706 . 03762
  * [53] J. Vig.  A multiscale visualization of attention in the transformer model.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 37–42. ACL, Florence, 2019\. doi: 10 . 18653/v1/P19-3007
  * [54] J. Vig and Y. Belinkov.  Analyzing the structure of attention in a transformer language model.  In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 63–76. ACL, Florence, 2019. doi: 10 . 18653/v1/W19-4808
  * [55] E. Voita, D. Talbot, F. Moiseev, R. Sennrich, and I. Titov.  Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5797–5808. ACL, Florence, 2019. doi: 10 . 18653/v1/P19-1580
  * [56] A. Wang, Y. Pruksachatkun, N. Nangia, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman.  Superglue: A stickier benchmark for general-purpose language understanding systems.  In Advances in Neural Information Processing Systems, vol. 32, pp. 3266–3280. Curran Associates Inc., Vancouver, 2019. doi: 10 . 5555/3454287 . 3454581
  * [57] K. Wang, A. Variengien, A. Conmy, B. Shlegeris, and J. Steinhardt.  Interpretability in the wild: a circuit for indirect object identification in gpt-2 small.  arXiv Preprint, 2022. doi: 10 . 48550/arXiv . 2211 . 00593
  * [58] Y. Wang, S. Liu, N. Afzal, M. Rastegar-Mojarad, L. Wang, F. Shen, P. Kingsbury, and H. Liu.  A comparison of word embeddings for the biomedical natural language processing.  Journal of Biomedical Informatics, 87:12–20, 2018. doi: 10 . 1016/j . jbi . 2018 . 09 . 008
  * [59] Z. J. Wang, R. Turko, and D. H. Chau.  Dodrio: Exploring transformer models with interactive visualization.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations, pp. 132–141. ACL, Online, 2021. doi: 10 . 18653/v1/2021 . acl-demo . 16
  * [60] J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, E. H. Chi, T. Hashimoto, O. Vinyals, P. Liang, J. Dean, and W. Fedus.  Emergent abilities of large language models.  Transactions on Machine Learning Research, 2022. doi: 10 . 48550/arXiv . 2206 . 07682
  * [61] S. Yang, Z. Quan, M. Nie, and W. Yang.  Transpose: Keypoint localization via transformer.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11802–11812. Montreal, 2021. doi: 10 . 1109/ICCV48922 . 2021 . 01159
  * [62] G. K. Zipf.  Selected studies of the principle of relative frequency in language.  Harvard University Press, Cambridge, 1932. doi: 10 . 4159/harvard . 9780674434929