# Disentangled Representation Learning for Text-Video Retrieval

Qiang Wang DAMO Academy, Alibaba Group
11email: {qishi.wq, yanhao.zyh, zhengyun.zy, panpan.pp,
<EMAIL_ADDRESS>Yanhao Zhang DAMO Academy, Alibaba Group
11email: {qishi.wq, yanhao.zyh, zhengyun.zy, panpan.pp,
<EMAIL_ADDRESS>Yun Zheng DAMO Academy, Alibaba Group
11email: {qishi.wq, yanhao.zyh, zhengyun.zy, panpan.pp,
<EMAIL_ADDRESS>Pan Pan DAMO Academy, Alibaba Group
11email: {qishi.wq, yanhao.zyh, zhengyun.zy, panpan.pp,
<EMAIL_ADDRESS>Xian-Sheng Hua
DAMO Academy, Alibaba Group
11email: {qishi.wq, yanhao.zyh, zhengyun.zy, panpan.pp,
<EMAIL_ADDRESS>
###### Abstract

Cross-modality interaction is a critical component in Text-Video Retrieval
(TVR), yet there has been little examination of how different influencing
factors for computing interaction affect performance. This paper first studies
the interaction paradigm in depth, where we find that its computation can be
split into two terms, the interaction contents at different granularity and
the matching function to distinguish pairs with the same semantics. We also
observe that the single-vector representation and implicit intensive function
substantially hinder the optimization. Based on these findings, we propose a
disentangled framework to capture a sequential and hierarchical
representation. Firstly, considering the natural sequential structure in both
text and video inputs, a Weighted Token-wise Interaction (WTI) module is
performed to decouple the content and adaptively exploit the pair-wise
correlations. This interaction can form a better disentangled manifold for
sequential inputs. Secondly, we introduce a Channel DeCorrelation
Regularization (CDCR) to minimize the redundancy between the components of the
compared vectors, which facilitate learning a hierarchical representation. We
demonstrate the effectiveness of the disentangled representation on various
benchmarks, e.g., surpassing CLIP4Clip largely by $+2.9\%$, $+3.1\%$,
$+7.9\%$, $+2.3\%$, $+2.8\%$ and $+6.5\%$ R@1 on the MSR-VTT, MSVD, VATEX,
LSMDC, AcitivityNet, and DiDeMo, respectively.

###### Keywords:

Video Retrieval, Cross-modality Interaction, Decorrelation

##  1 Introduction

Figure 1: Illustrations of common underlying assumptions for Text-Video
Retrieval. (1) The description could be relevant to the local video context.
(2) Multiple distinct sentences can hint at the same video. Our disentangled
representation addresses both assumptions through token-wise interaction and
channel decorrelation.

Text-Video Retrieval (TVR) has significant worth with the explosive growth of
video content on the internet. Alignment between different modalities requires
particular consideration of both intra-modal representation and cross-modality
interaction. The inherent asymmetry across different modalities raises an
expression challenge for Text-Video Retrieval. The pioneering works [12, 26]
incorporate multi-modality features and aggregate information from different
pre-trained experts to boost performance. While the open-ended text queries
and diverse vision contents require tremendous labor.

With the great success of NLP pre-training [8, 27, 22, 36, 37], Vision-
Language Pre-Training (VLPT) has received increasing attention [23, 21, 24,
28, 2]. Recently, the Contrastive Language–Image Pre-training (CLIP [35])
leverages a massive amount of image-caption pairs to learn generic vision and
language representations, showing impressive improvements in diverse vision-
language (VL) tasks. Researchers [34, 2, 29, 11, 13] explore the power of this
design principle to text-video retrieval with additional temporal fusion
module. CLIP4Clip [29] utilizes a temporal transformer to aggregate sequential
features into a single high-dimensional representation, and retrievals by dot-
product search.

However, the text-video retrieval is uniquely characterized by its sequential
frames. Thus we highlight the common underlying assumptions behind TVR in
Figure 1: (1) the description related with the local segment still needs to be
retrieved, and (2) human-generated sentences naturally have a hierarchical
structure and can describe in different views. Based on these perspectives, we
can re-examine the existing algorithmic framework, especially the interaction
components.

Figure 2 shows typical interaction archetypes and the proposed variations. In
determining the process flow for the interaction block, only a few properties
are commonly considered. One is the granularity of input content, and the
other is the interaction function.

The single-vector representation is widely used in the fields of biometrics
[7, 46] and text retrieval [15], and its retrieval process is extremely
concise and efficient. While as shown in Figure 2 (a), the over-abstract
representation will introduce a lack of fine-grained matching capabilities.
Therefore, the Mixtures of Experts (MoE) approaches [26, 12, 25] take
advantage of individual domains to integrate generalizable aggregated
features. MMT [12] and CE [26] explicitly construct a multi-expert fusion
mechanism to boost retrieval performance. In addition, HIT [25] performs
hierarchical cross-modality contrastive matching at both feature-level and
semantic-level. They can all be summed up as a hierarchical fusion
architecture (Figure 2(b)).

In contrast to the above parameter-free dot-product function, the deeply-
contextualized interaction has emerged that fine-tunes MLP models for
estimating relevance [41, 48] (Figure 2(c)). The recent cross transformer
interaction approaches [23, 29] can inference the complex relationship between
arbitrary-length text and video. However, the neural network interaction lacks
the typical inductive bias for matching and usually suffers from optimization
difficulty and performance degradation [29]. Furthermore, these heavy
interactions will bring a prohibitively computational cost for real-world
deployments.

We present a disentangled framework to address the above challenges, where a
novel token-wise interaction and channel decorrelation regularization
collaboratively decouples the sequential and hierarchical representation.
Concretely, we propose a lightweight token-wise interaction that fully
interacts with all sentence tokens and video frame tokens (Figure 2(e-f)).
Compared with the single-vector interaction (Figure 2(a)(c)) and multi-level
interaction (Figure 2(b)), our method can preserve more fine-grained clues.
Compared to the cross transformer interaction (Figure 2(d)), the proposed
interaction mechanism significantly ease the optimization difficulty and
computational overhead. In addition to the interaction mechanism, we employ a
Channel DeCorrelation Regularization (CDCR) to minimize the redundancy between
the components of the compared vectors, which facilitate learning a
hierarchical representation. The proposed modules are orthogonal to the
existing pretraining techniques [23, 21, 24, 28, 2] and can be easily
implemented by a few lines of code in modern libraries.

We validate the retrieval ability of our approach on multiple text-video
retrieval datasets. Our experimental results outperform the state-of-the-art
methods under widely used benchmarks, e.g., surpassing CLIP4Clip [29] largely
by $+2.9\%$, $+3.1\%$, $+7.9\%$, $+2.3\%$, $+2.8\%$ and $+6.5\%$ R@1 on the
MSR-VTT [44], MSVD [43], VATEX [42], LSMDC [39], AcitivityNet [10], and DiDeMo
[1], respectively. Notably, using ViT-B/16 [35] and QB-Norm [3] post
processing, our best model can achieve $53.3\%$ T2V R@1 and $56.2\%$ V2T R@1,
surpassing all existing single-model entries.

Our empirical analysis suggests that there is vast room for improvement in the
design of interaction mechanisms. The findings used in this paper make some
initial headway in this direction. We hope that this study will spur further
investigation into the operational mechanisms used in modeling interaction.

Figure 2: Illustrations of the text-video retrieval architecture and six
categories of interaction methods. Each subplot shows a typical interaction
module.

##  2 Related Work

Feature Representation for Text-Video Retrieval. In recent studies [26, 35,
34, 29], the text and video inputs are usually considered separately for
efficient deployment through a Bi-Encoder architecture [38, 18]. The text
encoder absorbs progress in the NLP field, and is upgraded from the early
Word2Vec [30] to the BERT-like models [8, 27]. While for the video input, due
to its rich semantic content, researchers utilize multi-modality features [12]
and a variety of pre-trained experts [26] to boost performance. Recently, CLIP
[35] proposes a concise contrastive learning method and trains on a large-
scale dataset with 400 million pairs to obtain the generic cross-modal
representation. Indeed, in the span of just a few months, several CLIP-based
TVR methods [29, 13, 11] constantly refresh the state-of-the-art results on
all benchmarks. Our approach also benefits from existing pre-training, while
with mainly focusing on the design of the interaction module. We conduct
extensive and fair ablation studies to evaluate the effectiveness of our
approach.

Interaction Mechanism for Text-Video Retrieval. Over the past few years, most
TVR studies have focused on improving performance by increasing the power of
text and visual encoders [6, 12, 25, 26]. Thus the simple dot-product
interaction is widely adopted for computing global similarity [29]. The time-
series video features are aggregated with average pooling, LSTM [14], or
temporal transformer [29]. Recent works [6, 12, 26] on the Mixture of Experts
(MoE [16]) show that integrating diverse domain experts can improve the
overall model’s capability with a voting procedure. In [25], hierarchical
feature alignment methods help models utilize information from different
dimensions. The pioneering work, JSFusion [48], proposes to measure dense
semantic similarity between all sequence data and learn a sophisticated
attentions network. The Cross Transformer [23, 28, 29] establish the multi-
modality correspondence by joint encoding texts and videos, which can capture
both intra- and inter-modality context. We conduct the empirical study on the
latest instantiation of interaction. Our work is also related to several
approaches that analyze the interaction mechanism on Information Retrieval
(IR) [18] and Text-Image Retrieval [19]. This work targets a deeper
understanding of the interaction mechanism for Text-Video Retrieval in a new
perspective.

Contrastive Learning for Text-Video Retrieval. A common underlying theme that
unites retrieval methods is that they aim to learn closer representations for
the labelled text-video pairs. The triplet loss based models[33, 26] use a
max-margin approach to separate positive from negative examples and perform
intra-batch comparisons. With the popularity of self-supervised learning (SSL)
[5, 4, 31], InfoNCE [31] have dominated recent retrieval tasks, which mainly
maximizes the diagonal similarity and achieves superior performance for noisy
data. The Barlow Twins method [49] have recently been proposed as a new
solution for SSL. They use a covariance matrix to indicate inter-channel
redundancy. We employ a sequential channel decorrelation regularization to
reduce inter-channel redundancy and competition, which fits the assumptions of
hierarchical semantics in Text-Video Retrieval.

##  3 Methodology

Figure 3: Overview of the proposed disentangled representation learning design
for Text-Video Retrieval. The pipeline of our method consists of two
components: the Weighted Token-wise Interaction (WTI) block that exhaustively
matching all sequential representation for the sentence and video; the Channel
DeCorrelation Regularization (CDCR) minimizes the redundancy for the
sequential representation.

The framework of the proposed Text-Video Retrieval (TVR) is shown in Figure 3.
Our method first generates sequential feature representation for both text and
video. In the cross-modality interaction stage, the network will reason the
dense correlation for all pair-wise features and dynamically adjust weights
based on the token contents. We additionally introduce a channel decorrelation
regularization to minimize the redundancy of these features. Consequently,
given a text query $\mathbf{t}=\\{t^{i}\\}_{i=1}^{N_{t}}$ with $N_{t}$ tokens,
and a set of video documents $\mathcal{V}=\\{\mathbf{v}_{n}\\}_{n=1}^{N}$, the
text-video retrieval is formulated as a cross-modality similarity measurement,
$\mathcal{S}(\mathbf{t},\mathbf{v})$. We can extract text features
$\mathbf{e}_{t}$ and video features $\mathbf{e}_{v}$ through text encoder
$E_{t}(\mathbf{t},\theta_{t})$ and video encoder
$E_{v}(\mathbf{v},\theta_{v})$, respectively. Different from the global dot-
product operation with single vector, our embedding features retain the
sequential structure of text and video frames with
$\mathbf{e}_{t}\in\mathbb{R}^{N_{t}\times D}$,
$\mathbf{e}_{v}\in\mathbb{R}^{N_{v}\times D}$, where $N_{v}$ is the sampled
video frames and $D$ is the feature dimension. We further introduce a Weighted
Token-wise Interaction (WTI) module to estimate the cross-modality correlation
and discover potential activate tokens. In order to improve the domain
generalization, we leverage the covariance matrix to indicate the redundancy
of features and employ a simple Channel Decorrelation Regularization (CDCR) to
improve the performance. The decorrelation loss implicitly alleviates an
overlooked channel competition, achieving impressive performance in retrieval
more complex sentences.

###  3.1 Feature Extractor

We utilize the efficient Bi-Encoder architecture for feature extraction. For
each query sentence $\mathbf{t}$, we add the [CLS] and [SEP] token to indicate
the start and the end of the sentence, and adopt the pretrained BERT-BASE [8]
to encode the text representation $\mathbf{e}_{t}=E_{t}(\mathbf{t})$. For each
video $\mathbf{v}$, we uniformly select $N_{v}$ frames as keyframes and employ
off-the-shelf transformer-based networks, e.g. ViT [9] , to extract sequential
features $\mathbf{e}_{v}=E_{v}(\mathbf{v})$.

Retrieval across modalities benefits from large-scale pre-training tasks [18,
8, 35]. In this paper, we mainly focus on the design of the interaction module
rather than the pretrained network. Thus our feature extractors are
initialized from the CLIP [35], and we finetune in an end-to-end manner.

###  3.2 Study of Interaction Mechanisms

Table 1: Comparisons of different interaction mechanisms. $N$ denotes number of video documents ($N$ is large and depends on applications); $D$ denotes representation dimension ($D=512$, by default); $N_{t}$ and $N_{v}$ denote length of text token and frame token, $N_{v+t}$ is the sum of $N_{t}$ and $N_{v}$ ($N_{t}=32,N_{v}=12$); $L$ denotes number of network layer and $S$ denotes feature levels ($L=4,S=3$, by default). Interaction | Contents | Function | Computational Complexity | Memory
---|---|---|---|---
DP | single | parameter-free | $\mathcal{O}(ND)$ | $\mathcal{O}(ND)$
HI | multi-level | light-parameter | $\mathcal{O}(NSD)$ | $\mathcal{O}(NSD)$
MLP | single | black-box | $\mathcal{O}(N(D^{2}L+D))$ | $\mathcal{O}(ND)$
XTI | token-wise | black-box | $\mathcal{O}(N(D^{2}(N_{t+v})+N_{t+v}^{2}D)L)$ | $\mathcal{O}(NN_{v}D)$
TI | token-wise | parameter-free | $\mathcal{O}(NN_{t}N_{v}D)$ | $\mathcal{O}(NN_{v}D)$
WTI | token-wise | light-parameter | $\mathcal{O}(N(N_{t}N_{v}D+N_{t+v}))$ | $\mathcal{O}(NN_{v}(D+1))$

To facilitate our study, we develop a generalized interaction formulation that
is able to represent various module designs and show a comparison in Table 1.
We then show how the existing interaction mechanisms can be represented within
this formulation, and how ablations can be conducted using this formulation
with respect to different interaction module elements.

Single Vector Dot-product Interaction: When measuring feature similarity,
researchers use simple dot-product operation, which is an intuitive solution.
Specifically, two global representations are compared in the $\ell_{2}$
normalized embedding space:

$\operatorname{DP}(\mathbf{e}_{t},\mathbf{e}_{v})=\frac{(\mathbf{e}_{t}^{\text{[CLS]}})^{\top}\cdot\bar{\mathbf{e}}_{v}}{\|\mathbf{e}_{s}^{\text{[CLS]}}\|_{2}\cdot\|\bar{\mathbf{e}}_{v}\|_{2}},$
(1)

where $\mathbf{e}_{t}^{\text{[CLS]}}$ is the text feature for [CLS] token and
$\bar{\mathbf{e}}_{v}=\frac{1}{N_{v}}\sum_{i=1}^{N_{V}}\mathbf{e}_{v}^{i}$ is
the average video representation. The dot-product method is widely used in
biometrics [7, 46], text-image retrieval [47], and text document retrieval
[15]. This interaction method is extremely efficient and can be accelerated by
the ANN library, e.g., FAISS [17]. While, due to the single-vector
representation, the sequential structure is heavily coupled in encoder.
Compressing the long sequence into a single vector will push the network to
learn extremely complex abstractions and may easily miss out fine-grained
clues. The dot-product function also encourages smooth latent spaces, which
will discourage the hierarchical semantics learning.

Hierarchical Interaction: Similar to the global feature representation, multi-
layer [25], multi-modal [12] and multi-experts [26] use the gated fusion
mechanism to ensemble the hierarchical representation:

$\operatorname{HI}(\mathbf{e}_{t},\mathbf{e}_{v})=\sum_{s=1}^{S}w(\mathbf{e}_{t,s}^{\text{[CLS]},s})^{\top}\bar{\mathbf{e}}_{v,s},$
(2)

where $w_{s}$ can be a normalized weight or a binarized gated unit. We note
that this hierarchical interaction actually contains two essential factors:
one is the retrieval-oriented feature pool, and the other is the dynamic
fusion block. The content of the comparison has been greatly enriched from
expert models. While the learning of the experts requires labor intensive
manual annotation.

MLP on Global Vector: In order to improve the nonlinear measurement ability of
the interaction block, researchers [41, 29] propose to use neural network to
learn the metric, among which the representative work [41] builds a Multilayer
Perceptron (MLP) measurement. They directly concatenate the compared features
as a whole, and optimize the similarity through MLP:

$\operatorname{MLP}(\mathbf{e}_{t},\mathbf{e}_{v})=f_{\theta}([\mathbf{e}_{t}^{\text{[CLS]}},\bar{\mathbf{e}}_{v}]).$
(3)

where $[,]$ denotes concatenation operation. Although the neural network
brings a nonlinear metric space, it also operates in a black-box setting. The
pure-parameterizative method needs to consume massive amount of labelled data
in the training process. And when deployed on large-scale (billions)
documents, even two fully connected layers will bring prohibitively
computational overhead.

Cross Transformer Interaction: The standard self-attention [40] is adopted for
cross-modality matching, which can handle variable-length inputs:

$\operatorname{XTI}(\mathbf{e}_{t},\mathbf{e}_{v})=\text{MHA}_{\theta}([\mathbf{e}_{t},\mathbf{e}_{v}]).$
(4)

The Cross Transformer aims to capture and model the inter-modal relations
between texts and videos by exchanging key-value pairs in the multi-headed
attention (MHA) mechanism. In Section 4.2, we observe that the cross
transformer interaction is difficult to optimize and usually occurs
performance degradation on video retrieval datasets. Furthermore, compared
with the dot-product operation, multi-head attention brings thousand times of
the computational overhead, as shown in Table 1 and Table 2.

Token-wise Interaction: Recently, ColBERT [18] and FILIP [45] propose token-
wise interaction for document and image retrieval. We introduce this token-
wise interaction to TVR, which cleverly solves the local context matching
problem (Assumption (1) in Figure 1):

$\operatorname{TI}(\mathbf{e}_{t},\mathbf{e}_{v})=\left(\sum_{i=1}^{N_{t}}\max_{j=1}^{N_{v}}(\tilde{\mathbf{e}}_{t}^{i})^{\top}\tilde{\mathbf{e}}_{v}^{j}+\sum_{j=1}^{N_{v}}\max_{i=1}^{N_{t}}(\tilde{\mathbf{e}}_{t}^{i})^{\top}\tilde{\mathbf{e}}_{v}^{j}\right)/2,$
(5)

where $\tilde{\mathbf{e}}=\mathbf{e}^{i}/\|\mathbf{e}^{i}\|_{2}$ is the
channel-wise normalization operation. The token-wise interaction is a
parameter-free operation and allows efficient storing and indexing [18].

Weighted Token-wise Interaction: Intuitively, not all words and video frames
contribute equally. We provide an adaptive approach to adjust the weight
magnitude for each token:

$\operatorname{WTI}(\mathbf{e}_{t},\mathbf{e}_{v})=\left(\sum_{i=1}^{N_{t}}f_{tw,\theta}^{i}(\mathbf{e}_{t})\max_{j=1}^{N_{v}}(\tilde{\mathbf{e}}_{t}^{i})^{\top}\mathbf{e}_{v}^{j}+\sum_{j=1}^{N_{v}}f_{vw,\theta}^{j}(\mathbf{e}_{v})\max_{i=1}^{N_{t}}(\tilde{\mathbf{e}}_{t}^{i})^{\top}\tilde{\mathbf{e}}_{v}^{j}\right)/2,$
(6)

where $f_{tw,\theta}$ and $f_{vw,\theta}$ are composed of classic MLP and a
SoftMax function. The adaptive block is lightweight and takes a single-
modality input, allowing offline pre-computation for large-scale video
documents. In the online process, the attention module introduces negligible
computational overhead, as shown in Table 2. Our method inherits the general
matching priors and is empirically verified to be more effective and
efficient.

WTI can be easily implemented by a few lines of code in modern libraries.
Algorithm 1 shows a simplified code based on PyTorch [32].

Algorithm 1 PyTorch-style pseudocode for Weighted Token-wise Interaction.

⬇

# t: text input v: video input

# f_t: text encoder network f_v: video encoder network

# f_tw: text weight network f_vw: video weight network

# B: batch size D: dimensionality of the embeddings

# N_t: text token size N_v: frame token size

def weighted_token_wise_interaction(t, v):

# compute embeddings

e_t = f_t(t) # BxN_txD

e_v = f_v(v) # BxN_vxD

# generate fusion weights

text_weight = torch.softmax(f_tw(e_t), dim=-1) # BxN_t

video_weight = torch.softmax(f_tv(e_v), dim=-1) # BxN_v

# normalize representation

e_t = e_t / e_t.norm(dim=-1, keepdim=True) # BxN_txD

e_v = e_v / e_v.norm(dim=-1, keepdim=True) # BxN_vxD

# token interaction

logits = torch.einsum("atc,bvc->abtv", [e_t, e_v]) # BxBxN_txN_v

t2v_logits = logits.max(dim=-1)[0] # BxBxN_txN_v -> BxBxN_t

t2v_logits = torch.einsum("abt,at->ab", [t2v_logits, text_weight]) # BxBxN_t
-> BxB

v2t_logits = logits.max(dim=-2)[0] # BxBxN_v

v2t_logits = torch.einsum("abv,bv->ab", [v2t_logits, video_weight]) #
BxBxN_v-> BxB

retrieval_logits = (t2v_logits + v2t_logits) / 2.0 # BxB

return retrieval_logits

###  3.3 Channel Decorrelation Regularization

Given a batch of $B$ video-text pairs, WTI generates an $B\times B$ similarity
matrix. The Text-Video Retrieval is trained in a supervised way. We employ the
InfoNCE loss [31] to maximize the similarity between labelled video-text pairs
and minimize the similarity for other pairs:

$\begin{gathered}\mathcal{L}_{\text{InfoNCE}}=\mathcal{L}_{v2t}+\mathcal{L}_{t2v}\\\
\mathcal{L}_{v2t}=-\frac{1}{B}\sum_{i}^{B}\log\frac{\exp\left(\operatorname{WTI}\left(\mathbf{e}_{t,i},\mathbf{e}_{v,i}\right)/\tau\right)}{\sum_{j}^{B}\exp\left(\operatorname{WTI}\left(\mathbf{e}_{t,i},\mathbf{e}_{v,j}\right)/\tau\right)}\\\
\mathcal{L}_{t2v}=-\frac{1}{B}\sum_{i}^{B}\log\frac{\exp\left(\operatorname{WTI}\left(\mathbf{e}_{t,i},\mathbf{e}_{v,i}\right)/\tau\right)}{\sum_{j}^{B}\exp\left(\operatorname{WTI}\left(\mathbf{e}_{t,j},\mathbf{e}_{v,i}\right)/\tau\right)},\end{gathered}$
(7)

where $\tau$ is temperature hyper-parameter.

The contrastive loss provides a macro objective to optimize the global
similarity. Referring expression comprehension, the multi-modality retrieval
often requires semantic information from micro-views, e.g., the channel-level.
Inspired from self-supervised learning methods [49], we utilize the covariance
matrix to measure the redundancy between features and employ a simple
$\ell_{2}$-norm minimization to optimize the hierarchical representation:

$\begin{gathered}\mathcal{L}_{\text{CDCR}}=\sum_{i}\left(1-\mathcal{C}^{ii}\right)^{2}+\alpha\sum_{i}\sum_{j\neq
i}(\mathcal{C}^{ij})^{2}\\\
\mathcal{C}^{ij}\triangleq\frac{\sum_{b}\mathbf{e}_{t,b}^{(i)}\mathbf{e}_{v,b}^{(j)}}{\sqrt{\sum_{b}\left(\mathbf{e}_{t,b}^{(i)}\right)^{2}}\sqrt{\sum_{b}\left(\mathbf{e}_{v,b}^{(j)}\right)^{2}}},\end{gathered}$
(8)

where $\mathbf{e}_{t,b}^{(i)}$ is the $i$-th channel of $b$-th text feature
$\mathbf{e}_{t,b}$. The coefficient $\alpha$ controls the magnitude of the
redundancy term. The total training loss $\mathcal{L}_{\text{all}}$ is defined
as:

$\begin{gathered}\mathcal{L}_{\text{all}}=\mathcal{L}_{\text{InfoNCE}}+\lambda\mathcal{L}_{\text{CDCR}},\end{gathered}$
(9)

where $\lambda$ is the weighting parameter. Surprisingly, empirical results
show that the cross-modality channel decorrelation regularization brings
significant improvements for all interaction mechanisms, as shown in Table 2.

##  4 Experiments

In this section, we firstly present the configuration details of our algorithm
and then conduct thorough ablation experiments to explore the relative
importance of each component in our method on MSR-VTT [44]. Finally we conduct
comprehensive experiments on six benchmarks: MSR-VTT [44], MSVD [43], VATEX
[42], LSMDC [39], ActivityNet [10] and DiDeMo [1].

###  4.1 Experimental Settings

Dataset: We conduct experiments on six benchmarks for video-text retrieval
tasks including:

  * •

MSR-VTT [44] contains 10,000 videos with 20 captions for each. We report
results on 1k-A which adopts 9,000 videos with all corresponding captions for
training and utilizes 1,000 video-text pairs as test.

  * •

MSVD [43] includes 1,970 videos with 80,000 captions. We report results on
split set, where train, validation and test are 1200, 100 and 670 videos.

  * •

VATEX [42] contains 34,991 videos with multilingual annotations. The training
split contains 25,991 videos. We report the results on the split set includes
1500 videos for validation and 1500 videos for test.

  * •

LSMDC [39] contains 118081 videos and equal captions extracted from 202 movies
with a split of 109673, 7408, and 1000 as the train, validation, and test set.
Every video is selected from movies ranging from 2 to 30 seconds.

  * •

ActivityNet [10] consists of 20,000 YouTube videos. We concatenate all
descriptions of a video to a single query and evaluate on the ‘val1’ split.

  * •

DiDeMo [1] contains 10,000 videos annotated with 40,000 sentences. All
sentence descriptions for a video are also concatenated into a single query
for text-video retrieval.

Evaluation Metric: We follow the standard retrieval task [29] and adopt Recall
at rank K (R@K), median rank (MdR) and mean rank (MnR) as metrics. Higher R@K
and lower MdR or MnR indicates better performance.

Implementation Details: We utilize the standard Bi-Encoder from CLIP [35] as
pre-trained feature extractor. Concretely, the vision encoder is comprised of
a vanilla ViT-B/32 [9] and 4-layers of temporal transformer blocks. Each block
employs 8 heads and 512 hidden channels. The temporal position embedding and
network weight parameters are initialized from the CLIP’s text encoder. The
fixed video length and caption lengths are 12 and 32 for MSR-VTT, MSVD, VATEX,
LSMDC and 64 and 64 for ActivityNet and DiDeMo. Following [35], the special
tokens, [CLS] and [SEP], and the text tokens are concatenated as inputs to a
12-layer linguistic transformer [8, 35]. We leverage the MLP to capture the
weight factor for tokens, and our ablation studies suggest a 2-layer
structure. The adaptive module is initialized from scratch with random
weights.

Training Schedule: For fair comparisons with our baseline, we follow training
schedules from CLIP4Clip [29]. The network is optimized by Adam [20] with a
batch size of 128 in 5 epochs. The initial learning rate for vision encoder
and text encoder are set $1\times 10^{-7}$, and the initial learning rate for
the temporal transformer and the adaptive module are set to $1\times 10^{-4}$.
All learning rates follow the cosine learning rate schedule with a linear
warmup. We apply a weight decay regularization to all parameters except for
bias, layer normalization, token embedding, positional embedding and
temperature. During training, we set the temperature $\tau=100$, CDCR weight
$\alpha=0.06$ and overall weighting parameter $\lambda=0.001$.

###  4.2 Ablation Study

Table 2: Ablation of Disentangled Representation on the 1K validation set of MSR-VTT [44]. Time shows inference speed for indexing 1 million video documents on a Tesla V100 GPU. Interaction | InfoNCE [31] | +CDCL |  Time (ms)
---|---|---|---
R@1$\uparrow$ | R@5$\uparrow$ | R@10$\uparrow$ | MnR$\downarrow$ | R@1$\uparrow$ | R@5$\uparrow$ | R@10$\uparrow$ | MnR$\downarrow$
DP | 42.8 | 72.1 | 81.4 | 16.3 | 44.2 | 72.7 | 82.0 | 14.5 | 415
HI | 43.5 | 72.9 | 81.7 | 16.1 | 44.1 | 72.6 | 82.8 | 14.1 | 531
MLP | 29.3 | 54.8 | 64.2 | 33.8 | - | - | - | - | 25,304
XTI | 41.8 | 71.2 | 82.7 | 16.2 | - | - | - | - | 80,453
TI | 44.8 | 73.7 | 82.9 | 13.5 | 45.5 | 72.0 | 82.5 | 13.3 | 536
WTI | 46.3 | 73.7 | 83.2 | 13.0 | 47.4 | 74.6 | 83.8 | 12.8 | 565
DP${}_{\text{+ViT-B/16}}$ | 45.9 | 73.8 | 82.3 | 13.8 | 46.6 | 73.3 | 82.8 | 13.4 | 415
TI${}_{\text{+ViT-B/16}}$ | 47.3 | 76.7 | 84.8 | 13.7 | 49.1 | 75.7 | 85.1 | 12.7 | 536
WTI${}_{\text{+ViT-B/16}}$ | 48.8 | 76.1 | 84.3 | 13.5 | 50.2 | 76.5 | 84.7 | 12.4 | 565

We quantitatively evaluate the key components, Weighted Token-wise Interaction
(WTI) and Channel DeCorrelation Regularization (CDCR), on MSR-VTT 1K [44].
Table 2 summarizes the ablation results.

Effect of Weighted Token-wise Interaction: Compared with the Single-Vector
Dot-Product Interaction (DP), the Token-wise Interaction (TI) only adds a
dense correlation, while the performance boosts $+2.0\%$ R@1. Our Weighted
Token-wise Interaction (WTI) dramatically improves the performance by an R@1
of $3.5\%$, which shows that the proposed interaction structure indeed helps
the model to adequately exploit pair-wise correlations. Compared with other
light-parameter function, our method improves by $+2.8\%$ R@1 over
Hierarchical Interaction (HI). An explanation is that the token representation
is more valuable than the multi-level features for the single-scale ViT [9].
As observed in [29], the MLP interaction (MLP) and Cross Transformer
Interaction (XTI), under black-box function, would lead to degradation
problems. Further, by adopting ViT-B/16 [35], WTI greatly promotes the T2V R@1
to $48.8\%$.

Effect of Channel DeCorrelation Regularization: In Table 2, we evaluate the
importance of Channel Decorrelation Regularization on different interaction
structures except block-box function. Surprisingly, CDCR increases performance
significantly for all interactions, i.e. by $+1.4\%$, $+0.6\%$, $+0.7\%$ and
$+1.1\%$ for DP, HI, TI and WTI respectively. The micro channel-level
regularization makes it easier to leverage semantic hierarchy.

Table 3: Effect of dual-path and layers for weight model on MSR-VTT 1K [44].  Dual | t2v | v2t | t2v+v2t
---|---|---|---
R@1$\uparrow$ | R@5$\uparrow$ | R@10$\uparrow$ | MnR$\downarrow$ | R@1$\uparrow$ | R@5$\uparrow$ | R@10$\uparrow$ | MnR$\downarrow$ | R@1$\uparrow$ | R@5$\uparrow$ | R@10$\uparrow$ | MnR$\downarrow$
TI | 43.4 | 71.0 | 80.2 | 16.1 | 43.5 | 71.8 | 82.7 | 14.0 | 44.8 | 73.7 | 82.9 | 13.5
WTI | 45.4 | 72.3 | 81.7 | 13.4 | 45.3 | 74.6 | 83.3 | 13.6 | 46.3 | 73.7 | 83.2 | 13.0
Layers | 1FC | 2FC | 3FC
WTI | 46.3 | 73.9 | 82.9 | 13.7 | 46.3 | 73.7 | 83.2 | 13.0 | 45.7 | 72.9 | 81.4 | 14.0

Effect of Dual-path Token-wise Retrieval: In Table 3, we test the importance
of dual-path interaction of t2v and v2t, subset of Eq. 5. By adopting dual-
path to compute the logits, our model provides stable gains by $+1.4\%$ and
$+1.0\%$ for TI and WTI, respectively.

Effect of MLP layers: By default, we simply set the weighted function with
MLP, which is composed of FC+ReLU blocks. In Table 3, we observe that R@1 is
slightly better when applying 2FC layers and drop by 3FC, likely because a
heavier structure suffers from over-fitting.

Figure 4: Hyper-parameter for feature decorrelation. (a) $\alpha$ for
$\mathcal{L}_{\text{CDCR}}$ (b) $\lambda$ for $\mathcal{L}_{\text{all}}$.

Figure 5: The training loss comparisons (a) and the influence of frame number
(b).

Hyper-parameter Selection for DeCorrelation: The parameter $\alpha$ and
$\lambda$ specifies the scale and importance of the Channel DeCorrelation
Loss. We evaluate the scale range setting $\alpha\in[0,0.1]$ and
$\lambda\in[0,0.002]$ as shown in Figure 4. We find that R@1 is improved from
$44.8\%$ to $45.5\%$ when $\alpha=0.06$ and saturated with $\alpha=0.07$. We
can get the best R@1 when the $\lambda$ is around 0.001. As a result, we adopt
$\alpha=0.06$ and $\lambda=0.001$ in our model to achieve the best
performance.

Training Convergence Analysis: To further reveal the source of the performance
improvement, we present the training loss in Figure 5. The training of pure-
parameterizative interaction (Cross Transformer) is not trivial since it lacks
the basic inductive biases for distance measurement. Inspired by this
observation, we seek a light-weight way to leverage the heuristic property for
better convergence, and therefore achieve a faster convergence rate.

Benefit for More Frames: Figure 5(b) shows performance for DP and WTI with
different frames. Compared with the naïve baseline DP, our WTI can achieve
more significant gains, $+4.9\%$ v.s. $+2.8\%$, with more frames. For fair
comparisons with CLIP4Clip [29], we mainly report results at 12 frames.

Inference Speed: Besides recall metric, we report the inference speed of all
the interaction structure in Table 2. Our lightweight interaction introduces
negligible computational overhead, bringing remarkable improvements. Our
approach reduce the inference time by several orders of magnitude, compared
with the heavy cross transformer.

###  4.3 Comparison with State-of-the-Arts

Table 4: Retrieval results on the validation set of MSR-VTT 1K [44].  Method |  Text$\rightarrow$ Video |  Video $\rightarrow$Text
---|---|---
R@1$\uparrow$ |  R@5$\uparrow$ |  R@10$\uparrow$ |  MdR$\downarrow$ |  R@1$\uparrow$ |  R@5$\uparrow$ |  R@10$\uparrow$ |  MdR$\downarrow$
HERO [23] | 16.8 | 43.4 | 57.7 | - | - | - | - | -
UniVL [28] | 21.2 | 49.6 | 63.1 | 6.0 | - | - | - | -
ClipBERT [21] | 22.0 | 46.8 | 59.9 | 6.0 | - | - | - | -
MDMMT [12] | 26.6 | 57.1 | 69.6 | 4.0 | 27.0 | 57.5 | 69.7 | 3.7
SUPPORT [33] | 27.4 | 56.3 | 67.7 | 3.0 | 26.6 | 55.1 | 67.5 | 3.0
FROZEN [2] | 31.0 | 59.5 | 70.5 | 3.0 | - | - | - | -
CLIP4Clip [29] | 44.5 | 71.4 | 81.6 | 2.0 | 42.7 | 70.9 | 80.6 | 2.0
Ours | 47.4 | 74.6 | 83.8 | 2.0 | 45.3 | 73.9 | 83.3 | 2.0
+ViT-B/16 | 50.2 | 76.5 | 84.7 | 1.0 | 48.9 | 76.3 | 85.4 | 2.0
+QB-Norm [3] | 53.3 | 80.3 | 87.6 | 1.0 | 56.2 | 79.9 | 87.4 | 1.0

In this subsection, we compare the proposed model with recent state-of-the-art
methods on the six benchmarks, MSR-VTT [44], MSVD [43], VATEX [42], LSMDC
[39], ActivityNet [10] and DiDeMo [1].

For MSR-VTT in Table 4, our model significantly surpasses the ClipBERT [21] by
absolute $25.5\%$ R@1, reaching $47.4\%$ R@1 on MSR-VTT, indicating the
benefits and necessity of large scale image-text pre-training for video-text
retrieval. We achieve $2.9\%$ improvement compared to CLIP4Clip [29], which
shows the benefit from token-wise interaction. Our WTI, employing ViT-B/16 and
QB-Norm [3], yields a remarkable T2V R@1 $53.3\%$.

Table 5 shows results for other benchmarks. For MSVD, our model also
demonstrates competitive performance with top-performing CLIP based model
[29]. For VATEX, our approach achieves $+7.6\%$ R@1 improvement for text-video
retrieval and $+3.8\%$ improvement for video-text retrieval. For LSMDC, our
approach achieves $+2.3\%$ R@1 improvements for text-video retrieval.

For ActivityNet, we outperform the state-of-the-art method by a large margin
of $+2.8\%$ R@1 on text-video retrieval. For DiDeMo, we achieve remarkable
performance $47.9\%$ R@1 and a relative performance improvement of $16.5\%$ in
T2V compared to CLIP4Clip. It is worth noting that long sentences are used in
ActivityNet and DiDeMo. The significant improvements on both datasets further
proves the advantages of our disentangled representation.

Overall, the consistent improvements across different benchmarks strongly
demonstrate the effectiveness of our algorithm. We hope that our studies will
spur further investigation in modeling interaction.

##  5 Conclusion

In this work, we present an empirical study to give a better general
understanding of interaction mechanisms for Text-Video Retrieval. Then we
propose a novel

Table 5: Retrieval results on the validation set of MSVD [43], VATEX [42], LSMDC [39], ActivityNet [10] and DiDeMo [1].  |  Text$\rightarrow$ Video |  Video $\rightarrow$Text
---|---|---
Method |  R@1$\uparrow$ |  R@5$\uparrow$ |  R@10$\uparrow$ |  MdR$\downarrow$ |  R@1$\uparrow$ |  R@5$\uparrow$ |  R@10$\uparrow$ |  MdR$\downarrow$
Retrieval performance on MSVD [43]
CE [26] | 19.8 | 49.0 | 63.8 | 6.0 | - | - | - | -
SUPPORT [33] | 28.4 | 60.0 | 72.9 | 4.0 | - | - | - | -
FROZEN[2] | 33.7 | 64.7 | 76.3 | 3.0 | - | - | - | -
CLIP [35] | 37.0 | 64.1 | 73.8 | 3.0 | 54.9 | 82.9 | 89.6 | 1.0
CLIP4Clip[29] | 45.2 | 75.5 | 84.3 | 2.0 | 48.4 | 70.3 | 77.2 | 2.0
Ours | 48.3 | 79.1 | 87.3 | 2.0 | 62.3 | 86.3 | 92.2 | 1.0
Ours${}_{\text{+ViT-B/16}}$ | 50.0 | 81.5 | 89.5 | 2.0 | 68.7 | 92.5 | 95.6 | 1.0
Retrieval performance on VATEX [42]
CLIP [35] | 39.7 | 72.3 | 82.2 | 2.0 | 52.7 | 88.8 | 94.9 | 1.0
SUPPORT [33] | 44.9 | 82.1 | 89.7 | 1.0 | 58.4 | 84.4 | 91.0 | 1.0
CLIP4Clip [29] | 55.9 | 89.2 | 95.0 | 1.0 | 73.2 | 97.1 | 99.1 | 1.0
Ours | 63.5 | 91.7 | 96.5 | 1.0 | 77.0 | 98.0 | 99.4 | 1.0
Ours${}_{\text{+ViT-B/16}}$ | 65.7 | 92.6 | 96.7 | 1.0 | 80.1 | 98.5 | 99.5 | 1.0
Retrieval performance on LSMDC [39]
CE [26] | 11.2 | 26.9 | 34.8 | 25.3 | - | - | - | -
MMT [12] | 12.9 | 29.9 | 40.1 | 19.3 | - | - | - | -
CLIP [35] | 11.3 | 22.7 | 29.2 | 46.5 | - | - | - | -
MDMMT [12] | 18.8 | 38.5 | 47.9 | 12.3 | - | - | - | -
CLIP4Clip [29] | 22.6 | 41.0 | 49.1 | 11.0 | - | - | - | -
Ours | 24.9 | 45.7 | 55.3 | 7.0 | 24.9 | 44.1 | 53.8 | 9.0
Ours${}_{\text{+ViT-B/16}}$ | 26.5 | 47.6 | 56.8 | 7.0 | 27.0 | 45.7 | 55.4 | 8.0
Retrieval performance on ActivityNet [10]
CE [26] | 17.7 | 46.6 | - | 6.0 | - | - | - | -
MMT [12] | 28.9 | 61.1 | - | 4.0 | - | - | - | -
SUPPORT[33] | 28.7 | 60.8 | - | 2.0 | - | - | - | -
CLIP4Clip [29] | 41.4 | 73.7 | 85.3 | 2.0 | - | - | - |
Ours | 44.2 | 74.5 | 86.1 | 2.0 | 42.2 | 74.0 | 86.2 | 2.0
Ours${}_{\text{+ViT-B/16}}$ | 46.2 | 77.3 | 88.2 | 2.0 | 45.7 | 76.5 | 87.8 | 2.0
Retrieval performance on DiDeMo [1]
CE [26] | 15.6 | 40.9 | - | 8.2 | 27.2 | 51.7 | 62.6 | 5.0
ClipBERT [21] | 21.1 | 47.3 | 61.1 | 6.3 | - | - | - | -
FROZEN [2] | 31.0 | 59.8 | 72.4 | 3.0 | - | - | - | -
CLIP4Clip [29] | 41.4 | 68.2 | 79.1 | 2.0 | 42.8 | 69.8 | 79.0 | 2.0
Ours | 47.9 | 73.8 | 82.7 | 2.0 | 45.4 | 72.6 | 82.1 | 2.0
Ours${}_{\text{+ViT-B/16}}$ | 49.0 | 76.5 | 84.5 | 2.0 | 49.9 | 75.4 | 83.3 | 2.0

disentangled representation method, which are collaboratively implemented by a
Weighted Token-wise Interaction (WTI) to solve sequential matching problem
from macro-view and a Channel DeCorrelation Regularization (CDCR) that reduces
feature redundancy from a micro-view. We demonstrate the effectiveness of the
proposed approach for modeling better sequential and hierarchical clues on six
datasets. We wish our cross-modality interaction will inspire more theoretical
researches towards more powerful interaction design.

Appendix

This document brings additional details of Weighted Token-wise Interaction.
Additional qualitative and quantitative results are also given for
completeness. Figure 6 shows the internal operating mechanism of WTI, and also
powerfully explains the performance improvement of our algorithm.

Figure 6: Visualization of the internal operating mechanism of WTI on MSR-VTT
[44]. Figure 7: The improvement with different scenes number on the MSR-VTT 1K
val [44]. Figure 8: The word cloud of top-1 weighted ordinary tokens on the
MSR-VTT 1K val [44].

##  6 Discussion

How Weighted Token-wise Interaction (WTI) improve performance? In this
section, we discuss the reasons for considering token-wise interaction and the
adaptive weight module.

First, the text and video input are naturally sequential, and keeping the
sequential structure can reduce the information loss and preserve more fine-
grained cues. Using the sequential representation for interaction is
essentially an ensemble learning method. The Max operation in our token-wise
interaction can select matching video clips, as shown in Fig. 6, which can
help us resolve local context assumptions in the main paper. Otherwise, if
using average pooling [29], the overall similarity will be pulled down by
irrelevant segments. Quantitatively, we analyze the relationship between
recall improvement and the number of scenes in a video (Fig. 7), and find that
our algorithm achieves most improvements at 3 scenes. Therefore, token-wise
interaction is intuitively a sensible practice.

Second, learning of adaptive weights is necessary, especially when CLIP [35]
is used as a pre-trained network. CLIP performs large-scale training (on 256
GPUs for 2 weeks) with large-scale dataset (400 millions), and we cannot
completely change the distribution of position embedding and features by a
small-scale fine-tuning dataset. The default global token, [SEP], shows a
dominant role, achieving top1 important token for $98.7\%$ videos and an
average weight of 0.2838, far exceeding the average weight of other tokens
(0.0674) on MSR-VTT 1K validation set. Simply averaging all sequential
features is unfair to [SEP], resulting in a suboptimal solution. On the other
hand, the importance of words in the text are also quite different, and our
experimental analysis shows that nouns/verbs account for the main weight, as
shown in Fig. 8.

Therefore, a good practice is to consider the token-wise interaction and
adaptive weight module and devising a solution to effectively fuse these two
factors (as WTI).

##  7 More Implementation Details

In this section, we present more implementation details that were omitted in
the main paper for brevity.

###  7.1 Pseudocode with Mask Inputs

Please note we omit the mask inputs at the main paper for brevity in Section
3.2. Here we present complete pseudo-codes for our Weighted Token-wise
Interaction (WTI) and Channel Decorrelation Regularization (CDCR) in Algorithm
2\- 4. We believe the pseudocode would aid an independent researcher to better
replicate the proposed interaction.

Algorithm 2 PyTorch-style pseudocode for Weighted Token-wise Interaction.

⬇

# t: text input v: video input

# mask_t: text mask mask_v: video mask

# f_t: text encoder network f_v: video encoder network

# f_tw: text weight network f_vw: video weight network

# B: batch size D: dimensionality of the embeddings

# N_t: text token size N_v: frame token size

def weighted_token_wise_interaction(t, mask_t, v, mask_v):

# compute embeddings

e_t = f_t(t) # BxN_txD

e_v = f_v(v) # BxN_vxD

# generate fusion weights

text_weight = f_tw(e_t).squeeze() # BxN_t

# fill masked text with -inf

text_weight.masked_fill_(1 - mask_t, float("-inf")) # BxN_v

text_weight = torch.softmax(text_weight, dim=-1) # BxN_t

video_weight = f_tv(e_v).squeeze() # BxN_v

# fill masked video with -inf

vision_weight.masked_fill_(1 - mask_v, float("-inf")) # BxN_v

video_weight = torch.softmax(vision_weight, dim=-1) # BxN_v

# normalize representation

e_t = e_t / e_t.norm(dim=-1, keepdim=True) # BxN_txD

e_v = e_v / e_v.norm(dim=-1, keepdim=True) # BxN_vxD

# token interaction

logits = torch.einsum("atc,bvc->abtv", [e_t, e_v]) # BxBxN_txN_v

# mask for logits

logits = torch.einsum(’abtv,at->abtv’, [logits, mask_t])

logits = torch.einsum(’abtv,bv->abtv’, [logits, mask_v])

t2v_logits, t2v_max_idx = logits.max(dim=-1) # BxBxN_txN_v -> BxBxN_t

t2v_logits = torch.einsum("abt,at->ab", [t2v_logits, text_weight]) # BxBxN_t
-> BxB

v2t_logits, v2t_max_idx = logits.max(dim=-2) # BxBxN_txN_v -> BxBxN_v

v2t_logits = torch.einsum("abv,bv->ab", [v2t_logits, video_weight]) #
BxBxN_v-> BxB

retrieval_logits = (t2v_logits + v2t_logits) / 2.0 # BxB

return retrieval_logits

Algorithm 3 PyTorch-style pseudocode for Channel DeCorrelation Regularization
with single-vector representation.

⬇

# e_t: text feature e_v: video feature

# B: batch size D: dimensionality of the embeddings

# alpha: the magnitude of the redundancy term

def channel_decorrelation_regularization_single(e_t, e_v):

# batch norm

t_norm = (e_t - e_t.mean(0)) / e_t.std(0) # NxD

v_norm = (e_v - e_v.mean(0)) / e_v.std(0) # NxD

B, D = t_norm.shape

cov = torch.einsum(’ac,ad->cd’, t_norm, v_norm) / B # DxD

# loss

on_diag = torch.diagonal(cov).add_(-1).pow_(2).sum()

off_diag = cov.flatten()[1:].view(D - 1, D + 1)[:, :-1].pow_(2).sum()

cdcr = on_diag + off_diag * alpha

return cdcr

Algorithm 4 PyTorch-style pseudocode for Channel DeCorrelation Regularization
with sequential representation.

⬇

# e_t: text feature e_v: video feature

# mask_t: text mask mask_v: video mask

# B: batch size D: dimensionality of the embeddings

# N_t: text token size N_v: frame token size

def channel_decorrelation_regularization_sequential(e_t, mask_t, e_v, mask_v):

# selecet max indexs for each text-video pair

i4t = t2v_max_idx[torch.arange(B),torch.arange(B)] # BxBxN_t -> BxN_t

i4v = v2t_max_idx[torch.arange(B), torch.arange(B)] # BxBxN_v -> BxN_v

e_4_t = e_v[torch.arange(B).repeat_interleave(N_t), i4t.flatten()] # (BxN_t)xD

e_4_v = e_t[torch.arange(B).repeat_interleave(N_v), i4v.flatten()] # (BxN_v)xD

e_t = e_t.reshape(-1,D) # (BxN_t)xD

e_v = e_v.reshape(-1,D) # (BxN_v)xD

mask_t = mask_t.flatten().type(torch.bool)

mask_v = mask_v.flatten().type(torch.bool)

e_t = e_t[mask_t]

e_v = e_v[mask_v]

e_4_t = e_4_t[mask_t]

e_4_v = e_4_v[mask_v]

# cov for t2v

t_norm = (e_t - e_t.mean(0)) / e_t.std(0) # XxD

v_norm = (e_4_t - e_4_t.mean(0)) / e_4_t.std(0) # XxD

X = t_norm.shape

cov1 = torch.einsum(’ac,ad->cd’, t_norm, v_norm) / B # DxD

# cov for v2t

v_norm = (e_t - e_t.mean(0)) / e_t.std(0) # XxD

t_norm = (e_4_v - e_4_v.mean(0)) / e_4_v.std(0) # XxD

X= t_norm.shape[0]

cov2 = torch.einsum(’ac,ad->cd’, t_norm, v_norm) / B # DxD

cov = (cov1+cov2)/2

# loss

on_diag = torch.diagonal(cov).add_(-1).pow_(2).sum()

off_diag = cov.flatten()[1:].view(D - 1, D + 1)[:, :-1].pow_(2).sum()

cdcr = on_diag + off_diag * alpha

return cdcr

###  7.2 Network Architecture for Weight Branch

Figure 6 shows details of the network architectures for learning weights. We
use the single-modal input for each branch to avoid coupling between
modalities.

Figure 9: Network architectures for weight branch.

###  7.3 Complexity Analysis

Table 6: Comparisons of different interaction mechanisms. $N$ denotes number of video documents ($N$ is large and depends on applications); $D$ denotes representation dimension ($D=512$, by default); $N_{t}$ and $N_{v}$ denote length of text token and frame token, $N_{v+t}$ is the sum of $N_{t}$ and $N_{v}$ ($N_{t}=32,N_{v}=12$); $L$ denotes number of network layer and $S$ denotes feature levels ($L=4,S=3$, by default). Interaction | Contents | Function | Computational Complexity | Memory
---|---|---|---|---
DP | single | parameter-free | $\mathcal{O}(ND)$ | $\mathcal{O}(ND)$
HI | multi-level | light-parameter | $\mathcal{O}(NSD)$ | $\mathcal{O}(NSD)$
MLP | single | black-box | $\mathcal{O}(N(D^{2}L+D))$ | $\mathcal{O}(ND)$
XTI | token-wise | black-box | $\mathcal{O}(N(D^{2}N_{t+v}+N_{t+v}^{2}D)L)$ | $\mathcal{O}(NN_{v}D)$
TI | token-wise | parameter-free | $\mathcal{O}(NN_{t}N_{v}D)$ | $\mathcal{O}(NN_{v}D)$
WTI | token-wise | light-parameter | $\mathcal{O}(N(N_{t}N_{v}D+N_{t+v}))$ | $\mathcal{O}(NN_{v}(D+1))$

Single Vector Dot-product Interaction: For each video, we only need to store a
$D$-dimensional feature, and the the overall feature storage complexity is
$\mathcal{O}(ND)$. During the query process, a similarity calculation consists
of $D$ multiplication operations. The overall computational complexity is
$\mathcal{O}(ND)$ .

Hierarchical Interaction: For the hierarchical interaction, we consider the
simplest case, directly using the constant weighted average to fuse the multi-
layer features. Therefore, both computation and storage are increased by a
factor of $S$ compared to DP.

MLP on Global Vector: We adopt the structure of stacking $L\times$ FC+ReLU
blocks, where the input channel of the first layer is $2D$, the hidden
channels size is $D$, and the final output is a single similarity. The
computational complexity is
$\mathcal{O}(N(2DD+DD(L-2)+D))=\mathcal{O}(N(LD^{2}+D))$. We can observe that
with the default configuration ($D=512,L=4$), the computational complexity of
MLP is about 2,049$\times$ of the DP, which is why neural network-based
interactions are difficult to deploy. They are usually designed to be used as
a fine-grained ranking module. The storage complexity of MLP is same as DP.

Cross Transformer Interaction: It is usually assumed that a Multi-Head
Attention block (MHA) consists of linear layer and QKV structure. The length
of the sequence input is $N_{t+v}$, so the computational complexity of the
linear layer is $\mathcal{O}(D^{2}N_{t+v})$. An dot-product version of the QKV
operation requires $\mathcal{O}(N_{t+v}^{2}D)$ complexity. Therefore, the
overall computational complexity for $L\times$ MHA is
$\mathcal{O}(N(D^{2}N_{t+v}+N_{t+v}^{2}D)L)$. With the default configuration
($D=512,L=4,N_{t}=32,N_{v}=12,N_{t+v}=44$), the computational complexity of
XTI is about 24,464$\times$ of the DP. At the same time, due to the use of
sequential representation for video, the storage complexity rises to
$\mathcal{O}(NN_{v}D)$.

Token-wise Interaction: Token-wise interactions need to calculate an
$N_{t}\times N_{v}$ size of similarity matrix for text and video tokens. The
computational complexity is $\mathcal{O}(N(N_{t}N_{v}D))$ and the storage
complexity is $\mathcal{O}(NN_{v}D)$.

Weighted Token-wise Interaction: For weighted token-wise interaction, the
complexity for the online search process is a little more complicated. For the
query sentence, we need to calculate an additional MLP to generate the text
weights, which brings $\mathcal{O}(D^{2}LN_{s})$. For video documents, we can
pre-compute sequential video features and the corresponding video weights. The
total storage overhead is $\mathcal{O}(N(N_{v}D+N_{v}))$. Note that during the
online process, there is no need to dynamically calculate the weights for each
video. Compared with TI, only two additional weighting operations are added in
the online process. Therefore, the overall computational complexity is
$\mathcal{O}(N(N_{t}N_{v}D+N_{t+v}))$. Compared to XTI, our computational
complexity is reduced 63.69 times.

##  8 Visualizations

More visualization results of WTI are shown in Fig. 10 and 11.

Figure 10: Qualitative results on images from MSR-VTT [44].

Figure 11: Qualitative results on images from MSR-VTT [44].

## References

  * [1] Anne Hendricks, L., Wang, O., Shechtman, E., Sivic, J., Darrell, T., Russell, B.: Localizing moments in video with natural language. In: Proceedings of the IEEE international conference on computer vision. pp. 5803–5812 (2017)
  * [2] Bain, M., Nagrani, A., Varol, G., Zisserman, A.: Frozen in time: A joint video and image encoder for end-to-end retrieval. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 1728–1738 (2021)
  * [3] Bogolin, S.V., Croitoru, I., Jin, H., Liu, Y., Albanie, S.: Cross modal retrieval with querybank normalisation. arXiv preprint arXiv:2112.12777 (2021)
  * [4] Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: International conference on machine learning. pp. 1597–1607. PMLR (2020)
  * [5] Chen, X., He, K.: Exploring simple siamese representation learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 15750–15758 (2021)
  * [6] Croitoru, I., Bogolin, S.V., Leordeanu, M., Jin, H., Zisserman, A., Albanie, S., Liu, Y.: Teachtext: Crossmodal generalized distillation for text-video retrieval. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 11583–11593 (2021)
  * [7] Deng, J., Guo, J., Xue, N., Zafeiriou, S.: Arcface: Additive angular margin loss for deep face recognition. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 4690–4699 (2019)
  * [8] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
  * [9] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)
  * [10] Fabian Caba Heilbron, Victor Escorcia, B.G., Niebles, J.C.: Activitynet: A large-scale video benchmark for human activity understanding. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 961–970 (2015)
  * [11] Fang, H., Xiong, P., Xu, L., Chen, Y.: Clip2video: Mastering video-text retrieval via image clip. arXiv preprint arXiv:2106.11097 (2021)
  * [12] Gabeur, V., Sun, C., Alahari, K., Schmid, C.: Multi-modal transformer for video retrieval. In: European Conference on Computer Vision. pp. 214–229. Springer (2020)
  * [13] Gao, Z., Liu, J., Chen, S., Chang, D., Zhang, H., Yuan, J.: Clip2tv: An empirical study on transformer-based methods for video-text retrieval. arXiv preprint arXiv:2111.05610 (2021)
  * [14] Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural computation 9(8), 1735–1780 (1997)
  * [15] Huang, P.S., He, X., Gao, J., Deng, L., Acero, A., Heck, L.: Learning deep structured semantic models for web search using clickthrough data. In: Proceedings of the 22nd ACM international conference on Information & Knowledge Management. pp. 2333–2338 (2013)
  * [16] Jacobs, R.A., Jordan, M.I., Nowlan, S.J., Hinton, G.E.: Adaptive mixtures of local experts. Neural computation 3(1), 79–87 (1991)
  * [17] Johnson, J., Douze, M., Jégou, H.: Billion-scale similarity search with GPUs. IEEE Transactions on Big Data 7(3), 535–547 (2019)
  * [18] Khattab, O., Zaharia, M.: Colbert: Efficient and effective passage search via contextualized late interaction over bert. In: Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval. pp. 39–48 (2020)
  * [19] Kim, W., Son, B., Kim, I.: Vilt: Vision-and-language transformer without convolution or region supervision. In: International Conference on Machine Learning. pp. 5583–5594. PMLR (2021)
  * [20] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  * [21] Lei, J., Li, L., Zhou, L., Gan, Z., Berg, T.L., Bansal, M., Liu, J.: Less is more: Clipbert for video-and-language learning via sparse sampling. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7331–7341 (2021)
  * [22] Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., Zettlemoyer, L.: Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461 (2019)
  * [23] Li, L., Chen, Y.C., Cheng, Y., Gan, Z., Yu, L., Liu, J.: Hero: Hierarchical encoder for video+ language omni-representation pre-training. In: EMNLP (2020)
  * [24] Li, X., Yin, X., Li, C., Zhang, P., Hu, X., Zhang, L., Wang, L., Hu, H., Dong, L., Wei, F., et al.: Oscar: Object-semantics aligned pre-training for vision-language tasks. In: European Conference on Computer Vision. pp. 121–137. Springer (2020)
  * [25] Liu, S., Fan, H., Qian, S., Chen, Y., Ding, W., Wang, Z.: Hit: Hierarchical transformer with momentum contrast for video-text retrieval. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 11915–11925 (2021)
  * [26] Liu, Y., Albanie, S., Nagrani, A., Zisserman, A.: Use what you have: Video retrieval using representations from collaborative experts. arXiv preprint arXiv:1907.13487 (2019)
  * [27] Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V.: Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019)
  * [28] Luo, H., Ji, L., Shi, B., Huang, H., Duan, N., Li, T., Li, J., Bharti, T., Zhou, M.: Univl: A unified video and language pre-training model for multimodal understanding and generation. arXiv preprint arXiv:2002.06353 (2020)
  * [29] Luo, H., Ji, L., Zhong, M., Chen, Y., Lei, W., Duan, N., Li, T.: Clip4clip: An empirical study of clip for end to end video clip retrieval. arXiv preprint arXiv:2104.08860 (2021)
  * [30] Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013)
  * [31] Van den Oord, A., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding. arXiv e-prints pp. arXiv–1807 (2018)
  * [32] Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., Lerer, A.: Automatic differentiation in pytorch (2017)
  * [33] Patrick, M., Huang, P.Y., Asano, Y., Metze, F., Hauptmann, A., Henriques, J., Vedaldi, A.: Support-set bottlenecks for video-text representation learning. arXiv preprint arXiv:2010.02824 (2020)
  * [34] Portillo-Quintero, J.A., Ortiz-Bayliss, J.C., Terashima-Marín, H.: A straightforward framework for video retrieval using clip. In: Mexican Conference on Pattern Recognition. pp. 3–12. Springer (2021)
  * [35] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020 (2021)
  * [36] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al.: Language models are unsupervised multitask learners. OpenAI blog 1(8), 9 (2019)
  * [37] Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P.J.: Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683 (2019)
  * [38] Reimers, N., Gurevych, I.: Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084 (2019)
  * [39] Rohrbach, A., Rohrbach, M., Tandon, N., Schiele, B.: A dataset for movie description. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3202–3212 (2015)
  * [40] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30 (2017)
  * [41] Wang, G., Luo, C., Xiong, Z., Zeng, W.: Spm-tracker: Series-parallel matching for real-time visual object tracking. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3643–3652 (2019)
  * [42] Wang, X., Wu, J., Chen, J., Li, L., Wang, Y.F., Wang, W.Y.: Vatex: A large-scale, high-quality multilingual dataset for video-and-language research. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 4581–4591 (2019)
  * [43] Wu, Z., Yao, T., Fu, Y., Jiang, Y.G.: Deep learning for video classification and captioning. In: Frontiers of multimedia research, pp. 3–29 (2017)
  * [44] Xu, J., Mei, T., Yao, T., Rui, Y.: Msr-vtt: A large video description dataset for bridging video and language. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 5288–5296 (2016)
  * [45] Yao, L., Huang, R., Hou, L., Lu, G., Niu, M., Xu, H., Liang, X., Li, Z., Jiang, X., Xu, C.: Filip: Fine-grained interactive language-image pre-training. arXiv preprint arXiv:2111.07783 (2021)
  * [46] Ye, M., Shen, J., Lin, G., Xiang, T., Shao, L., Hoi, S.C.: Deep learning for person re-identification: A survey and outlook. IEEE Transactions on Pattern Analysis and Machine Intelligence (2021)
  * [47] Young, P., Lai, A., Hodosh, M., Hockenmaier, J.: From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics 2, 67–78 (2014)
  * [48] Yu, Y., Kim, J., Kim, G.: A joint sequence fusion model for video question answering and retrieval. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 471–487 (2018)
  * [49] Zbontar, J., Jing, L., Misra, I., LeCun, Y., Deny, S.: Barlow twins: Self-supervised learning via redundancy reduction. In: International Conference on Machine Learning. pp. 12310–12320. PMLR (2021)


# A Dataset for Large Language Model-Driven AI Accelerator Generation

Mahmoud Nazzal∗, , Deepak Vungarala∗, , Mehrdad Morsali, , Chao Zhang, , Arnob
Ghosh, , Abdallah Khreishah, , and Shaahin Angizi ∗ These authors contributed
equally.
This work is supported in part by the National Science Foundation under Grant
No. 2228028. This work has been submitted to the IEEE for possible
publication. Copyright may be transferred without notice, after which this
version may no longer be accessibleM. Nazzal, D. Vungarala, M. Morsali, A.
Ghosh, A. Khreishah, and S. Angizi are with the Department of Electrical and
Computer Engineering, New Jersey Institute of Technology, Newark, NJ 07102
USA. E-mail:{mn69, dv336, mm2772, arnob.ghosh, abadallah,
<EMAIL_ADDRESS>C. Zhang is with the School of Computational Science
and Engineering, Georgia Institute of Technology, Atlanta, GA, USA. E-mail:
<EMAIL_ADDRESS>
###### Abstract

In the ever-evolving landscape of Deep Neural Networks (DNN) hardware
acceleration, unlocking the true potential of systolic array accelerators has
long been hindered by the daunting challenges of expertise and time
investment. Large Language Models (LLMs) offer a promising solution for
automating code generation which is key to unlocking unprecedented efficiency
and performance in various domains, including hardware descriptive code.
However, the successful application of LLMs to hardware accelerator design is
contingent upon the availability of specialized datasets tailored for this
purpose. To bridge this gap, we introduce the Systolic Array-based Accelerator
DataSet (SA-DS). SA-DS comprises of a diverse collection of spatial arrays
following the standardized Berkeley’s Gemmini accelerator generator template,
enabling design reuse, adaptation, and customization. SA-DS is intended to
spark LLM-centred research on DNN hardware accelerator architecture. We
envision that SA-DS provides a framework which will shape the course of DNN
hardware acceleration research for generations to come. SA-DS is open-sourced
under the permissive MIT license at this https://github.com/ACADLab/SA-DS.

###### Index Terms:

Systolic array design, LLM-powered hardware synthesis, accelerator
architecture optimization, EDA

##  1 Introduction

Artificial Intelligence (AI) has shown a remarkable potential to address
complex design problems ranging from software development to drug discovery. A
key advantage of AI is a significant reduction of the manual effort and
expertise requirements. This promising capability of AI suggests its
application in hardware design, particularly for developing specialized AI
accelerators needed to keep pace with the rapid evolution of Deep Neural
Networks (DNNs) [1]. In hardware design for DNNs, the complexity and need for
expert knowledge have been major limitations [2, 3].

Systolic array accelerators, typically obtained with specialized AI hardware
generators such as Gemmini [3], have significantly advanced the processing
capabilities for DNNs, providing high throughput and energy efficiency. These
systems, integrated with architectures like the Rocket Chip processor [4],
demonstrate the scalability and flexibility necessary for contemporary AI
applications. Despite these appealing achievements, challenges such as the
low-level nature, the complex programming interfaces, memory usage, and the
need for extensive development times persist [5]. Moreover, systolic array
accelerator generators like Gemmini [3] generally face limitations in
efficiently handling diverse and irregular computational patterns beyond their
optimized standard operations [6]. These limitations underscore the need for
innovative solutions such as AI model-based solutions [5, 6].

At the frontier of AI advancement, Large Language Models (LLMs) [7] offer an
appealing solution for alleviating the challenges in hardware accelerator
design. Along this line, GPT4AIGChip [8] exemplifies using LLMs to automate
the hardware design process, from conceptual design to synthesis to
fabrication. However, the lack of specialized datasets of hardware accelerator
design artifacts presents a strong obstacle in fully leveraging the potential
of LLMs [9]. This limitation restricts usage to the vanilla LLMs without fine-
tuning or in-context learning [9]; two of the most effective approaches for
maximizing the LLM capabilities.

To bridge the gap, we introduce a Systolic Array Accelerator Dataset (SA-DS)
to facilitate effective learning and generation of optimized designs by LLMs.
Specifically, our contributions include:

(1) We create, curate, and release SA-DS, the first systolic array accelerator
dataset for DNN hardware accelerator design. Each data point in SA-DS features
a verbal description of an accelerator micro-architecture and a Chisel
description of the design itself. These accelerator designs are obtained using
the Gemmini generator [3].

(2) We demonstrate the potential of SA-DS in enabling LLM-based hardware
accelerator design by showcasing its suitability for generating viable
accelerator designs by providing 1-short prompts with multiple LLMs.
Experimental results validate the suitability of SA-DS in providing high-
quality and relevant accelerator design examples to contemporary LLMs
including GPT [7], Claude [10], and Google’s Gemini [11] as compared to
existing Hardware Description Language (HDL) datasets.

Figure 1: Gemmini architectural template for ASIC accelerator design [3] .

##  2 Background

Static Accelerator Generation Tools: A wide array of static DNN accelerator
design tools have been developed such as VTA [12], MAGNet [13], and DNNWeaver
[14] to suit various applications. These tools provide many hardware
architecture templates supporting vector/systolic spatial arrays, data flows,
software ecosystems, and OS support. Among these tools, Gemmini [3] is a
comprehensive and well-packaged open-source infrastructure tailored to design
full-stack DNN accelerators. Gemmini offers a versatile hardware framework, a
multi-layered software stack, and an integrated System-On-Chip (SoC)
environment based on the architectural template shown in Fig. 1. Central to a
design provided by Gemmini is a spatial array architecture, where the template
employs a 2-D array of tiles containing Processing Elements (PE). These PE
units operate in parallel, handling Multiply-Accumulate (MAC) operations
efficiently. To optimize the area, power, and performance trade-offs, the size
of the spatial array, input type, and function units such as non-linear
functions, pooling, normalization, and dataflow can be adjusted.

TABLE I: Comparison of the state-of-the-art LLM-based HDL/HLS generators.

Function/Property | Ours | ChatEDA [9] | VeriGen [15] | GPT4AIGChip [8] | ChipGPT [16] | Chip-Chat [17] | AutoChip [18]
---|---|---|---|---|---|---|---
Function | AI Accelerator Gen. | RTL-to-GDSII | Verilog Gen. | AI Accelerator Gen. | Verilog Gen. | Verilog Gen. | Verilog Gen.
Chatbot∗ | ✓ |  |  |  |  |  |
Dataset | ✓ | NA† | ✓(Verilog) |  | NA | NA | NA
Output format | Chisel | GDSII | Verilog | HLS | Verilog | Verilog | Verilog
Automated Verification | ✓ |  † |  |  |  |  |
Multi-shot examples | ✓ |  |  |  |  |  |
Human in Loop | Low | NA | Medium | Medium | Medium | High | Low

∗A user interface featuring Prompt Optimization for the input of LLM. † Not
applicable.

LLM for Hardware Design: LLMs are increasingly pivotal in generating HDL and
High-Level Synthesis (HLS) code. Table I provides a high-level comparison of
the state-of-the-art methods along this line. GitHub Copilot [19] pioneered
automatic code generation, setting the stage for domain-specific applications
like DAVE [20]. Expanding on these capabilities, VeriGen [15] and ChatEDA [9]
advance the field by refining hardware design workflows and automating the
Register-Transfer Level (RTL) to Graphic Data System version II (GDSII)
process with fine-tuned LLMs. ChipGPT [16] and Autochip [18] further this
evolution by integrating LLMs to generate and optimize hardware designs, with
Autochip producing accurate Verilog code through simulation feedback. Chip-
Chat [17] demonstrates the efficacy of interactive LLMs like ChatGPT-4 in
fast-tracking design space exploration. RTLLM [21] and GPT4AIGChip [8]
specifically target design process efficiency, highlighting LLMs’ capacity to
manage complex design tasks and expand access to AI accelerator design,
showcasing the broad potential of LLMs in hardware design. However, except for
GPT4AIGChip [8], these works do not address using LLMs for AI hardware
accelerator architecture design.

##  3 SA-DS and An Envisioned Framework

We propose a dataset for LLM-enhanced AI hardware accelerator design and
envision a general framework for its applicability. Due to space limitations,
we restrict our presentation to describing how it can be utilized. This
framework is outlined in Fig. 2. Since an LLM’s response is determined by both
the prompt and the model coefficients, the framework focuses on these two
aspects. An immediate usage of SA-DS in the envisioned framework is to help to
fine-tune a generic LLM for the task of hardware accelerator design (Step 2 in
Fig. 2). This can be achieved by allowing its samples to partially or fully
alter the model coefficients. Besides, multi-shot prompting techniques such as
in-context learning can be used where SA-DS will function as the source for
multi-shot examples. Also, an initial prompt conveying the user’s intent and
key software and hardware specifications of the intended design can still be
further engineered/optimized through the Prompt Optimizer step (Step 1).
However, administering this optimization requires the development of timely
and accurate evaluation techniques and metrics for the designs generated.
Since SA-DS combines verbal description and systolic array design pairs, a
systolic array accelerator is taken as an outcome from the LLM (step 3). Next,
a third-party quality evaluation tool can be utilized to provide a
quantitative evaluation of the design, verify functional correctness, and
integrate the design with the full stack. (step 4).

---
Figure 2: An envisioned framework for utilizing SA-DS.

In the proposed usage framework, once the LLM generates systolic array
accelerators (step 3), the process moves to quality and functional evaluation
(step 4). The design, often formulated in Chisel, undergoes conversion to RTL
using tools like Verilator to assess functional correctness and full-stack
integration, thus translating into a verifiable HDL. This step crucially
embeds an automated RTL-GDSII validation process, where generated designs are
assessed and flagged as Valid or Invalid based on their code sequence
completeness and input-output correctness. Valid designs advance to resource
validation, focusing on optimizing for Power, Performance, and Area (PPA)
metrics. Conversely, designs flagged as Invalid trigger a feedback loop for
error analysis and LLM retraining, facilitating iterative refinement (Steps 2
to 5) to meet the set performance criteria. Ultimately, the process culminates
in Step 6, where a script generates Tool Command Language (TCL) instructions
to automate RTL evaluation for HDL codes, integrating with synthesis tools to
comprehensively assess and validate the PPA metrics of the hardware design.

##  4 SA-DS Sample Creation

SA-DS uses the Gemmini generator to provide a variety of spatial array
designs, making it easier for users to adapt and reuse these designs for
different projects. The dataset and design tools are developed with Chisel, a
programming language embedded in Scala [22], known for its clear and efficient
coding style [23]. Gemmini’s configurable nature allows for significant
customization, suiting various application-specific requirements, thus
supporting the advancement in AI chip design [24]. This combination of a
versatile template like Gemmini and a powerful design language like Chisel
ensures that SA-DS can effectively meet the diverse needs of hardware design
in AI applications.

Algorithm 1 describes how SA-DS is created within the Chipyard framework [25],
which ensures that the designs are verifiable. The process focuses on
generating spatial array structures and function units from the Gemmini
codebase. Specifically, the algorithm iterates through various configurations
of these elements, as indicated in line 6, guided by insights from analyzing
the Gemmini code. Each modification made during this process is checked for
accuracy using Verilator, ensuring that each version of the design (M) is
properly annotated to highlight its key features. The variables and their
values used in this algorithm are carefully chosen based on extensive testing
with the Gemmini template, leading to a diverse set of potential Gemmini
designs, as shown in Fig. 3.

Algorithm 1 SA-DS Creation with the Gemmini Generator

1:Input: Source Code $S$

2:Output: List of Verified Modified Source Codes $M$

3:$P\leftarrow$ list of changeable variable parameters

4:$M\leftarrow$ empty list

5:function GenerateVariations($S,P$)

6: for each $combination$ in $P$ do

7: $S_{mod}\leftarrow S$

8: for each $(parameter,value)$ in $combination$ do

9: Replace $parameter$ in $S_{mod}$ with $value$

10: end for

11: $verified\leftarrow$ VerifyWithVerilator($S_{mod}$)

12: if $verified$ then

13: $M.append(S_{mod})$

14: end if

15: end for

16: return $M$

17:end function

18:function VerifyWithVerilator($S_{mod}$)

19: return Verilator verification result for $S_{mod}$

20:end function

The SA-DS, generated as detailed in Algorithm 1, offers a variety of
configurations influenced by Function Unit (FU) availability and spatial array
sizes, leading to a structured dataset easily navigable and applicable to
diverse hardware design needs. As illustrated in Fig. 3, the dataset organizes
these configurations into six main categories, each containing 1536 unique
samples. These samples are enriched with dataflow variations like Output
Stationary (OS), Weight Stationary (WS), and their combinations, accounting
for 512 samples per dataflow type. The distribution of these parameters and
their corresponding function units is systematically represented in Fig. 4,
facilitating an understanding of the dataset’s comprehensive nature and the
interplay between different function units in each configuration.

---
Figure 3: The design space parameters of the proposed SA-DS based on
Gemmini[3].

Significance of the Parameters: M represents the configurations derived from
Gemmini, maintaining full-stack compatibility. These configurations are
crucial for defining the hardware accelerator’s micro-architectural elements
based on Gemmini’s template, including scratchpad and accumulator sizes. Key
parameters include:

  * •

Spatial Array Size: Defines the number of PEs, crucial for computational
capacity.

  * •

DataFlow: Manages data movement among PEs, with options like OS, WS, or an
automatic selection during runtime.

  * •

Function Units: Additional units that support DNN functionalities like ReLU
and normalization.

  * •

Accumulation & Spatial Array Output Type: Affects computation precision,
primarily supporting signed integer types, with potential expansion to
floating-point and complex integer types.

These elements facilitate customization to meet specific application
requirements.

Figure 4: The frequency of sample sin SA-DS in terms of (a) function units in
each category based on the Data flow for systolic arrays, (b) Function units
available individually or in combination with the others.

##  5 Experimental Analysis

In this section, we evaluate the effectiveness of SA-DS in supporting the
design generation process for hardware accelerators via LLMs, referencing the
framework depicted in Fig. 2. Due to space limitations and given the
complexity of LLM fine-tuning and prompt optimization requiring further
research, our analysis is conducted conceptually. We initiate a proof-of-
concept experiment that benchmarks SA-DS against a recent HLS Dataset (HLSD)
from [26]. This experiment considers utilizing each dataset to supply one-shot
examples for LLM prompts, aiming to enhance the generation of hardware designs
from verbal descriptions. To objectively assess the impact of each dataset, we
analyze the code quality derived from representative prompt-code pairs
selected from SA-DS. The experiment extends across four prominent LLMs; GPT-4
[7], GPT-3.5 [27], Claude [10], and Gemini Advanced [11]. Reflecting the
diversity of hardware specifications covered by SA-DS, our methodology
includes randomly selecting test sets from six categories within SA-DS,
ensuring each category is represented by 30 samples.

Evaluation is conducted by manual code review with an HLS and Chisel expert.
Due to the manual nature of verification, a bi-state verification scheme is
adopted. A pass characterizes the generation of a complete and functional code
complying with the verbal description or the case where the LLM generates the
most crucial portions of the code and leaves redundant lines or if the code is
extended to have other unended functionalities beyond what is requested in the
verbal description. Conversely, a fail refers to generating incomplete code,
incorrect file headers, or incurring fatal errors of different types that
render the code unfunctional. Therefore, we use Verilator as an automated
design verification tool exclusively for codes marked as Pass. The results of
this experiment are summarized in Table II.

TABLE II: Suitability of 1-shot examples: SA-DS vs. HLSD

LLM | SA-DS | HLSD
---|---|---
Pass | Fail | Pass | Fail
GPT-4 | 135 | 45 | 72 | 108
Gemini Advanced | 144 | 36 | 57 | 123
GPT-3.5 | 155 | 25 | 68 | 112
Claude | 150 | 30 | 71 | 109

The comparison between SA-DS and HLSD datasets in generating one-shot prompts
for LLMs like GPT-4, Gemini Advanced, GPT-3.5, and Claude, as shown in Table
II, reveals a clear pattern. SA-DS consistently shows fewer failures and more
passes across all tested LLMs with around 46% more passes on average. This
suggests that SA-DS’s examples better align with the LLMs’ capabilities,
leading to more effective code generation. The higher Pass rates in SA-DS
imply that while not perfect, the generated code often needs fewer revisions
to meet design requirements, indicating its practical value in streamlining
the accelerator design process.

##  6 Conclusion

This study has introduced the first publicly accessible LLM prompt-Chisel code
dataset, dubbed SA-DS. The prompt and code examples in SA-DS cover wide
variety of applications and design criteria. A proof-of-concept experiment has
showcased the benefits of SA-DS in enabling the high-quality generation of
hardware accelerator designs with mere verbal descriptions of novice users.
This exemplifies the promising potential of enabling further research in the
area of utilizing LLMs for automated hardware design generation. Key examples
along this line include fine-tuning high-end LLMs for hardware design,
optimized multi-shot learning, and prompt engineering serving the objectives
of design efficiency in terms of execution time, hardware cost, and power
consumption.

## References

  * [1] Y.-H. Chen _et al._ , “Eyeriss: An energy-efficient reconfigurable accelerator for deep convolutional neural networks,” _IEEE JSSC_ , vol. 52, no. 1, pp. 127–138, 2016.
  * [2] W.-Q. Ren _et al._ , “A survey on collaborative dnn inference for edge intelligence,” _Machine Intelligence Research_ , vol. 20, no. 3, pp. 370–395, 2023.
  * [3] H. Genc _et al._ , “Gemmini: Enabling systematic deep-learning architecture evaluation via full-stack integration,” in _DAC_. IEEE, 2021, pp. 769–774.
  * [4] K. Asanovic, R. Avizienis, J. Bachrach, S. Beamer, D. Biancolin, C. Celio, H. Cook, D. Dabbelt, J. Hauser, A. Izraelevitz _et al._ , “The rocket chip generator,” _EECS Department, University of California, Berkeley, Tech. Rep. UCB/EECS-2016-17_ , vol. 4, pp. 6–2, 2016.
  * [5] P. Xu and Y. Liang, “Automatic code generation for rocket chip rocc accelerators,” 2020.
  * [6] R. Xu, S. Ma, Y. Wang, and Y. Guo, “Hesa: Heterogeneous systolic array architecture for compact cnns hardware accelerators,” in _2021 Design, Automation & Test in Europe Conference & Exhibition (DATE)_. IEEE, 2021, pp. 657–662.
  * [7] (2023) Open ai chatgpt. [Online]. Available: https://openai.com/research/gpt-4
  * [8] Y. Fu _et al._ , “GPT4AIGChip: Towards next-generation ai accelerator design automation via large language models,” in _ICCAD_ , 2023, pp. 1–9.
  * [9] Z. He _et al._ , “Chateda: A large language model powered autonomous agent for eda,” in _MLCAD_. IEEE, 2023, pp. 1–6.
  * [10] (2023) Anthropic. [Online]. Available: https://www.anthropic.com
  * [11] (2024) Gemini. [Online]. Available: https://deepmind.google
  * [12] T. Moreau _et al._ , “Vta: an open hardware-software stack for deep learning,” _arXiv preprint arXiv:1807.04188_ , vol. 10, 2018.
  * [13] R. Venkatesan _et al._ , “Magnet: A modular accelerator generator for neural networks,” in _ICCAD_. IEEE, 2019, pp. 1–8.
  * [14] H. Sharma _et al._ , “From high-level deep neural models to fpgas,” in _MICRO_. IEEE, 2016, pp. 1–12.
  * [15] S. Thakur _et al._ , “Verigen: A large language model for verilog code generation,” _ACM TODAES_ , 2023.
  * [16] K. Chang _et al._ , “Chipgpt: How far are we from natural language hardware design,” _arXiv preprint arXiv:2305.14019_ , 2023.
  * [17] J. Blocklove _et al._ , “Chip-chat: Challenges and opportunities in conversational hardware design,” _arXiv preprint arXiv:2305.13243_ , 2023.
  * [18] S. Thakur _et al._ , “Autochip: Automating hdl generation using llm feedback,” _arXiv preprint arXiv:2311.04887_ , 2023.
  * [19] N. Friedman, “Introducing github copilot: your ai pair programmer,” _URL: https://github. blog/2021-06-29-introducing-github-copilot-ai-pair-programmer_ , 2021.
  * [20] H. Pearce, B. Tan, and R. Karri, “Dave: Deriving automatically verilog from english,” in _Proceedings of the 2020 ACM/IEEE Workshop on Machine Learning for CAD_ , 2020, pp. 27–32.
  * [21] Y. Lu _et al._ , “Rtllm: An open-source benchmark for design rtl generation with large language model,” _arXiv preprint arXiv:2308.05345_ , 2023.
  * [22] (2024) Scala. [Online]. Available: https://www.scala-lang.org
  * [23] J. Bachrach _et al._ , “Chisel: constructing hardware in a scala embedded language,” in _DAC_ , 2012, pp. 1216–1225.
  * [24] M. Chen, W. Shao, P. Xu, M. Lin, K. Zhang, F. Chao, R. Ji, Y. Qiao, and P. Luo, “Diffrate: Differentiable compression rate for efficient vision transformers,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023, pp. 17 164–17 174.
  * [25] A. Amid _et al._ , “Chipyard: Integrated design, simulation, and implementation framework for custom socs,” _IEEE Micro_ , vol. 40, no. 4, pp. 10–21, 2020.
  * [26] Z. Wei _et al._ , “Hlsdataset: Open-source dataset for ml-assisted fpga design using high level synthesis,” in _ASAP_. IEEE, 2023, pp. 197–204.
  * [27] (2023) Chat gpt-3.5. [Online]. Available: https://openai.com/blog/chatgpt