# Correlation Debiasing for Unbiased Scene Graph Generation in Videos

Anant Khandelwal
Glance AI
Bangalore, India
<EMAIL_ADDRESS>
###### Abstract

Dynamic scene graph generation (SGG) from videos requires not only
comprehensive understanding of objects across the scenes that are prone to
temporal fluctuations but also a model the temporal motions and interactions
with different objects. Moreover, the long-tailed distribution of visual
relationships is the crucial bottleneck of most dynamic SGG methods, since
most of them focus on capturing spatio-temporal context using complex
architectures, which leads to the generation of biased scene graphs. To
address these challenges, we propose FloCoDe: Flow-aware temporal consistency
and Correlation Debiasing with uncertainty attenuation for unbiased dynamic
scene graphs. FloCoDe employs feature warping using flow to detect temporally
consistent objects across the frames. In addition, it uses correlation
debiasing to learn the unbiased relation representation for long-tailed
classes. Moreover, to attenuate the predictive uncertainties, it uses a
mixture of sigmoidal cross-entropy loss and contrastive loss to incorporate
label correlations to identify the commonly co-occurring relations and help
debias the long-tailed ones. Extensive experimental evaluation shows a
performance gain as high as 4.1% showing the superiority of generating more
unbiased scene graphs.

##  1 Introduction

Scene graph generation for videos (VidSGG) aims to represent the video in the
form of a dynamic graph that is able to capture the temporal evolution of the
relationships between pairs of objects. VidSGG has direct usage for various
downstream applications like visual question answering [1, 44, 52], video
captioning [53] and video retrieval [8, 40, 51] etc. VidSGG is considered more
challenging compared to its image-based counterpart since the relations
between identified object pairs are dynamic along the temporal dimension,
making this a multi-label problem. In the current stage, VidSGG is relatively
in its nascent stage as compared to SGG (scene graph generation) from static
images [7, 22, 23, 24, 27, 30, 43, 57, 58, 58]. Several works [6, 28, 36, 46,
11, 17, 26] have proposed to solve VidSGG mostly with spatio-temporal sequence
processing with transformers [3, 4, 12, 18, 32, 49, 42]. Most of these methods
simply focus on building complex models that can effectively aggregate the
spatio-temporal information in a video, but they eventually lack the ability
to address the data imbalance in the relation/predicate classes. Their
performance is quite good in terms of the Recall@K metric, which is biased
towards frequent classes. However, another metric mean-Recall@K is proposed
[5, 43] to quantify the performance in the presence of low-frequent classes,
since it gives the overall view of the SGG models rather than considering only
high-frequent classes. Although, recent methods [31, 35] has proposed to deal
with class imbalance using memory-based debiasing and uncertainty attenuation
for classification.But their method of debiasing is based on learnable
attention and transformer weights, which have a risk of getting biased towards
high-frequent classes. One such case of this issue has been shown in Figure 2
during qualitative comparison with our method. Moreover, the uncertainty
attenuation based on Gaussian Mixture Models (GMM) has several limitations, as
discussed in [34, 56, 38]. To overcome these limitations of GMM based
classification, some works have proposed the regularization with a mixture of
standard deviations [19] and some proposed loss function that can incorporate
label correlations [38]. Some more approaches [25, 54] also tried to address
the biased relation predictions. Li et al. [25] proposed to weaken the false
correlation between input data and the predicate labels. Xu et al. [54]
considered biases in a meta-learning paradigm. These approaches mitigate the
long-tail problem to some extent, but the performance is still not
satisfactory.
In this work, we not only focus on improving the unbiased predicate
classification but also on improving object detection. We proposed to use the
flow-warped features in the temporal dimension to compensate for the dynamic
fluctuations in a video. In our analysis (in Table 4), it is shown that the
major bottleneck in the existing dynamic SGG is the incorrect detection of
objects across the video frames. Moreover, as compared to previous methods[31]
which use memory correlation to debias the predicate embedding, it has the
risk of biasing the attention weights towards highly frequent classes. In
order to mitigate this, we propose to debias the predicate embeddings during
the generation stage itself, since the correlation between the predicate
embeddings and the entities is primarily driven by high-frequent relation
classes, and hence we make this correlation unbiased so that the learned
embeddings are themselves the debiased embeddings. Furthermore, the
uncertainty attenuation in existing methods[31] does not take into account the
label correlations. Hence, we proposed to use a mixture of logic networks
(MLNs) that can distinguish between two different types of predictive
uncertainty: aleatoric and epistemic uncertainty. In addition to the
uncertainty-aware classification loss, we introduced supervised contrastive
loss to take into account the label correlations. The objective of multi-label
contrastive loss is to pull together predicate representations having more
than one overlapping class while pushing apart negative samples that do not
share any classes. Further, we regularize the uncertainty-aware training loss
function with aleatoric and epistemic uncertainty to penalise the loss
function explicitly, which can reduce the uncertainty, especially for low-
frequent classes where the probability of incorrect prediction is high.
Combining all this, we named our framework FloCoDe: Flow-aware temporal
consistency and Correlation Debiasing with Uncertainty Attenuation for an
unbiased dynamic SGG. The major contributions of this paper are: 1) FloCoDe
models both (a) aleatoric and epistemic uncertainty associated with dynamic
SGG and (b) label correlations to produce more unbiased scene graphs. 2)
Utilising a novel correlation-guided debiased learning of predicate embeddings
that avoids the bias in learnable weights of attention and transformer
decoder. 3) Utilising flow-aware, temporally consistent object detection for
accurate classification of nodes in scene graphs 4) FloCoDe achieves
significant gains for mR@K[43] and R@K, highlighting its superiority in
generating unbiased scene graphs.

##  2 Related Work

Image Scene Graph Generation: Image-based scene graph generation (ImgSGG) is
the task of obtaining the structured graph summarization that involves objects
as nodes and their relationships (formally called predicates) as edges in an
image. There exists tremendous work on ImgSGG with comparison to a common
benchmark, Visual Genome (VG) [22]. Some of these focused on evolving
efficient ways of aggregating spatial context [24, 27, 30, 57, 58] while some
of the latest works are based on addressing fundamental problems such as
preventing biased scene graphs caused by long-tailed predicate distribution
and noisy annotations in dataset.
Video Scene Graph Generation (VidSGG): With the successful exploration of
spatial context within images, video researchers have started to explore the
spatial context and temporal correlation between objects detected across the
frames. Similar to ImgSGG, long-tailed predicates and noisy annotations still
exist in the VidSGG benchmark Action Genome [16]; further, it has the
additional challenge of addressing the temporal fluctuations across the
frames. Numerous approaches [36, 59, 48, 28, 46] have employed object-tracking
mechanisms to tackle the temporal fluctuations among different frames.
However, object-tracking models incur high computational costs and memory
consumption; they also accumulate information from irrelevant frames, leading
to suboptimal performance. STTran [6] proposes a strong baseline that adopts a
spatial encoder and a temporal decoder to extract implicitly spatial-temporal
contexts. Other works [26, 50] which are also based on extracting temporal
correlations. Some ensure temporal continuity by extracting the entire co-
occurrence pattern, while others propose pre-training paradigms to model the
temporal correlations implicitly. A lot of works [3, 32, 42, 49] are based on
the superior sequence processing ability of transformers for the spatio-
temporal context of visual relations. However, despite their success, these
are mostly providing gains for high-frequent classes and suffer from the
problem of long-tail bias. Recent work TEMPURA[31] tries to address the long-
tail problem using an uncertainty-guided loss function. We go beyond this and
explore label correlation to further reduce prediction uncertainty.

##  3 Method

###  3.1 Preliminary

Problem Statement: Given a video
$\mathcal{V}=\\{I_{1},I_{2},I_{3},...,I_{T}\\}$, the goal of dynamic SGG is to
generate scene graphs denoted as $\mathcal{G}=\\{G_{t}\\}_{t=1}^{T}$ of video
$V$ consisting of $T$ frames. $G_{t}=\\{V_{t},E_{t}\\}$ is the scene graph of
frame $I_{t}$, where $V_{t}$ is the set of nodes and $E_{t}$ is the relation
as edges between nodes in $V_{t}$. Nodes $V_{t}$ connected to each other using
predicate in $E_{t}$, forming multiple $<$subject-predicate-object$>$
triplets. The set of object and predicate classes are referred to as
$\mathcal{Y}_{o}=\\{{y_{o}}_{1},{y_{o}}_{2},{y_{o}}_{3},.....,{y_{o}}_{c_{o}}\\}$
and
$\mathcal{Y}_{r}=\\{{y_{r}}_{1},{y_{r}}_{2},{y_{r}}_{3},.....,{y_{r}}_{c_{r}}\\}$
respectively.
Object Detection and Relation Representation: With the use of an off-the-shelf
object detector (Faster-RCNN [37]), we obtain the set of objects
$O_{t}=\\{o_{i}^{t}\\}_{i=1}^{N(t)}$, where $N(t)$ is the number of objects
detected in frame $I_{t}$. Each object in a $t^{th}$ frame is denoted as
$o_{i}^{t}=\\{b_{i}^{t},v_{i}^{t},c_{o_{i}}^{t}\\}$ where
$b_{i}^{t}\in\mathbb{R}^{4}$ being the bounding box,
$v_{i}^{t}\in\mathbb{R}^{2048}$ the RoIAligned [14] proposal feature of
$o_{i}^{t}$ and $c_{o_{i}}^{t}$ is its predicted class. However, the object
class $c_{o_{i}}^{t}$ fluctuates across the frames and is not coherent even
for the same object. Existing works [46] address this by incorporating object
tracking algorithms; opposed to this, our strategy compensates for these
dynamic fluctuations using flow-warped features and ensures temporal
coherence. Additionally, we extracted the base features $f_{t},t\in[1,T]$ and
ROIs from Faster R-CNN using ResNet-101 [13]. We warp these base features
using the temporal flow and compute the RoIAligned warped object features
$v_{i}^{t\rightarrow t^{\prime}}$ ($t^{\prime}$ represents the immediate
previous frame that contains the $i^{th}$ object) using the predicted ROIs.

###  3.2 Temporal Flow-Aware Object Detection

Object detectors trained on static images are prone to misclassifying the same
object in different frames. Existing methods [6, 17, 26, 50] either use
$c_{o_{i}}^{t}$ obtained from object detection in each frame or use object
feature to classify objects. However, these methods do not compensate for
temporal fluctuations in the videos. Inspired by FPGA[60], which uses flow-
guided feature aggregation for object detection in videos, we propose to
leverage flow-warped features and temporal processing for consistent object
detection across frames.

Figure 1: FloCoDe: Each RGB frame is passed to object detector and object
proposals are passed to temporal flow-aware object classification and relation
classification (correlation debiasing and MLN for uncertainty attenuation) to
generate unbiased scene graphs

We introduce Temporal Flow-Aware Object Detection (TFoD), which utilises
transformer encoder [49] with masked self-attention (TEnc) to process the set
of temporal object sequences $\mathcal{O}_{\mathcal{V}}$, which is constructed
as follows:

$\mathcal{O}_{\mathcal{V}}=\\{\mathcal{O}_{t_{1},k_{1}}^{1},\mathcal{O}_{t_{2},k_{2}}^{2},...,\mathcal{O}_{t_{\hat{C}_{o}},k_{\hat{C}_{o}}}^{\hat{C}_{o}}\\},\textrm{
where }\\\
\mathcal{O}_{t_{j},k_{j}}^{j}=\\{v_{i}^{t},v_{i}^{t+1},......,v_{i}^{k}\\}$
(1)

Each entry of $\mathcal{O}_{t_{j},k_{j}}^{j}$ corresponds to an object of the
same detected class ${c_{o}}_{j}$, here $1\leq t_{j},k_{j}\leq T$ and
$\hat{\mathcal{C}}_{o}\leq\mathcal{C}_{o}$ denoting all the detected classes
in a video $\mathcal{V}$. However, the detected class labels can be noisy
since they are based only on frame-level predictions; hence, we use the flow-
warped feature $v_{i}^{t\rightarrow t^{\prime}}$ instead of $v_{i}^{t}$ before
feeding it to the transformer encoder. The flow-warped feature is computed as:

$f_{t\rightarrow
t^{\prime}}=\mathcal{W}(f_{t},\mathcal{F}(I_{t^{\prime}},I_{t}))$ (2)

where $\mathcal{W}$ is a bilinear warping function [61, 60] applied on all the
locations for each channel in the feature maps and flow field
$\mathcal{F}(I_{t^{\prime}},I_{t})$ is computed from the pre-trained Flow-Net
[9], where $t^{\prime}$ is the index of the immediate frame previous to the
$t^{th}$ frame having the same object as depicted in the object sequences
$\mathcal{O}_{\mathcal{V}}$. Using $f_{t\rightarrow t^{\prime}}$ and predicted
ROIs, the warped RoIAligned feature is computed, and then the
$\mathcal{O}_{t_{j},k_{j}}^{j}$ is denoted as:

$\mathcal{O}_{t_{j},k_{j}}^{j}=\\{v_{i}^{t},v_{i}^{t+1\rightarrow
t},......,v_{i}^{k}\\}$ (3)

Each of $\mathcal{O}_{t_{j},k_{j}}^{j}$ is zero-padded to prepare functional
input tensor. Tenc uses masked multi-head self-attention instead of multi-head
self-attention in transformer encoder [49]. Mask is introduced to learn the
temporal dependencies in a unidirectional manner so that the object at frame
index $t$ can only attend to objects in previous frames. Attending to future
context can be noisy since it is more probable for unrelated objects, and
hence it is noisy to attend to future context. For any input $\mathbf{X}$, the
single head masked attention $\mathbb{A}$ is given as:

$\mathbb{A}(\mathbf{Q,K,V})=softmax\left(\frac{mask(\mathbf{Q}\mathbf{K}^{T})}{\sqrt{D_{K}}}\right)\mathbf{V}$
(4)

where $D_{K}$ is the dimension of $\mathbf{K}$, and $\mathbf{Q,K,V}$ are the
query, key, and value vectors, respectively. Here, $\mathbf{Q=K=V=X}$, the
multi-head attention is $\mathbb{M}(X)=concat(a_{1},a_{2},...a_{H})W_{H}$,
where
$a_{i}=\mathbb{A}(\mathbf{X}{W_{Q}}_{i},\mathbf{X}{W_{K}}_{i},\mathbf{X}{W_{V}}_{i})$
where ${W_{Q}}_{i},{W_{K}}_{i},{W_{V}}_{i}\textrm{ and }W_{H}$ are the
learnable weight matrices. The rest of the components, like residual
connection, normalisation, and FFN (feed-forward network), remain the same as
in the transformer encoder [49]. The output of n-layered TEnc is given as:

$X_{out}^{(n)}=TEnc(X_{out}^{(n-1)}),\textrm{
}X_{out}^{(0)}=\hat{\mathcal{O}}_{V}$ (5)

where $\hat{\mathcal{O}}_{V}=\mathcal{O}_{V}+P_{o}^{T}$, where $P_{o}^{T}$ are
the fixed positional embeddings injecting the temporal position of objects.
Inspired by the properties of neural collapse [33] we prefixed the classifier
weights (forming Equiangular Tight Frame (ETF)) for each object class to
induce the maximal separable classifier even under the class imbalance
setting. The pre-fixed classifier weights $\mathbf{W}_{ETF}$ are given as:

$\mathbf{W}_{ETF}=\sqrt{\frac{\mathcal{C}_{o}}{\mathcal{C}_{o}-1}}\mathbf{U}\left(\mathbf{I}_{\mathcal{C}_{o}}-\frac{1}{\mathcal{C}_{o}}\mathbf{1}_{\mathcal{C}_{o}}\mathbf{1}_{\mathcal{C}_{o}}^{T}\right)$
(6)

where
$\mathbf{W}_{ETF}=[\mathbf{w}_{1},\mathbf{w}_{2},.....\mathbf{w}_{\mathcal{C}_{o}}]\in
R^{d\times\mathcal{C}_{o}}$, $\mathbf{U}\in R^{d\times\mathcal{C}_{o}}$,
allows a rotation and satisfies
$\mathbf{U}^{T}\mathbf{U}=\mathbf{I}_{\mathcal{C}_{o}},\mathbf{I}_{\mathcal{C}_{o}}$
is the identity matrix, and $\mathbf{1}_{\mathcal{C}_{o}}$ is an all-ones
vector. The object classification loss is then given as:

$\mathcal{L}_{o}(x_{o_{i}},\mathbf{W}_{ETF})=\frac{1}{2}\left(\mathbf{w}_{c_{o_{i}}}^{T}\hat{x}_{o_{i}}-1\right)$
(7)

where $\hat{x}_{o_{i}}=x_{o_{i}}/||x_{o_{i}}||\textrm{ and }x_{o_{i}}\in
X_{out}^{(n)}$ and $\mathbf{w}_{c_{o_{i}}}$ is the fixed prototype in
$\mathbf{W}_{ETF}$ for object class $c_{o_{i}}$ and we have
$||\mathbf{w}_{c_{o_{i}}}||=1$. Finally, the converged features will be
aligned with $\mathbf{W}_{ETF}$, and thus the ETF structure instructed by
neural collapse is attained. This results in pulling the object features of
the same class into a common prototype and pushing away the features of other
classes. The theoretical advantage of the loss has been proved in [55].

###  3.3 Correlation-Aware Predicate Embedding

The relationship between objects is governed by three types of correlations:
a) spatial correlation between predicates b) temporal correlation between
predicates c) predicate-object correlation across the video frames. We propose
to model these correlations using the Vanilla Transformer [49]. Since the
relations between objects are highly imbalanced, the relation representation
becomes biased towards popular ones, and hence, to produce unbiased relation
embeddings, we propose to update correlation matrices as a weighted average of
the current correlation matrix and the previous matrix, where the weight is
determined by the decay factor. For each object $o_{i}$ of predicted class
$c_{o_{i}}$ obtained from object detection (Section 3.2), the input to the
transformer encoder is the set of features describing the relation with each
object $o_{j}$ detected in all the frames where $o_{i}$ is detected. The input
is constructed as follows:

$r_{i,j}^{t}=concat(\mathbf{x}_{c_{o_{i}}},f_{u}(\mathbf{u}_{ij}^{t}+f_{box}(\mathbf{b}_{i}^{t},\mathbf{b}_{j}^{t})),f_{I}(t))$
(8)

where $\mathbf{x}_{c_{o_{i}}}\in X_{out}^{(n)}$ is the feature representation
of object $o_{i}$ belonging to class $c_{o_{i}}\in[1,\mathcal{C}_{o}]$,
$u_{ij}^{t}\in\mathbb{R}^{256\times 7\times 7}$ is the feature map of the
union box computed by RoIAlign [14]. $f_{u},f_{I}$ is the FFN based on non-
linear projections, and $f_{box}$ is the bounding box to feature map
projection of [57]. Both are configured to produce d-dimensional relation
features. $f_{I}$ serve as positional embeddings denoting the frame index. The
single encoder input consists of both spatial and temporal relation features
between object $o_{i}$ and all other objects. $\\{o_{1},o_{2},....o_{j}\\}$,
specifically it is constructed as
$R_{i}=\\{r_{i,1}^{t_{1}},r_{i,2}^{t_{2}},.....r_{i,j}^{t_{j}}\\},\textrm{
where }t_{j}$ are the frame indices where $o_{i}$ and $o_{j}$ are detected
simultaneously. The transformer decoder leverages masked self-attention, and
its input is a set of object representations.
$\\{\mathbf{x}_{c_{o_{1}}},\mathbf{x}_{c_{o_{2}}}...,\mathbf{x}_{c_{o_{j}}}\\}$,
corresponding to objects $\\{o_{1},o_{2},.....o_{j}\\}$, detected in all the
frames where $o_{i}$ is detected. The input to the transformer encoder
contains all the predicate (relation) features across the frames, and hence,
with multi-head self-attention, it models both spatial and temporal
correlation between predicates. Similarly, the cross-attention between the
encoder and decoder models the predicates-object correlation. The predicate
embeddings at the output of the transformer decoder are denoted as
$\hat{r}_{tem}^{k}=\hat{r}_{i,j}^{t}\forall k\in[1,N(t)],t\in[1,T]$. At
transformer decoder, we use the sliding window of size 10 for predicate
representation with relating objects.

###  3.4 Debiased Predicate Embedding

The relation embeddings at the output of the transformer decoder are biased by
the fact that popular relations drive the learning of attention weights,
causing the biased predicate embeddings. Hence, we update the cross-attention
matrix with the previous matrix corresponding to each triplet
$\\{o_{i},r_{i,j},o_{j}\\}$ obtained by attention between the transformer
encoder and decoder across all the layers. Since the input to the transformer
encoder is concatenated relation and object features, and hence using cross-
attention with querying objects, we get the predicate-object correlations.
Since some predicates are rare and hence more prone to getting biased, if we
update their correlation using previously observed correlation, it will
generate the debiased embeddings. Let’s denote the stored correlation matrix
between every object pair for all relations as $\mathcal{M}_{e-1}$ at the end
of the previous epoch $e-1$, and the attention matrix at the current ongoing
epoch $e$ is $A_{e}$. During training, we update the attention matrix (denoted
as $\hat{A}_{e}$) using the decaying factor $\eta$ as the training epoch
progresses, given as:

$\hat{A}_{e}(o_{i},r_{i,j},o_{j})=\eta*A_{e}(o_{i},r_{i,j},o_{j})+\\\
(1-\eta)*\mathcal{M}_{e-1}(o_{i},r_{i,j},o_{j})$ (9)

where $\mathcal{M}_{0}=A_{0}$ at the end of the zero epoch. We then use the
attention value from $\hat{A}_{e}$ in place of the attention value calculated
from $QK^{T}\in A_{e}$. This avoids debiasing the attention weights to avoid
biasing towards popular predicates. Once the debiased weights are learned
during training, we expect that they will generate debiased embeddings during
the inference phase. Hence, there is no modification of attention matrices
during inference time.

###  3.5 Predicate Classification

Following previous works [31], we model the classifier framework to model the
noisy annotations in the SGG data. Specifically, we model the classification
head as a mixture-of-experts model named mixture logit networks (MLN) and a
noise pattern estimation method utilizing the outputs of the MLN. The number
of mixtures is $\mathcal{K}$. Different from [31], we propose: 1) an
uncertainty-aware mixture of attenuated loss 2) supervised contrastive
learning, which incorporates label correlation to improve predicate
classification.
Uncertainty-Aware Mixture of Attenuated loss: For a sample embedding
$\mathbf{z}_{i}$, the class-specific aleatoric ($\sigma_{a}$) and epistemic
uncertainty ($\sigma_{e}$) are computed as:

$\displaystyle\sigma_{e}^{2}$
$\displaystyle=\sum_{p=1}^{\mathcal{C}_{r}}\sum_{k=1}^{\mathcal{K}}\pi_{i,p}^{k}||\mu_{i,p}^{k}-\sum_{j=1}^{\mathcal{K}}\pi_{i,p}^{j}\mu_{i,p}^{j}||_{2}^{2}$
(10) $\displaystyle\sigma_{a}^{2}$
$\displaystyle=\sum_{p=1}^{\mathcal{C}_{r}}\sum_{k=1}^{\mathcal{K}}\pi_{i,p}^{k}\Sigma_{i,p}^{k}\vspace{-4mm}$
(11)

where the mean, variance, and mixture weights for the $p^{th}$ predicate class
are estimated as follows:

$\mu_{i}^{k}=f_{\mu}^{k}(\mathbf{z}_{i}),\Sigma_{i}^{k}=\sigma(f_{\Sigma}^{k}(\mathbf{z}_{i})),\pi_{i}^{k}=\frac{e^{f_{\pi}^{k}(\mathbf{z}_{i})}}{\sum_{k=1}^{\mathcal{K}}e^{f_{\pi}^{k}(\mathbf{z}_{i})}}\vspace{-1.7mm}$
(12)

where $f_{\mu},f_{\Sigma},f_{\pi}$ are the FFN projection functions and
$\sigma$ is the sigmoid non-linearity which ensures $\Sigma_{i,p}^{k}\geq 0$
for the $p^{th}$ predicate class. During training,
$\mathbf{z}_{i}=\hat{r}_{tem}^{i}$, the mixture of attenuated loss
($\mathcal{L}_{MAL}$) is given as:

$\mathcal{L}_{MAL}=\frac{1}{N}\sum_{i=1}^{N}\sum_{p=1}^{\mathcal{C}_{r}}\sum_{k=1}^{\mathcal{K}}\pi_{i,p}^{k}\frac{\mathcal{L}(\mu_{i,p}^{k},y_{r_{p}}^{i})}{\Sigma_{i,p}^{k}}\vspace{-2mm}$
(13)

where $\mathcal{L}(\mu_{i,p}^{k},y_{r_{p}}^{i})$ is the sigmoidal cross
entropy loss, $y_{r_{p}}^{i}$ is the ground-truth predicate class mapped to
$\mathbf{z}_{i}$, $\mu_{i,p}^{k}$ is logit of label $p$ in $k^{th}$ mixture.
For the corrupted input it is more likely to make a false prediction, hence
$\Sigma_{i,p}^{k}$ will increase to reduce the overall loss function for such
instance, which in turn prevents over-fitting to the corrupted instances
making model more robust.
Uncertainty-aware Supervised Contrastive Learning: The MAL loss function
classifies labels independently; this makes it difficult to capture
correlations between co-occurring semantic labels. To address this limitation,
we propose kernel-based multi-label contrastive loss, i.e.,
$\mathcal{L}_{\textrm{KMCL}}$. The objective of this loss function is to pull
together the representations of predicates sharing at least one class with the
anchor representation $\hat{r}_{tem}^{n}$ while pushing apart negative samples
that do not share any classes. Let us consider the positive set
$\mathcal{A}(n)=\\{m\in\\{N\setminus
n\\}:\mathcal{Y}_{r}^{n}\cdot\mathcal{Y}_{r}^{m}\neq 0$, where $\cdot$ is a
dot product} contains samples that have at least one label in common with the
anchor $\hat{r}_{tem}^{n}$, while
$\mathcal{Y}_{r}(n,m)=\\{y_{r_{p}}\in\mathcal{Y}_{r}:y_{r_{p}}^{m}=y_{r_{p}}^{n}=1\\}$
represents the indices of shared labels between $\hat{r}_{tem}^{n}$ and
$\hat{r}_{tem}^{m}$. The loss is formulated as:

$\small\mathcal{L}_{\textrm{KMCL}}=\frac{1}{N}\sum_{n=1}^{n=N}\frac{-1}{|\mathcal{A}(n)|}\sum_{m\in\mathcal{A}(n)}J(n,m)\\\
\sum_{y_{r_{p}}\in\mathcal{Y}_{r}(n,m)}\left(\textrm{log}\frac{\textrm{exp}(\rho_{y_{r_{p}}}^{n,m}/\tau)}{\sum_{i\in
N\setminus n}\textrm{exp}(\rho_{y_{r_{p}}}^{n,i}/\tau)}\right)$ (14)

where kernel similarity is given as:

$\small\rho_{y_{r_{p}}}^{n,i}=\left(\prod_{k=1}^{\mathcal{K}}\left(\frac{(\Sigma_{n,p}^{k})^{2}+(\Sigma_{i,p}^{k})^{2}}{2(\Sigma_{n,p}^{k})(\Sigma_{i,p}^{k})}\right)^{-\frac{1}{2}}\right)\\\
\textrm{exp}\left(-\frac{1}{4}\sum_{k=1}^{\mathcal{K}}\frac{(\mu_{n,p}^{k}-\mu_{i,p}^{k})^{2}}{(\Sigma_{n,p}^{k})^{2}+(\Sigma_{i,p}^{k})^{2}}\right)$
(15)

EMA Teacher: During training, we adopt the EMA weight update [20, 45, 47, 2]
for transformers in Section 3.3. Let’s say $\phi_{T},\theta_{T}$ are the
weights of transformers for teacher and student, respectively. The weight
update is then given as:

$\phi_{T,e}=\alpha*\phi_{T,e-1}+(1-\alpha)*\theta_{T,e}$ (16)

where $e$ is the training epoch. The EMA teacher effectively an ensemble of
student models at different training steps, which is a most widely used
learning strategy in semi-supervised setting [10, 15, 41, 45]. With combined
effect of all the student models the teacher model is an unbiased estimator of
predicate embeddings, resulting in improved performance.

###  3.6 Training and Testing

Training: With flow based object predictor (Section 3.2) and correlation-aware
predicate embeddings, the debiased predicate embeddings (Section 3.3, 3.4) is
generated. The entire framework is trained end-to-end minimizing the loss
equation:

$\mathcal{L}=\mathcal{L}_{o}+\mathcal{L}_{\textrm{MAL}}+\mathcal{L}_{\textrm{KMCL}}-\lambda_{1}\sigma_{e}+\lambda_{2}\sigma_{a}$
(17)

Testing: During testing we utilize the EMA teacher $\phi_{T}$ to generate the
predicate embeddings $\hat{r}_{tem}^{i}$. These predicate embeddings are then
passed to MLN which outputs the predicate confidence scores,
$\hat{y}_{r_{p}}^{i}$. The predicate confidence scores from $\mathcal{K}$
mixtures is given as:

$\hat{y}_{r_{p}}^{i}=\sum_{k=1}^{\mathcal{K}}\pi_{i,p}^{k}\frac{\mu_{i,p}^{k}}{\Sigma_{i,p}^{k}}$
(18)

##  4 Experiments

###  4.1 Dataset and Implementation

Dataset: Following previous works [31, 6], we also evaluated our method on the
most widely used benchmark, Action Genome [16]. Action Genome is the largest
benchmark for video SGG; it is built on top of Charades[39]. It contains
476,229 bounding boxes for 35 object classes (without person) and 1,715,568
instances of 26 predicate classes annotated for 234,253 frames. For all
experiments, we use the same training and test split following [31, 6, 26].
Metrics and Evaluation Setup: We evaluated the performance of FloCoDe with
popular metrics, namely, recall@K (i.e., R@K) and mean-recall@K (i.e., mR@K),
for $K=[10,20,50]$. R@K measures the ratio of correct instances among the
top-K predicted instances with the highest confidence, but this is biased
towards frequent predicate classes [43], whereas mR@K averages out the R@K
over all relationships. Hence, mR@K is a more reliable metric for balanced
evaluation across predicates [43].
Tasks: Following previous works [6, 16, 22, 46], we also evaluated our method
on three different experimental tasks: 1) Predicate Classification (PREDCLS):
predict the predicate class of object pairs, given the ground-truth bounding
boxes and labels of objects. 2) Scene graph classification (SGCLS): predict
both predicate labels and the category labels of objects given the bounding
boxes of objects. 3) Scene graph detection (SGDET): simultaneously detects
objects appearing in the frame and the predicate labels of each object pair in
a frame. Following, we also evaluated our method using two setups: a) With
Constraint and b) No Constraints. Later one allows each object pair to have
more than one predicates simultaneously while the former one restricts to only
one predicate.

Table 1: Comparative results for SGDET task, on AG[16] in terms of mean-
Recall@K and Recall@K, best results are in bold.

Method | With Constraint | No Constraint
---|---|---
mR@10 | mR@20 | mR@50 | R@10 | R@20 | R@50 | mR@10 | mR@20 | mR@50 | R@10 | R@20 | R@50
RelDN | 3.3 | 3.3 | 3.3 | 9.1 | 9.1 | 9.1 | 7.5 | 18.8 | 33.7 | 13.6 | 23.0 | 36.6
HCRD supervised | - | 8.3 | 9.1 | - | 27.9 | 30.4 | - | - | - | - | - | -
TRACE | 8.2 | 8.2 | 8.2 | 13.9 | 14.5 | 14.5 | 22.8 | 31.3 | 41.8 | 26.5 | 35.6 | 45.3
ISGG | - | 19.7 | 22.9 | - | 29.2 | 35.3 | - | - | - | - | - | -
STTran | 16.6 | 20.8 | 22.2 | 25.2 | 34.1 | 37.0 | 20.9 | 29.7 | 39.2 | 24.6 | 36.2 | 48.8
STTran-TPI | 15.6 | 20.2 | 21.8 | 26.2 | 34.6 | 37.4 | - | - | - | - | - | -
APT | - | - | - | 26.3 | 36.1 | 38.3 | - | - | - | 25.7 | 37.9 | 50.1
TEMPURA | 18.5 | 22.6 | 23.7 | 28.1 | 33.4 | 34.9 | 24.7 | 33.9 | 43.7 | 29.8 | 38.1 | 46.4
FloCoDe | 22.6 | 24.2 | 27.9 | 31.5 | 38.4 | 42.4 | 28.6 | 35.4 | 47.2 | 32.6 | 43.9 | 51.6

Table 2: Comparative results for PREDCLS and SGCLS task, on AG[16] in terms of
mean-Recall@K, best results are in bold.

Method | With Constraint | No Constraints
---|---|---
PredCLS | SGCLS | PredCLS | SGCLS
mR@10 | mR@20 | mR@50 | mR@10 | mR@20 | mR@50 | mR@10 | mR@20 | mR@50 | mR@10 | mR@20 | mR@50
RelDN | 6.2 | 6.2 | 6.2 | 3.4 | 3.4 | 3.4 | 31.2 | 63.1 | 75.5 | 18.6 | 36.9 | 42.6
TRACE | 15.2 | 15.2 | 15.2 | 8.9 | 8.9 | 8.9 | 50.9 | 73.6 | 82.7 | 31.9 | 42.7 | 46.3
STTran | 37.8 | 40.1 | 40.2 | 27.2 | 28.0 | 28.0 | 51.4 | 67.7 | 82.7 | 40.7 | 50.1 | 58.8
STTran-TPI | 37.3 | 40.6 | 40.6 | 28.3 | 29.3 | 29.3 | - | - | - | - | - | -
TEMPURA | 42.9 | 46.3 | 46.3 | 34.0 | 35.2 | 35.2 | 61.5 | 85.1 | 98.0 | 48.3 | 61.1 | 66.4
FloCoDe | 44.8 | 49.2 | 49.3 | 37.4 | 39.2 | 39.4 | 63.2 | 86.9 | 98.6 | 49.7 | 63.8 | 69.2

Table 3: Comparative results for PREDCLS and SGCLS task, on AG[16] in terms of
Recall@K, best results are in bold.

Method | With Constraint | No Constraints
---|---|---
PredCLS | SGCLS | PredCLS | SGCLS
R@10 | R@20 | R@50 | R@10 | R@20 | R@50 | R@10 | R@20 | R@50 | R@10 | R@20 | R@50
RelDN | 20.3 | 20.3 | 20.3 | 11.0 | 11.0 | 11.0 | 44.2 | 75.4 | 89.2 | 25.0 | 41.9 | 47.9
TRACE | 27.5 | 27.5 | 27.5 | 14.8 | 14.8 | 14.8 | 72.6 | 91.6 | 96.4 | 37.1 | 46.7 | 50.5
STTran | 68.6 | 71.8 | 71.8 | 46.4 | 47.5 | 47.5 | 77.9 | 94.2 | 99.1 | 54.0 | 63.7 | 66.4
STTran-TPI | 69.7 | 72.6 | 72.6 | 47.2 | 48.3 | 48.3 | - | - | - | - | - | -
APT | 69.4 | 73.8 | 73.8 | 47.2 | 48.9 | 48.9 | 78.5 | 95.1 | 99.2 | 55.1 | 65.1 | 68.7
TEMPURA | 68.8 | 71.5 | 71.5 | 47.2 | 48.3 | 48.3 | 80.4 | 94.2 | 99.4 | 56.3 | 64.7 | 67.9
FloCoDe | 70.1 | 74.2 | 74.2 | 48.4 | 51.2 | 51.2 | 82.8 | 97.2 | 99.9 | 57.4 | 66.2 | 68.8

Implementation details: Following previous works [6, 16, 22, 46, 31], we
adopted FasterRCNN [37] with ResNet-101 [13] backbone as the object detector.
We train the object detector on the training set of Action Genome[16]; this
results in 24.6 mAP at 0.5 IoU with CoCo metrics. For a fair comparison, we
used this detector across all the baselines. Following previous works [6, 16,
22, 46, 31], per-class non-maximal suppression at 0.4 IoU (intersection over
union) is applied to reduce region proposals provided by RPN. The parameters
of the object detector (excluding the object classifier) are fixed during
training when training scene graph generation models. For correlation-aware
predicate embedding, it is required to match the object pairs across the
frames. If there are multiple objects in the same category, to distinguish
different pairs, we use IoU between the two objects across different images to
match the subject-object pair. We calculated the IoU between the bounding box
of the object in the previous frame and the object of the same category in the
next frame. If the IoU is higher than 0.8, we consider them to be the same
object. If there are multiple candidates, we choose the one with the highest
IoU. We use an AdamW optimizer [29] with a batch size of 1 and an initial
learning rate of $2e^{-5}$. The number of mixture components $\mathcal{K}$ is
set to $4$ for SGCLS and $6$ for PREDCLS and SGDET. The self-attention and
cross-attention layers in our framework have $8$ heads with $d=1536$ and
dropout$=0.1$. We set regularizer hyper-parameters as
$\lambda_{1}=1,\lambda_{2}=1$. For debaised predicate embedding, we set the
initial learning rate of $1e^{-}5$ and reduce it with patience to $3$. For the
EMA teacher update, we have used $\alpha=0.999$. All experiments are carried
out on a single NVIDIA RTX-3090.

Figure 2: Qualitative Comparison with TEMPURA[31] for both With Constraint and
No Constraints setup. From left to right: input video frames, ground truth
graphs, graphs generated by FloCoDe, graphs generated by TEMPURA[31].
Incorrect object and predicate predictions are shown in green and red,
respectively. Table 4: Ablation Studies: Importance of KMCL, Debiasing, TFoD,
Regularizer & EMA Teacher for SGCLS and SGDET.

Uncertainty-aware Contrastive Learning |  Correlation-aware Debiasing |  Flow-aware Temporal Consistency | Regularizer | EMA Teacher | With Constraint | No Constraints
---|---|---|---|---|---|---
SGCLS | SGDET | SGCLS | SGDET
mR@10 | mR@20 | mR@10 | mR@20 | mR@10 | mR@20 | mR@10 | mR@20
- | - | - | - | - | 27.2 | 28.0 | 16.5 | 20.8 | 40.7 | 50.1 | 20.9 | 29.7
✓ | - | ✓ | ✓ | ✓ | 34.1 | 33.8 | 19.6 | 22.1 | 46.9 | 61.1 | 26.6 | 32.7
- | ✓ | ✓ | ✓ | ✓ | 33.6 | 34.3 | 19.4 | 21.8 | 46.2 | 60.6 | 25.8 | 32.5
✓ | ✓ | - | ✓ | ✓ | 32.2 | 33.4 | 18.1 | 19.8 | 45.9 | 59.1 | 21.8 | 31.6
✓ | ✓ | ✓ | - | ✓ | 35.8 | 36.6 | 21.2 | 22.7 | 48.3 | 61.4 | 27.5 | 34.4
✓ | ✓ | ✓ | ✓ | - | 36.7 | 38.8 | 22.1 | 23.8 | 49.2 | 62.9 | 28.3 | 35.2
✓ | ✓ | ✓ | ✓ | ✓ | 37.4 | 39.2 | 22.6 | 24.2 | 49.7 | 63.8 | 28.6 | 35.4

###  4.2 Comparison with state-of-the-art

We compared our method FloCoDe with several state-of-the-art methods for
dynamic SGG, namely, TEMPURA [31], STTran [6], TRACE [46], STTran-TPI [50],
APT [26], and ISGG [21]. Additionally, we compared our method with ReLDN [58],
which is a static method. Performance comparisons in terms of mR@K and R@K for
K = [10, 20, 50] are reported in Tables 1, 2, and 3. These tables contain
comparisons with two experimental setups: a) With Constraint and b) No
Constraints. With these experimental setups and tasks, i.e., PREDCLS + SGCLS,
we presented results for these in terms of mR@K and R@K in Tables 2 and 3,
respectively. Table 1 compares results for task i.e., SGDET, with the same
metrics and under both experimental setups. Wherever available, we utilise the
source code for the respective SOTA methods for mR@K and R@K values. For
methods with no code, we take the values reported in [31]. From the tables, it
has been observed that our method consistently outperforms other methods
across all the tasks and for both experimental setups. Specifically, in
comparison to the best baselines, we observe improvements of 4.1% on SGDET-
mR@10, 3.4% on SGCLS-mR@10, and 1.9% on PREDCLS-mR@10 under the ”With
Constraint” setup. Under the ”No Constraints” setup, we observe improvements
even more significant, with 3.9% on SGDET-mR@10, 1.4% on SGCLS-mR@10, and 1.7%
on PREDCLS-mR@10. This demonstrated the capability of FloCoDe in generating
more unbiased SGG for videos incorporating dynamic fluctuations and long-
tailed relations. We further verified this in Figure 3 for With Constraint and
No Constraints. In these figures, we compared our method on HEAD, BODY, and
TAIL classes with mR@10 values. We split the classes into HEAD, BODY, and TAIL
with the same definition as mentioned in [31]. Clearly, FloCoDe improved the
performance across all the classes, but the improvement for TAIL classes is
more confirming the unbiased predictions. Per-class performance is shown in
Fig. 4, comparing with other methods STTran and TRACE showing the improvement
at class level. Additionally, our method outperforms in terms of R@K values,
as shown in Table 3, showing improvements overall as compared to existing
methods. This demonstrates that our method has better generalisation since it
performs better on both mR@K (long tail performance) and R@K (overall).
Qualitative visualisations are illustrated in Fig 2.

(a) SGDET

(b) SGCLS

(c) PREDCLS

(d) SGDET

(e) SGCLS

(f) PREDCLS

Figure 3: Comparison of mR@10 for the HEAD, BODY and TAIL classes for ”with constraint”(top) and ”no contraints”(bottom) Figure 4: Comparative per class performance for PREDCLS task in R@10 for ”with constraint” setup Table 5: Results (in mR@10) with varying number of mixtures $\mathcal{K}$ for With Constraint setup Task $\mathcal{K}$ | 1 | 2 | 4 | 6 | 8
---|---|---|---|---|---
PREDCLS | 39.8 | 41.2 | 43.4 | 44.8 | 44.2
SGCLS | 30.2 | 35.5 | 37.4 | 36.2 | 35.8
SGDET | 16.1 | 18.1 | 21.9 | 22.6 | 22.1

###  4.3 Ablation Studies

We have conducted extensive ablation studies on SGCLS and SGDET tasks.
Specifically, we studied the impact of KMCL (uncertainty-aware contrastive
learning), Debiasing (correlation-aware debiasing), TFoD (flow-aware temporal
consistency), Regularizer (aleatoric and epistemic regularizer), and EMA
Teacher. When all these components have been removed, the FloCoDe boils down
to the baseline STTran[6], where the object proposals and predicate embeddings
are fed to the FFN layers before finally predicting the predicate class using
a classification layer. The results for these ablation studies are presented
in Table 4. Uncertainty Attenuation and Debiasing: We first discuss the impact
of uncertainty-aware contrastive learning and correlation-aware debiasing. For
the first case, we remove the loss $\mathcal{L}_{\textsc{KMCL}}$ to study the
improvement on top of the loss based on MLN. With MLN mixtures, a similar
improvement is already demonstrated in [31]. In the second case, we remove the
correlation-aware debiasing during training and train end-to-end without any
debiasing. The results for both of these are present in Table 4 rows 1 and 2.
Clearly, comparing the resulting models with FloCoDe shows the significant
improvement in mR@10 values, showing the value addition from each of them in
generating an unbiased SGG. This also shows that both can address the noise
associated with TAIL classes, while contrastive learning deals with annotation
noise-debiasing focus to generate unbiased predicate embeddings. Temporally
Consistent Object Detection: The effect of flow-aware detection of temporally
consistent objects is shown in row 3 of Table 4. Comparing the performance
with FloCoDe, we can see that without TFoD results in a significant drop in
performance. This highlights the fact that incorrect object detection is the
major bottleneck for any SGG method. For PREDCLS task, it takes only the
ground-truth boxes and labels, and hence the mR@k and R@K values are much
higher as compared to other tasks. Uncertainty Regularizer and EMA Teacher:
Comparing the ablation of these two components from rows 4 and 5,
respectively, confirms the importance of regularizers in further reducing the
noise associated with the TAIL classes. For EMA teachers, it produces more
balanced predicate embeddings, hence providing improvement by just predicting
the unbiased embeddings corresponding to their class. Number of Mixtures
$\mathcal{K}$: The performance of FloCoDe with varying numbers of mixtures in
MLN is shown in Table 5. Number of mixtures b/w 4 to 6 is optimal.

##  5 Conclusion

We demonstrated the correlation debiasing (label correlation and debiased
predicate embedding) and resulting performance of FloCoDe with these debiasing
mechanism prove the fact that dynamic SGG requires more focus on long-tailed
classification rather than complex architecture for temporal sequence
processing.

## References

  * Antol et al. [2015] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh.  Vqa: Visual question answering.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2425–2433, 2015.
  * Araslanov and Roth [2021] Nikita Araslanov and Stefan Roth.  Self-supervised augmentation consistency for adapting semantic segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 15384–15394, 2021.
  * Arnab et al. [2021] Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid.  Vivit: A video vision transformer.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 6836–6846, 2021.
  * Carion et al. [2020] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In _European conference on computer vision_ , pages 213–229. Springer, 2020.
  * Chen et al. [2019] Tianshui Chen, Weihao Yu, Riquan Chen, and Liang Lin.  Knowledge-embedded routing network for scene graph generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6163–6171, 2019.
  * Cong et al. [2021] Yuren Cong, Wentong Liao, Hanno Ackermann, Bodo Rosenhahn, and Michael Ying Yang.  Spatial-temporal transformer for dynamic scene graph generation.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 16372–16382, 2021.
  * Desai et al. [2021] Alakh Desai, Tz-Ying Wu, Subarna Tripathi, and Nuno Vasconcelos.  Learning of visual relations: The devil is in the tails.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 15404–15413, 2021.
  * Dong et al. [2021] Jianfeng Dong, Xirong Li, Chaoxi Xu, Xun Yang, Gang Yang, Xun Wang, and Meng Wang.  Dual encoding for video retrieval by text.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 44(8):4065–4080, 2021.
  * Dosovitskiy et al. [2015] Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, and Thomas Brox.  Flownet: Learning optical flow with convolutional networks.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2758–2766, 2015.
  * French et al. [2019] Geoff French, Samuli Laine, Timo Aila, Michal Mackiewicz, and Graham Finlayson.  Semi-supervised semantic segmentation needs strong, varied perturbations.  _arXiv preprint arXiv:1906.01916_ , 2019.
  * Goyal12 and Sigal123 [2021] Raghav Goyal12 and Leonid Sigal123.  A simple baseline for weakly-supervised human-centric relation detection.  2021\.
  * Han et al. [2022] Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al.  A survey on vision transformer.  _IEEE transactions on pattern analysis and machine intelligence_ , 45(1):87–110, 2022.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * He et al. [2017] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2961–2969, 2017.
  * Hoyer et al. [2021] Lukas Hoyer, Dengxin Dai, Yuhua Chen, Adrian Koring, Suman Saha, and Luc Van Gool.  Three ways to improve semantic segmentation with self-supervised depth estimation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11130–11140, 2021.
  * Ji et al. [2020] Jingwei Ji, Ranjay Krishna, Li Fei-Fei, and Juan Carlos Niebles.  Action genome: Actions as compositions of spatio-temporal scene graphs.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10236–10247, 2020.
  * Ji et al. [2021] Jingwei Ji, Rishi Desai, and Juan Carlos Niebles.  Detecting human-object relationships in videos.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 8106–8116, 2021.
  * Ju et al. [2022] Chen Ju, Tengda Han, Kunhao Zheng, Ya Zhang, and Weidi Xie.  Prompting visual-language models for efficient video understanding.  In _European Conference on Computer Vision_ , pages 105–124. Springer, 2022.
  * Kendall and Gal [2017] Alex Kendall and Yarin Gal.  What uncertainties do we need in bayesian deep learning for computer vision?  _Advances in neural information processing systems_ , 30, 2017.
  * Khandelwal [2023] Anant Khandelwal.  Segda: Maximum separable segment mask with pseudo labels for domain adaptive semantic segmentation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 2158–2168, 2023.
  * Khandelwal and Sigal [2022] Siddhesh Khandelwal and Leonid Sigal.  Iterative scene graph generation.  _Advances in Neural Information Processing Systems_ , 35:24295–24308, 2022.
  * Krishna et al. [2017] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  _International journal of computer vision_ , 123:32–73, 2017.
  * Li et al. [2022a] Rongjie Li, Songyang Zhang, and Xuming He.  Sgtr: End-to-end scene graph generation with transformer.  In _proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 19486–19496, 2022a.
  * Li et al. [2017] Yikang Li, Wanli Ouyang, Bolei Zhou, Kun Wang, and Xiaogang Wang.  Scene graph generation from objects, phrases and region captions.  In _Proceedings of the IEEE international conference on computer vision_ , pages 1261–1270, 2017.
  * Li et al. [2021] Yicong Li, Xun Yang, Xindi Shang, and Tat-Seng Chua.  Interventional video relation detection.  In _Proceedings of the 29th ACM International Conference on Multimedia_ , pages 4091–4099, 2021.
  * Li et al. [2022b] Yiming Li, Xiaoshan Yang, and Changsheng Xu.  Dynamic scene graph generation via anticipatory pre-training.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 13874–13883, 2022b.
  * Lin et al. [2020] Xin Lin, Changxing Ding, Jinquan Zeng, and Dacheng Tao.  Gps-net: Graph property sensing network for scene graph generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3746–3753, 2020.
  * Liu et al. [2020] Chenchen Liu, Yang Jin, Kehan Xu, Guoqiang Gong, and Yadong Mu.  Beyond short-term snippet: Video relation detection with spatio-temporal global context.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10840–10849, 2020.
  * Loshchilov and Hutter [2017] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_ , 2017.
  * Lu et al. [2016] Cewu Lu, Ranjay Krishna, Michael Bernstein, and Li Fei-Fei.  Visual relationship detection with language priors.  In _Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part I 14_ , pages 852–869. Springer, 2016.
  * Nag et al. [2023] Sayak Nag, Kyle Min, Subarna Tripathi, and Amit K Roy-Chowdhury.  Unbiased scene graph generation in videos.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 22803–22813, 2023.
  * Nawhal and Mori [2021] Megha Nawhal and Greg Mori.  Activity graph transformer for temporal action localization.  _arXiv preprint arXiv:2101.08540_ , 2021.
  * Papyan et al. [2020] Vardan Papyan, XY Han, and David L Donoho.  Prevalence of neural collapse during the terminal phase of deep learning training.  _Proceedings of the National Academy of Sciences_ , 117(40):24652–24663, 2020.
  * Park et al. [2023] Jeongeun Park, Seungyoun Shin, Sangheum Hwang, and Sungjoon Choi.  Elucidating robust learning with uncertainty-aware corruption pattern estimation.  _Pattern Recognition_ , 138:109387, 2023.
  * Pu et al. [2023] Tao Pu, Tianshui Chen, Hefeng Wu, Yongyi Lu, and Liang Lin.  Spatial-temporal knowledge-embedded transformer for video scene graph generation.  _arXiv preprint arXiv:2309.13237_ , 2023.
  * Qian et al. [2019] Xufeng Qian, Yueting Zhuang, Yimeng Li, Shaoning Xiao, Shiliang Pu, and Jun Xiao.  Video relation detection with spatio-temporal graph.  In _Proceedings of the 27th ACM international conference on multimedia_ , pages 84–93, 2019.
  * Ren et al. [2015] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  _Advances in neural information processing systems_ , 28, 2015.
  * Sajedi et al. [2023] Ahmad Sajedi, Samir Khaki, Konstantinos N Plataniotis, and Mahdi S Hosseini.  End-to-end supervised multilabel contrastive learning.  _arXiv preprint arXiv:2307.03967_ , 2023.
  * Sigurdsson et al. [2016] Gunnar A Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta.  Hollywood in homes: Crowdsourcing data collection for activity understanding.  In _Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part I 14_ , pages 510–526. Springer, 2016.
  * Snoek et al. [2009] Cees GM Snoek, Marcel Worring, et al.  Concept-based video retrieval.  _Foundations and Trends® in Information Retrieval_ , 2(4):215–322, 2009.
  * Sohn et al. [2020] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li.  Fixmatch: Simplifying semi-supervised learning with consistency and confidence.  _Advances in neural information processing systems_ , 33:596–608, 2020.
  * Sun et al. [2019] Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid.  Videobert: A joint model for video and language representation learning.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 7464–7473, 2019.
  * Tang et al. [2019] Kaihua Tang, Hanwang Zhang, Baoyuan Wu, Wenhan Luo, and Wei Liu.  Learning to compose dynamic tree structures for visual contexts.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 6619–6628, 2019.
  * Tapaswi et al. [2016] Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler.  Movieqa: Understanding stories in movies through question-answering.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 4631–4640, 2016.
  * Tarvainen and Valpola [2017] Antti Tarvainen and Harri Valpola.  Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.  _Advances in neural information processing systems_ , 30, 2017.
  * Teng et al. [2021] Yao Teng, Limin Wang, Zhifeng Li, and Gangshan Wu.  Target adaptive context aggregation for video scene graph generation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 13688–13697, 2021.
  * Tranheden et al. [2021] Wilhelm Tranheden, Viktor Olsson, Juliano Pinto, and Lennart Svensson.  Dacs: Domain adaptation via cross-domain mixed sampling.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 1379–1389, 2021.
  * Tsai et al. [2019] Yao-Hung Hubert Tsai, Santosh Divvala, Louis-Philippe Morency, Ruslan Salakhutdinov, and Ali Farhadi.  Video relationship reasoning using gated spatio-temporal energy graph.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10424–10433, 2019.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Wang et al. [2022] Shuang Wang, Lianli Gao, Xinyu Lyu, Yuyu Guo, Pengpeng Zeng, and Jingkuan Song.  Dynamic scene graph generation via temporal prior inference.  In _Proceedings of the 30th ACM International Conference on Multimedia_ , pages 5793–5801, 2022.
  * Wei et al. [2019] Yinwei Wei, Xiang Wang, Weili Guan, Liqiang Nie, Zhouchen Lin, and Baoquan Chen.  Neural multimodal cooperative learning toward micro-video understanding.  _IEEE Transactions on Image Processing_ , 29:1–14, 2019.
  * Xiao et al. [2021] Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua.  Next-qa: Next phase of question-answering to explaining temporal actions.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 9777–9786, 2021.
  * Xu et al. [2015] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio.  Show, attend and tell: Neural image caption generation with visual attention.  In _International conference on machine learning_ , pages 2048–2057. PMLR, 2015.
  * Xu et al. [2022] Li Xu, Haoxuan Qu, Jason Kuen, Jiuxiang Gu, and Jun Liu.  Meta spatio-temporal debiasing for video scene graph generation.  In _European Conference on Computer Vision_ , pages 374–390. Springer, 2022.
  * Yang et al. [2023] Yibo Yang, Haobo Yuan, Xiangtai Li, Zhouchen Lin, Philip Torr, and Dacheng Tao.  Neural collapse inspired feature-classifier alignment for few-shot class incremental learning.  _arXiv preprint arXiv:2302.03004_ , 2023.
  * Yelleni et al. [2023] Sai Harsha Yelleni, Kumari Deepshikha, PK Srijith, and C Krishna Mohan.  Monte carlo dropblock for modelling uncertainty in object detection.  _Pattern Recognition_ , page 110003, 2023.
  * Zellers et al. [2018] Rowan Zellers, Mark Yatskar, Sam Thomson, and Yejin Choi.  Neural motifs: Scene graph parsing with global context.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 5831–5840, 2018.
  * Zhang et al. [2019] Ji Zhang, Kevin J Shih, Ahmed Elgammal, Andrew Tao, and Bryan Catanzaro.  Graphical contrastive losses for scene graph parsing.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11535–11543, 2019.
  * Zheng et al. [2022] Sipeng Zheng, Shizhe Chen, and Qin Jin.  Vrdformer: End-to-end video visual relation detection with transformers.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 18836–18846, 2022.
  * Zhu et al. [2017a] Xizhou Zhu, Yujie Wang, Jifeng Dai, Lu Yuan, and Yichen Wei.  Flow-guided feature aggregation for video object detection.  In _Proceedings of the IEEE international conference on computer vision_ , pages 408–417, 2017a.
  * Zhu et al. [2017b] Xizhou Zhu, Yuwen Xiong, Jifeng Dai, Lu Yuan, and Yichen Wei.  Deep feature flow for video recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2349–2358, 2017b.