# Co-segmentation Inspired Attention Module for Video-based Computer Vision
Tasks

Arulkumar Subramaniam Jayesh Vaidya Muhammed Abdul Majeed Ameen Athira Nambiar
Anurag Mittal Department of Computer Science and Engineering, Indian Institute
of Technology Madras, IIT P.O., Chennai, Tamil Nadu - 600036, India

###### Abstract

Video based computer vision tasks can benefit from estimation of the salient
regions and interactions between those regions. Traditionally, this has been
done by identifying the object regions in the images by utilizing pre-trained
models to perform object detection, object segmentation and/or object pose
estimation. Though using pre-trained models seems to be a viable approach, it
is infeasible in practice due to the need for exhaustive annotation of object
categories, domain gap between datasets and bias present in pre-trained
models. To overcome these downsides, we propose to utilize the common
rationale that a sequence of video frames capture a set of common objects and
interactions between them, thus a notion of co-segmentation between the video
frame features may equip the model with the ability to automatically focus on
salient regions and improve underlying task’s performance in an end-to-end
manner. In this regard, we propose a generic module called “Co-Segmentation
Activation Module” (COSAM) that can be plugged-in to any CNN to promote the
notion of co-segmentation based attention among a sequence of video frame
features. We show the application of COSAM in three video based tasks namely:
1) Video-based person re-ID, 2) Video captioning, & 3) Video action
classification and demonstrate that COSAM is able to capture salient regions
in the video frames, thus leading to notable performance improvements along
with interpretable attention maps.

###### keywords:

###### MSC:

41A05, 41A10, 65D05, 65D17 attention , co-segmentation , person re-ID , video-
captioning , video classification

††journal: Computer Vision and Image Understanding

##  1 Introduction

Visual scene understanding from video data is an important research problem
due to the large amount of videos generated and published everyday. Visual
scenes captured in terms of a video majorly focus on capturing one or more
objects in terms of their activities and/or interactions between them. Thus,
visual scene understanding is primarily aimed to localize & recognize those
objects and further determine interactions between them, if any. Several
video-based computer vision tasks such as action recognition (Simonyan and
Zisserman (2014a)), visual object segmentation & tracking (Ristani et al.
(2016)), and video captioning (Zhang and Peng (2019)) have been formulated to
reflect this notion. To perform these tasks, earlier works used hand-crafted
visual descriptors such as 2D/3D HoG (Dalal and Triggs (2005)), SIFT (Lowe
(2004)), Optical flow (Horn and Schunck (1981); Bruhn et al. (2005)) to
aggregate visual information from the video frames, followed by a
classification (or) regression layer according to the task at hand. Owing to
the dependency on domain expertise, such hand-crafted features are unable to
scale well in practice. The availability of large-scale datasets and
computational resources have given rise to deep learning methods that have
shown great promise in several of these tasks.

Deep learning approaches perform extremely well on the tasks that has
extensive labeled samples available for training. Though annotated datasets
are available for various tasks, not all datasets have fine-grained
annotations (e.g., object bounding boxes, segmentation masks or salient
regions) to enable the deep network to focus on important portions of image
features. A straightforward approach to get fine-grained annotations and
identify the object regions is to utilize pre-trained models to perform object
detection, object segmentation and/or object pose estimation. However, it is
infeasible in practice majorly due to three reasons: First, the object
categories in the pre-trained models’ training dataset may not cover all the
object categories exhaustively needed for general computer vision tasks. Thus
the object-related cues from pre-trained models may prove to be incomplete for
the underlying tasks. For instance, object detectors may not cover the objects
belonging to the tail distribution such as star fish or baseball bat that may
be present in tasks such as video captioning and video action classification.
Further, the pose estimation (or) pedestrian detection / segmentation models
can effectively locate the person’s key joint locations (e.g., head, torso and
legs). However, they miss out on the salient accessories associated with the
subject (e.g., backpack, bag, hat and coat) that are also important cues for
tasks such as video captioning and video-based person re-identification (Re-
ID).

Second, the deep networks are shown to be susceptible to domain difference
between the training dataset’s distribution and target task’s data
distribution. Learning domain-independent features is a research problem of
its own. Owing to this, the domain gap between the pre-trained models’
training dataset and target task’s dataset may negatively impact the
performance of the underlying task. For instance, in the case of Re-ID,
pedestrian detector / segmentation methods and pose estimation methods are
trained using frames from human activities and taken from YouTube, sports
activity videos. Hence, they may not cover the drastic viewpoint variations in
surveillance scenarios, e.g., top-view. Also, surveillance images may not have
sufficient resolution for stable pose estimation. For the tasks of video
captioning and video-action classification, the object detector / segmentation
methods are pre-trained using MS-COCO and PASCAL VOC datasets. In a realistic
setting, the data distributions of captioning (MSR-VTT, MSVD) and action
classification (HMDB51, UCF101) are different from those pre-trained models’
training data distribution and the performance may be influenced by this
domain gap.

Thirdly, the bias present in pre-trained models may propagate into the target
task leading to an inadvertently biased target model. The deep networks
exhibit different types of biases due to factors such as background, color,
racial (Gwilliam et al. (2021)), gender (Tang et al. (2021); Zhao et al.
(2017)), contextual (Singh et al. (2020)), co-occurrence (Petsiuk et al.
(2021)), spatial noise, dataset (Tommasi et al. (2017)) and object-size
(Nguyen et al. (2020)). For instance, Petsiuk et al. (2021) show that the
object detectors are vulnerable to learning the co-occurrence of an unrelated
adversarial marker. Such biases may lead to a model that doesn’t generalize
well in the target task. Added to these downsides, the pre-trained models are
affected by nuisance variables such as background clutters, pose/viewpoint,
illumination variations, and occlusions.

To alleviate the downsides mentioned above and to achieve localization of
salient regions for the underlying task, we exploit the assumption that the
sequence of frames belonging to a video often capture a set of common
object(s) and interactions among them. Specifically, we propose to supplement
the deep network with the ability to attend to the common set of salient
regions between the frames and hypothesize that this may help the network to
improve the performance in an end-to-end manner. In this regard, we propose a
novel generic attention module namely ”Co-Segmentation Activation
Module(COSAM)” for video-based applications. In particular, our proposed COSAM
module facilitates intra-video attention inspired by the co-segmentation
concept (Vicente et al. (2011); Li et al. (2018c)) by leveraging correlated
information in multiple frames of the same video. COSAM is capable of
activating task-relevant object regions depending on the performed vision
task. In this work, we show the application of the proposed COSAM module in
three major computer vision tasks: 1) Video-based person Re-ID, 2) Video
captioning, and 3) Video action classification. Although we (Subramaniam et
al. (2019)) originally proposed the idea towards video-based person Re-ID
task, in this work, we extend the application domains to demonstrate the
applicability of a generic ”co-segmentation based attention” in various vision
tasks.

As a general principle, we evaluate the effectiveness of the COSAM module by
incorporating it into recent state-of-the-art models in these three major
video-based computer vision tasks. We outline the contributions of the paper
as follows:

  * •

Video-based person Re-ID:  Person re-identification (Re-ID) is the task of
associating a person’s image(s) among two or more camera views. It is posed as
a fine-grained retrieval problem where the comparative scores between the
query and the gallery instances are sorted to determine the best matched (top
$k$) gallery instances for each query instance. In Subramaniam et al. (2019),
we proposed to incorporate the COSAM module into a typical video-based Re-ID
pipeline (Sec. 4.1.1) and demonstrated that the model performs superior to the
state-of-the-arts. In this paper, along with Subramaniam et al. (2019), we
extend the experimental analysis to compare COSAM to the closely related non-
local attention modules (Wang et al. (2018d)) to compare and contrast them in
terms of number of parameters, FLOPs and quantitative performance.

  * •

Video captioning: Video captioning is the task of generating human
understandable text about the given video automatically by capturing spatial &
temporal visual context along with a suitable mapping function to map the
visual context into the text modality. In our work (Vaidya et al. (2022)), we
consider the baseline as the state-of-the-art method (Pan et al. (2020)) that
utilizes a pre-trained object detector (Ren et al. (2015)) to obtain object-
related features to aid video-captioning. We replace the object detector based
local branch by incorporating the COSAM module to automatically capture task-
specific salient regions. Through extensive experiments, we evaluate the
performance of COSAM-based model and show that it is indeed possible to
perform on par or better than state-of-the-art methods without using object
detectors in the video captioning task.

  * •

Video action classification: Video classification is one of the fundamental
research topics in computer vision and considered as a natural extension of
the image classification task to videos i.e., associate the video to one of
the predefined classes. In this work, we make use of the recent state-of-the-
art video classification model (Kalfaoglu et al. (2020)) as the baseline.
Specifically, we demonstrate the significance of the COSAM module on this task
by simply plugging it into the baseline 3D CNNs with two different backbones
(I3D, ResNeXt-101) and show that COSAM is able to improve the performance of
baseline architectures by capturing salient object regions in video frames.

The paper is organized as follows. Section 2 details some of the related
works, Section 3 briefly describes the co-segmentation concept that our work
is based upon and formulates a generic co-segmentation based attention module
to activate common salient regions among a set of frames, Section 4
illustrates the applicability of COSAM module on three video based computer
vision tasks, then Section 5 shows the experiments performed on these tasks
along with the comparison of state-of-the-art methods. Finally, section 6
concludes with the summary and possible future directions.

##  2 Related works

###  2.1 Evolution of attention modules

In the early adoption of deep convolutional neural networks (DCNN) for image
recognition, the model predominantly consisted of a sequence of convolution
layers (typically convolution + BatchNorm + ReLU) followed by a linear
classifier to classify the input (images / videos). For instance, AlexNet
(Krizhevsky et al. (2012)) uses 5 conv layers followed by 2 linear layers,
VGG16 (Simonyan and Zisserman (2014b)) uses 16 conv layers followed by 2
linear layers, ResNet (He et al. (2016)) uses 5 conv blocks followed by
average pooling and a linear classifier in which each conv block is made up of
a sequence of conv layers with residual connections.

Explicit feature activation | SE-ResNet (Hu et al. (2018))
---|---
BAM (Park et al. (2018))
CBAM (Woo et al. (2018))
DiverseAttention (Li et al. (2018b))
COSAM (Ours)
Correlation and feature aggregation | ShowAttendTell (Xu et al. (2015))
SelfAttention (Vaswani et al. (2017))
NonLocal (Wang et al. (2018d))
BERT (Devlin et al. (2018))
CrissCrossAttention (Huang et al. (2019))
Table 1: Evolution of attention methodologies in the literature

Recent works try to improve the representation learning ability of backbone
networks by augmenting it with “attention” blocks (spatial, channel, temporal)
to improve representation learning. Based on the nature of attention, we
broadly categorize these methods into two types namely, 1) Explicit feature
activation, and 2) Correlation and feature aggregation methods as shown in the
Table 1.

Explicit feature activation (EFA) based methods strive to estimate an
“importance” mask (typically $\sim$[0, 1]) denoting the importance of the
input features and use these masks to reweigh the input features. The
motivation is to learn to boost the features that are important for the task
and suppress the noisy features in an end-to-end manner via back-propagation.
For instance, SE-ResNet (Hu et al. (2018)) proposed to learn a channel
attention from global average pooled (GAP) feature maps. In a similar spirit,
Block Attention Module (BAM) (Park et al. (2018)) proposed to learn a spatial
attention and channel attention masks at bottleneck layers of ResNet.
Different from BAM, Convolutional Block Attention Module (Woo et al. (2018))
proposed to use global statistics of feature maps such as average and max
pooled channel descriptors to learn channel attention followed by spatial
attention. Li et al. (2018b) proposed to learn diverse attention masks using
multiple branches to focus on salient regions for image-based Re-ID. Our work
aligns with this paradigm where COSAM module learns to estimate spatial and
channel importance (attention) masks conditioned on correlation cost volume
between current frame and a set of reference frames.

On the other hand, Correlation and feature aggregation (CFA) methods estimate
the mutual attention weights among the set of input features (e.g., word-level
features in Natural language processing (NLP), pixel-level features in a 2D
image, frame-level features in a video) and then, for each input feature, the
output feature is estimated by weighted sum of input features with attention
weights. Xu et al. (2015) is one of the earlier works in this paradigm for the
task of image captioning where at each time step of caption generation, the
input context is gathered using a CFA construct between the hidden layer
features of last time step and image features. Extending this style of feature
interaction, Vaswani et al. (2017) formulated a self-correlation and feature
aggregation method called “self-attention” for NLP tasks. In Vaswani et al.
(2017), three projections are obtained from each input features namely, query,
key, value. Then correlation is carried out between query and key to get
attention weights, followed by feature aggregation step to aggregate weighted
sum of values for each input feature based on attention weights. BERT (Devlin
et al. (2018)) follows the self-attention paradigm to learn a language model
by predicting missing words in a sentence. For video-based tasks, Non-local
attention module networks (NLM) (Wang et al. (2018d)) propose to aggregate the
frame features based on cross attention probabilities between frame-wise
pixel-level representations. The concept of COSAM is close to NLM, however
COSAM proposes to explicitly activate the frame-wise features at each pixel by
learning spatial and channel attention weights instead of aggregating the
features. By doing this, COSAM promotes feature interactions between multiple
frames in an interpretable and computationally effective way in terms of
number of parameters and FLOPs (refer Sec. 5.1.4 for more information).

###  2.2 Object co-segmentation approaches

Depending on the underlying algorithm, the co-segmentation approaches can be
grouped into two types: 1) Graph-based (Chang and Wang (2015); Jerripothula et
al. (2016); Li et al. (2018a)), and 2) Clustering-based (Joulin et al. (2012);
Tao et al. (2017)). The graph-based methods exploited the shared structural
representation of object instances from the given images to jointly segment
the common objects, while the cluster based methods approach co-segmentation
by grouping pixels/super-pixels to determine common object regions. Earlier
methods (Rother et al. (2006); Vicente et al. (2011)) used hand-crafted
feature descriptors such as SIFT (Lowe (2004)) and HOG (Dalal and Triggs
(2005)) to represent object instances, while the recent methods increasingly
use deep learning approaches. Among the deep learning based methods, Li et al.
(2018c) used a deep model to co-segment the regions based on the semantic
similarity. Hsu et al. (2018) proposed to co-segment a particular category of
objects in an unsupervised way by formulating contrastive objective functions
to differentiate between foreground and background. Further, some methods
(Chen et al. (2018b); Li et al. (2018c)) proposed spatial and/or channel-wise
attention-based bottleneck layers in an encoder-decoder architecture to
activate semantically related features. Despite having a rich literature, the
co-segmentation concept in other computer vision tasks is under explored, and
our work marks one of the first attempts exploring the application of co-
segmentation to aid other vision tasks.

###  2.3 Person re-identification

Earlier approaches formulated the solutions by extracting hand-crafted
features (Bazzani et al. (2014); Ma et al. (2012)) followed by suitable metric
learning (Bak and Carr (2017); Prosser et al. (2010); Hermans et al. (2017))
for the re-identification task. Recent approaches utilize deep CNN models
(Ding et al. (2015); Fu et al. (2018); Xiao et al. (2017); Hermans et al.
(2017); Zheng et al. (2018)) to extract features and learn suitable metric in
an end-to-end manner. A common approach in many of the video Re-ID approaches
is to extract frame-level features followed by temporal modelling / feature
aggregation and classification, i.e., LSTM/pooling (McLaughlin et al. (2016);
Zhou et al. (2017); Liu et al. (2018)). For instance, McLaughlin et al. (2016)
& Chung et al. (2017) use deep CNNs to extract features from RGB & optical
flow and a recurrent layer followed by a temporal average pooling (TPavg) to
aggregate features. These methods are easily affected by noisy background
clutters, occluded objects and misalignment of the person that results in a
drastic drop of Re-ID accuracy.

The solutions to overcome these nuisance factors broadly follow three types of
approaches: 1) using pose estimation methods to focus on the spatial region of
the person and avoid features from the background (Su et al. (2017), Suh et
al. (2018)), 2) using segmentation mask of the subject i.e., person from a
pre-trained segmentation model (Ren et al. (2015); He et al. (2017)) to avoid
background clutter and extract features only from the foreground (Qi et al.
(2018); Song et al. (2018a)), 3) using attention-driven approaches to focus on
salient regions in an end-to-end manner (Li et al. (2018b); Wang et al.
(2016b)). Among these approaches, using pre-trained pose and segmentation
models have the following downsides: 1) The cues from pose and segmentation
methods may help to locate the person’s joint locations (e.g., head, torso and
legs), however they miss out important salient accessories associated with the
person (e.g., backpack, bag, hat and coat) that are also important cues for
Re-ID, 2) pre-trained models may not cover the drastic pose/viewpoint
variations in surveillance scenarios, e.g., top-view, 3) Surveillance images
are typically of low resolution and may not be sufficient for reliable pose
estimation / segmentation. Further, the attention models (Zhou et al. (2017);
Li et al. (2018b); Wang et al. (2016b)) are still sub-optimal since they work
on “per-frame” basis without using rich spatiotemporal cues in the video.
Different from these methods, in our work, we utilize spatiotemporal features
to co-segment the salient regions and activate them in an unsupervised way to
aid the Re-ID task.

###  2.4 Video captioning

A typical methodology followed by deep learning methods for video captioning
task (Pan et al. (2016); Venugopalan et al. (2015); Rohrbach et al. (2017);
Venugopalan et al. (2014)) is to extract global 2D/3D CNN abstract features
(Simonyan and Zisserman (2014b); He et al. (2016); Xie et al. (2017); Carreira
and Zisserman (2017)) and then, use a temporal modeling scheme (i.e., LSTM) to
model the temporal relationship between frames and then use a language decoder
to generate captions (Hochreiter and Schmidhuber (1997); Chung et al. (2014);
Vaswani et al. (2017)). Though the results from global 2D/3D CNN features are
promising, they fail to capture local features needed for captioning. To gain
fine-grained local cues, some methods (Zhang and Peng (2019); Pan et al.
(2020); Zhang et al. (2020); Ma et al. (2018); Zhou et al. (2019)) have used
pre-trained object detectors (Ren et al. (2015); He et al. (2017); Redmon et
al. (2016)) to extract local object features and shown significant performance
improvement. Following this paradigm, one of the recent methods (Pan et al.
(2020)) proposed a two-branch network where a “global scene branch” extracts
global features using 2D and 3D CNNs and a “local branch” extracts object
specific features using a pre-trained object detector.

Though object detectors support video-captioning models with prior information
on objects, we perceive the following downsides: 1) Object detection datasets
may not exhaustively capture all the object categories needed for the
captioning task, 2) In a realistic setting, the performance may be influenced
by the domain gap (Chen et al. (2020); Wu et al. (2021)) between the object
detection dataset and the visual-captioning dataset, 3) The pre-trained object
detectors may introduce bias in the captioning model towards the object
categories that it has been trained on. Due to these reasons, the performance
of the captioning task may be capped by the performance of pre-trained models
in the video captioning dataset. In our work, we explore an alternative
solution to eliminate the use of object detector by equipping the model with
the COSAM module to learn and focus on salient regions in an end-to-end
manner.

###  2.5 Video classification

A typical pipeline for video classification task involves three core
components: 1) A feature extractor, 2) Temporal modeling layer and 3)
Classification layer. The feature extractor is used to capture abstract
features from the given video frames and can consist of a 2D (Krizhevsky et
al. (2012); He et al. (2016)) or 3D CNN with one (Karpathy et al. (2014)) or
multiple branches (Simonyan and Zisserman (2014a)) depending on the nature of
the input (i.e., RGB and/or optical flow). Further, the temporal modeling
layer learns temporal interactions between frame-wise features and summarizes
the features. Naive pooling (Wang et al. (2016a)), recurrent aggregation (Wu
et al. (2015)) or RNN/LSTM layers (Donahue et al. (2015)) are some of the
prevalent temporal modeling layers. Finally, the classification layer takes
the summarized features as input and classifies into a set of predefined
categories. Typically a set of linear layers with output nodes equal to the
number of classes is used as a classification layer.

Initial attempts (Yue-Hei Ng et al. (2015); Donahue et al. (2015)) to find
better temporal modeling scheme either used a naive temporal average pooling
or RNN/LSTM layers to capture temporal information. Recent methods use
attention mechanism to model temporal dependency due to its strong empirical
performance in other vision and NLP tasks. Kalfaoglu et al. (2020) used a
BERT-style self-attention mechanism and pooling strategy to learn temporal
modeling and shown promising performance improvement. In our work, we adopt
Kalfaoglu et al. (2020) as our baseline network and attempt to study the
effectiveness of the COSAM module in this method.

Striving towards better backbone networks, 3D CNNs have emerged as a natural
choice considering its inherent nature to capture spatiotemporal information.
Several works (Yao et al. (2015); Carreira and Zisserman (2017); Diba et al.
(2017)) utilize 3D CNNs to improve the performance by efficiently capturing
spatiotemporal context from both appearance (RGB) and motion (optical flow)
data. Recent methods employ attention schemes (self-attention / cross-
attention) (Xu et al. (2015); Girdhar et al. (2019); Chi et al. (2020); Wang
et al. (2018d)) and graph convolutional networks (GCN) to capture rich
contextual information between video frames. For instance, Barrett and Siskind
(2015) proposed to automatically identify the most discriminative temporal
sub-sequences in a video. Wang et al. (2018d) introduced non-local blocks to
aggregate the features from video frames by means of correlating frame
features. In our work, we attempt to incorporate the COSAM module into a
recent state-of-the-art (Kalfaoglu et al. (2020)) to automatically learn
salient regions for video classification by means of co-segmentation inspired
attention.

##  3 Co-segmentation activation module (COSAM)

###  3.1 Concept of object co-segmentation

Object co-segmentation is the task of identifying and segmenting common
objects from two or more images according to “some” common characteristics
(Vicente et al. (2011); Li et al. (2018c)) such as the similarity of the
object-class and appearance. An illustration of co-segmentation is depicted in
Fig. 1.

Fig. 1: An example illustration of object co-segmentation using images from
the Caltech-UCSD Birds 200 (Welinder et al. (2010)) dataset.

Inspired by this task, we encompass an assumption that a sequence of frames
belonging to a video often capture a common set of objects and interactions
between them. Thus, learning to attend to common salient regions between
frames in an end-to-end manner may aid a deep network to improve the
underlying task’s performance. To achieve this, we propose to incorporate some
common saliency associated with the captured object(s) among video frames that
can enhance the features from the objects and suppress irrelevant background
features. In this regard, we propose a novel Co-Segmentation Activation Module
(COSAM) that can be plugged-in into a variety of CNN models.

Fig. 2: Illustration of Co-Segmentation Activation Module (COSAM). The
exemplary images are taken from the MARS (Zheng et al. (2016)) person re-
identification dataset. (a) In the COSAM spatial attention step, the
dimensionality reduced ($D\rightarrow D_{R}$) feature maps are passed through
Normalized Cross Correlation (NCC) matching layer and a summarization layer to
get the corresponding spatial attention mask. (b) In the COSAM channel
attention step, the common channel characteristics are used to activate the
common features and suppress the noisy features.

We briefly review two deep networks in the literature for co-segmentation that
have inspired our work. Li et al. (2018c) proposed a siamese architecture with
an encoder and a decoder to co-segment the common objects in a supervised
learning framework. In this process, two or more images are passed through the
encoder to get abstract feature maps. Then, in the bottleneck layer, mutual
correlations of spatial feature descriptors are calculated to measure the
commonality between the image features and this correlation based cost matrix
is further passed to the decoder to estimate the co-segmentation mask. The
same work also mentioned the idea of group co-segmentation to handle a group
of images simultaneously. In another work, Chen et al. (2018b) explored co-
segmenting images by conditioning on common channel activations. To achieve
this, they proposed a Siamese encoder-decoder architecture in which the
channel activations of one image are conditioned on the channel activations of
the other image (in image-pairs) in the bottleneck layer. For co-segmenting
among a group of images, the channel activations are conditioned by taking the
average channel activations of all the images. Our COSAM layer is inspired by
the group co-segmentation approach from both of these papers, but is
reformulated for video-based tasks.

The motivation of the COSAM module is to activate common salient regions
between image frames of a video conditioned on the correlation cost volume
between the frames’ features to aid the underlying task. The COSAM module
takes the frame features $\\{F_{n}\\}_{n=1}^{N}$ of a video $V$ with dimension
$N\times D\times H\times W$ as input and outputs the activated features of the
same dimension as of the input ($N\times D\times H\times W$). Here, $N=$
number of input frames of the video, $D=$ number of channels in the feature
maps, $H=$ height, and $W=$ width respectively. Further, the COSAM module is
carefully designed to make it convenient to be plugged-in inside any deep CNN.
The functionality of the COSAM module illustrated in Fig. 2 consists of two
steps, namely: 1) Spatial attention, and 2) Channel attention.

###  3.2 COSAM Spatial Attention

In the spatial attention step (Fig. 2a), the goal is to estimate a spatial
mask for each frame that only activates the salient spatial locations
consulting with a set of reference frames. To achieve this, first the features
of the input frames with dimension $N\times D\times H\times W$ are passed
through a dimensionality reduction layer consisting of a $1\times 1$
convolution layer followed by BatchNorm (Ioffe and Szegedy (2015)) and ReLU
(Nair and Hinton (2010)) to perform dimensionality reduction ($D\rightarrow
D_{R}$, $D_{R}<<D$). Dimensionality reduction is specifically carried out to
reduce the computational overhead. Next, to activate salient locations in the
feature maps, we utilize the concept of co-segmentation (Li et al. (2018c))
among the dimensionality reduced features to induce the notion of common
objects. Specifically, for each frame’s feature map with dimension
$D_{R}\times H\times W$, we consider the channel-wise feature vector at every
spatial location $(i,j)$ ($1\leq i\leq H,1\leq j\leq W$) as a $D_{R}$
dimensional local descriptor of the frame at location $(i,j)$, denoted by
$F_{n}^{(i,j)}$. Further, for each feature map $F_{n}$, we select $K$
reference feature maps $\\{R_{k}\\}_{k=1}^{K}$ ($1\leq K\leq(N-1)$) from the
remaining frames. To match the local regions across frames, for each frame
feature map $F_{n}$ and its location $(i,j)$, we compare the local descriptor
$F_{n}^{(i,j)}$ with all the local descriptors of its corresponding reference
feature maps $R_{k}^{h,w}$ ($1\leq k\leq K,1\leq h\leq H,1\leq w\leq W$) to
get a cost volume of dimension $(KHW)\times H\times W$. Here, the comparison
is carried out using Normalized Cross Correlation (NCC) between the local
descriptors as it is robust to illumination variations and this was found to
be more robust than a simple correlation (Subramaniam et al. (2016)). The
comparison results are reshaped into a 3D cost volume where each spatial
location $(i,j)$ holds the comparison values. The idea of creating a cost
volume in terms of matching the descriptors in an end-to-end learning
framework has also been employed in other Computer Vision tasks such as
geometric matching (Rocco et al. (2017)), image-based Re-ID (Subramaniam et
al. (2016)) and stereo matching (Kendall et al. (2017)) among others.

Fig. 3: A standard Video-based Re-ID framework containing (a) a Feature
extractor and (b) Temporal feature aggregation components.

Mathematically, cost volume creation can be defined as:

$\begin{split}\text{Cost
volume}_{(n)}(i,j)=&\\{NCC\left(F_{n}^{(i,j)},R_{k}^{(h,w)}\right)\\\ &1\leq
k\leq K,1\leq h\leq H,1\leq w\leq W\\}\end{split}$ (1)

Given two descriptors $P,Q$ of $D_{R}$ dimension, the NCC operation is defined
as:

$\begin{split}\text{NCC}(P,Q)&=\frac{1}{D_{R}}\frac{\sum_{k=1}^{D_{R}}(P_{k}-\mu_{P}).(Q_{k}-\mu_{Q})}{\sigma_{P}.\sigma_{Q}}\end{split}$
(2)

Here, $(\mu_{P},\mu_{Q})$ denote the mean of the descriptors $(P,Q)$, &
$(\sigma_{P},\sigma_{Q})$ denote the standard deviations of the descriptors
$(P,Q)$ respectively. (A small value of $\epsilon=1e^{-4}$ is added to
$\sigma^{\prime}s$ to avoid numerical instability).

The cost volume of each frame is summarized using a $1\times 1$ convolution
layer followed by a sigmoid activation resulting in a spatial mask (dimension
$1\times H\times W$) for the corresponding frame. The spatial mask is
multiplied with the corresponding frame’s original input features $F_{n}$ to
activate only the local regions of images that are in consensus with its
reference frames. The output features after the spatial attention step are
passed on to the channel attention step.

###  3.3 COSAM Channel Attention

In the channel attention step (Fig. 2 (b)), the goal is to boost the common
important channels between the frames and suppress the noisy channels. To
achieve this, inspired by (Chen et al. (2018b); Hu et al. (2018)), we apply a
Global Average Pooling (GAP) layer on the spatially-attended feature maps from
the spatial-attention step and get a 1D feature of dimension $D$ for each
frame. This resulting feature vector is passed through an Encoder-decoder
style Multi-Layer Perceptron (MLP) followed by sigmoid activation to get a
$D-$dimensional vector signifying the importance of each channel for every
frame. The obtained channel importance vectors of all the $N$ frames are
average pooled together on each channel dimension to estimate the global
channel-importance. In essence, this step boosts the channels that are
important across all the frame features and suppresses the noisy channels.
Next, the averaged channel importance vector is then multiplied with
spatially-attended features to obtain the importance-weighted channel
activations that are passed to the next layer.

Fig. 4: COSAM-based model architecture for the task of video-based person re-
identification. COSAM modules are inserted between consecutive ResBlocks to
induce a notion of co-segmentation based attention.

##  4 Applications of COSAM

COSAM is carefully designed as a generic module to be plug-and-play compatible
within the CNN models in various video-based computer vision tasks. In this
work, we show the applicability of COSAM on three exemplary computer vision
tasks namely: 1) Video-based person re-identification, 2) Video captioning,
and 3) Video action classification. Specifically, we study the COSAM module by
incorporating it into state-of-the-arts methods for these tasks. The following
sections (4.1, 4.2, and 4.3) describe the baseline architectures for each of
these tasks and the modifications carried out to incorporate COSAM into the
baseline architectures.

###  4.1 Video-based person re-identification

In this section, we outline the integration of the COSAM module into a video-
based person re-identification pipeline.

####  4.1.1 Typical video-based Re-ID pipeline

We follow a recent line of research yielding a strong baseline in video-based
Re-ID that can be summarized into a template framework as shown in Fig. 3. It
consists of two primary components :  (a) A Feature extraction network: This
is capable of extracting a meaningful abstract spatial representation from
video frames either by hand-crafted features (SIFT, LBP, HoG, etc.) or
automatically extracted deep CNN features by using pre-trained ImageNet models
(He et al. (2016); Hu et al. (2018)) such as ResNet and SE-ResNet.  (b)
Temporal feature aggregation: Here, the extracted frame-level features are
aggregated to form a video-level feature vector to represent the person
identity in the video. The complexity of feature aggregation techniques in the
literature varies from a _simple_ temporal pooling (TPmax/avg) operation
(average/max pooling) to _complex_ temporal attention (TA) and recurrent layer
(RNN) based aggregation (Gao and Nevatia (2018)). The aggregated video-level
feature vectors are then used to compare (using $L_{2}$ distance or a learned
metric) against other video instances for matching and retrieval purpose. In
Table 2, we give a summary of prior work in video-based Re-ID using the
aforementioned framework.

Literature work
|  Feature extractor |  Feature aggregation
---|---|---
RCN for Re-ID (McLaughlin et al. (2016)) | Custom 3-layer CNN |  RNN + TPavg
Two Stream Siamese (Chung et al. (2017)) | Custom 3-layer CNN |  RNN + TPavg
Jointly attentive spatiotemporal pooling (Xu et al. (2017)) |  deep CNN + spatial pyramid pooling | Attentive TP
Comp. Snippet Sim. (Chen et al. (2018a)) | ResNet-50 |  LSTM, Co-attentive embedding
Part-Aligned (Suh et al. (2018)) | GoogLeNet | Bilinear pooling
Table 2: A collection of approaches in the literature that are following the
video-based Re-ID pipeline shown in Fig. 3.

In the following section, we describe the network architecture incorporated
with the COSAM module following the template framework in Fig. 3.

####  4.1.2 Network architecture

Modern state-of-the-art network architectures (ResNet50, SE-ResNet, etc.,)
that are used as feature extractors in video-based Re-ID contain multiple
consecutive CNN blocks, in which the convolution layers are grouped according
to the resolution of output feature maps: ResNet50 and SE-ResNet50 have five
blocks (one initial convolution block followed by four consecutive Residual
(or) Squeeze and Excitation (SE) residual blocks). We propose to plug-in the
COSAM layer after the CNN blocks in these network architectures. An
illustration of the proposed network architecture along with the COSAM layer
is shown in Fig. 4. After getting the output of every CNN block, the feature
extractor employs a COSAM layer to co-segment the features and then the co-
segmented features are passed to the next CNN block. At the end of the feature
extractor, the temporal aggregation layer (TPavg or TA or RNN) is applied to
summarize the frame-level descriptors into a video-level descriptor. This
video-level descriptor is used to predict the probability that the video
belongs to a particular person identity.

####  4.1.3 Objective functions

For a fair comparison with the baseline (Gao and Nevatia (2018)) and due to
their suitability for our task, we use the same loss functions as in Gao and
Nevatia (2018). The overall loss function can be written as:

$L=\sum_{i=1}^{B}\left\\{L_{CE}+\lambda
L_{triplet}(I_{i},I_{i_{+}},I_{i_{-}})\right\\}$ (3)

Fig. 5: Illustration of the proposed model architecture. The model consists of
two branch viz., 1) Global Scene branch (GSB) - Sec. 4.2.2, and 2) Co-
segmentation branch (CoSB) - Sec. 4.2.3. Both branches have a separate
Transformer as decoder (Sec. 4.2.4). GSB captures global scene by using 2D and
3D CNN global descriptors. CoSB attends to salient regions via COSAM module
(Sec. 3) and promotes interaction among salient regions through SRIM (Sec.
4.2.3). Generated captions are evaluated using standard cross entropy loss.
Further, the salient region information is distilled from CoSB to GSB via KL-
divergence loss. Here, GAP = Global average pooling, $\bigoplus$ = feature
concatenation.

Here, $L_{CE}$ & $L_{triplet}$ refer to the cross-entropy loss and batch
triplet loss respectively and $\lambda$ refers to the trade-off parameter
between the losses (we use $\lambda=1$, as per Gao and Nevatia (2018)), $B$ =
batch size & $(I_{i},I_{i_{+}},I_{i_{-}})$ refer to the $i$th image in the
batch and its hard positive and hard negative pair within the current batch,
respectively.

##### Cross-Entropy loss ($L_{CE}$):

This supervised loss is used to calculate the classification error among the
identities. The number of nodes in the softmax layer depends on the number of
identities in the training set.

##### Batch triplet loss ($L_{triplet}$):

To reduce the intra-class variation and to increase the inter-class variation,
the training instances are formed as a triplet where each triplet contains an
anchor, a positive instance that belongs to the same class as the anchor and a
negative instance that belongs to a different class than the anchor. Hard
negative mining is carried out on the fly in each batch to select the hardest
examples that pose a challenge for the model. Let
$\\{f_{I_{A}},f_{I_{+}},f_{I_{-}}\\}$ be the video-level descriptors of a
triplet, where $I_{A},I_{+},I_{-}$ are the anchor, positive and negative
examples respectively. The triplet loss function is defined as:

$\begin{split}L_{triplet}(I_{A},I_{+},I_{-})=\max\\{&D\left(f_{I_{A}},f_{I_{+}}\right)\\\
&-D\left(f_{I_{A}},f_{I_{-}}\right)+m,0\\}\end{split}$ (4)

Here, $m$ is the margin between the distances, $D(i,j)$ denotes the distance
function between two descriptors $i,j$.

The Cross-entropy loss function is applied on the softmax probabilities
obtained for the identities and the batch triplet loss is applied on the
video-level descriptors to back-propagate the gradients. The experimental
results along with the comparison of state-of-the-art methods are shown in
Section 5.1.

###  4.2 Video captioning

In this section, we describe the application of the COSAM module to the task
of video captioning. Object detectors are commonly used in the video
captioning literature to obtain object-specific features. In our work, we
propose to eliminate the usage of pre-trained object detectors from a state-
of-the-art video captioning pipeline (Pan et al. (2020)) by augmenting it with
COSAM. We hypothesize that the dependency on pre-trained object detectors
could be avoided if the model is supplemented with the ability to focus on
salient object regions automatically.

####  4.2.1 Co-segmentation aided two-stream architecture

The overall model architecture is illustrated in Fig. 5. It follows a two-
branch architecture: 1) Global scene branch (GSB) for capturing global scene
cues, 2) Co-segmentation branch (CoSB) for capturing salient region features
based on co-segmentation between frame-level spatial features. Each branch
follows an encoder-decoder architecture. First, the input frames
$\\{F_{i}\\}_{i=1}^{T}$ of a video $V$ with dimension $T\times 3\times H\times
W$ are passed through GSB and CoSB simultaneously to predict the captions
individually. Here $F_{i}$ = $i^{th}$ frame, $T$ = number of frames, $3$
channels belong to RGB, $H$ = height, and $W$ = width of the frames
respectively. Next, we employ cross-entropy loss to evaluate the predicted
captions from both the branches. Further, we employ a KL-divergence loss to
impose a knowledge distillation constraint that requires both the branches to
predict the same captions with similar confidence level. This ensures that the
knowledge of learned salient regions from CoSB is propagated to GSB. Pan et
al. (2020) follows a similar approach. However, they make use of an object-
detection branch that requires a pre-trained object detector. In the following
sections, we describe the constituting components of the model.

Fig. 6: Co-Segmentation branch (CoSB) consists of two sub-modules COSAM and
SRIM. 1) COSAM performs spatial and channel attention on the input feature
map(s), 2) Next, Salient Region Interaction Module (SRIM) promotes information
propagation between salient regions via multi-head self-attention mechanism.
Here, $\bigotimes$ = point-wise multiplication, $\bigoplus$ = feature
addition.

####  4.2.2 Global scene branch (GSB)

The goal of GSB is to capture the global scene-level information to aid the
captioning task. To achieve this, we utilize an ImageNet (Deng et al. (2009a))
pre-trained 2D CNN to capture frame-wise features and a Kinetics (Kay et al.
(2017)) pre-trained 3D CNN to capture temporal features of the video.
Specifically, given the input frames $\\{F_{i}\\}_{i=1}^{T}$ of dimension
$T\times 3\times H\times W$, we pass the frames through a 2D ResNet-101 (He et
al. (2016)) pre-trained on ImageNet to obtain frame-wise features of dimension
$T\times 2048$ after global-average pooling (GAP) layer. The input frames are
also passed through a 3D ResNeXt-101 (Xie et al. (2017)) pre-trained on
kinetics dataset to obtain temporal features of dimension $T\times 2048$ from
its fully-connected layer before the classification layer. These per-frame 2D
and 3D features are concatenated together and are projected to 512 dimensions
to result in features $F_{GSB}$ of dimension $T\times 512$ for each video.

####  4.2.3 Co-segmentation branch (CoSB)

Global-scene information captured by GSB (Sec. 4.2.2) may not be sufficient
for the task of video captioning which requires fine-grained understanding of
the video. Hence, different from GSB, the goal of CoSB is to focus on salient
spatial regions of the frames that serves the task better. To this end, we
propose to employ a COSAM module (Section 3) to focus on salient regions.
Specifically, the CoSB branch (Fig. 6) takes spatial feature maps of dimension
$T\times C_{L}\times H_{L}\times W_{L}$ and uses a COSAM module to capture
salient regions of frames, followed by a salient-region interaction module
(SRIM) to capture interaction between those regions. Here, $C_{L},H_{L},W_{L}$
= number of channels, height, width of feature maps after $L^{th}$ CNN block.
In our work, we obtain these input feature maps from 2D ResNet-101 used in GSB
to share weights and reduce the model complexity. We will briefly mention the
architecture of the COSAM and SRIM modules in the following sections.

##### Integration of COSAM

In the CoSB branch, first we apply the COSAM module within adjacent frames’
spatial feature maps to capture the salient regions. As demonstrated in
Section 3, COSAM consists of two consecutive attention blocks namely: 1)
Spatial attention block (COSAM-SAB)  to activate common spatial salient
regions and suppress non-informative regions, 2) Channel attention block
(COSAM-CAB)  to activate common informative channels. The design of COSAM-SAB
and COSAM-CAB follows the architecture shown in Fig. 2. Specifically, the
input feature maps of dimension $T\times C_{L}\times H_{L}\times W_{L}$ are
passed through COSAM-SAB to generate a spatial attention mask of dimension
$T\times 1\times H_{L}\times W_{L}$. For each frame $F_{i}$, COSAM-SAB
correlates every location’s feature with its adjacent frame features (i.e.,
$F_{i-1},F_{i+1}$) and creates a cost volume. A summary convolution layer
takes the cost volume as input and produces a spatial attention mask. The
spatial mask is multiplied with the corresponding original features to obtain
spatially refined features. Further, COSAM-CAB takes the spatially refined
features as input, then through a GAP followed by an MLP, it produces channel
attention weights ($T\times C_{L}$). These channel attention weights are
multiplied with spatially refined features to output co-attended features of
dimension $T\times C_{L}\times H_{L}\times W_{L}$.

Fig. 7: Illustration of the network architecture adopted for video action
classification task. The baseline model (Kalfaoglu et al. (2020)) consists of
single stream feature extractor (3D ResNext-101, I3D) accepting RGB frames
(dimension $T\times 3\times H\times W$) as input and passes it through a set
of 3D CNN blocks, followed by Feature Reduction with Modified Block (FRMB)
module (Kalfaoglu et al. (2020)), temporal modeling layer (LSTM or BeRT-CLS)
and a classification layer. COSAM module is inserted after the feature
extractor and before FRMB module to study the improvements. A cross entropy
loss is employed as the objective function during training.

##### Salient-region interaction module (SRIM)

The COSAM module may have selected the salient regions containing potentially
multiple objects. To promote interactions between individual object-like
regions, we propose to use a self-attention based salient region interaction
module inspired from GloRe (Chen et al. (2019b)). It takes the co-attended
spatial feature maps of dimension $T\times C_{L}\times H_{L}\times W_{L}$ from
the previous step (COSAM) as input, and passes it through a $1\times 1$ 2D
convolution layer to perform dimension reduction on channels
($C_{L}\rightarrow C_{R}$, $C_{R}<<C_{L}$) to reduce computations. Next, we
assume that every frame has $N_{o}$ objects (in this work, $N_{o}$ is set to
$5$) and each pixel may belong to one of these objects. To figure out the
object association of pixels, the dimension-reduced feature maps (dimension
$T\times C_{R}\times H_{L}\times W_{L}$) are passed through an object-
association module to output object-association maps $\\{O_{i}\\}_{i=1}^{T}$
that associates every pixel of the feature map to one of $N_{o}$ objects. We
utilize a $1\times 1$ 2D convolution with $N_{o}$ output channels to achieve
this. Further, based on this object association map, we apply weighted average
pooling (WAP) on the dimension-reduced feature maps to output object feature
descriptors of size $T\times N_{o}\times C_{R}$. i.e., for each object, the
model outputs a descriptor of size $C_{R}$. In the next step, we promote
interactions between these object features using multi-head self-attention
(Vaswani et al. (2017)). Each frame $F_{i}$’s object features ($N_{o}\times
C_{R}$) are restricted to interact with the previous, current and next frame’s
($F_{i-1},F_{i},F_{i+1}$) object features to avoid noisy feature interactions
(refer to Sec. 5.2.6 for ablation studies).

Next, these self-attended object features are distributed back to pixel-space
through a dimension expansion block ($C_{R}\rightarrow C_{L}$) consisting of a
$1\times 1$ 2D convolution followed by a reverse mapping module based on
object association maps $\\{{O_{i}}\\}_{i=1}^{T}$. These redistributed
features (dimension $T\times C_{L}\times H_{L}\times W_{L}$) are added to the
original input features (dimension $T\times C_{L}\times H_{L}\times W_{L}$) to
yield context-aware features $\\{F_{{c}_{i}}\\}_{i=1}^{T}$. Next, we apply GAP
on $\\{F_{{c}_{i}}\\}_{i=1}^{T}$ to get per-frame feature descriptors
$F_{CoSB}$ (dimension $T\times C_{L}$) to pass them to the decoder for caption
generation.

####  4.2.4 Caption generation decoder

For every video instance, after obtaining its features from GSB ($F_{GSB}$)
and CoSB ($F_{CoSB}$), we use two separate decoders to predict captions from
$F_{GSB}$ and $F_{CoSB}$ respectively, similar to Pan et al. (2020). The
decoder transformer generates a word at every time-step by attending to the
input features as well as previously generated words in the caption. The
predicted caption from the decoders is evaluated using cross-entropy loss.
Further, to transfer salient region specific learning from CoSB to GSB, we
impose a knowledge distillation based constraint that both CoSB and GSB shall
generate the caption words with similar confidence (refer Sec 4.2.5 for
objective functions). During test time, we use the captions from the Global
scene branch for evaluation, as followed in Pan et al. (2020).

####  4.2.5 Objective functions

We use standard cross entropy loss for evaluating captions generated from the
transformer decoders. Further, to promote knowledge propagation of salient
regions from CoSB to GSB, we apply an online knowledge distillation constraint
between GSB and CoSB branches via KL-divergence loss similar to Pan et al.
(2020). Given the word vocabulary $W$ and probabilities of caption words
generated from global-scene branch $P$, co-segmentation branch $Q$, the
knowledge distillation constraint is formulated using KL-divergence as
follows:

$\displaystyle D_{KL}(\ P||\ Q)\ =\sum\limits_{x\in W}P(x)\log\
\left(\frac{P(x)}{Q(x)}\ \right)$ (5)

The overall loss function is given by,

$L=L_{GSB}+\lambda L_{CoSB}+\lambda_{KL}L_{KL}$ (6)

Here, $L_{GSB}$ and $L_{CoSB}$ are the individual cross entropy losses for GSB
and CoSB branches respectively. $L_{KL}$ denotes KL-divergence between
probability distributions of GSB and CoSB branch. $\lambda$ and $\lambda_{KL}$
are hyper-parameters. We perform experiments on two datasets (MSR-VTT (Xu et
al. (2016)) and MSVD (Chen and Dolan (2011))) and illustrate the results along
with state-of-the-art comparison in Section 5.2.

###  4.3 Video action classification

In this section, we utilize the COSAM module to perform the task of video
classification. In our work, without loss of generality, we take a
straightforward approach to adopt a recent state-of-the-art baseline method
(Kalfaoglu et al. (2020)) with BERT based late temporal modeling for action
recognition and simply insert the COSAM module into the baseline method to
analyze the contribution of the COSAM module.

####  4.3.1 Network architecture

We briefly mention the network architecture of the baseline method (Kalfaoglu
et al. (2020)) in the following paragraphs. The network architecture (Fig. 7)
consists of three core components as follows:

##### 1) Feature extractor

It extracts abstract features $\\{F_{i}\\}_{i=1}^{T}$ from the given video
frames $\\{f_{i}\\}_{i=1}^{T}$ of dimension $T\times 3\times H\times W$ where
$T=$ number of frames in the video, 3 channels signify RGB channels, $H=$
height, $W=$ width respectively. Specifically, in our work, we study the
effect of the COSAM module with two different backbone networks namely: 1) 3D
ResNeXt-101, and 2) I3D. 3D ResNeXt-101 follows the architecture of ResNet-101
in which the 2D residual blocks are replaced by 3D inception-based residual
blocks. I3D follows the architecture of the Inception network in which the 2D
inception modules are replaced by 3D inception modules. These networks are
equipped with Feature Reduction with Modified Block (FRMB) (Kalfaoglu et al.
(2020) at the end and pre-trained with Kinetics dataset. We plug-in the COSAM
module after the final convolution block of the backbone network (i.e., before
FRMB).

##### 2) Temporal modeling layer

It is used to model temporal dependencies between frame-level features and to
finally obtain a summarised video-level feature. We follow the baseline method
(Kalfaoglu et al. (2020)) to experiment with two kinds of temporal modeling
layers namely: 1) BERT based and 2) LSTM based. In the BERT based temporal
modeling layer, the frame-wise features are added with the position-encoding
and further concatenated with a CLS token. These position-encoded features are
passed through a sequence of self-attention layers to perform multi-head self-
attention on the frame-level features. Finally, the output features
corresponding to the CLS token are considered as the summarized video-level
representation and passed to the next layer. A long short-term memory (LSTM)
layer takes as input the global-average pooled (GAP) frame-level features
sequentially and forwards it through time. The output at the last time step is
considered as the summarized video-level feature.

##### 3) Classification layer

The summarised video-level feature from the temporal modeling layer (LSTM or
BERT-CLS) is passed through a fully connected layer to output classification
logits corresponding to action classes.

##### Overall pipeline

The feature extractor takes an input of dimension $T\times 3\times H\times W$
and passes it through a set of $L$ 3D convolutional (or 3D residual) blocks
and results in output features $O_{L}$ of dimension $\frac{T}{S_{T}^{L}}\times
D_{L}\times\frac{H}{S_{H}^{L}}\times\frac{W}{S_{W}^{L}}$. Here,
$S_{T}^{L},S_{H}^{L},S_{W}^{L}$ are strides in time, height, & width
dimensions respectively. In the case of the baseline model, the output
features $O_{L}$ from the backbone network are passed through the FRMB module
to obtain the final convolutional features of the given video snippet. Then,
global average pooling (GAP) is applied to the convolutional features to
obtain frame-wise features. These frame-wise features are passed through a
temporal modeling layer (BERT or LSTM) to model temporal dependencies and to
get a video-level summarized feature. The video-level features are passed
through the classification layer to result in class-specific scores for the
classes in the dataset. In the COSAM-based model, the output features $O_{L}$
are passed through the COSAM module to activate the set of common features and
then passed through the FRMB module for further processing.

####  4.3.2 Objective functions

We employ a simple softmax cross entropy loss among the video action classes
as the objective function during training. Specifically, given the training
data $\\{\\{f^{i}_{k}\\}_{i=1}^{T},t_{k}\\}_{k=1}^{N}$ ($f_{k}$ = $k^{th}$
training video instance, $t_{k}$ = the target class one-hot vector, $T=$
number of frames, $N=$ number of videos), the class softmax probabilities
$\\{p^{j}_{k}\\}_{j=1}^{C}$ are obtained from the classification layer. Then,
the cross entropy loss is given by:

$L=-\sum_{k=1}^{N}\sum_{c=1}^{C}I(c=t_{k})\log p_{k}^{c}$ (7)

Here, $I(.)$ denotes an indicator function, $C$ = number of classes. We
perform experiments on two datasets (HMDB51 (Kuehne et al. (2011)) and UCF101
(Soomro et al. (2012))) and illustrate the results along with state-of-the-art
comparison in Section 5.3.

##  5 Experiments and results

###  5.1 Video-based person re-identification

In this section, we evaluate the performance of the proposed COSAM layer in
the video-based person re-ID pipeline by plugging it into two state-of-the-art
deep architectures: ResNet50 (He et al. (2016)) & SE-ResNet50 (Hu et al.
(2018)).

####  5.1.1 Datasets and Evaluation protocol

We evaluate the proposed COSAM module based Re-ID model on three commonly used
video-based person Re-ID datasets: MARS (Zheng et al. (2016)), DukeMTMC-
VideoReID (Wu et al. (2018)) and iLIDS-VID (Wang et al. (2014)). The MARS
dataset (Zheng et al. (2016)) is the largest sequence-based person Re-ID
dataset with 1261 identities and 20,478 video sequences, with multiple frames
per person captured across 6 non-overlapping camera views. Among the total
identities, 625 identities are used for training and the rest are used for
testing. Additionally, 3,248 identities (disjoint with the train and test set)
are used as distractors. DukeMTMC-VideoReID (Wu et al. (2018)) is a subset of
the DukeMTMC multi-camera dataset (Ristani et al. (2016)), which was collected
on outdoor scenario with varying viewpoint, illuminations, background and
occlusions using 8 synchronized cameras. It contains 702 identities, each for
training & testing, and 408 identities as the distractors. There are 369,656
tracklets for training, and 445,764 frames for testing & distractors. iLIDS-
LID (Wang et al. (2014)) is a small dataset containing 600 sequences of 300
persons from two non-overlapping camera views. The sequences vary in length
between 23 and 192 frames. As per the protocol followed in Wang et al. (2014);
Li et al. (2018b), 10 random probe-gallery splits are used to perform
experiments.

We use the standard evaluation metrics as followed in the literature (Zheng et
al. (2016); Li et al. (2018b); Liu et al. (2018); Suh et al. (2018)) viz., 1)
Cumulative Matching Characteristics (CMC) & 2) Mean average precision (mAP).
CMC is based on the retrieval capability of the algorithm to find the correct
identity within the top-k ranked matches. CMC is used when only one gallery
instance exists for every identity. We report rank-1, rank-5 and rank-20 CMC
accuracies. The mAP metric is used to evaluate algorithms in multi-shot re-
identification settings where multiple instances of same identities are
present in the gallery.

####  5.1.2 Implementation details

The proposed method is implemented using the PyTorch framework (Paszke et al.
(2017)) and is available
online111https://github.com/InnovArul/vidreid_cosegmentation. During training,
every video consists of $N=4$ frames (as in baseline Gao and Nevatia (2018))
and each frame is of height = $256$ and width = $128$. The images are
normalized using the RGB mean and standard deviation of ImageNet (Deng et al.
(2009b)) before passing to the network. The network is trained using the Adam
optimizer with the following hyper-parameters : number of reference feature
maps $K=3$, $\beta_{1}$ = 0.9, $\beta_{2}$ = 0.999, batch size = 32, initial
learning rate = 0.0001, trade-off parameter between losses $\lambda=1$ and
COSAM dimension reduction size $D_{R}=256$. We train the network for $\sim$
60K iterations and the learning rate is multiplied by 0.1 after every 15K
iterations. The implementation was done on a machine with NVIDIA GeForce GTX
1080 Ti GPUs and takes around 8 hours to train a model with one GPU.

####  5.1.3 Results & Discussion

(a)

(b)

(c)

(d)

Fig. 8: Visualization of co-segmentations. The second row shows the
segmentation maps corresponding to the images in the first row.

In our experiments, every video of the person is split into multiple non-
overlapping video-snippets of length $N$ frames and each snippet is passed
through the network to obtain a snippet-level descriptor. Further, the video-
snippet level descriptors are averaged to get the video-level descriptor.
Then, these video-level descriptors are compared using the L2 distance to
calculate the CMC and mAP performances.

First, we present the results from our baseline method (Gao and Nevatia
(2018)) that re-visited the effect of various temporal aggregation layers with
ResNet50 (He et al. (2016)) as the feature extractor. We extend this study by
including yet another state-of-the-art architecture SE-ResNet50 (Hu et al.
(2018)222Winner of ILSVRC 2017 Image Classification ChallengeDeng et al.
(2009b)) and present the quantitative results in Table 3.

Feature extractor |  Temp. Agg | MARS | DukeMTMC-VideoReID
---|---|---|---
mAP | R1 | R5 | R20 | mAP | R1 | R5 | R20
ResNet50 Gao and Nevatia (2018) |  TPavg | 75.8 | 83.1 | 92.8 | 96.8 | 92.9 | 93.6 | 99.0 | 99.7
ResNet50 Gao and Nevatia (2018) | TA | 76.7 | 83.3 | 93.8 | 97.4 | 93.2 | 93.9 | 98.9 | 99.5
ResNet50 Gao and Nevatia (2018) | RNN | 73.8 | 81.6 | 92.8 | 96.7 | 88.1 | 88.7 | 97.6 | 99.3
SE-ResNet50 |  TPavg | 78.1 | 84.0 | 95.2 | 97.1 | 93.5 | 93.7 | 99.0 | 99.7
SE-ResNet50 | TA | 77.7 | 84.2 | 94.7 | 97.4 | 93.1 | 94.2 | 99.0 | 99.7
SE-ResNet50 | RNN | 75.7 | 83.1 | 93.6 | 96.0 | 92.4 | 94.0 | 98.4 | 99.1
Table 3: Evaluation of a simple video-based Re-ID framework (as shown in Fig. 3) on different feature extractor networks, feature aggregation techniques and datasets. Best results are shown in Bold. | COSAMi | MARS | DukeMTMC-VideoReID
---|---|---|---
| mAP | R1 | R5 | R20 | mAP | R1 | R5 | R20
ResNet50 |  No COSAM (Gao and Nevatia (2018)) | 75.8 | 83.1 | 92.8 | 96.8 | 92.9 | 93.6 | 99.0 | 99.7
COSAM2 | 68.3 | 77.7 | 90.1 | 96.1 | 88.9 | 90.2 | 98.4 | 99.0
COSAM3 | 76.9 | 82.7 | 94.3 | 97.3 | 93.6 | 94.0 | 98.7 | 99.9
|  COSAM4 | 76.8 | 82.9 | 94.2 | 97.1 | 93.8 | 94.7 | 98.7 | 99.7
|  COSAM5 | 76.6 | 82.8 | 93.9 | 97.2 | 93.2 | 93.7 | 98.4 | 99.9
|  COSAM3,4 | 76.4 | 83.4 | 93.9 | 97.1 | 93.7 | 94.4 | 99.1 | 99.4
|  COSAM3,5 | 76.9 | 83.7 | 94.0 | 97.3 | 93.0 | 93.7 | 99.0 | 99.7
|  COSAM4,5 | 77.2 | 83.7 | 94.1 | 97.5 | 94.0 | 94.4 | 99.1 | 99.9
|  COSAM3,4,5 | 76.6 | 83.2 | 93.7 | 97.3 | 93.1 | 93.6 | 98.7 | 99.4
| No COSAM | 78.3 | 84.0 | 95.2 | 97.1 | 93.5 | 93.7 | 99.0 | 99.7
SE-ResNet50 |  COSAM2 | 67.0 | 77.9 | 90.4 | 94.9 | 92.2 | 94.0 | 98.9 | 99.7
COSAM3 | 79.5 | 85.0 | 94.7 | 97.8 | 93.6 | 94.7 | 99.0 | 99.9
COSAM4 | 79.8 | 84.9 | 95.4 | 97.8 | 94.0 | 95.4 | 99.0 | 99.9
|  COSAM5 | 79.9 | 84.5 | 95.7 | 97.9 | 93.9 | 94.9 | 99.1 | 99.9
|  COSAM3,4 | 79.5 | 84.8 | 94.7 | 97.6 | 93.7 | 94.7 | 98.7 | 99.7
|  COSAM3,5 | 79.8 | 85.2 | 95.5 | 98.0 | 93.9 | 94.2 | 99.3 | 99.9
|  COSAM4,5 | 79.9 | 84.9 | 95.5 | 97.9 | 94.1 | 95.4 | 99.3 | 99.8
|  COSAM3,4,5 | 80.5 | 85.2 | 95.5 | 98.0 | 94.1 | 95.4 | 99.3 | 99.9
Table 4: Evaluation of the backbone feature extractors with COSAM and temporal
aggregation layer as TPavg. Here, COSAMi = plugging in COSAM layer after
$i^{th}$ CNN block.

Based on our experiments in Table 3, we postulate certain key observations:
First, the selection of the backbone network can influence the holistic system
performance. This is quite noteworthy since not much research on the influence
of the backbone network on Re-ID performance has been conducted. Second, it is
observed that even a simple TPavg layer performs on-par with complex
attention/RNN based aggregation layers, as also reported in Chen and Wu
(2017).

##### Location of the COSAM layer within the network

Next, to evaluate the effect of the COSAM module, we experiment by plugging-in
the COSAM module after each CNN block of the feature extractors and TPavg is
used as the feature aggregation layer. The network is trained and evaluated on
the MARS & DukeMTMC-VideoReID datasets and the quantitative results are shown
in Table 4. From the results, it can be inferred that the inclusion of the
COSAM module improves the baseline network and it is effective in the deeper
layers (COSAM3, COSAM4, COSAM5), as the features in those layers are more
discriminative and abstract than the features at shallow layer(s). We also
experiment with the inclusion of multiple COSAM blocks simultaneously. It is
found that COSAM4,5 (plugging in COSAM4 & COSAM5) achieves the best accuracy
vs. computation trade-off and is treated as our default proposed architecture
in the rest of the experiments.

|  Temp. Agg. | COSAMi | MARS | Duke | iLIDS-VID
---|---|---|---|---|---
| mAP | R1 | R5 | mAP | R1 | R5 | R1 | R5
3  ResNet50 |  TPavg Gao and Nevatia (2018) | - | 75.8 | 83.1 | 92.8 | 92.9 | 93.6 | 99.0 | 73.9 | 92.6
TPavg |  COSAM4,5 | 77.2 | 83.7 | 94.1 | 94.0 | 94.4 | 99.1 | 75.5 | 94.1
2-113 |  TA Gao and Nevatia (2018) | - | 76.7 | 83.3 | 93.8 | 93.2 | 93.9 | 98.9 | 72.3 | 92.4
| TA |  COSAM4,5 | 76.9 | 83.6 | 93.7 | 93.4 | 94.6 | 98.9 | 74.9 | 94.4
2-113 |  RNN Gao and Nevatia (2018) | - | 73.8 | 81.6 | 92.8 | 88.1 | 88.7 | 97.6 | 68.5 | 93.2
| RNN |  COSAM4,5 | 74.8 | 82.4 | 93.9 | 90.4 | 91.7 | 98.3 | 68.9 | 93.1
3 3  SE-ResNet50 |  TPavg | - | 78.1 | 84.0 | 95.2 | 93.5 | 93.7 | 99.0 | 76.9 | 93.9
TPavg |  COSAM4,5 | 79.9 | 84.9 | 95.5 | 94.1 | 95.4 | 99.3 | 79.6 | 95.3
2-113 | TA | - | 77.7 | 84.2 | 94.7 | 93.1 | 94.2 | 99.0 | 74.7 | 93.2
| TA |  COSAM4,5 | 79.1 | 85.0 | 94.9 | 94.1 | 95.3 | 98.9 | 77.1 | 94.7
2-113 | RNN | - | 75.7 | 83.1 | 93.6 | 92.4 | 94.0 | 98.4 | 77.4 | 94.4
| RNN |  COSAM4,5 | 76.0 | 83.4 | 93.9 | 92.5 | 93.9 | 98.3 | 77.8 | 97.3
3 |  |  |  |  |  |  |  |  |  |
Table 5: Comparison of the baseline models with best performing COSAM-
configuration (COSAM4,5) along with different feature extractor networks,
feature aggregation techniques and datasets. Here, COSAM4,5 = COSAM layer is
placed after 4th and 5th CNN blocks of the baseline model, Duke = DukeMTMC-
VideoReID dataset. Best mAP & CMC Rank-1 per backbone network are shown in red
and blue colors respectively. mAP is not applicable for iLIDS-VID due to
single gallery instance per probe.

##### Visualizations

To demonstrate the interpretability of our proposed method, we visualize the
spatial attention masks of the COSAM4 layer in the SE-ResNet50+COSAM4,5 model
trained on the MARS dataset (Fig. 8). The frames exhibit varying conditions
such as scale, pose, viewpoint changes and partial occlusions. In Fig. 8(a),
the predicted attention mask is able to focus on the person and avoid
background features. In Fig. 8(b), despite the person occupying comparatively
a small region of the frame, the COSAM layer still successfully focuses on the
person based on task-relevant consensus. Although the buildings and trees are
common in all the frames, our Co-segmentation inspired architecture
specifically trained for Re-ID ignores the background regions. In Fig. 8(c),
it can be observed that the spatial attention identifies the accessory
(umbrella) carried by the person. Identifying the person with the aid of their
belongings is one of the significant ways to discriminate the person by
appearance. In Fig. 8(d), the partial occlusion scenario is handled
successfully by avoiding the occluding object (cycle).

Network |  Deep model? | MARS
---|---|---
mAP | R1 | R5 | R20
LOMO+XQDA (Liao et al. (2015)) |  No | 16.4 | 30.7 | 46.6 | 60.9
JST-RNN (Zhou et al. (2017)) |  Yes | 50.7 | 70.6 | 90.0 | 97.6
QAN (Liu et al. (2017)) |  Yes | 51.7 | 73.7 | 84.9 | 91.6
Context Aware Parts (Li et al. (2017)) |  Yes | 56.1 | 71.8 | 86.6 | 93.0
IDE+XQDA+ReRanking (Zhong et al. (2017)) |  Yes | 68.5 | 73.9 | - | -
TriNet (Hermans et al. (2017)) |  Yes | 67.7 | 79.8 | 91.4 | -
Region QEN (Song et al. (2018b)) |  Yes | 71.1 | 77.8 | 88.8 | 94.1
Comp. Snippet Sim.(Chen et al. (2018a)) |  Yes | 69.4 | 81.2 | 92.1 | -
Part-Aligned (Suh et al. (2018)) |  Yes | 72.2 | 83.0 | 92.8 | 96.8
RevisitTempPool (Gao and Nevatia (2018)) |  Yes | 76.7 | 83.3 | 93.8 | 97.4
2 (Gao and Nevatia (2018)) + SE-ResNet50 + TPavg |  Yes | 78.1 | 84.0 | 95.2 | 97.1
2  SE-ResNet50 + COSAM4,5 \+ TPavg(ours) |  Yes | 79.9 | 84.9 | 95.5 | 97.9
SE-ResNet50 + COSAM4,5 \+ TPavg(ours) + Re-ranking (Zhong et al. (2017)) |  Yes | 87.4 | 86.9 | 95.5 | 98.0
Network |  Deep model? | DukeMTMC-VideoReID
mAP | R1 | R5 | R20
ETAP-Net (Suh et al. (2018)) |  Yes | 78.34 | 83.62 | 94.59 | 97.58
RevisitTempPool (Gao and Nevatia (2018)) |  Yes | 93.2 | 93.9 | 98.9 | 99.5
2 (Gao and Nevatia (2018)) + SE-ResNet50 + TPavg |  Yes | 93.5 | 93.7 | 99.0 | 99.7
2  SE-ResNet50 + COSAM4,5 + TPavg(ours) |  Yes | 94.1 | 95.4 | 99.3 | 99.8
Table 6: Comparison of our best model with state-of-the-art methods on MARS & DukeMTMC-VideoReID datasets.  Method | iLIDS-VID
---|---
R1 | R5 | R20
Top push video Re-ID (You et al. (2016)) | 56.3 | 87.6 | 98.3
JST-RNN (Zhou et al. (2017)) | 55.2 | 86.5 | 97.0
Joint ST pooling (Xu et al. (2017)) | 62.0 | 86.0 | 98.0
Region QEN (Song et al. (2018b)) | 77.1 | 93.2 | 99.4
RevisitTempPool (Gao and Nevatia (2018)) | 73.9 | 92.6 | 98.41
Gao and Nevatia (2018) \+ SE-ResNet50 + TPavg | 76.87 | 93.94 | 99.07
SE-ResNet50 + COSAM4,5 \+ TPavg(ours) | 79.61 | 95.32 | 99.8
Table 7: Comparison of the state-of-the-arts in iLIDS-VID dataset.

##### Effect of COSAM in the baseline model

To understand the significance of the COSAM layer, we incorporate our best
performing COSAM4,5 into baseline video-based Re-ID pipelines with two feature
extractors (ResNet50 and SE-ResNet50) and three different temporal aggregation
layers (TPavg, TA, RNN) Gao and Nevatia (2018). Table 5 represents the
performance evaluation of the models. Our COSAM-based networks show consistent
performance improvement (both CMC Rank and mAP) over the baseline models in
all three datasets. Between the backbone networks, SE-ResNet50 outperforms
ResNet50 in both baselines and proposed case studies, highlighting the
importance of a better backbone network selection. Among the temporal
aggregation modules, although more or less similar performance is exhibited by
TPavg, TA and RNN, the former (TPavg) results in the best mAP values in both
MARS and DukeMTMC-VideoReID datasets & best CMC Rank-1 in iLIDS-VID. In
particular, COSAM improves the mAP by 1.4% (ResNet50) & 1.8% (SE-ResNet50) in
MARS and 1.1%(ResNet50) & 0.6% (SE-ResNet50) in DukeMTMC-VideoReID
respectively. Regarding the CMC Rank, we observe an improvement of 0.6%
(ResNet50) & 0.9% (SE-ResNet50) in MARS, 0.8% (ResNet50) & 1.7% (SE-ResNet50)
in DukeMTMC-VideoReID and 1.6% (ResNet50) & 2.7% (SE-ResNet50) in iLIDS-VID.

##### Comparison with state-of-the-art methods

We compare our method with the state-of-the-arts Liao et al. (2015); Hermans
et al. (2017); Liu et al. (2017); Li et al. (2017); Zhong et al. (2017);
Hermans et al. (2017); Song et al. (2018b); Chen et al. (2018a); Suh et al.
(2018); Gao and Nevatia (2018) in MARS and DukeMTMC-VideoReID datasets and the
results are shown in Table 6. It is observed that our proposed COSAM module
applied into SE-ResNet50 (COSAM4,5) along with TPavg achieves the best
performance. In particular, our approach has $\sim$0.9% improvement in CMC
Rank-1 as well as 1.8% improvement in mAP in the MARS dataset over the best
performing method (Gao and Nevatia (2018) \+ SE-ResNet50 + TPavg). Apart from
this, applying re-ranking (Zhong et al. (2017)) further increases the
performance to +2.0% CMC Rank-1 and +7.5% mAP. Intuitively, such an improved
CMC Rank-1 (86.9%) shows that majority of the subjects are correctly
identified in the first rank, whereas the improved mAP result (87.4%) denotes
that multiple instances of the person are ranked precisely at the top in a
multi-shot setting that is significant in retrieval problems. We attribute
this improvement to the effectiveness of the COSAM layer in suppressing noise
and aiding the network learn about identifying relevant common objects.
Similarly, our COSAM layer with SE-ResNet50 achieves 0.6% improvement in mAP
and 1.7% improvement in CMC Rank-1 with DukeMTMC-VideoReID dataset. The
performance comparison of iLIDS-VID dataset in the Table 7 shows that COSAM
layer with SE-ResNet50 improves +2.8% in CMC Rank-1 over the baseline method
(Gao and Nevatia (2018)).

####  5.1.4 Ablation studies

##### Effect of different frame lengths ($N$):

We study the effect of the number of frames in a video on the performance of
our best performing model. In particular, we analyze with frame lengths of $N$
= 2, 4 and 8 in SE-ResNet50+COSAM4,5+TPavg and the results are shown in Table
8. We found $N=4$ frames to be optimal similar to Gao and Nevatia (2018).

frame length | MARS | DukeMTMC-VideoReID
---|---|---
mAP | R1 | R5 | R20 | mAP | R1 | R5 | R20
$N=2$ | 78.1 | 83.5 | 94.3 | 98.1 | 94.0 | 94.3 | 99.1 | 99.9
$N=4$ | 79.9 | 84.9 | 95.5 | 97.9 | 94.1 | 95.4 | 99.3 | 99.8
$N=8$ | 77.4 | 84.6 | 94.2 | 97.0 | 92.1 | 91.9 | 99.0 | 99.6
Table 8: Evaluation of the influence of track length $T$ on Re-ID performance
of the best performing model SE-ResNet50+COSAM4,5+TPavg.

##### Attribute-wise performance gains:

To understand the importance of COSAM in capturing attributes, we conduct
attribute-wise empirical studies on the DukeMTMC-VideoReID dataset and present
the results in Table 9. The significant improvements on attributes such as
handbag, hat and backpack show that COSAM is indeed capturing the person’s
attributes.

Model | Handbag | Hat | Backpack
---|---|---|---
mAP |  R1 |  R5 |  mAP |  R1 |  R5 |  mAP |  R1 |  R5
R50+TP |  91.2 |  92.0 |  100.0 |  91.1 |  91.7 |  97.5 |  92.8 |  93.9 |  98.6
R50+C4,5+TP |  95.2 |  96.0 |  100.0 |  93.5 |  94.2 |  97.5 |  95.1 |  96.4 |  99.8
SE50+TP |  94.1 |  97.3 |  100.0 |  92.7 |  94.2 |  99.2 |  94.3 |  95.6 |  99.1
SE50+C4,5+TP |  96.0 |  100.0 |  100.0 |  93.9 |  96.7 |  99.5 |  95.4 |  97.1 |  100.0
Table 9: Attribute-wise perf. comparison on Duke reveals the effectiveness of
COSAM to capture features of person’s accessories. Here, R50=ResNet50,
SE50=SE-ResNet50, C4,5=COSAM4,5, TP = Temporal average pooling.

##### Selection of frames

We perform experiments with two schemes of frame selection during training
namely: 1) Sequential: a continuous sequence of $N$ frames are selected, 2)
Random: random sampled $N$ frames from the whole video sequence. The
quantitative results are shown in Table 10. It is observed that the
performance of training using randomly sampled frames is inferior to the
performance of training using sequentially sampled frames.

frame selection | MARS | DukeMTMC-VideoReID
---|---|---
mAP | R1 | R5 | R20 | mAP | R1 | R5 | R20
sequential | 79.9 | 84.9 | 95.5 | 97.9 | 94.1 | 95.4 | 99.3 | 99.8
random | 77.5 | 83.3 | 93.6 | 97.0 | 90.2 | 90.6 | 98.3 | 99.6
Table 10: Evaluation of the influence of frame selection on Re-ID performance
of the best performing model SE-ResNet50+COSAM4,5+TPavg.

##### Cross-dataset test performance

We analyze the cross-dataset performance of the best performing model (SE-
ResNet50+COSAM4,5\+ TPavg) against a base-model without using COSAM layer. The
results are shown in Table 11. By training on MARS dataset and testing on
DukeMTMC-VideoReID dataset, the former model outperforms the latter by 2.8%
mAP and 3.5% CMC Rank-1. Similar performance of COSAM-based model is observed
while training on DukeMTMC-VideoReID dataset and testing on MARS dataset
(Improvement of 0.9% in mAP and 0.7% in CMC Rank-1).

| Train set | Test set | mAP | R1 | R5 | R20
---|---|---|---|---|---|---
No COSAM | MARS |  DukeMTMC | 32.0 | 33.3 | 53.3 | 67.1
COSAM4,5 | MARS |  DukeMTMC | 34.8 | 36.8 | 54.1 | 67.9
No COSAM |  DukeMTMC | MARS | 25.0 | 41.7 | 54.4 | 65.3
COSAM4,5 |  DukeMTMC | MARS | 25.9 | 42.4 | 56.0 | 65.8
Table 11: Cross-dataset performance of the best performing model with SE-
ResNet50 as the feature extractor and TPavg as the temporal aggregation layer.
Here DukeMTMC = DukeMTMC-VideoReID.

##### Spatial vs. Channel attention

To evaluate the effect of the attention steps in the COSAM layer, we
investigate the individual attention steps (Spatial and Channel). The results
in Table 12 shows that the inclusion of both spatial and channel attention
steps together achieves superior performance.

Attention layer | MARS | DukeMTMC-VideoReID
---|---|---
mAP | R1 | R5 | R20 | mAP | R1 | R5 | R20
Only spatial att. | 78.8 | 84.1 | 94.9 | 97.7 | 93.6 | 93.9 | 99.0 | 99.9
Only Channel att. | 79.0 | 84.3 | 95.0 | 97.8 | 93.8 | 94.4 | 99.1 | 99..7
Both | 79.9 | 84.9 | 95.5 | 97.9 | 94.1 | 95.4 | 99.3 | 99.8
Table 12: Evaluation of the influence of Co-segmentation based attention
layers on Re-ID performance of the best performing model SE-
ResNet50+COSAM4,5\+ TPavg.

##### Comparison with Non-Local attention module (NLM)

In this section, we compare and contrast the COSAM module with non-local
attention modules (NLM) proposed by Wang et al. (2018d) for video-based tasks.
Though both modules are conceptually similar, there are succinct differences
between COSAM and NLM.

Module | #Params | #FLOPs
---|---|---
NLM | Gauss. | 4.2M | 4.3B
Gaussian embedding | 8.39M | 8.59G
Concatenation | 8.4M | 8.72G
Dot product | 8.39M | 8.59G
COSAM (ours) | 1.6M | 0.57G
Table 13: COSAM vs. NLM (Wang et al. (2018d)) (input = $4$$\times$$2048$$\times$$16$$\times$$8)$. Observation: COSAM uses $\sim 4x$ less memory and $\sim 16x$ less computation than NLM.  Model | #Params | #FLOPs | MARS
---|---|---|---
mAP | R1 | R5
R50+NLM4,5+TP | 34.31M | 27.11B | 76.9 | 83.2 | 94.2
R50+COSAM4,5+TP | 26.22M | 17.24B | 77.2 | 83.7 | 94.1
SE50+NLM4,5+TP | 36.85M | 26.74B | 77.9 | 83.3 | 94.7
SE50+COSAM4,5+TP | 28.76M | 16.86B | 79.9 | 84.9 | 95.5
Table 14: Comparison of COSAM vs. NLM (Wang et al. (2018d)) on MARS dataset.
Here, R50 = ResNet50, SE50 = SE-ResNet50, NLM4,5 = Non-local module after
$4th$ and $5th$ block, COSAM4,5 = COSAM module after $4th$ and $5th$ block.

NLM is designed as a residual module that aggregates the features from
neighboring / reference feature maps, where as COSAM activates common salient
regions between feature maps and consists of a mandatory spatial and channel
attention step to pass through only the common features. Both NLM and COSAM
incurs $O(N*n^{2})$ complexity for comparing descriptors of each frame with
its reference feature maps. Here N = number of frames, $n=H\times W$. However,
NLM involves channel reduction step followed by feature comparison and
aggregation, then channel expansion. On the other hand, COSAM only requires
dimension reduction step followed by feature comparison which makes COSAM
lightweight in terms of number of parameters and FLOPs. Table 13 illustrates
that COSAM uses $\sim 4x$ less memory and $\sim 16x$ less computation than NLM
(Wang et al. (2018d)) for an input of dimension $4\times 2048\times 16\times
8$. Further, we conduct experiments with Gaussian embedding instantiation of
NLM on MARS dataset and show the results in Table 14. Results from Table 14
reveals that COSAM performs superior to NLM in terms of mAP and rank-1
accuracy. Specifically, with SE-ResNet50 backbone, COSAM model (SE-ResNet50 +
COSAM4,5 \+ TP) performs +2.0% and +1.6% in mAP, rank-1 accuracy than the NLM
model (SE-ResNet50 + NLM4,5 \+ TP).

Methods | Year | Object detector | MSRVTT | MSVD
---|---|---|---|---
B | M | R | C | B | M | R | C
POS+VCT (Hou et al. (2019)) | ICCV-2019 | - | 42.3 | 29.7 | 62.8 | 49.1 | 52.8 | 36.1 | 71.8 | 87.8
ORG-TRL (Zhang et al. (2020)) | CVPR-2020 | Faster RCNN | 43.6 | 28.8 | 62.1 | 50.9 | 54.3 | 36.4 | 73.9 | 95.2
OA-BTG (Zhang and Peng (2019)) | CVPR-2019 | Mask RCNN | 41.4 | 28.2 | - | 46.9 | 56.9 | 36.2 | - | 90.6
STG-KD (Pan et al. (2020)) | CVPR-2020 | Faster RCNN | 40.5 | 28.3 | 60.9 | 47.1 | 52.2 | 36.9 | 73.9 | 93.0
SA-LSTM (Wang et al. (2018a)) | CVPR-2018 | - | 36.3 | 25.5 | 58.3 | 39.9 | 45.3 | 31.9 | 64.2 | 76.2
M3 (Wang et al. (2018b)) | CVPR-2018 | - | 38.1 | 26.6 | - | - | 52.8 | 33.3 | - | -
RecNet (Wang et al. (2018a)) | CVPR-2018 | - | 39.1 | 26.6 | 59.3 | 42.7 | 52.3 | 34.1 | 69.8 | 80.3
PickNet (Chen et al. (2018d)) | ECCV-2018 | - | 41.3 | 27.7 | 59.8 | 44.1 | 52.3 | 33.3 | 69.6 | 76.5
MARN (Pei et al. (2019)) | CVPR-2019 | - | 40.4 | 28.1 | 60.7 | 47.1 | 48.6 | 35.1 | 71.9 | 92.2
Ours | - | - | 41.4 | 27.8 | 61.0 | 46.5 | 50.7 | 35.3 | 72.1 | 97.8
Table 15: Performance analysis on MSRVTT and MSVD datasets. First group of
methods optimize on decoder. Second group of methods enhance encoder visual
features by making use of object detectors. Third group of methods enhance
encoder features without the usage of object detectors. Here, B = BLEU@4, M =
METEOR, R = ROUGE_L, C = CIDEr.

###  5.2 Video captioning

In this section, we describe the experiments and results of COSAM-based video
captioning model along with the comparison of state-of-the-arts for this task.

####  5.2.1 Datasets

We use two challenging benchmark datasets: MSR-VTT (Xu et al. (2016)) and MSVD
(Chen and Dolan (2011)).

##### MSR-VTT

It contains 10,000 video clips from 20 wide-range of categories (cooking,
animations, sports, etc) with an average video length of 20 seconds. Each
video has 20 human-annotated English captions. We follow the standard split
followed in Pan et al. (2016); Pei et al. (2019); Zhang and Peng (2019) for
train, validation and test as follows: 6513 video clips for training set, 497
for cross-validation set and remaining 2990 for the test set.

##### MSVD

It contains total of 1970 videos with approx. 40 English captions per video.
We follow standard protocol split (Pan et al. (2016); Pei et al. (2019); Zhang
and Peng (2019)) of dividing 1970 videos into train = 1200, validation = 100,
and test = 670 videos for performing experiments.

####  5.2.2 Evaluation metrics

We use the following metrics found in the literature to quantitatively
evaluate our models performance: BLEU@4 (Papineni et al. (2002)), METEOR
(Denkowski and Lavie (2014)), ROUGE_L (Lin (2004)) and CIDEr (Vedantam et al.
(2015)). BLEU@N matches n-grams between the generated captions and the ground
truth, while METEOR metric is based on word-to-word matching between the
generated captions and the ground truth, ROUGE_L is Longest common subsequence
based metric (LCS) and CIDEr calculates n-gram similarity between the
generated caption and the ground truth. The above mentioned metrics are
computed using the Microsoft COCO evaluation server (Chen et al. (2015)).

####  5.2.3 Implementation Details

We implement our model using the PyTorch (Paszke et al. (2019)) deep learning
framework. During training, for every video, we use uniformly subsampled
$T=10$ frames as input with correct temporal order. The frames are resized to
a spatial resolution of $256\times 256$ and are center cropped to $224\times
224$. We use the 2D CNN ResNet-101 pre-trained on ImageNet (Deng et al.
(2009a)) dataset to extract 2D features in GSB. Also, we extract intermediate
features after the $5^{th}$ block (dimension $T\times{1024\times 14\times
14}$) as input features to the CoSB branch. For acquiring 3D features in GSB,
we use 3D ResNeXt-101 pre-trained on Kinetics dataset. For every frame, a clip
of 16 adjacent frames is used as input for ResNeXt-101 to get per-frame 3D
feature. The following hyper-parameters are used in network design:
$C_{R}=512,N_{o}=5$.

For the transformer decoder, we adopt Chen et al. (2018c) as our transformer
architecture. We build a vocabulary of size $11$K and $6.5$K for MSR-VTT and
MSVD respectively. During this process, punctuations are removed from every
sentence and a word is removed if its count is one or less in whole dataset to
avoid imbalanced distributions. During training, the maximum length of the
sentence is set to 20. We use 2 multi-head self attention layers (one to
encode visual features and the other to generate captions) in the decoder. We
greedily output words with beam size of 1. The self-attention module has 8
attention heads and an MLP to output features with $1024$ dimension.

We use a cross validation set to tune the hyper-parameters. The hyper-
parameters are set as follows: batch size = $64$, learning rate = $10^{-4}$,
the learning rate is decayed by multiplying $0.8$ after every $200$ epochs,
weight decay = $5\times 10^{-4}$, dropout probability = $0.3$. The model is
optimized using Adam optimizer with default parameters ($\beta_{1}=0.9$,
$\beta_{2}=0.999$) for $650$ epochs with early stopping to get the best
performing model. For MSR-VTT dataset, we set $\lambda=1$ and for MSVD
dataset, $\lambda=2$. $\lambda_{KL}$ is set to $4$ for both datasets.

(a) STG-KD Pan et al. (2020): A woman is cooking. Ours: A person is cutting a
piece of meat. GT: A person is cutting mushroom.

(b) STG-KD Pan et al. (2020): A boy kicks a goal. Ours: A boy is kicking a
soccer ball. GT: A boy kicks a soccer ball.

(c) STG-KD Pan et al. (2020): A person is cooking. Ours: A woman is mixing
ingredients. GT: A woman is mixing water and flour.

Fig. 9: Qualitative visualization of CoSB branch’s COSAM spatial attention on
MSVD dataset (best viewed in color)

(a) STG-KD Pan et al. (2020): A man is pouring pasta on to a container. Ours:
A man is putting a lid on a plastic container. GT: A man puts a lid on a
plastic container.

(b) STG-KD Pan et al. (2020): A woman is slicing carrot. Ours: A woman is
slicing octopus. GT: A woman is slicing octopus.

Fig. 10: Qualitative visualization of CoSB branch’s COSAM spatial attention on
MSVD dataset (best viewed in color)

####  5.2.4 Comparison with state-of-the-art methods

We compare our model with recent state-of-the-art methods: SA-LSTM (Wang et
al. (2018a)), M3 (Wang et al. (2018b)), RecNet (Wang et al. (2018a)), PickNet
(Chen et al. (2018d)), MARN (Pei et al. (2019)), POS+VCT (Hou et al. (2019)),
ORG-TRL (Zhang et al. (2020)), OA-BGT (Zhang and Peng (2019)), STG-KD (Pan et
al. (2020)).

Table 15 shows the comparison of our model with the state-of-the-art methods.
Table 15 is split into three logical groups that signifies different line of
works in the video-captioning literature. First group (POS+VCT (Hou et al.
(2019)), ORG-TRL (Zhang et al. (2020))) focuses on improving decoder in their
video captioning model by directly optimizing the decoder with $<$POS$>$ tags
and pre-trained BERT (Devlin et al. (2018)) modules. The second group (OA-BTG
(Zhang and Peng (2019)), STG-KD (Pan et al. (2020))) employ a pre-trained
object detector to introduce the concept of objects inside the model. The
third group (RecNet ( Wang et al. (2018a)), PickNet (Chen et al. (2018d)),
MARN (Pei et al. (2019)), SA-LSTM (Wang et al. (2018a)), M3 (Wang et al.
(2018b))) focuses on encoders without the help of external pre-trained models
like object detectors / optical flow estimators. Our model belongs to third
category where we focus on getting a better encoder representation without
using external pre-trained models and dependencies. Note that we follow the
standard procedure in the literature (Pei et al. (2019)) to not compare with
models based on reinforcement Learning (RL) (Wang et al. (2019)) techniques.

From the Table 15, in MSRVTT dataset, we can see that our model gets
competitive performance in B@4, ROUGE_L measures. Specifically, our model
performs 1 and 0.3 points better than its most comparable method MARN in B@4
and ROUGE_L respectively. Further, our model performs 1.2 and 2.4 points over
PickNet (Chen et al. (2018d)) in ROUGE_L and CIDEr metrics respectively. We
hypothesize that the moderate performance of our model in MSR-VTT is due to
the high number of discontinuous clips in these videos, thus it may be
difficult for COSAM to determine salient regions.

In MSVD dataset, our model outperforms on 3 metrics in the third logical
group: METEOR, ROUGE_L and CIDEr showing the effectiveness of combination of
GSB and CoSB branches. We achieve state-of-the-art CIDEr score outperforming
even the models that use external object detectors (Chen et al. (2019a)) and
those that focuses on both encoder and decoder (Zhang et al. (2020)). CIDEr
being a metric that is better correlated with human judgement, this
performance improvement shows that our model is able to generalize to test set
and generate better captions.

Components | B@4 | M | R | C
---|---|---|---|---
GSB | CoSB
COSAM | SRIM
✓ | ✗ | ✗ | 48.0 | 34.5 | 71.2 | 93.3
✗ | ✓ | ✓ | 38.3 | 28.3 | 64.7 | 49.1
✓ | ✓ | ✗ | 50.4 | 34.4 | 71.7 | 92.7
✓ | ✗ | ✓ | 49.2 | 35.2 | 71.9 | 92.7
✓ | ✓ | ✓ | 49.4 | 35.0 | 71.5 | 96.9
✓ | ✓ | ✓(m) | 50.7 | 35.3 | 72.1 | 97.8
Table 16: Ablation study of the proposed model on MSVD dataset. Here, B@4 =
BLUE@4, M = METEOR, R = ROUGE_L, C = CIDEr. ’m’ in SRIM column indicates
masked multihead self-attention.

####  5.2.5 Qualitative Analysis

In this section, we illustrate some of the interesting qualitative
visualization from our model showing its ability to localize the important
salient regions to aid video captioning. As our model is similar to Pan et al.
(2020)333We reproduce and train the model on our own for producing these
results. but without using external object detector, we also provide captions
from Pan et al. (2020) in the figures to have a relative comparison. In the
qualitative visualizations (Fig. 9(a), 9(b), 9(c), 10(a) and 10(b)), we show
the visualizations of the spatial mask from COSAM module of CoSB branch. From
the visualizations, it is evident from the masks that our model is able to
attend key salient regions such as objects, persons in the video.
Specifically, in Fig. 9(a), our model attempts to generate more detailed
captions by adding a quantifier like ”a piece of”. Similarly, in Fig. 9(c),
”mixing ingredients” is more detailed about the particular action than
”cooking”. In Fig. 10(a), our model is able to capture the ”closing“ action
between the person and the box lid, as well as it is able to generate a
caption giving specific details about ”plastic container“. Further, in Fig.
9(b), our model is able to attend to the cross bar of the goal post which is
an important cue to showcase that the boy is hitting a ”soccer ball”. We
notice a key observation that our model is better at classifying/recognizing
objects that are important for captioning tasks. For example, Fig. 10(b) shows
that our model is correctly able to classify ”octopus”, where as Pan et al.
(2020) classifies it as ”carrot”. We hypothesize that object detectors may
face difficulty in detecting and recognizing such uncommon classes that are
not present in their training dataset. Hence, letting the model to figure out
the salient regions without biasing it with an external object detector is a
potent approach.

####  5.2.6 Ablation studies

In this section, we perform ablation studies of different components of the
CoSB branch. The ablation study results are shown in Table 16. In the first
experiment (Row 1), we use a simple model by having only the GSB with the
cross entropy loss and show the performance. In the next experiment (Row 2),
we train only CoSB with the cross entropy loss which performs inferior to GSB-
only model. It may be due to the use of only 2D intermediate features without
temporal context. Further, from the third experiment, along with GSB, we start
integrating the components of CoSB one by one. Row 3 shows the performance of
our model with GSB and COSAM. This variant improves the GSB-only performance
by 2.4, 0.5 points in B@4 and ROUGE_L metrics respectively. Similarly, we
train GSB and SRIM (Row 4) to showcase relevance of spatial interactions which
results in improved in METEOR and ROUGE_L. Row 5 shows the performance of the
model with GSB+COSAM+SRIM. It can be seen that this combination improves CIDEr
metric significantly by 3.6 points over GSB-only model. In the next experiment
(Row 6), we modify SRIM to restrict interaction of salient-region features
(i.e., use a mask in multi-head self-attention) to be only within adjacent
frames. We observe that this masked multi-head self-attention improves the
model’s performance. Specifically, it improves 1.3, 0.9 points on B@4 and
CIDEr metrics over the model where SRIM has all-to-all self attention. We
hypothesize that constraining salient region interaction only to adjacent
frames may avoid noisy feature interaction between frames with higher time
gap, thus improving the performance.

Backbone | COSAM? |  temporal modeling? |  #params (M) |  #Flops (G) | HMDB51 | UCF101
---|---|---|---|---|---|---
Top-1% | Top-3% | Top-1% | Top-3%
ResNeXt101 (Kalfaoglu et al. (2020)) | ✗ | LSTM | 47.6 | 38.64 | 73.68 | 87.46 | 93.90 | 98.05
ResNeXt101 | ✓ | LSTM | 48.41 | 38.77 | 75.16 | 89.22 | 94.59 | 98.52
ResNeXt101 (Kalfaoglu et al. (2020)) | ✗ | BERT | 47.4 | 38.37 | 76.08 | 90.46 | 95.50 | 98.23
ResNeXt101 | ✓ | BERT | 48.21 | 38.49 | 77.52 | 92.55 | 95.96 | 98.84
I3D (Kalfaoglu et al. (2020)) | ✗ | BERT | 13.57 | 110.6 | 68.63 | 87.78 | 92.50 | 98.26
I3D | ✓ | BERT | 14.23 | 110.7 | 69.38 | 87.95 | 93.05 | 98.63
Table 17: The performance comparison of single stream RGB model from Kalfaoglu
et al. (2020) with and without COSAM layer.

###  5.3 Video action classification

In this section, we present the experimental results of applying COSAM to
video action classification task along with comparison to the baseline
architecture.

####  5.3.1 Datasets

We conduct the experiments on two action classification datasets (HMDB51
(Kuehne et al. (2011)) and UCF101 (Soomro et al. (2012))) in the literature.

##### UCF101

This dataset includes 13220 videos amounting to more than 27 hours of video
data, belonging to 101 action classes. This dataset consists of realistic user
uploaded videos with minimum length of 28 frames and average length of 180
frames per video containing scenes with cluttered background and camera ego-
motion. The 101 action classes of the dataset can be broadly clustered into
five types namely: 1) Human-Object Interaction, 2) Body-Motion Only, 3) Human-
Human Interaction, 4) Playing Musical Instruments, and 5) Sports.

##### HMDB51

The dataset consists of 51 action categories, each containing $\sim$101 clips
for a total of 6,766 video clips extracted from movie scenes and YouTube. The
videos contain various nuisance factors such as camera motion, viewpoint, pose
variations, poor video quality and partial occlusions that may hinder action
classification. Every video in this dataset has a minimum of 18 frames. The 51
action categories of the dataset can be divided into five types namely: 1)
General facial actions, 2) Facial actions with object manipulation, 3) General
body movements, 4) Body movements with object interaction, and 5) Body
movements for human interactions.

####  5.3.2 Evaluation metrics

The datasets are organized into three splits containing predefined number of
training and testing videos. Every UCF101 split consists of $9.5K$ training
videos, where as an HMDB51 split contains $3.7K$ training videos. To calculate
the final performance of the model, we follow the standard evaluation
procedure followed in the baseline (Kalfaoglu et al. (2020)) by finding the
mean top-1, top-3 accuracy across three splits and report the results.

Two-stream | Method | use flow? | HMDB51 | UCF101
---|---|---|---|---
TwoStream (Simonyan and Zisserman (2014a)) | ✓ | 59.40 | 88.00
TwoStream Fusion + IDT (Feichtenhofer et al. (2016)) | ✓ | 69.20 | 93.50
ActionVlad + IDT (Girdhar et al. (2017)) | ✓ | 69.80 | 93.60
Temp. Segment (Wang et al. (2018c)) | ✓ | 71.00 | 94.90
R(2+1)D (Tran et al. (2018)) | ✓ | 78.70 | 97.30
I3D (Carreira and Zisserman (2017)) | ✓ | 80.90 | 97.80
MARS + RGB + Flow (Crasto et al. (2019)) | ✓ | 80.90 | 98.10
BubbleNet (LO et al. (2020)) | ✓ | 82.6 | 97.2
| ResNeXt101 BERT (Kalfaoglu et al. (2020)) | ✓ | 83.55 | 97.87
Single-stream | IDT (Wang and Schmid (2013)) | ✗ | 61.70 | -
R(2+1)D (Tran et al. (2018)) | ✗ | 74.50 | 96.80
MARS + RGB (Crasto et al. (2019)) | ✗ | 73.10 | 95.60
TemporalShift (Lin et al. (2019)) | ✗ | 73.50 | 95.90
ResNeXt101 BERT (Kalfaoglu et al. (2020)) | ✗ | 76.08 | 94.59
ResNeXt101 + COSAM + BERT (ours) | ✗ | 77.52 | 95.96
Table 18: State-of-the-art performance comparison of deep models for video
action classification task.

####  5.3.3 Implementation details

We implement the network architecture using PyTorch framework (Paszke et al.
(2019)). We keep the training settings same as the baseline method (Kalfaoglu
et al. (2020))444https://github.com/artest08/LateTemporalModeling3DCNN. We
utilize AdamW optimizer with $\beta 1=0.9$, $\beta 2=0.999$ and weight decay =
$1e^{-3}$. During training, the following hyper-parameters are used: the video
frames are resized to have height = $112$, width = $112$, 64 frames are
uniformly sampled from each video, batch size = 8 for ResNeXt101 and 4 for
I3D, initial learning rate $10^{-5}$, we drop the learning rate by multiplying
it with 0.1 when the loss plateaus after an epoch, The linear projection
layers of BERT, the classification token (CLS) and the learned positional
embeddings are initialized with the zero-mean normal distribution with
standard deviation of 0.02.

####  5.3.4 Comparison with baseline method

In this section, we illustrate the performance comparison between the baseline
3D CNN model (Kalfaoglu et al. (2020)) and COSAM-incorporated model. The
experimental results are shown in the Table 17. Incorporating COSAM module
into the ResNeXt-101 feature extractor (Tab. 17 row 2) with LSTM as temporal
modeling scheme improves the baseline by +1.5% and +0.7% top-1 accuracy on
HMDB51 and UCF101 datasets respectively. Further, the COSAM-based model with
ResNeXt-101 feature extractor and BERT pooling layer (Tab. 17 row 4) performs
better than its baseline model by +1.4% & +0.5% on HMDB51 and UCF101 datasets
respectively. Notably, the models with BERT layer outperform the models with
LSTM temporal modeling layer as shown in Kalfaoglu et al. (2020). From the
performance improvements of COSAM models as shown in Tab. 17, it is evident
that COSAM is able to generalize irrespective of the temporal modeling scheme
(LSTM or BERT). Next, we incorporate COSAM into I3D feature extractor along
with BERT temporal modeling layer to evaluate the generalization of COSAM
across different feature extractors. As shown in the Table 17, I3D + COSAM
model performs +0.75% and +0.5% better on HMDB51 and UCF101 top-1 accuracy
respectively. The improvements on two different network architectures shows
that COSAM is able to generalize the performance improvements and have
potential to be applied across various model architectures.

Fig. 11: Visualization of COSAM spatial attention output for “Sword Exercise”
class from HMDB51 dataset Fig. 12: Visualization of COSAM spatial attention
output for “Hit” class from HMDB51 dataset Fig. 13: Visualization of COSAM
spatial attention output for “Horse Riding” class from UCF101 dataset Fig. 14:
Visualization of COSAM spatial attention output for “Playing Guitar” class
from UCF101 dataset

####  5.3.5 Comparison with the state-of-the-art methods

We compare the COSAM-based model with the following state-of-the-art methods
with single-stream models (IDT (Wang and Schmid (2013)), R(2+1)D (Tran et al.
(2018)), TemporalShift (Lin et al. (2019)), ResNeXt101 BERT (Kalfaoglu et al.
(2020))) and models with two-streams (TwoStream (Simonyan and Zisserman
(2014a)), TwoStream Fusion + IDT (Feichtenhofer et al. (2016)), ActionVlad +
IDT (Girdhar et al. (2017)), Temp. Segment (Wang et al. (2018c)), R(2+1)D
(Tran et al. (2018)), I3D (Carreira and Zisserman (2017)), , MARS + RGB + Flow
(Crasto et al. (2019)), BubbleNet (LO et al. (2020)), ResNeXt101 BERT
(Kalfaoglu et al. (2020))).

The comparison with the state-of-the-art methods are shown in Table 18. Table
18 is split into two logical categories namely: 1) Single-stream models that
use only the RGB frames of the videos as input, 2) Two-stream models that use
both RGB and optical flow inputs. Overall, the two-stream methods perform
superior to the single-stream methods due to the explicit motion cues provided
in terms of optical flow. However, they suffer from the disadvantage of high
latency due to the need of optical flow estimation between video frames. In
our work, we are motivated to improve the performance single-stream approach
by incorporating COSAM into the model and thus we compare our method with
single-stream approaches. In the single-stream category with only appearance
input (RGB frames), our COSAM-based model (ResNeXt101+COSAM+BERT) performs
superior to the state-of-the-art methods with only RGB input. Specifically,
our method performs better than Kalfaoglu et al. (2020) with +1.44% and +1.37%
Top-1 accuracy on HMDB51 and UCF101 datasets respectively. Further, our method
shows significant performance improvement over TemporalShift (Lin et al.
(2019)) by +4.02%, MARS+RGB (Crasto et al. (2019)) by +4.42%, R(2+1)D (Tran et
al. (2018)) by +3.02% respectively on HMDB51 dataset. COSAM can be considered
as orthogonal approach to these methods and may be combined with some of the
methods to improve the performance further.

####  5.3.6 Qualitative Analysis

In this section, we illustrate some of the interesting visualizations (Fig.
11, 12, 13 & 14) of COSAM spatial attention maps from the COSAM-based model
with ResNeXt-101 backbone and BERT temporal modeling layer.

In general, we observe that COSAM is able to attend to the salient object
regions needed for the task (i.e., video action classification) and refine the
features by suppressing the background noisy features. For instance, in Fig.
11, to classify the class ”Sword Exercise”, COSAM is able to focus on the
regions that the object (i.e., sword) is present in the image. Further, it is
able to attend on the same salient regions coherently across all the video
frames. Next, in Fig. 12, the salient objects such as persons and baseball bat
are captured in the attention map for the class “Hit”. COSAM aids in
automatically attending to the objects that lie in tail distribution such as
baseball bat which may not be detected by pre-trained object detectors. Next,
Fig. 13 shows that COSAM is able to capture the person and horse regions to
classify the video frames into “Horse riding” category. Note that the salient
regions are detected regardless of partial occlusion and color ambiguity.
Finally, Fig. 14 illustrates the ability of COSAM to focus on key semantic
object (Guitar) needed to classify the action (Playing guitar) with high
confidence than the other objects (i.e., Person) present in the video.

##  6 Conclusion and Future work

In this work, we presented a generic co-segmentation based attention module
(COSAM) that can be plugged-in within any backbone network. COSAM enables the
model to automatically learn to attend to salient object regions for the
underlying task in an end-to-end manner. To illustrate the applicability of
COSAM in various video-based computer vision tasks, we have incorporated it
into three major tasks namely: 1) Video-based person re-ID, 2) Video
captioning & 3) Video action classification. Through a comprehensive set of
empirical experiments, we have shown that inclusion of COSAM consistently
improves the performance of the underlying baseline models with an added
advantage of interpretable visual results. Further, we have shown a set of
qualitative visualizations demonstrating the capability of COSAM to attend to
salient regions for the underlying task. Given the generic nature of the COSAM
module, its effectiveness on other video based tasks such as visual tracking,
video object segmentation can be pursued as a possible future work.

## Acknowledgments

This work is supported by grants from PM’s fellowship for Doctoral Research
(SERB, India) & Google PhD Fellowship to Arulkumar Subramaniam.

## References

  * Bak and Carr (2017) Bak, S., Carr, P., 2017.  One-shot metric learning for person re-identification, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2990–2999.
  * Barrett and Siskind (2015) Barrett, D.P., Siskind, J.M., 2015\.  Action recognition by time series of retinotopic appearance and motion features.  IEEE Transactions on Circuits and Systems for Video Technology 26, 2250–2263.
  * Bazzani et al. (2014) Bazzani, L., Cristani, M., Murino, V., 2014.  Sdalf: modeling human appearance with symmetry-driven accumulation of local features, in: Person re-identification. Springer, pp. 43–69.
  * Bruhn et al. (2005) Bruhn, A., Weickert, J., Schnörr, C., 2005.  Lucas/kanade meets horn/schunck: Combining local and global optic flow methods.  International journal of computer vision 61, 211–231.
  * Carreira and Zisserman (2017) Carreira, J., Zisserman, A., 2017\.  Quo vadis, action recognition? a new model and the kinetics dataset, in: proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6299–6308.
  * Chang and Wang (2015) Chang, H.S., Wang, Y.C.F., 2015\.  Optimizing the decomposition for multiple foreground cosegmentation.  Computer Vision and Image Understanding 141, 18–27.
  * Chen et al. (2020) Chen, C., Zheng, Z., Ding, X., Huang, Y., Dou, Q., 2020\.  Harmonizing transferability and discriminability for adapting object detectors, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8869–8878.
  * Chen and Dolan (2011) Chen, D., Dolan, W.B., 2011\.  Collecting highly parallel data for paraphrase evaluation, in: Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies, pp. 190–200.
  * Chen et al. (2018a) Chen, D., Li, H., Xiao, T., Yi, S., Wang, X., 2018a.  Video person re-identification with competitive snippet-similarity aggregation and co-attentive snippet embedding, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1169–1178.
  * Chen et al. (2018b) Chen, H., Huang, Y., Nakayama, H., 2018b.  Semantic aware attention based deep object co-segmentation.  arXiv preprint arXiv:1810.06859 .
  * Chen et al. (2019a) Chen, K., Wang, J., Pang, J., Cao, Y., Xiong, Y., Li, X., Sun, S., Feng, W., Liu, Z., Xu, J., et al., 2019a.  Mmdetection: Open mmlab detection toolbox and benchmark.  arXiv preprint arXiv:1906.07155 .
  * Chen et al. (2018c) Chen, M., Li, Y., Zhang, Z., Huang, S., 2018c.  Tvt: Two-view transformer network for video captioning, in: Asian Conference on Machine Learning, PMLR. pp. 847–862.
  * Chen and Wu (2017) Chen, Q., Wu, R., 2017.  Cnn is all you need.  arXiv preprint arXiv:1712.09662 .
  * Chen et al. (2015) Chen, X., Fang, H., Lin, T.Y., Vedantam, R., Gupta, S., Dollár, P., Zitnick, C.L., 2015\.  Microsoft coco captions: Data collection and evaluation server.  arXiv preprint arXiv:1504.00325 .
  * Chen et al. (2019b) Chen, Y., Rohrbach, M., Yan, Z., Shuicheng, Y., Feng, J., Kalantidis, Y., 2019b.  Graph-based global reasoning networks, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 433–442.
  * Chen et al. (2018d) Chen, Y., Wang, S., Zhang, W., Huang, Q., 2018d.  Less is more: Picking informative frames for video captioning, in: Proceedings of the European conference on computer vision (ECCV), pp. 358–373.
  * Chi et al. (2020) Chi, L., Yuan, Z., Mu, Y., Wang, C., 2020.  Non-local neural networks with grouped bilinear attentional transforms, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11804–11813.
  * Chung et al. (2017) Chung, D., Tahboub, K., Delp, E.J., 2017.  A two stream siamese convolutional neural network for person re-identification, in: Proceedings of the IEEE International Conference on Computer Vision, pp. 1983–1991.
  * Chung et al. (2014) Chung, J., Gulcehre, C., Cho, K., Bengio, Y., 2014\.  Empirical evaluation of gated recurrent neural networks on sequence modeling.  arXiv preprint arXiv:1412.3555 .
  * Crasto et al. (2019) Crasto, N., Weinzaepfel, P., Alahari, K., Schmid, C., 2019\.  Mars: Motion-augmented rgb stream for action recognition, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7882–7891.
  * Dalal and Triggs (2005) Dalal, N., Triggs, B., 2005\.  Histograms of oriented gradients for human detection, in: international Conference on computer vision & Pattern Recognition (CVPR’05), IEEE Computer Society. pp. 886–893.
  * Deng et al. (2009a) Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L., 2009a.  Imagenet: A large-scale hierarchical image database, in: 2009 IEEE conference on computer vision and pattern recognition, Ieee. pp. 248–255.
  * Deng et al. (2009b) Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L., 2009b.  ImageNet: A Large-Scale Hierarchical Image Database, in: CVPR09.
  * Denkowski and Lavie (2014) Denkowski, M., Lavie, A., 2014\.  Meteor universal: Language specific translation evaluation for any target language, in: Proceedings of the ninth workshop on statistical machine translation, pp. 376–380.
  * Devlin et al. (2018) Devlin, J., Chang, M.W., Lee, K., Toutanova, K., 2018\.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805 .
  * Diba et al. (2017) Diba, A., Fayyaz, M., Sharma, V., Karami, A.H., Arzani, M.M., Yousefzadeh, R., Van Gool, L., 2017.  Temporal 3d convnets: New architecture and transfer learning for video classification.  arXiv preprint arXiv:1711.08200 .
  * Ding et al. (2015) Ding, S., Lin, L., Wang, G., Chao, H., 2015.  Deep feature learning with relative distance comparison for person re-identification.  Pattern Recognition 48, 2993–3003.
  * Donahue et al. (2015) Donahue, J., Anne Hendricks, L., Guadarrama, S., Rohrbach, M., Venugopalan, S., Saenko, K., Darrell, T., 2015.  Long-term recurrent convolutional networks for visual recognition and description, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2625–2634.
  * Feichtenhofer et al. (2016) Feichtenhofer, C., Pinz, A., Zisserman, A., 2016.  Convolutional two-stream network fusion for video action recognition, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1933–1941.
  * Fu et al. (2018) Fu, Y., Wei, Y., Zhou, Y., Shi, H., Huang, G., Wang, X., Yao, Z., Huang, T., 2018.  Horizontal pyramid matching for person re-identification.  arXiv preprint arXiv:1804.05275 .
  * Gao and Nevatia (2018) Gao, J., Nevatia, R., 2018.  Revisiting temporal modeling for video-based person reid.  arXiv preprint arXiv:1805.02104 .
  * Girdhar et al. (2019) Girdhar, R., Carreira, J., Doersch, C., Zisserman, A., 2019\.  Video action transformer network, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 244–253.
  * Girdhar et al. (2017) Girdhar, R., Ramanan, D., Gupta, A., Sivic, J., Russell, B., 2017.  Actionvlad: Learning spatio-temporal aggregation for action classification, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 971–980.
  * Gwilliam et al. (2021) Gwilliam, M., Hegde, S., Tinubu, L., Hanson, A., 2021\.  Rethinking common assumptions to mitigate racial bias in face recognition datasets, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4123–4132.
  * He et al. (2017) He, K., Gkioxari, G., Dollár, P., Girshick, R., 2017\.  Mask r-cnn, in: Proceedings of the IEEE international conference on computer vision, pp. 2961–2969.
  * He et al. (2016) He, K., Zhang, X., Ren, S., Sun, J., 2016.  Deep residual learning for image recognition, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778.
  * Hermans et al. (2017) Hermans, A., Beyer, L., Leibe, B., 2017.  In defense of the triplet loss for person re-identification.  arXiv preprint arXiv:1703.07737 .
  * Hochreiter and Schmidhuber (1997) Hochreiter, S., Schmidhuber, J., 1997\.  Long short-term memory.  Neural computation 9, 1735–1780.
  * Horn and Schunck (1981) Horn, B.K., Schunck, B.G., 1981\.  Determining optical flow.  Artificial intelligence 17, 185–203.
  * Hou et al. (2019) Hou, J., Wu, X., Zhao, W., Luo, J., Jia, Y., 2019\.  Joint syntax representation learning and visual cue translation for video captioning, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8918–8927.
  * Hsu et al. (2018) Hsu, K.J., Lin, Y.Y., Chuang, Y.Y., 2018.  Co-attention cnns for unsupervised object co-segmentation., in: IJCAI, pp. 748–756.
  * Hu et al. (2018) Hu, J., Shen, L., Sun, G., 2018.  Squeeze-and-excitation networks, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132–7141.
  * Huang et al. (2019) Huang, Z., Wang, X., Huang, L., Huang, C., Wei, Y., Liu, W., 2019.  Ccnet: Criss-cross attention for semantic segmentation, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 603–612.
  * Ioffe and Szegedy (2015) Ioffe, S., Szegedy, C., 2015\.  Batch normalization: Accelerating deep network training by reducing internal covariate shift, in: International Conference on Machine Learning, pp. 448–456.
  * Jerripothula et al. (2016) Jerripothula, K.R., Cai, J., Yuan, J., 2016.  Image co-segmentation via saliency co-fusion.  IEEE Transactions on Multimedia 18, 1896–1909.
  * Joulin et al. (2012) Joulin, A., Bach, F., Ponce, J., 2012.  Multi-class cosegmentation, in: 2012 IEEE Conference on Computer Vision and Pattern Recognition, IEEE. pp. 542–549.
  * Kalfaoglu et al. (2020) Kalfaoglu, M.E., Kalkan, S., Alatan, A.A., 2020.  Late temporal modeling in 3d cnn architectures with bert for action recognition, in: European Conference on Computer Vision, Springer. pp. 731–747.
  * Karpathy et al. (2014) Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., Fei-Fei, L., 2014\.  Large-scale video classification with convolutional neural networks, in: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 1725–1732.
  * Kay et al. (2017) Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan, S., Viola, F., Green, T., Back, T., Natsev, P., et al., 2017\.  The kinetics human action video dataset.  arXiv preprint arXiv:1705.06950 .
  * Kendall et al. (2017) Kendall, A., Martirosyan, H., Dasgupta, S., Henry, P., Kennedy, R., Bachrach, A., Bry, A., 2017.  End-to-end learning of geometry and context for deep stereo regression, in: Proceedings of the IEEE International Conference on Computer Vision, pp. 66–75.
  * Krizhevsky et al. (2012) Krizhevsky, A., Sutskever, I., Hinton, G.E., 2012.  Imagenet classification with deep convolutional neural networks, in: Advances in neural information processing systems, pp. 1097–1105.
  * Kuehne et al. (2011) Kuehne, H., Jhuang, H., Garrote, E., Poggio, T., Serre, T., 2011.  Hmdb: a large video database for human motion recognition, in: 2011 International conference on computer vision, IEEE. pp. 2556–2563.
  * Li et al. (2017) Li, D., Chen, X., Zhang, Z., Huang, K., 2017.  Learning deep context-aware features over body and latent parts for person re-identification, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 384–393.
  * Li et al. (2018a) Li, L., Liu, Z., Zhang, J., 2018a.  Unsupervised image co-segmentation via guidance of simple images.  Neurocomputing 275, 1650–1661.
  * Li et al. (2018b) Li, S., Bak, S., Carr, P., Wang, X., 2018b.  Diversity regularized spatiotemporal attention for video-based person re-identification, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 369–378.
  * Li et al. (2018c) Li, W., Jafari, O.H., Rother, C., 2018c.  Deep object co-segmentation.  arXiv preprint arXiv:1804.06423 .
  * Liao et al. (2015) Liao, S., Hu, Y., Zhu, X., Li, S.Z., 2015.  Person re-identification by local maximal occurrence representation and metric learning, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2197–2206.
  * Lin (2004) Lin, C.Y., 2004.  Rouge: A package for automatic evaluation of summaries, in: Text summarization branches out, pp. 74–81.
  * Lin et al. (2019) Lin, J., Gan, C., Han, S., 2019.  Temporal shift module for efficient video understanding. 2019 ieee, in: CVF International Conference on Computer Vision (ICCV), pp. 7082–7092.
  * Liu et al. (2018) Liu, H., Jie, Z., Jayashree, K., Qi, M., Jiang, J., Yan, S., Feng, J., 2018.  Video-based person re-identification with accumulative motion context.  IEEE transactions on circuits and systems for video technology 28, 2788–2802.
  * Liu et al. (2017) Liu, Y., Yan, J., Ouyang, W., 2017.  Quality aware network for set to set recognition, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5790–5799.
  * LO et al. (2020) LO, B.I., HC, M.V., Schwartz, W.R., 2020.  Bubblenet: A disperse recurrent structure to recognize activities, in: 2020 IEEE International Conference on Image Processing (ICIP), IEEE. pp. 2216–2220.
  * Lowe (2004) Lowe, D.G., 2004.  Distinctive image features from scale-invariant keypoints.  International journal of computer vision 60, 91–110.
  * Ma et al. (2012) Ma, B., Su, Y., Jurie, F., 2012.  Local descriptors encoded by fisher vectors for person re-identification, in: European Conference on Computer Vision, Springer. pp. 413–422.
  * Ma et al. (2018) Ma, C.Y., Kadav, A., Melvin, I., Kira, Z., AlRegib, G., Graf, H.P., 2018\.  Attend and interact: Higher-order object interactions for video understanding, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6790–6800.
  * McLaughlin et al. (2016) McLaughlin, N., Martinez del Rincon, J., Miller, P., 2016.  Recurrent convolutional network for video-based person re-identification, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1325–1334.
  * Nair and Hinton (2010) Nair, V., Hinton, G.E., 2010\.  Rectified linear units improve restricted boltzmann machines, in: Proceedings of the 27th international conference on machine learning (ICML-10), pp. 807–814.
  * Nguyen et al. (2020) Nguyen, N.D., Do, T., Ngo, T.D., Le, D.D., 2020.  An evaluation of deep learning methods for small object detection.  Journal of Electrical and Computer Engineering 2020\.
  * Pan et al. (2020) Pan, B., Cai, H., Huang, D.A., Lee, K.H., Gaidon, A., Adeli, E., Niebles, J.C., 2020\.  Spatio-temporal graph for video captioning with knowledge distillation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10870–10879.
  * Pan et al. (2016) Pan, Y., Mei, T., Yao, T., Li, H., Rui, Y., 2016\.  Jointly modeling embedding and translation to bridge video and language, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4594–4602.
  * Papineni et al. (2002) Papineni, K., Roukos, S., Ward, T., Zhu, W.J., 2002\.  Bleu: a method for automatic evaluation of machine translation, in: Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311–318.
  * Park et al. (2018) Park, J., Woo, S., Lee, J.Y., Kweon, I.S., 2018.  Bam: Bottleneck attention module.  arXiv preprint arXiv:1807.06514 .
  * Paszke et al. (2017) Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., Lerer, A., 2017\.  Automatic differentiation in pytorch, in: NIPS-W.
  * Paszke et al. (2019) Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al., 2019\.  Pytorch: An imperative style, high-performance deep learning library.  Advances in neural information processing systems 32, 8026–8037.
  * Pei et al. (2019) Pei, W., Zhang, J., Wang, X., Ke, L., Shen, X., Tai, Y.W., 2019.  Memory-attended recurrent network for video captioning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8347–8356.
  * Petsiuk et al. (2021) Petsiuk, V., Jain, R., Manjunatha, V., Morariu, V.I., Mehra, A., Ordonez, V., Saenko, K., 2021.  Black-box explanation of object detectors via saliency maps, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11443–11452.
  * Prosser et al. (2010) Prosser, B.J., Zheng, W.S., Gong, S., Xiang, T., Mary, Q., 2010.  Person re-identification by support vector ranking., in: BMVC, p. 6.
  * Qi et al. (2018) Qi, L., Huo, J., Wang, L., Shi, Y., Gao, Y., 2018\.  Maskreid: A mask based deep ranking neural network for person re-identification.  arXiv preprint arXiv:1804.03864 .
  * Redmon et al. (2016) Redmon, J., Divvala, S., Girshick, R., Farhadi, A., 2016\.  You only look once: Unified, real-time object detection, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 779–788.
  * Ren et al. (2015) Ren, S., He, K., Girshick, R., Sun, J., 2015.  Faster r-cnn: Towards real-time object detection with region proposal networks, in: Advances in neural information processing systems, pp. 91–99.
  * Ristani et al. (2016) Ristani, E., Solera, F., Zou, R., Cucchiara, R., Tomasi, C., 2016.  Performance measures and a data set for multi-target, multi-camera tracking, in: European Conference on Computer Vision, Springer. pp. 17–35.
  * Rocco et al. (2017) Rocco, I., Arandjelovic, R., Sivic, J., 2017.  Convolutional neural network architecture for geometric matching, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6148–6157.
  * Rohrbach et al. (2017) Rohrbach, A., Torabi, A., Rohrbach, M., Tandon, N., Pal, C., Larochelle, H., Courville, A., Schiele, B., 2017\.  Movie description.  International Journal of Computer Vision 123, 94–120.
  * Rother et al. (2006) Rother, C., Minka, T., Blake, A., Kolmogorov, V., 2006\.  Cosegmentation of image pairs by histogram matching-incorporating a global constraint into mrfs, in: 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), IEEE. pp. 993–1000.
  * Simonyan and Zisserman (2014a) Simonyan, K., Zisserman, A., 2014a.  Two-stream convolutional networks for action recognition in videos.  arXiv preprint arXiv:1406.2199 .
  * Simonyan and Zisserman (2014b) Simonyan, K., Zisserman, A., 2014b.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556 .
  * Singh et al. (2020) Singh, K.K., Mahajan, D., Grauman, K., Lee, Y.J., Feiszli, M., Ghadiyaram, D., 2020\.  Don’t judge an object by its context: Learning to overcome contextual bias, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11070–11078.
  * Song et al. (2018a) Song, C., Huang, Y., Ouyang, W., Wang, L., 2018a.  Mask-guided contrastive attention model for person re-identification, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1179–1188.
  * Song et al. (2018b) Song, G., Leng, B., Liu, Y., Hetang, C., Cai, S., 2018b.  Region-based quality estimation network for large-scale person re-identification, in: Thirty-Second AAAI Conference on Artificial Intelligence.
  * Soomro et al. (2012) Soomro, K., Zamir, A.R., Shah, M., 2012.  Ucf101: A dataset of 101 human actions classes from videos in the wild.  arXiv preprint arXiv:1212.0402 .
  * Su et al. (2017) Su, C., Li, J., Zhang, S., Xing, J., Gao, W., Tian, Q., 2017.  Pose-driven deep convolutional model for person re-identification, in: Proceedings of the IEEE International Conference on Computer Vision, pp. 3960–3969.
  * Subramaniam et al. (2016) Subramaniam, A., Chatterjee, M., Mittal, A., 2016.  Deep neural networks with inexact matching for person re-identification, in: Advances in Neural Information Processing Systems, pp. 2667–2675.
  * Subramaniam et al. (2019) Subramaniam, A., Nambiar, A., Mittal, A., 2019.  Co-segmentation inspired attention networks for video-based person re-identification, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 562–572.
  * Suh et al. (2018) Suh, Y., Wang, J., Tang, S., Mei, T., Mu Lee, K., 2018\.  Part-aligned bilinear representations for person re-identification, in: Proceedings of the European Conference on Computer Vision (ECCV), pp. 402–419.
  * Tang et al. (2021) Tang, R., Du, M., Li, Y., Liu, Z., Zou, N., Hu, X., 2021.  Mitigating gender bias in captioning systems, in: Proceedings of the Web Conference 2021, pp. 633–645.
  * Tao et al. (2017) Tao, Z., Liu, H., Fu, H., Fu, Y., 2017.  Image cosegmentation via saliency-guided constrained clustering with cosine similarity, in: Thirty-First AAAI Conference on Artificial Intelligence.
  * Tommasi et al. (2017) Tommasi, T., Patricia, N., Caputo, B., Tuytelaars, T., 2017\.  A deeper look at dataset bias, in: Domain adaptation in computer vision applications. Springer, pp. 37–55.
  * Tran et al. (2018) Tran, D., Wang, H., Torresani, L., Ray, J., LeCun, Y., Paluri, M., 2018\.  A closer look at spatiotemporal convolutions for action recognition, in: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 6450–6459.
  * Vaidya et al. (2022) Vaidya, J., Subramaniam, A., Mittal, A., 2022.  Co-segmentation aided two-stream architecture for video captioning., in: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision.
  * Vaswani et al. (2017) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I., 2017\.  Attention is all you need, in: Advances in neural information processing systems, pp. 5998–6008.
  * Vedantam et al. (2015) Vedantam, R., Lawrence Zitnick, C., Parikh, D., 2015.  Cider: Consensus-based image description evaluation, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4566–4575.
  * Venugopalan et al. (2015) Venugopalan, S., Rohrbach, M., Donahue, J., Mooney, R., Darrell, T., Saenko, K., 2015\.  Sequence to sequence-video to text, in: Proceedings of the IEEE international conference on computer vision, pp. 4534–4542.