Unified Framework with Consistency across Modalities for Human Activity Recognition
Tuyen Tran Thao Minh Le Hung Tran Truyen Tran



Unified Compositional Query Machine with Multimodal Consistency for
Video-based Human Activity Recognition
    
    
        Qian Niu12, 
        Junyu Liu2,
        Ziqian Bi3,
        Pohsun Feng4,
        Benji Peng5,
        Keyu Chen5
    
    
        2Kyoto University
    
    
        3Indiana University
    
    
        4National Taiwan Normal University
    
    
        5Georgia Institute of Technology
    
    
        Corresponding Email: niu.qian.f44@kyoto-u.jp
    

    September 9, 2024
===========================================================================================================================================================================================================================================================================================================================================================================================

Unified Compositional Query Machine with Multimodal Consistency for
Video-based Human Activity Recognition
    
    
        Qian Niu12, 
        Junyu Liu2,
        Ziqian Bi3,
        Pohsun Feng4,
        Benji Peng5,
        Keyu Chen5
    
    
        2Kyoto University
    
    
        3Indiana University
    
    
        4National Taiwan Normal University
    
    
        5Georgia Institute of Technology
    
    
        Corresponding Email: niu.qian.f44@kyoto-u.jp
    

    September 9, 2024
===========================================================================================================================================================================================================================================================================================================================================================================================



empty

COMPUTER

HUB

§ ABSTRACT 
Recognizing human activities in videos is challenging due to the spatio-temporal
complexity and context-dependence of human interactions. Prior studies
often rely on single input modalities, such as RGB or skeletal data,
limiting their ability to exploit the complementary advantages across
modalities. Recent studies focus on combining these two modalities
using simple feature fusion techniques. However, due to the inherent
disparities in representation between these input modalities, designing
a unified neural network architecture to effectively leverage their
complementary information remains a significant challenge. To address
this, we propose a comprehensive multimodal framework for robust video-based
human activity recognition. Our key contribution is the introduction
of a novel compositional query machine, called 
(COMPositional hUman-cenTric quERy
machine), a generic neural architecture that models the interactions
between a human of interest and its surroundings in both space and
time. Thanks to its versatile design,  can be leveraged
to distill distinctive representations for various input modalities.
Additionally, we introduce a consistency loss that enforces agreement
in prediction between modalities, exploiting the complementary information
from multimodal inputs for robust human movement recognition. Through
extensive experiments on action localization and group activity recognition
tasks, our approach demonstrates superior performance when compared
with state-of-the-art methods. Our code is available at: https://github.com/tranxuantuyen/COMPUTERhttps://github.com/tranxuantuyen/COMPUTER.






§ INTRODUCTION




Human activity recognition in videos is a crucial area of focus within
the field of Artificial Intelligence (AI), enabling numerous practical
applications in real-world scenarios <cit.>.
However, this is a challenging task due to the spatio-temporal complexity
and context-dependence of human interactions. These factors require
AI systems to robustly interpret and generalize across a wide range
of behaviors and environmental conditions.

Previous studies have explored representing human activities in consideration
of contextual factors <cit.>.
Tang et al.<cit.> analyze human actions
from an object-centric perspective, modeling the relationships between
humans and the surrounding objects. These works all model the dynamics
of visual scenes by focusing on how the relationships between entities
evolve over time. However, they rely solely on computationally expensive
RGB image sequences, making temporal representation from video data
challenging <cit.>. Additionally, using only RGB
data limits the capability to capture subtle body movements. 

Human body key points and skeleton data offer advantages in computational
costs and temporal modeling due to their compactness and robustness
against lighting conditions and scene variations <cit.>.
However, skeleton data lacks contextual information, limiting its
capability to represent spatial relationships involved in human-object
interactions.

Given the complementary attributes of RGB and skeleton data, a natural
question arises: how to design a unified model to effectively
combine these modalities for the task of human activity recognition?
However, it is not straightforward to build a joint representation
that leverages both modalities' strengths due to their inherent disparities.
HIT <cit.> was among the earliest attempt to address
this challenge. Their approach involved designing separate components
to process each modality independently, followed by a late fusion
technique. Due to significant structural differences between modalities,
late fusion performed poorly. This is because one modality may negatively
impact the other, ultimately reducing the representational capabilities
of the joint features.

To address the limitations of current methods, we first propose a
unified feature representation framework for multiple modalities in
human activity recognition. Second, we introduce a novel self-supervised
mechanism to ensure consistency in prediction using different modalities
to avoid negative cross-modality impacts within their joint representation.
Overview of the proposed approach is illustrated in Fig. <ref>.
To the best of our knowledge, we are the first to propose a generic
and modality-agnostic architecture, along with a novel mechanism for
multi-modal consistency for human activity recognition. To evaluate
the effectiveness of the proposed approach, we conduct intensive experiments
on two human activity recognition tasks: Spatio-Temporal Action Localization
and Group Activity Recognition.

In summary, our contribution is three-fold: (1) Introduction of a
unified compositional query machine for simultaneously handling multi-modal
inputs for the task of human-centric video understanding; (2) Introduction
of a novel mechanism to encourage consistency in prediction across
modalities in a self-supervised manner; (3) Conducting extensive experiments
and analyses across two tasks in human activity recognition in videos.




§ RELATED WORK





 §.§ Multi-modal human action recognition 


Prior works on human activity recognition mostly rely solely on RGB
features, discarding valuable information from other modalities. For
instance, skeleton data offers distinct advantages in recognizing
actions that require superior temporal modeling such as running or
driving a car <cit.>. Recognizing the benefits of
multi-modal input, some studies have attempted to incorporate additional
modalities beyond RGB features <cit.>.
PCSC <cit.> proposes to use optical flow to capture
motion, designing an inception-like model with an early fusion mechanism
to combine RGB with flow features. In contrast, <cit.>
extracts RGB and motion features using I3D <cit.>,
and then combines them with a late fusion mechanism. Most recently,
HIT <cit.> utilizes both skeleton data and RGB
features for spatio-temporal action localization using a simple late
fusion technique. While these approaches have shown some benefits
of using multi-modal inputs, neither early fusion or late fusion are
capable of building a joint representation that captures the complementary
advantages across modalities. Different from these works, our approach
uses a novel mechanism to leverage the consistency in prediction across
different modalities for robust human activity recognition. More importantly,
our newly introduced consistency loss allows us to train our proposed
method in an unsupervised manner without the need for additional training
data.



 §.§ Contrastive self-supervised learning


Contrastive self-supervised learning has gained popularity for its
ability to avoid the need for large-scale datasets. It requires the
sampling of positive and negative pairs from raw, unlabeled data.
During the learning process, it encourages convergence of the positive
pair representations in latent space while enforcing divergence of
the negative pairs. A prominent example is CLIP <cit.>,
where it constructs positive pairs consisting of an image and a sentence
describing the same object, and negative pairs consist of an image
and a sentence that refer to different objects. While structurally
different, visual and text-based latent representations should contain
mutual information linked to the same concept. This strategy is also
applied to image-image pairs for data augmentation, e.g., SimCLR <cit.>.
The intuition is to bring different augmented views of the same image
closer in  latent space, while pushing different augmented views apart.
Proven highly effective for data representation, this technique has
pioneered subsequent works <cit.>
for robust feature representation learning. In this work, we applied
this technique to human activity recognition using multi-modal input,
enforcing convergence of latent representations from different modalities
originating from the same actor, despite their structural disparities.





§ METHOD





 §.§ Preliminaries 


Formulation: Our goal is to design a model that leverages
multi-modal inputs, e.g., RGB sequences and human body key points,
for human activity recognition in videos. We achieve this by formulating
the problem under a neural query machine. Our query machine
takes as input a human-centric query { q_i} that
probes different aspects regarding the movements of a specific human
actor and its relationships with the surrounding entities within a
video input V. The output is a prediction of an action label ỹ,
based on the collective human-centric attributes in response to the
queries. Formally, our query machine is given as:


    ỹ=𝒜({ g^m(q_i,t^m,V)} _m).

For each modality m, q_i,t^m is i-th query at time step
t; g^m(.) is a neural building block that retrieves
relevant information in V in response to q_i,t^m; 𝒜(.)
is a neural network that aggregates the attributes from the input
modalities and maps them to label space. 

Our work investigates human activity recognition in videos under two
specific applications: Spatio-Temporal Action Localization and Group
Activity Recognition. Since human activity is usually interpreted
through different layers of interactions, such as self movements and
cross-entity interactions, we hypothesize a compositional function
for each modality-wise query machine g^m(.). This
compositional design places humans at the center of relational modeling
of their interactions with the surroundings (Sec. <ref>). 

Spatio-temporal video representation: Following recent studies
<cit.>, we first
extract a spatio-temporal representation for each video input V
using video feature extractors such as Slowfast <cit.>
and MViT models <cit.>. The video
V is usually segmented into T non-overlapping clips, resulting
in video features X∈ℝ^T× FHW× D, where F
is the number of frames in each clip, and H,W,D are the height,
width, and channel dimension of the feature maps, respectively. 

Query representation: We use two input modalities as queries:
human-centric visual appearance and body key points. 

Human-centric visual appearance: Visual appearance of human
actors themselves plays a crucial role in interpreting their actions.
To capture this, we follow <cit.>
to use an actor localization module to extract the appearance saliency
of human actors. First, for each video segment t in the T non-overlapping
clips from a video input, we utilize a human detector <cit.>
to localize human actors within their center frame. This yields a
set of bounding boxes for all N detected actors. We then use RoI-Align
<cit.> to extract visual appearance features for the N
actors: Q_t^vis={ q_i,t^vis| q_i,t^vis∈ℝ^1× D} _i=1^N.



Body key points: We use the common framework Detectron2 <cit.>
to detect human body key points from RGB frames. Similar to visual
appearance feature extraction, we use the middle frame of a video
clip t for pose detection, resulting in a set Q_t^key
of N person skeletons: Q_t^key={ q_i,t^key| q_i,t^key∈ℝ^1× D} _i=1^N.



 §.§ Compositional Human-centric Query Machine


We propose a novel family of model architectures, dubbed COMPositional
hUman-cenTric quERy machine (),
for multi-modal human activity recognition in videos. 
leverages a modular design, combining several identical modality-wise
query machines that model the relationships between human actors and
their surroundings in both space and time. This modular design simplifies
the construction of  by stacking identical building blocks,
facilitating dynamic model sizes and model's representation capabilities
for efficient action prediction.

Our query machines focus on two main types of interactions: human-human
interactions and human-context interactions. Inspired by Dang
et al. <cit.>, each modality-wise query machine
in  adopts a two-stage design, where the output of the
first stage serves as the input for the second stage. Figure <ref>
(on the left) provides a general architecture of . One
of the key advantages of 's modular design is its inherent
scalability. The system can be easily extended to incorporate additional
input modalities and handle different types of interactions. In this
work, we demonstrate this capability in the specific context of human
activity recognition with two modalities (visual appearance
and body key points) and two types of interactions (human-human
and human-context interactions). Mathematically,  implements
each individual query machine g^m(.) for the m-th
modality in Eq. (<ref>) using a compositional function:

    g^m(q_i,t^m,V)=(Φ_h(q_i,t^m,X^H),X^C).

Here, X^H and X^C represent human-centric and contextual
features, respectively, derived from the embedding of the video input
V. We define Φ_h(.) and Φ_c(.)
as reusable computational units called HUman-centric query
Blocks (s). These  units play a
crucial role in modeling human-human interactions (HH-HUB)
and human-context interactions (HC-HUB). Since the operation of the
HUB is generic and does not depend on a specific modality, we omit
m for the brevity. We elaborate the design of  in
the following.

Central to the 's operation is the widely used scaled
dot-product attention layer <cit.>:


    Attn(q,K,V)=∑_μ=1^Msoftmax_μ(K_μW_k(qW_q)/√(d)^⊤)V_μW_v,

where query q∈ℝ^1× D, keys K∈ℝ^M× D,
values V∈ℝ^M× D. The output of Attn(q,K,V)
is a vector in ℝ^1× D and W_q∈ℝ^D× D,
W_k∈ℝ^D× D, W_v∈ℝ^D× D
are network parameters. Fig. <ref> (on the right) demonstrates
the operation of .  is comprised of stacked
attention layers that accounts for the similarity between the dynamics
of a human actor and its surroundings in space and time through three
information channels (past, current, future). It searches for relevant
information of the query q in memories X_past, X_current,
X_future storing past, current and future knowledge in
the form of key-value pairs. While the query is a specific actor representation
at time t, the information in the memories can consist either human-centric
or general contextual information of a video clip at different points
in time. The output of  is a refined representation of
the actor-specific feature in response to the given query. Denoting
q̃_i,t as the output representation of actor i, defined
as:



    q̃_i,t    =HUB(q_i,t,{ X_past,X_current,X_future}),

where X_past={ K_t-w:t-1,V_t-w:t-1},
X_current={ K_t,V_t}, X_future={ K_t+1:t+w,V_t+1:t+w}
are key-value stores that encapsulates information in past, current
and future times. The window size w indicates clips that are w
steps apart from the present clip t. To enable effective retrieval
of past/future information while reducing the computational costs,
we employ a pre-computed clip selection mechanism that allows
us to skip irrelevant clips. In particular, we assess the relevance
of all clips within the window w to the present clip at time t
using their feature similarity. We then select the top-k most relevant
clips and store them as as key-value memories in the past (X_past)
and future times (X_future).  computes each
pair (q,X) using a multi-layer attention in Eq. (<ref>),
followed by a linear aggregation layer which returns a single vector
q̃_i,t for each human actor i.

Human-human interactions with HH-HUB: This stage considers
an actor in the relation with other actors involved in the same visual
scene in space and time. The HH-HUB Φ_h(.) takes
as input a query q_i,t, either visual appearance or human body
key points, representing an individual actor i at video clip t
and three key-value stores X_past^h, X_current^h,
X_future^h denoting visual appearance features
of all other human actors detected in past, present and future video
clips, respectively (See Sec. <ref>). While
the pair (q_i,t,X_current^h) at
current clip t captures the spatial relationships between actors
in the current scene, the across-time pairs (q_i,t,X_past^h)
and (q_i,t,X_future^h) provide information
about how the relationships evolve over time. The output of the HH-HUB
is a refined representation q̃_i,t∈ℝ^1× D
for the actor i at clip t: 

    q̃_i,t=[(q_i,t,X_past^h),Attn(q_i,t,X_current^h),Attn(q_i,t,X_future^h)]W_a,

where [· , ·] indicates feature concatenation, and W_a∈ℝ^3D× D
is learnable parameters.

Human-context interactions with HC-HUB: Unlike the HH-HUB
module, the HC-HUB Φ_c(.) focuses on modeling human-context
relationships. It takes the output q̃_i,t of the HH-HUB
as an input query and video spatio-temporal representations X_past^c,
X_current^c and X_future^c
of past, present and future video clips (See Sec. <ref>)
as key-value stores. The computation of the output q̂_i,t∈ℝ^1× D
of the HC-HUB is similar to the HH-HUB as in Eq. <ref>.
It now incorporates both human-human and human-context interactions
over space and time.



 §.§ Cross-modality Consistency with Contrastive Loss




In multi-modal human activity recognition, models should leverage
complementary aspects across modalities for label prediction. However,
inherent representation disparities between input modalities make
finding a joint representation capturing the saliency across all modalities
challenging. Instead of directly fusing high-level features of these
modalities together, we introduce a consistency loss to encourage
the model to exploit mutual information across input modalities of
the same person, as they both lead to the same activity prediction.
We achieve this by maximizing the mutual information between any pairs
of input modalities. Specifically, we sample a positive pair by taking
the final representations q̂_i,t^vis,q̂_i,t^key
by , which belongs to the same person while treating
k augmented samples randomly paired from different individuals
within a mini-batch as negative samples. Our cross-modality consistency
loss ℒ_𝒞𝒞 is implemented similar to the contrastive
loss in <cit.>:

    ,

where, sim(·,·) is the cosine similarity
function between two input vectors. 𝕀(.) is an
indicator function iff k≠ i within mini batch B. Importantly,
our consistency loss allows us to train the proposed model in an unsupervised
manner without the need for additional training data. We train our
models with this consistency loss together with the usual cross-entropy
loss for label prediction. We detail the training of our two tasks
as below.

Spatio-temporal action localization: We use a classifier
of an MLP followed by a logistic function to predict action labels
by an actor at time step. Our network is trained end-to-end by jointly
minimizing the binary cross entropy loss and the consistency loss:
ℒ=ℒ_ℬ𝒞ℰ+ℒ_CC. 

Group activity recognition: As all actors share the same
action label throughout a video input, we first apply the arithmetic
mean function across actors and along the temporal axis on the actor-specific
representations q̂^vis and q̂^key
to obtain a single output vector. We then use an MLP layer to map
the video feature to label space before applying the soft-max function
to return action label probabilities.We jointly minimize the cross
entropy loss and the consistency loss to train the network.




§ EXPERIMENTS


We evaluate the effectiveness of the proposed framework on two major
applications of human behavior understanding in videos: Spatio-Temporal
Action Localization on AVA v2.2 <cit.> and Group Activity
Recognition on Collective Activity dataset <cit.>.



 §.§ Spatio-temporal Action Localization 




Quantitative results: We first demonstrate the efficacy of
 when using video features by different video representation
backbones. The results are displayed in Fig. <ref>.
In general,  consistently improves all the baselines
where the gaps are more significant on weaker baselines. 



We also compare  against the most recent SoTA methods
on AVA (See Tab. <ref>). We categorize the prior
works based on the respective datasets that their video feature extractors
are pre-trained on, following <cit.>.
As seen,  consistently outperforms all the recent approaches
across all categories. While these works either only focus on human-human
interactions such as <cit.> or human-object/human-context
interactions as in <cit.>, 
enjoys the benefits of these two types of interactions within a single
model. While  clearly outperform approaches using single
modalities such as MViTv2 <cit.>, ORViT <cit.>
and MemViT <cit.>, we wish to emphasize its superior
performance when comparing with the most recent approach HIT <cit.>
that leverages identical input modalities. This clearly demonstrates
the effectiveness of our proposed method in both architecture modeling
with  units and learning with the cross-modality consistency
loss.

Qualitative results: We showcase examples taken from the
AVA dataset in Fig. <ref>. Combining both modalities
significantly enhances prediction performance compared to using a
single modality. Actions that requires efficient temporal modeling
such as running and driving are among the ones that benefit the most
from leveraging human body key points. 



Ablation studies: We conduct a comprehensive analysis on
's computational costs (Tab. <ref>)
and the contributions of each input modality (Tab. <ref>).
We also provide additional analysis on the effects of ablating different
designated components from the full design in the Supp. All ablation
studies use the MViTv2-S backbone.

Computational complexity: To demonstrate the benefits
of , we compare it with stronger baselines of similar
representation capacity (a.k.a model size) in Tab. <ref>.
These baselines are implemented by fine-tuning extending MViTv2 baselines
with additional self-attention layers on AVA v2. Results show that
simply increasing model size offer minimal improvement (See Row 1
vs. 1.a/1.b, and Row 2 vs. 2.a/2.b). In contrast, COMPUTER significantly
enhances baseline performance with minimal additional costs. Specifically,
 improves MViTv2-S by 3.0 points (10.9%) , with only
6.0% increase in GFLOPs and around 3.0% additional inference time.
We obverse consistent behaviors on MViTv2-B. Importantly, 
with MViTv2-S baseline even outperforms MViTv2-B despite faster inference
time and only 1/3 of the GFLOPs, thanks to the sparsity of our human
input tokens (Row 1.c vs Row 2).

Effectiveness of each modality: We analyze the impact
of each modality on the performance in Tab. <ref>.
RGB sequences slightly outperform body key points thanks to its rich
information.  successfully leverages the advantages of
each modality to improve the performance when using them in combination
(Row 3). Additionally, the proposed consistency loss considerably
improves the performance by nearly 1.5 points (5.1%). 

 



Effectiveness of each component in :

To provide more insights of our architecture , we ablate
its components and observe its effect to the overall performance.
In general, ablating any designed components of  would
result in degradation in performance (See Table <ref>). 

  Effectiveness of the hierarchy design: In this experiment,
the target human query attends to both the human and context elements
simultaneously, without imposing a hierarchical order. The significant
performance drop by 1.5 points (5.1%) highlights the importance of
our hierarchical design.

  Effectiveness of the HC-HUB block: This experiment
removes all HC-HUB blocks out of the original design of .
This leads to a considerable decrease in performance by nearly 2.0
points (5.9%).

  Effectiveness of the HH-HUB block: Similarly, this
experiment removes all the HH-HUB blocks. With the absence of the
human-human interactions, we observe a similar level of performance
degradation.

  Effectiveness of temporal modeling: This experiment
limits all  blocks to consider only the present information
channel while ignoring the other channels in past and future times.
Without considering the temporal dynamics of information, the performance
drops by nearly 1.5 points (4.8%).

  Effectiveness of pre-computed clip selection: This
experiment justifies the benefit of our pre-computed clip selection.
Instead of selectively choosing top-k top relevant past/future
clips with the clip at present time, we take into account all video
clips within the window size w and merely take average over the
post attention layer outputs. This suffers from 1.0 points performance
decrease that highlights the necessity of the information selection
strategy for the sake of both performance and computational burden.



 §.§ Group Activity Recognition


The Collective Activity dataset <cit.> includes 44
clips of five types of group activities including crossing, queuing,
walking, waiting and talking. For fair comparisons with prior works
<cit.>, we use InceptionNet
<cit.> pre-trained on ImageNet <cit.> for feature
extraction.

Quantitative results: The results of our proposed 
model for action group recognition, shown in Tab. <ref>,
demonstrate the effectiveness of our approach. Our method clearly
outperforms existing works by successfully incorporating multiple
modalities, leading to a more comprehensive representation. 




§ CONCLUSION


We introduced a unified framework named  for multi-modal
human activity recognition. The framework features a generic architecture
effectively retrieving information about human movements and the relationships
between human actors and their surroundings from different input modalities.
We also introduced a novel consistency loss to leverage the complementary
information across modalities for robust prediction of human activity
in an unsupervised manner. Through extensive experiments on two applications,
our framework demonstrated high efficiency approach compared to existing
methods.



 
