# Abductive Action Inference

Clement Tan1, Chai Kiat Yeo1, Cheston Tan2, Basura Fernando2
1Nanyang Technological University, Singapore
2Agency for Science, Technology and Research (A*STAR), Singapore
{s190099<EMAIL_ADDRESS>{cheston$-$tan@i2r,
<EMAIL_ADDRESS>
###### Abstract

Abductive reasoning aims to make the most likely inference for a given set of
incomplete observations. In this work, given a situation or a scenario, we aim
to answer the question ’what is the set of actions that were executed by the
human in order to come to this current state?’, which we coin as abductive
action inference. We provide a solution based on the human-object relations
and their states in the given scene. Specifically, we first detect objects and
humans in the scene, and then generate representations for each human-centric
relation. Using these human-centric relations, we derive the most likely set
of actions that the human may have executed to arrive in this state. To
generate human-centric relational representations, we investigate several
models such as Transformers, a novel graph neural network-based encoder-
decoder and a new relational bilinear pooling method. We obtain promising
results using these new models on this challenging task on the Action Genome
dataset.

##  1 Introduction

Reasoning is an inherent part of human life, and in recent years we have seen
that computer vision models are also developed in a way that they can perform
reasoning at some level [18]. Abductive reasoning aims to infer the most
compelling explanation for a given set of observed facts. Abductive reasoning
is an extremely useful tool in our daily life, as we often rely on a set of
facts to form the most probable conclusion. The ability to perform such
abductive reasoning about humans’ actions is vital for human-robot
collaborations. Imagine the scenario where a rescue robot enters an elderly
person’s house to check on why he or she is not responding to the routine
automated phone call. Upon entering the house, the robot observes its
surroundings and notices that the back door is left open but nothing else is
out of the ordinary. These observations may form a basis for a rational agent
- the elderly might have opened the door and went into the garden. The robot
can immediately make its way to search for him/her in the back garden. In this
scenario, the rescue social robot may be able to verify if the person has been
safe based on the factual observations in the scene and make conclusions based
on logical abduction.

Figure 1: In this paper, we ask the following question. Given a snapshot, what
are the likely actions executed by the person before? As humans, we can infer
that the person may have removed the shoes, picked up a bottle. He may have
opened the door and picked the box from the cabinet above. Even if the cabinet
is partly visible, we can make such conclusions. This type of inference
requires abductive reasoning and hence we call this task abductive action
inference.

In recent years, there have been some great initiatives made in abductive
reasoning for computer vision. In particular, [20] generates the description
of the hypothesis (abductive) and the premises in natural language given a
video snapshot of events. Without the generation of hypothesis description,
these methods boil down to dense video captioning. A similar task is also
presented in [14] where given an image, the model must perform logical
abduction to explain the observation in natural language.

In contrast to these recent works, we challenge a model to predict multiple
human actions that may have occurred in the past, given a single snapshot
(image) –see Figure 1. Based on the visual information from the snapshot,
objects such as the person, bottle, box, and shoes, may provide clues from
which we humans can draw conclusions. We term this task, _Abductive Action
Inference_ and further investigate how deep learning models perform on the
task. The task is challenging as it relies on the model’s ability to perform
abductive reasoning based on factual evidence i.e., determining what actions
may have or have not been performed based on visual information in the scene
of the snapshot.

Visual Genome [19] introduced the idea of decomposing a scene into its
corresponding objects and their visual relationships. In relation to actions,
the Action Genome [16] dataset extends the idea by decomposing actions across
a video into spatio-temporal scene graph representations. Spatial-temporal
scene-graphs have been used for action recognition before [22]. Human action
can be viewed as an evolution of human-object relationships over time.
Therefore, the state of human-object relations in a scene may give away some
cues about the actions that may have been executed by the human in that scene.
We propose to perform logical abduction on past actions using visual and
semantic information of human-object relations in the scene. On the other
hand, there is also the duality in which the evidence should support those
conclusions (the actions inferred by the model). If the human executed a set
of actions $\mathcal{A}$ which resulted in a state whereby a human-object
relation set $\mathcal{R}$ is formed (i.e.,
$\mathcal{A}\rightarrow\mathcal{R}$), then using the relational information,
we formulate the task by aiming to infer $\mathcal{A}$ from $\mathcal{R}$.

In this work, our models rely on the human-centric object relation tuples such
as (person, bottle), (person, shoes), (person, box), and (person, doorway)
obtainable from a single image at the current point in time to perform
abductive action inference. One can see why these human-centric relations are
vital for identifying past actions: the (person, bottle) objects may lead to
deriving actions such as (person-drank-from-bottle, person-took-bottle-from-
somewhere) while (person, shoes) may imply actions such as (person-holding-
shoes, person-took-shoes-off, person-threw-shoes-down). We first detect the
objects in the scene and construct representations for these human-centric
object relations within each image. These relations are made up of both visual
and semantic features (such as glove embedding) of recognized objects. To
capture this relational information from the scene, we propose novel models
such as a Graph neural network-based encoder-decoder model and a relational
bilinear pooling model. The graph affinity of our model reasons about human-
centric relations in the scene effectively. Alternatively, our relational
bilinear model aims to model pairwise relational representations using both
linear and bilinear modeling and nonlinear mappings. Our contributions can be
summarized as follows:

  1. 1.

To the best of our knowledge, we are the first to propose the important task
of abductive action inference from a single snapshot or image.

  2. 2.

We validate some effective models such as a Transformer against some newly
proposed models such as Jaccard Vector Similarity-based Graph Neural Network
encoder-decoder model.

  3. 3.

We also investigate the impact of the relational bilinear model and combine
that with graph neural networks encoder-decoder.

##  2 Related work

Figure 2: This figure illustrates the high-level idea of our proposed task and
the method. Given a scene, we aim to infer the most likely actions executed by
the human to come to that state. This is called abductive action inference.
Given the scene, first we detect objects in the scene and then form a human-
centric relation representation using models such as Transformers, Graph
neural networks and bi-linear relational modeling. Then we perform inference
to identify the actions executed by the human to come to this state.

It should be noted that our work is different from action recognition [15, 13]
in a fundamental way. First, in action recognition, the objective is to
identify the actions executed in the visible data (e.g., a video or an image
in still image action recognition [10]). In action recognition, the models can
learn from visual cues what the action looks like and what constitutes an
action. In our work, we aim to infer actions that the model has never seen the
human performing. During training, our model only sees human-object relations
in the scene which is the outcome of executed actions. There are no motion
cues or visual patterns of actions that the model can rely on to abduct
actions. From a single static image, the machine should infer what actions may
have been executed. This is drastically different from the classical action
recognition and action anticipation tasks.

Abductive action inference has some similarity to short-term action
anticipation [7, 5] and long-term action anticipation [1]. However, there are
several notable differences between the two tasks. Firstly, in abductive
action inference, the goal of the model is to identify the most plausible
actions executed by a human based on the current evidence, whereas in action
anticipation, the model learns to predict future actions from current
observations. Next, unlike action anticipation where a short video clip is
typically observed (e.g. a 2-second-long video) before predicting future
actions, in abductive action inference, we only process a single snapshot
(frame) to perform abductive reasoning. In addition, predictive methods [11]
are mostly used for action anticipation [9], while in abductive action
inference we are unable to use such predictive methods during training as we
only have access to a single snapshot. While action anticipation has to deal
with the uncertainty of human behavior in the future, in abductive action
inference, the evidence should suggest what actions may have been executed. In
abductive action inference, the uncertainty arises from the fact that several
different actions may have resulted in similar states $\mathcal{R}$.

Another related line of research is Visual Commonsense Reasoning (VCR) [35,
32]. In VCR [35], given an image, object regions and a question, the task
requires the machine to answer the question regarding what is happening in the
given frame. The model has to also provide justifications for the selected
answer in relation to the question. [23] also studied a somewhat similar
problem where a dynamic story underlying the input image is generated using
commonsense reasoning. In particular, VisualCOMET [23] extends VCR and
attempts to generate a set of textual descriptions of events at present, a set
of commonsense inferences on events before, a set of commonsense inferences on
events after, and a set of commonsense inferences on people’s intents at
present. In this vein, given the complete visual commonsense graph
representing an image, they propose two tasks; (1) generate the rest of the
visual commonsense graph that is connected to the current event and (2)
generate a complete set of commonsense inferences. In contrast, given a
snapshot without any other natural language queries, we recognize visual
objects in the scene and how they are related to the human and then use the
human-centred relational representation to infer the most likely actions
executed by the human.

Recently, there are machine learning models that can also perform logical
reasoning. In particular, [4] specifically use abductive reasoning to improve
the hypothesis of the learner. Furthermore, a neurosymbolic action recognition
method that can generate some explanation for predicted actions is presented
in [17]. Perhaps, our work is orthogonal to these ideas where our method can
also benefit from these developed ideas.

Visual scene graph generation [30] and spatial temporal scene graph generation
[3] are also related our work. To be specific, any advancement made in scene
graph generation, in particular, modeling relations between humans and objects
should also help to improve the abductive action inference task as this task
heavily relies on the human-object relations to make an abductive inference.
Furthermore, situation understanding [33] is also related to our work. Graph
neural networks are also related to our work [26, 34, 27], however, our model
rely on Jaccard Vector Similarity to construct the affinity of the nodes and
use encoder-decoder model. Furthermore, our work is also related to bilinear
pooling methods such as [8, 6]. However, we use bilinear pooling for
relational modeling and abductive action inference.

##  3 Abductive action inference

Given a single snapshot of a situation, we aim to infer the actions executed
by the human up to the current point in time using abductive reasoning which
we coined Abductive Action Inference. To do this, we first detect humans and
objects in the snapshot and then generate representation for human-centered
object relations. Then using these human-centered object relations, we
summarize the visual snapshot and using neural models, we infer the most
likely human actions executed by the human. The overview of our model and task
is shown in Figure 2.

Let us denote the object by $o\in O$, predicate category by $p\in P$ and human
by $h$. Then the relation $R$ is a triplet of the form $\left<h,p,o\right>$.
In each snapshot we may have $n$ number of relations. Let us denote a human
action by $a_{i}\in A$ where $A$ is the set of all actions. Then in abductive
action inference, we aim to learn the following logical association

$a_{1}\vee a_{2}\vee a_{3},\cdots,\vee a_{K}\rightarrow R_{1}\land
R_{2}\land\cdots\land R_{N}$ (1)

where $R_{1}\land R_{2}\land\cdots\land R_{N}$ are the relation set
$\mathcal{R}$ present in a situation and $a_{1}\vee a_{2}\vee
a_{3},\cdots,\vee a_{K}$ are the action set $\mathcal{A}$ executed by the
human to arrive at that situation. Therefore, one might re-write this
association as follows:

$\mathcal{A}\rightarrow\mathcal{R}$ (2)

where $\mathcal{R}\subset\mathbb{R}$ and $\mathbb{R}$ is the set of all
possible relations (triplets). Therefore, given $\mathcal{R}$ we aim to infer
the most likely set of actions executed by the human, hence we term it
abductive action inference.

We learn this logical association using the deep neural network function
$\phi()$ of the following form by maximizing the conditional probability of

$P(a_{1},a_{2},\cdots,a_{K}|r_{1},r_{2},\cdots,r_{n})=\phi(r_{1},r_{2},\cdots,r_{n};\theta_{\phi})$
(3)

where $r_{k}$ is the representation of the relation triplet $R_{k}$ from the
training set. The training set consists of snapshots (images) and
corresponding action set $\mathcal{A}$. From each training snapshot, we
extract the relation set $\mathcal{R}_{i}$ and the action set
$\mathcal{A}_{i}$ that resulted in $\mathcal{R}_{i}$. Therefore, the training
set consists of

$\mathcal{D}=\bigcup_{i}\\{\mathcal{R}_{i},\mathcal{A}_{i}\\}$ (4)

Given the training set ($\mathcal{D}$), we learn the model function in
Equation 3 using backpropagation. During training we use the max-margin multi-
label loss. Next, we discuss how to obtain the human-object relational
representation.

###  3.1 Relational representation

To obtain the relation representation, we first extract features from the
human region and object regions of each snapshot (image) using FasterRCNN [25]
model with ResNet101 backbone [12]. Let us denote the human feature by
$x_{h}$, the object feature by $x_{o}$, and the features extracted from taking
the union region of both human and object features by $x_{u}$. As we do not
know the predicate or the relationship label for the relation between $x_{h}$
and $x_{o}$, we use concatenation of all three visual features $x_{h}$,
$x_{o}$ and $x_{u}$ as the joint relational visual feature
$x_{v}=[x_{h},x_{o},x_{u}]$. Using FasterRCNN, we can also obtain the object
and human categories. We use Glove [24] embedding to get a semantic
representation of each human and object. Let us denote the Glove embedding of
the human by $y_{h}$ and object by $y_{o}$. Then, the semantic representation
of the relation is given by $y_{s}=[y_{h},y_{o}]$.

Using both visual and semantic representations, we obtain a joint
representation for each human-centric relation in a given snapshot. Therefore,
the default relation representation for a relation $R=<h,p,o>$ is given by

$r=[x_{v},y_{s}].$ (5)

Note that we do not have access to the predicate class or any information
about the predicate. Next, we present several neural models that we compare in
the experiments for abductive action inference.

###  3.2 Transformer model

Transformers [31] are a popular class of models in deep learning. Transformers
are very effective at capturing relationships between far apart elements in a
set or a sequence. In particular, in this work, we use Transformers as a set
summarization model. We use a multi-head self-attention Transformer model with
one encoder layer and three decoders as the default setup. We do not use any
positional encoding as we are summarising a set. Given the set of relational
representation of an image $\mathcal{R}_{i}=r_{1},r_{2},\cdots,r_{n}$, the
Transformer model outputs a Tensor of size $n\times d$ where $d$ is the size
of the relational representation. Afterwards, we use max-pooling to obtain a
snapshot representation vector and use a linear classifier to estimate the
probability distribution (as in equation 3) given the relational
representation $\mathcal{R}_{i}$. Next, we describe our new graph neural
network-based encoder-decoder model that summarizes the relational information
for abductive action inference.

Figure 3: (Left) Graph neural network encoder model architecture, (right)
Graph neural network decoder model architecture. The residual connections are
shown with the $+$ sign. Graph module consist of equations 6-8. Figure 4: The
bilinear graph neural network encoder-decoder architecture.

###  3.3 GNNED: Graph Neural Network Encoder-Decoder

Given the relational data with slight notation abuse let us denote the
relational representations by a $n\times d$ matrix
$\mathcal{R}=[r_{1},r_{2},\cdots,r_{n}]$ where $r_{n}$ has $d$ dimensions. In
our graph neural network encoder-decoder (GNNED) model, we first project the
relational data using a linear function as follows:

$\mathcal{R^{\prime}}=\mathcal{R}W_{l}+b_{l}$ (6)

where
$\mathcal{R^{\prime}}=[r^{\prime}_{1},r^{\prime}_{2},\cdots,r^{\prime}_{n}]$.
Then we construct the affinity matrix $W_{A}$ as follows:

$W_{A}(i,j)=\frac{2*r^{\prime}_{i}\cdot r^{\prime}_{j}}{r^{\prime}_{i}\cdot
r^{\prime}_{i}+r^{\prime}_{j}\cdot r^{\prime}_{j}}$ (7)

where $W_{A}(i,j)$ shows the similarity/affinity between the i-th-relation and
the j-th-relation in the set. Here, we use Jaccard Vector Similarity which is
a smooth and fully differentiable affinity [5]. Note that Jaccard Vector
Similarity is bounded [-1,1]. Here, $a\cdot b$ is the dot product. Thereafter,
we obtain the graph encoded feature representation as follows:

$G_{e}=ReLU((W_{A}\mathcal{R^{\prime}})W_{g}+b_{g})$ (8)

where $W_{g}$ and $b_{g}$ are the weight matrix and bias term respectively. We
call equations 6-8 the graph module. Using the graph module as a base model,
we develop the graph encoding layer as shown in Figure 3 (left). The graph
encoding layer is very similar to the Transformer encoder layer [31]. The
graph encoding layer consists of drop-out layers, layer norm [2], linear
layers, and residual connections [12]. The graph decoder architecture is shown
in Figure 3 (right) where it is also similar to a Transformer decoder layer
except for the graph module. Our graph encoder-decoder model then consists of
one graph encoder, and three graph decoders by default. The graph module is
able to model the inter-relations between human-object relations and reasons
about them to perform abductive inference on actions. Finally, we apply max
pooling at the end of the graph-encoder-decoder model to obtain the final
snapshot representation and then use a classifier to obtain the actions as
shown in Equation 3. Next, we present our bilinear pooling method for
relational modeling.

###  3.4 RBP: Relational modeling with bilinear Pooling.

Given the human representation $x_{h}$ and the object representation $x_{o}$,
we use bilinear pooling with a weight matrix $W_{b}$ of size $d\times d\times
d$ and linear projection matrices $W_{bl},W_{jb}$ as follows:

$\displaystyle o^{\prime}$ $\displaystyle=ReLU(W_{o}x_{o}+b_{o})$ (9)
$\displaystyle h^{\prime}$ $\displaystyle=ReLU(W_{h}x_{h}+b_{h})$ (10)
$\displaystyle r_{b}$
$\displaystyle=ReLU(~{}[h^{\prime}W_{b}o^{\prime};([h^{\prime};o^{\prime}]W_{bl}+b_{bl})]~{})W_{jb}+b_{jb}$
(11)

where [;] represents the vector concatenation and $h^{\prime}W_{b}o^{\prime}$
is the bilinear pooling operator applied over human and object features. The
$([o^{\prime};h^{\prime}]W_{bl}+b_{bl})$ is the output of concatenation of
human and object features and applying linearly projection using weight matrix
$W_{bl}$ and bias term $b_{bl}$. Then, we concatenate the bilinear pooled
vector ($h^{\prime}W_{b}o^{\prime}$) and the output of linear projection
($([o^{\prime};h^{\prime}]W_{bl}+b_{bl})$). Then we use ReLU and apply another
projection by using another linear model ($W_{jb}+b_{jb}$). Finally, we
concatenate the overlap feature $x_{u}$ with the overall model output
($r_{b}$) and apply max-pooling across all relational features
($[r_{b};x_{u}]$) in the image and use a linear classifier to obtain the
action distribution.

###  3.5 BiGED: Bilinear Graph Encoder-Decoder

Finally, to take advantage of both bilinear relational modeling and graph
neural network encoder-decoder models, we combine both strategies. The main
idea is to replace the projection function in Equations 9 with a graph neural
network encoder-decoder model. Let us denote the graph neural network encoder-
decoder model by $f_{Ged}()$. Then, equations 9 will be replaced as follows:

$\displaystyle O^{\prime}$ $\displaystyle=f_{Ged}(X_{o})$ (12)

where $X_{o}$ is all the object features in the snapshot. Afterwards, we apply
equation 11 as before using the bilinear modeling to obtain the relational
representation. Note that as there is only one or two humans in the snapshot,
we do not use GNNED to model human features. Similarly, we concatenate the
original human and object features $[x_{h},x_{o}]$ to obtain a joint feature
and then pass through another GNNED model. The outputs of both bilinear and
the joint feature-based GNNED models are concatenated to obtain the final
relational representation. A visual illustration of this model is shown in
Figure 4. As shown in the figure, there are two separate GNNED models, one to
capture object features and another one to capture joint concatenated human-
object features. Note that we concatenate the overlap feature $x_{u}$ with
final model output as before to form the relational features. Afterwards, we
use max-pooling to obtain the representation for the snapshot and then finally
use the classifier to infer the actions.

##  4 Experiments and Results

###  4.1 Dataset

We train our proposed models on the Action Genome (AG) dataset [16] for
abductive action inference tasks. Built upon the Charades dataset [28], the AG
dataset contains 9848 videos with 476,000+ object bounding boxes and 1.72
million visual relationships annotated across 234,000+ frames. The original
objective of this dataset is to study spatial-temporal scene graph generation.
_It should be noted that not all video frames in the Charades dataset are used
in the AG dataset. Only a handful of key frames are used in AG, and we follow
the same._ The AG dataset does not provide action annotations. To obtain
action annotations for snapshots of AG, we leveraged on the Charades dataset
which contains action annotations of 157 action classes. To be specific, we
use frame-level action annotations from Charades to get action annotations for
AG snapshots. Then we construct the training and testing dataset for abductive
action recognition as explained in Equation 4.

###  4.2 Experimental Setup

After obtaining action annotations for all snapshots for a given video in AG,
we drop videos having only one snapshot as they do not contain enough past
action labels. For the remaining snapshots, we assign action labels from the
previous snapshots in two ways (two setups). In the first setup, action labels
from all the previous snapshots (frames) are added to the ground truth denoted
by $\mathcal{A}=\\{\mathcal{A}_{0},\cdots\mathcal{A}_{t}\\}$ (including
actions from the current snapshot) where $\mathcal{A}_{t}$ denotes all actions
of the $t^{th}$ snapshot. In the second setup, the actions are added only from
the current and previous snapshots to the ground truth denoted by
$\mathcal{A}=\\{\mathcal{A}_{t-1},\mathcal{A}_{t}\\}$. Note that in the Action
Genome dataset, the snapshots are sampled non-homogeneously from the Charades
dataset videos. Therefore, the previous snapshot occurs several seconds before
the current snapshot. It should be noted that the abductive action inference
task in our work is always constrained to a limited time horizon even in the
first setup as it is limited by the episode of the video.

Model | Mean Average Precision
---|---
Rule-based inference | 26.27
MLP | 27.73
Transformer | 33.59
GNNED | 34.38
RBP | 35.55
BiGED | 35.96
Table 1: Abductive action inference performance using the proposed methods. Results are reported for abducting all previous actions $\mathcal{A}=\\{\mathcal{A}_{0},\cdots\mathcal{A}_{t}\\}$ using all snapshots of the test set of Action Genome dataset [16]. Model | Mean Average Precision
---|---
Rule-based inference | 26.18
MLP | 25.99
Transformer | 30.13
GNNED | 30.95
RBP | 31.48
BiGED | 31.55
Table 2: Abductive action inference performance using the proposed methods.
Results are reported for abducting all previous actions
$\mathcal{A}=\\{\mathcal{A}_{0},\cdots\mathcal{A}_{t}\\}$ using the last
snapshot of each video of the test set of Action Genome dataset [16].

###  4.3 Implementation details

For fair comparison, we employ FasterRCNN [25] with a ResNet101 [12] backbone
to extract human and object features from each snapshot based on the ground
truth person and object bounding boxes provided by AG for all models. We load
pre-trained weights provided by [3] that were trained on the training set of
AG which obtained 24.6 mAP at 0.5 IoU with COCO metrics. The parameters of the
FasterRCNN during training and inference are fixed for the abductive action
inference task. Our default human and object visual representations have 512
dimensions obtained from 2048 dimensional visual features from FasterRCNN. We
use linear mappings to do this. During training, we train the models for 10
epochs and set the batch size to 1 video (there are many frames in a video).
We use the AdamW [21] optimizer with an initial learning rate of 1e-5 along
with a scheduler to decrease the learning rate by a factor of 0.5 to a minimum
of 1e-7. We utilize Glove [24] word embedding of size 200 for object and
subject (human) semantic features. In addition, gradient clipping with a
maximal norm of 5 is applied. Moreover, to ensure we report the most accurate
performance of our models, we ran and reported the mean across 3 different
runs for each configuration. All models are trained on a single RTX3090 or
single A5000 GPU. We also use two additional baseline models, 1) Multi-Layered
Perceptron (MLP) and 2) rule-based abductive inference. Details of these
methods are provided in the supplementary material.

Model | Mean Average Precision
---|---
Rule-based inference | 26.50
MLP | 30.46
Transformer | 35.77
GNNED | 36.00
RBP | 36.07
BiGED | 36.72
Table 3: Abductive action inference performance using the proposed methods.
Results are reported for abducting actions of the previous and current
snapshot $\mathcal{A}=\\{\mathcal{A}_{t-1},\mathcal{A}_{t}\\}$ using all
frames of the test set of Action Genome dataset [16].

###  4.4 Quantitative Results

Our main results for the abductive action inference task is shown in Table 1.
These results are obtained based on the first configuration where the goal of
the model is to infer the set of actions
$\mathcal{A}=\\{\mathcal{A}_{0},\cdots\mathcal{A}_{t}\\}$ from all previous
and current snapshots. During training, the model learns from every single
snapshot in the video sequence. Likewise, during inference, the model predicts
the past action set on every single snapshot. As can be seen from the results
in Table 1, the MLP model and rule-based inference performs the worst. The
Transformer model improves the results to 33.59 while GNNED further improves
the results to 34.38 and surprisingly, the RBP obtains a particularly good
result of 35.55. The GNNED and RBP models follow two different approaches that
capture relational information and the way the relational reasoning is
performed are fundamentally different. Yet, these models obtain better results
than the Transformer model. Finally, we see that the combination of both GNNED
and RBP, i.e., BiGED performs even better.

Next, we evaluate the model’s performance only on the last snapshot of each
video sequence and inferred all the previous actions executed by the human in
each video. However, we use the trained model in the previous setup and
perform inference only on the last snapshot of the video. As a myriad of
actions could have happened throughout a video sequence, this setup is
particularly challenging. The result of this task is shown in Table 2. We
observe that the results are lower than in the previous setup for all models,
which demonstrates that the task is indeed more challenging. The MLP model and
the rule-based inference have a relatively poor performance. Still, the GNNED,
RBP and BiGED methods have better performance than the Transformer model
despite GNNED has similar architecture to Transformer.

In the second setup, we retrain all models with the corresponding past action
set $\mathcal{A}=\\{\mathcal{A}_{t-1},\mathcal{A}_{t}\\}$. We report the
results in Table 3. Similar trends to the previous two tables can be seen here
as well. The GNNED model also performs comparatively well. Both GNNED and RBP
outperforms the Transformer model and BiGED is the best performer. The rule-
based approach is unable to obtain satisfactory results in all setups. We
conclude that the relational bilinear modeling obtained by the RBP and GNNED
models are effective for abductive action inference.

###  4.5 Qualitative Results

We compare qualitative results in Figure 5. Depending on the number of ground
truth past actions a snapshot has, we take the same number of top-k predicted
actions from each model. All the models demonstrate its ability to perform
abductive action inference. In the first snapshot, there are objects such as a
person, laptop, table, cup and dish. In the second snapshot, there are objects
such as a person, floor, blanket, bag, and vacuum. In both scenarios, out of
all the models, RBP and BiGED demonstrates that it can infer actions more
accurately.

Model | Mean Average Precision
---|---
MLP (visual only) | 23.00
MLP (visual + scheduler) | 22.87
MLP (visual + semantic) | 30.44
MLP (visual + semantic + scheduler) | 30.46
Transformer (visual only) | 21.42
Transformer (visual + scheduler) | 21.93
Transformer (visual + semantic) | 35.40
Transformer (visual + semantic + scheduler) | 35.77

Table 4: Ablation study for impact of semantic features and scheduler. Model | Mean Average Precision
---|---
MLP+mean pooling | 28.49
MLP+max pooling | 30.46
Transformer+mean pooling | 35.31
Transformer+max pooling | 35.77
Table 5: Ablation study for max pooling vs mean pooling using the MLP and
Transformer model for abductive action inference.

###  4.6 Ablation Study

We perform several ablation studies using 2-layer MLP and Transformer models.
These findings can also generalized to other compared models. We use the
second setup where the model must abduct actions of the current and previous
snapshot of the Action Genome dataset for this ablation study.

Impact of semantic features and learning scheduler.
We perform several ablation studies using 2-layer MLP and Transformer models.
These findings also generalize to other compared models. We use the second
setup where the model must abduct actions of the current and previous snapshot
of the Action Genome dataset for this ablation study. We evaluate the effect
of visual and semantic (Glove) features in Table 4. The use of semantic
features provided a huge performance boost - the MLP model obtains a 7.44%
(23.00 vs 30.44) increase while Transformer obtains a 13.98% (21.42 vs 35.40)
increase. We attribute the performance increase to the contextual information
provided by the semantics. The semantics of objects enable the model to
identify and relate the actions effectively and provide an easier means to
reason about actions. It is also interesting to see the impact of the learning
rate scheduler which provides considerable improvement. Therefore, we use
semantics and the learning rate scheduler for all our models.

Max pooling vs Mean pooling.
We ablate the use of mean and max pooling on the relational features before
feeding them into the classification layer to perform abductive action
inference – see Table 5. In the experiments, max pooling obtains better
results (28.49 vs 30.46) for MLP and (35.31 vs 35.77) for the Transformer.
Therefore, we use max pooling.

Glove vs ConceptNet.
We compare the effect of Glove [24] and ConceptNet [29] embedding for this
task. Our assumption was that ConceptNet embedding may be able to provide
better reasoning for abductive inference. We use embedding of sizes 200 and
300 for Glove and ConceptNet respectively 111The default ConceptNet embedding
size is 300.. We show results in Table 6. In both scenarios, glove embedding
outperforms ConceptNet embedding. We, therefore, utilize Glove word embeddings
for all our experiments.

Model | Mean Average Precision
---|---
MLP+ConceptNet | 26.80
MLP+Glove | 30.46
Transformer+ConceptNet | 32.87
Transformer+Glove | 35.77
Table 6: Ablation study for Glove [24] vs ConceptNet [29] embeddings for
abductive action inference.

##  5 Discussion & Conclusion

Snapshot | MLP | Transformer | GNNED | RBP | BiGED
---|---|---|---|---|---
|  |  |  |  |
Watching a laptop or something on a laptop |  |  |  |  |
Working/Playing on a laptop |  |  |  |  |
Drinking from a cup/glass/bottle |  |  |  |  |
Holding a cup/glass/bottle of something |  |  |  |  |
Taking a cup/glass/bottle from somewhere |  |  |  |  |
Holding a dish |  |  |  |  |
Working at a table |  |  |  |  |
Watching a laptop or something on a laptop |  |  |  |  |
Working/Playing on a laptop |  |  |  |  |
Drinking from a cup/glass/bottle |  |  |  |  |
Holding a cup/glass/bottle of something |  |  |  |  |
Taking a cup/glass/bottle from somewhere |  |  |  |  |
Taking a cup/glass/bottle from somewhere |  |  |  |  |
Working at a table |  |  |  |  |
Watching a laptop or something on a laptop |  |  |  |  |
Working/Playing on a laptop |  |  |  |  |
Drinking from a cup/glass/bottle |  |  |  |  |
Holding a cup/glass/bottle of something |  |  |  |  |
Taking a cup/glass/bottle from somewhere |  |  |  |  |
Holding a cup/glass/bottle of something |  |  |  |  |
Working/Playing on a laptop |  |  |  |  |
Working at a table |  |  |  |  |
Watching a laptop or something on a laptop |  |  |  |  |
Drinking from a cup/glass/bottle |  |  |  |  |
|  |  |  |  |
Taking a cup/glass/bottle from somewhere |  |  |  |  |
Holding a cup/glass/bottle of something |  |  |  |  |
Working/Playing on a laptop |  |  |  |  |
Working at a table |  |  |  |  |
Watching a laptop or something on a laptop |  |  |  |  |
Drinking from a cup/glass/bottle |  |  |  |  |

&

Holding a bag Snuggling with a blanket Lying on the floor Sitting on the floor
Holding a vacuum Someone is awakening somewhere

Holding a bag Snuggling with a blanket Lying on the floor Sitting on the floor
Holding a vacuum Someone is awakening somewhere

Holding a bag Snuggling with a blanket Lying on the floor Sitting on the floor
Holding a vacuum Someone is awakening somewhere

Holding a bag Snuggling with a blanket Sitting on the floor Lying on the floor
Holding a vacuum Someone is awakening somewhere

Holding a bag Snuggling with a blanket Sitting on the floor Lying on the floor
Holding a vacuum Someone is awakening somewhere

Figure 5: Qualitative results produced each model on the Abductive Action
Inference on the last snapshot of each video in the AG test dataset. The first
column shows the snapshot along with the object bounding boxes in the scene.
The remaining columns show the actions predicted by each model. Actions
highlighted in green are correctly inferred while actions in red are wrongly
inferred.

In this paper, we present a new task called abductive action inference. We
evaluated several methods such as rule-based inference, simple multi-layer
perceptron, Transformers, a new graph neural network model, relational
bilinear model, and a bilinear graph neural network model on this challenging
task using the challenging Action Genome dataset. We obtained encouraging
results using both the newly proposed Graph neural network encoder-decoder,
the relational bilinear models and its combination.

One of the biggest challenges in abductive action inference is how to generate
ground truth actions for a given scenario. In the Action Genome dataset, we
assume the ground truth actions of the Charades dataset provides a reasonable
logical abduction. However, it is entirely possible the provided ground truth
may not be the most ideal logical abductions for some snapshots. However, our
manual investigation shows that on most of the snapshots, the ground truth
provides a reasonable and accurate logical abduction about the snapshot. More
research is needed in how to evaluate these abductive reasoning methods. One
limitation of our paper is that it is only evaluated on one dataset. However,
in the future, we are hopeful there might be more datasets for abductive
action inference. We believe that abductive action inference is not only a
challenging task for Artificial Intelligence (AI), but it is also a useful
tool that future robots may need. We are hopeful this important research
question will attract some attention in the future.

## References

  * [1] Yazan Abu Farha, Alexander Richard, and Juergen Gall.  When will you do what?-anticipating temporal occurrences of activities.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5343–5352, 2018.
  * [2] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.  Layer normalization.  arXiv preprint arXiv:1607.06450, 2016.
  * [3] Yuren Cong, Wentong Liao, Hanno Ackermann, Bodo Rosenhahn, and Michael Ying Yang.  Spatial-temporal transformer for dynamic scene graph generation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16372–16382, 2021.
  * [4] Wang-Zhou Dai, Qiuling Xu, Yang Yu, and Zhi-Hua Zhou.  Bridging machine learning and logical reasoning by abductive learning.  Advances in Neural Information Processing Systems, 32, 2019.
  * [5] Basura Fernando and Samitha Herath.  Anticipating human actions by correlating past with the future with jaccard similarity measures.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13224–13233, 2021.
  * [6] Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach.  Multimodal compact bilinear pooling for visual question answering and visual grounding.  arXiv preprint arXiv:1606.01847, 2016.
  * [7] Antonino Furnari and Giovanni Maria Farinella.  What would you expect? anticipating egocentric actions with rolling-unrolling lstms and modality attention.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6252–6261, 2019.
  * [8] Yang Gao, Oscar Beijbom, Ning Zhang, and Trevor Darrell.  Compact bilinear pooling.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
  * [9] Rohit Girdhar and Kristen Grauman.  Anticipative video transformer.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13505–13515, 2021.
  * [10] Guodong Guo and Alice Lai.  A survey on still image based human action recognition.  Pattern Recognition, 47(10):3343–3361, 2014.
  * [11] Tengda Han, Weidi Xie, and Andrew Zisserman.  Video representation learning by dense predictive coding.  In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 0–0, 2019.
  * [12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [13] Samitha Herath, Mehrtash Harandi, and Fatih Porikli.  Going deeper into action recognition: A survey.  Image and vision computing, 60:4–21, 2017.
  * [14] Jack Hessel, Jena D Hwang, Jae Sung Park, Rowan Zellers, Chandra Bhagavatula, Anna Rohrbach, Kate Saenko, and Yejin Choi.  The abduction of sherlock holmes: A dataset for visual abductive reasoning.  arXiv preprint arXiv:2202.04800, 2022.
  * [15] Hueihan Jhuang, Juergen Gall, Silvia Zuffi, Cordelia Schmid, and Michael J Black.  Towards understanding action recognition.  In Proceedings of the IEEE international conference on computer vision, pages 3192–3199, 2013.
  * [16] Jingwei Ji, Ranjay Krishna, Li Fei-Fei, and Juan Carlos Niebles.  Action genome: Actions as compositions of spatio-temporal scene graphs.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10236–10247, 2020.
  * [17] Yang Jin, Linchao Zhu, and Yadong Mu.  Complex video action reasoning via learnable markov logic network.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3242–3251, 2022.
  * [18] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick.  Clevr: A diagnostic dataset for compositional language and elementary visual reasoning.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2901–2910, 2017.
  * [19] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  International journal of computer vision, 123(1):32–73, 2017.
  * [20] Chen Liang, Wenguan Wang, Tianfei Zhou, and Yi Yang.  Visual abductive reasoning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15565–15575, 2022.
  * [21] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  arXiv preprint arXiv:1711.05101, 2017.
  * [22] Yangjun Ou, Li Mi, and Zhenzhong Chen.  Object-relation reasoning graph for action recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20133–20142, 2022.
  * [23] Jae Sung Park, Chandra Bhagavatula, Roozbeh Mottaghi, Ali Farhadi, and Yejin Choi.  Visualcomet: Reasoning about the dynamic context of a still image.  In European Conference on Computer Vision, pages 508–524. Springer, 2020.
  * [24] Jeffrey Pennington, Richard Socher, and Christopher D Manning.  Glove: Global vectors for word representation.  In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543, 2014.
  * [25] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  Advances in neural information processing systems, 28, 2015.
  * [26] Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini.  The graph neural network model.  IEEE transactions on neural networks, 20(1):61–80, 2008.
  * [27] Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling.  Modeling relational data with graph convolutional networks.  In European semantic web conference, pages 593–607. Springer, 2018\.
  * [28] Gunnar A Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta.  Hollywood in homes: Crowdsourcing data collection for activity understanding.  In European Conference on Computer Vision, pages 510–526. Springer, 2016.
  * [29] Robyn Speer, Joshua Chin, and Catherine Havasi.  Conceptnet 5.5: An open multilingual graph of general knowledge.  In Thirty-first AAAI conference on artificial intelligence, 2017\.
  * [30] Kaihua Tang, Yulei Niu, Jianqiang Huang, Jiaxin Shi, and Hanwang Zhang.  Unbiased scene graph generation from biased training.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3716–3725, 2020.
  * [31] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [32] Aming Wu, Linchao Zhu, Yahong Han, and Yi Yang.  Connective cognition network for directional visual commonsense reasoning.  Advances in Neural Information Processing Systems, 32, 2019.
  * [33] Mark Yatskar, Luke Zettlemoyer, and Ali Farhadi.  Situation recognition: Visual semantic role labeling for image understanding.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5534–5542, 2016.
  * [34] Changqian Yu, Yifan Liu, Changxin Gao, Chunhua Shen, and Nong Sang.  Representative graph neural network.  In European Conference on Computer Vision, pages 379–396. Springer, 2020.
  * [35] Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi.  From recognition to cognition: Visual commonsense reasoning.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6720–6731, 2019.