11affiliationtext: Pattern Analysis and Computer Vision (PAVIS), Istituto
Italiano di Tecnologia (IIT)22affiliationtext: Department of Information
Engineering and Computer Science (DISI), University of Trento,
Italy33affiliationtext: Department of Computer Science, University of Verona,
Italy44affiliationtext: Department of Electrical, Electronics and
Telecommunication Engineering and Naval Architecture (DITEN), University of
Genoa, Italy

# Anticipating Next Active Objects
for Egocentric Videos

Sanket Thakur  Cigdem Beyan  Pietro Morerio  Vittorio Murino  Alessio Del Bue

###### Abstract

This paper addresses the problem of anticipating the next-active-object
location in the future, for a given egocentric video clip where the contact
might happen, before any action takes place. The problem is considerably hard,
as we aim at estimating the position of such objects in a scenario where the
observed clip and the action segment are separated by the so-called “time to
contact” (TTC) segment. Many methods have been proposed to anticipate the
action of a person based on previous hand movements and interactions with the
surroundings. However, there have been no attempts to investigate the next
possible interactable object, and its future location with respect to the
first-person’s motion and the field-of-view drift during the TTC window. We
define this as the task of Anticipating the Next ACTive Object (ANACTO). To
this end, we propose a transformer-based self-attention framework to identify
and locate the next-active-object in an egocentric clip. We benchmark our
method on three datasets: EpicKitchens-100, EGTEA+ and Ego4D. We also provide
annotations for the first two datasets. Our approach performs best compared to
relevant baseline methods. We also conduct ablation studies to understand the
effectiveness of the proposed and baseline methods on varying conditions. Code
and ANACTO task annotations will be made available upon paper acceptance.

##  1 Introduction

Figure 1: The goal of our work is to anticipate the next-active-object, i.e.
to localize the object that the person will interact with in the first frame
of an action segment, based on the evidence of video clip of length
$\tau_{o}$, located $\tau_{a}$ seconds (anticipation time) before the
beginning of an action segment at time-step $t=\tau_{s}$.

The widespread use of wearable cameras prompted the design of egocentric
(first-person) systems that can readily support and help humans in their daily
activities, by augmenting their abilities [6, 7, 20]. In order to assist
users, a fundamental problem is to predict, forecast and even anticipate what
the person will do in the next few second(s). Among all the possible tasks,
one of the most relevant is to understand from an egocentric video stream,
which object a user will interact with or manipulate in the near future.
Besides, it is not just enough to localize the next-active-object (_NAO_) but
also to model the motion and Field-of-View (FoV) drift till the contact with
the object actually happens. Solving this task can help to gain more
understanding about the future activity of the person as well as the usage of
the objects. However, compared to other tasks performed with egocentric
videos, anticipating interactable objects is notably challenging since humans
interact with the environment based on their final goals and the responses
they get from the environment. On the other hand, performing this task is
useful, for example, by doing so a robot can prevent a collision between
object(s) and human(s) in a warehouse by analysing the past observation and
estimating the future point of contact or provide support in human-robot
interactions for instance in factories where objects are also moving to
anticipate the contact location based on robot movement wrt objects.

In this paper, we call this task “Anticipating Next ACTive Object” (ANACTO) by
following the nomenclature of the most recent literature [16]. In [16], _NAO_
is defined for the object which is identified in the _last observed frame_.
Instead, our ANACTO task further expands this definition to formulate the
motion and FoV drift of the interactant to anticipate the _NAO_ at its contact
point. According to [29], active objects are those which are in contact
(usually with hands) with the first-person. However, our work focuses on the
localization of the _NAO_ after a certain time at its contact point _before_
any interaction(s) begin, as shown in Figure 1. We have past evidence from
observed video clip segment of length $\tau_{o}$, which precedes the actual
action by a time to contact window $\tau_{a}$. We define ANACTO as the task of
predicting the bounding box of _NAO_ involved in the action in its starting
frame(s) at its contact point ($t=0$). Notice that, ANACTO task refers to not
only detect/localize the NAO in the last observed frame (which is the case for
Ego4D’s Short-term anticipation (STA)) but also anticipating the final
location of NAO at which the contact/interaction actually happens even in much
later upcoming frames. Instead, Ego4D STA does not aim to identify the final
interaction with the object. In Ego4D STA, it is assumed that object are
static because of the fact that the last observed frame is considered only.

We propose to address the ANACTO task by exploring the combination of object-
centered and human-centered cues while leveraging the self-attention mechanism
of vision transformers (VIT) [9]. In detail, the proposed method analyzes RGB
frames to gain an understanding of hand’s position and their motion without
explicitly using hand information. At the same time, it exploits an object
detector to include spatial positioning of objects in the observed clip. Since
ego-actions are mainly characterized as the interaction between the user’s
hands and objects in the scene, we claim that VIT’s self-attention is a good
candidate for capturing such relationships, both on frame-level and across
frames. Indeed, the correctness of this claim is shown by quantitative (which
also includes comparisons with several relevant methods) and qualitative
analysis. The main contributions of this work are the following:

  * •

A new task called Anticipating the Next ACTive Object (ANACTO) in egocentric
videos is introduced.

  * •

A novel method to address ANACTO, which is based on vision transformers,
encoding the interactions between the first-person and the objects, and
accounting for the time to contact window, is proposed.

  * •

Existing action anticipation state-of-the-art (SOTA) methods are extended to
perform ANACTO task.

  * •

Our method as well as the SOTA are benchmarked on EpicKitchens-100 [4]
(EK-100), EGTEA+ [25] and Ego4D [16] datasets. The performance comparisons
among all methods prove the effectiveness of the proposed method in all cases.
For the EK-100 and EGTEA+ datasets, we also provide annotations for the ANACTO
task.

##  2 Related Work

We first review studies on egocentric action anticipation, since our problem
follows a similar approach. Yet, instead of action classification, we focus on
regressing the location of _NAO_. Then, we review the definition of “active”
objects, which are also closely related to _NAO_ , and then investigate the
existing works on it.

Action Anticipation in Egocentric Videos. Action anticipation is the task of
predicting future actions _before_ they occur. The anticipation problem has
been well-explored for various actions from _third person videos_ [23, 24, 17,
18, 11, 14, 1, 24, 38, 33]. Instead, its application in _first-person videos_
, which is formalized in [5], has only recently gained popularity [26, 13, 28,
8] due to its applicability on wearable computing platforms [15]. We discuss
works that are closely related to our anticipation task such that perform
short-term (i.e., “recent”, see [35] for its definition) egocentric action
anticipation since we have evaluation protocols and datasets in common.

Lui et al. [26] define the egocentric action anticipation problem in terms of
human-object interaction forecasting, in which the hand movement is used as a
feature representation to predict the egocentric hand motion, interaction
hotspots and the future action. Dessalene et al. [8] perform hand-object
contact and activity modeling to anticipate partially observed and/or near
future action. For hand-object contact modeling, the short-term dynamics is
learned with 3D Convolutions. The localization of boundaries between the hands
and objects in contact is performed by applying segmentation through a U-Net
[34]. The activity modeling stage embeds the output of contact modeling
through Graph Convolutional Network (GCN) layers [22] and then fed to an LSTM,
which is followed by a fully-connected layer to make action predictions. On
the other side, there exist methods relying on the aggregation of the
information from the past frames in an observed video clip [13, 28]. For
example, [13] propose RU-LSTM, a method composed of a “rolling” LSTM (R-LSTM)
encoding the past observations, and the “unrolling” LSTM (U-LSTM) taking over
the current hidden and cell states of the R-LSTM and producing hypotheses of
future actions. Differently, the model in [28] uses a predictive model (a CNN)
and a transitional model (a CNN pre-trained on action recognition). The
predictive model directly anticipates the future action while the transitional
model is constrained to the output of the currently happening action that is
later on used to anticipate the future actions. Recently, [15] presented an
architecture based on transformers to encode the data performed by the
backbone and predict the future actions performed by the head network. [15]
achieves superior results compared to [13] and shows the better performance of
transformer backbone with respect to using many other backbones such as TSN
[39] and Faster R-CNN [31]. Compared to [15], our transformer based
architecture additionally aims to exploit the object-centric features with
spatial and temporal attention along with two losses introduced to model past
observation and learn about active object(s) to anticipate _NAO_ at its
contact point using an autoregressive decoder.

Since our ANACTO task is novel, to obtain relevant baselines to compare with,
we have modified several action anticipation SOTA tested on egocentric videos
[26, 13, 15] and tested on third-person videos [39] (we include [39] due to
its promising results demonstrated in [15] for egocentric settings). For the
baselines [26, 39], we append our decoder (see 3.3 for its definition) to
aggregate the frame level information gathered from their backbone in order to
perform the ANACTO task. In terms of encoder design, as we propose a
transformer based architecture, our method differs from [26, 13, 39] which are
based on I3D-Res50 [3], LSTMs [41], and Temporal Segment Networks, (i.e.,
Spatial and Temporal ConvNets), respectively.

Active Objects. For the first time, [29] defined _active_ and _passive_
objects in an egocentric setup. Their method is based on the appearance
differences among the objects (e.g., an opened fridge is an active object
which looks different from a closed fridge called a passive object), and the
location of the active object (i.e., active objects tend to appear close to
the center of an egocentric image). By definition, active objects are those,
which are currently involved in an interaction, e.g., being touched by humans,
whilst, the passive objects are the background objects that the human agent is
not in an interaction with, e.g., not manipulating them [29]. Dessalene et al.
[8] adapted these definitions to describe _NAO_ , which stands for the object
that will be contacted with a hand. Their method requires the visibility of
the _NAO_ and the existence of the hands in the current frames. It was also
only tested when some specific action classes (take, move, cut and open) were
considered. Instead, our method processes the frames independent to the
hand(s) visibility or presence in the current frames. Importantly, we do not
specifically restrict the possible (inter)actions between the human and the
objects, i.e., we use all the verb classes supplied by the benchmark datasets.
[19] also explored _NAO_ prediction using cues from visual attention and hand
position, but by only using a single frame for the prediction. That approach
[19] is not able to differentiate between the past or future active object,
since it does not account for the temporal information acquired by the videos.
Furnari et al. [12] also explored the _NAO problem_ by taking into account the
active/passive objects definition of [29]. Their method [12] uses an object
tracker to extract the object trajectories for a small video clip till the
last frame precedes an action. This trajectory is later used to classify
whether a given object is going to be active or passive in next frame. Such
methodology [12] is restricted to predicting the immediate NAO instead of
predicting the location of the active objects in several future frames as our
proposed method can do. Moreover, it requires an observation time which is
till the penultimate frame of an action segment, which is unpredictable in
real-life implementations. Very recently, Liu et al. [27], proposed a similar
setup but to forecast hand trajectories for interaction hotspots on next-
active-objects, i.e. confining to human hands interactions. Instead, our setup
is more generic, e.g., can include interactions of robot.

##  3 ANACTO in Egocentric Videos

In this section, we first formalize the ANACTO problem, and then we provide
details about the proposed model. Specifically, let $V$ be a given video clip,
we split the video clip into three sequential parts: the observed segment of
length $\tau_{o}$, the time to contact (TTC) window of length $\tau_{a}$ and a
given action segment which starts at timestep $t=\tau_{s}$. The goal is to
localize _NAO_ at the beginning of an action segment at timestep $t=\tau_{s}$
where the contact happens, using $\tau_{o}$ length observed video clip
$\tau_{a}$ seconds _before_ the beginning of the action segment involving
_NAO_ (see Fig. 1). In other words, ANACTO is a combination of two tasks
merged into one: (1) identifying _NAO_ from past observed segment, and (2) by
using the past observation(s), modelling the motion of a person to estimate
_NAO_ ’s location after the TTC window where actual contact happens. Notice
that, this definition assumes that for every action to be performed, a person
interacts with an object either with their hands or with a tool such that the
object becomes active at the starting point of the action. Therefore, our
problem description is not bounded with “hand”-object interactions only,
consequently our approach does not include/require the detection of hands
(e.g. the physical interactions can be performed by a tool as well).

Figure 2: Our T-ANACTO model is an encoder-decoder architecture. Its encoder
is composed of an _object detector_ and a _Vision Transformer_ [9]. The object
detector [31] takes an input frame (e.g., size of 1920$\times$1080) and
predicts the location of objects in terms of bounding boxes ($x$, $y$, $w$,
$h$) and detection confidence scores ($c$). The input of VIT are the frame(s),
first resized to, 224$\times$224 and then divided into the patches
(16$\times$16). The object detections ($x$, $y$, $w$, $h$) are also converted
to match the scaled size of the frame (i.e., 224$\times$224), reshaped, and
are then passed through a MLP to convert it into the same dimension as the
embeddings from the transformer encoder, which are later concatenated together
to be given to the decoder. There exist a linear layer between the decoder and
the T-ANACTO encoder, which adjusts the feature dimensions to be fed to the
transformer decoder. Transformer decoder uses temporal aggregation to predict
the next active object. For each frame, the decoder aggregate the features
from the encoder for current and past frames along with the embeddings of last
predicted active objects and then predicts the next active object for the
future frames.

###  3.1 Proposed Method: T-ANACTO

We propose a method, which regresses the location of _NAO_ from egocentric
videos by analyzing the past video frames, and incorporating object detections
for the input frames. Object detections refer to the location of the object
bounding box ($x_{c}$, $y_{c}$, $w$, $h$), and a confidence score ($c_{s}$)
produced by the detector. Fig. 2 illustrates the proposed method.

The proposed method (called T-ANACTO stands for Transformer-based Anticipating
Next ACTive Object) leverages the self-attention mechanism of VIT to construct
a encoder network that operates on individual frames or short clips, followed
by a transformer decoder. The _T-ANACTO encoder_ consists of a vision
transformer (VIT) [9] and an object detector [31] which are used to extract
the feature embeddings from each video frame. Our decoder is inspired from
[15], such that we exploit its _causal_ structure - to tackle a predictive
task based on past observations and make it autoregressive for an egocentric
setting. This model choice was supported by the fact that Transformer-based
end-to-end attention methods are efficient not only in recognizing actions in
given-video segments, but also in predictive video modelling. There also exist
promising results in anticipation and object detection-based tasks on static
images [2, 15, 10, 21, 32]. The T-ANACTO decoder aggregates the information
acquired in temporal dimension to collectively understand the first-person’s
movements with a final goal of predicting the location of the _NAO_. Herein,
we also introduce 2 losses to enforce the model to attend to past active
objects to predict for NAO in future frames based on previous observations.

###  3.2 T-ANACTO Encoder

The encoder of our model consists of an object detector [31] (called as object
detection head, ${H_{o}}$) combined with a VIT [9] (i.e., a video backbone).
The object detector identifies the positions of the objects, while VIT
analyzes a RGB video frame to understand the context. Different from [15], we
demonstrate the importance of object-centric features with temporal attention
along with two losses introduced to model past observation and anticipate
future contact point, described in detail below.

Given a video clip $V$ = $\\{X_{1},X_{2},\dots X_{T}\\}$ with $T$ frames,
where ${X_{t}}$ is the RGB image at time step $t$ and an action segment, we
trim the video clip into: (1) observed segment length of $\tau_{o}$, (2) TTC
window ($\tau_{a}$), before the beginning of the action segment at
$t=\tau_{s}$. Frames from the observed segment are then sampled at a frame
rate which is equal to $\tau_{a}$ to maintain consistency between frame
intervals as described in Fig. 3. Each frame extracted from the observed
segment is an input of an individual T-ANACTO Encoder. Our object detection
head ${H_{o}}$ follows a Faster R-CNN [31] architecture and consists of a
region proposal network and a regression head. It takes as input each RGB
frame $X_{t}$ and generate bounding boxes $b_{i,t}$ $\in$ $\mathbb{R}^{4}$
with corresponding confidence score ${cs}_{i,t}$ $\in$ $(0,1)$ such that:

$b_{i,t},{cs}_{i,t}=H_{o}(X_{t}),\>\>\>i\in\\{1,\ldots N\\},\vspace{-3pt}$ (1)

where $N$ is the total categories of objects for a dataset. For a category,
detections with the highest confidence score are used.

The object detections are performed for the original size of the image frames,
$X_{t}$ (e.g. $1920\times 1080$) and then the bounding boxes are scaled to
match the resized image size, $X^{r}_{t}$ of $224\times 224$ to match with the
input size of VIT [9]. The detections are then reshaped
$(BS,N,5)\xrightarrow{}(BS,-1)$ to be passed through an MLP, ${f_{MLP}}$ to
convert them to the same dimensions as the T-ANACTO encoder’s output.

For our video backbone $V_{b}$, we adopt ViT-B/16 using $224\times 224$
images, where ${X^{r}_{t}}$ is an image at a time $t$. We split each input
frame into $16\times 16$ non-overlapping patches, which are later flattened
into a 256-dimensional vector. The vector representation is then projected to
a 768-dimensional vector to be used as the input for our transformer encoder.
The feature dimensions are kept constant throughout the encoder. We also
append a learnable [cls] token in the patch features, which can later be used
to identify active object(s) label in the current frame, if any. All the other
patches are also allocated a spatial positional embedding with their patch
embedding. The resulting patch embeddings are then passed through a standard
VIT Encoder with pre-norm. Finally, the feature representations learnt for
each frame from the visual backbone are concatenated with the detections
obtained from the object detection head as follow:

$z_{t}=V_{b}(X^{r}_{t})+f_{MLP}(H_{o}(X_{t})).\vspace{-3pt}$ (2)

In the end, we add a temporal position encoding to the extracted features from
the T-ANACTO encoder for each frame, which are further given to the decoder
network.

###  3.3 T-ANACTO Decoder

We argue that the past observations can provide a lot of context to produce
hypothesis regarding the _NAO_. Therefore, for the decoder network, we take
inspiration from [15], and extending it to make it autoregressive at each
step, to aggregate the features of the past frames and exploit the last
predicted active object location which allows us to perform ANACTO.

The decoder network $D$ is designed to produce attentive features
corresponding to the future frames: $\hat{z_{1}},\dots,\hat{z_{t}}$ to
anticipate the location of the _NAO_ for each input frame as:
$\hat{z_{t}}=\textit{D}(z_{0},\dots,z_{t};\hat{h_{0}},\dots,\hat{h_{t-1}})$
(see also Fig. 2). Here $\hat{z_{t}}$ is the predicted features of the _future
frame_ at t+1 obtained after attending to all other encoded features belonging
to the frames before t+1 (i.e., $z_{0},z_{1}..z_{t}$). At each frame, the
decoder takes the previously predicted active object location $\hat{h_{t}}$ in
previous frames along with RGB features to estimate the next-active-object
position, $\hat{y_{t}}$ in future frames. Both these features are concatenated
together and are then fed to the next step. This helps in aggregating features
of the past frames and understanding the intention and final goal of the
first-person, which is defined by the action segment ground-truth label. These
features are passed through multiple decoder layers, each consisting of masked
multi-head attention, LayerNorm ($LN$), and a multi-layer perceptron (MLP) as
in [30]. The final output is then passed through another $LN$ to obtain the
final embeddings. For each decoder output $\hat{z_{t}}$, it is used to regress
the _NAO_ in the corresponding frame at t+1. The predicted features are then
fed to a linear layer $\theta$, to regress the bounding box coordinates
$\hat{y_{t}}$ $\in$ $\mathbb{R}^{8}$, i.e. $\hat{y_{t}}$ =
$\theta(\hat{z_{t}})$. The final prediction $y_{t}$ represents the model’s
output at each frame.

Figure 3: The observed video segment of length $\tau_{o}$ is sampled at a
frame rate equal to the TTC time (shown as $\tau_{a}$) to maintain consistency
in (1) the frame interval of sampled frames and (2) between the last observed
frame and the starting frame of the action segment, which starts at
$t=\tau_{s}$.

###  3.4 Loss Calculation

To train T-ANACTO, we sample a clip preceding each labeled action segment in a
given dataset, ending $\tau_{a}$ seconds before the start of the action. The
clip is then sampled with the same frame rate as $\tau_{a}$ seconds to remain
consistent with frame intervals as described in Fig. 3. The sampled frames are
then passed through our T-ANACTO model and train the network in a supervised
manner with three loss functions, described as follows.

${L}_{feat}$ defined in Eq. 3 aims at leveraging the predictive structure of
the model by supervising the future frame features predicted by the decoder to
match the true future frame features that are extracted as embeddings from the
encoder.

$\mathcal{L}_{feat}=\sum_{t=0}^{N}{||\hat{z}_{t}-z_{t+1}||}^{2}_{2},\vspace{-5pt}$
(3)

where $N$ is the number of frames in training. It is to be noted that our
model does not need the presence of hand or any active object to be present in
the observed segment. However, any active object found in the observed segment
provides additional supervision using ${L}_{cao}$, stands for current active
object loss, is a Mean Squared Error (MSE) Loss used for the prediction of
active objects _in the observed segment of the video clip_. In addition,
$\mathcal{L}_{nao}$, stands for the next-active-object loss, forces the model
to identify the location of the _NAO_ _at the start of an action_. . It is
supported by, $\mathcal{L}_{cao}$ which helps T-ANACTO to identify and keep
track of active object(s) found at the end of the observed video segment.

$\mathcal{L}_{cao}=\sum_{t=0}^{N-1}||y_{t}-\hat{y_{t}}||^{2},\mathcal{L}_{nao}=||y_{n}-\hat{y_{n}}||^{2},\vspace{-5pt}$
(4)

where $y_{t}$ $\in$ $\mathbb{R}^{8}$ and $\hat{y_{t}}$ $\in$ $\mathbb{R}^{8}$
are the ground-truth and predicted bounding boxes for active objects in _the
current frame_ , respectively. Whereas $y_{n}$ $\in$ $\mathbb{R}^{8}$ and
$\hat{y_{n}}$ $\in$ $\mathbb{R}^{8}$ are the ground-truth and predicted
bounding box for _NAO_ in _the starting frame_ of an action after
$\tau_{a}sec$, respectively. The final loss is a linear combination of the
aforementioned three losses:

$\mathcal{L}=\mathcal{L}_{feat}+\lambda_{1}\mathcal{L}_{cao}+\lambda_{2}\mathcal{L}_{nao},$
(5)

where $\lambda_{1}$, $\lambda_{2}$ are fixed weights.

##  4 Experimental Analysis

The experimental analyses were conducted on three major egocentric video
datasets, described in Sec. 4.1. As this is the first time ANACTO task is
being benchmarked, there is no existing method performing it. Therefore, we
adapted the SOTA action anticipation methods to perform comparisons in Sec.
4.2. We described the implementation details of T-ANACTO in Sec. 4.3.

###  4.1 Datasets

EK-100 [4]. Consists of about 100 hours of recordings with over 20M frames
comprising daily activities in kitchens, recorded with 37 participants. It
includes 90K action segments, labeled with 97 verbs and 300 nouns (i.e.
manipulated objects). It supplies the annotations regarding the hand and
object interactions, which are used for ANACTO. In detail, the aforementioned
annotations are in terms of the prediction results of a hand-object
interaction detector [36], which provides the hand location, side, contact
state, and a bounding box surrounding the object that the hand is in contact.
Such detector [36] was trained on EK-55 [5], EGTEA [25] and CharadesEgo [37]
datasets, and applied on EK-55 [5] dataset to annotate it with respect to the
hand-object interactions. We use the following annotations: the locations of
both hands (i.e., the bounding boxes $b\in$ $\mathbb{R}^{8}$), and the
locations of the objects along with the contact state information at each
frame of each video and, then curate the final ground-truth data for ANACTO
problem. It is important to mention that the videos in this dataset were
collected with different frame rates. In order to apply the methods: [13, 39,
26] requiring frame rates fixed to 30 frame per second, we converted each
video to this constant frame rate, thus the annotations regarding the hand
locations and active objects’ locations are also interpolated accordingly.

EGTEA+ [25]. Includes 28 hours of videos containing 106 action categories,
which corresponds to 2.4M frames. There exist 10325 action segments associated
to 19 verbs and 53 nouns (i.e., objects) that were recorded with 32
participants. It is important to notice that yet there exist no publicly
available source supplying annotations needed to perform ANACTO for EGTEA+.
Therefore, we created the hand-object interaction annotations following the
annotation pipeline of EK-100 [4] dataset. These include: the hand locations
(bounding boxes b $\in$ $\mathbb{R}^{8}$ and the corresponding detection
confidence scores) at each frame, the active object locations and their
contact state. First, all the videos are converted to a constant frame rate of
30 fps. Then, each frame is fed to the hand-object interaction detector model
from [36]. The hand and object threshold is kept at 0.5 to produce better
qualitative results, which is also the same when extracting the annotations
for EpicKitcen-100 dataset [4]. Additionally, we provide annotations for the
videos with original frame rate for its original frame size.

Ego4D [16]. This is the largest first-person dataset recently released. The
dataset is split into 5 different categories, each focusing on a different
task, combining for a total of 3,670 hours of egocentric videos across 74
locations. For this task, we focus on the forecasting split, containing 1000
videos for a total of 960 hours, annotated at 30 fps for the short term
interaction anticipation task. The annotations are for the _NAO_ in the _last
observed frame_.

###  4.2 Baseline Methods

We compare T-ANACTO with SOTA action anticipation methods, namely AVT [15],
RULSTM [13], Liu et al. [26] and TSN [39]. For RULSTM [13], we used pre-
extracted RGB, flow and object features as in their paper, for EK-100 and
EGTEA+. For Ego4D, we computed the flow and RGB features by following the same
TSN model mentioned in [13], which were then fed as the inputs to the RULSTM
model. We also tested individual modalities with TSN [39] (ResNet101) for RGB
frames and RULSTM-object centric path for object modality. Moreover, we used
object detections as well as their confidence score from the object detector
[31] to be used as object features in RULSTM(fusion) and RULSTM(obj). We
modified and re-trained all these aforementioned methods in order to perform
ANACTO task. We explored these methods (noticed that they were used for action
anticipation in egocentric videos, a.k.a. a _classification task_) because our
problem formulation is very much related to action anticipations, and we claim
that these methods can provide effective learning for ANACTO _regression task_
by modelling past motion. For each model, we replace the last classification
layer with a regression layer to predict the bounding boxes $\hat{y_{n}}$
$\in$ $\mathbb{R}^{8}$ regarding the next active object. Since TSN [39] method
processes individual frames and not a video clip, for the corresponding
experiments, we appended the whole T-ANACTO decoder layer to the TSN [39]
method allowing the aggregation of information from all frames (i.e., tuning
the task from frame-level processing to video processing). Throughout this
paper, we refer to these methods as _baselines_.

###  4.3 Implementation Details of T-ANACTO

T-ANACTO was trained with an SGD optimizer for 50 epochs with a learning rate
of $1e-5$. Recall that a linear layer exist after the output of the decoder to
regress the bounding box coordinates $\hat{y_{t}}$ $\in$ $\mathbb{R}^{8}$
(here the results are in $\mathbb{R}^{8}$, confining to a single active object
for each hand. We kept the values of $\lambda_{2}$ as 1.0 and $\lambda_{1}$ as
0.5 (see Eq. 5) respectively, while training T-ANACTO. Also, and the weight
for feature loss is set to 1.0 For training and testing, our model takes 10
sampled frames as input and takes 1s to process a batch of 4 clips during
inference. We keep the required input number of frame for each baseline method
as proposed in their original paper.

We used annotations from [36] detector for identifying active objects in
observed segment and to train the model for all datasets with
$\mathcal{L}_{cao}$ loss. Specifically, for EK-100 [4] and EGTEA+ datasets,
during training, we maintained a lookup window of 10 frames from starting
frame of action to look for first identified location of active objects i.e;
bounding boxes (if visible) to be labeled as ground truth for ANACTO task. It
is also possible that for some clips, _true contact_ i.e., the actual
interaction with an object can start sometime later after our lookup window.
For those cases, we do not get bounding box labels for the location of active
object. This means no object was actually active during the start of the
action segment. However, we checked whether if this situation does lead to any
inconsistency and observed that an active object is present _94% and 92%_ of
the times in the first 10 frames of the action segment for the EK-100 and
EGTEA dataset, respectively. It is important to notice that EK-100 and EGTEA
do not supply object detections. As mentioned before, to obtain this
information, we rely on Faster-RCNN [31] provided by [4] pre-trained on EK-55
[5] to detect the location of every object in the scene with a confidence
score associated with each prediction b $\in$ $\mathbb{R}^{5}$. For both
datasets, we use the training and test splits provided by [13] for the
evaluations of the T-ANACTO and the baseline methods. On the other hand, for
Ego4D, we used the forecasting split for training and validation provided by
[16]. It is important to notice that the annotations provided for _NAO_ are
wrt. only _the last observed frame_. As Ego4D is highly big-scaled, it was not
possible to annotate it as we performed for other datasets. Therefore, we
utilized only the supplied data as the ground-truth. On the other hand, this
allowed us to show another utility of the ANACTO task, i.e., its setup also
works for the model(s) to forecast _NAO_ in the last observed frame.

##  5 Results

As the evaluation metrics, Average Precision ($AP$) with various IoU
thresholds: 5, 10, 20 and 50 as well as their average shown as $AP_{avg}$ were
used.

The Effect of Losses and The Backbone. We first present an ablation study to
evaluate losses given in Eq. 5 as well as testing a different backbone (i.e.,
ResNet101, notice that this is the backbone used by TSN [39]) while keeping
the other settings of T-ANACTO the same. The corresponding results are given
in Table 1, when the experiments were performed on EK-100 dataset and the
anticipation length $\tau_{a}=0.25s$. As seen, using the transformer backbone
compared to ResNet101 improves the results for all cases (ResNet101 vs.
T-ANACTO w/ $\mathcal{L}_{nao}$ and ResNet101 vs. T-ANACTO w/
$\mathcal{L}_{cao}$+$\mathcal{L}_{nao}$). Moreoever, $\mathcal{L}_{cao}$
brings in important performance improvements to the ANACTO task, highlighting
the importance of using the object-centric features.

Ablation | AP5 | AP10 | AP20 | AP50 | $AP_{avg}$
---|---|---|---|---|---
ResNet101 | 31.2 | 28.1 | 17.4 | 2.3 | 19.75
T-ANACTO w/ $\mathcal{L}_{nao}$ | 33.5 | 29.6 | 19.3 | 2.4 | 21.2
T-ANACTO w/ $\mathcal{L}_{cao}$+$\mathcal{L}_{nao}$ (FULL) | 37.1 | 32.6 | 21.1 | 4.1 | 23.7

Table 1: Ablation study performed on EK-100 [4] to investigate the effect of
losses and the backbones of T-ANACTO.

Anticipation time |  $\tau_{a}=$ 1.0 s |  $\tau_{a}=$ 0.5 s |  $\tau_{a}=$ 0.25 s
---|---|---|---
Models | AP5 | AP10 | AP20 | AP50 | $AP_{avg}$ | AP5 | AP10 | AP20 | AP50 | $AP_{avg}$ | AP5 | AP10 | AP20 | AP50 | $AP_{avg}$
AVT [15] | 25.2 | 19.1 | 13.6 | 1.5 | 14.9 | 30.0 | 26.4 | 17.2 | 3.1 | 19.2 | 32.3 | 27.1 | 18.4 | 3.3 | 20.2
RULSTM [13] | 27.6 | 21.3 | 14.2 | 2.1 | 16.3 | 29.5 | 24.2 | 15.5 | 3.0 | 18.0 | 31.6 | 25.8 | 16.6 | 3.1 | 19.3
TSN(rgb) [39] | 17.2 | 12.1 | 7.6 | 0.7 | 9.4 | 20.2 | 16.4 | 8.6 | 1.7 | 11.7 | 25.6 | 19.1 | 11.8 | 1.8 | 14.6
RULSTM(obj) [39] | 24.4 | 19.3 | 11.1 | 1.7 | 14.1 | 24.4 | 19.1 | 11.3 | 1.8 | 14.1 | 27.0 | 20.2 | 14.7 | 1.9 | 16.0
Liu et al. [26] | 13.1 | 9.8 | 5.2 | 0.4 | 7.1 | 13.4 | 10.7 | 5.6 | 0.6 | 7.6 | 14.7 | 10.4 | 5.6 | 0.7 | 7.9
T-ANACTO | 34.4 | 28.8 | 18.1 | 3.2 | 21.2 | 35.4 | 29.7 | 20.2 | 3.3 | 22.1 | 37.1 | 32.6 | 21.1 | 4.1 | 23.7

Table 2: Results of our T-ANACTO model and other baseline methods for
different TTC duration, i.e., 1, 0.5 and 0.25 seconds, tested on the EK-100
[4]. Best result of each column are given in bold.

#### Effect of Anticipation Length.

We compare the performances of T-ANACTO and the baseline methods for various
anticipation lengths for the ANACTO task in the unobserved scenes. This set of
experiments was realized on EK-100 dataset [4] and the corresponding results
are given in Table 2. It is important to mention that since we keep the total
number of sampled frames from a given observed clip as constant throughout the
experiments, the change in anticipation time $\tau_{a}$ also changes the
observed length $\tau_{o}$ of the clip. In other words, in these sets of
experiments, the decrease in anticipation length $\tau_{a}$ also reduces the
respective observed length $\tau_{a}$ of time duration. The results given in
Table 2 show that changing the anticipation lengths from higher values to
lower values (e.g., from 1s to 0.5s or from 0.5s to 0.25s), as expected,
increases the performance of T-ANACTO as well as all baseline methods.

Comparisons among T-ANACTO and Baselines. Table 2 presents a performance
comparison among T-ANACTO and baseline methods on EK-100 dataset. As seen, our
method T-ANACTO surpasses all the other methods in all metrics, for all TTC
durations, while the second-best method is chaining for different TTC
durations. We also present comparisons on EGTEA+ and Ego4D datasets in Tables
3 and 4, respectively, when the TTC duration $\tau_{a}$ is 0.25s for EGTEA and
rate of sampling frames is 0.25s for Ego4D. To do so, for EGTEA+, we used
training and testing splits-1 (see [13] for details) and for Ego4D, the
experiments were conducted with the training and validation splits provided
for the forecasting task. As mentioned in Sec. 4.1, the _NAO_ for Ego4D is
identified at the end of the past observed segment. Even for this setup, we
notice that the attention-based mechanism elevated by object centric
information performs better, compared to other baselines. The obtained results
in the aforementioned tables are in line with the results obtained for the
EK-100 dataset, showing that T-ANACTO outperforms the other baseline methods,
while the performance improvement can be up to 12% in terms of $AP_{avg}$.

Models | AP5 | AP10 | AP20 | AP50 | $AP_{avg}$
---|---|---|---|---|---
AVT [15] | 19.7 | 16.5 | 10.2 | 2.6 | 12.2
RULSTM [13] | 18.8 | 13.4 | 7.7 | 1.4 | 10.3
TSN(rgb) [39] | 14.8 | 12.1 | 7.4 | 1.4 | 9.0
RULSTM(obj) [13] | 15.1 | 12.4 | 6.8 | 1.3 | 9.0
Liu et al. [26] | 11.8 | 8.5 | 5.7 | 1.0 | 6.8
T-ANACTO | 26.6 | 21.0 | 14.7 | 2.8 | 16.3

Table 3: T-ANACTO and the baseline methods’ performances when they are tested
on EGTEA+ dataset [25] with the TTC duration $\tau_{a}=0.25s$. Best result of
each column are given in bold.

Models | AP5 | AP10 | AP20 | AP50 | $AP_{avg}$
---|---|---|---|---|---
AVT [15] | 38.8 | 28.7 | 12.9 | 2.9 | 20.8
RULSTM [13] | 37.6 | 27.4 | 10.3 | 1.7 | 19.3
TSN(rgb) [39] | 35.5 | 23.2 | 8.5 | 1.5 | 17.1
RULSTM(obj) [13] | 34.6 | 21.3 | 8.2 | 1.5 | 16.4
Liu et al. [26] | 15.2 | 11.1 | 7.4 | 1.1 | 8.7
T-ANACTO | 41.2 | 31.4 | 18.6 | 4.6 | 24.0

Table 4: T-ANACTO and the baseline methods’ performances when they are tested
on Ego4D dataset [16] to identify NAO. Frames are sampled from observed
segment at $0.25s$. Best result of each column are given in bold.

#### Qualitative Results.

We visualize the effective spatial attention by our T-ANACTO encoder on the
last observed frame in Fig. 4 for EK-100 [4]. The red regions demonstrate the
regions of interest to the model, which correspond to human-object interaction
in the future frames and help in anticipating the _NAO_. The results show that
our model learns to focus on objects which are likely to be in contact with
human hands based on observation till last observed frame, and thus the
inference can also be performed before the contact happens. Notice in the
second column, even though the object is not active in the starting frame, our
model learns to focus on a possible object which becomes active later. We also
notice that the model performs equally well for different lighting conditions.
Besides, it is also interesting to note that T-ANACTO model is also able to
identify human-interaction hotspots for an object in some case. In the Supp.
Material, we provide more qualitative results of our model for identifying
objects in last observed frame, different TTC $\tau_{a}$, and discuss failure
cases for the model.

Figure 4: The top row shows the “last observed frame”, the middle row shows
“the region of interest of T-ANACTO”, and the bottom row shows “the starting
frame of an action”. The green box(es) in the last row represent the location
of _NAO_ bounding box in the starting frame(s) of action.

##  6 Conclusions

We have investigated the problem of anticipating next active object
localization. First, we discussed the formulation of the ANACTO task. We then
presented a new vision transformer based model, T-ANACTO which learns to
encode hand-object interactions with the help of an object detector. We proved
its effectiveness by comparing it against relevant strong anticipation based
baseline methods. The experimental evaluation highlights that: (1) the object-
centered cues help in elevating the performance to locate the next possible
active object; (2) the effectiveness of the model increases when the
anticipation time for the prediction before the beginning of an action is kept
short. Besides, we also discuss the effect of observation length on the
performance of model(s). (3) Our model effectively learns to identify and
allocate attention to possible action objects in the future, as realized from
qualitative results. (4) Importantly, T-ANACTO is also able to detect _NAO_
location even in the last observed frame. Finally, we also supply the ANACTO
task annotations for EGTEA+ and EK-100 datasets, i.e., hand and active object
bounding box annotations along with their contact state as well as providing
the object annotations for the entire dataset using an object detector pre-
trained on EK-55 [5].

As future work, we will extend the ANACTO task to predict the dynamic TTC,
noun and verb for _NAO_ , and investigate the use of an object tracker with
other human-centered cues such as gaze and the appearance of objects over
time. We will also investigate the effect of action recognition on _NAO_
identification and localization.

Anticipating Next Active Objects for Egocentric Videos: Supplementary Material

This supplementary material includes the visualization of the attention maps
of our T-ANACTO encoder, which is given for different time to contact window
(Section 7) for EpicKitchen [4, 5] and EGTEA [25] datasets. The attention maps
provide an intuition on learning of T-ANACTO to identify “interactable”
objects (i.e; possible next-active-object) in the scene and then model the
motion of the person till its TTC to anticipate its contact location in future
frame. We also provide further visualizations for Ego4D [16] dataset when the
next-active-object is identified at the last observed frame irrespective of
time to contact with that object. Then, we discuss the failure cases of our
model in Section 8 through multiple exemplary images. In addition, we also
provide a video giving the details of the transition of attention over past
frames till the last observed frame in a video clip.

##  7 Visualization of the Attention Maps

For training purposes, the vision transformer (VIT) [9] model was implemented
using the timm [40] based pytorch-image model, which does not provide
attention weights for the output of transformer encoder. To visualize the
spatial attention of our T-ANACTO encoder, we implemented a similar model with
the same nomenclature for the layers to load the trained weights from the
training of the model. The attention weights are then extracted from each
block layer and then stacked together to project the learning of our encoder.

We show the effectiveness of our model T-ANACTO for anticipating next active
object task (ANACTO) as spatial attention of our encoder in additional figures
for both EpicKitchen-100 [4] and EGTEA+ [25] datasets in Fig. 5, 6, 7 and 9.
Using this visualization, one can understand how the confidence of the model
differs as it analyzes frames that are temporally distant from the beginning
of an action segment for different TTC window $\tau_{a}$. In other words, we
are able to compare the diversity of attention for different TTC window,
$\tau_{a}$ v/s observed $\tau_{o}$ time of video clips for EpicKitchen [4]
dataset.

In detail, extending on the visualization provided in the main paper for
EpicKitchen-100 dataset, herein, we report additional results for that dataset
for different TTC window $\tau_{a}=$ 0.25 seconds, 0.5 seconds, 1.0 second in
Fig. 5, 6, 7, respectively. In the mentioned figures, we report the last
observed frame by the model and the attention map generated for that
particular frame by our T-ANACTO encoder to predict the location of next
active object. We also report the ground truth results for the active object
at the starting frame of action segment. This attention map(s) is generated
after considering the past frames and the last frame of observed segment
$\tau_{o}$. As mentioned in the main paper, the change in $\tau_{a}$ for a
video clip also affects the observed video segment length $\tau_{o}$
proportionally.

Figure 5: Results showing the attention map generated by our T-ANACTO encoder
for last observed frame of video clip with TTC $\boldsymbol{\tau_{a}=0.25}$
seconds before the beginning of the action. The red regions depicts the region
of interest to identify the next active object in the starting frame of the
action. The green bounding box for the starting frame of the action (row)
shows the localization of the active object for that frame. It is interesting
to note that for segments which there is no active object at the start of the
action, our encoder is able to identify the possible area of interest for next
future frames post the starting frame of the action.  Figure 6: Results
showing the attention map generated by our T-ANACTO encoder for last observed
frame of video clip with TTC $\boldsymbol{\tau_{a}=0.5}$ seconds before the
beginning of the action.  Figure 7: Results showing the attention map
generated by our T-ANACTO encoder for last observed frame of video clip with
TTC $\boldsymbol{\tau_{a}=1.0}$ second before the beginning of the action.
Figure 8: Our T-ANACTO model is designed to identify and locate the hand-
object interaction location for future frames. In the process, it also learns
to attribute to hand position in an image frame without explicitly providing
hand location. The figures provided illustrate the spatial attention of our
model to hand positions besides possible next active object. Figure 9: Results
shows the spatial attention map for EGTEA+ dataset. The green bounding box
specifies the location of the active object at the starting of an action.
Figure 10: Results shows the spatial attention map for ego4d dataset when
trained to identify next active object wrt last observed frame. The green
bounding box specifies the location of the object which will become active in
the future. The highlighted region specifies the attention stress by the model
in the last observed frame.  Figure 11: In the figure, we report results for
some of the failure cases of our model as discussed in Section 8.1. (a) The
model fails to attribute attention to objects which are light colored or
easily camouflaged with the background. (b) When the scene completely changes
at the beginning of action from the past observed segment. Figure 12: In the
figure, we report results for some of the failure cases of our model as
discussed in Section 8.2 for Ego4D dataset. (a) The model fails to attribute
for higher TTC time for a given next active object. (b) For objects which are
tiny or transparent or scattered around multiple objects it is difficult to
identify next active object for larger TTC. Figure 13: Results show the
diversity of spatial attention for the last observed frame preceding the
beginning of an action segment for different setups of TTC window for
$\tau_{a}=0.25,0.5,1.0$ seconds. The attention corresponds to the red region
in the image. The regions tend to appear more assertive as the model examines
frame closer to action segments i.e; as the $\tau_{a}$ is decreased. This also
attributes to a higher accuracy of the model for shorter time to contact
window.

To qualitatively understand the improved performance of the model as the
$\tau_{a}$ is reduced from $1.0s$ to $0.25$ seconds, we report the comparison
in Fig. 13. It is visible that as the model is fed with frames that are closer
to the beginning of an action segment, i.e lower $\tau_{a}$, its confidence
for the next active object increases and so the performance gain can be
justified.

It is also important to mention that, for most of the results, one can notice
that our model is also able to identify the hand’s positions and interaction
hotspots for certain objects, although our model does not explicitly require
the hand’s position as an input. We confirm this by reporting our results for
the EpicKitchen-100 dataset [25] in Fig. 8. Since our method learns to
identify the future hand-object interaction it focuses on locating the
position of hands and respectively locate the next active object in consequent
starting frame of an action segment.

In Fig. 9, we provide the attention map for the learning of our model for
EGTEA+ dataset [25] when trained on train split 1 and tested on test split 1.
We also provide a video visualizing the transition of attention on all past
frames till the last observed frame for different clips.

Ego4D [16]. In Fig. 10, we provide the attention map for the learning of our
model for Ego4D dataset when trained on training set of forecasting split to
predict the next active object location wrt last observed frame.

##  8 Success and Failure cases

All the visualizations discussed in the previous section, is given for the
cases T-ANACTO is successful to anticipate the next active object correctly.
In this section, we discuss the cases which can be considered as failure for
T-ANACTO.

###  8.1 Epic Kitchen and EGTEA

We were able to identify two major cases for EpicKitchen and EGTEA:

#### 1) Light colored objects.

We noticed that the model is not able to confine its attention to those areas
in the video clips where a light colored or transparent object is used for
human-object interaction (see Fig. 11(a)). It could perhaps be a failure of
the object detection model which is not able to identify items due to the
transparent nature of object and camouflage with the background of frames(s).
However, for most of the video clips consisting of light colored objects, our
model is able to identify the hand’s positioning in the frames as described in
Fig. 8 which can be exploited to further extend the work in this domain.

#### 2) Scene transition.

As stated earlier, the next active object detection is a challenging task due
to the consistent nature of humans to continuously interact with the
environment. In the process, a person does the interaction with the objects
based on the activities being performed which can lead to sudden change of
scenes from one moment to another. Therefore, a current scene at the start of
action segment might be drastically different wrt past observed frames. In
those cases, it is extremely difficult for the model to locate ”interactable”
objects in the scene which has not be observed by the model (see Fig. 11(b)).

###  8.2 Ego4D

We provide the visualization of cases in Fig. 12

#### 1) Sampling of frames.

Since our model take input frames at a sampled interval, it is trained to
output predictions after the the sampled interval time after the last observed
frame. However, in Ego4D dataset the TTC for a next active object varies
drastically for each clip, which is one of the main reasons our model suffers
for those objects whose TTC are much higher than sampled frame rate for our
input frames.

#### 2) Tiny and clustered objects.

We also notice that our model fails for tiny / transparent objects in the
scene or where multiple objects are scattered in the frame.

## References

  * [1] Yazan Abu Farha, Alexander Richard, and Juergen Gall.  When will you do what? - anticipating temporal occurrences of activities.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
  * [2] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I, volume 12346 of Lecture Notes in Computer Science, pages 213–229. Springer, 2020\.
  * [3] João Carreira and Andrew Zisserman.  Quo vadis, action recognition? a new model and the kinetics dataset.  2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4724–4733, 2017.
  * [4] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, , Antonino Furnari, Jian Ma, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray.  Rescaling egocentric vision.  CoRR, abs/2006.13256, 2020.
  * [5] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray.  Scaling egocentric vision: The epic-kitchens dataset.  In European Conference on Computer Vision (ECCV), 2018.
  * [6] Dima Damen, Teesid Leelasawassuk, Osian Haines, Andrew Calway, and Walterio Mayol-Cuevas.  You-do, i-learn: Discovering task relevant objects and their modes of interaction from multi-user egocentric video.  In Proceedings of the British Machine Vision Conference. BMVA Press, 2014.
  * [7] Dima Damen, Teesid Leelasawassuk, and Walterio Mayol-Cuevas.  You-do, i-learn: Egocentric unsupervised discovery of objects and their modes of interaction towards video-based guidance.  Computer Vision and Image Understanding, 149:98–112, 2016.  Special issue on Assistive Computer Vision and Robotics - ”Assistive Solutions for Mobility, Communication and HMI”.
  * [8] Eadom Dessalene, Chinmaya Devaraj, Michael Maynord, Cornelia Fermuller, and Yiannis Aloimonos.  Forecasting action through contact representations from first person video.  IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1–1, 2021.
  * [9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In International Conference on Learning Representations, 2021.
  * [10] Yuxin Fang, Bencheng Liao, Xinggang Wang, Jiemin Fang, Jiyang Qi, Rui Wu, Jianwei Niu, and Wenyu Liu.  You only look at one sequence: Rethinking transformer in vision through object detection.  Advances in Neural Information Processing Systems, 34, 2021.
  * [11] Panna Felsen, Pulkit Agrawal, and Jitendra Malik.  What will happen next? forecasting player moves in sports videos.  In Proceedings of the IEEE international conference on computer vision, pages 3342–3351, 2017.
  * [12] Antonino Furnari, Sebastiano Battiato, Kristen Grauman, and Giovanni Maria Farinella.  Next-active-object prediction from egocentric videos.  Journal of Visual Communication and Image Representation, 49:401–411, 2017.
  * [13] Antonino Furnari and Giovanni Maria Farinella.  What would you expect? anticipating egocentric actions with rolling-unrolling lstms and modality attention.  In International Conference on Computer Vision, 2019.
  * [14] Jiyang Gao, Zhenheng Yang, and Ram Nevatia.  Red: Reinforced encoder-decoder networks for action anticipation.  BMVC, 2017.
  * [15] Rohit Girdhar and Kristen Grauman.  Anticipative Video Transformer.  In ICCV, 2021.
  * [16] Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Ilija Radosavovic, Santhosh Kumar Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Mengmeng Xu, Eric Zhongcong Xu, Chen Zhao, Siddhant Bansal, Dhruv Batra, Vincent Cartillier, Sean Crane, Tien Do, Morrie Doulaty, Akshay Erapalli, Christoph Feichtenhofer, Adriano Fragomeni, Qichen Fu, Abrham Gebreselasie, Cristina González, James Hillis, Xuhua Huang, Yifei Huang, Wenqi Jia, Weslie Khoo, Jáchym Kolář, Satwik Kottur, Anurag Kumar, Federico Landini, Chao Li, Yanghao Li, Zhenqiang Li, Karttikeya Mangalam, Raghava Modhugu, Jonathan Munro, Tullie Murrell, Takumi Nishiyasu, Will Price, Paola Ruiz, Merey Ramazanova, Leda Sari, Kiran Somasundaram, Audrey Southerland, Yusuke Sugano, Ruijie Tao, Minh Vo, Yuchen Wang, Xindi Wu, Takuma Yagi, Ziwei Zhao, Yunyi Zhu, Pablo Arbeláez, David Crandall, Dima Damen, Giovanni Maria Farinella, Christian Fuegen, Bernard Ghanem, Vamsi Krishna Ithapu, C. V. Jawahar, Hanbyul Joo, Kris Kitani, Haizhou Li, Richard Newcombe, Aude Oliva, Hyun Soo Park, James M. Rehg, Yoichi Sato, Jianbo Shi, Mike Zheng Shou, Antonio Torralba, Lorenzo Torresani, Mingfei Yan, and Jitendra Malik.  Ego4d: Around the world in 3,000 hours of egocentric video.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18995–19012, June 2022.
  * [17] De-An Huang and Kris M Kitani.  Action-reaction: Forecasting the dynamics of human interaction.  In European Conference on Computer Vision, pages 489–504. Springer, 2014.
  * [18] Ashesh Jain, Hema S Koppula, Bharad Raghavan, Shane Soh, and Ashutosh Saxena.  Car that knows before you do: Anticipating maneuvers via learning temporal driving models.  In Proceedings of the IEEE International Conference on Computer Vision, pages 3182–3190, 2015.
  * [19] Jingjing Jiang, Zhixiong Nan, Hui Chen, Shitao Chen, and Nanning Zheng.  Predicting short-term next-active-object through visual attention and hand position.  Neurocomputing, 433:212–222, 2021.
  * [20] Takeo Kanade and Martial Hebert.  First-person vision.  Proceedings of the IEEE, 100:2442–2453, 2012.
  * [21] Bumsoo Kim, Junhyun Lee, Jaewoo Kang, Eun-Sol Kim, and Hyunwoo J Kim.  Hotr: End-to-end human-object interaction detection with transformers.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 74–83, 2021.
  * [22] Thomas N Kipf and Max Welling.  Semi-supervised classification with graph convolutional networks.  arXiv preprint arXiv:1609.02907, 2016.
  * [23] Kris M Kitani, Brian D Ziebart, James Andrew Bagnell, and Martial Hebert.  Activity forecasting.  In European conference on computer vision, pages 201–214. Springer, 2012.
  * [24] Tian Lan, Tsung-Chuan Chen, and Silvio Savarese.  A hierarchical representation for future action prediction.  In European Conference on Computer Vision, pages 689–704. Springer, 2014.
  * [25] Yin Li, Miao Liu, and James M. Rehg.  In the eye of beholder: Joint learning of gaze and actions in first person video.  In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
  * [26] Miao Liu, Siyu Tang, Yin Li, and James Rehg.  Forecasting human object interaction: Joint prediction of motor attention and actions in first person video.  In ECCV, 2020.
  * [27] Shaowei Liu, Subarna Tripathi, Somdeb Majumdar, and Xiaolong Wang.  Joint hand motion and interaction hotspots prediction from egocentric videos.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3282–3292, June 2022.
  * [28] Antoine Miech, Ivan Laptev, Josef Sivic, Heng Wang, Lorenzo Torresani, and Du Tran.  Leveraging the present to anticipate the future in videos.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 0–0, 2019.
  * [29] Hamed Pirsiavash and Deva Ramanan.  Detecting activities of daily living in first-person camera views.  In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 2847–2854, 2012.
  * [30] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners.  2019\.
  * [31] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
  * [32] Ivan Rodin, Antonino Furnari, Dimitrios Mavroeidis, and Giovanni Maria Farinella.  Predicting the future from first person (egocentric) vision: A survey.  Computer Vision and Image Understanding, 211:103252, 2021.
  * [33] Cristian Rodriguez, Basura Fernando, and Hongdong Li.  Action anticipation by predicting future dynamic images.  In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, September 2018.
  * [34] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer, 2015.
  * [35] Fadime Sener, Dipika Singhania, and Angela Yao.  Temporal aggregate representations for long term video understanding.  CoRR, abs/2006.00830, 2020.
  * [36] Dandan Shan, Jiaqi Geng, Michelle Shu, and David Fouhey.  Understanding human hands in contact at internet scale.  2020\.
  * [37] Gunnar A. Sigurdsson, Abhinav Kumar Gupta, Cordelia Schmid, Ali Farhadi, and Alahari Karteek.  Actor and observer: Joint modeling of first and third-person videos.  2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7396–7404, 2018.
  * [38] Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba.  Anticipating visual representations from unlabeled video.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
  * [39] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Val Gool.  Temporal segment networks: Towards good practices for deep action recognition.  In ECCV, 2016.
  * [40] Ross Wightman.  Pytorch image models.  https://github.com/rwightman/pytorch-image-models, 2019.
  * [41] Yong Yu, Xiaosheng Si, Changhua Hu, and Jianxun Zhang.  A review of recurrent neural networks: Lstm cells and network architectures.  Neural computation, 31(7):1235–1270, 2019.


# NLU-STR at SemEval-2024 Task 1: Generative-based Augmentation and Encoder-
based Scoring for Semantic Textual Relatedness

Sanad Malaysha, Mustafa Jarrar, Mohammed Khalilia
Birzeit University, Palestine
{smalaysha, mjarrar<EMAIL_ADDRESS>

###### Abstract

Semantic textual relatedness is a broader concept of semantic similarity. It
measures the extent to which two chunks of text convey similar meaning or
topics, or share related concepts or contexts. This notion of relatedness can
be applied in various applications, such as document clustering and
summarizing. SemRel-2024, a shared task in SemEval-2024, aims at reducing the
gap in the semantic relatedness task by providing datasets for fourteen
languages and dialects including Arabic. This paper reports on our
participation in Track A (Algerian and Moroccan dialects) and Track B (Modern
Standard Arabic). A BERT-based model is augmented and fine-tuned for
regression scoring in supervised track (A), while BERT-based cosine similarity
is employed for unsupervised track (B). Our system ranked 1st in SemRel-2024
for MSA with a Spearman correlation score of 0.49. We ranked 5th for Moroccan
and 12th for Algerian with scores of 0.83 and 0.53, respectively.

NLU-STR at SemEval-2024 Task 1: Generative-based Augmentation and Encoder-
based Scoring for Semantic Textual Relatedness

Sanad Malaysha, Mustafa Jarrar, Mohammed Khalilia Birzeit University,
Palestine {smalaysha, mjarrar<EMAIL_ADDRESS>

##  1 Introduction

The literature commonly examines semantic similarity, where the focus is on
whether two linguistic units (words, phrases, sentences, etc.) share similar
meanings (Bentivogli et al., 2016). However, semantic textual relatedness
(STR) is less explored due to its complexity and the scarcity of datasets
(Abdalla et al., 2023; Darwish et al., 2021). While the former task checks for
the presence of similar meaning or paraphrase, STR takes a more comprehensive
approach, evaluating relatedness across multiple dimensions, spanning topical
similarity, conceptual overlap, contextual coherence, pragmatic connection,
themes, scopes, ideas, stylistic conditions, ontological relations,
entailment, temporal relation, as well as semantic similarity itself (Miller
and Charles, 1991; Halliday and Hasan, 2014; Jarrar, 2021, 2011). For example,
consider the two sentences (The Earth orbits the sun at a speed of ~110,000
km/h.) and (Earth rotates at ~1670 km/h around its axis.). They hold semantic
relatedness through the shared topic of Earth’s speeds. In contrast, both
sentences are not semantically similar as they possess distinct meanings. This
illustrates the broader range of STR as described by Abdalla et al. (2023),
which ranges from highly relevant sentences, expressing the same idea with
different wording, to entirely unrelated sentences, discussing unrelated
topics.

Semantic relatedness has proven to be useful in evaluating sentence
representations generated by language models (Asaadi et al., 2019), in
addition to question answering (Tsatsaronis et al., 2014), machine translation
(Mi and Xie, 2024), plagiarism detection (Sabir et al., 2019), word-sense
disambiguation (Al-Hajj and Jarrar, 2021a; Malaysha et al., 2023), among
others. Exploring the relatedness and similar tasks in languages other than
English is hindered by the lack of data (Jarrar et al., 2023b; Al-Hajj and
Jarrar, 2021b). The SemRel-2024 shared task (Ousidhoum et al., 2024a) provided
datasets in fourteen languages and offered three tracks. In the supervised
track (A), training and testing are performed on the same language. In the
unsupervised track (B), the use of labeled data for training is prohibited;
and in the cross-lingual track (C), testing is conducted on a different
language than the one used for training.

This paper presents our contribution to track A and track B. In track A, we
fine-tuned BERT models using the Algerian and Moroccan sentence pairs to
produce similarity scores. To enrich the data, we augmented the SemRel-2024
dataset (Ousidhoum et al., 2024a) by generating additional sentence pairs from
Google Gemini 111https://gemini.google.com/, a generative model, using a
predefined prompt template. These generated pairs imitated the style and
meaning of the existing pairs, and we assigned them scores corresponding to
the originals. We used the same datasets provided by the Shared Task in
addition to a ~760 augmented Moroccan pairs to fine-tune BERT models,
AraBERTv2 (Antoun et al., 2020) and ArBERTv2 (Abdul-Mageed et al., 2021),
which resulted in a performance enhancement of 0.05 points. In track B, as
training on labeled data is not allowed, we used cosine similarity using
average pooling embedding (Zhao et al., 2022) on top of each model. Our
approaches achieved Spearman scores (Tsatsaronis et al., 2014) of 0.49 for MSA
(ranked first), 0.83 for Moroccan (ranked fifth), and 0.53 for Algerian
(ranked twelfth).

##  2 Related Work

Semantic textual relatedness (STR) has proven to be a valuable task in
numerous NLP applications, including the evaluation of LLMs (Asaadi et al.,
2019; Naseem et al., 2021). Determining the degree of relatedness in STR,
however, remains a challenging task in computational semantics. That is
because STR encompasses a broader range of commonalities beyond just meaning,
including shared viewpoint, topic, and period, demanding a deeper
understanding than semantic similarity alone (Asaadi et al., 2019; Abdalla et
al., 2023). For example, consider reading these two sentences (He heard the
waves crashing gently) and (Making him feel calm and peaceful). While humans
easily recognize their strong relatedness and shared description of the same
view (a beach scene), machines require advanced lexical and statistical
methods to achieve the same level of understanding. STR techniques mainly come
from four approaches: lexical similarity (Chen et al., 2018; Jarrar and
Amayreh, 2019; Alhafi et al., 2019), semantic similarity (Hasan et al., 2020;
Ghanem et al., 2023), deep learning (Zhang and Moldovan, 2019), and LLMs Li et
al. (2021).

Recently, Abdalla et al. (2023) introduced their STR-2022 dataset, which uses
fine-grained scores ranging from 0 (least related) to 1 (completely related).
Their dataset consists of 5,500 scored English sentence pairs. They framed the
task as supervised regression, where they fine-tuned two language models,
BERT-base (Kenton and Toutanova, 2019) and RoBERTa-base (Liu et al., 2019),
and applied average pooling on top of the final embedding layer. Their testing
of these models on the STR-2022 dataset yielded an average Spearman
correlation of 0.82 for BERT-base and 0.83 for RoBERTa-base. On the other
hand, their unsupervised experiments using Word2Vec (Mikolov et al., 2013)
achieved a correlation score of 0.60, outperforming both BERT-base (0.58) and
RoBERTa-base (0.48) by 0.02 and 0.12 points, respectively.

Asaadi et al. (2019) created the Bi-gram Semantic Relatedness Dataset (BiRD)
for examining semantic composition. To avoid inconsistencies and biases from
traditional 1-5 rating scales, they employed fine-grained scoring of bi-gram
pairs (0-1) using the best-worst scaling (BWS) annotation technique
(Kiritchenko and Mohammad, 2017). The dataset consists of 3,345 scored English
term pairs. They utilised three models to generate word representations: GloVe
(Pennington et al., 2014), FastText (Grave et al., 2018), and a word-context
co-occurrence matrix (Turney et al., 2011). To calculate relatedness scores
between pairs, they employed cosine similarity between the generated addition-
pooled vectors. The FastText model achieved the highest performance with a
Pearson correlation of 0.60.

The semantic relatedness between noun-pairs was studied using contextual
similarity by Miller and Charles (1991). They attempted to understand
distinctions between nouns in contextual discourse and how the similarity can
be broader than just the meaning. Additional ideas could rely on extracting
named entities (Liqreina et al., 2023; Jarrar et al., 2022) to measure the
relatedness Ghosh et al. (2023). However, the task evolved, leading to the
creation of the up-to-date dataset presented by the SemRel-2024 shared task
(Ousidhoum et al., 2024b). Their dataset annotation scores are at the level of
sentence pairs. They shared baseline results for fourteen languages and
dialects using Spearman correlation score. Since our focus is on Arabic, we
have chosen its results to show. For example, their baseline is 0.42 for MSA
in track B using multi-lingual BERT (mBERT) (Kenton and Toutanova, 2019), 0.60
for Algerian and 0.77 for Moroccan in track A using Label Agnostic BERT
Sentence embeddings (LaBSE) (Feng et al., 2022). Specifically, their Algerian
Arabic dataset offers 1,261 training and 583 test instances, Moroccan Arabic
dataset includes 924 training and 425 test instances, and MSA Arabic dataset
has 595 instances for testing.

Many efforts have been made to understand Arabic dialects, such as dialect
identification, intent detection, and morphological annotations Haff et al.
(2022); Nayouf et al. (2023); Jarrar et al. (2023c, 2017, a), but none studied
STR between dialects.

Figure 1: BERT-based Supervised Architecture (A).

##  3 System Overview

This section presents the techniques, datasets, and the augmentation we
employed in tracks A and B.

###  3.1 Supervised Track (A)

Since the datasets use continuous scoring values, we tackled STR as a
regression problem. We fine-tuned BERT with Mean Squared Error (MSE)
objective. The model uses a regressor output layer, represented by a single
neuron to predict the scores of the sentence-pairs. The data was pre-processed
using the technique presented in (Antoun et al., 2020) to achieve standardized
word forms. Before supplying the sentence pairs to the model, each was
concatenated using the special tokens of the model input in this
format:[CLS]Sentence1[SEP]Sentence2[SEP]. Figure 1 depicts our method
architecture for the supervised track (A). Since we focused on the Algerian
and Moroccan dialects in this track, we investigated various model parameters
including learning rates, number of epochs, and pre-trained models to
understand which model is better suited for each dialect. We found that both
models, AraBERTv2 222https://github.com/aub-mind/arabert and ArBERTv2
333https://huggingface.co/UBC-NLP/ARBERTv2, best fits the Moroccan dialect
more than Algerian. Nonetheless, we used same models for the Algerian dataset.

Figure 2: BERT-based Unsupervised Architecture (B). Figure 3: Example of the
augmented sentence-pairs.

###  3.2 Unsupervised Track (B)

The STR using MSA is covered in track B (unsupervised learning), where
training (or fine-tuning) on labeled data is not permitted. We employed cosine
similarity (Reimers and Gurevych, 2019) as an unsupervised technique to
calculate the sentence-pair scores. Figure 2 illustrates our architecture. We
conducted initial experiments using the same aforementioned models, ArBERTv2
and AraBERTv2, for generating sentence representations. Various pooling
options (CLS, average, max, and min) (Zhao et al., 2022) were applied on the
final embedding layer in each (frozen) model, and found that AraBERTv2 with
average-pooling is better suited for MSA in this track. The same data pre-
processing used in track A is applied in B.

###  3.3 Datasets

The datasets provided by the SemRel-2024 shared task cover fourteen languages
and dialects. In the paper, we used three Arabic datasets (Algerian, Moroccan,
and MSA). Table 1 presents their data splits, including train, development,
and testing. MSA has no labeled train data as it is included in Track B.
However, for the other two dialects, we employed BERT-based models, that
requires large train data (Bevilacqua et al., 2021).

| MSA | Algerian | Moroccan
---|---|---|---
Train |  |  |
Original | – | 1,261 | 924
Augments | – | – | 757
Total | – | 1,261 | 1,681
Dev. | 32 | 97 | 70
Test | 595 | 583 | 425
Table 1:  The original and augmented datasets splits.

Different methods can be used for data augmentation, such as back-translation
Lin and Giambi (2021) and generative models (Saidi et al., 2022). The back-
translation technique was tested by (Malaysha et al., 2023) and showed minor
improvement in performance. The availability of high-quality generative
models, such as ChatGPT 444https://chat.openai.com/ and Google Gemini,
encouraged us to employ them in automatic augmentation. We employed in-context
learning (Min et al., 2022) by prompting both models with the request depicted
in Figure 4.

Figure 4: The prompt template employed for Gemini.

The initial manual reviews and tests for twenty prompts of Moroccan and
Algerian sentences showed that both models are weak in Algerian comprehension.
ChatGPT is also weak in the Moroccan, while Gemini demonstrated a high
understanding of the Moroccan. Therefore, we decided to employ Gemini to
augment the Moroccan train split. From every sentence-pair, we took each
sentence and prompted it using the template in Figure 4. We mapped the
augmented (new) sentence from the model with the other sentence in the same
pair using the same score of the pair, as illustrated in Figure 3. By manually
reviewing all the model replies, we found cases that were not valid (wrong
content), and accordingly, we defined filters to exclude the not applicable
data per the following rules:

  * •

The model admits in the reply that it is just a language model and cannot
fulfill the request. The model reply in such case has common format to rely on
for the filter comparison.

  * •

The case when the reply goes far from the original meaning. This option is
achieved by manually reviewing the paraphrased contents.

  * •

When the model rejects augmentation because the requested sentence contains
information that breaks the model policy, i.e., talking about public figures
or sensitive discussions. Similar to first rule, it has common reply format to
automatically compare with.

Finally, after filtering the invalid augmentations, we reached 757 accepted
sentences which we added to the Moroccan training set (See Table 1), reaching
a total of 1,681 instances.

Development Phase | Track A | Track B
---|---|---
| Algerian | Moroccan | Augmented Moroccan | MSA
ArBERTv2 | 0.55 | 0.82 | 0.88$\uparrow$ | 0.42
AraBERTv2 | 0.69 | 0.84 | 0.79$\downarrow$ | 0.58
Table 2:  Our results on the development phase (i.e., on development split).  TEST Phase | Track A | Track B
---|---|---
| Algerian | Moroccan | Augmented Moroccan | MSA
Baseline (Ousidhoum et al., 2024a) | 0.60 | 0.77 | 0.77 | 0.42
ArBERTv2 | 0.42$\downarrow$ | 0.78$\uparrow$ | 0.83$\uparrow$ | 0.34$\downarrow$
AraBERTv2 | 0.53$\downarrow$ | 0.79$\uparrow$ | 0.77$\uparrow$ | 0.49$\uparrow$
Table 3:  The evaluation results on the test data. Our official ranked scores
are in bold.

##  4 Experimental Setup

Our experiments fine-tuned two language models for Algerian and Moroccan,
where we used the following pre-trained models: maubmindlab/bert-base-
arabertv02 (Antoun et al., 2020) and UBC-NLP/ARBERTv2 (Abdul-Mageed et al.,
2021). We employed the training data provided by the shared task, in addition
to the data generated by our augmentation technique, when applied. The
development data is excluded from either training or testing in the official
evaluation phase, and testing is done on the shared task test set (See Table
1). The data pairs were concatenated using special tokens ([CLS] and [SEP]),
as depicted in Figure 1, and digested by the models. The fine-tuning was done
as a regression task using one neuron in the output layer, optimized using MSE
as the loss function, and we used R-squared (Miles, 2005) to measure the
improvement. The final hyper-parameters in the fine-tuning process were: 10
epochs for training, 4 epochs for early stopping, a batch size of 16, 512 is
the maximum sequence length, a learning rate of $2e^{-5}$, 50 evaluation
steps, a seed of 42, and train ($\pm$ augmented data) split.

In the experiments of B track for the MSA, no supervised fine-tuning is
needed. Therefore, we neither used labeled data nor augmentation. We employed
average-pooling on the embeddings of the sentence tokens from the final layer
in each model. Then, we calculated the cosine similarity between the average
embeddings of the sentences in each pair. This was done to estimate the fine-
grained scores for the test (or development) data provided by the shared task.
The shared task considers Spearman correlation score to evaluate the submitted
predictions against their ground truth.

##  5 Results

Our approaches have achieved competitive ranks in the SemRel-2024 shared task.
The official results of the tracks we participated in, as well as the
baselines that were introduced by Ousidhoum et al. (2024a), are shown in Table
3. Additionally, our results on the development data are presented in Table 2.
In the test evaluation, we ranked first in Track B for the MSA, with a
Spearman correlation score of 0.49 using the AraBERTv2 model, outperforming
the baseline by 0.07 points. However, ArBERTv2 did not perform well in Track B
for MSA on both test and development splits. In contrast, ArBERTv2 achieved a
high score in Track A for the Moroccan dialect when fine-tuned on both the
train split and augmentation data, outperforming the baseline by 0.06 points
on test split, ranking 5th among the submitted systems. Nonetheless, neither
of the models, ArBERTv2 or AraBERTv2, surpassed the baseline for the Algerian
dialect in Track A, where our rank is 12. Similarly, both models achieved low
performance on the Algerian development split. It is possible that if we were
able to augment the Algerian data as well, it could have performed better,
similar to the improvement achieved in the Moroccan dataset. It is worth
noting that AraBERTv2 outperformed both the baseline and ArBERTv2 on the
original training data of the Moroccan dataset. However, its performance
degraded on both test and development splits once the augmentation was
included in the fine-tuning, unlike what happened with the ArBERTv2 model, on
both splits. This could be due to the nature of the data utilized in the pre-
training phase of the model. Due to the anisotropy problem (Baggetto and
Fresno, 2022) inherent in BERT-based pre-trained models, we noted that
computing cosine similarity directly between sentence representations is
insufficient for discerning relatedness.

##  6 Conclusion

We presented our contributions to the SemRel-2024 shared task. We targeted
three Arabic dialects covered by the shared task datasets, including MSA,
Algerian, and Moroccan. Our approaches employed supervised and unsupervised
techniques using commonly known language models, namely ArBERT and AraBERT. We
augmented the training data using generative models, which enhanced the
models’ performance. Our system ranked first (MSA), fifth (Moroccan), and
twelfth (Algerian) across the different tracks. We plan to augment additional
data of Moroccan and Algerian using other models than what we used in this
work. We will use the augmentations to experiment with both Arabic mono-
dialect and cross-dialect fine-tuning.

## References

  * Abdalla et al. (2023) Mohamed Abdalla, Krishnapriya Vishnubhotla, and Saif M. Mohammad. 2023.  What makes sentences semantically related? A textual relatedness dataset and empirical study.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2023, Dubrovnik, Croatia, May 2-6, 2023_ , pages 782–796. Association for Computational Linguistics.
  * Abdul-Mageed et al. (2021) Muhammad Abdul-Mageed, AbdelRahim A. Elmadany, and El Moatez Billah Nagoudi. 2021.  ARBERT & MARBERT: deep bidirectional transformers for arabic.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021_ , pages 7088–7105. Association for Computational Linguistics.
  * Al-Hajj and Jarrar (2021a) Moustafa Al-Hajj and Mustafa Jarrar. 2021a.  Arabglossbert: Fine-tuning bert on context-gloss pairs for wsd. In _Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021)_ , pages 40–48, Online. INCOMA Ltd.
  * Al-Hajj and Jarrar (2021b) Moustafa Al-Hajj and Mustafa Jarrar. 2021b.  Lu-bzu at semeval-2021 task 2: Word2vec and lemma2vec performance in arabic word-in-context disambiguation. In _Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021)_ , pages 748–755, Online. Association for Computational Linguistics.
  * Alhafi et al. (2019) Diana Alhafi, Anton Deik, and Mustafa Jarrar. 2019.  Usability evaluation of lexicographic e-services.  In _The 2019 IEEE/ACS 16th International Conference on Computer Systems and Applications (AICCSA)_ , pages 1–7. IEE.
  * Antoun et al. (2020) Wissam Antoun, Fady Baly, and Hazem Hajj. 2020.  Arabert: Transformer-based model for arabic language understanding.  In _Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection_ , pages 9–15.
  * Asaadi et al. (2019) Shima Asaadi, Saif M. Mohammad, and Svetlana Kiritchenko. 2019.  Big bird: A large, fine-grained, bigram relatedness dataset for examining semantic composition.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , pages 505–516. Association for Computational Linguistics.
  * Baggetto and Fresno (2022) Alejandro Fuster Baggetto and Víctor Fresno. 2022.  Is anisotropy really the cause of BERT embeddings not being semantic? In _Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022_ , pages 4271–4281. Association for Computational Linguistics.
  * Bentivogli et al. (2016) Luisa Bentivogli, Raffaella Bernardi, Marco Marelli, Stefano Menini, Marco Baroni, and Roberto Zamparelli. 2016.  SICK through the semeval glasses. lesson learned from the evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment.  _Lang. Resour. Evaluation_ , 50(1):95–124.
  * Bevilacqua et al. (2021) Michele Bevilacqua, Tommaso Pasini, Alessandro Raganato, Roberto Navigli, et al. 2021.  Recent trends in word sense disambiguation: A survey.  In _Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21_ , pages 4330–4338.
  * Chen et al. (2018) Zugang Chen, Jia Song, and Yaping Yang. 2018.  An approach to measuring semantic relatedness of geographic terminologies using a thesaurus and lexical database sources.  _ISPRS Int. J. Geo Inf._ , 7(3):98.
  * Darwish et al. (2021) Kareem Darwish, Nizar Habash, Mourad Abbas, Hend Al-Khalifa, Huseein T. Al-Natsheh, Houda Bouamor, Karim Bouzoubaa, Violetta Cavalli-Sforza, Samhaa R. El-Beltagy, Wassim El-Hajj, Mustafa Jarrar, and Hamdy Mubarak. 2021.  A panoramic survey of natural language processing in the arab worlds.  _Commun. ACM_ , 64(4):72–81.
  * Feng et al. (2022) Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2022.  Language-agnostic BERT sentence embedding.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022_ , pages 878–891. Association for Computational Linguistics.
  * Ghanem et al. (2023) Sana Ghanem, Mustafa Jarrar, Radi Jarrar, and Ibrahim Bounhas. 2023.  A benchmark and scoring algorithm for enriching arabic synonyms.  In _Proceedings of the 12th International Global Wordnet Conference (GWC2023)_ , pages 215–222. Global Wordnet Association.
  * Ghosh et al. (2023) Mirna El Ghosh, Nicolas Delestre, Jean-Philippe Kotowicz, Cecilia Zanni-Merk, and Habib Abdulrab. 2023.  Reltopic: A graph-based semantic relatedness measure in topic ontologies and its applicability for topic labeling of old press articles.  _Semantic Web_ , 14(2):293–321.
  * Grave et al. (2018) Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomás Mikolov. 2018.  Learning word vectors for 157 languages.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation, LREC 2018, Miyazaki, Japan, May 7-12, 2018_. European Language Resources Association (ELRA).
  * Haff et al. (2022) Karim El Haff, Mustafa Jarrar, Tymaa Hammouda, and Fadi Zaraket. 2022.  Curras + baladi: Towards a levantine corpus.  In _Proceedings of the International Conference on Language Resources and Evaluation (LREC 2022)_ , Marseille, France.
  * Halliday and Hasan (2014) Michael Alexander Kirkwood Halliday and Ruqaiya Hasan. 2014.  _Cohesion in english_.  9\. Routledge.
  * Hasan et al. (2020) Ali Muttaleb Hasan, Noorhuzaimi Mohd Noor, Taha H Rassem, and Ahmed Muttaleb Hasan. 2020.  Knowledge-based semantic relatedness measure using semantic features.  _International Journal_ , 9(2).
  * Jarrar (2011) Mustafa Jarrar. 2011.  Building a formal arabic ontology (invited paper).  In _Proceedings of the Experts Meeting on Arabic Ontologies and Semantic Networks_. ALECSO, Arab League.
  * Jarrar (2021) Mustafa Jarrar. 2021.  The arabic ontology - an arabic wordnet with ontologically clean content.  _Applied Ontology Journal_ , 16(1):1–26.
  * Jarrar and Amayreh (2019) Mustafa Jarrar and Hamzeh Amayreh. 2019.  An arabic-multilingual database with a lexicographic search engine.  In _The 24th International Conference on Applications of Natural Language to Information Systems (NLDB 2019)_ , volume 11608 of _LNCS_ , pages 234–246. Springer.
  * Jarrar et al. (2023a) Mustafa Jarrar, Ahmet Birim, Mohammed Khalilia, Mustafa Erden, and Sana Ghanem. 2023a.  Arbanking77: Intent detection neural model and a new dataset in modern and dialectical arabic.  In _Proceedings of the 1st Arabic Natural Language Processing Conference (ArabicNLP), Part of the EMNLP 2023_ , pages 276–287. ACL.
  * Jarrar et al. (2017) Mustafa Jarrar, Nizar Habash, Faeq Alrimawi, Diyam Akra, and Nasser Zalmout. 2017.  Curras: An annotated corpus for the palestinian arabic dialect.  _Journal Language Resources and Evaluation_ , 51(3):745–775.
  * Jarrar et al. (2022) Mustafa Jarrar, Mohammed Khalilia, and Sana Ghanem. 2022.  Wojood: Nested arabic named entity corpus and recognition using bert.  In _Proceedings of the International Conference on Language Resources and Evaluation (LREC 2022)_ , Marseille, France.
  * Jarrar et al. (2023b) Mustafa Jarrar, Sanad Malaysha, Tymaa Hammouda, and Mohammad Khalilia. 2023b.  Salma: Arabic sense-annotated corpus and wsd benchmarks.  In _Proceedings of the 1st Arabic Natural Language Processing Conference (ArabicNLP), Part of the EMNLP 2023_ , pages 359–369. ACL.
  * Jarrar et al. (2023c) Mustafa Jarrar, Fadi Zaraket, Tymaa Hammouda, Daanish Masood Alavi, and Martin Waahlisch. 2023c.  Lisan: Yemeni, irqi, libyan, and sudanese arabic dialect copora with morphological annotations.  In _The 20th IEEE/ACS International Conference on Computer Systems and Applications (AICCSA)_. IEEE.
  * Kenton and Toutanova (2019) Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of NAACL-HLT_ , pages 4171–4186.
  * Kiritchenko and Mohammad (2017) Svetlana Kiritchenko and Saif M. Mohammad. 2017.  Best-worst scaling more reliable than rating scales: A case study on sentiment intensity annotation.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 2: Short Papers_ , pages 465–470. Association for Computational Linguistics.
  * Li et al. (2021) Xiaotao Li, Shujuan You, and Wai Chen. 2021.  Enhancing accuracy of semantic relatedness measurement by word single-meaning embeddings.  _IEEE Access_ , 9:117424–117433.
  * Lin and Giambi (2021) Guan-Ting Lin and Manuel Giambi. 2021.  Context-gloss augmentation for improving word sense disambiguation.  _arXiv preprint arXiv:2110.07174_ , abs/2110.07174.
  * Liqreina et al. (2023) Haneen Liqreina, Mustafa Jarrar, Mohammed Khalilia, Ahmed Oumar El-Shangiti, and Muhammad Abdul-Mageed. 2023.  Arabic fine-grained entity recognition.  In _Proceedings of the 1st Arabic Natural Language Processing Conference (ArabicNLP), Part of the EMNLP 2023_ , pages 310–323. ACL.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized BERT pretraining approach.  _CoRR_ , abs/1907.11692.
  * Malaysha et al. (2023) Sanad Malaysha, Mustafa Jarrar, and Mohammed Khalilia. 2023.  Context-gloss augmentation for improving arabic target sense verification.  In _Proceedings of the 12th Global Wordnet Conference, GWC 2023, University of the Basque Country, Donostia - San Sebastian, Basque Country, Spain, 23 - 27 January 2023_ , pages 254–262. Global Wordnet Association.
  * Mi and Xie (2024) Chenggang Mi and Shaoliang Xie. 2024.  Language relatedness evaluation for multilingual neural machine translation.  _Neurocomputing_ , 570:127115.
  * Mikolov et al. (2013) Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013.  Efficient estimation of word representations in vector space.  In _1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings_.
  * Miles (2005) Jeremy Miles. 2005.  R-squared, adjusted r-squared.  _Encyclopedia of statistics in behavioral science_.
  * Miller and Charles (1991) George A Miller and Walter G Charles. 1991.  Contextual correlates of semantic similarity.  _Language and cognitive processes_ , 6(1):1–28.
  * Min et al. (2022) Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022.  Rethinking the role of demonstrations: What makes in-context learning work? In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022_ , pages 11048–11064. Association for Computational Linguistics.
  * Naseem et al. (2021) Usman Naseem, Imran Razzak, Shah Khalid Khan, and Mukesh Prasad. 2021.  A comprehensive survey on word representation models: From classical to state-of-the-art word representation language models.  _ACM Trans. Asian Low Resour. Lang. Inf. Process._ , 20(5):74:1–74:35.
  * Nayouf et al. (2023) Amal Nayouf, Mustafa Jarrar, Fadi zaraket, Tymaa Hammouda, and Mohamad-Bassam Kurdy. 2023.  Nâbra: Syrian arabic dialects with morphological annotations.  In _Proceedings of the 1st Arabic Natural Language Processing Conference (ArabicNLP), Part of the EMNLP 2023_ , pages 12–23. ACL.
  * Ousidhoum et al. (2024a) Nedjma Ousidhoum, Shamsuddeen Hassan Muhammad, Mohamed Abdalla, Idris Abdulmumin, Ibrahim Said Ahmad, Sanchit Ahuja, Alham Fikri Aji, Vladimir Araujo, Abinew Ali Ayele, Pavan Baswani, Meriem Beloucif, Chris Biemann, Sofia Bourhim, Christine De Kock, Genet Shanko Dekebo, Oumaima Hourrane, Gopichand Kanumolu, Lokesh Madasu, Samuel Rutunda, Manish Shrivastava, Thamar Solorio, Nirmal Surange, Hailegnaw Getaneh Tilaye, Krishnapriya Vishnubhotla, Genta Winata, Seid Muhie Yimam, and Saif M. Mohammad. 2024a.  Semrel2024: A collection of semantic textual relatedness datasets for 14 languages.
  * Ousidhoum et al. (2024b) Nedjma Ousidhoum, Shamsuddeen Hassan Muhammad, Mohamed Abdalla, Idris Abdulmumin, Ibrahim Said Ahmad, Sanchit Ahuja, Alham Fikri Aji, Vladimir Araujo, Meriem Beloucif, Christine De Kock, Oumaima Hourrane, Manish Shrivastava, Thamar Solorio, Nirmal Surange, Krishnapriya Vishnubhotla, Seid Muhie Yimam, and Saif M. Mohammad. 2024b.  SemEval-2024 task 1: Semantic textual relatedness.  In _Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024)_.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014.  Glove: Global vectors for word representation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL_ , pages 1532–1543. ACL.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_ , pages 3980–3990. Association for Computational Linguistics.
  * Sabir et al. (2019) Ahmed Sabir, Francesc Moreno, and Lluís Padró. 2019.  Semantic relatedness based re-ranker for text spotting.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_ , pages 3449–3455. Association for Computational Linguistics.
  * Saidi et al. (2022) Rakia Saidi, Fethi Jarray, Jeongwoo Jay Kang, and Didier Schwab. 2022.  GPT-2 contextual data augmentation for word sense disambiguation.  In _Proceedings of the 36th Pacific Asia Conference on Language, Information and Computation, PACLIC 2022, Manila, Philippines, October 20-22, 2022_ , pages 455–462. De La Salle University.
  * Tsatsaronis et al. (2014) George Tsatsaronis, Iraklis Varlamis, and Michalis Vazirgiannis. 2014.  Text relatedness based on a word thesaurus.  _CoRR_ , abs/1401.5699.
  * Turney et al. (2011) Peter D. Turney, Yair Neuman, Dan Assaf, and Yohai Cohen. 2011.  Literal and metaphorical sense identification through concrete and abstract context.  In _Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, EMNLP 2011, 27-31 July 2011, John McIntyre Conference Centre, Edinburgh, UK, A meeting of SIGDAT, a Special Interest Group of the ACL_ , pages 680–690. ACL.
  * Zhang and Moldovan (2019) Linrui Zhang and Dan Moldovan. 2019.  Multi-task learning for semantic relatedness and textual entailment.  _Journal of Software Engineering and Applications_ , 12(6):199–214.
  * Zhao et al. (2022) Shuai Zhao, Tianyu Zhang, Man Hu, Wen Chang, and Fucheng You. 2022.  AP-BERT: enhanced pre-trained model through average pooling.  _Appl. Intell._ , 52(14):15929–15937.