# From Actions to Events: A Transfer Learning Approach Using Improved Deep
Belief Networks

Mateus Roder1, Jurandy Almeida2, Gustavo H. de Rosa1, Leandro A. Passos1,
André L. D. Rossi1, João P. Papa1  1Department of Computing, São Paulo State
University – UNESP, Bauru, Brazil
{mateus.roder, gustavo.rosa, leandro.passos, andre.rossi<EMAIL_ADDRESS>2Instituto de Ciência e Tecnologia, Universidade Federal de São Paulo –
UNIFESP, São José dos Campos, Brazil
<EMAIL_ADDRESS>
###### Abstract

In the last decade, exponential data growth supplied machine learning-based
algorithms’ capacity and enabled their usage in daily-life activities.
Additionally, such an improvement is partially explained due to the advent of
deep learning techniques, i.e., stacks of simple architectures that end up in
more complex models. Although both factors produce outstanding results, they
also pose drawbacks regarding the learning process as training complex models
over large datasets are expensive and time-consuming. Such a problem is even
more evident when dealing with video analysis. Some works have considered
transfer learning or domain adaptation, i.e., approaches that map the
knowledge from one domain to another, to ease the training burden, yet most of
them operate over individual or small blocks of frames. This paper proposes a
novel approach to map the knowledge from action recognition to event
recognition using an energy-based model, denoted as Spectral Deep Belief
Network. Such a model can process all frames simultaneously, carrying spatial
and temporal information through the learning process. The experimental
results conducted over two public video dataset, the HMDB-51 and the UCF-101,
depict the effectiveness of the proposed model and its reduced computational
burden when compared to traditional energy-based models, such as Restricted
Boltzmann Machines and Deep Belief Networks.

##  I Introduction

Machine Learning (ML) techniques emerged in the last decades as revolutionary
tools capable of solving or slightly alleviating the burden imposed by
repetitive and tedious tasks. Recently, the advent of Deep Learning (DL)
algorithms powered up such advances, providing astonishing predictions over
complex domains. On the other hand, video-based domain tasks still pose
challenging assignments to intelligent algorithms, e.g., recognizing human
actions in videos [1].

An action recognition task is characterized by an observation of a complete
sequence of movements performed by a human followed by its classification [2].
Such a task plays a fundamental role in video surveillance-based security
systems, content-based video retrieval, and self-driving cars, among others.
Event recognition is a similar task which models the ability to retrieve
specific actions from sequences of videos, focusing on learning behaviors
related to events of interest, i.e., events which comprise specific activities
and objects in a given scene. Thus, there is a slight difference between
action and event recognition tasks, i.e., while the former attempts to
identify any action performed in the scene, the latter remarks specific
movements, such as “is there anybody drinking in the street?” or “does this
video contains any unusual behavior?”. Event recognition approaches are
commonly employed for monitoring public or private areas in search for
anomalous behaviors, such as violent assaults on sports events, abandoned
objects on train stations, and route obstruction on industrial environments,
among others.

Despite their similarities, both action and event recognition tasks present
their particularities, posing distinct challenges while training an
intelligent model. An interesting approach used to solve event recognition
tasks is importing some correlated knowledge extracted over action
recognition-based models, denoted as transfer learning.

Transfer learning studies the possibilities of transferring knowledge from
source domains to different contexts (target domains). To illustrate such an
idea, consider an autonomous-driving car trained with road-traffic data from
New Zealand. It will not work effectively on Brazilian streets due to
different signage and road-traffic rules, a distinct influx of vehicles, and
right-lane-based driving, among other issues. However, adapting the knowledge
learned in New Zealand to Brazil can reduce computational costs and time
needed to train new models [3]. In a nutshell, the transfer learning considers
two or more feature space distributions been related or equal, where the task
related to each one can be also related or equal, providing useful information
for the target task [4].

Several works addressed the problem of action and event recognition through
transfer learning strategies. Farajidavar et al. [5], for instance, proposed a
transductive transfer learning method for action recognition in tennis games.
Further, Shao et al. [6] published a survey comprising several approaches for
these tasks, such as space approximation [7], Gaussian mixture model [8], and
geometric reasoning [9]. Recently, novel approaches considered DL methods,
such as Wang et al. [10], who introduced Generative Adversarial Networks for
action recognition using partial-modalities. Tas et al. [11] employed a
Convolutional Neural Network (CNN) for action recognition and supervised
domain adaptation on 3D body skeletons. Furthermore, Gao et al. [12]
introduced a two-stream graph CNN for zero-shot action recognition, while Liu
et al. [13] proposed image-to-video adaptation and fusion networks in the same
context.

Among several DL algorithms, an energy-based model known as Deep Belief
Network (DBN) [14] obtained considerable popularity in the last years due to
its notable results in a wide variety of applications [15, 16]. It is composed
of multiple hidden layers, such that each layer is a greedily trained
Restricted Boltzmann Machine (RBMs) [17]. Even though some works proposed DBNs
for action [18, 19] and a specific type of event recognition [20], as far as
we know, there is still no work that has successfully addressed the concepts
of transfer learning from actions to events with video through DBNs, i.e., to
learn useful features from highly structured actions and movements to the
generalization of high-level events. Therefore, the main contributions of this
work are threefold: (i) to introduce DBNs in the video event classification
domain, (ii) to propose two approaches, denoted as Aggregative-DBN and
Gradient-DBN, that employ frame fusion and image gradient respectively, and
(iii) to support the lack of video-based event recognition in literature.

Additionally, we consider the following hypotheses: (i) DBNs are able to learn
useful correlations that map actions to high-level events in video-based
domain tasks; (ii) the overall accuracy can be improved using the proposed
approaches; and (iii) the overall training time can be reduced with the
Aggregative-based approach.

The remainder of this paper is organized as follows. Section II introduces the
main theoretical concepts used in the manuscript, while Section III presents
the proposed approaches. Further, Sections IV and V describe the methodology
adopted in this work and the experimental results. Finally, Section VI states
conclusions and future work.

##  II Theoretical Background

###  II-A Video-based Domain

Let $\mathcal{F}=\\{F_{1},F_{2},\dots,F_{n}\\}$ be a temporal sequence of
frames (images) $F_{i}$, which represents a soundless scene and possible
movements of its components. Such frames can be classified according to the
complexity of their internal representations and the interaction level between
their entities. Thus, it is possible to generate four classification-based
categories: attributes and movements, low-level events and actions,
interaction, and high-level events [21].

Figure 1 illustrates the hierarchical form of the categories discussed
hereafter. The movement characterizes the lowest representation level of a
frame and is widely employed to recognize human actions, such as body movement
[22]. On the other hand, low-level events and actions represent a particular
chain of movements, usually carried out by an entity, e.g., car or person.
Additionally, if such actions are performed or interacted by more than one
entity or object, it is possible to categorize them as an interaction [21].
Finally, the highest-level category, denoted as complex events, represents the
interaction of entities or sequence of actions in a specific time window in
the video. For instance, one can identify a birthday party as an event
composed of several actions and entities in a single scene.

Therefore, the event recognition task attempts to detect the complex events’
spatial and temporal locations in a sequence of frames [21]. The literature
lacks in standardizing the difference between actions and events, making them
interchangeable in most applications [21].

Figure 1: Video-based domain hierarchical complexity.

###  II-B Applied Transfer Learning

Transfer learning has recently received attention due to the advent of the
ImageNet111https://image-net.org/ dataset and the increased processing power
of GPUs. Such a task consists in transferring the knowledge from the source
domain to the target domain, providing information that can be useful on the
target domain, mainly when data are insufficient or the computational
resources are scarce. In this way, it is possible to train deep neural
networks in large-scale image/video datasets and use them to fine-tune more
specific tasks [23].

Given the previous concepts, it is possible to elucidate the mathematical
formulation regarding the problem addressed here. Let $\Gamma$ be a high-level
event recognition task, as well as let $\mathcal{D}_{S}$ and $\mathcal{D}_{T}$
be the source (action domain) and the target space domains (event domain),
respectively. Additionally, the source domain is composed of the subspaces
$\mathcal{A}\in\mathbb{R}^{d_{a}}$, $\mathcal{M}\in\mathbb{R}^{d_{m}}$, and
$\mathcal{I}\in\mathbb{R}^{d_{i}}$, where
$\\{\mathcal{A},\mathcal{M},\mathcal{I}\\}\subset\mathcal{D}_{S}$, while the
target domain is composed of $\mathcal{E}\in\mathbb{R}^{d_{e}}$, where
$\\{\mathcal{E}\\}\subset\mathcal{D}_{T}$. The subspace $\mathcal{A}$ stands
for the $d_{a}$-dimensional base actions, $\mathcal{M}$ stands for the
$d_{m}$-dimensional movements, $\mathcal{I}$ represents the interactions
between $d_{i}$-dimensional entities, and $\mathcal{E}$ stands for the
$d_{e}$-dimensional high-level events.

From the transfer learning theory [24], a specific case is when the data from
the source domain and target domain keep their probabilities, letting
$\mathcal{D}_{S}$ to be identical to $\mathcal{D}_{T}$, differing only on the
target task, which is the problem addressed in this paper. Finally, it is
possible to formulate the proposed approach for the given task using Equation
1, as follows:

$\Gamma=\\{y_{\mathcal{D}_{T}},f(\mathcal{D}_{S})\\},$ (1)

where $y_{\mathcal{D}_{T}}$ stands for the target domain labels and
$f(\mathcal{D}_{S})$ for the function that learns features from the source
domain. Such learned features are useful for the target domain and its
respective event classification task, i.e., a neural network that learns from
$\mathcal{D}_{S}$ and is fine-tuned in $\mathcal{D}_{T}$ with labels
$y_{\mathcal{D}_{T}}$. Here, $\mathcal{D}_{T}$ has the same probability
distribution that $\mathcal{D}_{S}$, as aforementioned.

###  II-C Restricted Boltzmann Machines

Restricted Boltzmann Machines (RBMs) [25, 17] are described as a bipartite
graph composed of two layers of neurons, i.e., a visible layer
$\textbf{v}\in\\{0,1\\}^{m}$, which is responsible for the input data, and a
hidden layer $\textbf{h}\in\\{0,1\\}^{n}$, whose units map the data
representation into a latent space. This interaction is modeled by a weight
matrix, $\textbf{W}\in\Re^{m\times n}$, which connects each visible unit
$v_{i}$ to all hidden units $h_{j}$, and vice-versa, denoted by the arc
$w_{ij}$.

The RBM learning procedure is performed by the minimization of an energy
function concerning some intrinsic variables, described as follows:

$E(\textbf{v},\textbf{h})=-\sum_{i=1}^{m}b_{i}v_{i}-\sum_{j=1}^{n}c_{j}h_{j}-\sum_{i=1}^{m}\sum_{j=1}^{n}v_{i}h_{j}w_{ij},$
(2)

where $\textbf{b}\in\Re^{m}$ and $\textbf{c}\in\Re^{n}$ stand for the bias
vector considering the visible and hidden layers, respectively.

Computing the joint probability of the system poses an intractable task due to
the increasing number of possible states. However, since the model is
represented as a bipartite graph, one can compute both the visible and hidden
units’ activation in a mutually independent fashion, performed as follows:

$P(v_{i}=1|\textbf{h})=\phi\left(\sum_{j=1}^{n}w_{ij}h_{j}+b_{i}\right),$ (3)

and

$P(h_{j}=1|\textbf{v})=\phi\left(\sum_{i=1}^{m}w_{ij}v_{i}+c_{j}\right).$ (4)

Note that $\phi(\cdot)$ stands for the logistic-sigmoid function. Finally, we
can solve the equations above by iteratively sampling over a Markov Chain,
using the well-known Contrastive Divergence (CD) algorithm [17].

The learning process in RBMs consists of an optimization problem whose goal is
to minimize the energy function given in Equation 2. In other words, such a
process ends up maximizing the marginal probability distribution of the
visible units, defined as follows:

$P(\textbf{v})=\frac{\displaystyle\sum_{\textbf{h}}e^{-E(\textbf{v},\textbf{h})}}{\displaystyle\sum_{\textbf{v},\textbf{h}}e^{-E(\textbf{v},\textbf{h})}},$
(5)

which is commonly handled in its natural logarithm version, i.e., more
precisely, we aim at maximizing the negative logarithm of the likelihood
function (Negative Log-Likelihood - NLL). Moreover, regarding the visible
units, such a procedure can be easily extended to the continuous domain, which
is useful to model any type of input. The changes occur on the energy
function, as follows:

$E(\textbf{v},\textbf{h})=\sum_{i=1}^{m}\dfrac{(v_{i}-b_{i})^{2}}{2\sigma^{2}_{i}}-\sum_{j=1}^{n}c_{j}h_{j}-\sum_{i=1}^{m}\sum_{j=1}^{n}\dfrac{v_{i}}{\sigma_{i}}h_{j}w_{ij}.$
(6)

Considering the derivatives, it is straightforward to show that the visible
prior becomes:

$P(v_{i}=1|\textbf{h})\sim\mathcal{N}\left(\sum_{j=1}^{n}w_{ij}h_{j}+b_{i},\sigma^{2}_{i}\right),$
(7)

and, when a Gaussian input with zero mean and one unit standard deviation is
employed, such a prior becomes simple and easy to sample, since the whole
procedure is still the same.

###  II-D Deep Belief Networks

Deep Belief Networks (DBNs) [14] are generative graphical models composed of a
stack of RBMs, thus providing multiple layers of latent variables, such that
the hidden layer of the bottommost RBM is employed to feed the subsequent
input units successively until reaching the topmost layer. DBNs are trained
greedily, meaning that an RBM at a specific layer does not consider others
during its learning procedure. Thus, a DBN is composed of $L$ layers, where
$\textbf{W}^{l}$ is the weight matrix of an RBM at layer $l$. Additionally, we
can observe that the hidden units at layer $l$ become the input units of layer
$l+1$.

The aforementioned procedure stands for the generative pre-training.
Afterward, it is possible to attach fully-connected (FC) layers with softmax
outputs at the topmost hidden layer for a discriminative fine-tuning, which
can be used for classification tasks.

##  III Proposed Approaches

This work proposes to employ DBNs as non-linear functions $f(\mathcal{D}_{S})$
to learn from the source domain, $\mathcal{D}_{S}$, the information that can
be used to map the target domain, $\mathcal{D}_{T}$, i.e., to extract
information from videos and use them to classify high-level events.
Additionally, two approaches are shown to the respective task.

###  III-A Aggregative Deep Belief Networks

The first alternative architecture is denoted as Aggregative (A- prefix in
models), which modifies the DBNs’ first layer. Such a variation is designed
considering two main concepts: (i) to capture general spatio-temporal
information without additional techniques, such as optical flow algorithms,
and (ii) reduce the overall computational burden.

The proposed A-DBN processes all frames simultaneously instead of processing
one frame at a time, enabling a complete parameter update at each iteration.
In other words, A-DBN aggregates all frames $\\{F_{1},F_{2},\dots,F_{n}\\}$
into a single frame denoted as $F_{r}$, which represents their summation. This
procedure carries spatial information (contour and edges) along their temporal
trajectories, highlighted as “spectrum” in the resulting frame. Figure 2
depicts the $F_{r}$ aggregation process, as well as its highlighted region.
Afterward, A-DBN first layer infers the posterior distribution given $F_{r}$,
as follows:

$P(h_{j}=1|\bm{F_{r}})=\phi\left(\sum_{i=1}^{m}w_{ij}F_{ri}+c_{j}\right).$ (8)

Figure 2: Frames aggregation for the Aggregative-based approach.

###  III-B Gradient Deep Belief Networks

The second alternative architecture denoted as Gradient (G- prefix in models)
also modifies the DBNs first layer. Such a variation is designed considering
one main concept, i.e., to capture general motion information between two
frames.

The proposed G-DBN processes two consecutive frames instead of processing one
frame at a time. In other words, G-DBN defines a resultant frame, $F_{r}$, as
the direct subtraction as follows: given $F_{1}$ and $F_{2}$, $F_{r}$ stands
for $F_{2}-F_{1}$. This procedure carries motion cues from the spatial domain
along their trajectories. Figure 3 depicts the $F_{r}$ generation process, as
well as its highlighted region. Afterward, G-DBN first layer infers the
posterior distribution given $F_{r}$, as follows:

$P(h_{j}=1|\bm{F_{r}})=\phi\left(\sum_{i=1}^{m}w_{ij}F_{ri}+c_{j}\right).$ (9)

Figure 3: Frames generation for the Gradient-based approach.

##  IV Experiments

In this section, we describe the dataset and the experimental setup employed
to apply and compare DBNs with A-DBN, proposed in this paper, for the task of
event recognition from complex actions domain.

###  IV-A Dataset

We opted to use two well-known datasets, UCF-101 [26] and HMDB-51 [27], as
they represent a big challenge and are well-established video action
recognition datasets. Both datasets comprises a significant amount of data
from real-world action videos collected from YouTube and classified among
$101$ and $51$ distinct classes, respectively. Moreover, such a diversity
becomes more expressive due to the substantial variations in camera motion,
object appearance and pose, scaling, viewpoint, cluttered background, and
illumination conditions.

The $13,320$ videos from UCF-101 are grouped in 5 macro-categories, which are
easily interpreted as high-level events. Also, this mapping expects inter-
class videos to share standard and essential features, which helps in
recognizing common actions and interactions. Following the authors’ guideline,
the high-level events used: (0) sports practice; (1) musical practice with an
instrument; (2) human-object interaction; (3) human body-motion; and (4)
people interacting. Random clips depicting such classes are presented in
Figure 4, each class is represented by the color of the border: green for $0$,
light-blue for $1$, blue for $2$, red for $3$, and purple for $4$.

Figure 4: Random clips from UCF-101 [26].

Similarly, the $6,766$ videos from HMDB-51 are grouped in 5 macro-categories,
easily interpreted as high-level events. Also, following the authors’
guideline, the high-level events are (0) human facial expression; (1)
manipulation of objects in the face region; (2) body movement; (3) interaction
between people and object(s); and (4) person interacting with each other,
where numbers in parentheses represent classes. Additionally, frames of clips
from the dataset are shown in Figure 5, where the color of the border
represents the event class: green for $0$, light blue for $1$, blue for $2$,
red for $3$, and purple for $4$.

Figure 5: Random clips from HMDB-51 [27].

Both datasets provide three partitions, each with separate data for training
and testing, where the first partition was used in this study, as it seems to
have the most difficult test samples to evaluate as cluttered background or
fewer interactions and actions. The process of splitting/acquiring the frames
is performed in a similar way to the work of Ng et al. [28], using $6$ frames
per video clip uniformly distributed over time. In their work, Ng et al.
showed that $6$ frames per video are enough to ensure a good performance,
achieving the same results as $20$ frames, for example, in addition to
imprinting a lower computational load on the action classification task.

Regarding the pre-processing step, two transformations were employed before
the image conversion to grayscale. The first transformation concerns cutting
operations, removing black regions that do not carry information, and resizing
from the original size ($240\times 320$) to $72\times 96$, to facilitate the
processing of energy-based models. The second transform stands for feature-
normalization using a Gaussian distribution with zero mean and unit variance.

###  IV-B Experimental Setup

Regarding the experimental setup hardware, we employed an Intel 2x Xeon(R)
E5-2620 @ 2.20GHz (40 cores), a GTX 1080 Ti, and 128 GB of RAM. For the
unsupervised pre-training process, we opted to use mini-batches of $128$
samples and $3$ epochs per-layer. Finally, Table I describes the employed
architectures and hyper-parameters.

TABLE I: Configuration of the models used in this work. Model | Layers | Hidden Neurons | Momentum | Learning Rate
---|---|---|---|---
RBM | $1$ | $[2,000]$ | $[0.5]$ | $[1\cdot 10^{-3}]$
A-RBM | $1$ | $[2,000]$ | $[0.5]$ | $[1\cdot 10^{-3}]$
G-RBM | $1$ | $[2,000]$ | $[0.5]$ | $[1\cdot 10^{-3}]$
$\text{DBN}_{\alpha}$ | $2$ | $[2,000-2,000]$ | $[0.5;0.5]$ | $[1\cdot 10^{-3};5\cdot 10^{-4}]$
$\text{DBN}_{\beta}$ | $3$ | $[2,000-2,000-2,000]$ | $[0.5;0.5;0.5]$ | $[1\cdot 10^{-3};5\cdot 10^{-4};5\cdot 10^{-4}]$
$\text{DBN}_{\iota}$ | $2$ | $[4,000-4,000]$ | $[0.5;0.5]$ | $[5\cdot 10^{-4};5\cdot 10^{-4}]$
$\text{DBN}_{\zeta}$ | $3$ | $[4,000-4,000-4,000]$ | $[0.5;0.5;0.5]$ | $[5\cdot 10^{-4};5\cdot 10^{-4};5\cdot 10^{-4}]$
$\text{A-DBN}_{\alpha}$ | $2$ | $[2,000-2,000]$ | $[0.5;0.5]$ | $[1\cdot 10^{-3};5\cdot 10^{-4}]$
$\text{A-DBN}_{\beta}$ | $3$ | $[2,000-2,000-2,000]$ | $[0.5;0.5;0.5]$ | $[1\cdot 10^{-3};5\cdot 10^{-4};5\cdot 10^{-4}]$
$\text{A-DBN}_{\iota}$ | $2$ | $[4,000-4,000]$ | $[0.5;0.5]$ | $[5\cdot 10^{-4};5\cdot 10^{-4}]$
$\text{A-DBN}_{\zeta}$ | $3$ | $[4,000-4,000-4,000]$ | $[0.5;0.5;0.5]$ | $[5\cdot 10^{-4};5\cdot 10^{-4};5\cdot 10^{-4}]$
$\text{G-DBN}_{\alpha}$ | $2$ | $[2,000-2,000]$ | $[0.5;0.5]$ | $[1\cdot 10^{-3};5\cdot 10^{-4}]$
$\text{G-DBN}_{\beta}$ | $3$ | $[2,000-2,000-2,000]$ | $[0.5;0.5;0.5]$ | $[1\cdot 10^{-3};5\cdot 10^{-4};5\cdot 10^{-4}]$
$\text{G-DBN}_{\iota}$ | $2$ | $[4,000-4,000]$ | $[0.5;0.5]$ | $[5\cdot 10^{-4};5\cdot 10^{-4}]$
$\text{G-DBN}_{\zeta}$ | $3$ | $[4,000-4,000-4,000]$ | $[0.5;0.5;0.5]$ | $[5\cdot 10^{-4};5\cdot 10^{-4};5\cdot 10^{-4}]$

Each model is connected to two additional fully-connected layers that are
fine-tuned using the well-known Adam optimizer [29] with the learning rate
equals to $10^{-3}$ and the same number of epochs and mini-batch size. Such FC
layers have two configurations since they depend on the number of hidden
neurons from the last RBM/DBN layer, i.e., $2,000-1,000-5$ and
$4,000-2,000-5$. It is important to highlight that the RBMs were employed for
$2,000$ hidden neurons only as this work primarily focuses on using the
hierarchical information learned by DBNs.

Furthermore, we opted to use the same approach employed by transfer learning,
i.e., to freeze the connections from the first hidden layer and make a gentle
adjustment of subsequent hidden layers (learning rate equals to $10^{-6}$).
The cross-entropy loss was used when adjusting model weights during the fine-
tuning process, while the final measure was the accuracy on the testing set.
Finally, to mitigate any stochastic nature, each model was fully trained and
fine-tuned for $6$ repetitions.

##  V Experimental Results

This section presents the experimental results concerning the DBNs and the
proposed approaches, i.e., the A-DBN and the G-DBN, applied to the task of
event recognition. All models follow the work of Ng et al. [28], using $6$
frames per video clip uniformly distributed over time.

###  V-A Model Evaluation for Event Recognition

Regarding the main task, i.e., learning from actions to classify high-level
events, Tables II and III show the predictive performance for all models and
architectures. The highlighted result stands for the best mean accuracy.
Besides, it also presents the average running time for each model, averaged
over the six repetitions to analyze the models’ computation impact and
efficiency.

TABLE II: Mean accuracies (%) and running times (minutes) over the UCF-101 test set (fold 1). Architecture | Accuracy | Time
---|---|---
RBM | $38.71\pm 1.04$ | $315.00\pm 5.00$
A-RBM | $42.48\pm 0.94$ | $270.00\pm 5.00$
G-RBM | $44.04\pm 1.82$ | $314.00\pm 5.00$
$\text{DBN}_{\alpha}$ | $37.72\pm 4.67$ | $765.00\pm 5.00$
$\text{A-DBN}_{\alpha}$ | $44.66\pm 1.28$ | $540.00\pm 5.00$
$\text{G-DBN}_{\alpha}$ | $40.16\pm 11.63$ | $764.00\pm 5.00$
$\text{DBN}_{\beta}$ | $40.55\pm 3.54$ | $1,215.00\pm 5.00$
$\text{A-DBN}_{\beta}$ | $44.80\pm 2.02$ | $810.00\pm 5.00$
$\text{G-DBN}_{\beta}$ | $44.84\pm 0.08$ | $1,211.00\pm 5.00$
$\text{DBN}_{\iota}$ | $41.92\pm 2.65$ | $775.00\pm 6.00$
$\text{A-DBN}_{\iota}$ | $\bm{45.01\pm 1.39}$ | $550.00\pm 6.00$
$\text{G-DBN}_{\iota}$ | $44.84\pm 0.04$ | $773.00\pm 6.00$
$\text{DBN}_{\zeta}$ | $42.33\pm 4.51$ | $1,225.00\pm 6.00$
$\text{A-DBN}_{\zeta}$ | $44.87\pm 2.81$ | $820.00\pm 6.00$
$\text{G-DBN}_{\zeta}$ | $44.86\pm 0.14$ | $1,220.00\pm 6.00$

From Table II, we can notice prominent results, mainly for the proposed
Aggregative version of DBNs. Starting from the base model, A-RBM achieved a
better accuracy than RBM, i.e., $42.48\%$ against $38.71\%$, representing a
meaningful difference ($3.77$ points of mean percentual accuracy), while the
G-RBM model achieved $44.04\%$, outperforming also the Aggregative approach.
In addition, A-RBM has a significantly lower computational burden,
approximately $14\%$ less running time. To clarify, hereafter, the percentual
difference between models’ accuracy stands for the absolute mean value of the
proposed model (A- or G-) subtracted to its standard version, and for the
running time, it is the mean proposed approach time divided by its standard
model.

Regarding the second architecture, i.e., the $\alpha$ models, the Aggregative
version overcomes the standard DBN and G-DBN in mean accuracy by almost $7\%$
and $4.5\%$, respectively, representing a meaningful improvement. However, the
standard model and the G-DBN do not overpass its parameterless version (RBM
and G-RBM) in mean accuracy. Moreover, the running time for A-DBNα was $30\%$
smaller than DBNα, impacting positively to a lighter training burden.

Concerning the $\beta$ models, a similar behavior was observed, i.e., the
Aggregative version overpassed the standard DBN in approximately $4\%$ in mean
accuracy, and almost $33\%$ less running time. Also, the G-DBN model achieved
the best mean accuracy, $44.84\%$, with a small standard deviation. However,
such results show that adding more hidden layers does not lead to an
impressive performance improvement for A-DBN since its mean accuracy was close
to the A-DBNα, while for the G-DBNβ the performance was increased. Moreover,
even with the previous observation, the A-DBNβ still gives a better running
time than its baselines.

Regarding the fourth architecture ($\iota$ models), the same behavior was
observed, highlighting that A-DBNι achieved a remarkable mean accuracy of
$45.01\%$, the highest average value over the baselines. Moreover, the running
time for A-DBNι was $30\%$ better than its standard version. Here, the
Aggregative models showed that more hidden units might benefit the overall
performance for the event classification task with transfer learning.

Finally, the $\zeta$ models showed almost the same results as the $\iota$
models, mainly for the proposed approaches, A-DBN and G-DBN, which achieved a
mean accuracy of $44.87\%$ and $44.86\%$, respectively. Such results indicate
that more hidden neurons improve the models’ performance. However, the larger
versions may demand more epochs of pre-training and/or data. Also, the running
time of A-DBN overpasses its standard and the Gradient version by
approximately $33\%$.

Nonetheless, it is essential to notice that A-DBN models have no difficulty in
overpassing the A-RBM mean accuracy; however, the G-DBNα do not overpass its
simpler version on mean accuracy due to one specific run that pushed down the
performance (note the standard deviation). Overall, the performance
improvement observed can be directly linked to the higher abstraction achieved
by hidden layers. Such approaches can improve the DBN lower bound and provide
a further improvement in discriminative fine-tuning. Besides, they were also
pre-trained with a relatively small number of epochs, which can induce a less
efficient overall lower bound optimization. However, the results showed that
more hidden units in hidden layers improve the mean accuracy rate, as the
A-DBNι and A-DBNζ models have shown.

TABLE III: Mean accuracies (%) and running times (minutes) over the HMDB-51 test set (fold 1). Architecture | Accuracy | Time
---|---|---
RBM | $34.60\pm 3.90$ | $45.00\pm 5.00$
A-RBM | $35.19\pm 4.24$ | $30.00\pm 5.00$
G-RBM | $38.05\pm 0.36$ | $44.00\pm 5.00$
$\text{DBN}_{\alpha}$ | $34.49\pm 3.98$ | $144.00\pm 5.00$
$\text{A-DBN}_{\alpha}$ | $37.68\pm 4.19$ | $132.00\pm 5.00$
$\text{G-DBN}_{\alpha}$ | $38.40\pm 0.01$ | $143.00\pm 5.00$
$\text{DBN}_{\beta}$ | $33.83\pm 4.06$ | $216.00\pm 5.00$
$\text{A-DBN}_{\beta}$ | $38.70\pm 4.33$ | $195.00\pm 5.00$
$\text{G-DBN}_{\beta}$ | $38.40\pm 0.01$ | $214.00\pm 5.00$
$\text{DBN}_{\iota}$ | $34.41\pm 4.03$ | $150.00\pm 6.00$
$\text{A-DBN}_{\iota}$ | $38.23\pm 4.25$ | $138.00\pm 6.00$
$\text{G-DBN}_{\iota}$ | $38.40\pm 0.01$ | $148.00\pm 6.00$
$\text{DBN}_{\zeta}$ | $34.53\pm 4.04$ | $225.00\pm 6.00$
$\text{A-DBN}_{\zeta}$ | $\bm{38.86\pm 4.26}$ | $207.00\pm 6.00$
$\text{G-DBN}_{\zeta}$ | $37.41\pm 2.43$ | $223.00\pm 6.00$

From Table III, one can notice interesting results, mainly for the proposed
approaches. The base model, A-RBM achieved a better accuracy rate than RBM,
i.e., $35.19\%$ against $34.60\%$, representing a meaningful difference
($0.60$ points of mean percentual accuracy), while the G-RBM model achieved
$38.05\%$, beating also the Aggregative approach. Here, it is interesting to
note that the time difference between models was not so impressive, explained
by the low data volume.

Regarding the second architecture, i.e., the $\alpha$ models, the Aggregative
version overcomes the standard DBN in mean accuracy by almost $3\%$, while the
G-DBN overpassed its standard version in approximately $4\%$, respectively,
representing a meaningful improvement. Moreover, it is important to notice
that the running time for A-DBNα was $8\%$ smaller than DBNα, impacting
positively to a lighter training burden.

Concerning the $\beta$ models, was observed a similar behavior, i.e., the
Aggregative version overpassed the standard DBN in approximately $5\%$ in mean
accuracy, and almost $10\%$ less running time. Also, the G-DBN model
overpassed its standard version in approximately $5\%$ of mean accuracy, with
a lower standard deviation. However, such results show that adding more hidden
layers does not cause an impressive performance improvement for G-DBN since
its mean accuracy was close to the G-DBNα, while for the A-DBNβ the
performance was increased. Moreover, even with the previous observation, the
A-DBNβ still gives a better running time than its baselines.

Regarding the $\iota$ models, the baseline model slightly increased its
performance, however, the G-DBN achieved the highest mean accuracy, $38.40\%$,
overpassing the DBN and A-DBN models. The Aggregative version was also better
than DBN, with $38.23\%$, and $8\%$ less running time than DBN. However, such
results show that adding more hidden layers does not cause an impressive
performance improvement for the proposed approach, keeping the hidden neurons
in $2,000$ units, since the mean accuracies were close to the A-DBNβ and
G-DBNβ.

Finally, the $\zeta$ models showed an interesting performance improvement
regarding the Aggregative approach, which achieved a remarkable mean accuracy
of $38.86\%$. On the other hand, G-DBN suffers from a performance decreasing,
achieving $37.41\%$ of mean accuracy. This fact pointed out that more hidden
neurons improve the Aggregative’s performance while keeping the lowest mean
time for training ($207$ minutes).

In general, one can observe two interesting behaviors for the proposed
approaches, the Aggregative-based models were able to improve the models’
performance, resulted from the total frames aggregation that carries general
motion information. On the other hand, the Gradient-based models did not
improve their performance as the Aggregative-based, explained by the fact that
two-by-two frames on the employed datasets may not carry as much information
like the overall sum of the six frames.

##  VI Conclusions and Future Works

In this paper, we addressed a transfer learning approach on two well-known
video datasets, learning from the action- to the event-based domain, through
energy-based models, such as RBMs and DBNs. Furthermore, we proposed
Aggregative-based and Gradient-based approaches that modify the processing of
the frames, simplifying the model’s complexity and giving robustness, saving
processing time, and improving the model’s generalization. Experimental
results show that the proposed approach can reduce the computational time by
as much as $33\%$ regarding the A-models.

The results were promising since most A-DBN architectures achieved a feasible
mean accuracy rate, highlighting the models A-DBNι and A-DBNζ. Also, the
Aggregative models showed a meaningful reduction in running time during the
unsupervised pre-training phase. The Gradient models showed a stable behavior,
varying almost nothing regarding the different architectures. Therefore, these
experimental results support our hypothesis that it is possible to transfer
the knowledge from actions to events with the employed energy-based models,
without complex inputs such as optical flow or convolutions. Finally, one can
highlight that by increasing the number of hidden neurons the overall
performance improved, pointing out the models’ opportunity to extract more
information.

Regarding future works, we plan to investigate the effect of combining
convolution operators in energy-based models, such as the Convolutional
Restricted Boltzmann Machine (CRBM). Additionally, we aim at employing more
complex models, such as Deep Boltzmann Machines (DBMs), trained with a more
expressive number of epochs. Finally, a future step is also to analyze how the
proposed approach can be applied to data augmentation tasks in the context of
event classification in videos.

## Acknowledgments

This research was supported by São Paulo Research Foundation - FAPESP (grants
#2013/07375-0, #2014/12236-1, #2019/07665-4, #2019/07825-1 and #2019/02205-5),
FAPESP-Microsoft Research Virtual Institute (grant #2017/25908-6), and
Brazilian National Council for Scientific and Technological Development - CNPq
(grants #314868/2020-8, #307066/2017-7 and #427968/2018-6).

## References

  * [1] Y. Kong and Y. Fu, “Human action recognition and prediction: A survey,” _CoRR_ , vol. abs/1806.11230, 2018. [Online]. Available: http://arxiv.org/abs/1806.11230
  * [2] A. F. Bobick and J. W. Davis, “The recognition of human movement using temporal templates,” _IEEE Transactions on pattern analysis and machine intelligence_ , vol. 23, no. 3, pp. 257–267, 2001.
  * [3] H. Venkateswara, S. Chakraborty, and S. Panchanathan, “Deep-learning systems for domain adaptation in computer vision: Learning transferable feature representations,” _IEEE Signal Processing Magazine_ , vol. 34, no. 6, pp. 117–129, 2017.
  * [4] S. Sun, H. Shi, and Y. Wu, “A survey of multi-source domain adaptation,” _Information Fusion_ , vol. 24, pp. 84–92, 2015.
  * [5] N. FarajiDavar, T. De Campos, J. Kittler, and F. Yan, “Transductive transfer learning for action recognition in tennis games,” in _2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops)_. IEEE, 2011, pp. 1548–1553.
  * [6] L. Shao, F. Zhu, and X. Li, “Transfer learning for visual categorization: A survey,” _IEEE transactions on neural networks and learning systems_ , vol. 26, no. 5, pp. 1019–1034, 2014.
  * [7] A. Quattoni, M. Collins, and T. Darrell, “Transfer learning for image classification with sparse prototype representations,” in _2008 IEEE Conference on Computer Vision and Pattern Recognition_. IEEE, 2008, pp. 1–8.
  * [8] L. Cao, Z. Liu, and T. S. Huang, “Cross-dataset action detection,” in _2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition_. IEEE, 2010, pp. 1998–2005.
  * [9] T. J. Darrell, I. A. Essa, and A. P. Pentland, “Task-specific gesture analysis in real-time using interpolated views,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 18, no. 12, pp. 1236–1242, 1996.
  * [10] L. Wang, C. Gao, L. Yang, Y. Zhao, W. Zuo, and D. Meng, “Pm-gans: Discriminative representation learning for action recognition using partial-modalities,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 384–401.
  * [11] Y. Tas and P. Koniusz, “Cnn-based action recognition and supervised domain adaptation on 3d body skeletons via kernel feature maps,” _arXiv preprint arXiv:1806.09078_ , 2018.
  * [12] J. Gao, T. Zhang, and C. Xu, “I know the relationships: Zero-shot action recognition via two-stream graph convolutional networks and knowledge graphs,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 33, 2019, pp. 8303–8311.
  * [13] Y. Liu, Z. Lu, J. Li, T. Yang, and C. Yao, “Deep image-to-video adaptation and fusion networks for action recognition,” _IEEE Transactions on Image Processing_ , vol. 29, pp. 3168–3182, 2019.
  * [14] G. E. Hinton, S. Osindero, and Y.-W. Teh, “A fast learning algorithm for deep belief nets,” _Neural Computation_ , vol. 18, no. 7, pp. 1527–1554, 2006\.
  * [15] H. Ali, S. N. Tran, E. Benetos, and A. S. d. Garcez, “Speaker recognition with hybrid features from a deep belief network,” _Neural Computing and Applications_ , vol. 29, no. 6, pp. 13–19, 2018.
  * [16] M. Roder, L. A. Passos, L. C. F. Ribeiro, B. C. Benato, A. X. Falcão, and J. P. Papa, “Intestinal parasites classification using deep belief networks,” in _International Conference on Artificial Intelligence and Soft Computing_. Springer, 2020, pp. 242–251.
  * [17] G. E. Hinton, “Training products of experts by minimizing contrastive divergence,” _Neural Computation_ , vol. 14, no. 8, pp. 1771–1800, 2002\.
  * [18] S. N. Gowda, “Human activity recognition using combinatorial deep belief networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , 2017, pp. 1–6.
  * [19] K. H. Ali and T. Wang, “Learning features for action recognition and identity with deep belief networks,” in _2014 International Conference on Audio, Language and Image Processing_. IEEE, 2014, pp. 129–132.
  * [20] Y. Zhang, Z. Liu, and W. Zhou, “Event recognition based on deep belief network,” _Acta Electronica Sinica_ , vol. 45, no. 6, pp. 1415–1423, 2017\.
  * [21] Y.-G. Jiang, S. Bhattacharya, S.-F. Chang, and M. Shah, “High-level event recognition in unconstrained videos,” _International Journal of Multimedia Information Retrieval_ , vol. 2, no. 2, pp. 73–101, 2013.
  * [22] J. Liu, B. Kuipers, and S. Savarese, “Recognizing human actions by attributes,” in _Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition_ , ser. CVPR ’11. IEEE Computer Society, 2011, pp. 3337–3344.
  * [23] C. Tan, F. Sun, T. Kong, W. Zhang, C. Yang, and C. Liu, “A survey on deep transfer learning,” in _International conference on artificial neural networks_. Springer, 2018, pp. 270–279.
  * [24] K. Weiss, T. M. Khoshgoftaar, and D. Wang, “A survey of transfer learning,” _Journal of Big data_ , vol. 3, no. 1, pp. 1–40, 2016.
  * [25] P. Smolensky, “Parallel distributed processing: Explorations in the microstructure of cognition,” D. E. Rumelhart, J. L. McClelland, and C. PDP Research Group, Eds. Cambridge, MA, USA: MIT Press, 1986, vol. 1, ch. Information Processing in Dynamical Systems: Foundations of Harmony Theory, pp. 194–281.
  * [26] K. Soomro, A. R. Zamir, and M. Shah, “Ucf101: A dataset of 101 human actions classes from videos in the wild,” _arXiv preprint arXiv:1212.0402_ , 2012\.
  * [27] H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre, “Hmdb: a large video database for human motion recognition,” in _2011 International Conference on Computer Vision_. IEEE, 2011, pp. 2556–2563.
  * [28] J. Yue-Hei Ng, M. Hausknecht, S. Vijayanarasimhan, O. Vinyals, R. Monga, and G. Toderici, “Beyond short snippets: Deep networks for video classification,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 4694–4702.
  * [29] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.


# CRC-RL: A Novel Visual Feature Representation Architecture for Unsupervised
Reinforcement Learning

Darshita Jain1, Anima Majumder1, Samrat Dutta$1$, Swagat Kumar2

1 TATA Consultancy Services, Bangalore, India.
2 Edge Hill University, UK.

1(darshita.jain, anima.majumder<EMAIL_ADDRESS>, 2<EMAIL_ADDRESS>

###### Abstract

This paper addresses the problem of visual feature representation learning
with an aim to improve the performance of end-to-end reinforcement learning
(RL) models. Specifically, a novel architecture is proposed that uses a
heterogeneous loss function, called CRC loss, to learn improved visual
features which can then be used for policy learning in RL. The CRC-loss
function is a combination of three individual loss functions, namely,
contrastive, reconstruction and consistency loss. The feature representation
is learned in parallel to the policy learning while sharing the weight updates
through a Siamese Twin encoder model. This encoder model is augmented with a
decoder network and a feature projection network to facilitate computation of
the above loss components. Through empirical analysis involving latent feature
visualization, an attempt is made to provide an insight into the role played
by this loss function in learning new action-dependent features and how they
are linked to the complexity of the problems being solved. The proposed
architecture, called CRC-RL, is shown to outperform the existing state-of-the-
art methods on the challenging Deep mind control suite environments by a
significant margin thereby creating a new benchmark in this field.

In the recent past, deep reinforcement learning (DRL) algorithms have been
successfully used to learn action policies directly from visual observations,
thereby finding application in several interesting areas such as gaming [1,
2], robotics [3, 4, 5, 6] and autonomous vehicles [7, 8] etc. This success is
mostly driven by the agent’s ability to jointly learn feature representation
and policy decisions by using long-term credit-assignment capabilities of
reinforcement learning algorithms in an end-to-end fashion. In spite of this
success, RL algorithms are known to be sample-inefficient and suffer from poor
generalization for high-dimensional observations such as images [9, 10]. There
are several approaches to address these concerns, including methods such as,
transfer learning [11, 12], meta-learning [13] [14] and active learning [15].
_Feature representation learning_ [16] is an alternative, and sometimes
complementary, to these approaches which aims at learning useful features that
can simplify the intended task, e.g., classification or prediction. This paper
primarily focuses on this later approach as it is now widely accepted that the
problem of sample-inefficiency in RL can be solved to a great extent by
learning suitable feature representation which is shown to expedite the policy
learning process [17]. The feature representations are learned using self-
supervised methods which are increasingly becoming popular as they obviate the
need for manually generated labeled datasets thereby simplifying the practical
deployment of deep learning models [18]. Some of these approaches include
auto-encoders [19], GANs [20] [21], contrastive learning [22] and data
augmentation techniques [9] [23]. The features thus obtained have been shown
to greatly improve the sample efficiency and generalizability of RL methods as
demonstrated in [24] [10] [23].

Rather than decoupling the representation learning from policy learning as
done in [25] [23] [17], we continue working with end-to-end models because of
their simplicity and aim at improving their performance by performing
auxiliary tasks as demonstrated in [10] [26]. Since the feature
representations are learned along side the policy decisions in an end-to-end
fashion, the features learned are actually _action-dependent_. This is
because, the backward gradient flow from the policy-learning algorithm is
allowed to update the encoder weights. This makes the learned feature vectors
strongly correlated to the actions being taken by the agent [27]. Given this
hindsight, we are motivated by two factors. First, it is our belief that the
quality of the features learnt could be improved by using a better loss
function leading to improved RL performance. Secondly, we are keen to develop
a better understanding of the relationship between the feature and action
spaces. Towards fulfilling these objectives, we propose a new heterogeneous
loss function called _CRC loss_ for feature representation learning by
combining three different loss functions, namely, _contrastive loss_ [10],
_reconstruction loss_ and _consistency loss_. The reconstruction loss obtained
with an auto-encoder model helps in learning compact features that is
sufficient to reconstruct the original observations. On the other hand, the
consistency loss [23] helps in learning features that are invariant to image
augmentations. In other words, by minimizing the consistency loss, the encoder
is encouraged to learn _task-relevant_ features while ignoring irrelevant
aspects (such as, background color) thereby avoiding observational over-
fitting [28]. Similarly, the contrastive loss helps in learning _class-
invariant_ features from augmented images by contrasting them against a batch
of negative samples. In that sense, these three loss functions contribute
complementary information and hence, should improve the feature representation
learning when combined together. In order to implement feature training with
this loss function, a new architecture inspired by CURL [10], is proposed that
uses a Siamese Twin encoder model, a decoder network and a feature predictor
to generate these losses. Through empirical analysis including feature
visualizations, it is shown that the feature representations learnt by the CRC
loss function is different from those learnt with the baseline CURL model,
indicating the role played by the CRC-loss in learning new action-dependent
features. In addition, visualization of correlation matrices between latent
features generated by this model show increasingly complex patterns for
complex environments with higher-dimensional action spaces, thereby providing
a clue about how features are inherently linked with action in an end-to-end
RL model. Through rigorous experiments on the challenging Deep Mind Control
suite environments [29], it is shown that the proposed CRC-RL model
outperforms the existing state-of-the-art methods by a significant margin,
thereby establishing the efficacy of the approach. The design choices for the
proposed model are justified through extensive ablation studies.

In short, the major contributions made in this paper are as follows:

  1. 1.

A new self-supervised feature representation architecture along with a novel
loss function is proposed to improve the performance of RL models in learning
policies directly from image observations.

  2. 2.

Through empirical analysis involving latent feature visualization, an attempt
has been made to provide insights into the relationship between the action and
feature space thereby providing better standing of the role played of the new
loss function in learning _action-dependent_ features.

  3. 3.

The resulting architecture is shown to outperform existing state-of-the-art
methods on the challenging DMC environments by a significant margin.

The rest of this paper is organized as follows. Related works are reviewed in
the next section. The proposed architecture is described in Section 2. The
experimental results are discussed and analyzed in Section 3. The paper ends
with the concluding section 4.

Figure 1: CRC-RL Architecture: It consists of a Siamese Twin encoder along
with a decoder and a feature predictor network. The query encoder together
with the decoder forms an auto-encoder. The query encoder is used for learning
policy using SAC algorithm. Observations are data-augmented to form query and
key observations, which are then encoded into latent features by the
respective encoders. Only the query encoder weights are updated during the
training step. The weights of key encoder are exponential moving average of
query encoder weights.

##  1 Related Works

This section provides an overview of related literature in the following
subsections.

###  1.1 Deep RL architectures for policy learning

Reinforcement learning algorithms learn the optimal policy for a given task by
maximizing a measure of _cumulative discounted future reward_ for the task
while balancing between _exploration_ (of new possibilities) and
_exploitation_ (of post experiences) [30]. This cumulative discounted reward
function, represented as Q or _value_ function, is not known a priori and, is
used to evaluate a given action taken by the agent. Depending on how this
function is estimated and desirable actions are derived from it, the RL-based
methods can be broadly classified into two categories: _value-based_ methods
and _policy-based methods_. The value-based methods aim at estimating the
Q-function and then derive action from this by using a greedy policy. On the
other hand, policy- based methods directly estimate the policy function by
maximizing a given objective function. The traditional Q-learning algorithm
estimates the Q function iteratively by using an approximate dynamic
programming formulation based on Bellman’s equation starting from an initial
estimate [31]. The original Q-learning algorithm could be applied to problems
with discrete state (observation) and action spaces and hence, suffered from
the _curse-of-dimensionality_ problem with higher dimensions and range of
values. This limitation is overcome by using a deep network to estimate Q
function that can take arbitrary observation inputs, thereby, greatly
enhancing the capabilities of RL algorithms. The resulting approach is known
as Deep Q Networks (DQN) [32] [33] which has been applied successfully to a
wide range of problems while achieving superhuman level performances in a few
cases, such as ATARI video games [34], Go [35] etc. The success of DQN has
spawned a new research field known as deep reinforcement learning (DRL)
attracting a large following of researchers. Readers are referred to [36] for
a survey of this field. The DQN models were subsequently extended to
continuous action spaces by using policy gradient methods that used a
parameterized policy function to maximize DQN output using gradient ascent
methods [37] [38]. This has opened the doors for solving various robotics
problems that use continuous values such as joint angles, joint velocities or
motor torques as input. Since then, a number of methods have been proposed to
improve the performance of RL algorithms and have been applied successfully to
different robotic problems - manipulation [39] [40], grasping [41] [42],
navigation [43] etc. Some of the notable methods include actor-critic models -
A2C and A3C [44], soft actor-critic (SAC) [45] and proximal policy
optimization (PPO) [46]. In spite of the success of these methods, (deep)
reinforcement learning algorithms, in general, suffer from limitations such as
poor sampling efficiency leading to longer training time, poor generalization
and instability. The work presented in this paper aims to address some of
these concerns by focusing on learning better task-relevant features.

###  1.2 Feature Representation Learning in RL

It is now widely accepted that learning policies from low-dimensional feature
vectors based on physical states is much more sample efficient compared to
learning directly from image pixels [10] [47]. Hence, it is imperative to
learn suitable state representations from image observations that will reduce
the search space thereby improving the sample efficiency and stability of RL
algorithms. The field of self-supervised representation learning has seen
great progress in last few years. Auto-encoders [19] [48] learn the state
representation by compressing the observation into low-dimensional state that
is sufficient to reconstruct the observation. These have been used to improve
the performance of RL algorithms as demonstrated in [17][49] [50] [24]. On the
other hand, contrastive learning [22] [51] learns the class-relevant feature
representations by maximizing the agreement between the augmented versions of
the same observation. It has been shown to greatly improve the sample
efficiency of RL algorithms as in [10]. Similarly, recent studies have shown
that the right kind of data augmentation techniques can improve the sample
efficiency and generalization capabilities of RL algorithms learning _task-
relevant_ features which remain unaffected by distractions introduced by the
augmentation [9] [52]. This can be further enforced by making the encoder
minimize the consistency loss as suggested in [23]. In short, learning
suitable feature representation plays a significant role in improving the
performance of RL algorithms by increasing sample efficiency, improving
generalization and stability. The work presented in this paper contributes to
this field by proposing a novel loss function that leads to superior learning
performance for continuous control tasks as will be demonstrated later in this
paper.

##  2 Method

This section provides details of the proposed CRC-RL model that uses a novel
heterogeneous loss function to extract useful information from visual images
to be used for learning optimal policy using an end-to-end RL framework. The
discussion is organized in the following subsections. The architecture of the
proposed model is described next.

###  2.1 The Model Architecture

The overall architecture of the proposed model is shown in Figure 1. The
observation is available in the form of images which are stacked together to
act as input to the model. Stacking of frames is a heuristic approach to
incorporate temporal information in the learning process [53]. The
observations obtained from the environment is stored in a replay buffer
$\mathcal{D}$ and a batch is sampled from this replay buffer during the
training process. A Siamese Twin encoder model is employed for extracting
features from the input images. These two encoders, termed as _query_ and
_key_ encoders, are used for computing contrastive and consistency losses. The
query encoder with a decoder is used for computing the reconstruction loss. A
combination of these three losses, known as the CRC loss, is used for updating
the parameters of the query encoder and decoder network. The input images are
augmented before applying to the key encoder. The features obtained from the
query encoder is used for policy estimation using soft-actor-critic algorithm
[45]. The parameters of the query encoder and decoder networks are updated
using error signals obtained from their own outputs as well as from the RL
algorithm. Since the encoder networks are getting influenced by the RL policy
algorithm, the features learnt in the process are _action-dependent_. This
aspect will be analyzed in more detail in the experiment section presented
later in this paper. The weights of the key encoder network is the exponential
moving average of the query encoder weights. The proposed CRC loss function
used for learning the feature embeddings is discussed in the next subsection.

###  2.2 The loss function for feature extraction

The query encoder is trained using the proposed CRC loss function which is a
combination of the following three loss components as described below.

####  2.2.1 Contrastive loss

In contrastive learning, we have a query $q$ observation and a set of key
observation samples $\mathbb{K}=\\{k_{0},k_{1},...\\}$ consisting of both
positive samples $(k_{+})$ and negative samples
$(\mathbb{K}\setminus{k_{+}})$. The positive samples are those that belong to
the same class as that of the query observation and the rest are considered to
be the negative samples. The goal is to learn embeddings such that $q$ is
relatively more similar to the positive keys $k_{+}$ than the negative keys in
the latent space. The query and key observations, generated by applying data
augmentation on sampled observations, are encoded using the query and key
encoder respectively. The contrastive loss depends on the output of both the
encoders (Siamese Twin) represented by the symbols $\mathbf{s}$ and
$\mathbf{s}^{\prime}$ respectively. The idea behind using the contrastive loss
is that the different augmentations of the same image will have the same
underlying information and hence their high-level representations will be
mapped together in the latent space. The similarity between the query and the
key embeddings is computed using the bilinear inner-product $q^{T}Wk>0$ where
$W$ is a symmetric matrix of parameters to be estimated [54] along with other
parameters during the training process. The objective of training is to reduce
this similarity measure so that the query embeddings become more distinct from
the key embeddings over time ($q^{T}Wk\approx 0\Rightarrow q\perp k,\;W>0$).
This is achieved by minimizing the InfoNCE loss [55] given by:

$L_{q}=\log\frac{\exp(q^{T}Wk_{+})}{\sum_{k_{i}\in\mathbb{K}}\exp(q^{T}Wk_{i})}$
(1)

####  2.2.2 Reconstruction loss

A well-trained encoder-decoder network is expected to reconstruct the input
image at the output of the decoder network. The reconstruction loss is
computed based on the inaccuracy in the reconstructed image. A convolutional
encoder $f_{\theta}$ maps an input observation
$\mathbf{x}\in\mathbb{R}^{m\times n\times 3}$ to a lower-dimensional latent
vector $\mathbf{s}\in\mathbb{R}^{l}$, and a deconvolutional decoder $g_{\phi}$
then reconstructs $\mathbf{s}$ back to $\mathbf{\hat{x}}\in\mathbb{R}^{m\times
n\times 3}$ such that

$\displaystyle f_{\theta}:\mathbf{x}\rightarrow\mathbf{s}$ (2) $\displaystyle
g_{\phi}:\mathbf{s}\rightarrow\mathbf{\hat{x}}$ (3)

Both the encoders and decoder are trained simultaneously by maximizing the
expected log-likelihood. The reconstruction loss checks how well the image has
been reconstructed from the input. The reconstruction loss forces the update
such that the latent representation preserves the core attributes of the input
data. An $L_{2}$ penalty is imposed on the learned representation $\mathbf{s}$
and a weight-decay is imposed on the decoder parameters to incorporate the
regularization affects as proposed in [56].

$L_{r}=\mathbb{E}_{\mathbf{x}\sim D}[\log
p_{\theta}(\mathbf{x}|\mathbf{s})+\lambda_{s}\|\mathbf{s}\|^{2}+\lambda_{\theta}\|\theta\|^{2}]$
(4)

where $\lambda_{s}$, and $\lambda_{\theta}$ are hyper-parameters.

####  2.2.3 Consistency loss

The consistency loss depends on the output of both the query and key encoder
$f_{\theta}$ and $f^{\prime}_{\theta}$. Here, the query encoder takes the
original non-augmented observation $\mathbf{x}$ and the key encoder uses the
augmented observation $\mathbf{\tilde{x}}$ as input. The output of the Key
encoder $\mathbf{s}^{\prime}$ is then used as an input to a feature predictor
module, which is nothing but an MLP, to estimate the non-augmented embedding
$\mathbf{\hat{s}}$. The consistency loss is designed to minimize the error
between the non-augmented embedding $\mathbf{s}$ and the augmented embedding
$\mathbf{s^{\prime}}$, thereby enabling the encoder to learn essential _task-
relevant_ features while ignoring irrelevant distractions (such as background
clutter or texture). This eliminates the need of using negative samples for
the computation of consistency loss. The consistency loss function can,
therefore, be mathematically written as:

$L_{c}(\mathbf{\hat{s}},\mathbf{s},\theta)=\mathbb{E}_{\mathbf{x}\sim\mathcal{D}}[\|\mathbf{\hat{s}}-\mathbf{s}\|^{2}]$
(5)

###  2.3 The CRC loss function

It is our conjecture that each of the above three loss functions enables the
encoder to extract non-redundant and complementary information from the higher
dimensional input. Thus, a combination of these three should improve the
overall RL performance in learning optimum policy. The resulting loss
function, called CRC loss, has the following mathematical form:

$L_{CRC}=c_{1}L_{q}+c_{2}L_{r}+c_{3}L_{c}$ (6)

where $c_{i}>0,\sum_{i}c_{i}=1,i=1,2,3$ are hyper-parameters that control the
relative importance of individual components. The RL model for policy learning
takes the query encoder output as its input. The SAC algorithm used for
learning policy is also allowed to affect the query encoder weights
$f_{\theta}$ during the backward gradient update step. At regular intervals,
the key encoder $f^{\prime}_{\theta}$ weights are updated using the
exponential moving average (EMA) of the weights of the query encoder
$f_{\theta}$. The feature learning and policy learning takes place in jointly
in parallel. The latent representations learned by the query encoder
$f_{\theta}$ receives gradients from both the CRC loss and the SAC algorithm
losses. This makes the feature representations _action-dependent_ , an aspect
which will be analyzed in some more detail in the next section.

##  3 Experimental Results and Discussions

The proposed CRC-RL model architecture takes its inspiration from the original
CURL implementation by Laskin et al. [10]. The original model is extended by
incorporating additional decoder and feature predictor to facilitate computing
the CRC loss function as described in the previous section. The model is
implemented using PyTorch [57] deep learning framework. The reinforcement
learning framework for policy estimation makes use of the publicly released
implementation of the SAC algorithm by Yarats et al. [58]. The query encoder
and decoder architecture is similar to the ones used in the above work. The
query encoder weights are tied between the actor and critic so that they both
use the same encoder to embed input image observations. The feature predictor
module is a MLP network which consists of cascaded linear layers and ReLU
activation function. The complete list of hyper-parameters is shown in Table
LABEL:tab:hyper_params. A number of experiments are carried out to establish
the efficacy of the proposed model. The design choices are justified through
several ablation studies as discussed below.

Table 1: Hyper-parameters used for DMControl experiments. Most hyper-
parameters values are unchanged across environments with the exception for
action repeat, learning rate, and batch size.

Hyper-parameter | Value
---|---
Pre transform image size | (100, 100)
Image size | (84, 84)
Action repeat | 8
Frame stack | 3
Transform | Random crop
Replay buffer capacity | 100000
Initial steps | 1000
Batch size | 512
Hidden layers | 1024
Evaluation episodes | 10
Optimizer | Adam
Learning rate $(f_{\theta},\pi_{\psi},Q_{\phi})$ | 1e-3
Learning rate $(\alpha)$ | 1e-4
Critic target update frequency | 2
Convolution layers | 4
Number of filters | 32
Latent dimension | 50
Discount $(\gamma)$ | 0.99
Initial temperature | 0.1

Table 2: Mean episodic reward (with standard deviation) over 10 evaluation
runs on DMControl environments after training for 100k environment steps. The
best scores are shown in bold letters.

100K Step | Our Method | CURL [10] | SODA [23] | PLANET [59] | DREAMER [60] | SAC+AE [17] | PIXEL | STATE | % Increase
---|---|---|---|---|---|---|---|---|---
Scores |  |  |  |  |  |  | SAC [45] | SAC [45] | over CURL
FINGER, SPIN | 793${\pm}$36 | 767${\pm}$56 | 363${\pm}$185 | 136${\pm}$216 | 341${\pm}$70 | 740${\pm}$64 | 179${\pm}$66 | 811${\pm}$46 | 3.38
CARTPOLE, SWINGUP | 813${\pm}$45 | 582${\pm}$146 | 474${\pm}$143 | 297${\pm}$39 | 326${\pm}$27 | 311${\pm}$11 | 419${\pm}$40 | 835${\pm}$22 | 39.6
REACHER, EASY | 636${\pm}$301 | 538${\pm}$233 | - | 20${\pm}$50 | 314${\pm}$155 | 274${\pm}$14 | 145${\pm}$30 | 746${\pm}$25 | 18.2
CHEETAH, RUN | 355${\pm}$31 | 299${\pm}$48 | - | 138${\pm}$88 | 235${\pm}$137 | 267${\pm}$24 | 197${\pm}$15 | 616${\pm}$18 | 18.7
WALKER, WALK | 490${\pm}$52 | 403${\pm}$24 | 635${\pm}$48 | 224${\pm}$48 | 277${\pm}$12 | 394${\pm}$22 | 42${\pm}$12 | 891${\pm}$82 | 21.5
BALL IN CUP, CATCH | 832${\pm}$81 | 769${\pm}$43 | 539${\pm}$111 | 0${\pm}$0 | 246${\pm}$174 | 391${\pm}$82 | 312${\pm}$63 | 746${\pm}$91 | 8.19

###  3.1 Performance Comparison

The performance of the proposed CRC-RL model is compared with the current
state-of-the-art methods on the challenging Deep mind control suite
(DMControl) environments [29]. The outcome is shown in Table 2 and 3 after
training for 100K and 500K environment steps respectively. It can be observed
that the proposed CRC-RL model outperforms the current state-of-the-art
methods, such as CURL [10], SODA [23], PlaNet [59], Dreamer [60], SAC+AE [17],
pixel-based SAC [45] on most of the DMControl environments, thereby
establishing the superiority of our approach. The environments shown in Table
3 are difficult compared to those shown in Table 2 and hence require longer
training time. In this case, our proposed model outperforms the baseline CURL
model in only 300K training steps.

Table 3: Mean episodic score (with standard deviation) for 10 evaluation runs
on DMControl environments obtained after training for 500k environment steps.
The best scores are shown in bold letters.

Environment | Our Method* | CURL | % Increase over CURL
---|---|---|---
QUADRUPED, WALK | $\mathbf{88\pm 51}$ | 39${\pm}$22 | 125.6
HOPPER, HOP | $\mathbf{61\pm 33}$ | 10${\pm}$17 | 510
WALKER, RUN | $\mathbf{306\pm 5}$ | 245${\pm}$32 | 24.8
FINGER TURN, HARD | $\mathbf{423\pm 78}$ | 207${\pm}$32 | 104.3

  * *

shows values for 300K training steps

###  3.2 t-SNE Visualizations

To better understand the relationship between the learned latent
representations and the action generated by the RL policy, we generate the
two-dimensional t-SNE plots of feature embeddings obtained from the query
encoder for 3 different environments as shown in Figure 2. These features are
assigned with the corresponding action labels generated by partitioning the
action space into five clusters by using the k-mean clustering algorithm. As
one can observe, the proposed CRC-RL model leads to more pristine clusters
with lesser outliers compared to the baseline CURL [10] algorithm for some
amount of training. Compared to the Cartpole environment, other two are
comparatively more complex and require larger amount of time for training.
This shows that the proposed model leads to better correlation between the
feature embeddings and agent actions. This aspect has not been empirically
investigated extensively in the existing literature and thus, the current work
makes a novel contribution by filling this void.

CURL |  |  |
---|---|---|---
CRC-RL |  |  |
|  (a) Cartpole-Swingup |  (b) Cheetah-Run |  (c) Walker-walk
Figure 2: t-SNE visualization of latent feature embeddings obtained from query encoder at 49K training steps. Colors correspond to cluster labels in the action space. One can observe that CRC-RL leads to more pristine clusters with less outliers compared to CURL. |  |
---|---|---
(a) | (b) | (c)
Figure 3: Effect of incorporating various loss components. CRC loss function
performs better than other combinations for more difficult environments such
as ’Cheetah-Run’ and ’Walker-walk’.

###  3.3 Feature Correlation Heat Maps

Another study is performed to validate our hypothesis that the proposed CRC
loss contribute new information resulting in learning new feature
representations which are distinct from those obtained using individual
losses. In this study, the correlation matrices between the latent features
obtained with the baseline CURL algorithm (that uses contrastive loss) and
that with the proposed CRC-RL model (that uses CRC loss) is plotted as heat-
maps as shown in Figure 4. These matrices are generated by collecting $200$
sample embeddings from both models trained with 49000 environment steps.
Since, each image sample is encoded into a $50\times 1$ feature vector, the
features generated by the above two methods are grouped into two feature
matrices (say, $F_{1}$ and $F_{2}$) of size $200\times 50$. The correlation
between these two feature matrices results in a $400x400$ matrix is visualized
as a heat map in the above figure where the darker regions shows higher
correlation and lighter regions show lower correlation. It is observed that
off-diagonal regions have lower correlation (lighter regions) indicating that
the two feature embeddings ($F1$ and $F2$ ) are very distinct from each other.
The diagonal regions are highly correlated (darker regions) as they correspond
to the features from the same method. Another interesting finding of this
study is that these heat maps show increasingly complex patterns for difficult
environments such as ’Walker-walk’ or ’Cheetah-Run’ compared to simpler
environments such as ’Cartpole-Swingup’. These patterns evolve over time and
stabilize as the training performance saturates. This is an interesting
insight that may provide clue to better understand the relationship between
features and action policies learned in an end-to-end RL framework.

|  |
---|---|---
(a) Cartpole-Swingup |  (b) Cheetah-Run |  (c) Walker-walk
Figure 4: Feature correlation heat-maps for three environments showing the
correlation between the latent features obtained with CURL and CRC-RL models.
The models are trained for 49K environment steps and 200 latent features are
used to generate this plot.

###  3.4 Ablation Study

Three separate ablation studies are carried out to justify the design choices
made in this paper as described below.

####  3.4.1 Usefulness of CRC loss function

First study validates the usefulness of the proposed CRC loss function
comprising of contrastive, reconstruction and consistency losses. The outcome
is shown in Figure 3. We start with the baseline CURL model [10] that uses
contrastive loss to learn the feature presentations. Then this model is
trained with a combined loss function of contrastive and reconstruction loss
and finally, with the CRC loss function comprising of contrastive,
reconstruction and consistency losses. The inclusion of these losses require
modifying the existing CURL model leading to the formation of the CRC-RL model
proposed in this paper. The figure shows that CRC-RL performs better than the
other two for the benchmark problems from the Deepmind control suite (DMC),
namely, ‘Cheetah-Run’ or ‘Walker-walk’. These two are comparatively difficult
problems to solve compared to simpler problems such as the ‘Cartpole-Swingup’
problem which does not benefit from the proposed CRC-RL model. This
observation clearly establishes the usefulness of the proposed approach. This
becomes more evident in the second ablation study discussed in the next
section below.

####  3.4.2 Relative weights of loss components in the CRC loss function

In this study, the relative weights of three loss components, namely,
contrastive, reconstruction and consistency loss, are varied and its effect on
the validation performance is compared as shown in Figure 5. The weights are
varied with the constraint of forming a convex sum. In other words,
$c_{i}>0,i\in{1,2,3}$ and $\sum_{i}c_{i}=1$. The figure shows that the
individual loss functions do not always provide the best performance. The best
performance is obtained by a combination of all the three losses. Having equal
weights ($c_{1}=c_{2}=c_{3}=0.33$) for all the three losses have a
regularizing effect on model performance in the sense that the performance is
traded for better generalizability. This is evident from the fact that the
validation performance curve with this combination of weights lie somewhere in
the middle of the all the curves. The generalization capability of the
proposed model is demonstrated in the third ablation study discussed next.

####  3.4.3 Generalization Capability of the CRC-RL model

In order to test the generalization capabilities of the proposed CRC-RL model,
another experiment is performed where the RL models are trained on images
augmented with _random crop_ effect and then validated on images augmented
with _Video-Easy_ and _Color-Hard_ artifacts [23]. The outcome is shown in
Figure 6. Comparing with the validation plots in Figure 5, it is observed that
the overall performance has come down significantly due to these complex
augmentations which makes it difficult for the model to generalize when
trained only with images augmented with only _random crop_ artefact. However,
even in this case, the RL model trained with CRC loss function provides the
best or closer-to-the-best evaluation performance compared to the models that
use individual loss components for training. This demonstrates the superior
generalization capability of the proposed CRC-RL model over the existing
models that use one of the above loss functions for training. It also
corroborates the earlier finding mentioned in the previous subsection that the
equal weights for individual loss components in the CRC loss function have a
regularizing effect on the model performance.

|  |
---|---|---
(a) | (b) | (c)
Figure 5: Effect of varying weighting parameters for different loss functions on the Evaluation performance. $c_{1}$, $c_{2}$ and $c_{3}$ are the weights to the contrastive loss, the reconstruction loss and the consistency loss respectively in the CRC loss function. The environments used are: (a) Cartpole-Swingup, (b) Cheetah-Run and (c) Walker-walk. Varying these parameters have a regularizing effect on the training performance. A smoothing factor of 0.5 is applied to the plot. |  |  |
---|---|---|---
|  |  |
(a) | (b) | (c) | (d)
Figure 6: Generalization capabilities of CRC-RL algorithm. $c_{1}$, $c_{2}$
and $c_{3}$ are the weights to the contrastive loss, the reconstruction loss
and the consistency loss respectively in the CRC loss function. The
environments used are: (a) Cartpole-Swingup, (b) Cheetah-Run and (c) Ball-in-
Cup-Catch and (d) Walker-walk. The RL models are trained on images with
_random-crop_ augmentation and evaluated on images with _Video-Easy_ and
_Color-Hard_ augmentations. Compared to the individual losses, CRC loss
provide best or second-best evaluation performance for these new augmentations
thereby establishing the superior generalization capabilities of the CRC-RL
model.

##  4 Conclusions

The paper addresses the problem of feature representation learning in end-to-
end reinforcement learning models with visual observations. Specifically, a
new loss function, called CRC loss, is proposed to learn action-dependent
features leading to superior performance in learning optimal action policies.
This loss function is a combination of three different loss functions, namely,
the image reconstruction loss, the contrastive loss and the consistency loss.
Through empirical analysis including latent feature visualization, an attempt
is made to generate new insights that can better explain the relationship
between the features being learnt and the actions being taken by the RL agent.
The resulting architecture is shown to outperform the existing state-of-the-
art methods in solving the challenging DMC problems by a significant margin
thereby forming a new benchmark in this field. The future work will involve
carrying out more in-depth analysis and evaluation of the individual loss
components on the overall performance as well as on the quality of features
being learned.

## References

  * [1] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al.  Human-level control through deep reinforcement learning.  nature, 518(7540):529–533, 2015.
  * [2] Shan Xue, Biao Luo, Derong Liu, and Ying Gao.  Event-triggered integral reinforcement learning for nonzero-sum games with asymmetric input saturation.  Neural Networks, 152:212–223, 2022.
  * [3] Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel.  End-to-end training of deep visuomotor policies.  The Journal of Machine Learning Research, 17(1):1334–1373, 2016\.
  * [4] Ahmed Hussain Qureshi, Yutaka Nakamura, Yuichiro Yoshikawa, and Hiroshi Ishiguro.  Intrinsically motivated reinforcement learning for human–robot interaction in the real-world.  Neural Networks, 107:23–33, 2018.
  * [5] Jiexin Wang, Stefan Elfwing, and Eiji Uchibe.  Modular deep reinforcement learning from reward and punishment for robot navigation.  Neural Networks, 135:115–126, 2021.
  * [6] Yutaka Nakamura, Takeshi Mori, Masa-aki Sato, and Shin Ishii.  Reinforcement learning for a biped robot based on a cpg-actor-critic method.  Neural networks, 20(6):723–735, 2007.
  * [7] Wei Yang, Xiaolong Wang, Ali Farhadi, Abhinav Gupta, and Roozbeh Mottaghi.  Visual semantic navigation using scene priors.  arXiv preprint arXiv:1810.06543, 2018.
  * [8] Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J Lim, Abhinav Gupta, Li Fei-Fei, and Ali Farhadi.  Target-driven visual navigation in indoor scenes using deep reinforcement learning.  In 2017 IEEE international conference on robotics and automation (ICRA), pages 3357–3364. IEEE, 2017.
  * [9] Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas.  Reinforcement learning with augmented data.  Advances in neural information processing systems, 33:19884–19895, 2020.
  * [10] Michael Laskin, Aravind Srinivas, and Pieter Abbeel.  Curl: Contrastive unsupervised representations for reinforcement learning.  In International Conference on Machine Learning, pages 5639–5650. PMLR, 2020.
  * [11] Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He.  A comprehensive survey on transfer learning.  Proceedings of the IEEE, 109(1):43–76, 2020.
  * [12] Tariqul Islam, Dm Mehedi Hasan Abid, Tanvir Rahman, Zahura Zaman, Kausar Mia, and Ramim Hossain.  Transfer learning in deep reinforcement learning.  In Proceedings of Seventh International Congress on Information and Communication Technology: ICICT 2022, London, Volume 1, pages 145–153. Springer, 2022.
  * [13] Yesmina Jaafra, Jean Luc Laurent, Aline Deruyver, and Mohamed Saber Naceur.  A review of meta-reinforcement learning for deep neural networks architecture search.  arXiv preprint arXiv:1812.07995, 2018.
  * [14] Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey.  Meta-learning in neural networks: A survey.  IEEE transactions on pattern analysis and machine intelligence, 44(9):5149–5169, 2021.
  * [15] Burr Settles.  Active learning literature survey.  2009\.
  * [16] Yoshua Bengio, Aaron Courville, and Pascal Vincent.  Representation learning: A review and new perspectives.  IEEE transactions on pattern analysis and machine intelligence, 35(8):1798–1828, 2013.
  * [17] Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus.  Improving sample efficiency in model-free reinforcement learning from images.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 10674–10681, 2021.
  * [18] Linus Ericsson, Henry Gouk, Chen Change Loy, and Timothy M Hospedales.  Self-supervised representation learning: Introduction, advances, and challenges.  IEEE Signal Processing Magazine, 39(3):42–62, 2022.
  * [19] Dor Bank, Noam Koenigstein, and Raja Giryes.  Autoencoders.  arXiv preprint arXiv:2003.05991, 2020.
  * [20] Daoyu Lin, Kun Fu, Yang Wang, Guangluan Xu, and Xian Sun.  Marta gans: Unsupervised representation learning for remote sensing image classification.  IEEE Geoscience and Remote Sensing Letters, 14(11):2092–2096, 2017\.
  * [21] Yuxin Peng and Jinwei Qi.  Cm-gans: Cross-modal generative adversarial networks for common representation learning.  ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 15(1):1–24, 2019.
  * [22] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In International conference on machine learning, pages 1597–1607. PMLR, 2020.
  * [23] Nicklas Hansen and Xiaolong Wang.  Generalization in reinforcement learning by soft data augmentation.  In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 13611–13617. IEEE, 2021.
  * [24] Chelsea Finn, Xin Yu Tan, Yan Duan, Trevor Darrell, Sergey Levine, and Pieter Abbeel.  Learning visual feature spaces for robotic manipulation with deep spatial autoencoders.  arXiv preprint arXiv:1509.06113, 25:2, 2015.
  * [25] Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin.  Decoupling representation learning from reinforcement learning.  In International Conference on Machine Learning, pages 9870–9879. PMLR, 2021.
  * [26] Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu.  Reinforcement learning with unsupervised auxiliary tasks.  arXiv preprint arXiv:1611.05397, 2016.
  * [27] Konstantia Xenou, Georgios Chalkiadakis, and Stergos Afantenos.  Deep reinforcement learning in strategic board game environments.  In European Conference on Multi-Agent Systems, pages 233–248. Springer, 2018.
  * [28] Behnam Neyshabur, Stephen Tu, Xingyou Song, Yiding Jiang, and Yilun Du.  Observational overfitting in reinforcement learning.  2020\.
  * [29] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al.  Deepmind control suite.  arXiv preprint arXiv:1801.00690, 2018.
  * [30] Richard S Sutton and Andrew G Barto.  Reinforcement learning: An introduction.  MIT press, 2018.
  * [31] Andrew G Barto.  Reinforcement learning and dynamic programming.  In Analysis, Design and Evaluation of Man–Machine Systems 1995, pages 407–412. Elsevier, 1995.
  * [32] Hado Van Hasselt, Arthur Guez, and David Silver.  Deep reinforcement learning with double q-learning.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016.
  * [33] Mohit Sewak.  Deep q network (dqn), double dqn, and dueling dqn.  In Deep Reinforcement Learning, pages 95–108. Springer, 2019.
  * [34] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller.  Playing atari with deep reinforcement learning.  arXiv preprint arXiv:1312.5602, 2013.
  * [35] Sean D Holcomb, William K Porter, Shaun V Ault, Guifen Mao, and Jin Wang.  Overview on deepmind and its alphago zero ai.  In Proceedings of the 2018 international conference on big data and education, pages 67–71, 2018.
  * [36] Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath.  Deep reinforcement learning: A brief survey.  IEEE Signal Processing Magazine, 34(6):26–38, 2017.
  * [37] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra.  Continuous control with deep reinforcement learning.  arXiv preprint arXiv:1509.02971, 2015.
  * [38] Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel.  Benchmarking deep reinforcement learning for continuous control.  In International conference on machine learning, pages 1329–1338. PMLR, 2016.
  * [39] Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine.  Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates.  In 2017 IEEE international conference on robotics and automation (ICRA), pages 3389–3396. IEEE, 2017.
  * [40] Hai Nguyen and Hung La.  Review of deep reinforcement learning for robot manipulation.  In 2019 Third IEEE International Conference on Robotic Computing (IRC), pages 590–595. IEEE, 2019.
  * [41] Deirdre Quillen, Eric Jang, Ofir Nachum, Chelsea Finn, Julian Ibarz, and Sergey Levine.  Deep reinforcement learning for vision-based robotic grasping: A simulated comparative evaluation of off-policy methods.  In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 6284–6291. IEEE, 2018.
  * [42] Shirin Joshi, Sulabh Kumra, and Ferat Sahin.  Robotic grasping using deep reinforcement learning.  In 2020 IEEE 16th International Conference on Automation Science and Engineering (CASE), pages 1461–1466. IEEE, 2020.
  * [43] Pengyu Yue, Jing Xin, Huan Zhao, Ding Liu, Mao Shan, and Jian Zhang.  Experimental research on deep reinforcement learning in autonomous navigation of mobile robot.  In 2019 14th IEEE Conference on Industrial Electronics and Applications (ICIEA), pages 1612–1616. IEEE, 2019.
  * [44] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu.  Asynchronous methods for deep reinforcement learning.  In International conference on machine learning, pages 1928–1937. PMLR, 2016.
  * [45] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al.  Soft actor-critic algorithms and applications.  arXiv preprint arXiv:1812.05905, 2018.
  * [46] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  arXiv preprint arXiv:1707.06347, 2017.
  * [47] Nicolò Botteghi, Mannes Poel, and Christoph Brune.  Unsupervised representation learning in deep reinforcement learning: A review.  arXiv preprint arXiv:2208.14226, 2022.
  * [48] Walter Hugo Lopez Pinaya, Sandra Vieira, Rafael Garcia-Dias, and Andrea Mechelli.  Autoencoders.  In Machine learning, pages 193–208. Elsevier, 2020.
  * [49] Antonin Raffin, Ashley Hill, René Traoré, Timothée Lesort, Natalia Díaz-Rodríguez, and David Filliat.  S-rl toolbox: Environments, datasets and evaluation metrics for state representation learning.  arXiv preprint arXiv:1809.09369, 2018.
  * [50] Sascha Lange and Martin Riedmiller.  Deep auto-encoder neural networks in reinforcement learning.  In The 2010 international joint conference on neural networks (IJCNN), pages 1–8. IEEE, 2010.
  * [51] Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer.  Revisiting self-supervised visual representation learning.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1920–1929, 2019.
  * [52] Roberta Raileanu, Max Goldstein, Denis Yarats, Ilya Kostrikov, and Rob Fergus.  Automatic data augmentation for generalization in deep reinforcement learning.  arXiv preprint arXiv:2006.12862, 2020.
  * [53] Wenling Shang, Xiaofei Wang, Aravind Srinivas, Aravind Rajeswaran, Yang Gao, Pieter Abbeel, and Misha Laskin.  Reinforcement learning with latent flow.  Advances in Neural Information Processing Systems, 34:22171–22183, 2021.
  * [54] Olivier Henaff.  Data-efficient image recognition with contrastive predictive coding.  In International conference on machine learning, pages 4182–4192. PMLR, 2020.
  * [55] Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  arXiv preprint arXiv:1807.03748, 2018.
  * [56] Partha Ghosh, Mehdi SM Sajjadi, Antonio Vergari, Michael Black, and Bernhard Schölkopf.  From variational to deterministic autoencoders.  arXiv preprint arXiv:1903.12436, 2019.
  * [57] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  Advances in neural information processing systems, 32, 2019.
  * [58] Denis Yarats, Ilya Kostrikov, and Rob Fergus.  Image augmentation is all you need: Regularizing deep reinforcement learning from pixels.  In International Conference on Learning Representations, 2020.
  * [59] Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson.  Learning latent dynamics for planning from pixels.  In International conference on machine learning, pages 2555–2565. PMLR, 2019.
  * [60] Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi.  Dream to control: Learning behaviors by latent imagination.  arXiv preprint arXiv:1912.01603, 2019.