#  On Transforming Reinforcement Learning by Transformer: The Development
Trajectory

Shengchao Hu,Li Shen, Ya Zhang, Yixin Chen, , and Dacheng Tao,  This work is
supported by Science and Technology Innovation 2030 –“Brain Science and Brain-
like Research” key Project (No. 2021ZD0201405). Shengchao Hu is with Shanghai
Jiaotong University, China. E-mail<EMAIL_ADDRESS>Li Shen is with JD
Explore Adacemy, China. Email<EMAIL_ADDRESS>Ya Zhang is with Shanghai
Jiaotong University, China. Email<EMAIL_ADDRESS>Yixin Chen is with
Washington University in St Louis, America. Email<EMAIL_ADDRESS>Dacheng
Tao is with JD Explore Adacemy, China and the University of Sydney, Australia.
Email<EMAIL_ADDRESS>Manuscript received April 19, 2005; revised
August 26, 2015.

###### Abstract

Transformer, originally devised for natural language processing, has also
attested significant success in computer vision. Thanks to its super
expressive power, researchers are investigating ways to deploy transformers to
reinforcement learning (RL) and the transformer-based models have manifested
their potential in representative RL benchmarks. In this paper, we collect and
dissect recent advances on transforming RL by transformer (transformer-based
RL or TRL), in order to explore its development trajectory and future trend.
We group existing developments in two categories: architecture enhancement and
trajectory optimization, and examine the main applications of TRL in robotic
manipulation, text-based games, navigation and autonomous driving. For
architecture enhancement, these methods consider how to apply the powerful
transformer structure to RL problems under the traditional RL framework, which
model agents and environments much more precisely than deep RL methods, but
they are still limited by the inherent defects of traditional RL algorithms,
such as bootstrapping and “deadly triad”. For trajectory optimization, these
methods treat RL problems as sequence modeling and train a joint state-action
model over entire trajectories under the behavior cloning framework, which are
able to extract policies from static datasets and fully use the long-sequence
modeling capability of the transformer. Given these advancements, extensions
and challenges in TRL are reviewed and proposals about future direction are
discussed. We hope that this survey can provide a detailed introduction to TRL
and motivate future research in this rapidly developing field.

###### Index Terms:

Reinforcement learning, Offline reinforcement learning, Decision transformer.

##  1 Introduction

Recently, transformer architecture has made substantial progress for natural
language processing (NLP) tasks [1]. For example, GPT series [2] and BERT
models [3] achieve state-of-the-art performance on a wide range of downstream
tasks (e.g. question answering, sentence classification, and so on). Inspired
by the success of transformer architecture in NLP, researchers also try to
apply transformers to computer vision tasks. Chen et al. [4] utilize the
transformer to auto-regressively predict pixels and ViT [5] directly applies
the transformer to sequences of image patches to classify the full image which
has achieved state-of-the-art performance on multiple image recognition
benchmarks. Considering the image as a sequence of sub-images [5, 6],
transformer architecture is able to extract representative features via the
self-attention mechanism [7, 8] and further can be used to generate more
powerful multi-modal visual-language models, such as the DALL-E [9], Flamingo
[10], and GATO [11].

Figure 1: A detailed overview of the transformers’ involvement in RL and
realistic application. The architecture enhancement block demonstrates the
specific part of transformer participation in traditional RL methods. The
trajectory optimization block shows the usage of the transformer and
corresponding research categories. Finally, the application block introduces
several realistic applications of TRL. (e.g. robotic manipulation, text-based
games, and so on.)

Reinforcement learning [12] (RL) is a powerful control strategy, which is
usually consisted of an environment and an agent. The agent observes the
current state of the environment, makes actions, and obtains the reward of the
current action and the state of the next moment, while the goal of the agent
is to maximize the cumulative reward it gets. With the success of deep neural
networks, deep RL has gradually become popular, which can make use of the
high-capacity capabilities of deep neural networks to make function
approximation methods more accurate, and enable agents to operate normally in
unstructured environments. However current deep RL mainly relies on
interacting with the environment to dynamically collect data, which limits the
amount of training data and may encounter difficulty when it is expensive to
interact with the environment, such as the robotic manipulation [13],
education [14], healthcare [15], and autonomous driving [16]. Thus offline RL
[17, 18] has attracted researchers’ interest which is a data-driven learning
method that can extract policies from static previously collected datasets
without interacting with the environment. Offline RL setting can make full use
of the ability of deep networks to extract the optimal policy from a large
amount of offline data, while it also needs to deal with the distributional
discrepancy between the offline training data and the target policy.

Due to the sequential decision process of RL, a natural idea is to apply
transformers to augment deep RL methods, which have been fully developed in
recent years [19, 20]. Initially, the transformer is usually used as an
alternative architecture for replacing CNN or LSTM in deep RL and is mainly
used to provide memory information for the agent network in environments where
the critical observations often span the entire episode. However, these
methods suffer from the instability issue empirically when directly applying
the original transformer structure to decision process [21], thus GtrXL [19]
firstly focuses on modifying the architecture to better adapt it to RL, and
motivates follow-on works to provide more powerful transformer architecture
[22, 23]. While DT [24] and TT [25] simultaneously treat the RL problem as a
conditional sequence modeling problem and leverage the transformer
architecture to model the sequential trajectory, which releases the powerful
long-sequence modeling capability of the transformer architecture and inspires
many following works. However, the current works enhance the performance of
transformer-based RL (TRL) from various perspectives, and there is a lack of a
unified understanding and summarizing for TRL. In order to keep pace with the
rate of new progress and promote the development of TRL, a survey of existing
works would be helpful for researchers in the community.

In this paper, we focus on providing a comprehensive overview of recent
advances in TRL and discussing the challenges and potential directions for
future improvement (shown in Figure 1). In order to facilitate future research
on different topics, we categorize the TRL by the perspective of the methods
and application, as listed in Table I, which includes architecture
enhancement, trajectory optimization, and applications.

For architecture enhancement, we mainly consider how to apply a more powerful
transformer structure to RL problems under the traditional RL framework.
According to the specific part of transformer participation, we further divide
it into feature representation and environment representation. Feature
representation mainly introduces the methods where the transformer is used to
extract feature representations from the multi-modal input and then the agent
utilizes these representations to make decisions through value- or policy-
based methods. Environment representation mainly illustrates how to leverage
the transformer architecture for the dynamics and reward modeling, which can
be used for better decision processes with additional planning algorithms.

For trajectory optimization methods, they all regard RL as a sequence modeling
problem and use a decision transformer to learn the policy under the behavior
cloning framework. According to the motivation and target tasks, we further
classify trajectory optimization methods as conditional BC, canonical RL,
pretraining, and generalization: 1) Conditional BC treats the RL as
conditional behavior cloning and adopts the transformer architecture to model
the state-action-reward sequence, including the pioneering works and their
conditions for optimality. 2) Canonical RL attempts to combine the advantages
of sequence modeling methods and traditional RL algorithms, which has
significantly improved the performance compared to the original algorithm. 3)
Pretraining focuses on how to pretrain the decision transformers for better
performance on downstream RL tasks, especially in terms of the pretraining
datasets and objectives. 4) Generalization mainly studies algorithms through
which a single agent is able to perform well in multiple environments which
may include the unseen one. Finally, we summarize four types of typical
applications of TRL, including robotic manipulation, text-based games, vision-
language navigation, and autonomous driving, each of them is highly related to
TRL and can be solved by RL methods. Towards the end of this paper, we
introduce the TRL in multi-agent RL extension, discuss several challenges and
provide insights on the prospects.

The rest of the paper is organized as follows. Section 2 introduces the
preliminaries of this survey, which includes a brief overview of RL and the
foundation of the standard transformer. Section 3 and 4 are the main parts of
the paper, in which we summarize the TRL methods and its applications,
respectively. Then in Section 5 we briefly describe TRL in the multi-agent RL
domain, an extension of single-agent RL, and discuss several future directions
and challenges in TRL. Finally, we give a summary to conclude this work.
Notice that in this survey we mainly include the representative works and some
recent preprint works on arXiv could be missed due to the rapid development of
this field.

Figure 2: The development trajectory of RL. At first, the RL agents use the
tabular Q-learning to make decisions at the current moment, then the DRL
agents begin to use the deep network to estimate the value and policy
functions, and finally, the TRL agents leverage the ability of transformer
architecture to evaluate the policy. There is a similar trend for environment
modeling in the model-based RL. TABLE I: Representative works of TRL

Category | Sub-category | Method | Highlights | Publication
---|---|---|---|---
Architecture Enhancement | Feature Representation | GTrXL [19] | Replace LSTM in V-MPO with Gated Transformer-XL | ICML 2020
WMG [26] | Transformer-based RL with factored observations | ICML 2020
RFWPs [22] | Add recurrence to the FWP | NeurIPS 2021
ALD [27] | Actor-Learner Distillation | ICLR 2021
COBERL [28] | Combine contrastive loss and a hybird LSTM-transformer | ICLR 2022
Environment Representation | IRIS [29] | World model composed of a autoencoder and a transformer | arXiv 2022
TransDreamer [30] | Transformer-based stochastic world model | arXiv 2022
VQM [31] | Latent representations transition model | ICML 2021
PlaTe [32] | Learn state and action spaces from videos | arXiv 2021
MINECLIP [33] | An automatic evaluation metric | arXiv 2022
Trajectory Optimization | Conditioned BC | DT[24] | Return-to-go, BC conditioning desired RTG | NeurIPS 2021
TT[25] | Modeling distribution of trajectory, beam search | NeurIPS 2021
ESPER [34] | Cluster trajectories, average cluster returns | arXiv 2022
RCSL [35] | Theoretical analysis for capabilities and limitations | NeurIPS 2022
Canonical RL | ODT[36] | Blends offline pretraining with online finetuning | ICML 2022
StARformer[37] | Markovian-like inductive bias | ECCV 2022
AD[38] | DT with in-context reinforcement learning | arXiv 2022
BooT[39] | Self-generate data | NeruIPS 2022
QDT[40] | Utilize Q function to relabel the RTG | arXiv 2022
Pretraining | ChibiT[41] | Pre-train on the Wikipedia | arXiv 2022
LID[42] | Pre-trained LMs as a general scaffold | NeruIPS 2022
Effect of Pre-training[43] | Investigate the effect of pre-training | NeruIPS 2022
MaskDP[44] | A pretraining method to learn generalizable models | NeurIPS 2022
ACL[45] | What representation matters in the offline pretraining | ICML 2021
Generalization | Multi-Game DT [46] | Power-law performance trend | NeuIPS 2022
Gato[11] | Multi-modal, multi-task, multi-embodiment model | arXiv 2022
Switch TT[47] | Sparsely activated model and trajectory value estimator | arXiv 2022
GDT [48] | Hindsight information matching generalization | ICLR 2022
FlexiBiT [49] | Randomly masking and predicting | NeurIPS 2022
Prompt-DT [50] | Trajectory prompt for generalization | ICML 2022
Transfer-DT [51] | Changes in the environment dynamics, causal reasoning | arXiv 2021
Application | Robotic Manipulation | T-OSIL [52] | Transformer for OSIL | CoRL 2021
LATTE [53] | Modify the trajectory based on the multi-modal transformer | arXiv 2022
Hiveformer [54] | History-aware instruction-conditioned multi-view transformer | CoRL 2022
PERACT [55] | Language goals with voxel observation | CoRL 2022
TTP [56] | Prompt-situation Transformer, multi-preference learning | arXiv 2022
Text-based Games | GATA [57] | Learn to construct and update graph-structured beliefs | NeurIPS 2020
Q*BERT [58] | Build a knowledge graph by answering questions | arXiv 2020
OOTD [59] | Model-based methods for TBGs | ICLR 2022
LTL-GATA [60] | Equip GATA with instruction-following capabilities | NeurIPS 2022
Navigation | PREVALENT [61] | Pretraining with finetuning for VLN tasks | CVPR 2021
VLNBERT [62] | Equip BERT model with recurrent function | CVPR 2021
HAMT [63] | Hierarchical transformer encoding long-range history | NeurIPS 2021
AVLEN [64] | Agent for audio-visual-language embodied navigation | NeurIPS 2022
Autonomous Driving | SPLT [65] | Disentangling the policy and world models | ICML 2022
BeT [66] | Modeling unlabeled demonstration data with multiple modes | NeurIPS 2022
TransFuser [67] | Fusion of intermediate features of the front view and LiDAR | CVPR 2021
InterFuse [68] | Safety-enhanced framework with multi-modal, view sensors | CoRL 2022

##  2 Preliminaries

In this section, we would go over the basic concept of reinforcement learning
and transformer, which is the preliminary knowledge of this survey.

###  2.1 Reinforcement Learning

Reinforcement learning (RL, refer to Figure 2 for the development trajectory
of RL) is learning to make the optimal action based on the current state at
every timestep, to maximize the numerical reward signal when each state-action
pair is assigned with a reward according to the definition of the problem. The
essence of RL is learning through interaction with environments. An agent
interacts with the environment and receives the corresponding reward, then the
agent could alter its behavior for better policy based on the interaction
consequence. So the agent is not guided by the "correct" actions (compared to
supervision learning) in most settings, but instead by the actions’ estimated
future cumulative reward, since an action may affect all subsequent rewards.
What makes the RL different and hard is the trial-and-error search and delayed
reward characteristics[12].

Below, we first illustrate the basic model formulation Markov decision process
and point out the essential property of it; then we detail the elements of RL,
giving out the most used component for a usual RL algorithm; finally, we
discuss the challenges faced in RL to describe what the latest RL research
focuses on.

####  2.1.1 Markov Decision Process

As a mathematical model of the sequential decision problem, Markov Decision
Process (MDP) is considered as the ideal modeling in dynamic environments and
provides the theoretical basis for the RL. To be specific, as shown in Figure
3, an agent is asked to make decision $a_{t}\in\mathcal{A}$ based on the
current state $s_{t}\in\mathcal{S}$, then the environment would respond to the
action made by the agent and transform to the next state
$s_{t+1}\in\mathcal{S}$ with the reward $r_{t}\in\mathbb{R}$.

Figure 3: Diagram of MDP. Observing the current state $s_{t}$, the agent would
make decision $a_{t}$, then the environment would transform to the next state
$s_{t+1}$ with the reward $r_{t}$.

Formally, A MDP can be defined by a 6-tuple notation
$\mathcal{M}=(\mathcal{S},\mathcal{A},\mathcal{T},\rho_{0},\mathcal{R},\gamma)$,
where $\mathcal{S}$ is the state space, $\mathcal{A}$ is the action space,
$T(s_{t+1}|s_{t},a_{t})$ denotes the transition probability from state $s_{t}$
to $s_{t+1}$ after taking action $a_{t}$, $\rho_{0}$ denotes the initial state
distribution, $R(s_{t},a_{t})$ denotes the reward value after taking action
$a_{t}$ at the state $s_{t}$ and $\gamma\in(0,1]$ denotes the discount factor.
Under the MDP setting, a trajectory is defined as the experience of the agent,
denoted by
$\tau=(s_{1},a_{1},r_{1},s_{2},a_{2},r_{2},\dots,s_{t},a_{t},r_{t})$, and a
policy is defined as a probability function $\pi(a_{t}|s_{t})$ which means the
probability of taking action $a_{t}$ at the state $s_{t}$. We are looking
forward to finding the optimal policy $\pi^{*}(a|s)$ that maximizes the
expected total reward for all the possible trajectories induced by the policy,
such that:

$\pi^{*}=\arg\max_{\pi}\mathbb{E}_{\tau\sim\pi}[\sum_{t=1}^{T}r_{t}].$ (1)

Note that in MDP, the environment’s dynamics are described by the transition
distribution $T(s_{t+1}|s_{t},a_{t})$, which means the environment is only
determined by the current state $s_{t}$ and action $a_{t}$, independent of the
history, which is called the Markov property. Thus the state information
should capture all the necessary information for the agent to make the
decision and the policy $\pi(a_{t}|s_{t})$ is only conditioned on the current
state.

There are also some cases in which the agent cannot access to the full-
observable state, which is defined as the partially-observable MDP (POMDP). In
the POMDP setting, the agent is only accessed to the local observation
$o_{t}\in\mathcal{O}$, where the distribution of the observation
$p(o_{t+1}|s_{t+1},a_{t})$ is dependent on the current state and the previous
action [69]. Though we can still apply MDP solving process to the policy
$\pi(a_{t}|o_{t})$ and assume the Markov property is also valid for the
observations, better modeling can be formulated as
$\pi(a_{t}|o_{t},o_{t-1},\cdots,o_{t-h+1})$ which leverages the history
information to mitigate the impact of not being able to access to the true
states. POMDP is a common modeling method in RL tasks and real environments,
such as text-based games, navigation, and so on.

####  2.1.2 Elements of Reinforcement Learning

Except for the agent and the environment, there are four main sub-elements of
a RL system: a policy, a reward function, a value function, and a model of the
environment based on the MDP definition[12]. The policy specifies the action
of the agent given the current state, which is the core of a RL agent since it
is the only knowledge source that guides the agent’s action. Whereas a reward
function is an immediate return after taking the action, a value function
specifies the total reward an agent can expect to accumulate over the future
given the current state. We seek the action that brings to the state with the
highest value, other than the highest reward because we wish to obtain good
final results which are correlated to the cumulative reward. A model of the
environment with a planning method helps the agent decide the action by
considering possible future situations.

There are three quantities of interest that are usually used to find the
optimal policy through different methods: action-value function
$Q^{\pi}(s_{t},a_{t})$, state-value function $V^{\pi}(s_{t})$ and their
difference known as the advantage function $A^{\pi}(s_{t},a_{t})$. State-value
function $V^{\pi}(s_{t})$ maps a state to the expected reward when starting
from state $s_{t}$ under the policy $\pi$, similarly action-value function
$Q^{\pi}(s_{t},a_{t})$ maps a pair of (state-action) to the expected reward.
The advantage function $A^{\pi}(s_{t},a_{t})$ is usually used as the lower-
variance to the action-value function to distinguish the impact of state and
action[70].

Based on these quantities, there are roughly three classes of methods: dynamic
programming, model-free, and model-based methods [18]. Dynamic programming is
used to compute the optimal policy given the known MDP. In model-free methods,
since not knowing the MDP, we use the sampling method instead. We can directly
optimize the policy through the policy gradient method (e.g. REINFORCE [71],
PG[72], NPG[73]), or use the value iteration method to conduct the policy from
the value function (e.g. DQN [74], Double DQN [75], Dueling DQN [70]), or use
the actor-critic methods which adopt the critic’s value to improve the actor’s
policy (e.g. DDPG [76], PPO [77], SAC[78]). For the model-based methods (e.g.
MOReL[79], MOPO [80], COMBO[81]), we learn a model of the environment that
simulates the MDP process, then we can either use the dynamic programming for
planning under the simulated MDP, or use the model-free methods by sampling
from the simulated MDP.

Recently offline RL has attracted many interests where an agent is to extract
return maximizing policies from some fixed limited datasets consisting of
trajectory rollouts of arbitrary policies [17]. Formally, the static datasets
are defined as follows: $\mathcal{D}=\\{(s_{t},a_{t},s_{t+1},r_{t})_{i}\\}$,
where $i$ is the index, the actions and states come from the distribution
induced by the behavior policy $(s_{t},a_{t})\sim d^{\pi_{\beta}}(\cdot)$, and
the next states and rewards are determined by the dynamics
$(s_{t+1},r_{t})\sim(T(\cdot|s_{t},a_{t}),r(s_{t},a_{t}))$. In contrast to
most online RL methods, offline RL makes itself possible for domains where it
is currently expensive to collect data online, such as healthcare[82],
inventory maintenance[83] and autonomous driving[65]. However offline RL is
harder since it cannot interact with the environments to explore new states
and transitions, thus RL approaches often need pessimistic value or policy
constraint to restrict access to the out-of-distribution phenomena, otherwise,
when encountering the unseen state, agents would make mistakes until the
policy diverges largely from the training policy which is called the
distribution shift.

####  2.1.3 Challenges in Reinforcement Learning

There are several basic challenges emphasized in RL[84]. Firstly the agent
needs to do a trial-and-error search to find the optimal policy by considering
the reward signal, however, it may cost a lot or be in danger when interacting
with the real-world environment actively. Then the state (or observation)
mainly depends on the agent’s action and may correlate with other agents’
actions (e.g. a car driving in the lane), and the state may correlate to the
history, which means the environment is not strictly following the MDP
formulation. Finally, the agent needs to deal with the long-term dependencies,
where the consequence of an action can only be shown through many transitions
of the environment, which is called the (temporal) credit assignment problem
[12].

###  2.2 Transformer Architecture

Transformer[85] is proposed as a network architecture for machine learning
tasks in the field of natural language processing (NLP) at first. The overall
structure diagram is shown in Figure 4, which consists of an encoder and a
decoder using stacked self-attention and point-wise, fully connected layers.
The encoder maps an input sequence of tokens to latent representations then
the decoder uses it to generate the desired results in an auto-regressive
manner, which consumes the previously generated results as additional input
for generating the next. In the following, we will describe each component of
the transformer in detail.

Figure 4: Structure of the original transformer (image from [85])

####  2.2.1 Self-Attention

Self-attention [86, 8] is the core component of the transformer, which models
pairwise relations between tokens. In order to calculate the self-attention,
the input token representation is linearly mapped into three vectors: the
query vector $q$, the key vector $k$, and the value vector $v$ with dimension
$d_{q}=d_{k}=d_{v}$. Then we pack vectors derived from different inputs
together into three different matrices: $Q,K,V$ and compute the attention map
as follows:

$\text{Attention}(Q,K,V)=\text{softmax}(\frac{QK^{T}}{\sqrt{d_{k}}})V.$ (2)

Eq.2 illustrates that we first compute the score between each pair of
different input vectors and then translate them into probabilities via softmax
operation, lastly we obtain the weighted sum of value where the weight is the
corresponding probability.

The self-attention layer in the encoder module and decoder module is similar
except that the layer in the decoder can only access positions less than the
current predicted position which is achieved by the masking technique. While
the encoder-decoder attention layer in the decoder module allows every
position in the decoder to query all positions in the input sequence, which is
achieved by the key matrix $K$ and value matrix $V$ coming from the encoder
module, and the query matrix $Q$ coming from the previous layer.

In order to boost the performance of the vanilla self-attention layer, a
multi-head attention technique is widely used which allows the model to
jointly compute the input representation tokens from different sub-spaces.
Especially, given the input vector and the number of head $h$, all inputs are
first linearly mapped into three groups of vectors and packed together: the
query group matrix $\\{Q_{i}\\}_{i=1}^{h}$, the key group matrix
$\\{K_{i}\\}_{i=1}^{h}$, and the value group matrix $\\{V_{i}\\}_{i=1}^{h}$.
In each group, there are $h$ vectors with dimension
$d_{q^{\prime}}=d_{k^{\prime}}=d_{v^{\prime}}=d_{model}/h$, and the multi-
attention process is shown as follows:

$\displaystyle\text{MultiHead}(Q^{\prime},K^{\prime},V^{\prime})=\text{Concat}(\text{head}_{1},\cdots,\text{head}_{h})W^{0},$
$\displaystyle\textbf{where}~{}\text{head}_{i}=\text{Attention}(Q_{i},K_{i},V_{i}),$
(3)

where the $Q^{\prime},K^{\prime},V^{\prime}$ are the concatenation of
corresponding group matrix $\\{Q_{i}\\}_{i=1}^{h}$, $\\{K_{i}\\}_{i=1}^{h}$,
$\\{V_{i}\\}_{i=1}^{h}$ and the $W^{0}$ is the projection weight.

####  2.2.2 Feed-Forward Network

A feed-forward network (FFN) is a position-wise fully connected network that
is applied after the self-attention layer. It consists of two linear
transformation layers with a non-linear activation function in between:

$\text{FFN}(X)=W_{2}\sigma(W_{1}X),$ (4)

where $W_{1}$ and $W_{2}$ are the learned parameters of the two linear
transformation layers, and $\sigma$ denotes the non-linear activation
function, such as the GELU[87], ReLU[88].

####  2.2.3 Positional Encoding

Since the input sequence is ordered and the preceding process is invariant to
the position, a positional encoding with dimension $d_{model}$ is added to the
original input embedding. Vanilla transformer positional embedding is encoded
as:

$\displaystyle PE_{(pos,2i)}$
$\displaystyle=\sin(pos/10000^{2i/d_{\text{model}}}),$ (5) $\displaystyle
PE_{(pos,2i+1)}$ $\displaystyle=\cos(pos/10000^{2i/d_{\text{model}}}),$ (6)

where $pos$ is position and $i$ is the dimension. In this way, each dimension
of the positional encoding corresponds to a sinusoid which makes it easy for
the model to learn to attend by relative positions and makes it possible to
extrapolate to a longer sequence length during inference. There are many other
choices of positional encodings, such as learned positional encoding [89] and
relative positional encoding[90].

####  2.2.4 Residual Connection

In order to promote the information flow from the input to the output, a
residual connection followed by a layer-normalization[91] is applied after
each layer in the encoder and decoder modules, shown in Figure 4. The above
process can be represented by the formula:

$\text{LayerNorm}(X+\text{Attention}(X)),$ (7)

where $X$ is the input to the self-attention layer. There are also other
variants of the residual connection, such as the pre-layer normalization (Pre-
LN)[92, 93] which put the layer-normalization before the attention layer, and
different normalization algorithms[94, 95].

##  3 Transformer-based RL

Transformer-based RL (TRL), whose goal is to leverage the powerful
representation ability of transformer architecture for better decision
processes in RL tasks, has attracted a lot of attention in recent years.
Initially, considering the success of the transformer architecture in domains
where the sequential information process is critical to performance, it is an
ideal candidate architecture for partially observable RL problems, where the
critical observations often span the entire episode, and currently RL
literature often uses the LSTMs architecture to provide memory to the agent.
These transformer-based methods are able to provide more representative
features for agents and environments due to the transformer’s superior
performance over LSTMs, but they only treat the transformer as a tool for
architecture enhancement and are still limited by the inherent defects of
traditional RL algorithms, such as bootstrapping and "deadly triad". While due
to the sequential decision process of RL, it is possible to directly involve
the transformer architecture in the decision-making process and get rid of the
limitations of traditional RL frameworks, which is achieved by treating the
sequential decision making progress of RL problems as the sequence modeling
process. Under the behavior cloning structure, the perspective of sequence
modeling is able to fully use the long-sequence modeling capability of the
transformer and avoid limitations of traditional RL, such as regularization,
conservatism, and the need for discounting future reward. These methods with
only a simple transformer structure are able to achieve the performance of
traditional RL algorithms with complex optimization processes, which arises
the interest of the RL community. Since then, many works on the decision
transformer have been proposed, studying how to further improve performance,
how to apply it to other tasks, and the reasons for the success and failure of
such methods, etc. In this section, we introduce the above two TRL methods
separately, and name them architecture enhancement (Subsection 3.1) and
trajectory optimization (Subsection 3.2) respectively according to the
characteristics of the algorithms and how the transformer architecture is
used, which is visualized in Figure 1.

###  3.1 Architecture Enhancement

In this subsection, we consider how to apply the powerful transformer
structure to RL problems under the traditional RL framework (shown in Figure
5). According to the specific part of transformer participation, we further
divide this subsection into two parts: feature representation and environment
representation. For the feature representation, we mainly introduce the
methods where the transformer is used to extract feature representations from
the multi-modal input and then the agent utilizes these representations to
make decisions through value- or policy-based methods. For the environment
representation, we mainly illustrate how to leverage the transformer
architecture for the dynamics and reward modeling, which can be used for
better decision processes with additional planning algorithms.

Figure 5: A generic framework for transformer in architecture enhancement.
Left: examples of using transformer architecture to enhance feature
representations [19, 23], right: examples of implementing transformer
architecture to enhance environment representations [30, 29].

####  3.1.1 Feature Representation

For the Partially Observable Markov Decision Processes (POMDPs) [96],
conditioning on simple observation is not enough to select the optimal action,
thus the agent often needs to consider the history of past observations.
Instead of directly conditioning the policy on the $N$ most recent
observation, using the gated RNNs such as LSTM [97] and GRU [98] to provide
memory to the agent is a better choice [99, 100]. While recent work
empirically demonstrates that transformer gains better performance in many
domains than the gated RNNs methods [101, 102], a natural question that how to
incorporate the transformer architecture into the RL domain sparks interest.
Motivated by factored representations [103] and factored MDPs [104], Loynd et
al. [26] introduce Working Memory Graph (WMG), a transformer-based agent which
takes the input a variable number of factor vectors and outputs features to
the actor-critic network, and the WMG is trained via the A3C algorithm [105]
with entropy-regularized policy gradient:

$\displaystyle\nabla_{\theta}\mathcal{J}(\theta)=\mathbb{E}_{\pi}[\sum_{t=0}^{\infty}\nabla_{\theta}\log\pi(a_{t}$
$\displaystyle|h_{t};\theta)A^{\pi}(Obs_{t},a_{t})$
$\displaystyle+\beta\nabla_{\theta}H(\pi(h_{t};\theta))],$ (8)

where $\pi(a_{t}|h_{t};\theta)$ is the policy head, $h_{t}$ is the output of
WMG, $H$ is the entropy and $\beta$ is the hyper-parameter. WMG also stores
the history information, called Memos, which is the basis of WMG’s shortcut
recurrence.

Concurrently, Parisotto et al. [19] try to leverage the transformer
architecture to solve the partially observable RL problems. Different from
WMG, Parisotto et al. modify the standard transformer architecture and XL
variant [101], called the Gated Transformer-XL (GTrXL), which substantially
improves the stability and learning speed (shown in Figure 6).

Figure 6: Transformer variants. Left: Standard Transformer(-XL) architecture.
Center: TrXL-I puts the layer normalization to the input stream of the
submodules. Right: GTrXL replaces the residual connection with the gating
layer.

Previous work has shown that the original transformer model suffers from
critical instability[21] and Parisotto et al. also find that it is difficult
to optimize the canonical transformer, which results in poor performance. Thus
they replace the residual connections with the gating mechanisms and move the
layer normalization onto the ”skip” stream of the residual connection which is
called Identity Map Reordering [106, 107]. The gating layer has many options,
such as highway connection [108], sigmoid-tanh (SigTanh) gate [109], Gated
Recurrent Unit (GRU) [98]. For the experiment on DMLab-30 [110], the GtrXL
transformer network is used to replace the LSTM-based agents in V-MPO [111]
algorithms, and the GRU-type gating layer performs best across all metrics.

Following GTrXL architecture which functions as a drop-in replacement to the
LSTM networks used in RL, there are many works designing more powerful memory
transformers for RL. For example, Rae et al. [23] build on TransformerXL [101]
and compress the old memories rather than discarding them, then they use the
compressive transformer as a drop-in replacement to an LSTM in the IMPALA
algorithms [99] which successfully compresses and makes use of past
observations. Irie et al. [22] notice that linear transformers are equivalent
to outer product-based FWPs [112] and original FWP formulation is more general
than some linear transformer, thus they augment FWP [113] with recurrent
connections, called recurrent FWPs (RFWPs) and use it as a drop-in replacement
to the IMPALA algorithm [99] which obtains powerful performance on standard
Atari 2600 setting [114] with small model size and promising scaling
properties for a larger model.

Compared to LSTM, one disadvantage of transformer is its significant
computation cost, which is important in actor-latency constrained settings.
Thus Parisotto et al. [27] develop an "Actor-Learner Distillation" (ALD)
procedure and use the limited capacity actor to collect data for large
capacity transformer learner learning, then optimize the actor through a
policy distillation loss:

$L_{ALD}^{\pi}=\mathbb{E}_{s\sim\pi_{A}}[\mathcal{D}_{KL}(\pi_{A}(\cdot|s)||\pi_{L}(\cdot|s))].$
(9)

For training, they use V-MPO [111] as the main RL algorithm and IMPALA [99]
distributed RL framework to parallelize acting and learning.

Although less able to capture long-range dependencies compared to the
transformer, LSTM has shown a strong ability on capturing recent dependencies
effectively. Banino et al. [28] propose a novel agent, named Contrastive BERT
for RL (CoBERL), which combines LSTMs with a transformer for architecture. By
putting a transformer before LSTM, the LSTM benefits from the transformer’s
ability to process long contextual dependencies, and the transformer can
reduce the memory size due to the existence of the LSTM [115]. Banino et al.
also combine the paradigm of masked prediction from BERT[3] with the
contrastive approach of RELIC [116] to improve the performance of agents. For
experiments on DeepMind Control Suite [117] and DMLab-30 [110], they evaluate
CoBERL both on the on-policy and off-policy setting, where the corresponding
RL algorithms are the V-MPO [111] and R2D2 [118] respectively, and optimize a
weighted sum of the RL objective and the contrastive loss.

####  3.1.2 Environment Representation

In order to to achieve high data efficiency, the model-based methods attract
attention. Recently, there are several ways to utilize the world model in RL:
pure representation learning [119], lookahead search [120], and learning in
imagination [121, 122]. Inspired by the success of Dreamer agent [122] which
belongs to the learning in imagination, Chen et al. [30] try to incorporate a
more powerful transformer to the RNN-based Dreamer and propose a transformer-
based model-based RL agent, called TransDreamer. They introduce the first
transformer-based stochastic world model, Transformer State-Space Model (TSSM)
by replacing the RNN in RSSM [121] with a transformer that is able to support
effective stochastic action-conditioned transitions and preserve the parallel
computation ability. Empirically, TransDreamer is more powerful than Dreamer
on tasks that require long-term and complex memory interactions, which
illustrates TSSM is better than RSSM at predicting the future.

In succession, Micheli et al. [29] also study the transformer ability in the
learning in imagination pattern. For the visually complex and partially
observable environment, Micheli et al. introduce Imagination with auto-
Regression over an Inner Speech (IRIS), an agent is trained on the imaginary
world which is generated by a discrete autoencoder and a GPT-like auto-
regressive Transformer. The autoencoder maps the raw pixels to a much smaller
amount of image tokens and the transformer simulates the environment dynamics
by casting dynamics learning as a sequence generation problem. During
training, the real interaction with the world is used to learn the environment
dynamics and the agents only learn from the imaginary, and the policy learning
opts for the learning objectives and hyper-parameters of DreamerV2 [122]. In
this way, IRIS outperforms a line of methods [123, 124] for sample-efficient
RL on the Atari 100k benchmark [125]. There are also several methods that use
autoencoders to encode the actions and states of the environment into discrete
latent variables and train a transformer-based transition model using the
latent variable. Ozair et al.[31] use Monte Carlo tree search (MCTS [126]) for
planning over future actions and observations, while Sun et al. [32] adopt the
beam search [127] to reduce the performance degradation.

A task-specific reward function is important for guiding the agents toward
specialists which is meticulously crafted by humans, however, sometimes it is
hard to define in the real-world or complex simulation environment. Fan et al.
[33] introduce MINECLIP which is a contrastive video-language model to provide
reward signal without any domain adaptation techniques for tasks in MINEDOJO
[33]. Formally, MINECLIP reward function can be defined as
$\Phi_{R}:(G,V)\rightarrow\mathbb{R}$ which maps the language goal $G$ and
current observation $V$ to a scalar reward. During training, it can be
optimized via the InfoNCE objective [128], which encourages the behavior in
the current snippet to follow the language description. MINECLIP achieves
strong performance and a high-reward snippet conforms to language
descriptions, which means MINECLIP is a good automatic metric for MINEDOJO
simulator.

###  3.2 Trajectory Optimization

Considering the RL problem as the conditional sequence modeling is first
proposed by the DT [24] and TT [25] simultaneously. Then following these
pioneering works, a series of new studies have been proposed (shown in Figure
7). In this subsection, we first give a brief introduction to the DT and TT
algorithms and introduce some works about the inner limitations and optimality
conditions of the DT algorithm. Next, we present some works that combine
sequence modeling methods with traditional reinforcement learning concepts,
which is a natural way to improve performance. Then we introduce the
pretraining mechanism, which is epidemic in NLP and CV, to the RL and
demonstrate the benefits of incorporating it into the training process.
Finally, we illustrate some algorithms through which a single agent is able to
perform well in multiple environments which may include the unseen one. We
point out that though these methods sometimes would trap in sub-optimum, the
usage of the transformer architecture and conditional sequence modeling
mechanism makes the RL agents have strong expansibility and generalization.

Figure 7: A generic framework for transformer in trajectory optimization. The
middle structure is a combination of Prompt-DT[50] and GDT[48]. A series of
works achieve different purposes by modifying the input form (the left part,
mainly for pretraining [41] and generalization [129]) and the structure of the
transformer (the right part, mainly to improve performance combining with
canonical RL [37] and increase generalization [47]) .

####  3.2.1 Conditioned BC

Chen et al. [24] recently propose Decision Transformer (DT) method for offline
RL which treats the policy learning process as a sequence modeling problem
(shown in Figure 8a).

(a)

(b)

Figure 8: (a) Decision Transformer architecture: States, actions, and RTGs are
input into the causal transformer which outputs the next timestep actions.
(b)Trajectory Transformer architecture: Each dimension of states, actions, and
RTGs is input into the causal transformer which outputs the next dimension of
states, actions, and RTGs.

In their formulation, they represent the trajectory as a sequence of state,
action, and returns-to-go (RTG) tuples collected at different time steps which
are amenable to auto-regressive training and generation:

$\tau=(\widehat{R}_{1},s_{1},a_{1},\widehat{R}_{2},s_{2},a_{2},\dots,\widehat{R}_{T},s_{T},a_{T}).$
(10)

The RTG is the cumulative rewards from the current time step till the end of
the episode $\widehat{R}_{t}=\sum_{t^{\prime}=t}^{T}r_{t^{\prime}}$. Instead
of the one-step reward $r_{t}$, RTG guides the model to generate actions
towards the future desired returns. During training, they sample trajectories
from the offline datasets and minimize the cross-entropy loss for discrete
actions or mean-squared error for continuous actions, arguing that predicting
the states and RTGs does not help improve the performance. During the
evaluation, they initiate the model with a specific target return based on the
desired performance on a given task as well as the environment state, then the
DT selects an action, observes the new state and reward, updates the RTG value
by subtracting the observed reward, and repeats until episode termination.
Though its training procedure is similar to behavior cloning, it is more
effective than simply performing imitation learning, performs well in sparse
settings, and avoids the stability issues related to bootstrapping for the
long-term credit assignment [130, 131].

Concurrently, Janner et al. [25] propose Trajectory Transformer (TT) which
also uses the transformer architecture to model the trajectory distributions,
but TT additionally models the state and reward transition over the trajectory
and discretizes each dimension independently, which incorporates model-based
components (shown in Figure 8b). In their formulation, assuming
$N$-dimensional states and $M$-dimensional actions, it represents the
trajectory as a sequence of state, action, and reward with discretization:

$\tau=(s_{t}^{1},s_{t}^{2},\dots,s_{t}^{N},a_{t}^{1},a_{t}^{2},\dots,a_{t}^{M},r_{t})_{t=1}^{T}.$
(11)

During training, they maximize the log-likelihood of each token from the
sequence in an auto-regressive manner. Once learning the trajectory
distribution, they use beam search algorithm[127] together with the RTG signal
to find the reward-maximizing behavior. Compared to the DT method, TT predicts
all the state, action, and reward tokens and uses an additional beam search
algorithm for planning, which can be seen as the model-based method, while DT
directly uses the predicted action for planning, which belongs to the model-
free methods, and supports the possibility that there is no need to include
components of traditional RL algorithms for a high-capacity sequence model
applying to RL problems.

Though methods such as DT which reduce RL to a prediction task and solve it
via supervised learning (RvS) become popular due to its simplicity and strong
overall performance on a set of tasks, Paster et al. [34] point out that in a
stochastic environment, the return reward corresponding to the trajectory is
unstable, therefore it may be failed when simply conditioning a probabilistic
model on the desired return and taking the predicted action. They show that
only when the training goals are independent of environment stochasticity,
these RvS methods would consistently achieve optimal policies. Thus Paster et
al. propose a method conditioned on the expected returns which is called
ESPER, short for environment-stochasticity-independent representations, to
meet the above condition. It first learns a discrete representation $I(\tau)$
acting as a cluster assignment for each trajectory, then a model learns to
predict the average return from the learned representation, which would be the
condition for a RvS agent to predict the final actions. Through this method,
ESPER is fully conditioned on outcomes that are determined by the actions of
the agents and independent of the stochasticity of the environment.

In succession, Brandfonbrener et al. [35] further provide a rigorous study of
the capabilities and limitations of the return-conditioned supervised learning
algorithms (RCSL). They find that the necessary assumptions for the RCSL to
obtain the near-optimal policy are more strict than those for classic DP
methods (e.g. Q-learning). In particular, RCSL requires nearly deterministic
dynamics, knowledge of the target return to condition, and conditioning value
to be consistent with the distribution of returns in the dataset. However
classic DP methods are capable of learning good policy when the dataset does
not contain high-return trajectories and the environment is stochastic. Paster
et al. think that though RCSL methods such as DT are simple and have strong
performance on many tasks, DP methods still act as the general learning
paradigm. However, the introduction of transformer structure and the training
paradigm of RCSL makes the RL agents have strong expansibility and
generalization, so that they can be used as general agents with good
performance in multi-task settings, which is hard for DP methods.

####  3.2.2 Canonical RL

Since the offline dataset is always limited and finite, RL methods usually
involve an online component, where policies are finetuned via task-specific
interactions with the environment, and online finetuning needs to acquire data
via exploration. Moreover, it has been observed that for standard online
algorithms, it is not helpful to access the offline data for the online
performance [132], hence it needs careful consideration to design an overall
pipeline. Zheng et al. [36] introduce Online Decision Transformer (ODT), a
learning framework for RL that incorporates online finetuning to the
pretraining DT algorithm. In order to balance the exploration-exploitation
trade-off, ODT explicitly imposes a lower bound on the policy entropy to
encourage exploration, which is motivated by the max-env RL frameworks [133]:

$\displaystyle\min_{\theta}J(\theta)$
$\displaystyle=\mathbb{E}_{(a,s,g)\sim\tau}[-\log\pi_{\theta}(a|s,g)],$
$\displaystyle\text{subject to }~{}~{}H^{\tau}_{\theta}[a|s,g]\geq\beta,$ (12)

where $\beta$ is a hyperparameter and $g$ is the RTG, and the policy entropy
$H^{\tau}_{\theta}[a|s,g]$ is defined at the level of sequences rather than
transitions which are different from the one for SAC [78]. Empirically, ODT
shows many extra significant gains during the finetuning procedure which
demonstrates the effectiveness and is competitive with the state-of-the-art
approaches on the D4RL benchmark [134]. Similarly, Li et al. [135] investigate
combining offline behavior cloning with online PPO [77] algorithm and focus on
sparse-reward goal-conditioned robotic manipulation problem. They adopt the
phasic approach by alternating online RL and offline SL with task reduction
and intrinsic reward to improve sample efficiency.

Moving away from online finetuning, how to incorporate the markovian-like
inductive bias into the transformer is also important. Since the sequences of
states, actions, and rewards generally have strong connections due to
potential causal relations, it may suffer from excess information or dilute
the essential relations when simply using a transformer attending to all
tokens. And it is also hard to learn the markovian-like dependencies between
tokens from scratch due to the weak relationship between non-adjacent tokens
[25]. To alleviate such issues, Shang et al. [37] propose the State-Action-
Reward Transformer (StARformer), which consists of two interleaving
components: a step transformer learning the local markovian representations by
self-attending a triple of state-action-reward, and a sequence transformer
modeling the long-term dependency. Notice that StARformer can directly operate
on the step-wise reward without performance degradation, while it is important
for DT to carefully design the RTG in the inference stage.

Afterward, Laskin et al. [38] indicate that the policy learned by the sequence
modeling methods does not improve incrementally from additional interaction
with the environment. They hypothesize that the reason these methods cannot be
improved through trial and error is that the training data does not show
learning progress: either the data contains no learning (e.g. collected by
fixed expert policies) or data with learning but with a context size that is
too small to capture the policy improvement. Thus, Laskin et al. present
Algorithm Distillation (AD) which learns an in-context policy improvement
operator on the learning history of an RL algorithm. The offline training
dataset needs to be large enough (i.e. across episodes) to capture the policy
improvement, then a causal transformer uses the learning history as its
context to model the actions, the learning loss function is as follows:

$\mathcal{L}(\theta):=-\sum_{n=1}^{N}\sum_{t=1}^{T-1}\log
P_{\theta}(A=a_{t}^{(n)}|h_{t-1}^{(n)},o_{t}^{(n)}).$ (13)

In particular, during training, AD first randomly samples a multi-episodic
sub-sequence of length $c$, then auto-regressively predicts the actions with
the transformer. During evaluation on DMLab Watermazer environment [136], AD
stores the sequential transitions to the context queue when unrolling the
transformer in the environment. Through this method, AD can learn new tasks
entirely in-context without updating its network parameters since all updated
information is contained in the historical context.

For the offline RL tasks, there usually exist two major limitations which
might cause sub-optimal training: one is that the offline dataset coverage is
limited which cannot guarantee to cover all the state-action-reward
transitions [137, 138], the other is the limited amount of training data which
may deteriorate the performance of model [139]. Except for training an
additional model learning environment dynamics to expand the data coverage,
Wang et al. [39] propose a novel algorithm, Bootstrapped Transformer (BooT)
which first generates data via the learned model itself and then uses the data
to further train the model for bootstrapping. Due to the ability of decision
transformer architecture, the data generated by the model is consistent with
the original offline data, thus avoiding the overfitting training [140]. To
stabilize bootstrapping training, BooT only trains on a part of the generated
trajectories with the highest confidence scores to prevent training with
inaccurately generated data, where the confidence is defined as:

$c(\tau)=\frac{1}{T^{\prime}(N+M+2)}\sum_{t=T-T^{\prime}+1}^{T}\log
P_{\theta}(\tau_{t}|\tau_{<t}),$ (14)

where $N,M$ are the dimensions of the state and action similar to TT,
respectively. It shows great effectiveness in the extensive experiments, which
are based on the D4RL offline dataset [134], but with high training time
consumption due to the pseudo data generation.

Similarly, Yamagata et al. [40] point out that if the offline dataset only
contains the sub-optimal trajectories, the DT method would fail to find the
optimal method which means it lacks the stitching ability - the ability to
learn the optimal policy from sub-optimal trajectories, contained by
conventional RL approaches (e.g. Q-learning). Thus Yamagata et al. propose the
Q-learning Decision Transformer (QDT) leveraging the Q-learning to improve DT
by improving the quality of the dataset, which is to relabel the RTG tokens in
the offline training data. In particular, since conservative Q learning (CQL)
[141] framework learns lower bounds of the true Q-functions values, they only
replace the RTG value when it is lower than the lower bound calculated by the
learned value function, and also replace all RTG tokens prior to the changed
RTG in order to keep consistency $R_{t}=r_{t}+R_{t+1}$. QDT is able to perform
well across all environments (e.g. Maze2D, Hopper, HalfCheetah, and Walker2d
in D4RL [134]), while both Q-learning and DT have poor performance in some
environments, which means QDT combines the strength of these approaches.

####  3.2.3 Pretraining

Towards sequence modeling, large pretraining models have made impressive
success in natural language process [3] and vision [5] tasks since pretraining
is an essential technique for alleviating high computation costs from using
expressive models such as transformers. Treating RL as sequence modeling
allows us to take advantage of the pretraining technique, however there lacks
of large off-the-shelf RL datasets for pretraining [142, 143]. Instead, Reid
et al. [41] demonstrate that pretraining on natural language can also provide
drastic improvements in convergence speeds and final policy performances on
the offline RL tasks that are unrelated to language (e.g. MuJoco [144]), which
provides a unified view of the sequence modeling domains. In order to better
adapt the pretrained language model to model trajectories, Reid et al. use a
similarity-based objective to maximize the similarity between the language
embeddings and the trajectories input representations:

$\mathcal{L}_{cos}=-\sum_{i=0}^{3N}\max_{j}\mathcal{C}(I_{i},E_{j}),$ (15)

where $E=[E_{1},\dots,E_{V}]$ are the language embeddings with vocabulary size
$V$, $I=I_{1},\dots,I_{3N}$ are the trajectories input tokens, and
$\mathcal{C}(z_{1},z_{2})=\frac{z_{1}}{||z_{1}||_{2}}\cdot\frac{z_{2}}{||z_{2}||_{2}}$
is the cosine similarity function, respectively. Reid et al. also show visual
initialization does not help performance improvement, and even has a negative
effect, which illustrates the large difference between image modeling and
trajectory modeling.

Following this work, Li et al. [42] further point out that using pretrained
language models as a general scaffold improves combinatorial generalization in
policy learning, especially on the out-of-distribution tasks. They also
perform several ablation experiments on BabyAI [145] and VirtualHome [146] and
show that there is no need to convert the states and actions history to
natural language strings to benefit from pretraining on the language, but the
sequential input encoding and pretraining on the language are important. While
Takagi [43] empirically investigates the impact of pretraining on data of
different modalities and studies what information is important when
pretraining on the language data. He shows that the transformer after
pretraining largely changes its representations and does not pay attention to
the downstream tasks. He also points out that the reason pretraining with
image data attains the bad performance [41] is the large gradients and
gradient clipping. And the reason why pretraining with language data performs
well may be that it makes the transformer get context-like information which
is used to solve the downstream tasks.

Concurrently, Liu et al. [44] randomly mask state and action tokens and
reconstruct these missing data, and they focus on solving the downstream tasks
via masked pretraining. They propose masked decision prediction (MaskDP) and
find that masked pretraining on unsupervised data leads to better general
agents. They also find that the mask ratios during pretraining are important
for downstream task performance since the states and actions in one sequence
are highly correlated temporally.

Yang et al. [45] also study the effect of pretraining for downstream tasks:
what unsupervised objectives are able to improve the performance of downstream
tasks, and what components of the objectives are most important? They leverage
the D4RL [134] offline dataset and mainly consider three categories for
downstream tasks: low-data imitation learning, offline RL, and online RL. And
they use the transformer-based architecture which is trained to predict the
randomly masking subset with a corresponding contrastive loss for pretraining,
then combined different algorithms for different downstream tasks, such as
BRAC [147] for offline RL and SAC [78] for offline RL. Results on Gym-MuJoCo
from D4RL [134] show that the use of pretraining with unsupervised learning
objective can improve the performance of policy learning, and the best
representation learning objective for pretraining depends on the downstream
task which means no one is optimal for all downstream tasks.

####  3.2.4 Generalization

It is hard for the traditional RL agents to build large-scale generalist
agents by training on massive datasets since they tend to use small models to
solve single tasks, or different tasks in one dynamics [148, 149]. Initially,
for the Atari suite [150], deep Q-learning [151] and actor-critic methods
[152] try to play all games with one algorithm, but they both require separate
training for each game. Then later works try to learn a single neural network
agent to play multiple Atari games simultaneously [99, 153, 154]. Lee et al.
[46] also try to learn a single agent for playing multiple Atari games. They
investigate several approaches, including decision transformer-based sequence
modeling (e.g. DT), online RL [74], offline temporal difference methods [141],
contrastive representations [155] and behavior cloning [156], and find that
decision transformer-based sequence modeling offers the best scalability and
performance. Since the training dataset usually contains expert and non-expert
trajectories, simply imitating the data cannot guarantee to produce expert
actions all the time. Inspired by the discriminator-guided generation problem
in language models [157, 158], Lee et al. propose an inference-time method
that assumes a binary classifier $P(\text{expert}^{t}|...)$ and use the
following Bayes’ rule to generate expert-level returns and actions:

$P(R^{t}|\text{expert}^{t},\dots)\propto
P_{\theta}(R^{t}|\dots)P(\text{expert}^{t}|R^{t},\dots),$ (16)

where the expert is proportional to future return
$P(\text{expert}^{t}|R^{t},\dots)\equiv\exp(\kappa R^{t})$, similarly to the
probabilistic inference [159, 160]. Experiment on 46 Atari games with existing
offline dataset [161] shows that the scaling rule is also adapted to the RL:
the larger size of the model, the better its performance and the faster it
adapts to new games via finetuning.

Concurrently, Reed et al. [11] aim to build a transformer-based generalist
agent based on offline data. They propose an agent, named Gato, to work as a
multi-modal, multi-task, multi-embodiment generalist policy that trains on the
widest variety of relevant data possible. In order to process the multi-modal
data through one agent, they first serialize all data into a flat sequence of
tokens, then use the transformer model to output the distribution over the
next discrete token through the following training loss:

$\mathcal{L}(\theta)=-\sum_{b=1}^{|\mathcal{B}|}\sum_{l=1}^{L}m(b,l)\log
p_{\theta}(s_{l}^{(b)}|s_{1}^{(b)},\dots,s_{l-1}^{(b)}),$ (17)

where $\mathcal{B}$ is the training batch sequences, and the masking function
$m(b,l)=1$ if the token at the index $l$ is from the text or logged action of
an agent. Notice that the training dataset is exclusively comprising near-
optimal agent experience in both simulated and real-world environments and
Gato needs the extra prompting information by expert trajectories to infer the
next token during evaluation [162, 163], which is different from the Lee et
al.’s work.

Similarly, Lin et al. [47] use the high-capacity sequential models to solve
the multi-task RL problem and indicate three significant algorithmic
challenges: high computation cost and model capacity, sharing parameters
causing a negative impact on single-task training, and the poor performance
caused by the Monte Carlo value estimator which suffers from poor sample
complexity. They propose the Switch Trajectory Transformer (SwitchTT), which
enhances two features of the TT method. First, it exploits a sparsely
activated model by replacing the FFN layer with a switch layer consisting of
multiple FFN layers and a router function to route the input to a specific
expert [164]. This modeling allows efficient computation and promotes
parameter sharing in multi-task learning. Second, it models the value
distribution of trajectory rather than the expected value to mitigate the
effect of the Monte Carlo value estimator and incorporate the uncertainty into
the reward. In practice, SwitchTT uses the categorical distribution and
minimizes the cross-entropy loss between the labeled class and discrete
distribution, and in the planning phase, it uses the weighted sum of the
discrete atoms as the value of each trajectory:

$V(x)=\sum_{i=1}^{N}p_{i}(x)z_{i},$ (18)

where $z_{i}$ is the discrete value and $p_{i}$ is the corresponding
probability given by the model, respectively. The multi-task training dataset
is composed of the replay buffer generated by the PPO [77] algorithm which
contains both the expert and non-expert trajectories. And the experimental
results on 10 tasks of the gym-mini-grid environment (e.g. FourRoom, DoorKey,
KeyCorridor) show that switchTT achieves comparable performance to TT while
saving 90% of the training time.

Furthermore, Furuta et al. [48] demonstrate that a lot of algorithms (e.g.
hindsight multi-task RL [165, 166], upside-down RL [167, 131]) conditioned on
future trajectory information can be unified as the hindsight information
matching (HIM), which trains policies to output the future tokens while
satisfying some statistics of future state information. They define the
information matching (IM) problems as to learn the conditional policy
$\pi(a|s,z)$ by minimizing the divergence between the statistics of the
generated trajectory and desired information $z$:

$\min_{\pi}\mathbb{E}_{z\sim
p(z),\tau\sim\rho_{z}^{\pi}(\tau)}[D(I^{\Phi}(\tau),z)],$ (19)

where $I^{\Phi}(\tau)$ denotes the information statistics of the trajectory
$\tau$. In practice, it can be the identity function, the reward function
$r(s,a)$, and so on. Furuta et al. accordingly introduce a generic template,
Generalized Decision Transformer (GDT), which adjusts its feature function and
anti-causal aggregator to adapt to different choices of $I^{\Phi}(\tau)$
(shown in Figure 9a).

(a)

(b)

Figure 9: (a) Structure of Generalized Decision Transformer, which is a
generalization of the DT structure including an anti-causal aggregator and a
feature function $\Phi$. (b) Structure of Prompt-DT, which takes the
trajectory prompt augmentation and the history sequence as input, and auto-
regressively predicts the actions corresponding to each state.

In succession, Carroll et al. [129] provide a unified view of different tasks,
such as BC, goal and waypoint conditioned imitation [168, 169], offline RL,
dynamics prediction [170, 171], initial-state inference [172] and more, as
sequence masking tasks. They propose a FlexiBiT framework which is trained on
a variety of inference tasks by a single agent. Since each task corresponds to
a unique masking scheme, in order to obtain a generalist agent which performs
well on any arbitrary sequence inference, FlexiBiT is trained by the random
masking scheme which masks each token independently with probability
$p_{\text{mask}}\sim\text{Uniform}(0,1)$. Such a training method enables the
FlexiBiT to perform well on any inference tasks, which is based on the mini-
grid environment framework [173], as the generalist agent, and even
outperforms specialized models that only train on the given tasks, which means
single-task models via unified multi-task training further improves the
performance.

Another difficulty of the generalization for RL is to generalize to unseen
tasks, a common method is through offline meta-reinforcement learning (e.g.
MAML [174], MACAW [175]) to achieve quick adaptation via algorithmic
designing. Inspired by the prompt-based framework for adaptation in NLP [176,
2], Xu et al. [50] propose the Prompt-based Decision Transformer (Prompt-DT)
which focuses on the power of architecture inductive bias. Instead of text
description as a prompt which needs large human labor to annotate [177], Xu et
al. adopt the trajectory segments as the prompt and force the agent to imitate
these demonstrations without finetuning (shown in Figure 9b). Further
experiments (e.g. Cheetah-dir [174], Ant-dir [178], Dial [143]) show that
given few-shot trajectories, Prompt-DT is able to beat strong meta offline RL
baselines and can generalized to out-of-distribution tasks.

Similarly, Boustati et al. [51] focus on training agents such that they are
robust to changes in the environment dynamics via causal reasoning. They
enrich the training dataset with the counterfactual trajectories to boost the
agent’s adaptation to structural changes. Mathematically, followed by
causality [179], Boustati et al. model the transitions as
$p(s^{\prime}|s,a,\Phi)$ with $\Phi$ denoting the structural features of the
environment, and one can generate a set of counterfactual trajectories by
intervening the $\Phi$. In order to train on more effective counterfactual
trajectories, they use the ATE measurement to order these trajectories:

$\text{ATE}[\phi_{CF}]=\sum_{t=1}^{T}\mathbb{E}_{p_{\text{CF}}(\tau)}[r_{t}]-\mathbb{E}_{p_{\text{Source}}(\tau)}[r_{t}].$
(20)

Empirical results on gym_minigrid environment suite [173] prove its efficacy
on the gym_minigrid environment suite [173].

There are also some works focusing on generalization from other perspectives,
such as AnyMorph [180] training a policy to generalize to new agent
morphologies without finetuning, AttentionNeuron [181] dealing with sudden
random reordering of inputs while performing a task, etc.

##  4 Applications of TRL

There are many specific applications for TRL. Here we mainly introduce four
popular environments, including robotic manipulation, text-based games,
vision-language navigation, and autonomous driving. In each of these fields,
agents usually need to process multi-modal input in a complex environment,
which is suitable to fully utilize the ability of the transformer
architecture. Below, we would describe the problem formulation of each field
and corresponding solutions with the help of transformer architecture.

###  4.1 Robotic Manipulation

For one-shot robotic imitation learning, it is important to consider which
pattern to represent the goal/intention from visual demonstrations and how to
incorporate it with current observations to the policy network. Dasari et al.
[52] explore task-driven features for the one-shot learning strategy and use
transformer (non-local self-attention block [182] in multi-head version ) to
extract relational features as the input state vector for the policy function.
The transformer takes as input both the features from demonstrations and
current observations via ResNet [183] process which is beneficial to preserve
spatial and temporal information in the attention module. However, in test
time the transformer features fail to weigh the important details, Dasari et
al. further add an unsupervised inverse dynamics loss which forces the
transformer to model the underlying dynamics. In particular, the context and
observations are first randomized and then passed to the representation model
to generate features $\tilde{\phi}_{t}$ which is used to predict a discretized
logistic mixture distribution [184]. The inverse loss is:

$\displaystyle\mathcal{L}_{inv}(\mathcal{D},\theta)$
$\displaystyle=-\ln(\sum_{i=1}^{k}\alpha_{k}(\tilde{\phi}_{t},\tilde{\phi}_{t+1})$
$\displaystyle\cdot\text{logistic}(\mu_{i}(\tilde{\phi}_{t},\tilde{\phi}_{t+1}),\sigma_{i}(\tilde{\phi}_{t},\tilde{\phi}_{t+1})).$
(21)

Results on multi-agent MuJoCo environments [144] show that both the network
design and the inverse loss can help policies perform better during test-time,
and further ablations show that transformer models are also more powerful than
LSTM with traditional "embedding level" attention mechanism. In succession,
Zhao et al. [185] point out that prior works [52, 186] tend to assume a very
strong similarity between train and test at the one-shot imitation learning
(OSIL) setting and introduce a new simulated robotic manipulation benchmark
and a new method MOSAIC. Compared to T-OSIL [52], MOSAIC uses the self-
attention-based policy network but adopts the temporal contrastive loss
objective [155] which is motivated by that the representations of two adjacent
frames should be as similar as possible and in other cases far away, i.e.,

$\mathcal{L}_{Rep}=\log\frac{\exp(q^{T}Wk_{+})}{\exp(q^{T}Wk_{+})+\sum_{i=1}^{F-1}(q^{T}Wk_{i})},$
(22)

where $q,k_{+}$ are the anchor and its positive, respectively, $W$ is the
projection matrix and $F$ is the total frame count. Experiments on Robosuite
[187] and MetaWorld [143] show that MOSAIC is powerful at learning efficiency
and performance, and is able to learn a promisingly general multi-task policy
via fine-tuning.

In the human-robot interaction setting, natural language is one of the most
intuitive ways to express human intent to the robot. However, it is not an
easy task to guide the robot to generate the target trajectory under human
instructions and commands. Rather than defining rigid templates with a static
set of actions and commands, Bucker et al. [188] provide a flexible language-
based interface where a user can reshape the existing robotic trajectory
according to the human commands. By treating the trajectory generation process
as a sequence modeling problem, it is natural to leverage the powerful
language model, transformer. Specifically, Bucker et al. first employ large
pretrained language models to encode the human instructions providing rich
semantic representations, and the pretrained language model can also lower the
requirement for the number of training examples. Then they align the
geometrical trajectory data with language embedding jointly with the use of
the multi-modal attention mechanism. Experiments show that the combination of
the pretrained large language model with the multi-modal transformers can
create more intuitive interfaces between robots and humans than traditional
methods such as kinesthetic teaching and programming obstacle avoidance.
Afterward, Bucker et al. [53] further enhance the architecture to include the
actual images encoder [189] and modeled the trajectory in 3D space with
velocity rather than just XY movement (shown in Figure 10).

Figure 10: The framework of LATTE (image from [53]), which includes a language
and image encoder, a geometry encoder, and a multi-modal transformer decoder.

Similarly, Sharma et al. [190] also use the transformer architecture model
encoding the instruction, but to create a semantic cost map that guides the
planner to produce trajectory.

Furthermore, Guhur et al. [54] focus on designing generic instruction-
following agents which produce actions other than modifying existing
trajectories and notice that recent works mainly rely on current observations
[191, 192] which may have a problem in sequential tasks requiring to track
object and remember previous actions, thus they introduce a new model,
Hiveformer, which jointly models the human instructions, multiple views and
history via a multi-modal transformer. The Hiveformer simultaneously encodes
the instructions, combining past and current visual observations and
proprioception, and predicts 7-DoF actions. Experiments on RLBench [193]
demonstrates the strong generalization ability of the Hiveformer whose history
input contributes to long-term tasks and multiple views input contributes to
the task requiring high precision or with occluded objects. Similarly,
Shridhar et al. [55] use a Perceiver transformer [194] to exploit the 3D
structure voxel patches and language goals to predict 6-DoF actions.

For task-specific settings, the robot needs to know the task constraints and
the dynamic environments. Jain et al. [56] think the robot should also adapt
to users’ different preferences and propose a prompt-situation architecture,
called TTP, which is able to generalize at test time to unseen preferences
with a single user demonstration. Assuming that the demonstrations implicitly
contain the task structure and preference, TTP takes as input one
demonstrations trajectory and current state, and learns to imitate the
preference in the demonstrations:

$\min_{m\sim M,\tau\sim D_{m},(S,a)\sim
D_{m}}\mathcal{L}_{CE}(a,\pi(S,\psi(\tau)))$ (23)

where $m$ is the preference, $D$ is the multi-preference dataset and $\tau$ is
the prompt contains state-action pairs. TTP successfully validates its
property both in the simulated kitchen environment and real-world Franka Panda
robotic arm.

A common use of the transformer in the robotic manipulation domain is to serve
as feature extraction modules for one or more modalities simultaneously. There
are still many works designing the transformer architecture to better extract
representations from visual/language/text input for downstream works, such as
gripper [195, 196], legged locomotion [197], dual-arm robot [198], and these
methods are mainly trained on BC pattern.

###  4.2 Text-based Games

Text-based games (TBGs) are interactive simulations where an agent plays a
game by processing text observations and acting text commands. Formally, TBGs
can be seen as partially observable MDP, the agents do not know the end goal
from the start and earn sparse rewards by completing a sequence of subgoals.
TBGs’ difficulty mainly depends on the environment’s complexity, game length,
and verbosity [199]. There are also several frameworks which are to promote
the development of TBGs, such as the TextWorld [199], LIGHT [200], Jericho
[201] and TextWorld with QA [202].

Standard RL algorithms are poor to explore the entire spaces effectively since
most games have over a billion possible actions per step [201, 203].
Ammanabrolu et al. [204] assume that there exists an oracle agent capable of
playing games perfectly and first use the question answering (QA) in TBGs
using a network to determine the next action, and the network is pretrained on
the experience of the oracle agent. They use the knowledge graphs as state
spaces for TBGs agents and then showcase the KG-A2C [203] which is able to
tackle the fully combinatorial action space for the first time. In order to
overcome the bottlenecks problem and improve the sample efficiency,
Ammanabrolu et al. [58] further propose the Q*BERT, which uses the pretrained
language model ALBERT [139] for question-answering. For correctly answering
the question, Q*BERT finetunes ALBERT on the SQuAD [205] and Jericho-QA
dataset which are specific to the text-game domain. Given the observation,
Q*BERT first produces a fixed set of questions which are sent to the
finetuning ALBERT, the predicted answers are then used to update the knowledge
graph as the current state to predict the next action. Compared to KG-A2C,
Q*BERT reaches asymptotic performance faster which indicates the QA system
leads to faster learning and thus improves the sample efficiency.

Concurrently, Adhikari et al. [57] point out that prior works highly rely on
the heuristic that exploits games’ inherent structure, for example, KG-A2C
[203] uses the hand-crafted heuristics to update the KG, thus they try to
design more general and effective representations in an autonomous way via
replacing heuristics with learning. Adhikari et al. propose the graph-aided
transformer agent (GATA), which learns to construct and update the graph-
structure beliefs in an entirely data-driven manner. GATA combines the
transformer version of LSTM-DQN with a dynamic belief graph which is used as
the long-term memory to improve the action selection by capturing the
underlying game dynamics. In particular, it consists of two modules: a graph
updater and an action selector. For the graph updater, it is pretrained using
two self-supervised training regimes. The first method involves a decoder
model to reconstruct text observations from the belief graph by viewing this
problem as sequence-to-sequence tasks and using the transformer-based model:

$\mathcal{L}=-\sum_{i=1}^{L_{O_{t}}}\log
p(O^{i}_{t}|O^{1}_{t},\dots,O_{t}^{i-1},\mathcal{G}_{t},A_{t-1})$ (24)

where $\mathcal{G}_{t}$ is the belief graph, $A_{t-1}$ is the previous action,
$O_{t}$ is the observation. The second method reformulates the first method as
a contrastive prediction task and learns to maximize the mutual information
between the predicted $\mathcal{G}_{t}$ and the text observations $O_{t}$. For
the Action selector, it encodes the belief graph $\mathcal{G}_{t}$ and text
observation $O_{t}$ into hidden representations, and aggregates them using a
bi-directional attention-based aggregator [206], which is trained via Double
DQN [75] method. GATA achieves great performance and outperforms baseline with
ground-truth graphs, which evinces the effectiveness of generating graph-
structured representations.

In succession, Tuli et al. [60] think that successful play also requires the
ability to follow instructions embedded within the text. Though instructions
may not be bound to reward directly, they can guide the actions of the agent
so as to achieve the ultimate goal as much as possible. However, GATA [57] is
not sensitive to the presence or absence of instructions and GATA usually
cannot complete tasks, thus Tuli et al. propose to equip GATA with the Linear
Temporal Logic (LTL) [207], a formal language which provides a mechanism to
monitor progress towards instructions completion. Specifically, Tuli et al.
first use the LTL to augment reward and define the episode termination. When
the LTL instruction is satisfied, a bonus reward is given, and a penalty is
given when the agent fails to complete an instruction. And if the LTL
instruction is violated, the agent would directly end in a terminal state even
if the environment has not indicated so. Then Tuli et al. augment the original
architecture to adapt to the LTL instructions, which includes the LTL encoding
of instructions and their in-game update process (shown in Figure 11).

Figure 11: The framework of LTL-GATA (image from [60]), which includes four
encoders and one action selector.

Finally, the overall policy can be defined as $\pi(a|o,g,\phi)$, where
$g,\phi$ are belief state and LTL instruction respectively. Experiments showed
that the LTL augmentation results in large performance improvements and more
effective instruction following and task completion.

Moving away from model-free methods, Liu et al. [59] focus on model-based
planning and try to learn the dynamics model from the TBGs which often
includes partially observable states with noisy text dynamics. Motivated by
the Object-Oriented MDP (OO-MDP) [208] that factorizes world states into
object states, Liu et al. design the Object-Oriented Text Dynamics (OOTD)
model, which includes graph representation for objects and independent
transition layers, to predict belief states without knowing rewards and
observations during planning. In particular, the OOTD models include a
transition model to predict the belief states of objects and a reward model
that maps sampled object states to rewards, both of them are learned under the
object-supervised setting and self-supervised setting. The transition model
first uses a graph decoder with the ComplEx scoring function [209] to map the
object states into memory graph $h_{t-1}$ given the states of K objects
$z_{t-1}=[z_{1,t-1},\dots,z_{K,t-1}]$, then encodes the $h_{t-1}$ into node
representations $e_{t-1}=[e_{1,t-1},\dots,e_{K,t-1}]$ with R-GCN [210].
Finally, to identify the affected objects given an action, OOTD uses a BIDAF
network [211] and a group of transition layers to predict the belief state of
objects by ICM framework [179]. The reward model is formalized as:

$p_{r}(r_{t}|z_{t},g_{t})=\psi^{r}[z_{t}^{pool},\psi^{g}(g_{t})],$ (25)

where $g_{t}$ is the goal and $\psi^{g}$ is a transformer-based text embedding
function and $\psi^{r}$ is implemented by MLP. Empirical results on Text-World
benchmark [199] demonstrated that OOTD-based planning methods are better than
model-free baselines in terms of sample efficiency and performance.

###  4.3 Navigation

Learning navigation in a photo-realistic environment with visual-linguistic
clues (VLN) has drawn increasing research interest [212], where the agents
need to understand the language instructions, perceive the surrounding
environment and perform navigation actions to reach the target goal. The VLN
task can be formulated as a partially observable MDP, where future
observations are only dependent on the current state and action of the agent,
and the visual observation only corresponds to partial instruction, requiring
the agent to memorize the navigation state and correctly localize the useful
information for current decision making. In order to promote the development
of the VLN field, there are also several datasets and simulators, such as
natural language guided navigation R2R [212] and Touchdown [213], dialog-based
navigation CVDN [214], VNLA [215] and HANNA [216], navigation for localizing
remote objects REVERIE [217] and so on.

Since each instruction only partially characterizes the trajectory and visual
state shares various relationships with language instructions, it may be
problematic for several methods [218, 219] which learn to understand the
instructions from scratch. Hao et al. [61] propose to pretrain an encoder to
align the representations of language instructions and visual states, which
can alleviate the ambiguity of instructions, called PREVALENT. Formally, the
instructions are represented as a set of $\mathcal{X}=\\{x_{i}\\}_{i=1}^{M}$,
and each instruction contains $L_{i}$ word tokens
$x_{i}=[x_{i,1},\dots,x_{i,L_{i}}]$, and the agent learns to navigate through
the policy $\pi$ given the training dataset $\mathcal{D}_{E}=\\{\tau,x\\}$
where $\tau$ is the expert trajectory:

$\max_{\theta}\mathcal{L}_{\theta}(\tau,x)=\log\pi_{\theta}(\tau|x)=\sum_{t=1}^{T}\log\pi_{\theta}(a_{t}|s_{t},x),$
(26)

where $\theta$ is the policy parameter. Hao et al. use two single-modal
encoders with a cross-modal encoder to construct the backbone which is all
based on the transformer, and propose two main tasks to pretrain the policy
network: Image-attended masked language modeling (MLM) and action prediction
(AP). MLM is similar to the pretraining in BERT which is to predict the masked
words based on the surrounding words by minimizing the negative log-
likelihood:

$\mathcal{L}_{MLM}=-\mathbb{E}_{s\sim p(\tau),(\tau,x)\sim\mathcal{D}_{E}}\log
p(x_{i}|x_{\backslash i},s),$ (27)

where the recovery process is also based on the visual states $s$. AP is to
predict the action without referring to the trajectory history by:

$\mathcal{L}_{AP}=-\mathbb{E}_{(a,s)\sim
p(\tau),(\tau,x)\sim\mathcal{D}_{E}}\log p(a|x_{[CLS]},s),$ (28)

where the $[CLS]$ is the position whose output is used to predict the action.
The full pretraining objective is :

$\mathcal{L}_{\text{Pretraining}}=\mathcal{L}_{MLM}+\mathcal{L}_{AP}.$ (29)

There are also other works that emphasize the importance of pretraining, for
example, PRESS [220] understands the language instructions through an off-the-
shelf BERT model, VLN-BERT [62] finetunes the ViLBERT [221] on instruction-
trajectory pairs to measure the compatibility, and there are also a lot of
methods [222, 223, 224] pretraining the transformer on a large number of
image-text pairs to learn generic cross-modal representations, known as V&L
BERT.

In succession, Hong et al. [225] point out that instead of using the
pretrained model only for encoding language [61, 220] or measuring
compatibility [62], they directly use the pretrained BERT model for learning
to navigate with modification for better adaptation to VLN. Hong et al.
propose the recurrent vision-and-language BERT for navigation which is able to
effectively use the pretrained V&L BERT and no longer requires additional
pretraining for the VLN task. They augment the original architecture with a
recurrent function to memorize the history representations and consider the
language tokens as keys and values but not queries in the self-attention
modules to reduce the computation consumption [226]. Experiments on R2R [212]
and REVERIE [217] shows that simple V&L BERT plus recurrent function is able
to replace the complex encoder-decoder models to achieve state-of-the-art
results.

Furthermore, Chen et al. [63] intend to explicitly encode the history instead
of using the recurrent states which might be sub-optimal for capturing
essential information for subsequent trajectory [227]. They propose the
History Aware Multimodal Transformer (HAMT), a fully transformer-based network
for VLN tasks. HAMT uses a cross-modal transformer to capture the long-range
dependencies of current observation and instruction from the history sequence,
and adopts a hierarchical version of the transformer to mitigate the
computation expenses, which progressively learns the representations of single
views, the spatial relationships among views within a panorama, and the
temporal relationships across different panoramas [228]. In order to learn
better visual representations, HAMT is also pretraining on several proxy
tasks, such as AP, MLM, spatial relationship prediction (SPREL), and
instruction-trajectory matching (ITM), the finetuning for sequential action
prediction with RL+IL objective:

$\displaystyle\theta\leftarrow\theta$
$\displaystyle+\mu\frac{1}{T}\sum_{t=1}^{T}\nabla_{\theta}\log\pi(\hat{a}_{t}^{h};\theta)(R_{t}-V_{t})$
$\displaystyle+\lambda\mu\frac{1}{T^{*}}\sum_{t=1}^{T^{*}}\nabla_{\theta}\log\pi(a_{t}^{*};\theta).$
(30)

Through this method, HAMT has strong performance both in seen and unseen
environments in all tasks. Similarly, Pashevich et al. [229] also propose a
multi-modal transformer model (E.T.) to encode the full episode history and
leverage the synthetic instructions as the interface between the human and the
agent to help the model learn more easily and generalize better.

Afterward, Paul et al. [64] consider a more complex and stochastic
environment, the audio-vision-language environment, which is also closer to
real-world navigation. They present AVLEN which guides the agent to localize
an audio source in a realistic visual world. The agents have to not only learn
the audio-visual information [230], but also seek help from an oracle for
navigation instructions in the form of natural language. It is important to
decide when to query the oracle and when to follow audio-visual cues to reach
the goal, Paul et al. propose a multi-modal hierarchical reinforcement
learning framework, which consists of a high-level policy to select which
modality to use for navigation based on the RL framework, and two low-level
policy corresponding to different modality based on the transformer
architecture (shown in Figure 12).

Figure 12: The framework of AVLEN (image from [64]), which is a combination of
query policy $\pi_{q}$, goal-based navigation policy $\pi_{g}$, and language-
based navigation policy $\pi_{l}$

Experiments on Matterport 3D simulator [212] enriched with audio events via
the SoundSpaces framework [231] show that the audio-visual-language modeling
agent leads to a clear improvement in performance, especially the existence of
oracle helps the agent deal with many challenging cases.

###  4.4 Autonomous Driving

For autonomous driving, the problem can be defined as point-to-point
navigation in an urban setting while keeping a safe distance from other
dynamic agents and following traffic rules. There are mainly two categories of
approaches to learning self-driving vehicles: supervised learning and
reinforcement learning. From the development of the driving simulator, CARLA
[232] and NoCrash [233], a series of works have been proposed, such as SAM
[234] mimicking driving methods, ST-P3 [235] with explicitly intermediate
representations for driving and NEAT [236] adopting the neural attention
fields for reasoning scene structure However these supervised learning methods
require large human-labeled driving data and lack robust interpretability, the
performance is also limited by the offline handcrafted trajectory, hence deep
RL methods attract researchers’ interest. For example, Roach [237] uses the
RL-based model to provide a demonstration for the IL agent, and Toromanoff et
al. [238] present a value-based Rainbow-IQN-Apex training method based on the
visual input.

The self-driving vehicle (SDV) usually takes as input both the images from the
cameras and the point cloud from the LiDAR sensors, a natural question is how
to integrate representations from the two or more modalities and maintain
their respective advantages, called sensor fusion. Prakash et al. [67] propose
the Multi-Modal Fusion Transformer (TransFuser) model which integrates
representations from different modalities through a self-attention transformer
and predicts the future way-points in an auto-regressive manner based on the
GRU model. TransFuser is trained under imitation learning where the policy is
learning to imitate the expert trajectory:

$\underset{\pi}{\text{argmin}}~{}~{}\mathbb{E}_{(\mathcal{X},\mathcal{W})\sim\mathcal{D}}[\mathcal{L}(W,\pi(\mathcal{X}))],$
(31)

where $\mathcal{W}=\\{(x_{t},y_{t})\\}_{t=1}^{T}$ is the expert trajectory,
$\mathcal{X}$ is the high-dimensional observations, loss function can be
$L_{1}$ loss, $L_{2}$ loss and so on. However, Shao et al. [68] think that the
TransFuser process harms sensor scalability and is limited to fusion between
LiDAR and a single-view image which may have a problem in fully perceiving the
surrounding environment. They further propose an Interpretable Sensor Fusion
Transformer (InterFuser), which is able to fuse information from multi-modal
multi-view sensors for comprehensive feature representations. What’s more,
motivated by humans’ information processing[239], Shao et al. output the
intermediate interpretable feature, called safety mind map, which provides
information about surrounding dynamic agents and traffic signs and is
exploited to constrain the actions within a safe action set by predefined
criteria. Extensive experiments on CARLA show that InterFuser achieves great
performance in complex and adversarial urban scenarios.

Towards the training dataset, algorithms need to learn behavioral priors from
the pre-collected data. One option is the offline RL, which learns a
conservative policy for task-specific behaviors, however, it cannot solve
problems where reward labels are not present. Thus behavior cloning is a more
fitting option [67, 68]. However these methods often make a fundamental
assumption: the data only consists of unimodal expert trajectory, while the
natural pre-collected data is often sub-optimal and contains multiple modes of
behavior. In order to train models that can naively learn multi-modal policy
behavior, Shafiullah et al. [66] present Behavior Transformers (BeT), which is
able to learn behaviors from distributionally multi-modal data. In particular,
BeT divides the continuous actions into two parts: a categorical variable
denoting an "action center" by k-means [240] and a corresponding "residual
action", then uses the transformer to map each observation to a categorical
distribution over $k$ discrete action bins with Focal loss [241]:

$\mathcal{L}_{focal}(p_{t})=-(1-p_{t})^{\gamma}\log(p_{t}),$ (32)

and an extra head to predict the offset with a loss akin to the masked multi-
task loss [242]:

$\text{MT-
Loss}(\mathbf{a},(\langle\hat{a}_{i}^{(j)}\rangle)_{j=1}^{k})=\sum_{j=1}^{k}\mathbb{I}[\lfloor\mathbf{a}\rfloor=j]\cdot||\langle\mathbf{a}\rangle-\langle\hat{a}^{(j)}\rangle||_{2}^{2},$
(33)

where the $\mathbb{I}[]$ denotes the Iverson bracket, ensuring the loss is
only incurred from the ground truth class of action $\mathbf{a}$. Experiments
on CARLA show that BeT is able to cover all the modes of demonstration data.

More recently, Villaflor et al. [65] try to apply sequence modeling RL methods
on stochastic safety-critical domains. However, these methods jointly model
the states and actions as a single sequencing problem which leads to overly
optimistic behavior and may be dangerous in the autonomous driving domain.
Thus Villaflor et al. develop a method called SeParated Latent Trajectory
Transformer (SPLT Transformer), to disentangle the effects of the policy and
world dynamics on the outcomes (shown in Figure 13).

Figure 13: The framework of SPLT (image from [65]), which includes a world
model to reconstruct the discounted returns, rewards, and states, and a policy
model to reconstruct the action sequence.

Following the prior work [243] on the autonomous driving domain, SPLT trains
two separate transformer-based discrete latent VAEs to represent the world and
policy model, which is able to produce a range of candidate behaviors for the
SDV and cover a majority of different potential response from the environment.
By leveraging the two models, SPLT conducts robust planning at the test time
with the intuition of picking a policy that will be robust to any realistic
possible future in the environment. Experiments on CARLA demonstrate that SPLT
is able to outperform these baseline approaches [24, 25, 244] on a variety of
autonomous driving tasks.

There are also many works in trajectory prediction leverage Transformer neural
networks [245, 246, 247] to model the long-range dependencies for prediction
in complex traffic situations. The transformer can also be used to model the
relationship between the SDV with other dynamic agents and the static elements
(e.g. lanes, stop lines, traffic light dynamic state) in the driving
environment [248, 249, 250], which is beneficial to consistent prediction with
the surrounding traffic. The VAE models [243, 247] are also usually
incorporated to facilitate covering the different possible modes of future
traffic trajectories.

###  4.5 Other Application

Automatic diagnosis is usually formulated as MDP and solved by RL methods for
policy learning [251, 252], however, these methods try to learn the symptom
inquiry and disease diagnosis policy with the max reward rather than the
doctor’s diagnostic logic directly, and there are no optimal reward functions
which make it hard to learn the policy. Considering the diagnosis process can
be seen as sequence generation which includes the symptoms and diagnoses, Chen
et al. [82] formulate the automatic diagnosis as a sequence modeling problem
leveraging the transformer architecture (Diaformer) and train it in an auto-
regressive manner. They also put forward an attention mask mechanism for the
automatic diagnosis: each explicit symptom can only see the explicit symptoms,
while each implicit symptom can see all the explicit symptoms and previously
generated implicit symptoms in an auto-regressive manner. In order to mitigate
the discrepancy between the order of symptoms sequence learned and the
disorder of realistic symptom inquiry, Chen et al. further put forward three
orderless training mechanisms: sequence shuffle, synchronous learning, and
repeated sequence. Extensive experiments on MuZhi [253], Dxy [254] and
Synthetic dataset [255] demonstrate the efficacy of the Diaformer model.

The maintenance problem is to make maintenance decisions that minimize costs
and keep the operators safe. There are several approaches that provide the
failure prediction and remaining useful life (RUL) estimation for human [256,
257], however many operators believe that it is not enough to make maintenance
decisions knowing the likelihood of failure in the future and RUL. Motivated
by the success of DT[24] and TT[25], Khorasgani et al. [258] try to solve
maintenance decision-making problems through offline RL methods to directly
generate the optimal maintenance actions such as "continuation" or "repair".
Given the history of observations, actions, and RTGs, Khorasgani et al. learn
to predict the action at each state. And it is common to develop reliable RUL
estimation models in industries, when the RUL estimation is available, it can
also be input into the decision model to improve accuracy and simplify the
training process, which is formulated as:

$\hat{a}_{k}=\texttt{model}(O_{k}^{T},A_{k-1}^{T},R_{k}^{H},RUL_{k}),$ (34)

where $T$ is the length of history window, $H$ is the future horizon.
Experiments on C-MAPSS datasets [259] show that using offline RL to directly
generate the optimal maintenance actions is competitive to using RUL
estimation.

##  5 Extension and Discussion

The transformer is becoming a hot topic in the field of RL due to its strong
performance and tremendous potential, as summarized in this survey. A natural
follow-up development direction is how to apply transformer-based methods to a
wider range of RL systems, such as multi-agent reinforcement learning, which
has been studied in several works. Nevertheless, the potential of transformer
for RL has not yet been fully explored, meaning that several challenges still
need to be resolved and several future prospects need to be considered. In
this section, we first introduce the transformer in multi-agent RL extension,
then discuss several challenges and provide insights on the future prospects.

###  5.1 Extension

The methods discussed above are all based on single-agent RL, a natural
extension is how to use transformers to solve multi-agent RL (MARL) [260] in
the form of sequence modeling. The first work is conducted by Meng et al.
[261] which studies the paradigm of offline pretraining with online finetuning
in the MARL domain and provides the large-scale datasets which are collected
from running the MAPPO [262] on the well-known SMAC task [263]. Treating the
decision making process as sequence modeling activates a novel pathway toward
training RL systems on diverse datasets which is important for the MARL
problem since online exploration may not be feasible in many settings [264].
Then Meng et al. propose Multi-Agent Decision Transformer (MADT) which casts
MARL as conditional sequence modeling through auto-regressive transformer
architecture (shown in Figure 14a).

(a)

(b)

Figure 14: (a) The detailed structure for MADT (image from [261]) which
includes offline pretraining and online finetuning. (b) The encoder-decoder
architecture for MAT (image from [265]) which inputs a sequence of agents’
observations and generates each agent’s optimal action in an auto-regressive
manner.

Since MARL includes the local state for each agent, they formulate the
trajectory as follows:

$\tau^{i}=(x_{1},\dots,x_{t},\dots,x_{T})~{}~{}~{}\text{where}~{}x_{t}=(s_{t},o_{t}^{i},a_{t}^{i}),$
(35)

where $s_{t}$ denotes the global state, $o_{t}^{i}$ denotes the local
observation for agent $i$ at time step $t$ and similar for $a_{t}^{i}$. And
they employ the Cross-entropy as the offline pretraining loss:

$L_{CE}(\theta)=\frac{1}{T}\sum_{t=1}^{T}P(a_{t})\log
P(\hat{a}_{t}|\tau_{t},\hat{a}_{<t};\theta),$ (36)

where $a_{t}$ is the ground truth action, $\tau_{t}$ includes
$\\{s_{1:t},o_{1:t}\\}$ and $\hat{a}$ denotes the output of MADT. After
pretraining on the offline dataset, it would get poor performance directly
using the above training method in the online environment, since the agent
would still try to select the actions which are similar to the offline dataset
even if they are bad actions. Thus Meng et al. treat the pretrained
transformer as the backbone of the actor and critic networks in the PPO
algorithm [77]. Experimental results on SMAC tasks show that pretrained MADT
enjoys high sample efficiency and generalisability enhancements.

Afterward, Wen et al. [265] propose Multi-Agent Transformer (MAT) which casts
cooperative MARL into sequence modeling from another perspective through an
encoder-decoder transformer architecture (shown in Figure 14b). Since the
agents’ interactions are important in multi-agent systems, it is not
guaranteed the joint performance would improve when endowing each agent with a
transformer policy and training independently [266]. Motivated by the multi-
agent advantage decomposition theorem [267] which transforms the joint policy
optimization into a sequential policy search process according to Eq.37, MAT
solves the problem through sequence modeling naturally and enjoys the
monotonic performance improvement guarantee.

$A_{\pi}^{i_{1:n}}(o,a^{i_{1:n}})=\sum_{m=1}^{n}A_{\pi}^{i_{m}}(o,a^{i_{1:m-1}},a^{i_{m}})$
(37)

What’s more, unlike DT adopts offline imitation learning, MAT is trained by
online trials and errors through PPO-likely objective. Extensive experiments
on SMAC, Multi-Agent MuJoCo[268], and Google Research Football [269]
demonstrate that MAT obtains superior performance and high data efficiency,
and is also an excellent few-shot learner.

In order to tackle the partial observability, gated RNNs are used to represent
memory information in the single-agent settings [270, 100] and so does for the
MARL settings [271, 263]. Since transformer achieves more powerful performance
than RNNs, Yang et al. [272] propose Agent Transformer Memory (ATM) network
which introduces the transformer-based working memory [26, 273] into MARL and
Entity-Bound Action Layer to incorporate the semantic inductive bias to the
action space. In particular, ATM first provides each agent with a memory
buffer to store past information and updates the buffer through a first-in-
first-out queue. Then ATM gives each action its own semantic meaning to
interact with different entities and uses the corresponding agents’ embedding
to compute the Q-values or action probabilities:

$Q=Q(h_{a},a),$ (38)

where $h_{a}$ is the embedding feature of the agent correlated to the action
$a$. For experiments, ATM is used as the agent network plugging into QMIX
[274] and QPLEX [275] algorithms, which are the most representative algorithms
on SMAC, and the results demonstrate ATM’s ability to speed up and improvement
in performance. However one of the limitations is that ATM needs to map
actions to different agents manually which may be hard or impossible in some
environments.

Furthermore, Hu et al. [276] develop a model, called Universal Policy
Decoupling Transformer (UPDeT), which is capable of handling multiple tasks at
a time. UPDet is based on Centralized training with decentralized execution
(CTDE) architecture and has no limitation on input or output dimension, which
makes zero-shot transfer possible. Each agent’s value function is based on the
transformer architecture and finally is mapped to a global Q-function through
the credit assignment function $F$:

$Q_{\pi}(s_{t},u_{t})=F(q_{1}^{t},\dots,q_{n}^{t}),$ (39)

where $u_{t}$ is the candidate actions, $q_{i}^{t}$ is the individual
Q-function for agent $i$ at time step $t$, respectively. UPDet further matches
the individual entity in observations to action groups and learns the
relationship between the matched entity and other observation-entity, which is
called policy decoupling and similar to the Entity-Bound Action Layer [272].
The final loss for optimizing the entire framework is the standard squared TD
error in DQN [74]:

$L(\theta)=\sum_{i=1}^{b}[(y_{i}^{DQN}-Q(s,u;\theta))^{2}],$ (40)

where $b$ represents the batch size and $y_{i}^{DQN}$ denotes the target in
DQN algorithm, respectively. Combining the transformer architecture and the
policy decoupling strategy, UPDet is able to be plugged into any existing
algorithms. Here, Hu et al. combine the UPDet with VDN [277], QMIX [274] and
QTRAN [278]. Experimental results on SMAC [263] demonstrate the transfer
capability in terms of both performance and speed.

###  5.2 Challenges

Although sequence modeling methods like DT [24] has achieved comparable
performance on a set of tasks, these works are only the first steps in the RL
domain and still have much room for improvement.

As shown in [34, 35], these methods have a high probability of failure in
stochastic environments, and many preconditions need to be satisfied in order
to reach the optimal policy, such as deterministic dynamics, known target
return, and the target return is corresponding to the distribution of the
dataset. Although Paster et al. [34] try to condition the expected return to
mitigate the negative effect of the uncertainty in the training dataset of the
stochastic environment, the results are still inferior to the traditional RL
algorithm and it is less appropriate to condition the model on the mean return
when the input to the agent is the visual goals. There is still a long way to
go in order to overcome these preconditions by changing the training method or
improving the transformer structure.

Then the transformer architecture used in most works still follows the
standard transformer for NLP [85], and a transformer structure designed
specifically for RL remains to be explored. There have been several works
trying to improve the architecture for better adaptation to RL. For example,
Parisotto et al. [19] modify the standard transformer architecture for more
stable and faster training, Shang et al. [37] try to incorporate the
markovian-like inductive bias into the transformer, and Villaflor et al. [65]
disentangle the policy and world dynamics generation for safer outcomes. These
improvements are based on a task-driven manner rather than designing a general
transformer architecture with high performance in RL, thus it is still an open
problem waiting to be solved.

Last but not least, developing efficient transformer models for RL remains an
open problem. The network used in the deep reinforcement learning algorithm
usually contains only a few layers of MLP structure, so it has a short
training time and testing time, but the transformer is usually large and
computationally expensive. For example, the transformer for generalization
task[46, 11] usually needs to train on large-scale data, and its structure
needs to be correspondingly very large to cover all possible situations.
Although some methods try to reduce the network structure through compression
or distillation [23, 27], they are still computationally expensive.
Consequently, efficient transformer models are urgently needed so that the
transformer-based RL methods can match the speed of traditional RL algorithms
and be able to deploy on resource-limited devices.

###  5.3 Future Prospects

In order to fully apply transformer capabilities to RL, we provide several
potential directions for further study.

Treating RL as sequence modeling problems simplifies many limitations in
traditional RL algorithms, but also loses their advantages. How to add the
advantages of traditional RL algorithms to the sequence modeling with a
transformer is a problem worth studying. At the same time, it is noted that
the original DT method [24] is based on RTG to model the future sequence. The
introduction of the RTG conditions makes DT unable to learn the optimal
results in many environments. How to correctly set the condition and what
function to use are all promising directions for future research. Finally, how
to conduct a pretraining process under the condition that the training dataset
is not all expert data is also a problem worth studying.

For traditional RL algorithms, the first problem that needs to be solved is
the computational consumption of transformers. Secondly, how to better use the
transformer structure to adapt to the memory module is worth studying. What’s
more, in addition to serving as a memory module (replacing the LSTM) under
POMDP conditions, what other roles the transformer can take on is also worthy
of further study. At the same time, it is mentioned in [38] that when the
training dataset contains learning data and is large enough, all reinforcement
learning algorithms can be imitated by the transformer structure, then it is
possible to design a general model that automatically switches the
corresponding traditional RL algorithm according to the given tasks, which has
not been studied yet.

For real-world applications, in addition to using the transformer structure as
a multi-modal multi-task feature extraction, it is also worth studying how to
apply the transformer structure to the decision network. And it is necessary
to fully understand the causes of errors and possible negative consequences
generated by transformers, which are critical to the security of practical
applications.

Note that the experimental environments used by different methods in this
survey are basically different, so it is difficult to have a unified benchmark
to intuitively compare different methods. Here we recommend that follow-up
research adopt a unified baseline and metric. For architecture enhancement,
these methods are usually evaluated in environments where the critic
observations often span the entire episode, thus the DMLab-30 environment
[110] is a better choice where memory information is important to make optimal
decisions for the agent. For trajectory optimization, since most methods are
based on offline RL, the D4RL dataset [134] is currently the most used
environment which has a large of offline trajectories. While in order to
verify the generalization, it is necessary to train a single agent for
different environments, where the Atari games suite and gym_minigrid
environment [173] are currently fully utilized since they both contain a lot
of different tasks. For different applications, each of them has its own
representative test environment, which would not be further introduced here.
In addition to the content introduced above, TRL still has great potential in
other fields and deserves to be fully studied.

##  6 Conclusion

In this survey article, we provide a comprehensive overview of the
transformers’ involvement in RL and realistic application. First, we give the
preliminary of RL, especially the offline RL, and the architecture of the
transformer. Then we review existing works on transformer in architecture
enhancement methods and trajectory optimization methods, and give a brief
overview of TRL in robotic manipulation, text-based games, navigation, and
autonomous driving domains, respectively. Finally, we share our perspective on
the open problems of the field, including possible domains for extension,
several potential challenges, and promising future directions for research.

## References

  * [1] P. M. Nadkarni, L. Ohno-Machado, and W. W. Chapman, “Natural language processing: an introduction,” _Journal of the American Medical Informatics Association_ , 2011.
  * [2] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell _et al._ , “Language models are few-shot learners,” _NeurIPS_ , 2020.
  * [3] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [4] M. Chen, A. Radford, R. Child, J. Wu, H. Jun, D. Luan, and I. Sutskever, “Generative pretraining from pixels,” in _ICML_. PMLR, 2020.
  * [5] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” _arXiv preprint arXiv:2010.11929_ , 2020.
  * [6] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in _ICCV_ , 2021.
  * [7] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” _arXiv preprint arXiv:1409.0473_ , 2014\.
  * [8] A. P. Parikh, O. Täckström, D. Das, and J. Uszkoreit, “A decomposable attention model for natural language inference,” _arXiv preprint arXiv:1606.01933_ , 2016.
  * [9] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever, “Zero-shot text-to-image generation,” in _ICML_. PMLR, 2021.
  * [10] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds _et al._ , “Flamingo: a visual language model for few-shot learning,” _arXiv preprint arXiv:2204.14198_ , 2022.
  * [11] S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y. Sulsky, J. Kay, J. T. Springenberg _et al._ , “A generalist agent,” _arXiv preprint arXiv:2205.06175_ , 2022.
  * [12] R. S. Sutton and A. G. Barto, _Reinforcement learning: An introduction_. MIT press, 2018.
  * [13] B. Singh, R. Kumar, and V. P. Singh, “Reinforcement learning in robotic applications: a comprehensive survey,” _Artificial Intelligence Review_ , 2022.
  * [14] A. Singla, A. N. Rafferty, G. Radanovic, and N. T. Heffernan, “Reinforcement learning for education: Opportunities and challenges,” _arXiv preprint arXiv:2107.08828_ , 2021.
  * [15] S. Liu, K. C. See, K. Y. Ngiam, L. A. Celi, X. Sun, M. Feng _et al._ , “Reinforcement learning for clinical decision support in critical care: comprehensive review,” _Journal of medical Internet research_ , 2020.
  * [16] B. R. Kiran, I. Sobh, V. Talpaert, P. Mannion, A. A. Al Sallab, S. Yogamani, and P. Pérez, “Deep reinforcement learning for autonomous driving: A survey,” _IEEE Transactions on Intelligent Transportation Systems_ , 2021\.
  * [17] S. Levine, A. Kumar, G. Tucker, and J. Fu, “Offline reinforcement learning: Tutorial, review, and perspectives on open problems,” _arXiv preprint arXiv:2005.01643_ , 2020.
  * [18] R. F. Prudencio, M. R. Maximo, and E. L. Colombini, “A survey on offline reinforcement learning: Taxonomy, review, and open problems,” _arXiv preprint arXiv:2203.01387_ , 2022.
  * [19] E. Parisotto, F. Song, J. Rae, R. Pascanu, C. Gulcehre, S. Jayakumar, M. Jaderberg, R. L. Kaufman, A. Clark, S. Noury _et al._ , “Stabilizing transformers for reinforcement learning,” in _ICML_. PMLR, 2020.
  * [20] M. Siebenborn, B. Belousov, J. Huang, and J. Peters, “How crucial is transformer in decision transformer?” _arXiv preprint arXiv:2211.14655_ , 2022.
  * [21] N. Mishra, M. Rohaninejad, X. Chen, and P. Abbeel, “A simple neural attentive meta-learner,” _arXiv preprint arXiv:1707.03141_ , 2017.
  * [22] K. Irie, I. Schlag, R. Csordás, and J. Schmidhuber, “Going beyond linear transformers with recurrent fast weight programmers,” _NeurIPS_ , 2021.
  * [23] J. W. Rae, A. Potapenko, S. M. Jayakumar, C. Hillier, and T. P. Lillicrap, “Compressive transformers for long-range sequence modelling,” in _ICLR_ , 2020.
  * [24] L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and I. Mordatch, “Decision transformer: Reinforcement learning via sequence modeling,” _NeurIPS_ , 2021.
  * [25] M. Janner, Q. Li, and S. Levine, “Offline reinforcement learning as one big sequence modeling problem,” _NeurIPS_ , 2021.
  * [26] R. Loynd, R. Fernandez, A. Celikyilmaz, A. Swaminathan, and M. Hausknecht, “Working memory graphs,” in _ICML_. PMLR, 2020.
  * [27] E. Parisotto and R. Salakhutdinov, “Efficient transformers in reinforcement learning using actor-learner distillation,” _arXiv preprint arXiv:2104.01655_ , 2021.
  * [28] A. Banino, A. P. Badia, J. Walker, T. Scholtes, J. Mitrovic, and C. Blundell, “Coberl: Contrastive bert for reinforcement learning,” _arXiv preprint arXiv:2107.05431_ , 2021.
  * [29] V. Micheli, E. Alonso, and F. Fleuret, “Transformers are sample efficient world models,” _arXiv preprint arXiv:2209.00588_ , 2022.
  * [30] C. Chen, Y.-F. Wu, J. Yoon, and S. Ahn, “Transdreamer: Reinforcement learning with transformer world models,” _arXiv preprint arXiv:2202.09481_ , 2022\.
  * [31] S. Ozair, Y. Li, A. Razavi, I. Antonoglou, A. Van Den Oord, and O. Vinyals, “Vector quantized models for planning,” in _ICML_. PMLR, 2021.
  * [32] J. Sun, D.-A. Huang, B. Lu, Y.-H. Liu, B. Zhou, and A. Garg, “Plate: Visually-grounded planning with transformers in procedural tasks,” _RAL_ , 2022.
  * [33] L. Fan, G. Wang, Y. Jiang, A. Mandlekar, Y. Yang, H. Zhu, A. Tang, D.-A. Huang, Y. Zhu, and A. Anandkumar, “Minedojo: Building open-ended embodied agents with internet-scale knowledge,” _arXiv preprint arXiv:2206.08853_ , 2022\.
  * [34] K. Paster, S. McIlraith, and J. Ba, “You can’t count on luck: Why decision transformers fail in stochastic environments,” _arXiv preprint arXiv:2205.15967_ , 2022.
  * [35] D. Brandfonbrener, A. Bietti, J. Buckman, R. Laroche, and J. Bruna, “When does return-conditioned supervised learning work for offline reinforcement learning?” _arXiv preprint arXiv:2206.01079_ , 2022.
  * [36] Q. Zheng, A. Zhang, and A. Grover, “Online decision transformer,” _arXiv preprint arXiv:2202.05607_ , 2022.
  * [37] J. Shang, K. Kahatapitiya, X. Li, and M. S. Ryoo, “Starformer: Transformer with state-action-reward representations for visual reinforcement learning,” in _ECCV_. Springer, 2022.
  * [38] M. Laskin, L. Wang, J. Oh, E. Parisotto, S. Spencer, R. Steigerwald, D. Strouse, S. Hansen, A. Filos, E. Brooks _et al._ , “In-context reinforcement learning with algorithm distillation,” _arXiv preprint arXiv:2210.14215_ , 2022.
  * [39] K. Wang, H. Zhao, X. Luo, K. Ren, W. Zhang, and D. Li, “Bootstrapped transformer for offline reinforcement learning,” _arXiv preprint arXiv:2206.08569_ , 2022.
  * [40] T. Yamagata, A. Khalil, and R. Santos-Rodriguez, “Q-learning decision transformer: Leveraging dynamic programming for conditional sequence modelling in offline rl,” _arXiv preprint arXiv:2209.03993_ , 2022.
  * [41] M. Reid, Y. Yamada, and S. S. Gu, “Can wikipedia help offline reinforcement learning?” _arXiv preprint arXiv:2201.12122_ , 2022.
  * [42] S. Li, X. Puig, Y. Du, C. Wang, E. Akyurek, A. Torralba, J. Andreas, and I. Mordatch, “Pre-trained language models for interactive decision-making,” _arXiv preprint arXiv:2202.01771_ , 2022.
  * [43] S. Takagi, “On the effect of pre-training for transformer in different modality on offline reinforcement learning,” _NeurIPS_ , 2022.
  * [44] F. Liu, H. Liu, A. Grover, and P. Abbeel, “Masked autoencoding for scalable and generalizable decision making,” _NeurIPS_ , 2022.
  * [45] M. Yang and O. Nachum, “Representation matters: offline pretraining for sequential decision making,” in _ICML_. PMLR, 2021.
  * [46] K.-H. Lee, O. Nachum, M. Yang, L. Lee, D. Freeman, W. Xu, S. Guadarrama, I. Fischer, E. Jang, H. Michalewski _et al._ , “Multi-game decision transformers,” _arXiv preprint arXiv:2205.15241_ , 2022.
  * [47] Q. Lin, H. Liu, and B. Sengupta, “Switch trajectory transformer with distributional value approximation for multi-task reinforcement learning,” _arXiv preprint arXiv:2203.07413_ , 2022.
  * [48] H. Furuta, Y. Matsuo, and S. S. Gu, “Generalized decision transformer for offline hindsight information matching,” _arXiv preprint arXiv:2111.10364_ , 2021.
  * [49] M. Carroll, O. Paradise, J. Lin, R. Georgescu, M. Sun, D. Bignell, S. Milani, K. Hofmann, M. Hausknecht, A. Dragan _et al._ , “Unimask: Unified inference in sequential decision problems,” _arXiv preprint arXiv:2211.10869_ , 2022.
  * [50] M. Xu, Y. Shen, S. Zhang, Y. Lu, D. Zhao, J. Tenenbaum, and C. Gan, “Prompting decision transformer for few-shot policy generalization,” in _ICML_. PMLR, 2022.
  * [51] A. Boustati, H. Chockler, and D. C. McNamee, “Transfer learning with causal counterfactual reasoning in decision transformers,” _arXiv preprint arXiv:2110.14355_ , 2021.
  * [52] S. Dasari and A. Gupta, “Transformers for one-shot visual imitation,” in _CoRL_. PMLR, 2021.
  * [53] A. Bucker, L. Figueredo, S. Haddadin, A. Kapoor, S. Ma, and R. Bonatti, “Latte: Language trajectory transformer,” _arXiv preprint arXiv:2208.02918_ , 2022.
  * [54] P.-L. Guhur, S. Chen, R. Garcia, M. Tapaswi, I. Laptev, and C. Schmid, “Instruction-driven history-aware policies for robotic manipulations,” _arXiv preprint arXiv:2209.04899_ , 2022.
  * [55] M. Shridhar, L. Manuelli, and D. Fox, “Perceiver-actor: A multi-task transformer for robotic manipulation,” _arXiv preprint arXiv:2209.05451_ , 2022.
  * [56] V. Jain, Y. Lin, E. Undersander, Y. Bisk, and A. Rai, “Transformers are adaptable task planners,” _arXiv preprint arXiv:2207.02442_ , 2022.
  * [57] A. Adhikari, X. Yuan, M.-A. Côté, M. Zelinka, M.-A. Rondeau, R. Laroche, P. Poupart, J. Tang, A. Trischler, and W. Hamilton, “Learning dynamic belief graphs to generalize on text-based games,” _NeurIPS_ , 2020\.
  * [58] P. Ammanabrolu, E. Tien, M. Hausknecht, and M. O. Riedl, “How to avoid being eaten by a grue: Structured exploration strategies for textual worlds,” _arXiv preprint arXiv:2006.07409_ , 2020.
  * [59] G. Liu, A. Adhikari, A.-m. Farahmand, and P. Poupart, “Learning object-oriented dynamics for planning from text,” in _ICLR_ , 2022.
  * [60] M. Tuli, A. C. Li, P. Vaezipoor, T. Q. Klassen, S. Sanner, and S. A. McIlraith, “Learning to follow instructions in text-based games,” _arXiv preprint arXiv:2211.04591_ , 2022.
  * [61] W. Hao, C. Li, X. Li, L. Carin, and J. Gao, “Towards learning a generic agent for vision-and-language navigation via pre-training,” in _CVPR_ , 2020.
  * [62] A. Majumdar, A. Shrivastava, S. Lee, P. Anderson, D. Parikh, and D. Batra, “Improving vision-and-language navigation with image-text pairs from the web,” in _ECCV_. Springer, 2020.
  * [63] S. Chen, P.-L. Guhur, C. Schmid, and I. Laptev, “History aware multimodal transformer for vision-and-language navigation,” _NeurIPS_ , 2021.
  * [64] S. Paul, A. K. Roy-Chowdhury, and A. Cherian, “Avlen: Audio-visual-language embodied navigation in 3d environments,” _arXiv preprint arXiv:2210.07940_ , 2022.
  * [65] A. R. Villaflor, Z. Huang, S. Pande, J. M. Dolan, and J. Schneider, “Addressing optimism bias in sequence modeling for reinforcement learning,” in _ICML_. PMLR, 2022.
  * [66] N. M. M. Shafiullah, Z. J. Cui, A. Altanzaya, and L. Pinto, “Behavior transformers: Cloning $k$ modes with one stone,” _arXiv preprint arXiv:2206.11251_ , 2022.
  * [67] A. Prakash, K. Chitta, and A. Geiger, “Multi-modal fusion transformer for end-to-end autonomous driving,” in _CVPR_ , 2021.
  * [68] H. Shao, L. Wang, R. Chen, H. Li, and Y. Liu, “Safety-enhanced autonomous driving using interpretable sensor fusion transformer,” _arXiv preprint arXiv:2207.14024_ , 2022.
  * [69] L. P. Kaelbling, M. L. Littman, and A. R. Cassandra, “Planning and acting in partially observable stochastic domains,” _Artificial intelligence_ , 1998\.
  * [70] Z. Wang, T. Schaul, M. Hessel, H. Hasselt, M. Lanctot, and N. Freitas, “Dueling network architectures for deep reinforcement learning,” in _ICML_. PMLR, 2016.
  * [71] R. J. Williams, “Simple statistical gradient-following algorithms for connectionist reinforcement learning,” _Machine learning_ , 1992.
  * [72] R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour, “Policy gradient methods for reinforcement learning with function approximation,” _NeurIPS_ , 1999.
  * [73] S. M. Kakade, “A natural policy gradient,” _NeurIPS_ , 2001.
  * [74] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski _et al._ , “Human-level control through deep reinforcement learning,” _nature_ , 2015\.
  * [75] H. Van Hasselt, A. Guez, and D. Silver, “Deep reinforcement learning with double q-learning,” in _AAAI_ , 2016.
  * [76] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” _arXiv preprint arXiv:1509.02971_ , 2015.
  * [77] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” _arXiv preprint arXiv:1707.06347_ , 2017\.
  * [78] T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel _et al._ , “Soft actor-critic algorithms and applications,” _arXiv preprint arXiv:1812.05905_ , 2018.
  * [79] R. Kidambi, A. Rajeswaran, P. Netrapalli, and T. Joachims, “Morel: Model-based offline reinforcement learning,” _NeurIPS_ , 2020.
  * [80] T. Yu, G. Thomas, L. Yu, S. Ermon, J. Y. Zou, S. Levine, C. Finn, and T. Ma, “Mopo: Model-based offline policy optimization,” _NeurIPS_ , 2020.
  * [81] T. Yu, A. Kumar, R. Rafailov, A. Rajeswaran, S. Levine, and C. Finn, “Combo: Conservative offline model-based policy optimization,” _NeurIPS_ , 2021.
  * [82] J. Chen, D. Li, Q. Chen, W. Zhou, and X. Liu, “Diaformer: Automatic diagnosis via symptoms sequence generation,” in _AAAI_ , 2022.
  * [83] H. Khorasgani, H. Wang, C. Gupta, and A. Farahat, “An offline deep reinforcement learning for maintenance decision-making,” _arXiv preprint arXiv:2109.15050_ , 2021.
  * [84] K. Arulkumaran, M. P. Deisenroth, M. Brundage, and A. A. Bharath, “Deep reinforcement learning: A brief survey,” _IEEE Signal Processing Magazine_ , 2017.
  * [85] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _NeurIPS_ , 2017.