# Towards Generalist Robot Learning from Internet Video: A Survey

Robert McCarthy  Daniel C.H. Tan  Dominik Schmidt  Fernando Acero  Nathan Herr
Yilun Du  Thomas G. Thuruthel  Zhibin Li

###### Abstract

This survey presents an overview of methods for _learning from video_ (LfV) in
the context of reinforcement learning (RL) and robotics. We focus on methods
capable of scaling to large internet video datasets and, in the process,
extracting foundational knowledge about the world’s dynamics and physical
human behaviour. Such methods hold great promise for developing general-
purpose robots.

We open with an overview of fundamental concepts relevant to the LfV-for-
robotics setting. This includes a discussion of the exciting benefits LfV
methods can offer (e.g., improved generalization beyond the available robot
data) and commentary on key LfV challenges (e.g., challenges related to
missing information in video and LfV distribution shifts). Our literature
review begins with an analysis of video foundation model techniques that can
extract knowledge from large, heterogeneous video datasets. Next, we review
methods that specifically leverage video data for robot learning. Here, we
categorise work according to which RL knowledge modality benefits from the use
of video data. We additionally highlight techniques for mitigating LfV
challenges, including reviewing action representations that address the issue
of missing action labels in video.

Finally, we examine LfV datasets and benchmarks, before concluding the survey
by discussing challenges and opportunities in LfV. Here, we advocate for
scalable approaches that can leverage the full range of available data and
that target the key benefits of LfV. Overall, we hope this survey will serve
as a comprehensive reference for the emerging field of LfV, catalysing further
research in the area, and ultimately facilitating progress towards obtaining
general-purpose robots.

††Correspondence to<EMAIL_ADDRESS>C.H. Tan is supported
by the Agency for Science, Technology, and Research (A*STAR), Singapore.
††Fernando Acero and Nathan Herr are supported by the UKRI CDT in Foundational
AI [EP/S021566/1] at the UCL Centre for Artificial Intelligence.

###### Contents

  1. 1 Introduction
  2. 2 Background
    1. 2.1 Reinforcement Learning
    2. 2.2 General Machine Learning
    3. 2.3 Related Surveys
  3. 3 LfV-for-Robotics: Preliminaries
    1. 3.1 The LfV Setting
    2. 3.2 Potential Benefits
    3. 3.3 Challenges
    4. 3.4 Evaluating LfV Methods
  4. 4 Towards Video Foundation Models
    1. 4.1 Video Encoders
    2. 4.2 Video Prediction Models
    3. 4.3 Video-to-Text Models
  5. 5 LfV-for-Robotics: Methods
    1. 5.1 Mitigating LfV Challenges
      1. 5.1.1 Recovering Action Information from Video
      2. 5.1.2 Representations to Address Distribution-Shift
    2. 5.2 Applications to RL Knowledge Modalities
      1. 5.2.1 Representation Transfer
      2. 5.2.2 Policies
      3. 5.2.3 Dynamics Models
      4. 5.2.4 Reward Functions
      5. 5.2.5 Value Functions
  6. 6 Datasets
    1. 6.1 Desiderata
    2. 6.2 Curating Video Data
    3. 6.3 Existing Datasets
  7. 7 Benchmarks
    1. 7.1 Designing Benchmarks for LfV
    2. 7.2 Existing Benchmarks
  8. 8 Challenges & Opportunities
    1. 8.1 High-level Recommendations
    2. 8.2 Video Foundation Models for LfV
    3. 8.3 Overcoming LfV Challenges
    4. 8.4 Other Generalist Robotics Challenges: Is Scaling Enough?
  9. 9 Conclusion

Figure 1: An overview of the key narratives, concepts, and taxonomies
contained in this survey on Learning from Videos (LfV) for generalist
robotics. The top green box presents the high-level motivation behind LfV
approaches. The middle orange boxes highlight the potential benefits LfV can
provide (Section 3.2) and the challenges in LfV (Section 3.3). The bottom blue
box visualises possible components in a pipeline for learning from large-scale
internet video, as per the taxonomies presented in the survey. Large internet
video datasets (Section 6) can be used to help pretrain (video) foundation
models (Section 4). These foundation models can further be adapted into
reinforcement learning (RL) ‘knowledge modalities’ and used in the robot
domain (Section 5.2). The diagram additionally highlights that action
representations (Section 5.1.1) can be used to mitigate the issue of missing
action labels in video.

##  1 Introduction

This survey is concerned with the goal of obtaining generalist robots. These
are robots capable of performing a diverse range of physical tasks in
unstructured real-world environments. Such robots would be highly useful and
have a wide range of commercial applications (e.g., household robots, factory
robots, or autonomous driving). Nevertheless, the generalist robot setting
presents several challenges. First, a generalist robot must be highly
competent. This includes maintaining capabilities ranging from the high-level
(e.g., reasoning and planning) to the low-level (e.g., dexterity and skill).
Second, to operate in unstructured settings, a generalist robot must rely on
imperfect partial observations (e.g., visual and tactile sensing) to perceive
the world.

How could we obtain such a robot? Classical robotics techniques are
insufficient as they typically rely on hand-crafted idealisations of physics
models and cannot usually handle unstructured and unseen scenarios [Krotkov et
al., 2018]. In contrast, machine learning (ML) techniques are more promising
and have been increasingly used in robotics, leading to the coining of the
term _robot learning_ [Argall et al., 2009, Peters et al., 2016, Kroemer et
al., 2021, Ibarz et al., 2021]. Now, it is commonly argued that progress in ML
is driven by improvements in data, algorithms, and computing power.
Fortunately, the cost of compute is ever decreasing [Moore, 1998, Mack, 2011],
and highly effective algorithms have recently been developed — including
expressive deep learning architectures, such as transformers [Vaswani et al.,
2017] and diffusion models [Ho et al., 2020] — whose performances consistently
and predictably improve with increased compute and data [Kaplan et al., 2020].
Combining these algorithms with massive, diverse datasets scraped from the
internet has led to remarkable improvements in language understanding and
generation [OpenAI, 2023], image generation [Betker et al., 2023], and, most
recently, video generation [Brooks et al., 2024].

Promisingly, these deep learning methods are transferable to robotics [Brohan
et al., 2022, Team et al., 2023b]. However, unlike other domains, robotics is
missing a key ingredient required for success: suitably large, diverse
datasets. Indeed, robotics faces a _chicken-and-egg_ problem. First, we cannot
easily collect real-world robot data due to the limited capabilities of our
robots. These limited capabilities mean deploying robots to collect data can
be ineffective and dangerous. Subsequently, we cannot easily improve our
robots due to the lack of data. Thus, arguably, data is currently _the_ key
bottleneck to progress in robotics.

How can we overcome this data bottleneck? To give insight into potential
solutions, we now briefly discuss the primary sources of data for robotics.
(1) _Real robot data:_ This is the exact data we want. With high-quality real
robot data, supervised learning or offline reinforcement learning (RL) can be
used to train our robot control policy. However, collecting real-world robot
data is expensive and difficult, regardless of whether collection is performed
by human teleoperation or an automated policy. (2) _Simulated robot data:_
Compared to real-world data collection, simulated collection is significantly
faster and cheaper [Kaufmann et al., 2023a]. However, simulation comes with a
number of issues. Simulated physics can be inaccurate. Additionally, creating
a diversity of simulated environments and tasks suitable for training a
generalist policy is not trivial. Moreover, we still often lack access to an
automated policy capable of collecting the simulated data. (3) _Internet
data:_ The internet is a massive and diverse data source. It has laid the
foundation for recent progress in deep learning [OpenAI, 2023, Betker et al.,
2023]. Internet text, image, and video data contain vast amounts of
information relevant to a generalist robot. However, internet data is not
trivially or directly applicable to robotics. This is due to distribution
shifts between internet data and the robot domain, and crucial missing
information in internet data (e.g., text contains no visual information,
whilst video contains no action labels).

Now, while any practical approach may utilise all three of the above data
sources, in this paper we focus on learning from internet data. Specifically,
our interest lies in _internet video data_. Our reasoning here is threefold.

  1. 1.

Information content of internet video. The information content of video is
highly relevant to generalist robotics. Compared to text or image data, video
can uniquely offer information about the physics and dynamics of the world,
and information about human behaviours and actions [Yang et al., 2024]. Like
images, video can provide visual and spatial information. Meanwhile, video can
ground knowledge and concepts learned from text data. Crucially, internet
video has excellent coverage over many behaviours and tasks relevant to a
generalist robot. For example, there are many videos of humans performing
household chores, which would be relevant to a general-purpose household
robot.

  2. 2.

Quantity and diversity of internet video. There are _huge_ quantities of video
data freely available on the internet (e.g., YouTube alone contains over ten
thousand years of footage [Sjöberg, 2023]). Importantly, this data is highly
diverse. The largest open-source robot dataset [Padalkar et al., 2023] pales
in comparison (see Figure 9), both in terms of quantity and diversity of the
data.

  3. 3.

Internet video is relatively untapped. The use of real and simulated robot
data has been extensively explored [Brohan et al., 2022, Team et al., 2023b,
Akkaya et al., 2019, Kaufmann et al., 2023a]. Meanwhile, internet text and
image data have been heavily exploited to train foundation models [OpenAI,
2023, Betker et al., 2023], and leveraging these text and image foundation
models for robotics has become increasingly common [Ahn et al., 2022, Liang et
al., 2023, Brohan et al., 2023, Shah et al., 2023]. However, the use of
internet video data is in more nascent stages, both in the general ML
literature, and in robotics. As such, there are opportunities for rapid
progress to be obtained via research focused on utilising internet video data.

Given its abundant quantities and relevant content, internet video data can
help mitigate the data bottleneck issue in robotics, and drive progress
towards creating generalist robots. More specifically, we hope to obtain the
following benefits from internet video: (1) improved generalization beyond the
available robot data, (2) improved data-efficiency and performance in-
distribution of the robot data, and, speculatively, (3) obtain emergent
capabilities not attainable from robot data alone. Indeed, recent progress in
the emerging field of LfV has been promising, demonstrating evidence of these
benefits. This has included works that leverage large-scale video prediction
models to act as robot dynamics models [Yang et al., 2023c, Bruce et al.,
2024], or works that leverage both robot data and internet video to train
foundational robot policies [Sohn et al., 2024].

Nevertheless, utilising internet video for robotics comes with a number of
fundamental and practical challenges. First, in general, video is a
challenging data modality. Video data is high-dimensional, noisy, stochastic,
and poorly labelled. These issues have seen progress in video foundation
models lag behind that of language and image models. Second, utilising video
data specifically for robotics introduces its own set of issues. Video lacks
information critical to robotics, including explicit action information and
low-level information such as forces and proprioception. Moreover, there may
be various distribution shifts between internet video and the downstream robot
setting, including disparities in environments, embodiments, and viewpoints.
Given these challenges, we outline two key questions for LfV research:

  1. 1.

How to extract relevant knowledge from internet video?

  2. 2.

How to apply video-extracted knowledge to robotics?

In this survey (see Figure 1), we review existing literature that attempts to
answer these questions. For the first question, we survey video foundation
modelling techniques promising for extracting knowledge from large-scale
internet video. We note that improved video foundation models, and video
foundation model pipelines customized for robotics, will likely be a key
driver of future LfV progress. For the second question, we perform a thorough
analysis of literature that leverages video data to aid robot learning. We
taxonomise this literature according to which RL Knowledge Modality (KM)
(i.e., which of representations, policy, dynamics model, reward function, or
value function) directly benefits from the use of video data. Additionally, we
review common techniques used to mitigate LfV challenges, such as the use of
action representations to address missing action labels in video.

We conclude by discussing problems and opportunities for future LfV research.
This includes advocating for scalable approaches that can best provide the
promised benefits of LfV. Here, we recommend targeting the policy and dynamics
model KMs. Additionally, we discuss directions for utilising video foundation
model techniques for LfV, before touching on directions for overcoming key LfV
challenges.

These promising opportunities, combined with encouraging recent advances in
LfV [Yang et al., 2023c, Bruce et al., 2024], strongly suggest that the
promised benefits of LfV are well within reach. We hope this comprehensive
survey can encourage and inform future LfV research, ultimately serving to
accelerate our progress towards the creation of generalist robots.

##### Survey structure.

The remainder of the paper proceeds as follows.

  * •

Background (Section 2). First, we introduce key RL concepts, including
formalising the RL setting and providing more detail on RL KMs (Section 2.1).
Second, a broad overview of relevant machine learning literature provides the
reader with additional context to LfV (Section 2.2). Finally, details of
related surveys and papers are discussed (Section 2.3).

  * •

LfV-for-robotics: Preliminaries (Section 3). Here, we give a more detailed
introduction to the LfV setting, providing useful preliminary information in
advance of our literature review in Sections 4 and 5. We outline the potential
benefits of LfV, the challenges in LfV, and metrics for evaluating LfV
methods.

  * •

Towards Video Foundation Models (Section 4). Improved foundational video
models and techniques will be vital for future progress in LfV. As such, we
dedicate this section to surveying literature regarding three relevant types
of video foundation models: video encoders (Section 4.1), video prediction
models (Section 4.2), and video-to-text models (Section 4.3). Associated
limitations and promising future directions are discussed.

  * •

LfV-for-robotics: Methods (Section 5). This section offers a review of
literature that utilizes video data to enhance robot learning. First, we
identify and taxonomise common strategies used to address LfV challenges
(Section 5.1). This includes methods that use alternative representations of
actions to tackle the missing action label problem (Section 5.1.1), and
methods that aim to explicitly address LfV distribution shifts (Section
5.1.2). Second, we present our main taxonomy of the LfV-for-robotics
literature (Section 5.2), categorising methods based on which component of the
RL algorithm benefits from the use of video. For each component, we detail the
different strategies in the literature and discuss their corresponding
advantages, disadvantages, and promising directions.

  * •

Datasets (Section 6). High quality video datasets are essential for LfV
progress. Here, we outline the desiderata for LfV video datasets (Section
6.1), provide in-depth details on techniques for curating video data (Section
6.2), and review and critique relevant existing video datasets (Section 6.3).

  * •

Benchmarks (Section 7). We discuss how an LfV benchmark should be designed
(Section 7.1), and present details of relevant existing benchmarks whilst
suggesting LfV-specific improvements (Section 7.2).

  * •

Challenges & Opportunities (Section 8). We discuss the key challenges and
opportunities for future LfV research, as identified during our analysis of
the literature. First, we give high-level recommendations for future LfV
research (Section 8.1). Second, we discuss in more detail how video foundation
model techniques can be benefit LfV (Section 8.2). Third, we identify paths
towards overcoming key LfV challenges (Section 8.3). Finally, we direct
attention towards challenges that may not be resolved via naive scaling to
internet video data (Section 8.4).

  * •

Conclusion (Section 9). We conclude a summary of the key takeaways of the
survey.

##### Contributions.

We now summarize the key contributions of this survey.

  * •

Advocacy for Learning from Videos (LfV): This survey highlights the promise of
LfV methodologies for developing general-purpose robots, and should serve to
encourage further research in the area. Specifically, we advocate for
approaches that can scale to large, diverse internet video datasets.

  * •

Formalization of Fundamental Concepts: We explicitly discuss and formalize
fundamental LfV concepts and notions in a single contained document.

  * •

Enumeration and Taxonomization: We synthesise the relevant literature into
useful and comprehensive taxonomies. This can facilitate a holistic
understanding of the LfV research landscape, and provides a structured
framework in which future LfV research can be placed and assessed.

  * •

Critical Analysis: Throughout the survey, we conduct critical analyses and
discussions of existing approaches; identifying their advantages,
disadvantages, and corresponding promising future directions.

  * •

Identification of Key Challenges and Opportunities: After a thorough analysis
of the LfV setting and literature, we outline key challenges and opportunities
for future LfV research. This should serve to encourage further progress in
LfV along these directions.

##  2 Background

In this section we introduce relevant formalisms and concepts related to
reinforcement learning (RL) (Section 2.1), and discuss relevant prior work in
ML (Section 2.2) and LfV-related surveys (Section 2.3).

###  2.1 Reinforcement Learning

In this section, we first formalise the reinforcement learning (RL) setting.
We then summarise some RL concepts introduced by Wulfmeier et al. [2023] that
we make use of throughout the rest of the survey. Specifically, we will
introduce the notion of a reinforcement learning Knowledge Modality (KM), and
will refer to several mechanisms of transfer for transferring pretrained KMs
to a downstream domain.

##### Formalism.

In reinforcement learning (RL), an agent observes its environment, takes an
action, and receives a reward after the state of the environment changes. This
can be formalised as a Markov Decision Process (MDP) consisting of the state
space $\mathcal{S}$, the action space $\mathcal{A}$, and the transition
probability $p(s_{t+1}|s_{t},a_{t})$ of reaching state $s_{t}+1$ from state
$s_{t}$ when executing action $a_{t}$. The agent’s behaviour is given in terms
of a policy $\pi(a_{t}|s_{t})$. When aspects of the state cannot be observed,
the environment is termed a Partially Observable MDP (POMDP), and the agent
only has access to observations $o_{t}\in\mathcal{O}$ that are partial
mappings of the state $o_{t}=f(s_{t})$. Unless stated otherwise, we will
always assume a generalist robot must operate in a POMDP. Throughout this
paper we will simplify the notation and denote any agent observations as
$s_{t}$.

The agent aims to maximise its sum of discounted future rewards, commonly
referred to as its ‘returns’. This is captured by the objective in Equation 1:

$J(\pi)=\mathbb{E}_{[p(s_{0}),p(s_{t+1}|s_{t},a_{t}),\pi(a_{t}|s_{t})]_{t=0...T}}\left[\sum_{t=0}^{\infty}\gamma^{t}r_{t}\right]$
(1)

where $\gamma$ is the discount factor, $r_{t}=r(s_{t},a_{t})$ is the reward,
and $\rho(s_{0})$ is the initial state distribution. The optimal policy
$\pi^{*}(a_{t}|s_{t})$ maximises Equation 1.

##### Reinforcement Learning Knowledge Modalities.

An RL Knowledge Modality (KM) is some function learned from data that
represents specific types of RL-related knowledge. We now give a brief summary
of the KMs we will refer to throughout this survey. Notably, we use the notion
of a KM to help define our main taxonomy of the LfV-for-robotics literature in
Section 5.2.

  * Policy, $\pi(a_{t}|s_{t})$. This is a mapping from states to actions directly describing the agent’s behaviour. $\pi(a_{t}|s_{t})$ can be learned via imitation learning from $(s_{t},a_{t})$ tuples or via offline RL [Levine et al., 2020] from trajectories of $(s_{t},a_{t},r_{t},s_{t+1})$ tuples.

  * State or State-Action Value Function, $V_{\pi}(s_{t})$ or $Q_{\pi}(s_{t},a_{t})$. These functions map from states or state-action pairs to the expected future return when acting under a particular policy $\pi$. The optimal value functions $V^{*}$ and $Q^{*}$ are the value functions for the optimal policy $\pi^{*}$. Given transition tuples $(s_{t},a_{t},r_{t},s_{t+1})$ for $Q$, or $(s_{t},r_{t},s_{t+1})$ for $V$, the value functions can learned using Monte Carlo estimates (determined over complete trajectories), or Temporal Difference estimates (determined over individual transitions) [Sutton and Barto, 2018]. The value function estimates the quality of states or actions, and thus can be used to aid planning [Schrittwieser et al., 2019] or learning of the policy [Lillicrap et al., 2015, Schulman et al., 2017].

  * Dynamics Model, $p(s_{t+1}|s_{t},a_{t})$. This function predicts the next state given the current state and action. It captures the underlying transition dynamics of the agent’s embodiment and its environment, and can be learned using transition tuples of the form $(s_{t},a_{t},s_{t+1})$. A dynamics model can be used: (i) for planning – for example, via model predictive control (MPC) [Garcia et al., 1989, Hafner et al., 2018]; (ii) to generate synthetic data for improved sample efficiency [Sutton, 1990, Janner et al., 2019]; or (iii) to help generate value gradients for policy training by backpropagating through a learned dynamics model [Hafner et al., 2023].

  * Reward Model, $r(s_{t},a_{t})$. This function predicts the reward provided by the environment for a specific state-action pair. Note, other formulations, such as $r(s_{t},s_{t+1})$ or $r(s_{t+1})$ are also acceptable. Depending on the formulation, models of the reward may be learned via supervised learning on tuples containing some, or all, of the following information: $(s_{t},a_{t},s_{t+1})$. In some cases, reward models can be inferred from expert reward-free trajectories, e.g., via inverse reinforcement learning [Arora and Doshi, 2021].

  * Representations. More generally, useful knowledge can be stored in the form of learned representations. Such representations can be learned via any learning objective, including whilst learning any of the above KMs. Representation transfer (see more details in the following paragraph), is a common method for transferring knowledge from one dataset to another setting [Nair et al., 2022, Majumdar et al., 2023].

##### Transfer Mechanisms.

RL KMs can be learned from a source MDP or dataset and subsequently
transferred to a target MDP. In the case of LfV, this involves learning a KM
from $D_{\text{video}}$ and adapting it to the robot MDP using
$D_{\text{robot}}$. There are several possible transfer mechanisms here.
Wulfmeier et al. [2023] distinguish between direct and indirect mechanisms.

Direct transfer mechanisms. (1) _Generalisation and zero-shot transfer_
involves using the pretrained KM directly in the target setting, without any
further fine-tuning or adaptation [Escontrela et al., 2023]. (2) _Fine-tuning_
involves continuing the training of the KM on data from the target domain. (3)
_Representation transfer_ involves a new model being defined that is composed
of parts of the pretrained KM and elsewhere is randomly initialised [Nair et
al., 2022, Majumdar et al., 2023] (see Figure 7). This new model is then
trained using data from the source domain. We note that, unlike [Wulfmeier et
al., 2023], we may use the terms fine-tuning and representation transfer
interchangeably in this survey. (4) _Hierarchy: conditioning_ involves using
the pretrained KM to condition a new KM being trained in the target MDP
[Schmidt and Jiang, 2023, Wang et al., 2023a, Wen et al., 2023]. (5)
_Hierarchy: composition_ involves composing solutions from multiple pretrained
KMs [Wulfmeier et al., 2019]. (6) _Meta-learning_ methods pretrain the KM such
that it can adapt quickly to unseen domains. General categories of meta-
learning include gradient-based approaches [Finn et al., 2017] and in-context
learning [Akkaya et al., 2019, Laskin et al., 2022].

Indirect transfer mechanisms. (1) _Transfer via (auxiliary) objectives:_ Here,
the pretrained KM is used to help define a learning objective for the
downstream KM. For example, one can distill knowledge by training with
regularisation towards the previous KM [Tirumala et al., 2022, Ye et al.,
2023]. (2) Transfer via data (e.g., by shaping exploration): Here, for
example, we can use previous policies for data collection in the target
domain, and learn a new KM off-policy from this experience [Campos et al.,
2021].

###  2.2 General Machine Learning

Here, we give a general overview of machine learning literature relevant to
the goal of utilising internet video data to help obtain generalist robots.
This provides additional context to the video foundation model research
(Section 4) and LfV-for-robotics research (Section 5.2) we review later in
this survey.

##### Foundation Models.

Progress in deep learning has continually been driven by scaling up datasets
and model sizes [Krizhevsky et al., 2012, He et al., 2016, Radford et al.,
2019, Brown et al., 2020]. ‘Foundation models’ are large models pretrained on
large diverse datasets. They demonstrate relatively general capabilities
(within their respective domains). Whereas previously ML research would train
task-specific models, it is now common to employ general foundation models
(either zero-shot or finetuned) to solve wide varieties of downstream tasks.
It is important to note here that, in addition to scaling, improved model
architectures (i.e., transformers [Vaswani et al., 2017] and diffusion models
[Ho et al., 2020]) have been crucial for advances in foundation models. We now
give an overview of foundation model research across several domains.

  * Language. In the domain of natural-language-processing (NLP), foundational large language models (LLMs) have led to remarkable progress [OpenAI, 2023]. Huge, diverse text datasets scraped from the internet have been used to pretrain transformer models with billions of parameters via simple self-supervised next-token prediction objectives [Brown et al., 2020, Narang and Chowdhery, 2022]. This pretraining scheme allows the model to learn vast amounts of declarative and procedural knowledge from internet data. These models can be further finetuned via supervised learning on smaller high-quality datasets, and via reinforcement learning from human or AI feedback (RLHF) [Ouyang et al., 2022, Bai et al., 2022]. This finetuning results in commercially viable models [OpenAI, 2023, Team et al., 2023a] that can be used, for example, as a chatbots or coding assistants.

  * Images. Image understanding and generation has similarly benefited from foundational training schemes. Deep learning exploded in popularity after successful image recognition results were obtained via the use of the relatively large ImageNet dataset [Deng et al., 2009, Krizhevsky et al., 2012]. Image recognition was further improved via training dual-encoder image-language models on large image-text datasets scraped from the internet [Radford et al., 2021]. More recently, text-conditioned image generation has excelled due to the use of large diffusion models and internet-scale data [Betker et al., 2023]. Recent efforts have sought to train foundational vision-language models (VLMs) that can take images and text as input, and output text [Alayrac et al., 2022, OpenAI, 2023, Team et al., 2023a]. Results here are promising, but these models still struggle with fine-grained and spatial visual understandings [Cui et al., 2023]. Note, we discuss _video_ foundation models efforts in Section 4.

  * Agents. Preliminary efforts have been made to develop agentic foundation models that can output low-level actions. When such models handle visual and textual inputs, they are sometimes referred to as vision-language-action models (VLAs). Three distinct approaches have been seen here. (1) An internet-pretrained VLM can be finetuned on action-labelled robot data [Brohan et al., 2023]. (2) A sequence model can be jointly pretrained on internet data and action-labelled agentic data [Reed et al., 2022, Sohn et al., 2024]. (3) A large model can be trained solely on agentic (robot) data [Brohan et al., 2022, Team et al., 2023b]. We note that the agentic datasets used in these approaches have been small relative to internet-scale datasets. In general, these approaches have used supervised learning (i.e., imitation learning) on the agentic data. In related work, LLMs have been prompted to act as agents or planners [Ahn et al., 2022, Yao et al., 2022, Park et al., 2023].

##### Deep Reinforcement Learning.

Progress in deep reinforcement learning, including the development of improved
on-policy [Schulman et al., 2017] and off-policy [Lillicrap et al., 2015,
Haarnoja et al., 2018] continuous control algorithms, has been promising for
robotics. However, online RL is impractical for real-world robotics; it is
time-consuming, costly, and potentially dangerous. Nevertheless, solutions
have been proposed in the RL literature to address these issues.

Offline RL algorithms have been proposed for learning from previously
collected data and avoiding issues with online learning [Levine et al., 2020,
Kumar et al., 2020, Zhou et al., 2021]. Model-based RL approaches – which
learn an explicit model of their environment – have been proposed to improve
data-efficiency (and overall performance) versus model-free counterparts
[Janner et al., 2019, Hafner et al., 2023]. When learning from pixel
observations, efficiency can be improved via auxiliary learning objectives
[Yarats et al., 2019], pretraining on prior image data [Wang et al., 2022a],
and data-augmentation [Yarats et al., 2021]. Recently, the use of modern
diffusion architectures has improved learning from diverse offline data [Wang
et al., 2022c], whilst the use of transformer architectures has allowed for
in-context learning [Lee et al., 2023, Laskin et al., 2022]. Elsewhere,
curiosity-based exploration methods have previously been proposed to
accelerate online RL [Burda et al., 2018, Pathak et al., 2017].

##### Scaling Robot Learning.

A lack of available data, combined with the impracticalities of real-world
online RL, has limited progress in robot learning. We now discuss a number of
proposed approaches that seek to scale up robot learning in light of these
issues.

  * Simulation. The use of simulation has often been proposed as a solution to the difficulties of real-world RL. This has led to impressive results in narrow settings, including legged locomotion [Zhuang et al., 2023] and drone racing [Kaufmann et al., 2023a]. However, simulated learning presents a number of issues. (1) Innaccuracies in low-level simulation physics creates a ‘sim-to-real’ gap [Zhao et al., 2020] that must be overcome. A common solution here is to employ domain randomization in simulation [Tobin et al., 2017]. (2) Manually creating a suitable diversity of simulated environments and tasks for generalist robotics is a challenge. Recent works seek to tackle this using procedurally generated environments [Deitke et al., 2022], or LLM-assisted environment design [Xian et al., 2023]. (3) Even if the previous issues were resolved, we would still lack a policy capable of collecting high-quality data in the simulated environment. Solutions here have included using humans to collect the data [Mees et al., 2022], or using LLMs with access to privileged simulation information as policies [Ha et al., 2023].

  * Scaling real world data collection. Recent efforts have sought to collect larger real-world robot datasets. Brohan et al. [2022] use human teleoperation to collect a dataset of 130k short episodes covering 700 table-top manipulation tasks. Khazatsky et al. [2024] collect a similar sized dataset, but with improved diversity. Padalkar et al. [2023] pool data from 33 different academic labs to create a dataset consisting of 22 different embodiments, 500 skills, and 150,000 tasks across more than 1 million episodes. Other work has investigated methods for automating data collection to improve scalablity versus human teleoperation [Bousmalis et al., 2023, Ahn et al., 2024, Yang et al., 2023a]. For example, Ahn et al. [2024] use VLMs and LLMs to orchestrate a fleet of data collecting robots. Finally, we note that several commercial robotics companies have demonstrated evidence of infrastructure for large-scale robot data-collection [Sohn et al., 2024, Jang, 2024].

  * Internet data. Robot learning can be aided via the use of internet data. Note, this may be done indirectly via the use of pretrained foundation models. Image and video data has been used to pretrain visual representations for robotics [Wang et al., 2022a, Nair et al., 2022]. Foundational VLMs and LLMs have been used to help define reward functions for the robot learner [Tam et al., 2022, Du et al., 2023c, Yu et al., 2023c, Klissarov et al., 2023]. LLMs have been employed as planners in long-horizons tasks, leaving execution of low-level skills to specialized robot controllers [Ahn et al., 2022, Huang et al., 2022]. Finally, as touched on above, internet data has been used to help train agentic robot foundation models [Brohan et al., 2023, Sohn et al., 2024]. We elaborate on how internet _video_ data has been used to aid robot learning in Section 5.2.

###  2.3 Related Surveys

Here, we note prior surveys and works with content directly related to ours.
This is done to guide the reader to other relevant works, and to provide
further context to the contributions of our work.

  * _Reinforcement and robot learning._ In the realm of RL, Wulfmeier et al. [2023] highlight the promise of transferring pretrained knowledge from a source to a target domain. The notions of RL KMs and mechanisms of transfer are introduced here, as we detail in Section 2.1. Other RL-centric surveys relevant to the LfV setting include a review of offline RL methods [Prudencio et al., 2023], an analysis of zero-shot generalization in RL [Kirk et al., 2023], and surveys regarding learning from demonstration data [Argall et al., 2009, chaandar Ravichandar et al., 2020]. Robot learning and RL in robotics have previously been surveyed by Kober et al. [2013], Peters et al. [2016], whilst Kroemer et al. [2021] focus on robot learning for manipulation.

  * _Foundation models for robotics._ Other surveys have emphasised the promise of foundation models for robotics. Hu et al. [2023b] assess how foundational VLMs and LLMs can be used for robotics, and additionally detail methods for developing robot-specific foundation models. Yang et al. [2023d] assess how foundation models can be used for general decision making applications, including robotics. They briefly touch on LfV techniques, including the use of video generation models as policies (see Section 5.2.2).

  * _Machine learning for video._ There exist relevant surveys of the video ML literature. Schiappa et al. [2023] present an analysis of methods for self-supervised learning from video data. Ming et al. [2024] survey existing video prediction methods. Tang et al. [2023] survey methods for utilising LLMs in video understanding, whilst Zhang et al. [2024] provide more details on how LLMs can be adapted to additional modalities.

  * _Learning from video for robotics._ There has also been work directly focused on LfV. First, we note Torabi et al. [2019], who review imitation learning over observational data. However, this focuses solely on methods that assume access to expert demonstrations (and thus are not scalable to internet video). Recently, Yang et al. [2024] advocate for the use of video (and video generation methods, in particular) as a unified interface to absorb internet knowledge and represent diverse tasks. Relevant to our work, they discuss the robotics-relevant information in video, several use-cases of video generation for robotics, and challenges in video generation that are applicable to LfV. Unlike this work, we: (i) focus solely on generalist robotics applications (versus the other potential applications of video); (ii) more broadly assess different methods for using video data in robotics (i.e., we go beyond video generation methods); and (iii) present a thorough survey of LfV-related literature.

Finally, the existing work most relevant to ours is a recent survey on video-
based learning approaches for robot manipulation [Eze and Crick, 2024]. In
contrast to this work, we place a stronger emphasis on the goal of obtaining
generalist robots, and on the promise of approaches that can scale to large-
scale internet video data. This includes presenting a more holistic analysis
of the literature (including dedicating Section 4 to video foundation models),
and providing different and richer taxonomies that are better suited to this
emphasis.

##  3 LfV-for-Robotics: Preliminaries

Whilst our main review of the LfV-for-robotics literature is presented in
Section 5, here we provide useful preliminary information. This includes
discussions of crucial LfV concepts, which lay foundations for LfV research.
We first formalise and add clarifications regarding the LfV and generalist
robot settings we are interested in (Section 3.1). We then discuss the
exciting potential benefits video data may provide to robotics (Section 3.2),
before discussing key fundamental and practical challenges in LfV (Section
3.3). Finally, we provide recommendations regarding how LfV methods should be
evaluated in light of these benefits and challenges (Section 3.4).

###  3.1 The LfV Setting

##### Formalising the LfV setting.

In the learning from videos for robotics setting, we will assume access to a
video dataset $D_{\text{video}}$. Here, we denote a video clip as
$\tau=(s_{0},s_{1},...,s_{T})$, where $\tau$ is the full clip and each $s$ is
an observation in the form of an image. We use $s$ to represent images for
compatibility with the RL observation notation established in Section 2.1.
Optionally, $D_{\text{video}}$ may come paired with language annotations or
annotations from other modalities. In general, we will also assume access to a
robot dataset $D_{\text{robot}}$. This dataset usually contains trajectories
of transition tuples $(s_{t},a_{t},r_{t},s_{t+1})$ – though $r_{t}$ may be
missing. The goal of LfV is to learn from $D_{\text{video}}$ in order to
improve performances versus when learning only from $D_{\text{robot}}$.

##### LfV for generalist robotics.

Although $D_{\text{video}}$ may come from any source, in this survey we are
primarily interested in methods that can leverage large-scale video data
gathered from the internet. We will generally assume that such an internet
dataset consists primarily of videos of humans and has good coverage over the
full range of physical tasks that humans commonly perform.

A ‘generalist robot’ is a general-purpose robot that can perform a diverse
range of physical tasks in unstructured real-world settings. Such settings are
POMDPs where the robot must rely heavily on visual observations. Throughout
this survey, unless stated otherwise, we assume the generalist robot has an
embodiment and affordances similar to those of a human. We also assume we want
the robot to perform tasks similar to those humans commonly perform, and to do
so in a comparable manner. Under these assumptions, internet video data can be
particularly useful for training a generalist robot; it provides a large and
diverse set of videos featuring embodiments similar to the robot, which are
performing tasks relevant to the robot.

We note some limitations to these assumptions. First, for certain robotics
tasks, non-human-like embodiments may prove more effective. Second, the robot
may need to perform specific tasks that are not commonly performed by humans.
In these cases, internet video will be less useful for the specific embodiment
and task. Nevertheless, it can still be generally informative about the world
and physical behaviour. Moreover, we believe aiming for humanoid robots that
can perform human-like tasks is a good starting point for generalist robot
efforts.

###  3.2 Potential Benefits

Robotic datasets are expensive to acquire and thus are currently task-specific
or relatively narrow [Padalkar et al., 2023]. In contrast, diverse video data
is freely available in vast quantities on the internet. To achieve our goal of
obtaining generalist robots, we advocate for the use of methods that can
leverage this data in a scalable manner. In this section, we briefly outline
the specific benefits we hope to obtain from methods that do so.

Figure 2: Generalization in the Learning from Videos (LfV) setting. The x-axis
indicates the full set of behaviours expected from a generalist robot. The
y-axis indicates the ‘levels’ of information contained in data. The figure
demonstrates that internet data has better coverage over desired behaviours
than small robot datasets, but lacks crucial low-level information essential
to robotics. Generalising beyond the robot data despite this missing low-level
information is a key LfV challenge. See Sections 3.2 and 3.3 for further
discussion.

##### Generalization beyond $\mathbf{D_{\text{robot}}}$.

LfV offers the exciting possibility of generalization beyond narrow robot
datasets $D_{\text{robot}}$ to the full space of tasks covered in the video
data $D_{\text{video}}$. We now explain the rationale behind this expectation.
First, consider a $D_{\text{robot}}$ that is sufficiently diverse such that it
contains most of the low-level skills or ‘atomic’ actions that will ever be
required from the robot (e.g., specific grasping motions or locomotion
skills). Now consider a task unseen in $D_{\text{robot}}$ but seen in
$D_{\text{video}}$. In this case, our combined dataset
($D_{\text{combined}}=D_{\text{robot}}+D_{\text{video}}$) contains all
information required to know how to complete the the task. $D_{\text{robot}}$
provides information regarding how to execute the low-level skills, whilst
$D_{\text{video}}$ provides higher-level information regarding how to complete
the task (e.g., visually, what movements are required and what steps are
involved). Thus, a suitable LfV method may be capable of generalising beyond
$D_{\text{robot}}$ to solve the task. There is preliminary evidence of this in
LfV-related literature [Brohan et al., 2023, Du et al., 2023a, Wu et al.,
2023a, Wang et al., 2023a]. Figure 2 explores this generalization setting in
more detail.

##### Emergent capabilities.

Learning from internet-scale video may yield capabilities qualitatively beyond
what can be obtained when learning only from a narrow $D_{\text{robot}}$. We
expect this for two reasons. First, in other domains, large quantities of
internet data have allowed for unexpected ‘emergent’ capabilities [Radford et
al., 2019, Brown et al., 2020]. Second, diverse internet video paired with
language annotations offers a path towards stitching together the lower-level
knowledge obtained from robotic and video data with the rich abstractions and
comprehensive world knowledge obtained from textual data [OpenAI, 2023].

##### Improvements in-distribution of $\mathbf{D_{\text{robot}}}$.

Finally, we also expect video data to yield improvements in tasks that (to
some extent) are in-distribution of the robot dataset. First, utilising a
large video dataset can allow for improved data-efficiency with respect to
$D_{\text{robot}}$ [Nair et al., 2022]. Second, LfV approaches may obtain
higher absolute task performance (e.g., higher success rates) in settings in-
distribution of $D_{\text{robot}}$, versus non-LfV approaches [Wu et al.,
2023a].

###  3.3 Challenges

There are several challenges that may be encountered when attempting to learn
from internet video for robotics. Here we discuss the key fundamental and
practical LfV challenges (see Figure 3 for visualizations).

##### Missing action labels.

A major challenge is that raw video data lacks the action labels required by
existing methods for learning from demonstrations (e.g., imitation learning
[Brohan et al., 2022] and offline RL [Chebotar et al., 2023]). Moreover,
adding robotic action labels retrospectively to internet video data is
generally not an option. This is because the low-level actions of a robot are
incompatible with the unrestricted action space of heterogeneous internet
video data. One solution to this problem is to use _alternative action
representations_ (see Section 5.1.1). Pure video data also lacks other RL-
relevant metadata, such as reward labels (which can inform on the quality of
the data), goal labels (which can be useful for goal-conditioning), or end-of-
episode labels.

##### Distribution-shift.

There may be a distributional shift between an internet video dataset and the
downstream robot domain. This can include differences in physical embodiments,
camera viewpoints, or environments. Additionally, humans in videos may be
performing behaviours which are sub-optimal or irrelevant to the downstream
robot task. These shifts present a challenge to deep learning methods.
Mitigations to this problem include: (i) scaling to ever larger and diverse
video data to aid generalization, (ii) leveraging robot data in addition to
internet videos, and (iii) using explicit methods to address the distribution
shifts (see Section 5.1.2).

##### Missing low-level information.

Videos are missing crucial low-level information for robotics. For example,
for certain skillful or dexterous behaviours, robots require low-level
percepts such as tactile sensing, proprioception, or depth sensing. This
information is not available in internet video. Thus, whilst internet video
and text data may have excellent coverage over behaviours and concepts
relevant to a generalist robot, it does not do so for the lowest levels of
information. A key challenge in LfV is to obtain generalization beyond
$D_{\text{robot}}$, despite the missing low-level information in
$D_{\text{video}}$. This challenge is visualized in Figure 2.

##### Controllability, stochasticity, and partial observability.

A challenging aspect of modelling behaviour purely from observational data is
that the boundaries between the agent and the environment are unknown. In
particular, it becomes impossible to distinguish which parts of transitions
are affected (i.e., controlled) by the agent’s actions and which effects are
simply to due to the external environment or noise. This can be problematic
for methods that attempt to extract action information from video (Section
5.1.1). Furthermore, stochasticity and partial-observability of the underlying
environment can make it challenging to perform accurate video prediction.

##### High-dimensionality, noise, and redundancy.

Methods that learn from or generate video data are typically computationally
demanding due to the high-dimensional nature of video data. Additionally,
video can contain significant noise and redundant information. This is in
stark comparison to language data, which is highly compressed and structured.
These characteristics make it more difficult to extract meaningful information
from video data. To overcome this, many works attempt to learn and operate in
a latent representation-space of video [Yan et al., 2021, Bardes et al.,
2023], rather than in raw pixel-space.

##### Dataset limitations.

Finally, the limitations of existing video datasets present a practical
challenge to LfV methods. First, we note that open-source curated video
datasets are still small relative to internet scales (see Figure 9). Second,
we note that language annotations of video are often sparse, coarse, noisy, or
non-existent. We give details on desiderata for video datasets, and methods
for curating improved video data, in Section 6.

Figure 3: Key challenges in LfV (see Section 3.3) are visualised, including:
missing information in video, LfV distribution shifts, and the high-
dimensional nature of video data.

###  3.4 Evaluating LfV Methods

Here, we give guidelines on how LfV methods should be evaluated. These
recommendations are predicated on our goal of scaling to diverse internet
video data to help us obtain generalist robots. They are also informed by the
LfV potential benefits and challenges discussed in the previous sections. We
encourage the reader to bear the following criteria in mind when assessing the
LfV literature reviewed in Section 5.

##### Scalability.

First, we should assess the _scalability_ of the LfV method. Below we outline
characteristics of an LfV method that influence its scalability. These
characteristics cannot usually be measured quantitatively, but, to an extent,
can be assessed qualitatively.

  1. 1.

Can the method scale to diverse internet data, and to the generalist robot
setting? First, we note we are interested in LfV methods that can leverage the
full scale of internet video, and extract as much information as possible from
this heterogeneous data. We thus consider methods that make limiting
assumptions on the nature of the video data to be less promising (e.g., some
methods assume minimal domain shift between the video data and the robot
domain [Stadie et al., 2017, Xiong et al., 2021, Baker et al., 2022]). Second,
we note we are interested in LfV methods that can be applied to unstructured
generalist robot settings. Some past LfV methods are only applicable in
narrow, contained robot settings [Torabi et al., 2018b, Stadie et al., 2017,
Sermanet et al., 2018].

  2. 2.

Can the method benefit from advances in video foundation modelling? There are
many commercial forces that will encourage improvements in video foundational
modelling in the coming years. LfV methods that can benefit from these
advances [Du et al., 2023a, Yang et al., 2023c, Sohn et al., 2024] (by
leveraging either the improved models or the improved techniques and datasets)
are particularly promising.

##### Downstream performance gains.

Second, we should evaluate the extent to which the LfV method can provide the
potential benefits (see Section 3.2) of LfV. Here, we can obtain more concrete
quantitative metrics. We now detail these metrics (which are inspired
partially by those outlined by Wulfmeier et al. [2023] in the RL transfer
setting):

  1. 1.

Performance in-distribution of $\mathcal{D}_{robot}$. We can measure the
performance (e.g., task success rate) of the robot in settings in-distribution
of $\mathcal{D}_{robot}$, after training on $\mathcal{D}_{video}$ and a fixed-
size $\mathcal{D}_{robot}$.

  2. 2.

Data-efficiency in-distribution of $\mathcal{D}_{robot}$. We can measure the
amount of data in $\mathcal{D}_{robot}$ required to reach a certain
performance in settings in-distribution of $\mathcal{D}_{robot}$, when also
training on $\mathcal{D}_{video}$.

  3. 3.

Generalization beyond $\mathcal{D}_{robot}$. We can measure the performance of
the robot in settings out-of-distribution of $\mathcal{D}_{robot}$, after
training on $\mathcal{D}_{video}$ and a fixed size $\mathcal{D}_{robot}$.

These metrics cover the benefits of _generalization beyond
$\mathcal{D}_{robot}$_ and _improvements in-distribution of
$\mathcal{D}_{robot}$_, as outlined in Section 3.2. However, the benefit of
_emergent capabilities_ is not accounted for. Indeed, this benefit likely must
be evaluated qualitatively. We finally note that it will be beneficial to use
these metrics to compare LfV methods to non-LfV baselines (i.e., to equivalent
methods that do not use video data). This will provide a concrete measure of
the benefits $\mathcal{D}_{video}$ is providing.

##  4 Towards Video Foundation Models

We established some preliminaries for the LfV setting in the previous Section
3. We now turn our attention to video foundation modelling as a general means
for extracting robotics-relevant knowledge (e.g., knowledge regarding physics
and behaviours) from large-scale internet video data. In this section, we will
review the general video foundation model literature, in advance of reviewing
literature that applies video data specifically to robotics in Section 5.

Now, foundation models are large deep learning models trained on large,
diverse datasets. They maintain general knowledge and capabilities useful in a
wide range of downstream settings. This has most notably been demonstrated in
natural language processing with large language models (LLMs) [OpenAI, 2023,
Team et al., 2023a]. In this section, we focus on _video foundation models_
(see Figure 4). We are interested in these models due to the following LfV
use-cases:

  * •

Using pretrained video foundation models: A pretrained video foundation model
can be adapted for downstream robot applications. For example, a video
prediction model can be used as a robot dynamics model [Yang et al., 2023c].

  * •

Using video foundation model techniques and datasets: Techniques and datasets
used originally for video foundation models can be customized for robotics
purposes. For example, a robot foundation model can be trained on both video
and robot data, using techniques inspired by video foundation modelling [Sohn
et al., 2024].

We categorise the literature here according to three specific functionalities
a video foundation model can provide: (i) video encoding (Section 4.1), (ii)
video prediction (Section 4.2), and (iii) video-to-text generation (Section
4.3). We explain in each subsection why these functionalities are relevant to
LfV. We note that, in practice, some models may perform multiple of these
functions (e.g., any-to-any sequence models can perform video prediction and
video-to-text generation [Liu et al., 2024b]).

Figure 4: Video Foundation Models (see Section 4) for LfV. The top green box
presents different categories of video foundation models and their potential
utility to robotics. The bottom blue boxes illustrate two approaches by which
video foundation modelling can contribute to LfV: (left) pretrained video
foundation models can be finetuned into robot foundation models; (right) video
foundation model techniques and datasets can be used to train robot foundation
models.

###  4.1 Video Encoders

Foundational video encoders can provide robust video representations useful
for downstream robotic applications. Representation transfer approaches (see
Section 5.2.1) can take a pretrained video encoder and finetune it into an RL
KM (e.g., into a policy) that can be used in the robot domain. Several works
have used frozen pretrained video representations to help define a robot
reward function [Fan et al., 2022, Sontakke et al., 2023, Nair et al., 2022]
(see Section 5.2.4). In this section, we first give a broad overview
techniques that can be used to train video encoders, before giving more
details on recent promising techniques for training foundational video
encoders.

##### Video Representation Learning Overview: Derive $Y$ from $\bar{X}$, apply
a supervised loss.

We now give a broad overview of techniques that can be used to learn video
encoders. Let $\bar{X}$ be a video clip $X$ paired with some corresponding
labels and data modalities. Most methods here can be framed as first deriving
a ‘label’ $Y$ from $\bar{X}$ and then using $Y$ to define a learning
objective. We now give an overview of the various ways in which $Y$ can be
defined and used for video representation learning.

  * •

How to Define $Y$? (1) Transforms of $X$: Commonly, we can derive a pseudo-
label from the video $X$ itself. This transform could be some augmentation of
the video (e.g., spatial or temporal shifts), a masked version of the video,
or a slice of the video. Schiappa et al. [2023] identify a number of
categories of augmentation that can be used for video data, including view
augmentations and spatio-temporal-augmentations. (2) Paired information: $Y$
can be derived from additional information paired with the video. This can
include labels or data of another modality. Labels relevant to robotics could
include the class of action performed in the video [Kay et al., 2017], object
masks and bounding boxes [Ding et al., 2023a], or human hand poses and
affordances [Shan et al., 2020]. Different modalities may include
corresponding audio, speech transcriptions, language annotations, or 3D depth
information [Grauman et al., 2021]. Finally, we note that occasionally $X$ may
be paired with another video, for example an equivalent video from a different
viewpoint [Grauman et al., 2023]. (3) Meta-data: Meta-data associated with the
video can also be used for $Y$. Some examples here include appearance
statistics and playback speed [Schiappa et al., 2023].

  * •

How to Use $Y$? (1) Prediction: Here, the learning objective is to predict
$Y$. One option is to predict $Y$ from $X$. For example, predicting a language
description of a video [Karamcheti et al., 2023], or predicting the next frame
in a video (i.e., video prediction [Gao et al., 2022]). Another variation is
to predict $Y$ from $Y$, i.e., auto-encoding. Here $Y$ is encoded into a
bottleneck (a compressed representation of the video) and then decoded [Tong
et al., 2022, Villegas et al., 2022]. (2) Joint-embeddings: Instead of
learning to predict the raw version of $Y$, we can define objectives where the
loss is calculated via embeddings of $X$ and $Y$. Contrastive learning
approaches have been heavily explored here [Schiappa et al., 2023]. Noise-
constrastive estimation objectives (NCE) [Oord et al., 2018] have been
popular, most commonly used to contrast the video with positive and negative
pairs of language descriptions [Xu et al., 2021, Zhao et al., 2024,
Papalampidi et al., 2023]. Other contrastive objectives include those that use
binary cross-entropy losses, or score-based approaches [Schiappa et al.,
2023]. Non-contrastive joint-embedding approaches also exist. BYOL [Grill et
al., 2020] define $Y_{1},Y_{2}$ (two augmentations of $X$) and train an
‘online’ network to predict ‘target’ network embedding of $Y_{1}$ from
$Y_{2}$. V-JEPA [Bardes et al., 2023] similarly define a learning objective by
predicting the embedding of $X$ from an embedding of $Y$.

The taxonomy presented above covers many schemes that can be used to train
video encoders. However, it may neglect some learning objectives. We identify
temporal difference learning from video [Bhateja et al., 2023] as one
neglected objective, however there may be others.

##### Foundational Video Encoders: Learning Objectives.

We now detail promising learning objectives and techniques for training
foundational video encoders from internet-scale video data. We focus primarily
on _video-first_ architectures that jointly represent the spatio-temporal
information in video [Arnab et al., 2021, Yu et al., 2023c, Zhao et al.,
2024]. In contrast to models that rely on strong hand-crafted inductive biases
to process video [Simonyan and Zisserman, 2014, Girdhar et al., 2017], we
consider video-first methods to be more promising: when scaled appropriately
to internet video data, these methods may learn richer, more informative
representations (since they are not limited by strong inductive biases).

  * •

Video-language objectives. Text annotations are useful for learning semantic
representations of video. We note several objectives in the literature that
leverage text annotations: (i) video-text contrastive losses [Xu et al.,
2021], (ii) video-text matching [Li et al., 2023c], (iii) masked language
modelling [Li et al., 2023c], and (iv) video-to-text losses [Papalampidi et
al., 2023]. Video-text contrastive NCE losses have been very popular [Xu et
al., 2021, Zhao et al., 2024, Papalampidi et al., 2023, Li et al., 2023c, Wang
et al., 2023f]. Wang et al. [2023f] accelerate contrastive training by
randomly masking input video in the early stages of training. Papalampidi et
al. [2023] use a learning schedule, moving from shorter to longer videos over
the course of the contrastive training. Note, the strategy for sampling
negative examples and mini-batches is crucial for contrastive approaches. Zhao
et al. [2024] alternate between mini-batches from different datasets during
training, whilst Xu et al. [2021] use retrieval augmented sampling to chose
mini-batches. Meanwhile, Bagad et al. [2023] artificially create negative
samples to improve temporal representations. Elsewhere, other video-text
losses have generally been used in conjunction with an NCE loss. Li et al.
[2023c] combine video-text contrastive, video-text matching, and masked
language modelling objectives. Yan et al. [2022], Papalampidi et al. [2023]
combine a video-to-text objective with the video-text contrastive objective.

  * •

Masked auto-encoding (MAE). Video MAE involves encoding and reconstructing a
masked video, with the reconstruction loss acting as the learning objective
[Tong et al., 2022, Wang et al., 2023c, Girdhar et al., 2022]. Tong et al.
[2022] tokenise the video and mask in token space. Two important design
decisions are made in this masking scheme: (i) due to temporal redundancy in
video, an extremely high masking ratio (90-95%) is employed; (ii) a tube
masking scheme – which extends masks along the temporal dimension – mitigates
issues related to easy reconstruction in areas with minimal motion. Wang et
al. [2023c] extend this approach, performing masking in the decoder to further
improve computational efficiency. Li et al. [2023c] perform semantic masking,
where semantically relevant parts of images (as determined by an image-
language model) are prioritized for masking. Related methods use masked
distillation schemes [Wang et al., 2022b], or vector-quantized auto-encoding
[Yu et al., 2023a] (both of which we discuss in more detail below).

  * •

Vector-quantized (VQ) auto-encoding. VQ-AEs [van den Oord et al., 2017] use
codebooks to encode video to discrete representations in their AE bottleneck.
The reconstruction objective can either be a pixel-error loss (i.e., a VQ-VAE
[van den Oord et al., 2017]), or an adversarial loss (i.e., a VQ-GAN [Esser et
al., 2020]). When deciding the encoder-decoder architecture, crucial decisions
must be made regarding how to fuse the spatio-temporal information in video. A
simple choice is to encode each frame individually using a 2D VQ-AE [Seo et
al., 2022b]. However, this can neglect the importance of fusing information
along the temporal dimension. To fuse spatio-temporal information in video, 3D
convolutions have been explored [Yan et al., 2021, Yu et al., 2022], as have
various spatio-temporal attention schemes [Arnab et al., 2021, Bertasius et
al., 2021]. When training a video VQ-AE, Yu et al. [2023c] improves upon Yu et
al. [2022] by introducing lookup-free quantization and using temporally causal
3D convolutions. Villegas et al. [2022] use a C-ViViT (a causal variation of
the ViVit architecture [Arnab et al., 2021]) encoder-decoder, compressing
video in space and time, whilst staying auto-regressive in time. Bruce et al.
[2024] improve the computational efficiency of their VQ-AE via the use of a
ST-transformer [Xu et al., 2020] encoder-decoder. These VQ-AEs models can be
framed as video tokenizers and are often used to provide a compressed latent
space for video prediction models [Yan et al., 2021] (see Section 4.2 for more
details).

  * •

Distillation losses. A number of works have explored the use of student-
teacher distillation losses [Wang et al., 2022b, Zhao et al., 2024, Li et al.,
2023c]. Wang et al. [2022b] pretrain separate image and video teachers (which
provide better guiding spatial and temporal features respectively) and train a
student to reconstruct the teacher’s features. Zhao et al. [2024] freeze a
video encoder from an initial video-text contrastive learning stage, and use
it to provide a distillation loss during a second MAE stage. Li et al. [2023c]
uses a frozen image-language model to provide a distillation loss throughout
training, improving the learning of semantic features.

  * •

Joint-embedding Prediction Architectures (JEPA). JEPA approaches have recently
been applied to video [Bardes et al., 2023] (a brief description of JEPA
approaches is found in the above paragraph). Versus pixel-based approaches,
JEPA methods may better mitigate issues related to the high-dimensionality of
video and noise in video. Bardes et al. [2023] show their video JEPA method to
be computationally efficient and competitive under frozen downstream
evaluation settings and in tasks requiring fine-grained understandings.

We note that the different objectives above can be combined to complement each
other. For example, video-text objectives that capture semantic features (but
suffer from noisy labels), can be combined with masked modelling objectives
that better capture low-level features (and do not require language labels)
[Li et al., 2023c, Zhao et al., 2024]. This is often done in a multi-stage
training frame-work [Zhao et al., 2024]. Other multi-stage pipelines have been
employed, including: pretraining on diverse lower-quality data before
finetuning on higher-quality data [Wang et al., 2023c], or starting with
shorter videos before moving to longer videos [Liu et al., 2024b].

Finally, we note that these _video-first_ techniques still often bootstrap
from image data; image data is generally more available and comes with
improved language annotations. Some techniques here include: using image data
jointly with video data during training [Papalampidi et al., 2023]; using
pretrained image models to provide a distillation loss [Li et al., 2023c, Wang
et al., 2022b]; or adapting pretrained image models into video models [Yan et
al., 2022, Yang et al., 2023e].

##### SOTA models.

Here, we give brief details on some representative state-of-the-art (SOTA)
video encoder models.

  * •

VideoPrism [Zhao et al., 2024]: A 1B parameter model is trained initially via
video-text contrastive learning and then via masked video modelling. The
training dataset consists of 36M well-captioned videos (closed-source) and
582M video clips with noisy parallel text (composed from open-sourced datasets
[Stroud et al., 2020, Zellers et al., 2021, Wang et al., 2023f] and additional
closed-source data).

  * •

VideoMAE-V2 [Wang et al., 2023c]: A 1B parameter model is first trained using
a masked auto-encoding objective. This is done on an unlabelled video dataset
of 1.35M clips obtained from public datasets [Kay et al., 2017, Goyal et al.,
2017, Gu et al., 2018, Bain et al., 2021], and from uncurated videos crawled
from Instagram. After this intial stage, the model is supervised finetuned on
labelled data from Kay et al. [2017].

  * •

Short/Long-ViViT [Papalampidi et al., 2023]: A 1B parameter model is
pretrained using a video-language contrastive loss. The pretraining dataset
consists of 27M video-text pairs including closed-source data and data from
the open-sourced Miech et al. [2019]. The model is then combined with a
pretrained LLM head and finetuned on a video-to-text objective.

  * •

InternVid [Wang et al., 2023f]: 50M video-text pairs from the InternVid
dataset [Wang et al., 2023f] are used to train a model via a video-language
contrastive loss. The model is open-sourced.

  * •

UMT [Li et al., 2023c]: A 300M parameter model is trained in two stages.
First, unmasked token alignment is performed using a pretrained image-language
model as a teacher and high quality video data based on Kay et al. [2017].
Next, video-text contrastive, video-text matching, and masked language
modelling objectives are added. The second stage leverages 25M image-text and
video-text [Bain et al., 2021, Sharma et al., 2018] pairs. The model is open-
sourced.

  * •

V-JEPA [Bardes et al., 2023]: A 630M parameter model is trained using a JEPA
objective. The dataset consists of 2M videos from [Miech et al., 2019, Kay et
al., 2017, Goyal et al., 2017]. The model is open-sourced.

##### Discussion.

Promising initial steps have been taken towards obtaining foundational video
encoders [Zhao et al., 2024]. However, a major bottleneck to further advances
is the computational difficulties of processing large-scale video data.
Indeed, many of the methods above employ specific mechanisms (e.g., masking
[Wang et al., 2023c]) to mitigate computational demands. Papalampidi et al.
[2023] perform a relevant analysis of the trade-offs between model accuracy
and reducing computational demands. Another bottleneck is the quality of the
video datasets themselves. Video-text datasets are used to train several
leading models, but language annotations are often of low quality. Finally, we
note that representing long video sequences remains a major challenge
[Papalampidi et al., 2023]. Improved architectures (e.g., recent state-space
models [Li et al., 2024]) may help with both computational demands and long-
term video modelling.

###  4.2 Video Prediction Models

We dedicate this section to discussing models that can perform next-frame
video prediction $p(s_{t+1}|s_{t-k:t})$. From internet video, video prediction
models can learn information regarding world dynamics and human behaviours.
They can thus be useful for downstream robotics in the following ways:

  * •

Dynamics: A video prediction model can be adapted into a robotics dynamics
model to serve as a planner [Du et al., 2023b] or simulator [Yang et al.,
2023c] (see Section 5.2.3).

  * •

Policies: The video prediction objective implicitly allows the model to learn
the distribution of behaviours in the video dataset [Escontrela et al., 2023].
As such, video prediction models can act as policies by generating videos of
proposed action-sequences [Du et al., 2023a] (see Section 5.2.2).

  * •

Representations: Due to the relevant information they represent, video
predictors can be used for LfV representation transfer approaches [Wu et al.,
2023a].

  * •

Rewards: Finally, a reward signal can be defined that encourages the robot to
match the behaviour expected by the video predictor [Escontrela et al., 2023]
(see Section 5.2.4).

We elaborate more on how video prediction models can be applied to robotics in
Section 5.2. We note that LfV methods that can utilise video prediction models
are particularly promising as video generation has a number of commercial
applications which will likely drive progress in capabilities in the near-
future [Brooks et al., 2024].

In this section, we give an overview of literature relevant to foundational
video prediction models. We focus on the most promising recent techniques:
diffusion, autoregressive transformers, and masking transformers. We note that
we are most concerned with models that can perform next-frame video prediction
$p(s_{t+1}|s_{t-k:t})$. However, also relevant are models that can more
generally perform some form of conditional video generation $p(\tau|c)$ (where
$\tau$ is a video clip and $c$ is some conditioning information) [Yang et al.,
2024].

##### Technique: Diffusion.

Diffusion models [Ho et al., 2020] have become popular in video generation due
to their expressiveness and controllability. Indeed, many SOTA video
generation models leverage diffusion [Brooks et al., 2024, Bar-Tal et al.,
2024]. We now give an overview of diffusion-based video prediction methods.

  * Pixel vs latent-space prediction. Some works perform the diffusion process directly in pixel-space [Ho et al., 2022b, Singer et al., 2022, Ho et al., 2022a], whilst others do so in a learned latent space [Brooks et al., 2024, Bar-Tal et al., 2024, Blattmann et al., 2023a]. Blattmann et al. [2023b], Zhou et al. [2022] use a pretrained image VQ-AE to define the latent space, but add temporal layers into the pretrained decoder to reduce flickering artifacts. More information on learning VQ-AEs for video can be found in Section • ‣ 4.1. Predicting in latent space aids computational efficiency, but can result in worse alignment of the generated video with the text prompt. Zhang et al. [2023a] combine both pixel-level and latent diffusion into a hybrid model in an attempt to obtain the benefits of both.

  * Leveraging image data. Video data is more limited in quantity and quality than image data. In particular, video often comes with inferior text annotations. As such, most diffusion video prediction models attempt to utilise image data in some way. Ho et al. [2022b, a] jointly train on image and video data. Bar-Tal et al. [2024], Ge et al. [2023], Blattmann et al. [2023b] take a pretrained image diffusion model and “inflate” it into a video model. This involves modifying the architecture to include temporal connections and finetuning it on video data. Meanwhile, Dai et al. [2023] factorise the video generation process; first generating an image with a pretrained image diffusion model, then generating a video conditioned on the image.

  * Video generation pipelines. The pipeline for generating video with diffusion models often involves some combination of the following steps [Ho et al., 2022a, Ge et al., 2023, Zhou et al., 2022]: key-frame generation, interpolation between key-frames, and spatial super-resolution upsampling. Usually, a separate diffusion model is used for each step. The hierarchical approach of generating key-frames before interpolating can serve to simplify generation of longer videos. However, Bar-Tal et al. [2024] recently generate the entire temporal duration in a single forward pass, obtaining improved global temporal consistency. Meanwhile, the use of ‘cascaded’ spatial super resolution can help keep the base model simple, reducing computational requirements and aiding generalization abilities [Ho et al., 2022a].

  * Other techniques and findings. Ge et al. [2023] find that naive extension of the image noise prior to video leads to sub-optimal performance. They instead use a scheme that better preserves natural correlations in video. Blattmann et al. [2023a] perform an in-depth analysis of the effects of data-quality on video diffusion models, finding pretraining on large diverse data and finetuning on smaller high-quality data to be an effective scheme. Beyond video prediction, video diffusion models can be fused for video editing, video in-filling and in-painting [Brooks et al., 2024, Bar-Tal et al., 2024], and can be flexibly conditioned on different input modalities [Chen et al., 2023c, Xing et al., 2024, Wang et al., 2024b].

##### Technique: Autoregressive and masking transformers.

These transformer-based methods have recently achieved promising results in
video prediction. We now give an overview of different methods seen in the
literature.

  * Pixel vs latent-space predictions. Whilst early work made predictions directly in pixel-space [Weissenborn et al., 2019], subsequent methods do so in a learned latent space [Yan et al., 2021, Yu et al., 2023a, Ge et al., 2022, Kondratyuk et al., 2023]. This latent space is generally pretrained via vector-quantized auto-encoding (VQ-AE), providing a discrete token-space suitable for the transformer architecture. Predicting in a learned latent space provides a number of benefits, including improving computational efficiency and removing pixel-level noise and redundancy from the video prediction objective [Yan et al., 2021]. Deciding the architecture of the VQ-AE tokenizer is an important design decision [Yu et al., 2023a]. State-of-the-art tokenizers jointly represent spatio-temporal video using 3D convolutions [Yan et al., 2021, Yu et al., 2022, Ge et al., 2022] or causal spatio-temporal attention mechanisms [Villegas et al., 2022, Bruce et al., 2024]. We provide more details on various VQ-AE architectures in Section • ‣ 4.1.

  * Autoregressive training. Autoregressive next-token prediction is a standard objective for training transformer sequence models. However, in video the embedding of a single image frame may consist of multiple tokens. Thus, decisions must be made regarding the order in which the tokens are predicted. One option is to simply flatten the discrete tokens in ‘raster scan’ order, and predict these tokens sequentially [Yan et al., 2021, Seo et al., 2022b, Hu et al., 2023a]. Rather than sequentially decoding, other approaches [Bruce et al., 2024] predict the next-frame autoregressively using MaskGit [Chang et al., 2022a] scheduled parallel decoding. MaskGit inference involves decoding all masked tokens of an image simultaneously, then refining the image iteratively conditioned on the previous generation (note, this draws strong parallels to the diffusion process). This parallel decoding process is more computationally efficient than standard autoregressive prediction. However, the sampling process must be carefully designed to ensure the generated tokens are consistent with each other.

  * Masked decoding training. Rather than training via autoregressive next-frame prediction, prior works mask part of the video and train the model to decode the masked tokens in parallel [Yu et al., 2022, 2023c, Gupta et al., 2022]. Usually, decoding is performed following a MaskGit process. This mask decoding training allows for multiple tasks to be performed at inference time, including: video prediction, video in-painting, frame-interpolation, and video editing [Yu et al., 2022]. Masked models are also more computationally efficient and do not suffer from the ‘drifting’ effect that can occur in auto-regressive models. However, they may suffer from sampling bias introduced by the independence assumptions within individual sampling steps [Yang et al., 2024].

  * Other notable techniques and findings. Several techniques have been proposed to improve long-horizon video predictions. Yan et al. [2023] use aggressive spatio-temporal compression to aid consistency in long-horizon predictions. Ge et al. [2022] improve long video prediction by first predicting key-frames, then interpolating. Meanwhile, different transformer architectures have been proposed that may reduce computational requirements on longer sequence lengths [Dao et al., 2022, Hawthorne et al., 2022, Liu et al., 2024b]. Recent advances in state-space models may be relevant here [Gu and Dao, 2023, Tong et al., 2022]. Elsewhere, Kondratyuk et al. [2023], Liu et al. [2024b] use separate tokenizers for different modalities, allowing their transformer prediction model to input and output other modalities (e.g., text), in addition to video.

##### Technique: Other.

Here, we briefly outline less promising video prediction techniques from the
literature. Initial video prediction techniques were based on recurrent or
convolutional architectures [Srivastava et al., 2015, Lotter et al., 2016,
Chiappa et al., 2017]. However, these often make a limiting assumption that
the environment is deterministic. Nevertheless, Gao et al. [2022] demonstrate
that a CNN trained with an MSE loss can act as a strong baseline. Latent-
variable (VAE) video models [Babaeizadeh et al., 2017, Denton and Fergus,
2018, Villegas et al., 2019] attempt to account for stochasticity in videos,
but tend to generate blurry outputs due to limited representational power and
underfitting. An interesting extension of these approaches is the use of
hierarchical VAE models [Saxena et al., 2021, Wu et al., 2021]. GAN-based
methods [Vondrick et al., 2016, Clark et al., 2019, Tulyakov et al., 2018] can
produce more realistic videos, but are known to suffer from training
instability and limited generation diversity. Elsewhere, Wu et al. [2022],
Jiang et al. [2023] propose object-centric video prediction models, Lin et al.
[2022] use neural radiance field (NeRF) representations, and Whitney et al.
[2023] learn a 3D particle-based simulator from RGB-D videos.

##### Conditioning video predictions.

The use of conditioning information can simplify the video prediction problem
and allow for more control over generated videos. Such conditioning is
valuable for downstream robotics usage; it can allow us to simulate the
effects of different action strategies. Yang et al. [2024] outline the various
popular conditioning schemes for video generation models $p(\tau|c)$. Here, we
briefly outline the different modalities of information that can be used as
conditioning information.

  * Language is a popular choice for conditioning video generations. Indeed, text-to-video models $p(\tau|c=text)$ represent much of the state-of-the-art in video generation [Brooks et al., 2024, Bar-Tal et al., 2024, Kondratyuk et al., 2023]. Language can allow for flexible and intuitive control over generated video, at varying degrees of detail.

  * Images and Video can be used to condition video predictions. The next-frame prediction problem requires conditioning on the previous frame(s) $p(s_{t+1}|s_{t-k:t})$ [Gao et al., 2022]. Video in-filling predicts a video that joins initial and final conditioning frames $p(s_{t+1:t+H-1}|s_{t},s_{t+H})$ [Höppe et al., 2022]. Video editing, in-painting, and stylization applications $p(\tau|\bar{\tau})$ are all conditioned on an initial input video $\bar{\tau}$ [Brooks et al., 2024, Bar-Tal et al., 2024].

  * Action information can be used to condition video predictions $p(s_{t+1}|c=\\{s_{t-k:t},action\\})$. Bruce et al. [2024] use a learned single-step latent action-space to condition next-frame predictions. Yang et al. [2023c] use robot actions (when labels are available) to condition next-frame predictions. The next-frame predictions of stochastic video prediction models can be conditioned via their latent variable [Rybkin et al., 2018]. We outline various action representations that may be suitable for conditioning video predictions in Section 5.1.1.

  * Other. Many other modalities have been used to control video generations. Briefly, recent methods have used hand-drawn sketches [Wang et al., 2024b], depth information [Xing et al., 2024], (hand-drawn) motion guidance [Wang et al., 2024b], motion priors [Chen et al., 2023c], or structured scene information [Wang et al., 2023e].

##### SOTA models.

Here, we give details of representative state-of-the-art video prediction
models. Note, we focus on models trained on diverse internet data that are
capable of generating realistic videos.

  * •

Sora [Yang et al., 2023c]: This recent model represented a major breakthrough
in video generation. Whilst experimental results (at the time of writing) are
limited, released videos qualitatively demonstrate a large jump in
capabilities. Improvements include higher visual quality, physical realism,
and longer generations (up to 1 minute) versus previous models. The model
employs a transformer-based latent diffusion architecture [Peebles and Xie,
2023]. However, a number of limitations are noted. The model: (i) can
hallucinate, generating video with inaccurate physics and incorrect cause-and-
effect relationships; and (ii) struggles to recreate precise descriptions or
spatial details in its prompt.

  * •

Lumiere [Bar-Tal et al., 2024]: This is a space-time UNet [Ronneberger et al.,
2015] diffusion model. It is trained to generate the entire temporal duration
of the video through a single pass of the model. It can generate video up to
80 frames at 16 fps (5 seconds long). No details are provided regarding the
datasets used or model parameter counts.

  * •

VideoPoet [Kondratyuk et al., 2023]: An 8B parameter decoder-only transformer
is trained to input and output several data modalities. This includes images,
videos, text, and audio. Separate auto-encoders are used to tokenise each
modality. The model is trained on 1B image-text pairs and 270M videos from
public internet and other sources (the curated dataset is closed source). Like
many diffusion models, this transformer model can be used for various
applications, such as text-to-video, image-to-video, video stylization, and
video outpainting.

  * •

Emu Video [Dai et al., 2023]: Here, the video generation process is factorized
into two steps: first an image is generated conditioned on text, then a video
is generated conditioned on the text and image. This cascade of diffusion
models totals to 6B parameters. The training dataset consists of 34M licenced
video-text pairs.

  * •

Runway [Esser et al., 2023]: Several closed-source video generation models are
available as commercial products [Wang et al., 2024a, pik, , moo, ]. One
representative example with published details is the Runway Gen-1 model, which
is a text-guided diffusion model trained on (closed source) large-scale image
and video data [Esser et al., 2023].

  * •

Stable Video Diffusion (SVD) [Blattmann et al., 2023a]: This is a leading
open-source video generation model. The 1.5B parameter model is trained in
three stages: (i) image pretraining; (ii) video pretraining on curated
datasets of up to 50M samples; and finally (iii) finetuning on 250k high-
quality video samples. However, the training data is closed-source. Another
notable open-source model is that of Hong et al. [2022].

##### Discussion.

First, we note there are various pros and cons between diffusion,
autoregressive transformer, and masked transformer video prediction methods.
Diffusion can model continuous spaces and sample multiple frames in parallel.
However, sampling speeds are slow and generating long sequences is a
challenge. Autoregressive models are easier to train than diffusion, and scale
well with context length. However, autoregressive decoding is computationally
expensive and predictions can suffer from a drifting effect. As noted, versus
autoregressive models, masked models are more computationally efficient but
can suffer from sampling bias. Yang et al. [2024] advocate for improved future
models that combine the advantages of these different schemes.

Now, whilst progress in foundational video prediction models has been
promising [Brooks et al., 2024], a number of challenges remain. This includes
issues with computational efficiency, long video generation, limited
generalization, and hallucination. Nevertheless, there are promising avenues
here. Improved architectures can deal with computational issues [Liu et al.,
2024b] and long video generation [Yan et al., 2023]. Yang et al. [2024] note
that RL finetuning [Black et al., 2023a] may improve hallucination issues. In
general, gains in generated-video quality may be obtained via improved
datasets (following the dataset desiderata outlined in Section 6.1). More
speculative directions include leveraging 3D information to improve the
physical realism of video generations [Zhen et al., 2024], or exploring the
use of hierarchy and temporal abstractions for improved long video generation.

A future direction of particular interest to robotics is to pursue methods
that allow for fine-grained, single-step conditioning of video predictions.
Such conditioning simplifies adaptation of a video prediction model into a
robot dynamics model. Bruce et al. [2024] use learned latent action
representations, and we outline various other action representations suitable
for conditioning video predictions in Section 5.1.1. Finally, we advocate for
progress in open-sourced foundational video prediction models – open-sourced
models will make LfV research more accessible to the wider community.

###  4.3 Video-to-Text Models

Here we refer to models with video-to-text capabilities. A capable video-to-
text model can perform, for example, video question answering or video
summarization. There are numerous commercial applications for such models and
we are likely to see increasingly capable models in the near future. Certain
models with video-to-text capabilities can benefit from internet text-only
data, in addition to video data and video-text data (e.g., ‘any-to-any’
sequence models [Liu et al., 2024b]); from this text-only data, these models
can learn powerful capabilities and knowledge similar to those seen in LLMs
[OpenAI, 2023].

Now, (capable) video-to-text foundation models could be valuable to robotics
in several ways:

  * •

High-quality representations: A capable video-to-text model will have robust,
high-quality video representations. Versus a video-only model, it will likely
have improved high-level semantic representations. Versus an image-language
model, it will have improved temporal-dynamic representations. Robotics can
bootstrap from such models via representation transfer [Brohan et al., 2023],
or by adding robot data directly into the model’s pretraining corpus [Reed et
al., 2022].

  * •

Grounded reasoning and planning: LLMs have proven useful as planning modules
in robotics [Ahn et al., 2022], but their lack of grounding in the physical
world is limiting. In contrast, video-to-text models can perceive the
environment through information-rich video, potentially allowing for improved
and closed-loop reasoning and planning.

  * •

Annotating robot data: High-quality language annotations can act as valuable
conditioning information in many ML domains [Betker et al., 2023, Brooks et
al., 2024]. Robotics is no different [Team et al., 2023b]. Capable video-to-
text models could serve as useful language annotators for robotic datasets.

  * •

Rewards: A sufficiently capable video-to-text model can provide reward or
value estimates for a robot learner. For example, this could be done through a
visual-question answering framework [Du et al., 2023c], or via an RL-from-AI-
feedback framework [Klissarov et al., 2023] (see Section 5.2.4).

In this section, we give an overview of existing video-to-text methods and
models. We do so with an emphasis on methods promising for obtaining
foundational video-to-text models. We note that research here is preliminary:
low-quality video captions and the difficulties of machine learning with video
data (e.g., due to issues related to high-dimensionality and noise) mean
progress lags behind foundation models in other domains.

##### Zero-shot combination of pretrained models.

Due to the difficulties of training a monolithic video-to-text model, previous
work has explicitly decomposed the problem, assigning the sub-tasks to frozen
pretrained models. These pretrained models often communicate via language. A
simple example is given by Chen et al. [2023b]. Here, an image-language model
answers questions about individual video frames, and an LLM synthesized this
information to produce a global summary of the video. Zeng et al. [2022] take
a similar approach, making use of a wider range of pretrained models,
including audio-language models and object detectors. Li et al. [2022c] solve
multimodal problems, including video-to-text tasks, using pre-trained models
as “generators” or “scorers” and composing them via a closed-loop iterative
consensus optimization. Whilst these compositional approaches can be
effective, their modular structure and lack of end-to-end video training mean
they can lack nuanced video representations and understandings.

##### Leveraging pretrained LLMs via adaptors and finetuning.

Alayrac et al. [2022], Li et al. [2022a] introduce schemes for adapting
pretrained LLMs to be additionally conditioned on image inputs. Recent
improvements in open-source LLMs [Touvron et al., 2023] have seen these
approaches extended into the realm of video. Such approaches typically involve
the following steps: (i) obtain a pretrained LLM and an (often pretrained)
video encoder; (ii) define an adaptor module to channel information from the
video encoder output into the LLM; and finally (iii) finetune the combined
model on video-text data. We now give more details on these methods.

  * Pretrained LLMs. Works in this space [Zhang et al., 2023b, Li et al., 2023a] have often leveraged open-source LLMs from the LLaMa [Touvron et al., 2023] family. More recent SOTA open-source LLMs would also be suitable [Jiang et al., 2024, Team et al., 2024].

  * Video encoders. The encoder used can be frame-based (i.e., each frame is encoded separately) [Maaz et al., 2023] or can jointly encode spatio-temporal information [Papalampidi et al., 2023]. Image pretrained encoders based on CLIP [Radford et al., 2021] have commonly been used [Maaz et al., 2023, Li et al., 2023b]. We note that any of the state-of-the-art encoder from Section 4.1 may be suitable (as demonstrated by [Zhao et al., 2024, Papalampidi et al., 2023]), though encoders pretrained with language-based losses likely are more suitable.

  * Adaptors. Tang et al. [2023] identify two main categories of adaptors used in the literature. (1) Connective adaptors connect the outputs of the video encoder to the input space of the LLM. This has been done, for example, with linear projections [Chen et al., 2023a], MLP layers [Lin et al., 2023], and more complex adaptors such as Q-formers [Zhang et al., 2023b]. (2) Insertive adaptors insert the outputs of the video encoder into the internal layers of the LLM. This can be done, for example, via cross-attention [Alayrac et al., 2022, Papalampidi et al., 2023]. Note that, if the LLM remains frozen downstream, then the adaptors must map video information directly into the embedding spaces of the LLM. This may be limiting as a frozen LLM lacks rich representations of lower-level video information. We direct the reader to Zhang et al. [2024] for more information on different adaptor types.

  * Training pipelines. Once the new video-to-text architecture is defined, a common training scheme is: (i) convert large diverse video-text data into token sequences and perform next-token prediction pretraining, then (ii) perform supervised instruction-tuning on small high-quality instruction datasets [Lin et al., 2023, Zhang et al., 2023b]. Following trends in LLMs, future work may investigate a third RLHF finetuning stage [Kaufmann et al., 2023b]. Finally, we note that during training the LLM and/or video encoder may be finetuned [Lin et al., 2023], or kept frozen [Maaz et al., 2023].

##### Natively multi-modal models.

The previously discussed methods have involved combining and finetuning
pretrained models not initially intended for video-to-text purposes. Here, we
highlight end-to-end training pipelines that (loosely) are more _natively_
multi-modal. We note this is a particularly preliminary area of research.
Nevertheless, recent trends towards any-to-any sequence models – where video-
to-text is formulated as an interleaved video and text sequence modelling
problem – are promising. Liu et al. [2024b], Team et al. [2023a], Jin et al.
[2023] all train any-to-any (or any-to-text) autoregressive transformers via
next token prediction. This requires the use of modality specific encoders and
decoders. Liu et al. [2024b] use a pretrained VQ-GAN to tokenise individual
video frames. Note, in practice, these methods may still initialise their
model with a pretrained LLM [Jin et al., 2023], or perform an initial stage of
training on text-only data [Liu et al., 2024b].

##### SOTA models.

Here we give brief details on some representative state-of-the-art video-to-
text models.

  * •

Video-LLaVa [Lin et al., 2023]: A pretrained LLM [Chiang et al., 2023] is
connected to pretrained video and image encoders [Zhu et al., 2023a] via a two
layer connective adaptor. The new architecture is first trained on 558k image-
text pairs [Liu et al., 2024c] and 702K video-text pairs [Luo et al., 2023],
before instruction tuning on 665k image-text [Liu et al., 2023b] and 100k
video-text [Maaz et al., 2023] instructional examples. The video encoder
remains frozen, but the LLM is finetuned. The model only takes eight frames of
video input. The authors note the final model has difficulty with temporal
relationships and spatio-temporal localization. The model is open-source.

  * •

VideoChatGPT [Maaz et al., 2023]: This model is intialized from LLaVa [Liu et
al., 2023b], an LLM adapted to take images as input. Both the visual encoder
and LLM from LLaVa are inherited. To adapt the visual encoder to video, frames
are encoded individually before spatial and temporal pooling. The pooled
features are projected into the LLM input space using a linear projection.
Video-to-text training is performed on a curated dataset of 100,000 custom
video-text instruction pairs. Both the image encoder and LLM are frozen during
the video-text training. The authors note that the final model struggles to
understand subtle temporal relationships and the visual details of small
objects. The model is open-sourced.

  * •

LWM [Liu et al., 2024b]: Here, ring-attention [Liu et al., 2023a] is used to
scale a transformer model up to million-length sequences. The model is
initialized using a Llama-2 7B [Touvron et al., 2023]. A first stage of
autoregressive text-only training is performed with a curriculum of increasing
context lengths. The second stage of training incorporates interleaved
video/images with text. A pretrained VQ-GAN is used to tokenize the images and
videos. Note, this is an any-to-any sequence model that can also perform image
and video generation. The model is open-sourced.

  * •

VideoPrism+PALI [Zhao et al., 2024]: The 1B parameter VideoPrism video encoder
[Zhao et al., 2024] (see Section 4.1) is connected to the PaLM-2 LLM [Anil et
al., 2023]. This is done following the scheme introduced by Alayrac et al.
[2022], where a randomly initialized Perceiver-resampler projects video
representations into the LLM embedding space, with randomly initialized cross-
attention layers added at each layer of the LLM (i.e., this is an insertive
adaptor). The backbones of the video encoder and LLM are kept frozen during
the video-to-text training. Video-text data from 10 different datasets are
employed, including some open-sourced datasets [Grauman et al., 2021, Damen et
al., 2018, Monfort et al., 2021]. This totals to 4.4M video clips.

  * •

Short/LongViViT [Papalampidi et al., 2023]: The Short/LongViViT video encoder
is first pretrained via a video-language contrastive loss (see Section 4.1)
[Papalampidi et al., 2023]. The frozen encoder is then plugged into a frozen
pre-trained LLM for video-to-text adaptation. As per the previous model, the
adaptation scheme follows Alayrac et al. [2022]. The video-to-text model is
trained with an autoregressive video-captioning loss, using close to 27M
(primarily closed source) video-text pairs.

##### Discussion.

Whilst there has been progress in video-to-text models, state-of-the-art
capabilities are still limited. Schemes that combine pretrained models in a
zero-shot manner can be useful for some applications, but a lack of end-to-end
training means they do not maintain rich video representations. Previous
methods that leverage pretrained LLMs have generally only fine-tuned on
relatively small amounts of video-text pairs and can suffer from
hallucinations. More generally, video-to-text models currently struggle with
spatial relationships, fine-grained understandings, and long-term
understandings.

A key bottleneck to progress here is the quality (and quantity) of available
video-text data. Versus existing image datasets, available video datasets are
smaller and can have inferior text annotations. Annotations are often sparse,
and can be coarse or noisy. Improved annotations will be essential for
improving video-to-text capabilities. This will enhance fine-grained
understandings, which is particularly relevant to robotic applications. We
give details on methods for captioning video data in Section 6.2. Progress in
long video understanding is not only bottlenecked by data quality, but also
computational burdens. More efficient architectures for handling longer
contexts are promising here [Liu et al., 2023a, Gu and Dao, 2023, Balažević et
al., 2024]. Finally, we advocate for more research into ‘natively’ multi-modal
models, where the full training pipeline is designed specifically for video-
to-text purposes.

##  5 LfV-for-Robotics: Methods

We have previously outlined preliminary information for the LfV-for-robotics
setting (Section 3), and reviewed video foundation models as a general-purpose
approach for extracting knowledge from internet video (Section 4). In this
section, we will review literature that has specifically utilised video data
for robotics. We begin by reviewing some common methods used to mitigate LfV
challenges (Section 5.1). We then move to our primary categorization and
description of the LfV-for-robotics literature (Section 5.2), where we
classify methods according to which component of the RL algorithm benefits
from the use of video data.

###  5.1 Mitigating LfV Challenges

We outlined key LfV challenges in Section 3.3. In this section, we detail two
categories of techniques that each address a key challenge: (1) The use of
alternative action representations to mitigate missing action labels in video
data (Section 5.1.1). (2) The use of certain representations that explicitly
address LfV distribution-shift issues (Section 5.1.2). These techniques
reoccur across various LfV methods, and are usually used as a single component
within a larger LfV pipeline. As such, here we have separated out details of
these techniques into contained sections ahead of our main analysis of the LfV
literature in Section 5.2.

####  5.1.1 Recovering Action Information from Video

(a) Utilising action representations to overcome the missing action-label
problem in LfV. Action-free videos can be labelled with alternative action
representations $\hat{a}$. This labelled video data can then be used to train
an alternative action RL knowledge modality (e.g., a policy, value function,
or dynamics model).

(b) Categories of action representations for LfV. This survey identifies and
analyses six distinct categories of action representations that can be learned
or obtained from video datasets. Details of each category are elaborated upon
in Section 5.1.1.

Figure 5: Recovering action information from video (Section 5.1.1). (a) An
overview of why action representations are useful in LfV. (b) Visualisations
of the different categories of action representations that can be used.

Transition tuples of robot data commonly come in the form of
$(s_{t},a_{t},r_{t},s_{t+1})$. With this data, some of the key RL knowledge
modalities (KMs) (see Section 2.1) can be trained. For example, one can use
behaviour cloning or RL to train an action-generating policy
$\pi(a_{t}|s_{t})$, use supervised learning to train an action-conditioned
dynamics model $p(s_{t+1}|s_{t},a_{t})$, or TD-learning to train an action-
conditioned value function $Q(s_{t},a_{t})$. However, video data transition
tuples come in the form of $(s_{t},s_{t+1})$ – i.e., video data is missing
action labels, meaning one cannot naively use it to train the KMs listed
above.

In this section, we review works that use _alternative representations of
actions_ to directly address this missing action label problem. These methods
define or learn some representation space that is analogous to the notion of
an action. We use $\hat{a}\in\mathcal{\hat{A}}$ to denote a single alternative
action and its underlying alternative action space. An intuitive example here
is the use of language to represent action information; e.g.,
$\hat{a}=\text{``pick up the cube"}$ . Once $\mathcal{\hat{A}}$ is learned or
defined, video data can be relabelled from $(s_{t},s_{t+1})$ to
$(s_{t},\hat{a}_{t},s_{t+1})$. This relabelled data can be used to train an
alternate action version of an RL KMs (see Figure 5(a)). In the literature,
alternative action policies $\pi_{\text{alt}}(\hat{a}_{t}|s_{t})$ [Schmidt and
Jiang, 2023], dynamics model $p_{\text{alt}}(s_{t+1}|\hat{a}_{t},s_{t})$
[Bruce et al., 2024], and value functions $Q_{\text{alt}}(s_{t},\hat{a}_{t})$
[Bhateja et al., 2023] have all been trained. These alternative action KMs can
be useful downstream for representation transfer [Bhateja et al., 2023,
Schmidt and Jiang, 2023], or if a mapping from alternative to robot action
space $f:\mathcal{A}\rightarrow\mathcal{\hat{A}}$ is obtained [Wang et al.,
2023a, Wen et al., 2023, Du et al., 2023a].

Before detailing the different categories of alternative action
representations $\mathcal{\hat{A}}$ in the literature, we first touch on some
preliminary discussion points:

  * •

What are some desirable properties of alternative actions? First, we note that
$\mathcal{\hat{A}}$ must be inferable purely from video data (or video data
paired with labels). Beyond this, we identify several additional desirable
properties. (1) $\mathcal{\hat{A}}$ can be used to label transitions from
$(s_{t},s_{t+1})$ to $(s_{t},\hat{a}_{t},s_{t+1})$, enabling meaningful
learning of alternative-action RL KMs. (2) $\mathcal{\hat{A}}$ should contain
information regarding what will happen in future frames of a video – i.e., it
contains information regarding future ‘actions’ taken in the video. As a
consequence, learning $\pi(a_{t}|\hat{a}_{t},s_{t})$ should be easier than
learning $\pi(a_{t}|s_{t})$. (3) $\mathcal{\hat{A}}$ should capture a high-
level, general notion of an action, making it transferable across the specific
underlying action spaces of different human and robot embodiments. (4)
$\mathcal{\hat{A}}$ should be interpretable and disentangled. In particular,
$\mathcal{\hat{A}}$ should have a consistent meaning across the state space.
This can allow for learning of $\pi(a_{t}|\hat{a}_{t},s_{t})$ in a more data-
efficient manner [Rybkin et al., 2018].

  * •

What are other notable characteristics of alternative action representations?
We note two distinct characteristics that vary across the different categories
of alternative action representations. (1) Learned vs defined:
$\mathcal{\hat{A}}$ can be on a spectrum from being entirely learned from data
(e.g., single-step latent actions), to simply being manually defined (e.g.,
observations-as-action). (2) Time-horizons: $\mathcal{\hat{A}}$ can contain
only short horizon information (e.g., single-step latent actions) or longer
horizon information (e.g., multi-step latent actions).

We now outline the main categories of $\mathcal{\hat{A}}$ we have identified
in the literature (see Figure 5(b)). Here, we focus on detailing how
$\mathcal{\hat{A}}$ can be defined and learned from video data. More details
on how alternative action representations can be used downstream for robotics
are found throughout Section 5.2.

##### Single-step latent actions.

Here, we consider learned action representations where, like true robot
actions, each ‘latent’ action corresponds to exactly one transition. Thus, the
latent action $\hat{a}_{t}$, which occurs between observations $s_{t}$ and
$s_{t+1}$, contains information about the true action $a_{t}$. Methods that
learn such a latent representation commonly do so using a next-state
prediction objective $p(s_{t+1}|\hat{a}_{t},s_{t})$, such that $\hat{a}_{t}$
is informative for the predictions of the forward dynamics model (FDM)
[Edwards et al., 2018, Rybkin et al., 2018, Schmidt and Jiang, 2023, Bruce et
al., 2024].

Edwards et al. [2018] introduce the concept of latent policies, i.e., policies
$\pi_{\text{alt}}(\hat{a}_{t}|s_{t})$ that produce actions in a latent action
space. However, their method can only learn discrete latent action spaces, and
the proposed learning objective is susceptible to mode collapse [Struckmeier
and Kyrki, 2022, Schmidt and Jiang, 2023]. Approaches to learn continuous
latent action spaces [Rybkin et al., 2018, Schmeckpeper et al., 2019, Chang et
al., 2022b, Schmidt and Jiang, 2023, Bruce et al., 2024] generally work as
follows: (1) A latent inverse dynamics model (IDM) encodes past frames
$\\{s_{t-k},\ldots,s_{t},s_{t+1}\\}$ to infer $\hat{a}_{t}$. (2) $\hat{a}_{t}$
is then passed, along with $\\{s_{t-k},\ldots,s_{t}\\}$, to an FDM
$p(s_{t+1}|\hat{a}_{t},\\{s_{t-k},\ldots,s_{t}\\})$ which predicts $s_{t+1}$.
(3) This setup is trained end-to-end using the FDM prediction error.
Additionally, a regularising objective or mechanism is usually employed to
prevent the IDM from copying the entire observation $s_{t+1}$ into the latent
action. For this purpose, Rybkin et al. [2018] regularise latent actions
towards a Gaussian prior and additionally enforce a latent action
composability loss. Schmidt and Jiang [2023], Bruce et al. [2024] use a
vector-quantization bottleneck to constrain the flow of information. Rybkin et
al. [2018], Schmidt and Jiang [2023] both show their regularization techniques
to improve disentanglement of the latent action representations and thus
improve downstream performance.

We note some potential limitations of these approaches. First, learned latent
action spaces jointly model changes caused by the agent and the external
environment. Often, we would like to only represent changes due to actions
taken by a single agent. Second, these approaches model environment
transitions on a visual level, thus they may omit low-level information
relevant to robotics (such as forces). We finally note that more research into
learning single-step latent action-spaces from realistic internet video is
required.

##### Multi-step latent actions.

Here, we discuss learned action representations $\hat{a}$ that contain
information about multiple time steps (rather than a single timestep). One
line of work here uses variational auto-encoding of video trajectories to
learn _latent plans_ ; i.e., compressed representations of video. This has
been explored extensively with action-labelled demonstrations [Lynch et al.,
2020, Cui et al., 2022, Rosete-Beas et al., 2022], but less so with action-
free video. Wang et al. [2023a] train latent plan representations from action-
free videos via auto-encoding, but use 3D human hand trajectories as the
decoder target (rather than raw video).

There are a number of other approaches. When videos are labelled with language
descriptions, representations of video trajectories have been learned using
video-language contrastive losses [Fan et al., 2022, Lifshitz et al., 2023,
Sontakke et al., 2023]. Chane-Sane et al. [2023], Chen et al. [2021a] obtain
informative representations of video clips by performing supervised
contrastive learning on a dataset with clip-level action labels. Xu et al.
[2023] learn representations of video clips using a self-supervised learning
clustering framework. Other approaches for learning multi-step latent actions
are seen in Tomar et al. [2023], Pertsch et al. [2022], Cai et al. [2023].

We note that these methods are often used for _video-as-instruction_
approaches that specify the task via video, encoding the video instruction
into $\mathcal{\hat{A}}$ before feeding it to a low-level robot policy (see
Section 5.2.2). Finally, we note that any video encoding technique from
Section 4.1 could apply here, though certain representations may have more
suitable properties than others.

##### Observations-as-action.

A simple approach is to use future observations (i.e., future images in the
video) as the action representation: future observations (or an encoding
thereof) provide information on what actions will be taken next in the video.
A benefit here is that this $\mathcal{\hat{A}}$ is directly available from raw
video; unlike latent action representations, no additional learning steps are
required. Observations-as-actions can be used on varying time-horizons, as we
now describe.

  * _Next-observation-as-action_ : These methods use the next observation $s_{t+1}$ as $\hat{a}_{t}$; a proposed $s_{t+1}$ provides clear information about the action that should be taken at $s_{t}$. A number of works train alternative action policies $\pi(s_{t+1}|s_{t})$ from video data to propose observations the robot should reach in the next time-step. Such policies can, for example, be learned via supervised video prediction objectives [Du et al., 2023a, Thomas et al., 2023].

  * _Observations-as-subgoals_ : A subgoal can be viewed as a high-level action. In ‘Observations-as-subgoals’ methods [Black et al., 2023b, Park et al., 2024, Bhateja et al., 2023], an observation (or embedding thereof) from $H$ time-steps into the future is used as a sub-goal (or alternative-action) representation. In this framework, a sub-goal policy $\pi(s_{t+H}|s_{t})$ can be trained to propose an image observation the robot should reach [Black et al., 2023b]. Some simple strategies for defining sub-goals in the video data include: choosing a fixed time-horizon $H$ [Black et al., 2023b, Du et al., 2023a], or randomly sampling observations beyond the current timestep [Bhateja et al., 2023]. More complex strategies include using key-frame identification to identify bottleneck states in video [Pertsch et al., 2019]. Liu et al. [2023c] identify critical states in videos, but require access to reward labels.

##### Language-as-action.

Natural language can be used as a flexible and meaningful high-level action-
space (e.g., $\hat{a}_{t}=$ “pick up the cube”). Some video datasets come with
language annotations that can directly provide such language action
information [Grauman et al., 2021]. General-purpose language annotations can
be further processed to convert them into a more suitable form. Mu et al.
[2023] use LLMs to perform additional processing on Ego4D [Grauman et al.,
2021] language descriptions to convert them into more usable language plans.
If the video does not come with language descriptions, VLMs, LLMs, and other
off-the-shelf models (such as object detectors), can be used to label videos
with language-action information (see Section 6.2 for information regarding
methods for captioning video). Once the video dataset is annotated with
language actions, it can be used, for example, to train a high-level policy
$\pi_{\text{alt}}(\hat{a}_{t}|s_{t})$ that outputs language actions [Mu et
al., 2023, Du et al., 2023b], or to train language-conditioned video
predictors $\pi(s_{t+1}|s_{t},\hat{a}_{t})$ [Du et al., 2023a]. Language
actions are also generally useful as they allow for easy interfacing with
other language models [Du et al., 2023b]. However, language is coarse and
language actions may not contain important lower-level action information.

##### Visual motion information.

Other works have used visual motion information in video to define alternative
action-spaces. Wen et al. [2023] do so by labelling video data with 2D point
trajectories. Specifically, random points on objects are sampled and tracked
throughout the video using an off-the shelf point tracker [Karaev et al.,
2023]. Here an alternative action policy $\pi_{\text{alt}}(\hat{a}_{t}|s_{t})$
is trained to predict future point trajectories $\hat{a}_{t}$, and a low-level
policy $\pi(a_{t}|\hat{a}_{t},s_{t})$ is trained to decode these point
trajectories to robot actions. Yuan et al. [2024] follow a similar approach,
but use 3D point trajectories. The 3D annotations are obtained either from 3D
annotated datasets or videos with depth information (but they note depth
estimation [Bhat et al., 2023] techniques could also be used). Elsewhere, Ko
et al. [2023] use an off-the-shelf model [Xu et al., 2022] to predict optical
flow between two images, giving a pixel-level dense correspondence map between
two frames. This map can be used to infer robot action without any action
labels. In a related approach, Nasiriany et al. [2024] represents actions via
visual arrows in images, allowing a VLM to select actions via an iterative
refinement procedure. Finally, Wang et al. [2023b] use structure-from-motion
[Schonberger and Frahm, 2016] to help recover action information, Yuan et al.
[2021] use motions of object-centric representations, and Yang et al. [2023c]
optionally use camera frame motion/angle information to condition video
predictions.

##### Human-hand-aware actions.

Off-the-shelf human-hand detection models [Rong et al., 2020, Shan et al.,
2020] can extract hand poses or affordances representations from videos which
can act as an alternative action representation [Bharadhwaj et al., 2023, Bahl
et al., 2023, Shaw et al., 2022, Qin et al., 2022, 2021]. For example,
$\hat{a}_{t}$ can be defined as the pose that should be reached at time $t+1$.
Human-to-robot retargeting can be used to convert human poses to a robot
action-space [Qin et al., 2021, Shaw et al., 2022, Sivakumar et al., 2022],
whilst affordance representations can naturally transfer across embodiments
[Bahl et al., 2023]. Bharadhwaj et al. [2023] use object masks in addition to
human poses to define a $\mathcal{\hat{A}}$. We give more details on methods
for detecting human hand poses and affordances in Section 5.1.2.

##### IDM pseudo-action labels.

Finally, though not technically an _alternative_ action representation, we
mention these methods here as they also address the missing action label
problem. Several works train an inverse dynamics model
$p^{-1}(a_{t}|s_{t},s_{t+1})$ on action labelled robot data, and use it to
provide pseudo-action labels for action-free video data [Baker et al., 2022,
Torabi et al., 2018a, Schmeckpeper et al., 2020]. However, these simple
approaches are unlikely to scale to diverse internet video. This is because
they require either: (i) minimal domain-shift between the video data and the
robot domain (e.g., Baker et al. [2022] assume identical embodiments), or (ii)
an explicit mechanism to deal with domain-shift that may not scale to diverse
internet video [Schmeckpeper et al., 2020] (see Section 5.1.2).

##### Discussion.

Alternative action representations are promising for tackling the problem of
missing action labels in video data. We now discuss which categories of
alternative action representations can be most useful for LfV.

  * How scalable is $\mathcal{\hat{A}}$ to diverse internet video? The degree to which $\mathcal{\hat{A}}$ is readily available in video, or can scale to more unstructured heterogeneous video data, is important. _Observations-as-action_ are immediately available in videos. _Language-as-action_ , whilst very useful, requires the video data to be paired with suitable language descriptions that may not always be available. _Single-step_ and (some) _multi-step latent actions_ can be learned purely from raw unlabelled video, but this learning step adds extra complexity and these methods are largely untested on diverse internet video. _Visual motion information_ and _human-hand-aware_ action spaces can provide useful inductive biases, but it also unclear how well they will scale to more unstructured video datasets.

  * How easy is it to decode the robot action from $\mathcal{\hat{A}}$? Alternative action policies $\pi_{\text{alt}}(\hat{a}_{t}|s_{t})$ can be used help to obtain a downstream robot policy. This is commonly done via a hierarchical conditioning approach $\pi(a_{t}|s_{t},\pi_{\text{alt}}(\hat{a}_{t}|s_{t}))$ or representation transfer $\pi_{\text{alt}}(\hat{a}_{t}|s_{t})\rightarrow\pi(a_{t}|s_{t})$ (see Section 5.2 for more details). In these cases, the ease of decoding $a_{t}$ from $\hat{a}_{t}$ is likely a good proxy for downstream performance gains. We now discuss properties of $\hat{a}_{t}$ that may simplify this decoding: (1) Decoding may be simplified if $\hat{a}_{t}$ contains more mutual information with $a_{t}$. Considering the nature of the information in $a_{t}$, this perhaps advocates for the use of shorter-horizon, or less abstracted, versions of $\mathcal{\hat{A}}$. However, we note that more abstracted action-spaces can be beneficial in long-horizon tasks [Du et al., 2023b], and may suffer less from missing low-level information in video. (2) Minimising the information in $\hat{a}_{t}$ (as seen in Rybkin et al. [2018], Schmidt and Jiang [2023]) could simplify decoding and improve decoding generalization by reducing spurious correlations in the decoding. However, both (1) and (2) are relatively untested in the literature.

  * Levels of action information in video. Video is missing important low-level information for robotics (e.g., it lacks explicit force information). Thus, alternative action representations obtained from video may not be fully informative for decoding robot actions (and so, in many cases, the decoder should be defined as $\pi(a_{t}|\hat{a}_{t},s_{t})$ rather than $\pi(a_{t}|\hat{a}_{t})$). This perhaps suggests that video is best suited for extracting slightly abstracted or longer-horizon action representations. Indeed, research has explored the use of video as a mid-level action representation within a hierarchical framework that also leverages a high-level language-as-action model and a low-level robot action model [Du et al., 2023b].

####  5.1.2 Representations to Address Distribution-Shift

Distribution shift between internet videos and the target robot domain poses a
challenge to LfV approaches (see Section 3.3). For example, a common shift is
the embodiment gap between humans and robots. Such distribution shift can
hinder our ability to transfer knowledge from the video data to the robot.

We are aware of three strategies that may help overcome these distribution
shift issues.

  1. 1.

Scaling to large diverse datasets: Scaling a deep learning method to ever
larger and diverse internet video data may help. Higher coverage in the
pretraining video dataset will minimise unseen shifts encountered in the
downstream robot domain. Additionally, the increased scale of pretraining may
improve the overall generalization abilities of the model.

  2. 2.

Leveraging in-domain robot data: In-domain robot data can be incorporated into
pretraining, or the model can be finetuned on robot data. Robot data contains
information about the robot domain that is not covered in the video data.

  3. 3.

Explicitly addressing distribution-shifts: Techniques can be used to
explicitly address distribution-shifts between the video data and the robot
domain. For example, embodiment invariant representations can be learned
[Schmeckpeper et al., 2020].

Ideally, we hope to overcome distribution shifts using strategies (1) or (2),
as methods from strategy (3) may be less scalable to diverse internet video.
Nevertheless, a number of LfV works have explored strategy (3), and we review
these methods in this section. We identify four main categories of methods,
which we now elaborate on.

##### Method: Human-(hand)-aware approaches.

In LfV, we are most often interested in learning from human behaviour; we wish
for our robots to replicate this behaviour. In particular, for robot
manipulation, we are interested in replicating the behaviour and effects of
the human hand. As such, there is a line of LfV research that explicitly
detects human embodiment-centric information in video, and subsequently
transfers this information to the robot.

  * _What types of human embodiment-centric information can be detected?_

    * –

Poses: Several works explicitly estimate the pose of the human body or hand in
videos. This involves estimating the positions and orientations of various
joints or key points on the body. The human hand is often represented using
the MANO hand model [Romero et al., 2022]. The human-body can be represented
by, for example, by the SMPL/SMPL-X models [Loper et al., 2023, Pavlakos et
al., 2019]. In the LfV literature, often an off-the shelf model (e.g.,
OpenPose [Cao et al., 2017]) is used to crop around the hand, and a second
off-the-shelf model (e.g., FrankMocap [Rong et al., 2020]) is used to detect
the hand pose from the cropped image [Sivakumar et al., 2022, Shaw et al.,
2022]. Once detected, the human-hand pose may need to be retargeted to the
robot embodiment. This can be achieved by directly optimising a loss function
[Shaw et al., 2022], or by training a retargeting network to minimise a
retargeting loss function [Sivakumar et al., 2022].

    * –

Affordances: Affordances are a common notion in robotics [Ardón et al., 2020].
A common affordance detected in human videos in the LfV literature is the
combined use of: (i) grasp/contact points (i.e., where on the object does the
hand make contact?); and (ii) post-grasp/contact trajectories (i.e., once
contact is made, where will the hand move?) [Bahl et al., 2023, Mendonca et
al., 2023]. This affordance information can be extracted from videos using an
off-the-shelf model [Shan et al., 2020], and using further tricks to obtain
accurate, usable affordance labels [Bahl et al., 2023]. Note, 2D affordances
can be converted to 3D using depth estimation [Mendonca et al., 2023].

    * –

Masks and Bounding boxes: Masks or bounding boxes of human hands and objects
can be used [Bharadhwaj et al., 2023]. These can be obtained from videos using
off-the shelf models [Kirillov et al., 2023, Zhang et al., 2022] or via
labelled datasets [Darkhalil et al., 2022].

  * _How to use this human embodiment-centric information?_ With a video dataset labelled with human-centric information, one of the following can be performed: (1) With the labelled dataset, the process for extracting the human-centric information can be distilled into a single model [Mendonca et al., 2023] (i.e., a model is trained to predict the label given an image). This can be useful when the original extraction method is convoluted or unlikely to generalize. It may also be a good auxiliary representation learning objective for robot manipulation [Bahl et al., 2023]. (2) The extracted information can be treated as an ‘alternative action representation’ $\mathcal{\hat{A}}$ [Bharadhwaj et al., 2023, Bahl et al., 2023, Shaw et al., 2022, Qin et al., 2022, 2021] (see Section 5.1.1 for more details on alternative actions).

  * _Limitations._ Human hand poses can be directly retargeted to certain robot hands, and the affordances detailed above are (in theory) embodiment agnostic. However, we note there still can be difficulties transferring this information to the robot. For example, a model trained to propose future poses solely on human images is unlikely to generalise zero-shot to robot images. This has led to tricks being used in the LfV literature, such as predicting affordances only when the embodiment is out of frame [Bahl et al., 2023], or in-painting human embodiments over robot embodiments [Bharadhwaj et al., 2023].

##### Method: Learned invariant representations.

Here we refer to methods that _learn_ a representation invariant to a specific
distribution shift between the video dataset and robot domain. These methods
generally rely on access to data from both distributions. We now briefly list
some methods seen in the LfV literature, before commenting on limitations.