# Learning Real-world Autonomous Navigation by
Self-Supervised Environment Synthesis

Zifan Xu${}^{\ssymbol{1}}$, Anirudh Nair${}^{\ssymbol{1}}$, Xuesu
Xiao${}^{\ssymbol{2},\ssymbol{3}}$, and Peter
Stone${}^{\ssymbol{1},\ssymbol{4}}$ ${}^{\ssymbol{1}}$Department of Computer
Science, The University of Texas at Austin ${}^{\ssymbol{2}}$Department of
Computer Science, George Mason University ${}^{\ssymbol{3}}$Everyday Robots
${}^{\ssymbol{4}}$Sony AI  {zfxu<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Machine learning approaches have recently enabled autonomous navigation for
mobile robots in a data-driven manner. Since most existing learning-based
navigation systems are trained with data generated in artificially created
training environments, during real-world deployment at scale, it is inevitable
that robots will encounter unseen scenarios, which are out of the training
distribution and therefore lead to poor real-world performance. On the other
hand, directly training in the real world is generally unsafe and inefficient.
To address this issue, we introduce Self-supervised Environment Synthesis
(ses), in which, after real-world deployment with safety and efficiency
requirements, autonomous mobile robots can utilize experience from the real-
world deployment, reconstruct navigation scenarios, and synthesize
representative training environments in simulation. Training in these
synthesized environments leads to improved future performance in the real
world. The effectiveness of ses at synthesizing representative simulation
environments and improving real-world navigation performance is evaluated via
a large-scale deployment in a high-fidelity, realistic simulator111Due to the
lack of access to large-scale real-world deployment data, we use simulated
Matterport environments [1] as a surrogate of the real world. and a small-
scale deployment on a physical robot.

##  I INTRODUCTION

While classical navigation systems have been able to move mobile robots from
one point to another in a collision-free manner for decades [2, 3], learning-
based approaches to navigation have recently gained traction [4] due to their
ability to learn navigation behaviors purely from data without extensive
engineering effort. For example, learned navigation systems can learn from
human demonstrations [5] or self-supervised trial and error [6]; they can
learn navigation cost functions that consider social norms and human
preferences [7]. They can also be combined with classical navigation systems
to assure navigation safety and enable adaptive behaviors in different
scenarios [8, 9, 10, 11].

Figure 1: A navigation policy trained in simulation is expected to be deployed
in completely different domains of navigation environments in the real world
(e.g., households, factories, and parks). The policy may also face different
real-world inter-domain deployments, in which a navigation policy learned in
one real-world domain will be deployed in another.

Due to the expense of trial-and-error training in the real world (e.g., safety
concerns and sample efficiency), most navigation behaviors are learned in
artificially created environments in simulation, which may not generalize well
to the real world (see Fig. 1). Despite efforts to create simulation
environments similar to the real world or enable efficient sim-to-real
transfer, it is inevitable that robots will encounter unfamiliar scenarios,
especially in large-scale real-world deployments.

The goal of this work is to improve real-world autonomous navigation with
safety and efficiency requirements based on mobile robots’ own navigation
experiences during actual deployment. These conservative, potentially
suboptimal, real-world experiences (without risky real-world exploration) may
not be sufficient to directly train an RL agent, but may be sufficient to
reconstruct the real-world navigation scenarios which an RL agent can interact
with and actively explore in simulation. On the other hand, given the large
amount of real-world deployment experiences available to many robots deployed
in the field (consider 7 million connected iRobot Roombas vacuuming homes day
to day), it is infeasible to reconstruct all these deployment environments and
train in simulation on a daily basis.

With this motivation in mind, this paper introduces Self-supervised
Environment Synthesis (ses), which enables mobile robots deployed in the field
to first reconstruct navigation scenarios from experiences and then synthesize
a representative set of simulation environments that is feasible for RL
training. Training in these simulated environments enables robots to learn to
address real-world challenges that they are likely to encounter.

Importantly, the distribution of real-world navigation scenarios is often
unbalanced, including mostly trivial open scenarios. Therefore, we use an
efficient strategy that filters out the trivial scenarios by a measure of
navigation difficulty and focus learning on the challenging navigation
scenarios. To synthesize the training environment set from the reconstructed
challenging navigation scenarios, three different environment synthesis
approaches—Generative Adversarial Networks (gan), K-means clustering with
Principle Component Analysis (pca), and random sampling—are employed to
represent the challenging scenarios with a concise training environment set
that is feasible for an RL agent to learn from. We denote the pipelines with
three environment synthesis approaches as ses-gan, ses-pca and ses-rs
respectively. We evaluate the three ses pipelines in Matterport, a dataset of
many simulated realistic household environments (which serves as a surrogate
of real world), and show that ses improves the deployment in these
environments compared to policies trained in artificially generated
environments [12], and the best pipeline ses-gan generates more representative
training environments and enables better deployment in Matterport than the
pipelines with other synthesis approaches.

##  II RELATED WORK

###  II-A Classical and Learning-Based Navigation

Mobile robot navigation has been investigated by roboticists for decades [2,
3]. Classical approaches can move robots from one point to another with a
reasonable degree of confidence that they won’t collide with any obstacles.
However, these approaches require extensive engineering to develop in the
first place and to adapt to different environments. Moreover, when
encountering an environment in which a robot has failed or achieved suboptimal
behavior before, without re-engineering the system, the robot will likely
repeat the same mistake again.

Inspired by the success of machine learning in other domains, roboticists have
also applied machine learning to autonomous navigation [4]. Most learning
approaches to navigation adopt an end-to-end approach, i.e., learning a
mapping from perceptual input directly to motion commands. Such approaches
require comparatively less engineering effort, and learn navigation behaviors
purely from data [13], e.g., from expert demonstrations [5, 14] or from trial
and error [6, 15]. However, these approaches often lack safety guarantees and
explainability, as provided by their classical counterparts. Therefore,
roboticists have also investigated more structured ways of integrating
learning with classical navigation, such as learning local planners [16, 17,
18], terrain-based cost functions [19], planner parameters [10, 11], driving
styles [20], or social norms [7]. Their success notwithstanding, learning-
based navigation approaches inherit one drawback from machine learning
approaches in general: poor generalizability when facing out-of-distribution
data. When deployed in the real world, especially at large scale, it is
inevitable that mobile robots will encounter scenarios that are not included
in their training distribution.

ses combats classical navigation’s inability to improve from experience [21]
and learning approaches’ poor generalizability to real-world scenarios. It
improves navigation by synthesizing training environments from real-world
deployment experiences.

###  II-B Sim-to-real Transfer

Limited by the safety and efficiency requirements in the real world, a
learning-based navigation system is usually trained in simulation. However,
policies trained in simulation can perform poorly in the real world due to the
mismatch between the simulation and the real world. This phenomenon is
commonly referred to as the _sim-to-real gap_.

One major source of the _sim-to-real gap_ is the discrepancies between the
sensor input rendered in simulation and the real robot’s sensors. For example,
to bridge the gap between real-world and synthetic camera images of a robotic
system, prior work has employed techniques such as pixel-level domain
adaptation, which translates synthetic images to realistic ones at the pixel
level [22, 23]. These adapted pseudo-realistic images bridge the _sim-to-real
gap_ to some extent, so policies learned in simulation can be executed more
successfully on real robots by adapting the images to be more realistic.
Another source of the _sim-to-real gap_ is caused by dynamics mismatch between
simulation and the real world e.g., due to an imperfect physics engine. A
common paradigm to reduce the dynamics mismatch is Grounded Simulation
Learning (GSL), which either directly modifies (i.e., grounds) the simulator
to better match the real world [24], or learns an action transformer that
induces simulator transitions that more closely match the real world [25, 26].

In contrast to the two _sim-to-real gaps_ introduced above, this work
addresses a gap caused by the environmental mismatch (e.g., differences in the
configurations and shapes of obstacles, and start-goal locations). ses can be
thought of as an environmental grounding method that minimizes the differences
in navigation environments between simulation and the real world based on the
navigation experiences collected during real-world deployment.

##  III APPROACH

In this section, We first formulate large-scale real-world navigation as a
multi-task RL problem in an unknown navigation domain, which is defined as a
distribution of navigation tasks. Sec. III-A formally defines the navigation
task and describes how a distribution of navigation tasks forms a navigation
domain. Then, Sec. III-B and III-C discuss the two stages of ses: real-world
navigation domain extraction from real-world deployment data and environment
synthesis that generates a representative set of navigation tasks. The whole
pipeline of ses is summarized in Alg. 1.

###  III-A Navigation Task and Navigation Domain

We focus on a standard goal-oriented navigation task, in which a robot
navigates in a navigation environment $e$ from a provided starting pose
$\alpha$ to a goal pose $\beta$. Each navigation task $T$ is instantiated as a
tuple $T=(e,\alpha,\beta)$. In real-world applications, robots are not
deployed to navigate in one single environment or with the same start and goal
all the time. Instead, actual deployments in the real world usually entail a
distribution over multiple environments with many start and goal poses. In
this case, we represent the real-world deployment as a navigation domain
$p_{\text{real}}$ defined as a distribution of navigation tasks
$p_{\text{real}}(T)$. ses generates a new navigation domain $p_{\textsc{ses}}$
in simulation that, with a limited number of tasks, models the distribution of
tasks in ${p_{\text{real}}}$ so that the navigation performance of policies
trained in $p_{\textsc{ses}}$ is likely to be strong in the real-world
navigation domain ${p_{\text{real}}}$. ses uses applr [10] as the learning
approach that solves the navigation tasks by training a parameter policy that
dynamically adjusts the hyper-parameters of a classical navigation stack.
Although our implementation of ses is based on a specific learning approach
(applr), we leave the formulation sufficiently broad so that applr can be
replaced with any RL-based approach to autonomous navigation.

###  III-B Real-world Navigation Domain Extraction

ses seeks to explore the distribution of the real-world navigation domain
during real deployment of an existing navigation policy $\pi_{0}$ (e.g., a
policy pretrained on artificially created environments or a classical
navigation system). In each real-world deployment, a navigation task
$T_{n}\sim p_{\text{real}}(T)$ is sampled from the real-world navigation
domain. At each deployment time step $k$, ses records the sensory input
$x_{k}$, the robot’s position $s_{k}$ in the world frame, and the number of
suboptimal or failed navigation behaviors $c_{k}$ (negative velocity command
is used as the indication of failure in this paper).222The navigation policy
used in this paper is based on a classical navigation system dwa, in which,
suboptimal recovery behavior assigns negative linear velocity to the robot
Each deployment will return a trajectory
$\tau=(x_{0},s_{0},c_{0},...,x_{k},s_{k},c_{k},...,x_{K},s_{K},c_{K})$ with
$K$ being the total number of steps during a deployment. Then, the
trajectories are segmented into different scenarios $\eta$ as follows: (1)
starting from an _initial_ step $(x_{i},s_{i},c_{i})$, iterate the trajectory
until a _final_ step $(x_{f},s_{f},c_{f})$ at which the robot is 5m away from
the _initial_ step; (2) record a scenario $\eta$ as a sub-trajectory of all
the steps between step $i$ and step $f$; (3) after one scenario is recorded,
set the _final_ step as the new _initial_ step for the next iteration. The
procedure is repeated until the whole trajectory is processed. Assuming a
total of $N$ rounds of deployments with trajectories segmented into $J$
scenarios, a trajectory set $\\{\tau_{n}\\}_{n=1}^{N}$ is collected and
converted into a scenario set $\\{\eta_{j}\\}_{j=1}^{J}$.

For each of the scenarios, we reconstruct a 5m-by-5m navigation environment
with start and goal set to be the positions of the initial and final steps,
and we re-orient the environment so that the start and goal are located at the
middle points of the bottom and top edges respectively. The obstacles and free
spaces can be reconstructed from the recorded sensory inputs or from available
maps. The constructed environment set is denoted as
$\\{e_{j},\bar{c}_{j}\\}_{j=1}^{J}$. Each environment $e_{j}$ is associated
with a total number of suboptimal navigation behaviors $\bar{c}_{j}$ between
the _initial_ step and _final_ step. Fig. 2 shows an example of such a
trajectory segmentation and environment construction process. Each environment
is represented as a 30$\times$30 binary matrix with each element denoting the
occupancy state of a 1.5m$\times$1.5m space. The benefits of such scenario
segmentation are: (1) encoding the start and goal into the orientation of the
environment so that a single environment distribution can represent the real-
world navigation domain distribution; (2) the segmented navigation
environments have roughly the same length, which makes it easier for an RL
agent to learn under roughly the same magnitudes of return and episode lengths
across different environments. Alg. 2 summarizes the above process of
generating an environment set that represents the real-world navigation domain
distribution.

Figure 2: An example of trajectory segmentation and environment construction:
dashed line denotes the actual trajectory of the robot. Three scenarios are
segmented and constructed as navigation environments at the bottom. Red dots
mark the positions of the start and goal in the environments. The numbers of
suboptimal behaviors $c$ are noted at the bottom.

A ”raw” navigation domain $p_{\text{raw}}=U(\\{e_{i}\\}_{j=1}^{J})$ can be
defined as a uniform distribution over all these constructed navigation
environments. However, it is likely that $p_{\text{raw}}$ includes many
trivial navigation tasks (e.g., open spaces), whereas, efficient training
should focus on the challenging navigation tasks. We use a threshold of 50
suboptimal behaviors within a scenario $\eta$ as an indication of easy and
challenging environments. The threshold of 50 is empirically picked such that
the trivial scenarios with mostly empty spaces will be filtered out while not
losing the challenging scenarios. Those challenging environments,
$\\{e_{j},\bar{c}_{j}|\bar{c}_{j}>50\\}_{j=1}^{J}$, form a set for further
environment synthesis. We denote the navigation domain with only challenging
navigation environments as
$p_{\text{chal}}=U(\\{e_{j}|\bar{c}_{j}>50\\}_{j=1}^{J})$. Our empirical
results indicate that training only on the navigation tasks with challenging
scenarios instead of all the real-world scenarios is essential for improving
real-world deployments.

###  III-C Environment Synthesis

While simply using all the challenging environments
$\\{e_{j}|\bar{c}_{j}>50\\}_{j=1}^{J}$ can precisely represent
$p_{\text{chal}}$ in real world, it is impractical to train an RL agent in all
these environments. In this case, a smaller set of training environments that
represents the real-world navigation domain distribution is required.
Therefore, we propose three methods of environment synthesis that perform this
training data selection process and generate a concise and representative
training environment set. The three environment synthesis methods are: (1)
Generative Adversarial Nets (gan) [27]; (2) K-means clustering with Principal
Component Analysis (pca) [28]; and (3) random sampling. We describe gan,
K-means clustering with pca, and random sampling as follows.

gan learns a generator’s distribution $p_{g}$ over all the possible
environments $e\in\\{1,0\\}^{30\times 30}$ to match a uniform distribution
$p_{\text{chal}}$ over the challenging environment set. Given a prior on the
input noise variables $p_{z}(z)$, a generator $G(z;\theta_{g})$ is defined as
a mapping from the input variable space to the environment space with
$\theta_{g}$ representing the parameters of the network. A discriminator
network $D(e;\theta_{d})$ outputs the probability that $e$ comes from
$p_{\text{chal}}$ rather than $p_{g}$. $D$ is trained to maximize the
probability of assigning the correct label to both challenging environments
and generated environments from $G$. Simultaneously, $G$ is trained to
minimize $\log(1-D(G(z)))$. To summarize, $D$ and $G$ play the following two-
player minimax game with value function $V(G,D)$:

$\displaystyle\min_{G}\max_{D}V(D,G)=\mathbb{E}_{e\sim
p_{\text{chal}}(e)}[\log(D(e))]+$ (1) $\displaystyle\mathbb{E}_{z\sim
p_{z}(z)}[\log(1-D(G(z)))].$

After a generator $G$ is learned, we query the generator $M$ times to draw $M$
environments as the training set, where $M$ is the size of the training
environment set dependent on the training budget of the RL algorithm.

K-means clustering with pca first reduces the dimensionality of the original
environment representation (30$\times$30 binary matrix) by pca so that each
environment can be represented by their principle components (we empirically
pick 100 principle components in our experiments because they can reasonably
reconstruct the original environments). Then, K-means is employed to cluster
the environments in the reduced space into $m$ clusters and sample $n$
environments from each cluster to select $M=m\times n$ representative
environments in the challenging environment set. We use the principle
components to reconstruct the representative environments and use them as the
training set.

Random sampling simply uses $p_{\text{SES}}=p_{\text{chal}}$ the uniform
distribution over all the challenging environments. Then, $M$ environments are
drawn from $p_{\text{SES}}$ to form the training set.

The synthesized navigation domain is denoted as $p_{\text{SES}}$. Finally, an
applr policy $\pi$ is trained in $p_{\text{SES}}$. In practice, the policy
$\pi$ is initialized to be the deployed policy $\pi_{0}$ that was trained on
the artificially created environments to speed up training.

Algorithm 1 Self-supervised Environment Synthesis

0: Original policy $\pi_{0}$, real-world navigation domain distribution
$p_{\text{real}}$, number of training environments $M$

1: Raw navigation domain $p_{\text{raw}}$ $\leftarrow$ Real-world Navigation
Domain Extraction based on $\pi_{0}$ and $p_{\text{real}}$ (Sec. III-B)

2: Synthesized navigation domain $p_{\text{SES}}$ $\leftarrow$ Environment
Synthesis based on $p_{\text{raw}}$ (Sec. III-C)

3: $\\{e_{i}\\}_{i=1}^{M}$ $\leftarrow$ Sample $M$ environments from
$p_{\text{SES}}$

4: $\pi\leftarrow$ Initialize with $\pi_{\text{0}}$, then train on
$\\{e_{i}\\}_{i=1}^{M}$

5: return $\pi$

Algorithm 2 Real-world Navigation Domain Extraction

0: Original policy $\pi_{0}$, real-world navigation domain distribution
$p_{\text{real}}$ and total number of deployments $N$.

1: $E=\emptyset$

2: for $n\in\\{1,...,N$} do

3: $T_{n}\sim p_{\text{real}}(T)$

4: Trajectory $\tau_{n}\leftarrow$ deploy $\pi_{0}$ on $T_{n}$

5: Initialize step $i=0$ and the scenario set $\eta=\emptyset$

6: for $x_{k},s_{k},c_{k}\in\tau_{n}$ do

7: $\eta\leftarrow\eta\cup\\{(x_{k},s_{k},c_{k})\\}$

8: if Euclidean($s_{k}$, $s_{i}$) $>5$ then

9: $e$ $\leftarrow$ Construct Environment from $\eta$

10: $\bar{c}=\sum_{i}^{k}c_{i}$

11: $E\leftarrow E\cup\\{e,\bar{c}\\}$

12: $i=k$, $\eta=\emptyset$

13: end if

14: end for

15: end for

16: return $E$

##  IV RESULTS

In this section, we describe an implementation of ses on a ground robot and
show that ses can efficiently synthesize representative environments based on
challenging navigation scenarios during real-world deployment and successfully
improve navigation by learning from the synthesized environments. Due to the
difficulty in accessing large-scale physical robot deployment data in the real
world, we use Matterport [1], a set of high-fidelity simulation environments
as a surrogate for a large-scale real-world deployment (see Fig. 3 (d)). We
assume that this environment set is not available before actual deployment,
and the robot needs to learn through its actual deployment experiences in
those environment to improve navigation.

###  IV-A Deployment in Matterport Navigation Domain

We employ ses on a simulated ClearPath Jackal differential-drive ground robot.
The specifications of the robot are kept the same as in applr [10], including
the ros move_base navigation stack, the underlying classical local planner
dwa, and the learned planner parameters.

The original applr policy was trained before any deployment in the barn
dataset [12]. The dataset contains 300 simulated navigation environments
randomly generated by Cellular Automata. Even though those environments are
designed to be diverse enough to cover different difficulty levels from
relatively open spaces to highly-constrained ones, when a controller learned
on these environments is deployed in Matterport, the randomly generated
environments in barn are found not to cover some specific obstacle shapes and
arrangements encountered by the robot, and therefore lead to the failures at
the deployment time. In Fig. 3, (a)-(c) show three example barn environments,
and (d) shows a Matterport environment where Jackal will be deployed. This
realistic Matterport environment may contain challenging navigaiton scenarios
that are unlikely to be generated by random sampling.

After training the original applr policy with the barn dataset, the robot will
be physically deployed in the real world at scale. Due to the lack of access
to large-scale real-world physical deployment data, we use 55 simulated house
layouts from the Matterport dataset [1] as a surrogate for physical real-world
deployment. Fig. 3 (d) shows an example of such a deployment environment. We
randomly sample 50 environments from the Matterport dataset [1] as the
environments where Jackal will be actually deployed. During actual deployment,
these environments serve as the Matterport navigation domain and navigation
experiences in those environments during deployment provide data for ses to
synthesize representative environments of the Matterport navigation domain and
to improve navigation. We leave five environments to test the learned policy
on these held-out Matterport environments.

Figure 3: Examples of training and deployment environments. (a), (b), (c):
example environments from the barn dataset; (d): a top-down view of a
Matterport environment created from the occupancy map shown at the top-right
corner.

###  IV-B ses Implementation

Matterport navigation domain representation: in the large-scale deployment in
Matterport environments, the robot with the original policy $\pi_{0}$ trained
in barn (artificially created navigation domain) is deployed 100 times to
navigate between the starting and goal locations randomly sampled from all the
navigable start-goal pairs in each Matterport environment, which accounts for
5000 deployments in total. We construct 16528 navigation scenarios from the
segmented trajectories using the method described in Sec. III-B as
$p_{\text{raw}}$ the distribution of the Matterport navigation domain. Then
3769 challenging environments are selected based on a threshold 50 of the
negative linear velocity count, which forms $p_{\text{chal}}$. Fig. 4 (left)
demonstrates some examples out of the 3769 challenging environments. Each of
the environments is represented as a binary matrix of size 30 $\times$ 30\.
Note that training in $p_{\text{chal}}$ with all 3769 environments is not
computationally practical due the limit of CPUs that run the environments in
parallel and the budget of total number of training steps. In the meantime,
realistic large-scale robot deployments in the real world (in contrast to our
surrogate) may generate orders of magnitude more challenging scenarios.

Figure 4: Examples of raw challenging environments (top) and generated
environments (bottom).

Environment synthesis: given the extracted Matterport navigation domain
$p_{\text{chal}}$, the environment synthesis method described in Sec. III-C
generates 100 training environments (the maximum number of environments that
can be trained in parallel in our training pipeline) by using the challenging
environments as an image dataset to train a gan. More specifically, we employ
a normal distribution prior on the input noise variables
$p_{z}(z):=\mathcal{N}(0,1)$, where $\mathcal{N}$ is the standard normal
distribution with mean and standard deviation equal to 0 and 1 respectively.
The generator $G(z;\theta_{g})$ is represented by a multilayer perceptron
(MLP) of 256, 512, and 1024-hidden-unit layers with batch normalization and
leaky ReLU activation function. The generator outputs a 900-dimensional vector
with a fully connected layer followed by a Tanh activation function. The
output vector is further rendered into a $30\times 30$ binary matrix using a
threshold of 0. The discriminator $D(x;\theta_{d})$ is represented by a second
MLP of 512, 256, and 256-hidden-unit layers with leaky ReLU activation
function. Each hidden layer is followed by a batch normalization layer and a
dropout layer with a dropout rate equal to 0.5. The dropout serves as an
ensemble strategy that prevents mode collapse in the generated images [29].
Fig. 2 (right) shows samples of generated images drawn from the generator
after training. The generator is queried 100 times to generate synthesized
navigation domain $p_{\text{SES}}$ with 100 training environments, which is in
the same order of magnitude as the barn dataset. We leave the investigation of
how ses performs with different numbers and sizes of training environments as
future work. To train the new policy $\pi$ that solves the navigation in
challenging Matterport environments, we use the same set of hyper-parameters
as when training $\pi_{0}$ except for a smaller training step of one fourth of
the original 4M training step, because the pre-trained policy $\pi$ needs
fewer training step to converge.

TABLE I: Average time costs and success rates | Time cost (s) | Success rate (%) |
---|---|---|---
ses-gan | 27.23 | 85.0 |
ses-pca | 32.21 | 80.6 |
ses-rs | 35.92 | 71.0 |
_Learn from scratch_ | 30.42 | 74.3 |
Original policy | 36.33 | 71.9 |
_dwa slow_ | 43.76 | 75.5 |
_dwa fast_ | 31.72 | 67.2 |

###  IV-C Test Results

We train the new applr policy $\pi$ in $p_{\text{SES}}$ as created by the
three ses methods: ses-gan, ses-pca, and ses-rs. We compare the methods with
four baselines: the applr policy learned from scratch on the training
environments generated by gan, the original applr policy $\pi_{0}$, and
vanilla dwa with two sets of static hyper-parameters, one having default
parameters and the other having a two times higher maximum linear velocity and
sampling rate. We call the two classical motion planners _dwa slow_ and _dwa
fast_ respectively. We use _learn from scratch_ to denote the first baseline
which is designed to verify the importance of initializing the policy with the
original policy $\pi_{\text{0}}$. Then we deployed the three adapted RL
policies and four baselines in 5 held-out Matterport environments to test
deployment performance. For each of the environments, we randomly sample five
navigable start-goal pairs that have at least 5-meter start-goal distances ,
and deploy each policy 20 times for each pair. During deployment, we measure
the success rate and the total time cost of the successful trials.

Table I presents the average time cost and success rate over all trials.
333Standard deviation is not included in the table due to the large variance
between different environments and start-goal pairs, but we analyze the
variance of the performance in every individual environment in Fig. 5. The
policy trained by ses-gan achieves the best performances of a 27.23s average
time cost and a 85% success rate. ses-pca follows with a 32.21s average time
cost and a 80.6% success rate. _Learn from scratch_ shows worse performance on
both metrics compared to ses-gan which indicates the importance of
initializing the model with the pre-trained policy $\pi_{\text{0}}$. The
policy trained by ses-rs performs comparably to the original policy that is
not trained on the Matterport environments at all, which indicates the poor
representation of the challenging Matterport environments with random
sampling. While _dwa slow_ achieves an acceptable success rate of 75.5%, it
takes much longer (43.76s) to finish the navigation. On the other hand, _dwa
fast_ can navigate relatively quickly, but more easily fails with the lowest
success rate of 67.2%.

To analyze performance variance, we randomly select one start-goal pair from
each environment and show the average time cost and success rate over 20
independent trials in Fig. 5. The policies trained by ses-gan and ses-pca, in
general, complete the navigation task more quickly and achieve higher success
rate. Among five test environments, the ses-gan policy achieves the lowest
time cost in three environments and the highest success rate in all five. ses
methods tend to show larger improvements in the more difficult environments (1
and 2), but also larger performance variance.

Figure 5: The average time cost (top) and the success rate (bottom) of 5
policies tested in 5 held-out environments.

###  IV-D Physical Experiment

To verify that ses is potentially applicable to real-world robots, we deploy
the pre-trained applr policy on a physical Jackal robot. The pre-trained
policy is deployed in a real-world environment and a map of the environment is
built during deployment using gmapping [30] (shown in Fig. 6). Using the same
criteria for suboptimal behavior as the Matterport experiments, the robot
identifies two challenging scenarios shown by the orange and yellow areas in
the map.

Considering the limited scale of our real-world deployment, we directly
construct two synthetic environments which are identical to the challenging
real-world scenarios (instead of using a gan) and fine-tune the pre-trained
policy in them. We denote the policies before and after fine-tuning as
_original policy_ and ses respectively. Table II reports the time cost
averaged over 5 trials of both policies to traverse the two difficult real-
world scenarios, where ses achieves significant improvement compared to the
_original policy_.

Figure 6: Real-world navigation path (red dashed line) and map. The orange and yellow areas mark the two challenging scenarios identified during the real-world deployment. TABLE II: Average time cost of the tested policies in real-world challenging scenarios | Scenario 1 | Scenario 2
---|---|---
Original policy |  $18.5\pm 7.6$ (s) |  $38.4\pm 9.8$ (s)
ses |  $11.3\pm 0.9$ (s) |  $15.6\pm 1.0$ (s)

##  V CONCLUSION

This paper introduces a self-supervised environment synthesis approach that
improves the real-world navigation performance of a learning-based navigation
system by synthesizing realistic simulation environments based on navigation
experiences during real-world deployment. To address the inevitable
environmental mismatch between simulation and real world deployment
environments, especially for large-scale real-world deployment, ses can
successfully identify challenging navigation scenarios during the deployment
in high-fidelity Matterport environments and synthesize representative
environments to be added to the training distribution. SES has also been
demonstrated in a small-scale real-world deployment.

## References

  * [1] A. Chang, A. Dai, T. Funkhouser, M. Halber, M. Niessner, M. Savva, S. Song, A. Zeng, and Y. Zhang, “Matterport3d: Learning from rgb-d data in indoor environments,” _International Conference on 3D Vision (3DV)_ , 2017.
  * [2] S. Quinlan and O. Khatib, “Elastic bands: Connecting path planning and control,” in _[1993] Proceedings IEEE International Conference on Robotics and Automation_. IEEE, 1993, pp. 802–807.
  * [3] D. Fox, W. Burgard, and S. Thrun, “The dynamic window approach to collision avoidance,” _IEEE Robotics & Automation Magazine_, vol. 4, no. 1, pp. 23–33, 1997.
  * [4] X. Xiao, B. Liu, G. Warnell, and P. Stone, “Motion control for mobile robot navigation using machine learning: a survey,” _arXiv preprint arXiv:2011.13112_ , 2020.
  * [5] M. Pfeiffer, M. Schaeuble, J. Nieto, R. Siegwart, and C. Cadena, “From perception to decision: A data-driven approach to end-to-end motion planning for autonomous ground robots,” in _2017 ieee international conference on robotics and automation (icra)_. IEEE, 2017, pp. 1527–1533.
  * [6] J. Zhang, J. T. Springenberg, J. Boedecker, and W. Burgard, “Deep reinforcement learning with successor features for navigation across similar environments,” in _2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2017, pp. 2371–2378.
  * [7] N. Pérez-Higueras, F. Caballero, and L. Merino, “Teaching robot navigation behaviors to optimal rrt planners,” _International Journal of Social Robotics_ , vol. 10, no. 2, pp. 235–249, 2018.
  * [8] H. Ma, J. S. Smith, and P. A. Vela, “Navtuner: Learning a scene-sensitive family of navigation policies,” in _2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2021, pp. 492–499.
  * [9] S. Daftry, N. Abcouwer, T. Del Sesto, S. Venkatraman, J. Song, L. Igel, A. Byon, U. Rosolia, Y. Yue, and M. Ono, “Mlnav: Learning to safely navigate on martian terrains,” _IEEE Robotics and Automation Letters_ , vol. 7, no. 2, pp. 5461–5468, 2022.
  * [10] Z. Xu, G. Dhamankar, A. Nair, X. Xiao, G. Warnell, B. Liu, Z. Wang, and P. Stone, “APPLR: Adaptive planner parameter learning from reinforcement,” in _2021 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2021.
  * [11] X. Xiao, Z. Wang, Z. Xu, B. Liu, G. Warnell, G. Dhamankar, A. Nair, and P. Stone, “Appl: Adaptive planner parameter learning,” _arXiv preprint arXiv:2105.07620_ , 2021.
  * [12] D. Perille, A. Truong, X. Xiao, and P. Stone, “Benchmarking metric ground navigation,” in _2020 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR)_. IEEE, 2020, pp. 116–121.
  * [13] D. A. Pomerleau, “Alvinn: An autonomous land vehicle in a neural network,” Carnegie Mellon University, Tech. Rep., 1989.
  * [14] X. Xiao, J. Biswas, and P. Stone, “Learning inverse kinodynamics for accurate high-speed off-road navigation on unstructured terrain,” _IEEE Robotics and Automation Letters_ , 2021.
  * [15] H. Karnan, G. Warnell, X. Xiao, and P. Stone, “Voila: Visual-observation-only imitation learning for autonomous navigation,” _arXiv preprint arXiv:2105.09371_ , 2021.
  * [16] X. Xiao, B. Liu, G. Warnell, and P. Stone, “Toward agile maneuvers in highly constrained spaces: Learning from hallucination,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 2, pp. 1503–1510, 2021.
  * [17] X. Xiao, B. Liu, and P. Stone, “Agile robot navigation through hallucinated learning and sober deployment,” in _2021 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2021.
  * [18] Z. Wang, X. Xiao, A. J. Nettekoven, K. Umasankar, A. Singh, S. Bommakanti, U. Topcu, and P. Stone, “From agile ground to aerial navigation: Learning from learned hallucination,” in _2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2021.
  * [19] M. Wigness, J. G. Rogers, and L. E. Navarro-Serment, “Robot navigation from human demonstration: Learning control behaviors,” in _2018 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2018, pp. 1150–1157.
  * [20] M. Kuderer, S. Gulati, and W. Burgard, “Learning driving styles for autonomous vehicles from demonstration,” in _2015 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2015, pp. 2641–2646.
  * [21] B. Liu, X. Xiao, and P. Stone, “A lifelong learning approach to mobile robot navigation,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 2, pp. 1090–1096, 2021.
  * [22] K. Bousmalis, A. Irpan, P. Wohlhart, Y. Bai, M. Kelcey, M. Kalakrishnan, L. Downs, J. Ibarz, P. Pastor, K. Konolige _et al._ , “Using simulation and domain adaptation to improve efficiency of deep robotic grasping,” in _2018 IEEE international conference on robotics and automation (ICRA)_. IEEE, 2018, pp. 4243–4250.
  * [23] K. Rao, C. Harris, A. Irpan, S. Levine, J. Ibarz, and M. Khansari, “Rl-cyclegan: Reinforcement learning aware simulation-to-real,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 11 157–11 166.
  * [24] A. Farchy, S. Barrett, P. MacAlpine, and P. Stone, “Humanoid robots learning to walk faster: From the real world to simulation and back,” in _Proceedings of the 2013 international conference on Autonomous agents and multi-agent systems_ , 2013, pp. 39–46.
  * [25] H. Karnan, S. Desai, J. P. Hanna, G. Warnell, and P. Stone, “Reinforced grounded action transformation for sim-to-real transfer,” in _IEEE/RSJ International Conference on Intelligent Robots and Systems(IROS 2020)_ , October 2020.
  * [26] J. P. Hanna, S. Desai, H. Karnan, G. Warnell, and P. Stone, “Grounded action transformation for sim-to-real reinforcement learning,” _Special Issue on Reinforcement Learning for Real Life, Machine Learning, 2021_ , May 2021.
  * [27] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” _Advances in neural information processing systems_ , vol. 27, 2014.
  * [28] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg _et al._ , “Scikit-learn: Machine learning in python,” _the Journal of machine Learning research_ , vol. 12, pp. 2825–2830, 2011.
  * [29] G. Mordido, H. Yang, and C. Meinel, “Dropout-gan: Learning from a dynamic ensemble of discriminators,” _arXiv preprint arXiv:1807.11346_ , 2018.
  * [30] B. Gerkey, “Ros wiki gmapping,” http://wiki.ros.org/gmapping, 2018.


HPs associated with this model in Section F.4. However these HPs poorly
satisfy our cardinality, independence and interpretability criteria from
Section 4.3, so in the Section G.2.1 we present a re-parametrization of these
HPs designed to better satisfy these points. In Section G.2.2 we then combine
these HPs with the final unit-scaled pre-norm residual scheme we derived in
Appendix F, resulting in our complete u-µP residual scheme.

####  G.2.1 Improved hyperparameters for transformer residuals

To avoid cluttered notation, in this section we rename

$\displaystyle\alpha_{\mathrm{res}}=\alpha_{r},\quad\alpha_{\mathrm{res{\text{-}}attn{\text{-}}ratio}}=\alpha_{\rho}$
$\displaystyle\alpha_{\mathrm{emb}}=\alpha_{e}$
$\displaystyle,\quad\alpha_{\mathrm{attn{\text{-}}residual}}=\alpha_{a}\quad\alpha_{\mathrm{ffn{\text{-}}residual}}=\alpha_{f}.$

To make the presentation more clear, we derive our new HPs using the standard
residual scheme from Equations 6, 7 and 8. For the actual unit scaled
implementation one needs to transform the multipliers following Equations 15,
16, 17 and 18, which we do in Section G.2.2.

To facilitate our analysis, we can view the transformer residual output as the
sum of three terms:

$\displaystyle R_{L}$ $\displaystyle=R_{L}^{(e)}+R_{L}^{(a)}+R_{L}^{(f)},$
$\displaystyle R_{L}^{(e)}$ $\displaystyle:=\alpha_{e}x,$ $\displaystyle
R_{L}^{(a)}$
$\displaystyle:=\sum_{l=1}^{L/2}\frac{\alpha_{a}}{\sqrt{L/2}}f_{2l-1}(R_{2l-1}(x)),$
$\displaystyle R_{L}^{(f)}$
$\displaystyle:=\sum_{l=1}^{L/2}\frac{\alpha_{f}}{\sqrt{L/2}}f_{2l}(R_{2l}(x)),$

and define the average residual scale,

$\sigma(R_{L}^{(a,f)})^{2}:=\frac{\sigma(R_{L}^{(a)})^{2}+\sigma(R_{L}^{(f)})^{2}}{2}.$

Note that we have added in the depth-µP multipliers here, though a similar
analysis can be performed for non-depth-µP models. As above, $f_{l}$ functions
alternate between self-attention layers and feed-forward layers.

With respect to our interpretability criterion, we propose two new multipliers
that correspond to dynamics in the network which we suggest are important to
control at initialization. The first is the ratio of the average scale of the
residuals’ contributions to those of the embedding,
$\alpha_{r}=\sigma(R_{L}^{(a,f)})/\sigma(R_{L}^{(e)})$. The second is the
ratio of the scale of the attention-residuals’ contributions to those of the
feed-forward-residuals,
$\alpha_{\rho}=\sigma(R_{L}^{(a)})/\sigma(R_{L}^{(f)})$. Not only do these two
ratios control key dynamics of our model, but we can use them to replace our
existing $(\alpha_{e},\alpha_{a},\alpha_{f})$ multipliers.

Let us first examine these two quantities for a standard (non-unit-scaled
model). Residual functions of the same kind have the same expected output
scale at initialization in pre-norm networks, meaning we can denote the output
scale $\sigma(f_{l}(R_{l}))$ of all self-attention functions as $\sigma_{a}$,
and of all feed-forward functions as $\sigma_{f}$. We thus have the following
scales at the output:

$\displaystyle\sigma(R_{L}^{(e)})$ $\displaystyle=\alpha_{e}\sigma(x),$
$\displaystyle\sigma(R_{L}^{(a)})$
$\displaystyle=\frac{\alpha_{a}}{\sqrt{L/2}}\,\sigma\\!\left(\sum_{i=1}^{L/2}f_{2l-1}(R_{2l-1})\right)=\alpha_{a}\sigma_{a},$
$\displaystyle\sigma(R_{L}^{(f)})$
$\displaystyle=\frac{\alpha_{f}}{\sqrt{L/2}}\,\sigma\\!\left(\sum_{i=1}^{L/2}f_{2l}(R_{2l})\right)=\alpha_{f}\sigma_{f},$
$\displaystyle\sigma(R_{L}^{(a,f)})$
$\displaystyle=\sqrt{\frac{(\alpha_{a}\sigma_{a})^{2}+(\alpha_{f}\sigma_{f})^{2}}{2}}.$

Recalling our definitions of $\alpha_{r},\alpha_{\rho}$ above, this gives us:

$\displaystyle\alpha_{\rho}$
$\displaystyle=\frac{\alpha_{a}}{\alpha_{f}}\frac{\sigma_{a}}{\sigma_{f}},$
$\displaystyle\alpha_{r}$
$\displaystyle=\sqrt{\frac{(\alpha_{a}\sigma_{a})^{2}+(\alpha_{f}\sigma_{f})^{2}}{2\,(\alpha_{e}\sigma(x))^{2}}},$
$\displaystyle=\sqrt{\frac{\alpha_{\rho}^{2}+1}{2}}\frac{\sigma_{f}}{\sigma(x)}\frac{\alpha_{f}}{\alpha_{e}}.$

The original $\alpha_{a},\alpha_{f}$ multipliers can then be written in terms
of $\alpha_{r},\alpha_{\rho}$:

$\displaystyle\alpha_{a}$
$\displaystyle=\alpha_{\rho}\alpha_{f}\frac{\sigma_{f}}{\sigma_{a}}$
$\displaystyle\alpha_{f}$
$\displaystyle=\alpha_{r}\alpha_{e}\frac{\sigma(x)}{\sigma_{f}}\sqrt{\frac{2}{\alpha_{\rho}^{2}+1}}$

We have replaced two of the three original multipliers, but still have a
dependence on $\alpha_{e}$ here in our expressions for $\alpha_{f}$ and
$R_{L}^{(e)}$, which we now remove by dividing it out of our residual branches
and embedding. We use the hat ($\hat{\cdot}$) symbol to denote terms that have
been divided-through by $\alpha_{e}$. This new system of equations is
equivalent to our old one thanks to the zero-homogeneity of the final post-
residual layer:

$\displaystyle R_{L+1}(x)$
$\displaystyle=f_{L+1}(R_{L}^{(e)}+R_{L}^{(a)}+R_{L}^{(f)})$
$\displaystyle=f_{L+1}((R_{L}^{(e)}+R_{L}^{(a)}+R_{L}^{(f)})/\alpha_{e})$
$\displaystyle=f_{L+1}(\hat{R}_{L}^{(e)}+\hat{R}_{L}^{(a)}+\hat{R}_{L}^{(f)})$

This gives $\hat{R}_{L}^{(e)}=\alpha_{e}x/\alpha_{e}=x$, removing our first
occurrence of $\alpha_{e}$. Following the division through $\hat{R}_{L}^{(a)}$
and $\hat{R}_{L}^{(f)}$, we obtain:

$\displaystyle\hat{R}_{L}^{(a)}$
$\displaystyle:=\sum_{l=1}^{L/2}\frac{\hat{\alpha}_{a}}{\sqrt{L/2}}f_{2l-1}(R_{2l-1}),$
$\displaystyle\hat{R}_{L}^{(f)}$
$\displaystyle:=\sum_{l=1}^{L/2}\frac{\hat{\alpha}_{f}}{\sqrt{L/2}}f_{2l}(R_{2l}),$
$\displaystyle\hat{\alpha}_{a}$
$\displaystyle=\alpha_{\rho}\hat{\alpha}_{f}\frac{\sigma_{f}}{\sigma_{a}},$
$\displaystyle\hat{\alpha}_{f}$
$\displaystyle=\alpha_{r}\frac{\sigma(x)}{\sigma_{f}}\sqrt{\frac{2}{\alpha_{\rho}^{2}+1}}.$

This system of equations is the same as the original, but with the two
$\alpha_{e}$ terms dropped, meaning our model’s multipliers can be expressed
in terms of only $\alpha_{r}$ and $\alpha_{\rho}$. Using the above equations,
any pair of values for $(\alpha_{r},\alpha_{\rho})$ can be translated back
into an equivalent set of values for $(\alpha_{e},\alpha_{a},\alpha_{f})$ such
that the output $R_{L+1}(x)$ is the same, meaning that our multipliers are no
less expressive than the original set. This satisfies our desired criteria of
minimizing the number of multipliers while maintaining expressivity.

We can simplify further in the case of unit-scaled models, which are designed
such that $\sigma(x),\sigma_{a},\sigma_{f}$ are all $1$ at initialization. In
this case our re-parametrization becomes:

$\displaystyle\hat{\alpha}_{a}$ $\displaystyle=\alpha_{\rho}\hat{\alpha}_{f},$
(19) $\displaystyle\hat{\alpha}_{f}$
$\displaystyle=\alpha_{r}\sqrt{\frac{2}{\alpha_{\rho}^{2}+1}},$ (20)
$\displaystyle\hat{\alpha}_{e}$ $\displaystyle=1.$ (21)

This is the basis of our claim that Unit Scaling is what enables a more
intuitive set of multipliers. Not only do the multipliers $\alpha_{r}$ and
$\alpha_{\rho}$ represent important dynamics in the network at initialization
(the ratio of residual-to-embedding scales, and the ratio of attention-to-
feed-forward scales), but it’s only via unit scaling that these equations
become simple enough to implement in practice. Using equations Equations 19,
20 and 21 for a non-unit scaled network may still be effective, but the
interpretation we’ve given to $\alpha_{r}$ and $\alpha_{\rho}$ no longer hold.

Our final desired property is an empirical one: that the most effective choice
of one multiplier depends as little as possible on the choice of the other
multiplier(s). We demonstrate that our multipliers satisfy this property
better than the standard set of residual multipliers in Section 5.2.

####  G.2.2 The full u-µP residual scheme

Here we give the full definition of our u-µP residual scheme, summarizing the
results of previous sections. A general pre-norm transformer is implemented
as:

$\displaystyle R_{0}(x)$ $\displaystyle=c\,x,$ (22) $\displaystyle R_{l}(x)$
$\displaystyle=a_{l}f_{l}(R_{l-1}(x))+b_{l}R_{l-1}(x),\quad l=1,..,L$ (23)
$\displaystyle R_{L+1}(x)$ $\displaystyle=f_{L+1}(R_{L}(x)),$ (24)

where $a_{l},b_{l}$ and $c$ are scalar multipliers, and the $f_{l}$ alternate
between self-attention and feed-forward layers. We consider our baseline set
of µP residual HPs here to be
$(\alpha_{\mathrm{emb}},\alpha_{\mathrm{attn{\text{-}}residual}},\alpha_{\mathrm{ffn{\text{-}}residual}})$,
which we implement (assuming depth-µP branch scaling) as:

$\displaystyle a_{l}$
$\displaystyle=\begin{dcases}\frac{\alpha_{\mathrm{attn{\text{-}}residual}}}{\sqrt{L/2}}&l\textrm{
is odd (self-attention)}\\\
\frac{\alpha_{\mathrm{ffn{\text{-}}residual}}}{\sqrt{L/2}}&l\textrm{ is even
(feed-forward)}\end{dcases}$ $\displaystyle b_{l}$ $\displaystyle=1$
$\displaystyle c$ $\displaystyle=\alpha_{\mathrm{emb}}.$

The corresponding u-µP set of residual HPs is
$(\alpha_{\mathrm{res}},\alpha_{\mathrm{res{\text{-}}attn{\text{-}}ratio}})$,
which we implement as:

$\displaystyle a^{2}_{l}$ $\displaystyle=\frac{\tau^{2}_{l}}{\tau^{2}_{l}+1}$
(25) $\displaystyle b^{2}_{l}$ $\displaystyle=\frac{1}{\tau^{2}_{l}+1}$ (26)
$\displaystyle c$ $\displaystyle=1,$ (27) $\displaystyle\tau^{2}_{l}$
$\displaystyle=\begin{dcases}\frac{\hat{\alpha}^{2}_{a}}{\frac{L}{2}+\ell\hat{\alpha}^{2}_{a}+\ell\hat{\alpha}^{2}_{f}}&l\textrm{
is odd}\\\
\frac{\hat{\alpha}^{2}_{f}}{\frac{L}{2}+(\ell+1)\hat{\alpha}^{2}_{a}+\ell\hat{\alpha}^{2}_{f}}&l\textrm{
is even}\end{dcases},\quad\ell=\left\lfloor{\frac{l-1}{2}}\right\rfloor$ (29)
$\displaystyle\hat{\alpha}^{2}_{a}$
$\displaystyle=\alpha^{2}_{\mathrm{res{\text{-}}attn{\text{-}}ratio}}\,\hat{\alpha}^{2}_{f}\
$ (30) $\displaystyle\hat{\alpha}^{2}_{f}$
$\displaystyle=\frac{2}{\alpha_{\mathrm{res{\text{-}}attn{\text{-}}ratio}}^{2}+1}\,\alpha^{2}_{\mathrm{res}}\,.$
(31)

This is the u-µP residual scheme. It satisfies the three properties that we
initially set out to achieve: the variance at initialization of our $R_{l}(x)$
is always 1, our HPs have a clear and useful interpretation, and our scheme is
as expressive as the baseline (which is neither unit-scaled or has
interpretable HPs).

##  Appendix H The cut-edge rule

In the section we review the notion of constraints used for scaling operations
in a computational graph. For a more thorough, generalized treatment, please
refer to Section 5.1 and Appendix E.4 of the original Unit Scaling paper [11].

For simplicity, we will only discuss the cut-edge rule in the context of a
typical neural network. For each operation $f$, parametrized by $\theta$
taking input $x$ and emitting output $y$, a user must choose how to scale $y$,
$\nabla_{x}$ and $\nabla_{\theta}$ (gradient of loss w.r.t. $x$ and $\theta$
respectively). In the simplest case, where there are no further data
dependencies, we can simply choose factors that preserve unit scale. In more
complex scenarios, we must balance the need for each tensor to be unit-scaled
and for gradients to be correct up to a constant factor.

In particular, a problem emerges in the presence of residual blocks in which
$y=x+f(x;\theta)$. In these circumstances, $\nabla_{x}$ is computed as the sum
of residual gradient $\nabla_{f}$ $\frac{\partial{f}}{\partial{x}}$ and skip
gradient $\nabla_{y}$. If we choose not to insert scaling factors into our
graph, $\nabla_{f}$ $\frac{\partial{f}}{\partial{x}}$ and $\nabla_{y}$ will
have some ratio of scale $r$. However, if we have chosen to rescale the
gradient of operations in $f$, then $\nabla_{f}$
$\frac{\partial{f}}{\partial{x}}$ will have been rescaled by some $s$. This
means the new ratio of $\nabla_{f}$ $\frac{\partial{f}}{\partial{x}}$ and
$\nabla_{y}$ will be $r\cdot s$. Therefore, when adding these together,
$\nabla_{x}$ is no longer a correct gradient up to a constant factor.

How do you remedy this? If we can ensure that the scaling factors are the same
for both the input gradients and outputs of an op, we will have $s=1$. This
ensures that gradients for inputs to residual blocks are correct up to a
constant factor.

How do you decide when you are free to preserve unit scale, and when to
constrain scaling factors to be the same? We previously define the cut-edge
rule [11] for computational graphs where nodes represent forward pass
operations and edges represent operation outputs. If an input edge is a cut-
edge, i.e., the number of connected components in the graph would increase
upon deletion (examples in a typical transformer model: output of embedding
gather, output of a residual add, output of final norm, output token logits,
weights), there is no need to constrain the scales of the operation’s output
edge and the input edge gradient. For all other input edges (e.g., inputs to a
residual add, intermediates computed along a residual branch), the scales of
gradients and outputs should be constrained.

##  Appendix I From µP to u-µP

Here we outline additional details to help readers follow the process of
deriving u-µP from the combination of Unit Scaling and µP. Our first step of
dropping $\sigma_{W}$ and $\mathrm{base{\text{-}}fan{\text{-}}in}$, and moving
$\alpha_{W}$s to functions, results in Table 9. This intermediate scheme does
not yet satisfy Unit Scaling, but simplifies the HP rules in preparation for
further changes. Note that we have also removed $\hat{\eta}_{\mathrm{emb}}$ as
we don’t include this HP in our u-µP extended HP set. We have included
residual scaling rules here, in accordance with depth-µP, which we intend u-µP
to satisfy, though our standard µP implementation doesn’t use it.

Table 9: An intermediate scheme resulting from dropping those HPs from µP which are not needed under u-µP. ABC-multiplier |  | Weight Type |  | Residual
---|---|---|---|---
| Input | Hidden | Output
parameter | ($A_{W}$) | $1$ | $1$ | $\frac{1}{\mathrm{fan{\text{-}}in}}$ |  $\frac{1}{\sqrt{0pt}}$*
initialization | ($B_{W}$) | $1$ | $\frac{1}{\sqrt{\mathrm{fan{\text{-}}in}}}$ | $1$ | —
Adam LR | ($C_{W}$) | $\eta$ | $\eta\,\frac{1}{\mathrm{fan{\text{-}}in}}$ | $\eta$ |  $\frac{1}{\sqrt{0pt}}$*

##  Appendix J Low-precision and its trade-offs

##### Number formats for deep learning

The standard numerical representations used in deep learning are the set of
formats defined by the IEEE 754 floating-point standard [64]. IEEE floats
comprise three elements: a sign bit, exponent bits, and mantissa bits. The
number of exponent bits determines the range of a format, while the mantissa
determines the precision555Confusingly, the term low-precision tends to
indicate using <32 bit-width formats, so in this context precision also
reflects the number of exponent bits as well as the usual mantissa bits.. We
refer readers to the original Unit Scaling paper ([11], Section 3.1) for a
comprehensive overview of floating-point representations.

The default format used for training is the single-precision floating-point
format, commonly known as FP32, with some hardware providers automatically
casting it to the smaller TF32 compute mode for accelerated arithmetic. The
16-bit FP16 and BF16 formats were later introduced, and more recently the FP8
E5 & E4 formats [65, 66, 67]. The higher range of E5 has typically been used
for gradients, while the higher precision of E4 has been seen as necessary for
weights and activations. Our particular implementation of FP8 training is
covered in Section A.7. Other aspects of training such as the optimizer state
and cross-device communication have also been put into FP8 [27], though not
all tensors are amenable to being run in the lowest precision [38] without
degradation. The use of multiple formats is known as mixed precision [68]. A
comparison of these formats is given in Table 10.

Table 10: A comparison of deep learning formats. E indicates exponent bits, and M mantissa bits. The smaller formats typically give more FLOPS, at the expense of reduced range and/or precision. Format | E | M | | max | | | min normal | | | min subnormal | | FLOPS (vs TF32)
---|---|---|---|---|---|---
FP32 | 8 | 23 | $3.4\times 10^{38}$ | $1.2\times 10^{-38}$ | $1.4\times 10^{-45}$ | $<1\times\hphantom{<}$
TF32 | 8 | 10 | $3.4\times 10^{38}$ | $1.2\times 10^{-38}$ | $1.1\times 10^{-41}$ | $1\times$
BF16 | 8 | 7 | $3.4\times 10^{38}$ | $1.2\times 10^{-38}$ | $9.2\times 10^{-41}$ | $2\times$
FP16 | 5 | 10 | $65504$ | $6.1\times 10^{-5\hphantom{0}}$ | $6.0\times 10^{-8\hphantom{0}}$ | $2\times$
FP8 E5 | 5 | 2 | $57344$ | $6.1\times 10^{-5\hphantom{0}}$ | $1.5\times 10^{-5\hphantom{0}}$ | $4\times$
FP8 E4 | 4 | 3 | $448$ | $1.6\times 10^{-2\hphantom{0}}$ | $2.0\times 10^{-3\hphantom{0}}$ | $4\times$

##### The benefits of low-precision

Using numerical representations with fewer bits facilitates the design of more
efficient arithmetic in hardware, typically leading to a linear increase in
peak FLOPS (as shown in Table 10). As large-scale training efforts are
typically compute-bound due to the size of matmuls [69], putting the inputs to
these operations in low-precision formats has a substantial impact on training
efficiency. Low-precision formats also reduce the other two common performance
constraints: for memory-bandwidth-bound models they require fewer bits to be
transmitted, and for memory-size-bound models they require fewer bits to be
stored.

##### The challenges of low-precision

Unfortunately, moving to low-precision formats also increases quantization
error. For values within the representable range this takes the form of
rounding error, and for values outside it, clipping error (both overflow and
underflow). Rounding error tends to be an intrinsic problem: the number of
mantissa bits dictates the expected accuracy of representations and this
cannot easily be changed. In contrast, clipping error is often eliminated by
scaling a tensor so that its values lie within the range of a format. Note
that a multiplicative change in values of this kind doesn’t affect the
(relative) rounding error, due to the exponential spacing of values. Most
research into making low-precision work has focused on the problem of scaling
tensors in this way.

Simply casting all tensors to FP16 or FP8 tends to impair training, largely
due to clipping error. For FP16, this primarily affects gradients. [68]
address this by introducing a fixed global loss-scale HP, which multiplies the
loss value in the backward pass, artificially up-scaling gradients to lie
within FP16 range [68]. Automatic loss scaling [70] builds upon this idea,
making the loss-scale a dynamic value that is tuned during training.

The later BF16 format has the same range as FP32, making loss scaling
unnecessary. For FP8 no such range-equivalent format can exist, so the problem
of clipping error must be addressed. Most FP8 implementations have done so by
moving from a global loss-scale to a local scale for each FP8 tensor. In
pseudo-code, this takes the form:

⬇

a = scale(A)

b = scale(B)

A = to_fp8(A / a)

B = to_fp8(B / b)

C = (a * b) * matmul(A, B)

where we assume that matmul takes inputs in FP8 and directly produces the
output in higher precision.

The result of the scale() operation can either be a fixed scale determined
before training [66], or in the case of Transformer Engine [71], computed
dynamically as a function of the ‘absmax’ of the input tensor (though they
introduce a delay across time-steps, to facilitate an efficient fused kernel).
Increasing granularity and computing scales dynamically using this kind of
method inevitably adds complexity (from both a logical and implementation
perspective), as well the potential for computational overhead. Unit Scaling
generally avoids the need for matmul input scaling.

##  Appendix K Benchmarking scaled matrix multiplication implementation in
PyTorch

Figure 18: Square matrix multiplication throughput in TFLOPs with and without
scaling factors applied to the output across 32-, 16-, and 8-bit float dtypes
on NVIDIA H100 PCIe. Naive implementation in PyTorch.

Standard strategies for FP8 training require expensive statistics gathering
(e.g., amax) per tensor. A key benefit of u-µP for FP8 training is that it
instead provides us with static scaling factors to rescale operation outputs.
Even a naive implementation in pytorch can achieve a minimal drop in hardware
utilization.

Figure 18 demonstrates hardware utilization for FP8, FP16, and FP32 matrix
multiplications on a single NVIDIA H100 PCIe card. For FP16 and FP32,
torch.matmul is used, whereas torch._scaled_mm is used for FP8. Comparing
"scaled" to "unscaled" matrix multiplication demonstrates a 30$\%$, 20$\%$,
and 10$\%$ drop in hardware utilization for each data type respectively. In
the case of FP8, where the drop in utilization is most pronounced, utilization
can be recovered by passing the scaling factor as a scale associated with one
of the two input tensors.

It should be noted that as of PyTorch version 2.3, torch._scaled_mm always
computes amax as well as the matrix multiplication. The performance of FP8
matrix multiplications could be higher without this overhead.