# i-Sim2Real: Reinforcement Learning of Robotic Policies in Tight Human-Robot
Interaction Loops

Saminda Abeyruwan∗, Laura Graesser∗, David B. D’Ambrosio, Avi Singh,
Anish Shankar, Alex Bewley, Deepali Jain, Krzysztof Choromanski, Pannag R.
Sanketi
Robotics at Google
{saminda,lauragraesser,ddambro,singhavi,phinfinity,bewley,
<EMAIL_ADDRESS>
###### Abstract

Sim-to-real transfer is a powerful paradigm for robotic reinforcement
learning. The ability to train policies in simulation enables safe exploration
and large-scale data collection quickly at low cost. However, prior works in
sim-to-real transfer of robotic policies typically do not involve any human-
robot interaction because accurately simulating human behavior is an open
problem. In this work, our goal is to leverage the power of simulation to
train robotic policies that are proficient at interacting with humans upon
deployment. But there is a chicken and egg problem — how to gather examples of
a human interacting with a physical robot so as to model human behavior in
simulation without already having a robot that is able to interact with a
human? Our proposed method, Iterative-Sim-to-Real (i-S2R), attempts to address
this. i-S2R bootstraps from a simple model of human behavior and alternates
between training in simulation and deploying in the real world. In each
iteration, both the human behavior model and the policy are refined. For all
training we apply a new evolutionary search algorithm called Blackbox Gradient
Sensing (BGS). We evaluate our method on a real world robotic table tennis
setting, where the objective for the robot is to play _cooperatively_ with a
human player for as long as possible. Table tennis is a high-speed, dynamic
task that requires the two players to react quickly to each other’s moves,
making for a challenging test bed for research on human-robot interaction. We
present results on an industrial robotic arm that is able to cooperatively
play table tennis with human players, achieving rallies of 22 successive hits
on average and 150 at best. Further, for 80% of players, rally lengths are 70%
to 175% longer compared to the sim-to-real plus fine-tuning (S2R+FT) baseline.
For videos of our system in action please see
https://sites.google.com/view/is2r.

**footnotetext: Indicates equal contribution.

> Keywords: sim-to-real, human-robot interaction, reinforcement learning

##  1 Introduction

Sim-to-real transfer has emerged as a dominant paradigm for learning-based
robotics. Real world training is often slow, cost-prohibitive, and poses
safety-related challenges, so training in simulation is an attractive
alternative and has been explored for a number of real world tasks, including
object manipulation [1, 2, 3, 4], legged robot locomotion [5, 6], and aerial
navigation [7, 8]. However, one element that is missing in this prior work is
that the policies are not trained to be proficient at interacting with humans
upon deployment. The utility of sim-to-real learning can be greatly increased
if we extend it to settings where the trained policies need to interact with
humans in a close, tight-loop fashion upon deployment. One of the major
promises of learning-based robotics is to deploy robots in human-occupied
settings, since non-learning robots already work well in deterministic, non-
human occupied settings, such as factory floors. However, simulating human
behavior is non-trivial (and indeed, one of the primary goals of artificial
intelligence research), making it a major bottleneck in sim-to-real research
for tasks involving human-robot interaction.

One approach to simulating human behavior is imitation learning. Given a few
examples of human behavior, we can use techniques such as behavior cloning [9,
10], or inverse reinforcement learning [11, 12] to distill that behavior into
a policy, and then use these policies to generate human behavior in
simulation. However, this approach presents a chicken and egg problem: in
order to obtain useful examples of human behavior (in the context of human-
robot interaction), we need a robot policy that already knows how to interact
with humans in the real world, but we cannot learn such a policy without the
ability to simulate human behaviors in the first place. The primary
contribution of this paper is a practical solution to this problem.

Our proposed method involves learning a coarse model of human behavior from
initial data collected in the real world to bootstrap reinforcement learning
of robotic policies in simulation. Deploying this learned policy in the real
world now allows us to collect data in which the human subjects meaningfully
interact with the robot. We then use this real world experience to improve our
human behavior model, and continue training the robot policy in simulation
under this updated model. We repeat this iterative process until a desired
level of performance is achieved.

Figure 1: Robot setup An ABB IRB 120T 6-DOF robotic arm is mounted to a two-
dimensional Festo linear actuator, creating an 8-DOF system.

We present results on a task involving a robot playing table tennis with non-
professional human players (see Figure 1). The goal for the robot is to
maximize rally length, i.e. the number of successive hits by the robot and
human before the ball goes out of play and policies are evaluated using rally
length. Table tennis is a high-speed, dynamic task that requires close, tight-
loop interactions between two players (in this case, a human and a robot).
Further, maximizing rally length requires the robot to cooperate with a human,
and vice versa. Thus we believe it to be a good instantiation of our problem
setting. We build an initial model of the human player’s ball trajectories
without a robot present and iteratively refine the robot and player models as
they play together, ultimately resulting in a robot policy that can hold
rallies of 22 successive hits on average and 150 at best.

While we demonstrate our approach on table tennis, we believe that our overall
pipeline can be applied to a broad range of tasks, and take into account the
various nuances of those tasks. The two characteristics a human behavior model
needs to be compatible with our approach are (a) it can be updated using human
data that is gathered whilst a human or humans are interacting with a robot,
and (b) the model can be used to sample human behavior in simulation.

In summary, the primary contributions of this paper are: (a) a framework for
training robotic policies in simulation that would need to interact with human
subjects upon deployment, (b) a real world instantiation of this framework on
a high-speed, dynamic task requiring tight, closed-loop interactions between
humans and robots, (c) a detailed assessment of how our method, which we call
Iterative-Sim-to-Real (i-S2R), compares with a baseline sim-to-real approach
in the domain of cooperative robotic table tennis, and (d) the first robotic
table tennis policy trained to control robot joints using reinforcement
learning (RL) that can handle a wide variety of balls and can rally
consistently with non-professional humans. i-S2R can apply any RL method,
however the only policy-optimization algorithm that so far led to the on-
robot-deployable policies is the so-called Blackbox Gradient Sensing (BGS)
that we introduce here. For videos of our system in action please see
https://sites.google.com/view/is2r.

##  2 Related Work

##### Sim-to-Real Learning for Robotics

RL is a powerful paradigm for learning increasingly capable and robust robot
controllers [13, 14, 15]. However, learning controllers from scratch on a
physical robot is often prohibitively time consuming due to the large number
of samples required to learn competent policies and potentially unsafe due to
the random exploration inherent in RL methods [16, 17]. Training policies in
simulation and transferring them to a physical robot, known as sim-to-real
transfer (S2R), is therefore appealing.

Whilst it is both fast and safe to train agents from scratch in simulation,
S2R presents its own challenge — persistent differences between simulated and
real world environments that are extremely difficult to overcome [17, 18]. No
single technique has been found to bridge the gap by itself. Instead a
combination of multiple techniques are typically required for successful
transfer. These include system identification [13, 19, 20, 21, 22] which may
involve iterating with a physical robot in the loop [2, 23], building hybrid
simulators with learned models [5, 13, 22], dynamics randomization [1, 2, 5,
6, 13, 14, 15], simulated latency [15, 22], and more complex network
architectures [13]. We use (1) system ID with a physical robot in the loop,
(2) dynamics randomization, (3) simulated latency, and (4) more complex
networks. Similarly to Lee et al. [13], we use a 1D CNN to represent control
policies. Yet a sim-to-real gap persists. Continuing to train in the real
world [24, 25, 26, 27] (known as fine-tuning) is an effective way to bridge
the remaining gap since the policy can adapt to changes in the environment. We
also utilize fine-tuning in this work, but unlike most past work, our learned
policy is expected to interact cooperatively with a real human during this
fine-tuning phase.

The closest sim-to-real approaches in prior work are Chebotar et al. [2] and
Farchy et al. [23] since they update simulation parameters based on multiple
iterations of real world data collection interleaved with simulated training.
However, both of these prior works focus on using real world interaction data
to learn improved _physical_ parameters for the simulator, whereas our method
focuses on learning better human behavior models. Unlike these prior works,
our learned policies are proficient at interacting with humans upon deployment
in the real world.

##### Reinforcement Learning for Table Tennis

Robotic table tennis is a challenging, dynamic task [28] that has been a test
bed for robotics research since the 1980s [29, 30, 31, 32, 33]. The current
exemplar is the Omron robot [34]. Until recently, most methods tackled the
problem by identifying a virtual hitting point for the racket [35, 36, 37, 38,
39, 40, 41, 42]. These methods depend on being able to predict the ball state
at time $t$ either from a ball dynamics model which may be parameterized [35,
36, 43, 44] or by learning to predict it [33, 38, 39]. This results in a
target paddle state or states and various methods are used to generate robot
joint trajectories given these targets [33, 35, 36, 43, 44, 45, 46, 47, 48,
49, 50]. More recently, Tebbe et al. [51] learned to predict the paddle target
using RL.

An alternative line of research seeks to do away with hitting points and ball
prediction models, instead focusing on high frequency control of a robot’s
joints using either RL [28, 39, 52] or learning from demonstrations [46, 53,
54]. Of these, Büchler et al. [28] is the most similar, training RL policies
to control robot joints from scratch at high frequencies given ball and robot
states as policy inputs. However Büchler et al. [28] restricts the task to
playing with a ball thrower on a single setting, whereas we focus on the
harder problem of cooperative play with different humans.

Most prior work simplifies the problem by focusing on play with a ball
thrower. Only a few [46, 49, 51, 55] focus on cooperative rallying with a
human. Of these, Tebbe et al. [51], is the most similar, evaluating policies
on various styles of human-robot cooperative play. However, Tebbe et al. [51]
simplify the environment to a single-step bandit and the policy learns to
predict the paddle state given the ball state at a pre-determined hit time
$t$. In contrast, we learn closed-loop policies that operate at a high
frequency (75Hz), removing the need for a learned policy to accurately predict
where the ball will be in the future, increasing the robustness of the system,
and enabling more dynamic play.

##### Human Robot Interaction

Although not a typical HRI benchmark, cooperative robotic table tennis
exhibits many of the features studied in the field: a human and robot working
together, complex interactions between the two, inferring actions based on
non-explicit cues, and so on. A major challenge in HRI is effectively modeling
the complexities of human behavior in simulation [56] in order to learn
without requiring an actual human. We employ several common techniques from
HRI to learn in simulation such as simplifying the human model [57],
specialized models for specific players [58], and refining our model based on
real world interactions. Finally we note that like us, Paleja et al. [59]
found policy performance varied depending on the skill of the human player.

##  3 Preliminaries

##### Problem Setting

We consider the problem of cooperative human-robot table tennis as a single-
agent sequential decision making problem in which the human is a part of the
environment. We formalize the problem as a _Markov Decision Process_ (MDP)
[60] consisting of a of a 4-tuple ($\mathcal{S}$, $\mathcal{A}$,
$\mathcal{R}$, $p$), whose elements are the state space $\mathcal{S}$, action
space $\mathcal{A}$, reward function
$\mathcal{R}:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$, and
transition dynamics $p:\mathcal{S}\times\mathcal{A}\rightarrow\mathcal{S}$. An
episode $(s_{0},a_{0},r_{0},...,s_{n},a_{n},r_{n})$ is a finite sequence of
$s\in\mathcal{S}$, $a\in\mathcal{A}$, $r\in\mathcal{R}$ elements, beginning
with a start state $s_{0}$ and ending when the environment terminates. We
define a parameterized policy $\pi_{\theta}:\mathcal{S}\to\mathcal{A}$ with
parameters $\theta$. The objective is to maximize
$\mathbb{E}\left[\sum_{t=1}^{N}r(s_{t},\pi_{\theta}(s_{t}))\right]$, the
expected cumulative reward obtained in an episode under $\pi_{\theta}$.

We make two simplifications to our problem. First, we focus on rallies
starting with a hit instead of a table tennis serve to make the data more
uniform. Second, an episode consists of a single ball throw and return.
Policies are therefore rewarded based on their ability to return balls to the
opposite side of the table. This reward structure encourages longer rallies,
as an agent that can return any ball can also rally indefinitely provided the
simulated single shots overlap with the real rally shots.

##### BGS & Evolutionary Search (ES)

i-S2R is compatible with any RL algorithm. In initial experiments we tried a
range of methods — PPO [61], QT-OPT [62], SAC [63], and Blackbox Gradient
Sensing (BGS) that we introduce here. Only BGS transferred well to a physical
robot, hence we continued with this approach and we leave to future work more
exhaustive research on other RL algorithms. BGS is an ES-method [64, 65, 66,
67, 68, 69] which have been shown to be an effective strategy for solving MDPs
[66, 68]. ES methods aim to optimize the smoothened version
$F_{\sigma}(\theta)$ of the original RL-objective $F(\theta)$, where $\theta$
stands for the policy parameters, given (for the parameter $\sigma>0$) as:

$F_{\sigma}(\theta)=\mathbb{E}_{\mathbf{\delta}\sim\mathcal{N}(0,\mathbf{I}_{d})}[F(\theta+\sigma\mathbf{\delta})].$
(1)

Different ES algorithms apply different Monte-Carlo strategies to approximate
the gradient of $F_{\sigma}(\theta)$. In BGS, following [64] we choose Monte
Carlo samples $\delta_{i}$ to form orthogonal-ensembles (to reduce the
variance of the estimation) and apply a novel technique for choosing a final
collection of samples $\delta_{i}$ for gradient estimation (the so-called
elite-choice process). The former technique improved convergence in training
and the latter was crucial for the overall effectiveness of training —
training in simulation failed without it. See Appendix B for details.

##  4 Method

i-S2R consists of two core components: (1) an iterative procedure for
progressively updating and learning from a human behavior model — the human
ball distribution in this setting — and (2) a method for modeling human
behavior in simulation given a dataset of human play gathered in the real
world (see Figure 2 for an overview). We first describe our iterative training
procedure, and then discuss how we model human ball distributions.

##### Iterative Training Procedure

Figure 2: Iterative-Sim-to-Real. left We start with a coarse bootstrap model
of human behavior (shown in yellow), and use it to train an initial robot
policy in simulation. We then fine-tune this policy in the real world against
a human player, and the human interaction data collected during this period is
used to update the human behavior model used in simulation. We then take the
fine-tuned policy back to simulation to further train it against the improved
human behavior model, and this process is repeated until robot and human
behaviors converge. right Specific i-S2R details used in this work. $x$-axis
represents the training iterations in sim, $y$-axis represents the fine-tuning
iterations in real with human-in-the-loop. Model names are in italics.

An overview of the method can be seen in Figure 2. First we gather an initial
dataset, $D_{0}$, from player $P$ hitting table tennis balls across the table
without a robot doing anything. From $D_{0}$, we build our first human
behavior model $M_{0}$ that defines a ball distribution (see below). A robot
policy is trained in simulation to return balls sampled from $M_{0}$. Once the
policy has converged, we transfer the parameters, $\theta_{0S}$, to a real
robotic system. The model is fine-tuned whilst player $P$ plays cooperatively
(i.e. trying to maximize rally length) with the robot for a fixed number of
parameter updates to produce $\theta_{0R}$. All of the human hits during this
fine-tuning phase are added to $D_{0}$ to form $D_{1}$, which is used to
define $M_{1}$. The policy weights, $\theta_{0R}$, are then transferred back
to simulation and training is continued with the new distribution $M_{1}$.
After training in simulation, the policy weights $\theta_{1S}$ are transferred
back to the real world. The fine-tuning process is repeated to produce the
next set of policy parameters $\theta_{1R}$, dataset $D_{2}$, and human model
$M_{2}$. This process can be repeated as many times as needed.

A useful check for assessing convergence was found by looking at the delta in
our human behavior model from one iteration to the next. We found the delta
between $M_{1}$ and $M_{2}$ was substantially smaller than between $M_{0}$ and
$M_{1}$ indicating that three iterations were enough for this task. For
details on the ball distribution parameters for different players see
subsection C.3.

##### Modeling Human Ball Distributions

One of our primary goals is to simulate human player behaviors from a set of
real world ball trajectories that have been subjected to air drag, gravity,
and spin. Due to perception challenges in the real world, we do not explicitly
model spin. The input to this procedure is a dataset of ball trajectories,
where each trajectory consists of a sequence of ball positions. The output is
a uniform ball distribution defined by 16 numbers: the minimum and maximum
initial ball position (6), velocity (6), and $x$ and $y$ ball landing
locations on robot side (4).

The ball distribution is derived from the dataset in two stages. The first
step is to estimate a ball’s initial position and velocity for each
trajectory. We do this by selecting the free flight part of the trajectory
(before the first bounce) and minimize the Euclidean distance between the
simulated and real trajectory using the Nelder-Mead method [70]. Please see
subsection C.4 for details on the model used to simulate a ball trajectory.

Next we remove outliers using DBSCAN [71] and take the minimum and maximum per
dimension to define the ball distribution. We sample an initial position and
velocity from this distribution and generate a ball trajectory in simulation
subject to the drag force. Other parameters needed for the simulation, such as
the coefficient of restitution, friction between the table and ball and the
robot paddle and the ball, and so on have been empirically estimated following
[72, 73].

##  5 System, Simulation, and MDP Details

Our real world robotic system (see Figure 1) is a combination of an ABB IRB
120T 6-DOF robotic arm mounted to a two-dimensional Festo linear actuator,
creating an 8-DOF system, with a table tennis paddle mounted on the end-
effector. The 3D ball position is estimated via a stereo pair of Ximea
MQ013CG-ON cameras from which we process 2D detections, triangulate to 3D, and
filter through a 3D tracker. See Appendix D for more details. We concatenate
the ball position with the 8-DOF robot joint angles to form an 11-dimensional
observation space. Along with the current observation, we pass the past seven
observations (a state space of $8\times 11$) as the input to the policy. The
policy controls the robot by outputting eight individual joint velocities at
75Hz. Following Gao et al. [52] we use a 3-layer 1-D dilated gated
convolutional neural network as our policy architecture. Details of the policy
architecture can be found in Appendix E.

Our simulation is built on the PyBullet [74] physics engine replicating our
real environment. We use PyBullet to model robot and contact dynamics whilst
balls are modeled as described in section 4. We add random uniform noise of
$2\times$ the diameter of a table tennis ball to the ball observation per
timestep to aid transfer to a physical system. We also found it necessary to
simulate sensor latency, otherwise sim-to-real transfer completely failed.
Robot actions as well as ball and robot observation latencies are modeled as
parameterized Gaussians based on measurements from the real system. Policies
are rewarded for hitting balls and for returning balls in a cooperative
manner. See Appendix G for details.

##  6 Experimental Results

##### Experimental Setup

To evaluate our method, we completed the procedure described in section 4 for
five different non-professional table tennis players, thus training five
independent i-S2R policies. We compare i-S2R with two baselines. First, the
standard sim-to-real (S2R) baseline in which a policy is transferred zero-shot
from simulation [1, 3, 5, 6, 7, 8]. Second, a stronger baseline of S2R plus
fine-tuning (S2R+FT) in which a policy is transferred in simulation and
training is continued in the real world. For fair comparison, S2R+FT is given
the same real world training budget as i-S2R. We follow the approach in [24]
using the same training algorithm throughout and implement an automatic reset
for autonomous training. Finally, each player trained a S2R-Oracle+FT policy
which was trained in simulation on the penultimate human behavior model
obtained through i-S2R and fine-tuned in the real world for 35% of the i-S2R
training budget. This is equivalent to the last round of fine-tuning for
i-S2R. (See Figure 2 right). S2R-Oracle+FT is intended to isolate the effect
of the human behavior modeling on final performance, enabling us to better
understand what aspects of the i-S2R process matter. Each policy was evaluated
by the model’s trainer. Select policies were cross-evaluated by two other
players. All policies were tested in random order and the identity of the
model was kept hidden from the evaluator (“blind eval”). Further details can
be found in Appendix H.

Due to the time needed to train and evaluate i-S2R, S2R+FT, and S2R-Oracle+FT
(roughly 20 hours per person) we note that 4 of the 5 players are authors on
this paper. The non-author player’s results appear consistent with our overall
findings (see Appendix K for details).

Figure 3: Aggregated results Boxplot details: white circle: mean, horizontal
line: median, box bounds: 25th and 75th percentiles. left When aggregated
across all players, i-S2R rally length is higher than S2R+FT by about 9%.
However, note that simple aggregation puts extra weight on higher skilled
players that are able to hold a longer rally. center The normalized rally
length distribution (see Appendix J for normalization details) shows a bigger
improvement between i-S2R and S2R+FT in terms of the mean, median and 25th and
75th percentiles. right The histogram of rally lengths for i-S2R and S2R+FT
(250 rallies per model) shows that a large fraction of the rallies for S2R+FT
are shorter (i.e. less than 5), while i-S2R achieves longer rallies more
frequently.

Figure 4: Results by player skill. When broken down by player skill, we notice
that i-S2R has a substantially longer rally length than S2R+FT and is
comparable to S2R-Oracle for beginner and intermediate players. The advanced
player is an exception. Note, S2R-Oracle+FT gets just 35% of i-S2R and S2R+FT
fine-tuning budget.

##### (1) Does i-S2R improve over S2R+FT in a human-robot interactive setting?

Figure 3 presents rally length distributions aggregated across all players
whilst Figure 4 splits the data by skill. Players are grouped into beginner
(40% players), intermediate (40% of players) and advanced (20% players). The
non-author player was classified as beginner. Please see Appendix I for skill
level definitions. When aggregated over all players, we see that i-S2R is able
to hold longer rallies (i.e. rallies that are longer than length 5) at a much
higher rate than S2R+FT, as shown in Figure 3. When the players are split by
skill level, i-S2R significantly outperforms S2R+FT for both beginner and
intermediate players (80% of the players). The improvement differs between the
two groups, with i-S2R yielding a $\approx 70\%$ and $\approx 175\%$
improvement for beginner and intermediate players respectively.

The policy trained by the advanced player has a different trend. Here, S2R+FT
dramatically outperforms i-S2R. We hypothesize that a good S2R model plays a
large part in the strong performance of S2R+FT since better transfer from
simulation improves the efficiency of subsequent fine-tuning (see Figure 5).
One possible explanation for the poor performance of i-S2R is that the policy
played fast. During evaluations, we observe the initial robot return is fast
with top spin, likely due to a combination of changes in the behavior model
from iteration 1 to 2 and 3 and inherent randomness in the training process.
In response, the advanced player returns the ball even faster, also with top
spin. This appears challenging for the robot to return. During evaluation,
most of the errors are made by the robot, where the rally ends with the ball
going over the human player’s end of the table. This suggests that fine-tuning
was not able to adjust in time to the top spin and fast speed of play, causing
the robot to hit over the table. One way to mitigate this would be to model
spin in simulation, so the policy could learn to respond to spin throughout
training, not just during fine-tuning. However, due to the time consuming
nature of repeating experiments on the physical system it is difficult to
fully explain this result, especially since both the training methodology and
involvement of humans introduces a high degree of variance.

##### (2) How many sim-to-real iterations does the human behavior model take
to converge?

For beginners we find that it only took two iterations for i-S2R to converge
(see Figure 5). In the leftmost chart showing beginner policy data, i-S2R
achieves comparable levels of performance at the end of the 2nd (fine-
tune-65%) and final (fine-tune-100%) iterations. However, for intermediate
skilled players this is not the case. The human behavior model from iteration
to iteration (Figure 6) offers a clue. For beginner players, the distribution
barely changes after the 2nd round as evidenced by the difference between the
left and right charts. Whereas for intermediate players the distribution
continues to change substantially from round 2 to 3 (specifically in y and z
velocities), which is perhaps why we see the strongest performance of i-S2R
after the 2nd iteration for beginners but after the 3rd iteration for
intermediate players.

The advanced player’s distribution hardly changes between the 2nd and 3rd
round and the performance of i-S2R is comparable across both. However this
does not explain why we observed the best i-S2R performance at the end of the
1st round for this player. Investigating the effect of playing style on
changes in ball distribution every iteration and hence on the sim-to-real gap
or training for more iterations for advanced players can shed light on this in
future work.

Figure 5: Policy performance at key checkpoints during training. For beginner
players i-S2R performance converges after just two iterations (see fine-
tune-65%). For intermediate players i-S2R takes three iterations to converge
(see fine-tune-100%). “S2R-Oracle-sim-3” here is same as “S2R-Oracle” in
Figure 4. Figure 6: The key distribution parameters change substantially from
initial ball distribution (sim1) to that after 1st round of sim training
(sim2). This is to be expected given we start from a simple human model (hits
across the table). The change in parameters between 1st and 2nd round of sim
training is much less (sim2 vs. sim3).

##### (3) What is the impact of the human behavior model?

For beginner and intermediate players, S2R-Oracle+FT is in line with i-S2R
performance. However S2R-Oracle+FT also achieved this level of performance
with just 35% of the real world training time compared to i-S2R and S2R+FT.
Therefore much of the benefit of i-S2R likely comes from improving the human
behavior model from iteration to iteration. It also suggests that if we had
access to the final human behavior model at the beginning of training, the
iterative sim-to-real training would not be needed. We could simply fine-tune
in the real world and achieve comparable performance with substantially less
human training time. S2R-Oracle+FT’s strong performance also validates our
motivation for this work, in which we hypothesized that the difficulty of
defining a good human behavior model a priori for human-robot cooperative
rallies was limiting performance.

This result indicates that i-S2R does not benefit from additional training
iterations in simulation over and above the improvements to the human behavior
model. The evaluations at earlier stages in training (shown in Figure 5)
suggest the remaining sim-to-real gap could be responsible. Figure 5 shows
that, in all cases, after both the second (sim-2) and third (sim-3) rounds of
simulated training, rally length drops noticeably. Reducing the sim-to-real
gap might improve i-S2R’s performance due to better starting points for the
last two rounds of fine-tuning.

Figure 7: Cross-evaluations mean rally lengths (with 95% CI) aggregated across
all players. i-S2R generalizes better to new players compared to S2R+FT.

##### (4) Does i-S2R offer any generalization benefits in this setting?

We evaluate the generalization capabilities of models trained with i-S2R, and
how they compare against models trained using S2R+FT by conducting cross-
evaluations. A “cross-evaluation” of a policy is an evaluation conducted by a
human who did not play with the policy during training. Each of the 5 policies
was cross-evaluated by randomly selecting 2 other humans from the human-
subject pool and averaging the results. As shown in Figure 7, i-S2R
substantially outperforms S2R+FT when the models are cross-evaluated by other
players (with similar blind evaluations as earlier) including for the advanced
player where S2R+FT was best in self evaluation (see Appendix K for details by
player). This observation holds whether we look at absolute or normalized
rally length (see Appendix J for normalization methodology). Performance with
other players is lower for all models, however i-S2R maintains around 70% of
performance on average compared to 30% for S2R+FT. We hypothesize that the
broader training distribution obtained by iterating between simulation and
reality leads to policies that can deal with a wider range of ball throws,
leading to better generalization to new players. Our confidence in this
hypothesis is strengthened by the fact that S2R-Oracle+FT also outperforms
S2R+FT in this setting.

##  7 Limitations

Having a human in the loop poses numerous challenges to robotic reinforcement
learning. It slows down the overall learning process to accommodate human
participants, and limits the scale at which one can experiment. As one
example, while we tested our method on five subjects, time limitations
prevented us from training with multiple random seeds for each subject. There
is significant variation in how people interact with robots (or sometimes even
the same person over time), which introduces extra variance into our
experiments. In our experiments, the trends we saw for one particular subject
were substantially different from all other subjects, and we could not fully
explain why.

It is possible for an expert human player to achieve long rallies by keeping
the ball in a very narrow distribution without really improving the inherent
capability of the agent to play beyond those balls. In our studies, since we
used non-professional players, this was not an issue.

Another limitation arising from training a policy with a human in the loop is
the possibility that some performance improvements are attributable to human
learning and not policy learning. We did our best to mitigate this by asking
players to evaluate all models “blind” (i.e. the player is unaware of what
model they are evaluating) and at the end of training, after which the
majority of human learning was likely to have occurred. Consequently, we think
that differences between models reflect differences in policy capability and
not human capability.

Finally, we represent humans in simulation in a simple way — by capturing all
initial position and velocity ranges during their play — and then we sample
each ball in simulation uniformly and independently. This ignores the
probability distribution of balls within those ranges and also results in a
loss of correlation between subsequent balls in a rally. The behavior model
also omits spin and human attributes such as stamina, skill level, intention,
and curiosity. These could be addressed by developing a more sophisticated
behavior model that takes these factors into account.

##  8 Conclusion

We present i-S2R to learn RL policies that are able to interact with humans by
iteratively training in simulation and fine-tuning in the real world with
humans in the loop. The approach starts with a coarse model of human behavior
and refines it over a series of fine-tuning iterations. The effectiveness of
this method is demonstrated in the context of a table tennis rallying task.
Extensive “blind” experiments shed light on various aspects of the method and
compare it against a baseline where we train and fine-tune in real only once
(S2R). We show that i-S2R outperforms S2R in aggregate, and the difference in
performance is particularly significant for beginner and intermediate players
(4/5). Moreover, i-S2R generalizes much better than S2R to other players.

#### Acknowledgments

We thank Pete Florence, Kamyar Ghasemipour, Andrew Silva, Ellie Sanoubari, and
Vincent Vanhoucke for their helpful and insightful feedback on earlier
versions of this manuscript. We are grateful to Michael Ahn, Sherry Moore, Ken
Oslund, and Grace Vesom for all their work on the robot control stack, for
Omar Cortes’ help in training models and for Justin Boyd and Khem Holden’s
help in calibrating our vision system. We thank Jon Abelian, Gus Kouretas,
Thinh Nguyen, and Krista Reymann for all that they do to help maintain our
robotic system. We would also like to thank Navdeep Jaitly, Peng Xu, Nevena
Lazic, and Reza Mahjourian for their work on early versions of this system.
Finally, we would like to thank Jon Abelian, Justin Boyd, Omar Cortes, Khem
Holden, Gus Kouretas and Thinh Nguyen for their help evaluating models.

## References

  * Peng et al. [2018] X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel.  Sim-to-real transfer of robotic control with dynamics randomization.  In _2018 IEEE International Conference on Robotics and Automation, ICRA 2018, Brisbane, Australia, May 21-25, 2018_ , pages 1–8. IEEE, 2018.
  * Chebotar et al. [2019] Y. Chebotar, A. Handa, V. Makoviychuk, M. Macklin, J. Issac, N. D. Ratliff, and D. Fox.  Closing the sim-to-real loop: Adapting simulation randomization with real world experience.  In _International Conference on Robotics and Automation, ICRA 2019, Montreal, QC, Canada, May 20-24, 2019_ , pages 8973–8979. IEEE, 2019.
  * Andrychowicz et al. [2020] M. Andrychowicz, B. Baker, M. Chociej, R. Józefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, J. Schneider, S. Sidor, J. Tobin, P. Welinder, L. Weng, and W. Zaremba.  Learning dexterous in-hand manipulation.  _Int. J. Robotics Res._ , 39(1), 2020.
  * Kataoka et al. [2022] S. Kataoka, S. K. S. Ghasemipour, D. Freeman, and I. Mordatch.  Bi-manual manipulation and attachment via sim-to-real reinforcement learning, 2022.  URL https://arxiv.org/abs/2203.08277.
  * Lee et al. [2019] J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun, and M. Hutter.  Learning agile and dynamic motor skills for legged robots.  _Sci. Robotics_ , 4(26), 2019.
  * Peng et al. [2020] X. B. Peng, E. Coumans, T. Zhang, T. E. Lee, J. Tan, and S. Levine.  Learning agile robotic locomotion skills by imitating animals.  In M. Toussaint, A. Bicchi, and T. Hermans, editors, _Robotics: Science and Systems XVI, Virtual Event / Corvalis, Oregon, USA, July 12-16, 2020_ , 2020.
  * Sadeghi and Levine [2017] F. Sadeghi and S. Levine.  CAD2RL: real single-image flight without a single real image.  In N. M. Amato, S. S. Srinivasa, N. Ayanian, and S. Kuindersma, editors, _Robotics: Science and Systems XIII, Massachusetts Institute of Technology, Cambridge, Massachusetts, USA, July 12-16, 2017_ , 2017.
  * Loquercio et al. [2021] A. Loquercio, E. Kaufmann, R. Ranftl, M. Müller, V. Koltun, and D. Scaramuzza.  Learning high-speed flight in the wild.  _Sci. Robotics_ , 6(59), 2021.
  * [9] D. Pomerleau.  ALVINN: an autonomous land vehicle in a neural network.  In D. S. Touretzky, editor, _Advances in Neural Information Processing Systems 1, [NIPS Conference, Denver, Colorado, USA, 1988]_ , pages 305–313.
  * Zhang et al. [2017] T. Zhang, Z. McCarthy, O. Jow, D. Lee, K. Goldberg, and P. Abbeel.  Deep imitation learning for complex manipulation tasks from virtual reality teleoperation.  _arXiv preprint arXiv:1710.04615_ , 2017.
  * Abbeel and Ng [2004] P. Abbeel and A. Y. Ng.  Apprenticeship learning via inverse reinforcement learning.  In C. E. Brodley, editor, _Machine Learning, Proceedings of the Twenty-first International Conference (ICML 2004), Banff, Alberta, Canada, July 4-8, 2004_ , volume 69 of _ACM International Conference Proceeding Series_. ACM, 2004.
  * Ziebart et al. [2008] B. D. Ziebart, A. L. Maas, J. A. Bagnell, and A. K. Dey.  Maximum entropy inverse reinforcement learning.  In D. Fox and C. P. Gomes, editors, _Proceedings of the Twenty-Third AAAI Conference on Artificial Intelligence, AAAI 2008, Chicago, Illinois, USA, July 13-17, 2008_ , pages 1433–1438. AAAI Press, 2008\.
  * Lee et al. [2020] J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter.  Learning quadrupedal locomotion over challenging terrain.  _CoRR_ , 2020.
  * OpenAI et al. [2019] OpenAI, I. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, J. Schneider, N. Tezak, J. Tworek, P. Welinder, L. Weng, Q. Yuan, W. Zaremba, and L. Zhang.  Solving rubik’s cube with a robot hand.  2019\.
  * Tan et al. [2018] J. Tan, T. Zhang, E. Coumans, A. Iscen, Y. Bai, D. Hafner, S. Bohez, and V. Vanhoucke.  Sim-to-real: Learning agile locomotion for quadruped robots.  _CoRR_ , abs/1804.10332, 2018.  URL http://arxiv.org/abs/1804.10332.
  * Zhao et al. [2020] W. Zhao, J. P. Queralta, and T. Westerlund.  Sim-to-real transfer in deep reinforcement learning for robotics: a survey.  _CoRR_ , 2020.
  * Höfer et al. [2021] S. Höfer, K. Bekris, A. Handa, J. C. Gamboa, M. Mozifian, F. Golemo, C. Atkeson, D. Fox, K. Goldberg, J. Leonard, C. Karen Liu, J. Peters, S. Song, P. Welinder, and M. White.  Sim2real in robotics and automation: Applications and challenges.  _IEEE Transactions on Automation Science and Engineering_ , 18(2):398–400, 2021.  doi:10.1109/TASE.2021.3064065.
  * Neunert et al. [2016] M. Neunert, T. Boaventura, and J. Buchli.  Why off-the-shelf physics simulators fail in evaluating feedback controller performance - a case study for quadrupedal robots.  2016\.
  * Zhu et al. [2017] S. Zhu, A. Kimmel, K. E. Bekris, and A. Boularias.  Model identification via physics engines for improved policy search.  _CoRR_ , 2017.
  * Kaspar et al. [2020] M. Kaspar, J. D. M. Osorio, and J. Bock.  Sim2real transfer for reinforcement learning without dynamics randomization.  _CoRR_ , abs/2002.11635, 2020.
  * Tan et al. [2016] J. Tan, Z. Xie, B. Boots, and C. K. Liu.  Simulation-based design of dynamic controllers for humanoid balancing.  In _2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 2729–2736, 2016.  doi:10.1109/IROS.2016.7759424.
  * Ota et al. [2020] K. Ota, D. K. Jha, D. Romeres, J. van Baar, K. A. Smith, T. Semitsu, T. Oiki, A. Sullivan, D. Nikovski, and J. B. Tenenbaum.  Towards human-level learning of complex physical puzzles.  _CoRR_ , abs/2011.07193, 2020.  URL https://arxiv.org/abs/2011.07193.
  * Farchy et al. [2013] A. Farchy, S. Barrett, P. MacAlpine, and P. Stone.  Humanoid robots learning to walk faster: From the real world to simulation and back.  In _Proceedings of the 2013 International Conference on Autonomous Agents and Multi-Agent Systems_ , AAMAS ’13, page 39–46, 2013.
  * Smith et al. [2021] L. M. Smith, J. C. Kew, X. B. Peng, S. Ha, J. Tan, and S. Levine.  Legged robots that keep on learning: Fine-tuning locomotion policies in the real world.  _CoRR_ , abs/2110.05457, 2021.
  * Barrett et al. [2010] S. Barrett, M. E. Taylor, and P. Stone.  Transfer learning for reinforcement learning on a physical robot.  In _AAMAS 2010_ , 2010.
  * Rusu et al. [2016] A. A. Rusu, M. Vecerík, T. Rothörl, N. Heess, R. Pascanu, and R. Hadsell.  Sim-to-real robot learning from pixels with progressive nets.  _CoRR_ , 2016.
  * Song et al. [2020] X. Song, Y. Yang, K. Choromanski, K. Caluwaerts, W. Gao, C. Finn, and J. Tan.  Rapidly adaptable legged robots via evolutionary meta-learning.  In _IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2020, Las Vegas, NV, USA, October 24, 2020 - January 24, 2021_ , pages 3769–3776. IEEE, 2020.  doi:10.1109/IROS45743.2020.9341571.  URL https://doi.org/10.1109/IROS45743.2020.9341571.
  * Büchler et al. [2020] D. Büchler, S. Guist, R. Calandra, V. Berenz, B. Schölkopf, and J. Peters.  Learning to play table tennis from scratch using muscular robots.  _CoRR_ , abs/2006.05935, 2020.  URL https://arxiv.org/abs/2006.05935.
  * Billingsley [1983] J. Billingsley.  Robot ping pong.  _Practical Computing_ , 1983.
  * Knight and Lowery [1986] J. Knight and D. Lowery.  Pingpong-playing robot controlled by a microcomputer.  _Microprocessors and Microsystems - Embedded Hardware Design_ , 1986\.
  * Hartley [1983] J. Hartley.  Toshiba progress towards sensory control in real time.  _The Industrial Robot 14-1_ , pages 50–52, 1983.
  * Hashimoto et al. [1987] H. Hashimoto, F. Ozaki, and K. Osuka.  Development of ping-pong robot system using 7 degree of freedom direct drive robots.  In _Industrial Applications of Robotics and Machine Vision_ , 1987\.
  * Muelling et al. [2010] K. Muelling, J. Kober, and J. Peters.  A biomimetic approach to robot table tennis.  _Adaptive Behavior_ , 2010.
  * Kyohei et al. [2020] A. Kyohei, N. Masamune, and Y. Satoshi.  The ping pong robot to return a ball precisely.  2020\.
  * Miyazaki et al. [2002] F. Miyazaki, M. Takeuchi, M. Matsushima, T. Kusano, and T. Hashimoto.  Realization of the table tennis task based on virtual targets.  _ICRA_ , 2002.
  * Miyazaki et al. [2006] F. Miyazaki et al.  Learning to dynamically manipulate: A table tennis robot controls a ball and rallies with a human being.  In _Advances in Robot Control_ , 2006.
  * Anderson [1988] R. Anderson.  _A Robot Ping-Pong Player: Experiments in Real-Time Intelligent Control_.  MIT Press, 1988.
  * Muelling et al. [2010] K. Muelling et al.  Simulating human table tennis with a biomimetic robot setup.  In _Simulation of Adaptive Behavior_ , 2010.
  * Zhu et al. [2018] Y. Zhu, Y. Zhao, L. Jin, J. Wu, and R. Xiong.  Towards high level skill learning: Learn to return table tennis ball using monte-carlo based policy gradient method.  _IEEE International Conference on Real-time Computing and Robotics_ , 2018.
  * Huang et al. [2015] Y. Huang, B. Schölkopf, and J. Peters.  Learning optimal striking points for a ping-pong playing robot.  _IROS_ , 2015.
  * Sun et al. [2011] Y. Sun, R. Xiong, Q. Zhu, J. Wu, and J. Chu.  Balance motion generation for a humanoid robot playing table tennis.  _IEEE-RAS Humanoids_ , 2011.
  * Mahjourian et al. [2018] R. Mahjourian, N. Jaitly, N. Lazic, S. Levine, and R. Miikkulainen.  Hierarchical policy design for sample-efficient learning of robot table tennis through self-play.  _arXiv:1811.12927_ , 2018.
  * Matsushima et al. [2003] M. Matsushima, T. Hashimoto, and F. Miyazaki.  Learning to the robot table tennis task-ball control and rally with a human.  _IEEE International Conference on Systems, Man and Cybernetics_ , 2003\.
  * Matsushima et al. [2005] M. Matsushima, T. Hashimoto, M. Takeuchi, and F. Miyazaki.  A learning approach to robotic table tennis.  _IEEE Transactions on Robotics_ , 2005.
  * Muelling et al. [2010] K. Muelling, J. Kober, and J. Peters.  Learning table tennis with a mixture of motor primitives.  _IEEE-RAS Humanoids_ , 2010.
  * Muelling et al. [2012] K. Muelling, J. Kober, O. Kroemer, and J. Peters.  Learning to select and generalize striking movements in robot table tennis.  _The International Journal of Robotics Research_ , 2012.
  * Huang et al. [2016] Y. Huang, D. Buchler, O. Koç, B. Schölkopf, and J. Peters.  Jointly learning trajectory generation and hitting point prediction in robot table tennis.  _IEEE-RAS Humanoids_ , 2016.
  * Koç et al. [2018] O. Koç, G. Maeda, and J. Peters.  Online optimal trajectory generation for robot table tennis.  _Robotics & Autonomous Systems_, 2018.
  * Tebbe et al. [2018] J. Tebbe, Y. Gao, M. Sastre-Rienietz, and A. Zell.  A table tennis robot system using an industrial kuka robot arm.  _GCPR_ , 2018.
  * Gao et al. [2019] Y. Gao, J. Tebbe, J. Krismer, and A. Zell.  Markerless racket pose detection and stroke classification based on stereo vision for table tennis robots.  _IEEE Robotic Computing_ , 2019.
  * Tebbe et al. [2021] J. Tebbe, L. Krauch, Y. Gao, and A. Zell.  Sample-efficient reinforcement learning in robotic table tennis.  _ICRA_ , 2021.
  * Gao et al. [2020] W. Gao, L. Graesser, K. Choromanski, X. Song, N. Lazic, P. Sanketi, V. Sindhwani, and N. Jaitly.  Robotic table tennis with model-free reinforcement learning.  _IROS_ , 2020.
  * Chen et al. [2020a] L. Chen, R. R. Paleja, and M. C. Gombolay.  Learning from suboptimal demonstration via self-supervised reward regression.  _CoRL_ , 2020a.
  * Chen et al. [2020b] L. Chen, R. R. Paleja, M. Ghuy, and M. C. Gombolay.  Joint goal and strategy inference across heterogeneous demonstrators via reward network distillation.  _CoRR_ , abs/2001.00503, 2020b.
  * Yu et al. [2013] Z. Yu, Y. Liu, Q. Huang, X. Chen, W. Zhang, J. Li, G. Ma, L. Meng, T. Li, and W. Zhang.  Design of a humanoid ping-pong player robot with redundant joints.  _2013 IEEE International Conference on Robotics and Biomimetics (ROBIO)_ , pages 911–916, 2013.
  * Aly et al. [2017] A. Aly, S. Griffiths, and F. Stramandinoli.  Metrics and benchmarks in human-robot interaction: Recent advances in cognitive robotics.  _Cognitive Systems Research_ , 43:313–323, 2017.
  * Huber et al. [2009] M. Huber, H. Radrich, C. Wendt, M. Rickert, A. Knoll, T. Brandt, and S. Glasauer.  Evaluation of a novel biologically inspired trajectory generator in human-robot interaction.  In _RO-MAN 2009-The 18th IEEE International Symposium on Robot and Human Interactive Communication_ , pages 639–644. IEEE, 2009.
  * Silva et al. [2022] A. Silva, K. Metcalf, N. Apostoloff, and B.-J. Theobald.  Fedembed: Personalized private federated learning, 2022.  URL https://arxiv.org/abs/2202.09472.
  * Paleja et al. [2021] R. Paleja, M. Ghuy, N. Ranawaka Arachchige, R. Jensen, and M. Gombolay.  The utility of explainable ai in ad hoc human-machine teaming.  In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, _Advances in Neural Information Processing Systems_ , volume 34, pages 610–623. Curran Associates, Inc., 2021.  URL https://proceedings.neurips.cc/paper/2021/file/05d74c48b5b30514d8e9bd60320fc8f6-Paper.pdf.
  * Puterman [2014] M. L. Puterman.  _Markov decision processes: discrete stochastic dynamic programming_.  John Wiley & Sons, 2014.
  * Schulman et al. [2017] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov.  Proximal policy optimization algorithms.  _CoRR_ , abs/1707.06347, 2017.
  * Kalashnikov et al. [2018] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, and S. Levine.  Scalable deep reinforcement learning for vision-based robotic manipulation.  In _Proceedings of The 2nd Conference on Robot Learning_ , pages 651–673. PMLR, 2018.
  * Haarnoja et al. [2018] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  In _Proceedings of the 35th International Conference on Machine Learning_ , pages 1861–1870. PMLR, 2018.
  * Choromanski et al. [2018] K. Choromanski, M. Rowland, V. Sindhwani, R. E. Turner, and A. Weller.  Structured Evolution with Compact Architectures for Scalable Policy Optimization.  In _Proceedings of the 35th International Conference on Machine Learning_ , pages 969–977. PMLR, 2018.
  * Wierstra et al. [2011] D. Wierstra, T. Schaul, T. Glasmachers, Y. Sun, and J. Schmidhuber.  Natural evolution strategies, 2011.
  * Salimans et al. [2017] T. Salimans, J. Ho, X. Chen, S. Sidor, and I. Sutskever.  Evolution strategies as a scalable alternative to reinforcement learning.  _arXiv:1703.03864_ , 2017.
  * Nesterov and Spokoiny [2017] Y. Nesterov and V. Spokoiny.  Random gradient-free minimization of convex functions.  _FoCM_ , 2017.
  * Mania et al. [2018] H. Mania, A. Guy, and B. Recht.  Simple random search provides a competitive approach to reinforcement learning.  _NeurIPS_ , 2018.
  * Choromanski et al. [2019] K. Choromanski, A. Pacchiano, J. Parker-Holder, Y. Tang, D. Jain, Y. Yang, A. Iscen, J. Hsu, and V. Sindhwani.  Provably robust blackbox optimization for reinforcement learning.  In L. P. Kaelbling, D. Kragic, and K. Sugiura, editors, _3rd Annual Conference on Robot Learning, CoRL 2019, Osaka, Japan, October 30 - November 1, 2019, Proceedings_ , volume 100 of _Proceedings of Machine Learning Research_ , pages 683–696. PMLR, 2019.  URL http://proceedings.mlr.press/v100/choromanski20a.html.
  * Nelder and Mead [1965] J. A. Nelder and R. Mead.  A simplex method for function minimization.  _Computer Journal_ , 7:308–313, 1965.
  * Schubert et al. [2017] E. Schubert, J. Sander, M. Ester, H.-P. Kriegel, and X. Xu.  Dbscan revisited, revisited: Why and how you should (still) use dbscan.  _ACM Transactions on Database Systems_ , (3), 2017.
  * Blank et al. [2017] P. Blank, B. H. Groh, and B. M. Eskofier.  Ball speed and spin estimation in table tennis using a racket-mounted inertial sensor.  In S. C. Lee, L. Takayama, K. N. Truong, J. Healey, and T. Ploetz, editors, _ISWC_ , pages 2–9. ACM, 2017.  ISBN 978-1-4503-5188-1.  URL http://dblp.uni-trier.de/db/conf/iswc/iswc2017.html#BlankGE17.
  * Gao et al. [2021] Y. Gao, J. Tebbe, and A. Zell.  Optimal stroke learning with policy gradient approach for robotic table tennis.  _CoRR_ , abs/2109.03100, 2021.  URL https://arxiv.org/abs/2109.03100.
  * Coumans and Bai [2016–2021] E. Coumans and Y. Bai.  Pybullet, a python module for physics simulation for games, robotics and machine learning.  http://pybullet.org, 2016–2021.
  * Nagabandi et al. [2018] A. Nagabandi et al.  Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning.  In _ICRA_ , 2018.
  * abb [Vasteras, 2020] Abb application manual - externally guided motion., Vasteras, 2020.
  * van der Maaten and Hinton [2008] L. van der Maaten and G. Hinton.  Visualizing data using t-SNE.  _Journal of Machine Learning Research_ , 9:2579–2605, 2008\.  URL http://www.jmlr.org/papers/v9/vandermaaten08a.html.

##  Appendix A Author Contributions

  * •

Saminda Abeyruwan co-led the project, developed i-S2R, ran initial i-S2R
experiments, helped build and maintain robotic infrastructure, introduced a
set of additional rewards to help with fine-tuning, helped with analyzing the
sim-to-real discrepancies, organized the human training and evaluation
protocols, and helped to write parts of the paper related to human behavior
modeling via ball trajectories. Was one of the test subjects.

  * •

Laura Graesser co-led the project, ran initial i-S2R experiments, advised on
experimental design, helped build and maintain robotic infrastructure,
analyzed the results, wrote the paper.

  * •

David B. D’Ambrosio helped build and maintain robotic infrastructure and
vision system. Analysis of human behavior parameters. Literature review. Post-
hoc evaluation of human-robot rallies. Discussion and writing of paper.

  * •

Avi Singh wrote the introduction and helped craft the overall narrative for
the paper. Made the project website. Was one of the test subjects.

  * •

Anish Shankar worked on the system’s hardware and software implementation,
data infrastructure used in analysis and overall system performance. Was one
of the test subjects.

  * •

Alex Bewley helped build the vision system, curate ball detection data and
trained the perception module used in this work. Contributed to writing the
vision related sections of the paper.

  * •

Deepali Jain co-developed with Krzysztof Choromanski: (a) the ES algorithm
used to learn policies presented in this paper and (b) distributed
optimization infrastructure to apply ES-based training. Conducted extensive
tests of ES in the simulator in the first phase of the project. Edited the
paper.

  * •

Krzysztof Choromanski co-developed with Deepali Jain: (a) the ES algorithm
used to learn policies presented in this paper and (b) distributed
optimization infrastructure to apply ES-based training. Conducted extensive
tests of ES in the simulator in the first phase of the project. Edited the
paper.

  * •

Pannag R. Sanketi managed the team. Set the research direction, co-led the
project, and wrote the paper. Was one of the test subjects.

##  Appendix B Details on the BGS Algorithm

As described in section 3, the ES objective is given by:

$F_{\sigma}(\theta)=\mathbb{E}_{\mathbf{\delta}\sim\mathcal{N}(0,\mathbf{I}_{d})}[F(\theta+\sigma\mathbf{\delta})],$
(2)

where $\sigma>0$ controls the precision of the smoothing, and $\delta$ is a
random normal vector with the same dimension as the policy parameters
$\theta$.

ES does not use derivatives or back-propagation to update policy parameters.
Instead, the gradient of the policy parameters $\theta$ with respect to the
objective is estimated with various Monte Carlo techniques. In this work we
apply Monte Carlo leveraging in addition the antithetic sampling trick, widely
applied by the community.

Specifically, $\theta$ is perturbed either by adding or subtracting Gaussian
perturbations $\delta_{R_{i}}$ and completing environment rollouts using the
perturbed parameters. As a result each perturbation is associated with a
reward, one for each direction $R^{+}_{i}$ and $R^{-}_{i}$.

Assuming the perturbations, $\delta_{R_{i}}$, are rank ordered with
$\delta_{R_{1}}$ being the top performing direction, then the policy update
can therefore expressed as follows.

$\theta^{{}^{\prime}}=\theta+\alpha\frac{1}{\sigma^{R}}\sum^{k}_{i=1}\Bigg{[}\Big{(}\Big{(}\frac{1}{m}\sum^{m}_{j=1}R^{+}_{i,j}\Big{)}-\Big{(}\frac{1}{m}\sum^{m}_{j=1}R^{-}_{i,j}\Big{)}\Big{)}\delta_{R_{i}}\Bigg{]},$
(3)

where $\alpha$ is the step size, $\sigma^{R}$ is the standard deviation of
each distinct reward (positive and negative direction), $k$ is the number of
top directions (elites), $N$ is the number of directions sampled per parameter
update, and $k<N$. $m$ is the number of repeats per direction and
$R^{+}_{i,j}$ is the reward corresponding to the j-th repeat of i-th in the
positive direction. $R^{-}_{i,j}$ is the same but in the negative direction.

Our BGS algorithm is built on two pillars that we describe in detail below.

##### Novel Elite-Choice Algorithm:

One of the key features of BGS is the novel algorithm of selecting top
directions (the elites). In ARS [68], the ranking of the elites is determined
by treating each antithetic direction separately. All rewards are ranked
yielding an ordering of directions based on the absolute rewards of either the
positive or negative directions (Equation 4). Whereas in BGS we take the
difference in rewards between the positive and negative directions and rank
the differences to yield an ordering over directions (Equation 5).

$\displaystyle ARS:\text{Sort }\delta_{R_{i}}\text{ by
max}\\{R^{+}_{1},...,R^{+}_{i},R^{-}_{1},...,R^{-}_{i}\\}.$ (4) $\displaystyle
BGS:\text{Sort }\delta_{R_{i}}\text{ by
max}\Bigg{\\{}\Big{(}\frac{1}{m}\sum^{m}_{j=1}R^{+}_{1,j}-\frac{1}{m}\sum^{m}_{j=1}R^{-}_{1,j}\Big{)},...,\Big{(}\frac{1}{m}\sum^{m}_{j=1}R^{+}_{i,j}-\frac{1}{m}\sum^{m}_{j=1}R^{-}_{i,j}\Big{)}\Bigg{\\}}.$
(5)

ARS can be interpreted as ranking directions in absolute reward space, whereas
BGS ranks directions according to reward curvature because it ranks based on
reward deltas. The new elite-choice algorithm was the game changer for all
policy-training experiments. We could not train efficient policies with ARS
(even in the simulator).

##### Orthogonal Perturbations (Samples):

The other key feature of the BGS algorithm is the use of the orthogonal
ensembles of perturbations (samples) $\delta_{R_{i}}$. This technique was
originally introduced in [64] and relies on constructing perturbations
$\delta_{R_{i}}$ in blocks, where each block consists of pairwise orthogonal
samples. Those samples are still of Gaussian marginal distributions, matching
those of the regular non-orthogonal variant. The feasibility of such a
construction comes from the isotropic property of the Gaussian distribution
(see: [64] for details). We observed that orthogonal perturbations led to
faster convergence in training.

In addition to our novel-elite choice algorithm and orthogonal perturbations,
we apply a number of common approaches used in ES methods; state normalization
[66, 75], reward normalization [68], and perturbation filtering [66]. We also
repeat and average rollouts with the same parameters to reduce variance.

##  Appendix C Iterative-Sim-to-Real Procedure

For the table tennis rallying task, we found 3 iterations to be sufficient.
The policy was trained for 30k to 45k updates for the first round of training
in simulation since it has to learn everything from scratch. For subsequent
simulation rounds, the policy was only trained for 5k updates, since we warm
start from latest real world policy weights and its primary task here is
_adaptation_ to a change in human behavior. Due to the human cost of real
world fine-tuning and evaluation, we did not experiment with shorter or longer
training cycles. In the real world, the policy was fine-tuned for 70 parameter
updates per cycle for the last two cycles and 60 updates for the first cycle
to make 200 updates in total. This is equivalent to approximately 2 hours of
wall clock time per cycle, which was our budget per player.

###  C.1 Seed Selection for Rounds of Simulated Training

We have used the following methodology when training in simulation. When
training in simulation is required, whether it is training from scratch or
intermediate steps of i-S2R, we train 3 models with 3 different random seeds.
Different random seeds were used for different players. When transferring to
the physical robot, each model is evaluated for 50 episodes according to the
training and evaluation instructions provided in subsection H.1. The model
with the highest average return is selected for fine-tuning and further
experiments. We have used a simple, sparse reward structure for evaluation: if
the robot hits the ball, a reward of +1 is given, and if the ball lands on the
human side, an additional +1 is given reward. Therefore, the maximum episode
reward is +2. If the robot misses the ball, there is no reward, and if the
robot faulted or stopped during an episode, a -2 reward is assigned to the
episode.

###  C.2 Bringing the Fine-Tuned Model from the Real World Back to Simulation

In i-S2R the fine-tuned model from the real world is brought back to
simulation in the next iteration for two reasons. First, the fine-tuned model
has been trained on the most up to date human behavior. As a result it is
likely better adapted to play with the updated human behavior model than the
latest set of policy weights from simulated training which were trained on the
prior human behavior model.

Second, there are aspects of our real world system which we have not been able
to model accurately in simulation on top of the challenges in modeling human
behavior. Our vision system does not detect spin, there are calibration
defects, variability in estimated delays, conditions of the surface materials,
wear and tear (of table tennis balls), physical robot properties mismatched
with the simulated robot. Therefore, our policies are subject to a sim2real
gap and real world fine-tuning adapts the policy to real world conditions.
When we transfer the fine-tuned policy weights back to simulation we observe
that some adaptation to real world conditions persist from iteration to
iteration, reducing the adaptation time in subsequent fine-tuning iterations.

However it is possible that this approach makes training in simulation more
difficult. It would be interesting to compare our approach with a variant in
which the policy weights are not transferred back to simulation from the real
world. Instead training in simulation would continue using the latest policy
weights from the previous iteration but using the latest human model after
real fine-tuning. We leave this to future work.

###  C.3 Human Behavior Models

Table 1 shows the changes of the ball behavior models, $M_{0}$, $M_{1}$, and
$M_{2}$, for each player. Skill levels: players 3 and 5 are beginners, players
2 and 4 are intermediate, and player 1 is advanced.

| player 1 | player 2 | player 3 | player 4 | player 5
---|---|---|---|---|---
| $M_{0}$ | $M_{1}$ | $M_{2}$ | $M_{0}$ | $M_{1}$ | $M_{2}$ | $M_{0}$ | $M_{1}$ | $M_{2}$ | $M_{0}$ | $M_{1}$ | $M_{2}$ | $M_{0}$ | $M_{1}$ | $M_{2}$
min z velocity ($ms^{-1}$) | 1.25 | -1.47 | -1.56 | 0.88 | -1.14 | -1.27 | 0.64 | -1.23 | -1.39 | 0.04 | -1.31 | -1.72 | 0.52 | -0.70 | -0.87
max z velocity ($ms^{-1}$) | 2.71 | 2.95 | 2.95 | 2.84 | 2.84 | 3.07 | 2.49 | 2.79 | 2.79 | 2.25 | 2.73 | 2.73 | 2.59 | 2.75 | 2.75
max x velocity ($|ms^{-1}|$) | 1.70 | 3.05 | 3.05 | 1.41 | 2.81 | 2.89 | 0.79 | 2.59 | 2.78 | 1.50 | 3.40 | 3.45 | 0.68 | 2.30 | 2.68
min y velocity ($|ms^{-1}|$) | 4.12 | 2.17 | 2.17 | 3.97 | 3.52 | 2.74 | 2.95 | 2.19 | 2.19 | 4.44 | 3.33 | 2.96 | 4.20 | 2.73 | 2.70
max y velocity ($|ms^{-1}|$) | 6.31 | 6.63 | 6.63 | 6.38 | 8.05 | 8.82 | 6.03 | 7.11 | 7.11 | 7.33 | 7.33 | 7.36 | 6.34 | 6.94 | 6.94
x start min ($m$) | -0.19 | -0.79 | -0.83 | 0.02 | -0.86 | -0.87 | 0.10 | -0.93 | -0.93 | -0.09 | -0.8 | -0.83 | 0.25 | -0.64 | -0.80
x start max ($m$) | 0.19 | 0.63 | 0.70 | 0.73 | 0.65 | 0.67 | 0.61 | 0.79 | 0.81 | 0.68 | 0.78 | 0.83 | 0.42 | 0.55 | 0.64
y start min ($m$) | 1.05 | 0.04 | 0.04 | 0.85 | 0.37 | 0.08 | 0.61 | 0.05 | 0.04 | 1.01 | 0.21 | 0.04 | 1.08 | 0.17 | 0.17
y start max ($m$) | 2.51 | 1.87 | 1.92 | 1.68 | 1.89 | 1.95 | 1.35 | 1.83 | 1.92 | 1.88 | 1.58 | 1.58 | 1.44 | 1.81 | 1.82
z start min ($m$) | 0.07 | 0.19 | 0.19 | 0.15 | 0.08 | 0.01 | 0.18 | -0.15 | -0.29 | 0.15 | 0.24 | 0.19 | 0.33 | 0.26 | 0.26
z start max ($m$) | 0.62 | 0.83 | 0.83 | 0.45 | 0.59 | 0.63 | 0.52 | 1.10 | 1.11 | 0.72 | 0.72 | 0.76 | 0.58 | 0.76 | 0.79
x land min ($m$) | -0.01 | -0.62 | -0.71 | -0.08 | -0.52 | -0.68 | -0.08 | -0.67 | -0.74 | -0.02 | -0.19 | -0.63 | 0.07 | -0.66 | -0.666
x land max ($m$) | 0.67 | 0.74 | 0.74 | 0.76 | 0.76 | 0.76 | 0.76 | 0.76 | 0.76 | 0.75 | 0.75 | 0.76 | 0.58 | 0.73 | 0.73
y land min ($m$) | -1.35 | -1.37 | -1.37 | -1.34 | -1.36 | -1.37 | -1.33 | -1.37 | -1.37 | -1.37 | -1.37 | -1.37 | -1.31 | -1.37 | -1.37
y land max ($m$) | -0.2 | -0.15 | -0.15 | -0.23 | -0.15 | -0.15 | -0.22 | -0.18 | -0.16 | -0.27 | -0.21 | -0.15 | -0.30 | -0.16 | -0.16
Table 1: Ball distribution changes, $M_{0}$, $M_{1}$, and $M_{2}$, per player
for i-S2R.

###  C.4 Details on Modeling Human Ball Distributions

We use the model, $\ddot{x}_{t}=g-K_{d}||\dot{x}_{t}||\dot{x}_{t}$,
$x_{t+1}=x_{t}+\Delta t(\dot{x}_{t}+\frac{\Delta t\ddot{x}_{t}}{2})$,
$\dot{x}_{t+1}=\dot{x}_{t}+\Delta t\ddot{x}_{t}$ to simulate a trajectory,
where (1) $x_{t}$, $\dot{x}_{t}$, and $\ddot{x}_{t}$ denote the position,
velocity, and acceleration of the ball at time t, (2)
$g=-9.81m/s{{}^{2}}[0,0,1]^{T}$ is the gravity, and (3)
$K_{d}=C_{d}\rho\frac{A}{2m}$. $m=0.0027kg$ is the ball’s mass,
$\rho=1.29kg/m^{3}$ is the air density, $C_{d}=0.47$ is the the drag
coefficient, and $A=1.256\times 10^{-3}m^{2}$ is the cross-sectional area for
a standard table tennis ball.

##  Appendix D Hardware Details

###  D.1 Robot Hardware Overview

Player Robot: The player robot (Figure 1) is a combination of an ABB IRB 120T
6-DOF robotic arm mounted to a two-dimensional Festo linear actuator, creating
an 8-DOF system. The robot arm’s end effector is a standard table tennis
paddle with the handle removed attached to a 174.3mm extension. The arm is
controlled with ABB’s Externally Guided Motion (EGM) interface at
approximately 248Hz by specifying joint position and speed targets [76]. The
2D linear actuator is independently controlled at up to 125Hz with position
target commands for each axis at a fixed velocity through Festo’s custom
Modbus interface. Position feedback from the robots is received at the command
rate. The policy outputs individual joint velocity commands which are
converted by a safety layer (to prevent collisions / stay within performance
limits) into raw hardware commands. The robot starts from a forehand-pose as
the home position and is controlled by the learned policy as soon as a ball is
in play. As soon as the policy either makes contact with the ball returning it
or misses it, the robot is returned to the home position and continues the
rally with the next or returned ball as fresh inputs to the policy.

###  D.2 Ball Vision Model

The ball location is determined through a stereo pair of Ximea MQ013CG-ON
cameras positioned above and to the side of the table and running at 125Hz. A
recurrent 2D detector model detects the ball position in each camera
independently. This detector was trained with $\approx 2$ hours of ball video
data with an additional $\approx 15$ minutes of humans pretending to play
without a ball which is used for hard negative mining. During training,
horizontal flipping augmentation are applied to video sequences to balance
detection performance across both directions. The 2D detections from each
camera are fed to standard OpenCV triangulation to produce 3D coordinates,
which are in turn run filtered through a 3D tracker and interpolated to the
75Hz frequency that the policy does inference on. There is roughly $\approx
15$ms of lag between image capture and 3D coordinate availability.

##  Appendix E Model Architecture

We represent our policy using a three layer 1D fully convolutional gated
dilated CNN with 976 parameters. Details are given in Table 2. The observation
space is 2-dimensional (timesteps x [ball position, robot joint position])
which is an (8 x 11) matrix. The networks outputs a vector (8,) representing
joint velocities.

| Layer
---|---
Parameter | 1 | 2 | 3
Convolution dimension | 1D | 1D | 1D
Number of filters | 8 | 12 | 8
Stride | 1 | 1 | 1
Dilation | 1 | 2 | 4
Activation function | tanh | tanh | tanh
Padding | valid | valid | valid
Table 2: CNN model architecture.

##  Appendix F Training Hyperparameters

Table 3 presents the ES hyper-parameters used for both simulated and real
world training.

Parameter | Simulation | Real fine-tuning
---|---|---
Step size | 0.00375 | 0.00375
Perturbation standard deviation | 0.025 | 0.025
Number of perturbations | 200 | 5
Number of rollouts per perturbation | 15 | 3
Percentage to keep (top x% rollouts) | 30% | 60%
Maximum environment steps per rollout | 200 | 200
Use orthogonal perturbations | True | True
Use observation normalization | True | True
Table 3: ES hyperparameters.

##  Appendix G Simulation Details

Our simulation handles robot dynamics and contact dynamics (via PyBullet), and
we model the ball using Newtonian dynamics, incorporating air drag but not
spin. At the beginning of an episode, a ball throw is sampled according the
the parameterized distribution described in section 4.

One major difference between simulated and real world robotic systems is the
existence of sensor latency and noise in the latter but not the former. We
seek to minimize this difference by measuring the latency of the major system
components and modeling them in our simulation. These components include (a)
ABB and Festo action latency, (b) ball observation latency, (c) ABB and Festo
observation latency. The latency of each component is modeled by
$\mathcal{N}(\mu,\sigma^{2})$ where $\mu$ and $\sigma^{2}$ were measured
empirically. The details are given in subsection G.1. At the beginning of each
episode during training in simulation the latency of each component is sampled
and remains fixed throughout the episode.

###  G.1 Sensor Latency Model

Table 4 details the parameters used in the simulated sensor latency model
described above.

| Latencies (ms)
---|---
Component | $\mu$ | $\sigma^{2}$
Ball observation | 40 | 8.2
ABB observation | 29 | 8.2
Festo observation | 33 | 9
ABB action | 71 | 5.7
Festo action | 64.5 | 11.5
Table 4: Sensor latency model parameters per component.

###  G.2 Rewards in Simulation and the Real World

Table 5 describes the rewards used in simulation to train and fine-tune in the
real world. Rewards 1 - 3 are common between simulation and the real world.
The fault reward (4) is only available on a physical robot. Rewards 6 - 8 are
proxies for this in simulation. Rewards 9 - 11 are used in simulation to
encourage the policy to learn safe style (e.g. paddle not coming close to the
table) to reduce the likelihood of collisions in the real world upon transfer.

Reward |  Range |  Sim weight |  Real weight |  Sim weighted max score |  Real weighted max score
---|---|---|---|---|---
(1) State transition plus bonus for landing the ball close to a target in the center of the table |  [0, 5] |  1 |  1 |  5 |  5
(2) Bonus for clearing the net with a target height |  [0, 1] |  1 |  1 |  1 |  1
(3) Bonus for hitting the ball and landing it on the opponent side of the table |  [0, 1] |  0.1 |  0.1 |  0.1 |  0.1
(4) Actual fault reward in real |  {-2, 0} |  0.0 |  1.0 |  0 |  0
(5) Episodic jerk reward (proxy for faulting in real) |  [0, 1] |  0.3 |  0 |  0.3 |  0
(6) Episodic acceleration reward (proxy for faulting in real) |  [0, 1] |  0.3 |  0 |  0.3 |  0
(7) Episodic velocity reward (proxy for faulting in real) |  [0, 1] |  0.4 |  0 |  0.4 |  0
(8) Episodic joint angle reward (safety reward, aimed to prevent faulting in real ) |  [0, 1] |  1 |  0 |  1 |  0
(9) Safety reward, penalty for robot colliding with itself or table |  [-1 * timesteps, 0] |  1 |  0 |  0 |  0
(10) Paddle height reward |  [-1 * timesteps, 0] |  0.5 |  0 |  0 |  0
(11) Style reward (sim only) |  [-1 * timesteps, 0] |  1 |  0 |  0 |  0
Total |  |  |  |  8.1 |  6.1
Table 5: Rewards used in simulation to train and fine-tune in the real world.

##  Appendix H Evaluation Methodology

Each model was evaluated by (a) the model’s trainer and (b) two other players.
In each evaluation, 50 rallies (defined as a sequence of consecutive hits
ending when one player fails to return the ball) were played with the human
always starting and the rally length calculated as the number of paddle
touches for both the human and robot. While the human can be responsible for a
rally ending, almost all ended with the robot failing to return the ball or
returning it such that the human could not easily continue the rally. The
model trainer also evaluated intermediate checkpoints (see Figure 2) using the
same methodology to shed light on the training dynamics. To ensure fair
evaluation, all models were tested in random order and the identity of the
model was kept hidden from the evaluator (“blind eval”).

We introduced a bijective model for anonymization to make it easier for the
players to evaluate the models fairly. Each player evaluated all their ten
models and three models trained by two other randomly selected players in the
roster. The identity of the models is revealed once all the evaluations have
been completed. A successful evaluation must contain at least 50 valid rally
balls (see subsection H.2 for further instruction on determining a valid rally
ball). In addition to rally length, we have also collected statistics such as
whether the player or robot is at fault for ending the rally.

All players trained and evaluated the following ten models:

  1. 1.

i-S2R sim 1

  2. 2.

i-S2R fine-tuned 35%

  3. 3.

i-S2R sim 2

  4. 4.

i-S2R fine-tuned 65%

  5. 5.

i-S2R sim 3

  6. 6.

i-S2R fine-tuned 100%

  7. 7.

S2R fine-tuned 65%

  8. 8.

S2R fine-tuned 100%

  9. 9.

S2R-Oracle sim

  10. 10.

S2R-Oracle fine-tuned

Each player cross evaluated three models each from two other players:

  1. 1.

i-S2R fine-tuned 100%

  2. 2.

S2R-Oracle fine-tuned

  3. 3.

S2R fine-tuned 100%

Table 6 shows the trainer and evaluator combinations for cross evaluations.

Trainer | Evaluators
---|---
player 1 | player 4 | player 5
player 2 | player 1 | player 3
player 3 | player 1 | player 4
player 4 | player 2 | player 5
player 5 | player 2 | player 3
Table 6: Trainer and evaluator combinations.

###  H.1 Instructions for Human Players

We have provided the following instructions while gathering initial ball
trajectories and rallying with the robot.

Initial Ball Distribution: The player lobs the ball over the net from the left
hand quadrant of the opponent side to the right hand quadrant of the robot
side. All the players used the same standard table tennis racket.

Training and Evaluation: The player always starts a rally. The player lobs the
ball from the left hand quadrant of the opponent side to the right hand
quadrant of the robot side as naturally as possible. During the play, for all
the return balls from the robot, the player tries to return the ball to the
right hand quadrant of the robot. In all cases, we have instructed the player
to cooperate with robot as much as possible.

###  H.2 Details on Rally Score Evaluations

Table 7 contains the rally length evaluation and end-of-rally attribution
instructions for raters. For each evaluation, the cases marked as “Filter” are
removed. Then, the top 60 rallies are selected and sorted by rally length. For
reporting, we have selected the top 50 rallies from this set.

Description |  did-robot-end-rally |  Instruction
---|---|---
Human hit the first ball to the net. |  - |  Filter
Human hit the first ball over the table. |  - |  Filter
Human hit the first ball out of distribution, robot did not return. |  Yes |
Human hit the first valid ball and the robot did not react. |  |  Filter
Human returning a ball out of distribution, robot did not return. |  Yes |
Human returns a ball that bounces multiple times on the human side (robot has returned the ball). |  No |
Human returning a ball over the table. |  No |
Human returning a ball to the edge of the table, robot did not return. |  Yes |
Human hit a ball that graces the net which robot did not contact. |  - |  Filter
Human hit a ping pong type service and the robot did not return. |  - |  Filter
Robot returns a ball which graces the net, but the human cannot return. |  No |
Robot returns a ball which lands at the corner of the table and the human cannot return. |  No |
Rally ends due to the robot cannot contact the ball and/or the encoder diff is high (obvious behavior change from a previous rally, if applicable) |  Yes |
Robot is in ABB home pose, not the episode start state. You throw a dummy ball by hand so that the robot moves to episode start state. |  - |  Filter
Robot is in ABB home pose, not the episode start state. You throw with a paddle so that the robot moves to episode start state. |  - |  Filter
Table 7: Rally score evaluation and end-of-rally attribution.

##  Appendix I Player Skill Level

Table 8 contains further details on player rally length, calculated over all
10 models that a player evaluated (see Appendix H). This data was used to
group players into three skill levels; beginner (players 3 and 5),
intermediate (players 4 and 2), and advanced (player 1). Note that player 5
was the non author player.

We grouped players according to empirical skill (i.e. how they actually
played) as opposed to using self-reported skill because non-professional
players’ perception of their skill level may not be well calibrated across
players. In future it would be interesting to consider self-reported skill in
addition to empirical skill.

| Rally length
---|---
Player | Min | 25th | Mean (Std.) | 75th | Max
3 (beginner) | 2 | 3.0 | 7.0 (5.9) | 9.0 | 52
5 (beginner) | 2 | 3.8 | 10.4 (10.9) | 13.0 | 85
4 (intermediate) | 2 | 4.0 | 14.0 (16.1) | 18.0 | 117
2 (intermediate) | 2 | 4.0 | 16.8 (27.1) | 15.0 | 190
1 (advanced) | 2 | 5.0 | 19.4 (27.6) | 22.0 | 345

Table 8: Rally length statistics by player. Values were calculated over all 10
models that a player evaluated (see Appendix H), making 500 (10 x 50) rallies
in total

.

##  Appendix J Rally Length Normalization Details

Let $x$ be the rally length, $\mu_{x}$ the mean rally length, and $\sigma_{x}$
the standard deviation of the rally lengths, then rally length is normalized
as follows:

$\frac{x-\mu_{x}}{\sigma_{x}}$ (6)

##### Evaluations

Here $\mu_{x}$ and $\sigma_{x}$ are calculated over all 10 evaluations (see
Appendix H), making 500 (10 x 50) rallies in total. The values per player are
given in Table 8. This can be interpreted as normalizing for player skill and
is intended to make rally length comparable between players of different skill
levels (e.g. beginner, advanced). This approach was used in Figure 3 and
Figure 8.

##### Cross-Evaluations

Here $\mu_{x}$ and $\sigma_{x}$ are calculated per model (50 rallies in total)
and rallies are normalized with respect to the player who trained the model.
This is intended to make rally length comparable across models and players
(e.g. S2R+FT player 3, i-S2R player 1). This approach was used in Figure 7 and
Figure 12 to estimate the % difference in performance when a model is
evaluated by different players (cross-evaluations) who did not train the
model.

##  Appendix K Additional Results

Here we present additional results. Figure 8 contains additional presentations
of the data aggregated over all five players; (a) mean normalized rally
length, (b) distribution of normalized rally length, and (c) mean rally
length. Figure 9 presents mean rally length by player skill level. Figure 10
contains additional presentations of the data aggregated over 4/5 players with
the outlier (advanced player) excluded; (a) mean normalized rally length, (b)
distribution of normalized rally length, and (c) mean rally length.

Figure 11 and Figure 12 break out results per player. Note that player 5 was
the non-author player and was categorized as a beginner. Figure 11 shows the
mean and distribution of rally length for each player, ordered from top to
bottom by skill level, beginner to advanced. Figure 12 shows cross evaluation
data by player with the same ordering by player skill.

Figure 13, Figure 14, and Figure 15 present additional details on ball
distributions per player during training and evaluation.

Figure 16 and Figure 17 present additional data on the robot return rate per
player in the form of heatmaps. The color of each square represents the robot
return rate (darker $=$ higher return rate) and the number in each square
represents the percentage of balls. The grid operates on two scales, a large 3
x 3 grid, and within each cell, a smaller 3 x 3 grid. In each heatmap, the
large scale grid represents where the incoming ball bounced on the robot side
of the table.

In Figure 16 the small scale grid represents the position on the player side
where the ball originated. So, Figure 16 shows the conditional return rate
given the start position of the incoming ball and where the ball bounced on
the robot side of the table. For example, let’s look at the player 3 i-S2R
(top left grid). The middle large grid represents the middle of the robot side
of the table, and shows that 48.6% of the balls land here, out of which 12.6%
are coming from the opponent (human) hitting the ball from the far left of the
human side of the table, and 0.3% from the middle right of the human side of
the table.

In Figure 17 the small scale grid represents the position on the player side
where the ball landed (i.e. where the robot returned the ball to). So, Figure
17 shows the conditional return rate given the landing position of the
returned ball (i.e. where the robot hit it to) and where the incoming ball
bounced on the robot side of the table. As an example, if we look at the
player 3 i-S2R middle grid, it accounts for 53.4% of the balls, out of which
17.4% of the returns are to the middle of the table.

##### Statistical Significance

We note that the un-normalized mean rally lengths for i-S2R, S2R+FT and
S2R-Oracle+FT are not statistically significantly different, since the 95%
confidence intervals overlap (Appendix K, Figure 8 (c)). However, the
histogram of rally lengths for i-S2R and S2R+FT (Figure 3, right) shows that a
large fraction of the rallies for S2R+FT are shorter (i.e. less than 5), while
i-S2R achieves longer rallies more frequently. This suggests i-S2R yields
policies that are more fun to play with on average.

When rally length is normalized to account for differences in skill level
between players (Appendix K, Figure 8 (a)), the mean rally length for
S2R-Oracle+FT is statistically significantly higher than S2R+FT, although the
difference is small.

Finally, when the advanced player (outlier) is excluded (Figure 10), the mean
rally length (normalized and un-normalized) for i-S2R is statistically
significantly higher than S2R+FT and the difference is large. The mean rally
length for i-S2R and S2R-Oracle+FT are not statistically significantly
different.

(a) Mean normalized rally length.

(b) Normalized rally distribution.

(c) Mean rally length.

Figure 8: Aggregated results across all 5 players after learning. Vertical
lines are 95% confidence intervals (CIs).

Figure 9: Mean rally length by player skill level. Vertical lines are 95%
confidence intervals (CIs). Note: S2R-Oracle+FT is only getting 35% of i-S2R
and S2R+FT fine-tuning budget.

(a) Mean normalized rally length.

(b) Normalized rally distribution.

(c) Mean rally length.

Figure 10: Aggregated results across 4/5 players, excluding the outlier
(advanced) player, after learning. Vertical lines are 95% confidence intervals
(CIs).

Figure 11: Breakdown by player, order top to bottom from lowest to highest
overall rally mean. left: Mean rally length. Vertical lines are 95% CIs.
right: Rally length distribution per model.

Figure 12: Cross evaluations. Ordered by model trainer, top to bottom from
lowest to highest overall rally mean. left: Mean rally length. right: Mean
normalized rally length. Vertical lines are 95% CIs.

Figure 13: Evolution of ball distributions for each player projected to 2D
using t-SNE [77] (up to 500 random ball trajectory are used for $\Delta D_{1}$
and $\Delta D_{2}$, and $D_{i}=D_{0}+\sum_{j=1}^{i}\Delta D_{j}$.)

Figure 14: Evolution of ball distribution for each player and the overlapping
evaluation distributions for i-S2R(2D projected using t-SNE [77] and up to 500
random ball trajectory are sample from each round).

Figure 15: Evolution of ball distribution for each player and the overlapping
evaluation distributions for i-S2R left and S2R+FT on right (2D projected
using t-SNE [77] and up to 500 random ball trajectory are sample from each
round).

Figure 16: Heatmaps of the robot hit rate with respect to the (x, y) position
where the episode initiated from. left: i-S2R right: S2R+FT. The outermost
block represents the robot side. Each 3x3 blue block represents the human
(opponent) side of the table. Each block shows, if the human throw landed on
the robot side, where would the human throw initiated from. The block color
represents the robot hit rate.

Figure 17: Heatmaps of the robot hit rate with respect to the (x, y) position
where the episode ends. left: i-S2R right: S2R+FT. The outermost block
represents the robot side. Each 3x3 blue block represents the human (opponent)
side of the table. Each block shows, if the human throw landed on the robot
side, where would the robot hit the ball such that it lands on the opponent
side. The block color represents the robot hit rate.

###  K.1 Oracle Ball Distribution Ablation

To assess the contributions of the human behavior model, we hand-designed
large, medium, and narrow ball distributions as shown in Table 9. The medium
distribution is restricted to throwing balls towards the forehand hand side of
the robot with a restricted velocity range, and the narrow distribution is
modeled based on our ball thrower machine. Figure 18 compares these
distributions with S2R-Oracle. Evaluations were done using a single human
subject and are zero-shot from simulation with no fine-tuning. The large model
performed $\approx 40$% lower then S2R-Oracle. Further, we observe that lower
zero-shot scores substantially increase the fine-tuning time required to match
final performance, which is costly when a human is in the loop. We also
observe that zero-shot transfer with the medium distribution is comparable to
S2R-Oracle. This is because there is a high overlap between the two human
behavior models. This indicates that human behavior modeling via ball
distributions plays an important part in the ability for cooperative
interaction with a table tennis playing robot.

Parameter | Large | Medium | Narrow | S2R-Oracle
---|---|---|---|---
min z velocity ($ms^{-1}$) | -10 | -0.1 | -1.2 | -1.72
max z velocity ($ms^{-1}$) | 10 | 2 | 1.5 | 2.72
max x velocity ($|ms^{-1}|$) | 10 | 1.5 | 0.9 | 3.45
min y velocity ($|ms^{-1}|$) | 2 | 3.5 | 5.0 | 2.96
max y velocity ($|ms^{-1}|$) | 35 | 8.5 | 9.4 | 7.35
x start min ($m$) | -0.76 | -0.75 | 0.15 | -0.82
x start max ($m$) | 0.76 | 0.4 | 0.55 | 0.82
y start min ($m$) | 0.1 | 1.2 | 1.01 | 0.03
y start max ($m$) | 2.0 | 1.37 | 1.57 | 1.58
z start min ($m$) | -0.4 | 0.15 | 0.25 | 0.19
z start max ($m$) | 1.2 | 0.6 | 0.64 | 0.75
x land min ($m$) | -0.76 | -0.2 | 0.18 | -0.62
x land max ($m$) | 0.76 | 0.7 | 0.62 | 0.75
y land min ($m$) | -1.37 | -1.3 | -1.26 | -1.36
y land max ($m$) | -0.1 | -0.5 | -0.33 | -0.15
Table 9: The ball distribution parameters for each of the ablated
distributions. land here implies landing on the robot side. Figure 18: Zero-
shot transfer of rally length for different ball distributions as defined in
Table 9.