††thanks: These two authors contributed equally

# Bootstrapping Motor Skill Learning
with Motion Planning

Ben Abbatematteo∗, Eric Rosen∗, Stefanie Tellex, George Konidaris
Department of Computer Science
Brown University
Providence, RI
<EMAIL_ADDRESS>

###### Abstract

Learning a robot motor skill from scratch is impractically slow; so much so
that in practice, learning must be bootstrapped using a good skill policy
obtained from human demonstration. However, relying on human demonstration
necessarily degrades the autonomy of robots that must learn a wide variety of
skills over their operational lifetimes. We propose using kinematic motion
planning as a completely autonomous, sample efficient way to bootstrap motor
skill learning for object manipulation. We demonstrate the use of motion
planners to bootstrap motor skills in two complex object manipulation
scenarios with different policy representations: opening a drawer with a
dynamic movement primitive representation, and closing a microwave door with a
deep neural network policy. We also show how our method can bootstrap a motor
skill for the challenging dynamic task of learning to hit a ball off a tee,
where a kinematic plan based on treating the scene as static is insufficient
to solve the task, but sufficient to bootstrap a more dynamic policy. In all
three cases, our method is competitive with human-demonstrated initialization,
and significantly outperforms starting with a random policy. This approach
enables robots to to efficiently and autonomously learn motor policies for
dynamic tasks without human demonstration.

**footnotetext: These authors contributed equally to this work

##  I Introduction

(a)

(b)

Figure 1: A robot using our method to autonomously learn to close a microwave.
(a) The robot uses a motion planner to generate an initial attempt at closing
the microwave door using a kinematic model of the microwave. The resulting
plan is unable to fully close the microwave door because of the robot’s
limited reach. (b) After bootstrapping a motor skill with the trajectory from
(a), the robot learns a motor skill that gives the door a push, exploiting its
dynamics to fully close the microwave.

Robots require motor policies for interacting with objects in their
environment. For example, a robot butler may need a motor skill that enables
it to open a drawer to fetch utensils for a table, for setting each element of
the table, and for pouring wine. While it is safe to assume that a robot will
have an accurate kinematic model of its own body, it is unlikely to have a
dynamics model of every object it will ever encounter. This lack of knowledge
means that the robot will have to learn how to manipulate the world around it
[20].

Reinforcement learning (RL) provides a framework for robots to acquire motor
policies without explicitly modeling the unknown world, but model-free RL
methods like policy search [9] have high sample-complexity, and often fail to
learn a reasonable policy from random initialization. Supervised approaches
for policy learning like Learning From Demonstration (LfD) [2] can encode
human prior knowledge by imitating expert examples, but do not support
optimization in new environments. Combining RL with LfD is a powerful method
for reducing the sample complexity of policy search, and is often used in
practice [23, 33, 42, 6]. However, this approach typically requires a human
demonstrator for initialization, which fundamentally limits the autonomy, and
therefore utility, of a robot which may need to acquire a wide range of motor
skills over its operational lifetime. More recently, model-based control
techniques (including Model Predictive Control [16, 30] and LQR [23]) have
been proposed as exploration methods for policy search; these methods still
require human demonstrations or complete dynamic models of both the robot and
every object in the scene.

We propose the use of kinematic motion planning to initialize motor skill
policies. While previous work have leveraged sample-based motion planners for
learning motor skills [40, 15, 13], they only focus on either free-space
motions or do not learn a closed-loop controller. To our knowledge, this is
the first use of motion planning to provide initial demonstrations for
learning closed-loop motor skill policies by leveraging estimated object
kinematics. Motion planning algorithms generate collision-free behavior and
generalize to novel scenarios when the robot has a good kinematic model of
itself and the object it aims to manipulate, making it useful for tasks like
pick-and-place. We show that given a (potentially approximate, and readily
estimated) kinematic description of the environment and the robot, off-the-
shelf motion planning algorithms can generate feasible (potentially successful
but inefficient) initial trajectories (Figure 1(a)) to bootstrap an object-
manipulation policy that can subsequently be optimized using policy search
(Figure 1(b)). This framework enables the robot to automatically produce its
own demonstrations for effectively learning and refining object manipulation
policies. Our work enables the robot to realize the benefits of an initial
demonstration fully automatically using kinematic planning, requiring no human
involvement.

To evaluate our method, we used two different motor policy classes (Dynamic
Movement Primitives (DMPs) [12] and deep neural networks [24]). We chose these
two different motor policy classes because deep neural networks are extremely
expressive in what policies they can represent, but are extremely sample
inefficient compared to structured motor primitives like DMPs, and we aim to
evaluate how our method performs in both contexts. Using these motor policy
classes, we compared bootstrapping with motion planning against learning from
scratch in three simulated experiments, and against human demonstrations in
real hardware experiments. Human demonstrations provide a baseline for how
effective these motor policies can do when bootstrapped with high-quality
demonstrations, and learning from scratch provides a baseline for how
difficult the task is without any prior information about the task.In the
first two experiments—opening a drawer with a dynamic movement primitive
representation, and closing a microwave door with a deep neural network
policy—we show that motion planning using a kinematic model produces a
reasonable initial policy,although suboptimal compared to a supervised human
demonstration, that learning adapts to generate efficient, dynamic policies
that exploit the dynamics of the object being manipulated. We also show how
our method can bootstrap a motor skill for the challenging dynamic task of
learning to hit a ball off a tee, which involves precise and agile movement.
In that case, treating the objects in the scene as static and applying
kinematic motion planning succeeds in generating a policy that makes contact
with the ball, which is sufficient to bootstrap a more dynamic policy that
learns to hit the ball several feet. Our method is competitive with human-
demonstrated initialization, but requires no human demonstration. It serves as
a suitable starting point for learning, and significantly outperforms starting
with a random policy. This approach enables robots to efficiently and
autonomously learn motor policies for dynamic tasks without human
demonstration.

In summary, our contributions are:

  1. 1.

A fully-autonomous paradigm for policy search, in which an agent first uses
goal-directed kinematic planning to devise feasible solution trajectories for
itself and objects in a scene.

  2. 2.

A novel algorithm that autonomously generates a set of initial demonstrations
for object manipulation via kinematic planning in object and robot
configuration spaces.

  3. 3.

An empirical evaluation of this algorithm in which we employ model-free policy
optimization after bootstrapping, demonstrating that our method’s performance
is comparable to human expert demonstration and superior to random
initialization in hardware and simulation tasks: closing a microwave, opening
a drawer, and hitting a ball off a tee.

##  II BACKGROUND

Our goal is to efficiently and autonomously learn robot motor skill policies.
To do so, we develop an approach that uses kinematic motion planning to
generate initial trajectories, fits a policy to those trajectories using
behavioral cloning, and subsequently optimizes that policy via policy search.
We now briefly describe policy search, policy representations, learning from
demonstration, and motion planning.

###  II-A Policy Search

Policy search methods [9] are a family of model-free reinforcement learning
algorithms that search within a parametric class of policies to maximize
reward. Formally, given a Markov Decision Process
$\mathcal{M}=\langle\mathcal{S},\mathcal{A},\mathcal{R},\mathcal{T},\gamma\rangle$,
the objective of policy search is to maximize the expected return of the
policy $\pi_{\theta}$:

$\max_{\theta}\mathop{\large{\mathbb{E}}}_{\mathcal{M},\pi_{\theta}}\left[\sum_{t=0}^{T}\gamma^{t}r_{t}\right].$
(1)

These approaches can learn motor skills through interaction, and therefore do
not require an explicit environment model, and are typically agnostic to the
choice of policy class (though their success often depends on the policy class
having the right balance of expressiveness and compactness). However, their
model-free nature leads to high sample complexity, which often makes them
infeasible to apply directly to robot learning problems.

Policy representation describes the class of functions used as the mapping
from states to actions. Our approach is agnostic to policy representation; we
demonstrate our approach enables efficient learning using two different common
policy representations: Dynamic Movement Primitives and neural network
controllers.

Dynamic Movement Primitives [12, 32] are a description of a non-linear second-
order differential equation that exhibits attractor dynamics modulated by a
learnable forcing function. DMPs are a popular representation for motor
policies because they are parameter-efficient, can express both point and
limit cycle attractors, enable real-time computation, and exhibit temporal
invariance that does not effect the attractor landscape. We refer the reader
to the work of Ijspeert et al. [12] for a more formal introduction to DMPs. If
we have $n$ joints we wish to control, we can model control for each joint
independently with $n$ DMPs for each one. Therefore, the multiple joints of a
robot are only coupled through time, which makes this representation very
compact.

Neural network controllers have received significant attention in recent
years; they are able to learn hierarchical feature representations for
approximating functions (in our case, motor skills) operating on high-
dimensional input such as robot sensor data. They are more expressive than
restricted policy classes such as DMPs and can operate directly on high-
dimensional state spaces (e.g. images), yet they typically exhibit higher
sample complexity [24].

Learning from Demonstration methods [2, 34] broadly consist of two families of
approaches that either mimic (Behavioral Cloning) or generalize (Inverse
Reinforcement Learning) the exemplified behavior. Inverse reinforcement
learning methods seek to estimate a latent reward signal from a set of
demonstrations; we assume a given reward function, and omit a discussion of
inverse reinforcement learning methods here.

Behavioral cloning methods [3, 31, 11] attempt to directly learn a policy that
reproduces the demonstrated policies. Given a dataset of expert demonstrations
$D$, the objective of behavioral cloning is: $\max_{\theta}\sum_{(s,a)\in
D}\pi_{\theta}(a|s).$

These methods often result in policies with undesirable behavior in states not
observed during demonstrations, though this can addressed with interactive
learning [35, 29, 39]. In our approach, the existence of a reward function
enables the agent to learn robust behavior in states outside of the initial
training distribution. Moreover, our experiments demonstrate our approach’s
ability to extrapolate beyond suboptimal initial demonstrations.

Many approaches investigate the incorporation of human-provided demonstrations
into policy search to drastically reduce sample complexity via a reasonable
initial policy and/or the integration of demonstrations in the learning
objective [32, 18, 6, 33, 42, 41, 23].

###  II-B Motion Planning

The pose of an articulated rigid body can defined by the state of each of its
movable joints. The space of these poses is called the configuration space
$\mathcal{C}$ [27]. Motion planning is the problem of finding a path (sequence
of poses) through configuration space such that the articulated object is
moved to a desired goal configuration, without encountering a collision.

While there exist many different families of motion planning algorithms, such
as geometric, grid-based, and probabilistic road maps [22], they all operate
in a similar fashion: given a configuration space $\mathcal{C}$ and start and
goal joint configurations $q_{0},q^{*}\in\mathcal{C}$, return a valid path of
joint configurations $\\{q_{t}\\}_{t=0}^{T}$ between the start and end
configurations. We focus on sample-based motion planning approaches.

Probabilistic motion planners provide a principled approach for quickly
generating collision-free robot trajectories. However, online replanning is
expensive, and kinematic motion planners are only as effective as their
kinematic models are accurate: they generate trajectories directly, and thus
cannot be improved through subsequent interaction and learning. Furthermore,
kinematic planners produce trajectories that only account for kinematics, not
dynamics: they explicitly do not account for forces involved in motion, such
as friction, inertial forces, motor torques, etc, which are important for
effectively performing contact-rich, dexterous manipulation.

The process of computing the position and orientation $p\in SE(3)$ of a link
in a kinematic chain for a given joint variable setting (a point in
configuration space) is termed forward kinematics. Inversely, computing a
configuration to attain a specific end effector pose $p$ is termed inverse
kinematics. We denote the forward kinematics functions $p=f(q)$.

##  III Bootstrapping Skill Learning with Motion Planning

Our methodology is inspired by how humans generate reasonable first attempts
for accomplishing new motor tasks. When a human wants to learn a motor skill,
they do not start by flailing their arms around in a random fashion, nor do
they require another person to guide their arms through a demonstration.
Instead, they make a rough estimate of how they want an object to move and
then try to manipulate it to that goal. For example, before being able to
drive stick shift, a human must first learn how to manipulate a gear shifter
for their car. Just by looking at the gear shifter, humans can decide (1) what
they should grab (the shaft), (2) where they want the shaft to go (positioned
in a gear location), and (3) how the shaft should roughly move throughout the
action (at the intermediate gear positions). Similarly, a robot that has a
good kinematic model of itself, and a reasonable kinematic model of the object
it wishes to manipulate, should be able to form a motion plan to achieve the
effect it wishes to achieve.

That plan may be inadequate in several ways: its kinematic model may be
inaccurate, so the plan does not work; object dynamics (like the weight of a
door, or the friction of a joint) may matter, and these are not represented in
a kinematic model; and a feasible and collision-free kinematic trajectory may
not actually have the desired effect when executed on a robot interacting with
a real (and possibly novel) object. But such a solution is a good start; we
therefore propose to use it to bootstrap motor skill learning.

Our approach, outlined in Figure 2, leverages the (partial) knowledge the
robot has about its own body and the object it is manipulating to bootstrap
motor skills. Our method first assumes access to the configuration space of
the robot, denoted as $\mathcal{C}_{R}$, as well as its inverse kinematics
function $f^{-1}_{R}$. This assumption is aligned with the fact that the robot
often has an accurate description of its own links and joints and how they are
configured during deployment. However, the world is comprised of objects with
degrees of freedom that can only be inferred from sensor data. Therefore, our
approach only assumes access to estimated kinematics of the object to be
manipulated, in the form of configuration space $\mathcal{C}_{O}$ and forward
kinematics $f_{O}$. Recent work has shown that estimating these quantities for
novel objects from sensor data in real environments is feasible [1, 25],
though state-of-the-art estimates still include noise.

Finally, our approach assumes that the task goal can be defined in terms of
kinematic states of the robot and environment. Examples of such tasks include
pick-and-place, articulated object manipulation, and many instances of tool
use. (Note that this requirement fails to capture reward functions defined in
terms of force, for example exerting a specific amount of force in a target
location.) Such a goal, together with object and robot kinematic descriptions,
enables us to autonomously generate useful initial trajectories for policy
search.

Our approach is outlined in Algorithm 1, and can broken down into five main
steps: 1) collect initial trajectories(s) from a motion planner using
estimated object kinematics, 2) fit a policy with these initial trajectories,
3) gather rollouts to sample rewards for the current policy based on the
kinematic goal, and 4) update the policy parameters based on the actions and
rewards, 5) repeat steps 3-4.

Algorithm 1 Planning for Policy Bootstrapping

1:procedure PPB($C_{R},f^{-1}_{R},C_{O},f_{O},q_{O}^{*}$)

2: $D\leftarrow\varnothing$

3: for $0$ to $N$ do

4: $D\leftarrow$ InitialMPDemos($C_{R},f^{-1}_{R},C_{O},f_{O},q_{O}^{*}$)
$\cup$ $D$

5: $\theta\leftarrow$ FitPolicy($D_{0},...,D_{N}$)

6: for $0$ to $E$ do

7: $T_{0},..,T_{n}\leftarrow$ Rollout($\pi,\theta,q_{O}^{*}$)

8: $\theta\leftarrow$ UpdatePolicy($T_{1},..,T_{n},\theta$)

Algorithm 2 Initial Motion Plan Demos

1:procedure InitialMPDemos($C_{R},f^{-1}_{R},C_{O},f_{O},q_{O}^{*}$)

2: $T_{O}\leftarrow$ MotionPlanner($C_{O},q_{O}^{*}$)

3: $g\leftarrow$ EstimateGrasp($C_{O}$,$f_{O}$)

4: $eepath\leftarrow$ GraspPath($T_{O},C_{O},f_{O},g$)

5: $T_{R}\leftarrow$ MotionPlanner($C_{R}$, $eepath$, $f^{-1}_{R}$)

6: return $T_{R}$

###  III-A Initial Trajectories from Motion Planner

Figure 2: System overview illustrating our proposed framework for generating
demonstrations with a motion planner and subsequently performing policy
search. The dashed box contains the steps from Algorithm 2.

To fit our policy class, we choose to sample $N$ demonstrations using a motion
planner. Our approach for collecting initial demonstrations with a kinematic
motion planner is outlined in Algorithm 2, and proceeds as follows. First, we
use a motion planner to find a path through the object’s configuration space
$C_{O}$ that moves the object from its initial state to a goal state
$q_{O}^{*}$ using an off-the-shelf motion planner. This produces a joint
trajectory in object configuration space, $T_{O}$, which transforms the object
from its current joint configuration to the desired one.

We then estimate a grasp point on the object to designate the contact point
for the robot during manipulation. This can be done by either generating
candidate grasps using off-the-shelf grasping algorithms [10, 28] or choosing
a part semantically. This produces a local 6D pose, $g$, that represents where
the robot should grasp the object during manipulation.

We then use the grasp point $g$, object joint trajectory $T_{O}$, and the
object’s forward kinematics $f_{O}$ to generate the series of 6D Cartesian
poses that the grasp point $g$ will go through as the object proceeds through
$T_{O}$. This produces a series of 6D Cartesian poses, $eepath$, which the
robot end effector must go through, assuming a fixed grasp pose to the object.

Finally, we solve for a path in robot joint space that achieves the end
effector path in Cartesian space using off-the-shelf sample-based motion
planners, using the robot’s inverse kinematics $f^{-1}_{R}$ and the sequence
of end-effector poses $eepath$.

Note that in our experiments, motion plans were generated offline, rather than
recomputed online based on the object’s tracked state, but online motion
planning is a trivial extension.

(a) Microwave closing (MLP)

(b) Drawer opening (DMP)

(c) T-ball (DMP)

Figure 3: Simulation Results. a) Comparison of our method optimized with DAPG
against Natural Policy Gradient starting with a random policy in a microwave
closing task using Gaussian multi-layer perception policies. b) Comparison of
our method against PI2-CMA starting with a random policy in a drawer opening
task with DMP policies. c) Our method compared with PI2-CMA with a initially
random policy in t-ball with DMP policies. Results are shown as mean and
standard error of the normalized returns aggregated across 20 random seeds.

###  III-B Fitting a Policy to a Demonstration

After collecting initial demonstrations from the motion planner, $D$, we can
bootstrap our motor policy by initializing the parameters to the policy
$\theta$. We can initialize a parameterized motor policy using any behavioral
cloning technique; in practice, for DMPs, we use Locally Weighted Regression
[36], and for neural networks, we maximize the likelihood of the demonstration
actions under the policy.

###  III-C Policy Search with Kinematic Rewards

To improve the motor policies after bootstrapping, we can perform policy
search based on the given (kinematic) reward function. Specifically, we choose
a number of epochs $E$ to perform policy search for. For each epoch, we
perform an iteration of policy search by executing the policy and collecting
rewards based on the goal $q_{O}^{*}$. We define our reward functions using
estimated object states $q_{O}$ and goal states $q_{O}^{*}$, and add a small
action penalty.

##  IV EXPERIMENTS

The aim of our evaluation was to test the hypothesis that motion planning can
be used to initialize policies for learning from demonstration without human
input. We tested this hypothesis in simulation, against learning from scratch,
and on real hardware, against human demonstrations, on three tasks: microwave-
closing, drawer-opening, and t-ball. We note that we do not show asymptotic
performance because our emphasis is on learning on real hardware from a
practical number of iterations. All the components of the motion planning
problem - state sampler, goal sampler, distance metrics, etc. - are reused
between problems without modification.

###  IV-A Simulation Experiments

The aim of our evaluation was to test the hypothesis that motion planning can
be used to initialize policies for learning from demonstration without human
input. We tested this hypothesis in simulation against learning from scratch,
and on real hardware against human demonstrations, on three tasks: microwave-
closing, drawer-opening, and t-ball.

###  IV-B Simulation Experiments

We used PyBullet [8] to simulate an environment for our object manipulation
experiments. We used URDFs to instantiate a simulated 7DoF KUKA LBR iiwa7 arm
and the objects to be manipulated, which gave us ground-truth knowledge of the
robot and object kinematics. For all our simulated experiments, we compared
implementations of our method against starting with a random policy.

For all three tasks, the state was represented as $s_{t}=[q_{R},q_{O}]^{T}$
where $q_{R}$ denotes robot configuration and $q_{O}$ denotes object
configuration. The action space $A$ was commanded joint velocity for each of
the 7 motors. The reward at each timestep $r_{t}$ was given as:

$r_{t}=-c\ ||q_{O}^{*}-q_{O}||_{2}^{2}-a_{t}^{T}Ra_{t},$ (2)

where $q_{O}$ denotes the object state at time $t$, $q_{O}^{*}$ denotes
desired object state, and $a_{t}$ denotes the agent’s action. We set $c=60$
and R = $I\times 0.001$ for all experiments. As such, maximum reward is
achieved when the object is in the desired configuration, and the robot is at
rest.

Our first simulated task was to close a microwave door, which consisted of
three parts: a base, a door, and a handle. The pose of the handle was used for
the EstimateGrasp method in Algorithm 2. The robot was placed within reaching
distance of the handle when the microwave door was in an open position, but
was too far to reach the handle in its closed configuration. Thus, the agent
was forced to push the door with enough velocity to close it. We used Gaussian
policies represented as multi-layer perceptrons with two hidden layers of
sizes (32,32) in this experiment. The randomly initialized policy was
optimized with natural policy gradient [17]. Ten demonstrations were generated
by perturbing the start state and initial kinematic plan with Gaussian noise.
The behavior cloning was performed by maximizing likelihood over the
demonstration dataset for 10 epochs. Our pretrained policy was optimized using
Demo Augmented Policy Gradient [33], which essentially adds the behavior
cloning loss to the natural policy gradient loss, annealing it over time. This
ensures that the agent remains close to the demonstrations early in learning,
but is free to optimize reward exclusively as learning progresses. Results are
shown in Figure 3(a).

Figure 4: Hardware experiment comparing our initialization scheme with human
demonstration. Results are shown as mean and standard error, aggregated across
three random seeds.

The second simulated task was to open a drawer. This task required the agent
to grasp the drawer’s handle and pull the drawer open. Again, the pose of the
object’s handle was used for EstimateGrasp method in our algorithm. In this
experiment, we used DMP policies. The weights, goals, and speed parameters of
the policies were optimized using PI2-CMA [38]. We used 32 basis functions for
each of the DMPs. The pretrained policy was initialized using Locally Weighted
Regression (LWR) [36] with a single demonstration. The results of this
experiment are shown in Figure 3(b).

The third simulated task was to hit a ball off a tee. The ball started at rest
on top of the tee. The pose of the ball was used in the EstimateGrasp method.
The object state was defined as the object’s $y$ position relative to its
initial pose. This experiment again used DMP policies initialized with LWR and
optimized with PI2-CMA. The results of this experiment are visualized in
Figure 3(c).

The results of our simulated tasks can be found in Figure 3. Across all three
tasks, we observe that policies initialized with our method dramatically
outperform starting learning with a random policy. This confirms our
hypothesis that using motion planning to generate demonstrations significantly
speeds the acquisition of motor skills in challenging tasks like articulated
object manipulation and t-ball.

(a) Motion Plan Demonstration

(b) Bootstrapped from (a)

Figure 5: Real-world Ball Hitting Images comparing an autonomous motion plan
demonstration generated from our method vs. a bootstrapped motor skill
initialized with that demonstration for a real-world robot hitting a ball off
a tee as far as possible. Qualitatively, the bootstrapped motor skill
outperforms the initial demonstration by learning to take advantage of the
latent tasks dynamics. Videos can be found in our supplemental video. (a) A
demonstration provided by the motion planner, which moves linearly towards the
ball (b) A motor skill bootstrapped by the motion planner demonstration that
learns a agile swooping motion to take advantage of the balls dynamics.

###  IV-C Real-world Experiments

For all our real-world experiments, we used a 7DoF Jaco arm [4] to manipulate
objects (Figure 1). We used ROS and MoveIt![7] as the interface between the
motion planner (RRT* [19] in our experiments) and robot hardware. For all
real-world experiments, we compared implementations of our method against
bootstrapping with a human demonstration, which we supplied. We acknowledge
this potential bias in expert trajectories, and qualify our decision by only
training on human demonstrations that at least accomplished the task. To
collect human demonstrations, we had an expert human teleoperate the robot
with joystick control to perform the task. For all tasks, the state space,
action space, and reward were defined in the same way as in our simulated
results (Section IV-B). Both experiments used DMP policies initialized with
LWR [36] and optimized with PI2-CMA [38] with 10 basis functions for each of
the DMPs.

Our first real-world task was to close a microwave door, similar to the one
described in our simulated domain (Section IV-B). As in the simulated
microwave task, we used the pose of the handle for the EstimateGrasp method in
Algorithm 2, and also the robot was similarly placed such that it was forced
to push the door with enough velocity to close. We placed an AR tag on the
front-face of the microwave to track the microwave’s state using a Kinect2.
Results are shown in Figure 4. We observe that the human demonstration is
better than the one produced by the motion planner, which we credit to the
fact that the motion of the door was heavily influenced by the dynamics of the
revolute joint which the motion planner did not account for. Nonetheless, both
policies converge to a similar final performance, with our method converging
slightly faster. Note the importance of the policy search phase: the motion
planner alone is insufficient for performing the task efficiently.

Our second real-world task was to hit a ball off a tee as far as possible
(Figure 5). Similar to our simulated task, the ball started at rest on top of
the tee. The pose of the ball was used in the EstimateGrasp method. The object
state was defined as the object’s $y$ position relative to its initial pose.
We placed scotchlite-reflective tape on the surface of the ball and conducted
our experiments within an OptiTrack motion-capture cage to track the object
pose. We observe that when using a motion planner to hit the ball, it moves
the bat in a linear motion to make contact, therefore transferring only
horizontal motion to the ball. We qualitatively observe that during policy
search, the robot learns a dynamic policy that accounts for the dynamics of
the ball by applying force under the ball to “scoop” the ball upwards and
forwards.

##  V RELATED WORK

To our knowledge, our method is the first to use an object’s estimated
kinematics in conjunction with a known robot dynamics model to bootstrap motor
policy learning, and we discover and discuss important problems that are only
introduced when leveraging policy-learning algorithms, behavioral-cloning, and
motion planning algorithms to do so. In this section, we discuss relevant
approaches to motor skill learning.

Recently, Model-Predictive Control (MPC) has been used in the context of
imitation learning and reinforcement learning to address the high sample
complexity of policy search [16, 30]. These approaches require a priori object
dynamics, or human demonstrations to fit learned models; in constrast, our
approach requires only object kinematics, which are much more readily
estimated from visual data at runtime [1, 25]. As such, our approaches enables
the learning of manipulation skills to be more autonomous than existing MPC-
based methods. Tosun et al. [40] proposed a neural network model for
generating trajectories from images, using a motion planner during training to
enable the robot to generate a trajectory with a single forward pass at
runtime. While this approach uses a motion planner for behavior cloning, it
stops short of optimization to improve the resulting policy. In constrast, our
method uses object kinematics to produce initial trajectories, while Tosun et
al. [40] only use the robot’s kinematic model, which is insufficient when the
task is to manipulate an object to a specific joint configuration.

While classic robot motor learning papers [3] leverage the known kinodynamics
of the robot, they do not discuss kinematics of external objects or grasp
candidates to bootstrap motor policies for object manipulation. We emphasize
that we cannot form dynamic plans in the problem setting we are interested in:
objects with unknown a priori dynamics.

Kurenkov et al. [21] proposed training an initially random RL policy with an
ensemble of task-specific, hand-designed heuristics. This improves learning
but the initial policy is still random, yielding potentially unsafe behavior
on real hardware, and delaying convergence to a satisfying policy. By
contrast, we choose to initialize the policy with demonstrations from a
kinematic planner, ensuring feasibility, safety, and rapid learning. Moreover,
we argue that motion planning is the principled heuristic to use to accelerate
learning, as it is capable of expressing manually programmed heuristics like
reaching and pulling. Finally, our approach can use the existing estimated
object kinematics to provide a principled reward signal for model-free
reinforcement learning.

Recently, residual reinforcement learning approaches have been developed which
learn a policy superimposed on hand-designed or model-predictive controllers
[37, 14]. Our method is compatible with these approaches, where demonstrations
from the motion planner can be used as a base policy on top of which a
residual policy can be learned based on kinematic rewards. These methods
typically suffer from the same limitations as MPC-based methods mentioned
above.

Guided Policy Search (GPS) [23] uses LQR to guide policy search into high-
reward regions of the state-space. The models employed are fundamentally local
approximations, and thus would benefit greatly from a wealth of suboptimal
demonstrations from the outset (as made evident by Chebotar et al. [5]). GPS
is one of the state-of-the-art algorithms we expect to be used within our
framework as the policy search implementation (Section III-C). A critical
distinction between our work and GPS is the notion of planning trajectories in
object configuration spaces and reasoning about grasp candidates to achieve a
desired manipulation. This is done using information available apriori, and
thus is immediately capable of generating high-value policies, whereas GPS is
estimating dynamics models given observed data (obtained either from
demonstration or random initialization). In the absence of a human
demonstrator, our method would provide far more useful data at the outset of
learning than running a naively initialized linear-gaussian controller (as
evidenced by our comparisons to random initialization). The ideas proposed in
our paper are distinct from those put forth in GPS: we present a method for
obtaining demonstrations under certain conditions in the absence of a human.

Most similar to our line of work are those that use sample-based motion
planners for improved policy learning. Jurgenson and Tamar [15] harness the
power of reinforcement learning for neural motion planners by proposing an
augmentation of Deep Deterministic Policy Gradient (DDPG) [26] that uses the
known robot dynamics to leverage sampling methods like RRT* to reduce variance
in the actor update and provide off-policy exploratory behavior for the replay
buffer. However, Jurgenson and Tamar [15] are only able to address domains
where they can assume good estimates of the dynamics model, such as producing
free-space motions to avoid obstacles. Our setting, in contrast, focuses on
object manipulation, where dynamics are not readily available, but are
critical for learning good policies. Jiang et al. [13] address learning to
improve plans produced by a motion planner, but do not bootstrap closed-loop
policies. Motion planners aren’t expressive enough to leverage the dynamics in
object-manipulation tasks, especially in the presence of unknown dynamics, and
traditionally are unable to handle perceptual data like RGB images. Our
method, on the other hand, enables motion planning to bootstrap policies that
are more expressive than the original planner.

##  VI CONCLUSION

We have presented a method that uses kinematic motion planning to bootstrap
robot motor policies. By assuming access to a potentially noisy description of
the object kinematics, we are able to autonomously generate initial
demonstrations that perform as well as human demonstrations, but do not
require a human, resulting in a practical method for autonomous motor skill
learning.

Our methodology is agnostic to the motion planner, motor policy class, and
policy search algorithm, making it a widely applicable paradigm for learning
robot motor policies. We demonstrate the power of our methodology by
bootstrapping different policy classes with demonstrations from humans and a
motion planner, and learn motor policies for three dynamic manipulation tasks:
closing a microwave door, opening a drawer, and hitting a ball off a tee. Our
framework is the first to enable robots to autonomously bootstrap and improve
motor policies with model-free reinforcement learning using only a partially-
known kinematic model of the environment.

## Acknowledgement

This research was supported by NSF CAREER Award 1844960 to Konidaris, and by
the ONR under the PERISCOPE MURI Contract N00014-17-1-2699. Disclosure: George
Konidaris is the Chief Roboticist of Realtime Robotics, a robotics company
that produces a specialized motion planning processor.

## References

  * Abbatematteo et al. [2019] Ben Abbatematteo, Stefanie Tellex, and George Konidaris.  Learning to generalize kinematic models to novel objects.  _Proceedings of the 3rd Conference on Robot Learning_ , 2019.
  * Argall et al. [2009] Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning.  A survey of robot learning from demonstration.  _Robotics and autonomous systems_ , 57(5):469–483, 2009.
  * Atkeson and Schaal [1997] Christopher G Atkeson and Stefan Schaal.  Robot learning from demonstration.  In _ICML_ , volume 97, pages 12–20. Citeseer, 1997.
  * Campeau-Lecours et al. [2019] Alexandre Campeau-Lecours, Hugo Lamontagne, Simon Latour, Philippe Fauteux, Véronique Maheu, François Boucher, Charles Deguire, and Louis-Joseph Caron L’Ecuyer.  Kinova modular robot arms for service robotics applications.  In _Rapid Automation: Concepts, Methodologies, Tools, and Applications_ , pages 693–719. IGI Global, 2019.
  * Chebotar et al. [2017] Yevgen Chebotar, Mrinal Kalakrishnan, Ali Yahya, Adrian Li, Stefan Schaal, and Sergey Levine.  Path integral guided policy search.  In _2017 IEEE international conference on robotics and automation (ICRA)_ , pages 3381–3388. IEEE, 2017.
  * Cheng et al. [2018] Ching-An Cheng, Xinyan Yan, Nolan Wagener, and Byron Boots.  Fast Policy Learning through Imitation and Reinforcement.  _arXiv e-prints_ , art. arXiv:1805.10413, May 2018.
  * Chitta et al. [2012] Sachin Chitta, Ioan Sucan, and Steve Cousins.  Moveit![ros topics].  _IEEE Robotics & Automation Magazine_, 19(1):18–19, 2012.
  * Coumans et al. [2013] Erwin Coumans et al.  Bullet physics library.  _Open source: bulletphysics. org_ , 15(49):5, 2013\.
  * Deisenroth et al. [2013] Marc Peter Deisenroth, Gerhard Neumann, Jan Peters, et al.  A survey on policy search for robotics.  _Foundations and Trends® in Robotics_ , 2(1–2):1–142, 2013.
  * Gualtieri et al. [2016] Marcus Gualtieri, Andreas Ten Pas, Kate Saenko, and Robert Platt.  High precision grasp pose detection in dense clutter.  In _2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 598–605. IEEE, 2016.
  * Ho and Ermon [2016] Jonathan Ho and Stefano Ermon.  Generative adversarial imitation learning.  In _Advances in neural information processing systems_ , pages 4565–4573, 2016.
  * Ijspeert et al. [2003] Auke J Ijspeert, Jun Nakanishi, and Stefan Schaal.  Learning attractor landscapes for learning motor primitives.  In _Advances in neural information processing systems_ , pages 1547–1554, 2003.
  * Jiang et al. [2019] Yuqian Jiang, Fangkai Yang, Shiqi Zhang, and Peter Stone.  Task-motion planning with reinforcement learning for adaptable mobile service robots.  In _IROS_ , pages 7529–7534, 2019.
  * Johannink et al. [2019] Tobias Johannink, Shikhar Bahl, Ashvin Nair, Jianlan Luo, Avinash Kumar, Matthias Loskyll, Juan Aparicio Ojea, Eugen Solowjow, and Sergey Levine.  Residual reinforcement learning for robot control.  In _2019 International Conference on Robotics and Automation (ICRA)_ , pages 6023–6029. IEEE, 2019.
  * Jurgenson and Tamar [2019] Tom Jurgenson and Aviv Tamar.  Harnessing reinforcement learning for neural motion planning.  _arXiv preprint arXiv:1906.00214_ , 2019.
  * Kahn et al. [2017] Gregory Kahn, Tianhao Zhang, Sergey Levine, and Pieter Abbeel.  Plato: Policy learning using adaptive trajectory optimization.  In _2017 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 3342–3349. IEEE, 2017.
  * Kakade [2002] Sham M Kakade.  A natural policy gradient.  In _Advances in neural information processing systems_ , pages 1531–1538, 2002.
  * Kang et al. [2018] Bingyi Kang, Zequn Jie, and Jiashi Feng.  Policy optimization with demonstrations.  In _International Conference on Machine Learning_ , pages 2469–2478, 2018.
  * Karaman et al. [2011] Sertac Karaman, Matthew R Walter, Alejandro Perez, Emilio Frazzoli, and Seth Teller.  Anytime motion planning using the rrt.  In _2011 IEEE International Conference on Robotics and Automation_ , pages 1478–1483. IEEE, 2011.
  * Kroemer et al. [2019] Oliver Kroemer, Scott Niekum, and George Konidaris.  A review of robot learning for manipulation: Challenges, representations, and algorithms.  _arXiv preprint arXiv:1907.03146_ , 2019.
  * Kurenkov et al. [2019] Andrey Kurenkov, Ajay Mandlekar, Roberto Martin-Martin, Silvio Savarese, and Animesh Garg.  Ac-teach: A bayesian actor-critic method for policy learning with an ensemble of suboptimal teachers.  _arXiv preprint arXiv:1909.04121_ , 2019.
  * LaValle [2006] Steven M LaValle.  _Planning algorithms_.  Cambridge university press, 2006.
  * Levine and Koltun [2013] Sergey Levine and Vladlen Koltun.  Guided policy search.  In _International Conference on Machine Learning_ , pages 1–9, 2013\.
  * Levine et al. [2016] Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel.  End-to-end training of deep visuomotor policies.  _Journal of Machine Learning Research_ , 17(39):1–40, 2016.  URL http://jmlr.org/papers/v17/15-522.html.
  * Li et al. [2019] Xiaolong Li, He Wang, Li Yi, Leonidas Guibas, A Lynn Abbott, and Shuran Song.  Category-level articulated object pose estimation.  _arXiv preprint arXiv:1912.11913_ , 2019.
  * Lillicrap et al. [2015] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra.  Continuous control with deep reinforcement learning.  _arXiv preprint arXiv:1509.02971_ , 2015.
  * Lozano-Perez [1981] Tomas Lozano-Perez.  Automatic planning of manipulator transfer movements.  _IEEE Transactions on Systems, Man, and Cybernetics_ , 11(10):681–698, 1981.
  * Mahler et al. [2017] Jeffrey Mahler, Jacky Liang, Sherdil Niyaz, Michael Laskey, Richard Doan, Xinyu Liu, Juan Aparicio Ojea, and Ken Goldberg.  Dex-net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics.  _arXiv preprint arXiv:1703.09312_ , 2017.
  * Nair et al. [2017] Ashvin Nair, Dian Chen, Pulkit Agrawal, Phillip Isola, Pieter Abbeel, Jitendra Malik, and Sergey Levine.  Combining self-supervised learning and imitation for vision-based rope manipulation.  In _2017 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 2146–2153. IEEE, 2017.
  * Pan et al. [2017] Yunpeng Pan, Ching-An Cheng, Kamil Saigol, Keuntaek Lee, Xinyan Yan, Evangelos Theodorou, and Byron Boots.  Agile autonomous driving using end-to-end deep imitation learning.  _arXiv preprint arXiv:1709.07174_ , 2017.
  * Pastor et al. [2009] Peter Pastor, Heiko Hoffmann, Tamim Asfour, and Stefan Schaal.  Learning and generalization of motor skills by learning from demonstration.  In _2009 IEEE International Conference on Robotics and Automation_ , pages 763–768. IEEE, 2009.
  * Peters and Schaal [2008] Jan Peters and Stefan Schaal.  Natural actor-critic.  _Neurocomputing_ , 71(7-9):1180–1190, 2008.
  * Rajeswaran et al. [2017] Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine.  Learning complex dexterous manipulation with deep reinforcement learning and demonstrations.  _arXiv preprint arXiv:1709.10087_ , 2017.
  * Ravichandar et al. [2019] Harish Ravichandar, Athanasios S Polydoros, Sonia Chernova, and Aude Billard.  Recent advances in robot learning from demonstration.  _Annual Review of Control, Robotics, and Autonomous Systems_ , 2019\.
  * Ross et al. [2011] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell.  A reduction of imitation learning and structured prediction to no-regret online learning.  In _Proceedings of the fourteenth international conference on artificial intelligence and statistics_ , pages 627–635, 2011.
  * Schaal and Atkeson [1998] Stefan Schaal and Christopher G Atkeson.  Constructive incremental learning from only local information.  _Neural computation_ , 10(8):2047–2084, 1998\.
  * Silver et al. [2018] Tom Silver, Kelsey Allen, Josh Tenenbaum, and Leslie Kaelbling.  Residual policy learning.  _arXiv preprint arXiv:1812.06298_ , 2018.
  * Stulp and Sigaud [2012] Freek Stulp and Olivier Sigaud.  Path integral policy improvement with covariance matrix adaptation.  _arXiv preprint arXiv:1206.4621_ , 2012.
  * Sun et al. [2017] Wen Sun, Arun Venkatraman, Geoffrey J Gordon, Byron Boots, and J Andrew Bagnell.  Deeply aggrevated: Differentiable imitation learning for sequential prediction.  In _Proceedings of the 34th International Conference on Machine Learning-Volume 70_ , pages 3309–3318. JMLR. org, 2017.
  * Tosun et al. [2019] Tarik Tosun, Eric Mitchell, Ben Eisner, Jinwook Huh, Bhoram Lee, Daewon Lee, Volkan Isler, H Sebastian Seung, and Daniel Lee.  Pixels to plans: Learning non-prehensile manipulation by imitating a planner.  _arXiv preprint arXiv:1904.03260_ , 2019.
  * Vecerik et al. [2017] Mel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nicolas Heess, Thomas Rothörl, Thomas Lampe, and Martin Riedmiller.  Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards.  _arXiv preprint arXiv:1707.08817_ , 2017.
  * Zhu et al. [2018] Yuke Zhu, Ziyu Wang, Josh Merel, Andrei Rusu, Tom Erez, Serkan Cabi, Saran Tunyasuvunakool, JÃ¡nos KramÃ¡r, Raia Hadsell, Nando de Freitas, and Nicolas Heess.  Reinforcement and imitation learning for diverse visuomotor skills.  In _Proceedings of Robotics: Science and Systems_ , Pittsburgh, Pennsylvania, June 2018.  doi: 10.15607/RSS.2018.XIV.009.


* Okamoto et al. (2015) Okamoto TJ, Antolin P, De Pontieu B, Uitenbroek H, Van Doorsselaere T, Yokoyama T (2015) Resonant Absorption of Transverse Oscillations and Associated Heating in a Solar Prominence. I. Observational Aspects. ApJ809:71, DOI 10.1088/0004-637X/809/1/71, 1506.08965
  * Oran et al. (2017) Oran R, Landi E, van der Holst B, Sokolov IV, Gombosi TI (2017) Alfvén Wave Turbulence as a Coronal Heating Mechanism: Simultaneously Predicting the Heating Rate and the Wave-induced Emission Line Broadening. ApJ845(2):98, DOI 10.3847/1538-4357/aa7fec
  * Pagano and De Moortel (2017) Pagano P, De Moortel I (2017) Contribution of mode-coupling and phase-mixing of Alfvén waves to coronal heating. A&A601:A107, DOI 10.1051/0004-6361/201630059
  * Pagano and De Moortel (2019) Pagano P, De Moortel I (2019) Contribution of observed multi frequency spectrum of Alfvén waves to coronal heating. A&A623:A37, DOI 10.1051/0004-6361/201834158, 1901.02310
  * Pagano et al. (2018) Pagano P, Pascoe DJ, De Moortel I (2018) Contribution of phase-mixing of Alfvén waves to coronal heating in multi-harmonic loop oscillations. A&A616:A125, DOI 10.1051/0004-6361/201732251, 1804.10562
  * Pagano et al. (2019) Pagano P, Van Damme HJ, Antolin P, De Moortel I (2019) MHD simulations of the in situ generation of kink and sausage waves in the solar corona by collision of dense plasma clumps. A&A626:A53, DOI 10.1051/0004-6361/201935539, 1905.03749
  * Pant et al. (2019) Pant V, Magyar N, Van Doorsselaere T, Morton RJ (2019) Investigating “Dark” Energy in the Solar Corona Using Forward Modeling of MHD Waves. ApJ881(2):95, DOI 10.3847/1538-4357/ab2da3, 1906.10941
  * Parker (1986) Parker EN (1986) Equilibrium of magnetic fields with arbitrary interweaving of the lines of force. II. Discontinuities in the field. Geophysical and Astrophysical Fluid Dynamics 35(4):277–301, DOI 10.1080/03091928608245896
  * Parker (1991) Parker EN (1991) The phase mixing of Alfven waves, coordinated modes, and coronal heating. ApJ376:355–363, DOI 10.1086/170285
  * Parnell and De Moortel (2012) Parnell CE, De Moortel I (2012) A contemporary view of coronal heating. Royal Society of London Philosophical Transactions Series A 370:3217–3240, DOI 10.1098/rsta.2012.0113, 1206.6097
  * Pascoe and De Moortel (2014) Pascoe DJ, De Moortel I (2014) Standing Kink Modes in Three-dimensional Coronal Loops. ApJ784:101, DOI 10.1088/0004-637X/784/2/101
  * Pascoe et al. (2009) Pascoe DJ, de Moortel I, McLaughlin JA (2009) Impulsively generated oscillations in a 3D coronal loop. A&A505:319–327, DOI 10.1051/0004-6361/200912270
  * Pascoe et al. (2010) Pascoe DJ, Wright AN, De Moortel I (2010) Coupled Alfvén and Kink Oscillations in Coronal Loops. ApJ711:990–996, DOI 10.1088/0004-637X/711/2/990
  * Pascoe et al. (2011) Pascoe DJ, Wright AN, De Moortel I (2011) Propagating Coupled Alfvén and Kink Oscillations in an Arbitrary Inhomogeneous Corona. ApJ731:73, DOI 10.1088/0004-637X/731/1/73
  * Pascoe et al. (2017) Pascoe DJ, Russell AJB, Anfinogentov SA, Simões PJA, Goddard CR, Nakariakov VM, Fletcher L (2017) Seismology of contracting and expanding coronal loops using damping of kink oscillations by mode coupling. A&A607:A8, DOI 10.1051/0004-6361/201730915
  * Pascoe et al. (2020) Pascoe DJ, Smyrli A, Van Doorsselaere T (2020) Tracking and Seismological Analysis of Multiple Coronal Loops in an Active Region. ApJ898(2):126, DOI 10.3847/1538-4357/aba0a6
  * Perez and Chandran (2013) Perez JC, Chandran BDG (2013) Direct Numerical Simulations of Reflection-driven, Reduced Magnetohydrodynamic Turbulence from the Sun to the Alfvén Critical Point. ApJ776:124, DOI 10.1088/0004-637X/776/2/124
  * Piddington (1956) Piddington JH (1956) Solar atmospheric heating by hydromagnetic waves. MNRAS116:314, DOI 10.1093/mnras/116.3.314
  * Poedts et al. (1990) Poedts S, Goossens M, Kerner W (1990) On the Efficiency of Coronal Loop Heating by Resonant Absorption. ApJ360:279, DOI 10.1086/169118
  * Qu et al. (2017) Qu ZN, Jiang LQ, Chen SL (2017) Observations of a Fast-mode Magnetosonic Wave Propagating along a Curving Coronal Loop on 2011 November 11. ApJ851:41, DOI 10.3847/1538-4357/aa9beb
  * Reale (2014) Reale F (2014) Coronal Loops: Observations and Modeling of Confined Plasma. Living Reviews in Solar Physics 11(1):4, DOI 10.12942/lrsp-2014-4
  * Reep et al. (2013) Reep JW, Bradshaw SJ, Klimchuk JA (2013) Diagnosing the Time Dependence of Active Region Core Heating from the Emission Measure. II. Nanoflare Trains. ApJ764(2):193, DOI 10.1088/0004-637X/764/2/193, 1303.4466
  * Rempel (2017) Rempel M (2017) Extension of the MURaM Radiative MHD Code for Coronal Simulations. ApJ834(1):10, DOI 10.3847/1538-4357/834/1/10, 1609.09818
  * Rial et al. (2010) Rial S, Arregui I, Terradas J, Oliver R, Ballester JL (2010) Three-dimensional Propagation of Magnetohydrodynamic Waves in Solar Coronal Arcades. ApJ713(1):651–661, DOI 10.1088/0004-637X/713/1/651, 1002.0469
  * Rosner et al. (1978) Rosner R, Tucker WH, Vaiana GS (1978) Dynamics of the quiescent solar corona. ApJ220:643–645, DOI 10.1086/155949
  * Ruderman et al. (1997a) Ruderman MS, Berghmans D, Goossens M, Poedts S (1997a) Direct excitation of resonant torsional Alfven waves by footpoint motions. A&A320:305–318
  * Ruderman et al. (1997b) Ruderman MS, Goossens M, Ballester JL, Oliver R (1997b) Resonant Alfven waves in coronal arcades driven by footpoint motions. A&A328:361–370
  * Russell et al. (2015) Russell AJB, Simões PJA, Fletcher L (2015) A unified view of coronal loop contraction and oscillation in flares. A&A581:A8, DOI 10.1051/0004-6361/201525746, 1506.07716
  * Sabri et al. (2018) Sabri S, Vasheghani Farahani S, Ebadi H, Hosseinpour M, Fazel Z (2018) Alfvén wave dynamics at the neighbourhood of a 2.5D magnetic null-point. MNRAS479(4):4991–4997, DOI 10.1093/mnras/sty1407
  * Schrijver et al. (1999) Schrijver CJ, Title AM, Berger TE, Fletcher L, Hurlburt NE, Nightingale RW, Shine RA, Tarbell TD, Wolfson J, Golub L, Bookbinder JA, Deluca EE, McMullen RA, Warren HP, Kankelborg CC, Handy BN, de Pontieu B (1999) A new view of the solar outer atmosphere by the Transition Region and Coronal Explorer. Sol. Phys.187:261–302
  * Sekse et al. (2013) Sekse DH, Rouppe van der Voort L, De Pontieu B, Scullion E (2013) Interplay of Three Kinds of Motion in the Disk Counterpart of Type II Spicules: Upflow, Transversal, and Torsional Motions. ApJ769:44, DOI 10.1088/0004-637X/769/1/44, 1304.2304
  * Selwa and Ofman (2009) Selwa M, Ofman L (2009) 3-D numerical simulations of coronal loops oscillations. Annales Geophysicae 27(10):3899–3908, DOI 10.5194/angeo-27-3899-2009
  * Shen et al. (2018) Shen Y, Liu Y, Song T, Tian Z (2018) A Quasi-periodic Fast-propagating Magnetosonic Wave Associated with the Eruption of a Magnetic Flux Rope. ApJ853:1, DOI 10.3847/1538-4357/aaa3ff, 1712.09045
  * Shen and Liu (2012) Shen YD, Liu Y (2012) Observational Study of the Quasi-periodic Fast-propagating Magnetosonic Waves and the Associated Flare on 2011 May 30. ApJ753:53, DOI 10.1088/0004-637X/753/1/53, 1204.6649
  * Shen et al. (2013) Shen YD, Liu Y, Su JT, Li H, Zhang XF, Tian ZJ, Zhao RJ, Elmhamdi A (2013) Observations of a Quasi-periodic, Fast-Propagating Magnetosonic Wave in Multiple Wavelengths and Its Interaction with Other Magnetic Structures. Sol. Phys.288:585–602, DOI 10.1007/s11207-013-0395-4, 1307.6099
  * Shen et al. (2017) Shen YD, Liu Y, Song T, Tian Z (2017) A quasi-periodic fast-propagating magnetosonic wave associated with the eruption of a magnetic flux rope. ArXiv e-prints 1712.09045
  * Shoda et al. (2019) Shoda M, Suzuki TK, Asgari-Targhi M, Yokoyama T (2019) Three-dimensional Simulation of the Fast Solar Wind Driven by Compressible Magnetohydrodynamic Turbulence. ApJ880(1):L2, DOI 10.3847/2041-8213/ab2b45, 1905.11685
  * Singh et al. (2006) Singh J, Sakurai T, Ichimoto K, Muneer S, Raveendran AV (2006) Spectroscopic Studies of Solar Corona VIII. Temperature and Non-Thermal Variations in Steady Coronal Structures. Sol. Phys.236:245–262, DOI 10.1007/s11207-006-0104-7
  * Soler and Terradas (2015) Soler R, Terradas J (2015) Magnetohydrodynamic Kink Waves in Nonuniform Solar Flux Tubes: Phase Mixing and Energy Cascade to Small Scales. ApJ803(1):43, DOI 10.1088/0004-637X/803/1/43, 1502.03949
  * Soler et al. (2017) Soler R, Terradas J, Oliver R, Ballester JL (2017) Propagation of Torsional Alfvén Waves from the Photosphere to the Corona: Reflection, Transmission, and Heating in Expanding Flux Tubes. ApJ840(1):20, DOI 10.3847/1538-4357/aa6d7f
  * Soler et al. (2019) Soler R, Terradas J, Oliver R, Ballester JL (2019) Energy Transport and Heating by Torsional Alfvén Waves Propagating from the Photosphere to the Corona in the Quiet Sun. ApJ871(1):3, DOI 10.3847/1538-4357/aaf64c, 1812.01323
  * Spicer (1977) Spicer DS (1977) An unstable arch model of a solar flare. Sol. Phys.53(2):305–345, DOI 10.1007/BF00160276
  * Srivastava et al. (2017) Srivastava AK, Shetye J, Murawski K, Doyle JG, Stangalini M, Scullion E, Ray T, Wójcik DP, Dwivedi BN (2017) High-frequency torsional Alfvén waves as an energy source for coronal heating. Scientific Reports 7:43147, DOI 10.1038/srep43147
  * Srivastava et al. (2019) Srivastava AK, Mishra SK, Jelínek P, Samanta T, Tian H, Pant V, Kayshap P, Banerjee D, Doyle JG, Dwivedi BN (2019) On the Observations of Rapid Forced Reconnection in the Solar Corona. ApJ887(2):137, DOI 10.3847/1538-4357/ab4a0c, 1901.07971
  * Stangalini et al. (2018) Stangalini M, Jafarzadeh S, Ermolli I, Erdélyi R, Jess DB, Keys PH, Giorgi F, Murabito M, Berrilli F, Del Moro D (2018) Propagating Spectropolarimetric Disturbances in a Large Sunspot. ApJ869(2):110, DOI 10.3847/1538-4357/aaec7b, 1810.12595
  * Stepanov et al. (2012) Stepanov AV, Zaitsev VV, Nakariakov VM (2012) Coronal Seismology: Waves and Oscillations in Stellar Coronae Flare Plasma. DOI 10.1002/9783527645985
  * Suzuki (2008) Suzuki TK (2008) Coronal heating and wind acceleration by nonlinear Alfvén waves ? global simulations with gravity, radiation, and conduction. Nonlinear Processes in Geophysics 15(2):295–304, URL https://hal.archives-ouvertes.fr/hal-00302984
  * Suzuki (2012) Suzuki TK (2012) Solar wind and its evolution. Earth, Planets, and Space 64(2):201–206, DOI 10.5047/eps.2011.04.012, 1104.3660
  * Suzuki and Inutsuka (2005) Suzuki TK, Inutsuka S (2005) Making the Corona and the Fast Solar Wind: A Self-consistent Simulation for the Low-Frequency Alfvén Waves from the Photosphere to 0.3 AU. ApJ632:L49–L52, DOI 10.1086/497536, arXiv:astro-ph/0506639
  * Suzuki and Inutsuka (2006) Suzuki TK, Inutsuka SI (2006) Solar winds driven by nonlinear low-frequency Alfvén waves from the photosphere: Parametric study for fast/slow winds and disappearance of solar winds. Journal of Geophysical Research (Space Physics) 111(A6):A06101, DOI 10.1029/2005JA011502, astro-ph/0511006
  * Terradas and Arregui (2018) Terradas J, Arregui I (2018) Temporal and Spatial Scales for Coronal Heating by Alfvén Wave Dissipation in Transverse Loop Oscillations. Research Notes of the American Astronomical Society 2(4):196, DOI 10.3847/2515-5172/aaeb26
  * Terradas et al. (2007) Terradas J, Andries J, Goossens M (2007) Coronal loop oscillations: energy considerations and initial value problem. A&A469:1135–1143, DOI 10.1051/0004-6361:20077404
  * Terradas et al. (2008a) Terradas J, Andries J, Goossens M, Arregui I, Oliver R, Ballester JL (2008a) Nonlinear Instability of Kink Oscillations due to Shear Motions. ApJ687:L115–L118, DOI 10.1086/593203, 0809.3664
  * Terradas et al. (2008b) Terradas J, Arregui I, Oliver R, Ballester JL, Andries J, Goossens M (2008b) Resonant Absorption in Complicated Plasma Configurations: Applications to Multistranded Coronal Loop Oscillations. ApJ679:1611–1620, DOI 10.1086/586733, arXiv:0802.0591
  * Terradas et al. (2018) Terradas J, Magyar N, Van Doorsselaere T (2018) Effect of Magnetic Twist on Nonlinear Transverse Kink Oscillations of Line-tied Magnetic Flux Tubes. ApJ853:35, DOI 10.3847/1538-4357/aa9d0f
  * Thurgood and McLaughlin (2013) Thurgood JO, McLaughlin JA (2013) Nonlinear Alfvén wave dynamics at a 2D magnetic null point: ponderomotive force. A&A555:A86, DOI 10.1051/0004-6361/201321338, 1305.7073
  * Tian et al. (2012) Tian H, McIntosh SW, Wang T, Ofman L, De Pontieu B, Innes DE, Peter H (2012) Persistent Doppler Shift Oscillations Observed with Hinode/EIS in the Solar Corona: Spectroscopic Signatures of Alfvénic Waves and Recurring Upflows. ApJ759:144, DOI 10.1088/0004-637X/759/2/144, 1209.5286
  * Tian et al. (2014) Tian H, DeLuca EE, Cranmer SR, De Pontieu B, Peter H, Martínez-Sykora J, Golub L, McKillop S, Reeves KK, Miralles MP, McCauley P, Saar S, Testa P, Weber M, Murphy N, Lemen J, Title A, Boerner P, Hurlburt N, Tarbell TD, Wuelser JP, Kleint L, Kankelborg C, Jaeggli S, Carlsson M, Hansteen V, McIntosh SW (2014) Prevalence of small-scale jets from the networks of the solar transition region and chromosphere. Science 346(6207):1255711, DOI 10.1126/science.1255711, 1410.6143
  * Tomczyk and McIntosh (2009) Tomczyk S, McIntosh SW (2009) Time-Distance Seismology of the Solar Corona with CoMP. ApJ697:1384–1391, DOI 10.1088/0004-637X/697/2/1384, 0903.2002
  * Tomczyk et al. (2007) Tomczyk S, McIntosh SW, Keil SL, Judge PG, Schad T, Seeley DH, Edmondson J (2007) Alfven Waves in the Solar Corona. Science 317(5842):1192–1196, DOI 10.1126/science.1143304, URL http://www.sciencemag.org/cgi/content/abstract/317/5842/1192, %****␣review_heating_arxiv.bbl␣Line␣1350␣****http://www.sciencemag.org/cgi/reprint/317/5842/1192.pdf
  * Tomczyk et al. (2008) Tomczyk S, Card GL, Darnell T, Elmore DF, Lull R, Nelson PG, Streander KV, Burkepile J, Casini R, Judge PG (2008) An Instrument to Measure Coronal Emission Line Polarization. Sol. Phys.247(2):411–428, DOI 10.1007/s11207-007-9103-6
  * Tu et al. (2005) Tu CY, Zhou C, Marsch E, Xia LD, Zhao L, Wang JX, Wilhelm K (2005) Solar Wind Origin in Coronal Funnels. Science 308(5721):519–523, DOI 10.1126/science.1109447
  * Uchida and Kaburaki (1974) Uchida Y, Kaburaki O (1974) Excess Heating of Corona and Chromosphere Above Magnetic Regions by Non-Linear Alfvén Waves. Sol. Phys.35:451–466, DOI 10.1007/BF00151968
  * Usmanov et al. (2018) Usmanov AV, Matthaeus WH, Goldstein ML, Chhiber R (2018) The Steady Global Corona and Solar Wind: A Three-dimensional MHD Simulation with Turbulence Transport and Heating. ApJ865(1):25, DOI 10.3847/1538-4357/aad687
  * van Ballegooijen and Asgari-Targhi (2018) van Ballegooijen AA, Asgari-Targhi M (2018) The Heating of Coronal Loops in Solar Active Regions. In: Journal of Physics Conference Series, Journal of Physics Conference Series, vol 1100, p 012027, DOI 10.1088/1742-6596/1100/1/012027
  * van Ballegooijen et al. (2011) van Ballegooijen AA, Asgari-Targhi M, Cranmer SR, DeLuca EE (2011) Heating of the Solar Chromosphere and Corona by Alfvén Wave Turbulence. ApJ736:3, DOI 10.1088/0004-637X/736/1/3, 1105.0402
  * van Ballegooijen et al. (2017) van Ballegooijen AA, Asgari-Targhi M, Voss A (2017) The Heating of Solar Coronal Loops by Alfvén Wave Turbulence. ApJ849:46, DOI 10.3847/1538-4357/aa9118, 1710.05074
  * Van Damme et al. (2020) Van Damme HJ, De Moortel I, Pagano P, Johnston CD (2020) Chromospheric evaporation and phase mixing of Alfvén waves in coronal loops. A&A635:A174, DOI 10.1051/0004-6361/201937266
  * van der Holst et al. (2014) van der Holst B, Sokolov IV, Meng X, Jin M, Manchester WB IV, Tóth G, Gombosi TI (2014) Alfvén Wave Solar Model (AWSoM): Coronal Heating. ApJ782:81, DOI 10.1088/0004-637X/782/2/81, 1311.4093
  * Van Doorsselaere et al. (2008) Van Doorsselaere T, Nakariakov VM, Verwichte E (2008) Detection of waves in the solar corona: kink or Alfvén? ApJ676:L73–L75
  * Van Doorsselaere et al. (2014) Van Doorsselaere T, Gijsen SE, Andries J, Verth G (2014) Energy Propagation by Transverse Waves in Multiple Flux Tube Systems Using Filling Factors. ApJ795:18, DOI 10.1088/0004-637X/795/1/18
  * Vasheghani Farahani and Hejazi (2017) Vasheghani Farahani S, Hejazi SM (2017) Coronal Jet Collimation by Nonlinear Induced Flows. ApJ844(2):148, DOI 10.3847/1538-4357/aa7da5
  * Vasheghani Farahani et al. (2010) Vasheghani Farahani S, Nakariakov VM, Van Doorsselaere T (2010) Long-wavelength torsional modes of solar coronal plasma structures. A&A517:A29+, DOI 10.1051/0004-6361/201014502
  * Vasheghani Farahani et al. (2011) Vasheghani Farahani S, Nakariakov VM, van Doorsselaere T, Verwichte E (2011) Nonlinear long-wavelength torsional Alfvén waves. A&A526:A80+, DOI 10.1051/0004-6361/201016063
  * Vasheghani Farahani et al. (2012) Vasheghani Farahani S, Nakariakov VM, Verwichte E, Van Doorsselaere T (2012) Nonlinear evolution of torsional Alfvén waves. A&A544:A127, DOI 10.1051/0004-6361/201219569
  * Verth et al. (2010) Verth G, Terradas J, Goossens M (2010) Observational Evidence of Resonantly Damped Propagating Kink Waves in the Solar Corona. ApJ718(2):L102–L105, DOI 10.1088/2041-8205/718/2/L102, 1007.1080
  * Walsh and Ireland (2003) Walsh RW, Ireland J (2003) The heating of the solar corona. A&A Rev.12(1):1–41, DOI 10.1007/s00159-003-0021-9
  * Wang (2011) Wang T (2011) Standing Slow-Mode Waves in Hot Coronal Loops: Observations, Modeling, and Coronal Seismology. Space Sci. Rev.158:397–419, DOI 10.1007/s11214-010-9716-1, 1011.2483
  * Wang et al. (2012) Wang T, Ofman L, Davila JM, Su Y (2012) Growing Transverse Oscillations of a Multistranded Loop Observed by SDO/AIA. ApJ751(2):L27, DOI 10.1088/2041-8205/751/2/L27, 1204.1376
  * Wang (2016) Wang TJ (2016) Waves in Solar Coronal Loops. Washington DC American Geophysical Union Geophysical Monograph Series 216:395–418, DOI 10.1002/9781119055006.ch23, 1803.11329
  * Wang et al. (2021) Wang TJ, Ofman L, Yuan D, Reale F, Kolotkov DY, Srivastava AK (2021) Slow-Mode Magnetoacoustic Waves in Coronal Loops. Space Science Reviews DOI submitted
  * Warnecke et al. (2017) Warnecke J, Chen F, Bingert S, Peter H (2017) Current systems of coronal loops in 3D MHD simulations. A&A607:A53, DOI 10.1051/0004-6361/201630095, 1611.06170
  * Withbroe and Noyes (1977) Withbroe GL, Noyes RW (1977) Mass and energy flow in the solar chromosphere and corona. ARA&A15:363–387, DOI 10.1146/annurev.aa.15.090177.002051
  * Wójcik et al. (2017) Wójcik D, Murawski K, Musielak ZE, Konkol P, Mignone A (2017) Numerical Simulations of Torsional Alfvén Waves in Axisymmetric Solar Magnetic Flux Tubes. Sol. Phys.292(2):31, DOI 10.1007/s11207-017-1058-7, 1701.04594
  * Wójcik et al. (2020) Wójcik D, Kuźma B, Murawski K, Musielak ZE (2020) Wave heating of the solar atmosphere without shocks. A&A635:A28, DOI 10.1051/0004-6361/201936938
  * Woolsey and Cranmer (2015) Woolsey LN, Cranmer SR (2015) Time-dependent Turbulent Heating of Open Flux Tubes in the Chromosphere, Corona, and Solar Wind. ApJ811(2):136, DOI 10.1088/0004-637X/811/2/136, 1509.00377
  * Yang et al. (2020a) Yang Z, Bethge C, Tian H, Tomczyk S, Morton R, Del Zanna G, McIntosh SW, Karak BB, Gibson S, Samanta T, He J, Chen Y, Wang L (2020a) Global maps of the magnetic field in the solar corona. Science 369(6504):694–697, DOI 10.1126/science.abb4462, 2008.03136
  * Yang et al. (2020b) Yang Z, Tian H, Tomczyk S, Morton R, Bai X, Samanta T, Chen Y (2020b) Mapping the magnetic field in the solar corona through magnetoseismology. Sci China Tech Sci 63(11):2357–2368, DOI 10.1007/s11431-020-1706-9, 2008.03146
  * Yuan et al. (2013) Yuan D, Shen YD, Liu Y, Nakariakov VM, Tan B, Huang J (2013) Distinct propagating fast wave trains associated with flaring energy releases. A&A554:A144, DOI 10.1051/0004-6361/201321435
  * Zavershinskii et al. (2019) Zavershinskii DI, Kolotkov DY, Nakariakov VM, Molevich NE, Ryashchikov DS (2019) Formation of quasi-periodic slow magnetoacoustic wave trains by the heating/cooling misbalance. Physics of Plasmas 26(8):082113, DOI 10.1063/1.5115224, 1907.08168
  * Zhang et al. (2015) Zhang Y, Zhang J, Wang J, Nakariakov VM (2015) Coexisting fast and slow propagating waves of the extreme-UV intensity in solar coronal plasma structures. A&A581:A78, DOI 10.1051/0004-6361/201525621
  * Zhugzhda (1996) Zhugzhda YD (1996) Force-free thin flux tubes: Basic equations and stability. Physics of Plasmas 3:10–21, DOI 10.1063/1.871836
  * Zimovets and Nakariakov (2015) Zimovets IV, Nakariakov VM (2015) Excitation of kink oscillations of coronal loops: statistical study. A&A577:A4, DOI 10.1051/0004-6361/201424960