## I. Introduction

The discovery of new planets has long been a goal of astronomical research.
The field of exoplanets has currently found 4,940 examples of such
bodies111https://exoplanetarchive.ipac.caltech.edu/. However, while each new
exoplanet discovery is undoubtedly a great achievement, in situ studies of
planets around other stars are currently outside our abilities.

However, if an exoplanet were to be found closer, such as in orbit around the
Sun, it would be a much needed boon for the field. A literature search for
planets in our solar system has revealed between eight or nine currently-known
examples222We exclude dwarf planets from this count, because they are small
and no-one likes them., with one tantalising discovery within the sun’s
habitable zone [Yahweh 4004 BC, Munroe 2014]. However, in recent years, these
discoveries have tailed off; in fact, no new planets have been found to orbit
the sun within the last hundred years, despite numerous searches (For a
review, see [O’Malley & Rosset 2020]). Using the methods of popular geography,
this means that we are overdue for such an event.

The discovery of new exoplanets has recently been noted as following a process
known as Mamajek’s Law: the number of known exoplanets doubles every 39
months. A recent paper [Lund 2021] (hereafter L21) notes a minor concern with
this theory when taken to its logical extreme - i.e., the collapse of our
galaxy into an all-consuming black hole.

From this theory, we pose a possible upside. As the number of new exoplanets
increases, following the distribution laid out by Mamajek’s law, there will
come a point when an exoplanet must be discovered inside our solar system.
Naturally, this will be of great interest to all scientists in the field of
exoplanets (hereafter ‘exoplaneteers’).

In this paper, we will investigate when this discovery is most likely to
occur, consider possible additional pitfalls arising from L21, and also pose
several novel methods for directing these discoveries.

A note on terminology: In this paper we use the terms ‘planet’ and ‘exoplanet’
interchangeably. Since we do not live in an anthropocentric universe, we would
not dream of thinking ourselves so self-important as to assume planets are
different from exoplanets. After all, all planets are exoplanets - from a
certain point of view [Kenobi 4 ABY].

## II. Methods

A key premise from L21 was that the number of discovered exoplanets doubled
every 39 months. The author went on to deduce thus; the total mass of these
exoplanets would mean that the mass of the Milky Way, contained within a 10
kpc radius, would exceed that of the corresponding Schwarzchild mass in 2252.

However, we can make approximations to deduce when it is likely that one of
these exoplanets would be discovered within the solar system. We postulate a
simple model of the Milky Way as a 2-dimensional plane, emitting exoplanets in
all directions333https://en.wikipedia.org/wiki/Spherical_cow, and then find
the area of the Solar System compared to the total area of the 2D plane
contained within its orbit:

$P=\frac{A_{SS}}{A_{MW}}=\frac{R_{SS}^{2}}{R_{MW}^{2}}$ (1)

where P is the possibility of any given new exoplanet within the Milky Way
(MW) being within the boundaries of the Solar System (SS), A is the area, and
R is the radius.

We take the outer edge of the solar system as 4.5 $\times$ 109 km (Google
2021), this being the orbit of Sol-i (known colloquially as ’Neptune’), the
furthest of the Milky Way’s exoplanet population; this marks the rough
boundary that can be reached by our current technology. For the radius of the
Milky Way, we follow the example of L21, noting that all exoplanets are found
within 10 kpc. From these figures, we find that any given exoplanet has a 1 in
4.7 $\times$ 1015 chance of being found within our solar system.

We thus assume that once that many exoplanets are discovered, it becomes
likely that one of those will be within the solar system. We note that this is
just an approximation; more advanced statistical modelling, perhaps utilising
the curious phenomenon of most exoplanets being found to congregate around the
solar system [Huff 1954], is beyond the scope of this paper.

L21 gives the following equation for Mamajek’s law of the number of
exoplanets:

$N_{planets}=10.25*1.24^{\left(year-1990\right)}$ (2)

Reversing this equation tells us that we will reach 4.7 $\times$ 1015
exoplanets - and thus an exoplanet in our solar system will be discovered - on
Friday 9th December 2146, just after 22:36 UTC444Once again fitting the
tradition that all big discoveries must disrupt an astronomer’s weekend..

## III. Discussion

While an exoplanet being discovered inside our solar system will have great
potential for scientific discovery, it would be beneficial if this planet were
to be found close to Earth for logistical reasons. Fortunately, the history of
exoplanet research provides ample anecdotal evidence to aid in this fact.555We
realise that anecdotal evidence alone is not sufficient to claim a discovery.
Therefore, we took a lot of anecdotes, and averaged them out. This is called
science and is perfectly acceptable..

An extensive literature search reveals that the vast majority of papers
regarding exoplanet discoveries are about detections of new exoplanets -
conversely, there are relatively few papers about non-detections of
exoplanets. Ipso facto, the vast majority of exoplanet searches lead to a
discovery of exoplanets. We thus postulate that exoplanets will most likely be
found wherever exoplaneteers are observing (especially considering that there
are currently no known exoplanets in locations that have not been observed).
This targeted approach to exoplanet discovery builds on previous proposals in
the field [Lund 2016].

From this, we arrive at a logical conclusion: The more exoplaneteers we have
observing a location, the more likely it is that an exoplanet will be
discovered there.

The ramifications of this theory are multifarious. For instance, we can use
simultaneous observations on multiple different telescopes to ‘triangulate’ a
discovery on a particular location. The best locations to target should be
trivial to travel to, and ideally not upset the orbital motions of other Solar
System bodies; this could include Lagrange points L3 and L4 in the Earth-Sun
system666Other Lagrange points are available. An exoplanet discovered at L1,
for example, would be easily detected through the transit method and may
provide many benefits to observational astronomy. However, we stress that L2
in particular should not be selected, as various telescopes that reside there
- such as JWST - are not equipped for unexpected lithobraking..

However, we note that there are complications that may result from this theory
of exoplaneteer idealism - that is, that exoplaneteers may inadvertently
discover an exoplanet outside of the targeted locations. For example, it would
be disadvantageous if an exoplanet were to be discovered inside an
exoplaneteer’s living room. We thus recommend that all exoplaneteers should be
blindfolded when not directly observing Lagrange points.

We should also be aware of the nature of the exoplanet that may be discovered.
Planets appearing in our solar system have been posited before, with less than
stellar results [Pedler & Davis 1966], and other authors from the same
institute have noted possible events near to 2146 that may disrupt ongoing
studies [Subotsky 1966]. We also note a worry that every new exoplanet could
also host its own exoplaneteers, which may accelerate Mamajek’s law.

## IV. Conclusions

In this paper, we calculate the statistical likelihood that an exoplanet will
be discovered in our solar system by the end of 2146. This will give us
roughly 102 years to research it before the black hole posited by L21 is due
to consume all matter in the Milky Way.

We deduce that many exoplanet discoveries are due to the observational
abilities of exoplaneteers; i.e. new exoplanets are only found because they’re
looking for them777“It’s probably because it’s quantum” [Pratchett 1989]..

Ergo, we pose a novel solution to Lund’s black hole; when an exoplanet is
discovered in our solar system, we send all exoplaneteers to it, where they
will have ample opportunity to observe it in situ. With this, they will
hopefully no longer feel the need to find new exoplanets, thus finally
breaking Mamajek’s law.

Figure 1: Possible discovery method for new exoplanets.

## Acknowledgements

The authors would like to thank Michael B. Lund for inspiring this paper. We
thank the anonymous referee for not looking at this paper. GNU Terry
Pratchett. This research has made use of NASA’s Astrophysics Data System.

## References

  * [Huff 1954] Huff, D., ‘How to Lie with Statistics’, New York: Norton, 1954.
  * [Kenobi 4 ABY] Kenobi, O. W., ‘Return of the Jedi’, 4 ABY
  * [Lund 2016] Lund, M. B., arXiv:1603.09496, 2016
  * [Lund 2021] Lund, M. B., arXiv:2103.17079, 2021
  * [Munroe 2014] Munroe, R., xkcd, 1371, https://xkcd.com/1371/, 2014
  * [O’Malley & Rosset 2020] O’Malley, M., Rosset, B., ‘The Search for Planet X’, 2020
  * [Pedler & Davis 1966] Pedler, K., Davis, G., ‘The Tenth Planet’, Doctor Who, 1966
  * [Pratchett 1989] Pratchett, T., ‘Pyramids’, Gollancz, 1989
  * [Subotsky 1966] Subotsky, M., ‘Daleks’ Invasion Earth 2150A.D.’, Doctor Who, 1966
  * [Yahweh 4004 BC] Yahweh, ‘Genesis’, 1, 1, 4004 BC


# SAR: Generalization of Physiological Agility and Dexterity via Synergistic
Action Representation

Cameron Berg  Vittorio Caggiano∗ Vikash Kumar∗
Meta AI
∗Equal advising

###### Abstract

Learning effective continuous control policies in high-dimensional systems,
including musculoskeletal agents, remains a significant challenge. Over the
course of biological evolution, organisms have developed robust mechanisms for
overcoming this complexity to learn highly sophisticated strategies for motor
control. What accounts for this robust behavioral flexibility? Modular control
via muscle synergies, i.e. coordinated muscle co-contractions, is considered
to be one putative mechanism that enables organisms to learn muscle control in
a simplified and generalizable action space. Drawing inspiration from this
evolved motor control strategy, we use physiologically accurate human hand and
leg models as a testbed for determining the extent to which a Synergistic
Action Representation (SAR) acquired from simpler tasks facilitates learning
and generalization on more complex tasks. We find in both cases that SAR-
exploiting policies significantly outperform end-to-end reinforcement
learning. Policies trained with SAR were able to achieve robust locomotion on
a diverse set of terrains (e.g., stairs, hills) with state-of-the-art sample
efficiency (4M total steps), while baseline approaches failed to learn any
meaningful behaviors. Additionally, policies trained with SAR on in-hand
100-object manipulation task significantly outperformed ($\mathbb{>70\%}$
success) baseline approaches ($\mathbb{<20\%}$ success). Both SAR-exploiting
policies were also found to generalize zero-shot to out-of-domain
environmental conditions, while policies that did not adopt SAR failed to
generalize. Finally, using a simulated robotic hand and humanoid agent, we
establish the generality of SAR on broader high-dimensional control problems,
solving tasks with greatly improved sample efficiency. To the best of our
knowledge, this investigation is the first of its kind to present an end-to-
end pipeline for discovering synergies and using this representation to learn
high-dimensional continuous control across a wide diversity of tasks.

Project website: https://sites.google.com/view/sar-rl

##  I Introduction

An important precondition for generalist embodied agents is their ability to
exhibit diverse behaviors in response to dynamic environmental conditions,
akin to humans and animals. From the practical perspective of constructing
such agents, supporting behavioral diversity requires the integration of
mechanisms that can capably handle complex high-dimensional action spaces. For
example, humans are capable of an extremely large set of feasible motor
repertoires, but this behavioral diversity necessitates continuous control of
approximately 600 muscles in the human body [21].

Learning control policies for high-dimensional continuous action spaces are
significantly more challenging [42] than learning in reduced action spaces
that utilize idealized direct control, such as the Atari environments commonly
used for reinforcement learning (RL) experiments [28, 4]. However, even the
joint-based and end-effector continuous control dynamics commonly utilized in
robotics exhibit significant simplifications as compared to the structural and
functional physiology of actuated biological organisms [46]. As a case study,
consider the human hand, where joints are not controlled directly, but rather,
through the proxy of pull-only muscle forces [50]. A further complication
involved in physiological control is multiarticularity, meaning that there
exist many-to-one and one-to-many relationships between muscles and joints
(e.g., the flexor digitorum profundus muscle controls over 3 joints for each
digit of the hand) [27]. Critically, the human physiology is also
overactuated, meaning that there are more muscle forces than degrees of
freedom (e.g., approximately 39 muscles collectively control 23 joints in the
human hand) [49].

Despite the significant challenges inherent in learning musculoskeletal
control, humans and animals are able to rapidly learn motor repertoires that
are orders of magnitude more complex than the current state of the art in
embodied agents[2]. A question naturally emerges from this fact: what
strategies have humans and animals evolved that enable them to learn
musculoskeletal control so robustly—and can these strategies be utilized to
train policies that exhibit similarly robust behaviors in high-dimensional
continuous action spaces?

Modular control is one mechanism known to be fundamental to human and animal
behavioral flexibility, where motor output is computed as a product of a
reduced number of modular spatiotemporal muscle coactivation patterns rather
than as separate activations for each individual muscle [10, 8]. These muscle
co-contraction modules are referred to as muscle synergies. There exists
strong evidence not only that motor behavior is produced by coordinated muscle
activity that can be represented with high fidelity in terms of a smaller
number of synergistic activation patterns [40], but also that the spinal cord
explicitly encodes muscle activity at the level of synergies rather than as
individual muscle activations [7]. This built-in action representation might
be thought of as an embodied form of transfer learning, as it enables
efficient adaptations of new motor repertoires that leverage similar—but non-
identical—muscle activation patterns. It is suggested that these motor modules
are present even in newborns [19], that they can be finetuned and recombined
to yield more complex behaviors over the lifetime [14], and that they
facilitate skill transfer, wherein similar activation representations can be
reused for similar tasks (e.g., writing with a pen vs. writing with a pencil
would not require learning two unique motor repertoires from scratch) [6].

In this work, we explore whether the explicit discovery and utilization of
muscle synergies improves the performance and generalization power of high-
dimensional continuous control policies, with a specific focus on learning
physiological control. In this investigation, we train physiologically
accurate musculoskeletal models of the human hand (MyoHand) and legs (MyoLegs)
[11] on a diverse task suite. To the best of our knowledge, this is the first
study that demonstrates synergy-exploiting control policies that are able to
successfully learn manipulation and locomotion tasks unsolved by baseline
learning approaches. In particular, our core contributions are as follows:

  * •

We present a simple and generalizable paradigm to recover a synergistic action
representation from behaviors learned during play.

  * •

We demonstrate that training MyoLegs with a Synergistic Action Representation
on a task suite of diverse terrains and conditions yields robust locomotion
with state-of-the-art (SOTA) sample efficiency (3-5M total steps), while
baseline approaches were found to learn no meaningful behaviors within this
training budget.

  * •

We further demonstrate that training MyoHand with a Synergistic Action
Representation on a complex multi-object randomized manipulation task enables
strong performance ($>70\%$ success) that is otherwise unsolved by baselines
($<20\%$ success).

  * •

We show that both SAR-exploiting policies generalize zero-shot to out-of-
domain environmental conditions, while policies without access to this
representation fail to generalize.

  * •

Finally, we demonstrate that Synergistic Action Representations extend beyond
physiological control using robotic manipulation and full-body humanoid
locomotion tasks, notably achieving SOTA performance with an order of
magnitude improved sample efficiency on the latter.

##  II Related Works

We first provide a background on what is currently understood about muscle
synergies in humans and summarize previous work in leveraging synergy
representations for learning musculoskeletal control.

###  II-A Motor neuroscience of muscle synergies

Over five decades, the motor neuroscience literature on muscle synergies [22]
has elucidated their mathematical representation [57, 40], control dynamics
[45, 25], lifespan finetuning and recombination [19], and cross-species
similarity [60, 19]. Muscle synergies can be defined as motor activation
modules that impose a dimensionality-reducing spatiotemporal structure to
muscle activations by linearly combining (temporal) simplified activation
patterns with a set of (spatial) variable activation weights for discrete
muscles [19]. It is widely believed that muscle synergies are an evolved
mechanism for vertebrate motor control with a direct neurophysiological
implementation [51, 24, 32]. Accordingly, muscle synergies have been found to
be robust to perturbations within a specific motor repertoire; for instance,
as few as four muscle synergies in the lower limbs appear to control
locomotion in humans and are invariant to movement speed, grade, direction,
and skill [19, 45].

###  II-B Synergy representations for motor control

The neuroscience literature on muscle synergies is reminiscent of dynamic
movement primitives (DMPs), a popular concept in robotic control where
actuation dynamics are decomposed into a smaller set of stable dynamical
systems [48]. While these two concepts are similar in their simplification of
complex continuous control dynamics into more tractable modular
representations, the methods typically used to parameterize and stabilize DMPs
can often preclude their generalization power to novel tasks [44]. By
contrast, one core property of muscle synergies is their putative role as an
action representation template for learning related tasks [14].

Figure 1: MyoSuite musculoskeletal models: MyoLegs (16 DoF, 80 muscles) and
MyoHand (23 DoF, 39 muscles) [11]. Muscles are indicated in red.

A small number of investigations have previously assessed the feasibility of
utilizing muscle synergy representations for effectively learning to control
musculoskeletal or robotic systems (for a review, see [33]), albeit in fairly
constrained task settings. To the best of our knowledge, all previous studies
have explored whether synergy representations enable reaching behaviors in a
musculoskeletal arm with 7 degrees of freedom (DoF) [30, 1, 18, 44, 12] or
8-DoF thumb/index model [43]. These investigations have found that synergies
can enable faster learning and partial generalization. One study also found
that synergies improved task performance [18]. While these previous works
demonstrate the general promise of leveraging muscle synergies for
musculoskeletal control, the nature of these task demonstrations is arguably
too low-dimensional both structurally (i.e., a 7-DoF arm) and functionally
(i.e., reaching to a small number of fixed points) to yield strong conclusions
about the full scope of using synergies for control.

In order to more comprehensively understand whether muscle synergies can
enhance the training performance and generalization power of musculoskeletal
control policies, we believe that synergy action representations must be
tested in significantly more complex musculoskeletal task settings. For this
reason, we investigate significantly beyond the current scope of the muscle
synergy control literature by exploring (a) how synergistic action
representations can be computationally extracted from simulated
musculoskeletal models, and (b) how these representations can be leveraged to
learn control policies capable of (i) achieving robust locomotion across a
diverse set of terrains and (ii) manipulating thousands of complex objects.
Finally, we investigate (c) the extent to which this biologically-inspired
representation can more generally enable high-dimensional continuous control
in task formulations beyond human physiology.

##  III Physiological Dexterity

In this work, we seek to use learned action representations to acquire
generalizable behavior across diverse conditions on physiologically accurate
musculoskeletal models. We begin by introducing both models, MyoLegs and
MyoHand and proceed to then formulate the problem of learning the associated
behaviors for each model before detailing our experimental methods in Sec IV.

###  III-A Physiologically accurate musculoskeletal models

We investigated various options of musculoskeletal models [35, 31] before
ultimately selecting the MyoLegs and MyoHand models in favor of their
computational efficiency, physiological accuracy, and support for contact
dynamics, which is essential for the stated goal of yielding generalizable
manipulation and locomotion behaviors [11]. Both models are implemented in the
MuJoCo physics simulator [56] and also exhibit important mechanistic features
that differentiate biological actuation from the joint-based control approach
traditionally utilized in robotics, including overactuation, multi-
articulation, and pull-only actuation with third order muscle dynamics [50].
MyoLegs is a physiologically realistic model of adult left and right legs (see
Fig. 1), comprising 20 degrees of freedom and and 80 muscle-tendon units [11].
MyoHand is a physiologically realistic model of an adult right hand-wrist (see
Fig. 1), comprising 23 joints, 29 bones, and 39 muscle-tendon units [11].
Relevant to the current investigation, previous work has also demonstrated
that MyoHand can learn dexterous manipulation behaviors with single objects
(e.g., twirling a pen to a desired orientation) [11]. Here, we aim to go
beyond isolated task-specific single-object solutions, by using the MyoHand
model to learn in-hand manipulation behaviors that generalize from prior
experiences to handle new, complex tasks.

Before outlining our methods, we next turn to formalizing our problem
formulation.

###  III-B Problem formulation

We pose the problem of learning with MyoLegs and MyoHand as Markov Decision
Processes (MDP) [52], which can be defined as $\mathcal{M}\
=(\mathcal{S},\mathcal{A},\mathcal{T},\mathcal{R},\mathcal{\rho},\mathcal{\gamma})$,
where $\mathcal{S}\subseteq\mathbb{R}^{n}$ and
$\mathcal{A}\subseteq\mathbb{R}^{m}$ each represent continuous state and
action spaces. $\mathcal{T}$ represents the unknown distribution that
describes state transition dynamics such that
$s^{\prime}\sim\mathcal{T}(\cdot\mid s,a)$. The reward function is denoted as
$\mathcal{R}:\mathcal{S}\rightarrow\left[0,R_{\max}\right]$, $\gamma\in[0,1)$
indicates the reward discount factor, and $\rho$ denotes the initial
distribution of states. The goal in performing RL is to search for policy
parameters $\theta$ that map from states to a probability distribution over
actions $\pi_{\theta}:\mathcal{S}\rightarrow P(\mathcal{A})$ in order to
maximize long-run discounted returns [52]. We optimize our policy using Soft
Actor-Critic (SAC) [23], an off-policy RL algorithm used for continuous
control which optimizes for both long-run discounted returns,
$\sum_{t}\gamma^{t}R_{t}$, and policy entropy, $H(\pi)$, such that
$\pi_{\theta}^{*}(a\mid
s)=\operatorname{argmax}_{\theta}[J(\pi,\mathcal{M})]$, where

$J=\max_{\theta}\mathbb{E}\left[\sum_{t}\gamma^{t}\left(R\left(s_{t},a_{t},s_{t+1}\right)+\alpha
H\left(\pi\left(\cdot\mid s_{t}\right)\right)\right)\right].$

We later demonstrate (Fig. 8) that the direct optimization of the above
objective in an end-to-end learning paradigm does not lead to meaningfully
generalizable behavior. We attribute this failure to the challenges in
navigating an extremely high-dimensional search space that hosts all internal
degrees of freedom of the hand and the object under manipulation in addition
to third-order actuation dynamics, intermittent contact dynamics, and the high
likelihood of catastrophic failures due to uncoordinated movements during
early exploration.

In contrast to end-to-end learning, biological organisms faced with similar
search spaces are easily able to synthesize sophisticated movements that are
robust to this complexity. In the next section, we outline our method that
takes inspiration from mechanisms that putatively contribute to this evolved
behavioral sophistication.

##  IV SAR: Synergistic Action Representation

Exploration challenges in high-dimensional action manifolds naturally
associated with manipulation and locomotion are well-recognized [5], including
in the field of robotics [5, 42, 13] and computer vision [63, 36]. Multiple
techniques leveraging expert demonstrations [42], curriculum learning [37,
13], and even human-designed mappings to embed representations into the
learning process [29] have been proposed in order to aid and accelerate the
acquisition of competent behavior. In contrast to these methods, which lean
heavily on experts, we investigate if there exist learning paradigms that do
not require such interventions at any point. Our work is heavily inspired by
the increasingly strong evidence that the biological motor system enables
actuation by leveraging synergistic control of the musculoskeletal system [10,
6, 15, 38]. We proceed to ask:

  1. 1.

How can synergistic representations be automatically acquired?

  2. 2.

How can synergistic control can be embedded in behavior acquisition paradigms?

###  IV-A Synergy Acquisition

Researchers in the fields of biomechanics and neuroscience have long pursued
the search for synergies both at the joint-postural and muscle-activation
levels. Due to the relative ease of recording joint posture over muscle
activations, synergies at the level of joint postures [55, 47, 17] have been
more extensively used to synthesize behaviours in robotic control [20, 33].
Nevertheless, although muscle synergies provide direct insight into biological
motor control and learning [16, 15, 12, 14], they can only be practically
extracted from a limited number of muscles in human legs and hands. This
practical limitation represents the biggest limitation to directly using
muscle synergies to synthesize behaviors [53].

How can we address this challenge to compute and extract synergies for the
MyoLegs and MyoHand at the actuation level? We consider the underlying
properties of synergies during locomotion and manipulation: synergies act like
lower-dimensional submanifolds in the high-dimensional space that are
consistently visited during the course of the behavior. In order to capture
these recurrent pathways, we subject MyoLegs and MyoHand to a play phase,
during which relatively simple behaviors are learned.

After the play period, we record muscle activation time series data from the
resulting policies, $M^{act}\in\mathcal{R}^{a\times t}$. We posit that owing
to the diversity of the behaviors captured in the dataset, its visitation
density has a high degree of overlap with the synergistic submanifolds. While
there are multiple plausible strategies for recovering submanifolds from such
datasets, we resort back to the techniques common in motor neuroscience
investigations of synergies from experimental datasets [40]. Applying both PCA
and ICA to the time series data (taken together, ICAPCA) is accepted as a
robust method for capturing muscle synergies [57]. In this work, we take the
additional step of normalizing this output in the range [-1, 1] in order to
reconfigure the resultant submanifold as a control signal.

$SAR:=|T_{ICA}(T_{PCA}(M^{act}))|$

The resulting linear factorization is what we refer to as the Synergistic
Action Representation (SAR). Practically, this representation can also be
viewed as a way to inverse transform any point in lower-dimensional synergy
space to full muscle activation space $a_{full}^{syn}=SAR(a_{s}yn)$. While
$syn=full$ constitutes a complete reconstruction of the original signal, lower
$syn$ will allow for capturing the most informative muscle co-activation
patterns over time. For both MyoLegs and MyoHand, we define SAR based on
$syn=20$ (which for both models captured over 80% of the variance in
$M^{act}$, see Appendix: Fig A.17).

###  IV-B Behavior Acquisition with SAR

While SAR encodes preferred muscle synergies, we still require a method for
training locomotion and manipulation behaviors. The question thus becomes: how
can use embed SAR into a behavioral learning paradigm? One intuitive strategy
is to learn a policy directly in the space of synergies and project it to full
muscle activation space via SAR: $a^{syn}_{39}=SAR(\pi(a_{N}|\o_{t}))$. While
the reduced dimensionality helps with the exploration challenges, it also
theoretically restricts the policy from expressing behaviors significantly
outside of the synergistic manifolds. Accordingly, we develop a policy
architecture that preserves the benefits of directed search from the
synergistic pathways but is still able to optionally learn task-specific
behavioral finetuning in the original nonsynergistic manifold.

Figure 2: Policy architecture for training with SAR. The policy takes as input
$o_{t}$ and outputs an $O+N$-dimensional action vector, $a_{t}^{O+N}$, where
$O$ is the dimensionality of the original action manifold and $N$ is the
dimensionality of the synergistic manifold. The first $N$ synergistic actions
are passed through $SAR(a_{t}^{N})=a_{t}^{SAR}$ to recover the synergistic
muscle activations. The subsequent $O$ task-specific activations are mixed
with the task-general activations using a linear blend $\varphi$ to recover
the final action $a_{t}^{*}$ that steps the environment forward. Note that for
MyoLegs, it was found that purely relying on the synergistic, task-general
pathway ($\varphi=1$) was sufficient for learning robust locomotion.

In order to balance contributions from task-general and task-specific muscle
activations, we introduce SAR into our behavioral acquisition pipeline as an
alternative weighted action representation pathway for our policy to utilize
(Fig. 2). At each timestep, the policy is trained to output a
$O+N$-dimensional action vector continuous in the range $[-1,1]$, where $O$ is
the dimensionality of the original action space, and $N$ is the number of
synergistic modules in SAR. The first $N$ components of the action vector are
input into SAR, which transforms this representation back into synergy-
exploiting activations in the original muscle activation space
($a_{t}^{SAR}$). The subsequent $O$ components of the action vector
($a_{t}^{O}$) are additively combined with the synergy-exploiting activations,
returning a final action ($a_{t}^{*}$) that steps the environment forward.
This simple elementwise vector addition is weighted by $\varphi$, a parameter
that calibrates the relative influence of the synergistic and nonsynergistic
muscle activation vectors such that
$a_{t}^{*}=\varphi(a_{t}^{SAR})+(1-\varphi)(a_{t}^{O})$ (Fig. 2).

Contrary to conventional wisdom that suggests a reduction the dimensionality
of the action space will best aid exploration challenges, the proposed policy
architecture somewhat counterintuitively increases the dimensionality of the
search space. Aligned with [59, 62, 61], our policy architecture can be viewed
as providing the learning agent the option to mix behavior from task-general
synergistic representational pathways (encoded by the first $N$ components of
$a_{t}$) while simultaneously allowing for the task-specific modulations
(encoded by the subsequent $O$ components of $a_{t}$). As is later
demonstrated in our MyoHand experiments (Sec. VII-B), in accordance with this
intuition, our agents learn to leverage the synergistic pathways when helpful
to accelerate search and proceed to rely on task-specific pathways for task
customization, exploiting the benefits of both learning pathways (see Fig. 2).
It is also critical to note that this architecture trivially allows for
exclusively task-specific ($O$-dimensional) or task-general ($N$-dimensional)
learning by setting $\varphi=0$ or $1$, respectively. Accordingly, we later
find that purely relying on the synergistic, task-general pathway (i.e.,
$\varphi=0$) was sufficient for learning robust locomotion. As such, this
policy architecture demonstrates a flexibility that renders it an attractive
mechanism for using SAR for behavior acquisition.

##  V Acquiring Agility and Dexterity with SAR: Experimental Design

We structure our main experiment to assess the effectiveness of SAR and our
proposed policy architecture (Fig. 2) in yielding generalizable agility and
dexterity. For both of these experiments using MyoLegs and MyoHand, we
structure our experimental goals into three discrete phrases:

  1. 1.

Play phase: learn play behaviors in an environment similar to the target task
in order to acquire activation data for computing SAR.

  2. 2.

Training phase: acquire the target behavior(s) using the SAR-based policy
architecture (Fig. 2) in comparison to baseline learning methods (Fig. 3) that
do not exploit synergies.

  3. 3.

Testing phase: Utilize both SAR and the SAR-based policy to yield zero- or
few-shot generalizations to new tasks with out-of-domain environmental
properties.

Before presenting our results in Sec. VII, next we outline details of our
locomotion and manipulation task suites for MyoLegs and MyoHand, respectively,
including technical task specifications as well as the play phase, training
phase, and testing phase for both musculoskeletal modalities.

(a) the in-hand reorientation task suite in Sec. V-A), (b) SAR representation
acquisition in Sec. V-C, and (c) our SAR agent and choice of baselines in Sec.
V-F.

Figure 3: Experimental paradigm. Ours (left) trains policies using a computed
SAR. Curriculum (center), leverages the play policy and extends training
without using SAR. End-2-End (right) simply trains directly on the training
phase for the entire training budget. Policies are then zero- and few-shot
tested on a set of out-of-domain tasks. All policies use identical
hyperparameters, including total number of training samples.

###  V-A Task specifications

We describe the overarching goal of the MyoLegs and MyoHand physiological
control experiments before detailing the action, state, and reward functions
used for these experiments.

MyoLegs control experiment. We task our MyoLegs agent with learning robust
locomotion using a limited training budget, beginning with a simpler flat
terrain and significantly complexifying the terrain conditions later on. This
task requires the continuous coordination of 80 muscles across two limbs, a
highly challenging action space to learn in a small number of samples.

MyoHand control experiment. We task our MyoHand agent with reorienting an
object to a particular goal orientation. Both the specific object and the
desired reorientation angle are randomly initialized at the beginning of each
training episode. This task requires an agent capable of learning a highly
flexible motor control policy that is robust to high variance both in object
identity and desired reorientation.

Musculoskeletal action space. For both MyoLegs and MyoHand, the action vector
$a_{t}$ consists of continuous values $[-1,1]$ for contracting each of the
muscles in the musculoskeletal model (80 muscles in MyoLegs, 39 muscles in
MyoHand. Given that muscle activations cannot be negative [40], MyoSuite
transforms this action vector into the range $[0,1]$ using the sigmoid
transform,$\frac{1}{1+e^{-5*(a_{t}-0.5)}}$.

Musculoskeletal state space For both MyoLegs and MyoHand, the state vector
$s_{t}=\\{\phi_{t}^{act},\phi_{t}^{len},\dot{\phi}_{t},\phi_{t}^{F}\\}$
consists of continuous values for muscle activations $\phi^{act}$, muscle
length $\phi^{len}$, muscle velocity $\dot{\phi}$, and muscle force
$\phi^{F}$. Given the object manipulations required in the MyoHand experiment,
object rotation $\psi$, object velocity $\dot{\psi}$, object goal rotation
${\psi^{*}}$, and object rotational error ${\psi^{e}}$ were also specifically
included in $s_{t}$ for this experiment only. Note that muscle activations,
force, length, and velocity were included in $s_{t}$ in lieu of joint
positions because proprioceptive signals are instantiated in the human nervous
system [58], while there is no corresponding evidence that suggests precise
joint positions are encoded neurophysiologically.

MyoHand reward function We employ the following reward function for learning
multiobject reorientation with MyoHand:

$R(x_{t},\hat{x}_{t}):=-\lambda_{1}\|x_{t}^{(p)}-\hat{x}_{t}^{(p)}\|_{2}+\lambda_{2}|\angle
x_{t}^{(o)}-\hat{x}_{t}^{(o)}|\\\
-\lambda_{3}\\{dropped\\}-\lambda_{4}\left\|\overline{m}_{t}\right\|_{2}+\lambda_{5}\\{b1\\}+\lambda_{6}\\{b2\\}$
(1)

where $\angle$ is the quaternion angle between the two current and goal
orientations, $x_{t}^{(p)}$ is the object goal position, $x_{t}^{(o)}$ is the
object goal orientation, $1\\{dropped\\}$ penalizes object dropping,
$\\{b1\\}$ and $\\{b2\\}$ incentivize simultaneous rotational and positional
alignment above a threshold, and $\bar{m}_{t}$ encodes overall muscle effort.

MyoLegs reward function We employ a different reward function for learning
locomotion with MyoLegs:

$R(t):=\lambda_{1}v_{t}-\lambda_{2}c_{t}+\lambda_{3}r_{t}+\lambda_{4}j_{t}-\lambda_{5}d_{t}$

where $v_{t}$ denotes the velocity reward (quantifying the desired velocity of
the leg), $c_{t}$ represents the cyclic reward for the hip (aiming to
encourage rhythmic movements), $r_{t}$ is the reward based on the reference
rotation (encourages the leg to follow a given rotation pattern), $j_{t}$ is
the joint angle reward (penalizes deviation of specific joints from their
desired angles), and $a_{t}$ is the action magnitude, computed as the norm of
the action divided by the total number of actuators (aiming to encourage
efficient use of the actuators).

where $\angle$ is the quaternion angle between the two current and goal
orientations, $x_{t}^{(p)}$ is the object goal position, $x_{t}^{(o)}$ is the
object goal orientation, $1\\{dropped\\}$ penalizes object dropping,
$1\\{bonus1\\}$ and $1\\{bonus2\\}$ incentivize simultaneous rotational and
positional alignment above a threshold, and $\bar{m}_{t}$ encodes overall
muscle effort.

We considered a trial successful when object rotational error
${\psi^{e}}=|\cos{(\psi,\psi^{*})}|$ was $\leq 5\%$. All trials were 50
timesteps.

###  V-B MyoHand object datasets

For the play, training, and testing phases of the MyoHand reorientation
experiment, objects were sampled from two datasets. The Reorient dataset
consisted of four core geometries in the MuJoCo physics simulator [56]:
ellipsoid, cuboid, cylinder, and capsule (i.e., spherocylinder). Those objects
could be parameterized to be scaled across the X, Y, and Z axes. Geometrical
dimensions and target poses were determined given both the physiological
constraints of the hand and the parameters used in related works involving
manipulation [11].

Additionally, a set of real-world objects was obtained by using 8 objects from
the ContactDB dataset [9].

###  V-C Play phases

For both experiments, the purpose of the play phase is to train a policy on a
simple task similar to the target task in order to extract muscle activation
data that can be used to compute a Synergistic Action Representation that can
be used to learn the target task (training phase).

MyoLegs play phase. For the MyoLegs locomotion experiment, the play period
simply consisted of training a policy on a flat walking environment for a 1.5M
steps. Though this policy would not be expected to achieve locomotion in such
a small number of samples, it was hypothesized that the muscle activation data
yielded by this policy would be sufficient for computing a Synergistic Action
Representation that would itself ultimately facilitate locomotion.

MyoHand play phase. For the MyoHand manipulation experiment, the play period
consisted of training a policy to reorient a large and small variant of each
of the four core MuJoCo geometries (see V-B), totaling eight geometries (Table
A.2) for 1M steps. The agent is exposed randomly to all eight of the objects
to learn to reorient them. It was hypothesized that this simpler environment
would introduce the agent to the multiobject manipulation control problem
without subjecting it to the full complexity of learning to reorient 100
discrete objects.

###  V-D Training phases

After the MyoLegs and MyoHand play phases, we are able to extract muscle
activations from the resulting play policies, which will be transformed into a
Synergistic Action Representation (see Sec. IV-A) and subsequently used to
parameterize a new policy (Fig. 2).

MyoLegs training phase. We test whether SAR acquired from a play policy 1.5M
steps on a forward locomotion task is sufficient for actually yielding forward
locomotion. In other words, (unlike the MyoHand experiment) the MyoLegs
training environment is the same as that used in the play phase. However, it
is critical to emphasize, as is later described in Sec. VII, that the MyoLegs
play phase policy does not itself learn any meaningful walking behavior. This
new policy is trained for an additional 1.5M steps.

MyoHand training phase. After extracting SAR from the play phase policy
trained to reorient eight objects, we use this representation to facilitate
reorientation of a much larger set of objects: we generate 25 different
geometries (Table A.2) of the 4 objects in the Reorient dataset. This set of
100 objects was randomly presented and the policy was trained for 2M steps.

###  V-E Testing phases

After the MyoLegs and MyoHand training phases, we can assess the
generalizability both of SAR and of the SAR-based policy. First, the degree to
which the policy zero-shot generalizes to out-of-domain conditions can be
compared to baseline approaches in order to assess the generality of what has
been learned over the previous two phases. Second, we can assess how
effectively the original SAR transfers when used to few-shot train new
policies on these out-of-domain environments.

MyoLegs test phase. We introduce four new locomotion environments in order to
rigorously assess the generalizability of the locomotion-based SAR and the
associated policy: a ‘hilly’ terrain, a stair-like terrain, an uneven terrain,
and a directionality change environment (i.e., a diagonal walk). We begin by
zero-shot testing the trained policy on the uneven terrain to assess the
generalizability of the SAR-based policy, and we proceed use the locomotion-
based SAR to train new policies on these unseen environments to assess the
generality of the original representation extracted during the MyoLegs play
phase.

MyoHand test phase. After training the MyoHand on the 100-object reorientation
task, the policy was zero-shot tested using three new object sets (see Fig.
9). First, in an in-domain test, the policy was tested in a zero-shot
inference on 1000 novel objects generated from the same parameters used for
training (250 geometries for 4 objects (Table A.2). Second, in an out-of-
domain test, an additional 1000 objects (250 geometries for 4 objects) were
obtained from a range of parameters outside of those used for training (Table
A.2). Finally, the generalization was tested on a RealWorldObjs set (see Sec.
V-B; Fig. 9).

###  V-F Baselines

For both the MyoLegs and MyoHand experiments, the performance of the trained
SAR-based policy (SAR-RL) was compared against two baselines.

####  V-F1 Curriculum learning: RL+Curr

First, we compared SAR to standard curriculum learning, here named RL+Curr, by
directly finetuning (i.e., resuming training for) the policy trained on the
play phase for the main training phase. This baseline (see ‘Baseline:
Curriculum,’ Fig. 3) controls for whether utilizing _any_ form of pretraining
on a simpler environment modulates performance on the more complex target task
or whether the Synergistic Action Representation _per se_ leads to enhanced
policies.

####  V-F2 End-to-end RL: RL-E2E

In addition, we train a policy end-to-end (End2End, or RL-E2E for short, see
‘Baseline: End-2-End,’ Fig. 3) on training phase task without any play phase
or Synergistic Action Representation. This baseline controls for whether SAR-
RL’s indirect access to additional training experience from the play phase is
what is driving its performance. Accordingly, note that the total training
samples and algorithm hyperparameters are identical across all three
conditions.

##  VI MyoLegs experimental results

We present results from the MyoLegs experiments described in Sec. V-A, where a
Synergistic Action Representation extracted from a policy trained for a small
number of steps on a flat walking task successfully enables locomotion when
used to retrain on this same task. We also show that this same SAR can be
leveraged to successfully learn locomotion on a diverse set of new terrains
with SOTA sample efficiency.

###  VI-A SAR enables highly sample efficient locomotion

In spite of the general failure of the play phase MyoLegs policy to yield
meaningful behavior, we hypothesized that this policy may exhibit sufficient
visitation of the relevant submanifolds of the action space for SAR to contain
representations useful for learning forward locomotion. Figure 4 demonstrates
that while RL-E2E fails to yield locomotion, training with SAR-RL efficiently
yields robust walking behavior. SAR-RL learns to walk $15x$ times farther
(11.7m vs. 0.8m) than RL-E2E (Fig. 4).

Figure 4: SAR-RL yields highly sample efficient locomotion. Left: after 1.5M
play phase steps (dotted black line), SAR is computed and a new SAR-exploiting
policy is trained on the forward locomotion task for an additional 1.5M steps
(blue). This is compared to RL-E2E (i.e., training on the forward locomotion
task for all 3M steps; orange). Right: average distance traveled by the
trained policies, in meters.

###  VI-B SAR-RL zero-shot generalization to uneven terrain

We test the hypothesis that MyoLegs policies trained within the synergistic
manifold (see Sec. V-D) will demonstrate superior zero-shot generalization in
unseen terrains than policies trained without SAR. We specifically assess
zero-shot generalization on uneven terrain, as is standard in the literature
[23]. We find that policies trained with SAR-RL are able to zero-shot
generalize to this uneven terrain, traveling more than twice the distance as
compared to the zero-shot performance of the RL-E2E baseline policy (2.3 m vs.
0.8 m; see Fig. 5).

Figure 5: SAR-RL zero-shot generalization to uneven terrain. The SAR-based
MyoLegs training phase policy is able to walk an average of 2.3 meters zero-
shot on uneven terrain, while RL-E2E is able to walk an average of 0.8 meters.

###  VI-C SAR transfers to diverse terrains

Though SAR-RL was able to relatively successfully zero-shot generalize to
uneven terrain, we hypothesize that tasks with significantly out-of-
distribution features such as uneven terrain, stair-climbing, hilly terrain,
and directionally varied environments likely require transfer learning in
order to yield robustly successful locomotion. Accordingly, we test the extent
to which the same SAR from the MyoLegs play phase (see Sec. V-C) enables
locomotion on terrains significantly different from that used to train the
initial play phase policy from which SAR was extracted. We find that this same
SAR indeed enables locomotion across diverse terrains and walking directions
using only 2.5M additional samples (see Fig. 6), while training using baseline
approaches again fails to yield meaningful behavior: across the four out-of-
domain environmental conditions, RL-E2E was found to walk an average of only
0.5 meters, while SAR-RL was found to walk an average of approximately 8
meters, a 16-fold improvement in performance (Fig. 7).

Figure 6: SAR-RL transfer learning to diverse terrains. Across four out-of-
domain environmental conditions—hilly terrain, stair-like terrain, uneven
terrain, and diagonal walk—the same SAR enables highly sample efficient
locomotion, while RL-E2E fails to learn any meaningful behavior. Example
frames are provided from the SAR-RL hill locomotion policy. Figure 7: SAR-RL
travels 16x farther than RL-E2E on average. Across the four out-of-domain
environmental conditions,RL-E2E was found to walk an average of approximately
0.5 meters, while SAR-RL was found to walk an average of approximately 8
meters, a 16-fold improvement.

##  VII MyoHand experimental results

We present results to show how SAR uniquely enables learning a single policy
that can solve a large set of in-hand reorientations using MyoHand (Sec.
VII-A). The same policy is then able to generalize to reorient objects with
different shapes that were not part of the training set (Sec. VII-B). Also,
the same synergistic representation can be used to learn in-hand reorientation
on real objects (Sec. VII-D).

###  VII-A SAR enables multiobject manipulation

We started with the hypothesis that learning by means of a Synergistic Action
Representation (SAR) enables learning a single policy capable of solving a
large variety of manipulation tasks. We used SAR (extracted as described in
Section V-C) to train a policy to reorient 100 different objects from the
Reorient dataset (see Sec. V-B). Figure 8 shows that SAR-RL is able to solve
faster and achieve a significantly higher success rate ($>70\%$) than other
methods. In reality, neither learning to solve directly all those tasks
(RL-E2E, Fig. 8) nor pre-training (RL+Curr, Fig. 8 orange line) appears to
help the generalization to a larger set of objects as indicated by a success
rate $<20\%$ in Figure 8. This indicates that the success at reorienting the
object was due to the adoption of SAR.

Figure 8: Manipulation of 100 objects with SAR. Performance learning a multi-
task policy to reorient 100 different objects by means of 3 different methods.
SAR-RL, depicted in blue, uses the acquired synergistic action representation
from a simpler task (1M steps) to facilitate and accelerate learning (2M
steps). RL+Curr, depicted in orange, represents the use of curriculum learning
for 1M steps on a reduced number of objects, followed by finetuning this
policy on the full set of 100 objects (2M steps). RL+E2E, depicted in green,
represents the use of an End-2-End policy that attempts to solve the 100
objects for the whole duration of the learning budget considered (i.e., 3M
steps). SAR-RL and RL+Curr are depicted to begin at 1M timesteps in order to
account for the 1M steps of pretraining embedded in both methods. The shaded
area represents the variability of the policies over 3 different seeds.

###  VII-B SAR manipulation zero-shot generalization

Next, we test the hypothesis that the SAR-exploiting policy also enables
superior zero-shot generalization on 3 different sets of objects (see Fig. 9
\- left). First, from the Reorient dataset we generated a new set of 1000 in-
domain objects within the same range used for the training (Reorient-ID).
Second, from the Reorient dataset we generated a new set of 1000 objects by
sampling object dimensions from uniform distributions both above and below
those used for the training environment (see Table A.2). This procedure
yielded objects that are both larger and smaller than—but never the same size
as—those generated in the training environment (Reorient-OOD). Third, we
tested generalization on 8 different real-world objects (RealWorldObjs).
Figure 9 shows that the SAR-based policy allows generalizing to reorient both
in-domain and out-of-domain objects and real-world objects. Indeed, when
compared to either end-to-end or curriculum learning, the generalization is
$>3x$ greater for parametric objects and $>2x$ greater for the real-world
objects.

Figure 9: Left: Zero-shot testing on 1000 objects obtained within the same
parameter set (in-domain) or outside the set of parameters (out-of-domain)
used for generating the objects for training the SAR based policy. In
addition, a set of eight real-world objects were used. Color coded indicates
the SAR, Curriculum and End-2-End policies described in Figure 8. Right:
Behavioral examples of SAR-RL’s zero-shot generalization to in-domain and out-
of-domain parametric objects.

###  VII-C SAR stabilizes over learning

Given that SAR enables strong generalization, the question that arises is if
all synergies were contributing equally to this behavior. Figure 10 shows the
normalized mean contributions of each synergy in SAR throughout the training
at different times. Throughout learning, the first 8 synergies—those that
originally contributed to explaining most of the variance of the data
($~{}60\%$, see Fig. A.17)—have a greater contribution to the final solutions
as indicated by their larger weight (Fig. 10).

###  VII-D SAR transfer learning on Real World Objects

SAR-RL was able to generalize to both parameterized shapes and to real-world
objects. Nevertheless, performance on real-world objects seems to be inferior
to parameterized objects (see Fig. 9, likely due to the enhanced complexity of
their contact dynamics. Accordingly, we investigate if it is possible to use
the SAR to capture via few-shot learning the details of those real-world
objects to reorient them. In this experiment, we leveraged only the first
eight synergies as we observed to be the ones contributing most to the
generalization.

We find that SAR significantly accelerates learning reorientation policies as
compared to training without SAR, approximately doubling the speed of
dexterity acquisition in complex real-world object reorientation tasks (Fig.
11).

Figure 10: Relative contribution of synergies over 100 object training.
Representation of the normalized mean contributions of each synergy over 2M
training steps on the 100-object reorientation environment. Figure 11: Few-
shot learning with SAR on RealWorldObjs. Average policy success on the
RealWorldObjs with SAR-RL (blue) as compared to training without—i.e., RL-E2E
(orange).

##  VIII Extending SAR beyond physiological control

Thus far we demonstrated that SAR enables robust skill transfer for
physiological control. To explore the generalizability of the SAR framework to
other high-dimensional continuous control settings, we perform two additional
sets of experiments on standard continuous control benchmarks. These trials
successfully extend SAR to (1) the 20-DoF robotic Shadow Hand in simulation
[54], and (2) the Humanoid-v2 gym environment [39], a 17-DoF bipedal walking
task.

###  VIII-A SAR enables robotic skill transfer in Shadow Hand

First, we demonstrate SAR’s applicability to robotics problems, demonstrating
that SAR enables approximately 2x faster learning of randomized object
reorientation on Shadow Hand compared against end-to-end RL (see Fig. 12.

####  VIII-A1 Dexterity transfer between geometries

Our first Shadow Hand result demonstrates the generalizability of SAR using
synergies computed from policies trained on the official gymnasium-robotics
‘HandManipulateEggRotate-v1’ environment. It is important to note that the
control actions in this environment are absolute angular positions of the
actuated joints rather than continuous muscle activations (as in the MyoHand).
This requires that the synergistic action representation extracted from this
environment would comprise coordinated joint movements rather than muscle co-
contractions.

In this experiment, we first train an end-to-end policy to manipulate an egg
to a randomized target orientation. We then compute and extract SAR by rolling
out this policy. Next, we use SAR (see Fig. 2) to train reorientation policies
on two unseen geometries: capsules and cylinders. In comparison to end-to-end
RL on cylinder and capsule reorientation, we find that SAR-RL achieves
approximately 2x better performance using the same number of samples (see Fig.
12).

####  VIII-A2 Replication of Reorient100 experiment

Our second Shadow Hand result reimplements the Reorient100 experiment in the
robotic task setting. As in the MyoHand experiment (see Fig. 3), we use an
eight-object reorientation task (2 x 4 geometric shapes) to train a policy,
compute SAR from rollouts of this trained policy, and use the acquired
representation to facilitate learning of the significantly more challenging
100-object (25 x 4 geometric shapes) reorientation task (Fig. 12). We find
that, compared to end-to-end training, SAR-RL achieves approximately 2x better
performance using the same number of samples on the 100-object task.

Figure 12: SAR-enabled dexterity on Shadow Hand. Top left: Using synergies
computed from an egg manipulation policy, SAR-RL achieves approximately 2x
better performance on randomized capsule and cylinder reorientation compared
to end-to-end RL. Top right: SAR-RL achieves approximately 2x better
performance on the Reorient100 task compared to end-to-end RL with synergies
computed from an 8-object reorientation policy. Bottom: example manipulations
of the policy trained with SAR-RL on Reorient100.

###  VIII-B SAR efficiently yields robust humanoid locomotion

We proceed to extend SAR to full-body motor control on the Humanoid-v2 gym
environment. We find that by simply training on this environment for a small
number of steps, computing and extracting SAR from this early policy, and
using this representation to parameterize the training of a new policy on the
same environment, we are able to reach SOTA performance in only 1.25M total
training steps (Fig. 13), one to two orders of magnitude more efficiently than
competing approaches documented in the literature (see Table I). Of note, we
also find that training with SAR yields a significantly more natural gait
compared to baseline approaches (e.g., consistent cadence and step length;
presence of stance and swing phases; arm swinging and upright posture).

Experiment |  Algorithm |  Returns at 1.25M samples
---|---|---
SAR (ours) |  SAR+SAC |  6104
Wang & Ni, 2020 |  meta-SAC |  5610
Peng et al, 2010 |  SAC |  4100
Wu et al, 2022 |  SAC |  4600
Toklu et al, 2020 |  PGPE |  715
RL-Zoo baseline |  SAC |  1552
AI2 baseline |  PPO |  1120
TABLE I: Comparing SOTA performance on Humanoid-v2 at 1.25M samples. Figure
13: SAR-enabled locomotion on Humanoid. Left: SAR-RL yields 3x better
performance on Humanoid-v2 than RL-E2E in the same number of total timesteps.
Right: behavioral comparison of the locomotion behavior yielded by SAR-RL in
comparison to the top RL-Zoo benchmark.

##  IX Discussion

Leveraging SAR computed from simpler tasks as a weighted action representation
for learning more complex tasks is demonstrated to be a uniquely effective
means for enabling strong performance (a) for bipedal locomotion over diverse
terrains, (b) for in-hand multiobject manipulation, and (c) for high-
dimensional control problems beyond the scope of physiological control.
Alternative solutions that did not leverage synergies did not yield comparably
effective or generalizable policies. What accounts for this striking
difference in performance?

Framing muscle synergies as an embodied form of transfer learning proves
useful for understanding how they enable strong performance on high-
dimensional continuous control tasks. It is well-known that in humans, skill
transfer learning occurs across the entire lifespan and is an important
contributing factor to the uniquely sophisticated behavior of humans [3]. Such
learning prevents human learners from being required to ’reinvent the wheel’
with each new behavioral repertoire by leveraging already-existing stable
action representations accrued during lifelong learning. Analogously, in our
experiments, we show that (a) the exact same set of synergies can be reused
and effectively finetuned for diverse task settings, and that (b) normalized
contributions of the synergies stabilize over training to utilize a core
subset of coordinated muscle co-contraction patterns (Fig. 8). Together, these
findings constitute strong evidence that a physiologically coherent action
representation is transferred from one task to the next.

Additionally, the successful extension of SAR to robotic and humanoid control
demonstrates that the utility of the representation is not constrained to
musculoskeletal control and can more generally enable robust learning in high-
dimensional systems. Taken as a whole, these investigations demonstrate that
leveraging synergies—a core mechanism of human motor organization and
adaptation—enables a degree of dexterity and agility that was otherwise not
reached using baseline learning methods. Accordingly, we propose that learning
high-dimensional control using Synergistic Action Representations is a
promising mechanism for instantiating skill transfer learning and ultimately
bootstrapping towards a generalist embodied agent.

##  X Limitations and future research

There are a number of limitations and opportunities for future research from
the present study. While the MyoHand and MyoLegs are physiologically accurate
from a structural perspective, there are still fundamental differences between
the state and action spaces employed in human motor learning and those of the
musculoskeletal models utilized in this research. The most obvious difference
is that the these models lack two sensory modalities: touch and sight.
Previous work suggests that both of these sensory pathways in the nervous
system help facilitate human-level dexterity and agility [26] and that a loss
of touch in particular can severely compromise dexterity in humans [34]. These
facts reveal both a strength and a limitation of this investigation: it is
limited insofar as the policies yielded from the synergistic learning paradigm
still should not be expected to precisely mirror the behavioral style of human
learners given dissimilar sensory data. However, this perceptual deficit also
reveals a strength of the synergistic learning paradigm: namely, that sight
and touch _were not required_ to yield successful control policies for these
complex tasks.

This investigation naturally enables a number of opportunities for future
research. First, the method for computing and instantiating muscle synergies
developed in this investigation is entirely task-agnostic, meaning that our
framework could be conceivably leveraged for any musculoskeletal task set. It
is also worth noting that the strategy of using normalized ICAPCA to compute
muscle synergies is potentially viable for yielding useful, dimensionality-
reduced representations that can be conceivably leveraged for any RL skill
transfer problem. In terms of object-level future opportunities, more formally
studying the extent to which muscle synergies can be utilized as an embodied
form of transfer learning remains highly underexplored and is a promising area
of future research.

## References

  * Al Borno et al. [2020] Mazen Al Borno, Jennifer L. Hicks, and Scott L. Delp.  The effects of motor modularity on performance, learning and generalizability in upper-extremity reaching: a computational analysis.  _Journal of The Royal Society Interface_ , 17(167):20200011, June 2020.  doi: 10.1098/rsif.2020.0011.  URL https://royalsocietypublishing.org/doi/10.1098/rsif.2020.0011.  Publisher: Royal Society.
  * Balda et al. [1998] Russell P. Balda, Irene M. Pepperberg, and A. C. Kamil.  _Animal Cognition in Nature: The Convergence of Psychology and Biology in Laboratory and Field_.  Academic Press, September 1998.  ISBN 978-0-08-052723-9.  Google-Books-ID: 504iRS01AK0C.
  * Bassett and Gazzaniga [2011] Danielle S. Bassett and Michael S. Gazzaniga.  Understanding complexity in the human brain.  _Trends in Cognitive Sciences_ , 15(5):200–209, May 2011.  ISSN 1364-6613.  doi: 10.1016/j.tics.2011.03.006.  URL https://www.sciencedirect.com/science/article/pii/S1364661311000416.
  * Bellemare et al. [2013] Marc G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling.  The Arcade Learning Environment: An Evaluation Platform for General Agents.  _Journal of Artificial Intelligence Research_ , 47:253–279, June 2013.  ISSN 1076-9757.  doi: 10.1613/jair.3912.  URL http://arxiv.org/abs/1207.4708.  arXiv:1207.4708 [cs].
  * Bicchi and Kumar [2000] Antonio Bicchi and Vijay Kumar.  Robotic grasping and contact: A review.  In _Proceedings 2000 ICRA. Millennium conference. IEEE international conference on robotics and automation. Symposia proceedings (Cat. No. 00CH37065)_ , volume 1, pages 348–353. IEEE, 2000.
  * Bizzi et al. [2008] E. Bizzi, V. C. K. Cheung, A. d’Avella, P. Saltiel, and M. Tresch.  Combining modules for movement.  _Brain Research Reviews_ , 57(1):125–133, January 2008.  ISSN 0165-0173.  doi: 10.1016/j.brainresrev.2007.08.004.
  * Bizzi and Cheung [2013] Emilio Bizzi and Vincent CK Cheung.  The neural origin of muscle synergies.  _Frontiers in Computational Neuroscience_ , 7, 2013.  ISSN 1662-5188.  URL https://www.frontiersin.org/articles/10.3389/fncom.2013.00051.
  * [8] Emilio Bizzi, Ferdinando A. Mussa-Ivaldi, and Simon Giszter.  Computations underlying the execution of movement: A biological perspective.  253(5017):287–291.  doi: 10.1126/science.1857964.  URL https://www.science.org/doi/abs/10.1126/science.1857964.  Publisher: American Association for the Advancement of Science.
  * Brahmbhatt et al. [2019] Samarth Brahmbhatt, Cusuh Ham, Charles C. Kemp, and James Hays.  ContactDB: Analyzing and Predicting Grasp Contact via Thermal Imaging, April 2019.  URL http://arxiv.org/abs/1904.06830.  arXiv:1904.06830 [cs].
  * Caggiano et al. [2016] Vittorio Caggiano, Vincent C. K. Cheung, and Emilio Bizzi.  An Optogenetic Demonstration of Motor Modularity in the Mammalian Spinal Cord.  _Scientific Reports_ , 6(1):35185, October 2016\.  ISSN 2045-2322.  doi: 10.1038/srep35185.  URL https://www.nature.com/articles/srep35185.  Number: 1 Publisher: Nature Publishing Group.
  * Caggiano et al. [2022] Vittorio Caggiano, Huawei Wang, Guillaume Durandau, Massimo Sartori, and Vikash Kumar.  MyoSuite – A contact-rich simulation suite for musculoskeletal motor control, May 2022.  URL http://arxiv.org/abs/2205.13600.  arXiv:2205.13600 [cs].
  * [12] Jiahao Chen and Hong Qiao.  Muscle-synergies-based neuromuscular control for motion learning and generalization of a musculoskeletal system.  51(6):3993–4006.  ISSN 2168-2232.  doi: 10.1109/TSMC.2020.2966818.  Conference Name: IEEE Transactions on Systems, Man, and Cybernetics: Systems.
  * Chen et al. [2021] Tao Chen, Jie Xu, and Pulkit Agrawal.  A System for General In-Hand Object Re-Orientation.  November 2021.  URL https://openreview.net/forum?id=7uSBJDoP7tY.
  * Cheung et al. [2020] Vincent C. K. Cheung, Ben M. F. Cheung, Janet H. Zhang, Zoe Y. S. Chan, Sophia C. W. Ha, Chao-Ying Chen, and Roy T. H. Cheung.  Plasticity of muscle synergies through fractionation and merging during development and training of human runners.  _Nature Communications_ , 11(1):4356, August 2020\.  ISSN 2041-1723.  doi: 10.1038/s41467-020-18210-4.  URL https://www.nature.com/articles/s41467-020-18210-4.  Number: 1 Publisher: Nature Publishing Group.
  * [15] Andrea d’Avella and Emilio Bizzi.  Shared and specific muscle synergies in natural motor behaviors.  102(8):3076–3081.  ISSN 0027-8424, 1091-6490.  doi: 10.1073/pnas.0500199102.  URL https://pnas.org/doi/full/10.1073/pnas.0500199102.
  * [16] Andrea d’Avella, Philippe Saltiel, and Emilio Bizzi.  Combinations of muscle synergies in the construction of a natural motor behavior.  6(3):300–308.
  * [17] Cosimo Della Santina, Matteo Bianchi, Giuseppe Averta, Simone Ciotti, Visar Arapi, Simone Fani, Edoardo Battaglia, Manuel Giuseppe Catalano, Marco Santello, and Antonio Bicchi.  Postural hand synergies during environmental constraint exploitation.  11\.  ISSN 1662-5218.  URL https://www.frontiersin.org/articles/10.3389/fnbot.2017.00041.
  * Diamond and Holland [2014] A. Diamond and O. E. Holland.  Reaching control of a full-torso, modelled musculoskeletal robot using muscle synergies emergent under reinforcement learning.  _Bioinspiration & Biomimetics_, 9(1):016015, March 2014.  ISSN 1748-3190.  doi: 10.1088/1748-3182/9/1/016015.
  * Dominici et al. [2011] Nadia Dominici, Yuri P. Ivanenko, Germana Cappellini, Andrea d’Avella, Vito Mondì, Marika Cicchese, Adele Fabiano, Tiziana Silei, Ambrogio Di Paolo, Carlo Giannini, Richard E. Poppele, and Francesco Lacquaniti.  Locomotor primitives in newborn babies and their development.  _Science (New York, N.Y.)_ , 334(6058):997–999, November 2011.  ISSN 1095-9203.  doi: 10.1126/science.1210617.
  * [20] Fanny Ficuciello, Damiano Zaccara, and Bruno Siciliano.  Synergy-based policy improvement with path integrals for anthropomorphic hands.  In _2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 1940–1945.  doi: 10.1109/IROS.2016.7759306.  ISSN: 2153-0866.
  * Gray [1924] Henry Gray.  _Anatomy of the Human Body_.  Lea & Febiger, 1924.  Google-Books-ID: RcdqAAAAMAAJ.
  * Grillner [1985] S. Grillner.  Neurobiological bases of rhythmic motor acts in vertebrates.  _Science (New York, N.Y.)_ , 228(4696):143–149, April 1985.  ISSN 0036-8075.  doi: 10.1126/science.3975635.
  * Haarnoja et al. [2018] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine.  Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor.  In _Proceedings of the 35th International Conference on Machine Learning_ , pages 1861–1870. PMLR, July 2018.  URL https://proceedings.mlr.press/v80/haarnoja18b.html.  ISSN: 2640-3498.
  * Hart and Giszter [2010] Corey B. Hart and Simon F. Giszter.  A Neural Basis for Motor Primitives in the Spinal Cord.  _Journal of Neuroscience_ , 30(4):1322–1336, January 2010.  ISSN 0270-6474, 1529-2401.  doi: 10.1523/JNEUROSCI.5894-08.2010.  URL https://www.jneurosci.org/content/30/4/1322.  Publisher: Society for Neuroscience Section: Articles.
  * Ivanenko et al. [2004] Y. P. Ivanenko, R. E. Poppele, and F. Lacquaniti.  Five basic muscle activation patterns account for muscle activity during human locomotion.  _The Journal of Physiology_ , 556(1):267–282, 2004.  ISSN 1469-7793.  doi: 10.1113/jphysiol.2003.057174.  URL https://onlinelibrary.wiley.com/doi/abs/10.1113/jphysiol.2003.057174.  _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1113/jphysiol.2003.057174.
  * Johansson [1996] Roland S. Johansson.  19 - Sensory Control of Dexterous Manipulation in Humans.  In Alan M. Wing, Patrick Haggard, and J. Randall Flanagan, editors, _Hand and Brain_ , pages 381–414. Academic Press, San Diego, January 1996\.  ISBN 978-0-12-759440-8.  doi: 10.1016/B978-012759440-8/50025-6.  URL https://www.sciencedirect.com/science/article/pii/B9780127594408500256.
  * Kilbreath et al. [2002] S. L. Kilbreath, R. B. Gorman, J. Raymond, and S. C. Gandevia.  Distribution of the forces produced by motor unit activity in the human flexor digitorum profundus.  _The Journal of Physiology_ , 543(1):289–296, 2002.  ISSN 1469-7793.  doi: 10.1113/jphysiol.2002.023861.  URL https://onlinelibrary.wiley.com/doi/abs/10.1113/jphysiol.2002.023861.  _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1113/jphysiol.2002.023861.
  * Kormushev et al. [2013] Petar Kormushev, Sylvain Calinon, and Darwin G. Caldwell.  Reinforcement Learning in Robotics: Applications and Real-World Challenges.  _Robotics_ , 2(3):122–148, September 2013.  ISSN 2218-6581.  doi: 10.3390/robotics2030122.  URL https://www.mdpi.com/2218-6581/2/3/122.  Number: 3 Publisher: Multidisciplinary Digital Publishing Institute.
  * Kumar et al. [2014] Vikash Kumar, Yuval Tassa, Tom Erez, and Emanuel Todorov.  Real-time behaviour synthesis for dynamic hand-manipulation.  In _2014 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 6808–6815, May 2014.  doi: 10.1109/ICRA.2014.6907864.  ISSN: 1050-4729.
  * Kutsuzawa and Hayashibe [2022] Kyo Kutsuzawa and Mitsuhiro Hayashibe.  Motor synergy generalization framework for new targets in multi-planar and multi-directional reaching task.  _Royal Society Open Science_ , 9(5):211721, May 2022.  doi: 10.1098/rsos.211721.  URL https://royalsocietypublishing.org/doi/full/10.1098/rsos.211721.  Publisher: Royal Society.
  * Lee et al. [2015] Jong Hwa Lee, Deanna S. Asakawa, Jack T. Dennerlein, and Devin L. Jindrich.  Finger muscle attachments for an opensim upper-extremity model.  _PLOS ONE_ , 10(4):1–28, 04 2015.  doi: 10.1371/journal.pone.0121712.  URL https://doi.org/10.1371/journal.pone.0121712.
  * Levine et al. [2014] Ariel J. Levine, Christopher A. Hinckley, Kathryn L. Hilde, Shawn P. Driscoll, Tiffany H. Poon, Jessica M. Montgomery, and Samuel L. Pfaff.  Identification of a cellular node for motor control pathways.  _Nature Neuroscience_ , 17(4):586–593, April 2014\.  ISSN 1546-1726.  doi: 10.1038/nn.3675.
  * [33] Yinlin Li, Peng Wang, Rui Li, Mo Tao, Zhiyong Liu, and Hong Qiao.  A survey of multifingered robotic manipulation: Biological results, structural evolvements, and learning methods.  16\.  ISSN 1662-5218.  URL https://www.frontiersin.org/articles/10.3389/fnbot.2022.843267.
  * Luukinen et al. [2021] Patrick Luukinen, Olli V. Leppänen, and Jarkko Jokihaara.  The effect of digital sensory loss on hand dexterity.  _Journal of Hand Surgery (European Volume)_ , 46(3):253–259, March 2021.  ISSN 1753-1934.  doi: 10.1177/1753193420936598.  URL https://doi.org/10.1177/1753193420936598.  Publisher: SAGE Publications Ltd STM.
  * McFarland et al. [2022] Daniel C. McFarland, Benjamin I. Binder-Markey, Jennifer A. Nichols, Sarah J. Wohlman, Marije de Bruin, and Wendy M. Murray.  A musculoskeletal model of the hand and wrist capable of simulating functional tasks.  _IEEE Transactions on Biomedical Engineering_ , pages 1–12, 2022\.  doi: 10.1109/TBME.2022.3217722.
  * Mordatch et al. [2012] Igor Mordatch, Zoran Popović, and Emanuel Todorov.  Contact-invariant optimization for hand manipulation.  In _Proceedings of the ACM SIGGRAPH/Eurographics symposium on computer animation_ , pages 137–144, 2012.
  * OpenAI et al. [2019] OpenAI, Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, Jonas Schneider, Nikolas Tezak, Jerry Tworek, Peter Welinder, Lilian Weng, Qiming Yuan, Wojciech Zaremba, and Lei Zhang.  Solving Rubik’s Cube with a Robot Hand, October 2019.  URL http://arxiv.org/abs/1910.07113.  arXiv:1910.07113 [cs, stat].
  * [38] Simon A. Overduin, Andrea d’Avella, Jose M. Carmena, and Emilio Bizzi.  Microstimulation activates a handful of muscle synergies.  76(6):1071–1077.  ISSN 1097-4199.  doi: 10.1016/j.neuron.2012.10.018.
  * Plappert et al. [2018] Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, Vikash Kumar, and Wojciech Zaremba.  Multi-goal reinforcement learning: Challenging robotics environments and request for research, 2018.
  * Rabbi et al. [2020] Mohammad Fazle Rabbi, Claudio Pizzolato, David G. Lloyd, Chris P. Carty, Daniel Devaprakash, and Laura E. Diamond.  Non-negative matrix factorisation is the most appropriate method for extraction of muscle synergies in walking and running.  _Scientific Reports_ , 10(1):8266, May 2020.  ISSN 2045-2322.  doi: 10.1038/s41598-020-65257-w.  URL https://www.nature.com/articles/s41598-020-65257-w.  Number: 1 Publisher: Nature Publishing Group.
  * Raffin et al. [2021] Antonin Raffin, Jens Kober, and Freek Stulp.  Smooth Exploration for Robotic Reinforcement Learning, June 2021\.  URL http://arxiv.org/abs/2005.05719.  arXiv:2005.05719 [cs, stat].
  * Rajeswaran et al. [2018] Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine.  Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations, June 2018.  URL http://arxiv.org/abs/1709.10087.  arXiv:1709.10087 [cs].
  * [43] Eric Rombokas, Mark Malhotra, Evangelos A. Theodorou, Emo Todorov, and Yoky Matsuoka.  Reinforcement learning and synergistic control of the ACT hand.  18(2):569–577.  ISSN 1083-4435, 1941-014X.  doi: 10.1109/TMECH.2012.2219880.  URL http://ieeexplore.ieee.org/document/6341113/.
  * Rückert and d’Avella [2013] Elmar Rückert and Andrea d’Avella.  Learned parametrized dynamic movement primitives with shared synergies for controlling robotic and musculoskeletal systems.  _Frontiers in Computational Neuroscience_ , 7:138, October 2013.  ISSN 1662-5188.  doi: 10.3389/fncom.2013.00138.  URL https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3797962/.
  * Saito et al. [2018] Akira Saito, Aya Tomita, Ryosuke Ando, Kohei Watanabe, and Hiroshi Akima.  Muscle synergies are consistent across level and uphill treadmill running.  _Scientific Reports_ , 8(1):5979, April 2018.  ISSN 2045-2322.  doi: 10.1038/s41598-018-24332-z.  URL https://www.nature.com/articles/s41598-018-24332-z.  Number: 1 Publisher: Nature Publishing Group.
  * Salvietti [2018] Gionata Salvietti.  Replicating Human Hand Synergies Onto Robotic Hands: A Review on Software and Hardware Strategies.  _Frontiers in Neurorobotics_ , 12, 2018.  ISSN 1662-5218.  URL https://www.frontiersin.org/articles/10.3389/fnbot.2018.00027.
  * [47] Marco Santello, Martha Flanders, and John F. Soechting.  Postural hand synergies for tool use.  18(23):10105–10115.  ISSN 0270-6474, 1529-2401.  doi: 10.1523/JNEUROSCI.18-23-10105.1998.  URL https://www.jneurosci.org/content/18/23/10105.  Publisher: Society for Neuroscience Section: ARTICLE.
  * Saveriano et al. [2021] Matteo Saveriano, Fares J. Abu-Dakka, Aljaz Kramberger, and Luka Peternel.  Dynamic Movement Primitives in Robotics: A Tutorial Survey, February 2021.  URL http://arxiv.org/abs/2102.03861.  arXiv:2102.03861 [cs].
  * Schumacher et al. [2022] Pierre Schumacher, Daniel Häufle, Dieter Büchler, Syn Schmitt, and Georg Martius.  DEP-RL: Embodied Exploration for Reinforcement Learning in Overactuated and Musculoskeletal Systems, May 2022.  URL http://arxiv.org/abs/2206.00484.  arXiv:2206.00484 [cs].
  * Sobinov and Bensmaia [2021] Anton R. Sobinov and Sliman J. Bensmaia.  The neural mechanisms of manual dexterity.  _Nature Reviews Neuroscience_ , 22(12):741–757, December 2021.  ISSN 1471-0048.  doi: 10.1038/s41583-021-00528-7.  URL https://www.nature.com/articles/s41583-021-00528-7.  Number: 12 Publisher: Nature Publishing Group.
  * Song et al. [2022] Yunqing Song, Masaya Hirashima, and Tomohiko Takei.  Neural Network Models for Spinal Implementation of Muscle Synergies.  _Frontiers in Systems Neuroscience_ , 16, 2022.  ISSN 1662-5137.  URL https://www.frontiersin.org/articles/10.3389/fnsys.2022.800628.
  * Sutton and Barto [2018] Richard S. Sutton and Andrew G. Barto.  _Reinforcement learning: An introduction, 2nd ed_.  Reinforcement learning: An introduction, 2nd ed. The MIT Press, Cambridge, MA, US, 2018.  ISBN 978-0-262-03924-6.  Pages: xxii, 526.
  * [53] Juri Taborri, Valentina Agostini, Panagiotis K. Artemiadis, Marco Ghislieri, Daniel A. Jacobs, Jinsook Roh, and Stefano Rossi.  Feasibility of muscle synergy outcomes in clinics, robotics, and sports: A systematic review.  2018:1–19.  ISSN 1176-2322, 1754-2103.  doi: 10.1155/2018/3934698.  URL https://www.hindawi.com/journals/abb/2018/3934698/.
  * Tassa et al. [2012] Yuval Tassa, Tom Erez, and Emanuel Todorov.  Synthesis and stabilization of complex behaviors through online trajectory optimization.  In _2012 IEEE/RSJ International Conference on Intelligent Robots and Systems_ , pages 4906–4913, October 2012.  doi: 10.1109/IROS.2012.6386025.  ISSN: 2153-0866.
  * Todorov and Ghahramani [2004] Emanuel Todorov and Zoubin Ghahramani.  Analysis of the synergies underlying complex hand manipulation.  In _The 26th Annual International Conference of the IEEE Engineering in Medicine and Biology Society_ , volume 2, pages 4637–4640. IEEE, 2004.
  * Todorov et al. [2012] Emanuel Todorov, Tom Erez, and Yuval Tassa.  MuJoCo: A physics engine for model-based control.  In _2012 IEEE/RSJ International Conference on Intelligent Robots and Systems_ , pages 5026–5033, October 2012.  doi: 10.1109/IROS.2012.6386109.  ISSN: 2153-0866.
  * Tresch et al. [2006] Matthew C. Tresch, Vincent C. K. Cheung, and Andrea d’Avella.  Matrix factorization algorithms for the identification of muscle synergies: evaluation on simulated and experimental data sets.  _Journal of Neurophysiology_ , 95(4):2199–2212, April 2006.  ISSN 0022-3077.  doi: 10.1152/jn.00222.2005.
  * Tuthill and Azim [2018] John C. Tuthill and Eiman Azim.  Proprioception.  _Current Biology_ , 28(5):R194–R203, March 2018\.  ISSN 0960-9822.  doi: 10.1016/j.cub.2018.01.064.  URL https://www.sciencedirect.com/science/article/pii/S0960982218300976.
  * Ude et al. [2010] Aleš Ude, Andrej Gams, Tamim Asfour, and Jun Morimoto.  Task-Specific Generalization of Discrete and Periodic Dynamic Movement Primitives.  _IEEE Transactions on Robotics_ , 26(5):800–815, October 2010.  ISSN 1941-0468.  doi: 10.1109/TRO.2010.2065430.  Conference Name: IEEE Transactions on Robotics.
  * Wainwright [2002] Peter C. Wainwright.  The evolution of feeding motor patterns in vertebrates.  _Current Opinion in Neurobiology_ , 12:691–695, 2002.  ISSN 1873-6882.  doi: 10.1016/S0959-4388(02)00383-5.  Place: Netherlands Publisher: Elsevier Science.
  * Wang et al. [2011] T. Y. Wang, T. Bhatt, F. Yang, and Y. C. Pai.  Generalization of motor adaptation to repeated-slip perturbation across tasks.  _Neuroscience_ , 180:85–95, April 2011.  ISSN 0306-4522.  doi: 10.1016/j.neuroscience.2011.02.039.  URL https://www.sciencedirect.com/science/article/pii/S0306452211002090.
  * Yang et al. [2018] Chenguang Yang, Chao Zeng, Cheng Fang, Wei He, and Zhijun Li.  A DMPs-Based Framework for Robot Learning and Generalization of Humanlike Variable Impedance Skills.  _IEEE/ASME Transactions on Mechatronics_ , 23(3):1193–1203, June 2018.  ISSN 1941-014X.  doi: 10.1109/TMECH.2018.2817589.  Conference Name: IEEE/ASME Transactions on Mechatronics.
  * Zhang et al. [2021] He Zhang, Yuting Ye, Takaaki Shiratori, and Taku Komura.  Manipnet: neural manipulation synthesis with a hand-object spatial representation.  _ACM Transactions on Graphics (ToG)_ , 40(4):1–14, 2021.

##  Appendix A Appendix

###  A-A Real world objects set used in this study

We begin by presenting the complete set of real world object we used in our
experiments in Sec.VII-B.

Figure A.14: Eight objects used for RealWorldObjs data set (see Sec. VII-B).

###  A-B Ablations

We present a series of ablation experiments to understand how our policy and
representation design choices in the Reorient100 experiment impacted results.

####  A-B1 Relative contribution of synergies to the solution

Figure A.15: Post hoc modulation of weight assigned to SAR Analysis of the
relationship of $\varphi$ to the success rate. The value of $\varphi$ used for
the main experiment is depicted in orange.

First, we investigate the choice and sensitivity of our blend weight
($\varphi$, see Sec. IV-B) between the synergistic and non-synergistic
dimensions. We analyzed the contribution of SAR in terms of success rate in
Figure A.15. While an action contribution of SAR between 0.6 and 0.8 does not
change greatly the results, the solution greatly benefits from the use of SAR.
Indeed, choosing a very weak contribution or no contribution of the synergies
negatively impacts the overall performance, leading to success rate $<0.25$.
This result indicates that SAR is in fact utilized by our trained policy.

####  A-B2 Relative contribution of most vs least meaningful synergies

Figure A.16: Comparison SAR performance using more vs. less informative
synergies. Utilizing synergies that explain more variance in a muscle
activation dataset yields more robust policies. It should be noted that both
of these policies significantly outperform the two baselines tested in this
investigation (see Fig. 8).

We have demonstrated (Fig. 10) that during training, the earlier synergies
that explain most of the variance of the original signal have an outsized
contribution to the final policy. Nevertheless, a question arises whether any
choice of synergies produce the same results? Here, we tested the contribution
of the synergies that contribute most versus the ones that contributed the
least to the original tasks on which the SAR was built (see Appendix Fig.
A.17). Figure A.16 shows that when the least informative synergies for the
pre-training task were chosen to compute SAR, their ability to facilitate the
learning of a larger set of reorientation was heavily compromised, reaching
about half of the performance when more significant synergies were utilized
for SAR. Overall, this result is consonant with the observation of Figure 10
where the use of synergies that capture a disproportionate amount of the
information from the signal—i.e. first synergies have higher VAF (see Fig.
A.17)—also contribute a disproportionate amount to the solution.

###  A-C Dimensionality (N) of the Synergistic Action Representation (SAR)

Figure A.17: Proportion of variance in muscle activation data explained from
play phase policy as the number of computed muscle synergies increases.

For the purposes of this investigation, SAR is constructed from the first 20
synergies computed from the play phase described in Sec. IV-A. This quantity
of muscle synergies was selected because it balances (a) explaining a large
proportion of the variance ($>80\%$) in the initial activation data while
simultaneously (b) reducing the dimensionality of the original data by a
factor of 2 (see Fig. A.17). We also ablate in Fig. A.16 that training with
synergies that explain more variance in the the muscle activation data leads
to improved performance as compared to training with later synergies.

###  A-D Synergistic Action Representation vs random action representation

Figure A.18: Training performance when SAR is computed as described in Sec.
IV-A as compared to initializing a random representation.

Additionally, in order to further demonstrate the utility of the latent
representation captured by SAR, we compare performance between training with
SAR and training with randomized PCA and ICA component matrices that share the
same dimensions as SAR (see Fig. A.18). This analysis demonstrates that
randomized representations are ineffective for learning the multi-object
reorientation task, which further indicates that SAR per se has a
representational utility.

###  A-E Effect of blend weight ($\varphi$) on effectiveness of SAR

Figure A.19: Training performance at different values of $\varphi$. In bolded
red is $\varphi^{*}=.66$, the value used for the main experiment. We find that
as $\varphi$ is increased from this value (i.e., SAR is given more weight in
$a_{t}^{*}$), performance decrements slightly (0.7 → 0.5-0.6 success rate). As
$\varphi$ is decreased from this value (i.e., SAR is given less weight in
$a_{t}^{*}$), performance decrements significantly (0.7 → 0.1-0.4 success
rate).

An important consideration for our investigation was the optimal blend weight,
$\varphi^{*}$, for mixing synergistic and nonsynergistic activations for each
final action, $a_{t}^{*}$ (see Fig. 2). In Fig. A.19 we present an ablation
study involving various choices of blend weight. We observe that an
approximate ratio of $2/3$ SAR activations, $1/3$ task-specific activations
(i.e., $\varphi\approx.66$) facilitated best performance. More specifically,
we find that as as $\varphi$ is increased from this value and SAR is more
heavily weighted in $a_{t}^{*}$, performance decrements slightly (0.7 success
rate → 0.5-0.6 success rate). As $\varphi$ is decreased from this value and
SAR is less heavily weighted in $a_{t}^{*}$, performance decrements
significantly (0.7 success rate → 0.1-0.4 success rate) (see Fig. A.19).

This result can be thought as proactively searching over values of
$\varphi$—in other words, different values can be specified for $\varphi$
prior to training, and variation in performance can be compared. In contrast,
we can also consider retroactively searching over values of $\varphi$—in other
words, SAR-RL trained at some value (.66 in this case) of $\varphi$, but the
$\varphi$ passed into the trained policy can be modulated during test to
assess the impact of SAR on $a_{t}^{*}$ on the policy. This latter approach is
precisely what was done for SAR-RL in Fig. A.15.

Thus, comparing proactive and retroactive modulation of $\varphi$ yields a
similar result: training performance is best at $\varphi\approx.66$, decreases
slightly as $\varphi>.66$, and decreases significantly as $\varphi<.66$. This
result indicates that the combination of synergistic and nonsynergistic
activations leads to best performance (see Sec. IV-A), but that more heavily
weighting SAR within this mix is required for achieving this level of
performance. It should be emphasized that at the extremes, $\varphi=0$—i.e.,
training with no SAR—leads to poor performance (success rate $<0.2$), while
$\varphi=1$—i.e., training only with SAR—leads to significantly improved but
still suboptimal performance (success rate $\approx 0.6$) (see Fig. A.19).

###  A-F Parameter and hyperparameter selection

We present parameters utilized for the training and testing regimes of our
main experiments (see Secs. VII-B; V-C). We selected parameters for the X, Y,
and Z axes of the parametric objects used in Secs. VII-B and V-C by sampling
over ranges for each geometry that were conducive to in-hand manipulation
(i.e., excluding shapes that were too large to fit in-hand or too small to
properly manipulate). (see Table A.2; Fig. 3)

Additionally, we utilize hyperparameters for SAC (Table A.3) inpsired by
previous successful utilizations of this learning algorithm for continuous
control in robotic RL [41]. We also include a linearly-scheduled learning rate
to prevent unstable learning as the end of the training regime is approached.

TABLE A.2: Parameter distributions used for generating pretraining, in-domain, and out-of-domain parametric objects (see Sec. VII-B). Note that for out-of-domain samples, either a larger or smaller uniform distribution was first randomly selected from which to sample parameters for each specified axis. Geometry |  Pretraining |  In-domain |  Out-of-domain
---|---|---|---
Ellipsoid |  obj1=[0.011,0.025,0.025],
obj2=[0.019,0.040,0.040] |  $X\sim\mathcal{U}_{[0.008,0.02]}$,
$Y,Z\sim\mathcal{U}_{[0.020,0.045]}$ |  $X\sim\mathcal{U}_{[0.008,0.02]}$,
$Y,Z\sim\mathcal{U}_{[0.015,0.020]}\lor\mathcal{U}_{[0.045,0.050]}$
Box |  obj1=[0.017,0.017,0.017],
obj2=[0.023,0.023,0.023] |  $X,Y,Z\sim\mathcal{U}_{[0.015,0.025]}$ |  $X,Y,Z\sim\mathcal{U}_{[0.025,0.030]}\lor\newline \mathcal{U}_{[0.010,0.015]}$
Capsule |  obj1=[0.013,0.025,0.025],
obj2=[0.019,0.040,0.040] |  $X\sim\mathcal{U}_{[0.010,0.022]}$,
$Y,Z\sim\mathcal{U}_{[0.020,0.045]}$ |  $X\sim\mathcal{U}_{[0.010,0.022]}$,
$Y,Z\sim\mathcal{U}_{[0.015,0.020]}\lor\mathcal{U}_{[0.045,0.050]}$
Cylinder |  obj1=[0.013,0.025,0.025],
obj2=[0.019,0.040,0.040] |  $X\sim\mathcal{U}_{[0.010,0.022]}$,
$Y,Z\sim\mathcal{U}_{[0.020,0.045]}$ |  $X\sim\mathcal{U}_{[0.010,0.022]}$,
$Y,Z\sim\mathcal{U}_{[0.015,0.020]}\lor\mathcal{U}_{[0.045,0.050]}$
TABLE A.3: Hyperparameters used for training with SAC across all experiments.
The stable-baselines3 implementation of SAC was utilized for this
investigation (see Sec. III). Soft Actor-Critic (SAC) hyperparameters
---
Policy type |  MlpPolicy
Actor architecture |  $[400,300]$
Critic architecture |  $[400,300]$
Learning rate |  $linearschedule(.001)$
Start learning |  $t=3000$
Batch size |  256
$\tau$ |  .02
$\gamma$ |  .98