# APriCoT: Action Primitives based on Contact-state Transition for In-Hand
Tool Manipulation

Daichi Saito1,2, Atsushi Kanehira2, Kazuhiro Sasabuchi2, Naoki Wake2,
Jun Takamatsu2, Hideki Koike1 and Katsushi Ikeuchi2 1Authors are with
Department of Computer Science, Tokyo Institute of Technology, Tokyo, Japan,
<EMAIL_ADDRESS>are with Applied Robotics Research,
Microsoft, Redmond, WA, USA

###### Abstract

In-hand tool manipulation is an operation that not only manipulates a tool
within the hand (i.e., in-hand manipulation) but also achieves a grasp
suitable for a task after the manipulation. This study aims to achieve an in-
hand tool manipulation skill through deep reinforcement learning. The
difficulty of learning the skill arises because this manipulation requires (A)
exploring long-term contact-state changes to achieve the desired grasp and (B)
highly-varied motions depending on the contact-state transition. (A) leads to
a sparsity of a reward on a successful grasp, and (B) requires an RL agent to
explore widely within the state-action space to learn highly-varied actions,
leading to sample inefficiency. To address these issues, this study proposes
Action Primitives based on Contact-state Transition (APriCoT). APriCoT
decomposes the manipulation into short-term action primitives by describing
the operation as a contact-state transition based on three action
representations (detach, crossover, attach). In each action primitive, fingers
are required to perform short-term and similar actions. By training a policy
for each primitive, we can mitigate the issues from (A) and (B). This study
focuses on a fundamental operation as an example of in-hand tool manipulation:
rotating an elongated object grasped with a precision grasp by half a turn to
achieve the initial grasp. Experimental results demonstrated that ours
succeeded in both the rotation and the achievement of the desired grasp,
unlike existing studies. Additionally, it was found that the policy was robust
to changes in object shape.

##  I Introduction

In-hand tool manipulation [1, 2] is an operation that not only manipulates a
tool within the hand (i.e., in-hand manipulation) but also achieves a grasp-
type [3] suitable for a task after the manipulation. Since the inappropriate
grasp may cause the tool not to function as expected and task failure,
realizing a suitable grasp through the manipulation is crucial for the
subsequent task. For instance, when shaking a box with fingers to pour its
contents into a cup, we should ensure an opening of the box faces the cup and
the box is held with a precision grasp (Figure 1-(A)). If the object is not
properly grasped, the opening is blocked by fingers (Figure 1-(B)), or it is
impossible to change the object orientation (Figure 1-(C)). The in-hand tool
manipulation skill is essential for a dexterous operation with various types
of tools in daily life. The acquisition of the ability on a single hardware is
necessary for realizing a general-purpose robot.

Figure 1: An example of a task after in-hand tool manipulation: pouring the
contents of a box into a cup by shaking it with fingers. To orient an opening
of the box towards the cup, the hand must rotate the box counterclockwise (red
arrow in the second image from the left). To successfully complete this task,
the hand should grasp the object as shown in (A) after manipulation. If the
object is grasped as in (B) or (C), issues arise such as the fingers blocking
the opening or the inability to properly change the object orientation.

The objective of this study is to achieve the manipulation skill through deep
reinforcement learning (DRL). DRL allows for the acquisition of policies
robust to various uncertainties, such as object shape variations and observed
joint angle errors, making it the de facto standard in the field of in-hand
manipulation [4, 5, 6, 7, 8, 9].

Although DRL has the potential to facilitate in-hand tool manipulation,
learning this skill remains difficult. The difficulty arises from the need for
(A) long-term contact-state changes and (B) highly-varied motions depending on
the contact-state transition for the manipulation. With regard to (A),
fingertip positions relative to the object must change many times to achieve
the desired grasp, involving long-term contact-state transitions. As for (B),
required motions vary depending on current fingertip positions. For example,
in the left situation of Figure 1, if the ring finger moves to the opposite
side of the object, the others must adjust the object orientation as shown in
the lower of Figure 2. Conversely, when one of the others moves to the
opposite, actions of this finger significantly differ from those in the lower
of Figure 2.

When learning an in-hand tool manipulation skill using DRL, two main issues
corresponding to (A) and (B) arise. The first issue, due to (A), is a sparsity
of a reward for a successful grasp over time. This sparsity makes it hard to
map the reward to previous actions predicted by a policy and thus the learning
becomes difficult. The second issue, stemming from (B), is that an RL agent
must widely explore within the state-action space to learn the highly-varied
actions. As a result, the number of training samples to find the optimal
policy significantly increases. A possible solution for the sparsity is adding
a dense reward such as one on a discrepancy between the joint angles of the
current hand configuration and those of the target grasp [8, 9]. However, this
reward limits the exploration within the state-action space and further
worsens sample efficiency. Thus, learning manipulation characterized by both
the reward sparsity and sample inefficiency presents significant challenges.

These issues can generally be solved by decomposing a long-term motion into
short-term motions [10, 11, 12, 13]. Although in-hand tool manipulation
appears to involve long-term and complex motions overall, it consists of a
combination of three abstract action representations to describe contact-state
changes: detach, crossover, and attach [14]. Detach is to move a finger away
from the object. Crossover is a motion where a finger changes from one side of
the object to the other side of the object. Attach is to bring a finger into
contact with the object (Figure 2). The desired grasp can be defined as the
contact-state [15], and it is reasonable to regard in-hand tool manipulation,
which is the operation to change the grasp, as the contact-state transition.
Thus, we can describe the entire manipulation by combining the
representations, and decompose the manipulation into short-term motions based
on this description. Such temporal decomposition can alleviate the issue due
to the characteristic (A). In addition, we can mitigate the problem from the
characteristic (B) by spatially decomposing so that the fingers behave
similarly within each motion and separately training the policy for each
motion.

Figure 2: The explanation of action representations and primitives. I, M, R,
and T represent the index finger, middle finger, ring finger, and thumb,
respectively. This figure shows an example of a contact-state transition in
case of rotating the box counterclockwise. Detach, crossover, and attach are
the action representations to transition the contact-state. The initial
contact-state of a primitive is set to the most stable one where all fingers
are in contact with the object.

This study proposes the learning of in-hand tool manipulation using Action
Primitives based on Contact-state Transition (APriCoT). By sequentially
executing the action primitives, we achieve the manipulation. That is, we
consider the transitions to achieve the desired grasp, decompose the
manipulation into short-term primitives based on the transition, and introduce
the learning of each action primitive. This study defines the contact-state
based on not only the arrangement of fingers but also grasp stability [16] and
manipulability. In addition, the initial and final contact-states of each
primitive are set to the most stable one where all fingers are in contact with
the object. In this case, the primitives make transitions between these
contact-states by the series of detach-crossover-attach. Since the most stable
contact-state is less affected by dynamics such as object velocity and the
friction forces between the object and the fingers, this contact-state is
easily realized. Thus, this state can be stably sampled in simulations, which
improves the efficiency of initial-state sampling. Additionally, training the
policy to learn a set of representations rather than each individually
enhances the ability of the policy to perform a smooth motion, thereby
promoting stable and successful manipulation. The policies are trained using a
step-wise reward, promoting execution in the detach-crossover-attach order.
This reward can be used for all primitive policies.

We target an example of in-hand tool manipulation where the hand rotates an
elongated object half a turn to achieve a precision grasp, as shown in Figure
1-(A). This operation has long-horizon contact-state transition, and is
frequently utilized for daily-life tools including wrenches and cups with
lids. Simulation experiments using a multi-fingered hand demonstrated that
existing methods focusing solely on in-hand manipulation or targeting
pinchable objects fail to achieve the desired grasp or fail the rotation
itself. In contrast, ours achieves the desired grasp after the rotation with a
$90\%$ success rate for objects of various shapes.

The contributions of this study are as follows:

  * •

Proposed APriCoT for learning the in-hand tool manipulation skill.

  * •

Introduced reward design focusing on detach, crossover, and attach, which is
reused for training all primitive policies.

  * •

Demonstrated that our method is effective for the manipulation with long-term
contact-state changes and highly-varied actions.

Although this study targets primitives for four contact-state transitions, our
method can be easily applicable to various operations by increasing the number
of reusable primitives. For instance, transitioning from a precision grasp to
a power grasp [3] after rotating an object half a turn and vice versa becomes
feasible. Even with more primitives, the reward design remains reusable,
avoiding significant increases in reward-design costs. Thus, our method is
potentially applicable to other operations. This study is a first step toward
realizing diverse in-hand tool manipulation through the combination of
primitives.

##  II Related Work

In-hand manipulation has been extensively researched over the past few decades
[17]. Early studies relied on precise models of a robot hand and an object for
motion planning [18, 19, 20, 21]. However, uncertainties of the models in real
limits their practical applicability. To address these challenges, this study
utilizes DRL to train the policy to learn object manipulation robust to the
uncertainties.

Advancements in DRL [22] have accelerated the policy training across a wide
range of domains, including robot locomotion, grasping, and object
manipulation [23, 24, 25]. The field of in-hand manipulation has also adopted
DRL [4, 5, 6, 7, 8, 9]. In-hand tool manipulation requires not only in-hand
manipulation but also the achievement of the desired grasp after the
manipulation. Previous research has focused on acquiring the former skill,
such as rotating various shaped objects [4, 5, 6, 7, 26, 8, 9, 27, 28, 29].
These studies have primarily designed a reward only concentrating on in-hand
rotation, and not ensured realizing the desired grasp. Although there are
several studies that focus on learning in-hand tool manipulation [1, 2], they
require precise models of a hand and object, making real-world application
difficult. The research by Qi et al. [8, 9] potentially has the ability to
maintain a precision grasp after rotating pinchable objects, however, their
method is limited to operations with a narrow state-action space, failing to
achieve an operation that requires exploring a broader space. This study
addresses such limitations by decomposing the manipulation into several
primitives and by separately training each primitive policy. This approach
simplifies the acquisition of the manipulation.

Existing researches enable the learning of long-horizon tasks by decomposing
them into shorter motions [30, 10, 11, 12, 13, 31]. For example, some studies
have segmented a sequence of tasks from grasp to manipulation, and designed
policies or controllers for each motion [10, 13, 31]. Other research has
divided in-hand manipulation in a 2-dimensional space into multiple actions,
and designed controllers for each action assuming the known model [12]. This
study focuses on in-hand tool manipulation in a 3D space. We consider short-
term action primitives for the manipulation and design the learning for these
primitives.

##  III Method

This study aims to achieve in-hand tool manipulation skills using DRL. The
primary challenges in learning this kind of manipulation are:

  * •

The necessity for long-term exploration to achieve the desired grasp.

  * •

The highly-varied actions depending on the contact-state transition,
necessitating the exploration of a wide state-action space.

To address these challenges, we propose Action Primitives based on Contact-
state Transition (APriCoT). The operation is decomposed into several
primitives so that the fingers are required to perform short-term and similar
motions within each primitive. This temporal and spatial decomposition
mitigates these issues. We achieve the manipulation by sequentially executing
primitive policies.

###  III-A Contact-state transition

To decompose in-hand tool manipulation, we design the contact-state
transition-graph of the manipulation. This graph has contact-states as nodes
and contact-state transitions as edges. We construct a transition-graph based
on three action representations: detach, crossover, and attach. Before the
construction, we design the contact-state in the graph. This study defines the
contact-state based on not only the finger arrangement but also the grasp
stability and manipulability. The stability is evaluated with the closure
[16]. The manipulability ensures that the fingers do not limit each other’s
range of motion after the transition. Specifically, regarding the grasp
stability, we require that a prehensile grasp is achieved and that at least
three fingers are in contact with the object. As for the manipulability, the
fingers must be positioned clockwise in the order of thumb, index, middle, and
ring finger to avoid entanglement of the fingers. The graph is then
constructed by connecting the contact-states that can transition through
detach, crossover, and attach motions. Under these conditions, the transition
of contact-states for the targeted manipulation is shown in Figure 3.
Initially, the ring finger detaches from the object, performs crossover, and
attaches to change the contact-state. Next, the same motions are performed
sequentially by the thumb, middle, and index fingers to achieve the desired
grasp.

Figure 3: The contact-state transitions in the targeted manipulation. Each
section enclosed by dashed lines represents an action primitive.

We set the initial and final contact-states of each primitive as the most
stable one where all fingers contact the object. This stability, being less
affected by dynamics such as object velocity and friction forces, makes it
easier to achieve such contact-state. Thus, the contact-state can be stably
sampled in simulations, which improves the efficiency of initial-state
sampling. By training a policy to learn a set of representations rather than
each individually, we can promote the ability of the policy to perform a
smooth motion. The smoothness leads to stable and successful manipulation. For
instance, learning only the detach may cause the finger to move excessively
away from the object, adding unnecessary motion before the crossover. Whereas,
training the policy to execute the detach-crossover-attach sequence enables
actions based on subsequent representation, which leads to avoiding the
learning of the unnecessary motion. By the decomposition, four policies are
required: Policies A, B, C, and D as shown in Figure 3. Note that, although a
reverse transition also exists, the policies corresponding to this transition
can be trained using the proposed training method.

###  III-B Formulation of training

We formulate the policy training as a Markov Decision Process (MDP). Primitive
policies corresponding to each transition are learned, and the number of
policies is denoted as $N$. For each transition $i$ $(1\leq i\leq N)$, the MDP
is defined as follows. The MDP has the state
$\bm{s}^{(i)}\in\mathcal{S}^{(i)}$ (where $\mathcal{S}^{(i)}$ is the state
space), action $\bm{a}^{(i)}\in\mathcal{A}^{(i)}$ (where $\mathcal{A}^{(i)}$
is the action space), state transition
$\mathcal{T}^{(i)}:\mathcal{S}^{(i)}\times\mathcal{A}^{(i)}\rightarrow\mathcal{S}^{(i)}$,
initial state distribution $\rho_{0}^{(i)}$, and reward function
$r^{(i)}:\mathcal{S}^{(i)}\times\mathcal{A}^{(i)}\rightarrow\mathbb{R}$. Our
objective is to find the policy
$\pi_{\theta}^{(i)}(\bm{a}^{(i)}|\bm{s}^{(i)})$ that maximizes the cumulative
reward
$J(\pi_{\theta}^{(i)})=\mathbb{E}_{\pi_{\theta}^{(i)}}\left[\sum_{t=0}^{T-1}\gamma^{t}r^{(i)}\left(\bm{s}^{(i)}_{t},\bm{a}^{(i)}_{t}\right)\right]$
for a given discount factor $\gamma\in[0,1)$. Here, $T$ denotes the episode
length, $\bm{s}^{(i)}_{0}\sim\rho_{0}^{(i)}$,
$\bm{a}^{(i)}_{t}\sim\pi_{\theta}^{(i)}(\bm{s}_{t})$, and
$\bm{s}^{(i)}_{t+1}=\mathcal{T}^{(i)}\left(\bm{s}^{(i)}_{t},\bm{a}^{(i)}_{t}\right)$.
$\pi_{\theta}^{(i)}$ is parameterized by a weight $\theta$ of a neural
network.

The target manipulation is achieved by sequentially executing the policies
$\pi^{(i)}$. At this time, the initial state when switching policies is given
by
$\bm{s}^{(i+1)}_{0}=\mathcal{T}^{(i)}\left(\bm{s}^{(i)}_{T-1},\bm{a}^{(i)}_{T-1}\right)$.

###  III-C Training design

First, we explain the states and actions required to implement APriCoT. After
that, we describe the proposed reward design, which can be commonly reused for
all primitives, and initial-state distribution. Subsequently, we explain the
overview of the training process. To train the policy efficiently, this study
adopts the framework of teacher-student learning. The implementation of
teacher-student learning follows the existing research [8].

####  III-C1 State, Action design

Toward the application of policies in the real-world, it is desirable to train
policies that can adapt to various-shaped objects. For the training,
recognizing the shape from observed information is required. In fact, the
policy can implicitly utilize the shape information from the difference
between the commanded and actual joint angles. Therefore, the state
$\bm{s}_{t}$ is designed to include the commanded joint angles $\bm{q}_{t}$
and the actual joint angles $\hat{\bm{q}}_{t}$. The action $\bm{a}_{t}$ is set
as the change in the commanded joint angles $\Delta\bm{q}_{t}$.

####  III-C2 Reward design

We design a reward to be reusable for all training of primitive policies. In
addition, to promote execution in the appropriate order, we adopt a step-wise
reward structure, where the reward incrementally increases as each
representation is achieved. Equation (1) represents the reward which is
decomposed into terms $r_{\rm transition}$ related to conact-state
transitions, and terms $r_{\rm obj}$ related to maintaining the object’s
position and orientation.

$\begin{split}r=r_{\rm transition}+r_{\rm obj}\end{split}$ (1)

The term $r_{\rm transition}$ is expressed by equation (2).

$\begin{split}r_{\rm transition}=w_{\rm det}r_{\rm det}+r_{\rm X}+w_{\rm
att}r_{\rm att}\end{split}$ (2)

The components $r_{\rm det},r_{\rm X},r_{\rm att}$ constituting $r_{\rm
transition}$ depend on the contact-state of finger $F$ moving to the opposite
side of the object. $r_{\rm det}$ provides a reward to encourage detach until
the finger crosses over the object, where $r_{\rm det}=-d^{z}_{F}$, and
becomes $0$ after the crossover ($d^{z}_{F}$ is the distance from the object’s
upper surface to the finger). $r_{\rm X}$ incentivizes crossover motion. This
term is a positive constant reward $c_{\rm X1}$ when the finger spans above
the object’s upper surface; and $c_{\rm X1}+c_{\rm X2}$ when the finger spans
below the upper surface; and $0$ when the finger does not cross over the
object; $r_{\rm att}$ encourages attachment. It is $-d^{x}_{F}$ when the
finger spans below the object’s upper surface and becomes $-d^{x}_{F}+c_{\rm
att}$ upon achieving the desired contact-state, otherwise $0$ ($d^{x}_{F}$ is
the distance from the object’s surface to the finger). Continuous functions
for $r_{\rm det}$ and $r_{\rm att}$ increase reward granularity to enhance
training efficiency. $w_{\rm det},w_{\rm att}$ are weights of $r_{\rm
det},r_{\rm att}$, respectively.

$r_{\rm obj}$ is represented by equation (3).

$\begin{split}r_{\rm obj}=w_{\rm dir}r_{\rm dir}+w_{\rm rot}r_{\rm rot}+w_{\rm
pos}r_{\rm pos}+r_{\rm term}\end{split}$ (3)

$r_{\rm dir}$ promotes finger-object surface contact for grasp stability. It
is $-\sum_{f\neq F}\theta_{f}$ until the finger spans the object, and
$-\sum\theta_{f}$ afterward, where $\theta_{f}$ is the angle between the
direction pointed by finger $f$ and the direction perpendicular to the
object’s upper surface. $r_{\rm rot},r_{\rm pos}$ prevent changes in the
object’s position and rotation around axes other than the yaw axis from the
initial state. Here, $r_{\rm rot}=-(\theta^{\rm roll}+\theta^{\rm pitch})$,
$r_{\rm pos}=-\lVert\bm{p}-\bm{p}^{\rm init}\lVert_{2}$. $\theta^{\rm
roll},\theta^{\rm pitch}$ are rotation angles around axes penetrating from
behind to front and from side to side of the object respectively; $\bm{p}$ is
the object’s position in the global coordinate system; $\bm{p}^{\rm init}$ is
the initial position in the global coordinate system. $r_{\rm term}$ prevents
early termination. The termination occurs when improving the reward is almost
impossible. By the termination, we can enhance the training efficiency. It is
$c_{\rm term}$ if the episode hasn’t terminated, and $-c_{\rm term}$
otherwise. The termination occurs when the object falls from the hand, when
finger-surface contact cannot be maintained ($\sum\theta_{f}>\Theta_{1}$), or
when the object’s orientation significantly deviates ($\theta_{\rm
roll}>\Theta_{2}$). $w_{\rm dir},w_{\rm rot},w_{\rm pos}$ are weights of the
reward terms.

####  III-C3 Initial-state design

During the training, diverse initial states are crucial for the robustness of
policies. Furthermore, from the perspective of enhancing the overall
performance of policies in skill chaining, aligning the input distribution of
policies with the output distribution of preceding stages is important [30,
11]. Therefore, we initially train a policy using a large set of initial
states generated through simulation. Subsequently, in subsequent policies, we
use the states outputted by the preceding policy as initial states (Figure 4
right).

The generation of initial states through simulation follows the following
steps:

  1. 1.

Add noise sampled from a uniform distribution to the initial pose of objects,
as well as to joint commands.

  2. 2.

Advance the simulation for a sufficient number of timesteps ($50$ iterations).
During the simulation, we input the same joint commands.

  3. 3.

Select initial states where the grasp is maintained as candidates for
adoption.

Such collected initial states, generated through these steps, are utilized for
the training of Policy A.

Figure 4: Training overview. (A) explains the initial states used for
training. The initial states of Policy B, C, D are the output states of Policy
A, B, C, respectively. (B) illustrates the steps of teacher-student learning.
The teacher policy takes observed information $\bm{o}_{t-l_{1}:t}$ and a
latent variable $\bm{z}_{t}$, encoded from privileged information $\bm{h}_{t}$
by the encoder $\mu$, as input to predict an action $\bm{a}_{t}$. In contrast,
the student policy uses $\bm{o}_{t-l_{1}:t}$ and a latent variable
$\hat{\bm{z}}_{t}$, with $\hat{\bm{z}}_{t}$ being encoded from
$\bm{o}_{t-l_{2}:t}$ by the encoder $\phi$.

####  III-C4 Implementation

The training utilizes the framework of teacher-student learning as in existing
studies [32, 23, 8, 9]. Training a policy from scratch based solely on the
observed information is difficult due to the limited object-related
information. Training with only the observed information needs a large amount
of samples to map the observed with object information, resulting in sample
inefficiency. To mitigate the sample inefficiency, the framework is adopted.
In teacher-student learning, a teacher policy is first trained using the
privileged and observed ones. Subsequently, the student policy is trained
using only the observed, guided by the teacher policy. This approach
facilitates efficient learning. The teacher and student policy consist of a
common policy $\pi$ and different encoders $\mu$ and $\phi$ (Fig. 4-(B)). The
encoders $\mu,\phi$ encode the privileged and the multiple observed ones into
latent variables $\bm{z}_{t},\hat{\bm{z}}_{t}$, respectively. During the
training of the student policy, the objective is to minimize the following
$L2$ loss function $L$, encouraging the encoders $\phi$ to align these latent
variables:

$L=\|\bm{z}_{t}-\hat{\bm{z}}_{t}\|_{2}^{2}$ (4)

This facilitates learning the correspondence between the privileged and the
observed information.

The privileged information $\bm{h}_{t}$ has object information including the
pose, shape, friction coefficients, and binary-contact information between
each finger link and the object. To represent the shape, parameters of
superquadrics [33] are used. The observed information $\bm{o}_{t}$ includes
commanded joint angles $\bm{q}_{t}$, measured values $\hat{\bm{q}}_{t}$, and
the action $\bm{a}_{t-1}$ from the previous timestep. Action $\bm{a}_{t}$
includes the change in commanded joint angles $\Delta\bm{q}_{t}$. In the
teacher policy, privileged information $\bm{h}_{t}$ is input to an encoder
$\mu$ consisting of fully connected layers, which outputs latent variable
$\bm{z}_{t}$. Subsequently, $\bm{s}_{t}=\\{\bm{z}_{t},\bm{o}_{t-l_{1}:t}\\}$
is input to policy $\pi$ consisting of fully connected layers, which outputs
action $\bm{a}_{t}$. In the student policy, $\bm{o}_{t-l_{2}:t}$ is input to
an encoder $\phi$ constructed with a Temporal Convolutional Network [34],
which outputs $\hat{\bm{z}}_{t}$. Then,
$\bm{s}_{t}=\\{\hat{\bm{z}}_{t},\bm{o}_{t-l_{2}:t}\\}$ is input to $\pi$,
which outputs action $\bm{a}_{t}$. This study uses $l_{1}=2,l_{2}=29$.

Toward the application of the policies in the real world, this study
randomizes object shapes, friction coefficients, and measured joint angles.
Object shape and friction coefficients are varied when initializing
environments. The shape parameters are varied using the parameters of
superquadrics. Observed joint angles are randomized by adding noise sampled
from a uniform distribution at each timestep.

##  IV Experiment

###  IV-A Settings

To rapidly collect a large amount of data, we utilized the IsaacGym Simulator
[35] for training. During the training process, data collection was achieved
by running $32,768$ environments in parallel on NVIDIA RTX3090 GPU. Each
environment included a four-fingered hand, specifically the Shadow Dexterous
Hand Lite111https://www.shadowrobot.com/dexterous-hand-series/, and a various-
shaped object. The simulation frequency and control frequency were set at
$120$ Hz and $20$ Hz, respectively. Each policy was trained for a maximum of
$T=100$ iterations per episode. The parameter range utilized for the
randomization of object shape, friction coefficients, and observed joint
angles are shown in Table I. The parameters of the superquadrics include an
object depth, width, height, $\epsilon_{1}$, and $\epsilon_{2}$.
$\epsilon_{1}$ and $\epsilon_{2}$ determine the shape, with $\epsilon_{2}$
altering the shape of top-surface. As $\epsilon_{2}$ approaches $0$, the shape
becomes more rectangular, while it turns more elliptical as $\epsilon_{2}$
approaches $1$. The policy $\pi$ and the encoder $\mu$ multi-layer perceptrons
(MLP) with hidden layers consisting of $[512,256,128,64]$ and $[256,64]$
units, respectively. We use the exponential linear unit (ELU) [36] as the
activation function for these networks. The structure of encoder $\phi$ is the
same as used in [8]. The Proximal Policy Optimization (PPO) algorithm [37] was
used for training the teacher policy. The student policy was optimized using
the Adam optimizer [38]. The hyperparameters for reward and early termination
and reward coefficients used for each policy were set as shown in Table II.

TABLE I: The range of parameter randomization in the experiment

Parameter Range Object depth $0.02,0.0275,0.35$ [m] Object width $0.11$ [m]
Object height $0.1$ [m] Object shape ($\epsilon_{1}$) $\simeq 0$ Object shape
($\epsilon_{2}$) $\simeq 0,0.5,1$ Friction coefficients uniform([$0.7,1.3$])
Joint noise uniform([$-0.05,0.05$]) [rad]

TABLE II: Hyperparameters of reward and early termination.

$c_{\rm X1}$ $c_{\rm X2}$ $c_{\rm att}$ $c_{\rm term}$ $\Theta_{1}$
$\Theta_{2}$ A,B,C 0.1 0.2 1 0.1 $120^{\circ}$ $15^{\circ}$ D 0.2 0.5 0.5 0.1
$120^{\circ}$ $15^{\circ}$

$w_{\rm det}$ $w_{\rm att}$ $w_{\rm dir}$ $w_{\rm rot}$ $w_{\rm pos}$ A,B,C
0.1 0.1 0.1 0.1 0.01 D 2 1 0.1 0.1 0.01

To demonstrate the effectiveness of the proposed method for the target
manipulation, we compared it against the following two baseline methods in the
experiments:

  1. 1.

Baseline A: A modified version of the method that targets only the rotation of
objects within the hand [4]. A positive reward given when realizing the grasp
was simply added to the original reward. By comparing this baseline with ours,
we verify that the intended grasp can be achieved more effectively by
considering the transitions and decomposing into the primitives.

  2. 2.

Baseline B: A method designed for pinchable objects [8]. This baseline was
compared with ours to investigate that learning the manipulation requires a
wide exploration, and that the decomposition into short-term primitives makes
learning feasible. In this method, a negative reward is added for changes in
joint angles, encouraging the rotation of the object without significantly
altering the joints. As a result, this method can rotate objects while
maintaining the desired grasp only for pinchable objects.

###  IV-B Results

####  IV-B1 Comparison

Figure 5 shows the training results of the baselines and the proposed method.
The success is defined as achieving the target grasp after rotating the object
by half a turn at the final timestep. The baselines failed to train a policy
achieving both the in-hand half-rotation and the desired grasp. In Baseline A,
the rotation was achieved, but the intended grasp was not realized. This is
because a long step results in sparse rewards on the successful grasp. In
Baseline B, it rotated the object around the roll direction for crossovers,
reducing the contact area and causing the object to drop due to the inability
to maintain the grasp. This is caused by limiting the exploration of joint
angles through the reward. This limitation makes it difficult to perform
crossover while preventing the object rotation around the roll direction. In
contrast, our method successfully achieved the desired grasp after rotating
the object. We believe that designing the primitives based on the contact-
state transition helped achieve the target grasp and the decomposition
simplified the exploration of the actions to perform crossover.

Figure 5: The example of results. (A):Baseline A, (B):Baseline B, (C):Ours. At
the final timestep, (A) achieved to rotate the object but the intended grasp
is not realized. In (B), the object is fallen. On the other hand, (C) achieved
the rotation and desired grasp.

####  IV-B2 Robustness

To evaluate the robustness of the proposed policies, this study assessed the
success rates of operations for different object shapes. The depth and
$\epsilon_{2}$ are selected from $\\{0.02,0.023,0.026,0.029,0.032,0.035\\}[m]$
and $\\{\simeq 0,0.25,0.5,0.75,1\\}$, respectively. We conducted the
evaluation by randomly sampling $1,000$ initial states and calculating the
success rates for each shape. The success rates for each shape are presented
in Figure 6. We achieved success rates of over $90\%$ for almost all shapes.
The results indicate that our policies are robust to changes in object shape.
The success rate with the unused shapes in the training was almost the same as
with the used shapes. For objects with a large depth and a high $\epsilon_{2}$
(thicker ellipsoids), the success rate was around $75\%$. This lower success
rate is attributed to the increased curvature, making the objects more prone
to slipping sideways during manipulation.

Figure 6: Success rate. The vertical axis represents the success rate for each
object shape. The axes of the base plane represent the depth and
$\epsilon_{2}$ of the used objects.

####  IV-B3 Visualization of latent variables

To verify whether the features of different object shapes were learned, we
visualized the distribution of latent variables $\bm{\hat{z}_{t}}$ using t-SNE
[39] (Figure 7). The depth and $\epsilon_{2}$ are chosen from
$\\{0.02,0.0275,0.035\\}[m]$ and $\\{\simeq 0,0.5,1\\}$, respectively. The
latent variables were collected by running Policy A for $100$ iterations. It
was observed that points corresponding to similar shapes were clustered
closely together. This suggests that the features of the object shapes were
learned by the policy, resulting in the policy being robust to changes in the
shapes.

Figure 7: Visualization of latent variables from encoder $\phi$ using t-SNE.
The points represent variables collected during policy execution with six
objects. The object depth and $\epsilon_{2}$ are written below each image.
Distinct colors correspond to different objects. As seen in arrowed areas,
points with the same color are clustered closely together.

##  V Discussion

###  V-A Summary of the experiments

This study proposed APriCoT to execute in-hand tool manipulation. Experiments
showed that while the baselines could not achieve both in-hand rotation and
the desired grasp, our method successfully achieved the target grasp after the
rotation. Even with the known contact-state after manipulation, existing
methods struggle with long-term operations. Our approach simplifies learning
by decomposing manipulation based on the transition. By this decomposition, we
can train the policy to learn each primitive where fingers need to just
perform short-term and similar motions, which improves sample efficiency. In
addition, the experiment demonstrated that the policies were robust to the
changes in the object shape. This robustness is important to apply the
policies in the real-world where various-shaped objects exist.

Visualization of latent variables suggests the policy implicitly recognizes
the object shape from joint angles and adapts actions accordingly. However,
information of object pose, such as roll rotation, cannot be detected solely
from joints. Failure to detect this rotation can destabilize and drop the
object. In addition, for tasks with a goal pose of the object, precisely
achieving the goal after the manipulation is challenging. Focusing solely on
contact-state transitions, as in this study, does not ensure the desired
orientation. Therefore, policies must adjust actions based on observations
recognizing object orientation. In the future, we integrate the existing
methods using visual information [4, 5, 9] with ours to deal with this issue.

Although we conducted only simulation experiments, we believe that the
proposed policies are applicable in the real-world. Following previous
research that demonstrated feasibility in the real, we employed domain
randomization and the framework of teacher-student learning for adaptation
during policy training. These techniques might help bridge the sim-to-real
gap.

###  V-B Realizing diverse manipulation using reusable primitives

This study designed action primitives for rotating an elongated object to
achieve a precision grasp. Various grasp-types exist other than a precision
grasp, and achieving such diverse grasps by in-hand tool manipulation requires
many policies. An example of these operations is achieving a power grasp by
moving the object toward the palm after the manipulation which was dealt with
in this study. Common contact-state transitions occur in various manipulation.
Policies for specific transitions should be reusable across operations. By
focusing on reusability a wide range of operations could be achieved, reducing
the number of required primitives. This study is a first step toward enabling
such various operations by leveraging action-primitive reusability.

This study executed primitives based on the contact-state transition-graph.
Manually designing the graph for all operations is labor-intensive. This could
be mitigated by automating graph construction via methods like Learning-from-
Observation, which uses human demonstrations for task-sequence construction
[40, 41, 42, 43]. Additionally, we can deal with the issue using a learning-
based method, such as hierarchical reinforcement learning [44] to train a
high-level policy for choosing primitives based on the current state.

##  VI Conclusion

To achieve the skill of in-hand tool manipulation that not only manipulates a
tool within the hand but also achieves a grasp suitable for a task after the
manipulation, we proposed Action Primitives based on Contact-state Transition
(APriCoT). APriCoT decomposes the operation into several action primitives. In
each primitive, fingers need to just perform short-term and similar motions,
which simplifies the training of primitive policies using DRL. Experiments
demonstrated that our method achieves both in-hand rotation and the desired
grasp using various-shaped objects. Our method has potential for other
operations, provided primitive policies for different contact-state
transitions are prepared. This study is a first step toward achieving diverse
operations.

## ACKNOWLEDGMENT

This work was supported by Grant-in-Aid for JSPS Fellows Grant Number
JP24KJ1054.

## References

  * [1] Rana Soltani Zarrin, Rianna Jitosho, and Katsu Yamane.  Hybrid learning-and model-based planning and control of in-hand manipulation.  In IROS, pages 8720–8726. IEEE, 2023.
  * [2] Ethan K Gordon and Rana Soltani Zarrin.  Online augmentation of learned grasp sequence policies for more adaptable and data-efficient in-hand manipulation.  In ICRA, pages 5970–5976. IEEE, 2023.
  * [3] Thomas Feix, Javier Romero, Heinz-Bodo Schmiedmayer, Aaron M Dollar, and Danica Kragic.  The grasp taxonomy of human grasp types.  IEEE Transactions on human-machine systems, 46(1):66–77, 2015.
  * [4] OpenAI: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al.  Learning dexterous in-hand manipulation.  IJRR, 39(1):3–20, 2020.
  * [5] Ankur Handa, Arthur Allshire, Viktor Makoviychuk, Aleksei Petrenko, Ritvik Singh, Jingzhou Liu, Denys Makoviichuk, Karl Van Wyk, Alexander Zhurkevich, Balakumar Sundaralingam, et al.  Dextreme: Transfer of agile in-hand manipulation from simulation to reality.  In ICRA, pages 5977–5984. IEEE, 2023.
  * [6] Zhao-Heng Yin, Binghao Huang, Yuzhe Qin, Qifeng Chen, and Xiaolong Wang.  Rotating without seeing: Towards in-hand dexterity through touch.  arXiv preprint arXiv:2303.10880, 2023.
  * [7] Tao Chen, Jie Xu, and Pulkit Agrawal.  A system for general in-hand object re-orientation.  In CoRL, pages 297–307. PMLR, 2022.
  * [8] Haozhi Qi, Ashish Kumar, Roberto Calandra, Yi Ma, and Jitendra Malik.  In-hand object rotation via rapid motor adaptation.  In CoRL, pages 1722–1732. PMLR, 2023.
  * [9] Haozhi Qi, Brent Yi, Sudharshan Suresh, Mike Lambeta, Yi Ma, Roberto Calandra, and Jitendra Malik.  General in-hand object rotation with vision and touch.  In CoRL, pages 2549–2564. PMLR, 2023.
  * [10] Aditya Gudimella, Ross Story, Matineh Shaker, Ruofan Kong, Matthew Brown, Victor Shnayder, and Marcos Campos.  Deep reinforcement learning for dexterous manipulation with concept networks.  arXiv preprint arXiv:1709.06977, 2017.
  * [11] Alexander Clegg, Wenhao Yu, Jie Tan, C. Karen Liu, and Greg Turk.  Learning to dress: synthesizing human dressing motion via deep reinforcement learning.  ACM Trans. Graph., 37(6), dec 2018.
  * [12] Tingguang Li, Krishnan Srinivasan, Max Qing-Hu Meng, Wenzhen Yuan, and Jeannette Bohg.  Learning hierarchical control for robust in-hand manipulation.  In ICRA, pages 8855–8862. IEEE, 2020.
  * [13] Yuanpei Chen, Chen Wang, Li Fei-Fei, and Karen Liu.  Sequential dexterity: Chaining dexterous policies for long-horizon manipulation.  In CoRL, pages 3809–3829. PMLR, 2023.
  * [14] Phongtharin Vinayavekhin, Shunsuke Kudoh, Jun Takamatsu, Yoshihiro Sato, and Katsushi Ikeuchi.  Representation and mapping of dexterous manipulation through task primitives.  In ICRA, pages 3722–3729. IEEE, 2013.
  * [15] Sing Bing Kang and Katsushi Ikeuchi.  Toward automatic robot instruction from perception-recognizing a grasp from observation.  IEEE Transactions on Robotics and Automation, 9(4):432–443, 1993.
  * [16] Carlo Ferrari, John F Canny, et al.  Planning optimal grasps.  In ICRA, volume 3, page 6, 1992.
  * [17] Allison M Okamura, Niels Smaby, and Mark R Cutkosky.  An overview of dexterous manipulation.  In Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No. 00CH37065), volume 1, pages 255–262. IEEE, 2000.
  * [18] Li Han and Jeffrey C Trinkle.  Dextrous manipulation by rolling and finger gaiting.  In Proceedings. 1998 IEEE International Conference on Robotics and Automation (Cat. No. 98CH36146), volume 1, pages 730–735. IEEE, 1998.
  * [19] Daniela Rus.  In-hand dexterous manipulation of piecewise-smooth 3-d objects.  IJRR, 18(4):355–381, 1999.
  * [20] Igor Mordatch, Zoran Popović, and Emanuel Todorov.  Contact-invariant optimization for hand manipulation.  In Proceedings of the ACM SIGGRAPH/Eurographics symposium on computer animation, pages 137–144, 2012.
  * [21] Yunfei Bai and C Karen Liu.  Dexterous manipulation using both palm and fingers.  In ICRA, pages 1560–1565. IEEE, 2014.
  * [22] Julian Ibarz, Jie Tan, Chelsea Finn, Mrinal Kalakrishnan, Peter Pastor, and Sergey Levine.  How to train your robot with deep reinforcement learning: lessons we have learned.  IJRR, 40(4-5):698–721, 2021.
  * [23] Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter.  Learning quadrupedal locomotion over challenging terrain.  Science robotics, 5(47):eabc5986, 2020.
  * [24] Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine.  Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations.  In RSS, 2018.
  * [25] Philipp Wu, Alejandro Escontrela, Danijar Hafner, Pieter Abbeel, and Ken Goldberg.  Daydreamer: World models for physical robot learning.  In CoRL, pages 2226–2240. PMLR, 2023.
  * [26] Andrew Melnik, Luca Lach, Matthias Plappert, Timo Korthals, Robert Haschke, and Helge Ritter.  Using tactile sensing to improve the sample efficiency and performance of deep deterministic policy gradients for simulated in-hand manipulation tasks.  Frontiers in Robotics and AI, 8:538773, 2021.
  * [27] Max Yang, Chenghua Lu, Alex Church, Yijiong Lin, Chris Ford, Haoran Li, Efi Psomopoulou, David AW Barton, and Nathan F Lepora.  Anyrotate: Gravity-invariant in-hand object rotation with sim-to-real touch.  arXiv preprint arXiv:2405.07391, 2024.
  * [28] Johannes Pitz, Lennart Röstel, Leon Sievers, and Berthold Bäuml.  Dextrous tactile in-hand manipulation using a modular reinforcement learning architecture.  In ICRA, pages 1852–1858. IEEE, 2023.
  * [29] Gagan Khandate, Maximilian Haas-Heger, and Matei Ciocarlie.  On the feasibility of learning finger-gaiting in-hand manipulation with intrinsic sensing.  In ICRA, pages 2752–2758. IEEE, 2022.
  * [30] George Konidaris and Andrew Barto.  Skill discovery in continuous reinforcement learning domains using skill chaining.  NeurIPS, 22, 2009.
  * [31] Murtaza Dalal, Deepak Pathak, and Russ R Salakhutdinov.  Accelerating robotic reinforcement learning via parameterized action primitives.  NeurIPS, 34:21847–21859, 2021.
  * [32] Dian Chen, Brady Zhou, Vladlen Koltun, and Philipp Krähenbühl.  Learning by cheating.  In CoRL, pages 66–75. PMLR, 2020.
  * [33] Alan H Barr.  Superquadrics and angle-preserving transformations.  IEEE Computer graphics and Applications, 1(01):11–23, 1981.
  * [34] Colin Lea, Michael D Flynn, Rene Vidal, Austin Reiter, and Gregory D Hager.  Temporal convolutional networks for action segmentation and detection.  In CVPR, pages 156–165, 2017.
  * [35] Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al.  Isaac gym: High performance gpu-based physics simulation for robot learning.  arXiv preprint arXiv:2108.10470, 2021.
  * [36] Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter.  Fast and accurate deep network learning by exponential linear units (elus).  arXiv preprint arXiv:1511.07289, 2015.
  * [37] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  arXiv preprint arXiv:1707.06347, 2017.
  * [38] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [39] Laurens Van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  Journal of machine learning research, 9(11), 2008.
  * [40] Katsushi Ikeuchi and Takashi Suehiro.  Toward an assembly plan from observation. i. task recognition with polyhedral objects.  IEEE transactions on robotics and automation, 10(3):368–385, 1994.
  * [41] Naoki Wake, Riku Arakawa, Iori Yanokura, Takuya Kiyokawa, Kazuhiro Sasabuchi, Jun Takamatsu, and Katsushi Ikeuchi.  A learning-from-observation framework: One-shot robot teaching for grasp-manipulation-release household operations.  In SII, pages 461–466. IEEE, 2021.
  * [42] Daichi Saito, Kazuhiro Sasabuchi, Naoki Wake, Jun Takamatsu, Hideki Koike, and Katsushi Ikeuchi.  Task-grasping from a demonstrated human strategy.  In Humanoids, pages 880–887. IEEE, 2022.
  * [43] Naoki Wake, Daichi Saito, Kazuhiro Sasabuchi, Hideki Koike, and Katsushi Ikeuchi.  Text-driven object affordance for guiding grasp-type recognition in multimodal robot teaching.  MVA, 34(4):58, 2023.
  * [44] Shubham Pateria, Budhitama Subagdja, Ah-hwee Tan, and Chai Quek.  Hierarchical reinforcement learning: A comprehensive survey.  ACM Computing Surveys (CSUR), 54(5):1–35, 2021.


# Some New Possible Anticipated Signals for Existence of Magnetic Monopoles

Qiu-He Peng 11affiliationmark:  , Jing-Jing, Liu 22affiliationmark:  , and
Zhong-Qi, Ma33affiliationmark<EMAIL_ADDRESS>
###### Abstract

We summarize some predictions from the model of supermassive object with
magnetic monopoles which match up with recent astronomical observations
quantitatively. They may be the signals for existence of magnetic monopoles in
the supermassive objects, such as one at the Galactic Center.

††slugcomment: Not to appear in Nonlearned J., 45.00footnotetext:
Corresponding to Jing-Jing, Liu00footnotetext: Department of Astronomy,
Nanjing University, Nanjing, Jiangshu 210000, China.00footnotetext: College of
Marine Science and Technology, Hainan Tropical Ocean University, Sanya, Hainan
572022, China.00footnotetext: Institute of High Energy Physics, Chinese
Academy of Sciences, Beijing 100049, China.

Keywords Galaxy: center — Galaxies: magnetic fields — Physical Date and
Processes: black hole physics.

##  1 Introduction

Most of physicists believe that the existence of magnetic monopoles had been
ruled out by experiments. However, experiments only indicated that the flux of
magnetic monopoles on the earth is too low to be observed. In the primordial
universe, the electromagnetic interaction between magnetic monopoles and
plasma is so strong such that magnetic monopoles might store up in the center
of quasars and active galactic nuclei (AGN) during the collapsing process of
the original giant nebulae, including at the collapsed core of the Galactic
Center (GC). Due to the Rubakov-Callan (RC) effect(Rubakov, 1981; Callan,
1983), magnetic monopoles may catalyze nucleon decay and is invoked as the
energy source of quasars and AGN. The model of supermassive object with
magnetic monopoles (SMOMM) (Peng & Chou, 2001), presented by one (QP) of the
authors, gave five predictions, some of which match up with the recent
astronomical observations quantitatively. It may be the signals for existence
of magnetic monopoles in the supermassive objects.

From the symmetry and beauty of the Dirac equation, Dirac (Dirac, 1958)
suggested the existence of magnetic monopoles which is the most natural
explanation of the quantization of electric charge. It is a fundamental
problem in physics whether magnetic monopoles exist or not.

The plan of this letter is as follows. In section 2, we will sketch the model
of SMOMM and its predictions. In Sections $3\sim 5$, we will show some
predictions of our model in agreement quantitatively or basically with some
new astrophysical observations: a strong magnetic field around the GC, the
observation of the rate of the emitted positrons, and the frequency of the
spectrum peak of the thermal radiation from the GC. In the last Section, we
will discuss and try to explain two recent observations with the model of the
stars with the magnetic monopoles: the highly super-luminous supernova
ASASSN-15lh and the gravitational waves GW150914.

##  2 The model of SMOMM

The model of SMOMM (Peng & Chou, 2001) suggested that an amount of magnetic
monopoles stored up in the center of quasars and AGNs, including at the
collapsed core of the GC during the collapsing process of the original giant
nebulae in the primordial universe because the electromagnetic interaction
between magnetic monopoles and plasma was so strong. Due to the RC effect
(Rubakov, 1981; Callan, 1983), nucleon decay may be catalyzed by magnetic
monopoles,

$p+M\rightarrow~{}M+e^{+}+\pi^{0}+{\rm debris}\ (85\%),$ (1)
$p+M\rightarrow~{}M+e^{+}+\mu^{+}+\mu^{-}+{\rm debris}\ (15\%).$ (2)

In this model, the RC effect is invoked as the energy source. In the case of
GC, the SMOMM is located at the center with the radius of about $8.1\times
10^{15}cm\sim 1.2\times 10^{4}R_{S}$ ($R_{S}$ is its Schwarzschild radius).
The total mass of the SMOMM is derived to be $2.5\times 10^{6}M_{\bigodot}$
(now taken as $4.6\times 10^{6}M_{\bigodot}$) from the observed luminosity of
Sgr A${}^{*}(1\times 10^{37}$ ergs ${\rm s}^{-1}$).

The gravitational effect around SMOMM in the GC is similar to the model of
supermassive black holes. In the black hole model, accretion of matter results
in the huge luminosity, but the energy source in the model of SMOMM is
supplied by the RC effect.

The main predictions of the model of SMOMM are as follows (Peng & Chou, 2001).

1\. The production rate of positrons emitted from the SMOMM in the model is
$\sim 6.5\times 10^{42}e^{+}/sec$.

2\. High-energy gamma-ray radiation has energy higher than 0.5 MeV. The
integrated energy of these radiations would be much greater than both the
bolometric luminosity and the energy of positron annihilation line.

3\. The radial magnetic field at the surface of the SMOMM is estimated to be
$H(R)\sim(20-100)G$.

4\. The strong radial magnetic fields of the high-speed rotating SMOMM
transforms a strong electric field for a distant observer in the rest frame. A
variety of produced particles ($e^{+}$, $\mu^{\pm}$, $\pi^{\pm}$) would be
accelerated by the strong electric field to very high energy, say
$E_{\gamma}\sim 10^{21}$eV or greater. We predict that these could just be the
observed ultra-high-energy cosmic rays which have an initial energy of several
hundred MeV produced from the SMOMM.

5\. The surface temperature of the SMOMM is derived to be about 121 K and the
corresponding spectrum peak of the thermal radiation is at $10^{13}Hz$ in the
sub-mm wavelength regime.

##  3 A strong magnetic field around the galactic center

The recent observation (Eatough et al., 2013) in 2013 indicated that there is
a dynamically important magnetic field near the black hole. In particular, at
$r=0.12pc$ the lower limit of the outward radial magnetic field near the GC is

$B\geq
8\left[\displaystyle\frac{RM}{66.960m^{-2}}\right]\left[\displaystyle\frac{n_{0}}{26cm^{-3}}\right]^{-1}mG,$
(3)

where $n_{0}$ is the number density of electrons there, and RM denotes the
measurement of the Faraday rotation near the GC. The lower limit of the
observed data is in agreement with the the prediction 3 in the model of SMOMM
because the magnetic field strength decreases as the inverse square of the
distance from the source and has $B\approx(10-50)mG$ at $r=0.12pc$. Up to now
no other physical mechanism can produce this strong radial magnetic field.

As analyzed in (Zamaninasab et al., 2014), “jet magnetic field and accretion
disk luminosity are tightly correlated over seven orders of magnitude for
sample of 76 radio-loud active galaxies”. They pointed out that the black hole
models “may require significant changes”, and “models of the Galactic Center
accretion disk may also need to be revised, as a dynamically important
magnetic field has been reported (Eatough et al., 2013) within a distance of
$\sim 3\times 10^{7}r_{g}$ from the central black hole.”

##  4 Rate of emitted positrons

New observation (Knödlseder et al., 2003) reported that the measured 511 keV
line flux located at the GC at a distance of $8.5kpc$ converts into an
annihilation rate of $(3.4-6.3)\times 10^{42}s^{-1}$. “The observed flux is
compatible with previous measurements (Share et al., 1990; Cheng et al., 1997;
Purcell et al., 1997; Milne et al., 2000, 2001) that have been obtained using
telescopes with small or moderate fields-of-view, yet it is on the low side
when compared to OSSE measurements.” (Knödlseder et al., 2003) Those
observations are in agreement with the the prediction 1 in the model of SMOMM
quantitatively.

##  5 Frequency of spectrum peak of the thermal radition from the galactic
center

A review paper (Falcke & Markoff, 2013) pointed out that the radio flux
density $S_{\nu}$ from the GC shows a flat-to-inverted spectrum. i.e., it
raised slowly with frequency of the power peaking around $10^{12}$ Hz in the
sub-mm band. The observed power peak is in agreement basically with the
prediction 5 in the model of SMOMM.

##  6 Conclusions and discussions

The agreement of the predictions of our model of the SMOMM with three new
astrophysical observations quantitatively or basically issues the signals for
existence of magnetic monopoles. We are looking forward to seeing more
astrophysical observation which will meet the predictions of our model.

At the beginning of this year two important astrophysical observations were
reported: The highly super-luminous supernova ASASSN-15lh and the
gravitational waves GW150914. We believe that our model and its development
can be suitable to meet the new observations.

The recent observation (Dong et al., 2016) reported the discovery of
ASASSN-15lh (SN 2015L), which was interpreted as the most luminous supernova
to be found. “At redshift $z-0.2326$, ASASSN-15lh reached an absolute
magnitude of $M_{u,AB}=-23.5\pm 0.1$ and bolometric luminosity
$L_{bol}=(2.2\pm 0.2)\times 10^{45}$ erg s-1, which is more than twice as
luminous as any previously known supernova”. “In the 4 months since first
detection, ASASSN-15lh radiated $(1.1\pm 0.2)\times 10^{52}$ ergs, challenging
the magnetar model for its engine”.

Up to now, the supernova explosion mechanism has not been solved. In a model
of a supermassive star with magnetic monopoles, the phenomena of ASASSN-15lh
and the super-luminous supernova (SLSN) may be explained naturally. Although a
few magnetic monopoles may be stored in the core of stars during the process
of star formation from a neutral hydrogen nebulae due to very weak interaction
between magnetic monopoles and neutral hydrogens, the stars may capture some
flight magnetic monopoles in the space (Peng et al., 1985). The flux of the
flight magnetic monopoles is

$\displaystyle\Phi_{m}=n_{m}v_{m}=10^{-4}cn_{B}^{(0)}\zeta^{(0)}_{m}(\displaystyle\frac{v_{m}}{10^{-4}c})$
$\displaystyle\approx 7.5\times
10^{-19}(\displaystyle\frac{\zeta^{(0)}_{m}}{\zeta_{S}})(\displaystyle\frac{n_{B}^{(0)}}{1cm^{3}})(\displaystyle\frac{v_{m}}{10^{-4}c})cm^{-2}s^{-1}$
$\displaystyle\approx 2.5\times
10^{-4}(\displaystyle\frac{\zeta^{(0)}_{m}}{\zeta_{S}})(\displaystyle\frac{n_{B}^{(0)}}{1cm^{3}})(\displaystyle\frac{v_{m}}{10^{-4}c})\times(100m)^{-2}Yr^{-1},$
(4)

where $n_{m}$ is the number density of the magnetic monopoles, $v_{m}$ is the
average velocity of the flight magnetic monopoles in space, $n_{B}^{(0)}$ is
the number density of the baryons in the interstellar space, and
$\zeta_{S}=Gm_{B}m_{m}/g_{m}^{2}\simeq 1.9\times 10^{-25}$ denotes the Newton
saturation value of $\zeta_{m}=N_{m}/N_{B}$, where $N_{m}$ is the total number
of the flight magnetic monopoles and $N_{B}$ is the total number of nucleons.
$\zeta_{m}^{(0)}$ is the value of $\zeta_{m}$ in the interstellar space, and
its upper limit is $\zeta_{m}^{0}\leq 10^{-20\pm 1}$ (Parker, 1970; Lazarides
et al., 1981). After formation of stars the total number of the magnetic
monopoles captured by the stars from space is estimated to be

$\displaystyle N_{m}$ $\displaystyle=4\pi R^{2}\Phi_{m}T\approx 3\times
10^{24}\left(\displaystyle\frac{\zeta^{(0)}_{m}}{\zeta_{S}}\right)\left(\displaystyle\frac{n_{B}^{(0)}}{1cm^{3}}\right)$
(5)
$\displaystyle~{}~{}~{}\times\left(\displaystyle\frac{v_{m}}{10^{-4}c}\right)\left(\displaystyle\frac{R}{10^{3}R_{\bigodot}}\right)^{2}\left(\displaystyle\frac{T}{10^{7}Yr}\right).$

where $R$ is the radius of the star, $T$ is the age of the progenitor of the
supernova. The captured superheavy monopoles are gathered at the core of the
stars.

Due to the RC effect, the luminosity produced by the nucleon decay catalyzed
by magnetic monopoles in the core of the stars is

$L_{m}=N_{m}\langle\sigma v\rangle n_{B}^{(c)}m_{B}c^{2},$ (6)

where $\sigma\sim 10^{-25}-10^{-26}$ cm2 is the cross section of the RC effect
(Rubakov, 1981; Callan, 1983), $v$ is the thermal velocity of the nucleons,
and $n_{B}^{(c)}$ is the central number density of the baryons for the star.
The temperature would reach $10^{11}$K in the collapsed core of the supernova,
i.e., $v/c\geq 0.1$. Thus,

$\displaystyle L_{m}$
$\displaystyle\sim(10^{42}-10^{43})\left(\displaystyle\frac{n_{B}^{(c)}}{n_{nuc}}\right)\left(\displaystyle\frac{\sigma}{10^{-26}cm^{2}}\right)$
(7)
$\displaystyle~{}~{}~{}\times\left(\displaystyle\frac{R}{10^{3}R_{\bigodot}}\right)^{2}\left(\displaystyle\frac{T}{10^{7}Yr}\right).$

This luminosity is enough to explain the supernova explosion, when the density
of the supernova core is greater than the nuclear density ($n_{nuc}$).

As long as we assume that the initial mass of the progenitor of the SLSN is
greater than $150m_{\bigodot}$, then its radius before the supernova explosion
was greater than $10^{4}R_{\bigodot}$, and the life of the super-massive star
is only $10^{5}-10^{6}Yr$. Thus, the luminosity produced from the nucleon
decay catalyzed by the magnetic monopoles in the collapsed core of the
supernova, where the density of the core is much greater than the nuclear
density, will be greater than $10^{42}-10^{43}$ ergs. This luminosity is
enough to make the SLSN exploding, so that the SLSN is explained naturally.

On September 14, 2015 two detectors of the Laser Interferometer Gravitational-
Wave Observatory (LIGO) simultaneously observed a transient gravitational-wave
signal (GW150914) (Abbott et al., 2016). This is the first direct detection of
gravitational waves which was commonly explained as the first observation of a
binary black hole system mergering to form a single black hole. On December
26, 2015, a second gravitational-wave event, GW151226, was observed by the
twin detectors of LIGO(Abbott et al., 2016).

On February 16, 2016, Connaughton et al. (2016) reported that the Fermi Gamma-
ray Burst Monitor (GBM) had revealed the presence of a weak transient source
above 50 keV, 0.4 s after the GW event: “This weak transient lasting 1 s does
not appear connected with other previously known astrophysical, solar,
terrestrial, or magnetospheric activity. Its localization is ill-constrained
but consistent with the direction of GW150914”(Connaughton et al., 2016). They
also note that the electromagnetic signal from a stellar mass black hole
binary merger is not expected if the GBM transient is associated with
GW150914. Based on their measurement of the fluence seen by GMB, a luminosity
of $1.8^{+1.5}_{-1.0}\times 10^{49}$ erg s-1 is derived in hard X-ray emission
between 1 keV and 10 MeV.

In another paper (Lyutikov, 2016) Lyutikov analyzed the physical requirements
in some detail that the possible observation of the electromagnetic signal
contemporaneous with GW150914 imposes on the circum-merger environment, and
found that the required physical parameters at the source exceed by many
orders of magnitude what is expected in realistic astrophysical scenarios. He
concluded that Fermi GMB signal contemporaneous with GW150914 is unrelated to
the black hole merger.

The merger of two black holes can produce the GW only, not the gamma-ray burst
simultaneously. However, in our model of stars with magnetic monopoles, the
merger of two supermassive neutron stars with the magnetic monopoles, (They
cannot collapse to black holes through the RC effect), whose mass density
reaches to $(10^{5}-10^{6})\rho_{nuc}$ at the center core, not only produces
the GW, but also produces the gamma-ray burst whose luminosity may reach
$10^{49}ergs/s$ through the RC effect. After we have finished this paper,
Racusin et al. (2016) reported on June 15, 2016 that they had made
observations of the event GW151226 and candidate LVT151012 with both the Fermi
Gamma-ray Burst Monitor and the Large Area Telescope (LAT). No electromagnetic
counterparts were detected by either GBM or LAT. We are looking forward to the
future detections of the GW which may accompany with the gamma-ray burst. If
yes, it will verify the existence of magnetic monopoles.

Acknowledgements We would like to thank Prof. Daniel Wang, Prof. Y.F. Huang,
Prof. P.F. Chen and Prof. J. L. Han for their help to inform us some new
information of observations. This work was supported in part by the National
Natural Science Foundation of China under grants 10773005, 11565020, and the
Counterpart Foundation of Sanya under grant 2016PT43, the Special Foundation
of Science and Technology Cooperation for Advanced Academy and Regional of
Sanya under grant 2016YD28, and the Natural Science Foundation of Hainan
province under grant 114012.

## References

  * Abbott et al. (2016) Abbott, B. P., Abbott, R., Abbott, T. D., et al., 2016, Physical Review Letters, 116, 061102
  * Abbott et al. (2016) Abbott, B. P., et al., 2016, Phys. Rev. Lett., 116, 241103
  * Callan (1983) C. Callan, 1983, Nucl. Phys., 212, 391
  * Cheng et al. (1997) Cheng, L. X., Leventhal, M., Smith, D. M., et al., 1997, ApJL, 481, 43
  * Connaughton et al. (2016) Connaughton, V., Burns, E., Goldstein, A., et al. 2016, arXiv:1602.03920
  * Dirac (1958) P. A. M. Dirac., 1958, The Principle of Quantum Mechanics, (Clarendon Press, Oxford, 1958).
  * Dong et al. (2016) Dong, S., Shappee, B. J., Prieto, J. L., et al., 2016, Science, 351, 257
  * Eatough et al. (2013) R.P. Eatough, H. Falcke, R. Karuppusamy et al., 2013, nature, 591, 391
  * Falcke & Markoff (2013) Falcke, H., & Markoff, S. B., 2013, Classical and Quantum Gravity, 30, 244003
  * Knödlseder et al. (2003) Knödlseder, J., Lonjou, V., Jean, P., et al., 2003, Aap, 411, 457
  * Lazarides et al. (1981) Lazarides, G., Shafi, Q., & Walsh, T. F., 1981, Physics Letters B, 100, 21
  * Lyutikov (2016) Lyutikov, M., 2016, arXiv:1602.07352
  * Milne et al. (2000) Milne, P. A., Kurfess, J. D., Kinzer, R. L., Leising, M. D., & Dixon, D. D., 2000, American Institute of Physics Conference Series, 510, 21
  * Milne et al. (2001) Milne, P. A., Kurfess, J. D., Kinzer, R. L., & Leising, M. D., 2001, Gamma-Ray Astrophysics, 587, 11
  * Parker (1970) Parker, E. N., 1970, ApJ, 160, 383
  * Peng et al. (1985) Peng, Q., Lie, Z. and Wang, D., 1985, Scientia Sinica (Serries A), 28, 970
  * Peng & Chou (2001) Peng, Q. & Chou, C., 2001, ApJ, 551, 23
  * Purcell et al. (1997) Purcell, W. R., Cheng, L.-X., Dixon, D. D., et al., 1997, ApJ, 491, 725
  * Racusin et al. (2016) Racusin, J. L., Burns, E., Goldstein, A., et al., 2016, arXiv:1606.04901
  * Rubakov (1981) Rubakov, V. 1981, JETP Lett. 33,644
  * Share et al. (1990) Share, G. H., Leising, M. D., Messina, D. C., & Purcell, W. R., 1999, ApJL, 358, 45
  * Zamaninasab et al. (2014) Zamaninasab, M., Clausen-Brown, E., Savolainen, T., & Tchekhovskoy, A., 2014, Nature, 510, 126