# Revealing the nature of $\Omega_{c}$-like states from pentaquark perspective

Ulaş Özdem  0000-0002-1907-2894<EMAIL_ADDRESS>Health Services
Vocational School of Higher Education, Istanbul Aydin University, Sefakoy-
Kucukcekmece, 34295 Istanbul, Türkiye

###### Abstract

We systematically study the electromagnetic properties of controversial states
whose internal structure is not elucidated and we try to offer a different
point of view to unravel the internal structure of these states. Inspired by
the $\Omega_{c}$ states observed by the LHCb Collaboration, we study the
electromagnetic properties of the $\Omega_{c}$-like states as the compact
diquark-diquark-antiquark pentaquarks with both $J^{P}=\frac{1}{2}^{-}$ and
$J^{P}=\frac{3}{2}^{-}$ in the context of the QCD light-cone sum rule model.
From the obtained numerical results, we conclude that the magnetic dipole
moments of the $\Omega_{c}$-like states can reflect their inner structures,
which can be used to distinguish their spin-parity quantum numbers. Measuring
the magnetic moment of the $\Omega_{c}$-like states in future experimental
facilities can be very helpful for understanding the internal organization and
identifying the quantum numbers of these states.

Electromagnetic form factors, diquark-diquark-antiquark picture, QCD light-
cone sum rules, magnetic dipole moments

##  I Motivation

Many heavy baryon states have been discovered in recent years by experimental
collaborations. One of the main challenges in non-perturbative QCD is to
understand and shed light on the precise nature of these states. Through
further theoretical explorations involving the study of hadrons comprising a
single heavy quark, it offers an exquisite basis to probe the dynamics of a
light diquark in a heavy quark background, to enhance the understanding of the
non-perturbative nature of QCD, and to test the predictions of different
phenomenological models. In recent decades, there have been significant
experimental advancements in the field of singly-charm/bottom baryons,
resulting in a dramatic increase in the number of particles [1, 2, 3, 4, 5, 6,
7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26,
27, 28, 29, 30]. As the data for the presence of some of these states is
scarce and their internal structure, as well as quantum numbers, are not well
defined, additional experimental exploration is therefore required. Therefore,
researchers in hadron physics continue to study these topics through
theoretical and experimental research, as they have not yet been fully
understood.

In 2017, the LHCb collaboration studied the $\Xi_{c}^{+}K^{-}$ mass spectrum
and observed five new narrow excited $\Omega_{c}$ states, $\Omega_{c}(3000)$,
$\Omega_{c}(3050)$, $\Omega_{c}(3066)$, $\Omega_{c}(3090)$, $\Omega_{c}(3119)$
[22]. The parameters that have been measured are as follows

$\displaystyle M_{\Omega_{c}(3000)}=3000.4\pm 0.2\pm 0.1\mbox{
MeV}\,,\,\,\,\,\Gamma_{\Omega_{c}(3000)}=4.5\pm 0.6\pm 0.3\mbox{ MeV}\,,$
$\displaystyle M_{\Omega_{c}(3050)}=3050.2\pm 0.1\pm 0.1\mbox{
MeV},\,\,\,\,\Gamma_{\Omega_{c}(3050)}=0.8\pm 0.2\pm 0.1\mbox{ MeV}\,,\,\,$
$\displaystyle M_{\Omega_{c}(3066)}=3065.6\pm 0.1\pm 0.3\mbox{
MeV}\,,\,\,\,\,\Gamma_{\Omega_{c}(3066)}=3.5\pm 0.4\pm 0.2\mbox{ MeV}\,,$
$\displaystyle M_{\Omega_{c}(3090)}=3090.2\pm 0.3\pm 0.5\mbox{
MeV}\,,\,\,\,\,\Gamma_{\Omega_{c}(3090)}=8.7\pm 1.0\pm 0.8\mbox{ MeV}\,,$
$\displaystyle M_{\Omega_{c}(3119)}=3119.1\pm 0.3\pm 0.9\mbox{
MeV}\,,\,\,\,\,\Gamma_{\Omega_{c}(3119)}=1.1\pm 0.8\pm 0.4\mbox{ MeV}\,.$ (1)

Later, the former four states $\Omega_{c}(3000)$, $\Omega_{c}(3050)$,
$\Omega_{c}(3066)$ and $\Omega_{c}(3090)$ were confirmed by the LHCb and Belle
Collaborations [31, 32]. The discovery of the LHCb collaboration has led to a
new experimental situation, which requires a more detailed study of heavy
baryons and their properties. The discovery of these states, although their
quantum numbers have not been determined, could provide new insights into QCD
and its complex behavior. This could lead to a deeper understanding of the
underlying properties of QCD. However, our information about their properties
is still not sufficient and further suggestions for experimental exploration
of $\Omega_{c}$-like states should be addressed. The experimental discoveries
were followed by various theoretical studies, investigating them in the
conventional baryon, molecular, and compact pentaquark states to shed light on
their exact nature and quantum numbers (for details see the Refs. [33, 34, 35,
36, 37]).

The literature review indicates that the majority of research has concentrated
on computing the spectroscopic and decay parameters of these states. However,
it is evident that relying exclusively on spectroscopic and decay parameters
is insufficient for resolving the disputed nature of these states. Therefore,
to determine the precise internal configurations of these states, additional
studies such as obtaining electromagnetic multipole moments, radiative decays,
and weak decays are required. To unveil the nature and internal structures of
unconventional states, the physical quantities associated with the
electromagnetic properties of hadrons are valuable parameters. The charge and
magnetism distributions inside the hadrons can be studied using the
electromagnetic form factors (FFs) and the resulting multipole moments. This
information can be used to determine the distribution of quark-antiquark pairs
(both valence and sea-quarks) and gluons within the hadron volume. It also
provides valuable insights into the geometric forms, electric radii, and
magnetic radii of the hadrons. With this motivation, in this work, we will
derive the electromagnetic features of $\Omega_{c}$-like states using the
method of QCD light-cone sum rules [38, 39, 40]. Throughout the analysis,
these states are considered in the compact $[ss][qc][\bar{q}]$,
$[sq][sc][\bar{q}]$ and $[qs][sc][\bar{q}]$ diquark-diquark-antiquark
configurations, and are assumed to have both $J^{P}=\frac{1}{2}^{-}$ and
$J^{P}=\frac{3}{2}^{-}$ quantum numbers. There are several studies in the
literature that have extracted the electromagnetic features of the singly-
heavy nonconventional hadron states [41, 42, 43, 44, 45, 46, 47, 48].

The article is structured as follows: we derive the QCD light-cone sum rules
for the electromagnetic features of the $\Omega_{c}$-like states in Sect. II.
The numerical results of the magnetic dipole moments and discussions are
presented in Sect. III. Sect. IV is reserved for our summary and concluding
remarks.

##  II Electromagnetic form factors from QCD light-cone sum rules

To study the electromagnetic properties of hadrons in the low energy regime,
reliable and efficient non-perturbative methods are needed. The QCD light-cone
sum rules method is one of these methods and has been very successful in
predicting the static and dynamical properties of both conventional and
nonconventional hadrons. In this method, the correlation function, the key
ingredient of the model, is calculated in terms of both hadronic parameters
and QCD degrees of freedom. The Borel transform and the continuum subtraction
procedures are then applied to suppress the contribution of the effects caused
by the continuum and higher states. Together with these procedures, the QCD
light-cone sum rules for the physical parameter of interest are obtained via
dispersion integrals and by using the quark-hadron duality assumption.

The study of the electromagnetic form factors of the $\Omega_{c}$-like states
($\frac{1}{2}^{-}\rightarrow\Omega_{c}$,
$\frac{3}{2}^{-}\rightarrow\Omega_{c}^{*}$) in QCD light cone sum rules begins
with the writing of the correlation function in the weak electromagnetic
background field ($\gamma$) as follows

$\displaystyle\Pi(p,q)$ $\displaystyle=i\int d^{4}xe^{ip\cdot x}\langle
0|T\left\\{J^{\Omega_{c}}(x)\bar{J}^{\Omega_{c}}(0)\right\\}|0\rangle_{\gamma}\,,$
(2) $\displaystyle\Pi_{\mu\nu}(p,q)$ $\displaystyle=i\int d^{4}xe^{ip\cdot
x}\langle
0|T\left\\{J_{\mu}^{\Omega_{c}^{*}}(x)\bar{J}_{\nu}^{\Omega_{c}^{*}}(0)\right\\}|0\rangle_{\gamma}\,,$
(3)

where $T$ is the time ordered product; and the $J^{\Omega_{c}}(x)$ and
$J_{\mu}^{\Omega_{c}^{*}}(x)$ are the interpolating currents for the
$\Omega_{c}$-like states. The explicit form of the $J^{\Omega_{c}}(x)$ and
$J_{\mu}^{\Omega_{c}^{*}}(x)$ are written as follows

$\displaystyle J^{\Omega_{c}}(x)$
$\displaystyle=\varepsilon^{abc}\varepsilon^{ade}\varepsilon^{bfg}\big{[}q^{d^{T}}_{1}(x)C\gamma_{\mu}q_{2}^{e}(x)\,q_{3}^{f^{T}}(x)C\gamma^{\mu}c^{g}(x)\,C\bar{q}^{c^{T}}(x)\big{]},$
(4) $\displaystyle J_{\mu}^{\Omega_{c}^{*}}(x)$
$\displaystyle=\varepsilon^{abc}\varepsilon^{ade}\varepsilon^{bfg}\big{[}q^{d^{T}}_{1}(x)C\gamma_{\mu}q_{2}^{e}(x)\,q_{3}^{f^{T}}(x)C\gamma_{5}c^{g}(x)\,C\bar{q}^{c^{T}}(x)\big{]},$
(5)

where $q_{1}$, $q_{2}$, $q_{3}$ are u-, d- and/or s-quark, $q$ is u- or
d-quark, the $C$ is the charge conjugation operator and $a$, $b$… are color
indices. It should be noted here that the mass values obtained with these
interpolating currents are compatible with the mass values of the
$\Omega_{c}(3050)$, $\Omega_{c}(3066)$, $\Omega_{c}(3090)$, and
$\Omega_{c}(3119)$ states [49, 50].

After this brief introduction to the methodology, we begin the analysis of
electromagnetic form factors in terms of hadronic parameters, following the
above procedure. To evaluate the hadronic side, a complete set of hadronic
states with the same quantum numbers is inserted into the correlation
functions with the considered interpolating currents. Integration over x is
then performed, yielding

$\displaystyle\Pi^{Had}(p,q)$ $\displaystyle=\frac{\langle 0\mid
J^{\Omega_{c}}(x)\mid{\Omega_{c}}(p,s)\rangle}{[p^{2}-m_{{\Omega_{c}}}^{2}]}\langle{\Omega_{c}}(p,s)\mid{\Omega_{c}}(p+q,s)\rangle_{\gamma}\frac{\langle{\Omega_{c}}(p+q,s)\mid\bar{J}^{\Omega_{c}}(0)\mid
0\rangle}{[(p+q)^{2}-m_{{\Omega_{c}}}^{2}]}+\cdots,$ (6)
$\displaystyle\Pi^{Had}_{\mu\nu}(p,q)$ $\displaystyle=\frac{\langle 0\mid
J_{\mu}^{\Omega_{c}^{*}}(x)\mid{\Omega_{c}^{*}}(p,s)\rangle}{[p^{2}-m_{{\Omega_{c}^{*}}}^{2}]}\langle{\Omega_{c}^{*}}(p,s)\mid{\Omega_{c}^{*}}(p+q,s)\rangle_{\gamma}\frac{\langle{\Omega_{c}^{*}}(p+q,s)\mid\bar{J}_{\nu}^{\Omega_{c}^{*}}(0)\mid
0\rangle}{[(p+q)^{2}-m_{{\Omega_{c}^{*}}}^{2}]}+\cdots,$ (7)

where $\cdots$ stands for the continuum and higher states. The Eqs. (6) and
(7) contain matrix elements containing hadronic parameters like residues
($\lambda$), form factors ($F_{i}$), spinors ($u_{(\mu)}$) and so on. The
explicit expressions for these matrix elements are as follows

$\displaystyle\langle 0\mid J^{\Omega_{c}}(x)\mid{\Omega_{c}}(p,s)\rangle=$
$\displaystyle\lambda_{\Omega_{c}}\gamma_{5}\,u(p,s),$ (8)
$\displaystyle\langle{\Omega_{c}}(p+q,s)\mid\bar{J}^{\Omega_{c}}(0)\mid
0\rangle=$ $\displaystyle\lambda_{\Omega_{c}}\gamma_{5}\,\bar{u}(p+q,s),$ (9)
$\displaystyle\langle{\Omega_{c}}(p,s)\mid{\Omega_{c}}(p+q,s)\rangle_{\gamma}$
$\displaystyle=\varepsilon^{\mu}\,\bar{u}(p,s)\bigg{[}\big{[}f_{1}(q^{2})+f_{2}(q^{2})\big{]}\gamma_{\mu}+f_{2}(q^{2})\frac{(2p+q)_{\mu}}{2m_{\Omega_{c}}}\bigg{]}\,u(p+q,s),$
(10) $\displaystyle\langle 0\mid
J_{\mu}^{\Omega_{c}^{*}}(x)\mid{\Omega_{c}^{*}}(p,s)\rangle$
$\displaystyle=\lambda_{{\Omega_{c}^{*}}}u_{\mu}(p,s),$ (11)
$\displaystyle\langle{\Omega_{c}^{*}}(p+q,s)\mid\bar{J}_{\nu}^{\Omega_{c}^{*}}(0)\mid
0\rangle$ $\displaystyle=\lambda_{{\Omega_{c}^{*}}}\bar{u}_{\nu}(p+q,s),$ (12)
$\displaystyle\langle{\Omega_{c}^{*}}(p,s)\mid{\Omega_{c}^{*}}(p+q,s)\rangle_{\gamma}$
$\displaystyle=-e\bar{u}_{\mu}(p,s)\bigg{[}F_{1}(q^{2})g_{\mu\nu}\varepsilon\\!\\!\\!/-\frac{1}{2m_{{\Omega_{c}^{*}}}}\Big{[}F_{2}(q^{2})g_{\mu\nu}+F_{4}(q^{2})\frac{q_{\mu}q_{\nu}}{(2m_{{\Omega_{c}^{*}}})^{2}}\Big{]}\varepsilon\\!\\!\\!/q\\!\\!\\!/$
$\displaystyle+F_{3}(q^{2})\frac{1}{(2m_{{\Omega_{c}^{*}}})^{2}}q_{\mu}q_{\nu}\varepsilon\\!\\!\\!/\bigg{]}u_{\nu}(p+q,s).$
(13)

With the help of the above formulas and the application of some mathematical
simplifications, the following result is derived for the hadronic side of the
correlation functions,

$\displaystyle\Pi^{Had}(p,q)=$
$\displaystyle\lambda^{2}_{\Omega_{c}}\gamma_{5}\frac{\big{(}p\\!\\!\\!/+m_{\Omega_{c}}\big{)}}{[p^{2}-m_{{\Omega_{c}}}^{2}]}\varepsilon^{\mu}\bigg{[}\big{[}f_{1}(q^{2})+f_{2}(q^{2})\big{]}\gamma_{\mu}+f_{2}(q^{2})\,\frac{(2p+q)_{\mu}}{2m_{\Omega_{c}}}\bigg{]}\gamma_{5}\frac{\big{(}p\\!\\!\\!/+q\\!\\!\\!/+m_{\Omega_{c}}\big{)}}{[(p+q)^{2}-m_{{\Omega_{c}}}^{2}]},$
(14) $\displaystyle\Pi^{Had}_{\mu\nu}(p,q)$
$\displaystyle=-\frac{\lambda_{{}_{\Omega_{c}^{*}}}^{2}\,\big{(}p\\!\\!\\!/+q\\!\\!\\!/+m_{\Omega_{c}^{*}}\big{)}}{[(p+q)^{2}-m_{{}_{\Omega_{c}^{*}}}^{2}][p^{2}-m_{{}_{\Omega_{c}^{*}}}^{2}]}\bigg{[}g_{\mu\nu}-\frac{1}{3}\gamma_{\mu}\gamma_{\nu}-\frac{2\,p_{1\mu}p_{1\nu}}{3\,m^{2}_{\Omega_{c}^{*}}}+\frac{(p+q)_{\mu}\gamma_{\nu}-(p+q)_{\nu}\gamma_{\mu}}{3\,m_{\Omega_{c}^{*}}}\bigg{]}$
$\displaystyle\times\bigg{\\{}F_{1}(q^{2})g_{\mu\nu}\varepsilon\\!\\!\\!/-\frac{1}{2m_{\Omega_{c}^{*}}}\Big{[}F_{2}(q^{2})g_{\mu\nu}+F_{4}(q^{2})\frac{q_{\mu}q_{\nu}}{(2m_{\Omega_{c}^{*}})^{2}}\Big{]}\varepsilon\\!\\!\\!/q\\!\\!\\!/+\frac{F_{3}(q^{2})}{(2m_{\Omega_{c}^{*}})^{2}}q_{\mu}q_{\nu}\varepsilon\\!\\!\\!/\bigg{\\}}$
$\displaystyle\times\big{(}p\\!\\!\\!/+m_{\Omega_{c}^{*}}\big{)}\bigg{[}g_{\mu\nu}-\frac{1}{3}\gamma_{\mu}\gamma_{\nu}-\frac{2\,p_{\mu}p_{\nu}}{3\,m^{2}_{\Omega_{c}^{*}}}+\frac{p_{\mu}\gamma_{\nu}-p_{\nu}\gamma_{\mu}}{3\,m_{\Omega_{c}^{*}}}\bigg{]}.$
(15)

The magnetic dipole moments $(\mu)$ are determined from magnetic form factors
at zero momentum transfer,

$\displaystyle\mu_{\Omega_{c}}$ $\displaystyle=$
$\displaystyle\frac{e}{2m_{\Omega_{c}}}F_{M}(0),~{}~{}~{}~{}~{}~{}$ (16)
$\displaystyle\mu_{\Omega_{c}^{*}}$ $\displaystyle=$
$\displaystyle\frac{e}{2m_{\Omega_{c}^{*}}}G_{M}(0),~{}~{}~{}~{}~{}~{}$ (17)

where $F_{M}(0)$ and $G_{M}(0)$ stand for magnetic form factor of the
spin-$\frac{1}{2}^{-}$ and spin-$\frac{3}{2}^{-}$ $\Omega_{c}$-like states,
respectively. Explicit forms of these terms are given as,

$\displaystyle F_{M}(0)$ $\displaystyle=$ $\displaystyle f_{1}(0)+f_{2}(0),$
(18) $\displaystyle G_{M}(0)$ $\displaystyle=$ $\displaystyle
F_{1}(0)+F_{2}(0).$ (19)

Now that we have finished calculating the analysis regarding hadronic
parameters, it is time to calculate in terms of QCD parameters. The magnetic
dipole moments are analyzed in terms of QCD parameters using the interpolating
current in the correlation function. Applying Wick’s theorem, we contract all
the light and heavy quark fields to get the desired outcomes:

$\displaystyle\Pi^{\rm{QCD}}(p,q)=$
$\displaystyle-i\,\varepsilon^{abc}\varepsilon^{a^{\prime}b^{\prime}c^{\prime}}\varepsilon^{ade}\varepsilon^{a^{\prime}d^{\prime}e^{\prime}}\varepsilon^{bfg}\varepsilon^{b^{\prime}f^{\prime}g^{\prime}}\int
d^{4}xe^{ip\cdot x}\langle
0|\bigg{\\{}\rm{Tr}\Big{[}\gamma^{\mu}S_{c}^{gg^{\prime}}(x)\gamma^{\nu}\widetilde{S}_{q_{3}}^{ff^{\prime}}(x)\Big{]}\rm{Tr}\Big{[}\gamma_{\mu}S_{q_{2}}^{ee^{\prime}}(x)\gamma_{\nu}\widetilde{S}_{q_{1}}^{dd^{\prime}}(x)\Big{]}$
$\displaystyle-\rm{Tr}\Big{[}\gamma^{\mu}S_{c}^{gg^{\prime}}(x)\gamma^{\nu}\widetilde{S}_{q_{3}}^{ff^{\prime}}(x)\Big{]}Tr\Big{[}\gamma_{\mu}S_{q_{2}q_{1}}^{ed^{\prime}}(x)\gamma_{\nu}\widetilde{S}_{q_{1}q_{2}}^{de^{\prime}}(x)\Big{]}-\rm{Tr}\Big{[}\gamma^{\mu}S_{c}^{gg^{\prime}}(x)\gamma^{\nu}\widetilde{S}_{q_{2}q_{3}}^{ef^{\prime}}(x)\gamma_{\mu}S_{q_{1}}^{dd^{\prime}}(x)\gamma_{\nu}\widetilde{S}_{q_{3}q_{2}}^{fe^{\prime}}(x)\Big{]}$
$\displaystyle-\rm{Tr}\Big{[}\gamma^{\mu}S_{c}^{gg^{\prime}}(x)\gamma^{\nu}\widetilde{S}_{q_{1}q_{3}}^{df^{\prime}}(x)\gamma_{\mu}\widetilde{S}_{q_{2}}^{ee^{\prime}}(x)\gamma_{\nu}\widetilde{S}_{q_{3}q_{1}}^{fd^{\prime}}(x)\Big{]}+\rm{Tr}\Big{[}\gamma^{\mu}S_{c}^{gg^{\prime}}(x)\gamma^{\nu}\widetilde{S}_{q_{1}q_{3}}^{df^{\prime}}(x)\gamma_{\mu}S_{q_{2}q_{1}}^{ed^{\prime}}(x)\gamma_{\nu}\widetilde{S}_{q_{3}q_{2}}^{fe^{\prime}}(x)\Big{]}$
$\displaystyle+\rm{Tr}\Big{[}\gamma^{\mu}S_{c}^{gg^{\prime}}(x)\gamma^{\nu}\widetilde{S}_{q_{2}q_{3}}^{ef^{\prime}}(x)\gamma_{\mu}S_{q_{1}q_{2}}^{de^{\prime}}(x)\gamma_{\mu}\widetilde{S}_{q_{3}q_{1}}^{fd^{\prime}}(x)\Big{]}\bigg{\\}}\widetilde{S}_{q}^{c^{\prime}c}(-x)|0\rangle_{\gamma},$
(20)

$\displaystyle\Pi^{\rm{QCD}}_{\mu\nu}(p,q)=$
$\displaystyle-i\,\varepsilon^{abc}\varepsilon^{a^{\prime}b^{\prime}c^{\prime}}\varepsilon^{ade}\varepsilon^{a^{\prime}d^{\prime}e^{\prime}}\varepsilon^{bfg}\varepsilon^{b^{\prime}f^{\prime}g^{\prime}}\int
d^{4}xe^{ip\cdot x}\langle
0|\bigg{\\{}\rm{Tr}\Big{[}\gamma_{5}S_{c}^{gg^{\prime}}(x)\gamma_{5}\widetilde{S}_{q_{3}}^{ff^{\prime}}(x)\Big{]}\rm{Tr}\Big{[}\gamma_{\mu}S_{q_{2}}^{ee^{\prime}}(x)\gamma_{\nu}\widetilde{S}_{q_{1}}^{dd^{\prime}}(x)\Big{]}$
$\displaystyle-\rm{Tr}\Big{[}\gamma_{5}S_{c}^{gg^{\prime}}(x)\gamma_{5}\widetilde{S}_{q_{3}}^{ff^{\prime}}(x)\Big{]}Tr\Big{[}\gamma_{\mu}S_{q_{2}q_{1}}^{ed^{\prime}}(x)\gamma_{\nu}\widetilde{S}_{q_{1}q_{2}}^{de^{\prime}}(x)\Big{]}-\rm{Tr}\Big{[}\gamma_{5}S_{c}^{gg^{\prime}}(x)\gamma_{5}\widetilde{S}_{q_{2}q_{3}}^{ef^{\prime}}(x)\gamma_{\mu}S_{q_{1}}^{dd^{\prime}}(x)\gamma_{\nu}\widetilde{S}_{q_{3}q_{2}}^{fe^{\prime}}(x)\Big{]}$
$\displaystyle-\rm{Tr}\Big{[}\gamma_{5}S_{c}^{gg^{\prime}}(x)\gamma_{5}\widetilde{S}_{q_{1}q_{3}}^{df^{\prime}}(x)\gamma_{\mu}\widetilde{S}_{q_{2}}^{ee^{\prime}}(x)\gamma_{\nu}\widetilde{S}_{q_{3}q_{1}}^{fd^{\prime}}(x)\Big{]}+\rm{Tr}\Big{[}\gamma_{5}S_{c}^{gg^{\prime}}(x)\gamma_{5}\widetilde{S}_{q_{1}q_{3}}^{df^{\prime}}(x)\gamma_{\mu}S_{q_{2}q_{1}}^{ed^{\prime}}(x)\gamma_{\nu}\widetilde{S}_{q_{3}q_{2}}^{fe^{\prime}}(x)\Big{]}$
$\displaystyle+\rm{Tr}\Big{[}\gamma_{5}S_{c}^{gg^{\prime}}(x)\gamma_{5}\widetilde{S}_{q_{2}q_{3}}^{ef^{\prime}}(x)\gamma_{\mu}S_{q_{1}q_{2}}^{de^{\prime}}(x)\gamma_{\mu}\widetilde{S}_{q_{3}q_{1}}^{fd^{\prime}}(x)\Big{]}\bigg{\\}}\widetilde{S}_{q}^{c^{\prime}c}(-x)|0\rangle_{\gamma},$
(21)

where $\widetilde{S}_{c(q)}^{ij}(x)=CS_{c(q)}^{ij\rm{T}}(x)C$.
$S_{q_{i}q_{j}}$ exists when $q_{i}=q_{j}$ but it vanishes when $q_{i}\neq
q_{j}$. The $S_{c}(x)$ and $S_{q}(x)$ are the heavy and light-quark
propagators whose expressions in the coordinate space are given as [51, 52],

$\displaystyle S_{q}(x)$
$\displaystyle=S_{q}^{free}(x)-\frac{\langle\bar{q}q\rangle}{12}\Big{(}1-i\frac{m_{q}x\\!\\!\\!/}{4}\Big{)}-\frac{\langle\bar{q}q\rangle}{192}m_{0}^{2}x^{2}\Big{(}1-i\frac{m_{q}x\\!\\!\\!/}{6}\Big{)}-\frac{ig_{s}}{32\pi^{2}x^{2}}~{}G^{\mu\nu}(x)\Big{[}\hbox
to0.0pt{/\hss}{x}\sigma_{\mu\nu}+\sigma_{\mu\nu}\hbox
to0.0pt{/\hss}{x}\Big{]},$ (22) $\displaystyle S_{c}(x)$
$\displaystyle=S_{c}^{free}(x)-\frac{g_{s}m_{c}}{16\pi^{2}}\int_{0}^{1}dv\,G^{\mu\nu}(vx)\bigg{[}(\sigma_{\mu\nu}{x\\!\\!\\!/}+{x\\!\\!\\!/}\sigma_{\mu\nu})\frac{K_{1}\big{(}m_{c}\sqrt{-x^{2}}\big{)}}{\sqrt{-x^{2}}}+2\sigma_{\mu\nu}K_{0}\big{(}m_{c}\sqrt{-x^{2}}\big{)}\bigg{]},$
(23)

with

$\displaystyle S_{q}^{free}(x)$ $\displaystyle=\frac{1}{2\pi
x^{2}}\Big{(}i\frac{x\\!\\!\\!/}{x^{2}}-\frac{m_{q}}{2}\Big{)},$ (24)
$\displaystyle S_{c}^{free}(x)$
$\displaystyle=\frac{m_{c}^{2}}{4\pi^{2}}\Bigg{[}\frac{K_{1}\big{(}m_{c}\sqrt{-x^{2}}\big{)}}{\sqrt{-x^{2}}}+i\frac{{x\\!\\!\\!/}~{}K_{2}\big{(}m_{c}\sqrt{-x^{2}}\big{)}}{(\sqrt{-x^{2}})^{2}}\Bigg{]},$
(25)

where $m_{q}$ and $\langle\bar{q}q\rangle$ stand for the light-quark mass and
condensates, respectively, $G^{\mu\nu}$ being the gluon field-strength tensor,
$v$ for the line variable, and $K_{0}$, $K_{1}$, and $K_{2}$ for the modified
second type Bessel functions.

The correlation function in Eqs. (20) and (21) would both receive perturbative
and non-perturbative contributions. The interactions between photon and light
or heavy quarks at short distances are referred to as perturbative
contributions, whilst interactions between photon and quarks at large
distances are referred to as non-perturbative contributions. To ensure the
completeness and reliability of the magnetic dipole moment results obtained
from the QCD light-cone sum rule method, it is necessary to include
contributions from both regions. For the evaluation of the perturbative
contributions we use the following formula:

$\displaystyle S^{free}(x)\rightarrow\int d^{4}y\,S^{free}(x-z)\,\hbox
to0.0pt{/\hss}{\\!A}(z)\,S^{free}(z)\,,$ (26)

where one of the propagators for either the light or heavy quark interacts
with the photon at a short distance, and the surviving four propagators are
assumed to be free.

In the case of the non-perturbative contributions, the following equation is
to be used

$\displaystyle
S_{\alpha\beta}^{ab}(x)\rightarrow-\frac{1}{4}\big{[}\bar{q}^{a}(x)\Gamma_{i}q^{b}(0)\big{]}\big{(}\Gamma_{i}\big{)}_{\alpha\beta},$
(27)

where it is assumed that the four surviving propagators are full, with one of
the quarks interacting with the photon at a large distance. Here
$\Gamma_{i}=\\{\textbf{1}$, $\gamma_{5}$, $\gamma_{\mu}$,
$i\gamma_{5}\gamma_{\mu}$, $\sigma_{\mu\nu}/2\\}$. Substituting Eq. (27) into
Eqs. (20) and (21) yields matrix elements
$\langle\gamma(q)\left|\bar{q}(x)\Gamma_{i}G_{\alpha\beta}q(0)\right|0\rangle$
and $\langle\gamma(q)\left|\bar{q}(x)\Gamma_{i}q(0)\right|0\rangle$, which are
defined in terms of the photon distribution amplitudes. These matrix elements,
expressed by photon wave functions, play a crucial role in the evaluation of
non-perturbative contributions (See Ref. [53] for details on the distribution
amplitudes (DAs) of the photon). It should be noted that the photon DAs used
in this study only take into account the contributions of the light quarks.
However, in principle, photons can be emitted at a long distance from the
c-quark. The matrix elements of nonlocal operators are proportional to the
quark condensates, the product of DAs, and some constants. In the case of
c-quark, the contribution of non-perturbative constants is negligible and can
be neglected. The c-quark condensates are proportional to $1/m_{c}$. The
condensates for the c-quark are to a large extent suppressed because of their
large mass [54]. In our calculations, we will therefore not use DAs with
c-quark. We have only taken into account the short-distance photon emission
from the c-quark. By following the technical schemes mentioned above, we
arrive at the QCD representation of the analysis of the magnetic dipole
moments. Details of the procedure used to obtain the expression of the
perturbative and non-perturbative contributions can be found in Refs. [55, 56]
for those who are interested.

After completing the procedures described above, we have finished analytical
calculations of the magnetic dipole moments for the relevant states. The
results for the magnetic dipole moments of $\Omega_{c}$-like states are
presented in the following section through numerical analysis.

##  III Numerical results and discussion

In this section, we numerically study the magnetic dipole moments of
$\Omega_{c}$-like states. The physical parameters to be calculated are
analyzed using the following parameters: $m_{u}=2.16^{+0.49}_{-0.26}$ MeV,
$m_{d}=4.67^{+0.48}_{-0.17}$ MeV, $m_{s}=93.4^{+8.6}_{-3.4}\,\mbox{MeV}$,
$m_{c}=1.27\pm 0.02\,$GeV [57], $\langle\bar{q}q\rangle$=$(-0.24\pm
0.01)^{3}\,$GeV3, $\langle\bar{s}s\rangle$= $0.8\langle\bar{q}q\rangle$ [58],
$m_{0}^{2}=0.8\pm 0.1$ GeV2, and $\langle g_{s}^{2}G^{2}\rangle=0.48\pm
0.14~{}$GeV4 [59].The mass and residue values for these states have been
calculated using the two-point sum rules of QCD, which have been used in our
analysis [49, 50]. The photon distribution amplitudes and the parameters
utilized in these distribution amplitudes are borrowed from Ref. [53].

Beyond the parameters listed above, the QCD light-cone sum rules method
includes two other parameters known as the Borel mass parameter $\rm{M^{2}}$
and the continuum threshold $\rm{s_{0}}$, and in the ideal scenario, these two
parameters are expected to have a very small impact on the results. To do
this, it is necessary to find the working regions of these parameters, where
the variation of the results with respect to these parameters should be
minimal. The $\rm{s_{0}}$ is the point at which the correlation function
begins to include contributions from both excited states and the continuum. To
determine the working region of this parameter, the assumption
$\rm{s_{0}}=(M_{H}+0.4^{+0.1}_{-0.1})~{}\rm{GeV}^{2}$ is usually adopted. The
results are then analyzed for their dependence on slight variations of this
parameter. We search for the ideal Borel parameters $\rm{M^{2}}$ according to
the two criteria: pole contribution (PC) and OPE convergence (CVG). These two
conditions are both satisfied in the region
$11.4~{}\mbox{GeV}^{2}\leq\rm{s_{0}}\leq 12.8~{}\mbox{GeV}^{2}$ and
$3.0~{}\mbox{GeV}^{2}\leq\rm{M^{2}}\leq 4.2~{}\mbox{GeV}^{2}$ for the
$\Omega_{c}$ states; and $11.9~{}\mbox{GeV}^{2}\leq\rm{s_{0}}\leq
13.3~{}\mbox{GeV}^{2}$ and $3.2~{}\mbox{GeV}^{2}\leq\rm{M^{2}}\leq
4.6~{}\mbox{GeV}^{2}$ for the $\Omega_{c}^{*}$ states. Our numerical
evaluations show that considering these working intervals for the
$\rm{M^{2}}$, the magnetic dipole moments of the $\Omega_{c}$-like states PC
vary within the interval $34\%\leq\rm{PC}\leq 61\%$ and $31\%\leq\rm{PC}\leq
57\%$ for $\Omega_{c}$ and $\Omega_{c}^{*}$ states, respectively. Analyzing
the CVG, one sees that the contribution of the higher twist and higher
dimensional terms in the OPE is $<3\%$ of the total and the series shows good
convergence for both $\Omega_{c}$ and $\Omega_{c}^{*}$ states. As an example,
for fixed values of the continuum threshold $\rm{s_{0}}$, the variation of the
extracted magnetic dipole moments of the spin-$\frac{1}{2}^{-}$
$\Omega_{c}$-like states with Borel mass $\rm{M^{2}}$ is shown in Fig.1. As it
is seen from this figure, the magnetic dipole moments of these states exhibit
a slight dependence on the parameter $\rm{M^{2}}$. While the magnetic dipole
moments of the $\Omega_{c}$-like states exhibit some dependence on
$\rm{s_{0}}$, this is within the limits authorized by the approach and
constitutes the main part of the uncertainties.

Figure 1:  Dependencies of the magnetic dipole moments of
spin-$\frac{1}{2}^{-}$ $\Omega_{c}$ states on $\rm{M^{2}}$ at three values of
$\rm{s_{0}}$.

After setting all the relevant parameters, we are finally ready to estimate
the numerical values of the corresponding magnetic dipole moments of
$\Omega_{c}$-like states. The predicted results for the magnetic dipole
moments are shown in Table 1. The errors of the results, caused by errors in
the input parameters and uncertainties in the estimation of the working
intervals of the auxiliary parameters, are also shown in Table 1.

Table 1:  Results of the magnetic dipole moments of the $\Omega_{c}$-like states. Parameters |  $J^{P}$ |  $ssuc\bar{u}$ |  $susc\bar{u}$ |  $ussc\bar{u}$ |  $ssdc\bar{d}$ |  $sdsc\bar{d}$ |  $dssc\bar{d}$
---|---|---|---|---|---|---|---
$\mu[\mu_{N}]$ |  $\frac{1}{2}^{-}$ |  $-0.31^{+0.20}_{-0.16}$ |  $-1.10^{+0.23}_{-0.25}$ |  $-1.09^{+0.24}_{-0.23}$ |  $0.17^{+0.11}_{-0.08}$ |  $0.62^{+0.14}_{-0.14}$ |  $0.61^{+0.15}_{-0.13}$
$\mu[\mu_{N}]$ |  $\frac{3}{2}^{-}$ |  $0.28^{+0.17}_{-0.13}$ |  $0.88^{+0.15}_{-0.18}$ |  $0.87^{+0.16}_{-0.16}$ |  $-0.058^{+0.032}_{-0.031}$ |  $-0.37^{+0.05}_{-0.07}$ |  $-0.37^{+0.05}_{-0.07}$

The order of magnetic dipole moments can yield insights into their
experimental measurability. The analysis of the obtained results indicates
that the extracted magnetic dipole moments for both spin-1/2 and spin-3/2
$\Omega_{c}$-like states can be experimentally measured. Our analysis shows
that although hadrons have the same quark content, different diquark
components can significantly affect the magnetic dipole moment results. This
indicates that the magnetic dipole moment is a crucial parameter in
determining the internal structure of hadrons. As previously stated, the
interpolating currents used for both spin-1/2 and spin-3/2 $\Omega_{c}$-like
states predicted compatible with mass values. However, the magnetic dipole
moments obtained for these states have different magnitudes and signs. This
information may help us understand their quantum numbers and internal
structure in experimental measurements. Comparing the magnetic dipole moment
results obtained in this study with those extracted using different
theoretical approaches can provide insight into the consistency of our
predictions. The measurement of the magnetic dipole moments of unstable
hadrons is a challenging experimental task. Nevertheless, using models
approximating key features of QCD can give clues to the relevant dynamical
mechanisms behind the observed structure. Together with spectroscopic/decay
parameters and electromagnetic properties, it will also be interesting to
characterize the branching ratios of the different decay modes and decay
channels of $\Omega_{c}$-like states.

Finally, this section briefly discusses the measurement of the magnetic dipole
moment of the $\Omega_{c}$-like states in the experiment. Due to their short
lifetimes, measuring the magnetic dipole moments of $\Omega_{c}$-like states
by spin precession experiments is a challenging task. Instead, the magnetic
dipole moment of these unstable states can only be measured indirectly through
a three-step process. The particle is first produced, followed by the emission
of a low-energy photon that acts as an external magnetic field. As the final
step, the particle decays. There is a procedure for the indirect determination
of the magnetic dipole moments of these states, which is based on the soft
photon emission of hadrons proposed in Ref. [61]. The basic concept underlying
the procedure is that the photon carries information about the magnetic dipole
and other higher multipole moments of the emitted particle. From this point of
view, we can determine the magnetic dipole moment of the particles under
consideration through measuring the cross-section or decay width of the
radiative process. The magnetic dipole moment of states under investigation
can affect the cross sections, whether total or differential. Determining the
dipole moment of states under investigation requires comparing theoretical
predictions with measured cross-sections. The magnetic dipole moment of
$\Delta(1232)$ resonance has been achieved by using the experimental data
obtained in the $\gamma
N\rightarrow\Delta(1232)\rightarrow\Delta(1232)\gamma\rightarrow\pi N\gamma$
process employing this procedure [62, 63, 64, 65, 66, 67, 68, 69, 70].

##  IV summary and outlook

Electromagnetic form factors, which describe the response of composite
particles to electromagnetic surveys, provide an important tool for
understanding the structure of bound states in QCD. The magnetic dipole moment
is the leading-order response of a bound system to a weak external magnetic
field. It therefore offers an exquisite laboratory for studying the internal
structure of composite particles, which is governed by the quark-gluon
dynamics of QCD. Therefore, we systematically study the magnetic dipole moment
of controversial states whose internal structure is not elucidated and we try
to offer a different point of view to unravel the internal structure of these
states. Inspired by the $\Omega_{c}$ states observed by the LHCb
Collaboration, we focus on the scenario of the diquark-diquark-antiquark
pentaquark interpretation of the $\Omega_{c}$-like states with both
$J^{P}=\frac{1}{2}^{-}$ and $J^{P}=\frac{3}{2}^{-}$ quantum numbers, and study
the electromagnetic properties of these states with the QCD light-cone sum
rules by employing the distribution amplitudes of the photon. From the
obtained numerical results, we conclude that the magnetic dipole moments of
the $\Omega_{c}$-like states can reflect their inner structures, which can be
used to distinguish their spin-parity quantum numbers. The order of the
magnetic dipole moments obtained for these states implies that they are
accessible by experiments. Measuring the magnetic dipole moments of the
$\Omega_{c}$-like states in future experimental facilities can be very helpful
for understanding the internal organization and identifying the quantum
numbers of these states. We hope that in the future we will have more
experimental data on a variety of physical quantities of these states.

## References

  * [1] H. Albrecht, et al., Observation of a new charmed baryon, Phys. Lett. B 317 (1993) 227–232.  doi:10.1016/0370-2693(93)91598-H.
  * [2] K. W. Edwards, et al., Observation of excited baryon states decaying to Lambda(c)+ pi+ pi-, Phys. Rev. Lett. 74 (1995) 3331–3335.  doi:10.1103/PhysRevLett.74.3331.
  * [3] H. Albrecht, et al., Evidence for Lambda(c)+ (2593) production, Phys. Lett. B 402 (1997) 207–212.  doi:10.1016/S0370-2693(97)00503-0.
  * [4] P. L. Frabetti, et al., An Observation of an Excited State of the $\Lambda_{c}^{+}$ Baryon, Phys. Rev. Lett. 72 (1994) 961–964.  doi:10.1103/PhysRevLett.72.961.
  * [5] M. Artuso, et al., Observation of new states decaying into Lambda+(c) pi- pi+, Phys. Rev. Lett. 86 (2001) 4479–4482.  arXiv:hep-ex/0010080, doi:10.1103/PhysRevLett.86.4479.
  * [6] B. Aubert, et al., Observation of a charmed baryon decaying to D0p at a mass near 2.94-GeV/c**2, Phys. Rev. Lett. 98 (2007) 012001.  arXiv:hep-ex/0603052, doi:10.1103/PhysRevLett.98.012001.
  * [7] K. Abe, et al., Experimental constraints on the possible J**P quantum numbers of the Lambda(c)(2880)+, Phys. Rev. Lett. 98 (2007) 262001.  arXiv:hep-ex/0608043, doi:10.1103/PhysRevLett.98.262001.
  * [8] R. Aaij, et al., Study of the $D^{0}p$ amplitude in $\Lambda_{b}^{0}\to D^{0}p\pi^{-}$ decays, JHEP 05 (2017) 030.  arXiv:1701.07873, doi:10.1007/JHEP05(2017)030.
  * [9] V. V. Ammosov, et al., Observation of the production of charmed Sigma(c)*++ baryons in neutrino interactions at the SKAT bubble chamber, JETP Lett. 58 (1993) 247–251.
  * [10] G. Brandenburg, et al., Observation of two excited charmed baryons decaying into Lambda(c)+ pi+-, Phys. Rev. Lett. 78 (1997) 2304–2308.  doi:10.1103/PhysRevLett.78.2304.
  * [11] R. Mizuk, et al., Observation of an isotriplet of excited charmed baryons decaying to Lambda+(c) pi, Phys. Rev. Lett. 94 (2005) 122002.  arXiv:hep-ex/0412069, doi:10.1103/PhysRevLett.94.122002.
  * [12] P. Avery, et al., Observation of a narrow state decaying into Xi(c)+ pi-, Phys. Rev. Lett. 75 (1995) 4364–4368.  arXiv:hep-ex/9508010, doi:10.1103/PhysRevLett.75.4364.
  * [13] L. Gibbons, et al., Observation of an excited charmed baryon decaying into Xi(c)0 pi+, Phys. Rev. Lett. 77 (1996) 810–813.  doi:10.1103/PhysRevLett.77.810.
  * [14] P. L. Frabetti, et al., Observation of a narrow state decaying into xi0(c) pi+, Phys. Lett. B 426 (1998) 403–410.  doi:10.1016/S0370-2693(98)00348-7.
  * [15] S. E. Csorna, et al., Evidence of new states decaying into Xi-prime(c) pi, Phys. Rev. Lett. 86 (2001) 4243–4246.  arXiv:hep-ex/0012020, doi:10.1103/PhysRevLett.86.4243.
  * [16] J. P. Alexander, et al., Evidence of new states decaying into Xi(c)* pi, Phys. Rev. Lett. 83 (1999) 3390–3393.  arXiv:hep-ex/9906013, doi:10.1103/PhysRevLett.83.3390.
  * [17] R. Aaij, et al., Observation of New $\Xi_{c}^{0}$ Baryons Decaying to $\Lambda_{c}^{+}K^{-}$, Phys. Rev. Lett. 124 (22) (2020) 222001.  arXiv:2003.13649, doi:10.1103/PhysRevLett.124.222001.
  * [18] B. Aubert, et al., A Study of anti-B —$>$ Xi(c) anti-Lambda-(c) and anti-B —$>$ anti-Lambda+(c) anti-Lambda-(c) anti-K decays at BABAR, Phys. Rev. D 77 (2008) 031101.  arXiv:0710.5775, doi:10.1103/PhysRevD.77.031101.
  * [19] R. Chistov, et al., Observation of new states decaying into Lambda(c)+ K- pi+ and Lambda(c)+ K0(S) pi-, Phys. Rev. Lett. 97 (2006) 162001.  arXiv:hep-ex/0606051, doi:10.1103/PhysRevLett.97.162001.
  * [20] B. Aubert, et al., A Study of Excited Charm-Strange Baryons with Evidence for new Baryons Xi(c)(3055)+ and Xi(c)(3123)+, Phys. Rev. D 77 (2008) 012002.  arXiv:0710.5763, doi:10.1103/PhysRevD.77.012002.
  * [21] B. Aubert, et al., Observation of an excited charm baryon Omega(C)* decaying to Omega(C)0 gamma, Phys. Rev. Lett. 97 (2006) 232001.  arXiv:hep-ex/0608055, doi:10.1103/PhysRevLett.97.232001.
  * [22] R. Aaij, et al., Observation of five new narrow $\Omega_{c}^{0}$ states decaying to $\Xi_{c}^{+}K^{-}$, Phys. Rev. Lett. 118 (18) (2017) 182001.  arXiv:1703.04639, doi:10.1103/PhysRevLett.118.182001.
  * [23] R. Aaij, et al., Observation of excited $\Lambda_{b}^{0}$ baryons, Phys. Rev. Lett. 109 (2012) 172003.  arXiv:1205.3452, doi:10.1103/PhysRevLett.109.172003.
  * [24] R. Aaij, et al., Observation of a new baryon state in the ${\Lambda}_{\mathrm{b}}^{0}{\pi}^{+}{\pi}^{-}$ mass spectrum, JHEP 06 (2020) 136\.  arXiv:2002.05112, doi:10.1007/JHEP06(2020)136.
  * [25] R. Aaij, et al., Observation of New Resonances in the $\Lambda_{b}^{0}\pi^{+}\pi^{-}$ System, Phys. Rev. Lett. 123 (15) (2019) 152001.  arXiv:1907.13598, doi:10.1103/PhysRevLett.123.152001.
  * [26] R. Aaij, et al., Observation of Two Resonances in the $\Lambda_{b}^{0}\pi^{\pm}$ Systems and Precise Measurement of $\Sigma_{b}^{\pm}$ and $\Sigma_{b}^{*\pm}$ properties, Phys. Rev. Lett. 122 (1) (2019) 012001.  arXiv:1809.07752, doi:10.1103/PhysRevLett.122.012001.
  * [27] A. M. Sirunyan, et al., Observation of a New Excited Beauty Strange Baryon Decaying to $\Xi^{-}_{\mathrm{b}}\pi^{+}\pi^{-}$, Phys. Rev. Lett. 126 (25) (2021) 252003\.  arXiv:2102.04524, doi:10.1103/PhysRevLett.126.252003.
  * [28] R. Aaij, et al., Observation of a new $\Xi_{b}^{-}$ resonance, Phys. Rev. Lett. 121 (7) (2018) 072002.  arXiv:1805.09418, doi:10.1103/PhysRevLett.121.072002.
  * [29] R. Aaij, et al., Observation of a new $\Xi_{b}^{0}$ state, Phys. Rev. D 103 (1) (2021) 012004.  arXiv:2010.14485, doi:10.1103/PhysRevD.103.012004.
  * [30] R. Aaij, et al., First observation of excited $\Omega_{b}^{-}$ states, Phys. Rev. Lett. 124 (8) (2020) 082002.  arXiv:2001.00851, doi:10.1103/PhysRevLett.124.082002.
  * [31] J. Yelton, et al., Observation of Excited $\Omega_{c}$ Charmed Baryons in $e^{+}e^{-}$ Collisions, Phys. Rev. D 97 (5) (2018) 051102.  arXiv:1711.07927, doi:10.1103/PhysRevD.97.051102.
  * [32] R. Aaij, et al., Observation of New $\Omega$c0 States Decaying to the $\Xi$c+K- Final State, Phys. Rev. Lett. 131 (13) (2023) 131902\.  arXiv:2302.04733, doi:10.1103/PhysRevLett.131.131902.
  * [33] F.-K. Guo, C. Hanhart, U.-G. Meißner, Q. Wang, Q. Zhao, B.-S. Zou, Hadronic molecules, Rev. Mod. Phys. 90 (1) (2018) 015004, [Erratum: Rev.Mod.Phys. 94, 029901 (2022)].  arXiv:1705.00141, doi:10.1103/RevModPhys.90.015004.
  * [34] S. L. Olsen, T. Skwarnicki, D. Zieminska, Nonstandard heavy mesons and baryons: Experimental evidence, Rev. Mod. Phys. 90 (1) (2018) 015003.  arXiv:1708.04012, doi:10.1103/RevModPhys.90.015003.
  * [35] N. Brambilla, S. Eidelman, C. Hanhart, A. Nefediev, C.-P. Shen, C. E. Thomas, A. Vairo, C.-Z. Yuan, The $XYZ$ states: experimental and theoretical status and perspectives, Phys. Rept. 873 (2020) 1–154.  arXiv:1907.07583, doi:10.1016/j.physrep.2020.05.001.
  * [36] H.-Y. Cheng, Charmed baryon physics circa 2021, Chin. J. Phys. 78 (2022) 324–362.  arXiv:2109.01216, doi:10.1016/j.cjph.2022.06.021.
  * [37] H.-X. Chen, W. Chen, X. Liu, Y.-R. Liu, S.-L. Zhu, An updated review of the new hadron states, Rept. Prog. Phys. 86 (2) (2023) 026201.  arXiv:2204.02649, doi:10.1088/1361-6633/aca3b6.
  * [38] V. L. Chernyak, I. R. Zhitnitsky, B meson exclusive decays into baryons, Nucl. Phys. B 345 (1990) 137–172.  doi:10.1016/0550-3213(90)90612-H.
  * [39] V. M. Braun, I. E. Filyanov, QCD Sum Rules in Exclusive Kinematics and Pion Wave Function, Z. Phys. C 44 (1989) 157.  doi:10.1007/BF01548594.
  * [40] I. I. Balitsky, V. M. Braun, A. V. Kolesnichenko, Radiative Decay Sigma+ —$>$ p gamma in Quantum Chromodynamics, Nucl. Phys. B 312 (1989) 509–550.  doi:10.1016/0550-3213(89)90570-1.
  * [41] U. Özdem, Analysis of the $\mathrm{X_{AV}}$ state through its electromagnetic properties (12 2023).  arXiv:2401.00481.
  * [42] U. Özdem, Magnetic dipole moments of the $\Omega$c(3185)0 and $\Omega$c(3327)0 states from molecular perspective, Phys. Lett. B 849 (2024) 138432.  arXiv:2311.02925, doi:10.1016/j.physletb.2023.138432.
  * [43] U. Ozdem, Electromagnetic properties of the $\Sigma_{c}(2800)^{+}$ and $\Lambda_{c}(2940)^{+}$ states via light-cone QCD, Eur. Phys. J. C 83 (11) (2023) 1077.  arXiv:2309.00959, doi:10.1140/epjc/s10052-023-12251-x.
  * [44] U. Özdem, K. Azizi, Magnetic moment of the $X_{1}(2900)$ state in the diquark–antidiquark picture, Eur. Phys. J. A 58 (9) (2022) 171.  arXiv:2202.11466, doi:10.1140/epja/s10050-022-00815-6.
  * [45] U. Özdem, Magnetic moment of the $\Xi_{b}(6227)$ as a molecular pentaquark state, Eur. Phys. J. Plus 137 (1) (2022) 103.  arXiv:2109.09313, doi:10.1140/epjp/s13360-022-02339-w.
  * [46] K. Azizi, U. Özdem, Magnetic dipole moments of the Tcc+ and ZV++ tetraquark states, Phys. Rev. D 104 (11) (2021) 114002.  arXiv:2109.02390, doi:10.1103/PhysRevD.104.114002.
  * [47] K. Azizi, U. t. Özdem, The electromagnetic multipole moments of the possible charm-strange pentaquarks in light-cone QCD, Eur. Phys. J. C 78 (9) (2018) 698\.  arXiv:1807.06503, doi:10.1140/epjc/s10052-018-6187-0.
  * [48] K. Azizi, U. Özdem, The electromagnetic multipole moments of the charged open-flavor $Z_{\bar{c}q}$ states, J. Phys. G 45 (5) (2018) 055003.  arXiv:1802.07711, doi:10.1088/1361-6471/aab56b.
  * [49] H.-J. Wang, Z.-Y. Di, Z.-G. Wang, Analysis of the excited $\Omega$ c states as the $\pm$ pentaquark states with QCD sum rules, Commun. Theor. Phys. 73 (3) (2021) 035201.  doi:10.1088/1572-9494/abc7b1.
  * [50] Z.-G. Wang, J.-X. Zhang, Possible pentaquark candidates: new excited $\Omega_{c}$ states, Eur. Phys. J. C 78 (6) (2018) 503.  arXiv:1804.06195, doi:10.1140/epjc/s10052-018-5989-4.
  * [51] K.-C. Yang, W. Y. P. Hwang, E. M. Henley, L. S. Kisslinger, QCD sum rules and neutron proton mass difference, Phys. Rev. D 47 (1993) 3001–3012.  doi:10.1103/PhysRevD.47.3001.
  * [52] V. M. Belyaev, B. Y. Blok, CHARMED BARYONS IN QUANTUM CHROMODYNAMICS, Z. Phys. C 30 (1986) 151.  doi:10.1007/BF01560689.
  * [53] P. Ball, V. M. Braun, N. Kivel, Photon distribution amplitudes in QCD, Nucl. Phys. B 649 (2003) 263–296.  arXiv:hep-ph/0207307, doi:10.1016/S0550-3213(02)01017-9.
  * [54] D. Antonov, J. E. F. T. Ribeiro, Quark condensate for various heavy flavors, Eur. Phys. J. C 72 (2012) 2179.  arXiv:1209.0408, doi:10.1140/epjc/s10052-012-2179-7.
  * [55] U. Özdem, Electromagnetic form factors of the Bc-like tetraquarks: Molecular and diquark-antidiquark pictures, Phys. Lett. B 838 (2023) 137750.  arXiv:2211.10169, doi:10.1016/j.physletb.2023.137750.
  * [56] U. Özdem, Electromagnetic properties of doubly heavy pentaquark states, Eur. Phys. J. Plus 137 (2022) 936.  arXiv:2201.00979, doi:10.1140/epjp/s13360-022-03125-4.
  * [57] R. L. Workman, et al., Review of Particle Physics, PTEP 2022 (2022) 083C01.  doi:10.1093/ptep/ptac097.
  * [58] B. L. Ioffe, QCD at low energies, Prog. Part. Nucl. Phys. 56 (2006) 232–277.  arXiv:hep-ph/0502148, doi:10.1016/j.ppnp.2005.05.001.
  * [59] S. Narison, $\overline{\rm m}_{c,b,}<\alpha_{s}G^{2}>$ and $\alpha_{s}$ from Heavy Quarkonia, Nucl. Part. Phys. Proc. 300-302 (2018) 153–164.  doi:10.1016/j.nuclphysbps.2018.12.026.
  * [60] J. Rohrwild, Determination of the magnetic susceptibility of the quark condensate using radiative heavy meson decays, JHEP 09 (2007) 073.  arXiv:0708.1405, doi:10.1088/1126-6708/2007/09/073.
  * [61] V. I. Zakharov, L. A. Kondratyuk, L. A. Ponomarev, Bremsstrahlung and determination of electromagnetic parameters of particles, Yad. Fiz. 8 (1968) 783–792.
  * [62] V. Pascalutsa, M. Vanderhaeghen, Magnetic moment of the Delta(1232)-resonance in chiral effective field theory, Phys. Rev. Lett. 94 (2005) 102003.  arXiv:nucl-th/0412113, doi:10.1103/PhysRevLett.94.102003.
  * [63] V. Pascalutsa, M. Vanderhaeghen, Chiral effective-field theory in the Delta(1232) region: I. Pion electroproduction on the nucleon, Phys. Rev. D 73 (2006) 034003.  arXiv:hep-ph/0512244, doi:10.1103/PhysRevD.73.034003.
  * [64] V. Pascalutsa, M. Vanderhaeghen, Chiral effective-field theory in the Delta(1232) region. II. Radiative pion photoproduction, Phys. Rev. D 77 (2008) 014027.  arXiv:0709.4583, doi:10.1103/PhysRevD.77.014027.
  * [65] M. Kotulla, et al., The Reaction gamma p —$>$ pi zero gamma-prime p and the magnetic dipole moment of the delta+(1232) resonance, Phys. Rev. Lett. 89 (2002) 272001.  arXiv:nucl-ex/0210040, doi:10.1103/PhysRevLett.89.272001.
  * [66] D. Drechsel, M. Vanderhaeghen, Magnetic dipole moment of the Delta+ (1232) from the gamma p —$>$ gamma pi0 pi reaction, Phys. Rev. C 64 (2001) 065202.  arXiv:hep-ph/0105060, doi:10.1103/PhysRevC.64.065202.
  * [67] A. I. Machavariani, A. Faessler, A. J. Buchmann, Field-theoretical description of electromagnetic Delta resonance production and determination of the magnetic moment of the $\Delta^{+}(1232)$ resonance by the $ep\to e^{\prime}N^{\prime}\pi^{\prime}\gamma^{\prime}$ and $\gamma p\to N^{\prime}\pi^{\prime}\gamma^{\prime}$ reactions, Nucl. Phys. A 646 (1999) 231–257, [Erratum: Nucl.Phys.A 686, 601–603 (2001)].  doi:10.1016/S0375-9474(98)00611-3.
  * [68] D. Drechsel, M. Vanderhaeghen, M. M. Giannini, E. Santopinto, Inelastic photon scattering and the magnetic moment of the Delta (1232) resonance, Phys. Lett. B 484 (2000) 236–242.  arXiv:nucl-th/0003035, doi:10.1016/S0370-2693(00)00654-7.
  * [69] W.-T. Chiang, M. Vanderhaeghen, S. N. Yang, D. Drechsel, Unitary model for the gamma p —$>$ gamma pi0 p reaction and the magnetic dipole moment of the Delta+ (1232), Phys. Rev. C 71 (2005) 015204.  arXiv:hep-ph/0409078, doi:10.1103/PhysRevC.71.015204.
  * [70] A. I. Machavariani, A. Faessler, Propagator of the Delta resonance and determination of the magnetic moment of the Delta+ from the gamma p —$>$ gamma pi0 p reaction, Phys. Rev. C 72 (2005) 024002.  doi:10.1103/PhysRevC.72.024002.


# On the Role of the Action Space in Robot Manipulation
Learning and Sim-to-Real Transfer

Elie Aljalbout∗, Felix Frank∗, Maximilian Karl, and Patrick van der Smagt1
∗Shared first authorship.1All authors are with the Volkswagen Group, Machine
Learning Research Lab, Munich, Germany.

###### Abstract

We study the choice of action space in robot manipulation learning and sim-to-
real transfer. We define metrics that assess the performance, and examine the
emerging properties in the different action spaces. We train over 250
reinforcement learning (RL) agents in simulated reaching and pushing tasks,
using 13 different control spaces. The choice of action spaces spans popular
choices in the literature as well as novel combinations of common design
characteristics. We evaluate the training performance in simulation and the
transfer to a real-world environment. We identify good and bad characteristics
of robotic action spaces and make recommendations for future designs. Our
findings have important implications for the design of RL algorithms for robot
manipulation tasks, and highlight the need for careful consideration of action
spaces when training and transferring RL agents for real-world robotics.

##  I Introduction

Robot reinforcement learning (RL) provides a way to acquire manipulation
skills without requiring explicit programming of task plans or models of task
objects. This property can enable robot manipulation to be feasible in a wide
variety of tasks and applications. However, due to the complexity of real-
world manipulation tasks, making RL work in these domains can be very
challenging. During policy search, the robot acts almost randomly for a large
period of time in order for the agent to explore its state and action spaces.
This can be a very dangerous and expensive process and should be approached
with care. Alternatively, such skills can be safely learned in simulation and
transferred later to the real world. In recent years, many approaches have
been proposed to overcome the sim-to-real gap [14, 15, 9]. These methods have
been shown to enable the successful transfer of manipulation policies learned
in simulation. Despite all efforts, the gap could not yet be fully bridged or
even understood. This lack of understanding hinders the applicability of sim-
to-real transfer to a wide range of applications.

The majority of previous studies focused on the state space and perception
aspects of robot policy learning and sim-to-real transfer. Our goal is to
understand the action space and control aspects of this problem.

To provide more context, in recent years there has been a shift toward
embedding well-established control principles into the action space, such as
PD controlled joint positions [23] or impedance control in end-effector space
[21]. This was in contrast to earlier interests in end-to-end policies, which
directly output the lowest level possible of control commands such as joint
torques [31]. In this recent trend in robot learning, the policy outputs a
higher-level control command, such as desired joint velocities, which are then
fed to a low-level hand-engineered controller that handles the low-level
control. Such an approach reduces the complexity of the policy, effectively
making learning simpler and more efficient. Nevertheless, the introduction of
these low-level controllers can fundamentally change the nature of the
problem, in some cases even violating some basic assumptions made in the
design of RL algorithms, such as the Markov property. In practice, these
engineered action spaces could consist of multiple control loops and can
abstract some physical phenomena. In addition, there exist many options for
action spaces that encapsulate robot control algorithms.

In this paper, our aim is to understand the role of the action space in
manipulation learning. First, we are interested in the exploration properties
of different spaces. Moreover, we are interested in the emerging properties of
policies trained with different low-level controllers. In addition, we are
interested in the gap created because of this choice. Most importantly, we are
interested in the transferability of policies trained in different control
spaces to the real world. Hence, we design a large-scale study to quantify
these different aspects. Our study includes 13 different action spaces that we
evaluate using multiple metrics, measuring success rate transfer, the
usability of resulting behaviors, and the gap introduced.

##  II Related Work

Recent work in robot learning has shown that the choice of action space can
play an important role in the success and performance of manipulation [21, 4],
flying [17], and locomotion [22, 11] policies. These findings sparked the
development of novel action spaces suitable for different families of tasks.
For instance, for manipulation, the force exerted by the robot on its
environment plays a crucial role. Multiple proposed action spaces for
manipulation allow the policy to control this aspect, either by some direct
means of force application [16, 7, 29] or implicitly via impedance control
[21, 19, 8]. Furthermore, several methods explored the use of movement
primitives in the action space [6, 2]. Another important choice is between
configuration or task-space control. Robot learning literature includes
methods with both configuration [15, 32, 2, 25] and task space [19, 21, 11]
control. Ganapathi et al. [12] propose an approach to implicitly combine these
two modes using a differentiable forward kinematics model. Interestingly, it
seems like the majority of sim-to-real works use configuration space control
[32, 15, 1, 13, 28]. The reason behind this is yet unclear. More recently,
several approaches have been proposed for learning latent action spaces that
can either help reduce the dimensionality of the policy’s control to the task
manifold [33, 5] or serve a different purpose such as coordinating multi-robot
tasks [3].

To better understand this aspect of robot learning, there have been multiple
studies concerning the action space in robotics [30, 17, 22]. These studies
span robotic applications including locomotion [22], manipulation [30], and
flying robot [17]. However, the majority of these studies focus on a very
limited set of action spaces or tasks. Multiple of them, study this problem in
simulation only [22, 30]. In contrast, Kaufmann et al. [17] study the transfer
of flying policies from simulation to the real world with three different
control spaces. Similarly, in this work, we study the role of the action space
for learning arm manipulation policies in simulation, and also their transfer
to the real world. Our study includes 13 different action spaces, spanning new
novel combinations as well as commonly used ones in robot arm manipulation.

Figure 1: The policy outputs an action $a$, which is then transformed into
joint torques $\tau$ using a select controller $f_{as}$. The policy and
controller receive feedback from the environment. Each action space is defined
by the choice of the controller and the way the action is treated in the
controller. The policy runs at a 60Hz frequency and the controller runs at a
frequency of 120Hz and 1KHz in the simulation and the real world respectively.

##  III Methods

In robot learning, the choice of action space involves the choice of a
controller and the frequency and limits of the policy outputs. The frequency
directly defines the reactive capabilities of the policy, and the limits are
typically given by the choice of controller and robot. In this paper, we aim
to study low-level control policies. Hence, we fix the frequency of all action
spaces to 60 Hz. Furthermore, we use the limits that are given by the robot
for the corresponding control variables.

###  III-A Reinforcement Learning

RL automates policy learning by maximizing the cumulative reward in a given
environment [26]. Tasks are usually formulated as Markov Decision Processes
(MDP). A finite-horizon, discounted MDP is defined by the tuple
$\mathcal{M}=(\mathcal{S},\mathcal{A},\mathcal{P},r,\rho_{0},\gamma,T)$, where
$\mathcal{S}$ and $\mathcal{A}$ are the state and action spaces respectively,
$\mathcal{P}:\mathcal{S}\times\mathcal{A}\to\mathcal{S}$ is the transition
dynamics, and $r:\mathcal{S}\times\mathcal{A}\to\mathbb{R}$ the reward. In
addition, we have an initial state distribution $\rho_{0}$, a discount factor
$\gamma\in[0,1]$, and a horizon length $T$. The optimal policy
$\pi:\mathcal{S}\to P(\mathcal{A})$ maximizes the expected discounted reward:

$J(\pi)=E_{\pi}\left[\sum_{t=0}^{T-1}\gamma^{t}r(s_{t},a_{t})\right].$ (1)

In this study, we use the proximal policy optimization (PPO) algorithm [24]
for learning manipulation policies. This choice is based on the popularity of
this algorithm in recent works on sim-to-real transfer [27, 1, 10, 13, 28].

###  III-B Action Spaces

For arm manipulation, one of the most native RL action spaces is one that
expects joint torque (JT) commands. This corresponds to $\tau=a$, where $\tau$
is the vector of joint torques. Such an action space gives the policy full
control over the robot. When given the right observations, such a policy can
internally learn to control the motion and forces exerted by the robot. This
action space was popular in early works using deep RL [31, 18]. However,
learning a policy with this action space can be very complicated since the
policy would need to either understand or implicitly handle both the
kinematics and dynamics of the robot to fulfill the task. This is due to the
fact that the reward is typically expressed using task space properties.

Alternatively, a low-level controller $f_{\mathrm{as}}:\mathcal{A}\longmapsto
T$ can be integrated in the action space to convert higher-level policy
actions into the space of joint torques $T$ of the robot. This concept is
illustrated in Figure 1.

Configuration action spaces consist of all action spaces that expect a
configuration-space action from the policy. At the center of all these action
spaces is the same controller, namely a joint impedance controller (JIC). JIC
regulates the behavior of a robot manipulator’s joints and allows specifying
desired stiffness, damping, and inertia characteristics for each joint. This
allows the robot to be compliant with its environment. The JIC control law is
the following:

$\displaystyle\tau$ $\displaystyle=f_{\mathrm{JIC}}(a)$ (2)
$\displaystyle\tau$ $\displaystyle=K(q_{d}-q)+D(\dot{q_{d}}-\dot{q}),$ (3)

where $\tau$ denotes the commanded joint torque, $q_{d}$ and $\dot{q_{d}}$
denote the desired joint positions and velocities respectively, and $q$ and
$\dot{q}$ denote the actual joint positions and velocities of the robot given
as feedback. $K$ and $D$ are the stiffness and damping matrices. In practice,
we use isotropic gains, i.e., these matrices are diagonal. Given this control
law, we can define two different base configuration action spaces:

  * •

Joint Velocities (JV): sets $f_{\mathrm{JV}}\leftarrow f_{\mathrm{JIC}}$, and
$\dot{q_{d}}=s(a)$ and $q_{d}$ is a first-order integration of $\dot{q_{d}}$;

  * •

Joint Position (JP): sets $f_{\mathrm{JP}}\leftarrow f_{\mathrm{JIC}}$, and
$q_{d}=s(a)$ and $\dot{q_{d}}$ is a first-order differentiation of $q_{d}$.

$s(a)$ is a function that scales the action vector to the limits of the
corresponding output vector. The differentiation and integration steps are
often ignored in RL action space implementations, despite being present in the
most common robot control libraries. Not including these steps in the
simulation would create an additional sim-to-real gap.

Task action spaces are defined using variables that are in the task/Cartesian
space of the robot. In the absence of accurate system identification, we can
use the following Cartesian impedance controller:

$\tau=J(q)^{T}\bigl{(}K(x_{d}-x)+D(\dot{x_{d}}-\dot{x})\bigr{)},$ (4)

where $J(q)$ is the Jacobian matrix for the current robot configuration $q$,
relating joint velocities to Cartesian velocities, $x_{d}$ and $\dot{x_{d}}$
are the desired Cartesian poses and velocities, $x$ and $\dot{x}$ are the
current Cartesian poses and velocities respectively. However, this formulation
can be tricky to use when the $x_{d}$ and $\dot{x_{d}}$ are generated by an RL
policy. This is due to the non-smooth nature of RL action trajectories. Of
course, this problem can be handled by introducing interpolators or cubic
spline fitting. But such solutions involve multiple design choices and
hyperparameters, meaning that an ideal solution is task-specific. Instead, we
transform the Cartesian actions into joint velocities, and then use a joint
impedance controller. We found this approach to be very good at handling the
non-smooth policy actions, without introducing any additional sim-to-real gap.
We have two base task action spaces:

  * •

Cartesian Velocities (CV): sets $\dot{x_{d}}\leftarrow s(a)$, transforms
$\dot{x_{d}}$ into $\dot{q_{d}}$ using inverse kinematics (IK), and then uses
the $f_{\mathrm{JIC}}$ in the same fashion as in the joint velocities action
space;

  * •

Cartesian Position (CP): sets $x_{d}\leftarrow s(a)$, and then uses a
proportional control to obtain $\dot{x_{d}}$ from $x_{d}$. This step naturally
results in smooth $\dot{x_{d}}$. Given $\dot{x_{d}}$, this action space
proceeds as in the CV action space.

We use the pseudoinverse IK method with a null-space controller that pushes
the joints toward their default positions.

Delta action spaces are based on the base action spaces defined previously. In
contrast to the base action spaces, delta action spaces set the control
targets $v_{d}$ relative to the current system feedback or to the control
targets of the previous policy step. This distinction creates two classes of
delta action spaces:

  * •

One-step Integrator (OI$\Delta$): uses the robot feedback $v$ to set
$v_{d}\leftarrow v+c\cdot a\cdot dt$;

  * •

Multi-step Integrator (MI$\Delta$): recurrently sets $v_{d}\leftarrow
v_{d}+c\cdot a\cdot dt$.

Depending on the choice of base action space, $v_{d}$ is a control target
vector, and can correspond to $q_{d}$, $\dot{q_{d}}$, $x_{d}$, or
$\dot{x_{d}}$. Similarly, $v$ is a control feedback vector, and can correspond
to $q$, $\dot{q}$, $x$, or $\dot{x}$. $dt$ is the step duration and $c$ is a
positive constant hyperparameter. Instead of the scaling performed in non-
delta base action spaces, we clip the target $v_{d}$ to the limits of the
corresponding output space after updating it. This means that each base
(configuration or task) action space has two additional variants, resulting in
12 action spaces, not including the joint torques action space. Despite
relying on the base action spaces, the delta variants have their unique
properties. For instance, due to the relative changes in the control targets,
the magnitude of the target change in one step is bound by $c\cdot dt$ given
that $a\in[-1,1]$. This property can be helpful in imposing smoothness
constraints on control target trajectories, even if policy output is
unconstrained. Additionally, if we set $c$ to be the positive bound of the
derivative of the corresponding control feedback variable $v$, each delta
action space would approximately be equivalent to an action space of a higher-
order derivative than its base action space. For instance, a delta joint
velocity action space would be approximately similar to a joint acceleration
action space. However, when sampling actions $a$ from a uniform distribution,
the resulting distribution of control targets can differ between a delta
action space and the base action space it approximates (e.g. $\Delta$JP and
JV). The clipping can also lead to more probability mass on the borders of the
$v_{d}$ space.

###  III-C Metrics

For each action space, we aim to assess its training performance, resulting
sample efficiency, emerging properties such as the usability in the real-world
environment, and the sim-to-real gap it creates. Therefore, we propose
multiple metrics to quantify these different aspects. For assessing training
performance, we look at the episodic rewards (ER) in simulation. This metric
can also show us the sample efficiency of the different action spaces.

To assess the emerging properties of each policy, we look at the number of
times it violates robot constraints, such as the joint acceleration and jerk
constraints. This is especially important since simulated environments rarely
have any mechanisms for enforcing these constraints. In contrast, real-world
robot control implementations can include rate limiters, which ensure that the
control targets would not result in violations of these constraints. Hence, a
policy trained in simulation without such mechanisms could learn to violate
them for the sake of exploration. Implementing these mechanisms in simulation,
on the other hand, can also hinder the policy training since these mechanisms
typically break the Markov assumption. To quantify this property, we use the
expected constraints violations (ECV) defined as

$\mathrm{ECV}(\pi)=E_{s,a\sim\pi}\left[\mathds{1}\left(\sum_{c\in\mathcal{C}}c(s,a)>0\right)\right],$
(5)

where $\mathds{1}$ is an indicator function, $\mathcal{C}$ is the set of all
constraints, and each constraint function $c(s,a)$ returns 1 for violated
constrained and 0 otherwise. Furthermore, we report the normalized tracking
error (NTE) as a measure of the feasibility of the policy’s actions in the
environment,

$\mathrm{NTE}(\pi)=E_{s,a\sim\pi}\frac{|v_{d,t}-v_{t+1}|}{v_{\mathrm{max}}-v_{\mathrm{min}}},$
(6)

where $v_{\mathrm{min}}$ and $v_{\mathrm{max}}$ are respectively the lower and
upper bounds of the control variable $v$. NTE is useful for analysing why
certain action spaces result in better transfer. A high value means that the
policy outputs actions that are hard to achieve in one control step. This
could create an additional sim-to-real gap since control targets that are not
fulfilled in one step can be tracked differently in simulated and real
environments due to the gap in dynamics. We also report the task accuracy
(ACC) measured by the Euclidean distance to the goal. This metric gives a more
detailed view of the performance of a policy than just the success rate.

To assess the sim-to-real gap of each action space we report the offline
trajectory error (OTE) in configuration and task spaces. This metric measures
the joint trajectory error when replaying, in simulation, the actions produced
by the policy when queried in the real world,

$\mathrm{OTE}(\pi)=E_{a,q_{\mathrm{real}}\sim\mathcal{D}_{\mathrm{real}}}|q_{\mathrm{sim}}-q_{\mathrm{real}}|,$
(7)

where $a$ and $q_{\mathrm{real}}$ are actions and joint configurations that
are sampled from the dataset $\mathcal{D}_{\mathrm{real}}$. The latter is
collected by playing a policy $\pi$ in the real-world, and $q_{\mathrm{sim}}$
is the joint configuration obtained when executing $a$ in simulation in an
open-loop fashion, i.e. without a policy.

##  IV Experiments

Figure 2: We show our real-world robot setup for the reaching (left) and the
pushing (right) tasks.

We design experiments to understand the role of different characteristics of
the action space on learning efficiency, sim-to-real gap, emerging properties,
and sim-to-real transfer. Throughout this section, we pose different questions
related to these topics, and answer them to the best of our capability based
on empirical analysis. To answer these questions we evaluate all action spaces
on two arm manipulation tasks, using the 7-degree-of-freedom Franka Emika
Panda robot. The first task is goal reaching. At the beginning of each
episode, a Cartesian goal is sampled in the workspace of the robot, and the
policy needs to move the end-effector towards that goal. This task is ideal
for studying the behavior of policies from all action spaces in the lack of
force interactions with the robot’s environment. The second task is object
pushing. At the beginning of each episode, a goal position is sampled in a
predefined area on the table. The policy needs to push a wooden box towards
that goal. This task involves moving an external object. Unlike the reaching
task, pushing requires physical interaction with the environment. It allows us
to understand the reactive capabilities of each action space and whether it
creates any sim-to-real gap that hinders the transfer of the interaction
behavior. During policy training for pushing in simulation, we perform domain
randomization on the box’s friction and mass parameters. The real-world setup
for both tasks is shown in Figure 2. In both tasks, the observation space of
the policy consists of joint positions, joint velocities, end-effector
Cartesian position, and the goal position. Additionally, in the pushing task
the policy has access to the object’s position and orientation. We train PPO
policies in a simulated environment, using NVIDIA’s Isaac-sim simulator [20].

Action | Reaching (Epochs) |  | Pushing (Epochs)
---|---|---|---
space | 25 | 50 | 75 |  | 25 | 50 | 75
JP |  $470$ $\pm$ | $7$ |  $478$ $\pm$ | $3$ |  $481$ $\pm$ | $3$ |  |  $15$ $\pm$ | $5$ |  $122$ $\pm$ | $82$ |  $208$ $\pm$ | $116$
OI$\Delta$JP |  $474$ $\pm$ | $1$ |  $472$ $\pm$ | $1$ |  $464$ $\pm$ | $3$ |  |  $338$ $\pm$ | $35$ |  $376$ $\pm$ | $17$ |  $388$ $\pm$ | $21$
MI$\Delta$JP |  $387$ $\pm$ | $14$ |  $425$ $\pm$ | $11$ |  $436$ $\pm$ | $6$ |  |  $155$ $\pm$ | $114$ |  $175$ $\pm$ | $150$ |  $114$ $\pm$ | $133$
JV |  $438$ $\pm$ | $13$ |  $459$ $\pm$ | $4$ |  $455$ $\pm$ | $7$ |  |  $381$ $\pm$ | $17$ |  $400$ $\pm$ | $9$ |  $379$ $\pm$ | $39$
OI$\Delta$JV |  $453$ $\pm$ | $3$ |  $457$ $\pm$ | $2$ |  $461$ $\pm$ | $3$ |  |  $332$ $\pm$ | $74$ |  $385$ $\pm$ | $27$ |  $359$ $\pm$ | $42$
MI$\Delta$JV |  $455$ $\pm$ | $4$ |  $465$ $\pm$ | $3$ |  $465$ $\pm$ | $1$ |  |  $243$ $\pm$ | $157$ |  $383$ $\pm$ | $43$ |  $381$ $\pm$ | $16$
JT |  $479$ $\pm$ | $2$ |  $476$ $\pm$ | $3$ |  $474$ $\pm$ | $3$ |  |  $71$ $\pm$ | $158$ |  $116$ $\pm$ | $182$ |  $187$ $\pm$ | $257$
CP |  $456$ $\pm$ | $7$ |  $467$ $\pm$ | $2$ |  $465$ $\pm$ | $4$ |  |  $300$ $\pm$ | $157$ |  $322$ $\pm$ | $167$ |  $379$ $\pm$ | $34$
OI$\Delta$CP |  $470$ $\pm$ | $2$ |  $467$ $\pm$ | $3$ |  $466$ $\pm$ | $3$ |  |  $365$ $\pm$ | $26$ |  $387$ $\pm$ | $13$ |  $395$ $\pm$ | $5$
MI$\Delta$CP |  $409$ $\pm$ | $9$ |  $441$ $\pm$ | $2$ |  $442$ $\pm$ | $3$ |  |  $283$ $\pm$ | $29$ |  $311$ $\pm$ | $7$ |  $307$ $\pm$ | $21$
CV |  $478$ $\pm$ | $2$ |  $476$ $\pm$ | $2$ |  $470$ $\pm$ | $2$ |  |  $410$ $\pm$ | $8$ |  $416$ $\pm$ | $8$ |  $423$ $\pm$ | $5$
OI$\Delta$CV |  $471$ $\pm$ | $3$ |  $468$ $\pm$ | $4$ |  $464$ $\pm$ | $5$ |  |  $396$ $\pm$ | $15$ |  $422$ $\pm$ | $8$ |  $415$ $\pm$ | $13$
MI$\Delta$CV |  $467$ $\pm$ | $2$ |  $467$ $\pm$ | $2$ |  $462$ $\pm$ | $3$ |  |  $393$ $\pm$ | $29$ |  $417$ $\pm$ | $12$ |  $414$ $\pm$ | $7$

Figure 3:  We compare the episodic reward (ER) obtained during training in
simulation for all the studied action spaces. These include joint (J) and
Cartesian (C) action spaces for position (P), velocity (V) and torque (T)
control. They also include delta ($\Delta$) spaces, with one-step (OI) and
multi-step (MI) target integration schemes. _Left_ : Comparing all action
spaces while picking only the best-performing delta variants. The rewards are
shown for reaching and pushing at multiple stages of the training process.
_Right_ : We show learning curves for the pushing task and aggregate action
spaces according to some of the characteristics named above.

The results discussed in this paper are based on 250 agents with different
action spaces and different tasks. However, prior to the final training runs
(of the 250 agents) we spent considerable time optimizing other
hyperparameters, such as learning rates or different reward scales and terms,
to guarantee consistency and a fair comparison over all action spaces. We
matched the simulation to the real environment to the best of our knowledge,
eliminating sim-to-real gaps where possible. This includes, for example,
matching the stiffness of the impedance controllers, sharing implementations
for each action space across both environments, and finding good velocity
limits that allow for safe transfer of policies. This tuning process required
training more than 20,000 agents in simulation and testing more than a 1500
agents in the real world. After training, we evaluate the learned policies in
both tasks in simulation and in the real world. For reaching, we use a fixed
grid of target goals that span the feasible workspace of the robot. For
pushing, we randomly sample goal positions and use the object position from
the previous episode as the initial one from the new run. We only manually
reset the object’s position whenever it gets outside of the predefined area it
was trained to operate in or when the policy fails to push the object away
from its starting location once. We introduced the latter intervention to
avoid heavily skewing the data by randomly sampling a difficult-to-handle
initial object position. In both training and evaluation, we run the
simulation at 120 Hz and we use action repeat to work with the policy output
at 60 Hz. On the real robot, we need an additional low-level safety controller
running at 1 kHz, which also repeats the actions from the policy. In all
action spaces except torque we additionally use a 5 Hz low-pass filter and a
rate limiter to conform to motion limits given by the robot. We tried
implementing the rate limiter in simulation as well, however, the policy
learning deteriorated massively, which we attribute to the rate limiter
breaking the Markov property. For the policy network, we use a 4-layer feed-
forward neural network for all action spaces.

Does the choice of action space affect the exploration behavior during
training in simulation?
We first examine the training performance in simulation. The results can be
seen in the table in Figure 3. The episodic rewards are not directly
representative of the success rate of the policies. This can be seen when
comparing the rewards from Figure 3 to the success rates shown in Table I.
Therefore, we mostly focus on sample efficiency in the current analysis. Since
different action spaces have different characteristics, we aim to understand
the global effect of these characteristics on the sample efficiency during
training. First, we compare Cartesian and joint action spaces. In the reaching
task, both action space groups behave almost identically. However, in the
pushing task, Cartesian action spaces seem to have an advantage in terms of
sample efficiency and maximum reached reward, as can be seen in the top right
plot in Figure 3. This result is most likely due to the spatial nature of the
pushing task, which gives Cartesian action spaces a natural advantage in
exploration.

We next compare the order of derivative of the action space, i.e. positions
vs. velocities vs. torques. Among joint action spaces, joint velocity and its
derivatives have the overall best performance in both tasks. These action
spaces show better sample efficiency and converge to higher reward regions
than their counterparts in the joint action space group as can be seen in the
bottom right plot in Figure 3. JP reaches the highest reward in reaching but
struggles massively in pushing. The same tendency can be observed for
Cartesian action spaces, i.e. velocity action spaces perform better in terms
of sample efficiency and final rewards. The joint torque action space is the
fastest one to converge in the reaching task, however, it fails to solve the
pushing task reliably within the same data budget.

We also compare base action spaces with the two different kinds of delta
action spaces. We observe that multi-step integration delta action spaces
consistently perform the worst, while one-step methods seem to have a slight
advantage. This tendency is less pronounced in the velocity action spaces,
with MI$\Delta$JV being consistently one of the best-performing joint action
spaces in both tasks in simulation. Hence, the current simulation data does
not conclusively favor any of these characteristics (non-delta, OI and MI).

Overall, we note that Cartesian velocity (CV) is the best-performing action
space in simulation. The worst one is multi-step-integration joint position
action space.

TABLE I: Sim-to-real transfer evaluation. We evaluate the success rate (SR) in simulation and the real-world environment, the task accuracy (ACC), the expected constraints violations (ECV), and the offline trajectory error (OTE). Action | Reaching |  | Pushing
---|---|---|---
space |  SR (sim) $\uparrow$ |  SR (real) $\uparrow$ |  ACC [cm] $\downarrow$ |  ECV $\downarrow$ |  OTE [rad] $\downarrow$ |  |  SR (sim) $\uparrow$ |  SR (real) $\uparrow$ |  ACC [cm] $\downarrow$ |  ECV $\downarrow$
JP |  $100$ $\pm$ | $0$ |  $6$ $\pm$ | $6$ |  $1.76$ $\pm$ | $0.14$ |  $100$ $\pm$ | $0$ |  $0.22$ $\pm$ | $0.01$ |  |  $87$ $\pm$ | $12$ |  $4$ $\pm$ | $0$ |  $4.23$ $\pm$ | $0.66$ |  $100$ $\pm$ | $0$
OI$\Delta$JP |  $100$ $\pm$ | $0$ |  $86$ $\pm$ | $9$ |  $1.59$ $\pm$ | $0.16$ |  $14$ $\pm$ | $1$ |  $0.45$ $\pm$ | $0.01$ |  |  $99$ $\pm$ | $3$ |  $48$ $\pm$ | $12$ |  $3.16$ $\pm$ | $0.46$ |  $98$ $\pm$ | $2$
MI$\Delta$JP |  $100$ $\pm$ | $0$ |  $69$ $\pm$ | $2$ |  $2.14$ $\pm$ | $0.13$ |  $0$ $\pm$ | $0$ |  $0.03$ $\pm$ | $0.00$ |  |  $90$ $\pm$ | $7$ |  $36$ $\pm$ | $3$ |  $3.52$ $\pm$ | $0.36$ |  $3$ $\pm$ | $1$
JV |  $100$ $\pm$ | $0$ |  $100$ $\pm$ | $0$ |  $1.17$ $\pm$ | $0.11$ |  $0$ $\pm$ | $0$ |  $0.03$ $\pm$ | $0.00$ |  |  $97$ $\pm$ | $5$ |  $90$ $\pm$ | $5$ |  $2.04$ $\pm$ | $0.32$ |  $66$ $\pm$ | $9$
OI$\Delta$JV |  $100$ $\pm$ | $0$ |  $97$ $\pm$ | $5$ |  $1.35$ $\pm$ | $0.13$ |  $13$ $\pm$ | $11$ |  $0.18$ $\pm$ | $0.01$ |  |  $99$ $\pm$ | $3$ |  $88$ $\pm$ | $13$ |  $1.56$ $\pm$ | $0.13$ |  $75$ $\pm$ | $4$
MI$\Delta$JV |  $100$ $\pm$ | $0$ |  $93$ $\pm$ | $9$ |  $1.98$ $\pm$ | $0.08$ |  $37$ $\pm$ | $37$ |  $0.03$ $\pm$ | $0.00$ |  |  $100$ $\pm$ | $0$ |  $96$ $\pm$ | $4$ |  $1.66$ $\pm$ | $0.11$ |  $55$ $\pm$ | $25$
JT |  $100$ $\pm$ | $0$ |  $64$ $\pm$ | $27$ |  $1.97$ $\pm$ | $0.25$ |  $22$ $\pm$ | $6$ |  $0.52$ $\pm$ | $0.03$ |  |  $40$ $\pm$ | $49$ |  - $\pm$ | - |  - $\pm$ | - |  $89$ $\pm$ | $21$
CP |  $99$ $\pm$ | $2$ |  $25$ $\pm$ | $4$ |  $1.71$ $\pm$ | $0.29$ |  $26$ $\pm$ | $9$ |  $0.16$ $\pm$ | $0.01$ |  |  $100$ $\pm$ | $0$ |  $57$ $\pm$ | $9$ |  $2.34$ $\pm$ | $0.39$ |  $92$ $\pm$ | $3$
OI$\Delta$CP |  $89$ $\pm$ | $2$ |  $44$ $\pm$ | $17$ |  $2.02$ $\pm$ | $0.32$ |  $21$ $\pm$ | $1$ |  $0.27$ $\pm$ | $0.01$ |  |  $99$ $\pm$ | $3$ |  $55$ $\pm$ | $17$ |  $2.86$ $\pm$ | $0.49$ |  $35$ $\pm$ | $16$
MI$\Delta$CP |  $100$ $\pm$ | $0$ |  $68$ $\pm$ | $2$ |  $1.85$ $\pm$ | $0.08$ |  $13$ $\pm$ | $4$ |  $0.28$ $\pm$ | $0.01$ |  |  $100$ $\pm$ | $0$ |  $73$ $\pm$ | $7$ |  $2.36$ $\pm$ | $0.14$ |  $99$ $\pm$ | $0$
CV |  $100$ $\pm$ | $0$ |  $43$ $\pm$ | $13$ |  $1.76$ $\pm$ | $0.41$ |  $12$ $\pm$ | $2$ |  $0.35$ $\pm$ | $0.02$ |  |  $99$ $\pm$ | $3$ |  $73$ $\pm$ | $9$ |  $2.69$ $\pm$ | $0.58$ |  $93$ $\pm$ | $10$
OI$\Delta$CV |  $97$ $\pm$ | $2$ |  $24$ $\pm$ | $2$ |  $2.13$ $\pm$ | $0.13$ |  $17$ $\pm$ | $3$ |  $0.40$ $\pm$ | $0.01$ |  |  $99$ $\pm$ | $3$ |  $61$ $\pm$ | $19$ |  $2.67$ $\pm$ | $0.30$ |  $99$ $\pm$ | $0$
MI$\Delta$CV |  $99$ $\pm$ | $2$ |  $58$ $\pm$ | $8$ |  $1.82$ $\pm$ | $0.08$ |  $4$ $\pm$ | $0$ |  $0.13$ $\pm$ | $0.01$ |  |  $86$ $\pm$ | $5$ |  $74$ $\pm$ | $4$ |  $2.84$ $\pm$ | $0.45$ |  $6$ $\pm$ | $2$

What properties naturally emerge due to the choice of action space?
After training the policies in simulation we evaluate them on a real robotic
setup. For the pushing task, we were not able to obtain joint torque policies
that are safe to run in the real world without damaging the robot. We made
multiple attempts to produce safely deployable joint torque policies, for
instance, by introducing different penalties or increasing the policy’s
control frequency. Despite all efforts, all joint torque policies were very
jerky or aggressive when deployed on the real robot. The main difficulty was
to obtain policies that are safe to deploy in a task where the end effector
needs to remain in close proximity to a rigid surface, e.g. the table.
Therefore, we exclude the joint toque action space from our real-world pushing
experiments. The reaching task was less safety-critical. Hence, we managed to
evaluate torque policies on real-world reaching. For all other action spaces,
we look at their sim-to-real transfer capabilities. In Table I, we report the
metrics introduced in Section III-C to quantify the sim-to-real gap and the
performance in the real world. One common challenge when learning manipulation
skills is to obtain smooth policies that do not violate the velocity,
acceleration, and jerk constraints of the robot. Based on the results in Table
I, we observe that different action spaces yield different ECV metrics. JV and
its derivatives (OI$\Delta$JV and MI$\Delta$JV) have on average the lowest ECV
score in both tasks. Vanilla JP results in the highest possible ECV. This
means that deploying this action space in the real world would always result
in some form of constraint violation. In the absence of safety mechanisms
(such as rate limiters and low-pass filters) these violations can be very
harmful. If such mechanisms are implemented, this behavior would increase the
sim-to-real gap further because of how these violations trigger the safety
mechanisms. In contrast, MI$\Delta$JP, seems to have the lowest ECV, which was
also evident when evaluating this action space in the real world.

Figure 4: We show the the robustness of delta action spaces to the velocity
limit hyperparameter in the pushing task on our real setup. We compare one-
step and multi-step integration. The errorbars indicate one standard deviation
of the respective variables.

Does the action space affect the sim-to-real gap?
First, we look at the offline trajectory error in the reaching task, which
gives us a proxy measure of the sim-to-real gap. We observe that the OTE
varies tremendously from one action space to another, despite the fact that
the data used to compute this measure are based on the same starting and goal
positions in all action spaces. This confirms that the choice of action space
does indeed contribute to the sim-to-real gap. Looking closer at the results,
we observe that JT exhibits the highest OTE. This is due to the behavior of
this action space being dictated solely by the dynamics of the robot, which is
different in simulation and in the real world. Unlike all the other action
spaces, JT does not include any feedback loops outside of the policy. Based on
this result, one would expect that more feedback loops should help reduce the
sim-to-real gap. However, the opposite can be seen in the data. For instance,
Cartesian action spaces have on average one additional feedback loop compared
to the joint action spaces. Their OTE is, however, higher on average. This is
due to the fact that feedback loops have different effects in simulation than
in the real world. In turn, this means that a good, highly-reactive feedback
loop is beneficial to overcome the dynamics gap, but adding more could
potentially contribute further to the sim-to-real gap. Comparing OI$\Delta$
and MI$\Delta$ action spaces we notice that the latter consistently have a
smaller OTE. Their OTE is even smaller than the corresponding base action
spaces. This effect is potentially due to the integral term embedded in these
action spaces. Executing the same actions results in the same final goal given
to the lower-level feedback loops, which is unique compared to all other
action spaces.

Which action space characteristics are good for sim-to-real transfer?
We compare the success rates in simulation and the real world. We observe that
the success rate in simulation is not directly reflected in the real world.
This is clear when comparing the ordering of success rates in both domains.
Furthermore, we notice that certain action space characteristics are clearly
advantageous for sim-to-real transfer. For instance, velocity-based action
spaces tend to keep a high success rate when transferred to the real world. In
contrast, position-based action spaces do not typically transfer well. This is
especially the case for JP, which loses almost all of its performance when
transferred. These results are consistent across both studied tasks. In
addition, velocity-based action spaces are on average more accurate in
fulfilling the task as can be seen when comparing the ACC score in Table I.

Additionally, we can see that delta action spaces transfer better than their
corresponding base spaces. The difference between OI and MI delta action
spaces depends on the task and the exact variants of the action space.
However, OI$\Delta$ action spaces required less tuning than MI$\Delta$ spaces
and have shown to be more robust to the choice of hyperparameters. This can be
seen in Figure 4. When varying the velocity limits for $\Delta$JV action
spaces, OI$\Delta$JV showed to be less sensitive to the chosen value. We
attribute this to the lower tracking error of this action space as shown in
the bottom plot in Figure 4. In general, OI$\Delta$ action spaces naturally
lead to a lower tracking error since they integrate the policy actions into
control targets based on the current feedback of the system. In contrast,
MI$\Delta$ action spaces integrate the policy actions into control targets
based on the previous control target. This in turn leads to a bigger gap
between the control target and the state of the system, and hence a larger
tracking error. While a lower NTE helps make JV action spaces more robust to
hyperparameters, it has an even larger effect on the best possible transfer
performance in JP action spaces, as shown in Figure 5. This tendency is also
evident when comparing the base spaces, as shown in Figure 6. Finally, joint
velocity show better transfer than Cartesian velocity action spaces, while the
opposite is true for position-based action spaces.

Figure 5: We compare joint position-based action spaces in the pushing task
and show their influence on the normalized tracking error and the resulting
effect on success rate in the real robot setup.

In summary, our data shows that two characteristics mostly influence the
transfer capability of an action space. The first one is the order of the
derivative of the control variables. With the exception of joint torque
control, which is problematic for other reasons, an action space that controls
a higher order derivative transfers better. The second characteristic is the
emerging tracking error of the action space’s control variable(s). Our data
strongly indicates that an action space which yields or naturally limits the
tracking error transfers better. This last property can be enforced by the
action space design, for instance, by reducing the magnitude of jumps of the
corresponding control targets. The latter can be controlled by the scaling of
the actions in delta action space, or by increasing the stiffness if the task
allows. For action spaces that do not naturally allow for controlling this
property, one could attempt to enforce a smaller NTE by means of
rewards/penalties on the actions’ magnitude and smoothness. However, based on
our experience, the tuning process for the resulting additional
hyperparameters (of such reward terms) can be very difficult.

Figure 6: We show the influence of the normalized tracking error on sim-to-
real transfer in terms of success rate for non-delta action spaces in the
reaching task.

Is there a consistently best-performing action space?
Overall, we notice that joint velocity action spaces seem to have the best
performance in sim-to-real transfer. These action spaces have on average the
lowest OTE, ACC and ECV and transfer the best to the real world. They also
required the least tuning to work, making them the most suited action spaces
for manipulation learning (among the studied options). This result is
consistent with our previous findings concerning favorable characteristics,
i.e. JV-based action space benefits from a higher-quality feedback loop due to
their configuration-space control, and can more easily generate high forces
for interaction than position-based action space due to controlling a higher-
order derivative.

##  V Conclusion

We studied the role of the action space in learning robotic arm manipulation
skills. We designed a study that includes 13 action spaces that we used for
training RL policies in simulation. We looked at the exploration behavior
under the different action spaces and observed that Cartesian spaces as well
as ones that control a higher-order derivative have an exploration advantage.
Furthermore, we observe that different action spaces lead to different
emergent properties such as the safety of their transfer and their rate of
constraints violations. However, we notice that the success rate of policies
in simulation does not dictate their performance in the real world. Our study
further shows that the characteristics that mostly affect the sim-to-real
transfer are the order of derivative and the emerging tracking error of the
action space. Joint velocity-based action spaces show very favorable
properties, and overall the best transfer performance. Our results show that
the choice of action space plays a central role in learning manipulation
policies and the transfer of such policies to the real world. Future work can
build on our findings for selecting action spaces for manipulation
applications or for designing more suitable alternatives.

## ACKNOWLEDGMENT

The authors would like to thank Alexandros Paraschos and Philip Becker-Ehmck
for their help and support.

## References

  * [1] Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al.  Solving rubik’s cube with a robot hand.  arXiv preprint arXiv:1910.07113, 2019.
  * [2] Elie Aljalbout, Ji Chen, Konstantin Ritt, Maximilian Ulmer, and Sami Haddadin.  Learning vision-based reactive policies for obstacle avoidance.  In Conference on Robot Learning, pages 2040–2054. PMLR, 2021.
  * [3] Elie Aljalbout, Maximilian Karl, and Patrick van der Smagt.  Clas: Coordinating multi-robot manipulation with central latent action spaces.  In Learning for Dynamics and Control Conference, pages 1152–1166. PMLR, 2023.
  * [4] Marvin Alles and Elie Aljalbout.  Learning to centralize dual-arm assembly.  Frontiers in Robotics and AI, 9:830007, 2022.
  * [5] Arthur Allshire, Roberto Martín-Martín, Charles Lin, Shawn Manuel, Silvio Savarese, and Animesh Garg.  Laser: Learning a latent action space for efficient reinforcement learning.  In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 6650–6656. IEEE, 2021.
  * [6] Shikhar Bahl, Mustafa Mukadam, Abhinav Gupta, and Deepak Pathak.  Neural dynamic policies for end-to-end sensorimotor learning.  Advances in Neural Information Processing Systems, 33:5058–5069, 2020.
  * [7] Cristian Camilo Beltran-Hernandez, Damien Petit, Ixchel Georgina Ramirez-Alpizar, Takayuki Nishi, Shinichi Kikuchi, Takamitsu Matsubara, and Kensuke Harada.  Learning force control for contact-rich manipulation tasks with rigid position-controlled robots.  IEEE Robotics and Automation Letters, 5(4):5709–5716, 2020.
  * [8] Miroslav Bogdanovic, Majid Khadiv, and Ludovic Righetti.  Learning variable impedance control for contact sensitive tasks.  IEEE Robotics and Automation Letters, 5(4):6129–6136, 2020.
  * [9] João Borrego, Pechirra de Carvalho Borrego, Plinio Moreno López, José Alberto Rosado, and Luís M. M. Custódio.  Mind the gap! bridging the reality gap in visual perception and robotic grasping with domain randomisation.  2018\.
  * [10] Yevgen Chebotar, Ankur Handa, Viktor Makoviychuk, Miles Macklin, Jan Issac, Nathan Ratliff, and Dieter Fox.  Closing the sim-to-real loop: Adapting simulation randomization with real world experience.  In 2019 International Conference on Robotics and Automation (ICRA), pages 8973–8979. IEEE, 2019.
  * [11] Helei Duan, Jeremy Dao, Kevin Green, Taylor Apgar, Alan Fern, and Jonathan Hurst.  Learning task space actions for bipedal locomotion.  In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 1276–1282. IEEE, 2021.
  * [12] Aditya Ganapathi, Pete Florence, Jake Varley, Kaylee Burns, Ken Goldberg, and Andy Zeng.  Implicit kinematic policies: Unifying joint and cartesian action spaces in end-to-end robot learning.  In 2022 International Conference on Robotics and Automation (ICRA), pages 2656–2662. IEEE, 2022.
  * [13] Ankur Handa, Arthur Allshire, Viktor Makoviychuk, Aleksei Petrenko, Ritvik Singh, Jingzhou Liu, Denys Makoviichuk, Karl Van Wyk, Alexander Zhurkevich, Balakumar Sundaralingam, et al.  Dextreme: Transfer of agile in-hand manipulation from simulation to reality.  In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 5977–5984. IEEE, 2023.
  * [14] Eric Heiden, David Millard, Erwin Coumans, and Gaurav S. Sukhatme.  Augmenting differentiable simulators with neural networks to close the sim2real gap.  ArXiv, abs/2007.06045, 2020.
  * [15] Jemin Hwangbo, Joonho Lee, Alexey Dosovitskiy, Dario Bellicoso, Vassilios Tsounis, Vladlen Koltun, and Marco Hutter.  Learning agile and dynamic motor skills for legged robots.  Science Robotics, 4(26):eaau5872, 2019.
  * [16] Mrinal Kalakrishnan, Ludovic Righetti, Peter Pastor, and Stefan Schaal.  Learning force control policies for compliant manipulation.  In 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 4639–4644. IEEE, 2011.
  * [17] Elia Kaufmann, Leonard Bauersfeld, and Davide Scaramuzza.  A benchmark comparison of learned control policies for agile quadrotor flight.  In 2022 International Conference on Robotics and Automation (ICRA), pages 10504–10510. IEEE, 2022.
  * [18] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra.  Continuous control with deep reinforcement learning.  arXiv preprint arXiv:1509.02971, 2015.
  * [19] Jianlan Luo, Eugen Solowjow, Chengtao Wen, Juan Aparicio Ojea, Alice M Agogino, Aviv Tamar, and Pieter Abbeel.  Reinforcement learning on variable impedance controller for high-precision robotic assembly.  In 2019 International Conference on Robotics and Automation (ICRA), pages 3080–3087. IEEE, 2019.
  * [20] Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al.  Isaac gym: High performance gpu-based physics simulation for robot learning.  arXiv preprint arXiv:2108.10470, 2021.
  * [21] Roberto Martín-Martín, Michelle A Lee, Rachel Gardner, Silvio Savarese, Jeannette Bohg, and Animesh Garg.  Variable impedance control in end-effector space: An action space for reinforcement learning in contact-rich tasks.  In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1010–1017. IEEE, 2019.
  * [22] Xue Bin Peng and Michiel Van De Panne.  Learning locomotion skills using deeprl: Does the choice of action space matter?  In Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation, pages 1–13, 2017.
  * [23] N. Rudin, David Hoeller, Philipp Reist, and Marco Hutter.  Learning to walk in minutes using massively parallel deep reinforcement learning.  ArXiv, abs/2109.11978, 2021.
  * [24] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  arXiv preprint arXiv:1707.06347, 2017.
  * [25] Laura Smith, Ilya Kostrikov, and Sergey Levine.  A walk in the park: Learning to walk in 20 minutes with model-free reinforcement learning.  arXiv preprint arXiv:2208.07860, 2022.
  * [26] Richard S Sutton and Andrew G Barto.  Reinforcement learning: An introduction.  MIT press, 2018.
  * [27] Jie Tan, Tingnan Zhang, Erwin Coumans, Atil Iscen, Yunfei Bai, Danijar Hafner, Steven Bohez, and Vincent Vanhoucke.  Sim-to-real: Learning agile locomotion for quadruped robots.  Robotics: Science and Systems, 2018.
  * [28] Bingjie Tang, Michael A Lin, Iretiayo Akinola, Ankur Handa, Gaurav S Sukhatme, Fabio Ramos, Dieter Fox, and Yashraj Narang.  Industreal: Transferring contact-rich assembly tasks from simulation to reality.  arXiv preprint arXiv:2305.17110, 2023.
  * [29] Maximilian Ulmer, Elie Aljalbout, Sascha Schwarz, and Sami Haddadin.  Learning robotic manipulation skills using an adaptive force-impedance action space.  arXiv preprint arXiv:2110.09904, 2021.
  * [30] Patrick Varin, Lev Grossman, and Scott Kuindersma.  A comparison of action spaces for learning manipulation tasks.  In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 6015–6021. IEEE, 2019.
  * [31] Niklas Wahlström, Thomas B Schön, and Marc Peter Deisenroth.  From pixels to torques: Policy learning with deep dynamical models.  arXiv preprint arXiv:1502.02251, 2015.
  * [32] Zhaoming Xie, Patrick Clary, Jeremy Dao, Pedro Morais, Jonanthan Hurst, and Michiel Panne.  Learning locomotion skills for cassie: Iterative design and sim-to-real.  In Conference on Robot Learning, pages 317–329. PMLR, 2020.
  * [33] Wenxuan Zhou, Sujay Bajracharya, and David Held.  Plas: Latent action space for offline reinforcement learning.  In Conference on Robot Learning, 2020.

We provide more details concerning our environments. For the reaching task, we
define the reward function

$\displaystyle r_{reach}(s_{t},a_{t})$
$\displaystyle=rew_{reach}(s_{t},a_{t})-pen_{reach}(s_{t},a_{t})$
$\displaystyle rew_{reach}$ $\displaystyle=r_{dist}+r_{exact}$ $\displaystyle
pen_{reach}$ $\displaystyle=r_{vel}+r_{smooth}+r_{neutral}+r_{limit}$
$\displaystyle r_{dist}(s_{t},a_{t})$
$\displaystyle=\lambda_{r}\cdot\frac{1}{1+\left\lVert
s_{t}-g\right\rVert_{2}^{2}}$ $\displaystyle r_{exact}(s_{t},a_{t})$
$\displaystyle=\mathds{1}(\left\lVert
s_{t}-g\right\rVert_{2}<\epsilon)(\lambda_{\epsilon}+\frac{1}{1+100\dot{q}^{2}})$
$\displaystyle r_{vel}(s_{t},a_{t})$
$\displaystyle=\lambda_{q}\cdot\left\lVert\dot{q}_{t}\right\rVert_{2}^{2}$
$\displaystyle r_{neutral}(s_{t},a_{t})$
$\displaystyle=\lambda_{n}\cdot\left\lVert q_{def}-q\right\rVert_{2}$
$\displaystyle r_{limit}(s_{t},a_{t})$ $\displaystyle=\lambda_{l}\cdot
e^{-30(q-q_{lim})^{2}}$ $\displaystyle r_{smooth}(s_{t},a_{t})$
$\displaystyle=\lambda_{s}\cdot\left\lVert a_{t}-a_{t-1}\right\rVert_{2},$

where we use the Euclidean norm as a distance metric. $g$ is the end-effector
goal position, $q_{def}$ is the default joint positions vector, $\epsilon$ is
a small positive constant, $\lambda_{r}$ and $\lambda_{\epsilon}$ scale the
reach and exact reach reward, $\lambda_{q}$ and $\lambda_{s}$ are positive
scalars for the penalties on the velocity magnitude and smoothness of the
action respectively and $\mathds{1}$ is an indicator function, $lamda_{n}$ is
a scalar for the penalties on divergence from the default joint position, and
$\lambda_{l}$ is a scalar for the joint limit avoidance penalty. For pushing
we have a different reward,

$\displaystyle r_{\text{push}}(s_{t},a_{t})$
$\displaystyle=rew_{\text{push}}(s_{t},a_{t})-pen_{\text{push}}(s_{t},a_{t})$
$\displaystyle rew_{\text{push}}$
$\displaystyle=r_{dist}+r_{exact}+r_{\text{push}}$ $\displaystyle
pen_{\text{push}}$
$\displaystyle=r_{vel}+r_{smooth}+r_{neutral}+r_{limit}+r_{col}$
$\displaystyle r_{col}(s_{t},a_{t})$
$\displaystyle=\lambda_{c}\cdot\mathds{1}(z_{ee}<0.02),$

where $\lambda_{c}$ is a scalar for the table collision penalty, $z_{ee}$ is
the end-effector’s z-position, $r_{dist}$ and $r_{exact}$ use the object
position as goal, and $r_{\text{push}}$ is defined exactly as $r_{dist}$, but
measures the distance between the object’s position and the pushing goal
position.

##  Appendix A Recommendations & Takeaways