# Can a star be smaller than a black hole of the same mass?

Shoulong Li<EMAIL_ADDRESS>Department of Physics, Synergetic
Innovation Center for Quantum Effect and Applications, and Institute of
Interdisciplinary Studies, Hunan Normal University, Changsha, 410081, China
H. Lü<EMAIL_ADDRESS>Center for Joint Quantum Studies and Department of
Physics, School of Science, Tianjin University, 135 Yaguan Road, Tianjin
300350, China  Joint School of National University of Singapore and Tianjin
University, International Campus of Tianjin University, Binhai New City,
Fuzhou 350207, China  Yong Gao<EMAIL_ADDRESS>Kavli Institute for
Astronomy and Astrophysics, Peking University, Beijing, 100871, China
Department of Astronomy, School of Physics, Peking University, Beijing,
100871, China  Rui Xu<EMAIL_ADDRESS>Department of Astronomy, Tsinghua
University, Beijing, 100084, China  Kavli Institute for Astronomy and
Astrophysics, Peking University, Beijing, 100871, China  Lijing Shao
<EMAIL_ADDRESS>Kavli Institute for Astronomy and Astrophysics, Peking
University, Beijing, 100871, China  National Astronomical Observatories,
Chinese Academy of Sciences, Beijing, 100012, China  Hongwei Yu
<EMAIL_ADDRESS>Department of Physics, Synergetic Innovation Center for
Quantum Effect and Applications, and Institute of Interdisciplinary Studies,
Hunan Normal University, Changsha, 410081, China

###### Abstract

It is commonly believed that black holes are the smallest self-gravitating
objects of the same mass in the Universe. Here, we demonstrate, in a subclass
of higher-order pure gravities known as quasi-topological gravity, that by
modifying general relativity (GR) to reduce the strength of gravity in strong-
field regimes while keeping GR unchanged in weak-field regimes, it is possible
for stars to collapse to radii less than $2M$ while still maintaining
equilibrium between gravity and pressure gradients, leading to physically-
reasonable neutron stars smaller in size than a black hole of the same mass.
We present concrete solutions for such objects and discuss some of their
observational consequences. These objects may furnish new avenues for
understanding the nature of gravity in strong-field regimes and leave imprints
on gravitational wave echoes from compact binary mergers. An observation of
these imprints may constitute evidence for new physics beyond GR when effects
of gravity in strong-field regimes are concerned.

Black holes, as an inevitable consequence of Einstein’s theory of General
Relativity (GR) [1] and one of the most enigmatic objects in the cosmos,
possess two striking characteristics, i.e, all the mass is concentrated at the
center with an infinite density signifying a spacetime singularity, and there
exists an event horizon as a defining feature, which ensures that the
singularity remains unobservable, and thus preserves the integrity of the
external spacetime and keeping it free from anomalies. Black holes as
predicted by GR provide a platform to probe strong gravitational fields in
extreme regimes, and are found to be capable of successfully explaining
various astrophysical observations with effects of gravity in strong-field
regimes, including gravitational-wave signals from compact binary mergers [2,
3, 4, 5, 6] and images of light-encircled shadows [7, 8].

It is interesting to note that among self-gravitating objects of the same
mass, the behavior of a highly compact object without an event horizon
resembles that of a black hole, when its size is small enough to possess a
clean photon sphere [9, 10, 11, 12, 13, 14]. This resemblance makes it
challenging to distinguish, using current gravitational-wave and
electromagnetic observations, black holes and horizonless objects nearly as
compact, since these observations lack sensitivity to the internal structure
within the photon sphere. This suggests that the highly compact objects
believed to account for the observations might be horizonless objects with
sizes comparable to or even smaller than black holes.

Hence, on one hand, theoretically, there are fundamental puzzles arising from
black holes, i.e., the notorious information paradox [15] due to the presence
of an event horizon which results in the loss of unitarity in Hawking
radiation and the undesirable existence of the spacetime singularity where all
the known physical laws are presumably no longer valid. And on the other hand,
observationally, no direct evidence for existence of an event horizon has been
provided by observations to date [9, 14]. Therefore, it is highly desirable to
look for regular horizonless objects nearly as compact or even more compact
than black holes, which are capable of explaining the observations that are
currently assumed to be explained by black holes and providing new avenues for
probing effects of gravity in strong-field regimes.

It is well established that, within the framework of GR, horizonless objects
with sizes equal to or slightly larger than that of black holes of the same
mass may exist when matter with exotic properties is considered [12], but
achieving horizonless objects smaller in size than equivalently massive black
holes is impossible. The underlying physical reason is as follows. Take the
static spherically-symmetric (SSS) case as an example. Such objects cannot
form because the gravitational force is so strong that the star will
inevitably collapse into a singularity when the radius $R$ of a star with mass
$M$ becomes smaller than the Schwarzschild radius $2M$ as a result of the fact
that nothing can travel faster than the speed of light, leading to the
formation of a black hole [16]. Here, the “geometrized units” are used in
which the speed of light $c$ and Newton’s gravitational constant $G$ are set
to unity. Moreover, even if a star with a radius $R<2M$ were to form, the
vacuum region with the radius ranging from the star’s radius $R$ to $2M$ would
lack a physically-reasonable metric description since the Birkhoff’s theorem
indicates that the only SSS asymptotically-flat vacuum solution of GR is the
Schwarzschild metric. As a result, it is widely believed that black holes are
the smallest self-gravitating objects of the same mass in the Universe.

Now questions arise naturally as to whether there exist highly compact objects
in modified theories of gravity, which are smaller than black holes, and what
the observational consequences are, which may signal that we need to go beyond
Einstein when effects of gravity in strong-field regimes are concerned [17,
18, 19, 20], if they do exist. These are what we aim to address in the current
manuscript. We show that by modifying GR to reduce the strength of gravity in
strong-field regimes while keeping GR unchanged in weak-field ones, and break
the Birkhoff’s theorem, it is possible for stars to collapse to radii less
than $2M$ while still maintaining equilibrium between gravity and pressure
gradients, and allow for their external vacuum region to be described by non-
black-hole metrics at the same time. We present, for the first time,
physically-reasonable solutions for horizonless objects smaller than black
holes of the same mass within the framework of a subclass of high-order pure
gravities, known as quasi-topological gravity (QTG) [21] and discuss their
observational consequences.

The simplest example of QTG is obtained by modifying GR with an inclusion of
cubic Ricci polynomial invariants, and the Lagrangian for this theory is [21]

$L=R+\lambda(R^{3}-6RR_{\mu\nu}R^{\mu\nu}+8{R_{\mu}}^{\nu}{R_{\nu}}^{\gamma}{R_{\gamma}}^{\mu})\,,$
(1)

where $R$ and $R_{\mu\nu}$ represent the Ricci scalar and Ricci tensor
respectively, and $\lambda$ is the overall coupling constant of the quasi-
topological combination. The equations of motion (EOM) derived from Eq. (1)
with matter are given by

$P_{\mu\alpha\beta\gamma}{R_{\nu}}^{\alpha\beta\gamma}-\frac{1}{2}g_{\mu\nu}L-2\nabla^{\alpha}\nabla^{\beta}P_{\mu\alpha\beta\nu}=8\pi
T_{\mu\nu}\,,$ (2)

with $P_{\mu\alpha\beta\gamma}\equiv{\partial L}/{\partial
R^{\mu\alpha\beta\gamma}}$, where $T_{\mu\nu}$ and $R^{\mu\alpha\beta\gamma}$
represent the energy-momentum tensor and the Riemann tensor, respectively.

Now suppose that an SSS object is composed of matter with a one-parameter
equation of state (EOS) $p(r)=E(\rho(r))$, where $\rho$ and $p$ represent the
density and pressure respectively, and the energy-momentum tensor is expressed
as ${T}_{\mu\nu}=\textup{diag}\\{-\rho,p,p,p\\}$. Both interior and exterior
regions of the SSS object are assumed to described by the following generic
metric

$ds^{2}=-h(r)dt^{2}+f(r)^{-1}dr^{2}+r^{2}(d\theta^{2}+\sin^{2}\theta
d\phi^{2})\,,$ (3)

in the Schwarzschild coordinates $(t,r,\theta,\phi)$. By substituting the
metric (3) into Eq. (2), we obtain the modified Tolman-Oppenheimer-Volkoff
(TOV) equilibrium equations, which can be expressed as a coupled system of
nonlinear higher-derivative ordinary differential equations,

$\displaystyle h^{\prime\prime\prime}$ $\displaystyle=$ $\displaystyle
F_{1}(r,\rho,h^{\prime\prime},h^{\prime},h,f^{\prime\prime},f^{\prime},f)\,,$
$\displaystyle f^{\prime\prime\prime}$ $\displaystyle=$ $\displaystyle
F_{2}(r,\rho,h^{\prime\prime},h^{\prime},h,f^{\prime\prime},f^{\prime},f)\,,$
$\displaystyle\rho^{\prime}$ $\displaystyle=$ $\displaystyle
F_{3}(r,\rho,h^{\prime},h)\,,$ (4)

where a prime denotes the derivative with respect to $r$. Here, we have used
functions $F_{i}$ to avoid displaying the complex and non-instructive
expressions in the TOV equations. We now try to obtain numerical solutions for
Eq. (4) with specific boundary conditions including regularity at the center
of the star and asymptotic flatness at spatial infinity. Near the center of
the star, by considering power expansions for $h$, $f$, and $\rho$, and
substituting them into Eq. (4), we can express the behavior of the regular
solution as follows,

$h=h_{0}+h_{2}r^{2},\quad f=1+f_{2}r^{2},\quad\rho=\rho_{0}\,,$ (5)

where $(h_{0},h_{2},f_{2},\rho_{0})$ are independent free parameters. For a
given central density $\rho_{0}$, once the appropriate values for
$(h_{0},h_{2},f_{2})$ are chosen, the TOV equations (4) are integrated from
the center of the star outward until reaching the star’s surface with radius
$R$, where the pressure becomes zero, i.e., $p(R)=0$. Then, taking the
obtained values for $h$ and $f$ at the surface of the star as initial
conditions, which are ensured by the continuity condition, we integrate
continuously the vacuum equations outward from the star’s surface to infinity.
The requirement of asymptotic flatness at infinity gives

$\lim_{r\rightarrow\infty}h(r)=\lim_{r\rightarrow\infty}f(r)=1-2M/r\,,$ (6)

where $M$ represents the mass of the star. It should be noted that up to now
we have not specified in our discussion any particular state of matter.
However, given that neutron stars are the most compact observed horizonless
objects without a photon sphere, employing the EOS for neutron stars can
facilitate achieving solutions for objects without an event horizon that can
be more compact than black holes. We utilize a commonly employed moderately
stiff EOS for dense nucleon matter, known as SLy (Skyrme Lyon) [22], which is
derived from a Skyrme-type energy density functional. The analytical
expression for this EOS can be found in Eq. (14) in Ref. [23].

Following the aforementioned procedure, we have obtained solutions with
compactness higher than that of the Schwarzschild solution across a range of
central densities and coupling constants, Here, compactness is defined as
${\cal C}\equiv M/R$, with $M$ and $R$ representing respectively the mass and
radius of an SSS object. For illustrative purposes, we provide a specific
numerical solution for the functions $f,h,\rho$ and $p$ obtained with a
central density of $\rho_{0}=3\times 10^{15}\textup{g/cm}^{3}$ and a coupling
constant of $\lambda=500\lambda_{\star}$, as depicted in Figure 1.

Figure 1: The upper and lower plots illustrate the numerical solutions of the
functions ($f,h$) and ($\rho,p$) in the framework of GR and QTG with a
coupling constant $\lambda=500\lambda_{\star}$, respectively. The central
density is set to $\rho_{0}=3\times 10^{15}\textup{g/cm}^{3}$, and the EOS
used is SLy. The dashed curves represent the Schwarzschild black hole metric
functions $h_{\textup{Sch}}$ and $f_{\textup{Sch}}$, with masses of
$2.05M_{\odot}$ (GR) and $6.84M_{\odot}$ (QTG). The solid vertical lines
represent the radii of the stars, while the dashed vertical lines represent
the event horizon radii of the black holes.

Note that we have adopted the symbol “$\star$” as a subscript to represent the
dimension of various physical quantities, including $M_{\odot}\sim
r_{\star}\sim{p_{\star}}^{-1/2}\sim\rho_{\star}^{-1/2}\sim\lambda_{\star}^{1/2}$,
with the Solar mass $M_{\odot}$ serving as the reference unit. In the specific
example depicted in Figure 1, the mass $M$ and radius $R$ of the star are
$6.84M_{\odot}$ and $11.82r_{\star}$, respectively, and the corresponding
compactness is ${\cal C}=M/R\approx 0.58$, which is larger than the
compactness of the Schwarzschild black hole (${\cal C}=0.5$). Furthermore, in
Figure 2, we show how compactness ${\cal C}$, mass $M$, and radius
$R_{\textup{NS}}$ of neutron stars vary with the overall coupling constant
$\lambda$ in the context of QTG at the specific central density
$\rho_{0}=3\times 10^{15}\textup{g/cm}^{3}$. We also depict the relationship
between the radius $R_{\textup{BH}}=2M$ of Schwarzschild black holes and the
coupling constant $\lambda$ in Figure 2.

Figure 2:  The upper plot shows the relationship between the compactness
${\cal C}$ of the star and the overall coupling constant $\lambda$ of the
quasi-topological combination in QTG. The lower plot shows the
$M-\lambda,R_{\textup{BH}}-\lambda,R_{\textup{NS}}-\lambda$ relations,
respectively. The central density is set to $\rho_{0}=3\times
10^{15}\textup{g/cm}^{3}$, and the EOS used is SLy.

The ${\cal C}-\lambda$ relationship follows a logarithmic law as confirmed
through fitting.

After presenting the specific solution in Figures 1 and the relationships
between physical quantities and the coupling constant in Figure 2, let us now
provide an understanding of how QTG enables a physically-reasonable neutron
star to be smaller than a black hole of the same mass, from both mathematical
and physical perspectives. First, from a mathematical perspective, in the
framework of GR, metric functions $h$ and $f$ typically have $h\neq f$ near
the center, and gradually converge to become equal as the radius $r$ increases
before reaching the surface, smoothly connecting to the Schwarzschild metric
($h_{\textup{Sch}}=f_{\textup{Sch}}=1-2M/r$), and thus ensuring the external
vacuum region is uniquely described by the Schwarzschild solution. With quasi-
topological terms included in QTG, the EOM become high-order derivative
equations. This modification breaks Birkhoff’s theorem and allows the external
vacuum region to be described by diverse SSS asymptotically-flat solutions.
Meanwhile, the Schwarzschild black hole remains a solution within the
framework of QTG, as can be easily verified by substituting the Schwarzschild
solution into vacuum EOM. This means that the modifications caused by quasi-
topological terms do not alter the size of SSS black holes predicted by GR,
significantly simplifying the comparison of sizes between black holes and
objects without an event horizon of the same mass. Therefore, when we obtain
solutions with compactness ${\cal C}>0.5$, we can conclude that stars
described by these solutions are smaller than black holes of the same mass
within the framework of quasi-topological gravity. With the effects of quasi-
topological terms, $h$ and $f$ still gradually converge to become equal, yet
at a slower rate compared to that in GR. When the coupling constant $\lambda$
exceeds a certain threshold, then $h$ and $f$ will remain unequal upon
reaching the star’s surface. This leads to a scenario where the spacetime
outside the star will no longer be described by the Schwarzschild metric.
However, it should be noted that $h$ and $f$ will ultimately converge to
equality before reaching the weak-field region, regardless of how large the
value of the coupling constant $\lambda$ is, and will smoothly connect to
those in the Schwarzschild solution. This can be demonstrated by introducing
linear perturbations to backgrounds with maximally symmetric spaces and
studying linearized gravity. Considering a metric in the form of
$g_{\mu\nu}=\bar{g}_{\mu\nu}+\tilde{g}_{\mu\nu}$, with
$\bar{R}_{\mu\nu\rho\sigma}\propto\bar{g}_{\mu\rho}\bar{g}_{\nu\sigma}-\bar{g}_{\mu\sigma}\bar{g}_{\nu\rho}$,
where ‘bar’ and ‘tilde’ denote quantities associated with the background and
perturbations respectively, it becomes evident that the linearized field
equations in QTG are in complete agreement with those of GR in weak-field
regimes, as if QTG is purely topological [21, 24]. This ensures that the
Schwarzschild metric remains the unique SSS asymptotically-flat solution in
weak-field regimes.

Now, let us turn to the physical perspective. In the framework of GR, the
density $\rho$ and pressure $p$ gradually decrease as the radius $r$
increases, with the pressure reaching zero at the surface of the star. In the
framework of QTG, the density $\rho$ and pressure $p$ still decrease as the
radius increases, but their decrease is slower compared to that in GR. Both
the pressure and density gradients at the same radial position are smaller
compared to those in GR, indicating a repulsive effect of the quasi-
topological terms and resulting in larger mass and radius for the neutron
star. As the coupling constant $\lambda$ increases, this repulsive effect
gradually intensifies, leading to a significant growth in the star’s mass. It
is noteworthy that the star’s radius scales proportionally to the cubic root
of its mass ($R\propto M^{1/3}$). In contrast, the Schwarzschild black hole’s
gravitational radius is linearly related to its mass ($R\propto M$), resulting
in a relatively slower rate of increase in the star’s radius compared to that
of the black hole as mass increases. Consequently, with the increase of the
coupling constant $\lambda$ in QTG, the star’s radius will inevitably become
smaller than that of an equally massive black hole. Furthermore, within the
framework of QTG, the quasi-topological nature of the theory ensures that the
spacetime of such stars, as well as any other less compact SSS objects, is
identical to that of Schwarzschild black holes in the weak-field region,
without any ghosts or modifications. This ensures that QTG can successfully
pass any stringent tests in weak-field regimes, just as GR does.

Based on the preceding discussion, it is evident that, for a neutron star with
a given EOS and central density, the coupling constant $\lambda$ must be
greater than a certain value for the star’s compactness to exceed that of the
Schwarzschild black hole. Next, we will explore how the central density
$\rho_{0}$ influences the compactness of a neutron star with a given EOS. We
will continue to utilize the SLy EOS and compare results in both GR and QTG
with a fixed coupling constant $\lambda=65\lambda_{\star}$. We depict the
relationship between compactness ${\cal C}$ and central density $\rho_{0}$ in
the left panel of Figure 3. Meanwhile the relation between mass $M$ and radius
$R$ of the neutron stars for the respective central densities is depicted in
the right panel of Figure 3,

Figure 3:  The left panel shows the relation between compactness ${\cal C}$
and central density $\rho_{0}$ in both GR and QTG with a fixed coupling
constant $\lambda=65\lambda_{\star}$. The right panel shows the mass-radius
diagram for neutron stars (solid lines) and black holes (dashed line) in GR
and QTG with coupling constants $65\lambda_{\star}$ (SLy), $15\lambda_{\star}$
(DSQS) and $100\lambda_{\star}$ (SQSB56). For the SLy, the points from top to
bottom in the context of QTG correspond to the points from left to right in
GR. This sequence reflects a decreasing central density within the range of
($6\times 10^{15},3.2\times 10^{14}$) $\textup{g/cm}^{3}$, which can be
translated to dimensionless values of ($9.71\times 10^{-3},5.18\times
10^{-4}$) and corresponds to the points in the left panel. For the DSQS, the
central density of the point is $1.67\times 10^{16}\textup{g/cm}^{3}$. For the
SQSB56, the central densities are $9\times 10^{15}\textup{g/cm}^{3}$ (upper)
and $5.4\times 10^{14}\textup{g/cm}^{3}$ (lower). The error bar (black line)
represents a neutron star with mass $1.4M_{\odot}$ and radius
$R=7.45^{+0.61}_{-0.41}r_{\star}$ [26].

from which, one can see that the impact of the quasi-topological terms on the
compactness of neutron stars is relatively minor at lower central densities
(relatively weak gravity regimes) but becomes more pronounced at higher
central densities (relatively strong gravity regimes). Moreover, within the
range of central densities permitted by the EOS, this effect intensifies with
increasing central density. Consequently, with the same modifications from the
quasi-topological terms, neutron stars with relatively high central densities
may exhibit greater compactness than black holes. Meanwhile, neutron stars
with relatively low central densities, or other astrophysical objects with
densities lower than that of neutron stars, may display negligible differences
in terms of compactness, mass, and radius compared to that in GR.

Remarkably, it is worth noting that when the coupling constant takes a
specific value that allows the existence of neutron stars more compact than
black holes within the particular QTG theory, neutron stars consistent with
observational constraints can still be obtained. For instance, with a fixed
coupling parameter of $\lambda=100\lambda_{\star}$ and the EOS of SQSB56 [25],
we find that, for relatively high central density, there exists a neutron star
with a mass of $4.32M_{\odot}$ and a radius of $8.31r_{\star}$ which exceeds
the compactness of a Schwarzschild black hole. Meanwhile, for relatively low
central density, a neutron star with a mass of $1.4M_{\odot}$ and a radius of
$7.53r_{\star}$ that fulfills the stringent constraints derived from multi-
messenger observations [26] is also allowed. We plot both cases in the right
panel of Figure 3.

Our solutions allow for a broad range of values for masses and compactness, as
demonstrated in the Figures 2 and 3. Adopting the EOS of DSQS [25], we obtain
compact objects with masses less than $3M_{\odot}$, exemplified by an object
with a mass of $2.97M_{\odot}$ and a radius of $5.91r_{\star}$ as depicted in
the right panel of Figure 3. It is worth noting that the cubic Ricci quasi-
topological combination (1) used in this study represents the simplest example
of QTG. By considering higher-order or Riemann polynomial corrections [21,
24], it is possible to obtain solutions with both smaller and larger masses
and more extreme compactness than those presented in Figures 2 and 3. Note
that all solutions presented herein have been obtained with positive values
for the coupling constant $\lambda$ to avoid tachyon condensation.

Now we have demonstrated that stars more compact than a black hole, which are
impossible in GR, can exist in QTG. A couple of comments is then in order.
First, it will be interesting to study whether such objects also exist in
other modified gravities. To address this issue, let us note that since the
mass-radius relation for black holes in diverse gravities is approximately
linear, while that for horizonless objects follows a cubic root law, any
theory capable of reducing the strength of gravity in strong-field regimes may
in principle allow for the existence of such objects. However, such theories
must, at the same time, pass tests in weak-field regimes. For QTG, its quasi-
topological nature ensures that its weak-field limit is identical to GR,
guaranteeing that it passes any precise tests in weak-field regimes. In
contrast, other theories might require a screening mechanism for deviations in
weak-field regimes. This can be illustrated by using cosmological gravity
theories [27]. For example, some modified gravity theories deviate from GR at
cosmological scales, allowing them to explain the accelerated cosmic expansion
without introducing dark energy. However, the same deviations may not sustain
solar system tests, and various screening mechanisms such as the Vainshtein
mechanism, chameleon mechanism, symmetron mechanism, etc., have been proposed
to render the modified gravity to pass these tests. Therefore, if there is a
mechanism that maintains consistency with GR in weak-field regimes and
decreases the strength of gravity in the strong-field one in a modified
gravity, physical reasonable objects more compact than black holes might be
allowed.

Second, naturally, one may wonder what the observational consequences of these
objects will be, or whether there will be observational signatures that can
differentiate them from black holes and other horizonless objects with
compactness ${\cal C}<0.5$. In this regard, Cardoso et al. discovered, in
their pioneering studies [9, 10, 11, 12], that horizonless compact objects,
which are less compact than a black hole but possess a clean photon sphere,
exhibit a late-time gravitational-wave ringdown waveform that contains a
sequence of decaying echo pulses, despite their early-time dynamics being
identical to that of a black hole. These pulses serve as a distinct signature
to distinguish these objects from black holes. Furthermore, the time delay
between the echoes is associated with the distance between the object’s
surface and the unstable photon sphere. Since the photon sphere is always
located at $3M$, objects with higher compactness have a greater distance
between their surface and the photon sphere, resulting in a longer time delay.
By precisely modeling the gravitational waveforms in strong gravity processes,
these echo signals can potentially be detected as the sensitivity of current
detectors increases and the next generation of interferometers come into play.
Notably, our horizonless objects, despite having compactness greater than that
of a black hole, also feature an unstable photon sphere at $3M$ and a well-
defined surface. Consequently, they exhibit echo waveforms during late-time
ringdown, characterized by the longer time delay resulting from their greater
compactness. This unique time delay can serve as a distinguishing signature to
set them apart from black holes and other horizonless objects with ${\cal
C}<0.5$. If gravitational-wave echo signals corresponding to horizonless
objects with $C>0.5$ are observed, it would constitute compelling evidence for
new physics beyond GR.

###### Acknowledgements.

We are grateful to Chengjie Fu, Robert B. Mann, Samuele Silveravalle, Hao Wei,
Puxun Wu and Shuang-Qing Wu for useful discussions. S.L. also thanks Lijing
for the warm hospitality during the early stage of this work at PKU-KIAA. S.L.
and H.Y. were supported in part by the National Natural Science Foundation of
China (12105098, 12075084, 11690034, 11947216, 12005059) and Natural Science
Foundation of Hunan Province (2022JJ40264). H.L. was supported in part by the
National Natural Science Foundation of China (11935009, 11875200). Y.G., R.X.
and L.S. were supported in part by the National SKA Program of China
(2020SKA0120300), the National Natural Science Foundation of China (11991053,
11975027), and the Max Planck Partner Group Program funded by the Max Planck
Society.

## References

  * [1] A. Einstein, “The foundation of the general theory of relativity.,” Annalen Phys. 49, no.7, 769-822 (1916).
  * [2] B. P. Abbott et al. [LIGO Scientific and Virgo], “Observation of gravitational waves from a binary black hole merger,” Phys. Rev. Lett. 116, no.6, 061102 (2016) [arXiv:1602.03837 [gr-qc]].
  * [3] B. P. Abbott et al. [LIGO Scientific and Virgo], “GWTC-1: A gravitational-wave transient catalog of compact binary mergers observed by LIGO and Virgo during the first and second observing runs,” Phys. Rev. X 9, no.3, 031040 (2019) [arXiv:1811.12907 [astro-ph.HE]].
  * [4] R. Abbott et al. [LIGO Scientific and Virgo], “GWTC-2: Compact binary coalescences observed by LIGO and Virgo during the first half of the third observing run,” Phys. Rev. X 11, 021053 (2021) [arXiv:2010.14527 [gr-qc]].
  * [5] R. Abbott et al. [LIGO Scientific and VIRGO], “GWTC-2.1: Deep extended catalog of compact binary coalescences observed by LIGO and Virgo during the first half of the third observing run,” [arXiv:2108.01045 [gr-qc]].
  * [6] R. Abbott et al. [LIGO Scientific, VIRGO and KAGRA], “GWTC-3: Compact binary coalescences observed by LIGO and Virgo during the second part of the third observing run,” [arXiv:2111.03606 [gr-qc]].
  * [7] K. Akiyama et al. [Event Horizon Telescope], “First M87 Event Horizon Telescope results. I. The shadow of the supermassive black hole,” Astrophys. J. Lett. 875, L1 (2019) [arXiv:1906.11238 [astro-ph.GA]].
  * [8] K. Akiyama et al. [Event Horizon Telescope], “First Sagittarius A* Event Horizon Telescope results. I. The shadow of the supermassive black hole in the center of the Milky Way,” Astrophys. J. Lett. 930, no.2, L12 (2022)
  * [9] V. Cardoso, E. Franzin and P. Pani, “Is the gravitational-wave ringdown a probe of the event horizon?,” Phys. Rev. Lett. 116, no.17, 171101 (2016) [erratum: Phys. Rev. Lett. 117, no.8, 089902 (2016)] [arXiv:1602.07309 [gr-qc]].
  * [10] V. Cardoso, S. Hopper, C. F. B. Macedo, C. Palenzuela and P. Pani, “Gravitational-wave signatures of exotic compact objects and of quantum corrections at the horizon scale,” Phys. Rev. D 94, no.8, 084031 (2016) [arXiv:1608.08637 [gr-qc]].
  * [11] V. Cardoso and P. Pani, “Tests for the existence of black holes through gravitational wave echoes,” Nature Astron. 1, no.9, 586-591 (2017) [arXiv:1709.01525 [gr-qc]].
  * [12] V. Cardoso and P. Pani, “Testing the nature of dark compact objects: a status report,” Living Rev. Rel. 22, no.1, 4 (2019) [arXiv:1904.05363 [gr-qc]].
  * [13] P. V. P. Cunha and C. A. R. Herdeiro, “Shadows and strong gravitational lensing: a brief review,” Gen. Rel. Grav. 50, no.4, 42 (2018) [arXiv:1801.00860 [gr-qc]].
  * [14] P. V. P. Cunha, C. A. R. Herdeiro and M. J. Rodriguez, “Does the black hole shadow probe the event horizon geometry?,” Phys. Rev. D 97, no.8, 084020 (2018) [arXiv:1802.02675 [gr-qc]].
  * [15] S. D. Mathur, “The information paradox: A pedagogical introduction,” Class. Quant. Grav. 26, 224001 (2009) [arXiv:0909.1038 [hep-th]].
  * [16] R. Penrose, “Gravitational collapse and space-time singularities,” Phys. Rev. Lett. 14, 57-59 (1965).
  * [17] C. M. Will, “The confrontation between general relativity and experiment,” Living Rev. Rel. 17, 4 (2014) [arXiv:1403.7377 [gr-qc]].
  * [18] E. Berti, E. Barausse, V. Cardoso, L. Gualtieri, P. Pani, U. Sperhake, L. C. Stein, N. Wex, K. Yagi and T. Baker, et al. “Testing general relativity with present and future astrophysical observations,” Class. Quant. Grav. 32, 243001 (2015) [arXiv:1501.07274 [gr-qc]].
  * [19] L. Barack, V. Cardoso, S. Nissanke, T. P. Sotiriou, A. Askar, C. Belczynski, G. Bertone, E. Bon, D. Blas and R. Brito, et al. “Black holes, gravitational waves and fundamental physics: a roadmap,” Class. Quant. Grav. 36, no.14, 143001 (2019) [arXiv:1806.05195 [gr-qc]].
  * [20] D. Psaltis, “Probes and tests of strong-field gravity with observations in the electromagnetic spectrum,” Living Rev. Rel. 11, 9 (2008) [arXiv:0806.1531 [astro-ph]].
  * [21] Y. Z. Li, H. S. Liu and H. Lü, “Quasi-topological Ricci polynomial gravities,” JHEP 02, 166 (2018) [arXiv:1708.07198 [hep-th]].
  * [22] F. Douchin and P. Haensel, “A unified equation of state of dense matter and neutron star structure,” Astron. Astrophys. 380, 151 (2001) [arXiv:astro-ph/0111092 [astro-ph]].
  * [23] P. Haensel and A. Y. Potekhin, “Analytical representations of unified equations of state of neutron-star matter,” Astron. Astrophys. 428, 191-197 (2004) [arXiv:astro-ph/0408324 [astro-ph]].
  * [24] R. A. Hennigar, D. Kubizňák and R. B. Mann, “Generalized quasitopological gravity,” Phys. Rev. D 95, no.10, 104042 (2017) [arXiv:1703.01631 [hep-th]].
  * [25] D. Gondek-Rosinska and F. Limousin, “The final phase of inspiral of strange quark star binaries,” [arXiv:0801.4829 [gr-qc]].
  * [26] C. D. Capano, I. Tews, S. M. Brown, B. Margalit, S. De, S. Kumar, D. A. Brown, B. Krishnan and S. Reddy, “Stringent constraints on neutron-star radii from multimessenger observations and nuclear theory,” Nature Astron. 4, no.6, 625-632 (2020) [arXiv:1908.10352 [astro-ph.HE]].
  * [27] K. Koyama, “Cosmological Tests of Modified Gravity,” Rept. Prog. Phys. 79, no.4, 046902 (2016) [arXiv:1504.04623 [astro-ph.CO]].


# BlockLot: Blockchain-based Verifiable Lottery

Yongrae Jo1, and Chanik Park 2  Department of Computer Science and Engineering
Pohang University of Science and Technology
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

We propose BlockLot, a blockchain based verifiable lottery. BlockLot provides
transparent, immutable, fair, and verifiable lottery services enhanced by
recent blockchain technologies such as append-only (replicated) distributed
ledger and smart contract. In addition, BlockLot allows all participants to
perform various verification procedures to ensure that the system is actually
working as expected.

We implement BlockLot services on Hyperledger/Fabric blockchain platform with
web-based user interface.

###### Index Terms:

Blockchain, Lottery, Hyperledger/Fabric

##  I Introduction

Lottery events occur frequently. A lottery event requires trust between
participants and event organizers. In general, people participate in the event
because they believe the probability of winning lottery is fair. But the trust
between the two is sometimes broken. For example, an event organizer can
manipulate the results of the winner.

To solve the problem of fraudulent lottery, computer software can be used to
compute winners without human intervention. Lottery program, which is
determined solely by input and output, is believed to be more reliable, fair,
transparent and verifiable than human-involved one.

The existing lottery systems adopt a centralized approach, in which
calculations are performed on a single computer[1, 2, 15, 14]. However, in
such a case, a person who gains privileged access to the computer (e.g.,
Hacker), on which the lottery program is run, may harm its integrity,
transparency, and fairness. For example, an attacker who gains privileged
access can manipulate the winners after the results are issued. Therefore, the
centralized lottery system is unreliable in nature.

We define three attributes of the unreliable lottery which includes
predictability, modifiability and information hiding. Predictability means
that the random seed used in the lottery can be predicted. Modifiability means
manipulation of event information, lottery result, random seed, lottery
algorithm. Information hiding represents hiding important details of the
lottery (e.g., changing a prize).

Against the attributes of unreliable lottery system, we also defined four
attributes of reliable lottery which includes fairness, transparency,
immutability, and verifiability. Fairness means that the winner should not be
predictable in advance. Transparency indicates that the method of drawing
winners and the results of the draw should be transparent to all participants.
Immutability means that the registered information can not be modified at
one’s disposal and no one can change the result after it is announced.
Verifiability represents that the lottery system should satisfy the
aforementioned attributes and that the results should be verifiable.

We introduce BlockLot, a reliable lottery system based on blockchain. BlockLot
enables reliable lottery in a way of fair, transparent, immutable, and
verifiable. BlockLot uses a Bitcoin blockchain as a random beacon to choose
winners, which it is unpredictable and publicly verifiable[11, 12]. BlockLot
also uses a distributed-replicated ledger based on a blockchain and all
services in BlockLot are supported by smart contracts in distributed manner.
The lottery services provided by BlockLot includes _open_ , _query_ ,
_subscribe_ , _draw_ , _check_ , and _verify_. The event organizer can open a
lottery by providing required information such as the announcement date,
number of winners, and list of prizes. Participants can query the lottery
events registered and subscribe to the event to participate in it. When
conditions for drawing winners are satisfied, an event organizer draw winners
and participants can check and verify it.

The paper is organized as follows. Section II describes design and
implementation of BlockLot. Section IV presents discussion and related work.
Section V includes concluding remarks.

##  II BlockLot

###  II-A Overview

We developed BlockLot on Hyperledger/Fabric, a popular permissioned blockchain
platform for enterprise[3]. Hyperledger/Fabric uses peer nodes to maintain a
distributed ledger and execute chaincodes(a.k.a smart contracts). The Web
server and the Hyperledger/Fabric SDK interact with the user (web client) and
the blockchain network. Upon receiving the user’s transaction, the web server
sends it to the peer node and informs the user of the chaincode execution
result. Also, it fetches the target block number to be used as a random seed
from the Bitcoin random beacon. (Fig. 1)

Figure 1: Overview of BlockLot: Users can use BlockLot services through a web
server. The peer nodes in the back-end blockchain network maintain the lottery
ledger and execute transactions of lottery service. Peer uses Bitcoin block
hash for darwing winners.

BlockLot users can be divided into participants and event organizers. An event
organizer can register the lottery event and draw the winner with
authentication (Fig. 2). Participants can query the event, subscribe to the
lottery, and check and verify the lottery result.

Figure 2: Usecase diagram of BlockLot: Event organizer can open a lottery
event and draw winners. Participant can query events, subscribe to an event,
check and verify the result.

Chaincode is a special program that processes transactions and performs I/O
with blockchain database in Hyperledger/Fabric. The types of transaction of
BlockLot consists of query, open, draw, and subscribe, each of which is
implemented by chaincode. We used GoLang to write chaincode because
Hyperledger/Fabric supports GoLang as a primary language.

###  II-B Generating a randomness

###  II-C Lottery sequence

In this section, we introduce the detailed implementation of the lottery
service of BlockLot (Fig. 3).

####  II-C1 Open

A open transaction registers a lottery in the blockchain. An event organizer
provides event information such as name, announcement date, number of winners,
block offset. The block offset represents the offset from the most recent
block height in the main chain of the Bitcoin blockchain at the time of
registration(namely target block). We used developer API provided by [4].
During the process, a unique ID of the event is generated and inserted into
the key/value store of ledger with related information.

####  II-C2 Query

A query transaction queries all registered events in the blockchain. We use
the range query function GetStateByRange(), which is provided by the chain
code shim interface, to retrieve the strings of (key, value) pairs and display
the concatenated values to the participant.

####  II-C3 Subscribe

To subscribe to an event, a participant chooses an event, enters their
identity(e.g., email address) and gets a authentication token. The token is
used to prove that the participant is actually a winner after the drawing.
Subscribe transaction appends a participant to the list of participants of
given event. The hash value of the registration name concatenated by
authentication token is recorded on the blockchain, ensuring the anonymity of
the participant.

####  II-C4 Draw

A draw transaction can be invoked after the announcement date has arrived and
a target block has been published. The event organizer enters a pre-issued
authentication token, and if the verification is passed, winners will be
selected. Once the winners are selected, all the information about the lottery
event is fixed, so the information obtained by using the hash function is used
as a verifiable key.(Fig. 4). The winner list and the verifiable key are
stored in the blockchain. The script for draw uses the Fisher-Yates random
shuffle algorithm, which uses the block hash of the target block as a random
seed, to output the winner array in a deterministic way.

####  II-C5 Check

After the draw transaction has been completed, the participant can confirm the
winner by entering the name and authentication token. Like the participant
list, the winner list is also expressed in hash values.

####  II-C6 Verify

Participants also need to verify the result to ensure that the lottery is
conducted without compromising the attributes of a reliable lottery. Our
assumptions on an attacker contains a) predicting and manipulating random
seed, b) fabricating event information arbitrarily, c) Manipulating winner
list after draw.(i.e., recording a fraudulent winner list), d) malicious
lottery scripts(e.g., draw winners regardless of random seed).

##### Preventing an attack on random seed and verifying its integrity

We use Bitcoin block hash, which is inherently unpredictable and immutable.
Bitcoin miners spread around the world compete to get reward by solving
random-based hash puzzles, so they generate unpredictable random numbers.
Also, Bitcoin is the largest global blockchain at the moment, which it is
immune to control block hash. Using Bitcoin blockchain as randomness source is
described in [12, 11]. In Bitcoin, block hash is calculated using its
_version, previousHash, merkle_root, timestamp, bits, nonce_ fields of the
block. So we calculate the hash value of the block and compare it with the one
in the lottery in BlockLot.

##### Verifying event information integrity

We also recalculate and compare hash values with information determined at the
time of event registration and draw to verify integrity. Given the participant
list, random seed, and draw script, participant can calculate the winner list
and compare it to the winner list on the blockchain.

##### Verifying response results from multiple peers

A peer that holds a blockchain can be crashed or hacked, so we need to query
all peers in the network, and compare the response results. If more than half
of the peers return the same response results, users can accept it.

##### Verifying statistical fairness

The draw script is verified by z-test to verify that a particular participant
does not win more than a average number of times. Given $n$ times of running
lottery, and when the probability of winning lottery is $p$ (number of winners
/ number of participants), its binomial distribution should follow a normal
distribution if $n$ is large enough. If each participant’s z-score is less
than a pre-defined maximum z-value, the verification is successful, otherwise
it fails.

Figure 3: Lottery sequence: 1) When an event organizer opens a lottery,
participants can query the event, and 2) subscribe to the event. 3) After the
conditions for drawing winners are satisfied, 4) the event organizer draw
winners, and the participants can check and verify the results. Figure 4:
Deriving verifiable random key: The key is derived from concatenating the
result from HAMC(initital random key, random source) and SHA256(Lottery
informatino)

##  III Implementation

###  III-A Chaincode Implementation

Hereby we explain how to handle lottery transactions with chaincode
implementation in Hyperledger/Fabric. Shim package provides simple interface
to access and modify ledger. For example, GetState() is used to access to the
ledger and PutState() is used to modify it. Unless otherwise stated, all of
the functions below are included in the stub interface.

Figure 5: Lottery event representation in chaincode

####  III-A1 Open

Implementing an open transaction is straightforward. It creates a new
lottery_event struct, fills fields in the struct, marshals the struct, and
inserts it into a ledger using PutState() with its identifier as a key. Since
the ledger is maintained per channel in Hyperledger/Fabric, ChannelID should
be specified when the event is created. The ID of open transaction is recorded
as well. GetChannel() and GetTxID() is used to do these. DrawTxID and
SubscribeTxIDs are initialized as UNDEFINED. Status field is initialized as
REGISTERED.

####  III-A2 Query

We used GetStateByRange() to iterate all registered lottery events in the
ledger. By iterating all keys in the ledger, we can combine those key/value
pairs into one byte array. The byte array is returned to client.

####  III-A3 Subscribe

A subscribe transaction contains identifiers of event and member as function
arguments. It retrieves an event using GetState(), checks if the member is
already registered, compares timeliness(i.e., due date). If successful, it
adds the member to MemberList in lottery event. After that, it also records
subscribeTxID returned from GetTxID().

####  III-A4 Draw

It fetches target block from [4], extracts hash value from it, and uses it as
a random seed for the Fisher-Yates random shuffle algorithm[13]. We implement
the algorithm in GoLang. The algorithm receives the number of
participants$\mathsf{P}$, number of winners($\mathsf{W}$), and random source
as arguments. Since the algorithm requires a fresh random seed in each
iteration step to swap two elements in an array with size of $\mathsf{P}$, we
newly design randomOracle() to generate new random values from the random
source(i.e., block hash). randomOracle() simply hashes the random source and
extracts the first four 32-bit integers from the hashed value, and adds them
together to get a single value. Then, the result value is used to choose two
elements randomly in the array. After iterating the array by $\mathsf{P}$
times, the algorithm is terminated. As a result, first $\mathsf{W}$ elements
from the array are selected as winners.

Draw also uses two cryptographic hash functions to create VerifiableRandomKey.
The key is derived from HMAC with random source, initial random key, and
SHA256 with lottery information. (Fig. 4). The initial random key is issued
when the event organizer register a lottery event. Lottery information
includes all fields in lottery event struct (Fig. 5). Since the winner list is
determined, no information in the lottery event is changed. Therefore, the key
for verifying integrity can be derived from these values.

###  III-B User Interface

Main user interface for BlockLot is implemented in web browser (Fig. 6).
Lotteries are displayed as tables, which contain event name, announcement
date, number of participants, number of winners, buttons for subscribe, draw,
check, verify, and info.

An event organizer can register the lottery event by clicking the big button
with a plus sign at the bottom of the table. A user can participate in a
lottery event by clicking the subs. button and providing his/her name or email
address (Fig. 7). After that, an authentication token will be issued to prove
that the user is a winner when it actually is.

We implemented full-fledged BlockLot and it is publicly available on [10].
More implementation details as well as user interfaces that is not covered
here can be found on the github repository.

Figure 6: Main user interface: BlockLot services are available via web browser
based interface. Lotteries are displayd as tables. Users can use each services
by clicking the corresponding buttons. For example, a user can open a lottery
by clicking the big button with a plus sign at the bottom of the table. Figure
7: Open & Subscribe Interface: An event organizer can open a lottery event by
providing its name, number of winners, announcement date, block offset, and
note. Participants participate in a lottery event by providing his/her name or
e-mail address.

##  IV Discussion and Related work

###  IV-A Security

##### Anonymity

BlockLot assumes that the participants identities are known offline in prior,
meaning that their privacy may not be strictly guaranteed. For example,
participation information to an event or the final results can be publicly
available. We believe that this assumption is not a big problem because many
lottery events provided by an enterprise in real-world often require
participant’s identity. (e.g., customer’s name).

##### Duplicate participation

Current implementation does not strictly prevent one user from participating
in the same lottery multiple times. Because the user authentication is done
only by name or e-mail, a malicious user can easily generate multiple
identities to increase the chances of winning lottery. However, this concern
will not be a problem if participant’s identities are known offline to the
event organizer.

##### Lagged blockchain view

We used developer APIs from [4], a freely available blockchain service
provider to read the latest Bitcoin block. One significant problem here is
that the main server which receives blocks on that site sometimes see a lagged
blockchain view (e.g., temporary network delay). In that case, an attacker can
predict the random seed, and exploit this information to be winner. But this
limitation can be relaxed by choosing a target block which is much higher than
currently perceived latest block.

###  IV-B Deployment environment

BlockLot focuses on lottery in situations where participants’ identities are
known in advance. For example, when a company performs a lottery for
promotional purposes, the customer provides his/her identity to participate in
the lottery.

###  IV-C Related work

In traditional centralized lottery system, a lottery is executed in a single
computer. But this approach is vulnerable to a single point of failure as well
as to the existence of a hacker who can gain privileged access to the system.
Our work is basically a decentralized system, which means that a single
participant cannot manipulate any data on the blockchain.

Also, blockchain-based lottery systems such as Kibo[5], FireLotto[6],
Quanta[7], and SmartBilions[8] are based on Ethereum blockchain[9]. Ethereum
provides a turing-complete smart contract platform which is suitable for
executing lottery services. In contrast, our system is based on a permission-
based blockchain, mainly for enterprise, which is orthogonal to Ethereum-based
methods because our system requires identity to be known to participate in a
lottery.

##  V Conclusion

In this paper, we present BlockLot, a Blockchain based verifiable lottery that
records all information about the lottery event in a distributed ledger, and
conducts the event in a transparent and immutable manner. BlockLot uses
Bitcoin block hash as a unpredictable(i.e., fairness) random seed to draw
winners. BlockLot also offers a variety of verification methods to ensure that
the lottery is actually reliable. In conclusion, BlockLot is expected to
significantly reduce social costs due to fraudulent lotteries.

## Acknowledgment

This work was partially supported by The Institute for Information &
Communications Technology Promotion (IITP) in South Korea, 2018, funded by the
Korea government (Ministry of Science and ICT) (No.2017-0-00652, Development
of High-performance and High-reliable Blockchain for Distributed Autonomous
IoT Platform). We also would like to thank to undergraduate students in
POSTECH. Hyunseung Lee gave supplementary comments on this original document,
Byoungyun Park helped with testing BlockLot, and Minsoo Koo designed BlockLot
logo.

## References

  * [1] http://www.unipicker.com/unipicker/main.do.
  * [2] https://cse.snu.ac.kr/node/18358.
  * [3] https://www.hyperledger.org/projects/fabric.
  * [4] https://www.blockchain.com/ko/api.
  * [5] https://kiboplatform.net/_alldata/files/kibowhitepaper.pdf.
  * [6] https://firelotto.io/whitepaper_en.pdf.
  * [7] https://www.quantaplc.im/Quanta.pdf.
  * [8] https://smartbillions.com/docs/SmartBillions_Smartpaper.pdf.
  * [9] https://ethereum.github.io/yellowpaper/paper.pdf.
  * [10] Blocklot.  https://github.com/rupc/BlockLot, 2018.
  * [11] Iddo Bentov, Ariel Gabizon, and David Zuckerman.  Bitcoin beacon.  arXiv preprint arXiv:1605.04559, 2016.
  * [12] Joseph Bonneau, Jeremy Clark, and Steven Goldfeder.  On bitcoin as a public randomness source.  IACR Cryptology ePrint Archive, 2015:1015, 2015.
  * [13] Ronald A Fisher and Frank Yates.  Statistical tables for biological, agricultural and medical research.  Oliver and Boyd Ltd, London, 1943.
  * [14] David M Goldschlag and Stuart G Stubblebine.  Publicly verifiable lotteries: Applications of delaying functions.  In International Conference on Financial Cryptography, pages 214–226. Springer, 1998.
  * [15] Jianying Zhou and Chunfu Tan.  Playing lottery on the internet.  In International Conference on Information and Communications Security, pages 189–201. Springer, 2001.


Copyright 2022 IEEE. Published in ICASSP 2022 - 2022 IEEE International
Conference on Acoustics, Speech and Signal Processing (ICASSP), scheduled for
7-13 May 2022 Virtual; 22-27 May 2022 In-Person in Singapore. Personal use of
this material is permitted. However, permission to reprint/republish this
material for advertising or promotional purposes or for creating new
collective works for resale or redistribution to servers or lists, or to reuse
any copyrighted component of this work in other works, must be obtained from
the IEEE. Contact: Manager, Copyrights and Permissions / IEEE Service Center /
445 Hoes Lane / P.O. Box 1331 / Piscataway, NJ 08855-1331, USA. Telephone: +
Intl. 908-562-3966.

# Container Localisation and Mass Estimation with an RGB-D Camera

###### Abstract

In the research area of human-robot interactions, the automatic estimation of
the mass of a container manipulated by a person leveraging only visual
information is a challenging task. The main challenges consist of occlusions,
different filling materials and lighting conditions. The mass of an object
constitutes key information for the robot to correctly regulate the force
required to grasp the container. We propose a single RGB-D camera-based method
to locate a manipulated container and estimate its empty mass i.e.,
independently of the presence of the content. The method first automatically
selects a number of candidate containers based on the distance with the fixed
frontal view, then averages the mass predictions of a lightweight model to
provide the final estimation. Results on the CORSMAL Containers Manipulation
dataset show that the proposed method estimates empty container mass obtaining
a score of 71.08% under different lighting or filling conditions.

Index Terms—  Convolutional Neural Networks, Object detection, Mass estimation

##  1 Introduction

Estimating the physical properties of objects is fundamental for the success
and safety of human-robot collaboration [1]. A robot is a completely
autonomous system and is supposed to extract information of the target object
in order to assist the human correctly. Robots could be employed, for example,
to help people perform housework, lift heavy loads or even bring medicines to
the elderly. An incorrect prediction of the object properties could harm the
human, e.g., dropping the object or spilling dangerous substances [2].

In a typical human-to-robot handover scenario, the person exchanges a
container with a robot, which takes it from human hand(s) [1, 3]. The
estimation of container properties such as its width, height and mass
represents a crucial stage, since the robot regulates the force to hold the
object during the handover and the maneuvering [2]. Moreover, it is not a
trivial task since the object could be unknown [4, 3] or the physical
properties of the container could change based on the interaction, e.g.,
deformation due to the grasp, or different stiffness and filling amounts [5].

The container mass can be indirectly retrieved by combining two properties:
filling mass and empty container mass. The filling mass can be seen as the
result of three contributions: filling type classification, filling level and
container capacity estimation [6]. Recent exisiting solutions use the CORSMAL
Containers Manipulation (CCM) dataset [7], which includes annotated audio-
visual recordings of people interacting with containers [8]. To perform
filling type classification, audio is one of the most used modality, either
processing spectrograms [9] or classical audio features [10]. Solutions for
filling level estimation can exploit only audio modality [11] or the
combination with visual data [12, 13]. Visual clues represent the main
modality used to estimate the container capacity. Approaches can consider the
task as a regression problem, employing either simple CNNs on single fixed
frontal view depth data [9] or distribution fitting via Gaussian processes
using object category as a prior across multiple views [12]. Otherwise, the
segmented container can be approximated to a primitive shape in 3D, computing
capacity as a by-product [11], or using volume formulas [13].

Unlike previous works that focused on estimating the filling mass, we propose
an approach to localise the container manipulated by a person and then
estimate its empty mass (regardless of the content) from an RGB-D camera with
fixed frontal view. Adopting human-to-robot handover as a use case, the
strategy consists in automatically selecting a number of containers candidates
based on the average distance with respect to the frontal view and then
estimating the empty container mass using a lightweight CNN111Code available
at: https://github.com/CORSMAL/Visual. A similar CNN was previously devised to
regress the container capacity using only depth image crops as input [9].
However, depth images can be noisy, and can contain missing values even after
applying basic morphological operations (e.g., closing). Using both
geometrical information (distance from depth map and aspect ratio with respect
to the original resolution) and the colour information from the RGB frame
increases the probability of localising the object and estimating the empty
container mass.

Fig. 1: Block diagram of the proposed approach. For each frame, containers are
detected, then $K$ nearest patches are selected leveraging the raw depth maps
considered in the segmentation mask coordinates. The empty mass of each patch
($m$) is predicted by the model which takes as input the RGB patch and
triplets of values: aspect ratio width ($a$) and height ($b$), and average
distance ($d$). The final empty mass estimation ($\hat{m}$) is the average of
$K$ mass predictions.

##  2 Localisation and mass estimation

The proposed method uses RGB-D data and is divided into three steps (see Fig.
1). The first step localises the container every $n$ frames using Mask R-CNN
[14] pre-trained on COCO [15], selecting only the classes cup, book, wine
glass and bottle [13]. Assuming that the manipulated container is the nearest
with respect to a fixed frontal view, the method automatically selects the
nearest detected containers based on the average distance of the depth values
belonging to the container mask (K-nearest patches selection). The last step
is the final prediction in which the model performs one prediction per each
patch and then empty container mass ($\hat{m}$) is computed as the average of
the predictions.

###  2.1 Mass estimation model

The model, designed to be lightweight, takes as input RGB patches of the
container, the width and height aspect ratios with respect to the original
image resolution ($a$ and $b$) and the average distance of the object from the
camera ($d$). The rationale behind this last features ($f$) is to preserve
some geometrical information of the distance between the container and camera,
along with the aspect ratios. Similarly to [9], the model has four
convolutional layers, two Fully Connected (FC) layers, concatenation of the
output of the second FC layer with $f$, and one FC layer after the
concatenation. Batch normalization and ReLU activations are used after each
layer. Max Pooling is used after each convolutional layer. The size of
convolutional kernels is $(3,3)$, paddings and strides are $(1,1)$, and
channel dimensions are $(32,64,64,128)$. The size of Max pooling kernel is
$(2,2)$. The output of the first two FC layers has dimension $64$ and $6$,
respectively. Unlike [9] that uses depth patches and their aspect ratios
$[a,b]$, our model takes as input RGB patches and the features $f=[a,b,d]$.
The number of model parameters is 533,000.

###  2.2 Model training

The model is trained using the containers patches extracted using the first
two steps of the described procedure on the CCM dataset. Mask R-CNN is applied
to the entire dataset, similarly to Crop-CCM [16]. The main differences are
that we use a single view of the scene, we do not restrict the model solely to
cup and glass cases, and we do not perform a manual check of the results. The
selection of the containers patches is indeed performed automatically during
$K$-nearest patches selection phase. Images resolution is $1280\times 720$,
the detection threshold is set to 0.4 and every frame of the video is analysed
($n=1$). In each frame, Mask R-CNN could find in general more than one object
e.g. the pitcher used to fill the cup and the cup itself. The maximum number
of considered containers candidates ($K$) is set to 5 (note: the model could
also detect less patches in a recording). The rationale behind the parameter
value choice is to have a trade-off to obtain enough patches to train the
model and provide a more robust mass prediction, by averaging a number of
candidates, as well as minimizing computational overhead. Our extracted
dataset consists of 3,408 patches, some of them are shown in Fig. 2. The
proposed automatic retrieval leads to almost 8% of the patches containing the
pitcher used to fill containers, which are not annotated in CCM dataset. The
empty mass annotation of each video is applied to each extracted patch.

The model is trained on the regression task. The patches are resized to
$112\times 112$ resolution, using zero padding on the shorter dimension in
order to maintain the proportions, and are normalized to [0, 1] range. The
following transformations are employed to augment the patches dataset:
horizontal flip with probability 50%, vertical flip with probability 50%,
random rotation between 0 and 180 degrees without cropping the patch, color
jitter which consists in randomly changing the brightness into $[0.8,1.2]$
range, contrast into $[0.8,1.2]$ range, saturation into $[0.8,1.2]$ range and
hue into $[-0.2,0.2]$ range. The aspect ratios, average distance and empty
mass labels are normalized using the minimum and maximum values retrieved from
the training set. The following setup is common to the experiments: mean
square error loss, batch size 32, learning rate 0.0015 with an exponential
decay rate of 0.9985 and with decay steps equal to 20; weight decay is set to
0.001.

Fig. 2: Sample patches of the extracted dataset. Black padding is applied
before resizing to keep the same aspect ratio.

##  3 Experimental setup

We evaluate our proposed model on the CCM dataset [7] which consists of 1140
audio-visual recordings. During each recording, a person interacts with a
container (e.g., filling a cup with rice contained in a pitcher) and then
prepares for the handover. Videos differ in conditions such as lighting,
person’s clothing, hand occlusions. The total number of containers in the
dataset is 15: 9 (684 recordings) constitute the training set, the other 6 are
evenly split into a public test set (228 recordings) and private test set (228
recordings).

To assess the generalization performances of the method, we leave one instance
per category (box, glass, cup) out, creating three folds [13], see Fig. 3. For
each testing fold, the training set includes containers belonging to the other
two folds, which are split into training and validation sets using 80% and 20%
as respective percentages of data. The training set is augmented of 3 times
using the described transformations. The model is trained using 100 epochs.
The validation set is used for model selection and the best model, the one
with the lowest mean loss, is kept for testing.

Fig. 3: 3-fold cross-validation setups ($F1$, $F2$, $F3$) of the CCM training
set. Each fold selects videos from one instance of each container type as test
set ( ), while videos belonging to the other instances are used as training
set ( ).

In addition to this 3-fold cross-validation, we also randomly split the whole
training set in training and validation with the 80:20 ratio as before to
include all available containers in the training phase. In this case, the
method is validated through the CORSMAL Challenge which provides the results
for the public test set and privates test set, since the annotations are
private. Every patch in the training generates other 4 images using the
described transformations. The model is trained for 300 epochs, as the number
of training images increases with respect to 3-fold cross-validation. The same
rationale of model selection is applied for this experiment.

As performance measures, we compute for each recording $j$ the relative
absolute error between the estimated measure $\hat{m}^{j}$, and the true
measure $m^{j}$, as:

$\epsilon(\hat{m}^{j},m^{j})=\frac{|\hat{m}^{j}-m^{j}|}{m^{j}}$ (1)

The score $s\in[0,1]$ (where 1 is best), across all $J$ recordings for each
measure is222https://corsmal.eecs.qmul.ac.uk/challenge.html:

$s=\frac{1}{J}\sum_{j=1}^{J}\mathbbm{1}_{j}\>e^{-\epsilon(\hat{m}^{j},m^{j})}$
(2)

The value of the indicator function $\mathbbm{1}_{j}\in\\{0,1\\}$ is 0 only
when $\hat{m}$ in recording $j$ is not estimated.

##  4 Results

Fig. 4 (top) analyses the per-class scores (colored based on cup, glass and
box) and the total score for the three splits (F1, F2, F3) and the validation
set (VAL). The whole score can be obtained using Eq. 2 or by multiplying per-
class score by the number of instances, summing and then dividing by the total
number of instances. Fig. 4 (bottom) provides a complementary analysis through
the per-class mean of relative absolute error $\epsilon$. Overall, Fig. 4
shows that the model does not generalize to testing containers significantly
different from the training ones. The model achieves the highest score and the
lowest mean error for the class box, probably due to the fact that they
feature different colors and have a larger shape with respect to other
classes. The mean relative absolute error in the cup cases suggests that the
empty mass for this class is not properly learned. A possible explanation for
the low performance on the first test fold is that the training images contain
colored and opaque cups, whereas in the test set cups are transparent, and the
only transparent containers in the used training folds are glasses. Other
folds statistics point out that the presence of similar containers between
training and test sets helps reducing the error and improving the score. The
values of mean relative absolute error for glass and box classes fall in the
$[0,1]$ range, while the cup class drops with respect to fold F1, yet it
remains higher than 1. The last group of bars in Fig. 4 shows the results for
the validation set predictions. The score value underlines that the chosen
model is able to learn from recordings having the same containers, yet in
different configurations, e.g., lighting conditions or filling. Also in this
case, the class that impacts the most on the score is cup.

F1F2F3VAL$0$$20$$40$$60$$80$Score $s$ (%)

F1F2F3VAL$0$$2$$4$$6$$8$$10$Mean $\epsilon$

Fig. 4: Analysis per container type of 3-fold cross validation and random
cross-validation of our proposed model for container empty mass estimation.
Top: testing score $s$ in percentage. Bottom: mean of relative absolute error
$\epsilon$. The maximum y-axis value is set to 10 for visualization purpose,
the actual value for cup in F1 is 22.952. Legend:  cup,  glass,  box total

The performance score of CORSMAL challenge methods evaluated on private and
public test set are shown in Table 1. Method 1 (M1) exploits RGB-D data from
the fixed frontal view, extracts object patches using an object detector
(YoloV5333https://github.com/ultralytics/yolov5), then predicts the empty
container mass using an efficient model (MobileNetV2 [17]) enhanced with
attention mechanism and pre-trained on container dimension estimation [18].
The eventual mass prediction is obtained by averaging the mass predictions.
Method 2 (M2) regresses the empty container mass using a custom CNN which
combines: the patch of the container extracted using a formula to select the
most visible view (across fontal, left and right views of the scene), the
symmetrically restored object mask from left side fixed view, and information
about container dimensions (height, width at the bottom, width at the top)
[19]. The container detection is performed by Localisation and object
Dimensions Estimator (LoDE) [6]. Our model ($M3$) is the one providing VAL
results of Fig. 4. As baselines for comparison, we consider also a pseudo-
random generator (M4) that draws the predictions from a uniform distribution
in the interval $[1,351]$ based on the Mersenne Twister algorithm [20], and
average (M5) computed on mass labels. In general, some similar features across
the methods can be highlighted e.g. the employment of two-stage approach
(first detection then mass estimation) to tackle the problem and the use of
lightweight models to perform the mass prediction. Our method achieves a
higher score with respect to M2, M4 and M5. Contrary to M2, our method does
not use LoDE during the container detection phase and exploits only the fixed
frontal view. The generalization properties on private test set show that M1
performs better than other models, probably due to the employment attention
mechanisms. Compared to this solution, our model is not pre-trained on other
container properties estimation and features less parameters than MobileNetV2
model.

Table 1: Public and private test scores of container mass estimation solutions in percentage. Set | M1 | M2 | M3 (Ours) | M4 | M5
---|---|---|---|---|---
Public test | 55.25 | 43.61 | 53.14 | 30.59 | 21.88
Private test | 62.32 | 36.77 | 46.14 | 28.25 | 22.24
Combination | 58.78 | 40.19 | 49.64 | 29.42 | 22.06

##  5 Conclusion

This paper provides a method to analyze a video, extract the container subject
to manipulation based on its distance with respect to the fixed frontal view,
and estimate its mass regardless of the content using a lightweight model. The
low percentage of pitcher patches suggests that the proposed method is able to
locate the manipulated container in training recordings. In the experiments,
the model learns from similar containers, but poorly generalizes to unseen
containers, especially cups. As future work, we will investigate multi-task
learning to also classify the container type and improve generalization, as
well as audio-visual perception to exploit the complementarity of the audio
modality. Moreover, the case of camera mounted on the robot will be considered
to analyse how partial views of the container due to movement affect model
predictions.

## References

  * [1] V. Ortenzi, A. Cosgun, T. Pardi, W. P. Chan, E. Croft, and D. Kulić,  “Object Handovers: a Review for Robotics,”  IEEE Trans. Robotics, vol. 37, no. 6, pp. 1855–1873, 2021.
  * [2] Y. L. Pang, A. Xompero, C. Oh, and A. Cavallaro,  “Towards safe human-to-robot handovers of unknown containers,”  in IEEE Int. Conf. Robot and Human Interactive Comm., 2021.
  * [3] P. Rosenberger, A. Cosgun, R. Newbury, J. Kwan, V. Ortenzi, P. Corke, and M. Grafinger,  “Object-Independent Human-to-Robot Handovers using Real Time Robotic Vision,”  IEEE Robotics Autom. Lett., vol. 6, no. 1, pp. 17–23, 2021.
  * [4] W. Yang, C. Paxton, A. Mousavian, Y. Chao, M. Cakmak, and D. Fox,  “Reactive Human-to-Robot Handovers of Arbitrary Objects,”  in IEEE Int. Conf. Robotics Autom., 2021.
  * [5] R. Sanchez-Matilla, K. Chatzilygeroudis, A. Modas, N. Ferreira Duarte, A. Xompero, P. Frossard, A. Billard, and A. Cavallaro,  “Benchmark for Human-to-Robot Handovers of Unseen Containers with Unknown Filling,”  IEEE Robotics Autom. Lett., vol. 5, no. 2, 2020.
  * [6] A. Xompero, R. Sanchez-Matilla, A. Modas, P. Frossard, and A. Cavallaro,  “Multi-View Shape Estimation of Transparent Containers,”  in IEEE Int. Conf. Acoustics, Speech and Signal Process., 2020.
  * [7] A. Xompero, R. Sanchez-Matilla, R. Mazzon, and A. Cavallaro,  “CORSMAL Containers Manipulation,” 2020,  (1.0) [Data set]. Queen Mary University of London. https://doi.org/10.17636/101CORSMAL1.
  * [8] A. Xompero, S. Donaher, V. Iashin, F. Palermo, G. Solak, C. Coppola, R. Ishikawa, Y. Nagao, R. Hachiuma, Q. Liu, F. Feng, C. Lan, R. H. M. Chan, G. Christmann, J.-T. Song, G. Neeharika, C. K. T. Reddy, D. Jain, B. U. Rehman, and A. Cavallaro,  “The CORSMAL benchmark for the prediction of the properties of containers,” arXiv:2107.12719v2, 2021.
  * [9] G. Christmann and J. Song,  “2020 CORSMAL Challenge - Team NTNU-ERCReport,” 2020,  https://corsmal.eecs.qmul.ac.uk/resources/challenge/2020.11.30_CORSMAL_NTNU-ERC_Report.pdf.
  * [10] K. Cho, B. Van Merriënboer, D. Bahdanau, and Y. Bengio,  “On the Properties of Neural Machine Translation: Encoder-Decoder Approaches,” arXiv:1409.1259v2, 2014.
  * [11] R. Ishikawa, Y. Nagao, R. Hachiuma, and H. Saito,  “Audio-Visual Hybrid Approach for Filling Mass Estimation,”  in IEEE Conf. Pattern Recognit. Workshops and Challenges, 2021.
  * [12] Q. Liu, F. Feng, C. Lan, and R. H. M. Chan,  “VA2Mass: Towards the Fluid Filling Mass Estimation via Integration of Vision and Audio Learning,”  in IEEE Conf. Pattern Recognit. Workshops and Challenges, 2021.
  * [13] V. Iashin, F. Palermo, G. Solak, and C. Coppola,  “Top-1 CORSMAL Challenge 2020 Submission: Filling Mass Estimation Using Multi-Modal Observations of Human-Robot Handovers,”  in IEEE Conf. Pattern Recognit. Workshops and Challenges, 2021.
  * [14] K. He, G. Gkioxari, P. Doll’ar, and R. B. Girshick,  “Mask R-CNN,”  in IEEE Int. Conf. Comput. Vis., 2017.
  * [15] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick,  “Microsoft COCO: Common Objects in Context,”  in Eur. Conf. Comput. Vis., 2014.
  * [16] A. Modas, A. Xompero, R. Sanchez-Matilla, P. Frossard, and A. Cavallaro,  “Improving Filling Level Classification with Adversarial training,”  in IEEE Int. Conf. Image Process., 2021.
  * [17] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen,  “MobileNetV2: Inverted Residuals and Linear Bottlenecks,”  in IEEE Conf. Comput. Vis. Pattern Recognit., 2018.
  * [18] H. Wang, C. Zhu, Z. Ma, and C. Oh,  “Improving Generalization of Deep Networks for Estimating Physical Properties of Containers and Fillings,”  in IEEE Int. Conf. Acoustics, Speech and Signal Process., Grand Challenges: Audio-Visual Object Classification for Human-Robot Collaboration, 2022.
  * [19] T. Matsubara, S. Otsuki, Y. Wada, H. Matsuo, T. Komatsu, Y. Iioka, K. Sugiura, and H. Saito,  “Shared Transformer Encoder with Mask-Based 3D Model Estimation for Container Mass Estimation,”  in IEEE Int. Conf. Acoustics, Speech and Signal Process., Grand Challenges: Audio-Visual Object Classification for Human-Robot Collaboration, 2022.
  * [20] M. Matsumoto and T. Nishimura,  “Mersenne Twister: a 623-Dimensionally Equidistributed Uniform Pseudo-Random Number Generator,”  ACM Trans. Modeling Comput. and Simulation, vol. 8, no. 1, pp. 3–30, 1998.