[a]Antonio Smecca

# Inclusive semileptonic $B$-decays from lattice QCD

Paolo Gambino  Shoji Hashimoto  Sandro Mächler  Marco Panero  Francesco
Sanfilippo  Silvano Simula  Nazario Tantalo

###### Abstract

We present the lattice QCD calculation of inclusive semileptonic $B_{s}$-meson
decays. We follow a recently proposed method, which is based on the extraction
of smeared spectral densities from Euclidean correlation functions and on the
numerical reconstruction of the integration kernel relevant for the inclusive
decay rate calculation. We compute four-point Euclidean correlation functions
using JLQCD and ETM gauge ensembles with unphysically light $b$-quark masses,
and apply two different methods for the integration kernel reconstruction.
Finally, we show that the lattice results obtained in this work are in good
agreement with the analytic predictions of the operator-product-expansion.
This opens the path for a future full lattice QCD calculation to be used as
theoretical input for the determination of the magnitude of the CKM element
$V_{cb}$.

##  1 Introduction

The study of the semileptonic decays of heavy mesons is an important and
active area of research since their decay rates are proportional to the
modulus squared of the elements of the Cabibbo-Kobayashi-Maskawa (CKM) mixing
matrix [1, 2]. The CKM matrix is a complex unitary matrix in the Standard
Model (SM) and the precise determination of its elements is very important in
order to validate the SM, as any deviation from unitarity would suggest the
effect of new physics. For this reason, one of the main goals of flavour
physics is to overconstrain the CKM elements in order to check for any such
deviations. At the moment, one of the most persistent tensions is the $\simeq
3\sigma$ discrepancy between exclusive and inclusive determinations of the
magnitude of $V_{cb}$ [3, 4]. The determination of CKM elements is not easy as
they cannot be simply calculated from theory, instead, one has to combine
experimental measurements with theoretical non-perturbative calculations.

Lattice QCD has been extremely successful in providing precise and
systematically improvable calculations for the form factors of exclusive
semileptonic decays [7], while computations of inclusive quantities relied on
the analytic method known as the Operator Product Expansion (OPE) [5, 6]. In
this contribution we present our latest work [8], regarding one of the first
studies of inclusive semileptonic decays of heavy mesons using lattice QCD.
This work is based on recently proposed methods [9, 10] which allow for a
complete first-principle computation of inclusive semileptonic decay rates
starting from Euclidean correlation functions obtained from lattice
simulations. Although it is still a preliminary exploratory study, our results
obtained from the lattice show a good agreement with the analytic predictions
of the OPE, making us optimistic for a future full lattice study. A complete
lattice computation of inclusive decays would be an important step towards a
better understanding of the $V_{cb}$ problem.

##  2 Theoretical framework

We follow the formalism outlined in ref. [10], in which we consider the
semileptonic decay of a $B_{s}$ meson into a pair of massless leptons
$l\bar{\nu}$ and to charmed final states $X_{c}$. Working in the $B_{s}$ meson
rest frame, we define the differential decay rate as

$\displaystyle\frac{d\Gamma}{dq^{2}dq^{0}dE_{\ell}}=\frac{G_{F}^{2}|V_{cb}|^{2}}{8\pi^{3}}L_{\mu\nu}W^{\mu\nu},$
(1)

where $L_{\mu\nu}$ is the leptonic tensor and $W_{\mu\nu}$ is the hadronic
tensor. The latter can be written in its spectral representation

$\displaystyle
W_{\mu\nu}(\omega,\boldsymbol{q})=\frac{(2\pi)^{3}}{2m_{B}}\langle\bar{B}(\boldsymbol{0})|J_{\mu}^{\dagger}(0)\delta(\hat{H}-\omega)\delta^{3}(\hat{\boldsymbol{P}}+\boldsymbol{q})J_{\nu}(0)|\bar{B}(\boldsymbol{0})\rangle,$
(2)

where $\hat{H}$ is the QCD Hamiltonian and $\hat{\boldsymbol{P}}$ is the
momentum operator, while $J_{\mu}$ is the flavour changing current.

The hadronic tensor can be decomposed into Lorentz-invariant structure
functions $Z^{(l)}(\omega,\boldsymbol{q}^{2})$, such that after integrating
over the lepton energy $E_{l}$ the differential decay rate can be rewritten as

$\displaystyle\frac{d\Gamma}{dq^{2}}=\frac{G_{F}^{2}|V_{cb}|^{2}}{24\pi^{3}|\boldsymbol{q}|}\sum_{l=0}^{2}\Big{(}\sqrt{q^{2}}\Big{)}^{2-l}Z^{(l)}(\boldsymbol{q}^{2}),$
(3)

where $\omega$ is the energy of the final meson state in the rest frame of the
$B_{s}$ meson and $Z^{(l)}(\boldsymbol{q}^{2})$ is the integral over phase
space of the structure functions $Z^{(l)}(\omega,\boldsymbol{q}^{2})$
regulated by the integration kernel $\Theta^{l}(\omega_{max}-\omega)$:

$\displaystyle
Z^{(l)}(\boldsymbol{q}^{2})=\int_{0}^{\infty}d\omega\Theta^{l}(\omega_{\max}-\omega)Z^{(l)}(\omega,\boldsymbol{q}^{2}).$
(4)

Equation (4) plays a central role in our effort to calculate the differential
decay rate showed in eq. (3). For this reason, this work focuses on
calculating this object using Euclidean correlators computed in lattice QCD
simulations.

##  3 Lattice computation

The first step towards a lattice calculation of the inclusive semileptonic
decay rate is to make a connection between the hadronic tensor in eq. (2) and
an appropriate Euclidean correlation function. Indeed, while two- and three-
point correlation functions are sufficient for accessing the ground state of a
particle, required for exclusive calculations, they cannot access the full
spectrum of charmed final states, which can instead be achieved by four-point
correlation functions as showed in ref. [10]. The four-point correlator can be
written explicitly as

$\displaystyle
C_{\mu\nu}(t_{\mathrm{snk}},t_{2},t_{1},t_{\mathrm{src}};\boldsymbol{q})=\int
d^{3}x\,e^{i\boldsymbol{q}\cdot\boldsymbol{x}}\,T\langle
0|\,\tilde{\phi}_{B_{s}}(\boldsymbol{0};t_{\mathrm{snk}})J_{\mu}^{\dagger}(\boldsymbol{x};t_{2})J_{\nu}(\boldsymbol{0};t_{1})\tilde{\phi}_{B_{s}}^{\dagger}(\boldsymbol{0};t_{\mathrm{src}})\,|0\rangle,$
(5)

where the two currents are sandwiched between the $B_{s}$ meson states.

$B_{s}$$t_{snk}$$J_{\mu}^{\dagger}$$t_{2}$$J_{\nu}$$t_{1}$$B_{s}$$t_{src}$$b$$\bar{s}$$b$$c$
Figure 1: Schematic representation of the four-point Euclidean correlation
function defined in eq. (5)

The Euclidean correlator is saturated by the matrix element

$\displaystyle M_{\mu\nu}(t;\boldsymbol{q})=e^{-m_{B}t}\,\int
d^{3}x\,\frac{e^{i\boldsymbol{q}\cdot\boldsymbol{x}}}{2m_{B}}\langle\bar{B}(\boldsymbol{0})|J_{\mu}^{\dagger}(\boldsymbol{x},\\!t)J_{\nu}(\boldsymbol{0},\\!0)|\bar{B}(\boldsymbol{0})\rangle,$
(6)

which can be shown, after a few lines of algebra, to be equal to the integral
over the hadronic tensor,

$\displaystyle M_{\mu\nu}(t;\boldsymbol{q})=\int_{0}^{\infty}d\omega
W_{\mu\nu}(\omega,\boldsymbol{q})\,e^{-\omega t}.$ (7)

In our analysis we focus on the contribution to the correlator between $t_{1}$
and $t_{2}$, in which the multi-particle states of the spectrum propagate. We
also write a linear combination of the Euclidean correlator in terms of the
hadronic-tensor components $Z^{(l)}(\omega,\boldsymbol{q})$ as

$\displaystyle G^{(l)}(a\tau;\boldsymbol{q})=\int_{0}^{\infty}d\omega
Z^{(l)}(\omega,\boldsymbol{q}^{2})e^{-\omega a\tau},$ (8)

where we expressed the Euclidean time in units of the lattice spacing (denoted
by $a$) as $a\tau$.

In our work we used two gauge ensembles generated by the JLQCD collaboration
[11, 12] and by the ETM collaboration [13, 14, 15, 16, 17, 18], which use
Domain Wall fermions and Twisted-Mass Wilson fermions respectively. In the
lattice simulations neither collaboration is able to accommodate a
relativistic $b$ quark with its physical mass. As a consequence, our results
are obtained with an unphysically light $B_{s}$ meson.

$L^{3}\times T$ | $N_{cnfg}$ |  $a$ $(fm)$ |  |
---|---|---|---|---
$32^{3}\times 64$ | $150$ | $0.0815(30)$ |  |
$a\mu_{l}$ | $a\mu_{\sigma}$ | $a\mu_{\delta}$ |  $m_{\pi}$ $(\text{\rm MeV})$ |
$0.0025$ | $0.135$ | $0.170$ | $375(13)$ |

$L^{3}\times T\times L_{s}$ | $N_{cnfg}$ |  $a$ $(fm)$ |  |
---|---|---|---|---
$48^{3}\times 96\times 8$ | $100$ | $0.055$ |  |
$am_{ud}$ | $am_{s}$ | $m_{\pi}(\text{\rm MeV})$ |  |
$0.0042$ | $0.0025$ | $300(1)$ |  |

Table 1: Gauge ensemble details for the ETM (left) and JLQCD (right)
collaborations. The number of dynamical quarks is $N_{f}=2+1+1$ in ETMC case
and $N_{f}=2+1$ in JLQCD case. In the Twisted Mass simulation, $a\mu_{\sigma}$
and $a\mu_{\delta}$ are heavy doublet parameters related to the strange- and
charm-quark masses.

In table 1 the details of the gauge configurations are summarised. Further
details about the ensembles can be found in ref. [8] and references therein.

##  4 Kernel reconstruction

In order to compute the differential decay rate one has to extract the
hadronic spectral density from Euclidean correlators, which is an ill-posed
inverse problem and therefore needs to be tackled with appropriate numerical
methods. In this work we apply two methods, one based on Chebyshev polynomials
[19] and another one based on the variant of the Backus-Gilbert method
proposed in ref. [20], which are applied respectively to the JLQCD ensemble
and the ETMC ensemble. The first step in both methods is to use the fact that
any smooth function $f(\omega)$ can be approximated numerically by a series of
polynomials

$\displaystyle f(\omega)=\sum_{\tau}^{\infty}g_{\tau}e^{-a\omega\tau},$ (9)

so that eq. (8) can be rewritten as

$\displaystyle\sum_{\tau}^{\infty}g_{\tau}G^{(l)}(a\tau;\boldsymbol{q})=\int_{0}^{\infty}d\omega
Z^{(l)}(\omega;\boldsymbol{q}^{2})f(\omega)\;.$ (10)

One can then notice how the term on the r.h.s. of the above equation is
analogous to the expression in eq. (4), if the kernel
$\Theta^{l}(\omega_{max}-\omega)$ is substituted with $f(\omega)$. However,
since the kernel $\Theta^{l}(\omega_{max}-\omega)$ is defined in terms of a
step-function which, by definition, is not a smooth function, one cannot
perform the substitution straightforwardly. This problem can be solved
introducing a smeared version of the step function $\theta_{\sigma}$ which in
turns defines a smeared integration kernel
$\Theta^{l}_{\sigma}(\omega_{max}-\omega)$ that is sufficiently smooth such
that it can be reconstructed with a series of polynomials

$\displaystyle\Theta^{l}_{\sigma}(\omega_{max}-\omega)=(\omega_{max}-\omega)^{l}\theta_{\sigma}(\omega_{max}-\omega)=m_{B_{s}}^{l}\sum_{\tau}^{\infty}g_{\tau}(\omega_{max};\sigma)e^{-a\omega\tau}\;.$
(11)

This final step allows us to write eq. (4) in terms of products of the
Euclidean correlator and the coefficients of the smeared kernel:

$\displaystyle
Z_{\sigma}^{(l)}(\boldsymbol{q}^{2})=\sum_{\tau}^{\infty}g_{\tau}(\omega_{max};\sigma)G^{(l)}(a\tau;\boldsymbol{q})\;.$
(12)

It is important to notice that eq. (12) is not the same quantity showed in eq.
(4) because it is defined in terms of a smeared kernel and therefore cannot be
used to compute the differential decay rate. Moreover, hadronic spectral
densities, and therefore also the structure functions
$Z^{(l)}(\omega,\boldsymbol{q}^{2})$, contained in the definition of the
$G^{(l)}$ correlators are computed in a finite volume and as a consequence
have a discrete energy spectrum, and therefore cannot be directly associated
with physical observables.

###  4.1 The HLT algorithm

In the above section we discussed how one of the key ingredients for the
calculation of eq. (4) are the smeared kernel coefficients $g_{\tau}$. In this
section we briefly review the method proposed in ref. [20] and we direct the
reader interested in the Chebyshev method to ref. [19, 21].

In the method of ref. [20], the coefficients $g_{\tau}$ are obtained by
minimising the functional

$\displaystyle W_{\lambda}[g]=(1-\lambda)\frac{A[g]}{A[0]}+\lambda B[g]$ (13)

where the $A[g]$ and $B[g]$ functionals are given by

$\displaystyle
A[g]=a\int_{E_{0}}^{\infty}d\omega\left\\{\Theta^{l}(\omega_{max}-\omega)-\sum_{\tau=1}^{\tau_{\rm
max}}g_{\tau}\,e^{-a\omega\tau}\right\\}^{2}\,,\qquad
B[g]=\sum_{\tau,\tau^{\prime}=1}^{\tau_{\rm
max}}g_{\tau}g_{\tau^{\prime}}\,\frac{{\rm
Cov}\left[G(a\tau),G(a\tau^{\prime})\right]}{\left[G(0)\right]^{2}}\,.$ (14)

The parameter $\lambda$ is a trade-off parameter which regulates the
minimisation procedure between the $A[g]$ and $B[g]$ functionals. The
functional $A[g]$ is the reconstruction bias which, in the absence of errors,
after the minimisation procedure gives the best polynomial approximation of
the kernel $\Theta^{l}(\omega_{max}-\omega)$. However, considering that
lattice data always have an associated statistical error, what happens is that
minimising the $A[g]$ functional for small $\sigma$ results in huge values of
the coefficients $g_{\tau}$. This is an effect related to the fact that we are
dealing with an ill-posed numerical problem. The introduction of the $B[g]$
functional together with the parameter $\lambda$ solves this problem by
extending the minimisation procedure to the statistical variance at the
expense of a perfect kernel reconstruction. The coefficients
$g_{\tau}^{\lambda}$ that minimise $W[g]$ will then be a particular balance
between the systematic and statistical errors, influenced by the $\lambda$
parameter. Following the procedure of ref. [20], the optimal value of
$\lambda$ is obtained by finding the maximum of the functional $W[g]$ as a
function of $\lambda$, finding $\lambda_{\star}$. One can appreciate the role
of $\lambda$ in balancing the statistical and systematic errors in figure 2.

Figure 2:  Integral $\bar{Z}^{(0)}_{\sigma}(\boldsymbol{q})$ of the hadronic
correlator with three kernels, plotted as a function of $A[g^{\lambda}]/A[0]$.
No significant difference is observed within the statistical errors for values
$A[g^{\lambda}]/A[0]$ smaller than $A[g^{\lambda_{\star}}]/A[0]$.

##  5 Lattice results and comparison with the OPE

Having found the values of the $g_{\tau}$ coefficients, either with the method
of ref. [19] or that of ref. [20], one is then able to compute eq. (12). As
discussed in section 4, the quantity calculated in eq. (12) cannot be used to
compute any physical observable. In order to make a connection between
$Z^{(l)}_{\sigma}$ and the corresponding physical quantity one has to first
perform the infinite-volume limit and only then take the $\sigma\rightarrow 0$
limit:

$\displaystyle Z^{(l)}(\boldsymbol{q}^{2})=\lim_{\sigma\rightarrow
0}\left(\lim_{V\rightarrow\infty}\right)m_{B_{s}}^{l}\sum_{\tau}^{\infty}g_{\tau}(\omega,\sigma)G^{(l)}(a\tau;\boldsymbol{q})\;.$
(15)

This is because the study of the infinite-volume limit of smeared spectral
densities is well-defined only for fixed smearing functions and therefore the
two limits do not commute. However, due to the exploratory nature of this
work, we restricted our analysis only to one physical volume for each
configuration ensemble, so that a $V\rightarrow\infty$ extrapolation was not
possible. This choice can be justified by the fact that our present
statistical uncertainties are likely larger than finite-volume effects; we
plan to investigate these effects more thoroughly in future work with
simulations in multiple volumes. Figure 3 shows the $\sigma\rightarrow 0$
limit taken with three smeared kernels.

Figure 3: Combined $\sigma\rightarrow 0$ extrapolation of $Z^{0}_{\sigma}$
employing $10$ values of $\sigma\in[0.12m_{B_{s}},0.3m_{B_{s}}]$ and using the
smallest $5$ to perform the fit.

After taking the $\sigma\rightarrow 0$ limit, the $Z^{(l)}(\boldsymbol{q})$
can be used to compute the differential decay rate as shown in eq. (3). The
lattice results obtained using the JLQCD and ETM ensembles cannot be compared
with each other since they use different quark masses in their simulations.
However, both of them can be compared with the analytic predictions from the
OPE.

Figure 4: Differential $\boldsymbol{q}^{2}$ spectrum, divided by
$|\boldsymbol{q}|$, in the SM. Comparison of OPE predictions with lattice QCD
results from the JLQCD (top panel) and ETM (bottom panel) ensembles are shown.

As shown in figure 4, the agreement between the OPE curve (in blue) and the
lattice data (in red) is very good, for both the JLQCD and the ETM ensembles.

One can then obtain the inclusive decay rate by performing the integration
over $\boldsymbol{q}$. In table 2 we show the final results obtained using the
JLQCD and ETM ensembles compared with the OPE predictions. We also show the
results of the first lepton energy moment $\langle E_{l}\rangle$ and the first
hadronic mass moment $\langle M_{X}^{2}\rangle$.

| JLQCD | OPE
---|---|---
$\Gamma/|V_{cb}^{2}|\times 10^{13}$ (GeV) | $4.46(21)$ | 5.7(9)
$\langle E_{\ell}\rangle$ (GeV) | 0.650(40) | 0.626(36)
$\langle M_{X}^{2}\rangle$ (GeV2) | 3.75(31) | 4.22(30)

ETMC | OPE
---|---
0.987(60) | 1.20(46)
0.491(15) | 0.441(43)
3.62(14) | 4.32(56)

Table 2: Total widths and moments.

What can be concluded from this comparison is that the lattice data show a
good agreement with the OPE curves, especially at low $\boldsymbol{q}$ and the
final results are compatible within the uncertainties.

These findings provide an important and non-trivial test of the method used in
this work, making us optimistic about the possibility of soon having a full
lattice QCD computation of inclusive semileptonic decays, at a level of
precision competitive with those from the OPE. This will certainly give a big
contribution towards a more precise inclusive determination of $|V_{cb}|$,
which will hopefully clarify the origin of the current $3\sigma$ tension
between the inclusive and exclusive determination of this quantity. Another
reason why a full inclusive lattice QCD computation is important is the
possibility of testing the quark-hadron duality on which the OPE is based, by
comparing calculations of the same physical observable with both methods.

##  6 Conclusions

We presented our recent study of the inclusive semileptonic decay of the
$B_{s}$ meson from lattice QCD [8], which is based on the strategy proposed in
ref. [10]. As we discussed, our calculation is based on the numerical
reconstruction of the integration kernel that regulates the integral over
phase space of the structure functions contributing to the hadronic tensor. We
computed four-point Euclidean correlation functions which were obtained from
two different gauge ensembles, based on different discretisation schemes for
both gauge and quark fields and at $b$-quark masses lighter than in nature. We
used two different methods for the kernel reconstructions, which were applied
to the correlation functions in order to obtain an inclusive calculation of
the semileptonic decay rate. The results that we obtained are consistent and
can be successfully compared with analytic predictions from the OPE approach.
This opens the path for systematic _ab initio_ non-perturbative studies of
inclusive decays on the lattice, which hopefully will help solve the $V_{cb}$
puzzle.

## Acknowledgments

The numerical calculations of the JLQCD collaboration were performed on the
SX-Aurora TSUBASA at the High Energy Accelerator Research Organization (KEK)
under its Particle, Nuclear and Astrophysics Simulation Program, as well as on
the Oakforest-PACS supercomputer operated by the Joint Center for Advanced
High Performance Computing (JCAHPC). We thank the members of the JLQCD
collaboration for sharing the computational framework and lattice data, and
Takashi Kaneko in particular for providing the numerical data for the
exclusive decay form factors. The numerical simulations of the ETM
collaboration were run on machines of the Consorzio Interuniversitario per il
Calcolo Automatico dell’Italia Nord Orientale (CINECA) under the specific
initiative INFN-LQCD123. The work of S.H. is supported in part by JSPS KAKENHI
Grant Number 22H00138 and by the Post-K and Fugaku supercomputer project
through the Joint Institute for Computational Fundamental Science (JICFuS).
The work of P.G., S.M., F.S., S.S. is supported by the Italian Ministry of
Research (MIUR) under grant PRIN 20172LNEEZ. This project has received funding
from the Swiss National Science Foundation (SNF) under contract 200020_204428.

## References

  * [1] Particle Data Group collaboration, _Review of Particle Physics_ , _PTEP_ 2020 (2020) 083C01.
  * [2] HFLAV collaboration, _Averages of $b$-hadron, $c$-hadron, and $\tau$-lepton properties as of 2018_, _Eur. Phys. J. C_ 81 (2021) 226 [1909.12524].
  * [3] P. Gambino, M. Jung and S. Schacht, _The $V_{cb}$ puzzle: An update_, _Phys. Lett. B_ 795 (2019) 386 [1905.08209].
  * [4] P. Gambino et al., _Challenges in semileptonic $B$ decays_, _Eur. Phys. J. C_ 80 (2020) 966 [2006.07287].
  * [5] K.G. Wilson, _Nonlagrangian models of current algebra_ , _Phys. Rev._ 179 (1969) 1499.
  * [6] L.P. Kadanoff, _Operator Algebra and the Determination of Critical Indices_ , _Phys. Rev. Lett._ 23 (1969) 1430.
  * [7] Flavour Lattice Averaging Group collaboration, _FLAG Review 2019: Flavour Lattice Averaging Group (FLAG)_ , _Eur. Phys. J. C_ 80 (2020) 113 [1902.08191].
  * [8] P. Gambino, S. Hashimoto, S. Mächler, M. Panero, F. Sanfilippo, S. Simula et al., _Lattice QCD study of inclusive semileptonic decays of heavy mesons_ , _JHEP_ 07 (2022) 083 [2203.11762].
  * [9] S. Hashimoto, _Inclusive semi-leptonic $B$ meson decay structure functions from lattice QCD_, _PTEP_ 2017 (2017) 053B03 [1703.01881].
  * [10] P. Gambino and S. Hashimoto, _Inclusive Semileptonic Decays from Lattice QCD_ , _Phys. Rev. Lett._ 125 (2020) 032001 [2005.13730].
  * [11] JLQCD collaboration, _Determination of charm quark mass from temporal moments of charmonium correlator with Möbius domain-wall fermion_ , _PoS_ LATTICE2016 (2017) 192 [1702.01498].
  * [12] JLQCD collaboration, _Form factors of $B\to\pi\ell\nu$ and a determination of $|V_{ub}|$ with Möbius domain-wall-fermions_, _Phys. Rev. D_ 106 (2022) 054502 [2203.04938].
  * [13] R. Baron et al., _Light hadrons from lattice QCD with light ( $u$,$d$), strange and charm dynamical quarks_, _JHEP_ 06 (2010) 111 [1004.5284].
  * [14] ETM collaboration, _Light hadrons from $N_{f}=2+1+1$ dynamical twisted mass fermions_, _PoS_ LATTICE2010 (2010) 123 [1101.0518].
  * [15] Alpha collaboration, _Lattice QCD with a chirally twisted mass term_ , _JHEP_ 08 (2001) 058 [hep-lat/0101001].
  * [16] R. Frezzotti and G.C. Rossi, _Twisted mass lattice QCD with mass nondegenerate quarks_ , _Nucl. Phys. B Proc. Suppl._ 128 (2004) 193 [hep-lat/0311008].
  * [17] R. Frezzotti and G.C. Rossi, _Chirally improving Wilson fermions. 1. $O(a)$ improvement_, _JHEP_ 08 (2004) 007 [hep-lat/0306014].
  * [18] ETM collaboration, _Up, down, strange and charm quark masses with $N_{f}=2+1+1$ twisted mass lattice QCD_, _Nucl. Phys. B_ 887 (2014) 19 [1403.4504].
  * [19] G. Bailas, S. Hashimoto and T. Ishikawa, _Reconstruction of smeared spectral function from Euclidean correlation functions_ , _PTEP_ 2020 (2020) 043B07 [2001.11779].
  * [20] M. Hansen, A. Lupo and N. Tantalo, _Extraction of spectral densities from lattice correlators_ , _Phys. Rev. D_ 99 (2019) 094508 [1903.06476].
  * [21] S. Hashimoto and G. Bailas, _Composition of the inclusive semi-leptonic decay of $B$ meson_, _PoS_ LATTICE2021 (2022) 534 [2112.09295].


xx xx XX

# Beam Manipulation by Hybrid Plasmonic-Dielectric Metasurfaces

K. Arik1 1 School of Electrical Engineering, Sharif University of Technology

(XX)

###### Abstract

A hybrid plasmonic-dielectric metasurface is proposed in order to manipulate
beam propagation in desired manners. The metasurface is composed of patterned
hybrid graphene-silicon nano-disks deposited on a low-index substrate, namely
silica. It is shown that the proposed hybrid metasurface simultaneously
benefits from the advantages of graphene-based metasurfaces and dielectric
ones. Specially, we show that the proposed hybrid metasurface not only
provides reconfigurability, just like previously proposed graphene-based
metasurfaces, but also similar to dielectric metasurfaces it is low loss and
CMOS-compatible. Such exceptional features give the metasurface exceptional
potentials to realize high efficient optical components. To demonstrate the
latter point, focusing and anomalous reflection are performed making use of
the proposed hybrid structure as examples of two well-known optics
functionalities. This work opens up a new route in realization of
reconfigurable meta-devices with widely real-world applications which cannot
be achieved with their passive counterparts.

Beam manipulation, metasurface, graphene, focusing, anomalous reflection.

##  1 Introduction

Wavefront shaping of light has been of utmost increasing attention since the
very beginning time, as for example, lenses and prisms have been traditionally
employed to focus and refract the light [1]. These conventional optical
devices, which are indeed based on phase accumulation along the light
propagation path, are suffering from the bulky geometry as well as their
relatively high reflection loss, restricting their application in the planar
configurations [2, 3, 4].
The attempts to overcome the aforementioned restrictions of the conventional
optical devices have stimulated the concept of frequency selective surfaces
(FSS) [5]. Such appealing surfaces, which are usually composed of planar
arranged metallic patches or apertures, have the potential to control either
reflection or transmission of electromagnetic waves over a highly thin
surface. This can be accomplished by properly tuning the elements’ size,
shape, and thickness of the FSS [6].
In a different area of research, the attempts to improve the characteristics
of the conventional optical components has inspired introducing artificial
engineered composite materials, known as man-made materials or ”metamaterial”.
These engineered materials have attracted a lot of attention from physicist to
realize novel and interesting optical components due to the fact that they
provide a new degree of freedom to control electromagnetic waves in desired
manners [7, 8, 9, 10, 11]. However, their fabrication is still a challenge for
the current technology due to their bulky geometry [7, 12].
Very recently, by combining the concepts of metamaterilas and frequency
selective surfaces, two dimensional metamaterials, also known as metasurfaces,
have been proposed. Just the same as metamaterials, metasurfaces provide an
exceptional ability to control electromagnetic waves in unusual ways, while at
the same time their FSS-like planar configuration eases the fabrication
process [13, 14]. It has been demonstrated recently that by imparting a
controlled gradient of phase discontinuity over the surfaces, the reflected or
transmitted field of the metasurface can be manipulated to the desired
directions, apart from those expected by conventional Snell’s laws of
refraction [15].
There are basically two types of metasurfaces proposed so far to manipulate
electromagnetic waves: plasmonic metasurfaces among which those based on
graphne plasmonics has been of most interest [3, 14, 16, 17], and dielectric
metasurfaces [18, 19, 20, 21, 22, 23]. Each of these structures has its own
pros and cons. In particular, while plasmonic metasurfaces, more specifically
graphene-based ones, provide excellent features such as high confinement of
the field and reconfiguration, they suffer from high values of ohmic losses
[3]. In reverse, all-dielectric metasurfaces offer lower amount of non-
radiation losses as well as compatibility with CMOS technology, while they can
not be reconfigured [19].
In this contribution, we propose a hybrid plasmonic-dielectric structure in
order to simultaneously benefit from the advantages of plasmonic and
dielectric metasurfaces. We will show that our proposed hybrid metasurface not
only provides reconfigurability, but also is low loss and CMOS-compatible.
Such exceptional features enable us to realize efficient optical devices. To
illustrate the latter, two optics functionalities, namely focusing and
anomalous refraction are performed making use of the proposed hybrid
structure.

(a)

(b)

Figure 1: (a) Schematic of the proposed metasurface for beam manipulation: a
hybrid layer of silicon and graphene is deposited on a silica substrate placed
on top of a thick gold metallic layer, (b) schematic of a single unit-cell of
the proposed hybrid metasurfce.

(a)  (b)

Figure 2:  (a) Amplitude, and (b) phase of the reflection coefficient of a
unit cell of the proposed hybrid metasurface versus the Fermi level of the
graphene patch $E_{f}$, and the radius of the hybrid nano-disk r.

##  2 Structure Design

The schematic of the suggested hybrid plasmonic-dielectric is shown in Fig. 1.
As it is observed, the structure is composed of a periodic array of nano-disks
deposited on top of a dielectric spacer, namely silicon dioxide, with the
thickness of $t_{s}$. Each nano-disk, in turn, is composed of a layer of
silicon, with the thickness of $t_{d}$, and a graphene patch, with the surface
conductivity of $\sigma_{g}$ on the top. A metallic gold layer with the
thickness of $t_{g}$ is also embedded at the bottom of the whole structure so
as to enhance the interaction of light with the metasurfce [14].
In the proposed structure, the silicon layer and the graphene patch take the
role of dielectric and plasmonic parts of the metasurface, respectively. To
clarify this, we express the surface conductivity of graphene $\sigma_{g}$ in
terms of its Fermi level $E_{f}$ and scattering rate $\Gamma$ as [24, 25, 26]

$\displaystyle\sigma_{g}$
$\displaystyle=i\frac{e^{2}K_{B}T}{\pi\hbar^{2}(\omega+2i\Gamma)}[\frac{E_{f}}{K_{B}T}+2\ln(\exp(-\frac{E_{f}}{K_{B}T})+1)]$
(1)
$\displaystyle+i\frac{e^{2}}{4\pi\hbar^{2}}\ln[\frac{2|E_{f}|-\hbar(\omega+2i\Gamma)}{2|E_{f}|+\hbar(\omega+2i\Gamma)}],$

in which $\hbar$, $K_{B}$, $T$, and $\omega$ are reduced Planck constant,
Boltzman constant, temperature in Kelvin, and angular frequency, respectively,
and $i=\sqrt{-1}$. It can be deduced from Eq. (1) that if the frequency of
operation is limited to few terahertz, the surface conductivity $\sigma_{g}$
becomes inductive enabling the patch to support surface plasmons [24].
Another interesting feature of the surface conductivity of the graphene patch,
given in Eq. (1), is that it can easily be tuned via a change in the Fermi
level $E_{f}$ of the graphene. Such appealing feature has made graphene an
exceptional material to design reconfigurable optics and electronics
components [27, 28, 29]. In the following, we also have exploited this unique
feature to have a full control over the phase of the metasurface, which
provides the opportunity to demonstrate desired optical functionalities.
To design the proposed metasurface for a specific optical functionality, the
characteristics of a single unit cell should first be investigated. Here,
without loss of generality, the periodicity of the metasurface and the
thicknesses $t_{d}$ and $t_{s}$ are considered to be $L=3~{}\mu{\rm m}$,
$t_{d}=3~{}\mu{\rm m}$ and $t_{s}=0.05~{}\mu{\rm m}$. Moreover, we assume the
operation frequency to be $f=10~{}{\rm THz}$. The operation frequency is
chosen such that the mentioned criteria regarding the surface conductivity of
the graphene patch is maintained, and consequently it can guide surface
plasmons.

(a)  (b)  (c)  (d)

Figure 3: (a) Required phase distribution to perform focusing at the focal
length of $F=120~{}\mu{\rm m}$, (b) field profile of the incident field, (c)
field profile of the corresponding reflected field, and (d) profile of the
corresponding reflected power.

(a)  (b)  (c)  (d)

Figure 4: (a) Required phase distribution to perform focusing at the focal
length of $F=30~{}\mu{\rm m}$, (b) field profile of the incident field, (c)
field profile of the corresponding reflected field, and (d) profile of the
corresponding reflected power.

figs. 2(a-b) show the amplitude and phase of the reflection coefficient of the
unit cell versus the Fermi level of the graphene patch $E_{f}$, and the radius
of the nano-disk $r$. The results are obtained using a full-wave
electromagnetic solver, namely HFSS. In HFSS, the graphene patches were
modeled via impedance layers. The corresponding values of impedances were
calculated simply by reversing the parameter $\sigma_{g}$ in Eq. (1). As one
can observe from Figs. 2(a-b), the magnitude of the reflection coefficient is
higher than 0.75 and simultaneously its phase varies smoothly from $-\pi$ to
$\pi$. The full control over the phase of the reflected field is due to the
mentioned tunable conductivity of graphene. Moreover, the ohmic-lossless
dielectric layer, i.e., silicon, keeps the amplitude of the reflection high.
These features enable the proposed hybrid metasurface to perform high
efficient optical functionalities.
In order to perform a specific optical functionality, the phase of the
corresponding reflected field to each element of the metasurface should be
modulated properly. For instance, to perform focusing, the phase profile has
to be modulated in a way that the reflected fields from single elements
interfere with each other constructively. To maintain such condition, the
phase profile should be of the form [30]

$\phi_{\rm lens}(x)=\frac{2\pi}{\lambda}(\sqrt{F^{2}+x^{2}}-F)$ (2)

in which $F$ is the focal length and $\lambda$ is the wavelength of operation.
Here, we consider a focal length of $F=4\lambda=120~{}\mu{\rm m}$ and $x=mL$,
in which $L$ equals to the periodicity of the nano-disks and $m=0,\pm 1,\pm
2,...,\pm 29,\pm 30$. The required phase distribution is then calculated
making use of Eq. (2), and the result is depicted in Fig. 3(a). Based on the
calculated phase profile of a single unit-cell of the metasurface in Fig.
2(b), we can then appropriately determine the radii and Fermi levels of nano-
disks so that the phase of the reflected field of each element matches the
phase distribution of Fig. 3(a). It should also be noted that the values of
radii and the Fermi levels should be chosen in a way that the amplitude of
reflection, shown in Fig. 3(a), becomes as high as possible.
A plane wave with the field profile shown in Fig. 3(b) is then considered to
perpendicularly be incident on the designed hybrid metasurface. The field
profile of the corresponding reflected field is shown in Fig. 3(c). As it is
observed, the reflected light is efficiently focused at the focal point of
$F=120~{}\mu{\rm m}$, which is consistent with our expectation. The profile of
the reflected power is also depicted in Fig. 3(d).
In order to assess the tunability of the device, we then tune the values of
the Fermi levels such that the focal length is adapted to $F=\lambda=30\mu{\rm
m}$. Similar results to those illustrated in Fig. 3 are then obtained and
depicted in Figs. 4(a-d), respectively. As it is observed, the reflected field
has been efficiently focused at the desired focal length. The opportunity to
focus the reflected field in such a small focal length is as a result of the
plasmonic behaviour of the metasurface [3]. On the other hand, the overall
reflectivity is high, which is because of the ohmic loss-less dielectric part
of the metasurface.
As another example, we demonstrate anomalous reflection functionality by means
of the proposed metasurface. For bending the reflected field of an incident
field towards a specific direction, $\theta_{r}$, the following linear phase
profile has to be imprinted to the incident wave [31]

$\phi_{\rm
anomalous}(x)=\frac{2\pi}{\lambda}(\sin(\theta_{r})-\sin(\theta_{i}))x,$ (3)

(a)  (b)  (c)

Figure 5: (a) Required phase distribution to perform anomalous reflection for
$\theta_{i}=0^{\circ}$ and $\theta_{r}=30^{\circ}$, (b) field profile of the
incident field, and (c) field profile of the corresponding reflected field.

where $\theta_{i}$ is the angle of incidence and $x=mL$, in which $L$ is the
periodicity and $m$ is an integer. For the sake of simplicity, we consider the
case in which the incident wave is perpendicularly incident onto the
metasurface, i.e., $\theta_{i}=0^{\circ}$, and aim to bend the reflected wave
toward the angle $\theta_{r}=30^{\circ}$. Fig. 5(a) shows the required phase
distribution $\phi_{\rm anomalous}$. Based on this figure, and by employing
similar procedure presented previously, one can appropriately design the
metasurface to bend the reflected wave in the desired angle, $\theta_{r}$.
To evaluate the performance of the designed metasurface, a plane wave shown in
Fig. 5(b) is considered to be incident on the designed structure, and then its
associated reflected field is obtained and depicted in Fig. 5(c). As it is
observed, the reflected field is bent to the desired angle
$\theta_{r}=30^{\circ}$.
It is worth noting that with respect to the previously proposed plasmonic
metasurfaces, like those investigated in [3, 14, 32], our suggested hybrid
metasurface has two main advantages: (1) compatibility to the current CMOS
technology, and (2) low level of ohmic losses. On the other hand, in
comparison with the previously proposed dielectric metasurfaces [19, 20], the
proposed structure has the advantage of tunability, which is directly related
to the unique feature of graphene.

##  3 Conclusions

In this work, we proposed a novel hybrid plasmonic-dielectric metasurface for
beam manipulation. The metasurface was composed of patterned hybrid graphene-
silicon nano-disks deposited on a silica substrate. Two optical
functionalities, namely focusing and anomalous reflection were demonstrated by
means of the proposed hybrid structure. The proposed hybrid metasurface had
the advantages of previously proposed graphene-based metasurfaces such as
tunability, together with those of dielectric metasurfaces like low ohmic
lossees, and CMOS compability. One may also extend the proposed metasurface to
perform various 3D optical functionalities.

## References

  * [1] D. Brewster, “On an account of a rock-crystal lens and decomposed glass found in niniveh,” _Die Fortschritte der Physik im Jahre_ , pp. 355–356, 1852\.
  * [2] C. Pfeiffer and A. Grbic, “Metamaterial huygens’ surfaces: tailoring wave fronts with reflectionless sheets,” _Physical review letters_ , vol. 110, no. 19, p. 197401, 2013.
  * [3] S. AbdollahRamezani, K. Arik, S. Farajollahi, A. Khavasi, and Z. Kavehvash, “Beam manipulating by gate-tunable graphene-based metasurfaces,” _Optics letters_ , vol. 40, no. 22, pp. 5383–5386, 2015.
  * [4] N. Yu, P. Genevet, F. Aieta, M. A. Kats, R. Blanchard, G. Aoust, J.-P. Tetienne, Z. Gaburro, and F. Capasso, “Flat optics: controlling wavefronts with optical antenna metasurfaces,” _IEEE Journal of Selected Topics in Quantum Electronics_ , vol. 19, no. 3, pp. 4 700 423–4 700 423, 2013.
  * [5] B. A. Munk, “Frequency selective surfaces theory and design. john wiley&sons,” 2000.
  * [6] N. Yu, P. Genevet, M. A. Kats, F. Aieta, J.-P. Tetienne, F. Capasso, and Z. Gaburro, “Light propagation with phase discontinuities: generalized laws of reflection and refraction,” _science_ , vol. 334, no. 6054, pp. 333–337, 2011.
  * [7] N. Engheta and R. W. Ziolkowski, _Metamaterials: physics and engineering explorations_. John Wiley & Sons, 2006\.
  * [8] A. Silva, F. Monticone, G. Castaldi, V. Galdi, A. Alù, and N. Engheta, “Performing mathematical operations with metamaterials,” _Science_ , vol. 343, no. 6167, pp. 160–163, 2014.
  * [9] A. Youssefi, F. Zangeneh-Nejad, S. Abdollahramezani, and A. Khavasi, “Analog computing by brewster effect,” _Optics letters_ , vol. 41, no. 15, pp. 3467–3470, 2016.
  * [10] F. Zangeneh-Nejad and R. Fleury, “Performing mathematical operations using high-index acoustic metamaterials,” _arXiv preprint arXiv:1803.02589_ , 2018\.
  * [11] ——, “Acoustic birefringence via non-eulerian metamaterials,” _arXiv preprint arXiv:1803.09070_ , 2018.
  * [12] A. E. Dorche, S. Abdollahramezani, A. Chizari, and A. Khavasi, “Broadband, polarization-insensitive, and wide-angle optical absorber based on fractal plasmonics,” _IEEE Photonics Technology Letters_ , vol. 28, no. 22, pp. 2545–2548.
  * [13] L. Liu, X. Zhang, M. Kenney, X. Su, N. Xu, C. Ouyang, Y. Shi, J. Han, W. Zhang, and S. Zhang, “Broadband metasurfaces with simultaneous control of phase and amplitude,” _Advanced Materials_ , vol. 26, no. 29, pp. 5031–5036, 2014\.
  * [14] Z. Li, K. Yao, F. Xia, S. Shen, J. Tian, and Y. Liu, “Graphene plasmonic metasurfaces to steer infrared light,” _Scientific reports_ , vol. 5, p. 12423, 2015.
  * [15] A. V. Kildishev, A. Boltasseva, and V. M. Shalaev, “Planar photonics with metasurfaces,” _Science_ , vol. 339, no. 6125, p. 1232009, 2013.
  * [16] L. Huang, X. Chen, H. Mühlenbernd, H. Zhang, S. Chen, B. Bai, Q. Tan, G. Jin, K.-W. Cheah, C.-W. Qiu _et al._ , “Three-dimensional optical holography using a plasmonic metasurface,” _Nature communications_ , vol. 4, 2013.
  * [17] F. Zangeneh-Nejad and R. Safian, “A graphene-based thz ring resonator for label-free sensing,” _IEEE Sensors Journal_ , vol. 16, no. 11, pp. 4338–4344.
  * [18] Y. Yang, I. I. Kravchenko, D. P. Briggs, and J. Valentine, “All-dielectric metasurface analogue of electromagnetically induced transparency,” _Nature communications_ , vol. 5, 2014.
  * [19] Y. F. Yu, A. Y. Zhu, R. Paniagua-Domínguez, Y. H. Fu, B. Luk’yanchuk, and A. I. Kuznetsov, “High-transmission dielectric metasurface with 2$\pi$ phase control at visible wavelengths,” _Laser & Photonics Reviews_, vol. 9, no. 4, pp. 412–418, 2015.
  * [20] J. Cheng, D. Ansari-Oghol-Beig, and H. Mosallaei, “Wave manipulation with designer dielectric metasurfaces,” _Optics letters_ , vol. 39, no. 21, pp. 6285–6288, 2014.
  * [21] A. Chizari, S. Abdollahramezani, M. V. Jamali, and J. A. Salehi, “Analog optical computing based on a dielectric meta-reflect array,” _Optics letters_ , vol. 41, no. 15, pp. 3451–3454, 2016.
  * [22] T. Rajabzadeh, M. H. Mousavi, S. Abdollahramezani, M. V. Jamali, and J. A. Salehi, “Femtosecond cdma using dielectric metasurfaces: Design procedure and challenges,” _arXiv preprint arXiv:1712.00834_ , 2017.
  * [23] T. Guo and C. Argyropoulos, “Broadband polarizers based on graphene metasurfaces,” _Optics letters_ , vol. 41, no. 23, pp. 5592–5595, 2016.
  * [24] G. W. Hanson, “Dyadic green’s functions and guided surface waves for a surface conductivity model of graphene,” _Journal of Applied Physics_ , vol. 103, no. 6, p. 064302, 2008.
  * [25] F. Zangeneh-Nejad and A. Khavasi, “Spatial integration by a dielectric slab and its planar graphene-based counterpart,” _Optics letters_ , vol. 42, no. 10, pp. 1954–1957, 2017.
  * [26] C. Argyropoulos, “Enhanced transmission modulation based on dielectric metasurfaces loaded with graphene,” _Optics Express_ , vol. 23, no. 18, pp. 23 787–23 797, 2015.
  * [27] Y. Yao, M. A. Kats, P. Genevet, N. Yu, Y. Song, J. Kong, and F. Capasso, “Broad electrical tuning of graphene-loaded plasmonic antennas,” _Nano letters_ , vol. 13, no. 3, pp. 1257–1264, 2013.
  * [28] F. Zangeneh-Nejad and R. Safian, “A graphene-based THz ring resonator for label-free sensing,” _IEEE Sensors Journal_ , vol. 16, no. 11, pp. 4338–4344, 2016.
  * [29] A. Tredicucci and M. S. Vitiello, “Device concepts for graphene-based terahertz photonics,” _IEEE Journal of Selected Topics in Quantum Electronics_ , vol. 20, no. 1, pp. 130–138, 2014.
  * [30] J. Cheng and H. Mosallaei, “Optical metasurfaces for beam scanning in space,” _Optics letters_ , vol. 39, no. 9, pp. 2719–2722, 2014.
  * [31] F. Monticone, N. M. Estakhri, and A. Alù, “Full control of nanoscale optical transmission with a composite metascreen,” _Physical Review Letters_ , vol. 110, no. 20, p. 203903, 2013.
  * [32] S. AbdollahRamezani, K. Arik, A. Khavasi, and Z. Kavehvash, “Analog computing using graphene-based metalines,” _Optics letters_ , vol. 40, no. 22, pp. 5239–5242, 2015.


# A Systematic Study on Object Recognition Using Millimeter-wave Radar
††thanks:  Corresponding author: Maloy Kumar Devnath

Maloy Kumar Devnath, Avijoy Chakma, Mohammad Saeid Anwar, Emon Dey, Zahid
Hasan, Marc Conn*,
Biplab Pal, Nirmalya Roy
Information Systems Department, *Computer Engineering Department
University of Maryland, Baltimore County (UMBC), USA
{maloyd1, achakma1, saeid.anwar, edey1, zhasan3, mconn1, bpal1<EMAIL_ADDRESS>
###### Abstract

Millimeter-wave (MMW) radar is becoming an essential sensing technology in
smart environments due to its light and weather-independent sensing
capability. Such capabilities have been widely explored and integrated with
intelligent vehicle systems, often deployed in industry-grade MMW radars.
However, industry-grade MMW radars are often expensive and difficult to attain
for deployable community-purpose smart environment applications. On the other
hand, commercially available MMW radars pose hidden underpinning challenges
that are yet to be well investigated for tasks such as recognizing objects,
and activities, real-time person tracking, object localization, etc. Such
tasks are frequently accompanied by image and video data, which are relatively
easy for an individual to obtain, interpret, and annotate. However, image and
video data are light and weather-dependent, vulnerable to the occlusion
effect, and inherently raise privacy concerns for individuals. It is crucial
to investigate the performance of an alternative sensing mechanism where
commercially available MMW radars can be a viable alternative to eradicate the
dependencies and preserve privacy issues. Before championing MMW radar,
several questions need to be answered regarding MMW radar’s practical
feasibility and performance under different operating environments. To answer
the concerns, we have collected a dataset using commercially available MMW
radar, Automotive mmWave Radar (AWR2944) from Texas Instruments, and reported
the optimum experimental settings for object recognition performance using
several deep learning algorithms in this study. Moreover, our robust data
collection procedure allows us to systematically study and identify potential
challenges in the object recognition task under a cross-ambience scenario. We
have explored the potential approaches to overcome the underlying challenges
and reported extensive experimental results.

###### Index Terms:

MMW Radar, Cross-Ambience, Cross-Distance, Cross-Height, Robotics, Object
Recognition, Smart Sensing, Domain Adaptation.

##  I Introduction

Visual data, such as images and videos, has been extensively used in building
smart environments for various applications, including remote monitoring,
security management, object recognition, tracking mobile objects, autonomous
agents, and more [1, 2]. However, imaging as a sensing mechanism is
susceptible to environmental factors such as lighting and weather, and the
privacy of individuals may be compromised when captured by cameras. Under
certain circumstances, privacy is a non-negotiable constraint, and therefore,
the response to the concerns mentioned below related to visual data may not be
positive.

  1. 1.

would the tenants in an elderly care home be comfortable and agree to be
watched 24 hours by multiple cameras?

  2. 2.

how would a camera-based system respond, if a person is lying on the kitchen
floor due to sickness at night?

  3. 3.

can the tiny hazardous material in a playing grass field be identified by
images?

The above concerns have motivated us to explore an active sensing mechanism
such as RADAR-based technology that has the potential to bypass the
constraints of privacy and environmental dependencies. Note that, RADAR
technology with various capacities has been explored in numerous applications
such as remote monitoring [3], security management [4], object recognition
[5], tracking mobile objects [6, 7], autonomous agents (unmanned ground
vehicle) [8] and many more. We find that these radar technologies are often
expensive, difficult to develop in a timely manner, and require substantial
domain knowledge [3] which is a deterrent factor for a community-based
deployment. MMW radar is a recent addition to the radar technology pool that
projects short wavelength (30-300GHz) electromagnetic waves as short pulses
and receives the reflected pulses from the objects that lie in its projection
path [9] as depicted in Figure 1. MMW radar is commercially available and can
be integrated with research-grade programmable autonomous agents and vehicles
[10, 11, 8, 12]. However, MMW radar is not entirely free of constraints, which
leads us to conduct this systematic study and investigate those underpinning
challenges.

Figure 1: MMW radar’s basic principles.

Most of the literature that has been discussed in Section II involves large
objects [13] and ignores smaller objects under varying environmental
conditions where the next one pose several challenges. The challenges become
more severe when we use the sparse point cloud data of the MMW radar. First,
Impact of object size and shape \- hazardous objects such as broken metal
prongs, pets in a home environment, electronic devices, and lanyards are often
small in shape, therefore small surface area to receive and reflect the radar
signal. Second, Operating Environment \- often objects are composed of
materials with varying reflective characteristics. When radar projects a
millimeter wave on an object, the resulting reflected signal waves carry
varied signal-to-noise ratios. The reflected signal from the surrounding
objects complicates the final received signal and therefore the object
recognition task. Third, Radar Position and Distance \- radar position and the
relevant distance from the object are another impacting factor. Consider the
top and side view of a smartphone as depicted in Figure 2 and the change in
the surface area is obvious. In addition, as the distance between the radar
and the object increases, the projected signal diverges into a wider area,
resulting in an attenuation of the received signal. Currently, the object
recognition performance of a commercially available MMW radar under varying
shapes, environments, and dynamic settings is unknown.

Figure 2: Smart phone’s top and side view.

In this study, we make the following contributions -

  1. 1.

We develop models to recognize small-sized objects using low-resolution and
sparse point cloud MMW radar data in three indoor and outdoor environments
with varying settings, including static or dynamic radar projection and height
or distance.

  2. 2.

We are using domain adaptation techniques to improve the robustness of a deep
learning model on a large scale, specifically for recognizing small objects in
low-resolution and sparse point cloud data obtained from MMW radar. To the
best of our knowledge, this is the first study to explore domain adaptation
for these types of objects.

  3. 3.

We have validated our proposed system by conducting experiments in sunny
outdoor, well-lit indoor, and dark indoor settings with a large-scale dataset
collected using a TI AWR2944 MMW radar. Particularly, we focus on a
feasibility study to recognize varying-sized small objects such as dimes,
pencils, plastic sheets, quarters, Unnamed Ground Vehicle (UGV), water
bottles, wood, pencils, and paper under varying environmental settings
(distance, height, lighting, backgrounds) and collect around five hours of the
MMW dataset [14]. Our DA-based approaches have provided robustness against
dynamic factors’ variation between training and test data and achieved a 0.898
F1-score.

The study is organized as follows - Section II discusses recent works on MMW
radar. Data-gathering and processing techniques are described in Section III.
The model training mechanism is elaborated in Section IV. Section V reports
the experimental settings, results, and findings, and Section VII concludes
the study.

##  II Related Works

In this section, we briefly discuss the MMW radar-based approaches: First, we
discuss the literature where MMW radar is deployed as an individual sensor
unit and, second, when MMW radar is associated with other sensor units.

###  II-A MMW Radar-based approaches

MMW radar is widely used for various research purposes, including human motion
behavior detection [15], periodic heart rate measurements [16], navigation
[17], tracking multiple persons [7], pose detection [18] and estimation [19],
human activity recognition [20], security [4], autonomous driving [21, 11],
human face and emotion recognition [22, 23]. [15] senses the micro-Doppler
information of the user and detects the motion behavior. mBeats [16] measures
heart rate in different poses in an unobtrusive manner. [19] leverages two
radar data points for human pose estimation. Two radar points are used to
generate heatmaps, and Convolutional Neural Network is deployed to transform
the two-dimensional heatmaps into a human pose. [20] proposes RadHAR, a low-
cost human activity recognition framework that leverages sparse and non-
uniform point clouds from an MMW radar. [4] has proposed a secure method for
speaker verification using MMW radar to prevent an adversarial attack and
enhance the home security. Authors have utilized the radar to capture both
vocal cord vibration and lip motion for identifying speakers. [21] proposes a
new MMW radar point cloud classification algorithm to improve human-vehicle
classification accuracy in complex scenes of autonomous driving.

###  II-B MMW Radar with other sensor modalities

Aside from being deployed as a standalone sensor modality, MMW radar has also
been widely used as a vision-based sensor modality, mainly to overcome the
critical limitations of visibility of vision-based approaches in the absence
of light. [24] has proposed a fusion-based approach where MMW radar is used
for obstacle detection and features are fused with the vision sensor feature.
[25] proposes a new radar-camera fusion system that takes into consideration
the error bounds of the two different coordinate systems from the
heterogeneous sensors. The authors have utilized a new fusion-extended Kalman
filter to adapt to the heterogeneous sensors. [26] proposes a framework called
milliEye, a lightweight MMW radar and camera fusion system for robust object
detection on edge platforms. milliEye possesses several key advantages over
existing sensor fusion approaches. [5] proposes a deep radar object detection
network, named RODNet, which is cross-supervised by a camera-radar fused
algorithm. RODNet considers object detection in various driving scenarios.

###  II-C Differences with existing approaches

Advanced driver assistance systems (ADAS) and autonomous driving applications
in the automotive sector utilize MMW radar [11] for more user-centric
features. For example, MMW radar can identify obstacles like vehicles and
pedestrians on the road and notify the driver. Gao et al. have created a large
raw radar dataset for various objects using data from Angle of Arrival,
Doppler Velocity, and other sources under different scenarios [27]. [6]
detects cyclists, pedestrians, and vehicles by converting the position,
velocity, and RCS information into images obtained from the MMW radar. [11]
has utilized range velocity heatmaps for differentiating between vehicles and
pedestrians. Moreover, [13] has proposed mmWave-YOLO for detecting humans,
cones, bicycles, signboards, fences, and riders using radar cross-section
data. Most of the previous studies have focused on detecting cars, people, and
bicycles using MMW radar data. The previous studies have focused on larger
subjects, while our study examines smaller objects typically found indoors or
outdoors.

In this study, the statistical features of radial distance, signal-to-noise
ratio, and the statistical nature of the noise have been investigated. MMW
radar enabling object recognition may be a promising technology for the smart
home industry, which has potential uses in energy optimization, security, and
human-machine interfaces. Therefore, we are interested in learning more about
MMW radar’s particular modality’s ability to recognize objects. From an
implementation point of view, we have considered two scenarios. Firstly,
objects of smaller dimensions such as dimes, lead pencils, plastic sheets,
wood, and quarters have been investigated when the radar is in a static state.
For the dynamic state of MMW radar, we have considered five different objects:
Unnamed Ground Vehicle (UGV), water bottles, plastic sheets, paper, and
clothes. In a static state, we are thinking of finding objects on the floor or
in a green field. While the radar emits signals vertically, it requires
additional time to receive the signals. If it is in motion, it will be
difficult to receive the signal. For this reason, we position the radar
horizontally when it is static. The moving radar emits horizontal signals. At
the state of motion, the radar receives the signal in a vertical orientation.
That’s why we place the radar vertically when the radar is dynamic. According
to the position and status of the MMW radar, we want to measure how well it
performs. When the position and condition of the MMW radar change, different
objects are taken into consideration. Due to the size of the object-to-
wavelength ratio and different environmental ambience, detecting small objects
by MMW radar involves additional technical challenges such as a high signal-
to-noise ratio, obfuscation, etc.

As we are getting time-series MMW radar data, we have applied a 1D
Convolutional Neural Network (CNN) to recognize the objects. Other researchers
have used 2D CNN [6] instead of 1D CNN, but 1D CNN performs satisfactorily due
to the nature of our data. The 1D CNN has achieved 95% accuracy, which is
covered in section V.

##  III Dataset

mmWaves are short-wavelength electromagnetic waves that fall in the frequency
range of 30-300 GHz (1 millimeter to 1 centimeter) [9]. This radar is used for
precisely measuring the distance of an object, its velocity, and its angle
with no or little interference [9]. In this section, we describe the data
collection procedure and data preprocessing of this systematic study.

###  III-A Data Collection

Firstly, we consider MMW radar as static. We leverage an MMW radar that is
integrated with a commodity, low-cost robot tool kit called TurtleBot3
111https://emanual.robotis.com/docs/en/platform/turtlebot3/overview/ as
depicted in Figure 3(a). Secondly, we consider MMW radar as dynamic. Figure
3(b) depicts the vertical positioning of the radar for data collection.
Communication between TurtleBot3 and MMW radar is interfaced with the
TurtleBot3 via the Raspberry Pi 4B. The radar operates with a sampling
frequency of 76–81 GHz and a wavelength of about 4 mm (approximately 0.16
inches). MMW radar provides a 3-axis radial distance, signal-to-noise ratio
(SNR), and noise after blending the transmitted and reflected signals from the
object that lies in the radar projection path.

(a)

(b)

(c)

Figure 3: (a) The red MMW radar is integrated with Turtlebot3, positioned over
a black lead pencil, powered by black cables, and charged using a blue
charger, (b) integrated with Turtlebot3, the red MMW radar emits horizontal
signals to recognize objects, (c) collecting data at a distance of 84 inches
in lab light by moving the radar at a constant velocity.

In this study, the impact of height and lighting conditions on static radar
data collection is examined. Two different heights are considered: placing the
TurtleBot3 on the ground, where the radar is 7 inches vertically apart from
the ground, and placing it at a height where the radar is 53 inches from the
ground. Five different objects ( dime (10 cent US coin), quarter (25 cent US
coin), lead pencil, plastic sheet, and wood) made of four different materials
are used, and data is collected for two minutes at different angles for each
object under three different lighting conditions: sunny, indoor room
environment with light (Lablight), and indoor room environment without light
(Night). Data collection is done outside with natural light for the sunny
condition and inside a lab with white light or without white light for the
other two conditions. Figure 4 and 5 depict the data distribution under
various lighting conditions for static radar. To study the impact of distance
and lighting conditions on moving radar, two different distances are
considered: placing the TurtleBot3 on the ground, where the radar is 42 inches
horizontally apart from the objects, and placing it in a certain position
where the objects are 84 inches away from the radar. Five distinct objects
made (UGV, water bottle, plastic sheet, paper, and clothes) of four different
materials are used, and data is collected for two minutes at different angles
for each object under three different lighting conditions. The radar is moved
back and forth towards the objects at different angles with a uniform velocity
of 0.025 meters per second, which is shown in Figure 3(c). The sample count of
objects is lower when the MMW radar is in a dynamic state, as the receiver
cannot catch all of the signals. We put five different things either below or
in front of the radar, depending on whether the radar sends out signals
vertically or horizontally. Overall, six sets of datasets are needed to
continue this systematic study. Figure 6 and 7 depict the data distribution
under various lighting conditions for moving radar.

(a)  (b)  (c)

Figure 4: Data distribution in sunny (L), lablight (M), and night (R)
environment for height 7 inches (when the radar is static).

(a)  (b)  (c)

Figure 5: Data distribution in sunny (L), lablight (M), and night (R)
environment for height 53 inches (when the radar is static).

(a)  (b)  (c)

Figure 6: Data distribution in sunny (L), lablight (M), and night (R)
environment for distance 42 inches (when the radar is dynamic).

(a)  (b)  (c)

Figure 7: Data distribution in sunny (L), lablight (M), and night (R)
environment for distance 84 inches (when the radar is dynamic).

###  III-B Data Preprocessing

Several statistical features such as standard deviation, variation, mean,
minimum, and maximum are calculated from the MMW radar data, and the extracted
features are further standardized by applying standard scalars from the Sci-
kit python library [28]. The normalized dataset is being split in a stratified
manner into a 70–30% ratio, with 70% of the data used for training and 30% for
testing. This stratification ensures that both training and testing data are
representative of each class. The training data is directly fed to the Fully
Connected Layer (FCL), regardless of whether the radar is in a static or
dynamic state. For recognizing objects with dynamic or static radar at longer
distances or heights, we have applied 1D CNN for better recognition. For
generating the input for 1D CNN, we have combined 10 samples to create a
$40\times 16$ frame. Later, flatten the $40\times 16$ frame to feed into the
1D CNN layer. To create the $40\times 16$ frame, we have first taken samples
from the datasets spanning rows 1 through 40 and then rows 2 through 41, which
is true for the rest of the data samples. Here, 40 rows indicate 4s of data
captured by MMW radar. To ensure consistency, we apply windowing to process
the continuous data obtained from the same objects. Each window consists of 40
rows, which corresponds to a 4-second MMW radar data acquisition. Our analysis
shows that 4 seconds of data are necessary for accurate recognition.

In the next section, we describe the deep models that are used for our
feasibility studies.

##  IV Methodology

To explore the feasibility of using low-resolution and sparse point cloud data
for object recognition, we employ two distinct deep learning frameworks: one
that utilizes a FCL, and the other that is based on a CNN. Next, we have used
the domain adaptation methods [29] to increase the robustness of the CNN-based
deep learning model. We have described the results of FCL and CNN in section
V.

####  IV-1 Training Mechanism of Fully Connected Layer

A $3$-layer fully connected Multi-Layer Perceptron (MLP) as the object
recognizer train the model in a supervised manner which is illustrated in
Figure 8(a). The first two layers consist of $16$ computation units followed
by a Rectified Linear Unit (ReLU), and the final layer is followed by a
Softmax layer. We input the extracted features into the model and train it
using the categorical cross-entropy loss in a supervised fashion as described
in Equation 1.

$L_{cce}=-\sum_{i=1}^{C}t_{i}log(p_{i})$ (1)

where $C$ is the total number of classes (objects), $t_{i}$ is the ground
truth, and $p_{i}$ is the Softmax probability of the object recognizer.

Figure 8: Overall process of object recognition using MMW radar (a) processed
MMW radar’s data is fed into the FCL for object recognition, (b) processed MMW
radar’s data is fed into the CNN, which is used for feature extraction, and a
FCL is used for recognizing the objects.

####  IV-2 Training Mechanism of CNN-based Deep Learning Model

The CNN-based deep learning framework is illustrated in Figure 8(b) and can be
categorized into three distinct modules: the feature extraction module (shown
in green color), the object recognition module (shown in yellow color), and
the domain discriminator module (shown in maroon color). The feature
extraction module consists of three distinct 1D Convolutional (Conv1D) layers
with ReLU activation layers. On the other hand, the object recognizer module
comprises two FC Layers. Finally, the domain adaptation module comprises one
FC Layer along with two outputs in the final layer. We have provided details
of the architecture settings in Table I for easy reference.

TABLE I: Architecture hyper-parameters. Hyper-parameters |  |  | Values |
---|---|---|---|---
Convolution filter no. |  |  | 1x311,1x146,1x64 |
Convolution filter dimension |  |  | 1x20,1x20,1x20 |
Stride |  |  | 2 |
No. of units in object recognizer module |  |  | 64,32,16,5 |
No. of units in domain adaptation module |  |  | 64,32,2 |

The framework has demonstrated promising results in recognizing different
objects, and the domain adaptation mechanism has helped to improve the
adaptability of the deep learning model. For our feasibility study, we have
used the feature extractor and object recognizer modules. We have trained the
network in a supervised manner using a labeled dataset that has been
collected. A batch-wise categorical cross-entropy loss function has been used,
which is described in Equation 1.

Next, we extend the deep learning model’s robustness for object recognition
using domain adversarial methods [29]. We use the same CNN model for
recognizing the objects and add a domain classifier (shown in maroon color in
Figure 8(b)). The domain classifier and feature extractor in our investigation
both use an adversarial strategy. Both modules accept features from datasets
with and without labels. The domain classifier attempts to predict the source
originality of the incoming features, whereas the feature extractor operates
to negate this goal. The target dataset is an open dataset without available
labels, whereas the source dataset has labels. While both modules process
samples of labeled and unlabeled data during adaptation, the object recognizer
only uses the labeled dataset from the source domain. When the model has been
worked in a semi-supervised manner, fewer labels from the target domain have
been used. We use the gradient reversal layer implementation from the work
[29], as indicated in Equations 2, 3 and 4 to accomplish this strategy.

$\theta_{fe}=\theta_{fe}-\mu\left(\frac{\partial
L^{i}_{or}}{\partial\theta_{fe}}-\frac{\partial
L^{i}_{dc}}{\partial\theta_{fe}}\right)\vspace{-1ex}$ (2)

where $\frac{\partial L_{or}}{\partial\theta_{fe}}$ is the gradient of the
loss function object recognizer $L_{or}$ with respect to the feature extractor
$\theta_{fe}$, and $\frac{\partial L_{dc}}{\partial\theta_{fe}}$ is the
gradient of the loss function domain classifier $L_{dc}$ with respect to the
feature extractor $\theta_{fe}$ and $\mu$ is the learning rate.

$\theta_{or}=\theta_{or}-\mu\frac{\partial
L^{i}_{or}}{\partial\theta_{or}}\vspace{-1ex}$ (3)

where $\frac{\partial L_{or}}{\partial\theta_{or}}$ is the gradient of the
loss function $L_{or}$ with respect to the object recognizer $\theta_{or}$.

$\theta_{dc}=\theta_{dc}-\mu\frac{\partial
L^{i}_{dc}}{\partial\theta_{dc}}\vspace{-1ex}$ (4)

where $\frac{\partial L_{dc}}{\partial\theta_{dc}}$ is the gradient of the
loss function $L_{dc}$ with respect to the domain classifier $\theta_{dc}$.

##  V Systematic Study

We report our experimental findings in this section. We have executed the
experiments on an NVIDIA GeForce RTX 3060 graphics card. In different
experiments throughout this systematic study, we consider micro-F1 as the
evaluation metric as the data distribution is imbalanced [30]. Micro-F1 score
is not biased to the majority samples.

###  V-A When MMW radar is static

####  V-A1 Impact of Ambience

In this study, we investigate how different surface materials affect the
reflected radar signal and ultimately the recognition of objects. The radar
projects a signal from a high elevation onto the surface and receives the
reflected signal from various objects in its path. As the surface can be made
up of different materials, the reflectivity of the signal can be altered. We
conduct experiments to recognize objects in three different environments while
maintaining a fixed distance. We collect training and testing samples from the
same environment at a fixed height. The findings are represented in Figure
9(a) and Figure 9(b). For 7 inches and 53 inches of the MMW radar, we have
used the FCL and CNN. The recognition performance of FCL and CNN is
approximately 85%.

#### Key Takeaways

  * •

A sunny environment (outdoor) provides better recognition performance than an
indoor environment (lab environment with light and without light). It’s true
both for FCL and CNN methods. Here, the outdoor environment is made of a
concrete surface, whereas the indoor environment’s surface is an office floor
carpet that is typically made of nylon, olefin, or polypropylene.

  * •

Within an indoor environment, FCL’s performance drops even more when the
height is increased, but CNN can hold the performance due to its complex
feature-learning nature. We hypothesize that as the height increases, the
projected radar signal covers more area compared to the lower height. As the
area coverage increases, the reflected signal is impacted by the surrounding
objects in the lab that are not recognized by FCL. CNN can recover it using
more complex features, which are shown in Figure 9(b).

(a)  (b)

Figure 9: Performance of different deep learning models under two conditions:
same ambiance and cross-ambiance, for similar heights (a) for a vertical
height of 7 inches, (b) for a vertical height of 53 inches.

(a)  (b)

Figure 10: Performance of different deep learning models under two conditions:
same and cross-ambiance for similar distances. (a) For a horizontal distance
of 42 inches, (b) for a horizontal distance of 84 inches.

####  V-A2 Impact of Cross-Ambience

In this experiment, we study the impact of the collected data in different
environments. Similar to the previous study, during training and testing, we
leverage the dataset from the same height but vary in respect of the collected
environment. Consider two datasets of 53 inches in height—one from the sunny
environment and another one from the lablight environment. We train the model
using the sunny environment dataset and test it in the lablight environment.
This study’s experimental findings are summarized in Figure 9(a) and Figure
9(b).

#### Key Takeaways

  * •

When the ambience of the training and testing datasets differs, the
performance drops significantly, and it drops even more when the height is
increased. Figure 9(a) and Figure 9(b) indicate that under cross-ambience
evaluation, height 53 performs even worse than height 7. We have used an
unsupervised domain adaptation (UDA) and semi-supervised domain adaptation
(SSDA) method with 10% and 20% labeled data from the target domain to solve
this problem. Figure 9(a) and Figure 9(b) demonstrate that SSDA significantly
improves performance while UDA improves performance by less amount. The
F1-score rises from 0.443 to 0.822 after utilizing SSDA (shown in Figure 9(a))
with training samples from a sunny environment and testing samples from a
night environment. It is applicable to other cross-ambience situations as
well.

####  V-A3 Impact of Cross-Height

In the cross-ambience study, we find that the performance of the model is
severely affected due to the heterogeneity of the data distribution caused by
different ambiences. In this study, we aim to investigate the performance of
the model when using similar ambience data samples that have been collected
from different heights for training and testing purposes. Table II represents
the experimental findings.

#### Key Takeaways

  * •

Similar to cross-ambience, cross-height also yields sub-optimal results. We
observe a similar pattern for the cross-height and cross-ambience settings
used in model training and testing. Table II demonstrates that SSDA
significantly improves performance while UDA improves performance by less
amount. The F1-score rises from 0.238 to 0.834 (SSDA 10%) and 0.878 (SSDA 20%)
(shown in Table II) with training and testing samples from the sunny
environment but at different heights. It is applicable to other cases also.

TABLE II: Performance of different deep learning models under the same
ambience, for different heights.

| Training
---
environment
| Testing
---
environment
| F1-score (FCL)
---
| F1-score (CNN)
---
| F1-score
---
(UDA)
| F1-score
---
(SSDA 10%)
| F1-score
---
(SSDA 20%)
| Sunny(53)
---
| Sunny(7)
---
| 0.158
---
| 0.238
---
| 0.312
---
| 0.834
---
| 0.878
---
| Night(53)
---
| Night(7)
---
| 0.346
---
| 0.217
---
| 0.250
---
| 0.531
---
| 0.799
---
| Lablight(53)
---
| Lablight(7)
---
| 0.262
---
| 0.329
---
| 0.371
---
| 0.495
---
| 0.587
---
| Sunny(7)
---
| Sunny(53)
---
| 0.107
---
| 0.193
---
| 0.342
---
| 0.645
---
| 0.909
---
| Night(7)
---
| Night(53)
---
| 0.198
---
| 0.296
---
| 0.410
---
| 0.537
---
| 0.588
---
| Lablight(7)
---
| Lablight(53)
---
| 0.276
---
| 0.301
---
| 0.380
---
| 0.437
---
| 0.488
---
Sunny(53) indicates taking the samples from a sunny environment where the
distance is 53 inches.

###  V-B When MMW radar is dynamic

####  V-B1 Impact of Ambience

We consider recognizing objects in three different environments while keeping
the distance fixed. We consider taking training and testing samples from the
same environment at a fixed distance. The findings are represented in Figure
10(a) and Figure 10(b). For 42 inches and 84 inches of the MMW radar, the
recognition performance of the FCL and CNN is above 92% when the radar is
static or dynamic.

#### Key Takeaways

  * •

A lablight environment (indoor) provides better recognition performance than
an outdoor environment (lab environment with light and without light) when we
use a FCL for a distance of 42 inches. When we apply CNN, the F1-scores are
close in all environments. It is expected, as CNN is more powerful than FCL.

  * •

When using FCL for a distance of 84 inches, the F1-score is unsatisfactory.
However, the F1-score is better in sunny environments compared to other
conditions. By using CNN for a distance of 84 inches, complex features can be
learned and the F1-score improves from 0.754 to 0.873 in a lablight
environment, as shown in Figure 10(b). This improvement applies to other cases
because radar struggles to capture sufficient information in dynamic states.

####  V-B2 Impact of Cross-Ambience

Here, we test how data from various settings might be used to draw
conclusions. This study uses the same dataset and distance for both training
and testing as the previous one, but differs in that the surroundings are
significantly changed. Take into account two 42-inch data sets, one from a
sunny setting and the other from a lablight setting. Data that has been
collected in a sunny environment is used to train the model, which is then put
to the test under artificial lighting. Figure 10(a) and Figure 10(b) are
showing the experimental findings.

#### Key Takeaways

  * •

Significant performance losses occur when the environments of the training and
testing datasets are different. Figure 10(a) and Figure 10(b) show the
performance of the deep learning models. It’s to be anticipated since the
distribution of data varies depending on the environment, although the
distance is the same. We have used UDA and SSDA methods with 10% and 20%
labeled data from the target domain to solve this problem. Figure 10(a) and
Figure 10(b) illustrate that SSDA leads to a substantial improvement in
performance, whereas UDA results in a comparatively smaller improvement. The
F1-score rises from 0.691 to 0.898 (shown in Figure 10(a)) after utilizing
SSDA with training samples from a night environment and testing samples from a
lablight environment. It is applicable to other cross-ambience situations as
well.

####  V-B3 Impact of Cross-Distance

We have observed a significant decrease in performance due to the existence of
data distribution heterogeneity caused by different environments. Here, we
investigate what happens to the model’s F1-score when training and testing
data samples have the same ambient conditions but are collected from different
distances. Table III represents the experimental findings.

#### Key Takeaways

  * •

Similarly to cross-ambience, cross-distance leads to suboptimal outcomes. We
see a similar tendency for cross-distance and cross-ambience training and
testing conditions. When we apply UDA and SSDA, the performance ameliorates.
The F1-score rises from 0.521 to 0.768 (shown in Table III) for the night
environment. Here, training samples are taken from 84 inches, but testing
samples are taken from 42 inches. Now, in the following section, we describe
how far our future research plans have progressed.

TABLE III: Performance of different deep learning models under cross distances
for the same ambience settings.

| Training
---
environment
| Testing
---
environment
| F1-score (FCL)
---
| F1-score (CNN)
---
| F1-score
---
(UDA)
| F1-score
---
(SSDA 10%)
| F1-score
---
(SSDA 20%)
| Sunny(84)
---
| Sunny(42)
---
| 0.492
---
| 0.514
---
| 0.494
---
| 0.542
---
| 0.670
---
| Night(84)
---
| Night(42)
---
| 0.571
---
| 0.521
---
| 0.583
---
| 0.702
---
| 0.768
---
| Lablight(84)
---
| Lablight(42)
---
| 0.460
---
| 0.491
---
| 0.512
---
| 0.603
---
| 0.644
---
| Sunny(42)
---
| Sunny(84)
---
| 0.405
---
| 0.457
---
| 0.461
---
| 0.500
---
| 0.570
---
| Night(42)
---
| Night(84)
---
| 0.466
---
| 0.526
---
| 0.543
---
| 0.563
---
| 0.631
---
| Lablight(42)
---
| Lablight(84)
---
| 0.411
---
| 0.422
---
| 0.445
---
| 0.518
---
| 0.532
---
Sunny(42) indicates taking the samples from a sunny environment where the
distance is 42 inches.

##  VI Discussion

We address the issues of developing and implementing a deep learning model
that utilizes MMW radar. These include potential noise in the dataset,
challenges in real-time implementation with point cloud radar data, and the
system’s potential impact on daily life.

  * •

During the data collection process, surrounding objects can potentially
introduce noise into the dataset. Although such objects may be considered
noise, we believe that our model is robust enough to recognize actual objects
in the presence of noise. However, in the future, we may be able to improve
the performance of our model by reducing the noise introduced by these
surrounding objects.

  * •

Integrating the object recognition pipeline into a real-time system can be
straightforward, but there are various implementation challenges to consider,
such as power consumption and noise detection from surrounding objects. These
challenges present concrete obstacles to achieving real-time implementation.

  * •

Our deep learning methodology has exclusively utilized the processed sparse
point cloud data generated from the MMW radar. If we incorporate divergent
types of MMW radar data, it could lead to processing and computational
difficulties, potentially requiring significant alterations to the hyper-
parameters of our model.

  * •

The potential use of MMW radar technology to recognize both familiar and
unfamiliar objects, activities, and health issues concurrently may have a
profound influence on daily life. Given the limitations on camera placement
due to privacy concerns, radar may serve as an excellent substitute for
identifying criminal or other daily activities.

We are nearing the conclusion of our investigations, where we summarize our
current constraints and future research scopes.

##  VII Conclusion

Our study has shown promising results for static object recognition using MMW
radar. We have evaluated the performance of our proposed model under various
conditions and found that it performs well on data with a homogenous ambiance
but poorly on data with a heterogeneous ambiance. To address this issue, we
have introduced domain adaptation techniques to improve accuracy. However, our
current model only focuses on recognizing static objects, and to address real-
world applications, we need to develop more advanced techniques for
recognizing known or unknown objects, activities, and health issues
simultaneously. This technology has diverse practical applications, especially
in areas where privacy issues constrain the deployment of cameras, such as
security, surveillance, and monitoring.

## ACKNOWLEDGEMENT

This research is supported by the U.S. Army Grant #W911NF2120076, and NSF
Research Experience for Undergraduates (REU) grant #CNS-2050999. We want to
thank Dr. Kelly Sherbondy for his collaboration on this project.

## References

  * [1] Barry Brumitt, Brian Meyers, John Krumm, Amanda Kern, and Steven Shafer.  Easyliving: Technologies for intelligent environments.  In Handheld and Ubiquitous Computing: Second International Symposium, HUC 2000 Bristol, UK, September 25–27, 2000 Proceedings 2, pages 12–29. Springer, 2000.
  * [2] Emon Dey, Jumman Hossain, Nirmalya Roy, and Carl Busart.  Synchrosim: An integrated co-simulation middleware for heterogeneous multi-robot system.  In 2022 18th International Conference on Distributed Computing in Sensor Systems (DCOSS), pages 334–341. IEEE, 2022.
  * [3] Mostafa Alizadeh, George Shaker, João Carlos Martins De Almeida, Plinio Pelegrini Morita, and Safeddin Safavi-Naeini.  Remote monitoring of human vital signs using mm-wave fmcw radar.  IEEE Access, 7:54958–54968, 2019.
  * [4] Yudi Dong and Yu-Dong Yao.  Secure mmwave-radar-based speaker verification for iot smart home.  IEEE Internet of Things Journal, 8(5):3500–3511, 2021.
  * [5] Yizhou Wang, Zhongyu Jiang, Yudong Li, Jenq-Neng Hwang, Guanbin Xing, and Hui Liu.  Rodnet: A real-time radar object detection network cross-supervised by camera-radar fused object 3d localization.  IEEE Journal of Selected Topics in Signal Processing, 15(4):954–967, 2021.
  * [6] Ruidong Xu, Yuan Zhu, Ke Lu, and Hao An.  Object detection of millimeter wave radar based on computer vision.  In 2022 4th International Conference on Advances in Computer Technology, Information Science and Communications (CTISC), pages 1–8. IEEE, 2022.
  * [7] Peijun Zhao, Chris Xiaoxuan Lu, Jianan Wang, Changhao Chen, Wei Wang, Niki Trigoni, and Andrew Markham.  mid: Tracking and identifying people with millimeter wave radar.  In 2019 15th International Conference on Distributed Computing in Sensor Systems (DCOSS), pages 33–40. IEEE, 2019.
  * [8] Robin Amsters and Peter Slaets.  Turtlebot 3 as a robotics education platform.  In Robotics in Education: Current Research and Innovations 10, pages 170–181. Springer, 2020.
  * [9] Cesar Iovescu and Sandeep Rao.  The fundamentals of millimeter wave radar sensors.  Texas Instruments, 2020.
  * [10] David R Green, Jason J Hagon, Cristina Gómez, and Billy J Gregory.  Using low-cost uavs for environmental monitoring, mapping, and modelling: Examples from the coastal zone.  In Coastal management, pages 465–501. Elsevier, 2019.
  * [11] Onur Toker and Suleiman Alsweiss.  mmwave radar based approach for pedestrian identification in autonomous vehicles.  In 2020 SoutheastCon, pages 1–2. IEEE, 2020.
  * [12] Riadul Islam, Maloy K Devnath, Manar D Samad, and Syed Md Jaffrey Al Kadry.  Ggnb: Graph-based gaussian naive bayes intrusion detection system for can bus.  Vehicular Communications, 33:100442, 2022.
  * [13] Atsutake Kosuge, Satoshi Suehiro, Mototsugu Hamada, and Tadahiro Kuroda.  mmwave-yolo: A mmwave imaging radar-based real-time multiclass object recognition system for adas applications.  IEEE Transactions on Instrumentation and Measurement, 71:1–10, 2022\.
  * [14] Maloy Kumar Devnath, Avijoy Chakma, Mohammad Saeid Anwar, Emon Dey, Zahid Hasan, Marc Conn, Biplab Pal, and Nirmalya Roy.  Millimeter-wave object recognition dataset (mord), 2023.
  * [15] Renyuan Zhang and Siyang Cao.  Real-time human motion behavior detection via cnn using mmwave radar.  IEEE Sensors Letters, 3(2):1–4, 2018.
  * [16] Peijun Zhao, Chris Xiaoxuan Lu, Bing Wang, Changhao Chen, Linhai Xie, Mengyu Wang, Niki Trigoni, and Andrew Markham.  Heart rate sensing with a robot mounted mmwave radar.  In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 2812–2818. IEEE, 2020.
  * [17] Chris Xiaoxuan Lu, Stefano Rosa, Peijun Zhao, Bing Wang, Changhao Chen, John A Stankovic, Niki Trigoni, and Andrew Markham.  See through smoke: robust indoor mapping with low-cost mmwave radar.  In Proceedings of the 18th International Conference on Mobile Systems, Applications, and Services, pages 14–27, 2020.
  * [18] Arindam Sengupta, Feng Jin, Renyuan Zhang, and Siyang Cao.  mm-pose: Real-time human skeletal posture estimation using mmwave radars and cnns.  IEEE Sensors Journal, 20(17):10032–10044, 2020.
  * [19] Guangzheng Li, Ze Zhang, Hanmei Yang, Jin Pan, Dayin Chen, and Jin Zhang.  Capturing human pose using mmwave radar.  In 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pages 1–6, 2020.
  * [20] Akash Deep Singh, Sandeep Singh Sandha, Luis Garcia, and Mani Srivastava.  Radhar: Human activity recognition from point clouds generated through a millimeter-wave radar.  In Proceedings of the 3rd ACM Workshop on Millimeter-wave Networks and Sensing Systems, pages 51–56, 2019.
  * [21] Zihao Zhao, Yuying Song, Fucheng Cui, Jiang Zhu, Chunyi Song, Zhiwei Xu, and Kai Ding.  Point cloud features-based kernel svm for human-vehicle classification in millimeter wave radar.  IEEE Access, 8:26012–26021, 2020.
  * [22] Muralidhar Reddy Challa, Abhinav Kumar, and Linga Reddy Cenkeramaddi.  Face recognition using mmwave radar imaging.  In 2021 IEEE International Symposium on Smart Electronic Systems (iSES)(Formerly iNiS), pages 319–322. IEEE, 2021.
  * [23] Xiaochao Dang, Zetong Chen, and Zhanjun Hao.  Emotion recognition method using millimetre wave radar based on deep learning.  IET Radar, Sonar & Navigation, 2022.
  * [24] Shuo Chang, Yifan Zhang, Fan Zhang, Xiaotong Zhao, Sai Huang, Zhiyong Feng, and Zhiqing Wei.  Spatial attention fusion for obstacle detection using mmwave radar and vision sensor.  Sensors, 20(4):956, 2020.
  * [25] Renyuan Zhang and Siyang Cao.  Extending reliability of mmwave radar tracking and detection via fusion with camera.  IEEE Access, 7:137065–137079, 2019.
  * [26] Xian Shuai, Yulin Shen, Yi Tang, Shuyao Shi, Luping Ji, and Guoliang Xing.  millieye: A lightweight mmwave radar and camera fusion system for robust object detection.  In Proceedings of the International Conference on Internet-of-Things Design and Implementation, pages 145–157, 2021.
  * [27] Xiangyu Gao, Guanbin Xing, Sumit Roy, and Hui Liu.  Experiments with mmwave automotive radar test-bed.  In 2019 53rd Asilomar Conference on Signals, Systems, and Computers, pages 1–6, 2019.
  * [28] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay.  Scikit-learn: Machine learning in Python.  Journal of Machine Learning Research, 12:2825–2830, 2011.
  * [29] Yaroslav Ganin and Victor Lempitsky.  Unsupervised domain adaptation by backpropagation.  In International conference on machine learning, pages 1180–1189. PMLR, 2015.
  * [30] Rebeen Ali Hamad, Masashi Kimura, and Jens Lundström.  Efficacy of imbalanced data handling methods on deep learning for smart homes environments.  SN Computer Science, 1(4):204, 2020.