# Deeply Virtual Compton Scattering at HERA and perspectives at CERN

Laurent Schoeffel

###### Abstract

Standard parton distribution functions contain neither information on the
correlations between partons nor on their transverse motion, then a vital
knowledge about the three dimensional structure of the nucleon is lost. Hard
exclusive processes, in particular DVCS, are essential reactions to go beyond
this standard picture. In the following, we examine the most recent data in
view of the dipole model predictions and their implication on the
quarks/gluons imaging (tomography) of the nucleon.

###### Keywords:

deeply virtual scattering, dipole model, parton distributions, proton
tomography

###### :

11.25.Db, 12.38.Bx, 13.60.Fz

##  1 Introduction

Measurements of the deep-inelastic scattering (DIS) of leptons and nucleons,
$e+p\to e+X$, allow the extraction of Parton Distribution Functions (PDFs)
which describe the longitudinal momentum carried by the quarks, anti-quarks
and gluons that make up the fast-moving nucleons. These functions have been
measured over a wide kinematic range in the Bjorken scaling variable $x_{Bj}$
and the photon virtuality $Q^{2}$. While PDFs provide crucial input to
perturbative Quantum Chromodynamic (QCD) calculations of processes involving
hadrons, they do not provide a complete picture of the partonic structure of
nucleons. In particular, PDFs contain neither information on the correlations
between partons nor on their transverse motion, then a vital knowledge about
the three dimensional structure of the nucleon is lost. Hard exclusive
processes, in which the nucleon remains intact, have emerged in recent years
as prime candidates to complement this essentially one dimentional picture
lolopic .

Indeed, within the investigation of hard exclusive reactions in the Bjorken
limit, the probe provided by the photon works as a clean tool reactions in
order to extract reliable knowledge on the substructure of strongly
interacting particles complementar to exclusive process.

The simplest exclusive process is the deeply virtual Compton scattering (DVCS)
or exclusive production of real photon, $e+p\rightarrow e+\gamma+p$. This
process is of particular interest as it has both a clear experimental
signature and is calculable in perturbative QCD. The DVCS reaction can be
regarded as the elastic scattering of the virtual photon off the proton via a
colourless exchange, producing a real photon in the final state lolopic . The
recent measurements from the DESY $ep$ collider HERA at low $x_{Bj}$
($x_{Bj}<0.01$) and large $Q^{2}$ (above 2 GeV2) are thus a decisive
experimental step forward dvcsh1 ; dvcszeus . In the Bjorken scaling regime,
QCD calculations assume that the exchange involves two partons, having
different longitudinal and transverse momenta, in a colourless configuration.
These unequal momenta or skewing are a consequence of the mass difference
between the incoming virtual photon and the outgoing real photon. This
skewedness effect can be interpreted in the context of generalised parton
distributions (GPDs) diehl ; freund2 ; buk  or in the dipole model approach
robi ; marquet .

A considerable interest of the high energy situation at HERA is that it gives
the important opportunity to constrain the gluon contribution to GPDs and to
study the evolution in virtuality $Q^{2}$ of the quark and gluon
distributions. On the other hand, recently the color dipole formalism has
provided a simultaneous description of photon induced process. The inclusive
deep inelastic reaction and the photon diffractive dissociation has been
successfully described and the study of other exclusive processes such as DVCS
is an important test of the color dipole approach.

In the following, we examine the most recent data in view of the dipole model
predictions robi ; marquet  and their implication on the quarks/gluons imaging
of the nucleon dvcsh1 ; lolotom ; pire .

##  2 The color dipole model

The colour dipole model provides a simple unified picture of inclusive and
diffractive processes and enables hard and soft physics to be incorporated in
a single dynamical framework. At high energies, in the proton’s rest frame,
the virtual photon fluctuates into a hadronic system (the simplest of which is
a $q{\bar{q}}$ dipole) a long distance upstream of the target proton. The
formation time of this hadronic system, and of the subsequent formation of the
hadronic final state, is much longer than the interaction time with the
target.

DVCS is a good probe of the transition between soft and hard regimes in the
dipole model for two reasons. Indeed, the transverse photon wave function can
select large dipoles, even for large $Q^{2}$, and certainly for the $Q^{2}$
range $2<Q^{2}<20$ GeV2. Also, because the final photon is real, DVCS is more
sensitive to large dipoles than DIS at the same $Q^{2}$.

Then, in the colour dipole approach, the DIS (or DVCS) process can be seen as
a succession in time of three factorisable subprocesses: i) the virtual photon
fluctuates in a quark-antiquark pair, ii) this colour dipole interacts with
the proton target, iii) the quark pair annihilates into a virtual (or real)
photon. The imaginary part of the DIS (or DVCS) amplitude at $t=0$ is
expressed in the simple way robi ; marquet

$\displaystyle{\cal I}m\,{\cal
A}\,(W,Q_{1},Q_{2})=\sum_{T,L}\int\limits_{0}^{1}dz\\!\int\limits_{0}^{\infty}d^{2}\mbox{\boldmath$r$}\,\Psi_{T,L}^{*}(z,\,\mbox{\boldmath$r$},\,Q_{1}^{2})\,\sigma_{dip}\,(z,\mbox{\boldmath$r$})\Psi_{T,L}(z,\,\mbox{\boldmath$r$},\,Q_{2}^{2})\,,$
(1)

where $\Psi(z,\mbox{\boldmath$r$},Q_{1,2})$ are the light cone photon wave
functions for transverse and longitudinal photons. The quantity $Q_{1}$ is the
virtuality of the incoming photon, whereas $Q_{2}$ is the virtuality of the
outgoing photon. In the DIS case, one has $Q_{1}^{2}=Q_{2}^{2}=Q^{2}$ and for
DVCS, $Q_{1}^{2}=Q^{2}$ and $Q_{2}^{2}=0$. The relative transverse quark pair
(dipole) separation is labeled by $r$ whilst $z$ (respec. $1-z$) labels the
quark (antiquark) longitudinal momentum fraction.

It should be noticed that the non-forward kinematics for DVCS is encoded in
the colour dipole approach through the different weight coming from the photon
wavefunctions in Eq. (1). The off-diagonal effects, which affect the gluon and
quark distributions in GPDs models, should be included in the parameterisation
of the dipole cross section. At the present stage of the development of the
dipole formalism, we have no accurate theoretical arguments on how to compute
skewedness effects from first principles. A consistent approach would be to
compute the scattering amplitude in the non-forward case, since the non-
forward photon wave function has been recently obtained. In this case, the
dipole cross section,
$\sigma_{dip}\,(x_{1},x_{2},\mbox{\boldmath$r$},\vec{\Delta})$, depends on the
momenta $x_{1}$ and $x_{2}$ carried by the exchanged gluons, respectively, and
on the total transverse momentum transfer $\vec{\Delta}$. In this case,
additional information about the dependence upon $\vec{\Delta}$ is needed for
the QCD Pomeron and proton impact factor. A first attempt in this direction is
done in robi .

##  3 Geometric Scaling

At very small values of the Bjorken scaling variable $x$ the saturation regime
of QCD can be reached. In this domain, the gluon density in the proton is so
large that non-linear effects like gluon recombination tame its growth. In the
dipole model approach, the transition to the saturation regime is
characterised by the so-called saturation scale parametrised here as
$Q_{s}(x)=Q_{0}({{x_{0}}/{x}})^{-\lambda/2}$, where $Q_{0}$, $x_{0}$ and
$\lambda$ are parameters. The transition to saturation occurs when $Q$ becomes
comparable to $Q_{s}(x)$. An important feature of dipole models that
incorporate saturation is that the total cross section can be expressed as a
function of the single variable $\tau$:

$\sigma_{tot}^{\gamma^{\ast}p}(x,Q^{2})=\sigma_{tot}^{\gamma^{\ast}p}(\tau),\;\;\mbox{with}\;\
\ \ \tau=\frac{Q^{2}}{Q_{s}^{2}(x)}.$ (2)

This property, called geometric scaling, has already been observed to hold for
the total $ep$ DIS cross section golec  and in diffractive processes marquet .
It has also recently been addressed in the context of exclusive processes
including DVCS and extended to cases with non-zero momentum transfer to the
proton robi . It is therefore interesting to test if the present DVCS
measurements obey the geometric scaling laws predicted by such models.

##  4 Latest News from the experimental Front

Measurements of DVCS cross section have been realised at HERA within the H1
and ZEUS experiments dvcsh1 ; dvcszeus . As mentioned in the introduction,
these results are given in the specific kinematic domain of both experiments,
at low $x_{Bj}$ ($x_{Bj}<0.01$) but they take advantage of the large range in
$Q^{2}$, offered by the HERA kinematics, which covers more than 2 orders of
magnitude. It makes possible to study the transition from the low $Q^{2}$ non-
perturbative region (around $1$ GeV2) towards higher values of $Q^{2}$ where
the higher twists effects are lowered (above $10$ GeV2). The last data on DVCS
cross section as a function of $W\simeq\sqrt{Q^{2}/x}$ are presented on figure
1. They show a strong $W$ dependence with $\sigma_{DVCS}\propto W^{0.7}$,
characteristic of a hard process, which can thus be described in perturbative
QCD as described in previous sections.

Data and model comparisons are presented on figures 2 and 3. They show that
the dipole approach is very efficient in describing the DVCS measurements for
the HERA kinematics. It must be noticed here that the non-forward kinematics
for DVCS is encoded only through the different weights coming from the photon
wavefunctions in Eq. (1). It means that without taking into account non-
diagonal effects that drive all the physics of GPDs, we can get a good
descritption of all existing low $x_{Bj}$ data in the framework of the dipole
model.

Beside $W$ and $Q^{2}$ DVCS cross sections, a major experimental achievement
of H1 dvcsh1  has been the measurement of DVCS cross sections, differential in
$t=(p^{\prime}-p)^{2}$, the momentum transfer (squared) at the proton vertex.
Some results are presente on figure 4: we observe the good description of
$d\sigma_{DVCS}/dt$ by a fit of the form $e^{-b|t|}$. Hence, an extraction of
the $t$-slope parameter $b$ is accessible for different values of $Q^{2}$ and
$W$ (see figures 4 and 5). We exploit these values of $b$ in a section below.

Figure 1: DVCS cross section for positrons/electrons samples as a function of
$W$ for $Q^{2}=8$ GeV2.

Figure 2:  DVCS cross section measurements as a function of the scaling
variable $\tau={Q^{2}}/{Q_{s}^{2}(x)}$. Results are shown for the full $t$
range $|t|<$ 1 GeV2 (left) and at four values of $t$ (right). The dashed
curves represent the predictions of the dipole model marquet .

Figure 3: Predictions for the DVCS cross section robi  with H1 data. Figure 4:
DVCS cross section, differential in $t$ presented with a fit of the form
$e^{-b|t|}$.

Figure 5: The logarithmic slope of the $t$ dependence for DVCS exclusive
production, $b$ as a function of $Q^{2}$ and $W$, extracted from a fit
$d\sigma/dt\propto\exp(-b|t|)$ where $t=(p-p^{\prime})^{2}$.

##  5 Quantifying skewing effects

Let’s define a basic quantity giving an overall measurement of the skewing
properties lolocompass ; lolocompass2 , which includes both the non-forward
kinematics and the possible non-diagonal effects. Namely, we set the ratio
between the imaginary parts of the DIS and DVCS (forward) scattering
amplitudes at zero momentum transfer:

$R\equiv\frac{Im\,{\cal A}\,(\gamma^{*}+p\to\gamma^{*}+p)|_{t=0}}{Im\,{\cal
A}\,(\gamma^{*}+p\to\gamma+p)|_{t=0}}\ ,$ (3)

where $t$ is the square of the four-momentum exchanged at the proton vertex.

In lolocompass ; lolocompass2 , it has been shown how to extract this quantity
from the DVCS cross sections. The factor $R$ is presented as a function of
energy $W$ in figure 6. An almost flat $W$ dependence is observed within the
present precision. This feature can be easily understood since the $W$
dependence of both the DIS and DVCS cross section is power-like having a
proportional effective power. Namely, $\sigma_{\mathrm{DIS}}\propto
W^{2\lambda}$ and $\sigma_{\mathrm{DVCS}}\propto W^{4\lambda}$. The mean value
$R\simeq 0.5$ is consistent with its early theoretical estimates using the
aligned jet model and the colour dipole model (see figure 6). We also display
a dipole model prediction with an ’ad hoc’ parametrisation for the off-
diagonal effects. It does not improve the quality of the description of data
within the present uncertainties.

Figure 6: The skewing factor $R$ as a function of $W$ at $Q^{2}=8$ GeV2. The
curves represent the theoretical predictions of the dipole approach (see
text).

##  6 Nucleon Tomography

Measurements of the $t$-slope parameters $b$ are key measurements for almost
all exclusive processes, in particular DVCS. Indeed, a Fourier transform from
momentum to impact parameter space readily shows that the $t$-slope $b$ is
related to the typical transverse distance between the colliding objects buk .
At high scale, the $q\bar{q}$ dipole is almost point-like, and the $t$
dependence of the cross section is given by the transverse extension of the
gluons (or sea quarks) in the proton for a given $x_{Bj}$ range. More
precisely, from the generalised parton distribution defined in the
introduction, we can compute a parton density which also depends on a spatial
degree of freedom, the transverse size (or impact parameter), labeled
$R_{\perp}$, in the proton. Both functions are related by a Fourier transform

$PDF(x,R_{\perp};Q^{2})\;\;\equiv\;\;\int\frac{d^{2}\Delta_{\perp}}{(2\pi)^{2}}\;e^{i({\Delta}_{\perp}{R_{\perp}})}\;GPD(x,t=-{\Delta}_{\perp}^{2};Q^{2}).$

Thus, the transverse extension $\langle r_{T}^{2}\rangle$ of gluons (or sea
quarks) in the proton can be written as

$\langle r_{T}^{2}\rangle\;\;\equiv\;\;\frac{\int
d^{2}R_{\perp}\;PDF(x,R_{\perp})\;R_{\perp}^{2}}{\int
d^{2}R_{\perp}\;PDF(x,R_{\perp})}\;\;=\;\;4\;\frac{\partial}{\partial
t}\left[\frac{GPD(x,t)}{GPD(x,0)}\right]_{t=0}=2b$

where $b$ is the exponential $t$-slope. Measurements of $b$ presented in
figure 5 corresponds to $\sqrt{r_{T}^{2}}=0.65\pm 0.02$ fm at large scale
$Q^{2}$ for $x_{Bj}<10^{-2}$. This value is smaller that the size of a single
proton, and, in contrast to hadron-hadron scattering, it does not expand as
energy $W$ increases. This result is consistent with perturbative QCD
calculations in terms of a radiation cloud of gluons and quarks emitted around
the incoming virtual photon. In short, gluons are located at the preiphery of
the proton as measured here and valence quarks are assumed to form the core of
the proton at small value of $\sqrt{r_{T}^{2}}$.

In other words, the Fourier transform of the DVCS amplitude is the amplitude
to find quarks at $R_{\perp}$ in an image plane after focusing by an idealized
lens. The square of the profile amplitude, producing the PDF (in transverse
plane) is positive, real-valued, and corresponds to the image, a weighted
probability to find quarks in the transverse image plane.

##  7 Perspectives at CERN

The complete parton imaging in the nucleon would need to get measurements of
$b$ for several values of $x_{Bj}$, from the low $x_{Bj}<0.01$ till
$x_{Bj}>0.1$. Experimentally, it appears to be impossible. Is it the breakout
of quark and gluon imaging in the proton? In fact, there is one way to recover
$x_{Bj}$ and $t$ correlations over the whole $x_{Bj}$ domain: we need to
measure a Beam Charge Asymmetry (BCA) lolopic ; lolocompass ; lolobca .

A determination of a cross section asymmetry with respect to the beam charge
has been realised by the H1 experiment by measuring the ratio
$(d\sigma^{+}-d\sigma^{-})/(d\sigma^{+}+d\sigma^{-})$ as a function of $\phi$,
where $\phi$ is the azimuthal angle between leptons and proton plane lolopic ;
freund2 . The result is presented on figure 7 with a fit in $\cos\phi$. After
applying a deconvolution method to account for the resolution on $\phi$, the
coefficient of the $\cos\phi$ dependence is found to be $p_{1}=0.17\pm
0.03(stat.)\pm 0.05(sys.)$. This result represents obviously a major progress
in the understanding of the very recent field of the parton imaging in the
proton. We are at the hedge of the giving a new reading on the most
fundamental question to know how the proton is built up by quarks and gluons.

Feasabilities for future BCA measurements at COMPASS have been studied
extensively in the last decade dhose . COMPASS is a fixed target experiment
which can use 100 GeV muon beams and hydrogen targets, and then access
experimentaly the DVCS process $\mu p\rightarrow\mu\gamma p$. The BCA can be
determined when using positive and negative muon beams. One major interest is
the kinematic coverage from $2$ GeV2 till $6$ GeV2 in $Q^{2}$ and $x_{Bj}$
ranging from $0.05$ till $0.1$. It means that it is possible to avoid the
kinematic domain dominated by higher-twists and non-perturbative effects (for
$Q^{2}<1$ GeV2) and keeping a $x_{Bj}$ range which is extending the HERA
(H1/ZEUS) domain.

Figure 7: Beam charge asymmetry as a function of $\phi$ measured by H1 lolopic
; lolobca .

##  8 Summary

DVCS measurements in the HERA kinematics at low $x_{Bj}$ are well described
within a dipole approach, which encodes the non-forward kinematics for DVCS
only through the different weights coming from the photon wavefunctions. Note
that the dipole model presents the great advantage to define a unique
framework that gives a good description of all hard processes accessible at
HERA. For the first time, we have also shown that proton tomography enters
into the experimental domain of high energy physics, with a first experimental
evidence that gluons are located at the periphery of the proton. A new
frontier in understanding this structure would be possible at CERN within the
COMPASS experimental setup.

## References

  * (1) L. Schoeffel, proceedings of the 27th International Conference on Physics in Collision (PIC 2007), Annecy, France, 26-29 Jun 2007. arXiv:0707.3706 [hep-ph].
  * (2) F. D. Aaron et al. [H1 Collaboration], Phys. Lett. B 659 (2008) 796 [arXiv:0709.4114 [hep-ex]] ; A. Aktas et al. [H1 Collaboration], Eur. Phys. J. C 44, 1 (2005) [hep-ex/0505061] ; C. Adloff et al. [H1 Collaboration], Phys. Lett. B 517 (2001) 47 [arXiv:hep-ex/0107005].
  * (3) S. Chekanov et al. [ZEUS Collaboration], Phys. Lett. B 573, 46 (2003) [hep-ex/0305028].
  * (4) M. Diehl, AIP Conf. Proc. 842 (2006) 294 [arXiv:hep-ph/0512201].
  * (5) A. Freund, Phys. Rev. D 68 (2003) 096006 [hep-ph/0306012].
  * (6) M. Burkardt, Int. J. Mod. Phys. A 18 (2003) 173 [hep-ph/0207047].
  * (7) C. Marquet, R. B. Peschanski and G. Soyez, Phys. Rev. D 76 (2007) 034011 [arXiv:hep-ph/0702171].
  * (8) C. Marquet and L. Schoeffel, Phys. Lett. B 639 (2006) 471 [arXiv:hep-ph/0606079].
  * (9) L. Schoeffel, proceedings of the 16th International Workshop on Deep Inelastic Scattering and Related Subjects (DIS 2008), London, England, 7-11 Apr 2008. arXiv:0805.2672 [hep-ph].
  * (10) J. P. Ralston and B. Pire, Phys. Rev. D 66 (2002) 111501 [arXiv:hep-ph/0110075].
  * (11) K. J. Golec-Biernat and M. Wusthoff, Phys. Rev. D 60 (1999) 114023 [arXiv:hep-ph/9903358].
  * (12) L. Schoeffel, Phys. Lett. B 658 (2007) 33 [arXiv:0706.3488 [hep-ph]].
  * (13) L. Favart, M. V. T. Machado and L. Schoeffel, Braz. J. Phys. 37 (2007) 798.
  * (14) L. Schoeffel, proceedings of the 15th International Workshop on Deep-Inelastic Scattering and Related Subjects (DIS2007), Munich, Germany, 16-20 Apr 2007. arXiv:0705.2925 [hep-ph].
  * (15) N. d’Hose, E. Burtin, P. A. M. Guichon, S. Kerhoas-Cavata, J. Marroncle and L. Mosse, Nucl. Phys. A 711 (2002) 160.


IRS

    intelligent reflecting surface
URA

    uniform rectangular array
BS

    base station
UE

    user-equipment
KF

    Kronecker factorization
LoS

    line-of-sight
SVD

    singular value decomposition
MIMO

    multiple input multiple output
SE

    spectral efficiency
SNR

    signal-to-noise ratio
B5G

    beyond fifth generation
EE

    energy efficiency
mmWave

    millimeter wave
MMSE

    minimum mean squared error
MISO

    multiple input single output
CSI

    channel state information
AoD

    azimuth of departure
EoD

    elevation of departure
AoA

    azimuth of arrival
EoA

    elevation of arrival
HOSVD

    high order singular value decomposition
TOT

    third-order tensor
AWGN

    additive white Gaussian noise
LSKRF

    least square Khatri-Rao factorization

# Low-Complexity Joint Active and Passive Beamforming Design for IRS-Assisted
MIMO

Yuri S. Ribeiro, Fazal-E-Asim, , André L. F. de Almeida, , Behrooz Makki, ,
and Gabor Fodor This work was supported by the Ericsson Research, Sweden, and
Ericsson Innovation Center, Brazil, under UFC.51 Technical Cooperation
Contract Ericsson/UFC. This study was financed in part by Funcap edital No.
05/2022. André L. F. de Almeida thanks CNPq for its financial support under
grant 312491/2020-4. G. Fodor was partially supported by the Digital Futures
project PERCy. email<EMAIL_ADDRESS>
###### Abstract

In this letter, we consider an intelligent reflecting surface (IRS)-assisted
multiple input multiple output (MIMO) communication and we optimize the joint
active and passive beamforming by exploiting the geometrical structure of the
propagation channels. Due to the inherent Kronecker product structure of the
channel matrix, the global beamforming optimization problem is split into
lower dimensional horizontal and vertical sub-problems. Based on this
factorization property, we propose two closed-form methods for passive and
active beamforming designs, at the IRS, the base station, and user equipment,
respectively. The first solution is a singular value decomposition (SVD)-based
algorithm independently applied on the factorized channels, while the second
method resorts to a third-order rank-one tensor approximation along each
domain. Simulation results show that exploiting the channel Kronecker
structures yields a significant improvement in terms of computational
complexity at the expense of negligible spectral efficiency (SE) loss. We also
show that under imperfect channel estimation, the tensor-based solution shows
better SE than the benchmark and proposed SVD-based solutions.

###### Index Terms:

intelligent reflecting surface, channel factorization, joint active and
passive beamforming, MIMO, terahertz.

##  I Introduction

IRS is a candidate technology to achieve high data rates required for beyond
fifth generation (B5G) wireless networks [1, 2]. The IRS is a two-dimensional
planar array composed of multiple passive reflecting elements capable of
changing the electromagnetic properties of the impinging waves, e.g., the
phase and amplitude, so that the received signal can be added constructively
at the receiver. On the other hand, terahertz (THz) communications suffer from
high penetration and attenuation losses, which leads to sparse channels [3].
To combat the path loss effects, passive beamforming using IRS may be
introduced. Although it is an attractive solution for THz communications, the
joint design of the active precoder, at the transmitter, combiner, at the
receiver, and passive IRS phase shifts is challenging.

In this regard, [4] proposed two joint passive and active beamforming methods
to maximize the received signal power at the user in a multiple input single
output (MISO) network. Then, [5] and [6] provided two different approaches for
joint active and passive beamforming in a multi-user MISO networks. Regarding
IRS-assisted multiple input multiple output (MIMO), [7] proposed three
singular value decomposition (SVD)-based solutions for the joint active and
passive beamforming design to maximize the spectral efficiency (SE) at the
user-equipment (UE). Also, [8] and [9] proposed an alternating optimization
scheme to design the active and passive beamforming. In [10], the authors
proposed two low-complexity solutions for the beamforming design. Although
some of those papers focus on low-complexity solutions, they still do not
exploit the explicit geometrical structure of the channels to split the
optimization problem into horizontal and vertical sub-problems with lower
dimensions. The works in [11, 12, 13, 14, 15] focused on channel estimation
for IRS-assisted communications using tensor decomposition methods, such as
the canonical polyadic decomposition. Furthermore, [16] and [17] proposed a
rank-one tensor approximation to reduce the overhead associated with the IRS
phase shift feedback. To the best of our knowledge, there is no work that
proposes tensor modeling for the joint active and passive beamforming design.

Our contributions can be summarized as follows:

  1. 1.

We propose a novel signal modeling that exploits the geometrical channel
structure at the base station (BS), the intelligent reflecting surface (IRS),
and the UE by decomposing the received signal into horizontal and vertical
components. This approach allows splitting the joint active/passive
beamforming problem into independent sub-problems of lower dimensions. Each
sub-problem can be individually solved, and the solutions of which are
combined using the Kronecker product to obtain the overall solution.

  2. 2.

We propose two algorithms that exploit the Kronecker product structure of the
cascaded MIMO channel to design the active and passive beamforming. The first
method, referred to as Kronecker factorization (KF), directly exploits the
Kronecker structure by means of SVD-based rank-one approximations applied on
the factorized channels. Our second solution, referred to as third-order
tensor (TOT), recasts the cascaded channel along each domain as a third-order
rank-one tensor and resorts to the high order singular value decomposition
(HOSVD) algorithm to optimize the active and passive beamforming vectors.

We show that our proposed TOT and KF solutions can reduce the computational
complexity, respectively, by $15$ and $140$ times, compared to the benchmark
scheme of [7] with similar SE performance under perfect channel state
information (CSI) assumption. Otherwise, when imperfect CSI is considered the
TOT method shows a significant SE improvement compared to the KF and the
benchmark [7] methods.

Notation: Scalars, vector, matrices and tensors are denoted ($a$), ($a$),
($A$) and ($\mathcal{A}$), respectively. The superscripts $\\{\\}^{T}$,
$\\{\\}^{*}$, and $\\{\\}^{H}$ denote transpose, conjugate, and hermitian,
respectively. The operators $\otimes$, $\diamond$, $\circ$, $\odot$ and
$\angle$ are the Kronecker, the Khatri-Rao, the outer product, the Hadamard
products and the angle of a complex value, respectively.
$\text{vec}({\mbox{\boldmath$A$}})$ converts $A$ to a column vector by
stacking its columns. The $n$-mode product of a tensor
$\mathcal{X}\in\mathbb{C}^{I\times J\times K}$ and a matrix
${\mbox{\boldmath$A$}}\in\mathbb{C}^{I\times R}$ is denoted by
$\mathcal{Y}=\mathcal{X}\times_{n}{\mbox{\boldmath$A$}}$, $n=1,2,3$.

##  II System Model

We consider a MIMO-IRS communication system, where the BS has $M$ antennas
transmitting a single stream symbol towards a UE with $K$ antennas via an IRS
having $N$ reflecting elements, which are adjusted by a controller connected
to the BS. Initially, Perfect CSI of all channels is assumed at the BS (see
Section IV for the effect of imperfect CSI). We further assume there is no
direct link between the BS and the UE. The received signal $r$ at the UE can
be written as

$\displaystyle
r={\mbox{\boldmath$w$}}^{H}{\mbox{\boldmath$G$}}\text{diag}({\mbox{\boldmath$\theta$}}){\mbox{\boldmath$H$}}{\mbox{\boldmath$q$}}x+n,$
(1)

where ${\mbox{\boldmath$w$}}\in\mathbb{C}^{K\times 1}$ is the combiner,
${\mbox{\boldmath$G$}}\in\mathbb{C}^{K\times N}$ is the channel between the
IRS and the UE, ${\mbox{\boldmath$\theta$}}\in\mathbb{C}^{N}$ is the IRS phase
shift vector defined as
${\mbox{\boldmath$\theta$}}\doteq[e^{j\theta_{1}},\dots,e^{j\theta_{N}}]^{T}$,
where $\theta_{n}\in[-\pi,\pi]$ represents the phase shift of the $n$-th
reflecting element, ${\mbox{\boldmath$H$}}\in\mathbb{C}^{N\times M}$ is the
channel between the BS and the IRS,
${\mbox{\boldmath$q$}}\in\mathbb{C}^{M\times 1}$ is the transmit precoder, $x$
is the transmitted signal, with power $P_{t}$, and
$n\sim\mathcal{C}\mathcal{N}(0,\sigma_{n}^{2})$ is the additive white Gaussian
noise (AWGN) with zero mean and variance $\sigma_{n}^{2}$.

Our goal is to maximize the signal-to-noise ratio (SNR) at the UE subject to
the IRS phase shifts, precoder, and combiner constraints, which leads to the
following optimization problem

$\displaystyle\max_{{\mbox{\boldmath$w$}},{\mbox{\boldmath$q$}},{\mbox{\boldmath$\theta$}}}|({\mbox{\boldmath$w$}}^{H}{\mbox{\boldmath$G$}}\text{diag}({\mbox{\boldmath$\theta$}}){\mbox{\boldmath$H$}}{\mbox{\boldmath$q$}})|^{2}$
(2) $\displaystyle\text{s.t.}\
||{\mbox{\boldmath$w$}}||=||{\mbox{\boldmath$q$}}||=1\ \text{and}\
\theta_{n}\in[-\pi,\pi].$

A classical solution to this problem was proposed in [7] and related works,
which relies on SVD-based steps applied to the full MIMO channel matrices $H$
and $G$ to find their dominant eigenmodes, from which the active beamforming
vectors (precoder $q$ and combiner $w$), as well as the IRS phase shift vector
$\theta$ are determined.

##  III Proposed approaches

Here, we discuss our proposed low-complexity active and passive beamforming
design. We first derive a Kronecker-based model for the involved channels and
then recast the resulting two-dimensional optimization problem as a product of
two smaller one-dimensional problems for each channel dimension. Then, our two
algorithms exploiting the resulting optimization problem are derived.

###  III-A Kronecker-structured Channel Factorization

A uniform rectangular array (URA) is deployed both at the BS and the UE, where
the BS is equipped with $M_{y}$ antenna elements along the $y$ axis and
$M_{z}$ antenna elements along the $z$ axis, with $M=M_{y}M_{z}$ being the
total number of antenna elements. Similarly, the UE has $K=K_{y}K_{z}$ antenna
elements, where $K_{y}$ and $K_{z}$ are the number of elements along the $y$
axis and $z$ axis, respectively. The element spacing between antennas of the
URAs is $\lambda/2$. At the BS side, the response of the $m$th antenna element
is defined as [18]

$\displaystyle[{\mbox{\boldmath$a$}}(\phi_{\text{bs}},\theta_{\text{bs}})]_{m}=e^{-j\pi[m_{y}\sin{\theta_{\text{bs}}}\sin{\phi_{\text{bs}}}+m_{z}\cos\theta_{\text{bs}}]},$
(3)

where $m=m_{z}+(m_{y}-1)M_{z}$ with $m_{y}~{}\in~{}{[0,1,\dots,M_{y}-1]}$ and
$m_{z}~{}\in~{}{[0,1,\dots,M_{z}-1]}$, $\theta_{\text{bs}}$ and
$\phi_{\text{bs}}$ represent the elevation of departure (EoD) and azimuth of
departure (AoD), respectively. Furthermore, define the spatial frequencies as
$\mu_{\text{bs}}=\pi\sin\theta_{\text{bs}}\sin\phi_{\text{bs}}$ and
$\psi_{\text{bs}}=\pi\cos\theta_{\text{bs}}$. The overall array response in
(3) can be written as a Kronecker product between the associated horizontal
and the vertical components as
${\mbox{\boldmath$a$}}(\mu_{\text{bs}},\psi_{\text{bs}})={\mbox{\boldmath$a$}}_{y}(\mu_{\text{bs}})\otimes{\mbox{\boldmath$a$}}_{z}(\psi_{\text{bs}})\in\mathbb{C}^{M\times
1}$, where ${\mbox{\boldmath$a$}}_{y}(\mu_{\text{bs}})$ is the horizontal
steering vector as
${\mbox{\boldmath$a$}}_{y}(\mu_{\text{bs}})=[1,e^{-j\mu_{\text{bs}}},\dots,e^{-j(M_{y}-1)\mu_{\text{bs}}}]\in\mathbb{C}^{M_{y}\times
1},$ and ${\mbox{\boldmath$a$}}_{z}(\psi_{\text{bs}})$ is the vertical
steering vector as
${\mbox{\boldmath$a$}}_{z}(\psi_{\text{bs}})=[1,e^{-j\psi_{\text{bs}}},\dots,e^{-j(M_{z}-1)\psi_{\text{bs}}}]\in\mathbb{C}^{M_{z}\times
1}.$ Note that in a similar way, the IRS arrival, the UE, and IRS departure
channel steering vectors can be written as
${\mbox{\boldmath$b$}}(\mu_{\text{irs}_{A}},\psi_{\text{irs}_{A}})={\mbox{\boldmath$b$}}_{y}(\mu_{\text{irs}_{A}})\otimes{\mbox{\boldmath$b$}}_{z}(\psi_{\text{irs}_{A}})$,
${\mbox{\boldmath$c$}}(\mu_{\text{ue}},\psi_{\text{ue}})={\mbox{\boldmath$c$}}_{y}(\mu_{\text{ue}})\otimes{\mbox{\boldmath$c$}}_{z}(\psi_{\text{ue}})$
and
${\mbox{\boldmath$d$}}(\mu_{\text{irs}_{D}},\psi_{\text{irs}_{D}})={\mbox{\boldmath$d$}}_{y}(\mu_{\text{irs}_{D}})\otimes{\mbox{\boldmath$d$}}_{z}(\psi_{\text{irs}_{D}})$,
respectively.

Under the previous definitions and assumptions, the channel matrix $H$ linking
the BS to the IRS can be written as

$\displaystyle{\mbox{\boldmath$H$}}\\!=\\!\sum^{L}_{l=1}\alpha^{(l)}_{H}[{\mbox{\boldmath$b$}}_{y}(\mu^{(l)}_{\text{irs}_{A}})\\!\otimes\\!{\mbox{\boldmath$b$}}_{z}(\psi^{(l)}_{\text{irs}_{A}})][{\mbox{\boldmath$a$}}_{y}(\mu^{(l)}_{\text{bs}})\\!\otimes\\!{\mbox{\boldmath$a$}}_{z}(\psi^{(l)}_{\text{bs}})]^{T}$
(4)

where $L$ is the number of paths, and $\alpha^{(l)}_{H}$
$~{}\sim~{}$$\mathcal{CN}~{}(\mu,\sigma)$ is the complex gain of the $l$th
path. Applying the property
$({\mbox{\boldmath$A$}}\otimes{\mbox{\boldmath$B$}})({\mbox{\boldmath$C$}}\otimes{\mbox{\boldmath$D$}})={\mbox{\boldmath$A$}}{\mbox{\boldmath$C$}}\otimes{\mbox{\boldmath$B$}}{\mbox{\boldmath$D$}}$,
we can rewrite (4) as

$\displaystyle{\mbox{\boldmath$H$}}=\sum^{L}_{l=1}\underbrace{\alpha^{(l)}_{H}[{\mbox{\boldmath$b$}}_{y}(\mu^{(l)}_{\text{irs}_{A}}){\mbox{\boldmath$a$}}^{T}_{y}(\mu^{(l)}_{\text{bs}})]}_{{\mbox{\boldmath$H$}}^{(l)}_{y}\in\mathbb{C}^{N_{y}\times
M_{y}}}\otimes\underbrace{[{\mbox{\boldmath$b$}}_{z}(\psi^{(l)}_{\text{irs}_{A}}){\mbox{\boldmath$a$}}^{T}_{z}(\psi^{(l)}_{\text{bs}})]}_{{\mbox{\boldmath$H$}}^{(l)}_{z}\in\mathbb{C}^{N_{z}\times
M_{z}}},$ (5)

or, compactly,

$\displaystyle{\mbox{\boldmath$H$}}=\sum^{L}_{l=1}{\mbox{\boldmath$H$}}^{(l)}_{y}\otimes{\mbox{\boldmath$H$}}^{(l)}_{z}.$
(6)

Similarly, following the same construction and definitions in (4) and (5) the
IRS-UE channel $G$ can be written as

$\displaystyle{\color[rgb]{0,0,0}{\mbox{\boldmath$G$}}=\sum^{R}_{r=1}\underbrace{\alpha^{(l)}_{G}[{\mbox{\boldmath$c$}}_{y}(\mu^{(l)}_{\text{ue}}){\mbox{\boldmath$d$}}^{T}_{y}(\mu^{(l)}_{\text{irs}_{D}})]}_{{\mbox{\boldmath$G$}}_{y}^{(l)}\in\mathbb{C}^{K_{y}\times
N_{y}}}\otimes\underbrace{[{\mbox{\boldmath$c$}}_{z}(\psi^{(l)}_{\text{ue}}){\mbox{\boldmath$d$}}^{T}_{z}(\psi^{(l)}_{\text{irs}_{D}})]}_{{\mbox{\boldmath$G$}}_{z}^{(l)}\in\mathbb{C}^{K_{z}\times
N_{z}}},}$ (7)

and, analogously as in (6)

$\displaystyle{\mbox{\boldmath$G$}}=\sum^{L}_{l=1}{\mbox{\boldmath$G$}}^{(l)}_{y}\otimes{\mbox{\boldmath$G$}}^{(l)}_{z}.$
(8)

In practical scenarios, the channel shows less variation in the vertical
domain compared to the horizontal domain [19]. For example, in THz
communications the channels are dominated by their line-of-sight (LoS)
components, while some non-LoS scenarios exhibit small angular spreads [3]. In
such scenarios, the vertical spatial frequencies of both $H$ and $G$ channels,
defined in (6) and (8), are strongly correlated, which implies reasonably
assuming that
${\mbox{\boldmath$H$}}^{(1)}_{z}\approx\cdots\approx{\mbox{\boldmath$H$}}^{(L)}_{z}$,
and
${\mbox{\boldmath$G$}}^{(1)}_{z}\approx\cdots\approx{\mbox{\boldmath$G$}}^{(L)}_{z}$.
This leads to the following approximations

$\displaystyle{\mbox{\boldmath$H$}}\approx\underbrace{\Big{(}\sum^{L}_{l=1}{\mbox{\boldmath$H$}}^{(l)}_{y}\Big{)}}_{{\mbox{\boldmath$H$}}_{y}}\otimes\
{\mbox{\boldmath$H$}}_{z}={\mbox{\boldmath$H$}}_{y}\otimes{\mbox{\boldmath$H$}}_{z},$
(9)
$\displaystyle{\mbox{\boldmath$G$}}\approx\underbrace{\Big{(}\sum^{L}_{l=1}{\mbox{\boldmath$G$}}^{(l)}_{y}\Big{)}}_{{\mbox{\boldmath$G$}}_{y}}\otimes\
{\mbox{\boldmath$G$}}_{z}={\mbox{\boldmath$G$}}_{y}\otimes{\mbox{\boldmath$G$}}_{z}.$
(10)

Otherwise stated, the involved MIMO channel matrices can be factorized in
terms of the Kronecker product of their respective horizontal ($y$-domain) and
vertical ($z$-domain) components. Note that such a Kronecker-structured model
is exact in a pure LoS scenario, or when the non-LoS components are
negligible.

###  III-B Two-dimensional Active and Passive Beamforming

By adopting the Kronecker-product based MIMO channel factorizations in (9) and
(10), we can rewrite (1) as

$r={\mbox{\boldmath$w$}}^{H}({\mbox{\boldmath$G$}}_{y}\otimes{\mbox{\boldmath$G$}}_{z})\text{diag}({\mbox{\boldmath$\theta$}})({\mbox{\boldmath$H$}}_{y}\otimes{\mbox{\boldmath$H$}}_{z}){\mbox{\boldmath$q$}}x+n.$
(11)

To fully decouple the received signal as well as the optimization problem (2)
into horizontal and vertical components, we impose a Kronecker product
structure to the active and passive beamforming vectors (i.e., precoder,
combiner, and IRS phase shifts) by defining

${\mbox{\boldmath$w$}}\doteq{\mbox{\boldmath$w$}}_{y}\otimes{\mbox{\boldmath$w$}}_{z},\quad{\mbox{\boldmath$q$}}\doteq{\mbox{\boldmath$q$}}_{y}\otimes{\mbox{\boldmath$q$}}_{z},\quad{\mbox{\boldmath$\theta$}}\doteq{\mbox{\boldmath$\theta$}}_{y}\otimes{\mbox{\boldmath$\theta$}}_{z}.$
(12)

Using these definitions, we simplify (11) to

$\displaystyle r=s_{y}s_{z}+n,$ (13)

where
$s_{y}\doteq{\mbox{\boldmath$w$}}_{y}^{H}{\mbox{\boldmath$G$}}_{y}\text{diag}({\mbox{\boldmath$\theta$}}_{y}){\mbox{\boldmath$H$}}_{y}{\mbox{\boldmath$q$}}_{y}$
is the horizontal domain desired signal component, and
$s_{z}\doteq{\mbox{\boldmath$w$}}_{z}^{H}{\mbox{\boldmath$G$}}_{z}\text{diag}({\mbox{\boldmath$\theta$}}_{z}){\mbox{\boldmath$H$}}_{z}{\mbox{\boldmath$q$}}_{z}$
is the vertical domain desired signal component. From (13), the SE expression
can then be written as

$\displaystyle\text{SE}=\log_{2}\Big{(}1+\frac{|s_{y}s_{z}|^{2}}{\sigma_{n}^{2}}\Big{)}=\log_{2}\Big{(}1+\text{SNR}_{y}\text{SNR}_{z}\Big{)},$
(14)

where we have defined $\text{SNR}_{y}=s_{y}s_{y}^{*}/\sqrt{\sigma^{2}}$ and
$\text{SNR}_{z}=s_{z}s_{z}^{*}/\sqrt{\sigma^{2}}$, as the horizontal and
vertical SNRs, respectively. Although imposing a Kronecker structure on these
beamforming vectors restricts the solution space, a significant complexity
reduction is achieved compared to the conventional “full” design, especially
for large IRS panels. Exploiting the decoupled signal model in (13) two
solutions are now derived to find the two-dimensional beamforming sets
$({\mbox{\boldmath$w$}}_{y},{\mbox{\boldmath$w$}}_{z})$,
$({\mbox{\boldmath$q$}}_{y},{\mbox{\boldmath$q$}}_{z})$, and
$({\mbox{\boldmath$\theta$}}_{y},{\mbox{\boldmath$\theta$}}_{z})$.

###  III-C Kronecker Factorization (KF) Method

From the factorization of the received signal in (13), the full optimization
problem (2) can be replaced by two smaller optimization sub-problems along
each domain, i.e.,

$\displaystyle\max_{{\mbox{\boldmath$w$}}_{y},{\mbox{\boldmath$q$}}_{y},{\mbox{\boldmath$\theta$}}_{y}}|({\mbox{\boldmath$w$}}_{y}^{H}{\mbox{\boldmath$G$}}_{y}\text{diag}({\mbox{\boldmath$\theta$}}_{y}){\mbox{\boldmath$H$}}_{y}{\mbox{\boldmath$q$}}_{y})|^{2}$
(15) $\displaystyle\text{s.t.}\
||{\mbox{\boldmath$w$}}_{y}||=||{\mbox{\boldmath$q$}}_{y}||=1\ \text{and}\
\theta_{y}\in[-\pi,\pi],$
$\displaystyle\max_{{\mbox{\boldmath$w$}}_{z},{\mbox{\boldmath$q$}}_{z},{\mbox{\boldmath$\theta$}}_{z}}|({\mbox{\boldmath$w$}}_{z}^{H}{\mbox{\boldmath$G$}}_{z}\text{diag}({\mbox{\boldmath$\theta$}}_{z}){\mbox{\boldmath$H$}}_{z}{\mbox{\boldmath$q$}}_{z})|^{2}$
(16) $\displaystyle\text{s.t.}\
||{\mbox{\boldmath$w$}}_{z}||=||{\mbox{\boldmath$q$}}_{z}||=1\ \text{and}\
\theta_{z}\in[-\pi,\pi].$

individually maximizing the SNR in the $y$ and $z$ domains implies maximizing
the overall SNR (14). The solution to the individual problems (15) and (16)
can be obtained from any state-of-the-art method. In this work, we consider
the method of [7] but applied in each channel domain. The steps of the KF
solution are detailed in Algorithm 1.

1 Compute the truncated SVD of ${\mbox{\boldmath$G$}}_{y}$,
${\mbox{\boldmath$G$}}_{z}$, ${\mbox{\boldmath$H$}}_{y}$ and
${\mbox{\boldmath$H$}}_{z}$.

$\lambda_{\text{g}_{y}}{\mbox{\boldmath$u$}}_{\text{g}_{y}}{\mbox{\boldmath$v$}}_{\text{g}_{y}}^{H}\approx{\mbox{\boldmath$G$}}_{y}$,
$\lambda_{\text{g}_{z}}{\mbox{\boldmath$u$}}_{\text{g}_{z}}{\mbox{\boldmath$v$}}_{\text{g}_{z}}^{H}\approx{\mbox{\boldmath$G$}}_{z}$,

$\lambda_{\text{h}_{y}}{\mbox{\boldmath$u$}}_{\text{h}_{y}}{\mbox{\boldmath$v$}}_{\text{h}_{y}}^{H}\approx{\mbox{\boldmath$H$}}_{y}$
and
$\lambda_{\text{h}_{z}}{\mbox{\boldmath$u$}}_{\text{h}_{z}}{\mbox{\boldmath$v$}}_{\text{h}_{z}}^{H}\approx{\mbox{\boldmath$H$}}_{z}$.

2 Design the combiner as
${\mbox{\boldmath$w$}}={\mbox{\boldmath$u$}}_{\text{g}_{y}}\otimes{\mbox{\boldmath$u$}}_{\text{g}_{z}}$.

3 Design the precoder as
${\mbox{\boldmath$q$}}={\mbox{\boldmath$v$}}_{\text{h}_{y}}\otimes{\mbox{\boldmath$v$}}_{\text{h}_{z}}$.

4 Design the IRS phase shifts as

${\mbox{\boldmath$\theta$}}=-\angle\\{[{\mbox{\boldmath$v$}}_{\text{g}_{y}}^{*}\odot{\mbox{\boldmath$u$}}_{\text{h}_{y}}]\otimes[{\mbox{\boldmath$v$}}_{\text{g}_{z}}\odot{\mbox{\boldmath$u$}}_{\text{h}_{z}}]\\}$

Algorithm 1 Kronecker factorization (KF) method

###  III-D Third-Order Tensor (TOT) Method

The second method exploits the horizontal and vertical factorizations of the
received signal from a tensor modeling perspective. Starting from (13), let us
rewrite $s_{y}$ and $s_{z}$ components of the received signal as

$\displaystyle
s_{y}={\mbox{\boldmath$w$}}_{y}^{H}{\mbox{\boldmath$G$}}_{y}\text{diag}({\mbox{\boldmath$\theta$}}_{y}){\mbox{\boldmath$H$}}_{y}{\mbox{\boldmath$q$}}_{y}=({\mbox{\boldmath$q$}}_{y}^{T}\otimes{\mbox{\boldmath$w$}}_{y}^{H}){\mbox{\boldmath$F$}}_{y}{\mbox{\boldmath$\theta$}}_{y},$
$\displaystyle
s_{z}={\mbox{\boldmath$w$}}_{z}^{H}{\mbox{\boldmath$G$}}_{z}\text{diag}({\mbox{\boldmath$\theta$}}_{z}){\mbox{\boldmath$H$}}_{z}{\mbox{\boldmath$q$}}_{z}=({\mbox{\boldmath$q$}}_{z}^{T}\otimes{\mbox{\boldmath$w$}}_{z}^{H}){\mbox{\boldmath$F$}}_{z}{\mbox{\boldmath$\theta$}}_{z},$

where we have defined
${\mbox{\boldmath$F$}}_{y}\doteq{\mbox{\boldmath$H$}}^{T}_{y}\diamond{\mbox{\boldmath$G$}}_{y}\in\mathbb{C}^{K_{y}M_{y}\times
N_{y}}$ and
${\mbox{\boldmath$F$}}_{z}\doteq{\mbox{\boldmath$H$}}^{T}_{z}\diamond{\mbox{\boldmath$G$}}_{z}\in\mathbb{C}^{K_{z}M_{z}\times
N_{z}}$ as the combined $y$ and $z$ domains Khatri-Rao channels, respectively.
We can rearrange the elements of ${\mbox{\boldmath$F$}}_{y}$ and
${\mbox{\boldmath$F$}}_{z}$ in third-order tensors
$\mathcal{F}_{y}\in\mathbb{C}^{K_{y}\times M_{y}\times N_{y}}$ and
$\mathcal{F}_{z}\in\mathbb{C}^{K_{z}\times M_{z}\times N_{z}}$ via the
mappings
$[\mathcal{F}_{y}]_{k_{y},m_{y},n_{y}}\doteq[{\mbox{\boldmath$F$}}_{y}]_{(k_{y}-1)M_{y}+m_{y},n_{y}}$
and
$[\mathcal{F}_{z}]_{k_{z},m_{z},n_{z}}\doteq[{\mbox{\boldmath$F$}}_{z}]_{(k_{z}-1)M_{z}+m_{z},n_{z}}$,
where $k_{t}=1,\ldots,K_{t}$, $m_{t}=1,\ldots,M_{t}$, $n_{t}=1,\ldots,N_{t}$,
$t\in\\{y,z\\}$. The set of active and passive beamforming vectors that
individually maximize the horizontal ($y$-domain) and vertical ($z$-domain)
SNRs can be found by independently solving the following problems

$\displaystyle\max_{{\mbox{\boldmath$w$}}_{y},{\mbox{\boldmath$q$}}_{y},{\mbox{\boldmath$\theta$}}_{y}}||\mathcal{F}_{y}\times_{1}{\mbox{\boldmath$w$}}_{y}\times_{2}{\mbox{\boldmath$q$}}_{y}\times_{3}{\mbox{\boldmath$\theta$}}_{y}||^{2}$
(17) $\displaystyle\text{s.t.}\
||{\mbox{\boldmath$w$}}_{y}||=||{\mbox{\boldmath$q$}}_{y}||=1\ \text{and}\
\theta_{y}\in[-\pi,\pi]$
$\displaystyle\max_{{\mbox{\boldmath$w$}}_{z},{\mbox{\boldmath$q$}}_{z},{\mbox{\boldmath$\theta$}}_{z}}||\mathcal{F}_{z}\times_{1}{\mbox{\boldmath$w$}}_{z}\times_{2}{\mbox{\boldmath$q$}}_{z}\times_{3}{\mbox{\boldmath$\theta$}}_{z}||^{2}$
(18) $\displaystyle\text{s.t.}\
||{\mbox{\boldmath$w$}}_{z}||=||{\mbox{\boldmath$q$}}_{z}||=1\ \text{and}\
\theta_{z}\in[-\pi,\pi]$

These problems can be solved by applying the HOSVD [20] to the tensors
$\mathcal{F}_{y}$ and $\mathcal{F}_{z}$. The solutions to
${\mbox{\boldmath$w$}}_{y}$ ${\mbox{\boldmath$q$}}_{y}$, and
${\mbox{\boldmath$\theta$}}_{y}$ correspond respectively to the 1-mode,
2-mode, and 3-mode dominant left singular vectors of $\mathcal{F}_{y}$.
Likewise, ${\mbox{\boldmath$w$}}_{z}$ ${\mbox{\boldmath$q$}}_{z}$, and
${\mbox{\boldmath$\theta$}}_{z}$ are found as the 1-mode, 2-mode, and 3-mode
dominant left singular vectors of $\mathcal{F}_{z}$ (we refer the reader to
[20] for further details on the HOSVD algorithm). The steps of the TOT method
are summarized in Algorithm 2.

1 Compute the HOSVD of $\mathcal{F}_{y}$ and $\mathcal{F}_{z}$ to obtain

$\\{{\mbox{\boldmath$w$}}_{y},{\mbox{\boldmath$q$}}_{y},{\mbox{\boldmath$\theta$}}_{y}\\}$
and
$\\{{\mbox{\boldmath$w$}}_{z},{\mbox{\boldmath$q$}}_{z},{\mbox{\boldmath$\theta$}}_{z}\\}$,
respectively.

2 Design the combiner as
${\mbox{\boldmath$w$}}={\mbox{\boldmath$w$}}_{y}\otimes{\mbox{\boldmath$w$}}_{z}$.

3 Design the precoder as
${\mbox{\boldmath$q$}}={\mbox{\boldmath$q$}}^{*}_{y}\otimes{\mbox{\boldmath$q$}}^{*}_{z}$.

4 Design the IRS phase shifts as
${\mbox{\boldmath$\theta$}}=-\angle[{\mbox{\boldmath$\theta$}}_{y}\otimes{\mbox{\boldmath$\theta$}}_{z}]$.

Algorithm 2 Third-order tensor (TOT) method

###  III-E Complexity Analysis

Both KF and TOT methods rely on (truncated SVD-based) rank-one approximation
procedures. Note that a single low-rank matrix approximation has a complexity
$O(ijv)$ [21], where $i$ is the number of rows, $j$ is the number of columns
and $v$ is the rank. For the rank-one approximation, this complexity is
reduced to $O(ij)$. In the baseline solution of [7], the precoder, combiner,
and IRS phase shifts are determined from the dominant left singular vectors of
the full channel matrices $H$ and $G$, resulting in $O(N(M+K))$. The KF method
solves the beamforming problem for each channel dimension separately, which
implies solving four rank-one matrix approximation procedures to
${\mbox{\boldmath$G$}}_{y}$, ${\mbox{\boldmath$G$}}_{z}$,
${\mbox{\boldmath$H$}}_{y}$ and ${\mbox{\boldmath$H$}}_{z}$. Thus, the overall
complexities of the horizontal and vertical problems are respectively given as
$O~{}(N_{y}(M_{y}~{}+~{}K_{y}))$ and $O(N_{z}(M_{z}+K_{z}))$. Finally, the
complexity of the TOT method corresponds to that of computing two independent
HOSVDs for the horizontal and vertical optimization problems, where the
individual HOSVDs have complexities $O(3K_{y}M_{y}N_{y})$ and
$O(3K_{z}M_{z}N_{z})$, respectively. The complexities are summarized in Table
I.

##  IV Simulation Results

We consider a MIMO communication system where the BS is equipped with $M=128$
antenna elements, composed of $M_{y}=16$ and $M_{z}=8$ along horizontal and
vertical domains, serves a single user with $K=16$ antenna elements composed
of $K_{y}=K_{z}=4$ antenna elements along horizontal and vertical domains,
respectively. The communication is assisted by an IRS equipped with $N=100$
reflecting elements composed of $N_{y}=N_{z}=10$ reflecting elements along
horizontal and vertical domains, respectively. The elevation of arrival (EoA)
and the EoD are generated from the uniform distribution
$\theta_{\text{bs}},\theta_{\text{irs}_{A}},\theta_{\text{irs}_{D}},\theta_{\text{ue}}\sim\text{U}~{}[90^{\circ}-\delta,90^{\circ}+\delta]$,
where $\delta$ is the elevation spread. Also, the azimuth of arrival (AoA) and
the AoD are generated from the uniform distribution
$\phi_{\text{ue}},\phi_{\text{irs}_{A}},\phi_{\text{irs}_{D}},\phi_{\text{ue}}\sim\text{U}~{}[-60^{\circ},60^{\circ}]$.
Also, $4$ paths are considered for $G$ and $H$. For comparison, the SVD-based
joint active and passive beamforming algorithm proposed in [7] is used as the
baseline solution serving as a reference for comparison.

TABLE I: Computational complexity Solution | Complexity
---|---
Baseline [7] | $O(N(M+K))$
KF | $O(N_{y}(M_{y}+K_{y})+N_{z}(M_{z}+K_{z}))$
TOT | $O(3(K_{y}M_{y}N_{y}+K_{z}M_{z}N_{z}))$
Figure 1: SE of the proposed methods compared to the baseline [7] for
$\delta=2.5^{\circ}$ and $\delta=10^{\circ}$, varying the SNR.

Fig. 1 shows the performance of the proposed algorithms varying the SNR for
different elevation spread values. Note that the approximation error
associated with the separable channel structure assumption in (9) and (10)
increases as a function of the elevation spread. The results show that the
proposed KF and TOT solutions have a slight performance degradation compared
with the baseline method of [7]. For example, in Fig 1a, the performance
degradation is $0.3$ bit/s/Hz, while in Fig 1b, it is equal to $2.3$ bit/s/Hz.
Indeed, since the optimizations problems in (17) and (18) are the same as the
ones in (15) and (16), respectively, both solutions have the same SE. It is
important to highlight that, in scenarios where the approximations in (9) and
(10) become exact, the imposed Kronecker structure of the passive and active
beamforming vectors in (12) does not degrade the SE, i.e., the proposed KF and
TOT solutions achieve the same SE as the baseline solution [7].

Figure 2: Computational complexity of the proposed solutions and the baseline
[7] for fixed $M$ and $K$ while varying $N$.

Fig. 2 shows the computational complexity as a function of the number of IRS
reflecting elements. By exploiting the Kronecker factorization structure of
the channels, the KF and TOT solutions have lower slopes when compared to the
baseline algorithm [7]. Although KF and TOT have small degradation in
performance, they can significantly reduce the computational complexity when
moderate or large numbers of IRS elements are considered and, as shown
previously, depending on the scenario they can have very close performance
compared to the baseline solution. For example, for $1000$ reflecting elements
the baseline solution [7] is $140$ and $15$ times more complex than the
proposed KF and TOT solutions, respectively.

Figure 3: SE of the proposed methods and the baseline method of [7] under
imperfect CSI.

Figure 3 depicts the SE of the proposed algorithms when channel estimation
error is taken into account, considering $\text{SNR}=20$dB. To this end, let
${\mbox{\boldmath$\hat{F}$}}\doteq{\mbox{\boldmath$H$}}^{T}\diamond{\mbox{\boldmath$G$}}+{\mbox{\boldmath$Z$}}$
be the noisy estimate of the combined (Khatri-Rao) channel obtained after LS
estimation and matched filtering with the known orthogonal pilots and optimal
DFT IRS phase shift matrices (details can be found in [22]), where
${\mbox{\boldmath$Z$}}\sim\mathcal{C}\mathcal{N}(0,\sigma_{z}^{2})$ accounts
for the channel estimation error. Also, define
${\mbox{\boldmath$\hat{F}$}}_{y}$ and ${\mbox{\boldmath$\hat{F}$}}_{z}$ as the
noisy estimates of the $y$ and $z$ channel components of $\hat{F}$ (obtained
according to [14]). The baseline method of [7] optimizes the beamforming
vectors from the estimates of $H$ and $G$ obtained after a Khatri-Rao
factorization step applied to the noisy estimate $\hat{F}$. The KF method
relies on the decoupled estimates of
${\mbox{\boldmath$\hat{H}$}}_{y},{\mbox{\boldmath$\hat{H}$}}_{z},{\mbox{\boldmath$\hat{G}$}}_{y}$,
${\mbox{\boldmath$\hat{G}$}}_{z}$ obtained after Khatri-Rao and Kronecker
factorization steps, while the TOT method operates directly on the estimates
${\mbox{\boldmath$\hat{F}$}}_{y}$ and ${\mbox{\boldmath$\hat{F}$}}_{z}$,
avoiding the Khatri-Rao factorization step. The perfect CSI curve corresponds
to the baseline method [7] assuming perfect CSI knowledge. As shown in Fig. 3,
for higher values of $\sigma_{z}^{2}$ TOT outperforms the baseline and the KF
methods due to the processing required to obtain
${\mbox{\boldmath$\hat{F}$}}_{y}$ and ${\mbox{\boldmath$\hat{F}$}}_{z}$ which
rejects part of the noise. The KF has the worst performance due to the scaling
ambiguities in the Khatri-Rao factorization step that propagates to the
subsequent stage where the horizontal and vertical channel matrices are
estimated. Hence, although TOT is more computationally complex than KF, under
moderate channel estimation error, it has a better SE performance. For
example, assuming $\sigma_{z}^{2}=-8$dB the TOT, baseline, and KF solutions
present a SE gap of $1.6$, $7$, and $8.2$ bits/s/Hz compared to the perfect
CSI curve. These results show the involved tradeoffs between complexity and
performance under imperfect CSI.

##  V Conclusion

We proposed two low-complexity joint active and passive beamforming designs
for IRS-assisted MIMO systems. Our methods exploit the geometrical channel
structure to split the involved optimization problems into vertical and
horizontal domains. The proposed KF and TOT algorithms significantly reduce
the computational complexity with negligible SE degradation  compared to the
reference method under perfect CSI, while the TOT method provides the best SE
performance when channel estimation noise is taken into account in the
proposed optimization due to more efficient noise rejection.

## References

  * [1] E. Basar, M. Di Renzo, J. De Rosny, M. Debbah, M. Alouini, and R. Zhang, “Wireless Communications Through Reconfigurable Intelligent Surfaces,” _IEEE Access_ , vol. 7, pp. 116 753–116 773, 2019.
  * [2] S. Gong, X. Lu, D. T. Hoang, D. Niyato, L. Shu, D. I. Kim, and Y.-C. Liang, “Toward smart wireless communications via intelligent reflecting surfaces: A contemporary survey,” _IEEE Communications Surveys & Tutorials_, vol. 22, no. 4, pp. 2283–2314, June 2020.
  * [3] N. A. Abbasi, J. L. Gomez, R. Kondaveti, S. M. Shaikbepari, S. Rao, S. Abu-Surra, G. Xu, J. Zhang, and A. F. Molisch, “Thz band channel measurements and statistical modeling for urban D2D environments,” _IEEE Trans. Wireless Commun._ , vol. 22, no. 3, pp. 1466–1479, Mar 2023\.
  * [4] Q. Wu and R. Zhang, “Intelligent reflecting surface enhanced wireless network: Joint active and passive beamforming design,” in _Proc. IEEE Globecom_ , Dec. 2018, pp. 1–6.
  * [5] H. Song, M. Zhang, J. Gao, and C. Zhong, “Unsupervised learning-based joint active and passive beamforming design for reconfigurable intelligent surfaces aided wireless networks,” _IEEE Commun. Lett._ , vol. 25, no. 3, pp. 892–896, Mar. 2021.
  * [6] H. Ur Rehman, F. Bellili, A. Mezghani, and E. Hossain, “Joint active and passive beamforming design for IRS-assisted multi-user MIMO systems: A vamp-based approach,” _IEEE Trans. Commun._ , vol. 69, no. 10, pp. 6734–6749, Oct 2021.
  * [7] A. Zappone, M. Di Renzo, F. Shams, X. Qian, and M. Debbah, “Overhead-aware design of reconfigurable intelligent surfaces in smart radio environments,” _IEEE Trans. Wireless Commun._ , vol. 20, no. 1, pp. 126–141, Jan. 2021.
  * [8] W. Zhou, J. Xia, C. Li, L. Fan, and A. Nallanathan, “Joint precoder, reflection coefficients, and equalizer design for IRS-assisted MIMO systems,” _IEEE Trans. Commun._ , vol. 70, no. 6, pp. 4146–4161, June 2022\.
  * [9] X. Zhao, K. Xu, S. Ma, S. Gong, G. Yang, and C. Xing, “Joint transceiver optimization for IRS-aided MIMO communications,” _IEEE Trans. Commun._ , vol. 70, no. 5, pp. 3467–3482, May 2022.
  * [10] E. E. Bahingayi and K. Lee, “Low-complexity beamforming algorithms for IRS-aided single-user massive MIMO mmwave systems,” _IEEE Trans. Wireless Commun._ , vol. 21, no. 11, pp. 9200–9211, Nov 2022.
  * [11] G. T. de Araújo, A. L. F. de Almeida, and R. Boyer, “Channel estimation for intelligent reflecting surface assisted MIMO systems: A tensor modeling approach,” _IEEE JSTSP_ , vol. 15, no. 3, pp. 789–802, April 2021.
  * [12] G. T. de Araújo, P. R. B. Gomes, A. L. F. de Almeida, G. Fodor, and B. Makki, “Semi-blind joint channel and symbol estimation in IRS-assisted multiuser MIMO networks,” _IEEE Wireless Commun. Lett._ , vol. 11, no. 7, pp. 1553–1557, July 2022.
  * [13] Y. Lin, S. Jin, M. Matthaiou, and X. You, “Tensor-based algebraic channel estimation for hybrid IRS-assisted MIMO-OFDM,” _IEEE Trans. Wireless Commun._ , vol. 20, no. 6, pp. 3770–3784, June 2021.
  * [14] Fazal-E-Asim, B. Sokal, A. L. F. de Almeida, B. Makki, and G. Fodor, “Tensor-Based High-Resolution Channel Estimation for RIS-Assisted Communications,” _arXiv e-prints_ , p. arXiv:2304.05576, Apr. 2023.
  * [15] Fazal-E-Asim, A. L. F. de Almeida, B. Sokal, B. Makki, and G. Fodor, “Two-dimensional channel parameter estimation for IRS-assisted networks,” 2023\. [Online]. Available: http://arxiv.org/abs/2305.04393
  * [16] B. Sokal, P. R. B. Gomes, A. L. F. de Almeida, B. Makki, and G. Fodor, “IRS phase-shift feedback overhead-aware model based on rank-one tensor approximation,” in _Proc IEEE Globecom_ , Dec 2022, pp. 3911–3916.
  * [17] B. Sokal, P. R. B. Gomes, A. L. F. de Almeida, B. Makki, and G. Fodor, “Reducing the control overhead of intelligent reconfigurable surfaces via a tensor-based low-rank factorization approach,” 2022.
  * [18] Fazal-E-Asim, F. Antreich, C. C. Cavalcante, A. L. F. de Almeida, and J. A. Nossek, “Two-dimensional channel parameter estimation for millimeter-wave systems using butler matrices,” _IEEE Trans. Wireless Commun._ , vol. 20, no. 4, pp. 2670–2684, April 2021.
  * [19] Z. Wang, W. Liu, C. Qian, S. Chen, and L. Hanzo, “Two-dimensional precoding for 3-d massive mimo,” _IEEE Transactions on Vehicular Technology_ , vol. 66, no. 6, pp. 5485–5490, Jun. 2017.
  * [20] T. G. Kolda and B. W. Bader, “Tensor decompositions and applications,” _SIAM Review_ , vol. 51, no. 3, pp. 455–500, 2009.
  * [21] N. Kishore Kumar and J. Shneider, “Literature survey on low rank approximation of matrices,” _arXiv_ , p. arXiv:1606.06511, Jun. 2016.
  * [22] T. L. Jensen and E. De Carvalho, “An optimal channel estimation scheme for intelligent reflecting surfaces based on a minimum variance unbiased estimator,” in _Proc. ICASSP 2020_ , 2020, pp. 5000–5004.

  *[B5G]: beyond fifth generation
  *[MISO]: multiple input single output
  *[MIMO]: multiple input multiple output
  *[SVD]: singular value decomposition
  *[SE]: spectral efficiency
  *[UE]: user-equipment
  *[BS]: base station
  *[IRS]: intelligent reflecting surface
  *[KF]: Kronecker factorization
  *[TOT]: third-order tensor
  *[HOSVD]: high order singular value decomposition
  *[CSI]: channel state information
  *[AWGN]: additive white Gaussian noise
  *[SNR]: signal-to-noise ratio
  *[URA]: uniform rectangular array
  *[EoD]: elevation of departure
  *[AoD]: azimuth of departure
  *[LoS]: line-of-sight
  *[EoA]: elevation of arrival
  *[AoA]: azimuth of arrival


# Semantic Information for Object Detection

Jean-Francois Nies
RPTU Kaiserslautern-Landau
Gottlieb-Daimler str. 47, 67663
Kaiserslautern, Germany
<EMAIL_ADDRESS>
###### Abstract

In this paper, we demonstrate that the concept of Semantic Consistency and the
ensuing method of Knowledge-Aware Re-Optimization can be adapted for the
problem of object detection in intricate traffic scenes. Furthermore, we
introduce a novel method for extracting a knowledge graph from a dataset of
images provided with instance-level annotations, and integrate this new
knowledge graph with the existing semantic consistency model.

Combining both this novel ’hybrid’ knowledge graph and the pre-existing
methods of frequency analysis and external knowledge graph as sources for
semantic information, we investigate the effectiveness of knowledge-aware re-
optimization on the Faster-RCNN and DETR object detection models. We find that
limited but consistent improvements in precision and/or recall can be achieved
using this method for all combinations of model and method studied.

##  1 Introduction

The problem of Object Detection [Zou2019] may be expressed informally as the
question "What objects are where?". More formally, it incorporates both the
definition of bounding boxes encompassing objects of interest as closely as
possible, and the ability to assign correct labels to each such box. In the
last years, research in this domain, chiefly using convolutional neural
networks (CNN), has already led to object recognition systems capable of
surpassing human performance for selected tasks [Altenberger2018]. Such
systems typically employ deep CNNs and are widely deployed in a range of
practical applications in areas such as autonomous vehicles and facial
recognition systems.

However, unlike a human observer, a model such as a convolutional neural
network has no ability to reason about the scene it is observing, and dismiss
or correct erroneus identifications based on context [lecun2015deep]. Other
objects present, the time and location, and the position of objects in
relation to each other may all provide vital cues to an onlooker, but the
integration of this context into a machine learning model is a complex
problem: Contextual information must be made available in a machine-readable
form, frequently resulting in large amounts of additional data, and processed
alongside conventional inputs. Nonetheless, this approach has received a great
deal of attention. Aside from immediate gains in performance, such Informed
Machine Learning may also provide ’soft’ benefits in terms of explainability
and accountability. [Rueden2019]

While it is possible to design models which integrate external knowledge from
the ground up, there may also be situations in which the necessary training
data is not available, or where a proprietary model must be treated as a
’black box’. In either case, the ability to perform knowledge-based re-
optimization of the outputs of arbitrary models may be desirable.

In this report, we investigate one variant of this approach, using Semantic
Consistency Matrices. Basing our work on an implementation by Lemens et al.
[Lemmens2021] of the algorithm described by Fang et al. [Fang2017], we
evaluate the effectiveness of this method on the Cityscapes dataset, and
introduce an additional method for generating semantic consistency matrices
which acts as a compromise between the already existing methods. Furthermore,
we study the effectiveness of the different knowledge-aware re-optimization
methods on the performance of the more recent Deformable Transformer (DETR)
architecture.

##  2 Related Work

Object detection is an active field of research with numerous immediate
applications. It should therefore come as no surprise that several different
approaches are currently under active investigation.

Convolutional neural networks remain a fundamental part of object detection
models, but many recent models augment their capabilities by introducing
different additional mechanisms. For example, the Faster R-CNN model [Ren2016]
is widely used as a backbone for other object detection models. It makes use
of an RPN or region proposal network to identify regions of interest before
applying a convolutional neural network to the actual task of recognition on
these regions. By contrast, the DETR architecture [Carion2020] prepends its
own CNN backbone to another model, composed of encoding and decoding
transformers, such that the CNN acts as a dimensionality reduction mechanism.

Finally, the YOLO family of models [redmon2018yolov3] typifies the one-shot
object detection approach, named in contrast to the two stages of an FRCNN-
like detector with separate region proposal and classification stages. A YOLO-
style model, as indicated by expanding the acronym to "You Only Look Once",
instead uses a single stage.

All these models, however, suffer from a limitation when compared to human
perception: While they are able to extract large amounts of relevant
information from input images, they do not generally leverage external,
contextual knowledge such as causal or semantic relations between different
depicted objects (whereby, e.g., an image containing cars is more likely to
also contain an omnibus than a lobster) and image metadata. Given the
intuitive nature of this approach, it should come as no surprise that the
question of using additional knowledge in object detection has been considered
from many different angles beyond the one at the core of this paper.

Liu et al.(2021) [Liu2021] use a similarity network to measure the pairwise
semantic similarity between objects, in a method somewhat similar to the one
discussed in this report. However, unlike semantic consistency, this is not
related to the likelihood of co-occurence of concepts. Instead, semantic
similarity is a measure of how likely any two objects are to belong to the
same class, no matter which class this may actually be. Others, such as Zhu et
al. [Zhu2021SemanticRR] have put forward architectures capable of more
sophisticated integration not only of semantic, but also spatial information
to improve the performance of object detection models. Chen et al. [Chen2020]
demonstrate that such an approach can yield improved performance especially on
small objects.

Menglong et al. (2019) [Menglong2019] applied a related approach to the task
of image classification or object recognition. For this, they described a
method in which labels assigned by one or several existing classifiers are
used to construct a knowledge graph, in which semantic similarity between any
two categories is assigned on the basis of the frequency at which these
classifiers confuse them. This information is then used to refine image
classification by adjusting the classifier’s output to better reflect the
domain knowledge gathered, assigning greater or lower confidence to outputs
according to their plausibility.

The construction of knowledge graphs is moving beyond simple text databases,
and into the field of Multi-Model Knowledge Graphs, organically integrating
image and text data into a single structure and explicitly connecting images
to explicit visual properties, as in Zhu et al. [zhu2022multi].

Making use of the fact that many images available today are tagged with
temporal and geographic metadata, MacAodha et al. [MacAodha2019PresenceOnlyGP]
take a different direction and extend conventional object detection models to
consider this metadata as additional context. This is accomplished through the
use of an assessment of co-occurrence of objects in given times and places.

Von Rueden et al. [Rueden2019] introduce the notion of ’informed machine
learning’ as an umbrella term for all types of machine learning making use of
external information sources, and present a useful general taxonomy.

Finally, Castellano et al. [castellano2022leveraging] demonstrate a novel area
of application of these techniques, by applying a combination of knowledge
graphs and deep learning techniques to the analysis of artworks, and provide a
related knowledge graph.

##  3 Dataset

We evaluated the Knowledge-Aware Re-Optimization method on the Cityscapes
Dataset [Cordts2016], which was originally designed for scene understanding as
applied to urban scenes. It contains 5000 stereoscopic images with detailed
annotations at the instance and pixel levels, as well as additional
annotations with coarser annotations. In our case, we consider the instance-
level annotations.

In order to include depth information necessary for some applications, the
dataset contains pairs of stereoscopic images, i.e. pairs of images recorded
simultaneously by two cameras placed at a slight distance from each other but
facing the same direction. The left- and right-hand image thus differ
slightly, as the perspective of the left and right eye would differ in human
vision, enabling depth perception. As depth information was outside the scope
of our experiments, we followed the convention adopted by the implementation
by T. Beemelmanns [Beemelamanns2022]for dealing with cityscapes dataset
formatting: Of each pair of stereoscopic images, only the left-hand one was
retained, thus providing us with a dataset of single images.

Class | Training | Validation
---|---|---
Person | 12044 (33.1%) | 3450 (34.4%)
Rider | 1080 (2.9%) | 542 (5.4%)
Car | 19113 (52.6%) | 4378 (43.7%)
Truck | 312 (0.8%) | 93 (0.9%)
Bus | 245 (0.6%) | 98 (0.9%)
Train | 118 (0.3%) | 23 (0.2%)
Motorcycle | 492 (1.3%) | 148 (1.4%)
Bicycle | 2903 (7.9%) | 1281 (12.7%)
Total | 36307 | 10013

Table 1:  Distribution of classes across the partial Cityscapes dataset

The number of instances in the training and validation components of the
dataset is given in table 1. This table also illustrates an issue of the
dataset, namely the large class imbalance present across both the training and
validation sets. Thus, the number of e.g. pedestrians (designated as ’Person’
in the dataset label but distinct from a ’Rider’ on a bicycle or motorcycle)
in the validation set exceeds the number of trains or busses by two orders of
magnitude. The classes corresponding to Trucks, Buses and Trains are
especially underrepresented. However, the proportion of each class relative to
the total number of instances in both sets is consistent except for the
classes ’Car’ and ’Bicycle’, where the training and validation sets differ
from each other by 5 percentage points.

##  4 Methodology

In this paper, we adapt the concept of semantic consistency [Fang2017] for the
domain of autonomous driving to improve the performance of object detection
models in traffic scenes by using environmental knowledge. This knowledge-
aware re-optimization framework relies on a semantic consistency matrix. For
any two concepts, this matrix gives a semantic consistency value, which is an
indication of how likely instances of these two concepts are to occur
simultaneously in an image. This includes the semantic consistency of a
concept with itself, as multiple instances of the same concept in a single
image are more likely for some concepts than others.

The framework as such is independent of any particular object detection method
or implementation, instead treating the object detection model as a black box.
The model’s output is adjusted to better fit the semantic consistency matrix,
with the scores for each class being raised or lowered to increase the overall
consistency of the detection. To regularize the model and avoid being overly
restricted in cases which do not conform to the expected distribution, very
large changes in class scores are also penalized. For our experiments, we
employed FRCNN [Ren2016] and DETR [Carion2020] as our object detection model.

The main challenge for this approach lies in obtaining the semantic
consistency matrix, $S$. We investigated three different methods, two from the
original paper of Fang et al.[Fang2017] on re-optimization and one novel
hybrid method.

###  4.1 External Knowledge Sources

####  4.1.1 Frequency-Based Semantic Consistency

The frequency-based method creates a semantic consistency matrix directly from
the annotated training data also used for the backbone model, and therefore
requires this training set to be available. Fang et al.[Fang2017] note that
this may be a drawback in practical applications where this training data may
be proprietary or otherwise unavailable, and also report a lower performance
for this method. Conversely, however, it may be useful for situations where no
suitable knowledge graph is available. The frequency-based method for
determining semantic consistency is based on co-occurrences of concepts. Thus,
it is assumed that objects which frequently occur together in the training set
have a higher degree of semantic consistency, following the equation

$S_{l,l^{\prime}}=max(log\frac{n(l,l^{\prime})N}{(n(l)n(l^{\prime}))},0)$

where $n(l)$ and $n(l^{\prime})$ are the individual frequencies of the
respective concepts $l,l^{\prime}$, $n(l,l^{\prime})$ denotes the number of
co-occurrences of the two concepts and $N$ is the total number of instances.
As the number of co-occurrences of multiple instances of one and the same
class may also be relevant, a special ’handshake’ equation is used in this
case, following the same conventions:

$n(l,l)=n(l)\frac{n(l)-1}{2}$

####  4.1.2 Knowledge Graph-Based Semantic Consistency

Knowledge Graph-Based Semantic Consistency is the second method described by
Fang et al.[Fang2017] It is capable of being used even on pre-trained models
where no training data is available, and can give better performance than the
frequency-based method.

To this end, an external knowledge graph whose vertices represent different
concepts, and whose edges connect the semantically related concepts is
required. For processing the knowledge graph, we followed the implementation
of Lemmens et al. [Lemmens2021] , who provided some key clarifications to the
original paper. Their method involves two stages.

First, the pre-existing graph, in our case ConceptNet5 [Speer2012], is cropped
to include only positive relations (i.e., remove relations such as antonyms
and contradictions) and is limited, without loss of generality, to the
English-language versions of concepts.

Then, the semantic consistency matrix is derived from this knowledge graph
using a series of random walks starting from each concept of interest, i.e.
corresponding to a desired label of the object detection model. The random
walk then traverses the graph, but also has a low probability of resetting to
the original node with every step to avoid remaining stuck in local groups
(Random Walk with Restart, RWR) [Tong2006] The probability of reaching any
given other node from the starting node eventually converges towards a steady-
state after a sufficient number of iterations.

Figure 1: The basic principle of knowledge-based re-optimization. Using a
semantic consistency matrix, the model output $p$ is changed to a re-optimized
output $\hat{p}$

####  4.1.3 Hybrid Semantic Consistency

In this paper, we introduce another method for knowledge-based re-
optimization, derived from a combination of the two approaches described
above. Intuitively, the semantic consistency as derived from an external
knowledge graph will, for a well-designed graph, correspond to the degree to
which these concepts are related in the corpus the graph is based on. However,
this may not be ideal in every situation. If a model is to be applied in a
specific use-case instead of general-purpose object detection, the class
distribution it encounters may be different from that suggested by a
consistency matrix derived from a generic knowledge-graph. As a simple
example, a knowledge graph derived from an encyclopedia might contain
relatively few mentions of pedestrians, but greater emphasis on trains,
whereas an object detection model intended for self-driving vehicles is likely
to encounter the opposite situation.

However, as the knowledge graph-based method produced performance superior to
the frequency-based method overall, we investigated the possibility of
creating a knowledge graph tailored to our dataset based on frequency data.
Our assumption is that a well-constructed dataset can be more relevant to the
area of application of the model than a general knowledge model. Our approach
uses the following steps:

  * •

We generate a matrix $M_{1}$ of co-occurrences for each class across our
dataset.

  * •

We generate a knowledge graph $G$ based on $M_{1}$ by creating a vertex and
concept for each label present in the co-occurrence matrix, and connect them
when the corresponding number of co-occurrences exceeds a threshold $\gamma$.

  * •

A random walk is performed on this new graph $G$, and the semantic consistency
matrix is generated as in the case of ordinary graph-based semantic
consistency.

We thus arrive at a semantic consistency matrix with the same structure to
that produced by other methods.

###  4.2 Knowledge-Based Re-optimization

Having generated our semantic consistency matrix $S$, we now perform the
actual re-optimization. This is done by finding the minimum of a loss function
accounting for both semantic consistency and the original input.

Formally, given two bounding boxes $b,b^{\prime}$, we call
$P_{b,l},P_{b^{\prime},l^{\prime}}$ the probability returned by the model for
a label $l$ resp. $l^{\prime}$ for either box. The loss function to be
minimized is given by the equation below, where $L$ is the number of concepts
and $B$ number of bounding boxes.

$\displaystyle
E(\hat{P})=(1-\epsilon)\sum^{B}_{b=1}\sum^{B}_{b^{\prime}=1,b^{\prime}\neq
b}\sum^{L}_{l=1}\sum^{L}_{l^{\prime}=1}S_{l,l^{\prime}}(\hat{P}_{b,l}-\hat{P}_{b^{\prime},l^{\prime}})^{2}$
(1)
$\displaystyle+\epsilon\sum^{B}_{b=1}\sum^{L}_{l=1}BS_{l,*}(\hat{P}_{b,l}-P_{b,l})^{2}$
(2)

This function can be decomposed into a sum of two terms, weighted by the
parameter $\epsilon\in(0,1)$, which must be determined in practice for each
dataset. The first term in equation 1 effectively demands that the total
square error between the final output for semantically similar concepts be
minimized. The semantic consistency acts like a weight for the nested sum
operations, meaning that differences between wholly orthogonal concepts
($S_{l,l^{\prime}}=0$) will have no effect at all on the final value.

The second term, in equation 2, imposes a cost for overly large deviations
from the backbone model’s output. The square error between the final output
$\hat{P}$ and the model’s output $P$ is modified by a coefficient and added
for all classes and bounding boxes, effectively acting as a check on the re-
optimization. The cost function given in 1 is then minimized. This is
accomplished by setting the gradient of $E(\hat{P})$ w.r.t. $\hat{P}_{b,l}$ to
zero.

###  4.3 Metrics

The implementation of Lemmens et al. [Lemmens2021] made use of an Area-Under-
Curve (AUC) style of average precision calculation, computing class-wise
average precision for a range of recall thresholds running from $0$ to $1.0$.
We retained this system, as it is especially useful for automatic parameter
searches, where it implicitly balances the priority of precision and recall
even if only the former is explicitly targeted.

###  4.4 Experiment Structure

As a first step, we established a baseline of performance metrics by measuring
the performance of both architectures on the validation set of the cityscapes
dataset. We then attempted to find an optimal combination of re-otimization
hyperparameters. As there are several such parameters, we resorted to an
automatic parameter search using Optuna [optuna_2019], covering the weight
parametre $\epsilon$ discussed above, but also the number of adjacent boxes
and classes considered for re-optimization, and an internal score threshold
for re-optimized outputs. As the precision-recall curve metrics used already
encompassed recall in computing precision, we verified that choosing accuracy
as the sole optimization target had no negative effect and carried out our
experiments on this base.

Tests showed that the best results on the unoptimized baseline were obtained
when considering the 100 highest-scoring detections. As this criterion was
also used in the original paper [Fang2017], we opted to follow this
convention. However, we also considered an alternative method, using a score
threshold rather than a static number of detections, and the use of different
values for both number of detections and threshold. The results are detailed
in section 5.

###  4.5 Challenges

Lemmens et al. [Lemmens2021] have already pointed out some technical
difficulties which hindered their re-implementation based on the original
paper. These difficulties arose primarily from missing details, such as the
exact stage of the pipeline at which the top-scoring detections are selected.
During our investigation, we also observed that, while the re-optimization
model is mathematically capable of treating the backbone model as a black box,
practical implementations required additional care.

While investigating different state-of-the-art, or recent implementations, we
found that many object detection frameworks differ in terms of input formats,
output formats, or both, in a way which complicated integration with
knowledge-based re-optimization even where an explicit API is provided. As an
example, yolov3 [redmon2018yolov3], as implemented in the mmdetection
framework, requires image input to be given as a file path, and returns a list
of bounding boxes and confidence scores per class, while the implementation of
Faster R-CNN built into pytorch requires image inputs in the form of pytorch
tensors, and returns a dictionary holding labels, scores and bounding boxes in
continuous tensors. It is therefore necessary to implement explicit format
conversions for each architecture.

An unrelated but significant difficulty stemmed from the large imbalance of
classes within the cityscapes dataset. While FRCNN was not visibly affected by
this, we found that DETR suffered from degraded performance on the
underrepresented classes (see section 5.2). Based on the findings discussed
below, it seems probable that a larger, more balanced training set or data
augmentation focused on the least-represented classes would have greatly
strengthened the results achieved with DETR.

##  5 Results

###  5.1 Faster-RCNN

In this section, we present and analyze the key results obtained with the
Faster R-CNN architecture. These results were computed on the 100 top-scoring
detections, and are given for the best-performing configuration of
hyperparameters for the given experiment.

Figure 2: Examples of un-optimized output (red) being adjusted to a re-optimized label (green), with the corresponding confidence score. (Faces and license plates redacted manually) Figure 3: Example of spurious detections in the un-optimized output (red) being removed in the re-optimized output (green), with the corresponding confidence score.  Re-Op | mAP@100 | Recall@100 | Small | Medium | Large
---|---|---|---|---|---
Baseline | 25.35 | 35.60 | 12.26 | 34.77 | 57.43
Hybrid | 25.01 (-0.34) | 35.87 (+0.27) | 11.94 | 35.00 | 58.40
KG | 25.22 (-0.13) | 35.88 (+0.28) | 11.93 | 35.40 | 57.93
Freq. | 25.40 (+0.04) | 35.84 (+0.24) | 12.21 | 35.11 | 57.80

Table 2: Effects of different re-optimization methods on the FRCNN model’s
performance. Numbers in bracket give the difference from the baseline.

All methods, as shown in table 4, provided some gain in recall, with the
largest gains being made using the knowledge graph-based semantic consistency
matrix. However, the knowledge-graph (KG) method also resulted in some
decrease in mAP. The hybrid method fared worse in both respects, with smaller
recall gains and greater mAP losses. Finally, the frequency-based method
resulted in an increase in recall slightly smaller than the previous methods,
but also no loss (and, in fact, a marginal gain) of mAP. This is broadly
consistent with the results of Lemmens et al.[Lemmens2021]: Although reported
an decrease in mAP and an increase in recall with both knowledge- and
frequency-based methods on MSCOCO, the pattern is similar to the extent that
the knowledge-based method resulted in greater reported gains and losses
respectively. The effects of the re-optimization, both positive and negative,
reported by Lemmens et al. are on the same order of magnitude as ours, with
the overall effect not exceeding 1%. This is substantially less than what was
originally reported in the paper of Fang et al. [Fang2017], where recall gains
reported could reach 3%.

Method | AP | Recall
---|---|---
Baseline | KG | Hybrid | Freq | Baseline | KG | Hybrid | Freq
Person | 25.29 | 24.92 | 24.42 | 25.32 | 33.51 | 33.78 | 33.94 | 33.48
Rider | 29.76 | 29.40 | 29.73 | 29.75 | 41.31 | 40.26 | 40.18 | 41.25
Car | 44.28 | 43.74 | 42.73 | 44.26 | 49.87 | 50.67 | 50.76 | 49.82
Truck | 19.36 | 19.45 | 19.46 | 19.49 | 31.51 | 32.58 | 32.37 | 32.69
Bus | 36.83 | 36.86 | 36.90 | 36.86 | 46.37 | 46.84 | 46.63 | 46.73
Train | 12.14 | 12.15 | 11.82 | 12.15 | 23.48 | 23.91 | 23.91 | 23.91
Motorcycle | 14.87 | 15.00 | 15.11 | 15.04 | 27.18 | 27.65 | 27.99 | 27.72
Bicycle | 20.28 | 20.26 | 19.95 | 20.33 | 31.21 | 31.35 | 31.19 | 31.14

Table 3: Class-wise breakdown of Precision and Recall using Hybrid Re-
Optimization with an FRCNN backbone

Examining the results in greater detail, we find that the average values
discussed in the earlier do not arise from a homogeneous change to all classes
in the same direction. The knowledge graph-based and hybrid methods do give
poorer mAP values on some classes, but actually yield increased precision on
others. Specifically, it appears that the classes making up a smaller
percentage of the dataset, such as buses and motorcycles, do benefit from this
method, whereas precision scores for the classes with the greatest support in
the dataset, Person and Car, also show the greatest decrease in precision. The
frequency-based method was more consistent, with increases in precision across
the board, with the exception of the Car class. The latter effect may be
tentatively explained the fact that this class makes up a substantially larger
portion of the training set (53.6 %), from which the frequency-based
consistency matrix is derived, than of the validation set (43.7%), on which it
is applied.

###  5.2 Deformable Transformers

In this section, we consider the results achieved with the different re-
optimization methods applied to an implementation of the DETR architecture
[Carion2020] on the Cityscapes dataset. As before, we will focus primarily on
metrics computed for the 100 highest-scoring detections.

Performance for DETR was typically lower than for FRCNN, primarily due to
slightly lower recall. However, the effects of the knowledge-aware re-
optimization largely followed the same pattern.

Method | mAP@100 | Recall | Small | Medium | Large
---|---|---|---|---|---
Baseline | 21.84 | 35.48 | 17.46 | 44.63 | 62.71
Hybrid | 21.81 (-0.033) | 35.61 (+0.13) | 17.46 | 44.71 | 63.03
KG | 21.81 (-0.031) | 35.63 (+0.15) | 17.47 | 44.71 | 63.18
Freq. | 21.58 (-0.038) | 35.70 (+0.14) | 17.46 | 44.70 | 63.06

Table 4: Effects of different re-optimization methods on the DETR model’s
performance. Numbers in bracket give the difference from the baseline.

As with Faster R-CNN on Cityscapes and COCO[Lemmens2021], the same pattern as
on FRCNN is reproduced, albeit weakly. One oddity of the results on DETR,
which we discovered by accident and explored for the sake of completeness, lay
in an unexpected behaviour of the $\epsilon$ hyperparameter. $\epsilon$
governs the relative weight of the semantic consistency values and the
backbone’s output. It is therefore normally constrained to a range of
$0<\epsilon<1$. However, we found that that using a value of $\epsilon>1$ can
in fact also lead to an increase in model performance for certain parameter
combinations. The anomalous results obtained with $\epsilon>1$, consistently
display relatively large increases in precision at the expense of recall,
which, combined with the weaker gains made by the ordinary ($\epsilon<1$)
method, suggests that some backbone models may benefit more from treating the
knowledge-aware component as a ’pessimizer’ with an effectively negative
weight. This effect did not occur with the Faster-RCNN architecture when we
repeated the experiments using this architecture under the same conditions,
with the overall change in performance being an order of magnitude smaller
than the one achieved with the more orthodox approach. These results are
included in Appendix 1.

Class/Method | AP | Recall
---|---|---
Baseline | KG | Hybrid | Freq | Baseline | KG | Hybrid | Freq
Person | 28.58 | 28.54 | 28.40 | 28.53 | 40.75 | 40.77 | 40.10 | 40.77
Rider | 27.53 | 27.59 | 27.74 | 27.59 | 37.22 | 37.24 | 36.94 | 37.23
Car | 49.03 | 48.95 | 49.01 | 48.95 | 57.66 | 57.71 | 57.23 | 57.71
Truck | 11.88 | 11.88 | 12.32 | 11.91 | 29.73 | 30.57 | 28.51 | 29.46
Bus | 17.13 | 17.04 | 18.33 | 17.05 | 29.25 | 29.42 | 29.40 | 29.39
Train | 2.52 | 2.55 | 3.94 | 2.53 | 23.77 | 23.84 | 23.19 | 23.84
Motorcycle | 14.70 | 14.62 | 15.44 | 14.61 | 26.76 | 26.76 | 26.74 | 26.76
Bicycle | 23.36 | 23.33 | 23.71 | 23.33 | 38.71 | 38.72 | 37.88 | 38.71

Table 5: Class-wise breakdown of Precision and Recall using Hybrid Re-
Optimization with a DETR backbone

The apparent anomaly of the very low precision value for the ’Train’ class may
be explained by the fact that it is the least represented class in the
training set with 118 instances, but especially in the validation set, where
only 21 trains occur. Therefore, results for this class may not be
significant. Conversely, when comparing the performance of Faster RCNNN in
table 3 with that of DETR in table 5, we find that DETR’s precision and recall
for the highest-scoring classes ’Person’ and ’Car’ match or surpass FRCNN’s
performance despite lower average values. As shown in table 1, these are also
the most represented classes in the dataset.

###  5.3 Behaviour of Confidence Scores

In view of the comparatively limited effect of re-optimization on model
performance, we decided to investigate the way in which it modifies the
confidence scores assigned by the backbone model in greater detail. The
behaviour of the confidence scores, it was hoped, would help us to ensure that
the model was functioning properly, but might also highlight differences in
the behaviour of the FRCNN model as applied on the cityscapes dataset and the
original MS-COCO dataset.

For each configuration in table 6, we give descriptive statistics for the 100
results with the highest confidence scores (@100) for each image, consistent
with the overall evaluation methodology. As may be seen in the table 6, the
results using the FRCNN architecture as a backbone are largely consistent
between the two datasets, with some differences caused by different re-
optimization parameters. The DETR backbone displays a tendency towards more
extreme changes in score, but with an average change in confidence score
remaining very close to zero overall. The fact that the standard deviation is
larger than for FRCNN, but still far from the extreme values, suggests that
the minima and maxima of confidence score changes for DETR are outliers rather
than the rule.

Backbone | Dataset | Method | Mean Change | Maximum | Minimum | Std. Dev.
---|---|---|---|---|---|---
FRCNN | COCO | KG | -0.14 | 0.04 | -0.25 | 0.08
FRCNN | COCO | Freq. | -0.14 | 0.03 | -0.25 | 0.08
FRCNN | Cityscape | KG | -0.01 | 0.02 | -0.05 | 0.02
FRCNN | Cityscape | Freq. | -0.02 | 0.06 | -0.1 | 0.03
FRCNN | Cityscape | Hybrid | -0.08 | 0.02 | -0.4 | 0.13
DETR | Cityscape | KG | 0.00 | 0.94 | -0.94 | 0.29
DETR | Cityscape | Freq. | -0.01 | 0.89 | -0.94 | 0.28
DETR | Cityscape | Hybrid | -0.00 | 0.93 | -0.94 | 0.29

Table 6: Statistical analysis of changes in confidence scores for different
datasets, backbone architectures and re-optimization methods.

##  6 Conclusion

In this report, we investigated the possibility of using knowledge-aware re-
optimization for object detection in on the Cityscapes dataset, applying the
re-optimization model to a problem in the domain of autonomous driving. These
experiments were carried out using three different sources of external
knowledge, including a novel hybrid knowledge graph generated from frequency
data obtained from the training set. We reported results for two different
architectures.

Despite technical challenges in adapting the re-optimization model to
different backbone architectures, we are able to apply the methods described
in the original paper to the new configurations described above. We were able
to achieve marginal but consistent increases in recall for both FRCNN and DETR
on the cityscapes dataset with a favorable trade-off in precision.

##  Appendix A Appendix

The results of the experiments using a value of $\epsilon>1$ are collected
here. Note that the baseline is unaffected by this and is, therefore,
identical to the values given in the main part of this report. As some of
these configurations were unable to achieve any mAP gain on FRCNN, values
exhibiting a larger increase in recall were instead selected so as to show
some effect of the model. However, as these effects are marginal in size and
lack a robust mathematical foundation, we must caution against placing an
excessive burden of interpretation on them.

###  A.1 FRCNN

Re-Op | mAP | Recall | Small | Medium | Large
---|---|---|---|---|---
Baseline | 25.35 | 35.60 | 12.26 | 34.77 | 57.43
KG | 24.94 (-0.4) | 35.68 (+0.08) | 12.05 | 35.06 | 57.60
Hybrid | 25.27 (-0.07) | 35.63 (+0.03) | 12.19 | 34.88 | 57.43
Freq. | 25.38 (+0.02) | 35.59 (-0.01) | 12.26 | 34.75 | 57.44

Table 7: Effects of different re-optimization methods on the FRCNN model’s performance, with $\epsilon>0$. Numbers in bracket give the difference from the baseline. Method | AP | Recall
---|---|---
Baseline | Freq | Baseline | Freq
Person | 25.29 | 25.31 | 33.51 | 33.54
Rider | 29.76 | 29.83 | 41.31 | 41.25
Car | 44.28 | 44.33 | 49.87 | 49.87
Truck | 19.36 | 19.40 | 31.51 | 31.51
Bus | 36.83 | 36.83 | 46.73 | 46.73
Train | 12.14 | 12.13 | 23.48 | 23.48
Motorcycle | 14.87 | 15.0 | 27.18 | 27.18
Bicycle | 20.28 | 20.23 | 31.21 | 31.19

Table 8: Class-wise breakdown of Precision and Recall using Hybrid Re-
Optimization with an FRCNN backbone, with $\epsilon>1$

###  A.2 DETR

Re-Op | mAP | Recall | Small | Medium | Large
---|---|---|---|---|---
Baseline | 21.84 | 35.48 | 17.46 | 44.63 | 62.71
Hybrid | 22.08 (+0.24) | 35.50 (+0.02) | 17.42 | 44.51 | 63.03
KG | 21.81 (+0.51) | 35.00 (-0.48) | 17.15 | 44.06 | 62.45
Freq. | 22.19 (+0.35) | 35.36 (-0.11) | 17.36 | 44.37 | 62.82

Table 9: Effects of different re-optimization methods on the DETR model’s performance. Numbers in bracket give the difference from the baseline. Method | AP | Recall
---|---|---
Baseline | KG | Hybrid | Freq | Baseline | KG | Hybrid | Freq
Person | 28.58 | 28.54 | 28.65 | 28.40 | 40.75 | 40.10 | 40.69 | 40.53
Rider | 27.53 | 27.59 | 27.68 | 27.74 | 37.22 | 36.94 | 37.19 | 37.13
Car | 49.03 | 48.95 | 49.08 | 49.01 | 57.66 | 57.23 | 57.65 | 57.52
Truck | 11.88 | 11.88 | 12.19 | 12.32 | 29.73 | 28.51 | 29.77 | 29.27
Bus | 17.13 | 17.04 | 17.65 | 18.33 | 29.25 | 29.40 | 29.42 | 29.42
Train | 2.52 | 2.55 | 2.77 | 3.94 | 23.77 | 23.19 | 23.91 | 23.84
Motorcycle | 14.70 | 14.62 | 14.96 | 15.44 | 26.76 | 26.74 | 26.76 | 26.74
Bicycle | 23.36 | 23.33 | 23.69 | 23.71 | 38.71 | 37.88 | 38.65 | 38.49

Table 10: Class-wise breakdown of Precision and Recall using Hybrid Re-
Optimization with a DETR backbone