#  On The Dales–Żelazko Conjecture for Beurling Algebras on Discrete Groups

Jared T. White  Jared T. White, School of Mathematics and Statistics, The Open
University, Walton Hall, Milton Keynes MK7 6AA, United Kingdom
<EMAIL_ADDRESS>
(Date: 2023)

###### Abstract.

Let $G$ be a group which is either virtually soluble or virtually free, and
let $\omega$ be a weight on $G$. We prove that, if $G$ is infinite, then there
is some maximal left ideal of finite codimension in the Beurling algebra
$\ell^{1}(G,\omega)$ which fails to be (algebraically) finitely generated.
This implies that a conjecture of Dales and Żelazko holds for these Banach
algebras. We then go on to give examples of weighted groups for which this
property fails in a strong way. For instance we describe a Beurling algebra on
an infinite group in which every closed left ideal of finite codimension is
finitely generated, and which has many such ideals in the sense of being
residually finite dimensional. These examples seem to be hard cases for
proving Dales and Żelazko’s conjecture.

###### Key words and phrases:

Banach algebra, virtually free group, virtually soluble group, maximal left
ideal, finitely-generated, weight, Beurling algebra

###### 2020 Mathematics Subject Classification:

43A20, 46H10 (primary); 20E05, 20E08, 20F16 (secondary)

The Open University

##  1\. Introduction

In this article we explore some connections between the properties of a
discrete group $G$ and the properties of the left ideals of its Beurling
algebras $\ell^{1}(G,\omega)$. We shall show that the existence or not of a
weight $\omega$ on $G$ such that $\ell^{1}(G,\omega)$ has a finitely-generated
maximal left ideal of finite codimension depends on properties of the group
$G$. One of our key methods will be based on weights defined in terms of word
length with respect to a given generating set for the group.

We begin with some background and motivation. Let $A$ be an infinite
dimensional unital Banach algebra, and let $I$ be a closed left ideal of $A$.
There is a natural tension between finite generation of $I$ and various ways
that $I$ could be “large”, such as being maximal or having finite codimension.
It is this idea that motivates the following conjecture of Dales and Żelazko
[6]:

######  Conjecture 1.1 (Dales–Żelazko).

Let $A$ be an infinite dimensional unital Banach algebra. Then $A$ has a
maximal left ideal which is not finitely generated.

The conjecture has since attracted some attention, and is known to be true for
various special classes of Banach algebras, for example commutative Banach
algebras [7], C*-algebras [3], $\ell^{1}(G)$ for a group $G$ [10], and the
algebra of bounded linear operators on a large class of Banach spaces [5],
including the reflexive Banach spaces [11, Corollary 2.2.7]. At the time of
writing no counterexample is known.

In [10] the author showed that the Dales–Żelazko Conjecture holds for Beurling
algebras on discrete groups, provided that the weight is sufficiently well-
behaved. In the present work we instead consider arbitrary weights but a
restricted class of groups. Our first result is the following.

######  Theorem 1.2.

Let $G$ be an infinite group which is either

  1. (i)

a virtually soluble group, or

  2. (ii)

a virtually free group,

and let $\omega$ be an arbitrary weight on $G$. Then $\ell^{1}(G,\omega)$
contains a maximal left ideal of finite codimension which is not finitely
generated. In particular $\ell^{1}(G,\omega)$ satisfies the Dales–Żelazko
conjecture.

One innovation of the present work on [10] is a lemma which allows us to go up
finite-index group extensions. Then the fact that Dales and Żelazko’s
conjecture is known to hold in the commutative case means that it holds for
any Beurling algebra on an abelian group, regardless of the weight. Combining
these two ideas will prove the more general Proposition 3.3, which will imply
Theorem 1.2.

The results in [10], as well as Theorem 1.2, establish the Dales–Żelazko
Conjecture by finding examples of finite-codimension left ideals. These left
ideals are usually easier to work with than those of infinite codimension.
However, as we shall see, a Beurling algebra need not have a finite-
codimension maximal left ideal that fails to be finitely generated. Indeed,
our next result shows that Beurling algebras can have starkly different
behaviour to that displayed in Theorem 1.2. First of all, it can happen that
there is only one finite-codimension maximal left ideal.

######  Theorem 1.3.

Let $G$ be a finitely-generated, infinite simple group. Fix a finite,
symmetric generating set $X$ for $G$, and let $\omega(t)=2^{|t|_{X}}\ (t\in
G)$. Then the only proper, finite-codimension left ideal of
$\ell^{1}(G,\omega)$ is the augmentation ideal $\ell^{1}_{0}(G,\omega)$, and
it is finitely generated.

On the other hand, it can happen that the Beurling algebra has “many” finite-
codimension maximal left ideals, all of which are finitely generated. We do
not know of any other example of an infinite dimensional unital Banach algebra
satisfying (i) and (ii).

######  Theorem 1.4.

Let $G$ be a finitely-generated, just infinite, residually finite group, which
is not a linear group. Fix a finite, symmetric generating set $X$ for $G$, and
set $\omega(t)=2^{|t|_{X}}\ (t\in G)$. Then $\ell^{1}(G,\omega)$ has the
following properties:

  1. (i)

the intersection of the finite-codimension, maximal left ideals of
$\ell^{1}(G,\omega)$ is $\\{0\\}$;

  2. (ii)

every finite-codimension left ideal is finitely generated.

Examples of groups satisfying this theorem include finitely-generated, just
infinite branch groups, such as the Grigorchuk group and the Gupta–Sidki
$p$-groups; it follows from [1, Corollary 7] that branch groups cannot be
linear. For background on branch groups and their relationship to the theory
of just infinite groups we refer the reader to [2].

Note that in part (ii) of Theorem 1.4 we do not assume that the left ideals
are closed. However, as we shall remark after the proof, every finite
codimension left ideal of this Beurling algebra is in fact closed. Likewise,
in Theorem 1.3 the augmentation ideal is the only proper finite-codimension
left ideal, closed or otherwise.

These two theorems entail that, if the Dales–Żelazko conjecture is true for
Beurling algebras, then in order to prove it one needs to look beyond finite-
codimension left ideals. We have been unable to resolve the conjecture for
these examples.

##  2\. Notation and Terminology

We now fix our notation and terminology. Let $A$ be a unital Banach algebra,
and let $I$ be a left ideal of $A$. We say that $I$ is finitely generated if
there exists $n\in\mathbb{N}$ and $a_{1},\ldots,a_{n}\in I$ such that
$I=Aa_{1}+\cdots+Aa_{n}$. We shall usually study left ideals that are closed,
but note that we do not take any closure on the right hand side in our
definition of finite generation. Recall that a maximal left ideal in a unital
Banach algebra is automatically closed, and that a codimension-1 left ideal is
always maximal, although in general a finite-codimension left ideal need not
be closed.

By a weight on a group $G$ we mean a function $\omega\colon G\to[1,\infty)$
such that $\omega(st)\leq\omega(s)\omega(t)\ (s,t\in G)$ and $\omega(e)=1$. We
set

$\ell^{1}(G,\omega)=\left\\{f\colon G\to\mathbb{C}:\sum_{t\in
G}|f(t)|\omega(t)<\infty\right\\},$

which is a Banach algebra with multiplication given by convolution, which we
write as ‘$*$’, and the norm given by $\|f\|_{\omega}=\sum_{t\in
G}|f(t)|\omega(t)\ (f\in\ell^{1}(G,\omega)).$ Banach algebras of this form are
called Beurling algebras. We write $\mathbb{C}G$ for the set of finitely-
supported, complex-valued functions of $G$, which is a dense subset of every
Beurling algebra on $G$. Given $t\in G$ we write $\delta_{t}$ for the point-
mass at $t$, and given $f\in\ell^{1}(G,\omega)$ we write
$f^{t}:=\delta_{t}*f*\delta_{t^{-1}}$.

Given a group $G$, a weight $\omega$ on $G$, and a normal subgroup $N$, we
write $\widetilde{\omega}$ for the weight induced on the quotient $G/N$
defined by

(2.1) $\widetilde{\omega}(Nt)=\inf\\{\omega(st):s\in N\\}\qquad(t\in G).$

By [9, Theorem 3.7.13] the quotient map $q\colon G\rightarrow G/N$ extends to
a surjective bounded algebra homomorphism
$q\colon\ell^{1}(G,\omega)\to\ell^{1}(G/N,\widetilde{\omega})$, whose kernel
is given by

$J_{\omega}(G,N):=\left\\{f\in\ell^{1}(G,\omega):\sum_{s\in Nt}f(s)=0\ (t\in
G)\right\\}.$

In fact $\ell^{1}(G,\omega)/J_{\omega}(G,N)$ and
$\ell^{1}(G/N,\widetilde{\omega})$ are isometrically isomorphic as Banach
algebras. We define the augmentation ideal of $\ell^{1}(G,\omega)$ to be

$\ell^{1}_{0}(G,\omega):=J_{\omega}(G,G)=\left\\{f\in\ell^{1}(G,\omega):\sum_{t\in
G}f(t)=0\right\\}.$

Let $G$ be a group with finite generating set $X$. We write $|t|_{X}$ for the
word length of $t\in G$ with respect to $X$, and, given $r\in\mathbb{N}$, we
write

$B_{r}^{X}=\\{t\in G:|t|_{X}\leq r\\}.$

We also write $\dot{B}_{r}^{X}:=B_{r}^{X}\setminus\\{e\\}$. A generating set
$X$ is said to be symmetric if, for all $x\in X$, we have $x^{-1}\in X$. We
shall often insist that our generating sets are symmetric, both to ease
notation and to apply the results of [10]. Word length gives us an important
way to construct weights on the group $G$. Given $c>1$ we can define a weight
$\omega$ on $G$ by

$\omega(t)=c^{|t|_{X}}\qquad(t\in G).$

Such a weight is sometimes called a radial exponential weight. Weights of this
form with $c=2$ appear in two of our main results discussed above; the choice
$c=2$ has merely been made for concreteness, and the analogous results also
hold for arbitrary $c>1$.

Let $G$ be a group, and let $H$ be a subgroup. By a right/left transversal for
$H$ in $G$ we mean a set of right/left coset representatives respectively.
When $H$ is a normal subgroup every left transversal is a right transversal
and vice versa so we refer simply to a transversal for $H$ in $G$.

We write $\mathbb{F}_{n}$ for the free group on the set
$\\{a_{1},\ldots,a_{n}\\}$, where $a_{1},\ldots,a_{n}$ are just $n$ arbitrary
symbols. We say that a group $G$ is a linear group if it is isomorphic to a
subgroup of $GL_{n}(\mathbb{K})$, the group of invertible $n$-by-$n$ matrices
over a field $\mathbb{K}$. We say that an infinite group $G$ is just infinite
if all of its non-trivial quotients are finite.

##  3\. The Proofs

Our first task is to prove Theorem 1.2. We begin by proving a lemma that
allows us to use our understanding of a finite-index subgroup to study the
ambient group.

######  Lemma 3.1.

Let $G$ be a group, and let $\omega$ be a weight on $G$. Let $H$ be a finite-
index, normal subgroup of $G$, and suppose that $\ell^{1}(H,\omega|_{H})$ has
a closed finite-codimension left ideal which is not finitely generated. Then
so does $\ell^{1}(G,\omega)$.

###### Proof.

Let $I$ be a finite-codimension left ideal in $\ell^{1}(H,\omega|_{H})$ which
fails to be finitely generated. We consider $\ell^{1}(H,\omega|_{H})$ as a
subalgebra of $\ell^{1}(G,\omega)$. Let $n=[G:H]$, and let
$t_{1},\ldots,t_{n}$ be a transversal for $H$ in $G$. Define

$J=\left\\{\delta_{t_{1}}*f^{(1)}+\cdots+\delta_{t_{n}}*f^{(n)}:f^{(i)}\in I\
(i=1,\ldots,n)\right\\}.$

Note that $J$ is the $\ell^{1}$-direct sum of $\delta_{t_{i}}*I$ for
$i=1,\ldots,n$, so it is closed. We show that $J$ is a left ideal of
$\ell^{1}(G,\omega)$. Let $u\in G$ and for each $i=1,\ldots,n$ write
$ut_{i}=t_{j(i)}v_{i}$, where $v_{i}\in H\ (i=1,\ldots,n)$, and $j$ is a
permutation. Then given $f^{(i)}\in I\ (i=1,\ldots,n)$ we have

$\delta_{u}*\left(\sum_{i=1}^{n}\delta_{t_{i}}*f^{(i)}\right)=\sum_{i=1}^{n}\delta_{t_{j(i)}}*\delta_{v_{i}}*f^{(i)}\in
J,$

since each $\delta_{v_{i}}*f^{(i)}\in I$. It follows that $J$ is a left ideal
in $\ell^{1}(G,\omega)$.

We claim that $\operatorname{codim}J=[G:H]\operatorname{codim}I,$ which is
finite. Indeed, write $k=\operatorname{codim}I$, and suppose that
$g_{1},\ldots,g_{k}\in\ell^{1}(H,\omega|_{H})$ have the property that
$\\{g_{1}+I,\ldots,g_{k}+I\\}$ is a basis for $\ell^{1}(H,\omega|_{H})/I$.
Then it is easily checked that $\\{\delta_{t_{i}}*g_{j}+J:i=1,\ldots,n,\
j=1,\ldots,k\\}$ is a basis for $\ell^{1}(G)/J$.

Next we show that $J$ is not finitely generated. Assume towards a
contradiction that

$J=\ell^{1}(G,\omega)*f_{1}+\cdots+\ell^{1}(G,\omega)*f_{m},$

for some $m\in\mathbb{N}$, and some $f_{1},\ldots,f_{m}\in J$. Then each
$f_{i}$ may be written as

$f_{i}=\delta_{t_{1}}*f_{i}^{(1)}+\cdots+\delta_{t_{n}}*f_{i}^{(n)},$

for some $f_{i}^{(1)},\ldots,f_{i}^{(n)}\in I$. Let $g\in I$ be arbitrary.
Then in particular $g\in J$, so it may be written as
$g=h_{1}*f_{1}+\cdots+h_{m}*f_{m}$, for some
$h_{1},\ldots,h_{m}\in\ell^{1}(G,\omega)$. Again, we may write each $h_{i}$ as
$h_{i}=\delta_{t_{1}}*h_{i}^{(1)}+\cdots+\delta_{t_{n}}*h_{i}^{(n)}$, for some
functions $h_{i}^{(1)},\ldots,h_{i}^{(n)}\in\ell^{1}(G,\omega)$ supported on
$H$. We have

$h_{i}*f_{i}=\sum_{j,\,k=1}^{n}\delta_{t_{j}}*h_{i}^{(j)}*\delta_{t_{k}}*f_{i}^{(k)}=\sum_{j,\,k=1}^{n}\delta_{t_{j}t_{k}}*(h_{i}^{(j)})^{t_{k}^{-1}}*f_{i}^{(k)}=\sum_{j,\,k=1}^{n}\delta_{t_{j}t_{k}}*\phi_{ijk},$

where $\phi_{ijk}:=(h_{i}^{(j)})^{t_{k}^{-1}}*f_{i}^{(k)}$. Since $H$ is
normal, for all $i,j,k$ we have ${\rm supp\,}\phi_{ijk}\subset H$, so that
${\rm supp\,}\left\\{\delta_{t_{j}t_{k}}*\phi_{ijk}\right\\}\subset
t_{j}t_{k}H$. Since $g$ is supported on $H$ it must be equal to the sum of
those terms $\delta_{t_{j}t_{k}}*\phi_{ijk}$ which are supported on $H$. For
each $k=1,\ldots,n$, write $k^{\prime}\in\\{1,\ldots,n\\}$ for the natural
number satisfying $t_{k}^{-1}H=t_{k^{\prime}}H$. We see that

$g=g|_{H}=\sum_{k=1}^{n}\sum_{i=1}^{m}\delta_{t_{k^{\prime}}t_{k}}*(h_{i}^{(k^{\prime})})^{t_{k}^{-1}}*f_{i}^{(k)}.$

Hence, since $g$ was arbitrary, $I$ is generated by
$\left\\{f_{i}^{(k)}:i=1,\ldots,m,\ k=1,\ldots,n\right\\}$. This contradiction
concludes the proof. ∎

In the proof of the next lemma $\mathcal{B}(E)$ denotes the Banach algebra of
bounded linear operators on a Banach space $E$.

######  Lemma 3.2.

Let $A$ be a Banach algebra with a finite-codimension left ideal which is not
finitely generated. Then:

  1. (i)

$A$ contains a finite-codimension two-sided ideal which is not finitely
generated as a left ideal;

  2. (ii)

if $A$ is also unital, then $A$ contains a finite-codimension maximal left
ideal which is not finitely generated.

###### Proof.

(i) Let $I$ be a left ideal of finite codimension in $A$ which is not finitely
generated. Let $J$ be the kernel of the map $\theta\colon
I\rightarrow\mathcal{B}(A/I)$ given by $a\mapsto(x+I\mapsto ax+I)$. Then

$J=\\{a\in I:ax\in I\text{ for all }x\in A\\}$

and is a two-sided ideal contained in $I$ (in fact the largest one). The ideal
$J$ has finite codimension in $I$, since
$\dim(I/J)\leq\dim\operatorname{im}\theta\leq\dim\mathcal{B}(A/I)<\infty,$ and
hence also has finite codimension in $A$. In particular we can write $I=J+{\rm
span\,}\\{a_{1},\ldots,a_{n}\\}$, for some $n\in\mathbb{N}$, and some
$a_{1},\ldots,a_{n}\in I$. If $J$ were finitely generated as a left ideal,
then taking a finite generating set for $J$ together with $a_{1},\ldots,a_{n}$
would give a finite generating set for $I$. Hence $J$ cannot be finitely
generated.

(ii) This follows from part (i) and [6, Lemma 2.9]. ∎

We are now ready to prove the first of our main results.

######  Proposition 3.3.

Let $G$ be an infinite group with a finite-index normal subgroup $H$ which has
an infinite abelian quotient. Let $\omega$ be a weight on $G$. Then
$\ell^{1}(G,\omega)$ has a finite-codimension maximal left ideal that is not
finitely generated. In particular $\ell^{1}(G,\omega)$ satisfies the
Dales–Żelazko Conjecture.

###### Proof.

Since by [6] the Dales–Żelazko Conjecture holds in the commutative case, the
Banach algebra $\ell^{1}\left(H/H^{\prime},\widetilde{\omega|_{H}}\right)$ has
a codimension-1 ideal which fails to be finitely generated. By considering the
preimage of this ideal under the quotient map
$\ell^{1}(H,\omega|_{H})\rightarrow\ell^{1}(H/H^{\prime},\widetilde{\omega|_{H}})$
we see that $\ell^{1}(H,\omega|_{H})$ has a codimension-1 ideal which is not
finitely generated. It follows from Lemma 3.1 that $\ell^{1}(G,\omega)$ has a
finite-codimension left ideal which fails to be finitely generated. Hence, by
Lemma 3.2(ii), $\ell^{1}(G,\omega)$ has a maximal left ideal of finite
codimension which is not finitely generated, as required. ∎

###### Proof of Theorem 1.2.

(i) Suppose that $G$ is virtually soluble. Then $G$ has a finite-index soluble
subgroup $S$ that we may take to be normal. Set
$k=\min\\{i\in\mathbb{N}:[S^{(i)}:S^{(i+1)}]=\infty\\}$ and observe that
$H:=S^{(k)}$ is a finite-index subgroup of $G$ with an infinite abelian
quotient. Moreover, since $H$ is characteristic in $S$ it is normal in $G$, so
we we may apply Proposition 3.3.

(ii) Suppose that $G$ is virtually free. Then $G$ contains a finite-index free
subgroup $H$, which we may assume is normal. The result now follows from
Proposition 3.3. ∎

We now move on to our theorem about simple groups.

###### Proof of Theorem 1.3.

By [10, Corollary 1.9(ii)] (and its proof) the augmentation ideal
$\ell_{0}^{1}(G,\omega)$ is generated by finitely many elements of the form
$\delta_{e}-\delta_{t}\ (t\in G)$. We shall show that this is the only proper,
finite-codimension left ideal. Indeed, let $I$ be a proper, left ideal of
$\ell^{1}(G,\omega)$ of finite codimension. Then $G$ acts linearly on the
quotient $E:=\ell^{1}(G,\omega)/I$, and since $G$ is simple the action is
either trivial or faithful. If the action were faithful, then $G$ would be a
linear group, which is impossible by Malcev’s Theorem [8]. As such the action
is trivial, and so $\delta_{t}\cdot x=x$ for all $x\in E$ and all $t\in G$. It
follows that $(\delta_{e}-\delta_{t})\cdot x=0\ (x\in E)$ for all $t\in G$
and, since these elements generate $\ell^{1}_{0}(G,\omega)$ as a left ideal,
we must have $f\cdot x=0$ for all $f\in\ell^{1}_{0}(G,\omega)$. As such
$\ell^{1}_{0}(G,\omega)$ is contained in the kernel of the action of
$\ell^{1}(G,\omega)$ on $E$, which is contained in $I$, and hence $I$ must
equal the augmentation ideal by maximality. This completes the proof. ∎

Finally, we turn towards the proof of Theorem 1.4. Our strategy for showing
that the left ideals in question are finitely generated will involve reducing
to the case of an ideal of the form $J_{\omega}(G,H)$ for a finite-index
subgroup $H$. We shall show that, for radial exponential weights,
$J_{\omega}(\mathbb{F}_{m},H)$ is finitely generated for any finite-index
subgroup $H$ of $\mathbb{F}_{m}$. Then, exploiting the fact that any finitely-
generated group is the quotient of some $\mathbb{F}_{m}$, we shall show that
in fact $J_{\omega}(G,H)$ is finitely generated, whenever $G$ is a finitely-
generated group, $H$ is a finite-index subgroup, and $\omega$ is a radial
exponential weight. To ease the notation we fix the base of our radial
exponential weights to be $2$, although this plays no significant role in the
proofs.

To carry this out we first require a technical lemma about free groups. Note
that the subgroup $H$ of the lemma always has a generating set of the form
described.

######  Lemma 3.4.

Let $m\in\mathbb{N}$, let $\mathbb{F}_{m}$ be the free group on the set
$\\{a_{1},\ldots,a_{m}\\}$, and let $H$ be a finitely-generated subgroup of
$\mathbb{F}_{m}$. Let $X=\\{a_{1}^{\pm 1},\ldots,a_{m}^{\pm 1}\\}$, and let
$Y$ be a generating set for $H$ of the form $\dot{B}^{X}_{r}\cap H$, for some
$r\in\mathbb{N}$. Take $u\in H$, and write $u=y_{1}y_{2}\cdots y_{n}$, where
$n=|u|_{Y}$, and $y_{1},\ldots,y_{n}\in Y$. Write

$y_{i}=x_{i,1}x_{i,2}\cdots x_{i,k_{i}}\qquad(i=1,\ldots,n)$

where $x_{i,p}\in X$ and $k_{i}=|y_{i}|_{X}$ , for $i=1,\ldots,n$, and
$p=1,\ldots,k_{i}$.

  1. (i)

For each $i=1,\ldots,n$, when we reduce the expression for $y_{i}y_{i+1}$
given by

$x_{i,1}\cdots x_{i,k_{i}}x_{i+1,1}\cdots x_{i+1,k_{i+1}}$

at most
$\min\\{\lceil\frac{k_{i}}{2}\rceil,\lceil\frac{k_{i+1}}{2}\rceil\\}-1$ of the
elements $x_{i,1},\ldots,x_{i,k_{i}}$ can cancel.

Moreover for each $j\in\\{2,\ldots,n\\}$ the following hold.

  1. (ii)

When we convert the expression for $y_{1}y_{2}\cdots y_{j}$ given by

$x_{1,1}\cdots x_{1,k_{1}}x_{2,1}\cdots x_{2,k_{2}}\cdots x_{j,1}\cdots
x_{j,k_{j}}$

to a reduced word, the elements $x_{j,p}$ do not get canceled for
$p=\lceil\frac{k_{j}}{2}\rceil,\ldots,k_{j}$.

  2. (iii)

$|y_{1}y_{2}\cdots y_{j-1}|_{X}<|y_{1}y_{2}\cdots y_{j-1}y_{j}|_{X}$.

###### Proof.

We first prove part (i). Observe that if any more than $\lceil
k_{i+1}/2\rceil-1$ (that is to say, half or more) of the elements
$x_{i+1,1},\ldots,x_{i+1,k_{i+1}}$ are canceled when reducing $x_{i,1}\cdots
x_{i,k_{1}}x_{i+1,1}\cdots x_{i+1,k_{2}}$ then we remove at least as many
elements from $x_{i,1},\ldots,x_{i,k_{i}}$ as we add from
$x_{i+1,1},\ldots,x_{i+1,k_{i+1}}$, so that
$|y_{i}y_{i+1}|_{X}\leq|y_{i}|_{X}\leq r$. This implies that $y_{i}y_{i+1}\in
Y\cup\\{e\\}$ by our hypothesis on the special form of the generating set $Y$,
and this contradicts $n=|u|_{Y}$. By a symmetrical argument we cannot cancel
more than $\lceil k_{i}/2\rceil-1$ elements either.

We shall prove parts (ii) and (iii) together by induction on $j$. The base
case $j=2$ follows from part (i). Suppose that the induction hypothesis holds
for $j$. We shall now prove that (ii) holds for $j+1$. Write

$y_{1}y_{2}\cdots y_{j}=z_{1}z_{2}\cdots z_{q},$

for some $z_{1},\ldots,z_{q}\in X$, where $q=|y_{1}\cdots y_{j}|_{X}$ and
$z_{1}z_{2}\cdots z_{q}$ is a reduced word over $X$. By the induction
hypothesis

$z_{q}=x_{j,k_{j}},\quad z_{q-1}=x_{j,k_{j}-1},\quad\ldots,\quad z_{q-\lfloor
k_{j}/2\rfloor}=x_{j,\lceil k_{j}/2\rceil}.$

By part (i), when we reduce

$y_{j}y_{j+1}=x_{j,1}\cdots x_{j,k_{j}}x_{j+1,1}\cdots x_{j+1,k_{j+1}}$

at most $\lceil k_{j}/2\rceil-1$ of the elements
$x_{j+1,1},\ldots,x_{j+1,k_{j+1}}$ can cancel with the elements
$x_{j,1},\ldots,x_{j,k_{j}}$. This means that when we reduce

$z_{1}z_{2}\cdots z_{q}x_{j+1,1}\cdots x_{j+1,k_{j+1}}$

the only cancellations that can occur are between the elements
$x_{j+1,1},\ldots,x_{j+1,k_{j+1}}$ and the elements
$x_{j,1},\ldots,x_{j,k_{j}}$. However, part (i) also implies that at most
$\lceil k_{j+1}/2\rceil-1$ of these elements can cancel, and so $x_{j+1,p}$
will not be canceled, for $p=\lceil k_{j+1}/2\rceil,\ldots,k_{j+1}$ as
required for part (ii). It also follows that we are canceling fewer terms from
$z_{1},\ldots,z_{q}$ than we are adding from
$x_{j+1,1},\ldots,x_{j+1,k_{j+1}}$, so that $|y_{1}y_{2}\cdots
y_{j-1}|_{X}<|y_{1}y_{2}\cdots y_{j-1}y_{j}|_{X}$, and part (iii) follows. ∎

######  Lemma 3.5.

Let $m\in\mathbb{N}$, write $\mathbb{F}_{m}=\langle
a_{1},\ldots,a_{m}\rangle$, and let $X=\\{a_{1}^{\pm 1},\ldots,a_{m}^{\pm
1}\\}$. Set $\omega(t)=2^{|t|_{X}}\ (t\in\mathbb{F}_{m})$. Let $H$ be a
finite-index subgroup of $\mathbb{F}_{m}$. Then the left ideal
$J_{\omega}(\mathbb{F}_{m},H)$ of $\ell^{1}(\mathbb{F}_{m},\omega)$ is
generated by finitely many elements of the form $\delta_{e}-\delta_{t}$, where
$t\in H$.

###### Proof.

Let $k=[\mathbb{F}_{m}:H]$ and let $t_{1},\ldots,t_{k}$ be a right transversal
for $H$ in $\mathbb{F}_{m}$. Write $\gamma=\omega|_{H}$. Then
$J_{\omega}(\mathbb{F}_{m},H)$ can be written as

$J_{\omega}(\mathbb{F}_{m},H)=\ell^{1}_{0}(H,\gamma)*\delta_{t_{1}}+\cdots+\ell^{1}_{0}(H,\gamma)*\delta_{t_{k}}.$

As such, it suffices the prove that $\ell^{1}_{0}(H,\gamma)$ is finitely
generated as a left ideal of $\ell^{1}(H,\gamma)$. The weight $\gamma$ may not
be radial, so we cannot apply [10, Theorem 1.8] directly, but instead we adapt
its proof.

Since $H$ has finite index it is finitely generated. Let $Y$ be a finite
generating set for $H$ of the form $\dot{B}_{r}^{X}\cap H$ for some
$r\in\mathbb{N}$. Let given $u\in H$, and write $u=y_{1}y_{2}\cdots y_{n}$,
for $y_{1},\ldots,y_{n}\in Y$, where $n=|u|_{Y}$. By [10, Lemma 6.2(i)], we
can write

$\delta_{e}-\delta_{u}=\sum_{y\in Y}g_{y}*(\delta_{e}-\delta_{y}),$

where each $g_{y}\in\mathbb{C}H$ and has the form

(3.1) $g_{y}=\sum_{j=0}^{n-1}h_{y}^{(j)},$

where each $h_{y}^{(j)}$ is either $0$ or $\delta_{y_{1}\cdots y_{j}}$ in the
case that $j\neq 0$, and is either $0$ or $\delta_{e}$ in the case that $j=0$.
(Note that, contrary to what is stated immediately before it, [10, Lemma
6.2(i)] is a purely algebraic statement that is independent of the weight; we
shall see that we are able to prove that part (ii) of that lemma also holds
for $\gamma$ by applying Lemma 3.4 above).

For $j=1,\ldots,n-1$ we have $\|h_{y}^{(j)}\|_{\gamma}\leq 2^{|y_{1}\cdots
y_{j}|_{X}}$, and so $\|g_{y}\|_{\gamma}\leq 1+\sum_{j=1}^{n-1}2^{|y_{1}\cdots
y_{j}|_{X}}$. By Lemma 3.4(iii) this sum consists of strictly increasing
powers of $2$, each less than $2^{|u|_{X}}$ so that,

(3.2) $\|g_{y}\|_{\gamma}\leq\sum_{j=0}^{|u|_{X}-1}2^{j}\leq
2^{|u|_{X}}=\gamma(u)\qquad(y\in Y).$

We can now mimic the proof of [10, Theorem 1.8]. Enumerate
$H=\\{u_{0}=e,u_{1},u_{2}\ldots\\}$, and let
$f=\sum_{i=0}^{\infty}\alpha_{i}\delta_{u_{i}}\in\ell_{0}^{1}(H,\gamma)$. For
each $i\in\mathbb{N}$ write

$\delta_{e}-\delta_{u_{i}}=\sum_{y\in Y}g_{y}^{(i)}*(\delta_{e}-\delta_{y})$

for some functions $g_{y}^{(i)}\in\mathbb{C}H$ as in (3.1). Define functions
$\phi_{y}\ (y\in Y)$ by

$\phi_{y}=-\sum_{i=0}^{\infty}\alpha_{i}g_{y}^{(i)},$

and note that the series converges because, by (3.2),
$\|g_{y}^{(i)}\|_{\gamma}\leq\gamma(u_{i})\ (i\in\mathbb{N})$.

Then, since $\sum_{i=0}^{\infty}\alpha_{i}=0$, we have

$\displaystyle f$
$\displaystyle=\sum_{i=0}^{\infty}\alpha_{i}\delta_{u_{i}}-\left(\sum_{i=0}^{\infty}\alpha_{i}\right)\delta_{e}=-\sum_{i=0}^{\infty}\alpha_{i}(\delta_{e}-\delta_{u_{i}})$
$\displaystyle=-\sum_{i=1}^{\infty}\alpha_{i}\left(\sum_{y\in
Y}g_{y}^{(i)}*(\delta_{e}-\delta_{y})\right)=\sum_{y\in
Y}\phi_{y}*(\delta_{e}-\delta_{y}).$

As $f$ was arbitrary, it follow that $\ell^{1}_{0}(H,\gamma)$ is generated by
the finite set $\\{\delta_{e}-\delta_{y}:y\in Y\\}$, which completes the
proof. ∎

The next lemma is an interesting generalisation of [10, Corollary 1.9(ii)],
which states that $\ell^{1}_{0}(G,\omega)$ is finitely generated for any
radial exponential weight $\omega$ on $G$.

######  Lemma 3.6.

Let $G$ be a finitely-generated group with a finite, symmetric generating set
$X$, and let $H$ be a finite-index subgroup. Let $\omega$ be the weight on $G$
defined by $\omega(t)=2^{|t|_{X}}\ (t\in G)$. Then the left ideal
$J_{\omega}(G,H)$ is finitely generated in $\ell^{1}(G,\omega)$. Moreover, the
generators can be chosen to each have the form $\delta_{e}-\delta_{t}$ for
some $t\in H$.

###### Proof.

Write $X=\\{u_{1}^{\pm 1},\ldots,u_{n}^{\pm 1}\\}$, and let
$\\{v_{1},\ldots,v_{m}\\}$ be a finite generating set for $H$, where
$n,m\in\mathbb{N}$. Writing $\mathbb{F}_{n}=\langle
a_{1},\ldots,a_{n}\rangle$, let $q\colon\mathbb{F}_{n}\to G$ denote the
surjective homomorphism defined by $a_{i}\mapsto u_{i}\ (i=1,\ldots,n)$. Let
$K=q^{-1}(H)$, which is a finite-index subgroup of $\mathbb{F}_{n}$.

Let $\beta$ denote the weight on $\mathbb{F}_{n}$ given by
$\beta(t)=2^{|t|_{Z}}$, where $Z=\\{a_{1}^{\pm 1},\ldots,a_{n}^{\pm 1}\\}$,
and let $\widetilde{\beta}$ be the weight induced on $G$ by regarding it as a
quotient of $\mathbb{F}_{n}$ via $q$, as in (2.1). We claim that
$\widetilde{\beta}=\omega$. Indeed, note that

$|t|_{X}=\inf\\{|s|_{Z}:q(s)=t\\},$

and by (2.1) we have

$\widetilde{\beta}(t)=\inf\\{\beta(s):q(s)=t\\}=\inf\\{2^{|s|_{Z}}:q(s)=t\\}=2^{\inf\\{|s|_{Z}:q(s)=t\\}}=\omega(t).$

The map $q$ extends to a bounded algebra homomorphism, which we also denote by
$q$,

$q\colon\ell^{1}(\mathbb{F}_{n},\beta)\to\ell^{1}(G,\widetilde{\beta})=\ell^{1}(G,\omega).$

It is routinely checked that $q(J_{\beta}(\mathbb{F}_{n},K))=J_{\omega}(G,H)$.
By Lemma 3.5 $J_{\beta}(\mathbb{F}_{n},K)$ is finitely generated, and hence so
is $J_{\omega}(G,H)$. The final statement also follows from Lemma 3.5 since
$q(\delta_{e}-\delta_{t})=\delta_{e}-\delta_{q(t)}\ (t\in K)$. ∎

We now prove our final theorem.

###### Proof of Theorem 1.4.

(i) Let $f\in\ell^{1}(G,\omega)\setminus\\{0\\}$. We shall show that there is
a finite-codimension maximal left ideal $I$ in $\ell^{1}(G,\omega)$ for which
$f\notin I$. By multiplying on the left by $\delta_{t}$ for some $t\in G$, we
may assume that $f(e)\neq 0$. Let $\varepsilon=\frac{1}{2}|f(e)|$, and let $F$
be a finite subset of $G$ containing the identity for which

$\sum_{t\in G\setminus F}|f(t)|\omega(t)<\varepsilon.$

Since $G$ is residually finite there exists $H\lhd G$ of finite index such
that $H\cap F=\\{e\\}$.

Let $q\colon\ell^{1}(G,\omega)\to\ell^{1}(G/H,\widetilde{\omega})$ denote the
quotient map. Then

(3.3) $q(f)(e)=\sum_{s\in H}f(s),$

which is non-zero because

$\sum_{s\in H,s\neq e}|f(s)|\leq\sum_{s\in G\setminus F}|f(s)|<|f(e)|,$

so that $f(e)$ can never cancel with the rest of the terms in the sum (3.3).
Therefore in particular $q(f)\neq 0$. Since $G/H$ is a finite group,
$\ell^{1}(G/H,\omega)$ is equal to $\mathbb{C}(G/H)$, which is semisimple.
Therefore there exists a maximal left ideal $\widetilde{I}$ in
$\ell^{1}(G/H,\widetilde{\omega})$ such that $q(f)\notin\widetilde{I}$. Let
$I=q^{-1}(\widetilde{I})$. Then $I$ is a finite-codimension maximal left ideal
of $\ell^{1}(G,\omega)$, and $f\notin I$.

(ii) Let $I$ be a finite-codimension left ideal of $\ell^{1}(G,\omega)$. Then
$G$ acts linearly on the quotient $E:=\ell^{1}(G,\omega)/I$, and since $G$ is
not a linear group, the action cannot be faithful, so has a kernel, which we
shall denote by $H\lhd G$. Since $G$ is just infinite, $H$ must have finite
index in $G$.

We claim that $I\supset J_{\omega}(G,H)$. As the action of $H$ on $E$ is
trivial, we must have $\delta_{t}\cdot x=x$, and hence
$(\delta_{e}-\delta_{t})\cdot x=0$, for all $t\in G$ and $x\in E$. By Lemma
3.6 $J_{\omega}(G,H)$ is generated as a left ideal by elements of the form
$\delta_{e}-\delta_{t}\ (t\in H)$, so it follows that $f\cdot x=0$ for all
$f\in J_{\omega}(G,H)$, which implies that $I\supset J_{\omega}(G,H)$.

Finally, fix finitely many elements $f_{1},\ldots,f_{k}\in I$ such that
$I=J_{\omega}(G,H)+{\rm span\,}\\{f_{1},\ldots,f_{k}\\}$. Then a finite
generating set for $J_{\omega}(G,H)$ together with $f_{1},\ldots,f_{k}$ form a
finite generating set for $I$. ∎

Remark. Note that in Theorem 1.4(ii) the left ideals are not assumed to be
closed. However, it follows from the above proof that all of the finite-
codimension left ideals of the Banach algebra in Theorem 1.4 actually are
closed. Indeed, we have seen that an arbitrary finite-codimension left ideal
$I$ is equal to $J_{\omega}(G,H)+{\rm span\,}\\{f_{1},\ldots,f_{k}\\}$, for
some finite-index subgroup $H$ and some $f_{1},\ldots,f_{k}\in I$, and since
$J_{\omega}(G,H)$ is closed, $I$ must be as well. Not every unital Banach
algebra has the property that all of its finite-codimension left ideals are
closed: for intance, add a unit to any of the many interesting examples in
[4]. This property can be rephrased as an automatic continuity property of
$\ell^{1}(G,\omega)$, namely that every module map from $\ell^{1}(G,\omega)$
to a finite-dimensional Banach left $\ell^{1}(G,\omega)$-module is
automatically continuous.

### Acknowledgements

I would like to thank Ian Short for his support and hospitality during my time
so far at the Open University. I would also like to thank Ulrik Enstad for his
useful comments on an earlier draft of the article, and Karthik Rajeev her
help with the reference [1]. Finally, I would like to thank the anonymous
referee for their careful reading of the article.

### Competing Interests

The author declares none.

## References

  * [1] M. Albért, _Representing graphs by the non-commuting relation_ , Publ. Math. Debrecen 69/3 (2006), 261–269.
  * [2] L. Bartholdi, R. Grigorchuk, and Z. Sunik, _Branch groups_ , Handbook of algebra, Vol. 3, North-Holland, Amsterdam (2003), 989–1112.
  * [3] D. Blecher and T. Kania, _Finite generation in C*-algebras and Hilbert C*-modules_ , Studia Math. 224 (2014), 143–151.
  * [4] M. Cabrera García, H. G. Dales, and Á. Rodríguez Palacios, Maximal Left Ideals in Banach Algebras, Bul. London Math. Soc. 52 (2020), 1–15.
  * [5] H. G. Dales, T. Kania, T. Kochanek, P. Koszmider and N. J. Laustsen, _Maximal left ideals in the Banach algebra of operators on a Banach space_ , Studia Math. 218 (2013), 245–286.
  * [6] H. G. Dales and W. Żelazko, _Generators of maximal left ideals in Banach algebras_ , Studia Math. 212 (2012), 173–193.
  * [7] A. V. Ferreira and G. Tomassini, _Finiteness properties of topological algebras_ , Ann. Scuola Norm. Sup. Pisa 5 (1978), 471–488.
  * [8] A. I. Malcev, _On isomorphic matrix representations of infinite groups of matrices_ (Russian), Mat. Sb. 8 (1940) 405–422 & Amer. Math. Soc. Transl. 45 (1965), 1–18.
  * [9] H. Reiter and J. D. Stegeman, _Classical Harmonic Analysis and Locally Compact Groups_ , London Math. Soc. Monogr. 22, Oxford University Press, 2000.
  * [10] J. T. White, _Finitely-generated left ideals in Banach algebras on groups and semigroups_ , Studia Math. 239 (2017), 67–99.
  * [11] J. T. White, _Banach algebras on groups and semigroups_ , PhD thesis, University of Lancaster (2018).


# Sparse Points to Dense Clouds: Enhancing 3D Detection with Limited LiDAR
Data

Aakash Kumar1, Chen Chen1, Ajmal Mian2, Neils Lobo1, and Mubarak Shah1 1Center
for Research in Computer Vision, University of Central Florida, USA
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>of Western Australia<EMAIL_ADDRESS>
###### Abstract

3D detection is a critical task that enables machines to identify and locate
objects in three-dimensional space. It has a broad range of applications in
several fields, including autonomous driving, robotics and augmented reality.
Monocular 3D detection is attractive as it requires only a single camera,
however, it lacks the accuracy and robustness required for real world
applications. High resolution LiDAR on the other hand, can be expensive and
lead to interference problems in heavy traffic given their active
transmissions. We propose a balanced approach that combines the advantages of
monocular and point cloud-based 3D detection. Our method requires only a small
number of 3D points, that can be obtained from a low-cost, low-resolution
sensor. Specifically, we use only 512 points, which is just 1% of a full LiDAR
frame in the KITTI dataset. Our method reconstructs a complete 3D point cloud
from this limited 3D information combined with a single image. The
reconstructed 3D point cloud and corresponding image can be used by any multi-
modal off-the-shelf detector for 3D object detection. By using the proposed
network architecture with an off-the-shelf multi-modal 3D detector, the
accuracy of 3D detection improves by 20% compared to the state-of-the-art
monocular detection methods and 6% to 9% compare to the baseline multi-modal
methods on KITTI and JackRabbot datasets. Project website link
https://bravocharlie-ai.github.io/Sparce_to_Dense/

##  I INTRODUCTION

3D detection is essential in robotic vision for applications such as
autonomous driving and augmented reality, enabling machines to identify and
localize objects in 3D space with sensors like LiDAR. Unlike 2D detection
methods, which only provide a 2D bounding box and lack depth information, 3D
detection offers details on objects’ location, size, and orientation in 3D
space. This detailed spatial awareness is crucial for robots to interact
safely and effectively with their environments. Consequently, 3D detection
techniques are pivotal for advancing machine perception, surpassing the
limited accuracy and capabilities of 2D detection methods.

In autonomous driving, 3D detection is primarily achieved through point cloud-
based or multimodal-based approaches, each dependent on sensor type. Point
cloud-based methods, utilizing LiDAR data, are divided into voxel-based
methods [1, 2, 3], which grid point clouds for volumetric representation to
apply 3D convolutions, and point-based methods [4, 5, 6], focusing on raw
point clouds for iterative sampling and abstract representation, trading off
efficiency for accuracy. Multimodal methods [7, 8, 9, 10] integrate multiple
sensor data, like cameras and LiDAR, enhancing detection accuracy and
robustness. The choice among these methods is dictated by the application’s
specific needs and constraints.

Figure 1: Overview of the proposed 3D object detection approach. The
architecture accepts an input image and a set of sparse LiDAR points, which it
then processes to generate a high-resolution point cloud. This dense point
cloud, once reconstructed, is paired with the original image and fed into an
off-the-shelf 3D object detector, enabling the accurate detection and
localization of 3D objects within the scene.

High-resolution LiDAR and depth sensors have advantages in accuracy and
precision, but they are expensive, consume more power and can suffer from
interference problems given that they are mostly active sensors. As a result,
researchers have been exploring ways to improve 3D detection performance while
reducing costs and complexity. One approach that has gained increasing
research attention is monocular 3D detection [11, 12, 13, 14, 15, 16, 17],
which estimates the 3D properties of objects in a scene using only a single 2D
image.

Monocular 3D detection, while accessible due to its single-camera requirement,
faces significant accuracy and reliability issues, making it unsuitable for
practical use. Its limitations include poor depth estimation, sensitivity to
lighting, and difficulty with complex or occluded shapes. The method also
suffers from dependence on precise camera calibration and a restricted field
of view. These challenges severely hinder its effectiveness for real-world
applications.

To address the aforementioned challenges, we propose a novel approach to 3D
detection by combining monocular vision with a small set of 3D points from a
low-resolution and low-cost depth sensor. Our method uses only 512 points or
1% of a full LiDAR frame of the KITTI dataset [18] for each detection.
Traditional multimodal 3D detection methods relying on high-resolution LiDAR
data struggle with lower-resolution LiDAR frame and image data. This is
particularly relevant in robotics applications where high-resolution LiDAR is
impractical due to power and size constraints. Our method introduces a
transformer-based module capable of reconstructing a high-resolution LiDAR
frame by processing low-resolution LiDAR inputs accompanied with image data.
The reconstructed high-resolution LiDAR data, when combined with the
corresponding image, significantly enhances detection precision when processed
by standard off-the-shelf multimodal 3D detectors, resulting in a 6-9%
accuracy improvement over baseline methods and over 20% compared to monocular
3D detection. Overview of our proposed method is shown in figure 1. This
strategy reconstructs full 3D point clouds from few sparse 3D points,
improving object detection in autonomous driving and robotics. It seamlessly
integrates with existing systems, promising advancements in the field of
robotics.

Our major contributions are summarized below:

  * •

We formulate a novel balanced approach between monocular and point cloud-based
3D detection.

  * •

We propose a novel transformer based architecture for point cloud
reconstruction using a single image and a very small number of 3D points.

  * •

The proposed approach can integrate with 3D sensors of varying resolutions
e.g., we show results for 256 to 512 points per scene.

  * •

We achieve significantly higher mean average precision (mAP) on a popular 3D
detection benchmark compared to monocular methods.

##  II Related Work

###  II-A Monocular 3D Detection

Monocular 3D detection estimates the 3D features of objects in a scene from a
single 2D image, which is challenging due to the lack of depth information.
Techniques such as learning 2D features, adding 3D cues, and geometric
reasoning are commonly used to enhance the 3D perception from images.
Monocular 3D detection finds applications in fields such as autonomous
driving, robotics, and augmented reality. In recent years, there has been an
increase in the number of monocular 3D object detection methods that use a
single image to estimate the 3D location, dimension and angle of objects [19,
20, 21, 22]. These methods mainly rely on geometric consistency to predict
objects since they lack depth information. Deep3Dbox [22] uses a unique
MultiBin loss to accurately predict the orientation of objects, and enforces a
constraint between the 2D and 3D bounding boxes based on geometric principles.
MonoPair [23] uses a spatial pair-wise relationship among objects to enhance
the accuracy of detection.

Another approach is to learn some depth cues while training the model whereas,
inference is still fully monocular. Among these methods, one approach involves
combining depth features extracted from the depth map with image features to
obtain 2D depth-aware features [24]. Another approach constructs 3D feature
volumes from 2D features to improve the 3D detection capability [16, 25].
Finally, the Pseudo-LiDAR method converts estimated depth maps into Pseudo 3D
representations for monocular 3D object detection [26, 27].

###  II-B Pointcloud based 3D Detection

In 3D object detection research, voxel-based and point-based methods stand
out. Voxel-based approaches grid point clouds for efficient feature
extraction. VoxelNet [1] segments scenes into evenly spaced 3D voxels, encodes
them with feature layers, and uses 3D convolution to create a volumetric
representation for region proposal network processing. SECOND [28] accelerates
LiDAR detection training and inference with sparse convolution, introducing
angle loss regression. PointPillars [2] projects 3D clouds onto 2D, using a
pillar feature extractor and network for 3D object detection. Point-based
methods [4, 5, 6] focus on raw clouds, employing iterative techniques for
precision but at higher computational costs. 3DSSD [6] offers an anchor-free,
single-stage model with a fusion sampling strategy for efficient downsampling
and accuracy. PV-RCNN [3] merges 3D voxel CNNs with PointNet abstraction,
enhancing feature extraction via voxel set abstraction and RoI-grid pooling
for superior 3D proposals and context capture.

###  II-C Mutli-modal 3D Detection

There are different fusion mechanisms for 3D detection that leverage the
complementary information provided by LiDAR and camera inputs. Point-level
fusion [7, 8, 9] is one approach that combines data at the individual point
level, where image features are queried via raw LiDAR points and concatenated
back as additional point features. This technique has shown to improve
detection accuracy, but it heavily relies on the presence of LiDAR point
clouds, which can limit its effectiveness in scenarios where LiDAR input is
missing. On the other hand, feature-level [10] fusion methods have become the
state-of-the-art. However, these methods heavily rely on high resolution LiDAR
point clouds.

##  III Methodology

Figure 2: Proposed architecture for generating a dense point cloud from an
input image and a sparse set of 3D points from a low-cost sensor. Initially,
the image is split into 465 patches which are transformed into 256-dimensional
vectors by a CNN based feature extractor. These feature vectors combined with
sampled 3D points (Point Queries) are passed through a transformer encoder-
decoder framework. The encoder uses self-attention to understand patch
details, while the decoder employs cross-attention with image-tokens to
produce point-tokens for each query point. These tokens are processed by a
Point Cloud (PC) Generator that translates them into a dense point cloud.
Training involves a Chamfer distance loss function, comparing the predicted
point groups with ground-truth data, derived from nearest neighbors to the
query points. The outcome is a detailed point cloud useful for 3D object
detection and other applications.

###  III-A Problem Formulation

Given an image and a small set of 3D points as input, the aim is to generate a
dense point cloud of the scene corresponding to the image. The input 3D points
can be obtained from a low-resolution and low-cost sensor. To achieve this,
the image is divided into $465$ patches, and each patch is processed through a
feature extractor that converts it into a 256-dimensional feature vector.
These vectors serve as tokens for the transformer encoder. At the same time,
few 3D points are fed into the decoder to perform cross-attention across the
image tokens coming from the transformer encoder as shown in figure 2. This
results in a 256-dimensional token for each query point, generated by the
transformer decoder. These tokens are then passed through a neural network:
the PC generator. The PC generator converts each decoder token into a set of
points corresponding to the input queries, allowing for the reconstruction of
a dense point cloud corresponding to the input image.

For the input 3D points, we downsample the corresponding LiDAR frame of the
image as mentioned in III-D to mimic low-resolution and low-cost LiDAR frame.
Such frame consists of around $700$ to $800$ points then We do farthest point
sampling to select $n$ number of 3D query points
$Q_{n}=\\{q_{1},q_{2},q_{3},...q_{n}\\}$. We show results for $n=$ 256 ,and
512. Against each query point, we predict a group of points
$g_{i}^{p}=h(q_{i})$, where $q_{i}$ is the $i$-th query point, superscript $p$
stands for prediction and the function $h(.)$ represents the network
architecture. Hence, all groups can be represented as $G_{n}^{p}=h(Q_{n})$.
Each $i$-th group of points contains $k$ number of points
$g_{i}^{p}=\\{p_{i_{1}},p_{i_{2}},p_{i_{3}}...p_{i_{k}}\\}$ where $k$ is 32 or
64. We show results for these settings in the validation table.

For training the network architecture we need a ground-truth group of points
$g_{i}^{g}$ against every predicted group of points $g_{i}^{p}$. For every
query point $q_{i}$, we select the $k$ neighboring points in a radius of 1.2
meters through random point sampling. These points are then normalize using
the input query point, hence, the ground-truth points can be represented as
$g_{i}^{g}=((p^{\prime}_{i_{1}},p^{\prime}_{i_{2}},p^{\prime}_{i_{3}},...p^{\prime}_{i_{k}})-q_{i})/1.2$,
where the superscript ’ represents un-normalized points. All points are
normalized with respect to the center query point and diameter. Hence, each
normalized set of points range from -1 to 1 $-1<=|p_{i_{k}}|<=1$.

Each query point $q_{i}$ is converted to a group of $k$ points. The output
point cloud consists of $n*k$ points. This reconstructed point cloud combined
with image can be used for down-streaming tasks such as 3D object detection.
Through this method, we can either use low resolution LiDAR, or any other
depth sensor that provides some 3D points from the scene to reconstruct the
dense point cloud and perform 3D object detection.

###  III-B Network Architecture

####  III-B1 Feature Extractor

Image size in KITTI dataset is $375\times 1242$, after padding zeros
horizontally and vertically, we get a $384\times 1248$ image that can be
divided into equal size patches of $32\times 32$ resulting in a total of
$12\times 39=468$ patches. Padding is performed to avoid changing the 2D scale
of objects with respect to their actual 3D scale in the point cloud. Each
patch is passed through VGG-net [29] with modified skip pooling [30] to pass
features from different levels to the top. We employ this feature extractor
from [31]. The skip pooling layer is applied at the output of each pooling
layer. The number of channels at those pooling layers are 32, 64, 128 and 256.
First, global average pooling is applied at each level to gather spatial
information and then two point-wise-convolution layers are used to re-scale
the number of channels for every layer to 64. Afterwords, these layers are
concatenated to form a 256 dimensional feature vector which is fed to the
transformer encoder. This type of feature extractor gathers low and high level
features from a single patch.

####  III-B2 Image Encoder

To capture the global context of the image, we apply self-attention across all
the features. This enables the model to learn the global features of the
image, facilitating the comparison of objects’ appearances with respect to
each other and their surroundings, thereby enhancing the ability of the model
to discern depth and object size. To achieve this, we use a transformer
encoder with four layers of self-attention, each employing multiheaded
attention with eight heads. In self-attention mechanism, the input sequence is
transformed into three vectors: query, key, and value vectors. The query
vector determines the relevance of each part of the sequence to other parts,
the key vector identifies which parts the model should pay attention to, and
the value vector assigns importance weights to each part.

To compute the relevance of each part, the dot product of the query and key
vectors is calculated, and a softmax function is applied to normalize the
results into a probability distribution. The value vector is then multiplied
by this probability distribution to produce the final weighted sum of the
input sequence, where the attention scores determine the weights. Self-
attention is useful for learning long-range dependencies and helps the model
understand the global context.

Overall, the combination of CNN-based feature extraction and self-attention
mechanism enable the model to learn both local and global features of the
image, hence the model can better discern the depth and object size, which is
crucial for accurate reconstruction of the point cloud.

####  III-B3  Decoder

Transformer decoder takes two inputs, the learned representation from the
encoder and the point-query embeddings. The decoder performs both self-
attention and cross-attention to generate the output. Cross-attention is
performed between each query point and the output tokens of the encoder in the
Transformer architecture. This means that the decoder learns to attend to
specific regions of the input image corresponding to the query points while
generating the output point cloud.

The decoder learns the location of each query point by identifying the
relevant parts of the encoder output sequence that are important for that
query point. It does this by computing attention scores for each query point
based on the dot product between the query vector and key vectors. These
scores represent the similarity between the query point and each key vector in
the encoder output. The softmax function is then used to normalize the scores
to obtain attention weights that are multiplied by the value vectors to obtain
the context vector. The context vector represents the weighted sum of the
encoder output sequence and captures the relevant information about the image
which needed to generate the output point cloud for the corresponding query
point.

Figure 3: Ground truth point cloud (LiDAR) compared to point cloud predictions
generated using $512$ query points. Each query point generates 32 points. We
show the query points with increased point size for better visibility.

####  III-B4  PC Generator

PC generator consists of four MLP layers. It takes input from the decoder
which is of size $k*256$ where $k$ is the number of queries and 256 is the
dimension of each token. It converts this 256 dimension vector to a $32*3$
dimension vector representing 32 points, each with 3 dimensions i.e. a point
cloud of 32 points.

####  III-B5 Query Embeddings.

We utilize nonparametric embeddings obtained from XYZ locations. Each query
point is associated with a query embedding, where the coordinates are
converted into Fourier positional embeddings [32].

###  III-C Loss Function

We use Chamfer distance loss for pointcloud reconstruction. Chamfer loss
measures the distance between the ground-truth and reconstructed point clouds
as follows:

$d_{\mathrm{chamfer}}(X,Y)=\frac{1}{k}\sum_{x\in X}\min_{y\in
Y}\|x-y\|^{2}+\frac{1}{k}\sum_{y\in Y}\min_{x\in X}\|x-y\|^{2},$ (1)

where X and Y are the ground truth and predicted point cloud. In this case
$X=g_{i}^{g}$ and $Y=g_{i}^{p}$. Whereas, $x$ is point in sub-point cloud
$g_{i}^{g}$ and $y$ is point in sub-point cloud $g_{i}^{p}$. Where $k$ is the
total number of points in $X$ or $Y$ which is constant here. We provide
results for two settings where $k=32$ and $k=64$. We apply this loss across
all $n$ numbers of corresponding groups of prediction and ground truth points.

TABLE I:  Results of monocular 3D detection methods on the KITTI leader board. Our method stands by combining minimal depth information with images to perform 3D detection through point cloud reconstruction. By adding just 512 extra points, our method achieves a significant improvement in performance compared to the monocular and baseline multimodal methods. Method | Reference | Modality |<EMAIL_ADDRESS>|<EMAIL_ADDRESS>
---|---|---|---|---
Easy | Medium | Difficult | Easy | Medium | Difficult
DDMP [17] | CVPR 2021 | Image | 19.71 | 12.78 | 9.80 | 28.08 | 17.89 | 13.44
CaDNN [16] | CVPR 2021 | Image | 19.17 | 13.41 | 11.46 | 27.94 | 18.91 | 17.19
DFRNet [15] | ICCV 2021 | Image | 19.40 | 13.63 | 10.35 | 28.17 | 19.17 | 14.84
MonoEF [14] | CVPR 2021 | Image | 21.29 | 13.87 | 11.71 | 29.03 | 19.70 | 17.26
MonoFlex [13] | CVPR 2021 | Image | 19.94 | 13.89 | 12.07 | 28.23 | 19.75 | 16.89
GPUNet [12] | ICCV 2021 | Image | 20.11 | 14.20 | 11.77 | - | - | -
MonoGround [33] | CVPR 2022 | Image | 21.37 | 14.36 | 12.62 | 30.07 | 20.47 | 17.74
MonoDTR [11] | CVPR 2022 | Image | 21.99 | 15.39 | 12.73 | 28.59 | 20.38 | 17.14
Pseudo-Stereo [27] | CVPR 2022 | Image | 23.61 | 17.03 | 15.16 | 31.83 | 23.39 | 20.57
MonoATT [34] | CVPR 2023 | Image | 24.72 | 17.37 | 15.00 | 36.87 | 24.42 | 21.88
MonoDETR [35] | ICCV 2023 | Image | 25.00 | 16.47 | 13.58 | 33.60 | 22.11 | 18.60
MVX-Net [7] | - | Image + 1% Points | 34.24 | 21.23 | 21.34 | 47.41 | 33.53 | 31.81
Ours $+$ MVX-Net | - | Image + 1% Points | 42.61 (+8.37%) | 26.14 | 24.18 | 54.12 | 38.43 | 34.61
EPNET++ [36] | - | Image + 1% Points | 37.61 | 24.43 | 22.43 | 52.26 | 36.25 | 33.53
Ours $+$ EPNET++ | - | Image + 1% Points | 44.23 (+6.62) | 31.94 | 25.16 | 58.12 | 42.37 | 39.32
SFD [37] | - | Image + 1% Points | 39.42 | 27.23 | 24.94 | 55.23 | 38.51 | 34.18
Ours $+$ SFD | - | Image + 1% Points | 45.13 (+5.71) | 31.02 | 27.83 | 59.12 | 43.43 | 41.23

###  III-D Point-cloud Downsampling

In our research, we developed a technique to simulate the data characteristics
of a low-resolution LiDAR system, a necessity for one of our key experiments.
The initial data was captured using a high-precision LiDAR sensor, featuring
64 lasers/detectors. This sensor covered a vertical field of view spanning
$26.8$ degrees in elevation, segmented into $64$ angular subdivisions, each
measuring approximately $0.4^{\circ}$ degrees ranging $+2^{\circ}$ to
$-24.8^{\circ}$ from relative to the horizontal plane. To achieve a reduced
vertical resolution, we selectively utilized data from only 8 out of the 64
detectors. This selective sampling effectively narrowed down our focus to
$\frac{1}{8}$th of the original elevation data points, thereby reducing the
vertical resolution proportionately. Concurrently, we modified the azimuthal
resolution from a high resolution of $0.08^{\circ}$ to coarser $0.64^{\circ}$.
This was accomplished by downsampling the original 360-degree azimuthal sweep,
selecting every eighth data point from the 4500 high-resolution points
(calculated from the $\frac{360^{\circ}}{0.08^{\circ}}$ division), hence
sampling at intervals of $0.64^{\circ}$. Additionally, to mimic the
performance of a low-cost LiDAR sensor, we introduced an artificial noise
level of ±1 cm to the data. This method allowed us to replicate lower-
resolution LiDAR data accurately, providing an essential basis for our
experimental analysis.

###  III-E Off-the-shelf Detectors

We use three pre-existing detectors, namely MVX-Net, EPNet++ and SFD. These
are multimodal approaches that processes both image and point cloud data.

####  III-E1 MVX-Net

MVX-Net [7] improves 3D object detection by combining LiDAR points with image
features using two fusion techniques, namely PointFusion and VoxelFusion. MVX-
Net extends the VoxelNet algorithm by fusing LiDAR and image data at early
stages, and the VFE feature encoding is used to append the pooled image
features to every voxel.

####  III-E2 EPNet++

EPNet++ [36] introduces a Cascade Bi-directional Fusion (CB-Fusion) module and
Multi-Modal Consistency (MC) loss for multi-modal 3D object detection,
combining hight resolution LiDAR and camera data to enhance feature
representation and prediction reliability.

####  III-E3 SFD

SFD [37] introduces a multi-modal framework that enhances 3D object detection
by generating pseudo point cloud to perform depth completion to address LiDAR
data sparsity. It features a novel fusion strategy, 3D-GAF, for integrating
features from raw and pseudo point clouds with fine granularity.

##  IV Experiments

###  IV-A Datasets

####  IV-A1 KITTI Dataset

Our proposed approach is evaluated on the KITTI 3D object detection dataset
[18]. The dataset consists of a vast collection of high-resolution images,
laser scans, and other sensor data, annotated for various computer vision
tasks, including object detection, 3D object detection, and depth estimation.
It comprises 7,481 training images and 7,518 test images with corresponding
point clouds captured across a midsize city from rural areas and highways.

####  IV-A2 JackRabbot Dataset

The JackRabbot dataset (JRDB) was gathered using a social mobile robot
outfitted with a range of sensors, including LiDAR sensor, 360-degree
cylindrical stereo RGB cameras, a 360-degree spherical RGB-D camera, as well
as GPS and IMU sensors for data collection in diverse indoor and outdoor
environments. JRDB offers annotations for pedestrians, featuring 1.8 million
3D oriented bounding box annotations derived from the LiDAR sensors, along
with 2.4 million 2D bounding boxes in RGB images from various cameras.
Additionally, it includes identity links between the 2D and 3D bounding boxes
and across time.

###  IV-B Implementation Details

Our proposed model is implemented using the PyTorch library. We employ the
standard nn.MultiHeadAttention module to implement the Transformer model. The
encoder consists of 4 layers, and each layer uses multiheaded attention with
eight heads. Similarly, the decoder also has 4 layers, closely following the
encoder architecture, where each layer uses multiheaded attention with eight
heads. To encode the XYZ coordinates in the decoder, we use Fourier positional
encodings [32].

We apply a dropout [38] of 0.1 for all self-attention and MLP layers in the
model. However, we use a higher dropout of 0.3 for the decoder. The model is
optimized using the AdamW optimizer [39], and a cosine learning rate schedule
[40] is applied to decay the learning rate until it reaches $10^{-6}$.
Additionally, we apply a weight decay of 0.1 to prevent overfitting. To
prevent the issue of exploding gradients, we apply gradient clipping at an L2
norm of 0.1 [41].

###  IV-C Results and Discussion

Our method requires minimal 3D information and uses just 512 3D points along
with an image for 3D detection. Note that 512 points form only 1% of the full
LiDAR frame in the KITTI dataset. Our approach involves reconstructing a
complete 3D point cloud from an image and a limited number of 3D points,
followed by the use of off-the-shelf detectors for 3D object detection. This
allows us to leverage the advantages of both image and point cloud data while
mitigating the challenges of high resolution 3D data acquisition and
processing.

TABLE II: Results on JackRabbot Dataset Method | Modality | AP3D
---|---|---
<EMAIL_ADDRESS>|<EMAIL_ADDRESS>
MVX-Net | I + 1% Points | 46.52 | 18.65
Ours $+$ MVX-Net | I + 1% Points | 52.78 | 23.94
EPNET++ | I + 1% Points | 48.56 | 27.65
Ours $+$ EPNET++ | I + 1% Points | 59.54 | 33.54
SFD | I + 1% Points | 48.95 | 26.48
Ours $+$ SFD | I + 1% Points | 61.48 | 32.48

Figure 3 gives an insight into the quality of the point clouds generated by
our approach. The left column of the figure shows the ground truth point cloud
obtained with a LiDAR sensor, and hence considered to be the most accurate 3D
representation of the scene. The remaining columns show the input query points
(low resolution LiDAR) and the reconstructed point clouds.

Figure 4: Qualitative results of 3D detection.

####  IV-C1 Results on KITTI Test Dataset

Table I presents a comparative analysis of various 3D object detection methods
on the KITTI test dataset, focusing on their performance across different
levels of difficulty and employing distinct modality inputs. Our evaluation is
segmented into monocular methods, which rely solely on image data (I), and
hybrid methods, which combine image data with a sparse point cloud (I + $1\%$
Points), to demonstrate the impact of incorporating minimal depth information
on detection accuracy. Monocular methods, while innovative, illustrate a
performance ceiling when tasked with interpreting complex 3D environments from
2D images alone. For instance, MonoDETR, the top performer among monocular
approaches, achieves AP scores of 25.00, 16.47, and 13.58 in easy, medium, and
difficult categories, respectively, for 3D detection, and 33.60, 22.11, and
18.60 for bird’s eye view detection).

To address the limitations of monocular methods, we explored the integration
of our novel reconstruction module with off-the-shelf multi-modal detectors
(MVX-Net, EPNET++, and SFD), which utilize both image data and a low-
resolution point cloud (constituting merely 1% of a full LiDAR scan). This
hybrid approach demonstrates a significant enhancement in detection
performance. Notably, the combination of our method with MVX-Net resulted in
AP 3D scores of 42.61, 26.14, and 24.18 and BEV scores of 54.12, 38.43, and
34.61 in easy, medium, and difficult categories, respectively, marking a
substantial improvement over both the baseline MVX-Net (+8.37%) scores and the
best-performing monocular method, MonoDETR (+17.61%).

This trend of performance improvement is consistent across all hybrid
configurations. For instance, our method combined with EPNET++ and SFD
achieved notable gains, surpassing traditional monocular and baseline multi-
modal approaches, highlighting the effectiveness of utilizing proposed model
for point cloud reconstruction. Moreover, we also show the qualitative results
in Figure 4 which also justifies that by utilizing our proposed architecture
detection accuracy is improved.

TABLE III: Ablation study of number of query points(n) and neighbouring points(k). This table shows how varying number of query points and neighboring points affect the detection accuracy. When point cloud reconstructed with different number of query points and neighboring points is given to off-the-shelf-detectors their detection accuracy varies. Best detection accuracy is achieved when we select 512 query points and 32 neighboring points.  Query | Neighboring | MVX-Net | EPNet++ | SFD
---|---|---|---|---
Points | Points | AP3DIoU=0.7 | AP3DIoU=0.7 | AP3DIoU=0.7
256 | 32 | 63.16 | 67.87 | 69.93
512 | 32 | 68.13 | 71.08 | 73.67
256 | 64 | 62.63 | 66.29 | 67.83
512 | 64 | 67.32 | 69.64 | 71.77

####  IV-C2 Results on JRDB Test Dataset

We also provide results on JackRabbot (JRDB) dataset in Table II to check the
effectiveness of our method on different dataset. We see the similar trend in
detection accuracy over three off-the-shelf detectors. Incorporating our
module with established off-the-shelf 3D detectors MVX-Net, EPNET++, and SFD
demonstrates a significant improvement in 3D object detection on the
JackRabbot Dataset, with MVX-Net resulted in an increase in AP from 46.52 to
52.78 at an IoU threshold of 0.3, and from 18.65 to 23.94 at an IoU threshold
of 0.5. Similarly, when combined with EPNET++, our module contributed to a
substantial increase in AP scores, from 48.56 to 59.54 at IoU=0.3, and from
27.65 to 33.54 at IoU=0.5. SFD witnessed the most pronounced improvement, with
AP scores leaping from 48.95 to 61.48 at IoU=0.3, and from 26.48 to 32.48 at
IoU=0.5.

The results underscore the potential of our approach in significantly
enhancing the accuracy of 3D object detection. By leveraging a minimal set of
depth points alongside image data, we demonstrate the feasibility of achieving
high-precision detection performance, without incurring the cost and
complexity associated with full-resolution LiDAR data. This balanced
methodology not only bridges the gap between monocular and LiDAR-based
detection systems but also paves the way for more accessible and efficient 3D
object detection solutions suitable for a wide range of applications.

####  IV-C3 Ablation Studies

Table III presents an ablation study to evaluate the impact of varying the
number of query points and neighboring points on the performance of off-the-
shelf 3D detectors, specifically MVX-Net, EPNet++, and SFD, when operating on
reconstructed point clouds generated by our proposed method and combined with
image data. The study was conducted using the KITTI validation set, with a
focus on understanding how different configurations of query and neighboring
points influence detection accuracy as measured by the Average Precision (AP)
at an Intersection over Union (IoU) threshold of 0.7.

For the reconstruction of point clouds, our proposed our method takes few
number of query points and a corresponding image. For selecting few query
points, we employ a downsampling strategy III-D on the corresponding LiDAR
frame associated with an image which result in 700 to 800 points. The we use
Farthest Point Sampling (FPS) to select a specific number of 3D points,
exploring configurations of 256 and 512 query points. Each query point serves
as a seed to predict a group of points, effectively reconstructing the spatial
context (neighboring points) around each query location. The number of
neighboring points, either 32 or 64, determines the density of the local point
cloud around each query point. This setup allows us to simulate various levels
of point cloud density and evaluate the robustness of 3D detection methods
under different configurations.

TABLE IV: Ablation study of farthest point sampling and random point sampling. We use our method combined with off-the-shelf detector, MVXNet to check the detection accuracy. Farthest point sampling gives better results. Query | Neighboring | Farthest Point Sampling | Random point Samping
---|---|---|---
Points | Points | AP3DIoU=0.7 | AP3DIoU=0.7
256 | 32 | 63.16 | 61.45
512 | 32 | 68.13 | 64.76
256 | 64 | 62.63 | 57.74
512 | 64 | 67.32 | 63.51
TABLE V: This table show the Reconstruction metrics of our method.  Method | Chamfer Distance | PSNR
---|---|---
Our | 0.058 | 27.54

The results, as depicted in the Table III, show a clear trend: increasing both
the number of query points and the number of neighboring points generally
leads to higher AP scores across all three detectors. Specifically, when
comparing the configuration of 256 query points with 32 neighbors to 512 query
points with the same number of neighbors, a noticeable improvement in
detection performance is observed for MVX-Net (from 63.16 to 68.13), EPNet++
(from 67.87 to 71.08), and SFD (from 69.93 to 73.67). This improvement
underscores the value of higher query point counts in enhancing the quality of
the reconstructed point clouds, thereby facilitating more accurate 3D object
detection.

Conversely, the effect of increasing the number of neighboring points from 32
to 64, while keeping the number of query points constant, presents a more
nuanced outcome. For the 256 query point setup, the AP scores slightly
decrease across all detectors when the number of neighboring points is
doubled. This trend is similarly observed for the 512 query point
configuration. This indicates that while a denser local reconstruction around
each query point might intuitively suggest better performance, the addition of
neighboring points beyond a certain threshold may introduce noise or
redundancy that does not contribute to, or possibly detracts from, the overall
detection accuracy.

These findings highlight the critical balance between the quantity of query
points and the density of their associated local reconstructions (neighboring
points) in optimizing 3D detection performance.

We also conduct another ablation study in the Table IV to show that using
farthest-point-sampling (FPS) on down-sampled point-cloud helps compared to
random-point-sampling (RPS). FPS helps because of better coverage of the
space, reduced redundancy, and adaptability to different densities. Moreover,
to mimic to cost LiDAR frame we add random noise of $\pm$ 1 c.m. To understand
the affect of noise level we also perform ablation study on different noise
level as shown in table VI. Increasing noise level doesn’t affect the
performance that much because query points servers as a seed to reconstruct
neighbouring points around it. Adding different noise levels to query points
doesn’t affect the overall reconstruction hence detection accuracy is remains
intact. We also quantitative result of reconstruction in Table V. We achieve
chamfer distance of 0.058 and PSNR of 27.54.

TABLE VI: Ablation study of noise level to mimic the low resolution and low cost LiDAR frame. When different noise level is injected to low resolution LiDAR frame (query point), we see little change in detection accuracy because query points serves just as seed to generate the neighbouring points. This experiment is conduct with n=512 and k=32 Noise Level | MVX-Net | EPNet++ | SFD
---|---|---|---
centi-meters | AP3DIoU=0.7 | AP3DIoU=0.7 | AP3DIoU=0.7
$\pm$0.5 | 67.92 | 70.81 | 73.12
$\pm$1 | 68.13 | 71.08 | 73.67
$\pm$1.5 | 68.32 | 71.63 | 74.07

##  V Conclusion

Monocular 3D detection lacks the accuracy and robustness required for real
world applications and high resolution 3D scanners are expensive. We presented
a hybrid approach that combines the advantages of monocular and point cloud-
based 3D detection by using only a very small number of 3D points. Using as
low as 512 points (i.e., 1% of a full LiDAR frame in the KITTI dataset) and a
single image, our method reconstructs a complete 3D point cloud that can be
used to improve the performance of existing 3D detectors.

## References

  * [1] Y. Zhou and O. Tuzel, “Voxelnet: End-to-end learning for point cloud based 3d object detection,” in _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 4490–4499.
  * [2] A. H. Lang, S. Vora, H. Caesar, L. Zhou, J. Yang, and O. Beijbom, “Pointpillars: Fast encoders for object detection from point clouds,” in _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 12 689–12 697.
  * [3] S. Shi, C. Guo, L. Jiang, Z. Wang, J. Shi, X. Wang, and H. Li, “Pv-rcnn: Point-voxel feature set abstraction for 3d object detection,” in _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 10 526–10 535.
  * [4] S. Shi, X. Wang, and H. Li, “Pointrcnn: 3d object proposal generation and detection from point cloud,” in _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 770–779.
  * [5] Z. Yang, Y. Sun, S. Liu, and J. Jia, “3dssd: Point-based 3d single stage object detector,” in _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 11 037–11 045.
  * [6] C. He, H. Zeng, J. Huang, X.-S. Hua, and L. Zhang, “Structure aware single-stage 3d object detection from point cloud,” in _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 11 870–11 879.
  * [7] V. A. Sindagi, Y. Zhou, and O. Tuzel, “Mvx-net: Multimodal voxelnet for 3d object detection,” in _2019 International Conference on Robotics and Automation (ICRA)_ , 2019, pp. 7276–7282.
  * [8] C. Wang, C. Ma, M. Zhu, and X. Yang, “Pointaugmenting: Cross-modal augmentation for 3d object detection,” in _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 11 789–11 798.
  * [9] T. Yin, X. Zhou, and P. Krähenbühl, “Multimodal virtual point 3d detection,” in _Neural Information Processing Systems (NeurIPS)_ , 2021.
  * [10] X. Bai, Z. Hu, X. Zhu, Q. Huang, Y. Chen, H. Fu, and C.-L. Tai, “Transfusion: Robust lidar-camera fusion for 3d object detection with transformers,” _arXiv preprint arXiv:2203.11496_ , 2022.
  * [11] K. Huang, T. Wu, H. Su, and W. H. Hsu, “Monodtr: Monocular 3d object detection with depth-aware transformer,” in _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022, pp. 4002–4011.
  * [12] Y. Lu, X. Ma, L. Yang, T. Zhang, Y. Liu, Q. Chu, J. Yan, and W. Ouyang, “Geometry uncertainty projection network for monocular 3d object detection,” in _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2021, pp. 3091–3101.
  * [13] Y. Zhang, J. Lu, and J. Zhou, “Objects are different: Flexible monocular 3d object detection,” in _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 3288–3297.
  * [14] Y. Zhou, Y. He, H. Zhu, C. Wang, H. Li, and Q. Jiang, “Monocular 3d object detection: An extrinsic parameter free approach,” in _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 7552–7562.
  * [15] Z. Zou, X. Ye, L. Du, X. Cheng, X. Tan, L. Zhang, J. Feng, X. Xue, and E. Ding, “The devil is in the task: Exploiting reciprocal appearance-localization features for monocular 3d object detection,” in _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2021, pp. 2693–2702.
  * [16] C. Reading, A. Harakeh, J. Chae, and S. L. Waslander, “Categorical depth distribution network for monocular 3d object detection,” in _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 8551–8560.
  * [17] L. Wang, L. Du, X. Ye, Y. Fu, G. Guo, X. Xue, J. Feng, and L. Zhang, “Depth-conditioned dynamic message propagation for monocular 3d object detection,” in _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 454–463.
  * [18] A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous driving? the kitti vision benchmark suite,” in _2012 IEEE Conference on Computer Vision and Pattern Recognition_ , 2012, pp. 3354–3361.
  * [19] G. Brazil and X. Liu, “M3d-rpn: Monocular 3d region proposal network for object detection,” in _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2019, pp. 9286–9295.
  * [20] A. Simonelli, S. R. Bulò, L. Porzi, M. Lopez-Antequera, and P. Kontschieder, “Disentangling monocular 3d object detection,” in _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2019, pp. 1991–1999.
  * [21] Y. Liu, Y. Yixuan, and M. Liu, “Ground-aware monocular 3d object detection for autonomous driving,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 2, pp. 919–926, 2021.
  * [22] A. Mousavian, D. Anguelov, J. Flynn, and J. Košecká, “3d bounding box estimation using deep learning and geometry,” in _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017, pp. 5632–5640.
  * [23] Y. Chen, L. Tai, K. Sun, and M. Li, “Monopair: Monocular 3d object detection using pairwise spatial relationships,” in _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 12 090–12 099.
  * [24] M. ng, Y. Huo, H. Yi, Z. Wang, J. Shi, Z. Lu, and P. Luo, “Learning depth-guided convolutions for monocular 3d object detection,” in _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , 2020, pp. 4306–4315.
  * [25] D. Rukhovich, A. Vorontsova, and A. Konushin, “Imvoxelnet: Image to voxels projection for monocular and multi-view general-purpose 3d object detection,” in _arXiv preprint arXiv:2106.01178_ , 2021.
  * [26] X. Ma, Z. Wang, H. Li, P. Zhang, W. Ouyang, and X. Fan, “Accurate monocular 3d object detection via color-embedded 3d reconstruction for autonomous driving,” in _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2019, pp. 6850–6859.
  * [27] Y.-N. Chen, H. Dai, and Y. Ding, “Pseudo-stereo for monocular 3d object detection in autonomous driving,” in _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022, pp. 877–887.
  * [28] Y. Yan, Y. Mao, and B. Li, “Second: Sparsely embedded convolutional detection,” in _Sensors_ , 2018.
  * [29] S. Liu and W. Deng, “Very deep convolutional neural network based image classification using small training sample size,” in _2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR)_ , 2015, pp. 730–734.
  * [30] S. Bell, C. L. Zitnick, K. Bala, and R. Girshick, “Inside-outside net: Detecting objects in context with skip pooling and recurrent neural networks,” in _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016, pp. 2874–2883.
  * [31] W. Zhang, H. Zhou, S. Sun, Z. Wang, J. Shi, and C. C. Loy, “Robust multi-modality multi-object tracking,” in _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2019, pp. 2365–2374.
  * [32] M. Tancik, P. P. Srinivasan, B. Mildenhall, S. Fridovich-Keil, N. Raghavan, U. Singhal, R. Ramamoorthi, J. T. Barron, and R. Ng, “Fourier features let networks learn high frequency functions in low dimensional domains,” in _Proceedings of the 34th Conference on Neural Information Processing Systems (NeurIPS)_ , 2020, pp. 10 370–10 379.
  * [33] P. Liu, G. Yang, P. Li, Z. Wang, J. Shi, Z. Deng, and Y. Qiao, “Mp-mono: Monocular 3d detection using multiple priors for autonomous driving,” in _2021 International Conference on 3D Vision (3DV)_ , 2021, pp. 535–544.
  * [34] Y. Zhou, H. Zhu, Q. Liu, S. Chang, and M. Guo, “Monoatt: Online monocular 3d object detection with adaptive token transformer,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2023, pp. 17 493–17 503.
  * [35] R. Zhang, H. Qiu, T. Wang, Z. Guo, Z. Cui, Y. Qiao, H. Li, and P. Gao, “Monodetr: Depth-guided transformer for monocular 3d object detection,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , October 2023, pp. 9155–9166.
  * [36] Z. Liu, T. Huang, B. Li, X. Chen, X. Wang, and X. Bai, “Epnet++: Cascade bi-directional fusion for multi-modal 3d object detection,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 45, no. 7, pp. 8324–8341, 2023.
  * [37] X. Wu, L. Peng, H. Yang, L. Xie, C. Huang, C. Deng, H. Liu, and D. Cai, “Sparse fuse dense: Towards high quality 3d detection with depth completion,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2022, pp. 5418–5427.
  * [38] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: a simple way to prevent neural networks from overfitting,” _Journal of Machine Learning Research_ , vol. 15, no. 1, pp. 1929–1958, 2014\.
  * [39] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” _arXiv preprint arXiv:1711.05101_ , 2017.
  * [40] ——, “Sgdr: Stochastic gradient descent with warm restarts,” in _arXiv preprint arXiv:1608.03983_ , 2016.
  * [41] I. Misra, R. Girdhar, and A. Joulin, “An end-to-end transformer model for 3d object detection,” in _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2021, pp. 2886–2897.