# Classical Poincaré conjecture via 4D topology

Akio KAWAUCHI
Osaka City University Advanced Mathematical Institute
Sugimoto, Sumiyoshi-ku, Osaka 558-8585, Japan
<EMAIL_ADDRESS>
( )

###### ABSTRACT

The classical Poincaré conjecture that every homotopy 3-sphere is
diffeomorphic to the 3-sphere is proved by G. Perelman by solving Thurston’s
program on geometrizations of 3-manifolds. A new confirmation of this
conjecture is given by combining R. H. Bing’s result on this conjecture with
Smooth Unknotting Conjecutre for an $S^{2}$-knot and Smooth 4D Poincaré
Conjecture.

x

Keywords: Homotopy 3-sphere, Smooth unknotting, Smooth homotopy 4-sphere.

Mathematics Subject Classification 2010: Primary 57M40; Secondary 57N13, 57Q45

1\. Introduction

A homotopy 3-sphere is a smooth 3-manifold $M$ homotopy equivalent to the
3-sphere $S^{3}$. It is well-known that a simply connected closed connected
3-manifold is a smooth homotopy 3-sphere. The following theorem, called the
classical Poincaré Conjecture coming from [15, 16] is positively shown by
Perelman [13, 14] solving positively Thurston’s program [17] on
geometrizations of 3-manifolds (see [12] for detailed histrical notes).

x

Theorem 1.1. Every homotopy 3-sphere $M$ is diffeomorphic to the 3-sphere
$S^{3}$.

x

The purpose of this paper is to give an alternative proof to Theorem 1.1 by
combining R. H. Bing’s result in [1, 2] on the classical Poincaré conjecture
with Smooth Unknotting Conjecutre and Smooth 4D Poincaré Conjecture which are
explained from now. Smooth Unknotting Conjecture (for a surface-knot) is the
following conjecutre.

x

Smooth Unknotting Conjecture. Every smooth surface-knot $F$ in the 4-sphere
$S^{4}$ is a trivial surface-knot if the fundamental group
$\pi_{1}(S^{4}\setminus F,x_{0})$ is an infinite cyclic group.

x

The positive proof of this conjecture for any surface-knot is claimed by [7]
with supplement [8]. (See also [9] for a generalization to a surfece-link). A
homotopy 4-sphere is a smooth 4-manifold $X$ homotopy equivalent to the
4-sphere $S^{4}$. Smooth 4D Poincaré Conjecture is the following conjecutre.

x

Smooth 4D Poincaré Conjecture. Every 4D smooth homotopy 4-sphere $X$ is
diffeomorphic to the 4-sphere $S^{4}$.

x

The positive proof of this conjecture is claimed by [10]. The result of R. H.
Bing in [1, 2] used for the proof of Theorem 1.1 is in the form of the
following lemma.

x

Lemma 1.2. A homotopy 3-sphere $M$ is diffeomorphic to the 3-sphere if for
every knot $K$ in $M$, there is a meridian-longitude-preserving isomorphism
from the fundamental gorup $\pi_{1}(M\setminus K,x_{0})$ onto the fumndamental
group $\pi_{1}(S^{3}\setminus K^{\prime},x^{\prime}_{0})$ of a knot
$K^{\prime}$ in $S^{3}$.

x

The proof of Lemma 1.2 is given in Section 2 together with an explanation of
Bing’s result. The main result of this paper to prove the follwing lemma.

x

Lemma 1.3. Every homotopy 3-sphere $M$ meets the assumption of Lemma 1.2.
Namely, for every knot $K$ in $M$, there is a meridian-longitude-preserving
isomorphism from the fundamental gorup $\pi_{1}(M\setminus K,x_{0})$ onto the
fundamental group $\pi_{1}(S^{3}\setminus K^{\prime},x^{\prime}_{0})$ of a
knot $K^{\prime}$ in the 3-sphere $S^{3}$.

x

For the proof of Lemma 1.3, a homotopy 3-sphere version of Artin’s spinning
construction on $S^{3}$ is used. This explanation is done in Section 3. In
Section 4, the proof of Lemma 1.3 is done. Lemmas 1.2 and 1.3 complete the
proof of Theorem 1.1.

Troughout the paper, the notations of the unit disk
$D=\\{x\in{\mathbb{C}}|\,|x|\leq 1\\}$ and the unit circle $S^{1}=\partial D$
are fixed.

x

2\. Proof of Lemma 1.2

The proof of Lemma 1.2 is made as follows.

x

Proof of Lemma 1.2. R. H. Bing’s result in [1, 2] is stated as follows:

x

Bing’s Theorem. A homotopy 3-sphere $M$ is diffeomorphic to $S^{3}$ if, for
every knot $K$ in $M$, there is a 3-ball in $M$ containing the knot $K$.

x

Let $K$ be a knot in $M$ with a meridian-longitude preserving isomorphism

$\pi_{1}(M\setminus K,x_{0})\cong\pi_{1}(S^{3}\setminus
K^{\prime},x^{\prime}_{0})$

for a knot $K^{\prime}$ in $S^{3}$. Let $N(K)$ be a tubular neighborhood of
the knot $K$ in $M$. Let $E(K;M)=\mbox{cl}(M\setminus N(K))$ be the compact
exterior of the knot $K$ in $M$. By H. Kneser’s prime factorization of a
compact 3-manifold in [11] (see [4, p.31], the compact 3-manifold $E(K;M)$ is
written as a connected sum

$E(K;M)\cong E^{\prime}\\#M_{1}\\#M_{2}\\#\dots\\#M_{k}$

for an irreducible compact connected 3-manifold $E^{\prime}$ with torus
boundary and finitely many irreducible homotopy 3-spheres
$M_{i}\,(i=1,2,\dots,k)$. The 3-manifold $E^{\prime}$ is understood as the
compact exterior $E(K;M^{\prime})$ of a knot $K$ in a homotopy 3-sphere
$M^{\prime}$. Note that the meridian-longitude system of $E(K;M)$ is taken
over by a meridian-longitude system of the irreducible 3-manifold
$E^{\prime}$. Thus, there is a meridian-longitude-preserving isomorphism

$\pi_{1}(E^{\prime},x_{0})\cong\pi_{1}(E(K^{\prime}),x^{\prime}_{0})$

for the compact exterior $E(K^{\prime})=\mbox{cl}(S^{3}\setminus
N(K^{\prime}))$ of the knot $K^{\prime}$ in $S^{3}$.

If $\pi_{1}(E^{\prime},x_{0})\cong{\mathbf{Z}}$, then by Dehn’s lemma (cf.
e.g. [6]), the compact 3-manifold $E^{\prime}$ is diffeomorphic to the product
$S^{1}\times D$ by a diffeomorphism sending the meridian-longitude pair of
$E^{\prime}$ to the pair $(1\times S^{1},S^{1}\times 1)$. Then we see the knot
$K$ is a trivial knot in a 3-ball in $M$.

Assume that $\pi_{1}(E^{\prime},x_{0})\not\cong{\mathbf{Z}}$. Then the compact
3-manifold $E^{\prime}$ is a Haken manifold and hence by F. Waldhausen’s
result in [18] (see [4]), there is a meridian-longitude-preserving
diffeomorphism $E^{\prime}\cong E(K^{\prime})$ (cf. [6, p. 74]), which means
that the knot $K$ is also in a 3-ball in $M$.

Thus, it is shown that for any knot in $M$ with a meridian-longitude
preserving isomorphism $\pi_{1}(M\setminus K,x_{0})\cong\pi_{1}(S^{3}\setminus
K^{\prime},x^{\prime}_{0})$ for a knot $K^{\prime}$ in $S^{3}$, it is shown
that there is a 3-ball $B$ containing the knot $K$. Since the assumption of
Lemma 1.2 is that every knot in $M$ admits a meridian-longitude preserving
isomorphism $\pi_{1}(M\setminus K,x_{0})\cong\pi_{1}(S^{3}\setminus
K^{\prime},x^{\prime}_{0})$ for a knot $K^{\prime}$ in $S^{3}$, Bing’s theorem
means that $M$ is diffeomorphic to $S^{3}$. This completes the proof of Lemma
1.2. $\square$

x

3\. Artin’s pinning construction

Let $M$ be a smooth homotopy 3-sphere, and $M^{o}$ the compact once-punctured
manifold $\mbox{cl}(M\setminus B)$ of $M$ for a 3-ball $B$ in $M$. The closed
smooth 4-manifold

$\Sigma^{M}_{B}=(\partial B)\times D\cup M^{o}\times S^{1}$

is a smooth homotopy 4-sphere by van Kampen theorem and a homological
argument, which is called the Artin’s spinning construction of the pair
$(M,B)$.

The product $\Lambda^{\prime}\times D$ for a subspace
$\Lambda^{\prime}\subset\partial B$ in $(\partial B)\times D$, the product
$\Lambda^{\prime\prime}\times S^{1}$ for a subspace
$\Lambda^{\prime\prime}\subset M^{0}$ in $M^{o}\times S^{1}$, or the union
$\Lambda^{\prime}\times D\cup\Lambda^{\prime\prime}\times S^{1}$ in $S(M,B)$
is called a round object.

By Smooth 4D Poincaré Conjecture, the closed smooth 4-manifold
$\Sigma^{M}_{B}$ is deffeomorphic to the 4-sphere $S^{4}$. For a knot $K$ in
the interior of $M^{o}$, the spun torus-knot $K\times S^{1}$ in the 4-sphere
$\Sigma^{M}_{B}$ is a round object uniquely given by the inclusions

$K\times S^{1}\subset M^{o}\times S^{1}\subset\Sigma^{M}_{B}.$

Let $c:K\times[0,1]\to M^{o}$ be a collar embedding of $K$ in $M^{o}$ such
that $c(K,0)=K$ and $\ell(K)=c(K,1)$ is a longitude of $K$ in $M^{o}$. For a
base point $p\in K$, let $\bar{\ell}(S^{1})=c(p,1)\times S^{1}$ and
$\bar{\ell}(K)=c(K,1)\times 1\subset M^{0}\times S^{1}$ be the simple loops
around the spun torus-knot $K\times S^{1}$ in the 4-sphere $\Sigma^{M}_{B}$.
Take the point $x_{0}=c(p,1)\in M^{o}\setminus K$ as the base point of the
longitude $\ell(K)$ of $K$ and the point $\bar{x}_{0}=c(p,1)\times
1\in\Sigma^{M}_{B}\setminus K\times S^{1}$ as the base point of the loops
$\bar{\ell}(S^{1})$ and $\bar{\ell}(K)$. Let $m(K)$ be a meridian loop of $K$
in $M^{o}$ with base point $x_{0}$, and ${\bar{m}}(K\times S^{1})$ a meridian
loop of $K\times S^{1}$ in $\Sigma^{M}_{B}$ with base point $\bar{x}_{0}$,
respectively. The following lemma is directly obtained by van Kampen theorem.

x

Lemma 3.1. For every knot $K$ in the interior of $M^{o}$ and the spun torus-
knot $K\times S^{1}$ in the 4-sphere $\Sigma^{M}_{B}$, the fundamental group
$\pi_{1}(\Sigma^{M}_{B}\setminus K\times S^{1},\bar{x}_{0})$ is isomorphic to
the fundamental group $\pi_{1}(M^{o}\setminus K,x_{0})$ by an isomprphism
sending the homotopy class $[\bar{\ell}(S^{1})]$ to the trivial element
$\\{1\\}$, the homotopy class $[\bar{\ell}(K)]$ to the homotopy class
$[\ell(K)]$ and the meridian element $[\bar{m}(K\times S^{1})]$ to the
meridian element $[m(K)]$, respectively.

x

Proof of Lemma 3.1. Apply van Kampen theorem for $(M^{o}\setminus K)\times
S^{1}$ and $(\partial B)\times D$ to obtaine the desired result. $\square$

x

The following lemma is essentially a corollary to Lemma 3.1.

x

Lemma 3.2. For a 3-ball $B$ and the standard spin structure on the product
$B\times S^{1}$, let $\varphi:B\times S^{1}\to S^{4}$ be a spin-preserving
smooth embedding. For a knot $K$ in the interior of $B$, the torus-knot
$K\times S^{1}$ in $S^{4}$ given by the composite embedding

$K\times S^{1}\subset B\times
S^{1}\stackrel{{\scriptstyle\varphi}}{{\longrightarrow}}S^{4}$

has the fundamental group $\pi_{1}(S^{4}\setminus K\times
S^{1},x^{\prime}_{0})$ which is isomorphic to the fundamental group
$\pi_{1}(B\setminus K,x_{0})$ by an isomprphism sending the homotopy class
$[\bar{\ell}(S^{1})]$ to the trivial element $\\{1\\}$, the homotopy class
$[\bar{\ell}(K)]$ to the homotopy class $[\ell(K)]$ and the meridian element
$[\bar{m}(K\times S^{1})]$ to the meridian element $[m(K)]$, respectively.

x

Proof of Lemma 3.2. Let $S^{4}=(\partial B)\times D\cup B\times S^{1}$ be the
union of round objects $(\partial B)\times D$ and $B\times S^{1}$. The
embedding $\varphi$ is smoothly isotopic to an embedding
$\varphi^{\prime}:B\times S^{1}\to S^{4}$ with $\varphi^{\prime}(B\times
S^{1})=B\times S^{1}$. By the isotopy classification of the orientable handle
$S^{1}\times S^{2}$ by H. Gluck [3] and the spin assumption on $\varphi$, we
can assume that the restriction of $\varphi^{\prime}$ to the orientable handle
$(\partial B)\times S^{1}$ is the identity map. Hence the embedding
$\varphi^{\prime}:B\times S^{1}\to S^{4}$ extends to a diffeomorphism
$(\varphi^{\prime})^{+}:S^{4}\to S^{4}$ whose restriction to the round object
$(\partial B)\times D$ is the identity map. By Lemma 3.1, the desired result
is obtained. $\square$

x

For a connected spatial graph $\Gamma$ in $M$ and a maximal tree $T$ of
$\Gamma$, let $B$ be a regular neighborhood of $T$ in $M$ which is a 3-ball.
Let $a(\Gamma)=\mbox{cl}(\Gamma\setminus B\cap\Gamma)$ be a proper arc system
in $M^{o}=\mbox{cl}(M\setminus B)$, and $\dot{a}(\Gamma)=(\partial
B)\cap\Gamma$ the boundary point system in the 2-sphere $\partial
M^{0}=\partial B$. The spun $S^{2}$-link $L^{\Gamma}_{T}$ is a round object
uniquely constructed from the pair $(\Gamma,T)$ in the 4-sphere
$\Sigma^{M}_{B}$ by taking the union of the annulus system $a(\Gamma)\times
S^{1}$ and the disk system $\dot{a}(\Gamma)\times D$ as follows:

$L^{\Gamma}_{T}=a(\Gamma)\times S^{1}\cup\dot{a}(\Gamma)\times D.$

The following lemma is a generalization of Lemma 3.1 to the spun $S^{2}$-link,
which was observed in [6, p.204].

x

Lemma 3.3. For every connected spatial graph $\Gamma$ in $M$ and the spun
$S^{2}$-link $L^{\Gamma}_{T}$ in the 4-sphere $\Sigma^{M}_{B}$, the
fundamental group $\pi_{1}(\Sigma^{M}_{B}\setminus
L^{\Gamma}_{T},\bar{x}_{0})$ is isomorphic to the fundamental group
$\pi_{1}(M\setminus\Gamma,x_{0})$ by an isomprphism sending the meridian
system of $L^{\Gamma}_{T}$ to a meridian system of the proper arc system
$a(\Gamma)$ in $M^{o}$.

x

Proof of Lemma 3.3. Note that there is a canonical isomorphism

$\pi_{1}(M^{0}\setminus a(\Gamma),x_{0})\cong\pi_{1}(M\setminus\Gamma,x_{0}).$

Then apply van Kampen theorem between $(M^{o}\setminus a(\Gamma))\times S^{1}$
and $((\partial B)\setminus\dot{a}(\Gamma))\times D$. The result is obtained.
$\square$

Figure 1: The round object $Y=\Gamma\times S^{1}\cup\beta\times D$

x

Let $V$ be a handlebody of genus $n$ in $M$, and $F=\partial V$. Notions on
graphs in the surface $F$ are introduced as follows:

Let $\beta$ be a simple arc in $F$. Let $O_{i}\,(i=1,2,\dots,n)$ be a system
of mutually disjoint simple loops in the surface $F$ with $\beta\cap
O_{i}=\emptyset$ for every $i$, and $v_{i}\,(i=1,2,\dots,n)$ a system of
distinct points in $\beta$ enumerated in this order with
$\partial\beta=\\{v_{0},v_{n}\\}$. Let $a_{i}\,(i=1,2,\dots,n)$ be a system of
mutually disjoint simple arcs in $F$ such that $a_{i}$ spans the point $v_{i}$
and a point $p_{i}\in O_{i}$ with interior $\mbox{Int}(a_{i})$ disjoint from
$\beta\cup_{i=1}^{n}O_{i}$. The graph

$\Gamma=\beta\cup_{i=1}^{n}(a_{i}\cup O_{i})$

is called an $n$-bouquet graph with based arc $\beta$ in $F$. Let
$P=N(\Gamma;F)$ be a regular neighborhood of an $n$-bouquet graph $\Gamma$
with based arc $\beta$ in $F$, which is a planar surface in $F$. Slide the
based arc $\beta$ of the $n$-bouquet graph $\Gamma$ to a boundary component of
$P$. Then the $n$-bouquet graph $\Gamma$ is a spine graph of the handlebody
$V$ with based arc $\beta$ if the framings of the loops $O_{i}$ determined by
$P$ are $0$-framings in $M$ and there is a diffeomorphism

$c:P\times[0,1]\to V$

such that $c(x,0)=x$ for all $x\in P$. This planar surface $P$ in $F$ is
called a spine surface of $V$. We have always a spine graph $\Gamma$ and a
spine surface $P$ of $V$ with based arc $\beta$ for every handlebody $V$ of
genus $n$. For a maximal tree $T$ of $\Gamma$ around the arc $\beta$ and a
3-ball neighborhood $B$ of $T$ in $M$, the spun $S^{2}$-link $L^{\Gamma}_{T}$
in the 4-sphere $\Sigma^{M}_{B}$ is constructed from the pair $(\Gamma,T)$.

To construct a slightly different construction of a $S^{2}$-link in the
4-sphere $\Sigma^{M}_{B}$ equivalent to the spun $S^{2}$-link
$L^{\Gamma}_{T}$, consider a 3-ball $B$ in $M$ so that the intersection

$B\cap V=(\partial B)\cap F=\Delta$

is a disk and the intersection

$B\cap\Gamma=(\partial B)\cap\Gamma=\beta$

is an arc in the boundary circle $\partial\Delta$ of the disk $\Delta$ so that
the disk $\Delta$ is considered as the product $[0,1]\times\beta$ with
$0\times\beta=\beta$. Let

$Y=\beta\times D\cup\Gamma\times S^{1}$

be a round object in $\Sigma^{M}_{B}$ which is the union of the solid cylinder
$\beta\times D$ and the round object $\Gamma\times S^{1}$ for the spine graph
$\Gamma$ of $V$ with based arc $\beta$. The round object $Y$ is illustrated in
Fig. 1 where note that the solid cylinder $\beta\times D$ is fixed setwise
under the one rotation on $Y$ along the central axis $\beta\times 0$. In Fig.
1, we find mutually disjoint disks

$D_{i}=v_{i}\times D\cup a_{i}\times S^{1}\quad(i=1,2,\dots,n).$

Let $h^{2}_{i}\,(i=1,2,\dots,n)$ be mutually disjoint 2-handles on the tori
$O_{i}\times S^{1}\,(i=1,2,\dots,n)$ thickening the disks
$D_{i}\,(i=1,2,\dots,n)$, respectively. The 2-sphere components of
$L^{\Gamma}_{T}$ are equivalent to the 2-spheres $S_{i}\,(i=1,2,\dots,n)$
obtained from the tori $O_{i}\times S^{1}\,(i=1,2,\dots,n)$ by the surgeries
along the 2-handles $h^{2}_{i}\,(i=1,2,\dots,n)$, respectively. The
$S^{2}$-link $L^{\Gamma}_{\beta}=\cup_{i=1}^{n}S_{i}$ in $\Sigma^{M}_{B}$
constructed in this way is a round object and called the spun $S^{2}$-link
constructed from the $n$-bouquet graph $\Gamma$ with based arc $\beta$. The
2-handle $h^{2}_{i}$ on the torus $O_{i}\times S^{1}$ is considered as a
1-handle $h^{*1}_{i}$ on the 2-sphere component $S_{i}$. Note that this
construction is done in the 3-manifold

$A(P)=\beta\times D\cup P\times S^{1}.$

In the round object $Y$, let $\beta_{j}\,(j=1,2,\dots,n-1)$ be the arcs
obtained by dividing the central axis $\beta\times 0$ of the solid cylinder
$\beta\times D$ at the points $v_{i}\times 0\,(i=1,2,\dots,n)$ such that the
arc $\beta_{j}$ connects $v_{j}\times 0$ to $v_{j+1}\times 0$ for every
$j\,(j=1,2,\dots,n-1)$, so that the arc $\beta_{j}$ connects the disk $D_{j}$
to the disk $D_{j+1}$ (see Fig. 1). The arc $\beta_{j}$ is a core arc of the
1-handle $h^{1}_{j}=\beta_{j}\times D\,$ on $L^{\Gamma}_{\beta}$ connecting
the component $S_{j}$ to the component $S_{j+1}$ for all
$j\,(j=1,2,\dots,n-1)$. Let $\alpha_{i}\,(i=1,2,\dots,n)$ be the core arcs of
the 1-handles $h^{*1}_{i}\,(i=1,2,\dots,n)$ in the central axis $\beta\times
0$ of the solid cylinder $\beta\times D$, so that the union

$\cup_{i=1}^{n}\alpha_{i}\cup_{j=1}^{n-1}\beta_{j}$

is a division of the central axis $\beta\times 0$ of the solid cylinder
$\beta\times D$. Thus the $S^{2}$-link $L^{\Gamma}_{\beta}$ with divided
central axis $\beta\times 0=\cup_{i=1}^{n}\alpha_{i}\cup_{j=1}^{n-1}\beta_{j}$
is obtained (see Fig. 2). Note that this construction is also done in the
3-manifold $A(P)$. We have the following lemma.

x

Figure 2: The $S^{2}$-link $L^{\Gamma}_{\beta}$ with divided central axis
$\beta\times 0=\cup_{i=1}^{n}\alpha_{i}\cup_{j=1}^{n-1}\beta_{j}$

Lemma 3.4. The 3-manifold $A(P)$ is obtained from a regular neighborhood of
the $S^{2}$-link $L^{\Gamma}_{\beta}$ with divided central axis $\beta\times
0$ in $A(P)$ by adding a boundary collar $(\partial A(P))\times[0,1]$ of
$A(P)$ and the 4-manifold $\Delta\times D\cup V\times S^{1}$ is a collar
$A(P)\times[0,1]$ of the 3-manifold $A(P)$ in $\Sigma^{M}_{B}$.

x

Proof of Lemma 3.4. By constructuin, the round object $Y$ is naturally
contained in $A(P)$ ant the closed complement $\mbox{cl}(A(P)\setminus N(Y))$
for a regular neighborhood $N(Y)$ of $Y$ in $A(P)$ is a boundary collar
$(\partial A(P))\times[0,1]$. Since

$h^{2}_{i}\cup O_{i}\times S^{1}=h^{*1}_{i}\cup S_{i}$

for every $i$, the closed complement of a regular neighborhood of the union

$L^{\Gamma}_{\beta}\cup_{i=1}^{n}h^{*1}_{i}\cup_{j=1}^{n-1}h^{1}_{j}$

in $A(P)$ is also a boundary collar $(\partial A(P))\times[0,1]$. Thus, the
closed complement of a regular neighborhood of the union

$L^{\Gamma}_{\beta}\cup_{i=1}^{n}\alpha_{i}\cup_{j=1}^{n-1}\beta_{j}$

in $A(P)$ is also a boundary collar $(\partial A(P))\times[0,1]$. Since the
handlebodey $V$ is a collar $P\times[0,1]$ in $\Sigma^{M}_{B}$ and the disk
$\Delta$ is the product $[0,1]\times\beta$ with $0\times\Gamma=\beta$, we see
that the 4-manifold $\Delta\times D\cup V\times S^{1}$ is a collar
$A(P)\times[0,1]$. $\square$

x

The following lemma is technically important to the proof of Lemma 1.3.

x

Lemma 3.5. Let $L^{\Gamma}_{\beta}$ be the $S^{2}$-link with divided central
axis $\beta\times 0$ in $\Sigma^{M}_{B}$ for a handlebody $V$ in $M$, and
$\bar{S}$ the $S^{2}$-knot $\bar{S}$ obtained from the spun $S^{2}$-link
$L^{\Gamma}_{\beta}$ by the surgeries along the 1-handles
$h^{1}_{j}=\beta_{j}\times D\,(i=1,2,\dots,n-1)$. If the $S^{2}$-knot
$\bar{S}$ is a trivial $S^{2}$-knot in $\Sigma^{M}_{B}$, then the round object
$V\times S^{1}$ extends to the product $U\times S^{1}$ for a 3-ball $U$ which
is a smooth spin 4-submanifold of $\Sigma^{M}_{B}$.

x

Proof of Lemma 3.5. Note that the central axis $\beta\times 0$ spans the
trivial 2-sphere $\bar{S}$. Let $\tilde{B}$ be a 3-ball in $\Sigma^{M}_{B}$
bounded by the $S^{2}$-knot $\bar{S}$. By [5], the central axis $\beta\times
0$ is moved to an arc $\tilde{\beta}$ in $\tilde{B}$ which is paralell to the
2-sphere $\bar{S}$ by a diffeomorphism of the 4-sphere $\Sigma^{M}_{B}$
isotopic to the identity by an isotopy keeping the 2-sphere $\bar{S}$ fixed
and $L^{\Gamma}_{\beta}$ setwise fixed. Hence there is a diffeomorphism $f$ of
the 4-sphere $\Sigma^{M}_{B}$ isotopic to theidentity by an isotopy keeping
the 2-sphere $\bar{S}$ fixed and $L^{\Gamma}_{\beta}$ setwise fixed such that
$f$ sends $L^{\Gamma}_{\beta}\cup\beta\times 0$ to itself identically and and
the central axis $\beta\times 0$ is unknotted in the 3-ball $f(\tilde{B})$. By
the spin assumption on the spine $P$ of $V$, we can assume $f$ sends a tubular
neighborhood $\beta\times d$ of $\beta\times 0$ in $\beta\times D$ to a
tubular neighborhood $\beta\times d$ of $\beta\times 0$ in $f(\tilde{B})$
identically, so that $f$ sends $L^{\Gamma}_{\beta}\cup\beta\times d$ to itself
identically. Assume that

$L^{\Gamma}_{\beta}\cup\beta\times d\subset\tilde{B}$

in $\Sigma^{M}_{B}$. Let $d^{\prime}$ be a disk in the interior of $d$. Let
$L^{\Gamma}_{\beta}\times[0,1]$ be a collar of $L^{\Gamma}_{\beta}$ in
$\Sigma^{M}_{B}$ so that the collars of the disks $v_{i}\times
d,\,(i=1,2,\dots,n)$ belong to $\beta\times d$. Then the 3-manifold

$\mbox{cl}(L^{\Gamma}_{\beta}\times[0,1]\cup\beta\times d)\setminus\beta\times
d^{\prime})$

is canonically diffeomorphic to $P\times S^{1}$ in $\Sigma^{M}_{B}$ and is
included in the solid torus $\mbox{cl}(\tilde{B}\setminus\beta\times
d^{\prime})$. By the spin assumption on $P$, this means that there is an
inclusion

$P\times S^{1}\subset U_{1}\times S^{1}$

for a disk $U_{1}$ in $\Sigma^{M}_{B}$ with $U_{1}\times S^{1}$ as a spin
submanifold. Since $V\times S^{1}$ is a collar of $P\times S^{1}$ in $S(M,B)$
by Lemma 3.4, there is an inclusion $V\times S^{1}\to U\times S^{1}$ for a
3-ball $U$ in $S(M,B)$. By construction, the 4D solid torus $U\times S^{1}$ is
a smooth spin 4-manifold of $S(M,B)$. $\square$

x

4\. Proof of Lemma 1.3

Now we are in a position to the proof of Lemma 1.3.

x

Proof of Lemma 1.3. Let $M$ be a homotopy 3-sphere. Let $V\cup V^{\prime}$ be
a Heegaard splitting of $M$ of a Heegaard genus $n$ for some $n$ with Heegaard
surface $F=\partial V=\partial V^{\prime}$. Let $\Gamma$ be a spine graph of
the handlebody $V$ with a based arc $\beta$ in $F$. In the notation of Section
3, the $n$-bouquet graph $\Gamma$ with based arc $\beta$ is written as
$\Gamma=\beta\cup_{i=1}^{n}(a_{i}\cup O_{i})$ in $F$. Similarly, let
$\Gamma^{\prime}$ be a spine graph of the handlebody $V^{\prime}$ with the
same based arc $\beta$ as $\Gamma$ but with different vertices in $\beta$.
Push the part $\Gamma^{\prime}\setminus\beta$ into the interior of
$V^{\prime}$ by using a boundary collar of $F$ in $V^{\prime}$. Let
$\bar{\Gamma}=\Gamma\cup\Gamma^{\prime}$ be a $2n$-bouquet graph in $M$ with
the based arc $\beta$. Let $B$ be a 3-ball used for the construction of the
4-sphere $\Sigma^{M}_{B}$ such that

$B\cap V=(\partial B)\cap F=\Delta$

is a disk and

$B\cap\bar{\Gamma}=(\partial B)\cap\bar{\Gamma}=\beta$

is a boundary arc in the disk $\Delta$ so that the disk $\Delta$ is considered
as the product $[0,1]\times\beta$ with $0\times\beta=\beta$. Let
$M^{o}=\mbox{cl}(M\setminus B)$. Let $N(\bar{\Gamma})$ be a regular
neighborhood of $\bar{\Gamma}$ in $M^{0}$. The union
$\bar{N}(\bar{\Gamma})=N(\bar{\Gamma})\cup B$ is a regular neighborhood of
$\Gamma$ in $M$. By construction, the closed complement
$\mbox{cl}(M^{o}\setminus\bar{N}(\bar{\Gamma}))$ is diffeomorphic to the
product $F^{o}\times[0,1]$ for the once-punctured compact surface of $F$
obtained from $F$ by removing the interior of a disk $\Delta$ in $F$. This
means that the fundamental group
$\pi_{1}(\mbox{cl}(M\setminus\bar{N}(\bar{\Gamma})),x_{0})$ is a free group of
rank $2n$ given by a free product ${\mathbf{F}}*{\mathbf{F}}^{\prime}$ where
the free direct product summand ${\mathbf{F}}$ has a basis represented by a
meridian loop system of $O_{i}\,(i=1,2,\dots,n)$. Note that some meridian
loops of $O^{\prime}_{i}\,(i=1,2,\dots,n)$ do not always represent elements of
the free product summand ${\mathbf{F}}^{\prime}$ (see Example 4.1 later for an
example). Let $x^{\prime}_{i}\,(i=1,2,\dots,n)$ be a basis of the free group
${\mathbf{F}}^{\prime}$. Let

$L^{\bar{\Gamma}}_{\bar{\beta}}=L^{\Gamma}_{\beta}\cup
L^{\Gamma^{\prime}}_{\beta}$

be the $2n$-component spun $S^{2}$-link of the $2n$-bouquet graph
$\bar{\Gamma}$ with based arc $\beta$ in the 4-sphere $\Sigma^{M}_{B}$. The
$n$ component spun $S^{2}$-link $L^{\Gamma}_{\beta}$ of the $n$-bouquet graph
$\Gamma$ with based arc $\beta$ is an $S^{2}$-sublink of the $S^{2}$-link of
$L^{\bar{\Gamma}}_{\beta}$. By van Kampen theorem, the fundamental group
$\pi_{1}(\Sigma^{M}_{B}\setminus L^{\bar{\Gamma}}_{\beta},\bar{x}_{0})$ is
canonically isomorphic to the fundamental group
$\pi_{1}(\mbox{cl}(M\setminus\bar{N}(\bar{\Gamma}),x_{0})$ which is the free
product ${\mathbf{F}}*{\mathbf{F}}^{\prime}$ where a meridian system of
$L^{\Gamma}_{\beta}$ represents the basis $x_{i}\,(i=1,2,\dots,n)$ of
${\mathbf{F}}$.

Let $\alpha_{i}\,(i=1,2,\dots,n)$ and $\beta_{j}\,(j=1,2,\dots,n-1)$ be the
arc systems in the central axis $\beta\times 0$ used for the spun $S^{2}$-link
$L^{\Gamma}_{\beta}$ with central axis $\beta\times 0$. Let $\bar{S}$ be the
2-sphere obtained from the spun $S^{2}$-link $L^{\Gamma}_{\beta}$ by the
surgery along the 1-handles $h^{1}_{j}\,(j=1,2,\dots,n-1)$ with the core arc
system $\beta_{j}\,(j=1,2,\dots,n-1)$. The 2-sphere $\bar{S}$ is an
$S^{2}$-knot in $\Sigma^{M}_{B}$ which is a round object. By construction,
note that the free product summand ${\mathbf{F}}$ changes into the infinit
cyclic group ${\mathbf{Z}}$ by the epimorphism sending the basis element
$x_{i}$ to the generator $1\in{\mathbf{Z}}$ of ${\mathbf{Z}}$ for every $i$,
so that the $S^{2}$-link $L^{\bar{\Gamma}}_{\beta}$ changes into the
$(1+n)$-component $S^{2}$-link $\tilde{L}=\bar{S}\cup
L^{\Gamma^{\prime}}_{\beta}$ in $\Sigma^{M}_{B}$ with a fundamental group
isomorphism

$\pi_{1}(\Sigma^{M}_{B}\setminus\tilde{L},\bar{x}_{0})\cong{\mathbf{Z}}*{\mathbf{F}}^{\prime}$

sending the meridian element of the component $\bar{S}$ to the generator
$1\in{\mathbf{Z}}$. This means that the fundamental group
$\pi_{1}(\Sigma^{M}_{B}\setminus\bar{S},\bar{x}_{0})$ is a quotient group of
the infinite cyclic group ${\mathbf{Z}}$. Since the first homology group
$H_{1}(\Sigma^{M}_{B}\setminus\bar{S};{\mathbf{Z}})\cong{\mathbf{Z}}$, the
fundamental group $\pi_{1}(\Sigma^{M}_{B}\setminus\bar{S},\bar{x}_{0})$ must
be isomorphic to the infinite cyclic group ${\mathbf{Z}}$. By Smooth
Unknotting Conjecutre, the $S^{2}$-knot $\bar{S}$ is a trivial $S^{2}$-knot.
By Lemma 3.5, the round object $V\times S^{1}$ extends to a smooth spin
4-submanifold $U\times S^{1}$ of $\Sigma^{M}_{B}$ for a 3-ball $U$.

Every knot $K$ in $M$ is isotopic to a knot $K^{\prime}$ in $V$, so that we
have the following inclusions:

$K^{\prime}\times S^{1}\subset V\times S^{1}\subset U\times
S^{1}\subset\Sigma^{M}_{B}.$

By Lemmas 3.1 and 3.2, there are meridian-longitude-preserving isomorphisms

$\pi_{1}(M\setminus K,x)\cong\pi_{1}(U\setminus
K^{\prime},x^{\prime})\cong\pi_{1}(DU\setminus K^{\prime},x^{\prime})$

for the fundamental groups $\pi_{1}(M\setminus K,x)$ and $\pi_{1}(U\setminus
K^{\prime},x^{\prime})$, where $DU=\partial(U\times[0,1])=U\times 0\cup
U\times 1$ denotes the double of the 3-ball $U$ with $U\times 0=U$ which is
diffeomorphic to the 3-sphere $S^{3}$. This completes the proof of Lemma 1.3.
$\square$

x

This completes the proof of Theorem 1.1.

x

In the proof of Lemma 1.3, although the basis elements
$x_{i}\,(i=1,2,\dots,n)$ of the free product summand ${\mathbf{F}}$ are
represented by a meridian system of the $S^{2}$-sublink $L^{\Gamma}_{\beta}$
of the $S^{2}$-link $L(\bar{\Gamma},\beta)$, the $S^{2}$-sublink
$L^{\Gamma}_{\beta}$ itself is not necessarily a trivial $S^{2}$-link. Here is
an example.

x

Figure 3: A 2-bouquet graph $\Gamma$ with based arc $\beta$ whose funfamental
group is a free group of rank $2$

Example 4.1. Let $\Gamma$ be a $2$-bouquet graph with based arc $\beta$ in
$S^{3}$ illustrated in Fig. 3 whose fundamental group
$\pi_{1}(S^{3}\setminus\Gamma,x_{0})$ is a free group of rank $2$. A regular
neighborhood $V$ of $\Gamma$ in $S^{3}$ and the closed complement
$V^{\prime}=\mbox{cl}(S^{3}\setminus V)$ constitute a genus $2$ Heegaard
splitting $V\cup V^{\prime}$ of $S^{3}$ by noting that the 3-manifold
$V^{\prime}$ is shown to be a handlebody of genus $2$ by the loop theorem and
the Alexander theorem (cf. e.g. [6]). The graph $\Gamma$ with based arc
$\beta$ is used to be a spine graph $\Gamma$ of $V$ by moving the based arc
$\beta$ to the Heegaard surface $F=\partial V=\partial V^{\prime}$. Let
$\Gamma^{\prime}$ be any spine graph of $V^{\prime}$ with base arc $\beta$
obtained by pushing $\Gamma^{\prime}\setminus\beta$ into the interior of
$V^{\prime}$. Let $L^{\bar{\Gamma}}_{\beta}=L^{\Gamma}_{\beta}\cup
L^{\Gamma^{\prime}}_{\beta}$ be the 4-component spun $S^{2}$-link in the
4-sphere $\Sigma^{S^{3}}_{B}$ constructed from the pair
$(\bar{\Gamma},\beta)$. The fundamental group
$\pi_{1}(\Sigma^{S^{3}}_{B}\setminus L^{\bar{\Gamma}}_{\beta},\bar{x}_{0})$ is
canonically isomorphic to the free group $\pi_{1}(\mbox{cl}(S^{3}\setminus
N(\bar{\Gamma}),x_{0})$ of rank $4$ which is the free product
${\mathbf{F}}*{\mathbf{F}}^{\prime}$ of free groups ${\mathbf{F}}$ and
${\mathbf{F}}^{\prime}$ of rank $2$ such that a basis of ${\mathbf{F}}$ is
represented by a meridian system of $L^{\Gamma}_{\beta}$. The $2$-component
$S^{2}$-sublink $L^{\Gamma}_{\beta}$ is not a trivial $S^{2}$-link, because
$L^{\Gamma}_{\beta}$ contains a component of the spun trefoil $S^{2}$-knot in
$S^{4}$. Thus, the fundamental group $\pi_{1}(\Sigma^{S^{3}}_{B}\setminus
L^{\Gamma}_{\beta},\bar{x}_{0})$ is not any meridian-based free group. This
also implies that there is a meridian element of $L^{\Gamma^{\prime}}_{\beta}$
in the fundamental group $\pi_{1}(\Sigma^{S^{3}}_{B}\setminus
L^{\bar{\Gamma}}_{\beta},\bar{x}_{0})$ which is not conjugate to any element
of the free product summamd ${\mathbf{F}}^{\prime}$, because otherwise the
fundamental group $\pi_{1}(\Sigma^{S^{3}}_{B}\setminus
L^{\Gamma}_{\beta},\bar{x}_{0})$ would be a meridian-based free group.

x

Acknowledgments. This work was partly supported by Osaka City University
Advanced Mathematical Institute (MEXT Joint Usage/Research Center on
Mathematics and Theoretical Physics JPMXP0619217849).

x

## References

  * [1] R. H. Bing,Necessary and sufficient conditions that a 3-manifold be $S^{3}$, Ann. of Math. 68 (1958), 17-37.
  * [2] R. H. Bing, Some aspects of the topology of 3-manifolds related to the Poincaré conjecture, in Lectures on Modern Mathematics II (T. L. Saatyed.), Wiley, 1964.
  * [3] H. Gluck, The embedding of two-spheres in the four-sphere, Trans. Amer. Math. Soc. 104 (1962), 308-333.
  * [4] J. Hempel, 3-manifolds, Ann. Math. Studies 86 (1976), Princeton Univ. Press.
  * [5] F. Hosokawa and A. Kawauchi, Proposals for unknotted surfaces in four-space, Osaka J. Math. 16(1979), 233-248.
  * [6] A. Kawauchi, A survey of knot theory, Birkhäuser (1996).
  * [7] A. Kawauchi, Ribbonness of a stable-ribbon surface-link, I. A stably trivial surface-link, Topology and its Applications (to appear). arxiv.org/abs/1804.02654
  * [8] A. Kawauchi, Supplement to ribbonness of a stable-ribbon surface-link, I: A revised proof of uniqueness of an orthogonal 2-handle pair on a surface-link (preprint version). arxiv.org/abs/1804.02654v11
  * [9] A. Kawauchi, Triviality of a surface-link with meridian-based free fundamental group (preprint version). arXiv:1804.04269
  * [10] A. Kawauchi, Smooth homotopy 4-sphere (preprint version). arXiv:1911.11904
  * [11] H. Kneser, Geschlossene Flächen in dreidimensionalen Mannigfaltigkeiten, Jahresbericht der Deut. Math. Verein. 38 (1929), 248-260.
  * [12] J. Milnor, Towards the Poincaré conjecture and the classification of 3-manifolds, Notices AMS 50 (2003), 1226-1233.
  * [13] G. Perelman, The entropy formula for the Ricci flow and its geometric applications. arXiv:math.DG/0211159 v1, 11 Nov 2002.
  * [14] G. Perelman, Ricci flow with surgery on three-manifolds. arXiv:math. DG/0303109 v1, 10 Mar 2003.
  * [15] H. Poincaré, Second complément à l’Analysis Sitis, Proc. London Math. Soc. 32 (1900), 277-308.
  * [16] H. Poincaré, Cinquième complément à l’Analysis Sitis, Rend. Circ. Mat. Palermo 18 (1904), 45-110.
  * [17] W. P. Thurston, Three dimensional manifolds, Kleinian groups and hyperbolic geometry, Bull. Amer. Math. Soc. 6 (1982), 357-381.
  * [18] F. Waldhausen, On irreducible 3-manifolds which are sufficiently large, Ann. of Math. 87 (1968) 56-88.


# DwinFormer: Dual Window Transformers for End-to-End Monocular Depth
Estimation

Md Awsafur Rahman, , and Shaikh Anowarul Fattah M. A. Rahman and S. A. Fattah
are with the Department of Electrical and Electronic Engineering, Bangladesh
University of Engineering and Technology, Dhaka-1000, Bangladesh, e-mail:
<EMAIL_ADDRESS>and fattah@eee.buet.ac.bd).

###### Abstract

Depth estimation from a single image is of paramount importance in the realm
of computer vision, with a multitude of applications. Conventional methods
suffer from the trade-off between consistency and fine-grained details due to
the local-receptive field limiting their practicality. This lack of long-range
dependency inherently comes from the convolutional neural network part of the
architecture. In this paper, a dual window transformer-based network, namely
DwinFormer, is proposed, which utilizes both local and global features for
end-to-end monocular depth estimation. The DwinFormer consists of dual window
self-attention and cross-attention transformers, Dwin-SAT and Dwin-CAT,
respectively. The Dwin-SAT seamlessly extracts intricate, locally aware
features while concurrently capturing global context. It harnesses the power
of local and global window attention to adeptly capture both short-range and
long-range dependencies, obviating the need for complex and computationally
expensive operations, such as attention masking or window shifting. Moreover,
Dwin-SAT introduces inductive biases which provide desirable properties, such
as translational equvariance and less dependence on large-scale data.
Furthermore, conventional decoding methods often rely on skip connections
which may result in semantic discrepancies and a lack of global context when
fusing encoder and decoder features. In contrast, the Dwin-CAT employs both
local and global window cross-attention to seamlessly fuse encoder and decoder
features with both fine-grained local and contextually aware global
information, effectively amending semantic gap. Empirical evidence obtained
through extensive experimentation on the NYU-Depth-V2 and KITTI datasets
demonstrates the superiority of the proposed method, consistently
outperforming existing approaches across both indoor and outdoor environments.

###### Index Terms:

Attention Mechanism, Computer Vision, Depth Estimation, Transformer

††publicationid: pubid: This work has been submitted to the IEEE for possible
publication. Copyright may be transferred without notice, after which this
version may no longer be accessible.

##  I Introduction

Dpeth information has a wide variety of applications in different fields [1,
2, 3, 4], such as refocusing/bokeh, self-driving vehicles, robot motion,
augmented reality, and so on. Depth sensors e.g. LiDAR, and ToF are expensive
and can capture only low-resolution sparse depth information, which demands
the development of single-image (monocular) depth estimation for high-
resolution and dense depth information. However, estimating depth relying on a
single image is an ill-posed problem [5] as often there is not enough
information to accurately determine the depth of objects in a scene, thus
making it very challenging. In the past years, convolutional neural network
(CNN) based deep learning methods have dominated monocular depth estimation
[6]. But, CNN fails to produce global context-aware pixel-wise prediction due
to its intrinsic locality resulting in a trade-off between fine-detailed and
consistent depth map [7]. Several methods attempted to mitigate these
limitations [8] still the above-mentioned issues persist.

In recent years, transformers [9, 10, 11] have shown promising results for
monocular depth estimation. They are notable for their ability to capture
long-range dependencies in data. A few years back, Vision Transformer [12]
applied pure transformer to the image surpassing the CNN, which verified the
usability of transformers in computer vision. Transformers, which have been
applied to pixel-wise prediction, have addressed the issue of a local
receptive field. Nevertheless, they have also introduced new challenges,
including a lack of multi-scale features, the need for higher image resolution
than text, and the fact that the computational complexity of self-attention
scales quadratically with the size of the image. Later on, several attempts
have been made to address these issues with hierarchical transformer [13] but
they provide limited coverage for the global-receptive field (cross-window
connection) [14]. Recent studies such as [14] and [15] made progress towards
addressing these issues. Despite that, these studies present their own set of
challenges. The utilization of global sparse attention in [15] leads to a
deterioration in the quality of global features as the size of the image
increases. Meanwhile, the study presented in [14] suffers from ineffective
global attention, resulting in a loss of crucial global information and
suboptimal performance.

Figure 1: Graphical abstract of the proposed method.

Despite the success of the recent transformer based methods mentioned above in
aligning depth edges with object boundaries, they often struggle to accurately
assign depth labels to pixels due to problems in effectively fusing encoder
and decoder features. Typically a skip connection is used to fuse encoder and
decoder features, which applies convolution to features after concatenation.
Due to convolution’s intrinsic locality, the flow of semantic information is
restricted from long ranges affecting the ability of the model to predict the
correct depth label for a pixel. To mitigate this issue, skip-attention module
is introduced by [16] that integrates encoder-decoder features contextually
using local window-based cross-attention. Despite the benefits of transformer-
based fusion of encoder-decoder features, the skip-attention module is still
restricted by the limited receptive field of local-window attention, thereby
it is only able to effectively incorporate information from a limited range of
input pixels leaving the global information unused, potentially limiting its
overall performance. In this paper, a transformer based architecture is
proposed for end-to-end depth estimation while addressing above mentioned
issues of existing approaches. The main contributions of the proposed method
can be summarized as follows:

  1. 1.

A dual window transformer-based network, namely DwinFormer, is proposed for
end-to-end monocular depth estimation. Here dual window self-attention (Dwin-
SAT) and cross-attention (Dwin-SAT) transformers are introduced to effectively
capture long-range dependencies and local fine-grained details.

  2. 2.

Proposed Dwin-SAT introduces an effective design of a transformer-based
backbone, which utilizes both local and global window attentions to mitigate
the trade-off between fine details and consistency in depth maps by capturing
both local and global contextual information.

  3. 3.

To bridge the semantic gap between encoded and decoded features, the proposed
Dwin-CAT decoder seamlessly fuses encoder and decoder features with both fine-
grained local and contextually aware global information.

##  II Related Work

Eigen et al. [6] first proposed a coarse to-fine method to estimate depth.
Zhou et al. [17] first introduced a method for predicting the camera’s ego-
motion and depth map from monocular video. Later on, Laina et al. [18]
proposed a fully CNN-based residual network, incorporating up-projection block
for improved performance. Ranjan et al [19] trained depth estimation in
conjunction with optical flow estimation and motion segmentation to achieve
synergistic results. Yourun et al. [20] developed a novel multi-scale method
that downsamples the predicted depth map and performs image synthesis at
multiple resolutions for improved model performance.

In recent years, transformers have been a popular choice for encoders for
their ability to capture long-range dependencies. TransDepth [10], and DPT [9]
are the first to utilize vision-transformer (ViT) encoder and a convolutional
decoder for depth estimation. Later on, SwinDepth [21] utilizes a hierarchical
vision-transformer named Swin Transformer [13] as encoder and multi-scale
convolutional block as decoder. On the other hand, AdaBin [22] uses an
additional transformer-based block to adaptively estimate depth values for an
image by dividing the range of possible depths into bins and calculating the
final depth estimates as linear combinations of the bin centers. Very few
explorations have been made in this area. But, recently in PixelFormer [16], a
window attention-based transformer is utilized to effectively fuse the
features produced by encoder and decoder. This approach avoids the limitations
of simply concatenating these features, which can incite semantic gap [23].
Despite using a transformer-based fusion of encoder-decoder features, this
method fails to reach its true potential as its skip-attention module is
highly impeded by incorporation of information from a limited range of input
pixels due to local-window attention.

In essence, CNNs struggle with a trade-off between fine details and
consistency in depth maps, while transformers have shown promise in capturing
global information, yet they are hindered by high computational complexity,
lack of multi-scale features, and inductive bias. Recent attempts to overcome
these limitations through different types of window attention face new sets of
challenges, such as limited global coverage, impaired global context, and loss
of global context. Meanwhile, the fusion of encoder and decoder features
remains a challenge due to semantic-gap, and while skip-attention module [16]
has improved the fusion process, it is still limited by its local receptive
fields.

##  III Methodology

Figure 2: Graphical overview of the DwinFormer Architecture. The input image
is encoded into multiscale feature maps using a series of Dwin-SA blocks in
the Dwin-SAT. The encoded features are then reconstructed into a depth map by
the Dwin-CAT, which integrates features from Dwin-SA of the same level and
Dwin-CA of the previous level. The final depth map is produced by the Depth
Head, processing the decoded features.

###  III-A Problem Definition

The proposed method models the depth map of an RGB image as the probability of
maximum depth for all pixels. Given an image $I\in\mathbb{R}^{H\times W\times
3}$, a model $\mathcal{F}(\Theta)$ is developed to predict a probability mask
$\hat{p}\in[0,1]^{H\times W\times 1}$ indicating the likelihood of the maximum
depth for all pixels in the image. The mask is transformed into the depth map
$\hat{y}=\hat{p}\odot\textit{max\\_depth}$. The model’s parameters $\Theta$
are optimized by minimizing a chosen loss function
$\mathcal{L}(\Theta,y,\hat{y})$ with respect to the true depth map $y$ through
$\Theta^{*}=\arg\min_{\Theta}\mathcal{L}(\Theta,y,\hat{y})$.

###  III-B Overview of Proposed DwinFormer Architecture

The proposed architecture, as shown in Fig. 2, introduces the Dual Window Self
Attention Transformer (Dwin-SAT) backbone to process the input image $I$. The
Dwin-SAT backbone employs multiple layers of Dual Window Self Attention (Dwin-
SA) to extract feature maps representing the image at different resolution
scales. These resolution scales are represented by $i$ and are defined as
$\frac{1}{2^{i+1}}$ relative to the input image $I$, where
$i\in{\\{1,2,3,4\\}}$ and $i=5$ has same resolution as $i=4$. Dwin-SA
leverages both Local Window Self Attention (Lwin-SA) and Global Window Self
Attention (Gwin-SA), to extract both locally and globally contextualized
features from the input image. These features are subsequently refined through
the proposed Dual Window Cross Attention (Dwin-CA), which is employed at
different stages to merge the encoder-decoder feature maps hierarchically.
Dwin-CA exploits Local Window Cross Attention (Lwin-CA) and Global Window
Cross Attention (Gwin-CA) respectively to fuse encoder-decoder features with
both local and global contexts. Finally, the resulting decoded features from
the stack of Dwin-CA layers are processed by the Depth Head module to estimate
the per-pixel probability of maximum depth. In what follows, the blocks of
DwinFormer are explained.

###  III-C Proposed Encoder Architecture: Dwin-SAT

The proposed Dwin-SAT backbone, as depicted in Fig. 2, effectively extracts
features at multiple resolutions by iteratively downgrading the spatial
dimensions and upgrading the channel dimensions by factors of 2. The process
begins with the Stem block, which generates overlapping patches from the input
image $I$ to facilitate the application of the transformer. These patches are
then projected into a $C$-dimensional embedding space through the use of a
$3\times 3$ convolutional layer with a stride of $2\times 2$ and $1\times 1$
padding. Subsequently, the number of patches is reduced by a factor of 2 using
a strided convolution, thus allowing the Dwin-SAT backbone to continuously
extract features at different levels of detail. Finally, the resultant
features from Dwin-SAT are operated by a $1\times 1$ convolution to precisely
control the channel of the decoder. Specifically, Dwin-SAT utilizes the Dwin-
SA module to extract spatial characteristics by interleaving between Lwin-SA
and Gwin-SA modules with local and global contexts respectively. In contrast
to Gwin-SA, which operates global attention using both global and local
windows, Lwin-SA operates local attention using only local windows in the
manner described in Swin Transformer [13]. The global window in Gwin-SA is
generated by Global Window Generator (GWG) from the whole input feature maps
and shares across all local windows, thus resulting in a reduced number of
parameters and FLOPs. It confines information across the entire input feature
maps for interaction with local query features. Specifically, as shown in Fig.
3, the generator consists series of a Fused-MBConv [14] block followed by a
max pooling layer. Then, using a Downsample block at the end of each stage,
the spatial dimension of resulting features is decreased and the channel
dimension is increased by a factor of $2$. The Downsample block follows the
same architecture in [14] comprising a modified Fused-MBConv block, followed
by a $2\times 2$ strided convolution with a kernel size of $3\times 3$ and
layer normalization layer, which employs spatial feature contraction from CNN
that impose inductive bias and inter-channel interactions. At the end of the
Dwin-SAT, a $1\times 1$ convolution is used to control the number of output
channels of each decoder layer at each scale. For a given scale $i$, the
encoder can be expressed as

$\displaystyle{}_{1}Q^{e}_{l}$ $\displaystyle=LayerNorm(E_{i-1})$ (1)
$\displaystyle K^{e}_{l}$ $\displaystyle=MLP(E_{i-1});V^{e}_{l}=MLP(E_{i-1})$
$\displaystyle L_{i}$ $\displaystyle=\operatorname{Lwin-
SA}({}_{1}Q^{e}_{l},K^{e}_{l},V^{e}_{l})+E_{i-1}$ $\displaystyle L_{i}$
$\displaystyle=L_{i}+MLP(LayerNorm(L_{i}))$ $\displaystyle{}_{2}Q^{e}_{l}$
$\displaystyle=LayerNorm(L_{i});E^{\prime}_{i}=GWG(E_{i-1})$ $\displaystyle
K^{e}_{g}$ $\displaystyle=MLP(E^{\prime}_{i});V^{e}_{g}=MLP(E^{\prime}_{i})$
$\displaystyle E_{i}$ $\displaystyle=\operatorname{Gwin-
SA}({}_{2}Q^{e}_{l},K^{e}_{g},V^{e}_{g})+L_{i}$ $\displaystyle E_{i}$
$\displaystyle=E_{i}+MLP(LayerNorm(E_{i}))$ $\displaystyle E_{i}$
$\displaystyle=\operatorname{Downsample(E_{i})}$

Figure 3: The input feature map with $H,W$ dimensions, is fed into the Global
Window Generator (GWG), which uses the Fused-MBConv and MaxPool layer
repeatedly $K$ times to generate a global window with $h,w$ dimensions which
is reshaped then repeated $M$ times.

###  III-D Dual Window Self Attention (Dwin-SA)

The Dwin-SAT backbone’s primary computational operator is Dwin-SA. In general,
it operates through the process of contextualization, which involves computing
the pairwise relations between each query position and all key positions to
form an attention map. This map serves as a weighting function, which is used
to linearly combine the values to produce the output at each query position.
In short, the output at each query position is expressed as a weighted sum of
the values, where the weights are computed using the query and key through the
attention mechanism. The Dwin-SA block employs two sub-mechanisms for feature
extraction. Firstly, Lwin-SA mechanism partitions the image into smaller
windows (referred to as local windows) and performs self-attention between
patches within these windows to extract fine-grained local features. Secondly,
the Gwin-SA, a novel attention mechanism, exploits the global window to
interact with patches located anywhere in the image, allowing it to consider
information outside of the local window. Gwin-SA accesses global information
from the image through global keys and values generated from GWG, which
represents the entire image. This enables Gwin-SA to take into account the
global context when making attention decisions. Fig. 4 provides a visual
insight into these attentions and they also can be expressed as follows,

$\displaystyle\operatorname{Lwin-
SA}\left(Q^{e}_{l},K^{e}_{l},V^{e}_{l}\right)=S\left(Q^{e}_{l}{K^{e}_{l}}^{T}/\sqrt{d}+B\right)V^{e}_{l}$
(2) $\displaystyle\operatorname{Gwin-
SA}\left(Q^{e}_{l},K^{e}_{g},V^{e}_{g}\right)=S\left(Q^{e}_{l}{K^{e}_{g}}^{T}/\sqrt{d}+B\right)V^{e}_{g}$
(3)

where $Q^{e}_{l},K^{e}_{l},V^{e}_{l}$ denotes to query, key, and value in the
local window from encoder whereas $K^{e}_{g},V^{e}_{g}$ denotes to key and
value in the global window from encoder generated by GWG. For both local and
global windows in encoder, $Q,K,V\in\mathbb{R}^{M^{2}\times d}$; $d$ is the
query/key dimension, $M^{2}$ is the number of patches in a window, and $S$ is
the Softmax function. Following [13, 14], assuming relative position along
each axis lies in the range $[-M+1,M-1]$, learnable relative position bias $B$
is sampled from bias matrix $\hat{B}\in\mathbb{R}^{(2M-1)\times(2M-1)}$.

Figure 4: Comparison between Global Window Self Attention in proposed Dwin-SAT
and Global Attention in GCViT. While both methods use attention on encoded
window tokens, Gwin-SA leverages global key-values with local queries and
GCViT uses global queries with local key-values. Here, Global Window Generator
is used to create global query-key-values. Figure 5: The Local and Global
Window Cross Attention (Lwin-CA and Gwin-CA) blocks in the proposed Dwin-CAT
utilize cross-attention between the encoded and decoded features. Both blocks
employ decoded features as local queries and encode features as key-values.
However, Lwin-CA uses encoded features as local key-values, while Gwin-CA
employs them as global key-values via Global Window Generator.

The proposed Gwin-SA mechanism draws inspiration from GCViT [14] but utilizes
a distinctive approach to calculate attention as shown in Fig. 4. The global
attention in GCViT utilizes global query and a local key-value to
contextualize the global window tokens with respect to the local window
tokens. Even so, this process can lead to a loss of non-locality for the
global window tokens, as the global query only contributes to the weights of
the attention and the local value provides the image features. In this
approach, the output at each query position is computed as a weighted average
of the values, with the weights derived from the query and key through an
attention mechanism. In contrast, the Gwin-SA mechanism employs a reverse
approach by using a local query and a global key-value to contextualize the
local window tokens with respect to the global window tokens. This approach
enables the local window tokens to acquire non-locality in attention through
the contribution of the global value’s supply of image features. A further
advantage of the Gwin-SA mechanism lies in its ability to consider each
smaller zone by giving deeper effort, and maximize global information, the
Gwin-SA mechanism provides a more in-depth effort. The key differences between
the GCViT approach and the Gwin-SA mechanism are summarized as follows:

  * •

Global Attention in GCViT:

    * $\Rightarrow$

Global query + local key-value

    * $\Rightarrow$

Contextualize global window w.r.t local window

  * •

Gwin-SA in DwinFormer:

    * $\Rightarrow$

Local query + global key-value

    * $\Rightarrow$

Contextualize local window w.r.t global window

###  III-E Proposed Decoder Architecture: Dwin-CAT

The proposed Dual Window Cross Attention Transformer (Dwin-CAT) is depicted in
Fig. 2. Here, the number of output channels of each decoder layer at each
resolution scale $i$ is represented as $F_{i}=2^{i+4}$, where
$i\in{1,2,3,4,5}$. The decoder takes the lowest-resolution feature map from
the encoder and gradually enhances it to produce a final depth representation.
At each level, Dwin-CAT integrates details from the encoder’s feature map
through skip connections and upscaling. However, the encoded feature with its
rich information can sometimes mismatch with the decoded feature that holds
more semantic information, making the optimization process challenging. To
bridge this semantic gap, Dwin-CAT uses the Dwin-CA component that integrates
both local and global contexts, resulting in a more contextualized feature.
The encoded features are first passed through a $1\times 1$ convolution to
harmonize their channels with the previous decoded features and facilitate
attention. The Dwin-CA component in Dwin-CAT fuses the encoded and decoded
features using Local Window Cross Attention (Lwin-CA) and Global Window Cross
Attention (Gwin-CA). The resulting feature is passed through the Upsample
layer to simultaneously upscale the spatial dimension and downscale the
channel dimension. The decoder operates similarly to Dwin-SAT, but it uses
cross-attention between the encoded and decoded features, where queries come
from the encoder and keys-values from the decoder. In mathematical terms, the
decoder layer can be expressed as follows,

$\displaystyle{}_{1}Q^{d}_{l}$
$\displaystyle=LayerNorm(D_{i-1});E^{\prime}_{i}=Conv_{1\times 1}(E_{i})$ (4)
$\displaystyle K^{e}_{l}$
$\displaystyle=MLP(E^{\prime}_{i});V^{e}_{l}=MLP(E^{\prime}_{i})$
$\displaystyle L_{i}$ $\displaystyle=\operatorname{Lwin-
CA}({}_{1}Q^{d}_{l},K^{e}_{l},V^{e}_{l})+D_{i-1}$ $\displaystyle L_{i}$
$\displaystyle=L_{i}+MLP(LayerNorm(L_{i}))$ $\displaystyle{}_{2}Q^{d}_{l}$
$\displaystyle=LayerNorm(L_{i});E^{\prime\prime}_{i}=GWG(E^{\prime}_{i})$
$\displaystyle K^{e}_{g}$
$\displaystyle=MLP(E^{\prime\prime}_{i});V^{e}_{g}=MLP(E^{\prime\prime}_{i})$
$\displaystyle D_{i}$ $\displaystyle=\operatorname{Gwin-
CA}({}_{2}Q^{d}_{l},K^{e}_{g},V^{e}_{g})+L_{i}$ $\displaystyle D_{i}$
$\displaystyle=D_{i}+MLP(LayerNorm(D_{i}))$ $\displaystyle D_{i}$
$\displaystyle=\operatorname{Upsample(D_{i})}$

###  III-F Dual Window Cross Attention (Dwin-CA)

The Dual Window Cross Attention (Dwin-CA) is an effective technique that
seamlessly integrates encoder and decoder features while addressing semantic
discrepancies between them. Similar to Dual Window Self-Attention (Dwin-SA),
it utilizes local and global window attention mechanisms. However, instead of
self-attention within encoded features, it utilizes cross-attention between
encoded and decoder features. This allows Dwin-CA to bridge semantic gaps
between the rich and nuanced encoded feature map and the decoded feature map,
which holds the prevalence of information about the final depth
representation. The technique does this by incorporating both local and global
context through Local Window Cross Attention (Lwin-CA) and Global Window Cross
Attention (Gwin-CA) respectively. Lwin-CA extracts fine-grained local features
by performing cross-attention between the encoder and decoder using local
windows while Gwin-CA captures long-range dependencies by performing cross-
attention between the encoder and decoder using global windows. Fig. 5 shows
an illustration of the proposed cross-attention mechanism and they can also be
mathematically represented as

$\displaystyle\operatorname{Lwin-
CA}\left(Q^{d}_{l},K^{e}_{l},V^{e}_{l}\right)=S\left(Q^{d}_{l}{K^{e}_{l}}^{T}/\sqrt{d}+B\right)V^{e}_{l}$
(5) $\displaystyle\operatorname{Gwin-
CA}\left(Q^{d}_{l},K^{e}_{g},V^{e}_{g}\right)=S\left(Q^{d}_{l}{K^{e}_{g}}^{T}/\sqrt{d}+B\right)V^{e}_{g}$
(6)

The notations used in these equations, such as $Q^{d}_{l}$, $K^{e}_{l}$,
$V^{e}_{l}$, $K^{e}_{g}$, $V^{e}_{g}$, $d$, $B$, and $S$ are consistent with
the previously defined notations in equation 2 and 3, with the exception of
$Q^{d}_{l}$, which represents the query in the local window from the decoder.

###  III-G Depth Head

The Depth Head of the proposed architecture utilizes a depth feature map
generated by the Dwin-CAT of size $(\frac{H}{4},\frac{W}{4},C)$. The depth
feature map is then passed through a $3\times 3$ convolutional layer which
transforms it into a probability map of maximum depth for each pixel. Then the
depth map is obtained by applying the sigmoid function on this probability
map, resulting in a continuous estimate of the depth at each pixel. Finally,
the resultant depth map is upsampled $4\times$ to match the resolution of the
ground truth.

###  III-H Loss Function

The Scale-Invariant loss (SI) [6] is utilized in a scaled form in this paper.
The mathematical representation of the loss is given by the equation:

$\displaystyle\mathcal{L}_{\text{pixel
}}=\alpha\sqrt{\frac{1}{T}\sum_{i}g_{i}^{2}-\frac{\lambda}{T^{2}}\left(\sum_{i}g_{i}\right)^{2}}$
(7)

Where $d_{i}$ is the ground truth depth, $\hat{d}_{i}$ is the estimated depth,
$T$ represents the number of pixels with valid ground truth values and $g_{i}$
is computed as $g_{i}=\log_{e}(\hat{d}_{i})-\log_{e}(d_{i})$. The parameters
$\lambda$ and $\alpha$ used in the experiments are set to 0.85 and 10
respectively.

TABLE I: Result on NYUv2 Data. The best result is indicated in bold, second best is underlined, and symbols $\uparrow$ or $\downarrow$ denote higher/lower values are preferable Method | Abs Rel $\downarrow$ | RMS $\downarrow$ |  $\log_{10}$ $\downarrow$ | $\delta_{1}\uparrow$
---|---|---|---|---
Eigen et al. [6] | 0.158 | 0.641 | - | 0.769
DORN [24] | 0.115 | 0.509 | 0.051 | 0.828
Yin et al. [25] | 0.108 | 0.416 | 0.048 | 0.872
BTS [26] | 0.110 | 0.392 | 0.047 | 0.885
TransDepth [10] | 0.106 | 0.365 | 0.045 | 0.900
DPT [9] | 0.110 | 0.367 | 0.045 | 0.904
Adabins [22] | 0.103 | 0.364 | 0.044 | 0.903
P3Depth [27] | 0.104 | 0.356 | 0.043 | 0.898
SwinDepth [21] | 0.100 | 0.354 | 0.042 | 0.909
Depthformer [11] | 0.100 | 0.345 | - | 0.911
NeWCRFs [28] | 0.095 | 0.334 | 0.041 | 0.922
PixelFormer [16] | 0.090 | 0.322 | 0.039 | 0.929
DwinFormer (Proposed) | $\mathbf{0.081}$ | $\mathbf{0.280}$ | $\mathbf{0.034}$ | $\mathbf{0.951}$
TABLE II: Result on KITTI Data. The best result is indicated in bold, second best is underlined, and symbols $\uparrow$ or $\downarrow$ denote higher/lower values are preferable Method | Abs Rel $\downarrow$ | RMS $\downarrow$ | $\log_{10}\downarrow$ | $\delta_{1}\uparrow$
---|---|---|---|---
Eigen et al. [6] | $0.203$ | $6.307$ | $0.282$ | $0.702$
DORN. [24] | $0.072$ | $2.727$ | $0.120$ | $0.932$
Yin et al. [25] | $0.072$ | $3.258$ | $0.117$ | $0.938$
BTS [26] | $0.059$ | $2.756$ | $0.096$ | $0.956$
TransDepth [10] | $0.064$ | $2.755$ | $0.098$ | $0.956$
Adabins [22] | $0.058$ | $2.360$ | $0.088$ | $0.964$
DPT [9] | $0.060$ | $2.573$ | $0.092$ | $0.959$
SwinDepth [21] | 0.064 | 2.643 | - | 0.957
Depthformer [11] | 0.058 | 2.285 | - | 0.967
NeWCRFs [28] | $0.052$ | $2.129$ | $0.079$ | $0.974$
PixelFormer [16] | $\underline{0.051}$ | $\underline{2.081}$ | $\underline{0.077}$ | $\underline{0.976}$
DwinFormer (Proposed) | $\mathbf{0.047}$ | $\mathbf{1.959}$ | $\mathbf{0.073}$ | $\mathbf{0.980}$

##  IV Experiments

Numerous experiments using the KITTI [29] and NYU Depth V2 [30] datasets are
conducted to confirm the effectiveness of the novel method proposed. Through a
thorough analysis of both quantitative and qualitative criteria, the proposed
method is put through a thorough evaluation and comparison against current
state-of-the-art techniques. Additionally, an ablation study is carried out to
clearly show the importance of each individual component.

###  IV-A Datasets

NYU Depth V2 is an indoor dataset of 120K RGB-depth pairs from 464 scenes,
with an official training/testing split of 50K/654 images, and a depth upper
bound of 10 meters. The proposed method is trained with a resolution of
$480\times 640$.

KITTI is an outdoor dataset of stereo images from 61 scenes, with a
training/testing split of 26K/697 images defined by [6], and a depth upper
bound of 80 meters. The proposed method is trained with a resolution of
$704\times 352$.

###  IV-B Implementation Details

The present study employs the TensorFlow framework to implement its proposed
method. Adam optimizer [31], with a batch size of 8 is utilized during
training. Both the KITTI and NYUV2 datasets are trained for a period of 30
epochs, with an initial learning rate of $3\times 10^{-6}$, that is gradually
increased linearly to $2.5\times 10^{-5}$ and subsequently decreased with a
Cosine schedule. The proposed method utilizes 8 x NVIDIA V100 GPUs for
training. To prevent overfitting, various data augmentation techniques such as
random rotation, horizontal flipping, brightness-contrast-hue adjustments,
grayscale, and noise are applied. The proposed architectural design allows for
the utilization of pre-trained weights from GCViT for initializing the encoder
backbone, thus avoiding the need for training the backbone on ImageNet from
scratch. The number of output channels in each level for both the encoder and
decoder is dependent on the size of the architecture, where for the XXTiny,
XTiny, Tiny, Small, Base, and Large sizes, $C$ is set to $64$, $64$, $64$,
$96$, $128$ and $192$ respectively. The remaining architectural parameters are
kept similar to those of GCViT [14].

###  IV-C Evaluation Metrics

To evaluate and compare the performance of the proposed method in comparison
to existing methods five metrics [6] are employed, namely Average Relative
error (Abs Rel): $\frac{1}{n}\sum_{i\in
n}\frac{\left|d_{i}-\hat{d}_{i}\right|}{\hat{d}_{i}}$; Root Mean Squared error
(RMS): $\sqrt{\frac{1}{n}\sum_{i\in n}\left\|d_{i}-\hat{d}_{i}\right\|^{2}}$;
Average $\left(\log_{10}\right)$ error: $\frac{1}{n}\sum_{i\in
n}\left|\log_{10}\left(d_{i}\right)-\log_{10}\left(\hat{d}_{i}\right)\right|;$
Threshold Accuracy $\left(\delta_{i}\right)$ : $\%$ of $d_{i}$ s.t.
$\quad\max\left(\frac{d_{i}}{\hat{d}_{i}},\frac{\hat{d}_{i}}{d_{i}}\right)=\delta<thr$
for $thr=$ $1.25$; Squared Relative difference (Sq Rel):
$\frac{1}{n}\sum_{i\in
n}\frac{\left\|d_{i}-\hat{d}_{i}\right\|^{2}}{\hat{d}_{i}}$. Here $n$ is the
total number of pixels for each depth map, $d_{i}$, and $\hat{d}_{i}$ denotes
ground truth, and predicted depth for $i$-th pixel respectively.

###  IV-D Performance Comparison with Existing Methods

####  IV-D1 Quantitative Analysis

Tables I, and II present a quantitative analysis, respectively on the outdoor
dataset KITTI and indoor dataset NYUv2, comparing the proposed method to
existing techniques. The results of the comparison reveal that the proposed
method exhibits outstanding performance, surpassing the other methods by a
considerable margin across a wide range of metrics. The only exception is the
metric $\delta_{3}$ where the results are highly saturated, but even in that
case, the proposed method’s results are competitive. These results
conclusively demonstrate the exceptional performance of the proposed method in
the field of monocular depth estimation, and its ability to produce highly
accurate and precise depth maps.

####  IV-D2 Qualitative Analysis

Figure 6: Qualitative comparison between previous SOTA and proposed method for
KITTI dataset. Figure 7: Qualitative comparison between previous SOTA and
proposed method for NYUv2 dataset.

Figures 6 and 7 present an insightful comparison between the proposed method
and existing techniques, respectively, on the outdoor dataset KITTI and indoor
dataset NYUv2. As it is plainly evident from the figures, the proposed method
demonstrates an aptitude for effectively identifying depth cues such as object
boundaries, and sharp edges while producing both consistent and detailed depth
maps even for objects with missing information in the RGB image. This
highlights the superiority of the proposed method.

###  IV-E Ablation Study

####  IV-E1 Effectiveness of Dwin-SAT

Table III presents a comprehensive examination of the performance of the
proposed Dwin-SAT in comparison to other established backbones. In this
ablation study, Dwin-CAT is utilized as decoders across all encoders for
comparison purposes. It is manifestly evident that the proposed Dwin-SAT
outperforms the other methods significantly. This is attributed to its unique
ability to effectively extract features with both local and global contexts,
thus conclusively validating the efficacy of the proposed architecture.

TABLE III: Ablation study of Dwin-SAT on KITTI Eigen Split. Results are
denoted by $\uparrow$ for better and $\downarrow$ for worse. The best results
are in bold

Method | Abs Rel $\downarrow$ | Sq Rel $\downarrow$ | $\delta_{1}\uparrow$
---|---|---|---
EfficientNet-B7 [32] | $0.056$ | $0.165$ | $0.968$
SwinT-Large [13] | $0.053$ | $0.153$ | $0.972$
ConvNeXt-Large [33] | $0.052$ | $0.154$ | $0.974$
MaxViT-Large [15] | $0.051$ | $0.140$ | $0.975$
GCViT-Large [14] | $0.050$ | $0.138$ | $0.977$
Dwin-SAT (Proposed) | $\mathbf{0.047}$ | $\mathbf{0.136}$ | $\mathbf{0.980}$

####  IV-E2 Effectiveness of Dwin-CAT

Table IV presents the performance of our proposed Dwin-CAT module in
comparison to existing techniques. In this ablation study, Dwin-SAT is
utilized as encoders across all decoders for comparison purposes. The results
clearly demonstrate that the proposed Dwin-CAT module stands out as a clear
winner, exhibiting a significant superiority over the other methods, thus
validating the efficacy of the proposed module.

TABLE IV: Ablation study of Dwin-CAT on KITTI Eigen Split. Results are denoted
by $\uparrow$ for better and $\downarrow$ for worse. The best results are in
bold.

Method | Abs Rel $\downarrow$ | Sq Rel $\downarrow$ | $\delta_{1}\uparrow$
---|---|---|---
Add-Conv | $0.060$ | $0.176$ | $0.961$
Concat-Conv | $0.058$ | $0.174$ | $0.963$
Nested-Skip [34] | $0.056$ | $0.168$ | $0.965$
SAM [16] | $0.052$ | $0.160$ | $0.969$
Dwin-CAT (Proposed) | $\mathbf{0.047}$ | $\mathbf{0.136}$ | $\mathbf{0.980}$

##  V Conclusion

In this paper, a transformer-based design namely, DwinFormer is introduced for
end-to-end monocular depth estimation. The proposed encoder and decoder
incorporate both self-attention and cross-attention mechanisms, utilizing both
local and global receptive fields. This approach effectively addresses
challenges, such as trade-off between consistency and fine details in the
depth map and bridging semantic disparities between the encoded and decoded
features. As a result, efficient feature extraction and improved
reconstruction of depth maps are acquired, leading to more accurate depth
maps. The experimental results firmly establish the dominance of the proposed
method over existing approaches, as seen by the remarkable performance
improvement on the NYUv2 and KITTI datasets.

## References

  * [1] B. Busam, M. Hog, S. McDonagh, and G. Slabaugh, “Sterefo: Efficient image refocusing with stereo vision,” in _Proc. of the IEEE/CVF Int. Conf. on Comput. Vision Workshops_ , 2019, pp. 0–0.
  * [2] W. Chuah, R. Tennakoon, R. Hoseinnezhad, D. Suter, and A. Bab-Hadiashar, “Semantic guided long range stereo depth estimation for safer autonomous vehicle applications,” _IEEE Trans. on Intell. Transp. Syst._ , vol. 23, no. 10, pp. 18 916–18 926, 2022.
  * [3] M. Mancini, G. Costante, P. Valigi, and T. A. Ciarfuglia, “Fast robust monocular depth estimation for obstacle detection with fully convolutional networks,” in _2016 IEEE/RSJ Int. Conf. on Intell. Robots and Syst. (IROS)_. IEEE, 2016, pp. 4296–4303.
  * [4] J. Ping, Y. Liu, and D. Weng, “Comparison in depth perception between virtual reality and augmented reality systems,” in _2019 IEEE Conf. on Virtual Reality and 3D User Interfaces (VR)_. IEEE, 2019, pp. 1124–1125.
  * [5] V. Hedau, D. Hoiem, and D. Forsyth, “Thinking inside the box: Using appearance models and context based on room geometry,” in _Eur. Conf. on Comput. Vision_. Springer, 2010, pp. 224–237.
  * [6] D. Eigen, C. Puhrsch, and R. Fergus, “Depth map prediction from a single image using a multi-scale deep network,” _Advances in Neural Inf. Process. Syst._ , vol. 27, 2014.
  * [7] S. M. H. Miangoleh, S. Dille, L. Mai, S. Paris, and Y. Aksoy, “Boosting monocular depth estimation models to high-resolution via content-adaptive multi-resolution merging,” in _Proc. of the IEEE/CVF Conf. on Comput. Vision and Pattern Recognit._ , 2021, pp. 9685–9694.
  * [8] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs,” _IEEE Trans. on Pattern Anal. and Mach. Intell._ , vol. 40, no. 4, pp. 834–848, 2017.
  * [9] R. Ranftl, A. Bochkovskiy, and V. Koltun, “Vision transformers for dense prediction,” in _Proc. of the IEEE/CVF Int. Conf. on Comput. Vision_ , 2021, pp. 12 179–12 188.
  * [10] G. Yang, H. Tang, M. Ding, N. Sebe, and E. Ricci, “Transformer-based attention networks for continuous pixel-wise prediction,” in _Proc. of the IEEE/CVF Int. Conf. on Comput. Vision_ , 2021, pp. 16 269–16 279.
  * [11] A. Agarwal and C. Arora, “Depthformer: Multiscale vision transformer for monocular depth estimation with global local inf. fusion,” in _2022 IEEE Int. Conf. on Image Process. (ICIP)_. IEEE, 2022, pp. 3873–3877.
  * [12] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” _arXiv preprint arXiv:2010.11929_ , 2020.
  * [13] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in _Proc. of the IEEE/CVF Int. Conf. on Comput. Vision_ , 2021, pp. 10 012–10 022.
  * [14] A. Hatamizadeh, H. Yin, J. Kautz, and P. Molchanov, “Global context vision transformers,” _arXiv preprint arXiv:2206.09959_ , 2022.
  * [15] Z. Tu, H. Talebi, H. Zhang, F. Yang, P. Milanfar, A. Bovik, and Y. Li, “Maxvit: Multi-axis vision transformer,” _arXiv preprint arXiv:2204.01697_ , 2022.
  * [16] A. Agarwal and C. Arora, “Attention attention everywhere: Monocular depth prediction with skip attention,” in _Proc. of the IEEE/CVF Winter Conf. on Appl. of Comput. Vision_ , 2023, pp. 5861–5870.
  * [17] T. Zhou, M. Brown, N. Snavely, and D. G. Lowe, “Unsupervised learning of depth and ego-motion from video,” in _Proc. of the IEEE Conf. on Comput. Vision and Pattern Recognit._ , 2017, pp. 1851–1858.
  * [18] I. Laina, C. Rupprecht, V. Belagiannis, F. Tombari, and N. Navab, “Deeper depth prediction with fully convolutional residual networks,” in _2016 4th Int. Conf. on 3D vision (3DV)_. IEEE, 2016, pp. 239–248.
  * [19] A. Ranjan, V. Jampani, L. Balles, K. Kim, D. Sun, J. Wulff, and M. J. Black, “Competitive collaboration: Joint unsupervised learning of depth, camera motion, optical flow and motion segmentation,” in _Proc. of the IEEE Conf. on Comput. Vision and Pattern Recognit._ , 2019, pp. 12 240–12 249.
  * [20] Y. Zhang, M. Gong, J. Li, M. Zhang, F. Jiang, and H. Zhao, “Self-supervised monocular depth estimation with multiscale perception,” _IEEE Trans. on Image Process._ , vol. 31, pp. 3251–3266, 2022.
  * [21] Z. Cheng, Y. Zhang, and C. Tang, “Swin-depth: Using transformers and multi-scale fusion for monocular-based depth estimation,” _IEEE Sensors Journal_ , vol. 21, no. 23, pp. 26 912–26 920, 2021.
  * [22] S. F. Bhat, I. Alhashim, and P. Wonka, “Adabins: Depth estimation using adaptive bins,” in _Proc. of the IEEE/CVF Conf. on Comput. Vision and Pattern Recognit._ , 2021, pp. 4009–4018.
  * [23] N. Zioulis, G. Albanis, P. Drakoulis, F. Alvarez, D. Zarpalas, and P. Daras, “Hybrid skip: A biologically inspired skip connection for the unet architecture,” _IEEE Access_ , 2022.
  * [24] H. Fu, M. Gong, C. Wang, K. Batmanghelich, and D. Tao, “Deep ordinal regression network for monocular depth estimation,” in _Proc. of the IEEE Conf. on Comput. Vision and Pattern Recognit._ , 2018, pp. 2002–2011.
  * [25] W. Yin, Y. Liu, C. Shen, and Y. Yan, “Enforcing geometric constraints of virtual normal for depth prediction,” in _Proc. of the IEEE/CVF Int. Conf. on Comput. Vision_ , 2019, pp. 5684–5693.
  * [26] J. H. Lee, M.-K. Han, D. W. Ko, and I. H. Suh, “From big to small: Multi-scale local planar guidance for monocular depth estimation,” _arXiv preprint arXiv:1907.10326_ , 2019.
  * [27] V. Patil, C. Sakaridis, A. Liniger, and L. Van Gool, “P3depth: Monocular depth estimation with a piecewise planarity prior,” in _Proc. of the IEEE/CVF Conf. on Comput. Vision and Pattern Recognit._ , 2022, pp. 1610–1621.
  * [28] W. Yuan, X. Gu, Z. Dai, S. Zhu, and P. Tan, “Neural window fully-connected crfs for monocular depth estimation,” in _Proc. of the IEEE/CVF Conf. on Comput. Vision and Pattern Recognit._ , 2022, pp. 3916–3925.
  * [29] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, “Vision meets robotics: The kitti dataset,” _The Int. Journal of Robotics Research_ , vol. 32, no. 11, pp. 1231–1237, 2013.
  * [30] N. Silberman, D. Hoiem, P. Kohli, and R. Fergus, “Indoor segmentation and support inference from rgbd images,” in _Eur. Conf. on Comput. Vision_. Springer, 2012, pp. 746–760.
  * [31] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [32] M. Tan and Q. Le, “Efficientnet: Rethinking model scaling for convolutional neural networks,” in _Int. Conf. on Mach. Learn._ PMLR, 2019, pp. 6105–6114.
  * [33] Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie, “A convnet for the 2020s,” in _Proc. of the IEEE/CVF Conf. on Comput. Vision and Pattern Recognit._ , 2022, pp. 11 976–11 986.
  * [34] Z. Zhou, M. M. R. Siddiquee, N. Tajbakhsh, and J. Liang, “Unet++: Redesigning skip connections to exploit multiscale features in image segmentation,” _IEEE Trans. on Med. Imag._ , vol. 39, no. 6, pp. 1856–1867, 2019.


11institutetext: The University of Sheffield, Sheffield S10 2TN, UK 11email:
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
22institutetext: Universität Koblenz, 56072 Koblenz, Germany 22email:
<EMAIL_ADDRESS>
# An Analysis of Classification Approaches for Hit Song Prediction using
Engineered Metadata Features with Lyrics and Audio Features

Mengyisong Zhao 11 Morgan Harvey 11 David Cameron 11 Frank Hopfgartner 22
Valerie J. Gillet 11

###### Abstract

Hit song prediction, one of the emerging fields in music information retrieval
(MIR), remains a considerable challenge. Being able to understand what makes a
given song a hit is clearly beneficial to the whole music industry. Previous
approaches to hit song prediction have focused on using audio features of a
record. This study aims to improve the prediction result of the top 10 hits
among Billboard Hot 100 songs using more alternative metadata, including song
audio features provided by Spotify, song lyrics, and novel metadata-based
features (title topic, popularity continuity and genre class). Five machine
learning approaches are applied, including: k-nearest neighbours, Naïve Bayes,
Random Forest, Logistic Regression and Multilayer Perceptron. Our results show
that Random Forest (RF) and Logistic Regression (LR) with all features
(including novel features, song audio features and lyrics features)
outperforms other models, achieving 89.1% and 87.2% accuracy, and 0.91 and
0.93 AUC, respectively. Our findings also demonstrate the utility of our novel
music metadata features, which contributed most to the models’ discriminative
performance.

###### Keywords:

Hit song prediction Music Information Retrieval Machine learning Text
processing.

##  1 Introduction

Music labels spend more than $4.5 billion every year discovering new talented
artists and producing popular songs [1]. Precipitated by the growing
importance of online digital music platforms and recent advancements in
machine learning and big data technologies, a new research area called hit
song science has attracted increasing attention [2]. A successful hit song
prediction approach could bring considerable benefits to many music lifecycle
stakeholders. Early hit song prediction studies illustrate the complexity of
this problem, delivering only weak classification results [3, 4, 5, 6]. In
recent years, more advanced approaches have been able to accurately predict
hits and non-hits using audio features [7, 8, 9, 10, 11, 12, 13]; however,
many other potentially useful sources of information about the songs are also
available. In this study, we employ 12 Spotify audio features (energy,
liveness, tempo, speechiness, acousticness, time_signature, key, duration_ms,
loudness, valence, mode and danceability), these features are drawn directly
from Spotify, together with novel features based on Billboard music metadata
(popularity continuity, genre class and title topic), as well as the topics
extracted from the songs’ lyrics to identify Top 10 hits among Top 100 hits.
To our knowledge, this work is the first attempt to improve hit song
prediction by extracting features from the topic of song titles and by using a
song’s prior popularity information. We examine the effectiveness of these
novel features together with song audio and lyrics features for hit song
prediction using a variety of machine learning approaches, including k-nearest
neighbours (kNN), Naïve Bayes (NB), Random Forest (RF), Logistic Regression
(LR) and Multilayer Perceptron (MLP). Our findings demonstrate the utility of
the new features and provide state-of-the-art prediction performance, as well
as providing promising avenues for future work in this area.

##  2 Related Work

Hit song prediction (HSP) has been investigated frequently in recent decades.
Much seminal work failed to accurately predict hit songs, with some work even
suggesting that popularity was not predictable [4, 5]. An early approach by
Dhanaraj and Logan [3] achieved promising results by using a SVM model to
classify top 1 songs through acoustic and lyrics data. However, they provided
only scant details about their data gathering, feature engineering, model
training and parameter optimization procedure and found textual features to be
more predictive than audio analysis. Salganik et al.[4], and Pachet and Roy
[5] attempted to reproduce Dhanaraj and Logan’s work but failed to achieve a
similar level of accuracy.

Various algorithms have been applied to tackle this task, among them: Logistic
Regression (LR), Support Vector Machine (SVM) and Neural Networks (NN) are
commonly used [3, 6, 8, 11]. Ni et al. [7] gained promising results in
predicting UK Top 5 hits on the Top 40 single song charts, but again little
implementation detail was provided. Fan and Casey [11] used LR and SVM models
to predict British and Chinese hit songs but found that audio features worked
better for predicting Chinese hits than British ones, and that textual
features worked best overall.

Herremans et al. [6] focussed particularly on dance songs and classified hits
using five machine learning models. Their research affirmed the importance of
audio features; however, they achieved relatively poor accuracy results,
perhaps due to their use of a large number of features without performing any
feature selection. Georgieva et al. [8] compared six machine-learning
algorithms when conducting Billboard hit song prediction; the most successful
algorithms were LR and a NN with a single hidden layer. Their work also
demonstrated the utility of Spotify’s audio features for this task. Nasreldin
[12] did similar research but identified XGBoost as the top performing
classifier; in their study the SVM model performed the worst. As they only use
the raw data without any feature selection, they only achieved accuracy
results similar to those of Herremans et al. [6]

Recently, Zangerle et al. [15] adopted deep neural networks and treat HSP as a
regression task, and their experimental results show that the wide and deep
neural network-based approach performed best, achieving 72.04% accuracy.
However, the common problem with deep neural networks is that their results
are hard to interpret. Essa et al. [16] tried to solve the HIS task by using
both classification and regression models. They considered audio features
alone and, through adopting seven machine learning models, they achieved
results suggesting that both machine learning approaches (classification and
regression) can be used for HSP.

Although previous studies have made a large contribution to this topic, it is
still unclear which features can be used to successfully classify hit songs
when including audio features, music metadata and song lyrics, and in what
combination. Audio features have shown promise, but only raw terms have been
used to construct features to date [5, 6, 7]. Textual features have rarely
been adopted in hit song prediction tasks and, although Singhi and Brown [17]
did attempt to extract 31 song lyrics features and build SVM model to predict
hit songs, the performance achieved was not inspiring.

##  3 Data and Methodology

###  3.1 Data collection and preprocessing

To investigate hit song prediction, we obtained Billboard hot 100 songs data
from the open-source platform data.world named “Billboard Hot-100 Songs
2000-2018 w/Spotify Data+Lyric”111data.world/typhon/billboard-
hot-100-songs-2000-2018-w-spotify-data-lyrics . The dataset includes all songs
in the Billboard hot 100 weekly charts from 2007 to 2017, as well as audio
features, metadata and lyrics of each song provided by Spotify. The raw
dataset includes 33 attributes in total. We firstly remove the irrelevant
features (e.g., spotify_link, video_link, analysis_url). Then, we define
“hits” in this context to be songs whose highest position in the Billboard Hot
100 list was at rank 10 or above to produce a binary classification of “hit”
(1) that at some point reached the Top 10 or “not-hit” (0) that never reached
the Top 10. The features used in this study include those engineered based on
metadata (e.g., weeks, song title, music genre), 12 Spotify audio features, as
well as lyrics of each song. 273 songs had missing audio features data and/or
lyrics, and were subsequently removed as it would not be possible to
extrapolate or estimate such features. This left 3581 unique songs in the
final data set: 507 hits and 3074 non-hits.

###  3.2 Feature Engineering

We engineered several additional features to augment the existing metadata
features from the original Billboard data and the Spotify audio features.
Popularity continuity was created to represent the sum of each song’s popular
duration (i.e., how many weeks it had already been listed in the hot 100 chart
prior to the week of interest). Songs already present in the chart for more
than 50 weeks were assigned a 3; those present for between 20 and 50 were
assigned to 2; those between 10 and 20 were assigned 1; otherwise, a song was
assigned 0. Unlike classical music, popular music has relatively rapid
iterations [19]. The majority of songs only remain in the chart for a short
period, typically less than 20 weeks. Therefore, we assign a number based on 3
duration splits where the assigned numbers are only based on weekly duration
data. The song title topic feature was created based on the song title. We
removed symbols, punctuation, short terms (i.e., fewer than 4 chars) and
stopwords from the data, then, inspired by [3], used a bagofwords
representation with Latent Dirichlet Allocation (LDA) to extract topics from
the song titles. In total, ten topics were extracted, and each song was
assigned to the topic number with the highest probability for that song in
$\theta$. The numerical variable named genre class was created to replace the
existing string variable broad_genre in which each genre was assigned a
numerical value: 1 to 6 representing country, electronic dance music (edm),
pop, r&b, rock and rap music respectively. We treat song lyrics similar to how
we treat song titles, the only difference being the number of topics: 20
topics were extracted from the lyrics. This is because lyrics are far longer
than titles, thus providing sufficient data to extract a larger number of more
meaningful topics. Each song was assigned to the topic number with the highest
probability for that song in $\theta$.

###  3.3 Training Environment

Min-max normalization method was applied to accelerate the algorithm
convergence speed [16]. After preprocessing and feature engineering, a total
of 16 features were used for model building. We treat each song as an
individual, temporal factors were not considered in our experiment, the data
were split into training and testing sets using a ratio of 80:20 and, due to
the relatively small size of the overall data set, 5-fold cross validation was
applied instead of an individual validation set. Due to the highly imbalanced
classes (i.e., most songs are not top-10 hits), Synthetic Minority Over-
Sampling (SMOTE) was adopted inspired by Chawla et al. [18], which could
effectively increase the accuracy of minority class (hit song) prediction. In
this paper, 5 nearest neighbours have currently used to over-sampling the
minority class (hit songs). resulting in a final training set of 4918 songs
(including hits 2459 and 2459 non-hits). Forward feature selection was carried
out. All models were trained and tested using KNIME
4.4.0222https://www.knime.com/.

###  3.4 Model Setup and Optimisation

This study examined five commonly-used machine learning approaches from the
prior literature, and all the model parameter tuning has been using 5-fold
cross validation. The model hyperparameter and their optimum values are shown
in Table 1.

k-Nearest Neighbour (kNN). We tested values of k between 1 to 20 to seek for
more appropriate neighborhood distance when predicting hit songs, and when we
tuned the hyperparameter to k = 1 has achieved most effective accuracy.

Naïve Bayes (NB). We tested the default probability from 0.001 to 1 every
0.01, the best setting was default probability = 0.031.

Random Forest (RF). When using RF to train our model, the different split
criterion algorithm provides varied performance, which includes information
gain, information gain ratio, and Gini index. We tested the number of models
of all algorithms from 50 to 1000 every 50, and the Gini index achieved best
performance at 600 numbers of models.

Logistic Regression (LR). We tested four ways to solve the equation,
iteratively reweighted least squares with Gauss, iteratively reweighted least
squares with Laplace, stochastic average gradient with Gauss, stochastic
average gradient with Laplace. We find out using iteratively reweighted least
squares with Laplace regularization to solve the equation is more effective.
The Laplace equals to 3 has been accepted as best performance.

Neural Network (NN). Multilayer perceptron (MLP) model consisting of an input
layer, a hidden layer, and an output layer has been conducted in this study.
We tested the Maximum number of iterations from 500 to 5000 with 500 stop
sizes, Number of hidden layers and Number of hidden neurons per layer from 1
to 25 was measured every 3. The best parameter tuning result is 4500, 4, 22,
respectively.

Table 1: All model hyperparameter tuning optimization value. Classifier | Hyperparameter | Value
---|---|---
kNN | K value | 1
NB | Default probability | 0.031
RF | Gini index: number of models | 600
LR | Laplace | 3
NN | Maximum number of iterations | 4500
| Number of hidden layers | 4
| Number of hidden neurons per layer | 22

##  4 Findings, Results and Limitations

Our results include an analysis of accuracy, as well as AUC and the number of
features used as a measure of parsimony (see Table 2 and Table 3). We compare
models trained using all features, including our novel engineered ones, audio
features and lyrics features together, against three “baseline” models, audio
features alone, audio features and original metadata features, as well as
novel features and audio features model. It is notable that Random Forest
(Accuracy=89.1%, AUC=0.91) and Logistic Regression (Accuracy=87.2%, AUC=0.93)
with all features performed best according to both metrics. Logistic
Regression with Laplace regularisation achieves the best AUC score while only
using 4 features. According to Han et al. [20], the reason L1 regularisation
is more appropriate to this task could be it capable of reduce the
coefficients of some features to zero and generate a spare solution. Random
Forest achieved the best accuracy result, but required seven features to train
the model, which leads to longer training times, and poorer explainability.
MLP shows average performance in this task; this model requires a maximum
number of features according to Table 2, and longest training time to achieve
the best result, perhaps because the volume of the data available is
insufficient to train the network well. Naïve Bayes performs worst on
accuracy, but better on AUC score, which means this model has great ability on
identifying hits but weak on identifying non-hits.

Table 2: Features selected for each model.

Classifier | Accepted Feature Combination
---|---
kNN |  popularity continuity1, song title topic, genre class, energy, liveness, key, lyrics topic2
NB |  popularity continuity, genre class, key, loudness
RF |  popularity continuity, genre class, song title topic, key, valence, energy, lyrics topic
LR |  popularity continuity, genre class, lyrics topic, danceability
NN |  popularity continuity, genre class, key, song title topic, lyrics topic, acousticness, liveness, tempo, danceability

  * 1

Novel features are marked in italics.

  * 2

Lyrics feature is marked in bold.

Table 3: All model training and test results summarisation and comparison.

| 5-fold CV Accuracy | 5-fold CV AUC | Model Test Accuracy | Model Test AUC
---|---|---|---|---
KNN (Audio) | 83.98% | 0.847 | 79.92% | 0.530
KNN (Metadata+audio) | 90.85% | 0.917 | 82.08% | 0.748
KNN (NFE1 +audio) | 93.79% | 0.930 | 86.05% | 0.7752
KNN (NFE+audio+lyrics) | 94.31% | 0.937 | 86.38% | 0.745
NB (Audio) | 62.71% | 0.697 | 42.82% | 0.609
NB (Metadata+audio) | 82.78% | 0.915 | 71.13% | 0.899
NB (NFE+audio) | 86.26% | 0.924 | 74.76% | 0.922
NB (NFE+audio+lyrics) | 86.23% | 0.931 | 78.52% | 0.900
RF (Audio) | 79.22% | 0.876 | 71.13% | 0.629
RF (Metadata+audio) | 91.62% | 0.977 | 74.76% | 0.869
RF (NFE+audio) | 93.84% | 0.980 | 87.59% | 0.908
RF (NFE+audio+lyrics) | 95.1% | 0.989 | 89.12% | 0.912
LR (Audio) | 61.26% | 0.649 | 57.88% | 0.603
LR (Metadata+audio) | 84.83% | 0.917 | 83.54% | 0.927
LR (NFE+audio) | 86.15% | 0.928 | 86.47% | 0.923
LR (NFE+audio+lyrics) | 87.07% | 0.933 | 87.17% | 0.927
MLP (Audio) | 68.20% | 0.756 | 63.60% | 0.563
MLP (Metadata+audio) | 87.48% | 0.923 | 76.85% | 0.847
MLP (NFE+audio) | 88.0% | 0.929 | 79.36% | 0.734
MLP (NFE+audio+lyrics) | 90.04% | 0.931 | 84.66% | 0.808

  * 1

NFE stands for abbreviation of novel feature engineering.

  * 2

The best performance has been marked in bold.

Compared to the baseline method, all the model test accuracy results with our
novel metadata features provided significant performance improvement seen in
Table 3, which proved our novel metadata features have contributed impact to
HSP task. When adding song lyrics topic features, the accuracy score of all
models are slightly increased, the AUC score of kNN and NB are decreased for
.030 and .022 respectively, probably because the lyrics topic increase the
complexity of features, which might be hard for both algorithm to classify the
patterns of hits and non-hits. The novel variables almost frequently in the
list of automatically selected features as shown in Table 2, demonstrating
their discriminative power. The utility of popularity continuity indicates
that the longer a song in a particular genre can maintain a position in the
charts, the more likely it is to become a hit song. Certain topically-coherent
sets of terms, such as love, girls, life, and hearts are more likely to appear
in the hits than non-hits, as captured in the song title topic and lyrics
topic feature. Based on the ablation studies, some of the Spotify audio
features such as key, liveness, energy, and danceability are also important
when classifying hit songs but less consistently so than our novel features
and song lyrics feature. The contributed features are varied between each
model. Compared to song title topic, song lyrics feature shows more
contribution when using these two features together to identify hit songs.

The result of this study supports the findings of [6, 8, 13, 14] that music
metadata, audio features and lyrics can be used to classify hit songs through
machine learning approaches. Adding all features together has achieved the
best performance of all models. Moreover, we have been able to outperform the
baseline results of [8, 9, 10], as their work achieved an accuracy score
around 60% to 87% compared to our work, which gave accuracy scores around 79%
to 89%.

As future work, we intend to further enrich our models by developing more
features based on, for example, music reviews and social tags. More complex
and granular genre classifications such as different types of music from
various cultures, like Latin music or dance songs from India could be used to
extend our model. Furthermore, a larger dataset covering a longer period will
be examined. As the hit songs identified in our study can be defined as
extremely popular songs (top 10 among100), the model generalisation ability
may need more tests, particularly adding songs never achieved in billboard top
100. The substance of a hit song may change over time, and we will consider
more complex models that include temporal aspects to model changes in genres
and topical popularity over time. Other audio-based features could also be
considered, such as Mel-frequency Cepstral Coefficients (MFCC) and compared
with the Spotify audio features.

## References

  * [1] IFPI Global Music Report Homepage, http://www.ifpicr.cz/ifpi-global-music-report-2016/. last accessed 21 Nov 2022
  * [2] Greenberg, D.M., and Rentfrow, P.J.: Music and big data: a new frontier. Current opinion in behavioral sciences 18, 50-56 (2017)
  * [3] Dhanaraj, R, and Logan, B.: Automatic Prediction of Hit Songs. In ISMIR, 488-491 (2005)
  * [4] Salganik, M.J., Dodds, P.S. and Watts, J.D.: Experimental study of inequality and unpredictability in an artificial cultural market. science 311, 5762, 854-856 (2006)
  * [5] Pachet, F., and Roy, P.: Hit Song Science Is Not Yet a Science. In ISMIR, 355-360 (2008)
  * [6] Herremans, D., Martens, D. and Sörensen, K.: Dance hit song prediction. Journal of New Music Research 43(3), 291-302 (2014)
  * [7] Ni, Y., Santos-Rodriguez, R., Mcvicar, R. and De Bie, T.: Hit song science once again a science. In 4th International Workshop on Machine Learning and Music (2011)
  * [8] Georgieva, E., Marcella S. and Nicholas B. Hitpredict.: Predicting hit songs using Spotify data. STANFORD COMPUTER SCIENCE 229: MACHINE LEARNING (2018)
  * [9] Middlebrook, K. and Sheik, K.: Song hit prediction: Predicting billboard hits using spotify data. arXiv preprint arXiv:1908.08609 (2019)
  * [10] Kim, S.T. and Oh, J.H.: Music intelligence: Granular data and prediction of top ten hit songs. Decision Support Systems, 145 113535 (2021)
  * [11] Fan, J. and Casey, M.: Study of Chinese and UK hit songs prediction. In Proceedings of the International Symposium on Computer Music Multidisciplinary Research, 640-652 (2013)
  * [12] Song Popularity Predictor Homepage, https://towardsdatascience.com/song-popularity-predictor-1ef69735e380. last accessed 17th October 2021
  * [13] Kawawa-Beaudan, J. and Garza, G.: Predicting Billboard Top 100 Songs (2015)
  * [14] Borg, N. and Hokkanen, G.: What makes for a hit pop song? What makes for a pop song. Unpublished thesis, Stanford University, California, USA (2011)
  * [15] Zangerle, E., Vötter, M., Huber, R. and Yang, Y. H.: Hit Song Prediction: Leveraging Low-and High-Level Audio Features. In ISMIR 319-326 (2019)
  * [16] Essa, Y., Usman, A., Garg, T. and Singh, M. K.: Predicting the Song Popularity Using Machine Learning Algorithm (2022)
  * [17] Singhi, A. and Brown, D. G.: Can song lyrics predict hits. In Proceedings of the 11th International Symposium on Computer Music Multidisciplinary Research 457-471 (2015)
  * [18] Chawla, N. V., Bowyer, K. W., Hall, L. O. and Kegelmeyer, W. P.: SMOTE: synthetic minority over-sampling technique. Journal of artificial intelligence research 16, 321-357 (2002)
  * [19] Kinga, S.: The attributes and values of folk and popular songs. Journal of Bhutan Studies (2001)
  * [20] Han, J., Kamber, M. and Pei, J.: Data Mining: Concepts and Techniques (3rd ed.). Elsevier Inc (2012)