# Tunneling effect between radial electric wells in a homogeneous magnetic
field

Léo Morin  Department of Mathematics, University of Copenhagen,
Universitetsparken 5, DK-2100 Copenhagen Ø, Denmark<EMAIL_ADDRESS>
###### Abstract.

We establish a tunneling formula for a Schrödinger operator with symmetric
double-well potential and homogeneous magnetic field, in dimension two. Each
well is assumed to be radially symmetric and compactly supported. We obtain an
asymptotic formula for the difference between the two first eigenvalues of
this operator, that is exponentially small in the semiclassical limit.

##  1\. Introduction

This article is devoted to the spectral analysis of electromagnetic
Schrödinger operators with symmetries. Without magnetic fields, it is known
that symmetries of the potential induce tunneling. This translates into a
spectral gap between the two first eigenvalues that becomes exponentially
small in the semiclassical limit. This effect was studied in [11, 12] where
the spectral gap was estimated (see also the books [8, 4]). When adding a
homogeneous magnetic field to this model, first general results were obtained
in [13] (for weak magnetic fields) and [14] (upper bounds on the spectral
gap). The problem was recently reconsidered in [5], where some upper and lower
bounds on the spectral gap are obtained. These bounds were improved in [9],
giving a sharp exponential decay rate.

In this paper we consider a potential with two symmetric, radial, and
compactly supported wells, as in [5, 9]. We prove a sharp estimate on the
spectral gap in presence of a constant magnetic field. As explained in these
two references, the spectral gap is given by an integral term measuring the
interaction between the wells, the _hopping coefficient_. However, we suggest
here another approach to estimate this coefficient, which is closer to the
original spirit of [11]. This approach provides us with a shorter proof and
better estimates.

A similar strategy was recently implemented to prove a purely magnetic
tunneling formula, between radial magnetic wells [7]. The case of multiple
potential wells was also considered in [13] (without magnetic field) and
recently adapted to the magnetic case in [10]. In these articles it is
explained how to reduce the problem to many double-well interactions.
Therefore, the result we present below should also have applications to that
setting.

We consider the following Schrödinger operator acting in the plane
$\mathbf{R}^{2}$,

(1.1) $\mathcal{H}_{h}=(-ih\nabla-\mathbf{A})^{2}+V,$

where $V$ is a double-well potential, and $\mathbf{A}$ is a vector potential
generating a uniform magnetic field of strength $B>0$, i.e.
$\nabla\times\mathbf{A}=B$. Without loss of generality, we choose the gauge

(1.2) $A(x,y)=(0,Bx).$

The double-well potential is a sum of two disjoint single wells,

(1.3) $V(x,y)=v(x-L,y)+v(x+L,y),$

separated by a distance $2L>0$. On the single well we make the following
assumptions.

######  Assumption 1.

We assume that

  * •

$v$ is smooth, compactly supported, non-positive, and radial.

  * •

$v$ admits a unique minimum $v_{0}<0$ reached at $0$, and it is non-
degenerate.

Let $a>0$ denote the radius of the support of $v$, i.e. the smallest positive
number such that $\mathrm{supp}(v)\subset\overline{B}(0,a)$. We exploit the
radiality of the single well problem. We denote by $\varphi$ the ground state
of the single well Hamiltonian,

$\mathcal{H}_{h}^{sw}=(-ih\nabla-\mathbf{A}^{\rm{rad}})^{2}+v,$

in radial gauge $\mathbf{A}^{\rm{rad}}(x,y)=\frac{B}{2}(-y,x)$. Then $\varphi$
is radial, as explained in [9, Section 2.2], and the function
$u(|X|)=\varphi(X)$ satisfies the equation

(1.4)
$-h^{2}\frac{1}{r}\partial_{r}(r\partial_{r}u)+\Big{(}\frac{B^{2}r^{2}}{4}+v(r)\Big{)}u=\mu_{h}u.$

where $\mu_{h}$ is the eigenvalue of $\varphi$. Hence, the study of the
single-well problem is reduced to a radial Schrödinger equation with effective
potential $v_{B}(r)=\frac{B^{2}r^{2}}{4}+v(r)$. As recalled in [9], this is a
very standard setting. The decay of $\varphi$ can be optimally measured in
terms of the Agmon distance,

(1.5)
$d(r_{1},r_{2})=\int_{r_{1}}^{r_{2}}\sqrt{\frac{B^{2}r^{2}}{4}+v(r)-v_{0}}\mathrm{d}r.$

Moreover, we have explicit WKB approximations for $\varphi$ (see Lemma 4
below). However, the double well problem _is not_ directly reduced to a
Schrödinger equation with potential $v_{B}$. If that was the case, the
spectral gap would have an exponential rate of decay given by $S_{0}=2d(0,L)$,
as in the non-magnetic situation [11]. Here, consistently with [9], we prove
that the spectral gap is much smaller.

######  Theorem 1.

Let $v$ be a single well potential satisfying Assumption 1, and let
$\mathcal{H}_{h}$ be as in (1.1), with $V$ given by (1.3). Also assume that
$L>(1+\frac{\sqrt{3}}{2})a$. Then the spectral gap between the two smallest
eigenvalues of $\mathcal{H}_{h}$ satisfies

$\lambda_{2}(h)-\lambda_{1}(h)=2C(L,B,v)h^{\frac{1}{2}}e^{-\frac{S}{h}}(1+o(1))$

as $h\to 0$, for some constant $C(L,B,v)>0$, and with

(1.6)
$S=2d(0,L)+\int_{0}^{L}\sqrt{\frac{B^{2}(2L-r)^{2}}{4}-v_{0}}-\sqrt{\frac{B^{2}r^{2}}{4}-v_{0}}\mathrm{d}r.$

The constant $C(L,B,v)$ can be computed explicitly, even though we have no
simple interpretation, see (4.15).

Theorem 1 is the first result establishing an asymptotic formula for the
spectral gap of $\mathcal{H}_{h}$. First results in this direction appeared in
a general framework in [13], when the magnetic field is weak. Then first
bounds on $S$ were obtained in [14, 5]. Recently the sharp exponential decay
rate (1.6) was found in [9], where it is proven that

$h\ln(\lambda_{2}(h)-\lambda_{1}(h))\sim-S.$

Another related problem is the tunneling between purely magnetic wells, when
there is no potential and the confinement is generated by a non-homogeneous
magnetic field. This problem was studied in [7] in the case of radial wells,
using a similar strategy. Finally, other magnetic tunneling estimates have
recently been established in the case of Neumann boundary conditions [3],
vanishing magnetic fields [1], or discontinuous magnetic fields [6]. See also
[2] in a one-dimensional setting.

Remark 1. The additional integral term in $S$ is a purely magnetic effect,
with no analogue in the non-magnetic case, thus making this model especially
interesting. It is directly related to oscillations of the eigenfunctions.
Indeed, the eigenfunctions $\varphi_{\ell}$ and $\varphi_{r}$ generated by the
left and right well respectively are related by a magnetic translation (see
(3.1)),

(1.7) $\varphi_{\ell}(x,y)=e^{\frac{i}{h}BLy}\varphi_{r}(x+2L,y).$

Thus, they do not oscillate at the same frequency and the relative oscillation
is fast as $h\to 0$. This rapidly oscillating phase appears in the hopping
coefficient, making it smaller than what one could expect (see Section 4).

Remark 2. The condition $L>(1+\frac{\sqrt{3}}{2})a$ is a limit of our
strategy, which relies entirely on the radiality of the single well. Indeed,
the problem centered on the left well is only radial up to distance $2L-a$,
and not $2L$. This slight difference makes our decay estimates on the
eigenfunctions non-optimal (see Lemma 7). To overcome this problem, we would
need to understand very precisely the non-radial situation.

Remark 3. We recover the bounds on $S$ stated in [5, 9] as follows. The result
[9, Theorem 1.2] can be translated as

(1.8) $d(0,2L-a)+d(0,a)\leq S\leq d(0,2L),$

which follows from

(1.9)
$S-d(0,2L)=\int_{0}^{L}\sqrt{\frac{B^{2}r^{2}}{4}+v-v_{0}}-\sqrt{\frac{B^{2}r^{2}}{4}-v_{0}}\mathrm{d}r\leq
0,$

and

(1.10)
$S-\big{(}d(0,2L-a)+d(0,a)\big{)}=\int_{0}^{a}\sqrt{\frac{B^{2}(2L-r)^{2}}{4}-v_{0}}-\sqrt{\frac{B^{2}r^{2}}{4}-v_{0}}\mathrm{d}r\geq
0.$

Moreover, the main result from [5] is

(1.11) $BL^{2}-BLa\leq S\leq BL^{2}+2\sqrt{|v_{0}|}L+\gamma_{0},$

which follows from (1.8) with
$\gamma_{0}=\int_{0}^{a}\sqrt{v-v_{0}}\mathrm{d}r$.

Strategy. In Section 2 we describe the single well problem. Most of this
section is standard since the ground state solves a radial Schrödinger
equation without magnetic field. In Section 3 we prove that the spectral gap
is given by the hopping coefficient. Our proof is inspired by the non-magnetic
situation in [4, 11, 8], and is somewhat simpler that the one of [5, 9].
Finally, in Section 4 we estimate the hopping coefficient, thus proving
Theorem 1.

##  2\. The single well problem

Let $\varphi$ be the normalized ground state for the single well problem in
radial gauge,

(2.1) $(-ih\nabla-\mathbf{A}^{\rm{rad}})^{2}\varphi+v\varphi=\mu_{h}\varphi,$

with

(2.2) $\mathbf{A}^{\rm{rad}}(x,y)=\frac{B}{2}(-y,x).$

We first collect the following basic facts about $\varphi$.

######  Proposition 2.

The ground state energy $\mu_{h}$ of $(-ih\nabla-\mathbf{A}^{\rm{rad}})^{2}$
satisfies

$\mu_{h}=v_{0}+h\sqrt{B^{2}+2v^{\prime\prime}(0)}+\mathcal{O}(h^{2}).$

The associated eigenfunction $\varphi$ is radial and real-valued. Moreover,
the first excited eigenvalue $\mu_{h,1}$ satisfies

$\mu_{h,1}=v_{0}+3h\sqrt{B^{2}+2v^{\prime\prime}(0)}+\mathcal{O}(h^{2}).$

A detailed proof of Proposition 2 is given in [9, Section 2.3]. It follows
from a harmonic approximation of $v$ near its minimum. Indeed, when $v$ is
quadratic, the problem becomes more explicit and we can prove that the
minimizer is radial. The harmonic approximation gives radiality when $v$ has a
non-degenerate minimium. The function $\varphi$ being radial, equation (2.1)
can be rewritten in polar coordinates, using the notation $\varphi(X)=u(|X|)$,

(2.3)
$-h^{2}\frac{1}{r}\partial_{r}\big{(}r\partial_{r}u\big{)}+\frac{B^{2}r^{2}}{4}u+vu=\mu_{h}u.$

This is a radial Schrödinger equation, with effective potential
$v_{B}(r)=\frac{B^{2}r^{2}}{4}+v(r)$. From this observation you deduce
Proposition 2, by harmonic approximation of $v_{B}$ near its minimum (see [8,
Chapter 2] for instance). Moreover, the ground state $\varphi$ has the
following WKB approximation ([8, Theorem 2.3.1]).

######  Proposition 3.

The ground state $\varphi$ of the single well problem has the following WKB
approximation,

$\Big{|}e^{\frac{d(0,|X|)}{h}}\varphi(X)-h^{-\frac{1}{2}}K(|X|)\Big{|}=\mathcal{O}(h^{\frac{1}{2}}),$

uniformly on any compact, where $d$ is the Agmon distance introduced in (1.5),
and

(2.4)
$K(r)=\sqrt{\frac{B^{2}+2v^{\prime\prime}(0)}{4\pi}}\exp\Big{(}-\int_{0}^{r}\frac{v_{B}^{\prime}(s)}{4(v_{B}(s)-v_{0})}+\frac{1}{2s}-\frac{\sqrt{B^{2}+2v^{\prime\prime}(0)}}{2\sqrt{v_{B}(s)-v_{0}}}\mathrm{d}s\Big{)}.$

As observed in [5, Equation (2.9)], the function $\varphi$ also has an
explicit integral formula outside the support of $v$, since equation (2.3) is
related to some special functions.

######  Lemma 4.

The normalized ground state $\varphi$ of the single well problem, solution of
(2.1), is radial. Moreover, it has the following integral formula for $|X|>a$,

(2.5)
$\varphi(X)=C_{h}\exp\Big{(}-\frac{B|X|^{2}}{4h}\Big{)}\int_{0}^{\infty}\exp\Big{(}-\frac{B|X|^{2}t}{2h}\Big{)}t^{\alpha-1}(1+t)^{-\alpha}\mathrm{d}t,$

where $\alpha=\frac{1}{2}-\frac{\mu_{h}}{2Bh}$, and $C_{h}$ is a normalization
constant. Moreover,

(2.6) $C_{h}\sim
h^{-1}K(L)\sqrt{\frac{f^{\prime\prime}(t_{L})}{2\pi}}t_{L}^{1-\nu}(1+t_{L})^{\nu}e^{\frac{\tilde{d}(L)-d(0,L)}{h}}\quad\text{as}\quad
h\to 0,$

where $\tilde{d}(L)=\int_{0}^{L}\sqrt{\frac{B^{2}r^{2}}{4}-v_{0}}\mathrm{d}r$,
the Agmon distance $d(0,L)$ was defined in (1.5), $t_{L}$ and
$f^{\prime\prime}(t_{L})$ are given in (2.12), and
$\nu=\frac{1}{2}-\frac{\sqrt{B^{2}+2v^{\prime\prime}(0)}}{2B}$.

###### Proof.

Note that $\varphi$ satisfies equation (2.3) which can be solved by special
functions outside the support of $v$ (It is related to Kummer functions, see
[5]). To estimate $C_{h}$, we combine the WKB approximation from Proposition
3,

(2.7) $\varphi(L)=h^{-\frac{1}{2}}K(L)e^{-\frac{d(0,L)}{h}}(1+o(1)),$

with an estimate of (2.5) using the Laplace method. Indeed, using that

(2.8) $\alpha=\frac{|v_{0}|}{2Bh}+\nu+o(1),$

we find

(2.9)
$\varphi(L)=C_{h}\int_{0}^{\infty}\exp\Big{(}-\frac{f(t)}{h}\Big{)}t^{\nu-1}(1+t)^{-\nu}\mathrm{d}t\,\big{(}1+o(1)\big{)},$

with

(2.10)
$f(t)=\frac{BL^{2}}{4}(1+2t)+\frac{|v_{0}|}{2B}\ln\Big{(}\frac{1+t}{t}\Big{)}.$

The first and second derivatives of $f$ are

(2.11)
$f^{\prime}(t)=\frac{BL^{2}}{2}-\frac{|v_{0}|}{2B}\frac{1}{t(1+t)},\qquad
f^{\prime\prime}(t)=\frac{|v_{0}|}{2B}\frac{2t+1}{t^{2}(1+t)^{2}}.$

In particular, $f$ has a unique critical point $t_{L}>0$, which is also a
global minimum,

(2.12)
$t_{L}=\frac{1}{2}\Big{(}-1+\sqrt{1+\frac{4|v_{0}|}{B^{2}L^{2}}}\,\Big{)},\quad\text{with}\quad
f^{\prime\prime}(t_{L})=\frac{B^{2}L^{3}}{2|v_{0}|}\sqrt{B^{2}L^{2}+4|v_{0}|}.$

By the Laplace method, we deduce

(2.13) $\varphi(L)=C_{h}\sqrt{\frac{2\pi
h}{f^{\prime\prime}(t_{L})}}e^{-\frac{f(t_{L})}{h}}t_{L}^{\nu-1}(1+t_{L})^{-\nu}\big{(}1+o(1)\big{)}.$

Also note that $f(t_{L})=\tilde{d}(L)$. We combine this with (2.7) to get the
estimate on $C_{h}$. ∎

##  3\. The double well problem

In order to study the double well problem, we follow the strategy of [11] and
compute the matrix of $\mathcal{H}_{h}$ in a basis
$(\varphi_{\ell},\varphi_{r})$, where $\varphi_{\ell}$ (resp. $\varphi_{r}$)
is the ground state generated by the left well (resp. the right well). More
precisely, we define $\varphi_{\ell}$ and $\varphi_{r}$ as the normalized
solutions to

(3.1)
$\displaystyle(-ih\nabla-\mathbf{A})^{2}\varphi_{\ell}+v(x+L,y)\varphi_{\ell}=\mu_{h}\varphi_{\ell},$
(3.2)
$\displaystyle(-ih\nabla-\mathbf{A})^{2}\varphi_{r}+v(x-L,y)\varphi_{r}=\mu_{h}\varphi_{r},$

respectively. Note that $\varphi_{\ell}$ and $\varphi_{r}$ are related to the
radial solution $\varphi$ of the single-well problem (2.1) as follows. Let us
denote by $\mathbf{A}^{\ell}$ and $\mathbf{A}^{r}$ the radial gauges centered
at $(-L,0)$ and $(L,0)$ respectively, namely

$\displaystyle\mathbf{A}^{\ell}(x,y)=\frac{B}{2}(-y,x+L),\qquad\mathbf{A}^{r}(x,y)=\frac{B}{2}(-y,x-L).$

We define the two functions $\sigma_{\ell}$ and $\sigma_{r}$ by

(3.3)
$\sigma_{\ell}(x,y)=\frac{B}{2}y(L-x),\qquad\sigma_{r}(x,y)=-\frac{B}{2}y(L+x).$

They satisfy

(3.4)
$\nabla\sigma_{\ell}=\mathbf{A}^{\ell}-\mathbf{A},\qquad\nabla\sigma_{r}=\mathbf{A}^{r}-\mathbf{A},\qquad\text{and}\qquad\sigma_{\ell}-\sigma_{r}=BLy.$

Then $\varphi_{\ell}$ and $\varphi_{r}$ are related to $\varphi$ by a magnetic
translation,

(3.5)
$\displaystyle\varphi_{\ell}(x,y)=e^{-\frac{i\sigma_{\ell}}{h}}\varphi(x+L,y),\qquad\varphi_{r}(x,y)=e^{-\frac{i\sigma_{r}}{h}}\varphi(x-L,y).$

The difference between the two smallest eigenvalues of $\mathcal{H}_{h}$ can
be estimated using $\varphi_{\ell}$ and $\varphi_{r}$ through the hopping
coefficient, as stated in the following theorem, which can also be found in
[5, Section 4].

######  Theorem 5.

The two smallest eigenvalues of $\mathcal{H}_{h}$ satisfy

$\displaystyle\lambda_{1}$
$\displaystyle=\mu_{h}-|w_{h}|+\mathcal{O}\big{(}h^{-3}e^{-\frac{2d(0,2L-a)}{h}}\big{)},$
$\displaystyle\lambda_{2}$
$\displaystyle=\mu_{h}+|w_{h}|+\mathcal{O}\big{(}h^{-3}e^{-\frac{2d(0,2L-a)}{h}}\big{)},$

where

(3.6)
$w_{h}=\int_{\mathbf{R}^{2}}v(x+L,y)\varphi_{\ell}(x,y)\overline{\varphi_{r}(x,y)}\mathrm{d}x\mathrm{d}y.$

The proof of Theorem 5 is a standard application of the Helffer-Sjöstrand
strategy (see [11, Theorem 3.9] or [7, 8]). For the reader’s convenience, we
recall the main ideas in Sections 3.1 and 3.2 below.

###  3.1. An approximation lemma

First of all, the Agmon estimates give exponential localization of the
eigenfunctions of $\mathcal{H}_{h}$ inside the wells. This localization is
enough to prove that the spectrum of the double-well operator is the
superposition of the spectra of the one-well operators, modulo
$\mathcal{O}(h^{\infty})$ (as in [11, 8]). The proof of this standard result
is omitted.

######  Lemma 6.

There exists $c,h_{0}>0$ such that, for $h\in(0,h_{0})$,

$|\lambda_{1}(h)-\mu_{h}|=\mathcal{O}(h^{\infty}),\quad|\lambda_{2}(h)-\mu_{h}|=\mathcal{O}(h^{\infty}),$

and $\lambda_{3}(h)-\lambda_{1}(h)\geq ch$.

Let $\Psi_{1}$, $\Psi_{2}$ be the two first eigenfunctions of
$\mathcal{H}_{h}$, and $\Pi$ the spectral projector on
$\mathrm{Ran}(\Psi_{1},\Psi_{2})$. The lemma below shows that the single well
states $\varphi_{\ell}$ and $\varphi_{r}$ are close to this eigenspace.

######  Lemma 7.

There exists a $C>0$ such that, for $h$ small enough and $j=\ell,r$ we have

$\|\varphi_{j}-\Pi\varphi_{j}\|\leq
Ch^{-\frac{3}{2}}e^{-\frac{d(0,2L-a)}{h}},\quad\|(-ih\nabla-\mathbf{A})(\varphi_{j}-\Pi\varphi_{j})\|\leq
Ch^{-\frac{3}{2}}e^{-\frac{d(0,2L-a)}{h}}.$

###### Proof.

We focus on $\varphi_{\ell}$, the estimates on $\varphi_{r}$ being identical.
Let $\lambda_{3}$ be the third eigenvalue of $\mathcal{H}_{h}$. By definition
of $\Pi$, we have the lower bound

(3.7)
$\lambda_{3}\|(I-\Pi)\varphi_{\ell}\|^{2}\leq\langle\mathcal{H}_{h}(I-\Pi)\varphi_{\ell},(I-\Pi)\varphi_{\ell}\rangle.$

On the other hand, since $\mathcal{H}_{h}$ and $\Pi$ commute, we can use the
eigenvalue equation (3.1) to get

(3.8)
$\langle\mathcal{H}_{h}(I-\Pi)\varphi_{\ell},(I-\Pi)\varphi_{\ell}\rangle=\mu_{h}\|(I-\Pi)\varphi_{\ell}\|^{2}+\langle(I-\Pi)v_{r}\varphi_{\ell},(I-\Pi)\varphi_{\ell}\rangle,$

with $v_{r}(x,y)=v(x-L,y)$. Combining (3.7) and (3.8) we find

(3.9)
$\|(I-\Pi)\varphi_{\ell}\|\leq(\lambda_{3}-\mu_{h})^{-1}\|v_{r}\varphi_{\ell}\|.$

For the gradient estimate we start from

(3.10)
$\|(-ih\nabla-\mathbf{A})(I-\Pi)\varphi_{\ell}\|^{2}\leq\langle\mathcal{H}_{h}(I-\Pi)\varphi_{\ell},(I-\Pi)\varphi_{\ell}\rangle+2|v_{0}|\|(I-\Pi)\varphi_{\ell}\|^{2}.$

We then use (3.8) and (3.9) to deduce

(3.11)
$\|(-ih\nabla-\mathbf{A})(I-\Pi)\varphi_{\ell}\|^{2}\leq\big{(}(\lambda_{3}-\mu_{h})^{-1}+(\mu_{h}+2|v_{0}|)(\lambda_{3}-\mu_{h})^{-2}\big{)}\|v_{r}\varphi_{\ell}\|^{2}.$

With Proposition 3 we finally bound $\|v_{r}\varphi_{\ell}\|$ in (3.9) and
(3.11),

(3.12) $\|v_{r}\varphi_{\ell}\|\leq
Ch^{-\frac{1}{2}}e^{-\frac{d(0,2L-a)}{h}}.$

The result follows since $\lambda_{3}-\mu_{h}\geq ch$, by Lemma 6. ∎

###  3.2. Proof of Theorem 5

From Lemma 7 we deduce, for $i,j\in\\{\ell,r\\}$, with
$\psi_{j}=\Pi\varphi_{j}$,

(3.13) $\displaystyle\langle\psi_{i}|\psi_{j}\rangle$
$\displaystyle=\langle\varphi_{i}|\varphi_{j}\rangle+\mathcal{O}\big{(}h^{-3}e^{-\frac{2d(0,2L-a)}{h}}\big{)},$
(3.14) $\displaystyle\langle\psi_{i}|\mathcal{H}_{h}\psi_{j}\rangle$
$\displaystyle=\langle\varphi_{i}|\mathcal{H}_{h}\varphi_{j}\rangle+\mathcal{O}\big{(}h^{-3}e^{-\frac{2d(0,2L-a)}{h}}\big{)},$

and also

(3.15)
$\displaystyle\langle\varphi_{\ell}|\mathcal{H}_{h}\varphi_{\ell}\rangle$
$\displaystyle=\mu_{h}+\mathcal{O}\big{(}h^{-3}e^{-\frac{2d(0,2L-a)}{h}}\big{)},$
(3.16) $\displaystyle\langle\varphi_{r}|\mathcal{H}_{h}\varphi_{r}\rangle$
$\displaystyle=\mu_{h}+\mathcal{O}\big{(}h^{-3}e^{-\frac{2d(0,2L-a)}{h}}\big{)},$
(3.17) $\displaystyle\langle\varphi_{\ell}|\mathcal{H}_{h}\varphi_{r}\rangle$
$\displaystyle=\mu_{h}\langle\varphi_{\ell}|\varphi_{r}\rangle+w_{h}.$

We now use the orthonormal basis

(3.18)
$\hat{\psi}_{\ell}=\frac{1}{\|\psi_{\ell}\|}\psi_{\ell},\qquad\hat{\psi}_{r}=\frac{\psi_{r}-\langle\psi_{r},\hat{\psi}_{\ell}\rangle\hat{\psi}_{\ell}}{\|\psi_{r}-\langle\psi_{r},\hat{\psi}_{\ell}\rangle\hat{\psi}_{\ell}\|}.$

It follows from (3.13), (3.14), (3.15), (3.16) and (3.17) that the matrix of
$\mathcal{H}_{h}$ in the orthonormal basis
$(\hat{\psi}_{\ell},\hat{\psi}_{r})$ is

(3.19)
$\begin{pmatrix}\langle\hat{\psi}_{\ell}|\mathcal{H}_{h}\hat{\psi}_{\ell}\rangle&\langle\hat{\psi}_{\ell}|\mathcal{H}_{h}\hat{\psi}_{r}\rangle\\\
\langle\hat{\psi}_{r}|\mathcal{H}_{h}\hat{\psi}_{\ell}\rangle&\langle\hat{\psi}_{r}|\mathcal{H}_{h}\hat{\psi}_{r}\rangle\end{pmatrix}=\begin{pmatrix}\mu_{h}&w_{h}\\\
\overline{w}_{h}&\mu_{h}\end{pmatrix}+\mathcal{O}\big{(}h^{-3}e^{-\frac{2d(0,2L-a)}{h}}\big{)}.$

We deduce that the two first eigenvalues of $\mathcal{H}_{h}$ are

(3.20)
$\lambda_{\pm}=\mu_{h}\pm|w_{h}|+\mathcal{O}\big{(}h^{-3}e^{-\frac{2d(0,2L-a)}{h}}\big{)},$

and Theorem 5 follows. Note that, instead of $\hat{\psi}_{j}$, one could also
use a more symmetric orthonormalization as in [7]. The choice (3.18) is the
same as in [5].

##  4\. Estimates on the hopping coefficient $w_{h}$

We prove here the following estimate on the hopping coefficient, which governs
the spectral gap by Theorem 5.

######  Theorem 8.

There exists a constant $C(B,L,v)>0$ such that

$w_{h}=-C(B,L,v)h^{\frac{1}{2}}e^{-\frac{S}{h}}(1+o(1))$

as $h\to 0$, with

(4.1)
$S=2d(0,L)+\int_{0}^{L}\sqrt{\frac{B^{2}(2L-r)^{2}}{4}-v_{0}}-\sqrt{\frac{B^{2}r^{2}}{4}-v_{0}}\mathrm{d}r.$

The constant $C(B,L,v)$ has a long but explicit expression, see (4.15).

Contrary to the approach of [5], we use the representation of $w_{h}$ as an
explicit integral on a line separating the two wells, as in [11, Equation
(2.25)]. Compared to the non-magnetic case, the novelty here is that the phase
appearing in this integral takes complex values due to the magnetic flux. A
similar effect was observed in the purely magnetic situation [7]. However for
constant magnetic fields the estimate is simpler, since the resulting complex
integral is Gaussian. We give the details of the proof in Section 4.1 below.

Our main result, Theorem 1, follows from the reduction to the hopping
coefficient Theorem 5, together with Theorem 8. We only need to ensure that
the error from Theorem 5 is smaller than $w_{h}$. We discuss this condition in
Section 4.2.

###  4.1. Proof of Theorem 8

Since $v$ is supported in the ball of radius $a$, and $L>a$, the integral
defining $w_{h}$ can be restricted to the left half-plane,
$\Omega_{\ell}=\\{(x,y)|x<0\\}$,

(4.2)
$w_{h}=\int_{\Omega_{\ell}}v(x+L,y)\varphi_{\ell}\overline{\varphi_{r}}\mathrm{d}x\mathrm{d}y.$

Using the equations (3.1) and (3.2) satisfied by $\varphi_{\ell}$ and
$\varphi_{r}$ on $\Omega_{\ell}$, we find

(4.3)
$w_{h}=\int_{\Omega_{\ell}}\varphi_{\ell}\cdot\overline{(-ih\nabla-\mathbf{A})^{2}\varphi_{r}}\mathrm{d}x\mathrm{d}y-\int_{\Omega_{\ell}}(-ih\nabla-\mathbf{A})^{2}\varphi_{\ell}\cdot\overline{\varphi_{r}}\mathrm{d}x\mathrm{d}y.$

A partial integration yields

(4.4)
$w_{h}=ih\int_{\partial\Omega_{\ell}}\varphi_{\ell}\overline{(-ih\nabla-\mathbf{A})\varphi_{r}}\cdot\mathbf{n}+\overline{\varphi_{r}}(-ih\nabla-\mathbf{A})\varphi_{\ell}\cdot\mathbf{n},$

where $\mathbf{n}=(1,0)$ is the outward normal to $\Omega_{\ell}$. Now due to
our choice of gauge, $A_{1}=0$ and only remains

(4.5)
$w_{h}=h^{2}\int_{\mathbf{R}}\overline{\varphi_{r}(0,y)}\partial_{x}\varphi_{\ell}(0,y)-\varphi_{\ell}(0,y)\overline{\partial_{x}\varphi_{r}(0,y)}\mathrm{d}y.$

We recall that $\varphi_{\ell}$ and $\varphi_{r}$ are related to the radial
solution $\varphi$ of the single-well problem by

(4.6)
$\displaystyle\varphi_{\ell}(x,y)=e^{-\frac{i\sigma_{\ell}}{h}}\varphi(x+L,y),\qquad\varphi_{r}(x,y)=e^{-\frac{i\sigma_{r}}{h}}\varphi(x-L,y).$

We now use these relations, together with the integral representation formula
in Lemma 2.5, to calculate $w_{h}$. First of all,

(4.7)
$\varphi_{j}(0,y)=C_{h}e^{-\frac{i\sigma_{j}}{h}}\int_{0}^{\infty}e^{-\frac{B(L^{2}+y^{2})}{4h}(1+2t)}t^{\alpha-1}(1+t)^{-\alpha}\mathrm{d}t,$

and

$\displaystyle\partial_{x}\varphi_{\ell}(0,y)$
$\displaystyle=C_{h}e^{-\frac{i\sigma_{\ell}}{h}}\int_{0}^{\infty}e^{-\frac{B(L^{2}+y^{2})}{4h}(1+2t)}\Big{(}-\frac{i}{h}\partial_{x}\sigma_{\ell}-\frac{BL}{2h}(1+2t)\Big{)}t^{\alpha-1}(1+t)^{-\alpha}\mathrm{d}t,$
$\displaystyle\partial_{x}\varphi_{r}(0,y)$
$\displaystyle=C_{h}e^{-\frac{i\sigma_{r}}{h}}\int_{0}^{\infty}e^{-\frac{B(L^{2}+y^{2})}{4h}(1+2t)}\Big{(}-\frac{i}{h}\partial_{x}\sigma_{r}+\frac{BL}{2h}(1+2t)\Big{)}t^{\alpha-1}(1+t)^{-\alpha}\mathrm{d}t.$

We insert this in (4.5),

$\displaystyle w_{h}=-h^{2}C_{h}^{2}$
$\displaystyle\int_{\mathbf{R}\times\mathbf{R}_{+}^{2}}\Big{(}\frac{i}{h}(\partial_{x}\sigma_{r}+\partial_{x}\sigma_{\ell})+\frac{BL}{h}(1+t+s)\Big{)}e^{\frac{i}{h}(\sigma_{r}-\sigma_{\ell})}$
(4.8) $\displaystyle
e^{-\frac{B(L^{2}+y^{2})}{2h}(1+t+s)}(ts)^{\alpha-1}(1+t)^{-\alpha}(1+s)^{-\alpha}\mathrm{d}y\mathrm{d}t\mathrm{d}s.$

In (4.8) we use $\sigma_{r}-\sigma_{\ell}=-BLy$,

$w_{h}=-hC_{h}^{2}BL\int_{\mathbf{R}\times\mathbf{R}_{+}^{2}}e^{-\frac{B(L^{2}+y^{2})}{2h}(1+t+s)-\frac{iBLy}{h}}\frac{(1+t+s-iy/L)(ts)^{\alpha-1}}{(1+t)^{\alpha}(1+s)^{\alpha}}\mathrm{d}y\mathrm{d}t\mathrm{d}s.$

Here the $y$-integral is Gaussian: inside the exponential we have

$\frac{B(L^{2}+y^{2})}{2}(1+t+s)+iBLy=\frac{B(1+t+s)}{2}\big{(}y+\frac{iL}{1+t+s}\big{)}^{2}+\frac{BL^{2}}{2}\big{(}1+t+s+\frac{1}{1+t+s}\big{)},$

and this complex-centered Gaussian is integrated as

(4.9)
$\int_{\mathbf{R}}\exp\Big{(}-\frac{B(1+t+s)}{2h}\big{(}y+\frac{iL}{1+t+s}\big{)}^{2}\Big{)}\mathrm{d}y=\sqrt{\frac{2\pi
h}{B(1+t+s)}},$

and

(4.10)
$\int_{\mathbf{R}}\frac{-iy}{L}\exp\Big{(}-\frac{B(1+t+s)}{2h}\big{(}y+\frac{iL}{1+t+s}\big{)}^{2}\Big{)}\mathrm{d}y=\frac{-1}{1+t+s}\sqrt{\frac{2\pi
h}{B(1+t+s)}}.$

Thus,

$w_{h}=-h^{\frac{3}{2}}C_{h}^{2}\sqrt{2\pi
BL^{2}}\int_{\mathbf{R}_{+}^{2}}\exp\Big{(}-\frac{BL^{2}}{2h}\big{(}1+t+s+\frac{1}{1+t+s}\big{)}\Big{)}\frac{\omega(s,t)(ts)^{\alpha-1}}{(1+t)^{\alpha}(1+s)^{\alpha}}\mathrm{d}t\mathrm{d}s,$

with $\omega(s,t)=(1+t+s)^{1/2}-(1+t+s)^{-3/2}\geq 0$. We replace
$\alpha=\frac{|v_{0}|}{2Bh}+\nu+o(1)$ as $h\to 0$,

(4.11) $w_{h}=-h^{\frac{3}{2}}C_{h}^{2}\sqrt{2\pi
BL^{2}}\int_{\mathbf{R}_{+}^{2}}e^{-\frac{g(s,t)}{h}}\frac{\omega(s,t)(ts)^{\nu-1}}{(1+t)^{\nu}(1+s)^{\nu}}\mathrm{d}t\mathrm{d}s\,(1+o(1)),$

with

(4.12)
$g(s,t)=\frac{BL^{2}}{2}\Big{(}1+t+s+\frac{1}{1+t+s}\Big{)}+\frac{|v_{0}|}{2B}\Big{(}\ln\big{(}\frac{1+t}{t}\big{)}+\ln\big{(}\frac{1+s}{s}\big{)}\Big{)}.$

The function $g$ has a unique critical point, which is also a global minimum,
at $t=s=t_{\star}$, with

(4.13) $t_{\star}=\frac{1}{2}\sqrt{N}-\frac{1}{2}+\frac{1}{2}\sqrt{1+N},\qquad
N=\frac{|v_{0}|}{B^{2}L^{2}}.$

Moreover,

(4.14)
$g(t_{\star},t_{\star})=BL^{2}\Big{(}\sqrt{1+N}+N\ln\big{(}\frac{1+\sqrt{1+N}}{\sqrt{N}}\big{)}\Big{)}=\int_{0}^{2L}\sqrt{\frac{B^{2}r^{2}}{4}+|v_{0}|}\mathrm{d}r.$

Using the Laplace method, the integral (4.11) can be estimated as

(4.15)
$w_{h}\sim-h^{5/2}C_{h}^{2}\frac{(2\pi)^{3/2}\sqrt{BL^{2}}}{|g^{\prime\prime}(t_{\star},t_{\star})|^{\frac{1}{2}}}\frac{\omega(t_{\star},t_{\star})t_{\star}^{2\nu-2}}{(1+t_{\star})^{2\nu}}\exp\Big{(}-\frac{1}{h}\int_{0}^{2L}\sqrt{\frac{B^{2}r^{2}}{4}+|v_{0}|}\mathrm{d}r\Big{)},$

when $h\to 0$. We insert the estimate (2.6) on $C_{h}$, and we find an
explicit constant $C(B,L,v)>0$ such that

(4.16)
$w_{h}\sim-C(B,L,v)h^{\frac{1}{2}}\exp\Big{(}-\frac{1}{h}\int_{0}^{2L}\sqrt{\frac{B^{2}r^{2}}{4}+|v_{0}|}\mathrm{d}r+\frac{2}{h}\tilde{d}(L)-\frac{2}{h}d(0,L)\Big{)}.$

We find the exponential rate (4.1) since

$\displaystyle\int_{0}^{2L}$
$\displaystyle\sqrt{\frac{B^{2}r^{2}}{4}+|v_{0}|}\mathrm{d}r-2\tilde{d}(L)=\int_{0}^{L}\sqrt{\frac{B^{2}(2L-r)^{2}}{4}-v_{0}}-\sqrt{\frac{B^{2}r^{2}}{4}-v_{0}}\mathrm{d}r.$

###  4.2. Condition on the error terms

To ensure the error terms from Theorem 5 to be smaller than $w_{h}$, we need
$S<2d(0,2L-a)$. We recall formula (4.1) for $S$, which can be rewritten as

(4.17) $S=d(0,2L)+d(0,L)-\tilde{d}(L)=2d(0,2L)-\tilde{d}(2L).$

Thus,

$\displaystyle
S-2d(0,2L-a)=2d(2L-a,2L)-\tilde{d}(2L)=d(2L-a,2L)-\tilde{d}(2L-a).$

Hence the condition $S<2d(0,2L-a)$ is satisfied as soon as

(4.18) $d(2L-a,2L)<\tilde{d}(2L-a).$

We now show that (4.18) is true as soon as $L>(1+\frac{\sqrt{3}}{2})a$. First,
we bound the left-hand side by

(4.19)
$d(2L-a,2L)\leq\int_{2L-a}^{2L}\frac{Br}{2}+\sqrt{|v_{0}|}\mathrm{d}r=\sqrt{|v_{0}|}a+\frac{Ba}{4}(4L-a),$

and the right-hand side

(4.20)
$\tilde{d}(2L-a)\geq\int_{0}^{a}\sqrt{|v_{0}|}\mathrm{d}r+\int_{a}^{2L-a}\frac{Br}{2}\mathrm{d}r=\sqrt{|v_{0}|}a+BL(L-a).$

Thus, a sufficient condition for (4.18) to hold is

$BL(L-a)>BLa-\frac{Ba^{2}}{4},$

which is equivalent to $L>(1+\frac{\sqrt{3}}{2})a$.

## Acknowledgements

The author thanks Søren Fournais, Bernard Helffer, Ayman Kachmar, and Nicolas
Raymond for many enlightening discussions, and for encouraging this work.

This work is funded by the European Union. Views and opinions expressed are
however those of the author only and do not necessarily reflect those of the
European Union or the European Research Council. Neither the European Union
nor the granting authority can be held responsible for them.

## References

  * [1] K. Abou Alfa.  Tunneling effect in two dimensions with vanishing magnetic fields.  arXiv:2212.04289, 2023.
  * [2] V. Bonnaillie-Noël, F. Hérau, and N. Raymond.  Semiclassical tunneling and magnetic flux effects on the circle.  J. Spectr. Theory, 7(3):771–796, 2017.
  * [3] V. Bonnaillie-Noël, F. Hérau, and N. Raymond.  Purely magnetic tunneling effect in two dimensions.  Invent. Math., 227(2):745–793, 2022.
  * [4] M. Dimassi and J. Sjöstrand.  Spectral asymptotics in the semi-classical limit, volume 268 of London Mathematical Society Lecture Note Series.  Cambridge University Press, Cambridge, 1999.
  * [5] C. Fefferman, J. Shapiro, and M. I. Weinstein.  Lower bound on quantum tunneling for strong magnetic fields.  SIAM J. Math. Anal., 54(1):1105–1130, 2022.
  * [6] S. Fournais, B. Helffer, and A. Kachmar.  Tunneling effect induced by a curved magnetic edge.  In The physics and mathematics of Elliott Lieb—the 90th anniversary. Vol. I, pages 315–350. EMS Press, Berlin, 2022.
  * [7] S. Fournais, L. Morin, and N. Raymond.  Purely magnetic tunnelling between radial magnetic wells.  arXiv:2308.04315, 2023.
  * [8] B. Helffer.  Semi-classical analysis for the Schrödinger operator and applications, volume 1336 of Lecture Notes in Mathematics.  Springer-Verlag, Berlin, 1988.
  * [9] B. Helffer and A. Kachmar.  Quantum tunneling in deep potential wells and strong magnetic field revisited.  arXiv:2208.13030, 2023.
  * [10] B. Helffer, A. Kachmar, and M. P. Sundqvist.  Flux and symmetry effects on quantum tunneling.  arXiv:2307.06712, 2023.
  * [11] B. Helffer and J. Sjöstrand.  Multiple wells in the semiclassical limit. I.  Comm. Partial Differential Equations, 9(4):337–408, 1984.
  * [12] B. Helffer and J. Sjöstrand.  Puits multiples en limite semi-classique. II: Interaction moléculaire. Symétries. Perturbation. (Multiple wells in the semi-classical limit. II: Molecular interaction. Symmetry. Perturbation).  Ann. Inst. Henri Poincaré, Phys. Théor., 42:127–212, 1985\.
  * [13] B. Helffer and J. Sjöstrand.  Effet tunnel pour l’équation de Schrödinger avec champ magnétique. (Tunnel effect for the Schrödinger equation with a magnetic field).  Ann. Sc. Norm. Super. Pisa, Cl. Sci., IV. Ser., 14(4):625–657, 1987\.
  * [14] S. Nakamura.  Tunneling estimates for magnetic Schrödinger operators.  Comm. Math. Phys., 200(1):25–34, 1999.


# The topological order of the space111Essay written for the Gravity Research
Foundation 2021 Awards for Essays on Gravitation.

Jingbo Wang<EMAIL_ADDRESS>Institute for Gravitation and
Astrophysics, College of Physics and Electronic Engineering, Xinyang Normal
University, Xinyang, 464000, P. R. China

###### Abstract

Topological order is a new type order that beyond Landau’s symmetry breaking
theory. The topological entanglement entropy provides a universal quantum
number to characterize the topological order in a system. The topological
entanglement entropy of the BTZ black hole was calculated and found that it
coincides with that for fractional quantum Hall state. So the BTZ black holes
have the same topological order with the fractional quantum Hall state. We
conjecture that black holes in higher dimensions also have topological orders.
Next we want to study the topological order of ordinary spaces which can be
described by spin network states in loop quantum gravity. We advise to bring
in the methods and results in string-net condensation to loop quantum gravity
to solve some difficult problems.

black hole; information loss paradox; Hawking radiation

###### pacs:

04.70.Dy,04.60.Pp

In our world there are many different phases of matter. For a long time, it
was believed that those phases can be described by Laudau’s symmetry breaking
theory. But in late 1980s, it was found that in chiral spin liquid there may
exist a new kind of order, so-called topological order that beyond the usual
symmetry description [1]. The term “topological” was motivated by the fact
that the low energy effective theory for the chiral spin liquid is a
topological quantum field theory. Since then, the study of topological phases
of quantum matter slowly becomes more and more active, and then a main stream
in condensed matter physics now. It was proposed that such phase of matter can
be used to build powerful quantum computer [2, 3].

The notion of topological order can be defined macroscopically by topological
degeneracy and non-Abelian geometric phases of the ground states on spaces
with non-trivial topology. Those properties are all robust against any small
perturbations. It can also be defined microscopically as pattern of long range
entanglement. Topological order can produce quasiparticles with fractional
quantum numbers and fractional/Fermi statistics, robust gapless boundary modes
and emergent gauge excitations. So they can provide an unification of gauge
interaction and Fermi statistics [4].

A universal quantum number to characterize the topological order in a system
can be provided by the topological entanglement entropy (TEE)[5, 6]. For a
two-dimensional topological ordered system, the entanglement area law satisfy

$S\sim\alpha L-\gamma,$ (1)

where $L$ is the length of the boundary, and $\alpha$ is a parameter that
depend on the system, $\gamma>0$ is a constant term that indicates the
existence of the topological order. This term is called the topological
entanglement entropy. For example, the TEE for fractional quantum Hall state
with filling factor $v=\frac{1}{m}$ takes the form

$\gamma=\frac{1}{2}\ln m.$ (2)

In the previous paper [7], we started from the Chern-Simons theory formulation
of gravity with negative cosmological constant, and applied some suitable
boundary conditions, then found that for BTZ black holes the boundary degrees
of freedom can be described by two chiral massless scalar fields $\Psi_{L}$
(for left-moving) and $\Psi_{R}$ (for right-moving). A chiral massless scalar
field can also be used to describe the boundary modes of the fractional
quantum Hall states, so from this point of view, the BTZ black hole can be
considered as two copies of quantum Hall states, or a quantum spin Hall state.
Quantizing those scalar fields gives the microscopic states of the BTZ black
hole. The entropy for those scalar fields can be calculated to be

$\begin{split}S_{R}=\frac{\pi}{4}(r_{+}+r_{-})-\frac{1}{2}\ln\frac{r_{+}+r_{-}}{8},\\\
S_{L}=\frac{\pi}{4}(r_{+}-r_{-})-\frac{1}{2}\ln\frac{r_{+}-r_{-}}{8},\\\
S_{BH}=S_{R}+S_{L}=\frac{2\pi
r_{+}}{4}-\frac{1}{2}\ln\frac{r^{2}_{+}-r^{2}_{-}}{64}.\end{split}$ (3)

The logarithmic term in (3) for right sector is the same with the topological
entanglement entropy for fractional quantum Hall state with filling factor
$v_{R}=\frac{8}{r_{+}+r_{-}}$, and the same is true for left sector. So we can
say that the right (left) sector of the BTZ black hole has the same
topological order with the fractional quantum Hall state with filling factor
$v_{R}=\frac{8}{r_{+}+r_{-}}$ ($v_{L}=\frac{8}{r_{+}-r_{-}}$).

For higher dimensional black holes, such as Kerr black holes in four
dimension, the boundary modes can be described by massless scalar field, which
is also the same as higher dimensional topological insulators [8]. Just like
the BTZ black hole, we can also identify the quantum states of the scalar
field with the microstates of Kerr black holes. We conjecture that black holes
in higher dimensions can also have topological orders.

Since black holes can have topological orders, a natural question is: can an
ordinary space (without black hole) have topological order? A quantum
description of space can be given by spin network states. They play important
roles in loop quantum gravity, lattice gauge theory, topological quantum field
theory and so on. So the question becomes if a general spin network state has
topological order?

A physical picture for topological order is provided by string-net
condensation in Levin-Wen model[9]. The Levin-Wen model is believed to be a
Hamiltonian version of the Turaev-Viro topological quantum field theory in
three dimensional spacetime. This model can be generalized to higher
dimension. The underlying mathematical framework is tensor category theory.
They can realize all discrete gauge theories (in any dimension) and all
doubled Chern-Simons theories (in $2+1$ dimensions). Besides, they can provide
exactly soluble Hamiltonian and ground state wave functions for each of these
phases. Actually spin networks are very similar to string-nets, so in Ref.[10]
the authors suggest to bring in the string-nets into loop quantum gravity to
include the fermionic and bosonic excitations. Bringing in the string-nets
into loop quantum gravity can also have other advantages:

  * •

The Hamiltonian for the string-nets is exactly soluble. It takes the form

$H=-\sum_{I}Q_{I}-\sum_{p}B_{p},\quad B_{p}=\sum_{s=0}^{N}a_{s}B_{p}^{s},\quad
a_{s}=\frac{d_{s}}{\sum_{i=0}^{N}d_{i}^{2}},$ (4)

where the sums rum over vertices $I$ and the plaquettes $p$ of the lattice.
The first term $Q_{I}$ is an electric charge operator, and the second term
$B_{p}$ is a linear combination of magnetic flux operators $B_{p}^{s}$.

A difficult problem in loop quantum gravity is that due to the complicate form
of the Hamiltonian constraint, it is very difficult to find exact eigenstates.
One can modify the Hamiltonian for the string-nets to get a new Hamiltonian
which is exactly soluble and also in the same universal class with that in
loop quantum gravity. This is just like the case in fractional quantum Hall
effect. Due to the interaction between electrons the Hamiltonian is very
complicate and the exact ground states is difficult to find. One can construct
a toy Hamiltonian which gives the Laughlin states that lie in the same
universal class with that in true fractional quantum Hall effect.

  * •

In topological phase of matter, a widely observed phenomenon is the boundary-
bulk correspondence [11, 12], which relates the topological structure of the
bulk states to the presence of protected zero-energy boundary states. This
boundary-bulk correspondence can be considered as a weak version of the
holographic principle [13, 14] in gravity theory. Actually in Ref.[13], it was
suggest that “quantum gravity should be described entirely by a topological
quantum field theory”. And topological quantum field theory can describe the
low energy behavior of topological phase of matter.

  * •

On the other hand, the studies in loop quantum gravity can also be applied to
the researches in string-net condensation theory. A main problem in the
unification through string-net condensation is how to construct the graviton
[15]. But in loop quantum gravity, there are some attempts to construct
graviton form loop or spin network [16, 17, 18], which may be useful for
string-net condensation.

In summary, we can use the methods in string-net condensation to study the
spin network states in loop quantum gravity. If space has topological order,
it can support gauge field and Fermion field from simple bosonic qubits. It
may solve the origin of elementary particles in the standard model, and
resents an unification of matter and information [4].

###### Acknowledgements.

This work is supported by Nanhu Scholars Program for Young Scholars of XYNU.

## References

  * [1] X. G. Wen.  Topological Orders in Rigid States.  International Journal of Modern Physics B, 4(2):239–271, January 1990.
  * [2] A. Y. Kitaev.  Fault tolerant quantum computation by anyons.  Annals Phys., 303:2–30, 2003.
  * [3] C. Nayak, S. H. Simon, A. Stern, M. Freedman, and S. Das S.  Non-Abelian anyons and topological quantum computation.  Rev. Mod. Phys., 80:1083–1159, 2008.
  * [4] B. Zeng, X. Chen, D.-L. Zhou, and X. G. Wen.  Quantum Information Meets Quantum Matter: From Quantum Entanglement to Topological Phases of Many-Body Systems.  Springer, New York, 2019.
  * [5] A. Kitaev and J. Preskill.  Topological entanglement entropy.  Phys. Rev. Lett., 96:110404, 2006.
  * [6] M. Levin and X.-G. Wen.  Detecting Topological Order in a Ground State Wave Function.  Phys. Rev. Lett., 96:110405, 2006.
  * [7] J. Wang and C.-G. Huang.  Conformal field theory on the horizon of a BTZ black hole.  Chin. Phys., C42(12):123110, 2018.
  * [8] J. Wang.  Microscopic states of Kerr black holes from boundary-bulk correspondence.  Chin. Phys., C45(1):015107, 2021.
  * [9] M. A. Levin and X.-G. Wen.  String-net condensation:a physical mechanism for topological phases.  Physical Review B, 71(4), Jan 2005.
  * [10] S. Bilson-Thompson and D. Vaid.  LQG for the Bewildered.  2 2014.
  * [11] M. Z. Hasan and C. L. Kane.  Topological Insulators.  Rev. Mod. Phys., 82:3045, 2010.
  * [12] X. L. Qi and S. C. Zhang.  Topological insulators and superconductors.  Rev. Mod. Phys., 83(4):1057–1110, 2011.
  * [13] G. ’t Hooft.  Dimensional reduction in quantum gravity.  Conf. Proc. C, 930308:284–296, 1993.
  * [14] L. Susskind.  The World as a hologram.  J. Math. Phys., 36:6377–6396, 1995.
  * [15] Z.-C. Gu and X.-G. Wen.  Emergence of helicity +- 2 modes (gravitons) from qbit models.  Nucl. Phys. B, 863:90–129, 2012.
  * [16] Z. Joost.  Gravitons in loop quantum gravity.  Nuclear Physics B, 378(1):288–308, 1992.
  * [17] A. Ashtekar, C. Rovelli, and L. Smolin.  Gravitons and loops.  Phys. Rev. D, 44:1740–1755, 1991.
  * [18] E. Bianchi, L. Modesto, C. Rovelli, and S. Speziale.  Graviton propagator in loop quantum gravity.  Class. Quant. Grav., 23:6989–7028, 2006.


# STaR: Distilling Speech Temporal Relation for
Lightweight Speech Self-Supervised Learning Models

###### Abstract

Albeit great performance of Transformer-based speech self-supervised learning
(SSL) models, their large parameter size and computational cost make them
unfavorable to utilize. In this study, we propose to compress the speech SSL
models by distilling speech temporal relation (STaR). Unlike previous works
that directly match the representation for each speech frame, STaR
distillation transfers temporal relation between speech frames, which is more
suitable for lightweight student with limited capacity. We explore three STaR
distillation objectives and select the best combination as the final STaR
loss. Our model distilled from HuBERT Base achieves an overall score of 79.8
on SUPERB benchmark, the best performance among models with up to 27 million
parameters. We show that our method is applicable across different speech SSL
models and maintains robust performance with further reduced parameters.

Index Terms—  speech self-supervised learning, model compression, knowledge
distillation, speech temporal relation

††footnotetext: This work was supported by the National Research Foundation of
Korea grant funded by the Korea government (MSIT) (No. 2021R1A2C1014044).

##  1 Introduction

Transformer-based speech self-supervised learning (SSL) models [1, 2, 3] have
risen to prominence with great performance in various speech-related tasks [4,
5]. Nonetheless, the downside of these models mainly comes from the
requirement of substantial computational resources during the pre-training
stage—HuBERT Base takes over 82 GPU-days for pre-training, and 32 GPUs en
masse are utilized to shorten this [2]. Another downside is their large
parameter size which makes on-device application difficult and renders the
speech SSL models unfavorable for many practical scenarios. The above issues
make compression techniques essential for the speech SSL models, and several
studies have attempted to solve these issues by applying pruning [6] or
quantization [7] technique.

Alternatively, knowledge distillation is another approach for model
compression that trains a student model with a smaller parameter size to
imitate the behavior of a larger teacher model by matching the student’s
representation to the teacher’s. Knowledge distillation of the Transformer-
based speech SSL models is being actively studied, with previous works of
task-specific compression including automatic speech recognition (ASR) [8],
keyword spotting (KWS) [9], and automatic speaker verification (ASV) [10].
However, given that the speech SSL models are utilized in various downstream
tasks, their approaches are limited to only a specific task.

On the other hand, approaches that realize task-agnostic compression via
knowledge distillation include DistilHuBERT [11] and FitHuBERT [12], where the
former suggests shallow and wide student model design, and the latter suggests
deep and narrow one. ARMHuBERT [13] proposes to reuse attention maps across
Transformer [14] layers and utilizes both masked and unmasked speech frames
for masking distillation. Some approaches [15, 16] jointly conduct
distillation with $L_{0}$ regularization [17] and structured pruning.
LightHuBERT [18] implements masking distillation and architecture search, but
it stands apart from other methods due to its excessive computational demand
for creating a teacher-sized supernet.

While the aforementioned studies have demonstrated promising results, two
limitations still exist. First, most approaches neglect the weak
representation capacity of the student and directly match the complex
teacher’s representation for each speech frame by introducing additional
linear heads [11, 12, 13, 15, 19]. This can be an over-constraint for the
lightweight student model, necessitating the establishment of an alternative
distillation objective that better suits to the student. Some studies even
inefficiently discard these trained linear heads after distillation, although
they can convey the teacher’s knowledge [11, 12, 15, 19]. Second, while
pruning allows us to manage the size of model parameters by a sparsity ratio,
it cannot determine computational cost of the model at the same time. As a
result, computational overhead can be higher than the vanilla distillation
approaches, in which the model is specified before the training.

In this study, we explore effective distillation objectives that capture
temporal relation between speech frames for lightweight student model.
Additional parameters are not necessary during distillation, enabling the
construction of a more compact and computationally efficient student model. We
verify the task-agnostic compression of our proposed objectives using SUPERB
benchmark [4]. Our model distilled from HuBERT Base achieves the best overall
score [3] of 79.8 among models with $\sim$ 27 million parameters, while
requiring only 30.7% multiply-accumulates (MACs) and 28.1% parameters of its
teacher model. It even surpasses LightHuBERT [18], which demands extensive
computational resources for compression, in terms of overall score, number of
parameters, and MACs.

##  2 Speech Temporal Relation Distillation

Fig. 1:  We propose three STaR distillation objectives: average attention map,
layer-wise TGM, and intra-layer TGM. TGM captures the temporal relation by
aggregating the channel information at two time steps. Each individual loss is
summed up across all Transformer layers.

Speech SSL models are capable of representing speech frames into features that
function as specific acoustic units for every time step. This can be
attributed to the pre-training scheme of a speech SSL model, predicting the
corresponding cluster of each masked frame [2, 3]. The pre-processing
procedure of wav2vec-U [20] also involves grouping the speech frames from the
same cluster for phonemic segmentation. As such, speech SSL models generate
the representations for each frame that are closely tied to specific acoustic
units, and how to represent these units is the core knowledge of the models.
However, directly learning the teacher’s representations, which convey the
characteristics of these units, can be an over-constraint for the student with
limited representation capacity.

To this end, we propose to distill the knowledge from the teacher in a more
flexible manner by focusing on the relation between speech frames instead. We
explore three Speech Temporal Relation (STaR) distillation objectives (Fig.
1), by which the distillation transfers pairwise temporal relation of the
speech frames. We first suggest average attention map distillation, a naïve
way to distill the temporal relation. Then, temporal Gram matrix (TGM) is
introduced as a distillation target, and on top of this, we propose two TGM
distillation objectives: layer-wise TGM distillation and intra-layer TGM
distillation. We highlight that our methods are task-agnostic compression and
do not require additional parameters during distillation stage unlike previous
works [11, 12, 13, 15, 19].

###  2.1 Average Attention Map Distillation

Attention map in Transformer [14] structure captures the temporal relation
between key and query. Each entry in the attention map indicates a level of
the relationship between two associated frames of the sequence, therefore,
distilling this map can be an intuitive way to transfer the temporal relation
from the teacher. Given key and query matrices for head $h$, $\textbf{K}_{h}$,
$\textbf{Q}_{h}\in\mathbb{R}^{d_{h}\times N}$, in multi-head self-attention
(MHSA), the corresponding attention map $\textbf{A}_{h}\in\mathbb{R}^{N\times
N}$ is
$\textbf{A}_{h}=\textit{softmax}\big{(}\textbf{Q}^{\top}_{h}\textbf{K}_{h}/\sqrt{d_{h}}\big{)}$.
$d_{h}$ is the width of the key matrix for head $h$, and $N$ is the sequence
length. Several studies have leveraged the attention maps of all heads in the
last Transformer layer for knowledge distillation [21, 22]; nevertheless, this
is not ideal for the speech SSL models where each layer plays an essential
role in different speech-related task [3, 4]. Distilling the attention maps of
all heads across all layers would be an ideal way, but it leads to excessive
computational overhead for training.

Accordingly, we propose to distill the attention map averaged across all heads
from each Transformer layer as an alternative. From each Transformer layer,
the proposed loss calculates Kullback-Leibler (KL) divergence between the
average attention maps of teacher T and student S.

$\displaystyle\mathcal{L}_{\text{avg-attn}}$
$\displaystyle=\sum_{\ell=1}^{L}\sum_{t=1}^{N}D_{KL}\left(\frac{1}{H^{\text{T}}}\sum_{h=1}^{H^{\text{T}}}\textbf{A}^{\ell,\text{T}}_{h,t}\bigg{\lVert}\frac{1}{H^{\text{S}}}\sum_{h=1}^{H^{\text{S}}}\textbf{A}^{\ell,\text{S}}_{h,t}\right),$
(1)

where $H$ and $L$ denote the total number of attention heads and Transformer
layers, respectively. $\textbf{A}^{\ell}_{h,t}$ represents the attention
distribution at time step $t$ within the attention head $h$ of Transformer
layer $\ell$.

###  2.2 Temporal Gram Matrix Distillation

While distilling the attention maps can transfer their temporal relation to
the student, it might not provide sufficient hints since these maps are not
the direct features used for inference. In order to provide stronger hints, we
present the distillation objectives regarding temporal relation within each
Transformer [14] layer output by computing a Gram matrix. Correlation between
two samples can be expressed via Gram matrix, defined as an inner product
between sample representations [23]. Let $\textbf{F}\in\mathbb{R}^{d\times N}$
be a representation of the channel width $d$. Then, the Gram matrix
$\tilde{\textbf{G}}$ is

$\displaystyle\tilde{G}_{ij}=\sum_{k}F_{ik}F_{jk},\quad\tilde{\textbf{G}}\in\mathbb{R}^{d\times
d},$ (2)

where $F_{\cdot k}$ denotes the $k$-th time step’s representation.

Gram matrix has been utilized in speech and vision domain to represent the
properties of data, referred to as style. Beginning with the proposal of
artistic style transfer via Gram matrix [23], numerous studies advocate
leveraging the matrix for speaker embedding [24, 25] or voice conversion [26,
27]. The style loss [23] minimizes the correlation between channels by
aggregating positional information within a sample, as in Eq. 2. The key
strength of this loss lies in its objective which can be seen as a
distribution alignment process, minimizing maximum mean discrepancy [28]
between two sets [29].

Gram matrix or its variant is also exploited as an objective for conducting
knowledge distillation [30, 31]. Prior works share the idea that they
aggregate temporal or spatial information to illustrate the relation between
channels when computing the Gram matrix. In contrast, we propose a temporal
Gram matrix (TGM) which aggregates channel information at two time steps to
take into account temporal relation between speech frames. Note that this
differs from other works [32, 33] which capture relation between samples
within the same batch. The temporal Gram matrix G is defined as

$\displaystyle{G}_{ij}=\sum_{k}F_{ki}F_{kj},\quad\textbf{G}\in\mathbb{R}^{N\times
N}.$ (3)

Layer-wise TGM distillation Layer-wise TGM distillation takes the TGM of each
Transformer layer output as an objective. We also include the first
Transformer layer input to distill the front-end convolutional layers more
directly. This approach effectively transfers the information encoded in each
layer of the speech SSL models [34] to the corresponding layer of the student
model. Indeed, previous studies have confirmed the effectiveness of layer-wise
distillation for the speech SSL models that targets intermediate layers [12,
13, 19]. Regarding the first Transformer layer input as the zeroth output, the
layer-wise TGM distillation loss is the mean squared error (MSE) between the
TGMs of teacher and student across all layers.

$\displaystyle\mathcal{L}_{\text{layer-wise}}$
$\displaystyle=\sum_{\ell=0}^{L}\big{\lVert}\textbf{G}^{\ell,\text{T}}-\textbf{G}^{\ell,\text{S}}\big{\rVert}^{2}_{2}$
(4)

Intra-layer TGM distillation To offer a more flexible distillation objective
for the student model with reduced parameter size, we propose intra-layer TGM
distillation. Inspired by a flow of solution procedure matrix [30], we modify
the TGM as computing the temporal relation between the input and output of a
single Transformer layer, specifically emphasizing its intra-layer role. This
matrix captures the progression of speech representation within every
individual layer, providing a more flexible objective based on two different
representations. The modified TGM $\check{\textbf{G}}$ of the Transformer
layer $\ell$ and the intra-layer TGM distillation loss are defined as

$\displaystyle\check{G}^{\ell}_{ij}$
$\displaystyle=\sum_{k}F^{\ell-1}_{ki}F^{\ell}_{kj},\quad\check{\textbf{G}}^{\ell}\in\mathbb{R}^{N\times
N},$ (5) $\displaystyle\mathcal{L}_{\text{intra-layer}}$
$\displaystyle=\sum_{\ell=1}^{L}\big{\lVert}\check{\textbf{G}}^{\ell,\text{T}}-\check{\textbf{G}}^{\ell,\text{S}}\big{\rVert}^{2}_{2}.$
(6)

The advantage of the objectives introduced in this section is needlessness of
additional parameters for implementing distillation. Even if the channel width
of the student model is not equal to the teacher’s, our losses can be
formulated without additional linear projections, provided their time lengths
$N$ are the same. As a result, we can create a more compact student and fully
transfer the teacher’s temporal relation knowledge, unlike previous works that
discard the projection heads carrying the teacher’s knowledge [11, 12, 15,
19].

##  3 Results

###  3.1 Experimental Details

Training details HuBERT Base [2], comprising 12 Transformer [14] layers, is
our primary speech SSL model to distill. We train our student model with
LibriSpeech [35] train-clean-100 dataset (100h) for 200 epochs, and full
dataset (960h) for 100 epochs. We follow the student’s training recipe of
[13], except for the cosine scheduler and the learning rate of 1e-3. Effective
batch size including gradient accumulation is 48, using two NVIDIA RTX 4090
GPUs.

Student description We create the student model by reducing the width of
attention layer and feed-forward network (FFN) while retaining the number of
layers in the teacher model. This is to perform layer-to-layer (L2L)
distillation for every layer, as in Eqs. 1, 4, and 6. The width of (attention,
FFN) is set as (432, 976) or (432, 1392) to match the parameter size with
prior works [13, 15, 18].

Evaluation Once distilled, the student model is evaluated on the SUPERB
benchmark [4] to verify its task-agnostic characteristic. The benchmark
consists of 10 speech-related tasks, including phoneme recognition (PR), ASR,
KWS, query by example spoken term detection (QbE), speaker identification
(SID), ASV, speaker diarization (SD), intent classification (IC), slot filling
(SF), and emotion recognition (ER). We fine-tune the model with default SUPERB
recipes, except for enabling the learning rate scheduler and setting the
learning rate of the SID task as 5e-3. For the metrics that assess the entire
tasks, we use overall score from [3] and generalizability score from [5]; the
former is the absolute average of each task’s performance, while the latter
uses the linearly mapped scores in the range 0 (log mel filterbank) $\sim$
1000 (SOTA reported in [4]). We follow the implementation of [5] for
estimating MACs.

###  3.2 Selection of STaR Loss

Table 1 compares the performance of the student models including our proposed
STaR distillations. Average attention map distillation outperforms FitHuBERT
[12] and is on par with MaskHuBERT [13], while requiring 4.6% and 16.3% fewer
parameters than FitHuBERT and MaskHuBERT, respectively. It also reveals that
L2L distillation for every layer is more effective than addressing only the
last layer ($\mathcal{L}_{\text{last-attn}}$) [21]. Besides, layer-wise and
intra-layer TGM distillation outperform the average attention map
distillation, as matching the TGMs synchronizes temporal relation of the
features that serve as direct outputs in fine-tuning. Incorporating the
consideration of the average rank [11] among the STaR students, we set the
default STaR loss as the combination of layer-wise and intra-layer losses and
denote this student as STaRHuBERT.

Table 1:  Results of the SUPERB benchmark for different STaR distillation
objectives. “Overall” and “General.” denote the overall and generalizability
scores. “Ranking” is the average rank over all SUPERB tasks proposed in [11].
$\mathcal{L}_{\text{last-attn}}$ is the last layer’s attention map
distillation [21]. We have reproduced MaskHuBERT-100h [13].

Methods |  Overall $\uparrow$ |  General. $\uparrow$ |  Ranking $\downarrow$
---|---|---|---
LibriSpeech 100h distillation |  |  |
FitHuBERT [12] | 74.5 | 695 | -
MaskHuBERT | 76.3 | 789 | -
$\mathcal{L}_{\text{last-attn}}$ | 75.7 | 750 | -
$\mathcal{L}_{\text{avg-attn}}$ | 76.5 | 766 | 4.4
$\mathcal{L}_{\text{layer-wise}}$ | 77.8 | 829 | 2.7
$\mathcal{L}_{\text{intra-layer}}$ | 77.7 | 820 | 3.2
$\mathcal{L}_{\text{layer-wise}}+\mathcal{L}_{\text{intra-layer}}$ | 77.8 | 831 | 2.2
$\mathcal{L}_{\text{layer-wise}}+\mathcal{L}_{\text{intra-layer}}+\mathcal{L}_{\text{avg-attn}}$ | 77.7 | 827 | 2.6
LibriSpeech 960h distillation |  |  |
$\mathcal{L}_{\text{layer-wise}}$ | 79.4 | 887 | 2.1
$\mathcal{L}_{\text{layer-wise}}+\mathcal{L}_{\text{intra-layer}}$ | 79.5 | 887 | 1.7
$\mathcal{L}_{\text{layer-wise}}+\mathcal{L}_{\text{intra-layer}}+\mathcal{L}_{\text{avg-attn}}$ | 78.8 | 871 | 2.3

Table 2:  Detailed evaluation results on the SUPERB benchmark. Metrics include
number of parameters, number of MACs, phoneme error rate (PER,%), word error
rate (WER,%) without language model, accuracy (Acc,%), maximum term weighted
value (MTWV), equal error rate (EER,%), diarization error rate (DER,%), F1
score (F1,%), and concept error rate (CER,%). The best performance is bolded,
and the second place is underlined. LightHuBERT [18] stands apart from other
works, requiring extensive computational resources for compression.

| Computation | Performance | Content | Speaker | Semantics | Paral.
---|---|---|---|---|---|---
| Params | MACs | Overall | General. | PR | ASR | KS | QbE | SID | ASV | SD | IC | SF | ER
Models |  Millions $\downarrow$ |  Giga $\downarrow$ |  Score $\uparrow$ |  Score $\uparrow$ |  PER $\downarrow$ |  WER $\downarrow$ |  Acc $\uparrow$ |  MTWV $\uparrow$ |  Acc $\uparrow$ |  EER $\downarrow$ |  DER $\downarrow$ |  Acc $\uparrow$ |  F1 $\uparrow$ |  CER $\downarrow$ |  Acc $\uparrow$
Baselines |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
SOTA [4] | - | - | 82.8 | 1000 | 3.53 | 3.62 | 96.66 | 0.0736 | 90.33 | 5.11 | 5.62 | 98.76 | 89.81 | 21.76 | 67.62
FBANK (log mel filterbank) | 0 | 0.4791 | 46.5 | 0 | 82.01 | 23.18 | 8.63 | 0.0058 | 8.5E-4 | 9.56 | 10.55 | 9.1 | 69.64 | 52.94 | 35.39
HuBERT Base [2] | 94.70 | 1669 | 80.8 | 941 | 5.41 | 6.42 | 96.30 | 0.0736 | 81.42 | 5.11 | 5.88 | 98.34 | 88.53 | 25.20 | 64.92
LightHuBERT $a_{\text{Small}}$ [18] | 27.00 | 860.7 | 79.1 | 900 | 6.60 | 8.33 | 96.07 | 0.0764 | 69.70 | 5.42 | 5.85 | 98.23 | 87.58 | 26.90 | 64.12
LibriSpeech 960h distillation |  |  |  |  |  |  |  |  |  |  |
ARMHuBERT-S [13] | 22.39 | 449.4 | 77.5 | 828 | 8.63 | 10.82 | 96.82 | 0.0720 | 63.76 | 5.58 | 7.01 | 97.02 | 86.34 | 29.02 | 62.96
DPHuBERT [15] | 23.59 | 654.1 | 78.9 | 866 | 9.67 | 10.47 | 96.36 | 0.0693 | 76.83 | 5.84 | 5.92 | 97.92 | 86.86 | 28.26 | 63.16
STaRHuBERT (ours) | 22.31 | 463.5 | 79.5 | 887 | 8.16 | 9.35 | 96.27 | 0.0688 | 77.58 | 5.39 | 6.05 | 97.55 | 87.94 | 25.31 | 63.01
STaRHuBERT-L (ours) | 26.63 | 511.9 | 79.8 | 896 | 7.97 | 8.91 | 96.56 | 0.0677 | 78.66 | 5.45 | 5.83 | 97.50 | 88.01 | 25.36 | 63.48
LibriSpeech 100h distillation |  |  |  |  |  |  |  |  |  |  |
ARMHuBERT-S [13] | 22.39 | 449.4 | 76.8 | 800 | 9.17 | 11.83 | 96.01 | 0.0569 | 66.48 | 5.92 | 6.23 | 95.97 | 83.89 | 33.29 | 63.29
DPHuBERT [15] | 23.59 | 654.1 | 77.7 | 819 | 10.02 | 11.38 | 96.36 | 0.0634 | 73.37 | 6.25 | 6.03 | 97.42 | 84.83 | 33.03 | 62.78
STaRHuBERT (ours) | 22.31 | 463.5 | 77.8 | 831 | 9.17 | 10.92 | 96.46 | 0.0626 | 72.26 | 5.66 | 5.97 | 97.21 | 84.89 | 30.77 | 61.22

###  3.3 Detailed SUPERB Benchmark Results

In Table 2, STaRHuBERT outperforms other methods [13, 15] by a large margin
for both 100h and 960h distillations, presenting the superiority of our
distillation objectives. Among the models with less than 24 million
parameters, STaRHuBERT shows the best performance in both overall [3] and
generalizability [5] scores with the fewest parameters. STaRHuBERT-L, which
has the wider FFN width of 1392, attains an overall score of 79.8, even
surpassing LigthHuBERT [18] for the first time as an approach that does not
demand excessive computational resources for compression.

Delving into specific tasks, STaRHuBERT excels in 5 out of 10 downstream tasks
compared to ARMHuBERT-S [13] and DPHuBERT [15]. Especially for both PR and
ASR, the performance of STaRHuBERT notably surpasses these approaches,
verifying that the temporal relation is crucial in learning speech
representations associated with acoustic units for lightweight SSL model. Our
student models also achieve the outstanding performance in the speaker-related
tasks, indicating that the TGMs can also capture speaker information by
distinguishing the temporal relations of different speakers. Taken together,
our proposed STaR distillation allows us to construct a more compact model
with strong performance and lightweight characteristics.

Our methodology also involves fewer MACs for inference compared to approaches
that conduct both pruning and distillation. STaRHuBERT exhibits a parameter
reduction of only 5.43% compared to DPHuBERT [15], but requires 29.1% fewer
MACs. The relatively high number of MACs observed in pruning approach is also
evident in the comparison between STaRHuBERT-L and LightHuBERT [18], wherein
the former requires 40.2% fewer MACs. This implies that pruning technique can
achieve good model performance and desired number of parameters by adjusting
the sparsity ratio, however, it is challenging to control the computational
cost of the pruned model. Hence, our approach, which is one of the vanilla
distillation approaches with a predefined model configuration, has a benefit
in terms of efficient and lightweight inference.

###  3.4 Examination on Universality

We confirm the universality of the STaR loss by replacing the teacher model
with wav2vec 2.0 Base [1] or wavLM Base [3]. Table 3 depicts the SUPERB
benchmark results for LibriSpeech [35] 960h distillation. STaRW2V2 offers
stronger representation overall with only 70.5% parameters of FitW2V2 [12].
STaRwavLM also outperforms previous methods [13, 16], even with the fewest
parameters. These student models corroborate that the proposed STaR
distillation is agnostic to other Transformer-based speech SSL models.

###  3.5 Compression for Smaller Parameter Sizes

To further support that our proposed STaR distillation is effective for
lightweight models, we compare the performance of the models compressed to
less than 20 million parameters [15, 19]. Fig. 2 compares ASR, SID, and IC
performances on the SUPERB benchmark using LibriSpeech [35] 960h distillation.
Our model performs best in all three tasks among the models with 9 million
parameters and also shows the least performance degradation as the number of
parameters decreased. Based on our findings, distillation of speech temporal
relation is preferable than directly matching the complex teacher
representation for lightweight student model with limited capacity.

Table 3:  Comparisons of the SUPERB benchmark results for different speech SSL
models, wav2vec 2.0 Base and wavLM Base.

Models |  Params (M) $\downarrow$ |  Overall $\uparrow$ |  General. $\uparrow$
---|---|---|---
wav2vec 2.0 Base [1] | 95.04 | 79.0 | 818
FitW2V2 [12] | 31.63 | 76.5 | 766
STaRW2V2 (ours) | 22.31 | 77.2 | 797
wavLM Base [3] | 94.70 | 81.9 | 1019
Structured Pruning [16] | 26.57 | 78.9 | 863
ARMwavLM-S [13] | 22.39 | 78.9 | 851
STaRwavLM (ours) | 22.31 | 79.4 | 899

(a) ASR

(b) SID

(c) IC

Fig. 2: Performance comparisons of models with fewer parameters.

##  4 Conclusion

In this study, we have proposed to distill speech temporal relation, STaR,
from the Transformer-based speech SSL models. We combine the distillation of
layer-wise and intra-layer TGMs as the STaR loss, which does not require any
additional parameters for distillation. STaRHuBERT-L achieves the SOTA overall
score of 79.8 among models with $\sim$ 27 million parameters, in particular
showing favorable results on PR, ASR, and speaker-related tasks. Further
experiments reveal that our method is agnostic to other speech SSL models and
is more suitable to lightweight students. To sum up, our results present the
effectiveness of distilling STaR rather than directly matching the output
representation for lightweight SSL models.

##  5 References

## References

  * [1] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli,  “wav2vec 2.0: A framework for self-supervised learning of speech representations,”  in Proc. NeurIPS, 2020, pp. 12449–12460.
  * [2] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed,  “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 3451–3460, 2021.
  * [3] Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al.,  “Wavlm: Large-scale self-supervised pre-training for full stack speech processing,”  IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1505–1518, 2022.
  * [4] Shu wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y. Lin, Andy T. Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, Tzu-Hsien Huang, Wei-Cheng Tseng, Ko tik Lee, Da-Rong Liu, Zili Huang, Shuyan Dong, Shang-Wen Li, Shinji Watanabe, Abdelrahman Mohamed, and Hung yi Lee,  “Superb: Speech processing universal performance benchmark,”  in Proc. Interspeech, 2021, pp. 1194–1198.
  * [5] Tzu-hsun Feng, Annie Dong, Ching-Feng Yeh, Shu-wen Yang, Tzu-Quan Lin, Jiatong Shi, Kai-Wei Chang, Zili Huang, Haibin Wu, Xuankai Chang, et al.,  “Superb@ slt 2022: Challenge on generalization and efficiency of self-supervised speech representation learning,”  in Proc. SLT, 2023, pp. 1096–1103.
  * [6] Cheng-I Jeff Lai, Yang Zhang, Alexander H Liu, Shiyu Chang, Yi-Lun Liao, Yung-Sung Chuang, Kaizhi Qian, Sameer Khurana, David Cox, and Jim Glass,  “Parp: Prune, adjust and re-prune for self-supervised speech recognition,”  in Proc. NeurIPS, 2021, pp. 21256–21272.
  * [7] Naigang Wang, Chi-Chun (Charlie) Liu, Swagath Venkataramani, Sanchari Sen, Chia-Yu Chen, Kaoutar El Maghraoui, Vijayalakshmi (Viji) Srinivasan, and Leland Chang,  “Deep compression of pre-trained transformer models,”  in Proc. NeurIPS, 2022, pp. 14140–14154.
  * [8] Euntae Choi, Youshin Lim, Byeong-Yeol Kim, Hyung Yong Kim, Hanbin Lee, Yunkyu Lim, Seung Woo Yu, and Sungjoo Yoo,  “Masked token similarity transfer for compressing transformer-based asr models,”  in Proc. ICASSP, 2023, pp. 1–5.
  * [9] Hyungjun Lim, Younggwan Kim, Kiho Yeom, Eunjoo Seo, Hoodong Lee, Stanley Jungkyu Choi, and Honglak Lee,  “Lightweight feature encoder for wake-up word detection based on self-supervised speech representation,”  in Proc. ICASSP, 2023, pp. 1–5.
  * [10] Jungwoo Heo, Chan yeong Lim, Ju ho Kim, Hyun seo Shin, and Ha-Jin Yu,  “One-Step Knowledge Distillation and Fine-Tuning in Using Large Pre-Trained Self-Supervised Learning Models for Speaker Verification,”  in Proc. Interspeech, 2023, pp. 5271–5275.
  * [11] Heng-Jui Chang, Shu-wen Yang, and Hung-yi Lee,  “Distilhubert: Speech representation learning by layer-wise distillation of hidden-unit bert,”  in Proc. ICASSP, 2022, pp. 7087–7091.
  * [12] Yeonghyeon Lee, Kangwook Jang, Jahyun Goo, Youngmoon Jung, and Hoirin Kim,  “Fithubert: Going thinner and deeper for knowledge distillation of speech self-supervised learning,”  in Proc. Interspeech, 2022, pp. 3588–3592.
  * [13] Kangwook Jang, Sungnyun Kim, Se-Young Yun, and Hoirin Kim,  “Recycle-and-Distill: Universal Compression Strategy for Transformer-based Speech SSL Models with Attention Map Reusing and Masking Distillation,”  in Proc. Interspeech, 2023, pp. 316–320.
  * [14] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin,  “Attention is all you need,”  in Proc. NeurIPS, 2017, pp. 6000–6010.
  * [15] Yifan Peng, Yui Sudo, Shakeel Muhammad, and Shinji Watanabe,  “Dphubert: Joint distillation and pruning of self-supervised speech models,”  in Proc. Interspeech, 2023, pp. 62–66.
  * [16] Haoyu Wang, Siyuan Wang, Wei-Qiang Zhang, Suo Hongbin, and Yulong Wan,  “Task-agnostic structured pruning of speech representation models,”  in Proc. Interspeech, 2023, pp. 231–235.
  * [17] Christos Louizos, Max Welling, and Diederik P Kingma,  “Learning sparse neural networks through l_0 regularization,”  in Proc. ICLR, 2018.
  * [18] Rui Wang, Qibing Bai, Junyi Ao, Long Zhou, Zhixiang Xiong, Zhihua Wei, Yu Zhang, Tom Ko, and Haizhou Li,  “Lighthubert: Lightweight and configurable speech representation learning with once-for-all hidden-unit bert,”  in Proc. Interspeech, 2022.
  * [19] Takanori Ashihara, Takafumi Moriya, Kohei Matsuura, and Tomohiro Tanaka,  “Deep versus wide: An analysis of student architectures for task-agnostic knowledge distillation of self-supervised speech models,”  in Proc. Interspeech, 2022, pp. 411–415.
  * [20] Alexei Baevski, Wei-Ning Hsu, Alexis Conneau, and Michael Auli,  “Unsupervised speech recognition,”  in Proc. NeurIPS, 2021, pp. 27826–27839.
  * [21] Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou,  “Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers,”  in Proc. NeurIPS, 2020, pp. 5776–5788.
  * [22] Zhiyuan Fang, Jianfeng Wang, Xiaowei Hu, Lijuan Wang, Yezhou Yang, and Zicheng Liu,  “Compressing visual-linguistic model via knowledge distillation,”  in Proc. ICCV, 2021, pp. 1428–1438.
  * [23] Leon Gatys, Alexander S Ecker, and Matthias Bethge,  “Texture synthesis using convolutional neural networks,”  in Proc. NeurIPS, 2015, pp. 262–270.
  * [24] Themos Stafylakis, Johan Rohdin, and Lukáš Burget,  “Speaker embeddings by modeling channel-wise correlations,”  in Proc. Interspeech, 2021, pp. 501–505.
  * [25] Yuki Saito, Shinnosuke Takamichi, and Hiroshi Saruwatari,  “Perceptual-similarity-aware deep speaker representation learning for multi-speaker generative modeling,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 1033–1048, 2021.
  * [26] Yanping Li, Dongxiang Xu, Yan Zhang, Yang Wang, and Binbin Chen,  “Non-parallel many-to-many voice conversion with psr-stargan.,”  in Proc. Interspeech, 2020, pp. 781–785.
  * [27] Sheng Shi, Jiahao Shao, Yifei Hao, Yangzhou Du, and Jianping Fan,  “U-gat-vc: Unsupervised generative attentional networks for non-parallel voice conversion,”  in Proc. ICASSP, 2022, pp. 7017–7021.
  * [28] Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola,  “A kernel two-sample test,”  The Journal of Machine Learning Research, vol. 13, pp. 723–773, 2012.
  * [29] Yanghao Li, Naiyan Wang, Jiaying Liu, and Xiaodi Hou,  “Demystifying neural style transfer,”  in Proc. IJCAI, 2017, pp. 2230–2236.
  * [30] Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim,  “A gift from knowledge distillation: Fast optimization, network minimization and transfer learning,”  in Proc. CVPR, 2017, pp. 4133–4141.
  * [31] ZhaoJing Zhou, Yun Zhou, Zhuqing Jiang, Aidong Men, and Haiying Wang,  “An efficient method for model pruning using knowledge distillation with few samples,”  in Proc. ICASSP, 2022, pp. 2515–2519.
  * [32] Baoyun Peng, Xiao Jin, Jiaheng Liu, Dongsheng Li, Yichao Wu, Yu Liu, Shunfeng Zhou, and Zhaoning Zhang,  “Correlation congruence for knowledge distillation,”  in Proc. ICCV, 2019, pp. 5007–5016.
  * [33] Yufeng Jin, Guosheng Hu, Haonan Chen, Duoqian Miao, Liang Hu, and Cairong Zhao,  “Cross-modal distillation for speaker recognition,”  in Proc. AAAI, 2023, pp. 12977–12985.
  * [34] Ankita Pasad, Bowen Shi, and Karen Livescu,  “Comparative layer-wise analysis of self-supervised speech models,”  in Proc. ICASSP, 2023, pp. 1–5.
  * [35] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur,  “Librispeech: an asr corpus based on public domain audio books,”  in Proc. ICASSP, 2015, pp. 5206–5210.