# Laser induced Flqouet topological phases in a 2D weak topological insulator
with $\boldsymbol{\it unconventional}$ nonlinear dispersion

Aayushi Agrawal<EMAIL_ADDRESS>Department of Physics,
Birla Institute of Technology and Science, Pilani 333031, India  Jayendra N.
Bandyopadhyay<EMAIL_ADDRESS>Department of Physics, Birla Institute of
Technology and Science, Pilani 333031, India

###### Abstract

Recently, we presented a two-dimensional (2D) model of a weak topological
insulator formed by stacking an $N$ number of Su-Schrieffer-Heeger (SSH)
chains [1]. We now study the influence of periodic driving on the topological
properties of this system, which has all the fundamental symmetries, by
shining it with circularly polarized light (CPL). The CPL is chosen because it
breaks the time-reversal symmetry, which induces more exotic topological
properties in the system. We investigate two different formations of the $N$
stacked SSH chains: all the SSH chains are topologically trivial in one
formation and nontrivial in the other one. In contrast to the undriven or
static case, both formations exhibit distinct topological behaviors. Here, we
particularly derive the Floquet or the effective Hamiltonian using the replica
method, which facilitates the study of high- and low-frequency regimes. We
have discovered that this model exhibits laser-induced Floquet topological
phases with higher Chern numbers. This system has nonlinear dispersion along
both directions with additional $k_{x}-k_{y}$ coupling terms, which made the
dispersion of this system unconventional. We closely study the role of this
unconventional dispersion in the system at the low-energy limit and its
response to periodic driving. The low-energy Hamiltonian also reveals a
hierarchy in the gaps of the neighboring Floquet bands. Interestingly, though
this model has nonlinear quasi-energy dispersion, it still shows some
signatures of hierarchy, which was observed in the system with linear
dispersion like graphene. Furthermore, we study the effect of linearly
polarized light (LPL) on the topological properties of the system. In response
to the LPL driving, the band-touching point either opens up or splits into two
band-touching points.

##  I Introduction

Floquet engineering is becoming a vast area of research, where a desired
solid-state system is synthesized by periodic driving. A primary goal of this
research area is to design a periodic driving protocol to quickly introduce or
enhance desired exotic properties in a given system in a very controlled way
[2]. In the realm of topological insulators (TIs), periodically driven TIs or
Floquet TIs have been studied extensively. Periodic driving introduces many
exciting features in the TIs, which may not be possible to realize by any
static means [3, 4, 5, 6, 7, 8, 9, 10]. These periodically driven systems can
be realized experimentally by ultracold atoms in optical lattices [11, 12].

The quasi-nD systems, engineered by stacking numerous nD systems, came into
light after the discovery of weak topological insulators (WTIs). These
topological materials were discovered in quasi-2D quantum spin Hall systems
(QSH) in which 2D QSH layers were stacked and formed a 3D layered structure
[13, 14]. A few studies concentrated on quasi-2D systems, such as the 3D
layered structure of graphene and topological crystalline insulators (TCI)
[15, 16, 17, 18, 19, 20, 21, 22, 23]. The recent literature focuses primarily
on 2D and some quasi-2D materials. Very little attention is paid to the
quasi-1D systems formed by a systematic stacking of identical 1D systems.
Recently, present authors rigorously study an $N$ number of stacked SSH chains
[1]. Similarly, some studies have investigated the topological properties of
another quasi-1D model, an extended Su-Schrieffer-Heeger model (E-SSH) [24,
25]. The E-SSH model is a 1D SSH chain where all the hopping amplitudes are
modulated by a cyclic parameter $\theta$, and this cyclic parameter is
considered as another synthetic dimension. The Floquet version of the SSH
chain, the simplest 1D TI, displays interesting properties [26, 27, 28, 29].
An intriguing feature of the static E-SSH model is that the system’s phase
diagram resembles the Haldane model’s phase diagram. The interlink between the
E-SSH model and the $N$ stacked SSH model is that by promoting $\theta$ to an
actual dimension, one can obtain a $N$ stacked SSH model. The E-SSH model is
also studied with different periodic driving schemes such as Dirac-delta
kickings, and sinusoidal driving [25, 28].

Through Floquet, the quasi-2D system of $N$ stacked graphene layers is also
explored where circularly polarized light (CPL) is used as an external
periodic drive and topological phases with the high Chern number ($C$) are
observed [16]. The CPL is widely recognized for breaking the time-reversal
symmetry [30, 31, 32, 33, 34, 35, 36, 37]. As a result, it creates a new gap
at the band touching points, and the system harbors the quantum Hall effect
without using any external magnetic field or creating Landau levels [38].
Furthermore, the CPL can be used to detect optical chirality, thermoelectric
transport, photo-voltage, the dynamical Hall effect, and as a probe for high
harmonic generation [39, 40, 41, 42, 43].

This work aims to induce new Floquet topological phases through CPL in the $N$
stacked SSH chains model. As per our knowledge, the periodically driven 2D WTI
is not studied in the literature. Here, we study a 2D WTI under periodic
driving. This undriven $N$ stacked SSH chains follows all three fundamental
symmetries (chiral, particle-hole, and time-reversal), which host nontrivial
topology but with Chern number $C=0$. In the undriven case, two possible
constructions were addressed: $N$ stacked SSH model where each SSH chain is
topologically trivial (winding number, $w=0$), and each SSH chain is
nontrivial ($w=1$) [1]. These two cases exhibit different topological phases
as we turn on the driving. This model has an interesting feature because there
exists a coupling term between the momenta $k_{x}$ and $k_{y}$, which makes
the system complex and distinct from various other well-studied 2D systems
[44, 45]. At the low energy limit, near the band touching points, the
dispersion relation of this system becomes quadratic with a $k_{x}k_{y}$ like
coupling term.

In various studies, it has been observed that, in the case of linear
dispersion, the band crossing points or Dirac points are stable against any
small perturbation [46, 38, 47, 48]. However, the quadratic dispersion near
the band crossing points or semi-Dirac points is unstable for the small
perturbation: either a gap is opened or splits into two Dirac points [49, 35].
The emergence of the semi-Dirac point requires linear dispersion in one
direction and quadratic dispersion in the other. However, the dispersion
relation of the $N$ stacked SSH chains model is unconventional with nonlinear
dispersion along both directions with additional $k_{x}-k_{y}$ coupling terms.
This unconventional dispersion relation encourages us to investigate the
response of the $N$ stacked SSH chains model under the influence of linearly
polarized light (LPL) along the $x-$ and $y-$ directions.

This paper is organized as follows: In Sec. II, we briefly discuss the static
Hamiltonian. In the next section, Sec. III, we discuss the Floquet formalism
and the periodically driven Hamiltonian, which uses the exact Floquet replica
method. In the next section, Sec. IV, we calculate the low-energy Hamiltonian
to show a signature of hierarchy in Floquet band gaps. In Sec. V, we
demonstrate the role of linearly polarized light on the $N$ stacked SSH model.
Finally, we summarize in Sec. VI.

##  II Static Hamiltonian

We study the effect of periodic driving on a $N$ stacked SSH chain model,
which is composed of an $N$ number of stacked SSH chains [1]. In real space,
the mathematical expression for this static system is given as

$\begin{split}H_{N-SSH}&=(1-\eta)\sum_{n_{x},n_{y}}c_{n_{x},n_{y}}^{\dagger
A}c_{n_{x},n_{y}}^{B}+(1+\eta)\sum_{n_{x},n_{y}}c_{n_{x}+1,n_{y}}^{\dagger
A}c_{n_{x},n_{y}}^{B}\\\
&+\frac{\delta}{2}\,\sum_{n_{x},n_{y}}\bigg{[}c_{n_{x},n_{y}}^{\dagger
A}c_{n_{x},n_{y}+1}^{B}+c_{n_{x},n_{y}+1}^{\dagger
A}c_{n_{x},n_{y}}^{B}\bigg{]}\\\
&-\frac{\delta}{2}\,\sum_{n_{x},n_{y}}\bigg{[}c_{n_{x}+1,n_{y}}^{\dagger
A}c_{n_{x},n_{y}+1}^{B}+c_{n_{x}+1,n_{y}+1}^{\dagger
A}c_{n_{x},n_{y}}^{B}\bigg{]}+h.c.\end{split}$ (1)

Here, $\delta$ is the hopping amplitude between the inter-sublattices of
neighboring SSH chains, and $\eta$ is the dimerization constant of the
individual SSH chain. The parameter $\eta$ can be negative or positive values,
which results in a trivial or nontrivial SSH chain. To illustrate the energy
spectrum and topological properties, we write the Hamiltonian in the
quasimomentum space (or $\boldsymbol{k}$-space), which is given as

$H_{N-SSH}(\boldsymbol{k})=\boldsymbol{h}\boldsymbol{\cdot}\boldsymbol{\sigma}$
(2) $\displaystyle h_{x}(\textbf{k})=$ $\displaystyle[(1+\cos k_{x})+(1-\cos
k_{x})\,(\delta\cos k_{y}-\eta)]$ $\displaystyle h_{y}(\textbf{k})=$
$\displaystyle[(1+\eta)-\delta\cos k_{y}]\,\sin k_{x}$

In our previous study [1], we found that breaking chiral and time-reversal
symmetries is essential for this static system to be a Chern insulator (CI).
However, this static system follows all three fundamental symmetries and shows
topological properties even when the Chern number $C=0$. For this case, its
topological property is determined by nonzero 2D Zak phase $Z(k_{y})=-\pi$.
This study also revealed that the topological properties of this system are
not dependent on the topological properties of the individual SSH chain. We
now discuss this model under the periodic driving with CPL. It is well-known
that the CPL breaks the chiral and the TR symmetry in the system, leading to
nontrivial topological properties in the system with nonzero Chern numbers.

##  III Driven Hamiltonian

We now apply a laser field whose vector potential $\boldsymbol{A}(t)$ has the
form

$\boldsymbol{A}(t)=(A_{0x}\cos\Omega t,A_{0y}\sin\Omega t)$

with it satisfies $\boldsymbol{A}(t+T)=\boldsymbol{A}(t)$, where $T$ is the
time-period of the driving, and consequently the driving frequency
$\Omega=2\pi/T$. Here, $A_{0x}$ and $A_{0y}$ are the components of the vector
potential along $x$ and $y$ direction, respectively. If we set
$A_{0x}=A_{0y}=A_{0}$, the laser field will be the CPL. The LPL is a special
case of this laser field, when it has a form either
$\boldsymbol{A}(t)=(A_{0x}\cos\Omega t,0)$ or
$\boldsymbol{A}(t)=(0,A_{0y}\cos\Omega t)$. This driving is induced in the
system by the Peierls substitution, which modifies the form of the quasi-
momenta $k_{x}$ and $k_{y}$ as

$k_{x}(t)\rightarrow k_{x}+A_{x}(t)\,;\,k_{y}(t)\rightarrow k_{y}+A_{y}(t)$

The form of the time-periodic Hamiltonian in the $\boldsymbol{k}$-space reads

$\begin{split}H_{N-SSH}(\textbf{k},t)=&[(1+\cos k_{x}(t))+(1-\cos k_{x}(t))\\\
&\times(\delta\cos k_{y}(t)-\eta)]\sigma_{x}\\\ &+[(1+\eta)-\delta\cos
k_{y}(t)]\,\sin k_{x}(t)\,\sigma_{y}\end{split}$ (3)

In order to solve the time-periodic equation, we use the Floquet replica
method and calculate the effective Hamiltonian, an infinite dimensional matrix
in the frequency space.

###  III.1 Floquet theory

Periodically driven systems are studied under the Floquet formalism.
Therefore, these systems are also known as Floquet systems [50, 51]. Using
this theorem, we solve the time-periodic Schrödinger equation [51, 52]

$i\hbar\,\frac{d}{dt}|\psi(t)\rangle=H(t)\,|\psi(t)\rangle,$ (4)

where $H(t)=H(t+T)$. The Floquet theorem is the temporal version of the well-
known Bloch’s theorem of solid-state physics. Therefore, according to this
theorem, the solution of the time-periodic Schrödinger equation can be written
as

$|\psi_{n}(t)\rangle=e^{-i\epsilon_{n}t}\,|u_{n}(t)\rangle,$ (5)

where $n$ presents the Floquet band index, and $\epsilon_{n}$ is corresponding
quasienergy. The states $|u_{n}(t)\rangle$ are called Floquet modes, which are
periodic in time with the same period as the Hamiltonian, i.e.,
$|u_{n}(t+T)\rangle=|u_{n}(t)\rangle$. The Floquet states are the eigenstates
of the single period time-evolution operator, and therefore

$\hat{U}(t_{0}+T,t_{0})|\psi_{n}(t_{0})\rangle=e^{-i\epsilon_{n}T}|\psi_{n}(t_{0})\rangle.$
(6)

Solving the above eigenvalue problem, one can obtain the Floquet states and
the corresponding quasienergies. However, there is an alternative way to
calculate the Floquet states and quasienergies by substituting these solutions
in the time-periodic Schrödinger equation, which eventually takes the
following form

$\left[H(t)-i\frac{\partial}{\partial
t}\right]\,|u_{n}(t)\rangle=\epsilon_{n}\,|u_{n}(t)\rangle.$ (7)

Due to the time periodicity in $H(t)$ and $u_{n}(t)$, we can expand these in
Fourier series as

$H(t)=\sum_{m}\,e^{-im\Omega
t}\,H^{(m)}\,;~{}~{}\,|u_{n}(t)\rangle=\sum_{m}\,e^{-im\Omega
t}\,|u_{n}^{(m)}\rangle,$

where $m=0,\pm 1,\pm 2,\dots$. The bands corresponding to $m=0$ are the
central Floquet bands, whereas those with nonzero $m$ form side bands. The
central Floquet bands lie in the quasienergy range
$-\frac{\Omega}{2}\leq\epsilon\leq\frac{\Omega}{2}$, and this is known as the
‘first Floquet-Brillouin zone’ (FBZ). The Fourier component $H^{(m)}$ is
obtained as

$H^{(m)}=\frac{1}{T}\int_{0}^{T}H(\boldsymbol{k},t)e^{-im\Omega t}dt.$ (8)

In the Fourier space, the time-periodic Schrödinger equation given in Eq. (7)
can be written as

$\epsilon_{n}|u_{n}^{(m)}\rangle=\sum_{m}\left[H^{m-m^{\prime}}-m\Omega\delta_{mm^{\prime}}\right]|u_{n}^{(m^{\prime})}\rangle$
(9)

The above eigenvalue equation corresponds to an infinite dimensional effective
Hamiltonian, which is defined in the extended Hilbert space
$\mathcal{H}\otimes\mathcal{T}$, where $\mathcal{H}$ is the standard Hilbert
space and $\mathcal{T}$ is the Hilbert space which spans all the time-periodic
functions $\bigl{\\{}e^{-im\Omega t}\bigr{\\}}$ [53]. This infinite-
dimensional matrix is formed by an infinite number of duplicate copies of
undriven energy bands affected by external driving. Therefore, this is known
as the “Floquet replica method”, where each copy corresponds to a photon
sector. We obtain quasienergies and the corresponding Floquet modes by
diagonalizing the effective Hamiltonian. In numerical calculation, we have to
truncate this infinite dimensional matrix and consider only a finite number of
photon sectors, where the strength of the driving frequency $\Omega$ decides
the number of photon sectors. In the case of the higher frequencies, we need
to consider a number of photon sectors for the numerical convergence. On the
other hand, for lower frequencies, many photon sectors are to be included in
the computation to achieve the desired convergence. The solution of this
quasienergy problem is analogous to the dressed atom picture of the laser-atom
interaction. Hence, the matrix element or the Fourier component $H^{(m)}$
demonstrates the $m$-photon process [54].

###  III.2 Periodically driven ${\boldsymbol{N}}$ stacked SSH model: Floquet
formalism

First, we write the Fourier components $H^{(m)}$ for the $N$-stacked SSH model
in the form as

$H^{(m)}=\boldsymbol{d}^{(m)}\boldsymbol{\cdot}\boldsymbol{\sigma}.$

The driving modifies the undriven part as follows

$H^{(0)}_{N-SSH}=\,\boldsymbol{d}^{(0)}_{N-SSH}\cdot\boldsymbol{\sigma}$ (10a)
where
$\begin{split}\left(d^{(0)}_{N-SSH}\right)_{x}&=\,(1-\eta)+(1+\eta)\,\cos
k_{x}\,J_{0}(A_{0})+\delta\,\cos k_{y}\,J_{0}(A_{0})\\\ &-\delta\,\cos
k_{x}\,\cos k_{y}\,J_{0}^{2}(A_{0})\end{split}$ (10b)
$\left(d^{(0)}_{N-SSH}\right)_{y}=\,(1+\eta)\,\sin
k_{x}\,J_{0}(A_{0})-\delta\,\sin k_{x}\,\cos k_{y}\,J_{0}^{2}(A_{0});$ (10c)

where $J_{0}$ is the Bessel function of the first kind with zeroth order.

In order to calculate the other non-zero Fourier components, we choose the
driving amplitude $A_{0}$ such that the Bessel functions contribute only up to
an order of $A_{0}^{2}$. The higher order Fourier components are neglected
because we assume that the amplitude $A_{0}$ is small. Thus, the effective
Hamiltonian $H_{\rm eff}$ have only $H^{(1)}$ and $H^{(2)}$ Fourier
components, which are given as

$H^{(1)}_{N-SSH}=\,\boldsymbol{d}^{(1)}_{N-SSH}\cdot\boldsymbol{\sigma}$ (11a)
where $\begin{split}\left(d^{(1)}_{N-SSH}\right)_{x}=&\,-(1+\eta)\,\sin
k_{x}\,J_{1}(A_{0})+i\,\delta\sin k_{y}\,J_{1}(A_{0})\\\ &-\delta\left\\{-\sin
k_{x}\,\cos k_{y}+i\,\cos k_{x}\,\sin
k_{y}\right\\}J_{0}(A_{0})\,J_{1}(A_{0})\end{split}$ (11b)
$\begin{split}\left(d^{(1)}_{N-SSH}\right)_{y}=&\,(1+\eta)\,\cos
k_{x}\,J_{1}(A_{0})\\\ &-\delta\left\\{\cos k_{x}\,\cos k_{y}+i\,\sin
k_{x}\,\sin k_{y}\right\\}J_{0}(A_{0})\,J_{1}(A_{0});\end{split}$ (11c)

and

$H^{(2)}_{N-SSH}=\,\boldsymbol{d}^{(2)}_{N-SSH}\cdot\boldsymbol{\sigma}$ (12a)
where $\begin{split}\left(d^{(2)}_{N-SSH}\right)_{x}=&\,-(1+\eta)\,\cos
k_{x}\,J_{2}(A_{0})+\delta\,\cos k_{y}\,J_{2}(A_{0})\\\ &+i\,\delta\sin
k_{x}\,\sin k_{y}J_{1}^{2}(A_{0})\end{split}$ (12b)
$\left(d^{(2)}_{N-SSH}\right)_{y}=\,-(1+\eta)\,\sin
k_{x}\,J_{2}(A_{0})-i\,\delta\,\cos k_{x}\,\sin k_{y}J_{1}^{2}(A_{0}).$ (12c)

Here, $J_{1}$ and $J_{2}$ are the Bessel functions of the first kind.

Figure 1: Quasienergy bands for the periodically driven $N$ stacked SSH chains
are shown in cylindrical geometry. In subfigures (a) and (c), all the SSH
chains are considered trivial ($\eta=-0.5$), while in subfigures (b) and (d),
all the SSH chains are nontrivial ($\eta=0.5$). Here, when the quasienergy
bands are shown as a function of $k_{x}$ ($k_{y}$), then this suggests that
the PBC is considered along $x$-direction ($y$-direction), and the OBC is
considered along the other direction. In both cases, the driving generates new
Floquet topological phases with the Chern number $C=-1$, and the appearance of
the edge states is its consequence. We set the parameter values for both plots
as $\delta=1.0$, $A_{0}=0.5$, and $\Omega=6.0$.

Floquet energy bands corresponding to the Hamiltonian given in Eq. (3) are
shown in Fig 1. In subfigure 1(a) and 1(c), we consider each SSH chain is
topologically trivial ($\eta=-0.5$). Subsequently, in subfigure 1(b) and 1(d),
we consider individual SSH chain as nontrivial ($\eta=0.5$). The Floquet bands
are shown for cylindrical geometry, where its axis is along $x$-direction or
$y$-direction. Here and throughout the paper, the axis of the cylinder along
$x$-direction (or $y$-direction) means periodic boundary condition (PBC) is
considered along $y$-direction (or $x$-direction), and open boundary condition
(OBC) is assumed along the $x$-direction (or $y$-direction). These Floquet
bands are presented in the high-frequency regime with $\Omega=6.0$ and the
driving amplitude $A_{0}=0.5$. In both cases, we obtain nontrivial topology
with $C=-1$, which was topologically trivial with $C=0$ in the undriven case
[1]. With the application of the CPL, the $N$ stacked SSH model is transformed
into a Chern insulator. In the undriven case, we found that the topological
properties remained unchanged irrespective of whether underlying SSH chains
are topologically trivial or nontrivial. We observe qualitatively similar
characteristics for this specific driving amplitude and frequency in both
cases. To illustrate the complete behavior of the topological properties in
driving parameter space, we present a phase diagram in the next subsection.

###  III.3 Phase diagram

Figure 2: Different topological phases with different Chern numbers $C$ are
shown as the function of driving frequency $\Omega$ and driving amplitude
$A_{0}$. In this case, we set $\eta=-0.5$; hence, the individual SSH chain is
topologically trivial. Here, we consider $9$ photon sectors in the Hamiltonian
to get the desired convergence. In subfigure (b), the Floquet bands are
plotted in FBZ along the high symmetric path. Here, $\Gamma$ is $(\pi,\pi)$,
$K$ is $(2\pi,\pi)$ and $M$ is $(2\pi,2\pi)$. Figure 3: The results presented
here is similar to Fig. 2, but here we consider individual SSH chain as
topologically nontrivial.

In this section, we demonstrate two different phase diagrams depending on the
topological property of the individual SSH chain. The phase diagrams are
plotted in the driving parameter space of amplitude $(A_{0})$ and frequency
$(\Omega)$ to illustrate various topological phases. The result for the case
when the individual SSH chain is topologically trivial is presented in Fig.
2(a), whereas Fig. 3(a) shows the result when the individual SSH chains are
topologically nontrivial. In these phase diagrams, we present the sum of the
Chern number of all the Floquet bands below $\epsilon=0$ as a function of the
driving frequency $\Omega$ for four different values of driving amplitudes
$A_{0}=0.1,0.3,0.5\,\rm{and}\,0.7$. As stated in Ref. [55], in the frequency
domain, one can calculate the number of chiral edge states in a particular gap
by summing the Chern number of all the Floquet bands below that gap.

We consider $9$ photon sectors for all the calculations to achieve the
convergence. When the individual SSH chain is topologically trivial, we
observe five topological phases with $C=0,\pm 1,-2,$ and $-3$. In the low-
frequency regime ($\Omega<4.0$), we obtain topological phases with the high
Chern number, while in the high-frequency regime ($\Omega>4.0$), the system
exhibits topological phases with $C=\pm 1$. In Fig. 3, where the phase diagram
is plotted for the nontrivial case, we observe four different topological
phases with Chern number $C=0,-1,-2,$ and $-4$. For this case, the system
saturates at the topological phase with $C=-1$ in the high-frequency regime.
It is important to note that, for the different values of the driving
amplitude $A_{0}\in[0.1,0.7]$, the variation in the Chern number with the
driving frequency $\Omega$ is almost similar. The value of $\Omega$ determines
the variation in the Chern number. Nevertheless, both the phase diagrams show
that, compared to the undriven case, the Floquet version of the $N$ stacked
SSH chains displays much richer topological phases with high Chern numbers.

It is well known that the bulk boundary correspondence in the Floquet system
is not the same as the undriven cases. In the case of the undriven systems,
the edge states appear only in the energy gaps between the bulk bands.
However, in the Floquet version of these systems, we have infinite copies of
the undriven systems. The driving affects not only the band gap between the
original bands but also different copies or replicas. Consequently, edge
states can also appear in between Floquet replica bands. Here, we consider the
Floquet bands or quasi-energy bands only in the first FBZ, where the edge
states can exist at the central gap $\frac{\epsilon}{\Omega}=0$; and as well
as at the boundary of the FBZ, i.e., at $\frac{\epsilon}{\Omega}=0.5$. As a
consequence, the total number of chiral edge states is calculated from the
relation $C=C_{0}-C_{\pi}$, where $C_{0}$ and $C_{\pi}$ respectively measure
the number of chiral edge states in the central gap and the gap around the
Floquet zone boundary. For the frequency regime $2.0\leq\Omega\leq 6.0$, we
observe $C_{\pi}=0$ at the Floquet zone boundary. Therefore, the Chern number
is always $C=C_{0}$ in this frequency regime.

The topological phase transitions observed in the phase diagrams occur in the
system because of the closing and re-opening of some of the band gaps.
Therefore, in Figs. 2(b) and 3(b), we show the bands in the first FBZ along
the high symmetric path. Earlier, we mentioned that the topological phases are
almost independent of $A_{0}$; hence, we plot these band diagrams only for
$A_{0}=0.1$. The phase transition occurs due to the band gap closing at
$\epsilon/\Omega=0$ or at the Floquet zone boundary $\epsilon/\Omega=\pm 0.5$.
Since the maximum band gap in the undriven system is $4.0$, the band gap
closing occurs at $\epsilon/\Omega=\pm 0.5$ when $\Omega=4.0$.

###  III.4 Demonstration of the edge states of the Floquet topological phases
with high Chern number

This section discusses the edge states observed in the energy band diagrams
for the higher Chern numbers with $|C|>1$. In Fig. 2, we observed topological
phases with the Chern number $C=-2$ and $-3$. Here, we select two pairs of the
driving amplitudes and frequencies $(A_{0}=0.5,\Omega=2.5)$ and
$(A_{0}=0.5,\Omega=3.5$) from the phase diagram, where the corresponding Chern
numbers are $C=-2$ and $-3$, respectively. We choose $A_{0}=0.5$ so that the
band gap is prominent and the edge states can be clearly visible. In Fig.
4(a)-(d), we have shown the energy band diagram for the case of cylindrical
geometry. In this figure, we consider individual SSH chains to be
topologically trivial. In Fig. 5(a)-(d), the energy band diagrams for
cylindrical geometry are presented, where the individual SSH chain is
nontrivial. These figures show the edge states for the topological phases with
$C=-4$ and $C=-2$. The band diagrams (Figs. 4 and 5) exhibit edge states along
both directions; thus, these show an actual 2D-like system even though the
system is constructed as a weak topological insulator by stacking many SSH
chains.

Figure 4: The quasienergy bands for cylindrical geometry are shown for the
Chern numbers $C=-2$ and $C=-3$ corresponding to the phase diagram Fig. 2. In
subfigures (a) and (c), the bands are plotted for the driving amplitude
$A_{0}=0.5$ and the driving frequency $\Omega=2.5$. In subfigures (b) and (d),
the bands are plotted for $A_{0}=0.5$ and $\Omega=3.5$. In both cases, the
individual SSH chain is considered topologically trivial.

##  IV Low-energy Hamiltonian

Figure 5: The quasienergy bands for cylindrical geometry are shown for the
Chern numbers $C=-4$ and $C=-2$ corresponding to the phase diagram presented
in Fig. 3. In subfigures (a) and (c), the bands are shown for the driving
amplitude $A_{0}=0.5$ and the driving frequency $\Omega=2.5$. In subfigures
(b) and (d), the bands are presented for $A_{0}=0.5$ and $\Omega=2.9$. In both
cases, the individual SSH chain is considered as topologically nontrivial.
Figure 6: The quasienergy bands repulsion in the case of the low-energy
Hamiltonian is shown for the cylindrical geometry. In subfigure (a), the
cylinder’s axis is considered along the $y$-direction. The boundary condition
is interchanged in subfigure (b), i.e., the PBC is considered along the
$x$-direction, and the OBC is considered along the $y$-direction. Here, the
individual SSH chain is considered as nontrivial. The colored dashed lines
show the quasienergy bands for the undriven system, whereas the black solid
lines are used for the driven case. We set the driving parameters at
$\Omega=0.3$ and $A_{0}=0.05$. Figure 7: The results presented here are
similar to Fig. 6, but here the individual SSH chain is topologically trivial.

The low-energy Hamiltonian near the band touching point
$\boldsymbol{K}_{D}=\left[\pi,\cos^{-1}\frac{\eta}{\delta}\right]$, can be
derived by substituting $\boldsymbol{k}=\boldsymbol{q}+\boldsymbol{K}_{D}$ in
Eq. (2). Here, $|\boldsymbol{q}|\ll 1$ and under this condition the
Hamiltonian will be

$\begin{split}h_{x}^{(0)}&=\frac{q_{x}^{2}}{2}-\eta
q_{y}^{2}-2q_{y}\sqrt{\delta^{2}-\eta^{2}}\\\
h_{y}^{(0)}&=-q_{x}-q_{x}q_{y}\sqrt{\delta^{2}-\eta^{2}}\end{split}$ (13)

For the driven case, we obtain Fourier components of the low energy
Hamiltonian by substituting
$\boldsymbol{q}\rightarrow\boldsymbol{q}+\boldsymbol{A}(t)$

$\begin{split}h_{x}^{(1)}&=q_{x}\frac{A_{0}}{2}+i\eta
q_{y}A_{0}+iA_{0}\sqrt{\delta^{2}-\eta^{2}}\\\
h_{y}^{(1)}&=-\frac{A_{0}}{2}+iq_{x}\frac{A_{0}}{2}+q_{y}\frac{A_{0}}{2}\sqrt{\delta^{2}-\eta^{2}}\\\
h_{x}^{(2)}&=\frac{A_{0}^{2}}{8}+\eta\frac{A_{0}^{2}}{4}\\\
h_{y}^{(2)}&=i\frac{A_{0}^{2}}{4}\sqrt{\delta^{2}-\eta^{2}}\end{split}$ (14)

Here, we see that, unlike graphene, the dispersion relation of the low-energy
Hamiltonian of the $N$-stacked SSH model is still unconventional, having
quadratic dispersion with asymmetry along both $k_{x}$ and $k_{y}$ directions
due to the presence of a coupling term. The unconventional dispersion makes
the system more complex than any other 2D system. We now investigate in detail
the behavior of the low-energy Hamiltonian.

The effect of the asymmetry in the Hamiltonian can be seen in Figs. 6 and 7,
where we have projected the Floquet bands along one of the directions of the
quasi momenta and set the value of the other quasi momentum equals zero. In
this figure, the dotted lines represent energy bands of the undriven system,
where different colors denote different photon sectors. The black solid lines
are used for the driven case. As we turn on the driving, the Floquet bands
corresponding to different photon sectors repel each other and create band
gaps where Floquet edge states can appear. Our primary goal is to investigate
whether the Floquet bands of the $N$ stacked model have the same hierarchical
structure as observed in graphene [37]. Even though our system is very
different from graphene, we observe some hierarchical structure in the Floquet
band gaps at $\epsilon=0$, and also at $\epsilon=\pm\frac{\Omega}{2}$. We
observe that the behavior of the level repulsion around the central gap and
the Floquet zone boundary is qualitatively similar to graphene. Here, the
width of the Floquet gaps is approximately of the order of
$\left(\frac{A_{0}}{\Omega}\right)^{\Delta m}$, where $\Delta m$ is the
difference between the photon sectors.

In Figs. 6 and 7, we show by red and blue dotted lines that the largest
Floquet band gap occurs at $\epsilon=\pm\frac{\Omega}{2}$, due to the
repulsion between the bands with photon sectors $m=\pm 1$ and $m=0$. However,
the width of the Floquet band gap at $\epsilon=0$ between the bands in the
zero photon sector is of the order of $\left(\frac{A_{0}}{\Omega}\right)^{2}$
[37].

##  V  ${\boldsymbol{N}}$ stacked SSH chains model under linearly polarized
light

Figure 8: The quasienergy bands with a semi-Dirac point are shown for the
undriven case with $\delta=|\eta|$. In subfigures (a) and (c), we set
$\delta=0.5$ and $\eta=-0.5$, and hence the individual SSH chain is
topologically trivial. On the other hand, in subfigures (b) and (d), the
individual SSH chain is made topologically nontrivial by setting $\delta=0.5$
and $\eta=0.5$.

Earlier, we have shown that the $N$ stacked SSH model has non-linear
dispersion. Therefore, we choose the system parameters so that this model also
shows a semi-Dirac point, i.e., a band touching point where the dispersion
along one direction is linear and quadratic along the other. A similar semi-
Dirac point is observed in the hexagonal lattice. The semi-Dirac point of this
system is split into two Dirac points if the system is shined by an LPL [35].
For our study, we consider two cases of the LPL: along $x$ and $y$-directions.
Our goal is to observe whether the semi-Dirac point of the $N$ stacked SSH
model also splits into two Dirac points. Here, we set $\delta=|\eta|$ in Eq.
(2). When $N$ stacked SSH chain is constructed with trivial SSH chains, we
observe a semi-Dirac point at $\left[\pi,\pi\right]$ as shown in Figs. 8(a)
and 8(c). For the other case, when $N$ stacked SSH chain is constructed with
nontrivial SSH chains, we find the semi-Dirac point at $\left[\pi,0\right]$
(or at $\left[\pi,2\pi\right]$) as shown in Figs. 8(b) and 8(d). The emergence
of the semi-Dirac behavior can be identified by deriving the low-energy
Hamiltonian around these band touching points as follows:

$h_{x}^{(0)}=\frac{q_{x}^{2}}{2}-\eta q_{y}^{2};~{}h_{y}^{(0)}=-q_{x}$ (15)

The above low energy Hamiltonian of the $N$ stacked SSH model shows semi-
Dirac-like behavior for the condition $\left[\delta=|\eta|\right]$. We now
separately study the role of LPL along $x$ and $y$-directions.

Figure 9: The quasienergy bands are presented when the LPL is applied along
the $x$-direction for the case when the individual SSH chain is topologically
nontrivial. We fix the driving parameters at $A_{0}=0.5$ and $\delta=\eta=0.5$
in all the subfigures. In subfigures (a) and (c), we set $\Omega=3.0$, while
in subfigures (b) and (d), we set $\Omega=6.0$. Here, we observe the emergence
of the semi-Dirac-like point as LPL is applied along $x$-direction. Figure 10:
This figure is similar to Fig. 9, but here we consider stronger driving
amplitude $A_{0}=1.0$.

###  V.1 Linearly polarized light along ${\boldsymbol{x}}$-direction

First, we consider the case when the LPL is applied along $x$-direction of the
form $A(t)=A_{0}\cos\Omega t$. For this driving, we find the Fourier component
of the Hamiltonian for $m=0$ as:

$H^{(0)}_{N-SSH}=\,\boldsymbol{d}^{(0)}_{N-SSH}\cdot\boldsymbol{\sigma}$ (16a)
where
$\begin{split}\left(d^{(0)}_{N-SSH}\right)_{x}=&\,(1-\eta)+(1+\eta)\,\cos
k_{x}\,J_{0}(A_{0})+\delta\,\cos k_{y},\\\ &-\delta\,\cos k_{x}\,\cos
k_{y}\,J_{0}(A_{0})\end{split}$ (16b)
$\left(d^{(0)}_{N-SSH}\right)_{y}=\,(1+\eta)\,\sin
k_{x}\,J_{0}(A_{0})-\delta\,\sin k_{x}\,\cos k_{y}\,J_{0}(A_{0}).$ (16c)

The Fourier component of the Hamiltonian for $m=1$ is obtained as:

$H^{(1)}_{N-SSH}=\,\boldsymbol{d}^{(1)}_{N-SSH}\cdot\boldsymbol{\sigma}$ (17a)
where $\left(d^{(1)}_{N-SSH}\right)_{x}=\,-(1+\eta)\,\sin
k_{x}\,J_{1}(A_{0})+\delta\sin k_{x}\cos k_{y}J_{1}(A_{0})$ (17b)
$\left(d^{(1)}_{N-SSH}\right)_{y}=\,\left[(1+\eta)-\delta\cos
k_{y}\right]\,\cos k_{x}\,J_{1}(A_{0})$ (17c)

and for $m=2$ as:

$H^{(2)}_{N-SSH}=\,\boldsymbol{d}^{(2)}_{N-SSH}\cdot\boldsymbol{\sigma}$ (18a)
$\left(d^{(2)}_{N-SSH}\right)_{x}=\,-(1+\eta)\,\cos
k_{x}\,J_{2}(A_{0})+\delta\cos k_{x}\cos k_{y}J_{2}(A_{0})$ (18b)
$\left(d^{(2)}_{N-SSH}\right)_{y}=\,-\left[(1+\eta)-\delta\cos
k_{y}\right]\,\sin k_{x}J_{2}(A_{0}).$ (18c)

The Floquet energy band diagrams are shown in Figs. 9, 10 and 11. Here again,
we consider two different cases depending on the topological property of the
individual SSH chain. In Figs. 9 and 10, each SSH chain is considered as
nontrivial by setting $\eta=0.5$. Unlike the case of a hexagonal lattice, here
we observe that the semi-Dirac-like point splits into two band-touching points
with non-linear dispersion along $y$-direction. This behavior is observed for
both high and low-frequency regimes. In Figs. 9(a) and (c), we set the driving
frequency $\Omega=3.0$. On the other hand, in Figs. 9(b) and (d), we set
$\Omega=6.0$. The driving amplitude is fixed at $A_{0}=0.5$ for both
frequencies. As we increase the driving amplitude, the separation between the
two band touching points increases, as shown in Fig. 10. On the contrary, when
we consider each SSH chain as topologically trivial, the semi-Dirac-like point
does not split, but a band gap opens at that point. This result is shown in
Fig. 11.

Figure 11: The quasienergy bands are presented for the case when the LPL is
applied along the $x$-direction. Here, we fix the parameters
$\delta=-\eta=0.5$; hence, the individual SSH chain is topologically trivial.
We set the driving amplitude $A_{0}=0.5$. In subfigures (a) and (c), we set
$\Omega=3.0$, while in subfigures (b) and (d), we set $\Omega=6.0$. Here, we
see a gap opening in the quasienergy band at the semi-Dirac-like point.

###  V.2 Linearly polarized light along ${\boldsymbol{y}}$-direction

We now consider the other case where the LPL is applied along $y$-direction of
the same form as earlier, i.e., $A(t)=A_{0}\cos\Omega t$. Like the previous
case, for this driving, we find the Fourier component of the Hamiltonian for
$m=0$ as:

$H^{(0)}_{N-SSH}=\,\boldsymbol{d}^{(0)}_{N-SSH}\cdot\boldsymbol{\sigma}$ (19a)
where
$\begin{split}\left(d^{(0)}_{N-SSH}\right)_{x}&=\,(1-\eta)+(1+\eta)\,\cos
k_{x}+\delta\,\cos k_{y}\,J_{0}(A_{0})\\\ &-\delta\,\cos k_{x}\,\cos
k_{y}\,J_{0}(A_{0})\end{split}$ (19b)
$\left(d^{(0)}_{N-SSH}\right)_{y}=\,(1+\eta)\,\sin k_{x}-\delta\,\sin
k_{x}\,\cos k_{y}\,J_{0}(A_{0}).$ (19c)

Again, we derive the Fourier component of the Hamiltonian for $m=1$ as:

$H^{(1)}_{N-SSH}=\,\boldsymbol{d}^{(1)}_{N-SSH}\cdot\boldsymbol{\sigma}$ (20a)
$\left(d^{(1)}_{N-SSH}\right)_{x}=\,-\delta\,\sin
k_{y}\,J_{1}(A_{0})+\delta\cos k_{x}\sin k_{y}J_{1}(A_{0})$ (20b)
$\left(d^{(1)}_{N-SSH}\right)_{y}=\,\delta\sin k_{x}\,\sin
k_{y}\,J_{1}(A_{0})$ (20c)

and for $m=2$ as:

$H^{(2)}_{N-SSH}=\,\boldsymbol{d}^{(2)}_{N-SSH}\cdot\boldsymbol{\sigma}$ (21a)
$\left(d^{(2)}_{N-SSH}\right)_{x}=\,-\delta\,\cos
k_{y}\,J_{2}(A_{0})+\delta\cos k_{x}\cos k_{y}J_{2}(A_{0})$ (21b)
$\left(d^{(2)}_{N-SSH}\right)_{y}=\,\delta\sin k_{x}\cos k_{y}\,J_{2}(A_{0}).$
(21c)

In Fig. 12, the Floquet band diagrams are shown for the case where the
individual SSH chain is topologically nontrivial. On the other hand, in Fig.
13, the Floquet band diagrams are presented for the trivial case. For both
cases, the semi-Dirac-like point does not split, and a band gap opens at that
point.

Figure 12: The quasi-energy bands are shown for the case when the LPL is
applied along $y$-direction, and here we consider the individual SSH chain as
topologically nontrivial by setting the parameters $\delta=\eta=0.5$. The
driving amplitude is set at $A_{0}=0.5$ in all the subfigures. In the
subfigures (a) and (c), we consider $\Omega=3.0$, while $\Omega=6.0$ is
considered in subfigures (b) and (d). Here, we also observe a band gap opening
at the semi-Dirac-like point. Figure 13: Similar results as Fig. 12 are
presented, but here, the individual SSH chain is considered topologically
trivial by setting $\delta=-\eta=0.5$.

##  VI Summary

We study the effect of time-periodic driving on the $N$ stacked SSH model and
examine the topological properties of this system. We use circularly polarized
light as a periodic drive and compute the effective Hamiltonian using the
Floquet replica method. This method is exact, and one can determine the
topological phases in high-frequency and low-frequency regimes. This system
exhibits topological phases with high Chern numbers in the low-frequency
regime. We consider two different versions of the $N$ stacked SSH model: in
one case, the individual SSH chain is topologically trivial, and the chains
are nontrivial in the other case. Under periodic driving, both versions have
distinct topological properties. From the phase diagrams in the driving
parameter space, we observe topological phases with high ($|C|>1$) Chern
numbers in the low-frequency regime, whereas $|C|=1$ is observed for the high-
frequency cases. When the $N$ stacked SSH model has all trivial SSH chains,
the highest Chern number is $C=-3$, while for each nontrivial SSH chain, the
highest Chern number is $C=-4$. We have also observed that, with the varying
driving amplitude $A_{0}$, the topological phases remain the same but change
with the varying driving frequency $\Omega$. The topological transition occurs
when a band gap closing or reopening happens in the system. Therefore, with
the varying $\Omega$, we have shown that the quasienergy bands lie in the
first FBZ. We have also shown the band diagrams in cylindrical geometry to
present the results with the high Chern number.

In the Floquet systems, the total number of chiral edge states between the gap
at $\epsilon=0$ and $\epsilon=\frac{\Omega}{2}$ is calculated from the
relation $C=C_{0}-C_{\pi}$. Here, $C_{0}$ and $C_{\pi}$ are the sum of the
Chern number of all the Floquet bands below the energy at $\epsilon=0$ and
$\epsilon=\frac{\Omega}{2}$, respectively. For both cases, we always find
$C_{\pi}=0$, which gives $C=C_{0}$. The model we study in this paper is
nontrivial due to the presence of coupling between the quasimomenta $k_{x}$
and $k_{y}$. We construct the low-energy Hamiltonian around the band touching
points to observe the interplay of this coupling and the periodic driving more
prominently. The $k_{x}-k_{y}$ coupling term makes the dispersion relation of
this system unconventional. Even after the presence of a very different
dispersion relation than graphene, the $N$ stacked SSH model shares a similar
signature of hierarchy in the Floquet gaps with graphene.

Due to the quadratic dispersion, we observe a semi-Dirac-like band touching
point for a particular choice of the system parameter. We find that, for the
periodic driving with LPL along $x$-direction, the semi-Dirac-like point
splits into two band touching points with nonlinear dispersion. On the other
hand, when the LPL is applied along the $y$-direction, a band gap opens around
the semi-Dirac-like point. The same observation is shown for the hexagonal
lattice [35]. We have shown the band diagrams for the existence of the semi-
Dirac point, splitting into two band points and the opening of the band gap
for the cylindrical geometry.

###### Acknowledgements.

Authors acknowledge financial support from DST-SERB, India, through the Core
Research Grant CRG/2020/001701 and also through MATRICS Grant No.
MTR/2022/000691.

## References

  * Agrawal and Bandyopadhyay [2023] A. Agrawal and J. N. Bandyopadhyay, Cataloging topological phases of $N$ stacked Su-Schrieffer-Heeger chains by a systematic breaking of symmetries, Phys. Rev. B 108, 104101 (2023).
  * Oka and Kitamura [2019] T. Oka and S. Kitamura, Floquet engineering of Quantum materials, Annual Review of Condensed Matter Physics 10, 387 (2019).
  * Cayssol  _et al._ [2013] J. Cayssol, B. Dóra, F. Simon, and R. Moessner, Floquet topological insulators, physica status solidi (RRL) – Rapid Research Letters 7, 101 (2013).
  * Yang  _et al._ [2022] K. Yang, S. Xu, L. Zhou, Z. Zhao, T. Xie, Z. Ding, W. Ma, J. Gong, F. Shi, and J. Du, Observation of Floquet topological phases with large Chern numbers, Phys. Rev. B 106, 184106 (2022).
  * Xiong  _et al._ [2016] T.-S. Xiong, J. Gong, and J.-H. An, Towards large-Chern-number topological phases by periodic quenching, Phys. Rev. B 93, 184306 (2016).
  * Rudner and Lindner [2020] M. S. Rudner and N. H. Lindner, Band structure engineering and non-equilibrium dynamics in floquet topological insulators, Nature Reviews Physics 2, 229 (2020).
  * Lindner  _et al._ [2011] N. H. Lindner, G. Refael, and V. Galitski, Floquet topological insulator in semiconductor quantum wells, Nature Physics 7, 490 (2011).
  * Foa Torres  _et al._ [2014] L. E. F. Foa Torres, P. M. Perez-Piskunow, C. A. Balseiro, and G. Usaj, Multiterminal conductance of a Floquet topological insulator, Phys. Rev. Lett. 113, 266801 (2014).
  * Rechtsman  _et al._ [2013] M. C. Rechtsman, J. M. Zeuner, Y. Plotnik, Y. Lumer, D. Podolsky, F. Dreisow, S. Nolte, M. Segev, and A. Szameit, Photonic Floquet topological insulators, Nature 496, 196 (2013).
  * Kumar  _et al._ [2020] A. Kumar, M. Rodriguez-Vega, T. Pereg-Barnea, and B. Seradjeh, Linear response theory and optical conductivity of Floquet topological insulators, Phys. Rev. B 101, 174314 (2020).
  * Jotzu and Messer [2014] G. Jotzu and M. e. a. Messer, Experimental realization of the topological Haldane model with ultracold fermions, Nature 515, 237 (2014).
  * Bloch [2005] I. Bloch, Ultracold quantum gases in optical lattices, Nature Physics 1, 23 (2005).
  * Fu  _et al._ [2007] L. Fu, C. L. Kane, and E. J. Mele, Topological insulators in three dimensions, Phys. Rev. Lett. 98, 106803 (2007).
  * Moore and Balents [2007] J. E. Moore and L. Balents, Topological invariants of time-reversal-invariant band structures, Phys. Rev. B 75, 121306 (2007).
  * Das  _et al._ [2019] S. K. Das, B. Yan, J. van den Brink, and I. C. Fulga, Topological crystalline insulators from stacked graphene layers, Phys. Rev. B 99, 165418 (2019).
  * Li  _et al._ [2018] S. Li, C.-C. Liu, and Y. Yao, Floquet high chern insulators in periodically driven chirally stacked multilayer graphene, New Journal of Physics 20, 033025 (2018).
  * Mannaï  _et al._ [2023] M. Mannaï, J.-N. Fuchs, F. Piéchon, and S. Haddad, Stacking-induced Chern insulator, Phys. Rev. B 107, 045117 (2023).
  * Kim  _et al._ [2015] Y. Kim, C. L. Kane, E. J. Mele, and A. M. Rappe, Layered topological crystalline insulators, Phys. Rev. Lett. 115, 086802 (2015).
  * Qian  _et al._ [2020] Y. Qian, Z. Tan, T. Zhang, J. Gao, Z. Wang, Z. Fang, C. Fang, and H. Weng, Layer construction of topological crystalline insulator LaSbTe, Science China Physics, Mechanics & Astronomy 63, 107011 (2020).
  * Zou and Liu [2016] J.-Y. Zou and B.-G. Liu, Floquet Weyl fermions in three-dimensional stacked graphene systems irradiated by circularly polarized light, Phys. Rev. B 93, 205435 (2016).
  * Ladovrechis and Fulga [2019] K. Ladovrechis and I. C. Fulga, Anomalous Floquet topological crystalline insulators, Phys. Rev. B 99, 195426 (2019).
  * Liu  _et al._ [2016] S. Liu, T. Ohtsuki, and R. Shindou, Effect of disorder in a three-dimensional layered Chern insulator, Phys. Rev. Lett. 116, 066401 (2016).
  * Burkov and Balents [2011] A. A. Burkov and L. Balents, Weyl semimetal in a topological insulator multilayer, Phys. Rev. Lett. 107, 127205 (2011).
  * Li  _et al._ [2014] L. Li, Z. Xu, and S. Chen, Topological phases of generalized Su-Schrieffer-Heeger models, Phys. Rev. B 89, 085111 (2014).
  * Agrawal and Bandyopadhyay [2022] A. Agrawal and J. N. Bandyopadhyay, Floquet topological phases with high Chern numbers in a periodically driven extended Su–Schrieffer–Heeger model, Journal of Physics: Condensed Matter 34, 305401 (2022).
  * Dal Lago  _et al._ [2015] V. Dal Lago, M. Atala, and L. E. F. Foa Torres, Floquet topological transitions in a driven one-dimensional topological insulator, Phys. Rev. A 92, 023624 (2015).
  * Lü and Xie [2019] X.-L. Lü and H. Xie, Topological phases and pumps in the Su–Schrieffer–Heeger model periodically modulated in time, Journal of Physics: Condensed Matter 31, 495401 (2019).
  * Li  _et al._ [2020] C.-F. Li, L.-N. Luan, and L.-C. Wang, Topological properties of an extend Su-Schrieffer-Heeger model under periodic kickings, International Journal of Theoretical Physics 59, 2852 (2020).
  * Asbóth  _et al._ [2014] J. K. Asbóth, B. Tarasinski, and P. Delplace, Chiral symmetry and bulk-boundary correspondence in periodically driven one-dimensional systems, Phys. Rev. B 90, 125143 (2014).
  * Oka and Aoki [2009] T. Oka and H. Aoki, Photovoltaic Hall effect in graphene, Phys. Rev. B 79, 081406 (2009).
  * Ghosh  _et al._ [2020] A. K. Ghosh, G. C. Paul, and A. Saha, Higher order topological insulator via periodic driving, Phys. Rev. B 101, 235403 (2020).
  * Perez-Piskunow  _et al._ [2014] P. M. Perez-Piskunow, G. Usaj, C. A. Balseiro, and L. E. F. F. Torres, Floquet chiral edge states in graphene, Phys. Rev. B 89, 121401 (2014).
  * Wang  _et al._ [2013] Y. H. Wang, H. Steinberg, P. Jarillo-Herrero, and N. Gedik, Observation of Floquet-Bloch states on the surface of a topological insulator, Science 342, 453 (2013).
  * Kundu  _et al._ [2014] A. Kundu, H. A. Fertig, and B. Seradjeh, Effective theory of Floquet topological transitions, Phys. Rev. Lett. 113, 236803 (2014).
  * Chen  _et al._ [2018] Q. Chen, L. Du, and G. A. Fiete, Floquet band structure of a semi-Dirac system, Phys. Rev. B 97, 035422 (2018).
  * Kitagawa  _et al._ [2011] T. Kitagawa, T. Oka, A. Brataas, L. Fu, and E. Demler, Transport properties of nonequilibrium systems under the application of light: Photoinduced quantum Hall insulators without Landau levels, Phys. Rev. B 84, 235108 (2011).
  * Perez-Piskunow  _et al._ [2015] P. M. Perez-Piskunow, L. E. F. Foa Torres, and G. Usaj, Hierarchy of Floquet gaps and edge states for driven honeycomb lattices, Phys. Rev. A 91, 043625 (2015).
  * Haldane [1988] F. D. M. Haldane, Model for a quantum Hall effect without landau levels: Condensed-matter realization of the “parity anomaly”, Phys. Rev. Lett. 61, 2015 (1988).
  * Huang and Xu [2021] S. Huang and X. Xu, Optical chirality detection using a topological insulator transistor, Advanced Optical Materials 9, 2002210 (2021).
  * Karch  _et al._ [2010] J. Karch, P. Olbrich, M. Schmalzbauer, C. Zoth, C. Brinsteiner, M. Fehrenbacher, U. Wurstbauer, M. M. Glazov, S. A. Tarasenko, E. L. Ivchenko, D. Weiss, J. Eroms, R. Yakimova, S. Lara-Avila, S. Kubatkin, and S. D. Ganichev, Dynamic Hall effect driven by circularly polarized light in a graphene layer, Phys. Rev. Lett. 105, 227402 (2010).
  * Hatano  _et al._ [2009] T. Hatano, T. Ishihara, S. G. Tikhodeev, and N. A. Gippius, Transverse photovoltage induced by circularly polarized light, Phys. Rev. Lett. 103, 103906 (2009).
  * Tahir and Vasilopoulos [2015] M. Tahir and P. Vasilopoulos, Off-resonant polarized light-controlled thermoelectric transport in ultrathin topological insulators, Phys. Rev. B 91, 115311 (2015).
  * Baykusheva  _et al._ [2021] D. Baykusheva, A. Chacón, J. Lu, T. P. Bailey, J. A. Sobota, H. Soifer, P. S. Kirchmann, C. Rotundu, C. Uher, T. F. Heinz, D. A. Reis, and S. Ghimire, All-optical probe of three-dimensional topological insulators based on high-harmonic generation by circularly polarized laser fields, Nano Letters 21, 8970 (2021).
  * Bernevig  _et al._ [2006] B. A. Bernevig, T. L. Hughes, and S.-C. Zhang, Quantum spin Hall effect and topological phase transition in HgTe Quantum wells, Science 314, 1757 (2006).
  * Qi  _et al._ [2006] X.-L. Qi, Y.-S. Wu, and S.-C. Zhang, Topological quantization of the spin Hall effect in two-dimensional paramagnetic semiconductors, Phys. Rev. B 74, 085308 (2006).
  * Hatsugai  _et al._ [2006] Y. Hatsugai, T. Fukui, and H. Aoki, Topological analysis of the quantum hall effect in graphene: Dirac-fermi transition across van hove singularities and edge versus bulk quantum numbers, Phys. Rev. B 74, 205414 (2006).
  * Yu  _et al._ [2011] S.-L. Yu, X. C. Xie, and J.-X. Li, Mott physics and topological phase transition in correlated dirac fermions, Phys. Rev. Lett. 107, 010401 (2011).
  * Castro Neto  _et al._ [2009] A. H. Castro Neto, F. Guinea, N. M. R. Peres, K. S. Novoselov, and A. K. Geim, The electronic properties of graphene, Rev. Mod. Phys. 81, 109 (2009).
  * Sun  _et al._ [2009] K. Sun, H. Yao, E. Fradkin, and S. A. Kivelson, Topological insulators and nematic phases from spontaneous symmetry breaking in 2d fermi systems with a quadratic band crossing, Phys. Rev. Lett. 103, 046811 (2009).
  * Floquet [1883] G. Floquet, On linear differential equations with periodic coefficients, Scientific annals of the Ecole Normale Supérieure 2nd series, 12, 47 (1883).
  * Shirley [1965] J. H. Shirley, Solution of the Schrödinger equation with a Hamiltonian periodic in time, Phys. Rev. 138, B979 (1965).
  * Eckardt and Anisimovas [2015] A. Eckardt and E. Anisimovas, High-frequency approximation for periodically driven quantum systems from a Floquet-space perspective, New Journal of Physics 17, 093039 (2015).
  * Sambe [1973] H. Sambe, Steady states and quasienergies of a quantum-mechanical system in an oscillating field, Phys. Rev. A 7, 2203 (1973).
  * Eckardt and Holthaus [2008] A. Eckardt and M. Holthaus, Dressed matter waves, Journal of Physics: Conference Series 99, 012007 (2008).
  * Rudner  _et al._ [2013] M. S. Rudner, N. H. Lindner, E. Berg, and M. Levin, Anomalous edge states and the bulk-edge correspondence for periodically driven two-dimensional systems, Phys. Rev. X 3, 031005 (2013).


# Graph Expansion in Pruned Recurrent Neural Network Layers Preserve
Performance

Suryam Arnav Kalra
Department of Computer Science and Engineering
Indian Institute of Technology Kharagpur
Kharagpur, India.
<EMAIL_ADDRESS>
&Arindam Biswas
Research Scientist
Polynom
Paris, France.
<EMAIL_ADDRESS>
&Pabitra Mitra
Department of Computer Science and Engineering
Indian Institute of Technology Kharagpur
Kharagpur, India.
<EMAIL_ADDRESS>
&Biswajit Basu
School of Engineering,
Trinity College Dublin
Dublin, Ireland.
<EMAIL_ADDRESS>
Corresponding author

###### Abstract

Expansion property of a graph refers to its strong connectivity as well as
sparseness. It has been reported that deep neural networks can be pruned to a
high degree of sparsity while maintaining their performance. Such pruning is
essential for performing real time sequence learning tasks using recurrent
neural networks in resource constrained platforms. We prune recurrent networks
such as RNNs and LSTMs, maintaining a large spectral gap of the underlying
graphs and ensuring their layerwise expansion properties. We also study the
time unfolded recurrent network graphs in terms of the properties of their
bipartite layers. Experimental results for the benchmark sequence MNIST,
CIFAR-10, and Google speech command data show that expander graph properties
are key to preserving classification accuracy of RNN and LSTM.

##  1 Introduction

Analysis of Artificial Neural Networks (ANNs) following a connection base
approach is a topical research direction as this not only mimics brain
networks in neuroscience, but also can provide specific graph measures which
can be used for analysis of performance and robustness of the networks.

Researchers in the recent years have explored if there is a relation between
the functional aspects of an ANN and its graph structure, and if such a
relation does exist then are there any characterization that explains the
relationship between the structure of the graph and is performance LeCun et
al. (1998); Sermanet et al. (2013); Zeiler & Fergus (2014); Krizhevsky et al.
(2017); Simonyan & Zisserman (2014); He et al. (2016); Szegedy et al. (2015).
However, most of these studies have been performed on custom design
architectures for specific tasks. Some researchers have also proposed ANNs
with a better predictive performance by varying the connections (i.e., edges)
between nodes and the operations they perform Zoph & Le (2017); Zoph et al.
(2018); Ying et al. (2019). However, identifying measures which can quantify
the influence of the structure on the performance for generic graph models of
ANN is a matter of open research interest.

In spite of effectiveness and robustness of Deep Learning Networks (DNNs),
there are still shortcomings due to limitations on resources for PCs and
embedded systems, for implementation of these networks. DNNs put significantly
high demand on computational power and energy consumption. With an aim to
reduce the memory demand and also to accelerate the interface speed of neural
networks pruning approaches are widely employed. It has been observed that
many of these dense networks contain a subnetwork, termed as a ‘winning
lottery ticket’ which is sparse yet performs as well as the dense network
Frankle & Carbin (2018).

Recurrent neural networks and Long Short Term Memory (LSTM) are network
structures with feedback connection that are useful in learning sequences.
Computational requirements of such networks are often enormous, thus
highlighting the need for extensive pruning and sparsification.

Despite, several attempts being made, the most effective method to sparsify
and prune networks is still Iterative Magnitude-based Pruning (IMP) Blalock et
al. (2020). This method is computationally intensive to run and is required to
be rigorously executed for every different network. Recent advances in the
research on neural networks have made pruning large feed forward and
convolutional networks possible even with a small quantity of data leading to
sample efficiency. Yet, when attempting to recover sparse recurrent networks,
these techniques may not work suitably. The failure to achieve sparse
recurrent networks may be either quantitative or qualitative in nature.
Sometimes when more recent sophisticated pruning techniques are applied, RNNs
typically obtain worse performance quantitatively than they do under a simple
random pruning scheme. But, if the distribution of weights in a pruned Long
Short Term Memory (LSTM) or RNN network tend to be concentrated in specific
neurons and gates and not well dispersed across the entire architecture Zhang
& Stadie (2019), then it may lead to qualitative failure in the process of
achieving a sparse network. Also, if the pruning leads to irregular network
structure, then there are possibilities of degradation of performance/energy
efficiency Li et al. (2018). Network pruning based on regular graph structure
has been proposed in Chen et al. (2023).

It has been observed across a broad range of different types of neural network
architectures (such as deep CNNs, stacked LSTM, and seq2seq LSTM models), that
large-sparse models tend to consistently outperform small-dense models by
achieving up to ten times reduction in number of non-zero parameters while
preserving the accuracy Zhu & Gupta (2017). This provides a strong motivation
to explore the basis of such observations and support or explain this by
possible graph measures, if any. There would be possible benefits brought
about from such analysis as there is an increasing trend to deploy deep neural
networks at the edge for high-accuracy, applications involved are real-time
data mining and user interaction, speech recognition and language
understanding to name a few. The latter two often apply a DNN to encode an
input sequence and then use a decoder to generate the output sequence.

Fully connected structures of RNNs Grossberg (2013) and LSTMs Hochreiter &
Schmidhuber (1997) leading to the requirement of large memory footprint make
these difficult to be implemented on resource limited hardware, though these
recurrent networks are known to be useful for several applications as
previously mentioned. Previous structured pruning methods can effectively
reduce RNN size; however, how to find a good balance between high sparsity and
high task accuracy, remains an open question. Failing to answer this would
give us pruned models only, which lead to moderate speedup on custom hardware
accelerators.

In general, RNNs have a significant large number of parameters and hence the
overall process of training involves enormous computational costs, as it is
required to repeat the process over the recurrent structures for multiple time
steps. To overcome this difficulty, the approach of RNN pruning has attracted
increasing attention in recent years Li et al. (2022b). It is beneficial
leading to reduction of computational cost as the time step progresses.
Nevertheless, most existing methods of RNN pruning are heuristic Furuya et al.
(2022) and hence any quantitative measure to improve the performance would be
a development forward.

Search of measures which can unravel the relationship between the structure
and performance of the neural network has also been motivated by an important
aspect of implementation of deep learning technologies. Lightweight design of
learning models has become an important research direction in the application
of deep learning. Pruning or sparsification is an effective means to achieve a
large reduction in model parameters and Floating Point Operations. The
relation between the lottery ticket hypothesis and expander graph properties
for fully connected and convolutional neural networks has been explored in Pal
et al. (2022). Similar graph measures that are easily computable can
significantly assist in guiding the sparsification approach.

In this paper, we generalise the study on expander properties of recurrent
architectures, both theoretically and with experimental implementations. We
show that the ‘layer-wise expansion’ property of a graph is crucial to
maintain connectivity and flow of information. The criteria (based on bounds
on spectral properties) that preserve the expansion property can be used to
decide the adequacy of a sparsified RNN for lightweight architecture design.
In fact, this approach in combination with other attributes can form the basis
of a new cost-effective sparsification and pruning algorithms to be used for
real-time applications. The claims on the ‘expander’ property-based technique
for the analysis and testing of the suitability of a lightweight architecture
have been verified on RNN and LSTM architectures (including the effect of
noise) highlighting the Importance of different types of connectivity in these
two cases. Additionally, the theoretical analysis of a bi-partite structure of
the rolled out RNN graph leading to the spectral analysis of a Toeplitz matrix
structure has provided additional insights for computation of the properties.

Contributions: The contributions of the paper are the following:

  * •

We propose an approach for analyzing generic RNN/LSTM architectures with
respect to their graph expansion properties.

  * •

We have identified qualification criteria based on expansion properties for
identifying suitable lightweight RNN/LSTM architecture with a view to real-
time hardware applications.

  * •

We have experimentally observed that the RNN/LSTM architectures, even with
highly reduced density satisfying the layerwise bipartite Ramanujan graph
property for the rolled out network, represent resiliently connected global
networks achieving high accuracy as compared to the original high-density
networks.

  * •

Our approach for analysis and the proposed qualification criteria based on
expansion properties for identifying suitable lightweight RNN architecture is
robust to noise. This has been verified experimentally.

  * •

We have identified the importance of different types of connections at
different layers in an RNN architecture; and hence this guides the decision on
how to prune a dense RNN, if necessary.

  * •

The expander graph property used in our study may be used as a stopping
criteria for network pruning algorithms for RNNs and LSTMs.

##  2 Related Work

Graph structure of neural networks has been previously studied in literature
You et al. (2020). Several sparse neural network topology have been explored
in this context for deep neural networks Bourely et al. (2017). Statistical
network properties like the power law are seen to hold for such sparse
networks Feng et al. (2022). Recently, block sparse recurrent neural networks
Narang et al. (2017) and sparse LSTM Liu et al. (2019) topologies have been
suggested.

Sparsity in deep neural networks is often considered as a desirable property
in the regularization framework Thom & Palm (2013). Sparsity may map to
functional specializations Béna & Goodman (2021). It is definitely beneficial
from the point of view of fast inference in resource poor platforms like edge
devices Han et al. (2017). It has been widely reported in literature that such
sparse networks often retain performance of the original network Frankle &
Carbin (2018); Khona et al. (2022); Ding et al. (2021).

Connectivity of sparse neural network are recently being characterized by
their spectral graph properties You et al. (2020); McDonald & Shokoufandeh
(2019); Hod et al. (2021). Expander graph properties of deep fully connected
and convolutional networks (CNN0 has been first reported in Prabhu et al.
(2018). It was established in Pal et al. (2022) that Ramanujan graph
properties of sparse feedforward and CNN is essential for maintaining their
performance. Graph connectivity property of recurrent neural networks and
LSTMs has not been studied in literature so far.

Pruning RNNs has been explored for a long time in literature Giles & Omlin
(1994). Large implicit depth in RNN necessaites the pruning. Similarly,
pruning is also useful in reducing computational complexity of LSTMs Zhu &
Gupta (2017). Among the pruning methods stagewise magnitude based prunign is
among the most widely used technique Li et al. (2022a). Chatzikonstantinou et
al. proposed a novel RNN pruning method that considers the RNN weight matrices
as collections of time-evolving signals Chatzikonstantinou et al. (2021).
Spectral pruning algorithms that consider graph properties of the RNN
structure are proposed very recently in Furuya et al. (2022). Regular graph
structure of RNN has been explored in Chen et al. (2023). A one-shot pruning
approach based on spectral properties of a RNN and having reduced
computational effort as compared to iterative pruning has been proposed in
Zhang & Stadie (2019). Differential pruning of various RNN layers has been
proposed in Li et al. (2022b). Pruning algorithms based on supermasks have
also been studied for transformer networks Choromanski et al. (2022).

RNN and LSTM are often considered to be hard to prune due to the recurrent
structure of the underlying graph. No guideline exists in literature regarding
the degree of pruning possible while ensuring prediction performance. We
propose to use expander graph property as a criteria for this purpose. The
recurrent networks are pruned to a point till the sparse connectivity
properties are preserved.

##  3 Expander Graphs and Ramanujan Graphs

In this section, we discuss various properties of expanders which will be
pertinent for the rest of the article. An expander graph is a sparse graph
that has strong connectivity properties. The connectivity can be quantified in
different ways giving rise to different notions of expanders such as vertex
expanders, edge expanders and spectral expanders. These notions are
interrelated. Recall that a graph $\Gamma=(V,E)$ is a tuple consisting of a
vertex set $V$ and an edge set $E$ which is a subset of $V\times V$.

###  3.1 Combinatorial Expansion

######  Definition 1 (Expander and Cheeger constant)

A graph $\Gamma=(V,E)$ is said to be an $\epsilon$-vertex expander if for
every non-empty subset $X\subset V$ with $|X|\leq\frac{|V|}{2}$, we have
$\frac{|\delta(X)|}{|X|}\geq\epsilon$, where $\delta(X)$ denotes the outer
vertex boundary of $X$ i.e., the set of vertices in $\Gamma$ which are
connected to a vertex in $X$ but do not lie in $X$. The infimum as $X$ runs
over all subsets of $V$ satisfying the conditions above is known as the vertex
Cheeger constant and is denoted by $\mathfrak{h}(\Gamma)$.

Edge expanders and the edge Cheeger constant $\mathbf{h}(\Gamma)$ are defined
similarly, where in place of the vertex boundary, we consider the edge
boundary i.e., the set of edges which have one vertex in $X$ and the other
outside of $X$. The vertex Cheeger constant $\mathfrak{h}(\Gamma)$ and the
edge Cheeger constant $\mathbf{h}(\Gamma)$ are related by the following
equivalence

$\frac{\mathfrak{h}(\Gamma)}{D}\leq\mathbf{h}(\Gamma)\leq\mathfrak{h}(\Gamma),$

where $D$ denotes the maximum degree of the graph (the degree of each vertex
is the number of edges going out from the vertex). The equivalence allows us
to speak about vertex expansion and edge expansion interchangeably.
Intuitively, given a graph with high vertex (or edge) Cheeger constant, it is
more difficult to separate any subset of the vertices from the rest of the
graph. This allows for free flow of information throughout the network which
the graph modelises. In the literature, having a high Cheeger constant is also
known as having high combinatorial expansion.

###  3.2 Spectral Expansion

The notion of spectral expansion is a bit different from combinatorial
expansion. Given a finite undirected graph $\Gamma$ the eigenvalues
$\lambda_{n}\leq\cdots\leq\lambda_{1}$ of its adjacency matrix are all real
and $\lambda_{1}\leq D$ with equality iff the graph is $D$-regular. A graph
$\Gamma=(V,E)$ is said to be a spectral expander if the quantities
$\\{|\lambda_{1}|-|\lambda_{2}|,|\lambda_{1}|-|\lambda_{k}|\\}$ are both
bounded away from zero, where $k=n-1$ if the graph is bipartite and $k=n$
otherwise.

###  3.3 Discrete Cheeger–Buser Inequality

Ideally, to ensure free flow information within the network, our goal is to
ensure that the graphs which modelise the networks have high combinatorial
expansion. This is achieved via the discrete Cheeger–Buser inequality
discovered independently by Dodziuk (1984) and by Alon & Milman (1985). The
inequality states that

$\frac{\mathbf{h}(\Gamma)^{2}}{2}\leq\alpha_{2}\leq 2\mathbf{h}(\Gamma),$

where $\alpha_{2}$ denotes the second smallest eigenvalue of the normalised
Laplacian matrix of $\Gamma$ and is related to the eigenvalues of the
adjacency matrix via

$\frac{\lambda_{i}}{D}\leq 1-\alpha_{i}\leq\frac{\lambda_{i}}{d}\,\,\forall
i=1,2,\ldots,n.$

See Chung (2016) for details. From the above, it is easy to check that a high
$|\lambda_{1}|-|\lambda_{2}|$ ensures a high $\mathbf{h}(\Gamma)$ and vice-
versa. Thus, the two notions of expansion are inter-connected and every
spectral expander remains a combinatorial expander. They are actually
equivalent for some classes of graphs, for instance bipartite graphs (as the
adjacency spectrum is symmetric about the origin), variants of algebraic
graphs Breuillard et al. (2015); Biswas (2019); Biswas & Saha (2021; 2022;
2023; 2021) etc.

###  3.4 Ramanujan graph bounds

A $d$-regular graph is said to be a Ramanujan graph if
$\max\\{|\lambda_{2}|,|\lambda_{k}|\\}\leq 2\sqrt{d-1}$. In the case of
bipartite graphs, $\lambda_{k}=\lambda_{2}$, hence the previous expression
reduces to $|\lambda_{2}|\leq 2\sqrt{d-1}$. For fixed degree, with the sizes
of the graphs growing larger and larger, these are the best possible
expanders, as given by the Alon-Bopanna bound. We refer to
Hoory–Linial–Wigderson Hoory et al. (2006) for the details.

When the graphs modelising the network are irregular (and possibly weighted),
to guarantee large expansion, we use two closely related quantities for $d$.
The combinatorial quantity $d_{avg}$ which is the average degree taking into
account all vertices and the spectral quantity $\lambda_{1}$ which is the
largest eigenvalue of the adjacency operator. The use of these quantities is
justified by the work of Hoory Hoory (2005) and result in extremal families.
Further, they have the added advantage of being easy to compute. Using them,
we consider the following expressions $\Delta_{R},\Delta_{S}$ with

$\displaystyle\Delta_{R}$
$\displaystyle=\frac{2\sqrt{d_{avg}-1}-\lambda_{2}}{\lambda_{2}}$ (1)
$\displaystyle\Delta_{S}$
$\displaystyle=\frac{2\sqrt{\lambda_{1}-1}-\lambda_{2}}{\lambda_{2}}$ (2)

We recall that these bounds were also considered in Pal et al. (2022).

##  4 Network Structures of RNN and LSTM

Recurrent networks and LSTMs are cyclic structures where the output from
previous time step state as well as the current time step input is used to
compute the current output. The recurrent structure can be unfolded over time
steps of the sequence to obtain a graph without cycles. In the unrolled the
network the weights are copied over the time steps and only the state and the
input values change. Thus the unfolded RNN/LSTM can be viewed as a very deep
feedforward network. The network structures of unrolled RNNs and LSTMs can be
modelised using bipartite graphs. Given a RNN or a LSTM, we consider the
complete bipartite network through which the input data passes during
inference.

In these sort of architectures, the bipartite networks are of size $m\times m$
where $m$ denotes the number of vertices in the first layer. Consider the
(weighted) adjacency matrix $B$ of this network (modelised by the graph
$\Gamma$) i.e., if there is a weight $w$ from $i$ to $j$ in the graph
$\Gamma$, we shall substitute $|w|$ for the corresponding entry in the matrix.
We will use an iterated magnitude based pruning and check that at each step,
the Ramanujan graph bound $\Delta_{S}>0$ is maintained. This ensures that the
network remains a strong combinatorial expander via the discrete Cheeger
inequality. The consideration of layerwise expansion also has a global
implication (for small unrolling).

###  4.1 Representation of the time unfolded network

In this section we briefly discuss the eigenvalue bounds for a simplified
version of the time unfolded network. Let $\begin{pmatrix}\mbox{0}_{m,m}&B\\\
B^{T}&\mbox{0}_{m,m}\\\ \end{pmatrix}$ be the adjacency matrix of the
bipartite graph of size $2m$ modeling the initial layer $W_{xh}$ of the neural
network. Let $k$ denote the number of times the data loops through the
unfolded neural network, i.e., the sequence length. We consider a simplified
version of the unrolled network consisting of the layers $W_{xh}$. This
network is also bipartite and of size $(k+1)m$. Let us denote its weighted
adjacency matrix by $A$. The eigenvalues of $A$ can be obtained using the
eigenvalues of $B$ in certain cases. It is clear that $A$ is a
$(k+1)m\times(k+1)m$ matrix and

$A=\begin{pmatrix}\mbox{0}_{m,m}&B&\mbox{0}_{m,m}&\cdots&\mbox{0}_{m,m}\\\
B^{T}&\mbox{0}_{m,m}&B&\cdots&\mbox{0}_{m,m}\\\
\mbox{0}_{m,m}&B^{T}&\mbox{0}_{m,m}&\cdots&\mbox{0}_{m,m}\\\
\vdots&\ddots&\ddots&\ddots&\vdots\\\
\mbox{0}_{m,m}&\cdots&B^{T}&\mbox{0}_{m,m}&B\\\
\mbox{0}_{m,m}&\cdots&\mbox{0}_{m,m}&B^{T}&\mbox{0}_{m,m}\\\ \end{pmatrix}.$

$A$ is a block tridiagonal Toeplitz matrix. The eigenvalues of a tridiagonal
Toeplitz matrix are well known. Closed form expressions for eigenvalues of
general block tridiagonal Toeplitz matrices are unknown. However, eigenvalue
approximations exist see Nakatsukasa (2012). A particular case is when $B$ is
symmetric i.e., $B=B^{T}$. For these type of network structures, the
eigenvalues take the form,

$\left\\{2\lambda^{(B)}_{i}\cos\left(\frac{\pi
j}{k+2}\right)\right\\}_{i=1\ldots m,j=1\ldots k+1}.$

See Abderramán Marrero & Aiat Hadj (2020).

###  4.2 Relation between spectral Gap and accuracy

The hypothesis for our study is that the preservation of expander properties
of the recurrent neural network and LSTM is related to performance of these
networks. More precisely, we claim that the network classification accuracy is
high for the pruned sparse networks when the values of $\Delta{R}$ and
$\Delta{S}$ (Equation 1) are positive. Further pruning leads to drastic drop
in prediction accuracy.

##  5 Experimental Results

###  5.1 Datasets

We provide results for three benchmark sequential data, namely, the sequential
MNIST, the sequential CIFAR10, and the Google speech command data. For the
sequential MNIST data Le et al. (2015), each row of a $28\times 28$ MNIST
digit grey-scale image is presented as input at a single time step, leading to
a sequence of length $28$ for the entire image. The rows are presented in
scanline order. The task is to classify the sequence into one of the ten
classes. We have considered this data set as both long and short term
dependencies characterize the classification task. We also present results for
noisy MNIST sequence data. In the noisy data a Gaussian noise with zero mean
and $\sigma$ variance is added to $p$ fraction of the sequence points
(pixels). We considered $p=0.20$ and $\sigma=0.15,0.30,0.45,0.60$,
respectively to study the effect of varying degree of noise.

The CIFAR-10 dataset consists of 60000 $32$ colour images in 10 classes, with
6000 images per class. The sequential CIFAR-10 Chang et al. (2017) is
constructed by a similar process in scanline order from the CIFAR-10 images.
There are 50000 training images and 10000 test images. In a similar manner of
the MNIST data, we study noisy CIFAR-10 images too.

The Google Speech Commands dataset Warden (2018) has 65000 one-second long
utterances of 10 short words, by thousands of different people. It is an audio
dataset, with a sampling rate of 16kHz, of 35 spoken words designed to help
train and evaluate keyword spotting systems in the presence of background
noise or unrelated speech. The keywords e.g., backward, bed, bird, cat, dog,
down, eight, five, follow, forward, four, go, happy, house, learn, left,
marvin, nine, no, off, on, one, right, seven, sheila, six, stop, three, tree,
two, up, visual, wow, yes, zero, are commands used by a voice assistant. Mel-
scale spectogram extracted from the raw signal are used as input features. The
sequence length of the mel-scale input feature is $k=400$. If we set the
sequence length to $k$, then the input is passed as a $16000/k$ size vector at
each time step (and the number of time steps is $k$). The dataset is used as a
benchmark evaluate the performance of RNN/LSTM on long sequences.

###  5.2 Setup

Experiments are performed for both RNN and LSTM. The hyperparameters for the
network are shown in Table 1. Dense output layers are considered for each
network. Classification accuracy on a test data ($20$%) is used as the
performance measure of the network. While we perform experiments with both RNN
and LSTM for the MNIST data, only the LSTM is studied for the CIFAR-10 and
Google Speech Command datasets. This is because the accuracy of RNN is quite
low for these datasets.

Table 1: Hyperparameters for the experiments. Learning rate | $0.001$
---|---
Training epochs | $20$
Pruning epochs | $20$
Batch size | $100$
Optimizer | Adams
Initialization | Kaiming uniform
Activation | tanh
Loss | Cross entropy
Hidden nodes | 128

###  5.3 Results

The goal of our experiments is to study the effect of preserving expander
graph properties on the performance of sparse RNN and LSTM. Iterative
magnitude pruning Blalock et al. (2020) over a number of epochs is performed
to sparsify the networks. The weights between input to hidden layers, feedback
layers, and hidden to output layers are represented as $W_{xh}$, $W_{hh}$, and
$W_{hy}$ each representing a bipartite graph. We only prune the $W_{xh}$ and
$W_{hh}$ layers, leaving out the dense $W_{hy}$ layer unchanged. The time
unfolded networks effectively contains $k$ identical copies of the $W_{hh}$
bipartite graph, where $k$ is the input sequence length.

Preservation of expander property is characterized by the spectral gap of the
bipartite graphs of RNN layers. The spectral gap is obtained by Equation 1.
Spectral gaps for both unweighted ($\Delta_{R}$, $\Delta_{S}$ and weighted
($\Delta_{S}$) bipartite graph representation are reported. We plot the
variation in test set accuracy with the $q$% remaining fraction of network
weights for RNN and LSTM along with the spectral gaps $\Delta_{S}$ and
$\Delta_{R}$ (Equation 1). We present the following combination of results:

  * •

Dataset: MNIST, CIFAR-10, Google speech command

  * •

Architecture: RNN (MNIST), LSTM (all)

  * •

Graph representation: Weighted, Unweighted

  * •

Layer: $W_{xh}$, $W_{hh}$

  * •

Spectral gap: $\Delta_{S}$ (weighted, unweighted), $\Delta_{R}$ (unweighted)

Figures 1 and 2 shows the variation for RNN considering the unweighted and
weighted adjacency matrices respectively for the MNIST dataset. Similar plots
are shown for LSTM in Figures 3 and 4. Both clean and noisy input sequences
are considered in the plots.

The variation of spectral gaps for the CIFAR10 data for LSTM is shown in
Figures 5 and 6 for unweighted and weighted graph representations. Next, we
consider the speech command dataset with much longer sequence length of
$k=400$. The variation in spectral gaps for LSTM are shown in Figures 7 and 8
for unweighted and weighted graphs.

It is observed from the figures that as long as the expander graph property
holds for the networks the test set classification accuracy is almost
preserved as compared to the unpruned network. Whereas the accuracy starts
dropping when these properties are lost.

For the MNIST dataset, in the RNN, the $W_{xh}$ weights lose the Ramanujan
property at a remaining edge percentage of $50.0$%, and the $W_{hh}$ weights
lose the property at $50.0$% and $35.0$% for the unweighted graph
representation (Figure 1) and at $2.9$% remaining weights (Figure 2)
considering the weighted graph representation. For LSTM with the MNIST
dataset, with an unweighted representation, the remaining weights are $50.0$%
($W_{xh}$) and $35.0$% ($W_{hh}$) (Figure 3) when the expander graph property
is lost. The $W_{xh}$ weights lose the Ramanujan property at a remaining edge
percentage of $50.0$%, and the $W_{hh}$ weights lose the property at $12.9$%
remaining weights (Figure 4) for the weighted graph representation, for the
same dataset,

LSTM on the CIFAR-10 datset has a zero crossing of spectral gaps at a
remaining edge percentage of $47.0$% ($W_{xh}$) and $12.9$% ($W_{hh}$) for the
unweighted representation (Figure 5). The values are $16.9$%, for $W_{xh}$ and
$18.0$% for $W_{hh}$ (Figure 6) for the weighted graph representation. For the
Google speech command dataset, having a longer sequence length, we observe
that the zero crossing for the unweighted representation is at remaining
weight percentage of $50$% ($W_{xh}$) and $35$% ($W_{hh}$) for the unweighted
representation (Figure 7), and $14.9$% for both $W_{xh}$ and $W_{hh}$ in the
weighted graph representation (Figure 8).

We have added various noise quantities to the MNIST and CIFAR-10 dataset as
described in Section 5.1. The Google speech command data contains natural
background noise. We observe from Figures 1-6 that the degradation in
performance with loss of expander graph property becomes even more prominent
as noise quantity increases. This reinstates the fact that expander graph
property is crucial for noise robustness of the networks.

We observe in most of our experiments with RNN that the $W_{xh}$ layer lose
the expander graph property before the $W_{hh}$ layer. This points to the fact
that $W_{xh}$ edges play more significant role as compared to the $W_{hh}$.
Similar, results has been reported in Averbeck (2022).

The value of $\Delta_{S}$ oscillates for RNN (Figure 2) for very sparse
networks. This may be due the approximation errors for very sparse networks.
Other oscillations of test accuracy at lower lesser sparsity may be attributed
to underfitting or overfitting that we do not optimize.

(a) $W_{xh}$

(b) $W_{hh}$

Figure 1: Variation in test set accuracy (left vertical axis) and spectral gap
($\Delta_{S}$, $\Delta_{R}$) (right vertical axis) on sequence MNIST data for
RNN with remaining edges percentage $q$, considering unweighted graph
representation. The vertical lines shows the first zero crossing of
$\Delta_{S}$, $\Delta_{R}$.

(a) $W_{xh}$

(b) $W_{hh}$

Figure 2: Variation in test set accuracy and spectral gap ($\Delta_{S}$) on
sequence MNIST data for RNN with remaining edges percentage, considering
weighted graph representation. The vertical line shows the first zero crossing
of $\Delta_{S}$.

(a) $W_{xh}$

(b) $W_{hh}$

Figure 3: Variation in test set accuracy and spectral gap ($\Delta_{S}$,
$\Delta_{R}$) on sequence MNIST data for LSTM with remaining edges percentage,
considering unweighted graph representation. The vertical lines shows the
first zero crossing of $\Delta_{S}$, $\Delta_{R}$.

(a) $W_{xh}$

(b) $W_{hh}$

Figure 4: Variation in test set accuracy and spectral gap ($\Delta_{S}$) on
sequence MNIST data for LSTM with remaining edges percentage, considering
weighted representation. The vertical line shows the first zero crossing of
$\Delta_{S}$.

(a) $W_{xh}$

(b) $W_{hh}$

Figure 5: Variation in test set accuracy and spectral gap ($\Delta_{S}$,
$\Delta_{R}$) on sequence CIFAR-10 data for LSTM with remaining edges
percentage, considering unweighted graph representation. The vertical lines
shows the first zero crossing of $\Delta_{S}$, $\Delta_{R}$.

(a) $W_{xh}$

(b) $W_{hh}$

Figure 6: Variation in test set accuracy and spectral gap ($\Delta_{S}$) on
sequence CIFAR-10 data for LSTM with remaining edges percentage, considering
weighted representation. The vertical line shows the first zero crossing of
$\Delta_{S}$.

(a) $W_{xh}$

(b) $W_{hh}$

Figure 7: Variation in test set accuracy and spectral gap ($\Delta_{S}$,
$\Delta_{R}$) on Google speech command data for LSTM with remaining edges
percentage, considering unweighted graph representation. The vertical lines
shows the first zero crossing of $\Delta_{S}$, $\Delta_{R}$.

(a) $W_{xh}$

(b) $W_{hh}$

Figure 8: Variation in test set accuracy and spectral gap ($\Delta_{S}$) on
Google speech command data for LSTM with remaining edges percentage,
considering weighted representation. The vertical line shows the first zero
crossing of $\Delta_{S}$.

##  6 Conclusion

In this paper we study the expansion properties of sparse RNN and LSTM
networks obtained by pruning. We empirically observe that the sparse networks
that satisfy the expander graph properties preserve the classification
accuracy of the unpruned network. We also show using results from linear
algebra that bounds on the spectral gaps for recurrent network structures can
be obtained from the corresponding bounds for the layer bi-partite graphs. We
present results for the sequential MNIST, CIFAR-10, and the Google speech
command data with and without noise for both RNN and LSTM pruned networks
using iterative magnitude pruning to support our claims.

## References

  * Abderramán Marrero & Aiat Hadj (2020) J. Abderramán Marrero and D.A. Aiat Hadj.  Improving formulas for the eigenvalues of finite block-toeplitz tridiagonal matrices.  _Applied Mathematics and Computation_ , 382:125324, Oct 2020.  URL http://dx.doi.org/10.1016/j.amc.2020.125324.
  * Alon & Milman (1985) Noga Alon and Vitali D Milman.  $\lambda$1, isoperimetric inequalities for graphs, and superconcentrators.  _Journal of Combinatorial Theory, Series B_ , 38(1):73–88, 1985.
  * Averbeck (2022) Bruno B. Averbeck.  Pruning recurrent neural networks replicates adolescent changes in working memory and reinforcement learning.  _Proceedings of the National Academy of Sciences_ , 119(22):e2121331119, 2022.  doi: 10.1073/pnas.2121331119.  URL https://www.pnas.org/doi/abs/10.1073/pnas.2121331119.
  * Biswas (2019) Arindam Biswas.  On a cheeger type inequality in cayley graphs of finite groups.  _European Journal of Combinatorics_ , 81:298–308, October 2019.  doi: 10.1016/j.ejc.2019.06.009.  URL https://doi.org/10.1016/j.ejc.2019.06.009.
  * Biswas & Saha (2021) Arindam Biswas and Jyoti Prakash Saha.  A Cheeger type inequality in finite Cayley sum graphs.  _Algebraic Combinatorics_ , 4(3):517–531, 2021.  doi: 10.5802/alco.166.  URL https://alco.centre-mersenne.org/articles/10.5802/alco.166/.
  * Biswas & Saha (2021) Arindam Biswas and Jyoti Prakash Saha.  Expansion in Cayley graphs, Cayley sum graphs and their twists.  _arXiv e-prints_ , art. arXiv:2103.05935, March 2021.  doi: 10.48550/arXiv.2103.05935.
  * Biswas & Saha (2022) Arindam Biswas and Jyoti Prakash Saha.  Spectra of twists of cayley and cayley sum graphs.  _Advances in Applied Mathematics_ , 132:102272, January 2022.  doi: 10.1016/j.aam.2021.102272.  URL https://doi.org/10.1016/j.aam.2021.102272.
  * Biswas & Saha (2023) Arindam Biswas and Jyoti Prakash Saha.  A spectral bound for vertex-transitive graphs and their spanning subgraphs.  _Algebraic Combinatorics_ , 6(3):689–706, 2023.  doi: 10.5802/alco.278.  URL https://alco.centre-mersenne.org/articles/10.5802/alco.278/.
  * Blalock et al. (2020) Davis W. Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John V. Guttag.  What is the state of neural network pruning?  _CoRR_ , abs/2003.03033, 2020.
  * Bourely et al. (2017) Alfred Bourely, John Patrick Boueri, and Krzysztof Choromonski.  Sparse neural networks topologies.  _CoRR_ , abs/1706.05683, 2017.
  * Breuillard et al. (2015) Emmanuel Breuillard, Ben Green, Robert Guralnick, and Terence Tao.  Expansion in finite simple groups of lie type.  _Journal of the European Mathematical Society_ , 17(6):1367–1434, 2015.  doi: 10.4171/jems/533.  URL https://doi.org/10.4171/jems/533.
  * Béna & Goodman (2021) Gabriel Béna and Dan F. M. Goodman.  Extreme sparsity gives rise to functional specialization.  _arXiv_ , 2021.
  * Chang et al. (2017) Shiyu Chang, Yang Zhang, Wei Han, Mo Yu, Xiaoxiao Guo, Wei Tan, Xiaodong Cui, Michael Witbrock, Mark A Hasegawa-Johnson, and Thomas S Huang.  Dilated recurrent neural networks.  _Advances in neural information processing systems_ , 30, 2017.
  * Chatzikonstantinou et al. (2021) Christos Chatzikonstantinou, Dimitrios Konstantinidis, Kosmas Dimitropoulos, and Petros Daras.  Recurrent neural network pruning using dynamical systems and iterative fine-tuning.  _Neural Networks_ , 143:475–488, 2021.
  * Chen et al. (2023) Zhuangzhi Chen, Jingyang Xiang, Yao Lu, Qi Xuan, Zhen Wang, Guanrong Chen, and Xiaoniu Yang.  Rgp: Neural network pruning through regular graph with edges swapping.  _IEEE Transactions on Neural Networks and Learning Systems_ , pp. 1–13, 2023.  doi: 10.1109/TNNLS.2023.3280899.
  * Choromanski et al. (2022) Krzysztof Choromanski, Han Lin, Haoxian Chen, Tianyi Zhang, Arijit Sehanobish, Valerii Likhosherstov, Jack Parker-Holder, Tamas Sarlos, Adrian Weller, and Thomas Weingarten.  From block-toeplitz matrices to differential equations on graphs: towards a general theory for scalable masked transformers.  In _Proceedings of the 39th International Conference on Machine Learning_ , pp. 3962–3983, 2022.
  * Chung (2016) Fan Chung.  A generalized alon-boppana bound and weak ramanujan graphs.  _The Electronic Journal of Combinatorics_ , 23(3), July 2016.  doi: 10.37236/5933.  URL https://doi.org/10.37236/5933.
  * Ding et al. (2021) Shaojin Ding, Tianlong Chen, and Zhangyang Wang.  Audio lottery: Speech recognition made ultra-lightweight, noise-robust, and transferable.  In _International Conference on Learning Representations_ , 2021.
  * Dodziuk (1984) Jozef Dodziuk.  Difference equations, isoperimetric inequality and transience of certain random walks.  _Transactions of the American Mathematical Society_ , 284(2):787–794, 1984.
  * Feng et al. (2022) Fan Feng, Lu Hou, Qi She, Rosa HM Chan, and James T Kwok.  Power law in deep neural networks: Sparse network generation and continual learning with preferential attachment.  _IEEE Transactions on Neural Networks and Learning Systems_ , 2022.
  * Frankle & Carbin (2018) Jonathan Frankle and Michael Carbin.  The lottery ticket hypothesis: Finding sparse, trainable neural networks.  _arXiv preprint arXiv:1803.03635_ , 2018.
  * Furuya et al. (2022) Takashi Furuya, Kazuma Suetake, Koichi Taniguchi, Hiroyuki Kusumoto, Ryuji Saiin, and Tomohiro Daimon.  Spectral pruning for recurrent neural networks.  In _Proceedings of The 25th International Conference on Artificial Intelligence and Statistics_ , pp. 3458–3482, 2022.
  * Giles & Omlin (1994) C Lee Giles and Christian W Omlin.  Pruning recurrent neural networks for improved generalization performance.  _IEEE transactions on neural networks_ , 5(5):848–851, 1994.
  * Grossberg (2013) Stephen Grossberg.  Recurrent neural networks.  _Scholarpedia_ , 8(2):1888, 2013.
  * Han et al. (2017) Song Han, Junlong Kang, Huizi Mao, Yiming Hu, Xin Li, Yubin Li, Dongliang Xie, Hong Luo, Song Yao, Yu Wang, et al.  Ese: Efficient speech recognition engine with sparse lstm on fpga.  In _Proceedings of the 2017 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays_ , pp. 75–84, 2017.
  * He et al. (2016) K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 770–778, 2016.
  * Hochreiter & Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  _Neural computation_ , 9(8):1735–1780, 1997.
  * Hod et al. (2021) Shlomi Hod, Stephen Casper, Daniel Filan, Cody Wild, Andrew Critch, and Stuart Russell.  Detecting modularity in deep neural networks.  _arXiv preprint arXiv:2110.08058_ , 2021.
  * Hoory (2005) Shlomo Hoory.  A lower bound on the spectral radius of the universal cover of a graph.  _Journal of Combinatorial Theory, Series B_ , 93(1):33–43, January 2005.  doi: 10.1016/j.jctb.2004.06.001.  URL https://doi.org/10.1016/j.jctb.2004.06.001.
  * Hoory et al. (2006) Shlomo Hoory, Nathan Linial, and Avi Wigderson.  Expander graphs and their applications.  _Bulletin of the American Mathematical Society_ , 43(4):439–561, 2006.
  * Khona et al. (2022) Mikail Khona, Sarthak Chandra, Joy J. Ma, and Ila Fiete.  Winning the lottery with neurobiology: faster learning on many cognitive tasks with fixed sparse RNNs.  _arXiv_ , 2022.
  * Krizhevsky et al. (2017) A. Krizhevsky, I. Sutskever, and G. E. Hinton.  Imagenet classification with deep convolutional neural networks.  _Communications of the ACM_ , 60(6):84–90, 2017.
  * Le et al. (2015) Quoc V Le, Navdeep Jaitly, and Geoffrey E Hinton.  A simple way to initialize recurrent networks of rectified linear units.  _arXiv preprint arXiv:1504.00941_ , 2015.
  * LeCun et al. (1998) Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner.  Gradient-based learning applied to document recognition.  _Proceedings of the IEEE_ , 86(11):2278–2324, 1998.
  * Li et al. (2022a) Guiying Li, Peng Yang, Chao Qian, Richang Hong, and Ke Tang.  Stage-wise magnitude-based pruning for recurrent neural networks.  _IEEE Transactions on Neural Networks and Learning Systems_ , pp. 1–15, 2022a.
  * Li et al. (2022b) Guiying Li, Peng Yang, Chao Qian, Richang Hong, and Ke Tang.  Stage-wise magnitude-based pruning for recurrent neural networks.  _IEEE Transactions on Neural Networks and Learning Systems_ , pp. 1–15, 2022b.  doi: 10.1109/TNNLS.2022.3184730.
  * Li et al. (2018) Zhe Li, Shuo Wang, Caiwen Ding, Qinru Qiu, Yanzhi Wang, and Yun Liang.  Efficient recurrent neural networks using structured matrices in fpgas.  _CoRR_ , abs/1803.07661, 2018.
  * Liu et al. (2019) Shiwei Liu, Decebal Constantin Mocanu, and Mykola Pechenizkiy.  Intrinsically sparse long short-term memory networks.  _arXiv preprint arXiv:1901.09208_ , 2019.
  * McDonald & Shokoufandeh (2019) Andrew W. E McDonald and Ali Shokoufandeh.  Sparse super-regular networks.  In _2019 18th IEEE International Conference On Machine Learning And Applications (ICMLA)_ , pp. 1764–1770, 2019.
  * Nakatsukasa (2012) Yuji Nakatsukasa.  Eigenvalue perturbation bounds for Hermitian block tridiagonal matrices, Jan 2012.  URL http://dx.doi.org/10.1016/j.apnum.2011.09.010.
  * Narang et al. (2017) Sharan Narang, Greg Diamos, Shubho Sengupta, and Erich Elsen.  Exploring sparsity in recurrent neural networks.  In _International Conference on Learning Representations_ , 2017.  URL https://openreview.net/forum?id=BylSPv9gx.
  * Pal et al. (2022) Bithika Pal, Arindam Biswas, Sudeshna Kolay, Pabitra Mitra, and Biswajit Basu.  A study on the ramanujan graph property of winning lottery tickets.  In _International Conference on Machine Learning, ICML_ , volume 162, pp. 17186–17201, 2022.
  * Prabhu et al. (2018) Ameya Prabhu, Girish Varma, and Anoop Namboodiri.  Deep expander networks: Efficient deep networks from graph theory.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , September 2018.
  * Sermanet et al. (2013) P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, and Y. LeCun.  Overfeat: Integrated recognition, localization and detection using convolutional networks.  _arXiv preprint arXiv:1312.6229_ , 2013.
  * Simonyan & Zisserman (2014) K. Simonyan and A. Zisserman.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_ , 2014.
  * Szegedy et al. (2015) C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich.  Going deeper with convolutions.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 1–9, 2015.
  * Thom & Palm (2013) Markus Thom and Günther Palm.  Sparse activity and sparse connectivity in supervised learning.  _Journal of Machine Learning Research_ , 14(33):1091–1143, 2013.
  * Warden (2018) Pete Warden.  Speech commands: A dataset for limited-vocabulary speech recognition, 2018.
  * Ying et al. (2019) C. Ying, A. Klein, E. Christiansen, E. Real, K. Murphy, and F. Hutter.  Nas-bench-101: Towards reproducible neural architecture search.  In _International Conference on Machine Learning_ , pp. 7105–7114. PMLR, 2019.
  * You et al. (2020) Jiaxuan You, Jure Leskovec, Kaiming He, and Saining Xie.  Graph structure of neural networks.  In _International Conference on Machine Learning_ , pp. 10881–10891, 2020.
  * Zeiler & Fergus (2014) M. D. Zeiler and R. Fergus.  Visualizing and understanding convolutional networks.  In _European conference on computer vision_ , pp. 818–833. Springer, 2014.
  * Zhang & Stadie (2019) Matthew Shunshi Zhang and Bradly C. Stadie.  One-shot pruning of recurrent neural networks by jacobian spectrum evaluation.  _CoRR_ , abs/1912.00120, 2019.
  * Zhu & Gupta (2017) Michael Zhu and Suyog Gupta.  To prune, or not to prune: exploring the efficacy of pruning for model compression.  _arXiv preprint arXiv:1710.01878_ , 2017.
  * Zoph et al. (2018) B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le.  Learning transferable architectures for scalable image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 8697–8710, 2018.
  * Zoph & Le (2017) Barret Zoph and Quoc V. Le.  Neural architecture search with reinforcement learning.  In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net, 2017.  URL https://openreview.net/forum?id=r1Ue8Hcxg.