# Flat bands and magnetism in $\bm{\mathrm{Fe_{4}GeTe_{2}}}$ and
$\bm{\mathrm{Fe_{5}GeTe_{2}}}$ due to bipartite crystal lattices

Fuyi Wang  Haijun Zhang∗ National Laboratory of Solid State Microstructures,
School of Physics, Nanjing University, Nanjing 210093, China.  Collaborative
Innovation Center of Advanced Microstructures, Nanjing University, Nanjing
210093, China.

###### Abstract

$\mathrm{Fe_{n=4,5}GeTe_{2}}$ exhibits quasi-two-dimensional properties as a
promising candidate for a near-room-temperature ferromagnet, which has
attracted great interest. In this work, we notice that the crystal lattice of
$\mathrm{Fe_{n=4,5}GeTe_{2}}$ can be approximately regarded as being stacked
by three bipartite crystal lattices. By combining the model Hamiltonians of
bipartite crystal lattices and first-principles calculations, we investigate
the electronic structure and the magnetism of $\mathrm{Fe_{n=4,5}GeTe_{2}}$.
We conclude that flat bands near the Fermi level originate from the bipartite
crystal lattices and that these flat bands are expected to lead to the
itinerant ferromagnetism in $\mathrm{Fe_{n=4,5}GeTe_{2}}$. Interestingly, we
also find that the magnetic moment of the Fe5 atom in
$\mathrm{Fe_{5}GeTe_{2}}$ is distinct from the other Fe atoms and is sensitive
to the Coulomb interaction $U$ and external pressure. These findings may be
helpful to understand the exotic magnetic behavior of
$\mathrm{Fe_{n=4,5}GeTe_{2}}$.

††preprint: APS/123-QED

##  I Introduction

In recent years, $\mathrm{Fe_{n=4,5}GeTe_{2}}$ have been discovered as van der
Waals (vdW) itinerant ferromagnets with a high Curie temperature ${T_{c}}$ [1,
2, 3, 4, 5, 6, 7, 8, 9, 10]. They are promising for spintronic applications
due to the near-room-temperature ferromagnetism, magnetic anisotropy, and high
electric conductivity [4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 3, 14, 15, 16, 17,
18, 19, 20, 21, 22]. $\mathrm{Fe_{n=4,5}GeTe_{2}}$ also exhibit many
interesting properties, such as the Kondo effect [23], anomalous Hall effect
(AHE) [24, 25], butterfly-shaped magnetoresistance [26], controllable
topological magnetic transformations [27], and skyrmionic spin structures up
to the room temperature [28, 29]. However, the underlying physics of the
magnetic behaviors of $\mathrm{Fe_{n=4,5}GeTe_{2}}$ has not been well
understood.

Figure 1: Crystal structure and bipartite crystal lattices. (a) Crystal
structures of $\mathrm{Fe_{4}GeTe_{2}}$ with primitive lattice cell in black
solid line. (b) Top views of three BCLs of $\mathrm{Fe_{4}GeTe_{2}}$, labeled
$\mathrm{L_{1}}$, $\mathrm{L_{2}}$ and $\mathrm{L_{3}}$. The first BCL
($\mathrm{L_{1}}$) is a honeycomb lattice. The center BCL ($\mathrm{L_{2}}$)
is a dice lattice. The third BCL ($\mathrm{L_{3}}$) is equivalent to the
$\mathrm{L_{1}}$ due to the inversion symmetry. (c) The side view of a
septuple layer of $\mathrm{Fe_{4}GeTe_{2}}$. The triangle lattice has three
different stacked positions denoted as $A$, $B$ and $C$. (d) Crystal
structures of $\mathrm{Fe_{5}GeTe_{2}}$ with primitive lattice cell in black
solid line. (e) Top view of three BCLs of $\mathrm{Fe_{5}GeTe_{2}}$. The first
BCL ($\mathrm{L_{1}}$) and the center BCL ($\mathrm{L_{2}}$) are dice
lattices. The third BCL ($\mathrm{L_{3}}$) is a honeycomb lattice. (f) The
side view of the octuple layer of $\mathrm{Fe_{5}GeTe_{2}}$.

To investigate the electronic structure and the magnetism of
$\mathrm{Fe_{n=4,5}GeTe_{2}}$, we approximately decompose their crystal
lattices into three basic layers due to the layered structure. Each basic
layer contains at least one Fe layer and one Ge or Te layer, as shown in Fig.
1. Interestingly, we notice that these basic layers of
$\mathrm{Fe_{n=4,5}GeTe_{2}}$ can be approximately regarded as bipartite
crystal lattices (BCLs) which can be divided into two sublattices with
negligible intra-sublattice hopping [30, 31], since the hopping primarily
occurs between the Fe and Ge/Te sublattices. We determine that the stacked
BCLs in $\mathrm{Fe_{n=4,5}GeTe_{2}}$ give rise to flat bands [32, 33, 34, 35]
which may account for the ferromagnetism observed in these materials. It is
worth mentioning that the decomposition of BCLs for
$\mathrm{Fe_{n=4,5}GeTe_{2}}$ is a rough approximation, and the hoppings
between adjacent BCLs still require careful consideration.

In this work, we construct model Hamiltonians for the stacked BCLs of
$\mathrm{Fe_{n=4,5}GeTe_{2}}$. We determine that the flat bands can be
attributed to the BCLs, based on these model Hamiltonians. We also demonstrate
that the itinerant ferromagnetism in these materials arises from the nearly
flat bands near the Fermi energy driven by the Coulomb interaction $U$, which
is known as flat-band ferromagnetism [36, 37, 38, 39, 40, 41, 42]. The BCL-
induced ferromagnetism primarily depends on the lattice structure, orbitals,
and electron filling number. We expect that this conclusion could be extended
to other vdW ferromagnets. Furthermore, by combining the model Hamiltonians
and first-principles calculations, we find that the magnetic moment of Fe5 in
$\mathrm{Fe_{5}GeTe_{2}}$ is sensitive to both ${U}$ and external pressures.
The pressure-tunable magnetic moment transitions in $\mathrm{Fe_{5}GeTe_{2}}$
might be experimentally observed.

##  II Methods

First-principles calculations are carried out using the Perdew-Burke-
Ernzerhof-type (PBE) generalized gradient approximation (GGA) [43] of the
density functional theory (DFT), using the Vienna ab initio simulation package
(VASP) [44, 45, 46]. The $\mathrm{GGA+U}$ method with $U=3.0\ eV$ is employed
to treat the Fe’s ${d}$ orbitals. Self-consistent calculations including the
spin-orbit coupling (SOC) are performed. A kinetic energy cutoff of $500\ eV$
is used, and the $10\times 10\times 10$ k-point mesh is taken for the bulk
calculations. The inner atomic positions are obtained via full relaxation with
a total energy tolerance of ${10^{-6}}$ eV. The experimental lattice constants
of $\mathrm{Fe_{4}GeTe_{2}}$ ($a=4.03\ \mathrm{\AA}$, and $c=29.08\
\mathrm{\AA}$) [4] and $\mathrm{Fe_{5}GeTe_{2}}$ ($a=4.04\ \mathrm{\AA}$, and
$c=29.19\ \mathrm{\AA}$)[5] are adopted. The Wannier-based model Hamiltonians
are obtained from the projection of the $p$ orbitals of Bi and Te and the $d$
orbitals of Fe through employing the WANNIER90 package [47, 48, 49].

##  III Crystal structure and orbitals

###  III.1 The bipartite crystal lattice

$\mathrm{Fe_{4}GeTe_{2}}$ has a space group of ${R{\bar{3}}m(166)}$ that
includes an inversion symmetry. The Te (Fe1′, Fe2) atoms are related to the
Te′ (Fe1, Fe2′) atoms via an inversion operation, with the Ge atom serving as
the inversion center. The unit cell of $\mathrm{Fe_{4}GeTe_{2}}$ has a
septuple-layer structure, as shown in Fig. 1. We can decompose the unit cell
of $\mathrm{Fe_{4}GeTe_{2}}$ into three BCLs stacked along the $z$ direction,
denoted as $\mathrm{L_{1}}$, $\mathrm{L_{2}}$, and $\mathrm{L_{3}}$. The
$\mathrm{L_{1}}$ BCL comprises the Fe1 sublattice and the Te sublattice, while
the $\mathrm{L_{3}}$ BCL comprises the Fe1′ sublattice and the Te′ sublattice.
The $\mathrm{L_{1}}$ and $\mathrm{L_{3}}$ BCLs are related through the
inversion symmetry. The $\mathrm{L_{2}}$ BCL consists of the Fe2(Fe2′)
sublattice and the Ge sublattice, which can be viewed as a dice lattice [30,
50, 51, 52] with an inversion symmetry, where the Ge atom acts as the
inversion center.

$\mathrm{Fe_{5}GeTe_{2}}$ belongs to the ${R3m(160)}$ space group. The lattice
of $\mathrm{Fe_{5}GeTe_{2}}$ can be obtained by inserting a Fe5 layer between
the Fe1′ and Te1 layers of $\mathrm{Fe_{4}GeTe_{2}}$ [4], as depicted in Fig.
1(c, f). The inserted Fe5 layer breaks the inversion symmetry, resulting in
the unequivalence of Fe1 and Fe1′, as well as Fe2 and Fe2′. Consequently, Fe1′
and Fe2′ in $\mathrm{Fe_{5}GeTe_{2}}$ are renamed as Fe4 and Fe3,
respectively. The unit cell of $\mathrm{Fe_{5}GeTe_{2}}$ has an octuple-layer
structure, which can also be approximately decomposed into three BCLs stacked
along the $z$ direction. The $\mathrm{L_{1}}$ BCL in $\mathrm{Fe_{5}GeTe_{2}}$
is a dice lattice consisting of the Fe4 and Fe5 sublattice and the Te1
sublattice, though it is just a quasi-BCL due to the non-negligible nearest
hopping between Fe4 and Fe5. The $\mathrm{L_{2}}$ BCL in
$\mathrm{Fe_{5}GeTe_{2}}$ is a dice lattice but lacks the inversion symmetry.
Lastly, the $\mathrm{L_{3}}$ BCL in $\mathrm{Fe_{5}GeTe_{2}}$ is a honeycomb
lattice which is almost identical to the $\mathrm{L_{3}}$ BCL in
$\mathrm{Fe_{4}GeTe_{2}}$.

###  III.2 Orbitals and the site symmetry

The ferromagnetism in $\mathrm{Fe_{n=4,5}GeTe_{2}}$ is mainly due to the
partially filled $d$ orbitals of Fe, whereas Te and Ge do not exhibit major
magnetic behavior. Therefore, it is important to analyze the splitting of Fe’s
$d$ orbitals to reveal the underlying mechanism of the ferromagnetism.

Both $\mathrm{Fe_{4}GeTe_{2}}$ and $\mathrm{Fe_{5}GeTe_{2}}$ have the $C_{3v}$
site symmetry, which has three irreparable representations: two 1D irreps
${A_{1,2}}$ and a 2D irrep ${E}$. The ${p}$ and ${d}$ orbitals can be
classified based on the irreducible representations (irrep) of the site
symmetry group. Here, ${p_{z}}$ corresponds to ${A_{1}}$ irrep for ${p}$
orbitals of Te and Ge, while (${p_{x}}$, ${p_{y}}$) corresponds to ${E}$ irrep
and forms a doublet. Moreover, the $d$ orbitals of Fe are divided into a
singlet ${d_{z^{2}}}$ with ${A_{1}}$ irrep and two doublets (${d_{xz}}$,
${d_{yz}}$) and (${d_{xy}}$, ${d_{x^{2}-y^{2}}}$) with ${E}$ irrep. The
orbitals in doublets can be recombined as ${p_{x}\pm ip_{y}}$, ${d_{xz}\pm
id_{yz}}$, and ${d_{x^{2}-y^{2}}\pm id_{xy}}$, renamed according to their
quantum numbers of the angular momentum projection operator $\hat{l_{z}}$ as
${p_{m=\pm 1}}$, ${d_{m=\pm 1}}$, and ${d_{m=\pm 2}}$. Meanwhile, the singlet
orbitals are renamed as ${p_{m=0}}$ and ${d_{m=0}}$, respectively.

Table 1:  The character table of the point group ${C_{3v}}$ | ${E}$ | ${C_{3}(z)}$ | ${3\sigma_{z}}$ |  ${p}$ orbitals |  ${d}$ orbitals
---|---|---|---|---|---
${A_{1}}$ | 1 | 1 | 1 | ${p_{z}}$ | ${d_{z^{2}}}$
${A_{2}}$ | 1 | 1 | -1 | / | /
${E}$ | 2 | -1 | 0 | (${p_{x},p_{y}}$) | (${d_{xy},d_{x^{2}-y^{2}}}$); (${d_{xz},d_{yz}}$)
Figure 2:  Band structures by model Hamiltonians of $\mathrm{Fe_{4}GeTe_{2}}$.
(a, b) The band structures of $H_{m=0}$ and ${H_{m\neq 0}}$ in red and blue.
The dashed gray lines are the band structures calculated by first-principles
calculations (density functional theory, DFT). (c, d) Band structures by the
$H_{B_{1}}/H_{B_{3}}$ for $A^{\prime}(k)=0$ (c) and $A^{\prime}(k)\neq 0$ (d).
$H_{B_{1}}$ is equivalent to $H_{B_{3}}$ due to the inversion symmetry. The
projections of Fe1 are in green. (e, f) Band structures by the $H_{B_{2}}$ for
$A^{\prime}(k)=0$ (e) and $A^{\prime}(k)\neq 0$ (f). The projections of Fe1
are in orange.

##  IV Model Hamiltonians

###  IV.1 Construction of Model Hamiltonians

To understand the origin of flat bands, it is essential to formulate a model
Hamiltonian. Since the lattice structure of $\mathrm{Fe_{n=4,5}GeTe_{2}}$ can
be viewed as three stacked BCLs, we construct the tight-binding model
Hamiltonian $H_{tot}$ by placing the BCL Hamiltonians ${H_{L_{i}}}$ on the
diagonal. The general form of the tight-binding model Hamiltonian for
$\mathrm{Fe_{n=4,5}GeTe_{2}}$ is written as,

$H_{tot}(k)=\left(\begin{array}[]{ccc}H_{L_{1}}(k)&S_{12}(k)&S_{13}(k)\\\
S_{12}^{\dagger}(k)&H_{L_{2}}(k)&S_{23}(k)\\\
S_{13}^{\dagger}(k)&S^{\dagger}_{23}(k)&H_{L_{3}}(k)\end{array}\right)$ (1)

Figure 3:  Band structures by model Hamiltonians of $\mathrm{Fe_{5}GeTe_{2}}$.
(a, b) The band structures of $H_{m=0}$ and ${H_{m\neq 0}}$ in red and blue.
The dashed gray lines are the band structures calculated by first-principles
calculations. (c, d) Band structures by the $H_{B_{1}}$ for $A^{\prime}(k)=0$
(c) and $A^{\prime}(k)\neq 0$ (d). The projections of Fe4 and Fe5 are in blue
and red, respectively. (e, f) Band structures by the $H_{B_{2}}$ for
$A^{\prime}(k)=0$ (e) and $A^{\prime}(k)\neq 0$ (f). The projections of Fe4
and Fe5 are in orange and green, respectively. (g, h) Band structures by the
$H_{B_{3}}$ for $A^{\prime}(k)=0$ (g) and $A^{\prime}(k)\neq 0$ (h). The
projections of Fe1 are in purple.

where the ${S_{12}}$ and ${S_{23}}$ represent the hopping between adjacent
BCLs which usually have the same order of magnitude as the intra-BCL hoppings,
while the ${S_{13}}$ between the L1 and L3 BCLs is almost zero. Therefore, the
Hamiltonians of $\mathrm{L_{1}}$, $\mathrm{L_{2}}$, and $\mathrm{L_{3}}$ BCLs
cannot be treated independently. However, the hopping between adjacent BCLs
primarily occurs between orbitals along the ${z}$ direction, such as the
${p_{z}}$ and ${d_{z^{2}}}$ with ${m=0}$. Therefore, to simplify the model,
the orbitals can be categorized into two sets. The first set comprises all the
orbitals with ${m=0}$, while the second set consists of the remaining orbitals
with ${m\neq 0}$. By applying a unitary transformation, the original model
Hamiltonian is transformed to,

$H_{tot}(k)=\left(\begin{array}[]{cc}H_{m=0}(k)&S_{m}(k)\\\
S_{m}^{\dagger}(k)&H_{m\neq 0}(k)\end{array}\right)$ (2)

where the ${H_{m=0}(k)}$ and ${H_{m\neq 0}(k)}$ are the Hamiltonian with the
${m=0}$ orbitals and ${m\neq 0}$ orbitals respectively. ${S_{m}(k)}$ is the
hopping matrix between the ${m=0}$ and ${m\neq 0}$ orbitals. Fig. 2(b) and
Fig. 3(b) show that the band structures calculated by ${H_{m\neq 0}(k)}$ of
$\mathrm{Fe_{n=4,5}GeTe_{2}}$ can catch the main feature of the band
structures from the first-principles calculations, which validate the
partitioning of orbitals into ${m=0}$ and ${m\neq 0}$.

Since the hopping between adjacent BCLs is relatively weak for the ${m\neq 0}$
orbitals, the ${H_{m\neq 0}(k)}$ is given by,

$H_{m\neq 0}(k)=\left(\begin{array}[]{ccc}H_{B_{1}}(k)&S_{B_{12}}(k)&0\\\
S_{B_{12}}^{\dagger}(k)&H_{B_{2}}(k)&S_{B_{23}}(k)\\\
0&S^{\dagger}_{B_{23}}(k)&H_{B_{3}}(k)\end{array}\right)$ (3)

where ${H_{B_{i}}}$ is the Hamiltonian based on the ${m\neq 0}$ orbitals of
the Li BCL, and ${S_{B_{ij}}}$ is the hopping between the Li BCL and the Lj
BCL, which can be negligible (${S_{B_{ij}}}\approx 0$). Then, ${H_{m\neq
0}(k)}$ is further considered to made up of the three individual ${H_{B_{i}}}$
which is written as [31],

$H_{B_{i}}(k)=\left(\begin{array}[]{cc}A(k)&S(k)\\\
S^{\dagger}(k)&B(k)\end{array}\right)$ (4)

where ${A(k)}$/${B(k)}$ is a Hermitian matrix denoting the onsite energy and
intra-sublattice hopping and ${S(k)}$ denotes the inter-sublattice hopping for
each BCL. Since onsite energies lie on the diagonal, the matrix
$A^{\prime}(k)/B^{\prime}(k)$ obtained after removing the diagonal terms of
$A(k)/B(k)$ represents the intra-sublattice hoppings. As mentioned above, the
decomposition of BCLs of $\mathrm{Fe_{n=4,5}GeTe_{2}}$ is a rough
approximation due to the existence of the nonzero intra-sublattice hoppings
which leads to the dispersion of flat bands.

###  IV.2 Flat bands due to BCLs

Figure 4:  Electronic structures of $\mathrm{Fe_{4}GeTe_{2}}$ calculated by
first-principles calculations with ${U=3.0\ eV}$. (a) The non-SOC band
structure. (b) The SOC band structure. (c) The spin-polarized DOS without SOC.
(d) The Fermi surfaces with up spin are in red, while the Fermi surfaces with
down spin are in blue.

In general, a BCL Hamiltonian can induce ($N=N_{A}-N_{B}$) flat bands when
${N_{A}>N_{B}}$ [31], as shown in Appendix A. Here, $N_{A/B}$ denotes the
number of orbitals present on the $A/B$ sublattice. The emergence of flat
bands can be attributed to the destructive interference of wavefunctions
associated with the properties of the BCL [53, 54]. When ${N_{A}>N_{B}}$, the
hopping along different directions overlaps destructively at the $B$
sublattice, resulting in (${N_{A}-N_{B}}$) states solely on the $A$ sublattice
at every momentum $k$. Since there are no hoppings between states on the $A$
sublattice of BCL, these states form (${N_{A}-N_{B}}$) flat bands. However,
according to the proof in Appendix A, the crystal field splitting of orbitals
due to the crystal field effect and the intra-sublattice hoppings on $A$
sublattice may lead to slight bending or loss of degeneracy in the flat bands.
We will take into account the impact of the crystal field splitting and
$A^{\prime}(k)$ on the flat bands when performing calculations using these
model Hamiltonians.

We first analyze the flat bands according to the BCL Hamiltonian ${H_{B_{i}}}$
of $\mathrm{Fe_{n=4,5}GeTe_{2}}$. We first neglect the crystal field splitting
of the $d$ orbital due to the crystal field effect and the intra-sublattice
hoppings $A^{\prime}(k)$ for each BCL. The $\mathrm{L_{1}}$/$\mathrm{L_{3}}$
BCL of $\mathrm{Fe_{4}GeTe_{2}}$ and $\mathrm{L_{3}}$ BCL of
$\mathrm{Fe_{5}GeTe_{2}}$ are honeycomb lattices that consist of a Fe
sublattice (denoted as the $A$ sublattice) and a Te sublattice (denoted as the
$B$ sublattice). The $A$ sublattice comprises four degenerate $d$ orbitals,
while the $B$ sublattice comprises two degenerate $p$ orbitals. As a result,
the BCL Hamiltonian with (${N_{A}-N_{B}=2}$) degenerate flat bands. On the
other hand, the $\mathrm{L_{2}}$ BCL of $\mathrm{Fe_{4}GeTe_{2}}$ and
$\mathrm{L_{1}}$/$\mathrm{L_{2}}$ BCLs are dice lattices that consist of the
sublattice with two Fe (denoted as the $A$ sublattice) and a Te/Ge sublattice
(denoted as the $B$ sublattice). The $A$ sublattice contains eight degenerate
$d$ orbitals, while the $B$ sublattice contains two degenerate $p$ orbitals.
The BCL Hamiltonian for this dice lattice has (${N_{A}-N_{B}=6}$) degenerate
flat bands. Due to the relatively localized nature of the $d$ orbitals, we
anticipate that the intra-sublattice hopping ($A^{\prime}(k)$) will have small
magnitudes, thereby having a limited impact on the formation of flat bands.

Figure 5:  Magnetic properties of $\mathrm{Fe_{4}GeTe_{2}}$ calculated by
first-principles calculations. (a) The ${U}$ dependence of magnetic moments of
unequivalence Fe atoms. (b) The non-spin-polarized DOS.

Based on the model Hamiltonians, the flat bands are calculated, shown in Fig.
2(c, e) and Fig. 3(c, e, g) without considering the intra-sublattice hoppings
in $A$ sublattice ($A^{\prime}(k)=0$) and Fig. 2(d, f) and Fig. 3(d, f, h)
with considering the intra-sublattice hoppings($A^{\prime}(k)\neq 0$). The
clear flat bands have been shown in Fig. 2 and Fig. 3, though the
$A^{\prime}(k)\neq 0$ lead to the slight bending of the nearly flat bands. We
can see that the dispersion of the bands almost keep unchanged with
$A^{\prime}(k)=0$ and $A^{\prime}(k)\neq 0$ for $\mathrm{L_{1}/L_{3}}$ BCLs of
$\mathrm{Fe_{4}GeTe_{2}}$ and $\mathrm{L_{3}}$ BCL of
$\mathrm{Fe_{5}GeTe_{2}}$, whereas this is not the case for the
$\mathrm{L_{2}}$ BCL of $\mathrm{Fe_{4}GeTe_{2}}$ and $\mathrm{L_{1}/L_{2}}$
BCLs of $\mathrm{Fe_{5}GeTe_{2}}$ [Fig. 2(e, f), Fig. 3(c, d, e, f)] due to
the hopping between orbitals of Fe in dice lattice. We find that the bands
from BCL model Hamiltonians with $A^{\prime}(k)\neq 0$ can well reproduce the
bands obtained from first-principles calculations, which support that the flat
bands originate from the BCLs of $\mathrm{Fe_{n=4,5}GeTe_{2}}$.

It is worth discussing whether the flat bands of $\mathrm{Fe_{n=4,5}GeTe_{2}}$
are itinerant or local. Flat bands can be classified into two types: trivial
flat atomic bands and non-trivial flat bands [53]. Flat atomic bands originate
from the localization of orbitals or isolated atoms, resulting in negligible
overlaps between atomic wavefunctions [53]. Conversely, non-trivial flat bands
emerge from extended wavefunctions with substantial overlaps and hoppings
[53], indicating the itinerant character. In the case of
$\mathrm{Fe_{n=4,5}GeTe_{2}}$, the significant overlaps and hoppings between
the orbitals suggest that their flat bands are itinerant.

###  IV.3 Flat-Band Ferromagnetism

In the absence of spin polarization, all flat bands formed by the $d$ orbitals
of Fe are close to the Fermi energy due to the partial occupation of the $d$
orbitals. These flat bands result in sharp peaks of the non-spin-polarized
density of states (DOS) near the Fermi energy. According to the Stoner theory,
these peaks can lead to spontaneous magnetization [55, 56, 57]. The critical
condition for the instability is expressed as ${U>1/N_{E_{F}}}$ [55], and here
$N_{E_{F}}$ denotes the DOS at the Fermi energy.

Figure 6:  Electronic structures of $\mathrm{Fe_{5}GeTe_{2}}$ calculated by
first-principles calculations with ${U=3.0\ eV}$. (a) The band structure
without SOC. (b) The band structure with SOC. (c) The spin-polarized DOS
without SOC. (d) The Fermi surfaces with up spin are in red, while the Fermi
surfaces with down spin are in blue.

As the value of ${U}$ increases, the energies of states with up and down spin
will decrease and increase respectively, leading to a spin-polarized DOS.
Consequently, flat bands near the Fermi energy in non-spin-polarized band
structures can give rise to ferromagnetism. The spin-polarized DOS contributes
to the magnetic moment, which can be quantified as
${m=n_{\uparrow}-n_{\downarrow}}$, where $n_{\uparrow}$/$n_{\downarrow}$
represents the number of occupied states with the up/down spin. The magnetic
moment increases with increasing ${U}$, which is also confirmed by the results
of first-principles calculations [Fig. 5(a) and Fig. 7(a)].

##  V Electronic structure and magnetic properties

###  V.1 $\mathrm{Fe_{4}GeTe_{2}}$

We perform first-principles calculations to investigate the electronic
structure and magnetic properties of $\mathrm{Fe_{4}GeTe_{2}}$. In our
calculations, we employ ${U=3.0\ eV}$ to obtain the band structure, DOS, and
Fermi surfaces. The results suggest that the band structures with and without
SOC are similar, implying that SOC has a negligible effect on the electronic
structure of $\mathrm{Fe_{4}GeTe_{2}}$ [Fig. 4(a,b)]. The spin-polarization
DOS is consistent with the ferromagnetism [Fig. 4(c)]. The band structure and
Fermi surfaces [Fig. 4(d)] indicate the $\mathrm{Fe_{4}GeTe_{2}}$ is a
quasi-2D ferromagnetic metal.

Figure 7:  Magnetic properties of $\mathrm{Fe_{5}GeTe_{2}}$ calculated by
first-principles calculations. (a) The ${U}$ dependence of magnetic moments of
unequivalent Fe atoms. (b) The non-spin-polarized DOS.

By gradually increasing the ${U}$, we observe a gradual increase in the
magnetic moments of the Fe atoms of $\mathrm{Fe_{4}GeTe_{2}}$. As illustrated
in Fig. 5(a), the magnetic moments of Fe1 (Fe1′) and Fe2 (Fe2′) surpass ${1.5\
\mu_{B}}$ when ${U=0.0\ eV}$. Furthermore, we identify the presence of nearly
flat bands in the non-spin-polarized band structures [Fig. 2] and the
corresponding peaks in DOS [Fig. 5(b)]. These sharp peaks near the Fermi
energy suggest that $\mathrm{Fe_{4}GeTe_{2}}$ exhibits characteristics of an
itinerant flat-band ferromagnet [55].

###  V.2 $\mathrm{Fe_{5}GeTe_{2}}$

We further perform first-principles calculations to analyze the electronic and
magnetic properties of $\mathrm{Fe_{5}GeTe_{2}}$. As shown in Fig. 6 and Fig.
7, the band structures, DOS, and Fermi surfaces are similar to those of
$\mathrm{Fe_{4}GeTe_{2}}$. However, there is a significant difference in the
magnetic properties of Fe5. For ${U\leq 0.7\ eV}$, the magnetic moments of Fe5
are negligible, while it has a sudden increase between ${U=0.7\ eV}$ and
${0.8\ eV}$. We explain this phenomenon based on the band structure of the
$\mathrm{L_{1}}$ BCL which is a quasi-dice lattice. The energy levels of Fe5
orbitals are slightly lower than those of Fe4 due to their different coupling
to Te1. Then, bonding and anti-bonding bands are formed through the hopping
between Fe4 and Fe5 orbitals. The anti-bonding band is primarily composed of
Fe4 orbitals, whereas the bonding band is dominated by Fe5 orbitals. The
Fe4-dominated bands are very close to the Fermi energy, resulting in the
spontaneous magnetization of Fe4. As the value of ${U}$ increases, the
Fe5-dominated flat bands cross the Fermi level, leading to a pronounced
enhancement in Fe5’s magnetic moment.

Figure 8:  The dependence of the magnetism of $\mathrm{Fe_{5}GeTe_{2}}$ on
external pressures calculated by first-principles calculations. (a) The ${U}$
dependence of magnetic moments of Fe5’s magnetic moments under different
external pressures. The percentages indicate the proportion of deformation in
the $z$-direction. (b) The critical points of curves of Fe5’s magnetic moments
in (a) evolution with the external pressure.

We also investigate the effect of external pressure on the magnetic moment of
Fe5 for $\mathrm{Fe_{5}GeTe_{2}}$. For simplicity, the cell volume is kept
unchanged, applying pressure along the ${z}$-direction causes stretching in
the ${xy}$ plane. The compression along the ${z}$-direction is primarily
accommodated by the vdW gaps, resulting in negligible alteration to the
vertical spacing among atoms within each octuple layer. Consequently, the
pressure primarily influences the intra-layer hoppings due to the in-plane
stretching. Therefore, the hopping between Fe4 and Fe5 slightly decreases,
causing the energy level of Fe5 to approach the Fermi energy. Therefore, the
magnetic moment of Fe5 increases with the stretches in the ${xy}$ plane.
Furthermore, Fig. 8 illustrates that the expansion along the ${z}$-direction
can lead to a significant reduction for the magnetic moment of Fe5,
approaching almost zero. This pressure-induced modulation of the magnetic
moment might be experimentally observed in $\mathrm{Fe_{5}GeTe_{2}}$.

##  VI Conclusion

In this study, we investigate the origin of the nearly flat bands and
ferrimagnetism in $\mathrm{Fe_{n=4,5}GeTe_{2}}$. Our analysis reveals that the
lattice structure of these materials can be viewed as three stacked BCLs along
the ${z}$ direction. The presence of different orbital numbers on two
sublattices results in nearly flat bands. We demonstrate that the observed
ferromagnetism in $\mathrm{Fe_{n=4,5}GeTe_{2}}$ arises from these nearly flat
bands according to the Stoner theory. By combining model calculations with
first-principles calculations, we find that the magnetic moment of Fe5 in
$\mathrm{Fe_{5}GeTe_{2}}$ is sensitive to Coulomb interactions ${U}$ and
external pressure. Notably, pressure-induced transitions in the magnetic
moment of Fe5 may be experimentally observed in $\mathrm{Fe_{5}GeTe_{2}}$. The
emergence of flat-band ferromagnetism in $\mathrm{Fe_{n=4,5}GeTe_{2}}$
predominantly depends on the lattice structure, orbital characteristics, and
electron filling number. These findings contribute to our understanding of the
electronic and magnetic properties of vdW ferromagnets, specifically
$\mathrm{Fe_{n=4,5}GeTe_{2}}$.

###### Acknowledgements.

This work is supported by National Key Projects for Research and Development
of China (Grant No.2021YFA1400400 and No.2017YFA0303203), the Fundamental
Research Funds for the Central Universities (Grant No. 020414380185), Natural
Science Foundation of Jiangsu Province (No. BK20200007), the Natural Science
Foundation of China (No. 12074181, No. 12104217, and No. 11834006) and the Fok
Ying-Tong Education Foundation of China (Grant No. 161006).

##  Appendix A A brief proof of flat bands in BCLs

Firstly we ignore the intra-sublattice hoppings and the crystal field
splitting of orbitals on the $A$ sublattice, so $A_{k}=\epsilon I$, where $I$
is an identity matrix and $\epsilon$ is the onsite energies of orbitals on $A$
sublattice. We set $\epsilon$ as the zero energy point:

$H_{k}=\begin{pmatrix}O&S_{k}\\\ S_{k}^{\dagger}&B_{k}\end{pmatrix}$ (5)

Diagonalizing $N_{B}\times N_{A}$ rectangular matrix $S_{k}$, we have:

$S_{k}=W_{k}\Sigma_{k}V_{k}^{\dagger}$ (6)

Here, $\Sigma_{k}$ is a rectangular diagonal matrix with $N_{A}-N_{B}$ zero
rows:

$\Sigma_{k}=\begin{pmatrix}\epsilon_{1}&0&0&\dots&0\\\
0&\epsilon_{2}&0&\dots&0\\\ 0&0&\epsilon_{3}&\ddots&0\\\
\vdots&\ddots&\ddots&\ddots&0\\\ 0&0&\dots&0&\epsilon_{N_{B}}\\\
0&0&\dots&0&0\\\ \vdots&\vdots&\dots&\vdots&\vdots\\\
0&0&\dots&0&0\end{pmatrix}$ (7)

Then we perform a similarity transformation on $H_{k}$ as:

$H_{k}=\begin{pmatrix}W_{k}&O\\\
O&V_{k}\end{pmatrix}\begin{pmatrix}O&\Sigma_{k}\\\
\Sigma_{k}^{\dagger}&b_{k}\end{pmatrix}\begin{pmatrix}W_{k}^{\dagger}&O\\\
O&V_{k}^{\dagger}\end{pmatrix}$ (8)

where $b_{k}=V_{k}^{-1}B_{k}(V_{k}^{\dagger})^{-1}$. So that $H_{k}$ is
similar to a matrix that contains $N_{A}-N_{B}$ zero rows, which implies that
$H_{k}$ possesses at least $N_{A}-N_{B}$ zero-energy solutions with any $k$.
Then we could conclude that a BCL has at least $N_{A}-N_{B}$ degenerate flat
bands at the onsite energy of orbitals on $A$ sublattice. This proof does not
make requirements on the form of $B_{k}$. Therefore, intra-sublattice hoppings
and crystal field splitting of orbitals on the $B$ sublattice do not affect
the flat band.

## References

  * Bera  _et al._ [2023] S. Bera, S. K. Pradhan, M. S. Khan, R. Pal, B. Pal, S. Kalimuddin, A. Bera, B. Das, A. N. Pal, and M. Mondal, Unravelling the nature of spin reorientation transition in quasi-2d vdw magnetic material, $\mathrm{Fe_{4}GeTe_{2}}$, Journal of Magnetism and Magnetic Materials 565, 170257 (2023).
  * Chen  _et al._ [2022a] H. Chen, S. Asif, M. Whalen, J. Támara-Isaza, B. Luetke, Y. Wang, X. Wang, M. Ayako, S. Lamsal, A. F. May, M. A. McGuire, C. Chakraborty, J. Q. Xiao, and M. J. H. Ku, Revealing room temperature ferromagnetism in exfoliated $\mathrm{Fe_{5}GeTe_{2}}$ flakes with quantum magnetic imaging, 2D Materials 9, 025017 (2022a).
  * Chen  _et al._ [2023] H. Chen, S. Asif, K. Dolui, Y. Wang, J. Támara-Isaza, V. M. L. D. P. Goli, M. Whalen, X. Wang, Z. Chen, H. Zhang, K. Liu, D. Jariwala, M. B. Jungfleisch, C. Chakraborty, A. F. May, M. A. McGuire, B. K. Nikolic, J. Q. Xiao, and M. J. H. Ku, Above-room-temperature ferromagnetism in thin van der waals flakes of cobalt-substituted $\mathrm{Fe_{5}GeTe_{2}}$, ACS Applied Materials 15, 3287 (2023).
  * Seo  _et al._ [2020] J. Seo, D. Y. Kim, E. S. An, K. Kim, G. Y. Kim, S. Y. Hwang, D. W. Kim, B. G. Jang, H. Kim, G. Eom, S. Y. Seo, R. Stania, M. Muntwiler, J. Lee, K. Watanabe, T. Taniguchi, Y. J. Jo, J. Lee, B. I. Min, M. H. Jo, H. W. Yeom, S. Y. Choi, J. H. Shim, and J. S. Kim, Nearly room temperature ferromagnetism in a magnetic metal-rich van der waals metal, Science Advances 6, eaay8912 (2020).
  * Stahl  _et al._ [2018] J. Stahl, E. Shlaen, and D. Johrendt, The van der waals ferromagnets $\mathrm{Fe_{5-\delta}GeTe_{2}}$ and $\mathrm{Fe_{5-\delta-x}Ni_{x}GeTe_{2}}$ crystal structure, stacking faults, and magnetic properties, Zeitschrift für anorganische und allgemeine Chemie 644, 1923 (2018).
  * Wang  _et al._ [2023] H. Wang, H. Lu, Z. Guo, A. Li, P. Wu, J. Li, W. Xie, Z. Sun, P. Li, H. Damas, A. M. Friedel, S. Migot, J. Ghanbaja, L. Moreau, Y. Fagot-Revurat, S. Petit-Watelot, T. Hauet, J. Robertson, S. Mangin, W. Zhao, and T. Nie, Interfacial engineering of ferromagnetism in wafer-scale van der waals fe4gete2 far above room temperature, Nature Communications 14, 2483 (2023).
  * Tang  _et al._ [2023] M. Tang, J. Huang, F. Qin, K. Zhai, T. Ideue, Z. Li, F. Meng, A. Nie, L. Wu, X. Bi, C. Zhang, L. Zhou, P. Chen, C. Qiu, P. Tang, H. Zhang, X. Wan, L. Wang, Z. Liu, Y. Tian, Y. Iwasa, and H. Yuan, Continuous manipulation of magnetic anisotropy in a van der waals ferromagnet via electrical gating, Nature Electronics 6, 28 (2023).
  * Seo  _et al._ [2021a] J. Seo, E. S. An, T. Park, S.-Y. Hwang, G.-Y. Kim, K. Song, W.-s. Noh, J. Y. Kim, G. S. Choi, M. Choi, E. Oh, K. Watanabe, T. Taniguchi, J. H. Park, Y. J. Jo, H. W. Yeom, S.-Y. Choi, J. H. Shim, and J. S. Kim, Tunable high-temperature itinerant antiferromagnetism in a van der waals magnet, Nature Communications 12, 2844 (2021a).
  * May  _et al._ [2019a] A. F. May, C. A. Bridges, and M. A. McGuire, Physical properties and thermal stability of ${\mathrm{Fe}}_{5-x}{\mathrm{GeTe}}_{2}$ single crystals, Physical Review Materials 3, 104401 (2019a).
  * Chen  _et al._ [2022b] X. Chen, Y.-T. Shao, R. Chen, S. Susarla, T. Hogan, Y. He, H. Zhang, S. Wang, J. Yao, P. Ercius, D. A. Muller, R. Ramesh, and R. J. Birgeneau, Pervasive beyond room-temperature ferromagnetism in a doped van der waals magnet, Physical Review Letters 128, 217203 (2022b).
  * Mondal  _et al._ [2021] S. Mondal, N. Khan, S. M. Mishra, B. Satpati, and P. Mandal, Critical behavior in the van der waals itinerant ferromagnet ${\mathrm{Fe}}_{4}{\mathrm{GeTe}}_{2}$, Physical Review B 104, 094405 (2021).
  * May  _et al._ [2019b] A. F. May, D. Ovchinnikov, Q. Zheng, R. Hermann, S. Calder, B. Huang, Z. Fei, Y. Liu, X. Xu, and M. A. McGuire, Ferromagnetism near room temperature in the cleavable van der waals crystal $\mathrm{Fe_{5}GeTe_{2}}$, ACS Nano 13, 4436 (2019b).
  * Alahmed  _et al._ [2021] L. Alahmed, B. Nepal, J. Macy, W. Zheng, B. Casas, A. Sapkota, N. Jones, A. R. Mazza, M. Brahlek, W. Jin, M. Mahjouri-Samani, S. S. L. Zhang, C. Mewes, L. Balicas, T. Mewes, and P. Li, Magnetism and spin dynamics in room-temperature van der waals magnet $\mathrm{Fe_{5}GeTe_{2}}$, 2D Materials 8, 045030 (2021).
  * Zhang  _et al._ [2020] H. Zhang, R. Chen, K. Zhai, X. Chen, L. Caretta, X. Huang, R. V. Chopdekar, J. Cao, J. Sun, J. Yao, R. Birgeneau, and R. Ramesh, Itinerant ferromagnetism in van der waals ${\mathrm{Fe}}_{5-x}{\mathrm{Ge}\mathrm{Te}}_{2}$ crystals above room temperature, Physical Review B 102, 064417 (2020).
  * Liu  _et al._ [2022] Q. Liu, J. Xing, Z. Jiang, Y. Guo, X. Jiang, Y. Qi, and J. Zhao, Layer-dependent magnetic phase diagram in $\mathrm{Fe_{n}GeTe_{2}(3\leq n\leq 7)}$ ultrathin films, Communications Physics 5, 140 (2022).
  * Tan  _et al._ [2021] C. Tan, W.-Q. Xie, G. Zheng, N. Aloufi, S. Albarakati, M. Algarni, J. Li, J. Partridge, D. Culcer, X. Wang, J. B. Yi, M. Tian, Y. Xiong, Y.-J. Zhao, and L. Wang, Gate-controlled magnetic phase transition in a van der waals magnet $\mathrm{Fe_{5}GeTe_{2}}$, Nano Letters 21, 5599 (2021).
  * Seo  _et al._ [2021b] J. Seo, E. S. An, T. Park, S.-Y. Hwang, G.-Y. Kim, K. Song, W.-s. Noh, J. Y. Kim, G. S. Choi, M. Choi, E. Oh, K. Watanabe, T. Taniguchi, J. H. Park, Y. J. Jo, H. W. Yeom, S.-Y. Choi, J. H. Shim, and J. S. Kim, Tunable high-temperature itinerant antiferromagnetism in a van der waals magnet, Nature Communications 12, 2844 (2021b).
  * Gong and Zhang [2019] C. Gong and X. Zhang, Two-dimensional magnetic crystals and emergent heterostructure devices, Science 363, eaav4450 (2019).
  * Burch  _et al._ [2018] K. S. Burch, D. Mandrus, and J. Park, Magnetism in two-dimensional van der waals materials, Nature 563, 47 (2018).
  * Gong  _et al._ [2017] C. Gong, L. Li, Z. Li, H. Ji, A. Stern, Y. Xia, T. Cao, W. Bao, C. Wang, Y. Wang, Z. Q. Qiu, R. J. Cava, S. G. Louie, J. Xia, and X. Zhang, Discovery of intrinsic ferromagnetism in two-dimensional van der waals crystals, Nature 546, 265 (2017).
  * Huang  _et al._ [2017] B. Huang, G. Clark, E. Navarro-Moratalla, D. R. Klein, R. Cheng, K. L. Seyler, D. Zhong, E. Schmidgall, M. A. McGuire, D. H. Cobden, W. Yao, D. Xiao, P. Jarillo-Herrero, and X. Xu, Layer-dependent ferromagnetism in a van der waals crystal down to the monolayer limit, Nature 546, 270 (2017).
  * Chen  _et al._ [2022c] X. Chen, Y.-T. Shao, R. Chen, S. Susarla, T. Hogan, Y. He, H. Zhang, S. Wang, J. Yao, P. Ercius, D. A. Muller, R. Ramesh, and R. J. Birgeneau, Pervasive beyond room-temperature ferromagnetism in a doped van der waals magnet, Physical Review Letters 128, 217203 (2022c).
  * Huang  _et al._ [2022] Y. Huang, X. Yao, F. Qi, W. Shen, and G. Cao, Anomalous resistivity upturn in the van der waals ferromagnet $\mathrm{Fe_{5}GeTe_{2}}$, Applied Physics Letters 121, 162403 (2022).
  * Deng  _et al._ [2022] Y. Deng, Z. Xiang, B. Lei, K. Zhu, H. Mu, W. Zhuo, X. Hua, M. Wang, Z. Wang, G. Wang, M. Tian, and X. Chen, Layer-number-dependent magnetism and anomalous hall effect in van der waals ferromagnet $\mathrm{Fe_{5}GeTe_{2}}$, Nano Letters 22, 9839 (2022).
  * Yang  _et al._ [2021] X. Yang, X. Zhou, W. Feng, and Y. Yao, Strong magneto-optical effect and anomalous transport in the two-dimensional van der waals magnets ${\mathrm{Fe}}_{n}{\mathrm{GeTe}}_{2}$ ($n=3$, 4, 5), Physical Review B 104, 104427 (2021).
  * Ohta  _et al._ [2021] T. Ohta, M. Tokuda, S. Iwakiri, K. Sakai, B. Driesen, Y. Okada, K. Kobayashi, and Y. Niimi, Butterfly-shaped magnetoresistance in van der waals ferromagnet $\mathrm{Fe_{5}GeTe_{2}}$, AIP Advances 11, 025014 (2021).
  * Lv  _et al._ [2022] X. Lv, K. Pei, C. Yang, G. Qin, M. Liu, J. Zhang, and R. Che, Controllable topological magnetic transformations in the thickness-tunable van der waals ferromagnet $\mathrm{Fe_{5}GeTe_{2}}$, ACS Nano 16, 19319 (2022).
  * Zhang  _et al._ [2022] C. Zhang, C. Liu, S. Zhang, B. Zhou, C. Guan, Y. Ma, H. Algaidi, D. Zheng, Y. Li, X. He, J. Zhang, P. Li, Z. Hou, G. Yin, K. Liu, Y. Peng, and X.-X. Zhang, Magnetic skyrmions with unconventional helicity polarization in a van der waals ferromagnet, Advanced Materials 34, 2204163 (2022).
  * Schmitt  _et al._ [2022] M. Schmitt, T. Denneulin, A. Kovács, T. G. Saunderson, P. Rüßmann, A. Shahee, T. Scholz, A. H. Tavabi, M. Gradhand, P. Mavropoulos, B. V. Lotsch, R. E. Dunin-Borkowski, Y. Mokrousov, S. Blügel, and M. Kläui, Skyrmionic spin structures in layered fe5gete2 up to room temperature, Communications Physics 5, 254 (2022).
  * Sutherland [1986] B. Sutherland, Localization of electronic wave functions due to local topology, Physical Review B 34, 5208 (1986).
  * Călugăru  _et al._ [2021] D. Călugăru, A. Chew, L. Elcoro, Y. Xu, N. Regnault, Z.-D. Song, and B. A. Bernevig, General construction and topological classification of crystalline flat bands, Nature Physics 18, 185 (2021).
  * Wu  _et al._ [2023] B. Wu, S. Fang, J. Yang, S. Liu, Y. Peng, Q. Li, Z. Lin, J. Shi, W. Yang, Z. Luo, C. Wang, J. Yang, J. Lu, and H. Du, High-performance ${\mathrm{Fe}}_{x}{\mathrm{Ge}\mathrm{Te}}_{2}\text{$-$}$based (x = 4 or 5) van der waals magnetic tunnel junctions, Physical Review Applied 19, 024037 (2023).
  * Kim  _et al._ [2021] D. Kim, C. Lee, B. G. Jang, K. Kim, and J. H. Shim, Drastic change of magnetic anisotropy in $\mathrm{Fe_{3}GeTe_{2}}$ and $\mathrm{Fe_{4}GeTe_{2}}$ monolayers under electric field studied by density functional theory, Scientific Reports 11, 17567 (2021).
  * Joe  _et al._ [2019] M. Joe, U. Yang, and C. Lee, First-principles study of ferromagnetic metal $\mathrm{Fe_{5}GeTe_{2}}$, Nano Materials Science 1, 299 (2019).
  * Yamagami  _et al._ [2021] K. Yamagami, Y. Fujisawa, B. Driesen, C. H. Hsu, K. Kawaguchi, H. Tanaka, T. Kondo, Y. Zhang, H. Wadati, K. Araki, T. Takeda, Y. Takeda, T. Muro, F. C. Chuang, Y. Niimi, K. Kuroda, M. Kobayashi, and Y. Okada, Itinerant ferromagnetism mediated by giant spin polarization of the metallic ligand band in the van der waals magnet ${\mathrm{Fe}}_{5}\mathrm{Ge}{\mathrm{Te}}_{2}$, Physical Review B 103, L060403 (2021).
  * Lieb [1989] E. H. Lieb, Two theorems on the hubbard model, Physical Review Letters 62, 1927 (1989).
  * Derzhko  _et al._ [2015] O. Derzhko, J. Richter, and M. Maksymenko, Strongly correlated flat-band systems: The route from heisenberg spins to hubbard electrons, International Journal of Modern Physics B 29, 1530007 (2015).
  * Mielke [1991] A. Mielke, Ferromagnetism in the hubbard model on line graphs and further considerations, Journal of Physics A: Mathematical and General 24, 3311 (1991).
  * Mielke [1993a] A. Mielke, Ferromagnetism in the hubbard model and hund’s rule, Physics Letters A 174, 443 (1993a).
  * Tasaki [1998] H. Tasaki, From nagaoka’s ferromagnetism to flat-band ferromagnetism and beyond: An introduction to ferromagnetism in the hubbard model, Progress of Theoretical Physics 99, 489 (1998).
  * Tasaki [1996] H. Tasaki, Stability of ferromagnetism in hubbard models with nearly flat bands, Journal of Statistical Physics 84, 535 (1996).
  * Kusakabe and Aoki [1994] K. Kusakabe and H. Aoki, Ferromagnetic spin-wave theory in the multiband hubbard model having a flat band, Physical Review Letters 72, 144 (1994).
  * Kresse and Joubert [1999] G. Kresse and D. Joubert, From ultrasoft pseudopotentials to the projector augmented-wave method, Physical Review B 59, 1758 (1999).
  * Kresse and Furthmuller [1996] G. Kresse and J. Furthmuller, Efficient iterative schemes for ab initio total-energy calculations using a plane-wave basis set, Physical Review B 54, 11169 (1996).
  * Kresse and Furthmüller [1996] G. Kresse and J. Furthmüller, Efficiency of ab-initio total energy calculations for metals and semiconductors using a plane-wave basis set, Computational Materials Science 6, 15 (1996).
  * Kresse  _et al._ [1994] G. Kresse, J. Furthmüller, and J. Hafner, Theory of the crystal structures of selenium and tellurium: The effect of generalized-gradient corrections to the local-density approximation, Physical Review B 50, 13181 (1994).
  * Pizzi  _et al._ [2020] G. Pizzi, V. Vitale, R. Arita, S. Blügel, F. Freimuth, G. Géranton, M. Gibertini, D. Gresch, C. Johnson, T. Koretsune, J. Ibañez-Azpiroz, H. Lee, J.-M. Lihm, D. Marchand, A. Marrazzo, Y. Mokrousov, J. I. Mustafa, Y. Nohara, Y. Nomura, L. Paulatto, S. Poncé, T. Ponweiser, J. Qiao, F. Thöle, S. S. Tsirkin, M. Wierzbowska, N. Marzari, D. Vanderbilt, I. Souza, A. A. Mostofi, and J. R. Yates, Wannier90 as a community code: new features and applications, Journal of Physics: Condensed Matter 32, 165902 (2020).
  * Marzari and Vanderbilt [1997] N. Marzari and D. Vanderbilt, Maximally localized generalized wannier functions for composite energy bands, Physical Review B 56, 12847 (1997).
  * Souza  _et al._ [2001] I. Souza, N. Marzari, and D. Vanderbilt, Maximally localized wannier functions for entangled energy bands, Physical Review B 65, 035109 (2001).
  * Leykam  _et al._ [2018] D. Leykam, A. Andreanov, and S. Flach, Artificial flat band systems: from lattice models to experiments, Advances in Physics: X 3, 1473052 (2018).
  * Sukhachov  _et al._ [2023a] P. O. Sukhachov, D. O. Oriekhov, and E. V. Gorbar, Stackings and effective models of bilayer dice lattices (2023a), arXiv:2303.01452 [cond-mat.mes-hall] .
  * Sukhachov  _et al._ [2023b] P. O. Sukhachov, D. O. Oriekhov, and E. V. Gorbar, Optical conductivity of bilayer dice lattices (2023b), arXiv:2303.08258 [cond-mat.mes-hall] .
  * Regnault  _et al._ [2022] N. Regnault, Y. Xu, M.-R. Li, D.-S. Ma, M. Jovanovic, A. Yazdani, S. S. P. Parkin, C. Felser, L. M. Schoop, N. P. Ong, R. J. Cava, L. Elcoro, Z.-D. Song, and B. A. Bernevig, Catalogue of flat-band stoichiometric materials, Nature 603, 824 (2022).
  * Li  _et al._ [2018] Z. Li, J. Zhuang, L. Wang, H. Feng, Q. Gao, X. Xu, W. Hao, X. Wang, C. Zhang, K. Wu, S. X. Dou, L. Chen, Z. Hu, and Y. Du, Realization of flat band with possible nontrivial topology in electronic kagome lattice, Science Advances 4, eaau4511 (2018).
  * Kübler [2021] J. Kübler,  _Theory of Itinerant Electron Magnetism, 2nd Edition_ (Oxford University Press, 2021).
  * Wahle  _et al._ [1998] J. Wahle, N. Blümer, J. Schlipf, K. Held, and D. Vollhardt, Microscopic conditions favoring itinerant ferromagnetism, Physical Review B 58, 12749 (1998).
  * Mielke [1993b] A. Mielke, Ferromagnetism in the hubbard model and hund’s rule, Physics Letters A 174, 443 (1993b).


# Introducing Latent Timbre Synthesis

Kıvanç Tatar
Simon Fraser University
Vancouver, B.C., Canada
<EMAIL_ADDRESS>Daniel Bisig
Zurich University of the Arts
Zurich, Switzerland
<EMAIL_ADDRESS>Philippe Pasquier
Simon Fraser University
Vancouver, B.C., Canada
<EMAIL_ADDRESS>
###### Abstract

We present the Latent Timbre Synthesis (LTS), a new audio synthesis method
using Deep Learning. The synthesis method allows composers and sound designers
to interpolate and extrapolate between the timbre of multiple sounds using the
latent space of audio frames. We provide the details of two Variational
Autoencoder architectures for LTS, and compare their advantages and drawbacks.
The implementation includes a fully working application with graphical user
interface, called interpolate_two, which enables practitioners to explore the
timbre between two audio excerpts of their selection using interpolation and
extrapolation in the latent space of audio frames. Our implementation is open-
source, and we aim to improve the accessibility of this technology by
providing a guide for users with any technical background.

##  1 Introduction

Promising new research of Deep Learning (DL) for the musical applications of
audio transformation and sound synthesis has recently emerged in academia; in
conjunction with the increasing popularity of Deep Learning architectures [2].
Musical applications of these technologies have yet to become accessible for
composers and musicians who lack expertise in Machine Learning (ML) and
Artificial Intelligence (AI). Although Deep Learning has been applied to many
musical tasks [2], the research on incorporating Deep Learning architectures
for sound design applications in experimental electronic music is still in its
early stages.

This project focuses on the integration of modern ML and AI techniques into
tools for computer-assisted sound design and their applications within
composition practices. We concentrate on the fields of experimental electronic
music and sound art, which takes sound qualities, listening modalities and
experiences [38], and organized sound theories [28] into the center [36]. We
specifically focus on audio corpus-based sound synthesis approaches that rely
on large libraries of audio excerpts.

Latent Timbre Synthesis (LTS) aims to help composers by utilizing an abstract
latent timbre space that is generated by training unsupervised Deep Learning
(DL) models with a set of audio recordings. These new DL tools allow composers
to synthesize sounds using a latent space of audio that is constrained to the
timbre space of the audio recordings in the training set. Our development of
ML and AI tools for computer-assisted sound design and their subsequent
evaluation within composition practice serve to highlight the benefits and
shortcomings of the selected machine learning algorithms for creative ideation
and discovery.

In 1940s, physicist Dennis Gabor proposed [7] that a sound is composed of
acoustical quanta that is bounded by time and frequency. We are inspired by
this idea while applying this approach to digital audio, and asking,

  * •

What would be the latent space of audio frames?

  * •

Can we regenerate original audio recordings using that latent space?

  * •

Can we create new audio synthesis methods using the latent space for sound
design applications and composition practices?

  * •

Can a DL-based model be used to provide professional grade sound synthesis
tools?

  * •

How can an audio synthesis architecture using a Deep Learning model provide
the user the flexibility to generate sounds of any duration?

We are also inspired by the definition of music as “nothing but organized
sound” [39] involving sound objects [29] that situate on multiple layers [32],
where any sound can be used to produce music [19, 39], and strong connections
exist between pitch, noise, timbre, and rhythm [32, 31, 27, 28]. In that
sense, the Latent Timbre Synthesis project builds on our previous work titled
Musical Agents based on Self-Organizing Maps (MASOM) [35, 37]. MASOM combines
organizing sound samples in latent audio space with statistical sequence
models for musical structure. The latent audio space in MASOM is generated by
a Self-Organizing Map that organizes a set of audio excerpts. In LTS, we move
further by aiming for an audio synthesis framework where we can synthesize
sounds that do not exist in the training set.

Following the research questions and directions above, our contributions
presented in this paper include two Variational Auto-encoders (VAEs) to
generate a latent space of audio frames. Unlike other Deep Learning
architectures such as Generative Adversarial Networks (GANs), VAEs are
beneficial for our applications because these architectures can encode an
existing audio frame to a latent space, as well as synthesize audio frames
from latent vectors. VAEs also allow audio synthesis through interpolation and
extrapolation of timbres, by using the latent vectors of audio frames.

Latent Timbre Synthesis differs from the previous works such as Granma
MagNet[1] because we prioritize the flexibility to generate audio with any
duration, in comparison to outputting audio excerpts of fixed-duration. We
think that the flexibility of changing the duration of the generated audio is
crucial for our applications, which stands out as another contribution of LTS.
Our approach focus on creating a latent space of audio frames, where we can
represent an audio recording with any length as a time-series sequence of
latent vectors.

The LTS framework consists of three main modules, calculation of wavelet
transform based spectrogram representation, latent audio frame space
generation using two specific Variational Auto-encoders and inverse synthesis
using wavelet-based magnitude spectrogram generated by the decoder of the VAE.
We compare the advantages and drawback of two VAE architectures for designing
a synthesis tool for composition practices and sound design applications. In
addition, we present and share a fully working application, called
interpolate_two with a Graphical User Interface (GUI) that allows composers to
synthesize audio using timbre interpolation and extrapolation with multiple
sounds. In comparison to high computational complexity of previous works
mentioned in Section 2, the low computational complexity of interpolate_two
allows the incorporation of the sound design tool within composition practices
and real-time applications. The documentation of the setup of interpolate_two
is detailed to guide practitioners of all backgrounds. Our implementation is
open-source111The source code is available at
https://www.gitlab.com/ktatar/latent-timbre-synthesis., and sound examples are
available222We provide sound examples at https://kivanctatar.com/Latent-
Timbre-Synthesis.. We encourage our readers to dive into the code and
experiment with the framework for further audio synthesis possibilities.

##  2 Related Works

We limit this section to the previous works that utilize audio spectrogram as
an input for the Deep Learning architecture, with the exception of WaveNet. We
situate the Latent Timbre Synthesis project within the raw-audio generation
applications of Deep Learning, and WaveNet is one of the state of the art
systems in the area. We also omit Deep Learning systems for speech synthesis
or vocoder applications, such as MelGAN [16], while mentioning in Section 5
how we plan to incorporate them in LTS as a next step in our research.

WaveNet is a Deep Learning architecture that uses an audio corpus for the
tasks of music composition, multi-speaker speech and text to speech
generation, and speech recognition [24]. WaveNet applies Convolutional Neural
Networks (CNNs) with two strategies to handle temporality of raw audio data:
causal convolution and dilation. Causal convolutions ensure that the output
only depends on the past observations. Dilated causal convolutions skips a
number of inputs on each layer. The number of inputs that are skipped
exponentially increases with each layer; hence, the receptive field of the
network also increases exponentially [41]. Note that, the receptive field is
the number of neurons that affect a single neuron in deep networks. Oord et
al. [24] tested WaveNet on two audio corpora: the MagnaTagATune dataset and
the YouTube piano dataset. The authors point out that “Even with a receptive
field of several seconds, the models did not enforce long-range consistency
which resulted in second-to-second variations in genre, instrumentation,
volume and sound quality.” That is, WaveNet struggled to generate long-term
variations like in the case of interactive music systems that apply Markov
Models [36, Section 6.1]. There has been follow-up research on the WaveNet
architecture, where the authors stack multiple WaveNet architecture on top of
each other [25], or they combine WaveNet with Vector Quantized Variational
Autoencoders [4]. The main drawback of all WaveNet systems are their
computational complexity and high-usage of GPU memory [3]. The technology
requirements of WaveNet compromise its usage in compositional practices, where
composers do not necessarily have access to computers with the state of the
art GPUs.

Differentiable Digital Signal Processing (DDSP) is a toolbox made by the
Google for researching Digital Signal Processing (DSP) applications of Deep
Learning [5]. The authors describe the DDSP Autoencoder, which is a VAE
architecture where the input are the Mel-Frequency Cepstral Coefficients
(MFCCs) of an audio excerpt. We mention a comparison of using MFCCs and other
audio features as the representation of timbre for audio synthesis with VAEs
in Section 3.1. The architecture employs three autoencoders for fundamental
frequency (f-encoder), loudness (l-encoder), and the latent space of timbre
(z-encoder). The fundamental frequency and the loudness encoders use the CREPE
architecture that is originally presented as a pitch detector [10]. The
z-encoder architecture is inspired by the ResNet architecture in Computer
Vision research [11]. The decoder, on the other hand, controls the input
parameters of an additive synthesis module, a subtractive synthesis module,
and a reverb. These three synthesis modules generate the final audio. The loss
function compares the generated audio with the original one, using a specific
function called Multi-Scale Spectrogram Loss, which is similar to comparing
the spectrograms of original and generated audio.

Generative Timbre Spaces project [6] is perhaps one of the most similar
previous study to the Latent Timbre Synthesis project. The application of
Generative Timbre Synthesis focuses on generating a latent timbre space of
conventional musical instruments. This model uses a VAE where the encoder is a
3-layer feed-forward network with 2000 units in each layer. The latent space
has 64 dimensions. The authors introduce a new regularization item in the cost
function. The additional regularization loss tries to force the network to
satisfy perceptual similarity ratings of conventional musical instruments in
Western Classical Music. These perceptual ratings are proposed in previous
studies [8, 15, 12, 20, 17]. The training dataset of Generative Timbre Spaces
is audio recordings of conventional musical instruments where each file is an
instrument playing a note. The authors takes one frame from each audio file to
train the VAE model. Hence, the architecture aims to capture the generalized
timbre of a conventional musical instrument instead of the regeneration of an
arbitrary audio excerpt. Likewise, the cost function is not suitable to
regenerate a dataset with arbitrary audio recordings because there are no
perceptual ratings available. We further discuss the issues related to the
hyper-parameters of VAE in Generative Timbre Spaces in Section 3.2.

The DDSP Autoencoder as well as the Generative Timbre Spaces aims for the
synthesis applications of conventional music where the model is conditioned to
output an audio with a fundamental frequency constraint. In sound design,
experimental electronic music, and Sound Art applications, having a
fundamental frequency of a sound gesture is rather limiting. The music theory
of the contemporary electronic music emphasizes the continuum between noise,
pitch, and rhythm [32, 19, 31, 27, 28]. Hence, in LTS, we aim for a model that
could generate any audio so that the composers and practitioners are free to
explore the full potential of digital audio synthesis.

##  3 System Design

Figure 1: Latent Timbre Synthesis framework

The Latent Timbre Synthesis framework consists of three main parts,
spectrogram calculation using a type of wavelet transform, audio frame latent
space generation using Variational Autoencoders, and inverse synthesis of
audio using the magnitude spectrogram generated by the decoder of VAE (Figure
1).

###  3.1 Wavelet Transform based Spectrograms

The audio feature extraction module generates spectrogram frames using a type
of wavelet transform for audio, called Constant-Q Transform (CQT) [30], which
gained popularity in Music Information Retrieval (MIR) research in the recent
years. CQT, as well as its variant Non-stationary Gabor Transform (NSGT) [40],
have been compared to the other audio features such as Mel-Frequency Cepstral
Coefficients (MFCC); and previous studies showed that CQT and NSGT could
perform better in MIR applications such as segmentation and musical structure
analysis [23]. Naturally, segmentation and musical structure analysis tasks
require computing the audio similarity [22]; thus, they are suited to create a
latent space of audio frames.

Table 1: A previous study provided a comparison of audio frame reconstruction losses with Variational Autoencoders using spectrograms with fixed-length windows and wavelet transform based spectrograms [6]. | Spectrogram | $logp(x)$ | $||x-\tilde{x}||^{2}$
---|---|---|---
Fixed Window | STFT | -1.9237 | 0.2412
DCT | 4.3415 | 2.2629
Wavelet Transform | CQT | -2.8723 | 0.1610
NSGT-MEL | -2.9184 | 0.1602
NSGT-ERB | -2.9212 | 0.1511

A previous work [6] compared spectrograms computed with fixed windows and
wavelet transform based spectrograms for applications of latent audio frame
space generation using Deep Learning (DL). This comparison included Short-Time
Fourier Transform, Discrete Cosine Transform, Constant-Q Transform (CQT), and
Non-Stationary Gabor Transform (NSGT) variations using different frequency
scales. The wavelet based transforms in this previous study were CQT and NSGT
variants. The authors [6] found that wavelet transform based spectrogram
representations perform better than the spectrograms calculated using fixed-
length windows in regards to the log-likelihood and mean quality of the audio
frame reconstructions, as shown in Table 1; while the audio frame
reconstructions of wavelet transform based spectrograms gave similar results.
We utilize CQT in comparison to other wavelet-based spectrograms in Table 1
because a python library for audio analysis, titled
Librosa333https://librosa.github.io/librosa/ [21], includes a CQT and inverse
CQT implementation [30] combined with a Fast-Griffin-Lim phase estimation [26]
that we explain in Section 3.3. We aimed that the LTS framework would be
available for composers and sound designers of all technical levels. Hence, we
prioritized the options that simplify the installation of the LTS framework.

We delve into the details of Constant-Q Transform in the following. We can
calculate the CQT of an audio recording [30], a discrete time domain signal
$x(n)$, using the following formula:

$X^{CQ}(k,n)=\sum_{j=n-\lfloor N_{k}/2\rfloor}^{n+\lfloor
N_{k}/2\rfloor}x(j)a_{k}^{*}(j-n+N_{k}/2)$ (1)

where $k$ represents the CQT frequency bins with a range of $[1,K]$, and
$X^{CQ}(k,n)$ is the CQT transform. $N_{k}$ is the window length of a CQT bin,
that is inversely proportional to $f_{k}$ that we define in equation 3 Notice
that, $\lfloor\cdot\rfloor$ is the rounding towards negative infinity.
$a_{k}^{*}$ is the negative conjugate of the basis function $a_{k}(n)$ and,

$a_{k}(n)=\frac{1}{N_{k}}w(\frac{n}{N_{k}})exp[-i2\pi n\frac{f_{k}}{f_{s}}]$
(2)

where $w(t)$ is the window function, $f_{k}$ is the center frequency of bin
$k$, and $f_{s}$ is the sampling rate. CQT requires a fundamental frequency
parameter $f_{1}$, which is the center frequency of the lowest bin. The center
frequencies of remaining bins are calculated using,

$f_{k}=f_{1}2^{\frac{k-1}{B}}$ (3)

where $B$ is the number of bins per octave.

CQT is a wavelet-based transform because the window size is inversely
proportional to the $f_{k}$ while ensuring the same Q-factor for all bins $k$.
We can calculate the Q-factor using,

$Q=\frac{qf_{s}}{f_{k}(2^{\frac{1}{B}}-1)}$ (4)

where $q$ is scaling factor with the range [0,1] and equals to $1$ as the
default setting. We direct our readers to the original publication for the
specific details of the CQT [30], which also proposed a fast algorithm to
compute CQT and inverse CQT (i-CQT), given in Figure 2.

(a) Constant-Q Transform

(b) Inverse Constant-Q Transform

Figure 2: A fast algorithm to compute CQT and i-CQT, described in [30] and
implemented in Librosa [21].

All experiments in this paper used the same audio feature extraction
configurations. We tried several parameters to find the configuration that
could give the least amount of audio artifacts with the pipeline of
calculating the CQT spectrogram and then reconstructing the audio back using
the magnitude spectrogram of CQT combined with the phase estimation algorithm.
Notice that these artifacts would appear even with an ideal DL model because
the inverse audio synthesis introduces these artifacts to the LTS. We used
16-bit and 44.1 kHz stereo or mono audio recordings. We converted the stereo
files to mono first, and then calculated the CQT spectrograms using a hop-size
of 128 samples. The $f_{1}$ parameter was 32.7 Hz that corresponds to the
musical note, C1. $q$ value in equation 4 was equal to $1$, and the window
function was
“hann”444https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.windows.hann.html.
CQT included 48 bins per octave for a total range of 8 octaves; which sums up
to a 384 bins in total. Hence, the input of the DL models are vectors with 384
dimensions. These parameters resulted in the least amount of artifacts in our
experiments. Our source code is flexible to change these parameters, and we
encourage our readers to try different parameter configurations.

###  3.2 Deep Learning in Latent Timbre Synthesis

####  3.2.1 Autoencoders and Variational Autoencoders

Autoencoders are Deep Learning architectures for generative modelling. The
architecture consists of two main modules: an encoder and a decoder (Figure 1
and 4). The encoder maps the input data $x\in R^{L}$ to a latent vector $z\in
R^{M}$ where $z=encoder(x)$, and $M<L$. The decoder aims to convert a latent
vector back to the original data, and ideally, $decoder(encoder(x))=x$. The
vanilla Autoencoder architecture encodes the input data vector to a single
point, that is the latent vector. In comparison, Variational Autoencoder (VAE)
is an improved version of the Autoencoder architecture that converts the input
data vector to a stochastic distribution over the latent space. This
difference is also referred as the “reparametrization trick” [13, 14, 33].

In VAE, the encoder tries to generate a latent space by approximating $p(z|x)$
while the decoder tries to capture the true posterior $p(x|z)$. The vanilla
VAE approximates $p(z|x)$ using $q(z|x)\in Q$ with the assumption that
$p(z|x)$ is in the form of a Gaussian distribution $N(0,I)$. This
approximation is referred in the literature as Variational Inference [13].
Specifically, the encoder outputs the mean $\mu_{M}$ and the co-variance
$\sigma_{M}$ as the inputs of the Gaussian distribution function
$N(z;\mu_{M},\sigma^{2}_{M}I)$ over a latent space with $M$ number of
dimensions. Hence, the encoder approximates $p(z|x)$ using
$q^{*}(z|x)=N(z;f(x),g(x)^{2}I)$ where $\mu_{M}=f(x)$, $f\in F$,
$\sigma_{M}=g(x)$, and $g\in G$. The decoder’s input, the latent vector $z$ is
sampled from the latent distribution $q(z)=N(z;f(x),g(x)^{2}I)$. Hence, the
loss function consists of the reconstruction loss and the regularization term
of Kullback-Leibler divergence between $q^{*}(z|x)$ and $p^{*}(z)$,

$L_{f,g}=\mathbb{E}_{q^{*}(z)}[logp^{*}(x|z)]-\alpha\cdot
D_{KL}[q^{*}(z|x)||p^{*}(z)]$ (5)

We direct our readers to the original VAE publication [13] for the
mathematical induction of the loss function in equation 5 [14]. Note that,
some previous works introduced additional regularization terms to the loss
function to condition the VAE further, such as the introduction of perceptual
ratings of musical instruments in [6].

The LTS framework focuses on Variational Autoencoders in comparison to
Generative Adversarial Networks because we aim for audio synthesis by
interpolation and extrapolation in the latent space of audio frames (see
Section 4). The input vectors of the VAE model are CQT vectors calculated from
one audio frame where the window size varies with the frequency bins. We aim
to generate a latent space of audio frames so that we can synthesize audio
with any duration. Previous systems such as Grannma MagNet [1] utilizes audio
excerpts with fixed-duration, where the training input vectors of deep
learning model are 2D audio spectrograms with time along the x-axis and
frequency along the y-axis. This design choice constraints these DL models to
limited applications such as generating a fixed-length audio excerpt. Our
approach differs from the previous systems because the training observation of
DL model in LTS is one audio-spectrum vector that is calculated from one
audio-frame. This allows LTS to generate audio with any duration in sound
design applications.

We focus on two Deep Learning (DL) architectures in the current version of the
Latent Timbre Synthesis framework. Both models are Variational Autoencoders
(VAE); however, the layers and model parameters differ. In the first VAE
model, we were inspired by the previous work [6] where the authors trained a
Variational Autoencoder to generate conventional musical instrument timbres
with digital audio synthesis.

We initially tried the VAE architecture with the same hyper-parameter settings
that were used in the Generative Timbre Spaces project. The setting of
Generative Timbre Spaces [6] were unsuccessful in our experiments. The model
could not learn to regenerate the audio recordings in the training dataset,
and could only generate noise. Upon further investigation, we found that the
Kullback-Leibler Divergence regularization term multiplier caused the issue.
In Generative Timbre Spaces, the authors increase the multiplier from $0$ to
$2$ during the first $100$ epochs of the training, following the warm-up
procedure [33]. We suspect that the KLD multiplier range of $[0,2]$ is
specific to the application of Generative Timbre Spaces where the training
dataset consists of audio files with distinct harmonic content and low
noisiness in the spectrum. Furthermore, the training dataset size of
Generative Timbre Spaces is rather small, less than 100 MB, whereas we work
with GBs of audio to train the LTS models. For example, the erokia dataset
that we provide with our source code includes 2 GBs of audio that corresponds
to $3,084,591$ audio frames as training data for LTS models, using a hop-size
of $128$ samples for the CQT calculation.

We found that the range of $[0,2]$ for KLD multiplier was too high for our
application and prohibited the VAE to learn. In addition, the warm-up
procedure had adverse effects on the learning. We further investigated this
issue using the MNIST dataset [18]. MNIST gave us a visual understanding of
the effect of KLD multiplier. The images in MNIST are $28*28$ pixels, adding
up to a total of $784$ pixels. This is similar to LTS where the input of the
VAE is a vector of $384$ dimensions.

Like in the case of our tests with audio spectrograms, we obtained similar
results with the MNIST data. Higher KL-divergence values as well as the warm-
up procedure significantly deteriorated the reconstructions of the trained
model.

Figure 3(a) and 3(b) show the effect of KLD multiplier on the training of the
Variational Autoencoder. We used the same architecture depicted in Figure 1
and only changed the input and output dimensions to $784$ that corresponds the
flattened vector of $28x28$ images in the MNIST dataset. In addition, we
tested the warm-up procedure and the reverse settings of the warm-up
procedure; however, both were detrimental to the training in our experiments.
We proceeded our experiments with KLD values around $1e-5$, given the success
of this setting with the MNIST dataset.

(a) The effect of the KLD multiplier on the reconstructions

(b) The loss values during training where x-axis is the training epoch and
y-axis is the loss value

Figure 3: The effect of the KLD multiplier in the loss function on Variational
Autoencoder training

All VAE architectures in LTS use decoder networks that are the reversed
replicas of the encoder networks, as in most cases of VAEs. The first VAE
model in LTS is a feed-forward network with two Dense layers including 2048
neurons, The dense, dense_1, and dense_2 layers in Figure 4 apply Rectified
Linear Units (ReLU) as the neuron activation functions.

Figure 4: The first Deep Learning architecture available in Latent Timbre
Synthesis Figure 5: The second Variational Autoencoder architecture available
in Latent Timbre Synthesis

We train the network for 2000 epochs, while the improvements after epoch 50
are rather in the exploitation555Exploration and exploitation are two search
strategies in optimization applications [34, Section 5.3]. phase of the
learning, and help to minimize the floor noise in the generated CQT magnitude
spectrogram. The learning rate is $1e^{-4}$ and the KLD multiplier in the cost
function is $5e^{-4}$. The learning rate and the KLD multiplier parameters are
dependent on the training dataset. We recommend the readers who would be
interested to try their own dataset to start with the hyper-parameter settings
above, and change the parameters when needed.

The latent space of our first VAE model consists of $256$ dimensions. This
number is still acceptable given that the previous work [6] used a $64$
dimensional latent space to cluster a much smaller range of conventional
musical instrument timbres. Yet, we explored the possibility to find a deeper
network that could generate a latent space with a smaller number of
dimensions.

The second VAE network that we present in this paper aimed to decrease the
number of latent space dimensions of the first network using a deeper
architecture, shown in Figure 5. We pursued an iterative design procedure
where we tried to decrease the number of dimensions of the latent space while
maintaining the reconstruction quality and trying to achieve lower final loss
values in the training runs. Our experiments aimed for a 8 dimensional latent
space, and we first tested increasing the number of dense layers. Our
experiments found an optimum of 4 dense layers where the number of neurons is
changing by two-folds each layer (Figure 5). In addition to the Dense layers,
we also tried adding convolutional layers on top of the Dense layers. We
imagined that the convolutional layers could grasp the relationships between
frequency bins of a given CQT vector, such as harmonics. The convolutional
layer generates a 2-dimensional vector with size 24x16, which is in relation
to 48 bins per octave for 8 octaves while trying to remain as close as to a
square. Notice that, each row in the 2D vector generated by the convolutional
layer corresponds to half of an octave. We imagined that this could further
help the network to capture the harmonic content in the CQT vectors. Our
experiments found an optimum of one additional convolutional layer with $32$
filters and a kernel size of $3$ and a stride size of $2$ on top of the dense
layers, shown in Figure 5. The final second network could decrease the number
of latent space dimensions while giving a low final loss value. However, the
network introduced a consistent floor noise sound in the reconstructions. The
introduction of normalization techniques such as batch, instance, or layer
normalization could not eliminate the floor noise in the reconstructions. Our
findings suggest a balance between the number of latent space dimensions and
the floor noise in the reconstructions where increasing the number of latent
space dimensions is the solution to eliminate the noise.

Given that audio quality is of great importance in composition tasks, we
focused on using and disseminating the first network given in Figure 4.
Additionally, the low computational complexity of the first network can be an
advantage when we combine the VAE with Deep Learning models for time-series
sequence generation in our future work, detailed in Section 5.

###  3.3 Inverse Synthesis and Audio Reconstruction

This version of LTS uses the Fast Griffin-Lim [26] phase estimation algorithm
(GLA) for generating audio from CQT magnitude spectrograms that the VAE
decoder outputs. Briefly, the GLA estimates the phase component of a magnitude
spectrogram by iterating the inverse synthesis and the spectrogram calculation
multiple times, initially proposed in [9] and shown in Algorithm 1 [26]. Given
an an audio signal $x(n)$ and its frequency transform $X(i)$,

Algorithm 1 Griffin-Lim Algorithm

1:Set: $\angle X_{0}(i)$

2:Initialize: $X_{0}(i)=|X(i)|\cdot e^{j\angle X_{0}(i)}$

3:for $n=1,2,\ldots,N$ do

4: $X_{n}(i)=T(IT(|X(i)|\cdot e^{j\angle X_{n-1}(i)}))$

5:end for

6:$\hat{x}(n)=IT(X_{N}(i))$

where $N$ is the total number of GLA iterations, $T$ and $IT$ is the frequency
transform and inverse frequency transform function respectively; such as
Short-Fourier Transform, or Constant-Q Transform in our case. Note that, the
space of audio spectrograms is a subset of the complex number space. The
iterative process of Griffin-Lim moves the complex spectrogram of the
estimated signal $\hat{x}(n)$ towards the complex number space of audio
signals in each iteration, as proven in [9].

The Fast Griffin-Lim algorithm (F-GLA) [26] is a revision of the original
Griffin-Lim algorithm.

Algorithm 2 Fast Griffin-Lim Algorithm

1:Set: $\angle X_{0}(i)$

2:Initialize: $X_{0}(i)=|X(i)|\cdot e^{j\angle X_{0}(i)}$

3:Initialize: $Y_{0}(i)=T(IT(|X(i)|\cdot e^{j\angle X_{0}(i)}))$

4:for $n=1,2,\ldots,N$ do

5: $Y_{n}(i)=T(IT(|X(i)|\cdot e^{j\angle X_{n-1}(i)}))$

6: $X_{n}(i)=Y_{n}(i)+\alpha(Y_{n}(i)-Y_{n-1}(i))$

7:end for

8:$\hat{x}(n)=IT(X_{N}(i))$

where $\alpha$ is a constant. A previous study [26] showed that the F-GLA
revision significantly improves signal-to-noise ratio (SNR) compared to the
GLA, where the setting $\alpha=1$ resulted in the highest SNR value.

The F-GLA module has the highest computation time in the LTS framework. We are
aware that a revision of this module using another Deep Learning architecture
for vocoder applications can improve the computational complexity of LTS while
making LTS more lightweight within real-time applications. We further address
this in the Section 5.

This concludes the explanation of the LTS architecture, where we covered our
specific choice of Constant-Q Transform, two Variational Autoencoder
architectures, and the inverse synthesis using the GLA phase estimation. In
the following, we introduce the interpolate_two algorithm.

##  4 Interpolation and Extrapolation in Latent Timbre Space

The first sound design application of Latent Timbre Synthesis is the
interpolate_two framework that allows composers to synthesize sounds using
interpolation and extrapolation with two sounds. The framework requires a
trained model to generate sounds using the latent audio frame space. The user
can select the duration of the generated sound, and can choose an excerpt from
two audio files. These two excerpts have the same duration. The algorithm uses
these two excerpts for synthesis with interpolation and extrapolation in the
timbre space. The interpolation amount sets how much of the latent vector is
copied from one of the audio excerpts. For example, 30% interpolation is
adding 30% of the latent vectors of the first audio and 70% of the latent
vectors of the second audio. The percentages above 100 or below 0 corresponds
to extrapolations. For instance, 120% is moving 20% away from the second audio
in the direction the latent vector that points from audio 1 to audio 2. The
algorithm synthesizes the audio by calculating every audio frame using inverse
synthesis from a generated spectrogram. Hence, the user sets interpolation
amounts for each latent vector that corresponds to one audio frame. The user
can draw an interpolation curve to change the interpolation percentage in time
using the LTS framework.

Figure 6: The Max GUI of the interpolate_two application

The application, interpolate_two consists of two components: Max GUI and the
python engine. The Max GUI handles user interactions while the python engine
reacts to OSC messages coming from Max. The python engine runs the deep
learning model and audio feature extraction (CQT calculations), as well as
inverse synthesis that generates audio from CQT magnitude spectrogram combined
with Fast Griffin-Lim phase estimation.

The framework of interpolate_two is compatible with all variations of the VAE
architectures given in Figure 4 and 5. The B an C regions in the GUI in Figure
6 load the datasets and models that are produced within a particular run with
the dataset. Using the regions D and E, the user can select two audio files
from a dataset to choose excerpts. The file dropdown menu allows to select an
audio file from the dataset. ”Zoom to selection” sets the view to the
selection area. Clicking ctrl (or cmd on macos) and then dragging the mouse up
& down on the waveform views applies zoom in & out. The waveform in the region
F sets apply interpolation (or extrapolation) amounts per frame, where x-axis
is the time and y-axis sets the interpolation percentage for a frame. When
both waveform views are zoomed to the selection, the x-axis of the
interpolation curve corresponds to the x-axis of the waveform. The
interpolation curve view is a [waveform~] object. The ”Vertical Zoom”
parameter in the inspector of this Max object sets the maximum
interpolation/extrapolation amount. The default maximum is 1.3; hence,
[1.0,1.3] and [-1.0, -1.3] are the extrapolation regions. It is possible to
extrapolate even more by changing the vertical zoom parameter; however, the
higher amounts are likely to give audible distortions. The normalize toggle in
region H allows the user to normalize the generated audio to prevent audio
distortions or extremely high audio volumes while exploring extrapolation
possibilities.

The section H send messages to the python engine to handle output generation.
”Generate & Play” initiates the python engine to synthesize a sound using the
current interpolation curve and the audio selections. ”Play Again” plays the
previous generated sound, without going through the deep learning calculation.
”STOP” immediately stops the audio coming out of the python engine. Phase
iterations sets the number of iterations of the Fast Griffin-Lim algorithm.
Higher number of iterations (max. 64) gives better results; however, the
calculation takes significantly longer. The phase estimation algorithm is the
bottleneck of computational complexity of this framework. Still, the
calculation of the audio takes 50% of the audio duration with phase iteration
set to 1. That is, calculating a 2-second sound takes around 1-second on a
laptop with NVIDIA RTX 2080 Max-Q GPU and 2.20 GHz Intel i7-8750H CPU.

##  5 Future Work

We are currently finalizing our VAE model analysis where we visualize the
latent audio frame space. In addition, we are conducting a qualitative study
where composers and practitioners used the LTS framework for sound design
applications. We aim to obtain a better understanding of the creative
potential of these algorithms by evaluating them in real-world application
scenarios within composition practices. Our future work includes a publication
where we plan to include the DL model analysis and the qualitative study, as
well as a compilation album release.

## Acknowledgements

This research has been supported by the Swiss National Science Foundation,
Natural Sciences and Engineering Research Council of Canada, Social Sciences
and Humanities Research Council of Canada, and Compute Canada.

## References

  * [1] Akten, M.: Grannma MagNet (2018).  URL https://www.memo.tv/works/grannma-magnet/.  Library Catalog: www.memo.tv
  * [2] Briot, J.P., Pachet, F.: Deep learning for music generation: challenges and directions.  Neural Computing and Applications 32(4), 981–993 (2020).  DOI 10.1007/s00521-018-3813-6.  URL http://link.springer.com/10.1007/s00521-018-3813-6
  * [3] Dieleman, S.: Sander Dieleman: Generating music in the raw audio domain.  URL https://www.youtube.com/watch?v=y8mOZSJA7Bc
  * [4] Dieleman, S., Oord, A.v.d., Simonyan, K.: The challenge of realistic music generation: modelling raw audio at scale.  In: Proceedings of the 32nd Conference on Neural Information Processing Systems (NeurIPS 2018), p. 11. Montreal QC, Canada (2018)
  * [5] Engel, J., Hantrakul, L.H., Gu, C., Roberts, A.: DDSP: Differentiable Digital Signal Processing (2019).  URL https://openreview.net/forum?id=B1x1ma4tDr
  * [6] Esling, P., Chemla-Romeu-Santos, A., Bitton, A.: Generative timbre spaces: regularizing variational auto-encoders with perceptual metrics.  arXiv:1805.08501 [cs, eess] (2018).  URL http://arxiv.org/abs/1805.08501.  ArXiv: 1805.08501
  * [7] Gabor, D.: Acoustical Quanta and the Theory of Hearing.  Nature 159(4044), 591–594 (1947).  DOI 10.1038/159591a0.  URL http://www.nature.com/articles/159591a0
  * [8] Grey, J.M.: Multidimensional perceptual scaling of musical timbres.  The Journal of the Acoustical Society of America 61(5), 1270–1277 (1977).  DOI 10.1121/1.381428.  URL http://asa.scitation.org/doi/10.1121/1.381428
  * [9] Griffin, D.W., Lim, J.S.: Signal estimation from modified short-time Fourier transform.  IEEE Transactions on Acoustics, Speech, and Signal Processing 32(2), 236–243 (1984).  DOI 10.1109/TASSP.1984.1164317.  URL http://ieeexplore.ieee.org/document/1164317/
  * [10] Hantrakul, L., Engel, J., Roberts, A., Gu, C.: Fast and Flexible Neural Audio Synthesis.  In: Proceedings of the 20th International Society for Music Information Retrieval Conference (ISMIR 2019), p. 7 (2019)
  * [11] He, K., Zhang, X., Ren, S., Sun, J.: Deep Residual Learning for Image Recognition.  In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778. IEEE, Las Vegas, NV, USA (2016).  DOI 10.1109/CVPR.2016.90.  URL http://ieeexplore.ieee.org/document/7780459/
  * [12] Iverson, P., Krumhansl, C.L.: Isolating the dynamic attributes of musical timbrea.  The Journal of the Acoustical Society of America 94(5), 2595–2603 (1993).  Publisher: Acoustical Society of America
  * [13] Kingma, D.P., Welling, M.: Auto-Encoding Variational Bayes.  arXiv:1312.6114 [cs, stat] (2014).  URL http://arxiv.org/abs/1312.6114.  ArXiv: 1312.6114
  * [14] Kingma, D.P., Welling, M.: An Introduction to Variational Autoencoders.  Foundations and Trends® in Machine Learning 12(4), 307–392 (2019).  DOI 10.1561/2200000056.  URL http://arxiv.org/abs/1906.02691.  ArXiv: 1906.02691
  * [15] Krumhansl, C.L.: Why is musical timbre so hard to understand.  Structure and perception of electroacoustic sound and music 9, 43–53 (1989)
  * [16] Kumar, K., Kumar, R., de Boissiere, T., Gestin, L., Teoh, W.Z., Sotelo, J., de Brebisson, A., Bengio, Y., Courville, A.: MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis.  In: Proceedings of the 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), p. 12. Vancouver, BC, Canada (2019)
  * [17] Lakatos, S.: A common perceptual space for harmonic and percussive timbres.  Perception & psychophysics 62(7), 1426–1439 (2000).  Publisher: Springer
  * [18] LeCun, Y., Cortes, C., Burges, C.: MNIST handwritten digit database.  URL http://yann.lecun.com/exdb/mnist/
  * [19] Luigi, R.: The Art of Noise.  A Great Bear Pamphlet (1967)
  * [20] McAdams, S., Winsberg, S., Donnadieu, S., De Soete, G., Krimphoff, J.: Perceptual scaling of synthesized musical timbres: Common dimensions, specificities, and latent subject classes.  Psychological research 58(3), 177–192 (1995).  Publisher: Springer
  * [21] McFee, B., Raffel, C., Liang, D., Ellis, D.P., McVicar, M., Battenberg, E., Nieto, O.: librosa: Audio and Music Signal Analysis in Python.  In: Proceedings of The 14th Python in Science Conference (SCIPY 2015) (2015)
  * [22] Müller, M.: Fundamentals of Music Processing.  Springer International Publishing, Cham (2015).  URL http://link.springer.com/10.1007/978-3-319-21945-5
  * [23] Nieto, O., Bello, J.P.: Systematic Exploration Of Computational Music Structure Research.  In: Proceedings of the 17th International Society for Music Information Retrieval Conference (ISMIR 2016), p. 7. New York, NY, USA (2016)
  * [24] Oord, A.v.d., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., Kavukcuoglu, K.: Wavenet: A generative model for raw audio.  arXiv preprint arXiv:1609.03499 (2016)
  * [25] Oord, A.v.d., Li, Y., Babuschkin, I., Simonyan, K., Vinyals, O., Kavukcuoglu, K., Driessche, G.v.d., Lockhart, E., Cobo, L.C., Stimberg, F., Casagrande, N., Grewe, D., Noury, S., Dieleman, S., Elsen, E., Kalchbrenner, N., Zen, H., Graves, A., King, H., Walters, T., Belov, D., Hassabis, D.: Parallel WaveNet: Fast High-Fidelity Speech Synthesis.  arXiv:1711.10433 [cs] (2017).  URL http://arxiv.org/abs/1711.10433.  ArXiv: 1711.10433
  * [26] Perraudin, N., Balazs, P., Sondergaard, P.L.: A fast Griffin-Lim algorithm.  In: 2013 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics, pp. 1–4. IEEE, New Paltz, NY (2013).  DOI 10.1109/WASPAA.2013.6701851.  URL http://ieeexplore.ieee.org/document/6701851/
  * [27] Roads, C.: Microsound.  The MIT Press, Cambridge, Mass. (2004)
  * [28] Roads, C.: Composing electronic music: a new aesthetic.  Oxford University Press, Oxford (2015)
  * [29] Schaeffer, P.: Traité des objets musicaux, nouv. edn.  Seuil (1964)
  * [30] Schörkhuber, C., Klapuri, A.: Constant-Q Transform Toolbox For Music Processing.  In: Proceedings of the 7th Sound and Music Computing Conference (SMC 2010), p. 8. Barcelona, Spain (2010)
  * [31] Smalley, D.: Spectromorphology: explaining sound-shapes.  Organised Sound 2(02), 107–126 (1997).  DOI 10.1017/S1355771897009059.  URL http://journals.cambridge.org/article_S1355771897009059
  * [32] Stockhausen, K.: Four Criteria of Electronic Music with Examples from Kontakte (1972).  URL https://www.youtube.com/watch?v=7xyGtI7KKIY&list=PLRBdTyZ76lvAFOtZvocPjpRVTL6htJzoP
  * [33] Sønderby, C.K., Raiko, T., Maaløe, L., Sønderby, S.K., Winther, O.: How to Train Deep Variational Autoencoders and Probabilistic Ladder Networks.  In: Proceedings of the 23rd international conference on Machine learning (ICML 2016). ACM Press, Pittsburgh, Pennsylvania (2016)
  * [34] Tatar, K., Macret, M., Pasquier, P.: Automatic Synthesizer Preset Generation with PresetGen.  Journal of New Music Research 45(2), 124–144 (2016).  DOI 10.1080/09298215.2016.1175481.  URL http://dx.doi.org/10.1080/09298215.2016.1175481
  * [35] Tatar, K., Pasquier, P.: MASOM: A Musical Agent Architecture based on Self Organizing Maps, Affective Computing, and Variable Markov Models.  In: Proceedings of the 5th International Workshop on Musical Metacreation (MUME 2017). Atlanta, Georgia, USA (2017)
  * [36] Tatar, K., Pasquier, P.: Musical agents: A typology and state of the art towards Musical Metacreation.  Journal of New Music Research 48(1), 56–105 (2019).  DOI 10.1080/09298215.2018.1511736.  URL https://www.tandfonline.com/doi/full/10.1080/09298215.2018.1511736
  * [37] Tatar, K., Pasquier, P., Siu, R.: Audio-based Musical Artificial Intelligence and Audio-Reactive Visual Agents in Revive.  In: Proceedings of the joint International Computer Music Conference and New York City Electroacoustic Music Festival 2019 (ICMC-NYCEMF 2019), p. 8. International Computer Music Association, New York City, NY, USA (2019)
  * [38] Tuuri, K., Eerola, T.: Formulating a Revised Taxonomy for Modes of Listening.  Journal of New Music Research 41(2), 137–152 (2012).  DOI 10.1080/09298215.2011.614951.  URL http://www.tandfonline.com/doi/abs/10.1080/09298215.2011.614951
  * [39] Varese, E., Wen-chung, C.: The liberation of Sound.  Perspectives of New Music 5(1), 11–19 (1966).  URL https://www.jstor.org/stable/832385?origin=JSTOR-pdf&seq=1#page_scan_tab_contents
  * [40] Velasco, G.A., Holighaus, N., Dörﬂer, M., Grill, T.: Constructing An Invertible Constant-Q Transform With Nonstationary Gabor Frames.  In: Proceedings of the 14th International Conference on Digital Audio Effects (DAFx-11)), p. 7. Paris, France (2011)
  * [41] Yu, F., Koltun, V.: Multi-scale context aggregation by dilated convolutions.  arXiv preprint arXiv:1511.07122 (2015)