#  Quantum anomalous Hall insulator of composite fermions in twisted bilayer
graphene

Guangyue Ji  International Center for Quantum Materials, Peking University,
Beijing 100871, China  Junren Shi<EMAIL_ADDRESS>International Center
for Quantum Materials, Peking University, Beijing 100871, China  Collaborative
Innovation Center of Quantum Matter, Beijing 100871, China

###### Abstract

We theoretically study the realization of quantum anomalous Hall insulator
(QAHI) of composite fermions (CFs) in the twisted bilayer graphene (TBG)
system. We show that the moiré pattern in TBG is not only able to provide a
commensurate moiré superlattice, but also a tunable effective periodic
potential necessary for the realization, without the need of imposing an
additional superstructure as in the conventional GaAs system. These make the
TBG an ideal platform for realizing the QAHI of CFs. We establish the phase
diagram with respect to tunable experimental parameters based on the Dirac CF
theory. We find that the topological property of the system depends critically
on the orbital magnetic susceptibility of CFs, which is not specified in the
pristine Dirac CF theory. The experimental realization of QAHI of CFs would be
helpful for unveiling the magnetic property of CFs and clarifying the issue.

## Introduction

The moir$\acute{\text{e}}$ pattern in van der Waals (vdW) heterostructures has
attracted widespread investigations in both theoretical and experimental
condensed matter physics. It provides us a powerful way of precisely
controlling the electronic properties [1, 2, 3, 4] and realizing exotic
quantum states including the unconventional superconductor [5], Mott insulator
[6, 7], and fractional Chern insulator [8]. Moreover, in the presence of a
magnetic field, there is an interplay between the moir$\acute{\text{e}}$
potential and the magnetic length, and people have used this to study the
quantum Hall physics subjected to a periodic modulation. In contrast to
previous works mostly focused on the study of energy spectrums of non-
interacting electrons, i.e., the Hofstadter’s butterfly [9, 10, 11, 12], this
work focuses on the quantum anomalous Hall insulator (QAHI) of composite
fermions (CFs), which is a strongly-correlated topological state consisting of
emergent quasiparticles in a Landau level [13].

A CF consists of an electron and $2p$ quantum vortices, and subjects to a
reduced effective magnetic field [14]. In particular, at even-denominator
magnetic filling $\nu_{\text{m}}=1/2p$, the effective magnetic field vanishes,
and the system is proposed to form a Fermi liquid of CFs (CFL) [15]. The CFL
state has already been observed in various 2D electron systems including the
graphene and its vdW heterostructure [16, 17]. Moreover, inspired by the
Haldane model [18], it is proposed that imposing a proper electrostatic
potential modulation on the CFL state, which is equivalent to an effective
magnetic field modulation for CFs [15, 19], is able to induce the QAHI of CFs
[13]. The exotic state is proposed to be able to exhibit fractional quantum
Hall effect (FQHE) when $p>1$. As a topological phase consisting of emergent
quasiparticles in a strongly-correlated system, which is also proposed to lie
in a duality web in $2+1$ dimension [20, 21], its novel properties are urged
to be tested in experiment.

However, although the proposal of QAHI of CFs has attracted much attention, it
has yet to be realized in experiment as far as we know, let alone test its
novel physical properties. The reason is that realizing the QAHI of CFs
requires the length scale of the modulation potential to be commensurate with
the magnetic length, which is difficult to realize in the conventional GaAs
system. Besides, imposing a superstructure on the GaAs sample to construct the
required modulation potential will inevitably affect the quality of the
sample. These difficulties hinder the experimental realization of the QAHI of
CF in the conventional GaAs system.

In this work, we will focus on its experimental realization in a
moir$\acute{\text{e}}$ heterostructure, i.e., the twisted bilayer graphene
(TBG) system instead. We show that the moiré pattern in TBG is not only able
to provide a commensurate moiré superlattice, but also a tunable effective
periodic potential necessary for the realization, without the need of imposing
an additional superstructure as in the conventional GaAs system. These make
the TBG an ideal platform for realizing the QAHI of CFs. Furthermore, we
establish the phase diagram with respect to tunable experimental parameters
based on the Dirac CF theory. We find that the topological property of the
system depends critically on the orbital magnetic susceptibility of CFs, which
is not specified in the pristine Dirac CF theory [22]. The experimental
realization of QAHI of CFs will be helpful for unveiling the magnetic property
of CFs and classifying the issue.

## TBG system

The QAHI of CFs was first proposed in Ref. [13]. According to the CF theory,
we know CFs feel a reduced effective magnetic field:
$B^{*}=B-2pn_{\text{e}}\Phi_{0}$, where $B$ is the external magnetic filed,
$n_{\text{e}}$ is the electron density, $\Phi_{0}=h/e$ is a quantum flux. At
even-denominator magnetic filling $\nu_{m}\equiv n_{\text{e}}h/eB=1/2p$, CFs
feel zero effective magnetic filed and form the CFL state. Furthermore, if one
imposes a periodic scalar potential modulation on the system, it will induce
an electron density modulation $\delta n_{\text{e}}$, which will further
induce an effective magnetic field modulation $\delta B^{*}$. By tuning
$\delta B^{*}$ in analogy to the Haldane model, it is expected to realize the
QAHI of CFs when integer CF Bloch bands are fully filled. The band filling
factor is defined as $\nu_{\text{e}}\equiv n_{\text{e}}S$, where
$S=3\sqrt{3}L^{2}/2$ is the area of the hexagonal unit cell and $L$ is the
lattice constant. Ref. [13] shows that there are two key conditions in
realizing the QAHI of CFs. One is that the lattice constant $L$ should be
commensurate with the magnetic length $l_{B}=\sqrt{\hbar/eB}$. Concretely, it
requires $L=\sqrt{4\pi\nu_{\text{e}}/3\sqrt{3}\nu_{m}}l_{B}$. The other is a
proper periodic modulation, which should be able to fully open the gap at the
Fermi surface and drive the system into an insulator.

The required conditions can be fulfilled in TBG system with the set-up shown
in Fig. 1(a), where the TBG sample is encapsulated with two dual-gates [23].
This set-up has been widely used to study the properties of bilayer graphene
[23, 16, 24, 25]. By tuning the voltages applied to the top and bottom gates
$V_{t}$ and $V_{b}$, one can independently tune the total electron density
$n_{\text{e}}$ and electrical displacement field $D$. Concretely, the electron
density is given by $n_{\text{e}}=c_{t}V_{t}+c_{b}V_{b}$, where
$c_{t}=\varepsilon_{t}/d_{t}$ ($c_{b}=\varepsilon_{b}/d_{b}$) is the unit-area
capacitance of the top (bottom) gate and it depends on the dielectric constant
and thickness of the dielectric layer between TBG and the top (bottom) gate.
The electrical displacement field is given by $D=(c_{t}V_{t}-c_{b}V_{b})/2$,
which can be tuned to be as large as several volts per nanometer in experiment
[23]. The electrical displacement field will further induce an interlayer bias
$\Gamma$. For the AB-stacked bilayer graphene at the charge neutral point, a
large band gap $\Delta\approx\Gamma\approx 0.25\,\text{eV}$ induced by the
interlayer bias has already been observed in experiment [23]. This indicates
the interlayer bias $\Gamma$ is widely tunable. In the following, we will show
that the two key conditions for realizing the QAHI of CFs can be fulfilled
with the continuously tunable electron density $n_{\text{e}}$ and interlayer
bias $\Gamma$.

First, let’s look at the commensurate condition. For a typical magnetic field
$B\sim 10\,\text{T}$, $L\sim 18\,\text{nm}$ for $\nu_{m}=1/2$ and
$\nu_{\text{e}}=1$. The ratio between $L$ and the carbon-carbon distance of
graphene $a_{0}\approx 1.42\,\mathring{\text{A}}$ is $L/a_{0}\sim 126$, which
corresponds to a large superlattice. On the other hand, we know that the
lattice constant of a TBG superlattice is determined by its twist angle
$\theta$: $L=a_{0}\left|m-n\right|/2\sin(\theta/2)$ [26], where the
superlattice vector is set to be $\bm{L}_{1}=m\bm{a}_{1}+n\bm{a}_{2}$ and
$\bm{a}_{i}$ are the lattice vectors of the graphene. We have two ways to
construct a large superlattice: one is large $m$ and $n$ and small
$\left|m-n\right|$ with $\theta\sim 0$, and the other is large
$\left|m-n\right|$ with $\theta\sim 30^{\circ}$ [2]. By tuning the twist angle
$\theta$, one can easily realize a large superlattice as shown in Fig. 1(b).
Furthermore, by tuning the electron density $n_{\text{e}}$, one can realize
the desired electron filling $\nu_{e}$. Meanwhile, one can tune the applied
magnetic field and realize the desired magnetic filling $\nu_{m}$. In this
way, one can precisely realize the required commensurate condition in this
system.

Figure 1: (a) Schematic of the device with the TBG encapsulated between two
dual-gates. By tuning $V_{t}$ and $V_{b}$, one can independently tune the
electron density $n_{\text{e}}=c_{t}V_{t}+c_{b}V_{b}$ and the electrical
displacement field $D=(c_{t}V_{t}-c_{b}V_{b})/2$, where $c_{t}$ ($c_{b}$) is
the unit-area capacitance of the top (bottom) gate. (b) Moir$\acute{\text{e}}$
Superlattice of TBG. $\bm{L}_{1}=(\sqrt{3}L/2)(-1,\sqrt{3})$ and
$\bm{L}_{2}=-\sqrt{3}L(1,0)$ are its primitive lattice vectors. The lattice
constant $L$ can be tuned with the rotation angle $\theta$. (c) First
Brillouin zone. $\bm{G}_{1}$ and $\bm{G}_{2}$ are the reciprocal lattice
vectors of the moiré superlattice. $K$ ($K^{\theta}$) is the (rotated) Dirac
point. (d) Spatial profiles of the periodic potential $V(\bm{r})$ in Eq. (13)
with $|V_{0}|=1$ and $\nu_{\text{m}}/\nu_{\text{e}}=1/2$.

Next, let’s look at the periodic modulation condition. In the following, we
will show that apart from providing us a moir$\acute{\text{e}}$ superlattice,
the moir$\acute{\text{e}}$ pattern is also able to induce an effective period
potential with tunable strength. The TBG consists of two layers of graphene
with a relative rotation angle. The monolayer graphene is a honeycomb lattice
of carbon atoms, of which each unit cell contains two inequivalent atoms
denoted as A and B. In the reciprocal lattice space near the Dirac point
$\bm{K}=(4\pi/3\sqrt{3}a_{0})(1,0)$, it can be described by the Dirac equation
$H_{\text{G}}=-iv_{\text{e}}\bm{\sigma}\cdot\bm{\nabla}$, where
$\bm{\sigma}=(\sigma_{x},\sigma_{y})$ are Pauli matrices and
$v_{\text{e}}\approx 1\times 10^{6}\,\text{m/s}$ is the Fermi velocity [27].
For the other layer with rotation $\theta$, its Hamiltonian is
$H_{\text{G}}^{\theta}=\bm{\sigma}^{\theta}\cdot(-iv_{\text{e}}\bm{\nabla}-\Delta\bm{K})$,
where
$\bm{\sigma}^{\theta}=e^{i\theta\sigma_{z}/2}(\sigma_{x},\sigma_{y})e^{-i\theta\sigma_{z}/2}$
are the rotated Pauli matrices, $\Delta\bm{K}=\bm{K}^{\theta}-\bm{K}$ is the
displacement of the Dirac point (see Fig. 1c), and
$\bm{K}^{\theta}=R(\theta)\bm{K}$ with $R(\theta)$ being the rotation matrix.
Furthermore considering the interlayer hopping, we obtain the following
Hamiltonian describing the TBG [1, 3, 9]:

$\displaystyle H_{\text{TBG}}$
$\displaystyle=\left(\begin{array}[]{cc}-i\bm{\sigma}\cdot\bm{\nabla}&T(\bm{r})\\\
\left[T(\bm{r})\right]^{\dagger}&\bm{\sigma}^{\theta}\cdot(-i\bm{\nabla}-\Delta\bm{K})\end{array}\right),$
(3)

where $T(\bm{r})=\omega\sum_{\bm{G}}T_{\bm{G}}e^{i\bm{G}\cdot\bm{r}}$ is the
interlayer hopping matrix with

$\displaystyle T_{0}=\left(\begin{array}[]{cc}1&1\\\
1&1\end{array}\right),T_{-\bm{G}_{1}}=\left(\begin{array}[]{cc}e^{i\phi_{0}}&1\\\
e^{-i\phi_{0}}&e^{i\phi_{0}}\end{array}\right),$ (8) $\displaystyle
T_{-\bm{G}_{1}-\bm{G}_{2}}=\left(\begin{array}[]{cc}e^{-i\phi_{0}}&1\\\
e^{i\phi_{0}}&e^{-i\phi_{0}}\end{array}\right)$ (11)

for the initial AB stacked structure,
$\bm{G}_{1}=\sqrt{3}R(-\pi/6)\Delta\bm{K}$ and
$\bm{G}_{2}=\sqrt{3}R(\pi/2)\Delta\bm{K}$ are the reciprocal lattice vectors
of the moiré superlattice, $\omega\approx 0.11\,\text{eV}$ is the hopping
strength and $\phi_{0}=2\pi/3$ [1].

In the presence of a perpendicular magnetic field, the system will form Landau
levels. For a monolayer graphene with a linear dispersion, its LL spectrum is
$E_{n}=\text{sgn}(n)\sqrt{2|n|eB\hbar}v_{\text{e}}$ with $n=0,\pm 1,\ldots$.
In this work, we focus on the spin and valley-polarized zeroth LL (ZLL), which
is reasonable in experiment with strong magnetic field [16] and electric
displacement field [28]. The wavefunctions in the ZLL of layer $1$ have only A
atom component, and we denote them as $|\psi_{1,m}\rangle=(|m\rangle,0)^{T}$,
where $|m\rangle$ are the LLL eigenstates of the conventional massive electron
system. The ZLL eigenstates of layer $2$ are related to layer $1$ and given by
$|\psi_{2,m}\rangle=e^{i\theta\sigma_{z}/2}e^{i\Delta\bm{K}\cdot\bm{r}}|\psi_{1,m}\rangle$.
In the presence of the interlayer coupling $T(\bm{r})$, the ZLLs of the two
layers will couple with each other. In the following, we will show that the
interlayer coupling will induce an effective periodic potential.

In the presence of non-zero interlayer bias $\Gamma$, the layer ZLL energy
degeneracy will be lifted. When $\Gamma$ is much larger than the interlayer
hopping strength $\omega$, the two layers can be well seen as two isolated
layers with the interlayer coupling acting as an effective perturbation
potential. The effective potential induced by $T(\bm{r})$ can be derived by
employing the perturbation theory. Its matrix element (accurate to the second
order of $\omega$) between any two states $\psi_{1,m}$ and
$\psi_{1,m^{\prime}}$ in the ZLL of layer $1$ is

$\displaystyle V_{mm^{\prime}}$
$\displaystyle=\frac{\omega^{2}}{\Gamma}\sum_{l,\bm{G},\bm{G}^{\prime}}T_{\bm{G}}^{11}(T_{\bm{G}^{\prime}}^{11})^{*}\langle
m|e^{i\bm{q}\cdot\bm{r}}|l\rangle\langle
l|e^{-i\bm{q}^{\prime}\cdot\bm{r}}|m^{\prime}\rangle,$ (12)

where $\bm{q}=\bm{G}+\Delta\bm{K}$. Note that $\sum_{l}|l\rangle\langle l|$ is
in fact the conventional LLL projection operator $\hat{P}_{\text{LLL}}$ [29].
By using the commutation relation of projected plane wave factors [29], one
can show $\sum_{l}\langle m|e^{i\bm{q}\cdot\bm{r}}|l\rangle\langle
l|e^{-i\bm{q}^{\prime}\cdot\bm{r}}|m^{\prime}\rangle=e^{-\frac{1}{2}q^{\prime}\bar{q}}\langle
m|e^{i(\bm{q}-\bm{q}^{\prime})\cdot\bm{r}}|m^{\prime}\rangle$, where
$q=q_{x}+iq_{y}$ and $\bar{q}=q_{x}-iq_{y}$. By substituting it and the
hopping matrix Eq. (11) into Eq. (12), we derive $V_{mm^{\prime}}=\langle
m|V(\bm{r})|m^{\prime}\rangle$ with the effective potential

$\displaystyle V(\bm{r})$
$\displaystyle=V_{0}\left(\sum_{i=1}^{2}e^{i\bm{G}_{i}\cdot\bm{r}}+e^{-i(\bm{G}_{1}+\bm{G}_{2})\cdot\bm{r}}\right)+\text{h.c.},$
(13)

where
$V_{0}=\frac{\omega^{2}}{\Gamma}\exp\left[-(|q|^{2}l_{B}^{2}/2)e^{i2\pi/3}\right]$,
$|q|=|\Delta\bm{K}|=2|\bm{K}|\sin(\theta/2)$, and an irrelevant phase factor
$e^{-i\phi_{0}}$ has been removed through a translation of the original point
[30]. Furthermore, by using the commensurate condition, it can be simplified
to be

$V_{0}=\frac{\omega^{2}}{\Gamma}\exp\left[\alpha/\sqrt{3}\right]\exp\left[-i\alpha\right],$
(14)

where $\alpha=\frac{\pi}{3}\frac{\nu_{\text{m}}}{\nu_{\text{e}}}$ is the phase
factor and it is determined by the ratio of magnetic and band filling factors.
The spatial profile of $V(\bm{r})$ with $|V_{0}|=1$ and
$\nu_{\text{m}}/\nu_{\text{e}}=1/2$ is plotted in Fig. 1(d), and it is
triangular. Since the interlayer bias $\Gamma$ is widely tunable, the
effective potential $V_{0}$ induced by the moiŕe pattern is also widely
tunable. Thus, it is expected to fulfill the required periodic modulation
condition without much difficulty.

## Dirac composite fermions

In the following, we study the QAHI of CFs subjected to the periodic potential
Eq. (13) by adopting the Dirac CF theory, in which the particle-hole symmetry
is explicit [22]. The Dirac CF theory treats CFs as neutral massless Dirac
particles. The motion of Dirac CFs is governed by the Dirac equation
$i\partial_{t}\psi=\hat{H}\psi$, where $\psi$ is the Dirac CF spinor, and the
Hamiltonian

$\displaystyle\hat{H}$
$\displaystyle=\bm{\sigma}\cdot(-i\bm{\partial}-\bm{a}(\bm{r}))+a_{0}(\bm{r})\sigma_{0},$
(15)

where $a^{\mu}=(a^{0},\bm{a})$ are the internal gauge fields, $\sigma_{0}$ is
the identity matrix, we have set $e=\hbar=v_{\text{CF}}=1$ and will restore
these constants when appropriate. Meanwhile, the CF system subjects to the
constraints [22]

$\displaystyle\rho_{\textrm{CF}}$
$\displaystyle\equiv\left\langle\psi^{\dagger}\psi\right\rangle=\frac{B}{4\pi},$
(16) $\displaystyle\bm{j}_{\textrm{CF}}$
$\displaystyle\equiv\left\langle\psi^{\dagger}\bm{\sigma}\psi\right\rangle=-\frac{1}{4\pi}\hat{z}\times\bm{E},$
(17)

where $\rho_{\textrm{CF}}$ and $\bm{j}_{\textrm{CF}}$ are the CF density and
current, respectively, $\bm{B}=B\hat{z}$ and $\bm{E}$ are the external
electromagnetic fields, and $\left\langle\cdots\right\rangle$ denotes the
grand canonical ensemble average. The CF density is set by the external
magnetic field, but not the electron density as assumed in the HLR theory
[15]. It is a manifestation of the particle–vortex duality with the electron
system. In the presence of the scalar potential $V(\bm{r})$, the CF current is
fixed to be $\bm{j}_{\textrm{CF}}=(1/4\pi)\hat{z}\times\bm{\nabla}V(\bm{r})$
according to Eq. (17). To induce a CF current satisfying this constraint, the
internal magnetic field $\bm{b}(\bm{r})$ coupled to CFs needs to adjust
accordingly. By tuning it properly, it is expected to realize the QAHI of
Dirac CFs.

Different from systems governed by the Schrödinger equation, there exists a
Dirac sea consisting of all negative-energy states. When a modulation is
imposed on the system, not only positive-energy states will respond to it, but
also the Dirac sea. In Ref. [31], Principi et al. show that the current
response of the Dirac sea to a vector potential is divergent. Thus, to
evaluate the physical response current, we shall regularize the system first.

We regularize the system by employing the Pauli-Villars regularization scheme
for $(2+1)$-dimensional massless Dirac fermions proposed by Redlich in Refs.
[32, 33]. The regularization scheme is as follows. For a massless Dirac
system, one can introduce two auxiliary massive systems with the Hamiltonian
$\hat{H}_{m}=\hat{H}\pm m\sigma^{3}$, where $m$ is the mass and its magnitude
is required to be much larger than the chemical potential, i.e., $m\gg\mu$. By
canceling out the divergent term in our massless system with the auxiliary
systems, we obtain a finite regularized current

$\bm{j}^{\text{R}}\equiv\bm{j}_{m=0}-(\bm{j}_{m}+\bm{j}_{-m})/2+\nabla\times(\chi_{m}^{\text{mag}}\bm{b}),$
(18)

where $\bm{j}_{\pm m}$ are the response currents of the massive systems. In
addition to a divergent term, $\bm{j}_{\pm m}$ also contains finite Chern-
Simons current and magnetization current
$\nabla\times(\chi_{m}^{\text{mag}}\bm{b})$, where
$\chi_{m}^{\text{mag}}=1/12\pi m$ is the orbital magnetic susceptibility [34].
Since the Chern-Simons current is odd with respect to the mass, we eliminate
it by introducing two auxiliary massive systems with opposite masses. Because
the magnetization current is even with respect to the mass, we need to
subtract its contribution manually as shown in Eq. (18).

To check the correctness of the above regularization scheme, we numerically
calculate the orbital magnetization susceptibility
$\chi_{\text{CF}}^{\text{mag}}$ of our massless Dirac system in the linear
response region. First, we solve the eigenequation
$\hat{H}\psi_{n,\bm{k}}(\bm{r})=E_{n}(\bm{k})\psi_{n,\bm{k}}(\bm{r})$
subjected to a given $\bm{b}$ for both massive and massless systems by
employing the plane wave expansion method. Next, we calculate the current
$\bm{j}_{\textrm{m=0}}$ and $\bm{j}_{\textrm{m}}$ with derived eigenstates,
and derive the regularized CF current $\bm{j}^{\text{R}}$ by applying Eq.
(18). Then, we can derive $\chi_{\text{CF}}^{\text{mag}}$ directly. The
results show that $\chi_{\text{CF}}^{\text{mag}}\rightarrow 0$ with the
increase of the momentum cut-off $\Lambda=n_{\Lambda}|\bm{G}_{i}|$ (see Fig.
2(a)). This is consistent with the analytic result that the orbital magnetic
susceptibility of massless Dirac fermions at $\mu>0$ is zero [34].

Figure 2: (a) CF magnetic susceptibility $\chi_{\text{CF}}^{\text{mag}}$ vs
momentum cut-off $\Lambda=n_{\Lambda}|\bm{G}_{i}|$. The red line denotes the
magnetic susceptibility of massless Dirac CFs. It approaches zero with the
increase of the momentum cut-off. The horizontal cyan dashed line denotes
$\chi_{m}^{\text{mag}}=1/12\pi m$ with $m=5$. (b) Induced Chern-Simons field
strength (in unit of $\hbar/eL^{2}$) vs momentum cut-off. The modulation
strength $|V_{0}|$ is set to be $5\times 10^{-3}$. The induced Chern-Simons
magnetic field is divergent with the increase of the momentum cut-off. It
results from the zero CF magnetization susceptibility implicitly assumed in
the Dirac CF theory.

However, a zero CF magnetization susceptibility will lead to an infinite
electron density response. The density susceptibility of electrons
$\chi_{\text{e}}$ is inversely proportional to the magnetic susceptibility of
CFs $\chi_{\text{CF}}^{\text{mag}}$ as follows [22]

$\chi_{\text{e}}=\left(\frac{e}{2h}\right)^{2}\frac{1}{\chi_{\text{CF}}^{\text{mag}}}.$
(19)

Note that though different CF effective theories treat CFs as different types
of particles, the above relation is identical [22, 35, 36]. By substituting
$\chi_{\text{CF}}^{\text{mag}}=0$ into Eq. (19), one will obtain an infinite
density response. That is to say a finite scalar modulation $V$ will induce an
infinite electron density fluctuation, which is obviously unphysical. In our
numerical study, we find that the induced magnetic filed $\bm{b}$, which is
proportional to the electron density fluctuation given by
$b=B-4\pi\rho_{\textrm{e}}$, is divergent with the increase of the cut-off as
shown in Fig. 2(b). This also indicates that the density response of the
system is divergent.

As we can see, the unphysical infinite density response results from the zero
magnetization susceptibility of CFs implicitly assumed in the Dirac CF theory.
In fact, the Dirac theory is a long-wavelength low-energy effective theory,
while the magnetization is an effect of higher order [36, 37]. In the
following, we proceed by introducing a finite orbital magnetic susceptibility
of CFs $\chi_{\text{CF}}^{\text{mag}}$ into the Dirac CF theory and treating
it as an extra parameter. In this case, there is an extra orbital
magnetization current and the CF current in Eq. (17) needs to be modified to
be

$\bm{j}_{\textrm{CF}}=\left\langle\psi^{\dagger}\bm{\sigma}\psi\right\rangle+\nabla\times(\chi_{\text{CF}}^{\text{mag}}\bm{b}),$
(20)

where the $\chi_{\text{CF}}^{\text{mag}}\bm{b}$ is the orbital magnetization.
In this way, we solve the density response divergence problem in the pristine
Dirac CF theory. Furthermore, by carefully tuning the periodic modulation
potential $V(\bm{r})$, we show the system can form the QAHI of Dirac CFs.
Meanwhile, we find that the topological property of the system depends
critically on the magnetic susceptibility of CFs.

## QAHI of Dirac CFs

In the absence of modulation, CFs are free and form a Dirac cone with the
Fermi wave vector $k_{\text{F}}=\sqrt{eB/\hbar}$. By folding the linear
dispersion in the first BZ, we can obtain the energy bands of free Dirac CFs.
They will touch at points of high symmetry. In the presence of a periodic
modulation $V(\bm{r})$, energy degeneracies will be partially lifted. With an
appropriate modulation strength and phase, the gap around the Fermi surface
can be fully open, and the system forms an insulating state as shown in Fig.
3(b). The whole phase diagram for $\nu_{\text{m}}=1/2$ is shown in Fig. 3(a).
Note that the system will not form an insulating state at $\nu_{\text{e}}=1$,
because the degeneracy between the first two positive bands at $K$ point can
not be lifted with the triangular potential $V(\bm{r})$ shown in Fig. 1(d).

Figure 3: (a) Phase diagrams of Dirac CFs for $\nu_{\text{m}}=1/2$. The area
of insulating states is marked with cyan. $\mu_{\text{CF}}$ is the chemical
potential of CFs. The energy unit is $\hbar v_{\textrm{CF}}/L$. The magnetic
susceptibility of CFs is set to be $\chi_{\text{CF}}^{\text{mag}}=-1/8\pi
m_{\text{CF}}$. The critical bias (marked with red circles) is
$\Gamma_{\text{c}}\approx\omega^{2}/(0.6\hbar v_{\textrm{CF}}/L)\approx
3.4\epsilon B^{-1/2}\,\text{eV}$ for $\nu_{\text{e}}=2$ and
$\Gamma_{\text{c}}\approx 8.4\epsilon B^{-1/2}\,\text{eV}$ for
$\nu_{\text{e}}=3$. (b) Energy bands of Dirac CFs at $\omega^{2}/\Gamma=0.5$
and $\nu_{\text{e}}=3$ (marked with a cross in (a)). It can seen that the
energy degeneracy near the Fermi surface has been fully lifted with the given
periodic potential modulation. The CF Hall conductivity of the insulating
state is $-e^{2}/2h$ ($+e^{2}/2h$) $\text{for
}\Gamma\chi_{\text{CF}}^{\text{mag}}<0$
($\Gamma\chi_{\text{CF}}^{\text{mag}}>0$). The corresponding electron Hall
conductivity is $e^{2}/h$ (0).

In the process of deriving the effective potential $V(\bm{r})$, it’s assumed
that the interlayer hopping strength $\omega$ is much smaller than the
interlayer bias $\Gamma$ and can be treated as a perturbation. Now, let’s
check its rationality. In the case of $\nu_{\text{m}}=1/2$ and
$\nu_{\text{e}}=2$, the ratio between $\omega$ and the critical bias
$\Gamma_{\text{c}}$ is $\omega/\Gamma_{\text{c}}\approx 0.6\hbar
v_{\textrm{CF}}/L\omega$, where we have set the magnetic susceptibility of CFs
$\chi_{\text{CF}}^{\text{mag}}=-1/8\pi m_{\text{CF}}$ as done in Refs. [35,
36] and $m_{\text{CF}}$ is the effective mass of CFs [15, 19]. By substituting
the CF velocity $v_{\text{CF}}=\hbar
k_{\text{F}}/m_{\text{CF}}\approx(7.2/\epsilon)\times 10^{5}\,\text{m/s}$ and
the commensurate lattice constant
$L=\sqrt{4\pi\nu_{\text{e}}/3\sqrt{3}\nu_{m}}l_{B}$ into the ratio, we obtain
$\omega/\Gamma_{\text{c}}\approx(3.2/\epsilon)\times 10^{-2}B^{1/2}$, where
$\epsilon$ is the dielectric constant and it is determined by the substrates.
For the typical experimental parameters $\epsilon\sim 3-15$ and $B\sim
10\,\text{T}$ [38], $\omega/\Gamma_{\text{c}}\ll 1$. Thus, there is a wide
range of $\Gamma$ satisfying $\omega\ll\Gamma<\Gamma_{\text{c}}$. Since the
interlayer bias $\Gamma$ is widely tunable, e.g., $\Gamma\approx
0.25\,\text{eV}>\omega\approx 0.11\,\text{eV}$ has already been realized in
experiment [23], it is reasonable to treat the interlayer hopping as a
perturbation to the CFL state.

Next, we analyze the topological property of the insulating state by
calculating its Hall conductivity. For a non-degenerate band, one can evaluate
its Hall conductivity by applying the TKNN formula:
$\sigma^{xy}=-\mathcal{C}/2\pi=-(1/2\pi)\int_{\textrm{BZ}}(d^{2}k/2\pi)\Omega_{k_{x}k_{y}}$,
where $\Omega_{k_{x}k_{y}}$ is the Berry curvature in the momentum space, and
$\mathcal{C}$ is the Chern number of the band [39]. Naively, one expects to
derive the Hall conductivity of the system by summing up the Chern number of
all occupied positive bands. However, this is incorrect for the Dirac system
because it ignores the contribution of the filled Dirac sea. To correctly
evaluate the Hall conductivity of the Dirac system, we regularize the system
by introducing auxiliary massive systems as done in Eq. (18). The expression
of the regularized Hall conductance is given by

$\sigma_{\text{CF}}^{xy}\equiv\sigma_{m=0}^{xy}-(\sigma_{m}^{xy}+\sigma_{-m}^{xy})/2,$
(21)

where $\sigma_{m=0}^{xy}$ ($\sigma_{m}^{xy}$) is the total Hall conductivity
of all occupied bands below the chemical potential $\mu_{\text{CF}}$ ($\ll m$)
of the massless (massive) Dirac system. In numerical calculations, we evaluate
$\sigma_{m}^{xy}$ by setting a momentum cut-off
$\Lambda=n_{\Lambda}|\bm{G}_{i}|$ as done before. Since
$\sigma_{\text{CF}}^{xy}$ is a topological quantity independent of the cut-
off, it can be easily evaluated by setting a small $n_{\Lambda}$. Finally, we
obtain the Hall conductivity of the insulating state, and it is (in unit of
$e^{2}/h$)

$\sigma_{\text{CF}}^{xy}=\begin{cases}-\frac{1}{2},&\text{for
}\Gamma\chi_{\text{CF}}^{\text{mag}}<0\\\ +\frac{1}{2},&\text{for
}\Gamma\chi_{\text{CF}}^{\text{mag}}>0\end{cases}.$ (22)

The half-integer quantization is a character of Dirac particles [40, 41]. It
can be seen that the Hall conductivity of the system critically depends on the
orbital magnetization susceptibility of CFs. The reason is that the induced
internal magnetic field are opposite for positive and negative
$\chi_{\text{CF}}^{\text{mag}}$.

With the CF Hall conductivity determined, we can derive the electron Hall
conductivity by applying the conductivity transformation relation
$\sigma_{\text{e}}^{xy}=e^{2}/2h-(e^{2}/2h)^{2}/\sigma_{\text{CF}}^{xy}$ [22].
By substituting the CF conductivity Eq. (22) into the relation, we obtain the
electron Hall conductivity (in unit of $e^{2}/h$)

$\sigma_{\text{e}}^{xy}=\begin{cases}1,&\text{for
}\Gamma\chi_{\text{CF}}^{\text{mag}}<0\\\ 0,&\text{for
}\Gamma\chi_{\text{CF}}^{\text{mag}}>0\end{cases}.$ (23)

For $\Gamma\chi_{\text{CF}}^{\text{mag}}<0$, its Hall conductivity is nonzero
and the insulating state is a topological insulator. Thus, the effective
periodic potential induced by the interlayer hopping of the TBG system can
indeed drive the CFL state into the QAHI of CFs.

In this work, we focus on the TBG system and the effective potential given by
Eq. (13). In fact, the phase and strength of the effective potential can be
further tuned by coupling more layers [42, 43, 4]. By using the interference
between effective potentials induced by different layers, it is flexible to
realize a desired potential. In this case, it is also possible to construct a
hexagonal periodic potential and realize the QAHI of CFs at
$\nu_{\text{e}}=1$. Besides, one can also tune the effective potential by
other methods, including coupling to substrates and tuning the strain [44, 11,
24, 45]. In short, the moir$\acute{\text{e}}$ pattern in vdW heterostructures
provides us a powerful way of controlling the electronic properties to realize
the QAHI of CFs.

For the theoretical part, a full understanding of orbital magnetic
susceptibility of CFs is still absent and relevant investigations are rare.
There even exists a misconception that its effect on observable physical
quantities is negligible. However, we reveal that it in fact plays a central
role in the QAHI of CFs. In particular, the zero magnetic susceptibility of
CFs implicitly assumed in the Dirac theory will lead to an unphysical infinite
density response. Thus, the experimental realization of the QAHI of CFs would
be helpful for unveiling the magnetic property of CFs and clarifying the
issue.

## Summary

In summary, we show that the moiré superlattice and effective periodic
potential induced by the the moiré pattern make TBG an ideal platform for
realizing the QAHI of CFs. We establish the phase diagram with respect to
tunable experimental parameters based on the Dirac CF theory. We find that the
topological property of the system depends critically on the orbital magnetic
susceptibility of CFs, which is not specified in the pristine Dirac CF theory.
The experimental realization of QAHI of CFs would be helpful for clarifying
the issue.

###### Acknowledgements.

G.J. and J.S. thank Qian Niu, Di Xiao, Jing-Yuan Chen, Yinhan Zhang, Haoran
Chen and Xuesong Hu for valuable discussions. This work is supported by the
National Basic Research Program of China (973 Program) Grants No.
2018YFA0305603 and No. 2021YFA1401900 and the National Science Foundation of
China Grant No. 12174005.

## References

  * Lopes dos Santos  _et al._ [2007] J. M. B. Lopes dos Santos, N. M. R. Peres, and A. H. Castro Neto, Phys. Rev. Lett. 99, 256802 (2007).
  * Trambly de Laissardière  _et al._ [2010] G. Trambly de Laissardière, D. Mayou, and L. Magaud, Nano Lett. 10, 804 (2010).
  * Bistritzer and MacDonald [2011a] R. Bistritzer and A. H. MacDonald, Proceedings of the National Academy of Sciences 108, 12233 (2011a).
  * Carr  _et al._ [2020] S. Carr, S. Fang, and E. Kaxiras, Nature Reviews Materials 5, 748 (2020).
  * Cao  _et al._ [2018a] Y. Cao, V. Fatemi, S. Fang, K. Watanabe, T. Taniguchi, E. Kaxiras, and P. Jarillo-Herrero, Nature 556, 43 (2018a).
  * Cao  _et al._ [2018b] Y. Cao, V. Fatemi, A. Demir, S. Fang, S. L. Tomarken, J. Y. Luo, J. D. Sanchez-Yamagishi, K. Watanabe, T. Taniguchi, E. Kaxiras, R. C. Ashoori, and P. Jarillo-Herrero, Nature 556, 80 (2018b).
  * Chen  _et al._ [2019] G. Chen, L. Jiang, S. Wu, B. Lyu, H. Li, B. L. Chittari, K. Watanabe, T. Taniguchi, Z. Shi, J. Jung, Y. Zhang, and F. Wang, Nat. Phys. 15, 237 (2019).
  * Xie  _et al._ [2021] Y. Xie, A. T. Pierce, J. M. Park, D. E. Parker, E. Khalaf, P. Ledwith, Y. Cao, S. H. Lee, S. Chen, P. R. Forrester, K. Watanabe, T. Taniguchi, A. Vishwanath, P. Jarillo-Herrero, and A. Yacoby, Nature 600, 439 (2021).
  * Bistritzer and MacDonald [2011b] R. Bistritzer and A. H. MacDonald, Phys. Rev. B 84, 035440 (2011b).
  * Dean  _et al._ [2013] C. R. Dean, L. Wang, P. Maher, C. Forsythe, F. Ghahari, Y. Gao, J. Katoch, M. Ishigami, P. Moon, M. Koshino, T. Taniguchi, K. Watanabe, K. L. Shepard, J. Hone, and P. Kim, Nature 497, 598 (2013).
  * Hunt  _et al._ [2013] B. Hunt, J. D. Sanchez-Yamagishi, A. F. Young, M. Yankowitz, B. J. LeRoy, K. Watanabe, T. Taniguchi, P. Moon, M. Koshino, P. Jarillo-Herrero, and R. C. Ashoori, Science 340, 1427 (2013).
  * Ponomarenko  _et al._ [2013] L. A. Ponomarenko, R. V. Gorbachev, G. L. Yu, D. C. Elias, R. Jalil, A. A. Patel, A. Mishchenko, A. S. Mayorov, C. R. Woods, J. R. Wallbank, M. Mucha-Kruczynski, B. A. Piot, M. Potemski, I. V. Grigorieva, K. S. Novoselov, F. Guinea, V. I. Fal’ko, and A. K. Geim, Nature 497, 594 (2013).
  * Zhang and Shi [2014] Y. Zhang and J. Shi, Phys. Rev. Lett. 113, 016801 (2014).
  * Jain [1989] J. K. Jain, Phys. Rev. Lett. 63, 199 (1989).
  * Halperin  _et al._ [1993] B. I. Halperin, P. A. Lee, and N. Read, Phys. Rev. B 47, 7312 (1993).
  * Li  _et al._ [2017] J. I. A. Li, C. Tan, S. Chen, Y. Zeng, T. Taniguchi, K. Watanabe, J. Hone, and C. R. Dean, Science 358, 648 (2017).
  * Zibrov  _et al._ [2017] A. A. Zibrov, C. Kometter, H. Zhou, E. M. Spanton, T. Taniguchi, K. Watanabe, M. P. Zaletel, and A. F. Young, Nature 549, 360 (2017).
  * Haldane [1988] F. D. M. Haldane, Phys. Rev. Lett. 61, 2015 (1988).
  * Jain [2007] J. Jain,  _Composite Fermions_ (Cambridge University Press, 2007).
  * Seiberg  _et al._ [2016] N. Seiberg, T. Senthil, C. Wang, and E. Witten, Annals of Physics 374, 395 (2016).
  * Senthil  _et al._ [2019] T. Senthil, D. T. Son, C. Wang, and C. Xu, Physics Reports 827, 1 (2019).
  * Son [2015] D. T. Son, Phys. Rev. X 5, 031027 (2015).
  * Zhang  _et al._ [2009] Y. Zhang, T.-T. Tang, C. Girit, Z. Hao, M. C. Martin, A. Zettl, M. F. Crommie, Y. R. Shen, and F. Wang, Nature 459, 820 (2009).
  * Spanton  _et al._ [2018] E. M. Spanton, A. A. Zibrov, H. Zhou, T. Taniguchi, K. Watanabe, M. P. Zaletel, and A. F. Young, Science 360, 62 (2018).
  * Polshyn  _et al._ [2022] H. Polshyn, Y. Zhang, M. A. Kumar, T. Soejima, P. Ledwith, K. Watanabe, T. Taniguchi, A. Vishwanath, M. P. Zaletel, and A. F. Young, Nature Physics 18, 42 (2022).
  * Moon and Koshino [2012] P. Moon and M. Koshino, Phys. Rev. B 85, 195458 (2012).
  * Castro Neto  _et al._ [2009] A. H. Castro Neto, F. Guinea, N. M. R. Peres, K. S. Novoselov, and A. K. Geim, Rev. Mod. Phys. 81, 109 (2009).
  * Huang  _et al._ [2022] K. Huang, H. Fu, D. R. Hickey, N. Alem, X. Lin, K. Watanabe, T. Taniguchi, and J. Zhu, Phys. Rev. X 12, 031019 (2022).
  * Girvin and Jach [1984] S. M. Girvin and T. Jach, Phys. Rev. B 29, 5617 (1984).
  * Zhang  _et al._ [2020] Y. Zhang, Y. Gao, and D. Xiao, Phys. Rev. B 101, 041410 (2020), arXiv:1910.09001 [cond-mat].
  * Principi  _et al._ [2009] A. Principi, M. Polini, and G. Vignale, Phys. Rev. B 80, 075418 (2009).
  * Redlich [1984a] A. N. Redlich, Phys. Rev. Lett. 52, 18 (1984a).
  * Redlich [1984b] A. N. Redlich, Phys. Rev. D 29, 2366 (1984b).
  * Koshino and Ando [2011] M. Koshino and T. Ando, J. Phys.: Conf. Ser. 334, 012005 (2011).
  * Wang  _et al._ [2017] C. Wang, N. R. Cooper, B. I. Halperin, and A. Stern, Phys. Rev. X 7, 031029 (2017).
  * Ji and Shi [2021] G. Ji and J. Shi, Phys. Rev. Research 3, 043055 (2021).
  * Ji and Shi [2020] G. Ji and J. Shi, Phys. Rev. Research 2, 033329 (2020).
  * Tőke and Jain [2007] C. Tőke and J. K. Jain, Phys. Rev. B 75, 245440 (2007).
  * Thouless  _et al._ [1982] D. J. Thouless, M. Kohmoto, M. P. Nightingale, and M. den Nijs, Phys. Rev. Lett. 49, 405 (1982).
  * Schakel [1991] A. M. J. Schakel, Phys. Rev. D 43, 1428 (1991).
  * Mikitik and Sharlai [1999] G. P. Mikitik and Y. V. Sharlai, Phys. Rev. Lett. 82, 2147 (1999).
  * Lui  _et al._ [2011] C. H. Lui, Z. Li, K. F. Mak, E. Cappelluti, and T. F. Heinz, Nature Phys 7, 944 (2011).
  * Bao  _et al._ [2011] W. Bao, L. Jing, J. Velasco, Y. Lee, G. Liu, D. Tran, B. Standley, M. Aykol, S. B. Cronin, D. Smirnov, M. Koshino, E. McCann, M. Bockrath, and C. N. Lau, Nature Phys 7, 948 (2011).
  * Zhou  _et al._ [2007] S. Y. Zhou, G.-H. Gweon, A. V. Fedorov, P. N. First, W. A. de Heer, D.-H. Lee, F. Guinea, A. H. Castro Neto, and A. Lanzara, Nature Mater 6, 770 (2007).
  * Choi  _et al._ [2010] S.-M. Choi, S.-H. Jhi, and Y.-W. Son, Nano Lett. 10, 3486 (2010).