# Probing the Anisotropic Fermi Surface in Tetralayer Graphene via Transverse
Magnetic Focusing

Illias Klanurak  Department of Physics, Faculty of Science, Chulalongkorn
University, Patumwan, Bangkok 10330, Thailand  Kenji Watanabe  Research Center
for Electronic and Optical Materials, National Institute for Materials
Science, 1-1 Namiki, Tsukuba 305-0044, Japan  Takashi Taniguchi  Research
Center for Materials Nanoarchitectonics, National Institute for Materials
Science, 1-1 Namiki, Tsukuba 305-0044, Japan  Sojiphong Chatraphorn
Department of Physics, Faculty of Science, Chulalongkorn University, Patumwan,
Bangkok 10330, Thailand  Thiti Taychatanapat<EMAIL_ADDRESS>Department
of Physics, Faculty of Science, Chulalongkorn University, Patumwan, Bangkok
10330, Thailand

###### Abstract

Bernal-stacked tetralayer graphene (4LG) exhibits intriguing low-energy
properties, featuring two massive subbands and showcasing diverse features of
topologically distinct, anisotropic Fermi surfaces, including Lifshitz
transitions and trigonal warping. Here, we study the influence of the band
structure on electron dynamics within 4LG using transverse magnetic focusing.
Our analysis reveals two distinct focusing peaks corresponding to the two
subbands. Furthermore, we uncover a pronounced dependence of the focusing
spectra on crystal orientations, indicative of an anisotropic Fermi surface.
Utilizing the semiclassical model, we attribute this orientation-dependent
behavior to the trigonal warping of the band structure. This phenomenon leads
to variations in electron trajectories based on crystal orientation. Our
findings not only enhance our understanding of the dynamics of electrons in
4LG, but also offer a promising method for probing anisotropic Fermi surfaces
in other materials.

The distinct electronic properties of multilayer graphene, in contrast to
monolayer graphene, are shaped by unique features [1, 2, 3, 4]. Trigonal
warping (TW) plays a pivotal role in this differentiation, inducing a
distortion of the circular Fermi surface near $K$ and $K^{\prime}$ points in
the graphene Brillouin zone due to interlayer interactions [5]. This effect
introduces a discrete 3-fold rotational symmetry, breaking the continuous
rotational symmetry of the Fermi surface and causing anisotropy in the low-
energy band structure [5]. Numerous studies have emphasized the significance
of TW, associating it with diverse phenomena such as correlated phases in
bilayer graphene [6], emergent Dirac gullies [7], Landau level hybridization
in trilayer graphene [8], Lifshitz transitions in Bernal-stacked tetralayer
graphene (4LG) [9], and topological phase transitions in twisted double
bilayer graphene [10]. Recognizing the importance of the TW effect is crucial
for anticipating the transport characteristics of materials

Although scanning tunneling microscopy has facilitated quasi-direct
observation of the TW effect in graphene [11], its elucidation in transport
experiments remains challenging. Prior methods, including ballistic transport
in bilayer graphene with superimposed antidot arrays [12] and quantum point
contacts [13], have been employed. However, these approaches could introduce
inhomogeneity that can mask the intrinsic transport characteristics of bilayer
graphene [14]. In our study, we leverage transverse magnetic focusing (TMF) on
4LG to investigate the low-energy anisotropic band structure associated with
TW. By applying a transverse magnetic field to guide electron trajectories,
TMF enables us to derive insights into the electronic properties of materials
[15, 16, 17, 18, 19, 20, 21, 22, 23].

Figure 1: (a) Electronic band structure of 4LG. The inner and outer conduction
bands are labeled by $\alpha_{\mathrm{in}}$ and $\alpha_{\mathrm{out}}$ while
the inner and outer valence bands are labeled by $\beta_{\mathrm{in}}$ and
$\beta_{\mathrm{out}}$. (b) Fermi surfaces of 4LG at 4 different Fermi
energies. The shading indicates filled states for each band. (c) Electron
trajectories in real space at $E_{\mathrm{F}}$ = 40 meV and magnetic field of
$0.2$ T. The pink and orange trajectories corresponds to electrons from
$\alpha_{\mathrm{in}}$ and $\alpha_{\mathrm{out}}$ bands, respectively. (d)
Simulated trajectories of electrons injected through point O in all directions
under the same conditions as in (c). The angle $\phi$ measures the orientation
of the device’s edge relative to the zigzag edge of 4LG. The red dots indicate
the focusing points at which the caustic lines intersect the device’s edge.
(e) Focusing length at $0.2$ T and focusing field for focusing length of $1.6$
$\mu$m as a function of $\phi$ for electron trajectories in (d).

The advantage of using 4LG in this study lies in its band structure, which
hosts charge carriers with two distinct effective masses [24, 25]. We denote
the inner and outer electron (hole) bands as $\alpha_{\mathrm{in}}$
($\beta_{\mathrm{in}}$) and $\alpha_{\mathrm{out}}$ ($\beta_{\mathrm{out}}$),
respectively, as illustrated in Figure 1a. The band structure of 4LG is
parameterized by the Slonczewski-Weiss-McClure (SWMcC) parameters of graphite
[5] ($\gamma_{0}$-$\gamma_{5}$ and $\delta$). Notably, the interlayer hopping
parameter between non-dimer sites, $\gamma_{3}$, is identified as a key
contributor to the TW effect in multilayer graphene [5].

The influence of $\gamma_{3}$ on each band at specific Fermi energies
$E_{\mathrm{F}}$ can be visualized by the shape of Fermi surfaces, as shown in
Figure 1b. For energy $|\epsilon(\mathbf{k})|>8$ meV, the Fermi surfaces at
each $K$ point consist of two pockets, with those from $\alpha_{\mathrm{out}}$
and $\beta_{\mathrm{out}}$ bands exhibiting more distortion than those of
$\alpha_{\mathrm{in}}$ and $\beta_{\mathrm{in}}$ bands. We propose that the
distinct degrees of TW between the inner and outer bands can be discerned
through ballistic magnetotransport measurements utilizing the TMF technique.

In TMF technique, we inject charge carriers into a two-dimensional electron
gas subjected to a small perpendicular magnetic field $B$. Employing the
semiclassical framework[26], the trajectories of charge carriers are dictated
by the energy dispersion $\epsilon(\mathbf{k})$ at $E_{\mathrm{F}}$. The
evolution of position $\mathbf{r}$ and momentum $\hbar\mathbf{k}$ for the
carriers is governed by the semiclassical equation:

$\dot{\mathbf{r}}=\frac{1}{\hbar}\frac{\mathrm{d}\epsilon}{\mathrm{d}\mathbf{k}}\
,\ \hbar\dot{\mathbf{k}}=-e\dot{\mathbf{r}}\times B\hat{z}$

where $\hbar$ is the reduced Planck constant and $e$ is the elementary charge.
In essence, the real-space trajectory of charge carriers mirrors the shape of
their Fermi surfaces but is scaled by $\hbar/eB$ and rotated by 90∘. As
depicted in Figure 1c, electron trajectories in real space at
$E_{\mathrm{F}}=40$ meV under $B=0.2$ T are illustrated, with the $x$-axis
aligned with the zigzag edge of 4LG. Electrons within each pocket trace closed
orbits with their direction determined by the Lorentz force. Notably,
electrons from the $\alpha_{\mathrm{in}}$ pocket (pink-color path) exhibit
smaller orbits in real space compared to those in the $\alpha_{\mathrm{out}}$
pocket (orange-color path) due to the smaller size of their Fermi surface.

In Figure 1d, we depict the trajectories of electrons emanating from a point
$O$ in various directions, all under identical energy and magnetic field
conditions as those shown in Figure 1c. This illustration captures how each
set of trajectories within a band collectively evolves into a caustic, a zone
characterized by a significantly high concentration of carriers. This results
in a pronounced focusing pathway for these carriers. Notably, the
$\alpha_{\mathrm{out}}$ caustic exhibits heightened sensitivity to TW, leading
to its adoption of a hexagonal shape, in contrast to the
$\alpha_{\mathrm{in}}$ caustic, which retains a more circular form. The
caustic lines for each band converge on different focusing points along the
device’s edge, marked as red dots in Figure 1d.

Additionally, as a result of an anisotropic Fermi surface, we observe
variations in the focusing length as a function of device orientation, denoted
by $\phi$, relative to the zigzag edge. This dependency is clearly
demonstrated in Figure 1e for both the $\alpha_{\mathrm{in}}$ and
$\alpha_{\mathrm{out}}$ bands. Specifically, Figure 1e reveals that the
variability in focusing length with respect to $\phi$ is more pronounced for
the $\alpha_{\mathrm{out}}$ band, whereas it remains relatively stable for the
$\alpha_{\mathrm{in}}$ band. Notably, device orientations aligned with the
zigzag edge ($\phi=0^{\circ}$) and the armchair edge ($\phi=90^{\circ}$) yield
the maximum difference in focusing length and, consequently, focusing field,
as illustrated in Figure 1e.

Figure 2:  (a) Optical image of the 4LG device with nonlocal TMF measurement
configurations, where each contact is labeled by numbers 1-7. The scale bar is
2 $\mu$m. (b) Schematic diagram of device cross-section. We encapsulate a 4LG
flake with top and bottom hBN flakes. Highly doped silicon and Cr/Au are used
as bottom and top gates. (c) Resistivity (red line) and conductivity (blue
line) as functions of $n$ at $2.4$ K from the left side of the device. (d)
Longitudinal resistance as a function of $B$ and $n$. (e) Calculated density
of states of 4LG. The labeled numbers indicate filling factors of gap states
(black lines). (f) Crystal structure of 4LG and its hopping parameters.

Therefore, to investigate the anisotropy of the Fermi surfaces in 4LG with
TMF, we fabricate an L-shaped device, illustrated in Figure 2a. To ensure the
maximum variation of the TMF signal, the left and right sides of the device
are etched in parallel to the cleaved lines of the exfoliated 4LG flake, which
is expected to exhibit either zigzag or armchair edges (see Supporting
Information for details) [27, 28].

To explore the ballistic behavior of electrons in 4LG, we construct
hBN/4LG/hBN heterostructure using the dry transfer technique [29], depicted in
Figure 2b. The heterostructure is then etched in $\mathrm{O_{2}/CHF_{3}}$
plasma, while electron beam lithography along with thermal evaporation of
Cr/Au are employed to define top gate and electrical contacts. Both top and
back gates are utilized to independently control carrier density $n$ and
electric displacement field $D$. Each contact is referred to by the number
labels shown in Figure 2a. All the measurements in this study are conducted at
2.4 K using the standard lock-in technique with an excitation frequency of 17
Hz. We maintain $D$ at zero to preserve the intrinsic band structure of 4LG.

We initially evaluate the device quality by examining the longitudinal
resistivity as a function of $n$, as illustrated in Figure 2c (see Supporting
Information for details). The $n$ value is determined from the period of
Shubnikov-de Haas oscillations at a finite magnetic field. The resistivity
profile reveals two distinct peaks around the charge neutrality point (CNP),
along with a small shoulder at positive $n$, indicative of 4LG [9]. These
features coincide precisely with Lifshitz transition points, which correspond
to the local maxima or minima in the band structure. At large $n$, the
conductivity demonstrates linear behavior, corresponding to field-effect
mobilities of approximately 180,000 and 110,000 $\mathrm{cm^{2}V^{-1}s^{-1}}$
for electrons and holes, respectively. These values are comparable to prior
studies on ballistic transport in graphene [30, 17, 31, 12].

To obtain an accurate description of the band structure, we utilize Landau
level (LL) crossings to determine the values of the SWMcC parameters [32]. In
Figure 2d, the longitudinal resistance between contacts 5 and 6 is presented
as a function of $B$ and $n$. The LL crossings manifest as bright color spots
in Figure 2d. By comparing the LL crossings in the simulated density of states
(Figure 2e) with experimental data, we achieve excellent agreement using the
SWMcC parameters reported in Ref. 9 ($\gamma_{0}=3$, $\gamma_{1}=0.39$,
$\gamma_{2}=-0.02$, $\gamma_{3}=0.3$, $\gamma_{4}=0.04$, $\gamma_{5}=0.04$,
$\Gamma_{1}=0.32$, $\Gamma_{3}=0.25$, $\Gamma_{4}=0.032$, and $\delta=0.041$
eV). Additional parameters $\Gamma_{1}$, $\Gamma_{3}$, and $\Gamma_{4}$ are
introduced to accommodate interlayer hopping between bulk layers (see Figure
2f), reflecting differences in interlayer interactions between the bulk and
surfaces of 4LG [33].

Figure 3: (a–c) Comparison of measure and simulated TMF spectra at $n=4\times
10^{12}$ $\mathrm{cm^{-2}}$: (a) The measured TMF spectra at $2.4$ K with
vertical lines indicating the focusing peaks. (b) The simulated TMF spectra
(solid line) alongside ratios of $N_{\mathrm{c}}/N_{\mathrm{i}}$ for
$\alpha_{\mathrm{in}}$ and $\alpha_{\mathrm{out}}$ bands (pink dashed lines
and orange dotted lines, respectively). The black dashed-dotted lines
represent the combined $-N_{\mathrm{ref}}/N_{\mathrm{i}}$ from both bands. (c)
The simulated trajectories in real space at magnetic fields corresponding to
the peak positions in (b). The red and black thick horizontal lines mark the
positions of injectors and collectors, respectively. The top and bottom rows
of these panels correspond to data from the right and left sides of the
device, respectively. (d–e) Comparison of measured and simulated TMF spectra
at $n=-4\times 10^{12}$ $\mathrm{cm^{-2}}$: (d) The measured TMF spectra at
$2.4$ K. (e) The simulated TMF spectra (solid line) and the ratios of
$N_{\mathrm{c}}/N_{\mathrm{i}}$ for $\beta_{\mathrm{in}}$ and
$\beta_{\mathrm{out}}$ bands (blue dashed lines and teal dotted dashed lines,
respectively). (f) The simulated trajectories in real space.

We then perform nonlocal measurement to obtain TMF spectra for both sides of
the device. Figure 2a illustrates the specific measurement configurations
employed. Electron injection and collection are carried out through contacts 5
(1) and 6 (3) for the left (right) side, respectively, with a distance of
$1.6$ $\mathrm{\mu m}$ between the injector and collector on both sides. The
resulting TMF spectra for electrons at $n=4\times 10^{12}$ $\mathrm{cm^{-2}}$,
presented in Figure 3a as a function of $B$, reveal two distinct peaks at
positive $B$, as indicated by vertical lines. These peaks, denoted as
$B_{\mathrm{f}}$, correspond to the magnetic fields required to focus
electrons onto the collector. Notably, lower $B_{\mathrm{f}}$ values are
associated with the $\alpha_{\mathrm{in}}$ band, while higher values are
linked to the $\alpha_{\mathrm{out}}$ band. This difference in peak values is
attributed to the larger Fermi surface of the $\alpha_{\mathrm{out}}$ band
compared to the $\alpha_{\mathrm{in}}$ band, necessitating a stronger magnetic
field for electron focusing at the same distance.

Significantly, distinctions emerge in the TMF spectra between the two sides.
The $B_{\mathrm{f}}$ values for $\alpha_{\mathrm{in}}$ electrons remain
consistent on both sides, aligning with their nearly circular caustic curve
(see Figure 1d). Conversely, for the $\alpha_{\mathrm{out}}$ band, variations
in $B_{\mathrm{f}}$ values become apparent between the two sides, despite the
equal distances between the injector and collector on both sides. The high
mobility inherent in our device suggests that these discrepancies in
$B_{\mathrm{f}}$ are not a consequence of disorder. Further, quantum Hall
measurements for both sides yield identical results, confirming uniform Bernal
stacking across the device (see Supporting Information for details). These
observations collectively point to the anisotropic Fermi surface as the
primary contributor to the observed disparity, a consequence of TW.

To confirm our assumption, we perform numerical calculations to determine TMF
spectra by tracking the count of carriers reaching the collector [19, 21]. The
trajectories of carriers are extracted utilizing the semiclassical model from
the calculated Fermi surfaces at the specified $n$. These Fermi surfaces are
obtained from the band structure of 4LG, incorporating SWMcC parameters
derived from quantum Hall data (see Supporting Information for details). The
nonlocal resistance, represented as $V_{\mathrm{R}}/I_{\mathrm{R}}$, is
approximated through the ratio
$(N_{\mathrm{c}}-N_{\mathrm{ref}})/N_{\mathrm{i}}$, where $N_{\mathrm{i}}$ is
the number of injected carriers, $N_{\mathrm{c}}$ is the number of collected
carriers, and $N_{\mathrm{ref}}$ is the number of carriers accumulated at the
other reference voltage probe. We weigh the number of injected carriers in
each pocket by their electronic density of states (see Supporting Information
for details). In our simulation, we neglect the specular reflection of
electrons at the device’s edge, given the absence of higher-order peaks in our
TMF data. The resulting simulation, depicted by the solid lines in Figure 3b,
reveals that the best-fit values of $\phi$ for the right-side (RS) and left-
side (LS) contacts are $0^{\circ}$ (zigzag edge) and $90^{\circ}$ (armchair
edge), respectively. These values align with the device’s design, expected to
correspond to the zigzag or armchair edges of 4LG. The pink dashed lines and
orange dotted lines in Figure 3b illustrate the ratio
$N_{\mathrm{c}}/N_{\mathrm{i}}$ for electrons from the $\alpha_{\mathrm{in}}$
and $\alpha_{\mathrm{out}}$ bands, respectively. Additionally, the black
dashed-dotted line represents the combined ratio
$-N_{\mathrm{ref}}/N_{\mathrm{i}}$ from both bands which provides a negative
background in our TMF spectra.

To elucidate the physical significance of each peak, we compute the
trajectories of the ensemble of injected electrons at each $B_{\mathrm{f}}$,
represented by pink (for $\alpha_{\mathrm{in}}$) and orange (for
$\alpha_{\mathrm{out}}$) triangles in Figure 3b. The outcomes are presented in
Figure 3c, where zigzag lines denote the orientation of the zigzag edge of
4LG. The simulation unveils that $B_{\mathrm{f}}$ for each band aligns with
the magnetic field strength that focuses the maximum number of electrons onto
the collector, as depicted by the caustic in Figure 3c. Moreover, it
highlights that the peak positions for $\alpha_{\mathrm{in}}$ electrons remain
consistent for both sides, while the $\alpha_{\mathrm{out}}$ peak for the LS
contacts shifts towards lower $B$ compared to the RS contacts. This behavior
is attributed to the more pronounced TW effect experienced by the
$\alpha_{\mathrm{out}}$ electrons in contrast to the $\alpha_{\mathrm{in}}$
electrons, evident from the Fermi surfaces.

While most peak positions in the simulation generally exhibit excellent
alignment with the experimental data, a noticeable discrepancy arises in the
actual TMF data for the left side of the device. Here, the peak corresponding
to the $\alpha_{\mathrm{in}}$ band manifests as a subtle shoulder rather than
a distinct peak, in contrast to the simulation results (Figure 3a and 3b). In
addition, the focusing field for the $\alpha_{\mathrm{out}}$ band appears
smaller than in simulation. This disparity is likely rooted in the inherent
simplicity of the model employed for calculating trajectories and background
signal. To enhance the precision of our analysis, more advanced methodologies,
such as the Landauer-Büttiker scattering theory [34, 35], could be employed to
offer a more accurate portrayal of the experimental observations. Moreover,
considering the charge accumulation at boundaries resulting from graphene’s
finite size, which modifies carrier trajectories and impacts the focusing
field, can enhance accuracy of the simulation [17, 36]. However, pursuing such
refinements extends beyond the scope of the present paper.

Next, we analyze the TMF spectra for holes at $n=-4\times 10^{12}$ cm-2
utilizing the identical measurement configurations employed for electrons. The
resulting spectra are illustrated in Figure 3d. Notably, the focusing peaks
for holes manifest at negative values of $B$, aligning with expectations as
holes possess opposite charges to electrons. Consequently, they necessitate
the opposite sign of $B$ to guide them toward the collectors. Similar to the
electron regime, the spectra for the RS and LS contacts exhibit distinct
characteristics. The RS spectrum features a prominent peak at $B=-0.25$ T,
accompanied by a small shoulder peak around $-0.18$ T, consistent with the
presence of two Fermi surfaces. In contrast, the LS spectrum displays a
singular, broad peak at $-0.21$ T.

To understand the experimental data, we simulate the TMF spectra of holes as
shown in Figure 3e. The simulation provides insights into the features
observed in the experimental TMF spectra. Specifically, the small shoulder at
$-0.18$ T in the RS spectrum is attributed to the focusing of
$\beta_{\mathrm{in}}$ holes onto the collector, as evidenced by the trajectory
simulation in Figure 3f. Meanwhile, the more pronounced peak is linked to the
focusing of $\beta_{\mathrm{out}}$ holes. In the case of the LS spectrum, the
simulation reveals that the focusing fields for $\beta_{\mathrm{in}}$ and
$\beta_{\mathrm{out}}$ are closely matched (Figure 3e, bottom), causing the
two peaks to fully merge into a single peak, consistent with our experimental
data. The primary contributing factor to the broadening of the focusing peaks
in our device stems from the finite width of the injector and collector,
approximately $500$ nm, leading to poorly resolved focusing peaks. This
observation underscores the importance of considering spatial constraints in
the design and interpretation of TMF spectra in nanoscale devices.

Figure 4: (a, b) Experimental TMF spectra at $2.4$ K for RS and LS contacts,
respectively. (c, d) Numerical simulation of the TMF data for RS and LS
contacts. (e) Fermi surfaces at densities $1.2\times 10^{12}$ (blue),
$4.5\times 10^{11}$ (yellow), and $1.0\times 10^{11}$ cm-2 (black). (f, g)
Simulated trajectories of holes at $n=-2\times 10^{12}$ $\mathrm{cm^{-2}}$ and
$B=-0.05$ T for RS and LS contacts. The red dash line in (f) shows a caustic.
The inset shows real-space trajectories of holes in each condition.

We further investigate the density dependence of TMF spectra. Figure 4a and 4b
show the RS and LS TMF spectra as a function of $B$ and $n$. The focusing
peaks are represented as bright curved ridges along diagonal in the plot. At
high carrier density ($|n|>1\times 10^{12}$ $\mathrm{cm^{-2}}$), the TMF
spectra exhibit behavior consistent with those presented in Figure 3. The RS
spectra manifest two distinct peaks, whereas the LS spectra display poor
resolution due to the smaller difference in $B_{\mathrm{f}}$ between the two
Fermi surfaces. As $n$ decreases, $B_{\mathrm{f}}$ for each pocket also
decreases. This behavior arises from the shrinking Fermi surfaces at lower
carrier densities, resulting in a smaller cyclotron radius of carriers. The
$n$-dependent TMF spectra for both sides at high density align well with the
simulation, as demonstrated in Figure 4c and 4d. This agreement further
underscores the robustness and reliability of our experimental observations
and simulations across varying carrier densities.

As the electron density reduces below approximately $1\times
10^{12}~{}\mathrm{cm^{-2}}$, the inner peak associated with the
$\alpha_{\mathrm{in}}$ band disappears (Figure 4a), in agreement with our
simulation, depicted in Figure 4c. The vanishing of this inner peak results
from the Fermi level dropping below the bottom edge of the
$\alpha_{\mathrm{in}}$ band. At these lower densities, only electrons from the
$\alpha_{\mathrm{out}}$ band contribute, resulting in a Fermi surface that
includes both electron and hole orbits, as shown in Figure 4e (highlighted in
yellow). Our simulation reveals the coexistence of electron and hole orbits
within a certain density range, leading to the appearance of peaks at both
positive and negative $B$ values. As $n$ approaches zero, the simulation shows
a pronounced shift in the focusing field of the outer peak, indicative of a
Lifshitz transition. This transition marks a topological transformation of the
Fermi surface from a singular to three distinct smaller electron pockets
(Figure 4e, black). However, in our experimental data, this Lifshitz
transition is not directly observable as it occurs at densities akin to
disorder-induced density fluctuations (the horizontal ridge in Figure 4a).

The simulation further unveils an additional focusing peak, as a small
shoulder in the RS data near $B=-0.05$ T and $n=-2\times 10^{12}$ cm-2 (Figure
4a). This supplementary peak, elucidated by the simulation, emerges due to the
specific injection direction of holes in a device oriented along the zigzag
direction. Figure 4f illustrates this phenomenon, where a new caustic
generates a focusing path onto the collector. Notably, this extra peak is
absent in the LS data corresponding to the armchair orientation. The
simulation of carrier trajectories in Figure 4g aligns with this observation,
as the caustic does not direct carriers toward the collector in this
particular configuration. These findings underscore the orientation-dependent
nuances in the TMF behavior of anisotropic Fermi surfaces.

In summary, we employed TMF to investigate the anisotropic Fermi surface in
4LG. Our findings reveal that TMF spectra distinctly vary with crystal
orientations, a critical observation that underscores the sensitivity of TMF
in detecting subtle electronic structure changes linked to TW effects. This
variation in TMF spectra not only deepens the understanding of anisotropic
properties in 4LG but also illustrates TMF’s potential in elucidating the
complex relationship between crystal orientation and anisotropic electronic
properties in multilayer graphene. These insights have broader implications
for materials with anisotropic Fermi surfaces, expanding the scope of TMF
applications in material science research.

###### Acknowledgements.

This research has been primarily supported by the NSRF via the Program
Management Unit for Human Resources & Institutional Development, Research and
Innovation (Grant no. B05F640152), National Research Council of Thailand
(NRCT) and Chulalongkorn University (Grant no. N42A650266), and the Thailand
Toray Science Foundation (TTSF). K.W. and T.Taniguchi acknowledge support from
the JSPS KAKENHI (Grant Numbers 20H00354 and 23H02052) and World Premier
International Research Center Initiative (WPI), MEXT, Japan.

## References

  * McCann and Koshino [2013] E. McCann and M. Koshino, The electronic properties of bilayer graphene, Rep. Prog. Phys. 76, 056503 (2013).
  * Craciun  _et al._ [2009] M. F. Craciun, S. Russo, M. Yamamoto, J. B. Oostinga, A. F. Morpurgo, and S. Tarucha, Trilayer graphene is a semimetal with a gate-tunable band overlap, Nat. Nanotechnol. 4, 383 (2009).
  * Datta  _et al._ [2017] B. Datta, S. Dey, A. Samanta, H. Agarwal, A. Borah, K. Watanabe, T. Taniguchi, R. Sensarma, and M. M. Deshmukh, Strong electronic interaction and multiple quantum hall ferromagnetic phases in trilayer graphene, Nat. Commun. 8, 14518 (2017).
  * Hirahara  _et al._ [2018] T. Hirahara, R. Ebisuoka, T. Oka, T. Nakasuga, S. Tajima, K. Watanabe, T. Taniguchi, and R. Yagi, Multilayer graphene shows intrinsic resistance peaks in the carrier density dependence, Sci. Rep. 8, 13992 (2018).
  * Charlier  _et al._ [1991] J.-C. Charlier, X. Gonze, and J.-P. Michenaud, First-principles study of the electronic properties of graphite, Phys. Rev. B 43, 4579 (1991).
  * Seiler  _et al._ [2022] A. M. Seiler, F. R. Geisenhof, F. Winterer, K. Watanabe, T. Taniguchi, T. Xu, F. Zhang, and R. T. Weitz, Quantum cascade of correlated phases in trigonally warped bilayer graphene, Nature 608, 298 (2022).
  * Zibrov  _et al._ [2018] A. A. Zibrov, P. Rao, C. Kometter, E. M. Spanton, J. Li, C. R. Dean, T. Taniguchi, K. Watanabe, M. Serbyn, and A. F. Young, Emergent dirac gullies and gully-symmetry-breaking quantum hall states in a b a trilayer graphene, Phys. Rev. Lett. 121, 167601 (2018).
  * Datta  _et al._ [2018] B. Datta, H. Agarwal, A. Samanta, A. Ratnakar, K. Watanabe, T. Taniguchi, R. Sensarma, and M. M. Deshmukh, Landau level diagram and the continuous rotational symmetry breaking in trilayer graphene, Phys. Rev. Lett. 121, 056801 (2018).
  * Shi  _et al._ [2018] Y. Shi, S. Che, K. Zhou, S. Ge, Z. Pi, T. Espiritu, T. Taniguchi, K. Watanabe, Y. Barlas, R. Lake, and C. N. Lau, Tunable lifshitz transitions and multiband transport in tetralayer graphene, Phys. Rev. Lett. 120, 096802 (2018).
  * Mohan  _et al._ [2021] P. Mohan, U. Ghorai, and R. Sensarma, Trigonal warping, satellite dirac points, and multiple field tuned topological transitions in twisted double bilayer graphene, Phys. Rev. B 103, 155149 (2021).
  * Joucken  _et al._ [2020] F. Joucken, Z. Ge, E. A. Quezada-López, J. L. Davenport, K. Watanabe, T. Taniguchi, and J. Velasco Jr, Determination of the trigonal warping orientation in bernal-stacked bilayer graphene via scanning tunneling microscopy, Phys. Rev. B 101, 161103 (2020).
  * Oka  _et al._ [2019] T. Oka, S. Tajima, R. Ebisuoka, T. Hirahara, K. Watanabe, T. Taniguchi, and R. Yagi, Ballistic transport experiment detects fermi surface anisotropy of graphene, Phys. Rev. B 99, 035440 (2019).
  * Ingla-Aynés  _et al._ [2023] J. Ingla-Aynés, A. L. R. Manesco, T. S. Ghiasi, S. Volosheniuk, K. Watanabe, T. Taniguchi, and H. S. J. van der Zant, Specular electron focusing between gate-defined quantum point contacts in bilayer graphene, Nano Lett. 23, 5453 (2023).
  * Lee  _et al._ [2022] Y. K. Lee, J. S. Smith, and J. H. Cole, Influence of device geometry and imperfections on the interpretation of transverse magnetic focusing experiments, Nanoscale Res. Lett. 17, 31 (2022).
  * Van Houten  _et al._ [1989] H. Van Houten, C. W. J. Beenakker, J. G. Williamson, M. E. I. Broekaart, P. H. M. Van Loosdrecht, B. J. Van Wees, J. E. Mooij, C. T. Foxon, and J. J. Harris, Coherent electron focusing with quantum point contacts in a two-dimensional electron gas, Phys. Rev. B 39, 8556 (1989).
  * Rokhinson  _et al._ [2004] L. P. Rokhinson, V. Larkina, Y. B. Lyanda-Geller, L. N. Pfeiffer, and K. W. West, Spin separation in cyclotron motion, Phys. Rev. Lett. 93, 146601 (2004).
  * Taychatanapat  _et al._ [2013] T. Taychatanapat, K. Watanabe, T. Taniguchi, and P. Jarillo-Herrero, Electrically tunable transverse magnetic focusing in graphene, Nat. Phys. 9, 225 (2013).
  * Chen  _et al._ [2016] S. Chen, Z. Han, M. M. Elahi, K. M. M. Habib, L. Wang, B. Wen, Y. Gao, T. Taniguchi, K. Watanabe, J. Hone, A. W. Ghosh, and C. R. Dean, Electron optics with p-n junctions in ballistic graphene, Science 353, 1522–1525 (2016).
  * Lee  _et al._ [2016] M. Lee, J. R. Wallbank, P. Gallagher, K. Watanabe, T. Taniguchi, V. I. Fal’ko, and D. Goldhaber-Gordon, Ballistic miniband conduction in a graphene superlattice, Science 353, 1526 (2016).
  * Lo  _et al._ [2017] S.-T. Lo, C.-H. Chen, J.-C. Fan, L. W. Smith, G. L. Creeth, C.-W. Chang, M. Pepper, J. P. Griffiths, I. Farrer, H. E. Beere, G. A. C. Jones, D. A. Ritchie, and T.-M. Chen, Controlled spatial separation of spins and coherent dynamics in spin-orbit-coupled nanostructures, Nat. Commun. 8, 15997 (2017).
  * Berdyugin  _et al._ [2020] A. I. Berdyugin, B. Tsim, P. Kumaravadivel, S. G. Xu, A. Ceferino, A. Knothe, R. K. Kumar, T. Taniguchi, K. Watanabe, A. K. Geim, I. V. Grigorieva, and V. I. Fal’ko, Minibands in twisted bilayer graphene probed by magnetic focusing, Sci. Adv. 6, eaay7838 (2020).
  * Gupta  _et al._ [2021] A. Gupta, J. J. Heremans, G. Kataria, M. Chandra, S. Fallahi, G. C. Gardner, and M. J. Manfra, Precision measurement of electron-electron scattering in gaas/algaas using transverse magnetic focusing, Nat. Commun. 12, 5048 (2021).
  * Rao  _et al._ [2023] Q. Rao, W.-H. Kang, H. Xue, Z. Ye, X. Feng, K. Watanabe, T. Taniguchi, N. Wang, M.-H. Liu, and D.-K. Ki, Ballistic transport spectroscopy of spin-orbit-coupled bands in monolayer graphene on wse2, Nat. Commun. 14, 6124 (2023).
  * Koshino and McCann [2011] M. Koshino and E. McCann, Landau level spectra and the quantum hall effect of multilayer graphene, Phys. Rev. B 83, 165443 (2011).
  * Klanurak  _et al._ [2022] I. Klanurak, K. Watanabe, T. Taniguchi, S. Chatraphorn, and T. Taychatanapat, Magnetoconductance oscillations in electron-hole hybridization gaps and valley splittings in tetralayer graphene, Phys. Rev. B 106, L161405 (2022).
  * Ashcroft and Mermin [2022] N. W. Ashcroft and N. D. Mermin,  _Solid state physics_ (Cengage Learning, 2022).
  * Neubeck  _et al._ [2010] S. Neubeck, Y. M. You, Z. H. Ni, P. Blake, Z. X. Shen, A. K. Geim, and K. S. Novoselov, Direct determination of the crystallographic orientation of graphene edges by atomic resolution imaging, Appl. Phys. Lett. 97, 053110 (2010).
  * Almeida  _et al._ [2011] C. M. Almeida, V. Carozo, R. Prioli, and C. A. Achete, Identification of graphene crystallographic orientation by atomic force microscopy, J. Appl. Phys. 110, 086101 (2011).
  * Wang  _et al._ [2013] L. Wang, I. Meric, P. Y. Huang, Q. Gao, Y. Gao, H. Tran, T. Taniguchi, K. Watanabe, L. M. Campos, D. A. Muller, J. Guo, P. Kim, J. Hone, K. L. Shepard, and C. R. Dean, One-dimensional electrical contact to a two-dimensional material, Science 342, 614 (2013).
  * Mayorov  _et al._ [2011] A. S. Mayorov, R. V. Gorbachev, S. V. Morozov, L. Britnell, R. Jalil, L. A. Ponomarenko, P. Blake, K. S. Novoselov, K. Watanabe, T. Taniguchi, and A. K. Geim, Micrometer-scale ballistic transport in encapsulated graphene at room temperature, Nano Lett. 11, 2396 (2011).
  * Sandner  _et al._ [2015] A. Sandner, T. Preis, C. Schell, P. Giudici, K. Watanabe, T. Taniguchi, D. Weiss, and J. Eroms, Ballistic transport in graphene antidot lattices, Nano Lett. 15, 8402 (2015).
  * Taychatanapat  _et al._ [2011] T. Taychatanapat, K. Watanabe, T. Taniguchi, and P. Jarillo-Herrero, Quantum hall effect and landau-level crossing of dirac fermions in trilayer graphene, Nature Physics 7, 621–625 (2011).
  * Wu  _et al._ [2015] Z. Wu, Y. Han, J. Lin, W. Zhu, M. He, S. Xu, X. Chen, H. Lu, W. Ye, T. Han, Y. Wu, G. Long, J. Shen, R. Huang, L. Wang, Y. He, Y. Cai, R. Lortz, D. Su, and N. Wang, Detection of interlayer interaction in few-layer graphene, Phys. Rev. B 92, 075408 (2015).
  * Beconcini  _et al._ [2016] M. Beconcini, S. Valentini, R. K. Kumar, G. H. Auton, A. K. Geim, L. A. Ponomarenko, M. Polini, and F. Taddei, Scaling approach to tight-binding transport in realistic graphene devices: The case of transverse magnetic focusing, Phys. Rev. B 94, 115441 (2016).
  * Petrović  _et al._ [2017] M. D. Petrović, S. P. Milovanović, and F. M. Peeters, Scanning gate microscopy of magnetic focusing in graphene devices: quantum versus classical simulation, Nanotechnology 28, 185202 (2017).
  * Silvestrov and Efetov [2008] P. G. Silvestrov and K. B. Efetov, Charge accumulation at the boundaries of a graphene strip induced by a gate voltage: Electrostatic approach, Physical Review B 77, 155436 (2008).
  * Huang  _et al._ [2015] Y. Huang, E. Sutter, N. N. Shi, J. Zheng, T. Yang, D. Englund, H.-J. Gao, and P. Sutter, Reliable exfoliation of large-area high-quality flakes of graphene and other two-dimensional materials, ACS nano 9, 10612 (2015).
  * Wang  _et al._ [2012] Y. Y. Wang, R. X. Gao, Z. H. Ni, H. He, S. P. Guo, H. P. Yang, C. X. Cong, and T. Yu, Thickness identification of two-dimensional materials by optical imaging, Nanotechnology 23, 495713 (2012).
  * Purdie  _et al._ [2018] D. G. Purdie, N. M. Pugno, T. Taniguchi, K. Watanabe, A. C. Ferrari, and A. Lombardo, Cleaning interfaces in layered materials heterostructures, Nat. Commun. 9, 5387 (2018).

Supporting Information

Device fabrication

_2D materials preparation._ Tetralayer graphene (4LG) and hexagonal boron
nitride (hBN) flakes were exfoliated from their bulk counterparts using the
scotch tape technique [37] and subsequently deposited onto Si/SiO2 chips. The
optical contrast between the flakes and SiO2 substrates was used to identify
the thickness of the 2D materials [38]. Flakes of 4LG with perpendicular
cleaved edges and hBN flakes with a thickness of approximately 10-30 nm were
carefully selected from the chips for device fabrication. The hBN and 4LG
flakes used for the device in the main text are depicted in Figure S1(a-c).

_Heterostructure assembly._ To create the hBN/4LG/hBN heterostructure (BNGBN),
a stack comprising poly(bisphenol A carbonate) film on a block of
polydimethylsiloxane (PDMS) was utilized as a stamp. This stamp was employed
in picking up the selected flakes and precisely stacking them together.
Subsequently, the BNGBN heterostructure was released onto a new Si/SiO2 chip
at 180 ∘C to eliminate blisters that may have been trapped between the
interfaces of the heterostructure [39]. Figure S1(d) illustrates the optical
image of the heterostructure following the assembly process.

_Electron-beam lithography._ The blister-free region within the BNGBN (marked
by the dashed line in Figure S1(d)) was used as the basis for designing the
transport channel for the device discussed in the main text. A polymethyl
methacrylate (PMMA) film was used as the electron beam resist. Subsequently,
EBL and O2/CHF3 plasma etching techniques were employed to define an L-shaped
device, with two edges of the device aligned parallel to the cleaved edge of
the 4LG. In Figure S1(e), the device structure is shown, indicating the
deposition of the top gate and contacts using thermal evaporation of Cr/Au at
a thickness of 5/50 nm.

_Control of charge carrier density._ The L-shaped device depicted in Figure
S1(e) comprises a p-doped silicon back gate (BG) and a Cr/Au top gate (TG). By
applying voltages $V_{\mathrm{BG}}$ and $V_{\mathrm{TG}}$ to the BG and TG, we
can introduce a carrier density
$n=\frac{1}{e}(C_{\mathrm{BG}}V_{\mathrm{BG}}+C_{\mathrm{TG}}V_{\mathrm{TG}})$
and an electric displacement field
$D=\frac{1}{2\epsilon_{0}}(C_{\mathrm{BG}}V_{\mathrm{BG}}-C_{\mathrm{TG}}V_{\mathrm{TG}})$
into the 4LG channel. Here, $e$ is the elementary charge, $\epsilon_{0}$ is
the vacuum permittivity, and $C_{\mathrm{BG}}$ and $C_{\mathrm{TG}}$ are the
capacitance per unit area of the BG and TG, respectively (determined from
Shubnikov-de Haas oscillations at finite magnetic fields). Based on these
relationships, we can introduce charge carriers into the 4LG without an
external displacement field by simultaneously adjusting both $V_{\mathrm{BG}}$
and $V_{\mathrm{TG}}$ in a manner that satisfies the relation
$C_{\mathrm{TG}}V_{\mathrm{TG}}=C_{\mathrm{BG}}V_{\mathrm{BG}}$.

Figure S1: (a-c) Individual top hBN (a), 4LG (b), and bottom hBN (c) flakes
selected in constructing the device discussed in the main text. (d) The
hBN/4LG/hBN heterostructure formed from these flakes (a-c). The black dashed
line indicates the edge of 4LG. (e) The device following the lithography
processes. Scale bars are 20 $\mu$m.

Local transport measurement

Figure S2: (a) Four-probe longitudinal resistance as a function of filling
factor at $B$ = $2$ T and $2.4$ K. The black and red lines are measured using
contacts from the left and right sides of the device, respectively. (b)
Resistivity and (c) conductivity as functions of $n$ at $B$ = 0 T. The black
and red lines represent the data from left and right contacts, respectively

The stacking configuration of our 4LG device was determined by analyzing the
Landau-level (LL) crossings in the quantum Hall data. As illustrated in Figure
2d of the main text, the longitudinal resistance for the left-side contacts
reveals the positions of these LL crossings. Notably, we observed that these
LL crossing positions align precisely with those reported in Ref. 9,
confirming the ABAB stacking of our device. To ensure uniform stacking across
the entire device, we compared the resistance oscillations due to the
Shubnikov-de Hass effect between both sides. Figure S2 presents the resulting
data depicting the resistance oscillations for the left-side (black line) and
right-side (red line) contacts under a perpendicular magnetic field $B$ = 2 T
as a function of filling factor $\nu$, with a fixed current bias of 300 nA
applied to both sides. Notably, dips in the resistance correspond to integer
values of $\nu$, with the majority of these resistance dips exhibiting a
separation of $\Delta\nu$ = 4, attributed to spin and valley degeneracy.
However, certain resistance dips, such as those at $\nu$ = 12 and 20, are
separated by $\Delta\nu$ = 8 due to Landau level crossings. Given that the LL
crossing positions are consistent for both sides, we conclude that our 4LG
device maintains a uniform Bernal stacking configuration. Therefore, the
observed variation in the TMF spectra in the main text is not attributable to
differing stacking orders of 4LG between the left- and right-side contacts.

To evaluate the quality and density uniformity of the 4LG across both the left
and right sides of the device, we conducted resistivity measurements as
functions of carrier density ($n$) at $B=0$ T. The resistivity profiles for
both sides, depicted in Figure S2(b), exhibit characteristic double peaks
typical of ABAB-stacked 4LG [9]. Conductivity, illustrated in Figure S2(c),
was derived from the reciprocal of resistivity for the left and right
contacts. Notably, on the left side of the device, field-effect mobilities,
determined from the linear regime of the conductivity plots, were identified
to be 180,000 and 110,000 $\mathrm{cm^{2}V^{-1}s^{-1}}$ for electrons and
holes, respectively. Conversely, the right side exhibited slightly lower
mobilities of 110,000 and 100,000 $\mathrm{cm^{2}V^{-1}s^{-1}}$ for electrons
and holes, respectively. These notably high mobility values on both sides
suggest minimal potential fluctuation.

Band structure calculation

The low-energy band structure of tetralayer graphene (4LG) near the $K$ and
$K^{\prime}$ points in the graphene Brillouin zone was computed by
diagonalizing the effective Hamiltonian, which was parameterized using the
Slonczewski-Weiss-McClure parameters of graphite. The effective Hamiltonian
follows the details in Ref. 24. Our chosen basis consisted of $|A_{1}\rangle$,
$|B_{1}\rangle$, $|A_{2}\rangle$, $|B_{2}\rangle$, $|A_{3}\rangle$,
$|B_{3}\rangle$, $|A_{4}\rangle$, and $|B_{4}\rangle$, where $|A_{i}\rangle$
and $|B_{i}\rangle$ represent Bloch states for electrons in layer $i$ on
sublattices $A$ and $B$, respectively. In this basis, the Hamiltonian for the
4LG can be written as:

$H_{\mathrm{4LG}}=\begin{pmatrix}0&v\pi^{\dagger}&-v_{4}\pi^{\dagger}&v_{3}\pi&\gamma_{2}/2&0&0&0\\\
v\pi&\Delta^{\prime}&\gamma_{1}&-v_{4}\pi^{\dagger}&0&\gamma_{5}/2&0&0\\\
-v_{4}\pi&\gamma_{1}&\Delta^{\prime}&v\pi^{\dagger}&-V_{4}\pi&\Gamma_{1}&\gamma_{5}/2&0\\\
v_{3}\pi^{\dagger}&-v_{4}\pi&v\pi&0&V_{3}\pi^{\dagger}&-V_{4}\pi&0&\gamma_{2}/2\\\
\gamma_{2}/2&0&-V_{4}\pi^{\dagger}&V_{3}\pi&0&v\pi^{\dagger}&-v_{4}\pi^{\dagger}&v_{3}\pi\\\
0&\gamma_{5}/2&\Gamma_{1}&-V_{4}\pi^{\dagger}&v\pi&\Delta^{\prime}&\gamma_{1}&-v_{4}\pi^{\dagger}\\\
0&0&\gamma_{5}/2&0&-v_{4}\pi&\gamma_{1}&\Delta^{\prime}&v\pi^{\dagger}\\\
0&0&0&\gamma_{2}/2&v_{3}\pi^{\dagger}&-v_{4}\pi&v\pi&0\end{pmatrix},$

where $\Delta^{\prime}$ is the energy difference between dimer and non-dimer
sites, $\gamma_{i}$ are the hopping parameters as indicated by the diagram in
Figure 2f of the main text, and the velocities
$v_{i}=\sqrt{3}a\gamma_{i}/2\hbar$ (the lattice constant $a$ = 0.246 nm).
Using the convention that the x-axis is parallel to 4LG’s zigzag edge, we can
express $\pi=\xi p_{x}+ip_{y}$, where $\mathbf{p}=(p_{x},p_{y})$ is the
momentum operator and $\xi=\pm 1$ is the valley index.

In the Hamiltonian above, we have introduced an additional set of hopping
parameters $\Gamma_{1}$, $\Gamma_{3}$, and $\Gamma_{4}$, along with their
corresponding velocities $V_{i}=\sqrt{3}a\Gamma_{i}/(2\hbar)$, to account for
difference in interlayer interactions between the bulk and surfaces of 4LG
[33]. Diagonalizing this Hamiltonian at each momentum
$\mathbf{p}=\hbar\mathbf{k}$ using the same parameters as in Ref.9 yields the
low energy band structure as depicted in Figure 1a of the main text.

TMF spectra simulation

Figure S3: (a) Fermi surface of 4LG from band $\alpha_{\mathrm{out}}$ at
$E_{\mathrm{F}}=40$ meV. (b) Trajectory of $\alpha_{\mathrm{out}}$ electrons
in real space with $E_{\mathrm{F}}=40$ meV under an out-of-plane magnetic
field $B$ = 0.12 T. (c) Simulated trajectories of $\alpha_{\mathrm{out}}$
electrons, with the same conditions as in (b), injected at the red contact
from various initial angles. Orange and black paths represent trajectories
reaching the collector and reference contacts, respectively. The zigzag edges
of the 4LG in (b) and (c) are aligned parallel to the x-axis.

In our simulation of the TMF spectra, presented in Figure 3 of the main text,
we initiated the process by deriving the Fermi surface from the band
structure, corresponding to a specific Fermi energy, denoted as
$E_{\mathrm{F}}$. This is depicted in Figure S3(a), where we illustrate the
Fermi surface associated with electrons in the $\alpha_{\mathrm{out}}$ band at
an $E_{\mathrm{F}}$ value of 40 meV. When subjected to a magnetic field $B$,
we determine the semiclassical trajectory by implementing a 90∘ rotation of
the Fermi surface. This rotation is clockwise for electrons and counter-
clockwise for holes. Additionally, the Fermi surface is scaled by a factor of
$\hbar/eB$, as demonstrated in Figure S3(b). The direction of motion within
the orbit is influenced by the effective charge of the carriers and the
direction of magnetic field.

In our study, the non-local resistance for each $B$ is estimated to be
directly proportional to the ratio
$(N_{\mathrm{c}}-N_{\mathrm{ref}})/N_{\mathrm{i}}$. The simulation begins with
the modeling of electron trajectories, injected at various initial angles
through the injector, as depicted in Figure S3(c). These trajectories are
monitored to quantify the fraction that successfully arrives at the collector.
To accurately represent the density of states at each angle, each trajectory
is assigned a weight inversely proportional to
$|\nabla_{\mathbf{k}}\epsilon(\mathbf{k})|$. The total number of injected
electrons, denoted as $N_{\mathrm{i}}$, is calculated by summing the weights
of all trajectories corresponding to injected electrons. Conversely, the
number of electrons that reach the collector, $N_{\mathrm{c}}$, is determined
by summing the weights of those trajectories that successfully terminate at
the collector, as shown by the orange paths in Figure S3(c).

To estimate the number of electrons reaching the reference contact, denoted as
$N_{\mathrm{ref}}$, we adopt the premise that only those electron trajectories
which extend to the device’s edge beyond the collector contact are relevant
contributors to $N_{\mathrm{ref}}$. These trajectories are represented by the
black paths in Figure S3(c). Given the absence of second-order focusing peaks
in our TMF spectra, it is inferred that diffusive scattering occurs at the
device’s edge. Consequently, trajectories that scatter off the device’s edge
in the region between the injector and collector (illustrated as blue paths in
Figure S3(c)) are presumed to reach both the collector and the reference
contact equally. Therefore, these do not influence the TMF spectra. The value
of $N_{\mathrm{ref}}$ is derived by aggregating the weights of the black
trajectories depicted in Figure S3(c).


# Temporally Aligning Long Audio Interviews with Questions:
A Case Study in Multimodal Data Integration

Piyush Singh Pasi1 Karthikeya Battepati1 Preethi Jyothi1 Ganesh Ramakrishnan1
Tanmay Mahapatra2 Manoj Singh2 1Indian Institute of Technology, Bombay
2CARE India Solutions for Sustainable Development
{piyushsinghpasi, karthikeyabatte, pjyothi<EMAIL_ADDRESS>{tanmay,
<EMAIL_ADDRESS>
###### Abstract

The problem of audio-to-text alignment has seen significant amount of research
using complete supervision during training. However, this is typically not in
the context of long audio recordings wherein the text being queried does not
appear verbatim within the audio file. This work is a collaboration with a
non-governmental organization called CARE India that collects long audio
health surveys from young mothers residing in rural parts of Bihar, India.
Given a question drawn from a questionnaire that is used to guide these
surveys, we aim to locate where the question is asked within a long audio
recording. This is of great value to African and Asian organizations that
would otherwise have to painstakingly go through long and noisy audio
recordings to locate questions (and answers) of interest. Our proposed
framework, Indent, uses a cross-attention-based model and prior information on
the temporal ordering of sentences to learn speech embeddings that capture the
semantics of the underlying spoken text. These learnt embeddings are used to
retrieve the corresponding audio segment based on text queries at inference
time. We empirically demonstrate the significant effectiveness (improvement in
R-avg of about 3%) of our model over those obtained using text-based
heuristics. We also show how noisy ASR, generated using state-of-the-art ASR
models for Indian languages, yields better results when used in place of
speech. Indent, trained only on Hindi data is able to cater to all languages
supported by the (semantically) shared text space. We illustrate this
empirically on 11 Indic languages.

##  1 Introduction

Audio surveys and oral interviews are routinely used as a means for data
collection in many parts of the world Jones (2003), Reichmann et al. (2010).
Apart from ease of use, audio surveys are also very inclusive since they
naturally allow for data to be collected from illiterate or physically
challenged individuals Heinritz et al. (2022). Several audio surveys by
governmental and non-governmental organizations are conducted with the aim of
improving social outcomes such as health and education. These surveys are
typically accompanied by predefined questionnaires that guide the interviewer.
Temporally aligning questions to where each question was asked in an interview
becomes very important for organizations collecting the data, so as to quickly
retrieve answers to relevant questions from long audio files.

Figure 1: Illustration of speech in audio and relevant questions

In this work, we use long audio health surveys collected from mothers with
young children residing in rural areas of Bihar, India. This data was
collected by a non-governmental organization CARE India CARE (2023) as part of
their larger effort towards improving maternal and neonatal health. The
interviews are typically long, spanning more than 40 minutes on average, and
are based on a fixed questionnaire designed by CARE. Given a written question
drawn from the questionnaire, our main goal is to extract a short audio
snippet from the long recording that matches the given question. This task is
of utmost importance to organizations such as CARE who would otherwise have to
resort to tedious manual searches through these long audio recordings to
retrieve answers to various questions.

Most prior work on cross-modal segment retrieval relies on strong supervision
in the form of temporally-aligned text annotations Anne Hendricks et al.
(2017), Gao et al. (2017), Liu et al. (2018), Xu et al. (2018), Gupta et al.
(2021), Javed et al. (2022). For long audio files, creating such time-aligned
annotations using human annotators would require significant amount of time
and money. Since we are working with audio surveys derived from a fixed
sequence of questions in a questionnaire, we use a weaker form of supervision
involving larger audio segments paired with sets of temporally ordered
questions. We present a new architecture Indent 111Indent can be expanded as
“alIgning loNg auDio intErviews aNd quesTions”., that is trained using such
weak supervision of larger audio segments and sets of questions. Indent uses
an attention-based mechanism and is trained with contrastive losses across
audio-text modalities to ground each question in the larger segment with its
corresponding audio.

Our task of audio segment retrieval using textual queries is particularly
challenging for the following reasons:  1. The audio surveys with the young
mothers are conducted in noisy home environments, thus resulting in very noisy
audio recordings.  2. Questions from the questionnaire are not asked verbatim
and are typically paraphrased by the interviewer when posed to the mothers.
Figure 1 illustrates this phenomenon.  3. While the questions appear in formal
text in the official language of Bihar, i.e., Hindi, the audio recordings
could contain variations that are typical of the local dialects spoken by the
interviewees.

To summarize, our main contributions are: (i) We tackle a high-utility problem
of locating short audio segments from within long audio surveys based on
textual queries. These audio surveys are collected by CARE India in real
environments and present many interesting challenges that we outline above.
The dataset is enriched with fine-grained annotations on the evaluation splits
and coarse-grained annotations on the training data that is sufficient to
enable weak supervision. (ii) We design Indent, a new model for text-to-audio
retrieval from within long audio surveys given a question in text. Indent is
trained using weak supervision consisting of longer audio segments and sets of
temporally ordered questions.

The problem of question-to-audio retrieval can also serve as an intermediate
step for various downstream tasks such as audio-driven form filling and audio-
driven question answering, by extracting answers from the survey recordings
once the relevant question is isolated. We leave such extensions to future
work and focus on the challenging problem of retrieving audio segments from
long audios based on questions in text.

##  2 Related Work

Most relevant to our work are (i) approaches that focus on temporal sentence
grounding in videos, where the task is to retrieve a video that semantically
corresponds to a query in natural language, and (ii) audio text alignment for
Automatic Speech Recognition (ASR).

###  2.1 Video Moment Retrieval Using Text Queries

Prior work has explored various approaches for the task of temporally
grounding captions or descriptions in videos in both fully and weakly
supervised settings Zhang et al. (2022). Some of the prominent approaches are
discussed below. Moment Context Networks (MCN) Anne Hendricks et al. (2017)
learn a shared embedding for video temporal context features (i.e., video
features integrated with temporal endpoint features that indicate when a
moment occurs in a video) and LSTM-based language features. The CTRL
architecture Gao et al. (2017) constitutes a temporal regression network to
produce alignment scores and location offsets using combined representations
for visual and text domains. A language-temporal attention network has also
been proposed Liu et al. (2018) to learn word attention based on the temporal
context information in the video. Temporal boundary annotations for sentences
have been used to train a segment proposal network using attention Xu et al.
(2018). All these approaches assume the fully supervised setting. A Text-
Guided Attention (TGA) model was proposed for video moment retrieval using
text queries in a weakly supervised setting Mithun et al. (2019). However, TGA
does not utilize the temporal order of sentences unlike our approach, Indent
that employs a Gaussian-weighted attention to leverage this temporal
information.

###  2.2 Audio-Text Alignment For Speech Recognition

Most of the widely used ASR models that generate the best transcripts for
various languages require large amounts of labeled transcripts for training
Chadha et al. (2022); Javed et al. (2022). Methods proposed for low resource
settings Anguera et al. (2014) also expect access to some annotated
transcripts during training, which are not available in our setting. Models
that use CTC or sequence-to-sequence loss functions Synnaeve et al. (2019);
López and Luque (2022) require audio files with their corresponding
transcripts to be used during training. Such models cannot be employed with
our CARE India dataset, since we do not have any transcribed speech and the
questions asked in the audio recordings are only semantically similar to (and
often, paraphrases of) the questions listed in the questionnaire.

Recently, a few methods have been proposed for speech recognition using
unaligned speech and text Ao et al. (2021); Chen et al. (2022) and generating
speech embeddings that capture the semantics of the underlying spoken text by
aligning both text and speech embedding spaces using unsupervised techniques
Chung et al. (2018); Chen et al. (2018). These models require word-level audio
segments for semantically rich speech embeddings, which are generally not
easily available or extractable. Other prior work require very large amounts
of unpaired speech and text and do not currently serve many Indian languages
Ao et al. (2021); Chen et al. (2022).

##  3 CARE India Dataset

We create a new “CARE India Dataset” that contains speech interviews conducted
by the CARE India Organization222https://bihar.care.org/. These interviews
focus on analyzing maternal health and empowering women in rural areas in the
state of Bihar, India. Each interview is approximately 40 minutes long and is
recorded in the household premises of the interviewee, thus yielding very
noisy recordings. The interviewer refers to a questionnaire as a guide for the
duration of the interview. Searching for a specific question from the
questionnaire within the noisy long audio recording is not the only challenge.
Questions are not asked verbatim during the course of the conversation and
could be paraphrased. The questions are also drawn from a diverse pool
including multiple-choice type, numerical and subjective questions. All these
factors add to the overall difficulty of isolating the correct audio segment
that maps to a given question. CARE India (and other similar organizations)
could greatly benefit from an automated solution that given a question in text
can identify roughly where it appeared in a long audio recording. Without such
a tool, volunteers would have to tediously go through long and noisy audio
recordings to search for where a question appeared in it. This work is our
first step towards building such a tool. We present a new model Indent that
can be trained on longer audio segments spanning multiple questions and
identify potential audio segments within a test audio recording that could be
matched with a test query.

Each interview in the train set is annotated at the segment-level. A segment
contains $m$ consecutive questions and is of variable duration. Segments do
not overlap or share questions. If a question begins within a segment, it
would also end in that segment. We choose to set $m=5$, since this granularity
gives us segments that are not too long (which would make learning difficult)
and not too short (which would otherwise overwhelm the human annotators). For
each audio segment containing five questions, we annotate the start of the
first question and the end of the fifth question. Our development and test
sets have fine-grained annotations with start and end timestamps for each
question. However, such a per-question annotation takes much longer. We find
that annotating train files takes around 4 hours per audio recording, while it
takes more than 6 hours to annotate test files from a similarly-sized audio
recording. Detailed dataset statistics for our train, development and test
sets are presented in Table 1.

Train set |
---|---
Total no. of interviews | 34
Total no. of segments | 1223
Avg. no. of segments in an interview | 35.97
Avg. segment duration (sec) | 54.64
Avg. no. of chunks per segment | 14.31
Avg. chunk duration (sec) | 1.7
Avg. interview duration† (min) | 42.1
Total Train set duration† (hour) | 23.85
Total no. of questions asked | 6115
Avg. no. of questions asked in an interview | 179.85
Dev set |
Total no. of interviews | 3
No. of chunks in an interview | 634.33
Avg. interview duration† (min) | 34.2
Total Dev set duration† (hour) | 1.71
Avg. chunk duration (sec) | 2.04
Total no. of questions asked | 538
Avg. no. of questions asked in an interview | 179.33
Avg. question duration (sec) | 2.87
Test set |
Total no. of interviews | 5
No. of chunks in an interview | 603.4
Avg. interview duration† (min) | 36.28
Total Test set duration† (hour) | 3.02
Avg. chunk duration (sec) | 1.96
Total no. of questions asked | 629
Avg. no. of questions in an interview | 125.8
Avg. question duration (sec) | 1.78
No. of question in fixed questionnaire $Q$ | 1555

Table 1: CARE India Dataset Statistics, † denotes duration without any
preprocessing

##  4 Indent: Architecture and Methodology

We formally define our problem as follows. Given a question $q$ drawn from a
fixed questionnaire $Q$ and an audio interview $f$ within which $q$ is asked,
we want to retrieve the audio segment $s\in f$ that exactly matches $q$. On an
average, $f$ is more than 40 minutes long. In Table 1, we provide more
detailed statistics on the data distribution. As described in Section 3,
during training, we assume access to longer audio segments containing $m=5$
questions. These segmentations are manually annotated. At test time, we
extract non-overlapping audio segments of fixed duration from the test audio
interview $f$ and identify the segment that is the best match for a given
question using our trained Indent.

#### Overview of Workflow.

Indent consists of three main components:  1. Speech Encoder  2. Question
Encoder  3. Gaussian-weighted Cross-attention Network.  The speech encoder
consists of audio preprocessing and feature extractor modules. We use an audio
preprocessing module to remove noise from the speech interviews and break
segments into smaller chunks of roughly 2 seconds duration each. We extract
speech features at the chunk-level. The question encoder extracts question
features at the sentence-level. Hence, the time resolution of the audio chunk
features and the question features are very different (c.f. Figure 1). Using
both audio chunk and question feature sequences as inputs, we try to align
chunks with questions and simultaneously bridge the modality gap between
speech and text in a shared space using a Gaussian Weighted Cross Attention
module and contrastive learning. At inference, for a speech interview, we rank
segments using the dot-product scores of the chunks and the given question. In
Figure 2, we present a schematic diagram of the overall architecture of
Indent. Next, we elaborate on each of the model components.

###  4.1 Speech Encoder

The audio recordings contain speech both from the interviewer and the
interviewee; the latter is typically very faint and more difficult to isolate.
As mentioned in Section 3, the audio recordings are annotated with start and
end times of large segments containing $m=5$ questions each. Within each of
these segments, we are only interested in parts corresponding to the
interviewer. To isolate these parts, we attempted several techniques including
voice activity detection (VAD) Wiseman (2017), speaker diarization Bredin et
al. (2020) and source separation techniques Ravanelli et al. (2021). After
qualitatively analyzing the respective outputs, we found VAD to yield the best
quality outputs. Using VAD, we split each annotated segment $s$ into $n_{s}$
chunks $\\{c^{s}_{i}\\}_{i=1}^{n_{s}}$. This results in a varying number of
chunks across segments. We note here that the number of chunks $n_{s}$ can be
higher than the number of questions $m$ in a segment $s$. The microphone often
fails to adequately capture the interviewee’s voice, and we therefore assume
(after qualitative validation) that the chunks after VAD would primarily
contain the interviewer’s speech.

Once we have the chunk sequence $\\{{c^{s}_{i}}\\}_{i=1}^{n_{s}}$, we use a
frozen pretrained speech encoder (wav2vec2.0 Baevski et al. (2020)) to extract
features for chunks $\\{\boldsymbol{c}^{s}_{i}\\}_{i=1}^{n_{s}}$ where
${\boldsymbol{c}^{s}_{i}}\in{R}^{T_{i}\times d^{\prime}}$, $T_{i}$ is the
number of time frames for chunk $c_{i}^{s}$. The speech encoder produces a
$d^{\prime}$-dimensional feature vector for each speech frame in $c_{i}^{s}$.
To reduce ${\boldsymbol{c}^{s}_{i}}$ to one aggregate feature vector per
chunk, we apply a 1D convolution and take the mean across all $T_{i}$
features. Next, we project these chunk features to a $d$-dimensional shared
space using a linear layer. We then apply self-attention over the segment so
that each chunk can gather context from other chunks. We also add skip
connections wherever appropriate. The left block in Figure 2 represents the
speech encoder with layers and skip connections. More formally:

$\displaystyle\\{{{\boldsymbol{\hat{c}}}}_{i}^{s}\\}_{i=1}^{n_{s}}$
$\displaystyle=\mathrm{MeanPool}(\mathrm{Conv1D}\left(\\{\boldsymbol{c}^{s}_{i}\\}_{i=1}^{n_{s}}\right))$
$\displaystyle C^{s}$
$\displaystyle={\\{{{C_{i}^{s}}}}\\}_{i=1}^{n_{s}}=\mathrm{Linear}(\\{{{\boldsymbol{\hat{c}}}}_{i}^{s}\\}_{i=1}^{n_{s}})$
$\displaystyle\boldsymbol{C}^{s}$
$\displaystyle=\\{{\boldsymbol{C}_{i}^{s}}\\}_{i=1}^{n_{s}}=\mathrm{SelfAttn}({C^{s}},{C^{s}},{C^{s}})$

where ${C_{i}^{s}}\in\mathbb{R}^{d}$ is the chunk feature after a
$\mathrm{Linear}$ layer,
$\mathrm{SelfAttn}(k,q,v)=\mathrm{softmax}(\frac{k.q^{\intercal}}{\sqrt{d}})v$,
$q,k,v\in\mathbb{R}^{n_{s}\times d}$ and $\boldsymbol{C}_{i}^{s}$ is the chunk
feature after self-attention.

###  4.2 Question Encoder

We use a frozen pretrained sentence transformer network to extract a
$d$-dimensional feature vector. During training, for each segment $s$ we have
$m$ questions, and hence we compute
${\boldsymbol{q}^{s}=[\boldsymbol{q}^{s}_{1},\cdots,\boldsymbol{q}^{s}_{m}]}$
where $\boldsymbol{q}^{s}_{j}\in{\mathbb{R}}^{d}$. We opt for sentence-level
features for textual questions since the spoken and textual questions are
semantically similar at the sentence-level but need not have any
correspondence at the word-level (e.g., when the spoken question is a
paraphrase of the written question). Thus, extracting word-level features from
the textual questions can lead to sub-optimal alignments. We also note here
that we are doing weakly supervised training with no access to temporal
boundaries but only the order of the occurrence of questions within a segment.
This motivates our choice of using sentence-level features. The right block in
Figure 2 shows the question encoder. It is important to keep the question
encoder frozen since the pretrained sentence transformer is trained with
massive amounts of text data in comparison to our training data. We only want
to learn effective semantic alignments from the spoken utterances to the
already pretrained sentence embeddings.

For each segment $s$, we now have speech chunk features $\boldsymbol{C}^{s}$
and textual features from the questions $\boldsymbol{q}^{s}$ and we need to
learn a cross-modal alignment between these two feature sequences. Towards
this, we propose a Gaussian-weighted Cross Attention scheme.

###  4.3 Gaussian-weighted Cross Attention

Given $\boldsymbol{C}^{s}$ and $\boldsymbol{q}^{s}$, we aim to learn an
alignment between $\boldsymbol{q}^{s}$, representing $m$ sentence embeddings
and $\boldsymbol{C}^{s}$, representing $n_{s}$ chunk embeddings. We propose a
Gaussian-weighted cross attention module with a contrastive learning objective
Chen et al. (2020) in order to learn this alignment. Typically, $n_{s}>m$
implying that a question spans more than one speech chunk. To align a chunk
$\boldsymbol{C}_{i}^{s}$ with a question $\boldsymbol{q}_{i}^{s}$, we consider
an “anchor chunk” feature $\hat{\boldsymbol{C}_{i}^{s}}$ using
$\boldsymbol{q}^{s}$. While chunk is anchored to one question, the neighboring
questions can also provide context. We assume some consecutive chunks
$\boldsymbol{C}^{s}_{a:b}$ combine to form the question
$\boldsymbol{q}_{i}^{s}$, but we do not know the temporal boundaries of
$\boldsymbol{q}_{i}^{s}$ (given our weakly-supervised setting). Further,
speech features $\boldsymbol{C}_{i}^{s}$ are not semantically rich to guide us
with any weak boundaries. However, we know that each chunk unambiguously
belongs to a single underlying question. Hence, we represent each chunk
$\boldsymbol{C}_{i}^{s}$ as a linear combination of $\boldsymbol{q}^{s}$ but
anchor the chunk $\boldsymbol{C}_{i}^{s}$ to a single $\boldsymbol{q}_{i}^{s}$
by making one of the weights high and the rest much smaller to accumulate some
neighboring context. A Gaussian distribution with a moving mean and standard
deviation serves our purpose.

Start-of-segment chunks and end-of-segment chunks map to questions
$\boldsymbol{q}_{1}^{s}$ and $\boldsymbol{q}_{m}^{s}$, respectively, with high
probability. Hence, pivoting on $n_{s}$, we move the Gaussian mean and vary
the standard deviation as a function of the position of the chunk within the
segment:

$\displaystyle\mu_{i}$ $\displaystyle=\frac{(i-1)(m-1)}{n_{s}-1}$ (1)
$\displaystyle\sigma_{i}$ $\displaystyle=\alpha\cdot\mathrm{min}(i-1,n_{s}-i)$
(2) $\displaystyle\mathcal{G}_{i}$
$\displaystyle=\mathrm{Gaussian}(\mu_{i},\sigma_{i})$ (3)

where $\mu_{i}$ and $\sigma_{i}$ are the mean and standard deviation used to
generate the Gaussian distribution $\mathcal{G}_{i}$ for chunk $c_{i}^{s}$ and
$\alpha$ is a scaling factor. From Eqn 3, we see that for each position $i$,
the mean shifts by $\frac{m-1}{n_{s}-1}$ from the previously calculated mean.

Figure 2: End-to-End Indent Architecture. Modules filled with blue color are
frozen.

We sample values $\\{a^{i}_{j}\\}_{j=1}^{m}$ from $\mathcal{G}_{i}$ as shown
in Fig. 3, apply min-max normalization and generate an anchor chunk
${\hat{\boldsymbol{C}_{i}^{s}}}$ using $\boldsymbol{q}^{s}$ for each chunk
$\boldsymbol{C}_{i}^{s}$. Hence, $\\{a^{i}_{j}\\}_{j=1}^{m}$ has a peak value
$a^{i}_{j}$ and also has some weight allocated to the other questions. Thus,
we generate a representation corresponding to each audio chunk
$\boldsymbol{C}_{i}^{s}$ in the shared semantic space as
$\boldsymbol{\hat{C}}_{i}^{s}=\sum_{j=1}^{m}a^{i}_{j}{{q_{j}}^{s}}$. Since
$\boldsymbol{\hat{C}}_{i}^{s}$ and $\boldsymbol{C}_{i}^{s}$ are both chunk-
level representations, we apply a contrastive objective between them to bridge
the modality gap. Since the question encoder is frozen, contrastive learning
facilitates the speech encoder to better align with the text-based sentence
embeddings.

Figure 3: Gaussian-weighted Cross Attention (CA). Gaussian distribution of
weights ($a^{i}_{j}$) for fixed and varying $\sigma$ cases. We sample at
weights at each black-dot ($0,\cdots,m-1$) and the weight value is
intersection of the Gaussian distribution and the dotted line which results in
very peaky distribution for peripheral chunks & vice versa.

#### Dynamic In-Audio Negative Sampling.

For contrastive learning, we need negative chunks. Specifically, negative
chunks from the same interview are desirable to prevent the model from
learning easy solutions by merely exploiting speaker characteristics. During
training, we create groups by randomly shuffling all segments in the speech
interview and making groups of $n$ segments. If the number of segments in the
audio interview is not divisible by $n$, we also create a group with the last
$n$ segments in order to ensure that all the segments are covered during
training. For each chunk in segment $s_{x}$, the corresponding negative chunks
will be from segment $s_{y}$ ($x\neq y$) in the same group $g$. Chunks from
all such $s_{y}$ are potential negatives for each chunk of $s_{x}$. We denote
the number of all potential negatives for $s_{x}$ by $k_{x}$. To keep the
number of negatives for each chunk constant inside a batch, we first compute
the smallest $k_{x}$ across $s_{x}$’s in a batch and randomly sample those
many chunks as negative examples for each chunk in the batch. We also explore
data augmentation by performing reshuffling of all segments to create more new
groups. The number of times we perform data augmentation is denoted by the
hyperparameter $D$. Reshuffling is also performed across epochs.

#### Training Objective.

We define the following contrastive loss Chen et al. (2020) objective:

$\displaystyle L_{\text{st}}$
$\displaystyle=\frac{-1}{|C|}\sum_{C}\mathrm{log}\left(\frac{\mathrm{exp}({\boldsymbol{C}_{i}^{s}}^{\intercal}\hat{\boldsymbol{C}_{i}}^{s})}{\sum_{j\in
n_{g}}\mathrm{exp}({\boldsymbol{C}_{j}^{s}}^{\intercal}\hat{\boldsymbol{C}_{i}}^{s^{\prime}})}\right)$
$\displaystyle L_{\text{ss}}$
$\displaystyle=\frac{-1}{|C|}\sum_{C}\mathrm{log}\left(\frac{\mathrm{exp}({\boldsymbol{C}_{i}^{s}}^{\intercal}{\boldsymbol{C}_{i}}^{s})}{\sum_{j\in
n_{g}}\mathrm{exp}({\boldsymbol{C}_{j}^{s}}^{\intercal}{\boldsymbol{C}_{i}}^{s^{\prime}})}\right)$

where $n_{g}$ is a union of $i$ and dynamically sampled negative chunk indices
(as described earlier), $s,s^{\prime}\in g,s=s^{\prime}$ if $j=i$, $C$ denotes
all chunks in a batch; $L_{\text{st}}$ aligns the speech and text spaces while
$L_{\text{ss}}$ regularizes the network by explicitly encouraging negative
speech chunks $\boldsymbol{C}_{j}^{s}$ ($j\in n_{g},j\neq i$) to be far apart
from $\boldsymbol{C}_{i}^{s}$ in the shared space. Indent is optimized over
the loss $L=L_{\text{st}}+L_{\text{ss}}$. We apply the label smoothing
technique Reed et al. (2014) that forces Indent to lower the confidence of the
positive chunk by redistributing a small fraction of its probability mass
uniformly to all the negative chunks. (In the experiments, we set positive
probability to be 0.95 and redistribute 0.05 probability mass evenly among all
the negatives.)

##  5 Experiments and Results

###  5.1 Experimental Setup

#### Question Encoder.

We use pretrained Language-agnostic BERT Sentence Embedding (LaBSE) Feng et
al. (2022) as sentence embeddings. For each question $q$, LaBSE produces a
$768$-dimensional embedding. Since we keep the question encoder frozen, a
helpful consequence of using LaBSE is that we can use questions in any
language that are well-aligned in the LaBSE embedding space. In Table 2, we
present experiments on questions in multiple Indian languages.

#### Speech Encoder.

We use a pretrained voice activity detector (VAD) Wiseman (2017) to split
segment $s$ into $n_{s}$ chunks, $\\{c_{1}^{s},\cdots,c_{n_{s}}^{s}\\}$. To
extract the chunkwise speech features $\boldsymbol{c_{i}}^{s}$, we use a
state-of-the-art multilingual model IndicWav2vec-Hindi Javed et al. (2022).
Each chunkwise speech vector is a $1024$-dimensional feature vector. To
aggregate, we apply a convolution layer with 1 filter per channel (i.e., a
total of 1024 filters) of receptive-field 20 and stride 2, followed by GELU
activation Hendrycks and Gimpel (2016) and dropout and a final mean pooling on
$\boldsymbol{c}_{i}^{s}$ to derive $\boldsymbol{\hat{c}}_{i}^{s}$. We then
linearly project $\boldsymbol{\hat{c}}_{i}^{s}$ to a $768$-dimensional vector
to generate $C_{i}^{s}\in C^{s}$. We subsequently apply self attention on
$C^{s}$ to generate $\boldsymbol{C}^{s}$. We present experiments with
different speech feature extractors in Section 5.2.

#### Training.

We train Indent end-to-end keeping VAD Wiseman (2017), Question Encoder LaBSE
Feng et al. (2022) and speech feature extractor IndicWav2Vec-Hindi Javed et
al. (2022) modules frozen using the loss defined in Section 4.3. We use the
Adam optimizer with a learning rate 3e-4, a step scheduler with step-size 10
and decay factor gamma 0.1. After hyperparameter tuning (described in Section
5.2), we set batch size $|B|=4,n=4,D=2,\sigma=0.5$ and train for 40 epochs.

#### Model Variants.

We refer to the complete model with the speech encoder and question encoder,
trained using contrastive losses, as Indent. To compare how Indent performs
with text instead of speech, we train Indent-T by replacing the speech feature
extractor in the speech encoder with ASR predictions for which we extracted
LaBSE embeddings. No-Train is a learning-free technique that computes a simple
matching based on dot-products between LaBSE embeddings of questions with
LaBSE embeddings of ASR predictions from the speech.

#### Evaluation & Metrics.

During inference, we create fixed-chunk-size segments since we do not have
segment boundaries during inference. We need segment information for local
context; removing self-attention within a segment deteriorates the performance
significantly. We set segment-size to 14 (average number of chunks in the
train set). We compute dot-product score $q^{\intercal}\boldsymbol{C}_{i}^{s}$
for the given question $q$ across all chunks $\boldsymbol{C}_{i}^{s}$. We rank
each segment $s$ using $\max_{i=1\cdots
n_{s}}(q^{\intercal}\boldsymbol{C}_{i}^{s})$ (higher score is better). We
evaluate using the recall@k (R@k) metric. If the ground truth segment $s$ of
the question $q$ appears in the top $k$ highest scoring segments, the
algorithm is considered to have retrieved the correct segment for R@k. Our
reported R@k numbers are a mean across interviews of the ratio of the number
of questions correctly retrieved to the total number of questions asked
$\left(\frac{\mathrm{no.\ of\ correctly\ retrieved\ questions}}{\mathrm{total\
no.\ of\ questions}}\right)$. We report R@1, R@5, R@10 and average of the
three is R-avg.

###  5.2 Results & Analysis

Language | No-Train | Indent | Indent-T
---|---|---|---
R@1 | R@5 | R@10 | R-avg | R@1 | R@5 | R@10 | R-avg | R@1 | R@5 | R@10 | R-avg
Hindi (hi) | 19.8 | 38.5 | 51.6 | 36.7 | 21.1 | 46.5 | 61.8 | 43.1 | 40.9 | 69.0 | 79.4 | 63.1
Assamese (as) | 17.7 | 36.0 | 50.3 | 34.7 | 16.4 | 39.4 | 54.3 | 36.7 | 33.8 | 62.8 | 74.6 | 57.1
Bengali (bn) | 18.3 | 37.9 | 51.7 | 36.0 | 20.3 | 45.4 | 57.9 | 41.2 | 37.2 | 67.7 | 78.6 | 61.2
English (en) | 21.5 | 43.2 | 55.9 | 40.2 | 16.9 | 40.4 | 53.7 | 37.0 | 37.2 | 64.5 | 78.0 | 59.9
Gujarati (gu) | 19.2 | 38.5 | 50.9 | 36.2 | 20.0 | 45.1 | 60.3 | 41.8 | 38.7 | 68.6 | 78.8 | 62.0
Kannada (kn) | 19.1 | 37.1 | 52.8 | 36.3 | 18.7 | 45.0 | 58.6 | 40.8 | 39.2 | 66.6 | 78.1 | 61.3
Malayalam (ml) | 18.5 | 35.1 | 49.1 | 34.2 | 18.3 | 44.7 | 57.8 | 40.3 | 35.0 | 65.2 | 77.4 | 59.2
Marathi (mr) | 19.0 | 36.2 | 52.3 | 35.8 | 18.7 | 44.8 | 59.6 | 41.0 | 38.4 | 65.7 | 77.0 | 60.4
Oriya (or) | 20.3 | 39.6 | 52.4 | 37.4 | 18.9 | 44.4 | 58.4 | 40.5 | 39.0 | 67.0 | 78.0 | 61.4
Punjabi (pa) | 19.5 | 37.9 | 50.5 | 36.0 | 20.3 | 45.7 | 58.5 | 41.5 | 39.2 | 67.4 | 78.3 | 61.6
Tamil (ta) | 18.9 | 38.5 | 52.0 | 36.4 | 17.7 | 43.8 | 59.0 | 40.2 | 34.9 | 66.5 | 77.5 | 59.6
Telugu (te) | 17.8 | 37.8 | 50.9 | 35.5 | 18.1 | 43.8 | 58.3 | 40.1 | 37.3 | 65.5 | 76.4 | 59.7

Table 2: Test set results using models trained with Hindi questions and
evaluated on questions translated in 11 different languages.

In Table 3, we present our main results. Indent outperforms No-Train by an
absolute 6.4% indicating that our training method has successfully aligned
speech features to the semantic space of LaBSE embeddings Feng et al. (2022).
Indent is able to exploit very small amounts of weakly-annotated data to
successfully bridge the modality gap and generate semantically-rich speech
features. We also train Indent-T using ASR transcripts instead of speech using
our architecture and we see large (absolute) 26.4% and 19.98% gains in R-avg
performance compared to No-Train and Indent, respectively. This suggests:  1.
The quality of ASR transcripts is reasonably good with WER 17.8 for the base
model without any LM (as reported by Javed et al. (2022)).  2. Aligning across
speech and text modalities with small amounts of weakly labeled data is a
challenging task. Bootstrapping using jointly trained speech and text models
might be worth exploring as future work Ao et al. (2021); Chen et al. (2022).
Our model Indent still performs better than No-Train and can be quite
effective when used with low-resource languages that do not have good ASR. We
also perform experiments with other speech feature extractors such as Vakyansh
Chadha et al. (2022) and XLSR Conneau et al. (2020), but both underperform
compared to IndicWav2vec-Hindi Javed et al. (2022).

Feature extractor | Model Variant | R@1 | R@5 | R@10 | R-avg
---|---|---|---|---|---
No-Train | 19.8 | 38.5 | 51.6 | 36.7
XLSR Conneau et al. (2020) | Indent | 9.3 | 30.5 | 43.8 | 27.9
Vakyansh Chadha et al. (2022) | Indent-T | 35.6 | 62.9 | 74.9 | 57.8
Indent | 15.0 | 34.9 | 47.9 | 32.6
IndicASR Javed et al. (2022) | Indent-T | 40.9 | 69.0 | 79.4 | 63.1
Indent | 21.1 | 46.5 | 61.8 | 43.1

Table 3: Main test results. We experiment with different feature extractors
and model variants.

A useful consequence of training Indent using a frozen question encoder is
that we can query our model with any of the 109 languages supported by LaBSE.
We evaluate Indent on 11 different Indian languages unseen during training.
During inference, we translate the questionnaire using IndicTrans’s
Indic2English (for Hindi to English translation) and Indic2Indic (for Hindi to
other Indic language translation) models Ramesh et al. (2022). Note that
during training we use speech in Hindi matched to questions in Hindi, while
during inference we query using text in multiple Indian languages. From the
results shown in Table 2, we observe that Indent outperforms No-Train for all
Indian languages. Indent-T performs the best, thus pointing to high-quality
ASR transcriptions. Interestingly, performance on English is better with No-
Train compared to Indent. This could be because LaBSE was trained
predominantly on English data.

Tables 4, 5, 6, and 7 show results on the dev set using Indent from various
ablation experiments by tuning important hyperparameters. Table 4 shows
different group sizes. We see that having group sizes that are too large or
too small are detrimental to performance. Note that each group contains an
average of $14n$ chunks. We envisage that, with small groups there is not much
diversity in negatives and with large groups a lot of potential negatives are
never used since we only take minimum number of possible negatives across
chunks in a batch (as described in Section 4.3). Hence, we train with $n=4$.

$n$ | R@1 | R@5 | R@10 | R-avg
---|---|---|---|---
2 | 18.7 | 43.4 | 59.5 | 40.5
4 | 18.0 | 43.8 | 66.3 | 42.7
8 | 18.4 | 44.5 | 59.9 | 40.9
Table 4: Tuning hyperparameter group size $n$, $|B|=4,D=2,\sigma=2.5$. Metrics
reported on Dev set using Indent.

Data augmentation hyperparameter $D$ creates more groups of different segment
combinations. Table 5 shows how data augmentation helps, but at the cost of
overfitting. $D=3$ leads to overfitting; $D=2$ gives the best trade-off
between train and dev set performance.

$D$ | R@1 | R@5 | R@10 | R-avg
---|---|---|---|---
1 | 13.4 | 39.3 | 57.9 | 36.9
2 | 18.0 | 43.8 | 66.3 | 42.7
3 | 16.9 | 46.1 | 65.3 | 42.8
Table 5: Tuning hyperparameter data augmentation $D$, $|B|=4,n=4,\sigma=2.5$.
Metrics reported on Dev set using Indent.

In Table 6, we present experiments with varying batch sizes. Similar to group
size, we do not want the batches to be too large or too small. Although
negative sampling is within a group, the number of negatives are the same
within the batch and hence similar restrictions hold as with group sizes.
Based on the results in Table 6, we set batch size $|B|=4$.

$|B|$ | R@1 | R@5 | R@10 | R-avg
---|---|---|---|---
2 | 16.1 | 43.0 | 60.4 | 39.8
4 | 18.0 | 43.8 | 66.3 | 42.7
8 | 17.0 | 42.6 | 61.6 | 40.4
12 | 15.0 | 40.6 | 56.3 | 37.3
Table 6: Tuning hyperparameter batch size $|B|$, $n=4,D=2,\sigma=2.5$. Metrics
reported on the Dev set using Indent.

In Table 7, we show experiments on tuning the standard deviation $\sigma$ of
the Gaussian distribution from equation 3. Using equation 2, we vary $\sigma$
and scale it with $\alpha$. In Table 7, we present the metrics across various
$\alpha$ values and constant $\sigma$ values. Our intuition behind using high
$\sigma$ in the center was that central chunks are hard to anchor, but Table 7
suggests otherwise. We see that constant $\sigma$ is more effective by a
significant margin, indicating that even central chunks are strongly anchored
to some question.

Figure 4: Chunk vs. question dot-product scores using Indent. Same colored
chunks belong to the same segment; black refers to ground-truth. Note that the
question and the ASR transcript of the ground-truth segment are highly
correlated. Chunks 52 and 55 have high non-negative scores due to the presence
of the word mobile and the question asked about smart phone which matches
exactly with chunk 42 mobile phone.

#### Analysis.

We qualitatively analyze Indent by plotting all the chunk scores for a given
question. As shown in Figure 4, Indent is able to correctly identify the
ground truth segment (in black); the dot-product scores are also very high for
the ground truth segment compared to the rest.

Standard deviation | R@1 | R@5 | R@10 | R-avg
---|---|---|---|---
Fixed $\sigma$ | 0.2 | 19.7 | 47.1 | 66.8 | 44.5
0.5 | 20.3 | 46.9 | 67.5 | 44.9
1 | 19.6 | 42.7 | 62.2 | 41.5
1.5 | 19.6 | 44.1 | 64.7 | 42.8
2.5 | 18.0 | 43.8 | 66.3 | 42.7
3 | 18.1 | 43.5 | 65.2 | 42.3
Varying $\alpha$ | 0.1 | 15.8 | 41.8 | 57.3 | 38.3
0.25 | 15.9 | 42.4 | 58.8 | 39.0
0.4 | 18.0 | 43.7 | 60.0 | 40.6
0.6 | 17.9 | 43.6 | 63.0 | 41.5
0.75 | 18.1 | 43.0 | 65.2 | 42.1
0.9 | 17.5 | 40.4 | 63.6 | 40.5
Table 7: Metrics reported on Dev set with tuning $\sigma$ and $\alpha$;
$|B|=4,n=4,D=2$ and Indent architecture is used.

##  6 Conclusions and Future Work

We present a new end-to-end weakly-supervised approach Indent, to align audio
recordings with questions and an associated dataset of long health audio
surveys collected from young mothers residing in rural areas of Bihar (India).
Indent automatically grounds questions within long audio recordings with the
use of a Gaussian-weighted cross-attention mechanism that exploits the fact
that questions appear temporally ordered in the training audio segments. We
show through extensive experiments that we are able to isolate questions
within long audio recordings reasonably well and also demonstrate how this
framework can be used with queries in other Indian languages.

## Acknowledgments

Unlinked anonymized data used for the work reported here was collected by CARE
India Solutions for Sustainable Development, as part of system strengthening
and governmental program implementation tracking and supported by Bill and
Melinda Gates Foundation. The technical and legal support was provided by the
Koita Centre for Digital Health (KCDH), and specifically by Dr Raghavendran
Lakshmi Narayanan, the Health Information Manager at KCDH. Shubham Nemani and
Pranjal Saini’s invaluable contributions and assistance during the initial
data processing stages are deeply appreciated.

## References

  * Anguera et al. [2014] Xavier Anguera, Jordi Luque, and Ciro Gracia.  Audio-to-text alignment for speech recognition with very limited resources.  In Fifteenth Annual Conference of the International Speech Communication Association, 2014.
  * Anne Hendricks et al. [2017] Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell.  Localizing moments in video with natural language.  In Proceedings of the IEEE international conference on computer vision, pages 5803–5812, 2017.
  * Ao et al. [2021] Junyi Ao, Rui Wang, Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujie Liu, Tom Ko, Qing Li, Yu Zhang, et al.  Speecht5: Unified-modal encoder-decoder pre-training for spoken language processing.  arXiv preprint arXiv:2110.07205, 2021.
  * Baevski et al. [2020] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli.  wav2vec 2.0: A framework for self-supervised learning of speech representations.  Advances in neural information processing systems, 33:12449–12460, 2020.
  * Bredin et al. [2020] Hervé Bredin, Ruiqing Yin, Juan Manuel Coria, Gregory Gelly, Pavel Korshunov, Marvin Lavechin, Diego Fustes, Hadrien Titeux, Wassim Bouaziz, and Marie-Philippe Gill.  Pyannote. audio: neural building blocks for speaker diarization.  In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7124–7128. IEEE, 2020.
  * CARE [2023] CARE.  Care india website.  https://www.careindia.org/, 2023.  Accessed: 2023-03-02.
  * Chadha et al. [2022] Harveen Singh Chadha, Anirudh Gupta, Priyanshi Shah, Neeraj Chhimwal, Ankur Dhuriya, Rishabh Gaur, and Vivek Raghavan.  Vakyansh: Asr toolkit for low resource indic languages.  arXiv preprint arXiv:2203.16512, 2022.
  * Chen et al. [2018] Yi-Chen Chen, Chia-Hao Shen, Sung-Feng Huang, and Hung-yi Lee.  Towards unsupervised automatic speech recognition trained by unaligned speech and text only.  arXiv preprint arXiv:1803.10952, 2018.
  * Chen et al. [2020] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In International conference on machine learning, pages 1597–1607. PMLR, 2020.
  * Chen et al. [2022] Zhehuai Chen, Yu Zhang, Andrew Rosenberg, Bhuvana Ramabhadran, Pedro Moreno, Ankur Bapna, and Heiga Zen.  Maestro: Matched speech text representations through modality matching.  arXiv preprint arXiv:2204.03409, 2022.
  * Chung et al. [2018] Yu-An Chung, Wei-Hung Weng, Schrasing Tong, and James Glass.  Unsupervised cross-modal alignment of speech and text embedding spaces.  Advances in neural information processing systems, 31, 2018.
  * Conneau et al. [2020] Alexis Conneau, Alexei Baevski, Ronan Collobert, Abdelrahman Mohamed, and Michael Auli.  Unsupervised cross-lingual representation learning for speech recognition.  arXiv preprint arXiv:2006.13979, 2020.
  * Feng et al. [2022] Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang.  Language-agnostic BERT sentence embedding.  In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 878–891, Dublin, Ireland, May 2022. Association for Computational Linguistics.
  * Gao et al. [2017] Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia.  Tall: Temporal activity localization via language query.  In Proceedings of the IEEE international conference on computer vision, pages 5267–5275, 2017.
  * Gupta et al. [2021] Anirudh Gupta, Harveen Singh Chadha, Priyanshi Shah, Neeraj Chhimwal, Ankur Dhuriya, Rishabh Gaur, and Vivek Raghavan.  Clsril-23: cross lingual speech representations for indic languages.  arXiv preprint arXiv:2107.07402, 2021.
  * Heinritz et al. [2022] Florian Heinritz, Gisela Will, and Raffaela Gentile.  Surveying illiterate individuals: Are audio files in computer-assisted self-interviews a useful supportive tool?  Migration Research in a Digitized World, page 101, 2022.
  * Hendrycks and Gimpel [2016] Dan Hendrycks and Kevin Gimpel.  Gaussian error linear units (gelus).  arXiv preprint arXiv:1606.08415, 2016.
  * Javed et al. [2022] Tahir Javed, Sumanth Doddapaneni, Abhigyan Raman, Kaushal Santosh Bhogale, Gowtham Ramesh, Anoop Kunchukuttan, Pratyush Kumar, and Mitesh M Khapra.  Towards building asr systems for the next billion users.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 10813–10821, 2022.
  * Jones [2003] Rachel Jones.  Survey data collection using audio computer assisted self-interview.  Western journal of nursing research, 25:349–58, 05 2003.
  * Liu et al. [2018] Meng Liu, Xiang Wang, Liqiang Nie, Qi Tian, Baoquan Chen, and Tat-Seng Chua.  Cross-modal moment localization in videos.  In Proceedings of the 26th ACM international conference on Multimedia, pages 843–851, 2018.
  * López and Luque [2022] Fernando López and Jordi Luque.  Iterative pseudo-forced alignment by acoustic ctc loss for self-supervised asr domain adaptation.  arXiv preprint arXiv:2210.15226, 2022.
  * Mithun et al. [2019] Niluthpol Chowdhury Mithun, Sujoy Paul, and Amit K Roy-Chowdhury.  Weakly supervised video moment retrieval from text queries.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11592–11601, 2019.
  * Ramesh et al. [2022] Gowtham Ramesh, Sumanth Doddapaneni, Aravinth Bheemaraj, Mayank Jobanputra, Raghavan AK, Ajitesh Sharma, Sujit Sahoo, Harshita Diddee, Mahalakshmi J, Divyanshu Kakwani, Navneet Kumar, Aswin Pradeep, Srihari Nagaraj, Kumar Deepak, Vivek Raghavan, Anoop Kunchukuttan, Pratyush Kumar, and Mitesh Shantadevi Khapra.  Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages.  Transactions of the Association for Computational Linguistics, 10:145–162, 02 2022.
  * Ravanelli et al. [2021] Mirco Ravanelli, Titouan Parcollet, Peter Plantinga, Aku Rouhe, Samuele Cornell, Loren Lugosch, Cem Subakan, Nauman Dawalatabad, Abdelwahab Heba, Jianyuan Zhong, Ju-Chieh Chou, Sung-Lin Yeh, Szu-Wei Fu, Chien-Feng Liao, Elena Rastorgueva, François Grondin, William Aris, Hwidong Na, Yan Gao, Renato De Mori, and Yoshua Bengio.  SpeechBrain: A general-purpose speech toolkit, 2021.  arXiv:2106.04624.
  * Reed et al. [2014] Scott Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich.  Training deep neural networks on noisy labels with bootstrapping.  arXiv preprint arXiv:1412.6596, 2014.
  * Reichmann et al. [2010] William M Reichmann, Elena Losina, George R Seage III, Christian Arbelaez, Steven A Safren, Jeffrey N Katz, Adam Hetland, and Rochelle P Walensky.  Does modality of survey administration impact data quality: audio computer assisted self interview (acasi) versus self-administered pen and paper?  PloS one, 5(1):e8728, 2010.
  * Synnaeve et al. [2019] Gabriel Synnaeve, Qiantong Xu, Jacob Kahn, Tatiana Likhomanenko, Edouard Grave, Vineel Pratap, Anuroop Sriram, Vitaliy Liptchinsky, and Ronan Collobert.  End-to-end asr: from supervised to semi-supervised learning with modern architectures.  arXiv preprint arXiv:1911.08460, 2019.
  * Wiseman [2017] John Wiseman.  Python WebRTC-VAD website.  https://pypi.org/project/webrtcvad/, 2017.  Accessed: 2023-06-13.
  * Xu et al. [2018] Huijuan Xu, Kun He, Leonid Sigal, Stan Sclaroff, and Kate Saenko.  Text-to-clip video retrieval with early fusion and re-captioning.  arXiv preprint arXiv:1804.05113, 2(6):7, 2018.
  * Zhang et al. [2022] Hao Zhang, Aixin Sun, Wei Jing, and Joey Tianyi Zhou.  The elements of temporal sentence grounding in videos: A survey and future directions.  arXiv preprint arXiv:2201.08071, 2022.