# Noise Reduction of Stochastic Density Functional Theory for Metals

Jake P. Vu<EMAIL_ADDRESS>Department of Chemistry, Purdue University, West
Lafayette, Indiana 47906, USA  Ming Chen<EMAIL_ADDRESS>Department of
Chemistry, Purdue University, West Lafayette, Indiana 47906, USA

###### Abstract

Density Functional Theory (DFT) has become a cornerstone in the modeling of
metals. However, accurately simulating metals, particularly under extreme
conditions, presents two significant challenges. First, simulating complex
metallic systems at low electron temperatures is difficult due to their highly
delocalized density matrix. Second, modeling metallic warm-dense materials at
very high electron temperatures is challenging because it requires the
computation of a large number of partially occupied orbitals. This study
demonstrates that both challenges can be effectively addressed using the
latest advances in linear-scaling stochastic DFT methodologies. Despite the
inherent introduction of noise into all computed properties by stochastic DFT,
this research evaluates the efficacy of various noise reduction techniques
under different thermal conditions. Our observations indicate that the
effectiveness of noise reduction strategies varies significantly with the
electron temperature. Furthermore, we provide evidence that the computational
cost of stochastic DFT methods scales linearly with system size for metal
systems, regardless of the electron temperature regime.

##  I Introduction

Metals are ubiquitous in daily life, industry, and academic research, due to
their unique mechanical, magnetic, and electronic properties [1, 2, 3].
Theoretical modeling of metal electronic structures is crucial for
understanding their ground and excited state properties [4, 5]. Density
functional theory (DFT) is a widely used approach in materials science and
condensed matter physics to accurately model the ground-state properties of
metals at reasonable computational costs. [4, 6, 7] Recent studies have
focused on complex metal systems such as metal nanostructures, high-entropy
alloys, and metal-support interfaces, which exhibit intriguing properties and
have significant applications [8, 9, 10]. These materials often require large
supercells in theoretical modeling, sometimes containing more than 1000 atoms
[11, 12, 13]. While DFT has been effective for studying bulk metal systems,
the computational cost of conventional DFT scales as $O(N^{3}_{e})$, where
$N_{e}$ represents the number of electrons. This makes the application of
conventional DFT methods to complex metal systems challenging.

Another significant challenge arises in modeling metallic systems at high
electron temperatures, especially in the context of warm-dense materials
relevant to fusion energy [14]. Accurate modeling of these materials requires
the description of electronic structure at extremely high electron
temperatures [15]. Conventional DFT methods face difficulties in modeling warm
dense materials due to the requirement of a large number of partially occupied
orbitals [16]. Therefore, the development of DFT methodologies that can
minimize computational expenses is essential for modeling metals across
various system sizes and under different conditions.

Various linear-scaling DFT methods have been developed to address the
computational challenges associated with modeling complex materials [17, 18,
19, 20, 21, 22, 23, 24, 25, 26]. These methods generally employ one of two
strategies: exploiting the “nearsightedness” of the electronic structure, or
decomposing the system into smaller, manageable subsystems [19, 27, 24, 28,
29, 26]. The concept of nearsightedness in electronic structure implies that
the one-body density matrix, $\rho(\mathbf{r},\mathbf{r}^{\prime})$, decays
exponentially with the distance between points $\mathbf{r}$ and
$\mathbf{r}^{\prime}$ [30, 31]. This property allows for the truncation of the
density matrix, facilitating the use of sparse matrix techniques to achieve
linear-scaling computational efficiency [32, 33]. Such an approach is
particularly effective for systems with a large band gap or metallic systems
at high electron temperatures [34, 32]. However, challenges arise when dealing
with metals at low electron temperatures due to the slow decay of
$\rho(\mathbf{r},\mathbf{r}^{\prime})$, making the application of this
principle less straightforward [35, 36]. Alternatively, some methods focus on
the subsystem decomposition approach, which relies on embedding theory to
account for interactions between a subsystem and its environment [17]. This
technique has shown success with non-covalently bonded systems, such as
molecular clusters [24], but faces significant challenges when applied to
inorganic materials with covalent bonds [37]. Notwithstanding, certain linear-
scaling DFT methods have demonstrated the ability to efficiently model metals
with high electron temperatures by focusing on a localized density matrix.
Nonetheless, these methods often struggle to access high-energy orbital
information, which remains a limitation [18, 38, 39].

Stochastic DFT (sDFT) represents a significant advancement in linear-scaling
DFT methodologies, uniquely addressing challenges posed by systems
characterized by small or negligible fundamental gaps [40]. Unlike traditional
DFT approaches that depend on the Kohn-Sham (KS) orbitals, sDFT calculates
ground-state properties—including electron density, ground- state energy, and
forces on nuclei—through the statistical averaging over a set of stochastic
orbitals. Research into the application of sDFT on semiconductor materials has
revealed a particularly compelling advantage: the number of stochastic
orbitals required for accurate property calculation does not scale with the
size of the system. This characteristic enables sDFT to achieve linear or even
sublinear scaling efficiency for computations of electron density, energy per
particle, and nuclear forces. The independence from explicit KS orbitals
allows sDFT to effectively model systems at high electron temperatures, such
as warm-dense materials (WDM), with enhanced computational efficiency [41, 42,
39, 43].

Inherent to sDFT, stochastic noise affects all calculated properties,
presenting a significant computational challenge. To reduce this noise by an
order of magnitude, the number of stochastic orbitals must be increased by two
orders of magnitude, leading to a substantial rise in computational demand
[44]. To mitigate this issue, a variety of noise reduction techniques have
been developed, each leveraging different approaches to enhance computational
efficiency without compromising accuracy. Among these, “overlapped embedded-
fragmented sDFT” (o-efsDFT) which is based on real-space fragmentation
technique (oef-sDFT) [45], “energy-window sDFT” (ew-sDFT) which is an energy-
space fragmentation method [44], and “energy window embedded-fragmented sDFT”
(ew-efsDFT) which uses a hybrid strategy [46] have shown promise. These
methods facilitate the study of semiconductors, including those with minimal
band gaps, by effectively managing stochastic noise and computational
workload. Despite their success with semiconductors, the application of these
noise reduction techniques to metals, particularly at varying electron
temperatures, remains an area with limited exploration [42].

In this study, we undertake a comprehensive benchmarking of noise reduction
techniques in stochastic Density Functional Theory (sDFT), aimed at
efficiently modeling metals subjected to both low and high electron
temperatures. Our paper is structured to facilitate a clear understanding of
these techniques and their efficacy. We begin by offering a concise
introduction to sDFT, alongside a detailed overview of the various noise
reduction strategies that have been developed to date. This sets the
foundation for our subsequent analysis. Following the introduction, we delve
into an empirical evaluation of these noise reduction techniques, using bulk
aluminum as our test system. This evaluation encompasses simulations at both
room temperature and elevated temperatures, providing insights into the
performance of these methods across a range of thermal conditions.
Additionally, we extend our analysis to compare the computational costs
associated with the ew-efsDFT method across different system sizes and
temperatures. This comparison aims to elucidate the scalability and efficiency
of ew-efsDFT, offering valuable perspectives on its practical application in
materials science research. Through this structured approach, our study aims
to illuminate the capabilities and limitations of noise reduction techniques
in sDFT, contributing to the ongoing optimization of computational
methodologies for the modeling of metal systems under diverse thermal
conditions.

##  II Stochastic Density Functional Theory

We initiate our discussion with a consideration of a supercell of volume $V$,
encompassing an electron density $\rho(\mathbf{r})$ that is discretized over a
real-space grid consisting of $N_{G}$ grid points. For the scenario of spin-
unpolarized systems, the electron density is expressed as:

$\rho(\textbf{r})=2\Sigma_{i}f(\varepsilon_{i})\langle\textbf{r}|\psi_{i}\rangle\langle\psi_{i}|\textbf{r}\rangle$
(1)

where $\psi_{i}(\mathbf{r})=\langle\mathbf{r}|\psi_{i}\rangle$ represents the
Kohn-Sham (KS) orbitals, and $\varepsilon_{i}$ denotes the corresponding KS
orbital energies, according to the Kohn-Sham formulation of DFT. The function
$f(x)$, serving as a smearing function to accommodate the occupancy of states,
is crucial for DFT calculations in metallic systems. Specifically, we employ
the Fermi-Dirac distribution:

$f(x;\mu,\beta)=\frac{1}{1+e^{\beta(x-\mu)}}$ (2)

with $\mu$ symbolizing the chemical potential, and $\beta=1/k_{\mathrm{B}}T$
representing the inverse temperature factor, where $k_{\mathrm{B}}$ is the
Boltzmann constant and $T$ denotes the electron temperature. It is pertinent
to highlight that alternative smearing functions, such as the error function,
are equally viable for implementation within the scope of sDFT. The Kohn-Sham
Hamiltonian, $\hat{h}_{\mathrm{KS}}$, is defined as:

$\hat{h}_{\mathrm{KS}}=\hat{t}+\hat{v}_{\mathrm{loc}}+\hat{v}_{\mathrm{nl}}+\hat{v}_{\mathrm{H}}+\hat{v}_{\mathrm{XC}},$
(3)

where $\hat{t}$ delineates the kinetic energy term, $\hat{v}_{\mathrm{loc}}$
and $\hat{v}_{\mathrm{nl}}$ correspond to the local and nonlocal
pseudopotentials, respectively, $\hat{v}_{\mathrm{H}}$ represents the Hartree
potential, and $\hat{v}_{\mathrm{XC}}$ denotes the exchange-correlation
potential. Consequently, the one-body reduced density matrix is succinctly
described by $\hat{\rho}=f(\hat{h}_{\mathrm{KS}};\mu,\beta)$.

In sDFT, the electron density $\rho(\mathbf{r})$ is calculated using
stochastic orbitals $|\chi\rangle$ as follows:

$\rho(\mathbf{r})=\left\langle\left\langle\chi\middle|\hat{\rho}\delta(\mathbf{r}-\hat{\mathbf{r}})\middle|\chi\right\rangle\right\rangle_{\chi}=\left\langle\left|\xi(\mathbf{r})\right|^{2}\right\rangle_{\chi},$
(4)

where $\delta(\cdot)$ is the Dirac delta function, and
$|\xi\rangle=\sqrt{\hat{\rho}}|\chi\rangle$ represents a projected stochastic
orbital. The notation $\langle\cdots\rangle_{\chi}$ denotes averaging over all
samples of $\chi$. In practical sDFT calculations, a finite number
($N_{\chi}$) of stochastic orbitals is employed. The stochastic orbital
$\chi(\mathbf{r})$ is constructed to satisfy
$\langle\chi(\mathbf{r})^{\ast}\chi(\mathbf{r}^{\prime})\rangle_{\chi}=\delta(\mathbf{r}-\mathbf{r}^{\prime})$.
Practically, $\chi(\mathbf{r})=\pm(\Delta V)^{-\frac{1}{2}}$, where $\Delta
V=V/N_{\mathrm{G}}$ is the volume element of the real-space grid, and the sign
of $\chi$ is randomly, uniformly, and independently selected for each grid
point. Equation 4 becomes exact in the limit as the number of stochastic
orbitals $N_{\chi}\rightarrow\infty$; otherwise, the density obtained is a
stochastic approximation.

The projection of stochastic orbitals onto the operator $\sqrt{\hat{\rho}}$
can be efficiently approximated by expanding
$\sqrt{f(\hat{h}_{\mathrm{KS}};\mu,\beta)}$ through polynomial series, such as
Chebyshev or Newton’s Interpolation polynomials [47, 48]. Specifically, this
approximation takes the form:

$\sqrt{f(\hat{h}_{\mathrm{KS}};\mu,\beta)}\approx\sum^{N_{\mathrm{c}}}_{n=0}a_{n}(\mu,\beta)T_{n}(\hat{h}_{\mathrm{KS}}),$
(5)

where $N_{\mathrm{c}}$ denotes the length of the polynomial chain, and
$T_{n}(\cdot)$ are the Chebyshev polynomials. Within the framework of sDFT,
the application of $T_{n}(\hat{h}_{\mathrm{KS}})$ to a stochastic orbital is
computed utilizing the iterative formula associated with Chebyshev
polynomials, thereby facilitating a computationally efficient projection
algorithm.

A property $O$, other than electron density, can be calculated as follows. If
$O$ is an explicit functional of electron density, it can be calculated using
$\rho$ from Eq. (4). Otherwise, $O$ requires calculating the trace of an
operator, which can be evaluated by the stochastic trace formula. In the
stochastic trace formula, the trace of an operator $\hat{O}[\rho]$ is
calculated with

$\mathrm{Tr}(\hat{O}[\rho])=\langle\langle\chi|\hat{O}[\rho]|\chi\rangle\rangle_{\chi}\,.$
(6)

Here, $\hat{O}[\rho]$ indicates that $\hat{O}$ depends on the electron density
$\rho$. In a special case, $\hat{O}[\hat{\rho}]=\hat{\rho}[\rho]\hat{A}$,
where $\hat{A}$ is a one-body operator. In this case,

$O=2\operatorname{Tr}(\hat{\rho}\hat{A})=2\langle\langle\xi|\hat{A}|\xi\rangle\rangle_{\chi}\,.$
(7)

For some other properties, $\hat{O}$ is more complicated. For example, sDFT
uses the following formula to calculate the electron density of states at
energy $\varepsilon$:

$D(\varepsilon)=\langle\langle\chi|\delta(\hat{h}_{\mathrm{KS}}-\varepsilon)|\chi\rangle\rangle_{\chi}\approx\langle\langle\chi|G(\hat{h}_{\mathrm{KS}};\varepsilon,\sigma)|\chi\rangle\rangle_{\chi},$
(8)

where
$G(x;\varepsilon,\sigma)=\frac{1}{\sqrt{2\pi\sigma^{2}}}e^{-(x-\varepsilon)^{2}/2\sigma^{2}}$
is a Gaussian function. A pre-selected $\sigma$ is the broadening of the
Gaussian function, which determines the broadening of the density of states.
The second example, electron entropy ($S_{\mathrm{KS}}$), is essential for
finite temperature calculations involving metals. In sDFT, $S_{\mathrm{KS}}$
can be calculated using the formula:

$S_{\mathrm{KS}}=-2k_{\mathrm{B}}\langle\langle\chi|\hat{\rho}\log\hat{\rho}+(\hat{\mathrm{I}}-\hat{\rho})\log(\hat{\mathrm{I}}-\hat{\rho})|\chi\rangle\rangle_{\chi},$
(9)

where $\hat{\mathrm{I}}$ represents the identity operator. Both Eq. (8) and
Eq. (9) can be calculated with a Chebyshev polynomial expansion, similar to
Eq. (5).

Due to the use of finite stochastic orbitals $|\chi\rangle$, noise exists in
all quantities calculated with sDFT. This noise varies inversely with the
square root of the number of stochastic orbitals, $N_{\chi}$, in accordance
with the central limit theorem. Specifically, to reduce the stochastic
fluctuation by one order of magnitude, it is necessary to increase $N_{\chi}$
by two orders of magnitude. Therefore, noise reduction techniques are crucial
for accurately determining properties without compromising the efficiency of
sDFT. It is important to note that for many observables, such as electron
density, density of states, and atomic forces, the noise does not scale with
the system size. This allows sDFT to maintain linear scaling. Conversely, for
properties like the energy per particle, the noise diminishes as the system
size increases. This implies that fewer stochastic orbitals are required for
larger systems, making sDFT a sub-linear-scaling method. However, for total
energy calculations, noise amplifies with system size, challenging the linear-
scaling property of sDFT. Users are encouraged to carefully assess the
properties they intend to calculate with sDFT, considering the method’s
scalability and noise characteristics for their specific application.

###  II.1 Overlapped Embedded Fragmentation

Reduction in the statistical noise can be achieved by introduction of a
reference system that can be calculated by deterministic KS-DFT. This will
provide us a reasonable approximation of the electron density matrix which is
corrected with sDFT. One way to generate such reference system is to decompose
the whole system into overlapped fragments, leading to the implementation of
the overlapped embedded-fragmented stochastic DFT (o-efsDFT). We will outline
the method in the rest of this section.

Figure 1: Schematic for overlapped fragmentation is shown in this figure. [21]
The system is divided into overlapped fragments. Each fragment $f$ is composed
of a core region ($C_{f}$) and a buffer region ($B_{f}$)is defined around each
$C_{f}$. A deterministic DFT is performed for each dressed fragment ($D_{f}$),
which is the union of $C_{f}$ and $B_{f}$ ($C_{f}\cup B_{f}$)

.

The o-efsDFT method divides a supercell into fragments refereed to as core
regions ($C_{f}$) that is wrapped by a buffer region ($B_{f}$) to create a
dressed fragment ($D_{f}=C_{f}\cup B_{f}$, where $f$ is the fragment index.
The fragment density matrix, $\hat{\rho}_{f}$, is defined as

$\left\langle\mathbf{r}\middle|\hat{\rho}_{f}\middle|\mathbf{r}^{\prime}\right\rangle=\begin{cases}\sum_{i=1}\sqrt{f(\varepsilon_{i}^{f};\mu_{f},\beta)}\left\langle\mathbf{r}\middle|\varphi_{i}^{f}\right\rangle\left\langle\varphi_{i}^{f}\middle|\mathbf{r}^{\prime}\right\rangle,&\mathbf{r}^{\prime}\in
D_{f}\\\ 0,&\mathbf{r}^{\prime}\notin D_{f}\end{cases}$ (10)

where $\varphi_{i}^{f}$ are the KS orbitals for the $f$’th fragment obtained
from deterministic KS-DFT. $\varepsilon_{i}^{f}$ are fragment KS orbital
energies. $\mu_{f}$ is the chemical potential that keeps each fragment charge
neutral. We want to emphasize that using o-efsDFT to model a charge-separated
system needs further developments. In o-efsDFT, electron density is calculated
by

$\rho(\mathbf{r})=2\sum_{f}\rho_{f}(\mathbf{r})+2\left\langle|\xi(\mathbf{r})|^{2}\right\rangle_{\chi}-2\sum_{f}\left\langle\left|\xi_{f}(\mathbf{r})\right|^{2}\right\rangle_{\chi}\;\;,$
(11)

where the fragment electron density is
$\rho_{f}(\mathbf{r})=\sum_{i=1}f(\varepsilon_{i}^{f},\mu_{f},\beta)\left|\varphi_{i}^{f}(\mathbf{r})\right|^{2}$,
$\xi_{f}(\mathbf{r})=\sum_{i=1}\sqrt{f(\varepsilon_{i}^{f},\mu_{f},\beta)}\varphi_{i}^{f}(\mathbf{r})\left\langle\varphi_{i}^{f}\middle|\chi\right\rangle_{D_{f}}$,
and
$\left\langle\varphi_{i}^{f}\middle|\chi\right\rangle_{D_{f}}=\int_{D_{f}}\mathrm{d}\mathbf{r}\varphi_{i}^{f}(\mathbf{r})^{\ast}\chi(\mathbf{r})$.
If $\hat{\rho}_{f}$ is similiar with $\hat{\rho}$, noise in the second term of
Eq.(11) will almost cancel noise in the third term. As we will demonsrate
later, it is crucial to have a reasonably good fragment density matrix in
order to maximizing the noise-redcution efficiency of sDFT.

###  II.2 Energy Windows

Reduction in the stochastic noise can also be achieved by dividing the
occupied space into subspaces named as “energy windows”. In this method,
instead of projecting the stochastic orbitals onto the occupied space, the
stochastic orbitals are projected onto energy windows using a set of
projectors, $\hat{\mathbf{P}}_{1},\ldots,\hat{\mathbf{P}}_{N_{\mathrm{w}}}$.
Here, the projector $\hat{\mathbf{P}}_{w}$ is defined as

$\hat{\mathbf{P}}_{w}=f(\hat{h}_{\mathrm{KS}};e_{w},\beta)-f(\hat{h}_{\mathrm{KS}};e_{w-1},\beta)$
(12)

for $1\leq w\leq N_{\mathrm{w}}$, where
$-\infty=e_{0}<e_{1}<\cdots<e_{N_{\mathrm{w}}}=\mu$. In energy windows sDFT
(ew-sDFT), the electron density is obtained from the sum of all the projected
densities.

$\rho(\mathbf{r})=2\sum_{w=1}^{N_{\mathrm{w}}}\left\langle\left|\xi^{(w)}(\mathbf{r})\right|^{2}\right\rangle_{\chi}\equiv
2\sum_{w=1}^{N_{\mathrm{w}}}\rho^{(w)}(\mathbf{r})\;\;,$ (13)

where $\left|\xi^{(w)}\right\rangle=\sqrt{\hat{\mathbf{P}}_{w}}|\chi\rangle$
is a projected stochastic orbital for window
$w\cdot\left|\xi^{(w)}\right\rangle$ are calculated simultaneously with a
single polynomial expansion.

###  II.3 Energy Window Embedded Fragmentation

The final noise reduction method at our disposal combines the approaches used
in o-efsDFT and ew-sDFT with the resulting expression for the electron density
on each grid point r

$\displaystyle\rho(\mathbf{r})=$ $\displaystyle
2\sum_{f}\rho_{f}(\mathbf{r})+2\sum_{w=1}^{N_{\mathrm{w}}}\left\langle\left|\zeta^{(w)}(\mathbf{r})\right|^{2}\right\rangle_{\chi}$
$\displaystyle-2\sum_{f}\sum_{w=1}^{N_{\mathrm{w}}}\left\langle\left|\xi_{f}^{(w)}(\mathbf{r})\right|^{2}\right\rangle_{\chi}\;\;,$
(14)

where
$\zeta^{(w)}(\mathbf{r})=\left\langle\mathbf{r}\left|\sqrt{\hat{\rho}\hat{\mathbf{P}}_{w}}\right|\chi\right\rangle$
and
$\xi_{f}^{(w)}(\mathbf{r})=\left\langle\mathbf{r}\left|\hat{\rho}_{f}\sqrt{\hat{\mathbf{P}}_{w}}\right|\chi\right\rangle$
The projection operators on the energy windows are the same as ew-efsDFT
besides the last window is defined as

$\hat{\mathbf{P}}_{N_{\mathrm{w}}}=\hat{I}-\sum_{w=1}^{N_{\mathrm{w}}-1}\hat{\mathbf{P}}_{w},$
(15)

In ew-sDFT, $\sum_{w=1}^{N_{\mathrm{w}}}\hat{\mathbf{P}}_{w}$ equals the
density matrix $\hat{\rho}$, while in ew-efsDFT, it returns the identity
operator, $\hat{I}$. In addition, the highest energy window is set to
$\varepsilon_{N_{\mathrm{w}}}=\mu$ in ew-sDFT, while in ew-efsDFT, the energy
windows are held fixed for the entire self-consistent procedure and are chosen
to be independent of the chemical potential, $\mu$. This greatly simplifies
the on-the-fly calculations of the chemical potential. [46] The actions of
$\sqrt{\hat{\rho}\hat{\mathbf{P}}_{w}}$ and $\sqrt{\hat{\mathbf{P}}_{w}}$ on
$|\chi\rangle$ are obtained using a Chebyshev polynomial series. $\rho_{f}$ is
calculated in the same way as o-efsDFT.

##  III Results

sDFT, o-efsFT, ew-sDFT, and ew-efsDFT were tested on an aluminum crystal of
varying supercell sizes. Various temperatures ranging from 300 K to 60,000 K
were used to test metal calculations at ambient conditions as well as in warm-
dense metal systems. All calculations were performed using the
Becke–Lee–Yang–Parr (BLYP) functional [49, 50]. It is important to emphasize
that an exchange-correlation functional specifically tuned for warm-dense
matter should be used to accurately model materials with extremely high
electron temperatures [51, 52, 53]. However, this work mainly focuses on
demonstrating the computational efficiency of various sDFT implementations;
thus, the BLYP functional was used in all calculations. A wave function cutoff
of 60.0 Ry and a density cutoff of 96.0 Ry, corresponding to a real-space grid
spacing of 0.17 Å, were used for the calculations. A Troullier-Martins norm-
conserving pseudopotential in Kleinman-Bylander form was employed [54, 55].
Throughout this study, 256 stochastic orbitals were used, regardless of the
number of atoms, and 10 uniformly distributed energy windows were utilized in
the ew-sDFT and ew-efsDFT calculations. All statistical quantities were
obtained using 5 independent runs, unless explicitly stated otherwise.

Figure 2:  The density of states of Al256 at different electron temperatures
are shown in this figure. The green dashed line is the density of states
calculated from dDFT. The solid blue line and the solid red line are
calculated from sDFT at temperature 6,000K and 60,000K with Eq.(8).

Previous work by White and Collins has demonstrated the advantages of sDFT in
studying warm dense matter. [39] The deterministic DFT (dDFT) calculations
lack information on KS orbitals with high orbital energies. Therefore,
evaluating properties which requires knowledge of high energy orbitals, like
electron density of states, is not possbile with dDFT. In Fig.2, we
demonstrate that sDFT is capable of estimating the electronic density of
states even at extremely high energies. This is necessary for studying warm
dense matter with high electron temperatures. Similar results were observed by
White and Collins. [39]

Figure 3:  This figure shows standard deviations (STD) of the density in sDFT
and noise reduction techniques at different temperatures. We obtained the STDs
over 5 independent trials for sDFT (green), ew-sDFT (blue), o-ef-sDFT (red),
and ew-ef-sDFT (purple) at 300K (A), 6,000K (B), and 60,000K (C). We plotted
the STDs for the first 50 grid points along the x-axis with $y=z=0.0$ of
Al256.

Fig.3 shows the standard deviations (STD) of the electron densities calculated
with various sDFT implementations at different electron temperatures. Panel A
demonstrates that real-space fragmentation is not capable of reducing noise in
electron density at low electron temperatures; specifically, the noise from
o-efsDFT is comparable to that from sDFT, and the noise from ew-efsDFT is
comparable to that from ew-sDFT. On the other hand, the energy window method
works reasonably well and can reduce the noise in electron density by almost
50$\%$. This qualitative noise reduction performance is similar when the
electron temperature is increased to 6000 K, as demonstrated in Fig.3, Panel
B. Since the Fermi-Dirac function becomes smoother at higher electron
temperatures, the overlap between two neighboring energy windows becomes
larger. Therefore, both ew-sDFT and ew-efsDFT become less effective at
reducing noise. Real-space fragmentation still cannot efficiently reduce noise
in electron density at 6000 K. When the electron temperature is increased to
60,000 K, the efficiency of noise reduction becomes very different, as shown
in Panel C of Fig.3. First, all real-space-fragmentation-based methods,
including o-efsDFT and ew-efsDFT, can significantly reduce noise in electron
density. The efficiency changes for real-space fragmentation methods will be
explained in the following paragraph. Second, using only ew-sDFT is not
capable of reducing noise in the electron density because there is a
significant overlap between two neighboring window functions.

It has been demonstrated that the noise reduction efficiency of real-space
fragmentation methods depends on the difference between the fragment density
matrix $\sum_{f}\hat{\rho}_{f}$ and the system density matrix $\hat{\rho}$.
[45, 46] The difference between $\sum_{f}\hat{\rho}_{f}$ and $\hat{\rho}$
arises for two reasons. First, a $\Gamma$-point calculation is performed for
each fragment. The finite size error from using a small supercell in a
fragment DFT calculation is one reason for the discrepancy between
$\sum_{f}\hat{\rho}_{f}$ and $\hat{\rho}$. Second, the fragment density matrix
$\hat{\rho}_{f}$ is truncated at the boundary of $D_{f}$, according to
Eq.(10). At 300 K, $\hat{\rho}_{f}$ is very sensitive to fragment size, which
can be confirmed by performing a unit cell calculation with k-point sampling.
At this temperature, DFT calculations converge slowly with an increasing
number of k-points (see Fig.S1 in the Supplemental Information). Also,
$\hat{\rho}_{f}$ is highly delocalized at 300 K, as shown in Fig.4, leading to
a large truncation error at the boundary of $D_{f}$. However, at 60,000 K,
fragment DFT calculations converge quickly with an increasing number of
k-points, and $\hat{\rho}_{f}$ decays to zero much faster (see Fig.4).
Therefore, using a small supercell in a fragment DFT calculation results in
$\sum_{f}\hat{\rho}_{f}$ that is similar to $\hat{\rho}$. Errors associated
with truncating $\hat{\rho}_{f}$ are also negligible at 60,000 K.

Figure 4:  One-dimensional slices of density matrix is shown in this figure. The green line is the results from a 300K deterministic DFT calculation while the blue/magenta line is from a sDFT calculation at 6000 K/60000 K. Details of calculating the density matrix are presented in the support information.  Table 1:  The average energies per electron and corresponding standard deviations (STD) for various sDFT methods at different temperatures. The statistics were obtained from 5 independent trials for each method of sDFT used in this study. In order, the kinetic energy ($E_{\mathrm{K}}$), non-local pseudopotential energy ($E_{\mathrm{nl}}$), local pseudopotential energy ($E_{\mathrm{loc}}$), hartree energy ($E_{\mathrm{H}}$), exchange-correlation energy ($E_{\mathrm{XC}}$), entropy ($TS_{\mathrm{KS}}$), and total energy ($E_{\mathrm{tot}}$) are shown in units of eV in the table. Each entry represents the average energy up to 4 decimal places and in parenthesis next to it, is the STD on the order of 0.1 meV. Temperature(K) | Method | $E_{\mathrm{K}}$ | $E_{\mathrm{nl}}$ | $E_{\mathrm{loc}}$ | $E_{\mathrm{H}}$ | $E_{\mathrm{XC}}$ | $TS_{\mathrm{KS}}$ | $E_{\mathrm{tot}}$
---|---|---|---|---|---|---|---|---
300 | Deterministic | 7.7052 | -0.3639 | 5.4858 | 0.0381 | -6.8521 | -0.0011 | -18.7095
| sDFT | 7.7215(154) | -0.3666(12) | 5.4858(16) | 0.0494(3) | -6.8582(7) | -0.0009(0) | -18.6905(140)
| ew-sDFT | 7.7059(124) | -0.3645(11) | 5.4863(5) | 0.0397(2) | -6.8531(2) | -0.0011(0) | -18.7084(110)
| o-efsDFT | 7.7063(26) | -0.3643(3) | 5.4871(13) | 0.0434(2) | -6.8598(4) | -0.0010(0) | -18.7098(18)
| ew-efsDFT | 7.7097(27) | -0.3643(5) | 5.4861(3) | 0.0385(2) | -6.8528(2) | -0.0011(0) | -18.7054(24)
6,000 | Deterministic | 7.7702 | -0.3733 | 5.4846 | 0.0395 | -6.8534 | -0.1193 | -18.7731
| sDFT | 7.8035(250) | -0.3763(14) | 5.4854(12) | 0.0505(2) | -6.8595(3) | -0.1198(7) | -18.7377(227)
| ew-sDFT | 7.7842(205) | -0.3747(16) | 5.4853(8) | 0.0431(2) | -6.8556(3) | -0.1196(8) | -18.7587(191)
| o-efsDFT | 7.7696(45) | -0.3735(6) | 5.4852(9) | 0.0424(3) | -6.8576(4) | -0.1194(10) | -18.7748(40)
| ew-efsDFT | 7.7739(24) | -0.3737(3) | 5.4846(3) | 0.0401(1) | -6.8543(1) | -0.1199(11) | -18.7706(15)
20,000 | Deterministic | 8.4300 | -0.4342 | 5.4795 | 0.0368 | -6.8528 | -1.3115 | -19.3737
| sDFT | 8.4596(196) | -0.4372(16) | 5.4803(10) | 0.0482(4) | -6.8599(4) | -1.3159(30) | -19.3464(193)
| ew-sDFT | 8.4677(230) | -0.4375(17) | 5.4798(10) | 0.0452(6) | -6.8582(5) | -1.3161(36) | -19.3407(200)
| o-efsDFT | 8.4311(22) | -0.4344(4) | 5.4797(2) | 0.0373(1) | -6.8538(1) | -1.3144(28) | -19.3760(32)
| ew-efsDFT | 8.4300(37) | -0.4342(3) | 5.4797(1) | 0.0370(0) | -6.8532(1) | -1.3145(38) | -19.3766(52)

Noise reduction techniques can also be applied to other ground state
properties like ground state energy and atomic forces. We list different
energy terms calculated with various methods at different temperatures in
Table 1. We would like to emphasize that the reason we display results only up
to 20,000 K is due in part to deterministic calculations of Al256 at a
temperatures higher than 20,000 K is highly challenging. For all ground state
energies, including those for sDFT above 20,000 K, please refer to Table S1 in
the Supplemental Information. Some energy terms, such as local pseudopotential
energy, Hartree energy, and exchange-correlation energy, can be directly
evaluated with electron density. The noise in these energy terms is consistent
with the noise in electron density. At 300 K, the noise reduction efficiency
of o-efsDFT is limited, while o-efsDFT can significantly reduce noise at
20,000 K. ew-sDFT works well at 300 K, and its noise reduction efficiency is
dramatically reduced at 20,000 K. For $E_{\mathrm{loc}}$, $E_{\mathrm{H}}$,
and $E_{\mathrm{XC}}$, ew-efsDFT performs best at all three temperatures.

Kinetic energy and non-local pseudopotential energy can be calculated using
Eq. (7). Noise reduction methods for these energy terms have been developed in
previous studies [45, 44, 46]. Real-space-fragmentation-based methods, such as
o-efsDFT and ew-efsDFT, have demonstrated significant noise reduction in these
energy terms. However, ew-sDFT can only achieve marginal noise reduction, even
at low electron temperatures. Previous studies have indicated that, at low
electron temperatures, ew-sDFT can not reduce the noise in
$\mathrm{Tr}(\hat{\rho}\hat{O})$ only if the matrix representation of
$\hat{O}$ is diagonal-dominant in the deterministic Kohn-Sham (KS) orbital
basis [44]. Our results suggest that the kinetic operator $\hat{t}$ and the
non-local pseudopotential operator $\hat{v}_{\mathrm{nl}}$ tend to be
diagonal-dominant in the deterministic KS orbital basis for the tested system.

Noise in the electron entropy term, $S_{\mathrm{KS}}$, originates from two
primary sources. The first source is the noise in electron density, which
leads to noise in the Kohn-Sham Hamiltonian, $\hat{h}_{\mathrm{KS}}$, and
subsequently in the density matrix, $\hat{\rho}$. The second source is
inherent in the stochastic trace formula used to calculate entropy. Despite
significant reductions in electron density noise through methods such as
o-efsDFT, ew-sDFT, and ew-efsDFT, we observe that noise in $TS_{\mathrm{KS}}$
persists. This observation indicates that the dominant noise contribution
arises from the stochastic trace formula rather than electron density
fluctuations. To delve deeper into this issue, we utilized converged electron
densities, $\rho_{\mathrm{sDFT}}$, from sDFT or its noise-reduced variants,
and performed a deterministic diagonalization of
$\hat{h}_{\mathrm{KS}}[\rho_{\mathrm{sDFT}}]$ to evaluate Eq. (9). This
deterministic trace approach revealed a decrease in entropy noise when
employing noise reduction techniques (see Table S2 in the Supplemental
Information). This outcome underscores that the primary source of entropy
noise is the stochastic trace formula, overshadowing the impact of noise
reduction in electron density. Addressing the noise in the stochastic trace
formula for entropy thus necessitates further research and development.
Nevertheless, our findings indicate that the presence of significant noise in
the entropy term does not adversely affect the accuracy of calculated electron
densities and atomic forces.

In addition to noise-related challenges, we also identified the presence of
bias or systematic errors in certain energy terms, particularly those that are
non-linear functionals of electron density [56]. Notably, bias is evident in
$E_{\mathrm{H}}$, $E_{\mathrm{XC}}$, and $TS_{\mathrm{KS}}$. Such biases are
inherent to sDFT calculations, where, for instance, $E_{\mathrm{H}}$ obtained
from sDFT is significantly higher compared to dDFT calculations, despite
relatively small fluctuations in $E_{\mathrm{H}}$. Noise-reduction methods
play a crucial role in mitigating these biases, largely due to their
effectiveness in minimizing fluctuations in electron density. Specifically,
the systematic error in all energy terms is notably reduced when employing ew-
efsDFT.

Figure 5:  Standard deviations (STD) of the total nuclei force (eV/Å)in sDFT
and noise reduction techniques at different electron temperatures. We obtained
the STD over 5 independent trials for sDFT (green), ew-sDFT (blue), o-ef-sDFT
(red), and ew-ef-sDFT (purple) at 300 K (A), 6,000 K (B), and 60,000 K (C). We
plotted the STDs for the first 20 atoms of Al256.

Reducing noise in atomic forces is very important for determining equilibrium
structures.[57, 58] In Fig.5, the STD of the force on the nuclei along the
x-axis ($F_{x}$) for the first 20 atoms is plotted for sDFT and noise
reduction methods at electron temperatures of 300 K (Panel A), 6,000 K (Panel
B), and 60,000 K (Panel C). At 300K, both o-efsDFT and ew-sDFT can marginally
improve the fluctuation of nuclear forces, while ew-efsDFT performs best at
300 K. At 60,000 K, ew-sDFT is not capable of reducing noise in nuclear
forces. However, both methods, o-efsDFT and ew-efsDFT, can significantly
reduce the noise in nuclear forces at 60,000 K. This is consistent with the
noise reduction efficiency observed in electron density at different
temperatures. In Fig.6, we highlight the efficiency of sDFT and noise
reduction techniques by plotting the average STD of nuclear forces along the
x-axis as a function of temperature. The efficiency of ew-sDFT is consistent
with the work done by Hadad et al.[42].

To investigate the impact of system size on the noise reduction efficiency of
sDFT and ew-efsDFT, we conducted a series of tests focusing on the atomic
forces within different-sized systems. Our tests involved analyzing data from
five independent trials for both sDFT and ew-efsDFT across a range of system
sizes. However, due to resource constraints, we were limited to single runs
for Al500 and Al864 when using

Figure 6:  Noise reduction efficiency in nuclei forces (eV/Å) of various sDFT
methods with respect to temperature. We obtained the standard deviation (STD)
of atomic force on each atom over 5 independent trials for each method at 300
K, 6000 K, 20,000 K, 40,000 K, and 60,000 K. We then average all STDs of
different atoms and plotted the corresponding values for sDFT (green), ew-sDFT
(blue), o-ef-sDFT (red), and ew-ef-sDFT (purple).

ew-efsDFT. Given these limitations, it was crucial to validate the reliability
of our statistical analysis on the atomic forces for these larger systems. To
achieve this, we employed two distinct statistical methods for smaller system
sizes, which served as a basis for comparison. The first method calculated the
STD of the force on each atom across the independent trials, providing a
measure of variability for individual atomic forces. The second method
aggregated the forces from all atoms within a single trial and then calculated
the STD, offering a holistic view of force variability within a single system
snapshot. Our findings, presented in Table 2, reveal no significant difference
between the results obtained from these two statistical approaches. This
consistency suggests that, for larger systems where conducting multiple
calculations is not feasible due to resource constraints, a single calculation
can reliably represent the statistical behavior of atomic forces. This insight
is particularly valuable for efficiently assessing the noise reduction
efficiency of sDFT and ew-efsDFT in large-scale systems, indicating that even
with limited data, meaningful statistical conclusions can be drawn. We want to
emphasize that the equivalence of the two statistical methods may only be
applicable to a crystal structure, where atoms are equivalent in such a
system. We do not recommend calculating STD across different atoms in a slab
model or a nanocrystal. Aside from the sDFT Al4 results in Table 2, which
could be the result of artifacts from using a single unit cell, there appear
to be no significant changes in the efficiency of noise reduction in response
to system size. Since the number of stochastic orbitals is kept constant
across all calculations, this suggests that the number of stochastic orbitals
needed for a system is nearly independent of the system size.

Table 2:  Comparison of different methods of obtaining statistics for nuclei forces (eV/Å). Here we compare using sDFT and ewef-sDFT on various system sizes of Al all of which were ran at an electron temperature of 6,000K. The Numbers reported in column A was a result of taking the STD across all atoms for each trial, then taking the average of those STDs. Numbers reported in column B was a result of taking the STD for each atom across all trials, then taking the average of those STDs. For Al500 and Al864, statistics were obtained from only B. Temperature(K) | Method | System | A | B
---|---|---|---|---
6,000 | sDFT | Al4 | 0.3258 | 0.3332
|  | Al32 | 0.6225 | 0.5257
|  | Al108 | 0.5605 | 0.4764
|  | Al256 | 0.6051 | 0.5078
| ew-efsDFT | Al108 | 0.1193 | 0.1024
|  | Al256 | 0.1078 | 0.0917
|  | Al500 | —— | 0.1097
|  | Al864 | —— | 0.1146
60,000 | sDFT | Al4 | 0.3434 | 0.3476
|  | Al32 | 0.6756 | 0.5861
|  | Al108 | 0.5939 | 0.5079
|  | Al256 | 0.6222 | 0.5253
| ew-efsDFT | Al108 | 0.0165 | 0.0140
|  | Al256 | 0.0122 | 0.0104
|  | Al500 | —— | 0.0154
|  | Al864 | —— | 0.0170

Given the performance of ew-efsDFT in noise reduction for properties such as
density, ground-state energy, and force on nuclei, we tested its computational
efficiency for larger systems. Our tests were conducted on the Cori-KNL and
Perlmutter supercomputers at the National Energy Research Scientific Computing
Center (NERSC). Calculations on Cori-KNL included Al256, Al500, and Al864 at
900 K and 6,000 K. The same set of system sizes was tested on Perlmutter, but
at a system temperature of 60,000 K. The system parameters defined earlier in
the discussion were applied to all six runs. In Fig.7 and Fig.8, we plotted
the total and per iteration CPU time per core versus the number of electrons
in the system on a log-log scale. The top panel of Fig.7 presents the results
for the runs at 6,000 K, which shows that the time needed for each iteration
scales as $O(N{e}^{0.83})$, while the time needed for a single calculation
scales as $O(N_{e}^{0.91})$. The bottom panel of Fig.7 displays the results
for the runs at 900 K, indicating that the time needed for each iteration
scales as $O(N_{e}^{0.86})$, while the time needed for a single calculation
scales as $O(N_{e}^{0.62})$. In Fig.8, the run conducted on Perlmutter at
60,000 K showed that the time needed for each iteration scales as
$O(N_{e}^{0.63})$, and the time needed for a single calculation scales as
$O(N_{e}^{0.96})$. We continue to observe similar scaling on different
computational architectures, and the scaling does not seem to be impacted by
system size or electron temperature.

Figure 7:  Scaling of ew-efsDFT at 900 K and 6,000 K on NERSC Corri-KNL. In
panels A (6,000 K) and B (900 K), the average time it took to complete one SCF
iteration (blue) and the total time (green) to converge the SCF calculations
were plotted against their corresponding system size.

Fig.(8) suggests that our noise reduction method, ew-efsDFT, is significantly
different from the mixed stochastic density functional theory (MDFT) developed
by White and Collins. [39] In MDFT, stochastic orbitals are used to correct a
dDFT calculation at a lower electron temperature. Therefore, the scaling of
MDFT is still limited by the need to perform a dDFT calculation for the whole
system. In contrast, ew-efsDFT uses stochastic orbitals to correct a reference
system composed of fragments. By taking advantage of the localized density
matrix at high electron temperatures, the noise reduction efficiency of ew-
efsDFT is significant, and linear scaling is still achieved at high electron
temperatures. This enables the modeling of disordered warm dense matter with a
large supercell.

Figure 8:  Scaling of ew-efsDFT at 60,000 K on NERSC Perlmutter. The average
time it took to complete one SCF iteration (blue) and the total time (green)
to converge the SCF calculations were plotted against their corresponding
system size.

##  IV Conclusions

In this work, we present benchmark studies on noise-reduction techniques in
stochastic density functional theory (sDFT) for metallic systems. Our findings
indicate that the efficiency of noise reduction depends on the electron
temperature. Specifically, at low electron temperatures, the energy-window-
based method, ew-sDFT, outperforms the fragmentation-based approach, o-efsDFT.
Conversely, at extremely high electron temperatures, o-efsDFT demonstrates
superior performance over ew-efsDFT, attributed to the localized nature of the
one-body density matrix. At both low and high electron temperatures, ew-
efsDFT, which integrates both the energy-window and fragmentation approaches,
emerges as the most efficient method. Our results further reveal that ew-
efsDFT scales linearly, enabling the efficient simulation of large metallic
systems across a large range of electron temperatures. Despite ew-efsDFT’s
current status as the most effective noise-reduction method, we observe
significant residual noises in electron density, energy, and nuclear forces at
low electron temperatures. This underscores the necessity for further
advancements. Additionally, the development of a formula to mitigate noise in
electron entropy is imperative to enhance the accuracy of total energy
calculations at high electron temperatures.

## DATA AVAILABILITY

The data that support the findings of this study are available from the
corresponding author upon reasonable request.

## Supplementary Material

Discussions on convergence of electronic structure with system size and
electron temperatures are included in the Supplementary Material [59, 60]. The
Supplementary Material also includes a complete data on energies per electron
with different electron temperatures, analysis of noise in entropy, and
details in calculating the density matrix.

##  V Acknowledgments

J. V. and M. C. gratefully acknowledges support from the National Science
Foundation EAR-2246687. Resources of the National Energy Research Scientific
Computing Center (NERSC), a U.S. Department of Energy Office of Science User
Facility operated under Contract No. DE-AC02-05CH11231, are greatly
acknowledged.

## References

  * Bhaduri [2018] A. Bhaduri,  _Mechanical Properties and Working of Metals and Alloys_ , Springer Series in Materials Science (Springer Nature Singapore, 2018).
  * McHenry and Laughlin [2014] M. E. McHenry and D. E. Laughlin, “19 - magnetic properties of metals and alloys,” in  _Physical Metallurgy (Fifth Edition)_, edited by D. E. Laughlin and K. Hono (Elsevier, Oxford, 2014) fifth edition ed., pp. 1881–2008.
  * Ehrenreich and Schwartz [1976] H. Ehrenreich and L. Schwartz, “The electronic structure of alloy,”  Solid State Phys., 31, 149–286 (1976).
  * Aarons  _et al._ [2016] J. Aarons, M. Sarwar, D. Thompsett, and C.-K. Skylaris, “Perspective: Methods for large-scale density functional calculations on metallic systems,” J. Chem. Phys. 145, 220901 (2016).
  * Herring and Montemore [2023] C. J. Herring and M. M. Montemore, “Recent advances in real-time time-dependent density functional theory simulations of plasmonic nanostructures and plasmonic photocatalysis,” ACS Nanosci. Au 3, 269–279 (2023).
  * Connolly and Williams [1983] J. W. D. Connolly and A. R. Williams, “Density-functional theory applied to phase transformations in transition-metal alloys,” Phys. Rev. B 27, 5169–5172 (1983).
  * Greeley, Nørskov, and Mavrikakis [2002] J. Greeley, J. K. Nørskov, and M. Mavrikakis, “Electronic structure and catalysis on metal surfaces,” Annu. Rev. Phys. Chem. 53, 319–348 (2002).
  * Zhang  _et al._ [2019] Z. Zhang, C. Zhang, H. Zheng, and H. Xu, “Plasmon-driven catalysis on molecules and nanomaterials,” Acc. Chem. Res. 52, 2506–2515 (2019).
  * Gao  _et al._ [2022] X. Gao, R. Chen, T. Liu, H. Fang, G. Qin, Y. Su, and J. Guo, “High-entropy alloys: a review of mechanical properties and deformation mechanisms at cryogenic temperatures,” J. Mater. Sci. 57, 6573–6606 (2022).
  * Banin, Ben-Shahar, and Vinokurov [2014] U. Banin, Y. Ben-Shahar, and K. Vinokurov, “Hybrid semiconductor–metal nanoparticles: From architecture to function,” Chem. Mater. 26, 97–110 (2014).
  * Hakimioun  _et al._ [2023] A. H. Hakimioun, B. D. Vandegehuchte, D. Curulla-Ferre, K. Kaźmierczak, P. N. Plessow, and F. Studt, “Metal–support interactions in heterogeneous catalysis: Dft calculations on the interaction of copper nanoparticles with magnesium oxide,” ACS Omega 8, 10591–10599 (2023).
  * Stradi  _et al._ [2016] D. Stradi, U. Martinez, A. Blom, M. Brandbyge, and K. Stokbro, “General atomistic approach for modeling metal-semiconductor interfaces using density functional theory and nonequilibrium green’s function,” Phys. Rev. B 93, 155302 (2016).
  * Aarons  _et al._ [2017] J. Aarons, L. Jones, A. Varambhia, K. E. MacArthur, D. Ozkaya, M. Sarwar, C.-K. Skylaris, and P. D. Nellist, “Predicting the oxygen-binding properties of platinum nanoparticle ensembles by combining high-precision electron microscopy and density functional theory,” Nano Lett. 17, 4003–4012 (2017), pMID: 28644034.
  * Koenig  _et al._ [2005] M. Koenig, A. Benuzzi-Mounaix, A. Ravasio, T. Vinci, N. Ozaki, S. Lepape, D. Batani, G. Huser, T. Hall, D. Hicks, _et al._ , “Progress in the study of warm dense matter,” Plasma Phys. Control. Fusion 47, B441 (2005).
  * Smith, Sagredo, and Burke [2018] J. C. Smith, F. Sagredo, and K. Burke, “Warming up density functional theory,” in  _Frontiers of Quantum Chemistry_, edited by M. J. Wójcik, H. Nakatsuji, B. Kirtman, and Y. Ozaki (Springer Singapore, Singapore, 2018) pp. 249–271.
  * Blanchet, Torrent, and Clerouin [2020] A. Blanchet, M. Torrent, and J. Clerouin, “Requirements for very high temperature kohn–sham dft simulations and how to bypass them,” Phys. Plasmas 27, 122706 (2020).
  * Zhu, Pan, and Yang [1996] T. Zhu, W. Pan, and W. Yang, “Structure of solid-state systems from embedded-cluster calculations: A divide-and-conquer approach,” Phys. Rev. B 53, 12713–12724 (1996).
  * Prentice  _et al._ [2020] J. C. Prentice, J. Aarons, J. C. Womack, A. E. Allen, L. Andrinopoulos, L. Anton, R. A. Bell, A. Bhandari, G. A. Bramley, R. J. Charlton, _et al._ , “The onetep linear-scaling density functional theory program,” J. Chem. Phys. 152, 174111 (2020).
  * Kohn [1996a] W. Kohn, “Density functional and density matrix method scaling linearly with the number of atoms,” Phys. Rev. Lett. 76, 3168 (1996a).
  * Wesolowski and Wang [2013] T. A. Wesolowski and Y. A. Wang,  _Recent Progress in Orbital-free Density Functional Theory_ (WORLD SCIENTIFIC, 2013).
  * Artacho  _et al._ [1999] E. Artacho, D. Sánchez-Portal, P. Ordejón, A. Garcia, and J. M. Soler, “Linear-scaling ab-initio calculations for large and complex systems,” Phys. Status Solidi (b) 215, 809–817 (1999).
  * Hernández, Gillan, and Goringe [1996] E. Hernández, M. Gillan, and C. Goringe, “Linear-scaling density-functional-theory technique: the density-matrix approach,” Phys. Rev. B 53, 7147 (1996).
  * Henderson [2006] T. M. Henderson, “Embedding wave function theory in density functional theory,” J. Chem. Phys. 125 (2006).
  * Huang and Carter [2008] P. Huang and E. A. Carter, “Advances in correlated electronic structure methods for solids, surfaces, and nanostructures,” Annu. Rev. Phys. Chem. 59, 261–290 (2008).
  * Soler  _et al._ [2002] J. M. Soler, E. Artacho, J. D. Gale, A. García, J. Junquera, P. Ordejón, and D. Sánchez-Portal, “The siesta method for ab initio order-n materials simulation,” J. Phys. Condens. Matter 14, 2745 (2002).
  * Krishtal  _et al._ [2015] A. Krishtal, D. Sinha, A. Genova, and M. Pavanello, “Subsystem density-functional theory as an effective tool for modeling ground and excited states, their dynamics and many-body interactions,” J. Phys. Condens. Matter 27, 183202 (2015).
  * Yang [1991] W. Yang, “Direct calculation of electron density in density-functional theory,” Phys. Rev. Lett. 66, 1438–1441 (1991).
  * Jacob and Neugebauer [2014] C. R. Jacob and J. Neugebauer, “Subsystem density-functional theory,” Wiley Interdiscip. Rev. Comput. Mol. Sci. 4, 325–362 (2014).
  * Wesolowski, Shedge, and Zhou [2015] T. A. Wesolowski, S. Shedge, and X. Zhou, “Frozen-density embedding strategy for multilevel simulations of electronic structure,” Chem. Rev. 115, 5891–5928 (2015).
  * Kohn [1996b] W. Kohn, “Density functional and density matrix method scaling linearly with the number of atoms,” Phys. Rev. Lett. 76, 3168–3171 (1996b).
  * Schindlmayr [2000] A. Schindlmayr, “Decay properties of the one-particle green function in real space and imaginary time,” Phys. Rev. B 62, 12573–12576 (2000).
  * Baer and Head-Gordon [1997] R. Baer and M. Head-Gordon, “Sparsity of the density matrix in kohn-sham density functional theory and an assessment of linear system-size scaling methods,” Phys. Rev. Lett. 79, 3962 (1997).
  * Challacombe [1999] M. Challacombe, “A simplified density matrix minimization for linear scaling self-consistent field theory,” J. Chem. Phys. 110, 2332–2342 (1999).
  * Benzi, Boito, and Razouk [2012] M. Benzi, P. Boito, and N. Razouk, “Decay properties of spectral projectors with applications to electronic structure,” SIAM J. Matrix Anal. Appl. 33, 1299–1322 (2012).
  * Ismail-Beigi and Arias [1999] S. Ismail-Beigi and T. Arias, “Locality of the density matrix in metals, semiconductors, and insulators,” Phys. Rev. Lett. 82, 2127 (1999).
  * Ismail-Beigi and Arias [1999] S. Ismail-Beigi and T. A. Arias, “Locality of the density matrix in metals, semiconductors, and insulators,” Phys. Rev. Lett. 82, 2127 (1999).
  * Sun  _et al._ [2016] J. Sun, R. C. Remsing, Y. Zhang, Z. Sun, A. Ruzsinszky, H. Peng, Z.-H. Yang, A. Paul, U. Waghmare, X. Wu, M. Klein, and J. Perdew, “Accurate first-principles structures and energies of diversely bonded systems from an efficient density functional,” Nat. Chem. 8, 831–836 (2016).
  * Skylaris  _et al._ [2005] C.-K. Skylaris, P. D. Haynes, A. A. Mostofi, and M. C. Payne, “Introducing onetep: Linear-scaling density functional simulations on parallel computers,” J. Chem. Phys. 122, 084119 (2005).
  * White and Collins [2020] A. J. White and L. A. Collins, “Fast and universal kohn-sham density functional theory algorithm for warm dense matter to hot dense plasma,” Phys. Rev. Lett. 125, 055002 (2020).
  * Baer, Neuhauser, and Rabani [2013] R. Baer, D. Neuhauser, and E. Rabani, “Self-averaging stochastic kohn-sham density-functional theory,” Phys. Rev. Lett. 111, 106402 (2013).
  * Cytter  _et al._ [2018] Y. Cytter, E. Rabani, D. Neuhauser, and R. Baer, “Stochastic density functional theory at finite temperatures,” Phys. Rev. B 97, 115207 (2018).
  * Hadad  _et al._ [2024] R. E. Hadad, A. Roy, E. Rabani, R. Redmer, and R. Baer, “Stochastic density functional theory combined with langevin dynamics for warm dense matter,” (2024), arXiv:2401.11336 .
  * Liu and Chen [2022] Q. Liu and M. Chen, “Plane-wave-based stochastic-deterministic density functional theory for extended systems,” Phys. Rev. B 106, 125132 (2022).
  * Chen  _et al._ [2019a] M. Chen, R. Baer, D. Neuhauser, and E. Rabani, “Energy window stochastic density functional theory,” J. Chem. Phys. 151, 114116 (2019a).
  * Chen  _et al._ [2019b] M. Chen, R. Baer, D. Neuhauser, and E. Rabani, “Overlapped embedded fragment stochastic density functional theory for covalently-bonded materials,” J. Chem. Phys. 150, 034106 (2019b).
  * Chen  _et al._ [2021] M. Chen, R. Baer, D. Neuhauser, and E. Rabani, “Stochastic density functional theory: Real- and energy-space fragmentation for noise reduction,” J. Chem. Phys. 154, 204108 (2021).
  * Kosloff [1988] R. Kosloff, “Time-dependent quantum-mechanical methods for molecular dynamics,” J. Phys. Chem. 92, 2087–2100 (1988).
  * Kosloff [1994] R. Kosloff, “Propagation methods for quantum molecular dynamics,” Annu. Rev. Phys. Chem. 45, 145–178 (1994).
  * Becke [1988] A. D. Becke, “Density-functional exchange-energy approximation with correct asymptotic behavior,” Phys. Rev. A 38, 3098–3100 (1988).
  * Lee, Yang, and Parr [1988] C. Lee, W. Yang, and R. G. Parr, “Development of the colle-salvetti correlation-energy formula into a functional of the electron density,” Phys. Rev. B 37, 785–789 (1988).
  * Moldabekov  _et al._ [2023] Z. A. Moldabekov, M. Lokamani, J. Vorberger, A. Cangi, and T. Dornheim, “Non-empirical mixing coefficient for hybrid xc functionals from analysis of the xc kernel,” J. Phys. Chem. Lett. 14, 1326–1333 (2023).
  * Dornheim, Groth, and Bonitz [2018] T. Dornheim, S. Groth, and M. Bonitz, “The uniform electron gas at warm dense matter conditions,” Phys. Rep. 744, 1–86 (2018).
  * Bonitz  _et al._ [2020] M. Bonitz, T. Dornheim, Z. A. Moldabekov, S. Zhang, P. Hamann, H. Kählert, A. Filinov, K. Ramakrishna, and J. Vorberger, “Ab initio simulation of warm dense matter,” Phys. Plasmas 27, 042710 (2020).
  * Troullier and Martins [1991] N. Troullier and J. L. Martins, “Efficient pseudopotentials for plane-wave calculations,” Phys. Rev. B 43, 1993 (1991).
  * Kleinman and Bylander [1982] L. Kleinman and D. Bylander, “Efficacious form for model pseudopotentials,” Phys. Rev. Lett. 48, 1425 (1982).
  * Fabian  _et al._ [2019] M. D. Fabian, B. Shpiro, E. Rabani, D. Neuhauser, and R. Baer, “Stochastic density functional theory,” Wiley Interdiscip. Rev. Comput. Mol. Sci. 9, e1412 (2019).
  * Arnon  _et al._ [2020] E. Arnon, E. Rabani, D. Neuhauser, and R. Baer, “Efficient langevin dynamics for “noisy” forces,” J. Chem. Phys. 152 (2020).
  * Chen, Baer, and Rabani [2023] M. Chen, R. Baer, and E. Rabani, “Structure optimization with stochastic density functional theory,” J. Chem. Phys. 158 (2023).
  * Giannozzi  _et al._ [2009] P. Giannozzi, S. Baroni, N. Bonini, M. Calandra, R. Car, C. Cavazzoni, D. Ceresoli, G. L. Chiarotti, M. Cococcioni, I. Dabo, A. Dal Corso, S. Fabris, G. Fratesi, S. de Gironcoli, R. Gebauer, U. Gerstmann, C. Gougoussis, A. Kokalj, M. Lazzeri, L. Martin-Samos, N. Marzari, F. Mauri, R. Mazzarello, S. Paolini, A. Pasquarello, L. Paulatto, C. Sbraccia, S. Scandolo, G. Sclauzero, A. P. Seitsonen, A. Smogunov, P. Umari, and R. M. Wentzcovitch, “Quantum espresso: a modular and open-source software project for quantum simulations of materials,” J. Phys. Condens. Matter 21, 395502 (2009).
  * Giannozzi  _et al._ [2017] P. Giannozzi, O. Andreussi, T. Brumme, O. Bunau, M. Buongiorno Nardelli, M. Calandra, R. Car, C. Cavazzoni, D. Ceresoli, M. Cococcioni, N. Colonna, I. Carnimeo, A. Dal Corso, S. de Gironcoli, P. Delugas, R. A. J. DiStasio, A. Ferretti, A. Floris, G. Fratesi, G. Fugallo, R. Gebauer, U. Gerstmann, F. Giustino, T. Gorni, J. Jia, M. Kawamura, H.-Y. Ko, A. Kokalj, E. Küçükbenli, M. Lazzeri, M. Marsili, N. Marzari, F. Mauri, N. L. Nguyen, H.-V. Nguyen, A. Otero-de-la Roza, L. Paulatto, S. Poncé, D. Rocca, R. Sabatini, B. Santra, M. Schlipf, A. P. Seitsonen, A. Smogunov, I. Timrov, T. Thonhauser, P. Umari, N. Vast, X. Wu, and S. Baroni, “Advanced capabilities for materials modelling with quantum espresso,” J. Phys. Condens. Matter 29, 465901 (2017).


# DUALITY TEMPORAL-CHANNEL-FREQUENCY ATTENTION ENHANCED SPEAKER REPRESENTATION
LEARNING

###### Abstract

The use of channel-wise attention in CNN based speaker representation networks
has achieved remarkable performance in speaker verification (SV). But these
approaches do simple averaging on time and frequency feature maps before
channel-wise attention learning and ignore the essential mutual interaction
among temporal, channel as well as frequency scales. To address this problem,
we propose the Duality Temporal-Channel-Frequency (DTCF) attention to re-
calibrate the channel-wise features with aggregation of global context on
temporal and frequency dimensions. Specifically, the duality attention - time-
channel (T-C) attention as well as frequency-channel (F-C) attention - aims to
focus on salient regions along the T-C and F-C feature maps that may have more
considerable impact on the global context, leading to more discriminative
speaker representations. We evaluate the effectiveness of the proposed DTCF
attention on the CN-Celeb and VoxCeleb datasets. On the CN-Celeb evaluation
set, the EER/minDCF of ResNet34-DTCF are reduced by 0.63%/0.0718 compared with
those of ResNet34-SE. On VoxCeleb1-O, VoxCeleb1-E and VoxCeleb1-H evaluation
sets, the EER/minDCF of ResNet34-DTCF achieve 0.36%/0.0263, 0.39%/0.0382 and
0.74%/0.0753 reductions compared with those of ResNet34-SE.

Index Terms—  speaker representation, CNN, attention

##  1 Introduction

Speaker verification (SV) is to decide to accept or reject test utterances
according to the enrollment utterances of the claimed speaker [1]. Recently,
speaker verification based on neural networks has occupied the predominance
with the distinguished speaker representation learning ability [2, 3]. In
general, an SV system based on neural networks consists of a speaker
representation extractor and a supervised classification loss [4, 5, 6].
Specifically, the speaker representation extractor includes a frame-level
feature extractor and an utterance-level speaker embedding aggregator. The
most vital work in SV is to construct a speaker representation extractor that
can generate discriminative speaker embedding [7, 8, 9].

Currently, convolutional neural network (CNN) is the most popular neural
network topology of speaker representation extractor in speaker verification
[10]. Variants of CNN-based speaker representation models have been explored
because of their strong representation extracting ability. X-vector [8, 11,
12] based on 1D convolution is a pioneer work in speaker verification.
Additionally, ResNet [13, 14] based on 2D convolution has become predominant
in various speaker verification challenges [15, 16, 17, 18]. However,
convolution neural networks are not flawless. The mechanism of convolution
operation is to adopt a fixed-size convolution kernel to capture the local
time and frequency speaker patterns in speech. But the fixed-size kernel
limits the receptive field on speech features. Thus, representation extracting
ability [19, 20, 21] is constrained without the essential global context
information which is also important. Meanwhile, convolution neglects the
interaction among channels (convolutional filters) as well.

To mitigate the above problems, attention mechanism [22] has been introduced
in ResNet and x-vector. Zhou et al. [23] adopted channel-wise response feature
attention in ResNet to enhance the speaker representation ability, which was
the first work introducing Squeeze-and-Excitation (SE) [19] attention in the
speaker verification. The SE attention mainly learns the inter-dependency
relationships of channels. Its superior performance has triggered awareness of
adopting a variety of attention modules in the residual block of ResNet
further to promote CNN’s discriminative speaker representation learning
ability. Inspired by the convolutional block attention module (CBAM) [24] in
computer vision, Sarthak and Rai [25] proposed the multi-path (Channel,
Temporal and Frequency respectively) attention of intermediate tensors in the
residual blocks of ResNet. In addition to the SE attention, this work further
introduced a time-frequency domain attention mask learned by large-size
kernels. Xia and Hansen [26] further proposed to combine the SE attention and
non-local time-frequency attention to re-calibrate the channel and temporal-
frequency feature maps. Meanwhile, some recent works [9, 18] also introduced
the SE attention into TDNN (1D convolutional neural network) [12] with
excellent performance in speaker verification.

Although the above SE-attention approaches have achieved superior SV
performance, there is still substantial space to further improve the
discriminative power of attention-guided speaker embeddings. Note that these
approaches do averaging on time and frequency feature maps before channel-wise
attention learning and ignore the essential mutual interaction among temporal,
channel as well as frequency scales.

In this study, we propose the Duality Temporal-Channel-Frequency (DTCF)
attention to assemble the temporal and frequency information into the channel-
wise attention. Recent studies [18, 25] have shown that adopting attention
mechanism in frequency domain has great potential to improve the
representation ability of speaker embedding models. There are important
interactions between channel, temporal and frequency dimensions, which can
lead to clear performance gain. In this paper, instead of simple averaging on
time and frequency feature maps in previous studies, we specifically encode
temporal and frequency information into channel-wise attention, resulting in
the so-called duality attention – time-channel (T-C) attention as well as
frequency-channel (F-C) attention. The proposed duality attention mechanism
aims to focus on salient regions along the T-C and F-C feature maps that may
have more considerable impact on the global context, leading to more
discriminative speaker representations. Experimental results show that our
proposed attention module significantly outperforms the SE attention and other
competitive attention modules.

The most similar structure to ours is [27], in which coordinate attention is
introduced to mobile networks [28, 29] to solve computer vision problems.
Specifically, the coordinate attention embeds positional information of images
into the channel attention, achieving promising performance in the image
segmentation task. In this paper, we introduce a similar structure, the DTCF
attention to the popular ResNet in speaker verification, to explore the
salient regions in time-channel and frequency-channel dimensions in speech
signals, aiming to learn more discriminative speaker representation.

The rest of the paper is organized as follows. Section 2 overviews of the
speaker representation extractor framework. Section 3 details our proposed
attention module. Section 4 introduces the experimental setup and Section 5
summarizes the experimental results. Finally, conclusions are drawn in Section
6.

##  2 Speaker Representation Framework

Deep residual networks [13] are widely used in image recognition and have
recently been applied to speaker representation learning [2, 30]. In our
study, the backbone of speaker representation learning network is the
classical ResNet34 [16, 31], which has achieved good results [32, 33] on
different datasets of speaker verification.

The overview of the speaker representation learning framework is illustrated
in Fig. 1. It mainly consists of two modules, which are speaker representation
extractor and classifier respectively. The speaker representation extractor
includes two sub-modules, which are a frame-level representation extractor in
the dashed frame in Fig. 1 and an utterance-level aggregator. In general,
there are four residual blocks in ResNet34. The output of each residual block
flows through the attention block (e.g., SE-attention) to learn the inter-
dependence of channels before the skip connection [23]. The utterance-level
aggregator encodes various length frame-level speaker representations into
fixed-length utterance-level speaker representations. We use the attention
statistic pooling (ASP) [34] aggregator in the speaker representation
framework. In the classification module, we adopt the Additive Angular Margin
Softmax (AAM-Sotfmax) [35] as the loss function.

In this paper, we mainly focus on expanding the respective field of CNN with
innovative channel-wise attention as well as exploiting the global context
information of time and frequency dimensions. Thus, we introduce the Duality
Temporal-Channel-Frequency (DTCF) attention mechanism after each residual
block in ResNet34 to encode the temporal and frequency information into the
channel-wise attention masks. Compared with other channel-wise attention
modules or their variants, the DTCF attention can preserve the time and
frequency information as well as learn the interactions among temporal,
channel, and frequency.

Fig. 1: Overview of Speaker Representation Learning

##  3 Duality Temporal-Channel-Frequency Attention

###  3.1 SE Attention

Channel-wise attention modules are widely embedded in ResNet to learn the
inter-dependence of channels in intermediate feature maps. As shown in Fig. 2,
SE attention adopts global average pooling (GAP) to squeeze the 3D temporal-
channel-frequency tensor into a 1D channel-wise feature, then with two fully
connected (FC) layers and a sigmoid method to weigh different channel
importance for boosting speaker representation learning ability. Given the
intermediate tensor $X\in\mathbb{R}^{C\times T\times F}$ is the 3D speech
feature map in networks, $C$ is the number of channels (convolutional
kernels), $T$ is the number of frames in the temporal domain, $F$ is the
number of frequency bins. The average pooling on time and frequency dimension
of SE attention [19] is formulated as:

$\displaystyle X^{C}=\frac{1}{T\times F}\sum_{i=1}^{T}\sum_{j=1}^{F}X_{ij},$
(1)

where $X^{C}$ is the results of global average pooling of the intermediate
feature map $X$ on time and frequency dimensions. Next is to learn the
channel-wise attention weights of $X^{C}$. The channel-wise attention mask
learning is formulated as:

$\displaystyle M^{C}=\delta(W_{2}^{C\times
C^{\prime}}\times(relu((W_{1}^{C^{\prime}\times C}\cdot X^{C})))),$ (2)

where $\cdot$ means matrix multiplication. $W_{1}^{C^{\prime}\times C}$ and
$W_{2}^{C\times C^{\prime}}$ are two fully-connected layers which capture the
inter-dependence of channels in the intermediate speech feature map $X$. The
dimensional reduction factor $r=\frac{C}{C^{\prime}}$ indicates the bottleneck
output feature map reduction ratio to avoid the parameters overhead. Finally,
a sigmoid function $\delta$ is used to scale the channel-wise weights. The
attention mask is a set of attention factors predicted by supervised speaker
classification loss aiming to emphasize essential channels and compress the
useless channels. As shown in Fig. 2, the recalibration operation means that
the attention mask dot multiplies with the original tensor $X$.

Fig. 2: A Squeeze-and-Excitation (SE) Attention Block

###  3.2 Encode Time-Frequency Information

The above SE attention averages the time-channel-frequency feature map
$X^{T\times C\times F}$ into $X^{C}$, which may lose the discriminative
speaker information in temporal and frequency domain. Moreover, the SE
attention neglects the exploration in global context relationships on time and
frequency dimensions. To alleviate these issues, we firstly encode global time
and frequency information into the channel-wise feature and then conduct the
duality channel-wise attention with the assist of temporal and frequency
dimensions respectively. The DTCF attention module is illustrated in Fig. 3.
We average the time and frequency domains in parallel with 2D adaptive average
pooling (GAP(1D)) as Eq. 3 and Eq. 4. GAP(1D) denotes average operation on
time or frequency separately. After we concatenate $X^{C\times F}$ and
$X^{C\times T}$, a convolution operation with kernel size $(1\times 1\times
C^{\prime})$ is adopted to learn the inter-dependence relationships between
channels with the help of global time and frequency information. Here
$W^{C^{\prime}\times C}$ is the weights indicating the relationship of global
channels with aggregation of temporal and frequency dimensions.

$\displaystyle X^{C\times F}=\frac{1}{T}\sum_{i=1}^{T}X_{i},$ (3)

$\displaystyle X^{C\times T}=\frac{1}{F}\sum_{i=1}^{F}X_{j},$ (4)

$\displaystyle X_{1}^{C^{\prime}\times(T+F)}=relu(W_{1}^{C^{\prime}\times
C}\cdot[X^{C\times F},X^{C\times T}]),$ (5)

where $\cdot$ is matrix multiplication. The dimensional reduction factor
$r=\frac{C}{C^{\prime}}$ is used to avoid the exceed of parameters.

###  3.3 Duality Channel-wise Attention

After encoding the global temporal and frequency domains into the channel
attention mask learning, we split $X_{1}^{C^{\prime}\times(T+F)}$ into
$X_{2}^{C^{\prime}\times T}$ and $X_{2}^{C^{\prime}\times F}$ respectively.
Then we conduct channel-wise attention twice with the help of temporal and
frequency domain separately. The formulas of the duality channel-wise
attention are as follows:

$\displaystyle X_{2}^{C\times T}=\delta(W_{2}^{C\times C^{\prime}}\cdot
X^{C^{\prime}\times T}),$ (6)

$\displaystyle X_{2}^{C\times F}=\delta(W_{3}^{C\times C^{\prime}}\cdot
X^{C^{\prime}\times F}),$ (7)

where $\cdot$ is matrix multiplication. $W_{2}^{C\times C^{\prime}}$ and
$W_{3}^{C\times C^{\prime}}$ are channel-wise attention weights with the
convolutional operations of the kernel size of $(1\times 1\times C^{\prime})$.
$X^{C\times F}$ and $X^{C\times T}$ are the channel-wise attention masks
calculated with the help of frequency domain and temporal domain respectively.
$\delta$ is sigmoid function to normalize the attention weights into attention
masks $W_{2}^{C\times C^{\prime}}$ and $W_{3}^{C\times C^{\prime}}$. The above
proposed attention module is called as Duality Temporal-Channel-Frequency
(DTCF) attention, because it learns duality channel-wise attention with the
help of time and frequency information. As shown in Fig. 3, the attention
masks $X^{C\times F}$ and $X^{C\times T}$ dot multiply the original feature
map $X^{T\times C\times F}$ to enhance the necessary channels and suppress the
useless channels with consideration of global time and frequency information.

In a word, the proposed DTCF attention learns the global context information
of speech feature map with the consideration of both temporal and frequency
dimensions. At the same time, the DTCF attention learns the inter-dependence
channel relationship just like the SE attention mechanism [23].

Fig. 3: Duality Temporal-Channel-Frequency (DTCF) Attention Block

##  4 Experimental Setup

###  4.1 Datasets and Data Augmentation

To evaluate the performance of the DTCF attention, we conduct experiments on
two popular datasets CN-Celeb [36] and VoxCeleb [37, 38].

CN-Celeb There are 3000 speakers and 11 different genres in CN-Celeb, which
involves various real-world noise, cross-channel mismatch, and speaking styles
in the wild speech utterances. The training set has more than 600,000
utterances of 2800 speakers and 18,024 utterances of 200 speakers. There are
3,604,800 pairs in the test trials. Moreover, domain mismatch between the
enrollment and test in the trials makes this dataset a very challenging one in
speaker verification.

VoxCeleb VoxCeleb2 contains over 1 million utterances from 6,112 speakers
extracted from videos uploaded to YouTube. Short-term utterances (7.8s on
average) and diverse acoustic environments make it more challenging for SV. We
train the speaker verification models with VoxCeleb2 development part, which
has in-the-wild speech utterances from 5994 speakers. The evaluation set is
Voxceleb1, which contains over 100,000 utterances for 1,251 speakers. There
are three types of evaluation trials, which are VoxCeleb1-O, VoxCeleb1-H, and
Voxceleb1-E. The VoxCeleb1-O test set is composed of speech from 40 speakers.
The extended VoxCeleb1-E list uses the entire VoxCeleb1 set (1251 speakers),
and the hard VoxCeleb1-H list contains speakers with the same gender and
nationality.

Data Augmentation The experiments on CN-Celeb and VoxCeleb both do online data
augmentation [39] during training. The noise datasets used in data
augmentation are MUSAN [40] and RIRs [41]. Moreover, we adopt speed
perturbation and specaugment [42] during training. The speed perturbation
includes speeding up or down each utterance by 0.9 and 1.1 times respectively.
Note that online data augmentation has been successfully applied in various
speech and speaker recognition tasks [43, 44, 42].

###  4.2 Network Configurations

We validate the proposed DTCF attention module in ResNet34 [16], which only
has one-quarter of the channels in each residual block compared with the
original ResNet-34 [13] in order to reduce computational cost. We use
attention statistic pooling (ASP) [16] as the pooling layer with the
intermediate channel dimension of 128. We adopt the penultimate layer of the
network as the embedding extraction layer with 512 dimensions. The dimensional
reduction factor $r$ in attention modules is set to 8. The supervised loss of
training we use is Additive Angular Margin Softmax (AAM-Sotfmax) [35]. The
details of the model structure with approximately 9 million parameters are
shown in Table 1. In this paper, SE attention or its variants are embedded
into the same ResNet backbone except for the DTCF attention modules (DTCF-
Module).

Table 1: ResNet34-DTCF Structure

Layer | Kernel Size | Stride | Output Shape
---|---|---|---
Conv1 | $3\times 3\times 32$ | $1\times 1$ | $L\times 80\times 32$
Res1 | $3\times 3\times 32$ | $1\times 1$ | $L\times 80\times 32$
DTCF-Module | - | - | $L\times 80\times 32$
Res2 | $3\times 3\times 32$ | $1\times 1$ | $L\times 40\times 64$
DTCF-Module | - | - | $L\times 40\times 64$
Res3 | $3\times 3\times 64$ | $2\times 2$ | $L_{/2}\times 20\times 128$
DTCF-Module | - | - | ${L_{/2}\times 20\times 128}$
Res4 | $3\times 3\times 128$ | $2\times 2$ | ${L_{/4}\times 10\times 256}$
DTCF-Module | - | - | ${L_{/4}\times 10\times 256}$
Flatten | - | - | ${L_{/8}\times 2560}$
ASP | - | - | 5120
Linear | 512 | - | 512
AAMSoftmax | - | - | NumSpkrs

###  4.3 Training Details

Eighty-dimensional mel-filter bank features with 25ms window size and 10ms
window shift are extracted as model inputs. During training, the batch size of
every iteration is 300 with 2 GeForce RTX 3090. The learning rate of all model
training varies between 1e-8 and 1e-3 using the triangular2 policy [45]. The
optimizer during training is Adam optimizer [46]. The hyperparameter scale and
margin of AAM-softmax are set to 30 and 0.2 respectively. To prevent
overfitting, we apply a weight decay on all weights in the model of 2e-5.

###  4.4 Scoring Criterion

In the test phase, we use cosine similarity as the scoring criterion. The
performance metrics are equal error rate (EER) and minimum detection cost
funtion (minDCF) [47] which is evaluated with
$P_{target}=0.01,C_{miss}=C_{fa}=1$.

##  5 Experimental Results and Analysis

###  5.1 Results on CN-Celeb Dataset

The experimental results on CN-Celeb are shown in Table 2. The EER of the
baseline ResNet34 on CN-Celeb is 16.52%. After embedding the SE attention into
each residual block in ResNet34, the EER is reduced to 15.47% and the minDCF
becomes 0.6679. When we replace the SE attention with the proposed DTCF
attention module in ResNet34, the EER/minDCF further reduce 0.63%/0.0718
compared with the results of the ResNet34-SE model.

Table 2: Experimental Results on CN-Celeb (E)

Model | CN-Celeb (E)
---|---
EER (%) | minDCF
ResNet [48] | 16.52 | -
ResNet34-SE | 15.47 | 0.6679
ResNet34-DTCF | 14.84 | 0.5961

Table 3: Experimental Results on VoxCeleb

| VoxCeleb1-O | VoxCeleb1-E | VoxCeleb1-H
---|---|---|---
Methods | EER (%) | minDCF | EER (%) | minDCF | EER (%) | minDCF
Ecapa(C=1024) [9] | 0.87 | 0.1066 | 1.12 | 0.1318 | 2.12 | 0.2101
ResNet34-ft-CBAM [25] | 1.08 | 0.1655 | 1.43 | 0.1562 | 2.67 | 0.2633
ResNet34-L2-tf-GTFC [26] | 0.90 | 0.1278 | 1.22 | 0.1484 | 2.42 | 0.2497
ResNet34-SE | 1.15 | 0.1353 | 1.52 | 0.1641 | 2.83 | 0.2835
ResNet34-DTCF | 0.79 | 0.1090 | 1.13 | 0.1259 | 2.09 | 0.2082

###  5.2 Results on VoxCeleb Dataset

The experimental results on VoxCeleb are shown in Table 3. We compare our
results with Ecapa [9], ResNet34-ft-CBAM [25], ResNet34-SE [23],
ResNet34-L2-tf-GTFC [25]. The results of ResNet34-ft-CBAM, ResNet34-L2-tf-
GTFC, ResNet34-SE are reproduced with the same channels of residual blocks in
ResNet34. From Table 3, the DTCF attention module embedded in ResNet-34
outperforms Ecapa [9] on VoxCeleb1-H. The minDCF of Ecapa on VoxCeleb1-O and
EER of Ecapa on VoxCeleb1-E are better than those of our model. Compared with
ResNet34-SE which is the most classical speaker verification model structure,
ResNet34-DTCF achieves 0.36%/0.0263 reductions on EER/minDCF on VoxCeleb1-O.
The DTCF attention modules make 0.39%/0.0382 and 0.74%/0.0753 reductions on
EER/minDCF on VoxCeleb1-E and VoxCeleb1-H respectively. Moreover, our
attention mechanism is better than other recent competitive attention modules
in ResNet34.

###  5.3 Analysis

We visualize the speaker embeddings of the CN-Celeb evaluation set in Fig. 4
and Fig. 5. Specifically, we randomly select 55 speakers for analysis. One
hundred and fifty embeddings from each speaker are randomly chosen for
visualization with t-SNE [49]. Fig. 4 and Fig. 5 illustrate the embedding
distribution of ResNet34-DTCF and ResNet-SE respectively. Comparing the two
figures, we find that the embedding space of ResNet-SE is messier than that of
ResNet-DTCF. As shown in the black circle areas in two figures, the speaker
category represented by the red dots is clustered into three centers in Fig.
4, but there are many scattered points entangled with other categories in Fig.
5. This demonstrates that the speaker representation learned from ResNet-DTCF
is more discriminative than those learned from ResNet-SE.

Fig. 4: ResNet34-DTCF Embedding Space Visualization Fig. 5: ResNet34-SE
Embedding Space Visualization

The above experimental results have shown that the proposed DTCF attention
module outperforms other channel-wise attention modules both on CN-Celeb and
VoxCeleb. We believe that doing simple average pooling along time and
frequency domains may lose the necessary information for speaker
discriminative representation learning. Instead, our method preserves the time
and frequency information when learning the channel-wise attention mask.
Specifically, the method can explore the global context in the time and
frequency domains as well as make the inter-dependence channel-wise attention
mask focus on the salient regions and neglect insignificant parts, leading to
more discriminative speaker representation.

##  6 Conclusions

In this paper, we propose the Duality-Temporal-Channel-Frequency (DTCF)
attention to boost the representation extracting capability of CNN in speaker
verification. Different from other channel-wise attention learning with
average pooling on time and frequency dimensions, the DTCF attention module
firstly recalibrates the channel-wise features with aggregation of global
context on temporal and frequency dimensions, and then the duality channel-
wise attention is adopted with preservation of temporal and frequency
information respectively. Experimental results demonstrate that the proposed
DTCF attention outperforms other channel-wise attention modules.

## References

  * [1] Frédéric Bimbot, Jean-François Bonastre, Corinne Fredouille, Guillaume Gravier, Ivan Magrin-Chagnolleau, Sylvain Meignier, Teva Merlin, Javier Ortega-García, Dijana Petrovska-Delacrétaz, and Douglas A Reynolds,  “A tutorial on text-independent speaker verification,”  EURASIP Journal on Advances in Signal Processing, vol. 2004, no. 4, pp. 1–22, 2004.
  * [2] Douglas A Reynolds,  “An overview of automatic speaker recognition technology,”  in Proc. ICASSP. IEEE, 2002, vol. 4, pp. IV–4072.
  * [3] Zhongxin Bai and Xiaolei Zhang,  “Speaker recognition based on deep learning: An overview,”  Neural Networks, 2021.
  * [4] Li Wan, Quan Wang, Alan Papir, and Ignacio Lopez Moreno,  “Generalized end-to-end loss for speaker verification,”  in Proc. ICASSP. IEEE, 2018, pp. 4879–4883.
  * [5] Ying Liu, Yan Song, Ian McLoughlin, Lin Liu, and Li-rong Dai,  “An effective deep embedding learning method based on dense-residual networks for speaker verification,”  in Proc. ICASSP. IEEE, 2021, pp. 6683–6687.
  * [6] Dao Zhou, Longbiao Wang, Kong Aik Lee, Yibo Wu, Meng Liu, Jianwu Dang, and Jianguo Wei,  “Dynamic margin softmax loss for speaker verification,”  in Proc. INTERSPEECH, 2020, pp. 3800–3804.
  * [7] Tianyan Zhou, Yong Zhao, and Jian Wu,  “Resnext and res2net structures for speaker verification,”  in Spoken Language Technology Workshop (SLT). IEEE, 2021, pp. 301–307.
  * [8] David Snyder, Daniel Garcia-Romero, Gregory Sell, Daniel Povey, and Sanjeev Khudanpur,  “X-vectors: Robust dnn embeddings for speaker recognition,”  in Proc. ICASSP. IEEE, 2018, pp. 5329–5333.
  * [9] Brecht Desplanques, Jenthe Thienpondt, and Kris Demuynck,  “ECAPA-TDNN: Emphasized channel attention, propagation and aggregation in tdnn based speaker verification,”  in Proc. INTERSPEECH, 2020.
  * [10] Nikhil Ketkar,  “Convolutional neural networks,”  in Proc. Deep Learning with Python. 2017, pp. 63–78, Springer.
  * [11] Ahilan Kanagasundaram, Sridha Sridharan, Sriram Ganapathy, Prachi Singh, and Clinton Fookes,  “A study of x-vector based speaker recognition on short utterances,”  in Proc. INTERSPEECH, 2019, pp. 2943–2947.
  * [12] Mickael Rouvier, Richard Dufour, and Pierre-Michel Bousquet,  “Review of different robust x-vector extractors for speaker verification,”  in Proc. EUSIPCO, 2021, pp. 1–5.
  * [13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,  “Deep residual learning for image recognition,”  in Proc. CVPR, 2016, pp. 770–778.
  * [14] FA Rezaur rahman Chowdhury, Quan Wang, Ignacio Lopez Moreno, and Li Wan,  “Attention-based models for text-dependent speaker verification,”  in Proc. ICASSP. IEEE, 2018, pp. 5359–5363.
  * [15] Hossein Zeinali, Shuai Wang, Anna Silnova, Pavel Matějka, and Oldřich Plchot,  “BUT system description to voxceleb speaker recognition challenge 2019,”  in Proc. The VoxSRC Workshop, 2019.
  * [16] Hee Soo Heo, Bong-Jin Lee, Jaesung Huh, and Joon Son Chung,  “Clova baseline system for the voxceleb speaker recognition challenge 2020,”  in Proc. INTERSPEECH, 2020.
  * [17] Wei Yao, Shen Chen, Jiamin Cui, and Yaolin Lou,  “Multi-stream convolutional neural network with frequency selection for robust speaker verification,”  IEEEAccess, 20201.
  * [18] Jenthe Thienpondt, Brecht Desplanques, and Kris Demuynck,  “Integrating frequency translational invariance in tdnns and frequency positional information in 2d resnets to enhance speaker verification,”  Proc, Interspeech, 2021.
  * [19] Jie Hu, Li Shen, and Gang Sun,  “Squeeze-and-excitation networks,”  in Proc. CVPR, 2018, pp. 7132–7141.
  * [20] Jongchan Park, Sanghyun Woo, Joon-Young Lee, and In So Kweon,  “BAM: bottleneck attention module,”  in Proc. BMVC, 2018.
  * [21] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He,  “Non-local neural networks,”  in Proc. CVPR, 2018, pp. 7794–7803.
  * [22] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin,  “Attention is all you need,”  in Proc. NeurIPS, 2017, vol. 30.
  * [23] Jianfeng Zhou, Tao Jiang, Zheng Li, Lin Li, and Qingyang Hong,  “Deep speaker embedding extraction with channel-wise feature responses and additive supervision softmax loss function.,”  in Proc. INTERSPEECH, 2019.
  * [24] Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon,  “Cbam: Convolutional block attention module,”  in Proc. ECCV, 2018, pp. 3–19.
  * [25] Yadav Sarthak and Atul Rai,  “Frequency and temporal convolutional attention for text-independent speaker recognition,”  in Proc. ICASSP, 2020, pp. 6794–6798.
  * [26] Wei Xia and John HL Hansen,  “Speaker representation learning using global context guided channel and time-frequency transformations,”  in Proc. INTERSPEECH, 2020.
  * [27] Qibin Hou, Daquan Zhou, and Jiashi Feng,  “Coordinate attention for efficient mobile network design,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 13713–13722.
  * [28] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen,  “Mobilenetv2: Inverted residuals and linear bottlenecks,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 4510–4520.
  * [29] Daquan Zhou, Qibin Hou, Yunpeng Chen, Jiashi Feng, and Shuicheng Yan,  “Rethinking bottleneck structure for efficient mobile network design,”  in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16. Springer, 2020, pp. 680–697.
  * [30] Joon Son Chung, Jaesung Huh, and Seongkyu Mun,  “Delving into voxceleb: environment invariant speaker recognition,”  in Proc. The VoxSRC Workshop, 2019.
  * [31] Li Zhang, Qing Wang, Kong Aik Lee, Lei Xie, and Haizhou Li,  “Multi-level transfer learning from near-field to far-field speaker verification,”  in Proc. INTERSPEECH, 2021.
  * [32] Joon Son Chung, Jaesung Huh, Seongkyu Mun, Minjae Lee, Hee Soo Heo, Soyeon Choe, Chiheon Ham, Sunghwan Jung, Bong-Jin Lee, and Icksang Han,  “In defence of metric learning for speaker recognition,”  in Proc. INTERSPEECH, 2020.
  * [33] Nikita Torgashov,  “Id r&d system description to voxceleb speaker recognition challenge 2020,”  in Proc. INTERSPEECH, 2020.
  * [34] Koji Okabe, Takafumi Koshinaka, and Koichi Shinoda,  “Attentive statistics pooling for deep speaker embedding,”  in Proc. INTERSPEECH, 2018.
  * [35] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou,  “Arcface: Additive angular margin loss for deep face recognition,”  in Proc. CVPR, 2019, pp. 4690–4699.
  * [36] Yue Fan, JW Kang, LT Li, KC Li, HL Chen, ST Cheng, PY Zhang, ZY Zhou, YQ Cai, and Dong Wang,  “Cn-celeb: a challenging chinese speaker recognition dataset,”  in Proc. ICASSP. IEEE, 2020, pp. 7604–7608.
  * [37] Arsha Nagrani, Joon Son Chung, and Andrew Zisserman,  “Voxceleb: a large-scale speaker identification dataset,”  in Proc. INTERSPEECH, 2017.
  * [38] Joon Son Chung, Arsha Nagrani, and Andrew Zisserman,  “Voxceleb2: Deep speaker recognition,”  in Proc. INTERSPEECH, 2018.
  * [39] Weicheng Cai, Jinkun Chen, Jun Zhang, and Ming Li,  “On-the-fly data loader and utterance-level aggregation for speaker and language recognition,”  Transactions on Audio, Speech, and Language Processing, vol. 28, pp. 1038–1051, 2020.
  * [40] David Snyder, Guoguo Chen, and Daniel Povey,  “Musan: A music, speech, and noise corpus,”  preprint arXiv:1510.08484, 2015.
  * [41] Emanuel AP Habets,  “Room impulse response generator,”  Technische Universiteit Eindhoven, Tech. Rep, vol. 2, pp. 1, 2006\.
  * [42] Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le,  “Specaugment: A simple data augmentation method for automatic speech recognition,”  INTERSPEECH, 2019.
  * [43] Hitoshi Yamamoto, Kong Aik Lee, Koji Okabe, and Takafumi Koshinaka,  “Speaker augmentation and bandwidth extension for deep speaker embedding.,”  in Proc. INTERSPEECH, 2019.
  * [44] Tom Ko, Vijayaditya Peddinti, Daniel Povey, and Sanjeev Khudanpur,  “Audio augmentation for speech recognition,”  in Sixteenth Annual Conference of the International Speech Communication Association, 2015.
  * [45] Leslie N Smith,  “Cyclical learning rates for training neural networks,”  in Proc. WACV. IEEE, 2017, pp. 464–472.
  * [46] Diederik Kingma and Jimmy Ba,  “Adam: A method for stochastic optimization,”  in Proc. ICLR, 12 2014.
  * [47] Seyed Omid Sadjadi, Timothée Kheyrkhah, Audrey Tong, Craig S Greenberg, Douglas A Reynolds, Elliot Singer, Lisa P Mason, Jaime Hernandez-Cordero, et al.,  “The 2016 nist speaker recognition evaluation.,”  in Proc. INTERSPEECH, 2017, pp. 1353–1357.
  * [48] Lantian Li, Ruiqi Liu, Jiawen Kang, Yue Fan, Hao Cui, Yunqi Cai, Ravichander Vipperla, Thomas Fang Zheng, and Dong Wang,  “CN-Celeb: multi-genre speaker recognition,”  in Proc. ICASSP, 2020.
  * [49] Laurens Van der Maaten and Geoffrey Hinton,  “Visualizing data using t-sne,”  Journal of machine learning research, vol. 9, no. 11, 2008.