# Accelerating Representation Learning with View-Consistent Dynamics in Data-
Efficient Reinforcement Learning

Tao Huang1111Equal contribution. Jiachen Wang1111Equal contribution. Xiao
Chen2 1ShanghaiTech University
2Zhejiang University
{huangtao1<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Learning informative representations from image-based observations is of
fundamental concern in deep Reinforcement Learning (RL). However, data-
inefficiency remains a significant barrier to this objective. To overcome this
obstacle, we propose to accelerate state representation learning by enforcing
view-consistency on the dynamics. Firstly, we introduce a formalism of Multi-
view Markov Decision Process (MMDP) that incorporates multiple views of the
state into traditional MDP. Following the structure of MMDP, our method, View-
Consistent Dynamics (VCD), learns state representations by training a view-
consistent dynamics model in the latent space, where views are generated by
applying data augmentation to states. Empirical evaluation on DeepMind Control
Suite and Atari-$100$k demonstrates VCD to be the SoTA data efficient RL
algorithm on visual control tasks.

##  1 Introduction

Deep Reinforcement Learning (RL) harnesses the expressive power of deep neural
networks and the long-term reasoning ability of RL to solve sequential
decision-making problems Mnih et al. (2015). Recent years have witnessed the
sensational progress of it in various complex control tasks, such as playing
video games Hafner et al. (2020), robotic control Kalashnikov et al. (2018)
and autonomous driving Shalev-Shwartz et al. (2016).

Despite the notable success of deep RL, recent studies have revealed that
data-/sample- inefficiency severely impedes its performance when learning from
high dimensional observations Lake et al. (2016). This remains a significant
barrier to the real-world applicability of deep RL, where collecting
experiences is often costly and time-consuming Dulac-Arnold et al. (2019). For
instance, a successful RL agent requires several months to develop a decent
grasping skill, standing sharply in contrast to the human-level efficiency
Kalashnikov et al. (2018). Accordingly, elevating data efficiency is of
paramount importance for the broader progress of deep RL.

Many existing works approach this goal by augmenting deep RL with self-
supervised tasks. The motivation of that is two-fold: (i) the potentially
sparse reward signals are incapable of learning a good state representation
with limited data Yarats et al. (2020); Laskin et al. (2020a); (ii) Self-
Supervised Learning (SSL) unleashes the potential of massive unsupervised
signals for representation learning, which has achieved remarkable performance
in downstream vision and language tasks, particularly in low data regimes Chen
et al. (2020); Grill et al. (2020). Beyond that, there are proliferative
paradigms of designing SSL tasks in RL due to its interactive and temporal-
correlated training mechanism, such as maximally preserving predictive
information van den Oord et al. (2018); Lee et al. (2020b), modeling dynamics
Jaderberg et al. (2017); Schwarzer et al. (2020); Yu et al. (2021) and
discriminating features of instances at the spatial or temporal level Laskin
et al. (2020a); Stooke et al. (2021).

In this work, we propose to further accelerate representation learning in RL
by enforcing view-consistency on the learnt dynamics model within a latent
space. We posit that learning a view-consistent dynamics can encode
transition-relevant information into the state representations, thus enabling
an RL agent to efficiently exploit the environment dynamics into its decision-
making process. We formalize the above idea with a new decision-making
framework called Multi-view Markov decision process (MMDP), which takes the
view space of the state into consideration. With some realistic assumptions,
we point out that the transition dynamics holds the same dynamics
characteristics over multiple views given a state.

Following the structure of MMDP, our method, View-Consistent Dynamics (VCD),
learns state representations by training a view-consistent dynamics model
within a latent space, where views are generated by applying data augmentation
to the same underlying states. Composing it with RL objective teases out the
final objective in the whole course of policy learning. We demonstrate our
framework in Figure 1.

Figure 1: RL policy learning with VCD: (a) Overview of our framework.
Different views are generated from a given underlying state and encoded to
view representations. Both views are utilized in our auxiliary task for
representation learning. (b) Illustration of our auxiliary objective. The
view-consistency objective is calculated between the representations of two
views predicted by the dynamics module. And the prediction objective is
realized by enforcing the predicted next-state representations towards the
ground-truth state representation.

We evaluate VCD on a series of pixel-based control tasks from the DeepMind
control suite Tassa et al. (2018) to Atari games Bellemare et al. (2013). The
empirical evaluation shows that our VCD agents outperform prior state-of-the-
art baselines on different evaluation metrics. We also conduct extensive
ablation studies to validate the efficacy of the view-consistent dynamics.

We highlight our main contributions below:

  * •

We propose a new framework MMDP that extends a traditional MDP state to it
multiple views and present a novel but simple method VCD that exploits the
structure of MMDP to accelerate the representation learning in RL .

  * •

We demonstrate that VCD agents outperform prior state-of-the-art baselines on
the widely used pixel-based control tasks from DMControl and Atari games in
terms of both data-efficiency and asymptotic performance. We also adopt the
recently proposed rigorous deep RL evaluation metrics as our benchmarking
methodology.

  * •

Through careful ablation studies, we verify the efficacy of the view-
consistency module itself and other implementation details in incorporating
SSL to deep RL.

##  2 Related Work

Many algorithms have been proposed to improve the data efficiency of deep RL
agents that take input as high dimensional observations like images. We
classify the existing methods into three categories as follows.

#### Build world models.

The first line of work explicitly builds world models of the environment.
Representative works include PlaNet Hafner et al. (2019), Dreamer Hafner et
al. (2020) and SLAC Lee et al. (2020a) that perform planning or rollout-
sampling in the latent space through a learnt world model. On the contrary,
the dynamics model in UNREAL Jaderberg et al. (2017) are learnt to obtain
compact state representations without planning, which shares a similar idea
with our method. The main difference here is that our dynamics model is
located in the representation space, avoiding the inefficiency incurred by
pixel-level reconstruction.

#### Apply data augmentation.

The second line of works devotes to unleashing the potentials of data
augmentation. For instance, CURL Laskin et al. (2020a) learns contrastive
unsupervised representations from visual observations, achieving high data-
efficiency in DeepMind Control Suite and Atari games. The core of CURL is to
generate key-query pairs through data augmentation for a contrastive loss.
This idea is further explored in ATC Stooke et al. (2021) and ST-DIM Anand et
al. (2019) where a temporal contrast is adopted instead. We also perform an
ablation study to the contrastive loss in our method in Section 4.4. Beyond
that, Laskin et al. (2020b) observes that simply applying data augmentation on
the input observations can greatly improve the data-efficiency. Their method,
named RAD, is further extended by DrQ Yarats et al. (2020). It regularizes
model-free RL algorithms with multiple augmented states (views). However,
unlike DrQ, we use multiple views to regularize representation learning
instead of RL.

#### Design auxiliary tasks.

The third line of works leverage the recent advances in unsupervised and self-
supervised representation learning Chen et al. (2020); Grill et al. (2020) by
designing auxiliary tasks along with RL. One designing paradigm focuses on
reconstruction-based objectives, such as future prediction in UNREAL and
image-reconstruction in SAC-AE Yarats et al. (2021). Another vine of works
learns representations by discriminating features of instances at spatial or
temporal level, including CURL and ATC. Beyond that, some works propose to
harness the Markovian structure of the environment. For instance, CPC van den
Oord et al. (2018) and PI-SAC Lee et al. (2020b) maximally preserves the
predictive information in the state representations; SPR Schwarzer et al.
(2020) and PlayVirtual Yu et al. (2021) predict future state representation by
learning a latent dynamics model. Our method bears some resemblance to SPR in
learning dynamics, except that a new property, view-consistency, is being
enforced on our model.

##  3 Method

In this section, we propose our method named VCD to improve the data-
efficiency in (pixel) RL. Our key idea is to accelerate the state
representation learning by training a _view-consistent_ dynamics model in the
latent space. To achieve this, we incorporate a self-supervised task into RL
that includes two orthogonal but complementary parts: (i) train a latent
dynamics model to predict the future view representations; (ii) force the
dynamics model to be view-consistent.

We first formalize Multi-view Markov Decision Process (MMDP) and the
definition of view-consistency in Section 3.1. In Section 3.2, we then
introduce the _view-consistent_ dynamics model and show how it helps
representation learning in RL. Finally, we discuss the practical concerns of
implementing VCD in Section 3.3.

###  3.1 Multi-view Markov Decision Process

A Markov decision process (MDP) in RL is defined by a tuple
$\langle\mathcal{S},\mathcal{A},\mathcal{P},r,\mathcal{\gamma}\rangle$, where
$\mathcal{S}$ is the state space, $\mathcal{A}$ is the action space,
$\mathcal{P}$ is the transition dynamics, $r$ represents the reward function
whose element $r({\mathbf{s}}_{t},{\mathbf{a}}_{t})$ is the reward collected
by taking action ${\mathbf{a}}_{t}$ at state ${\mathbf{s}}_{t}$,
$\gamma\in[0,1)$ denotes the discount factor. Crucially, we stack $l$
consecutive image-based observations
$({\mathbf{o}}_{t-l+1},...,{\mathbf{o}}_{t})\in\mathcal{O}^{l}$ as the fully-
observed state ${\mathbf{s}}_{t}$. The agent’s objective is to find a policy
$\pi(\cdot|{\mathbf{s}}_{t})$ that maximizes the cumulative discounted return
$\mathbb{E}_{\pi}\left[\sum_{t=0}^{\infty}\gamma^{t}r({\mathbf{s}}_{t},{\mathbf{a}}_{t})\right]$.

Data augmentation is wildly adopted in RL to improve data-efficiency . These
methods share a key insight that data augmentation unleashes the potential of
the massive pixel observations (states). To further leverage its power, we
propose to model the view space $\mathcal{V}$ of the state. Specifically, the
views ${\mathbf{v}}_{t}^{i}$ are emitted from one underlying state
${\mathbf{s}}_{t}$ with the rendering function $q(\cdot|{\mathbf{s}}_{t})$.
And data augmentation like rotation, crop or translation are all specific
instances of such rendering functions. We further assume a block structure Du
et al. (2019) over the view space $\mathcal{V}$. That is, $\mathcal{V}$ can be
partitioned into disjoint blocks $\mathcal{V}_{s}$, each containing the
support of the conditional distribution $q(\cdot|{\mathbf{s}})$ and a perfect
decoding function $q^{\star}:\mathcal{V}\rightarrow\mathcal{S}$ that maps
${\mathbf{v}}_{t}^{i}$ to ${\mathbf{s}}_{t}$. This is a realistic assumption.
For instance, one can rotate an image to generate multiple views, and these
views can be rotated back to restore the original images. We illustrate the
ordinary MDP and the resulting Multi-view MDP (MMDP) in Figure 2.

The distinct structure of MMDP implies a core property that guides our
representation learning method: the transition dynamics is invariant to a
state ${\mathbf{s}}_{t}$ and its views ${\mathbf{v}}^{i}_{t}$:

$\mathcal{P}({\mathbf{s}}_{t+1}|{\mathbf{s}}_{t},{\mathbf{a}}_{t})=\mathcal{P}({\mathbf{s}}_{t+1}|{\mathbf{v}}^{i}_{t},{\mathbf{a}}_{t}),\,i\in\\{1,2\\},$

and we call it _View-Consistency_ of the transition dynamics $\mathcal{P}$. In
other words, the block structure on the view space retains all the dynamics
information among different views. In the following section, we extend this
property in the latent space to boost state representation learning.

Figure 2: The graphical model of MMDP. The state ${\mathbf{s}}_{t}$ generates
views ${\mathbf{v}}_{t}^{i}$ through the rendering function (blue arrow)
$q(\cdot|{\mathbf{s}}_{t},\upsilon_{t}^{i})$. The views ${\mathbf{v}}_{t}^{i}$
are in a block that can be decoded back to into their underlying state
${\mathbf{s}}_{t}$ through decoding function (red arrow)
$q^{\star}(\cdot|{\mathbf{v}}_{0}^{i})$.

###  3.2 How View-consistency Helps RL

We first describe the latent dynamics model. Then we discuss how to exert
view-consistent to this dyanmics model and how it helps representation
learning in RL.

#### Latent dynamics model.

Encoding predictive information into the state representation
${\mathbf{z}}^{s}_{t}\in\mathcal{Z}$ aids policy learning in RL van den Oord
et al. (2018), where ${\mathbf{z}}^{s}_{t}=f({\mathbf{s}}_{t})$ is encoded by
a feature encoder $f(\cdot)$. Some works realize this by training a latent
dyanmics model (DM) $h(\cdot,\cdot)$ that predicts the transition dynamics
$({\mathbf{z}}^{s}_{t},{\mathbf{a}}_{t})\rightarrow{\mathbf{z}}^{s}_{t+1}$ in
the representation space $\mathcal{Z}$. Following structure of MMDP, we
consider the DM that takes as input view representation
${\mathbf{z}}_{t}^{i}=f({\mathbf{v}}^{i}_{t})$ and an action
${\mathbf{a}}_{t}$, which then predicts the future view representation
${\mathbf{z}}_{t+1}=f({\mathbf{v}}_{t+1})$.111We remove the superscript here
for clarity, as ${\mathbf{z}}_{t+1}$ can denote any view representations given
ground state ${\mathbf{s}}_{t+1}$.

#### View-consistent dynamics model.

In our multi-view setting, the transition dynamics holds the view-consistency
property in the state space. It is natural to ask:

_What if a latent dynamics model holds view-consistency?_

To investigate this problem, we first provide a formal definition of the view-
consistent DM as follows:

######  Definition 1 (View-consistent dynamics model).

Given any two view representations ${\mathbf{z}}_{t}^{1}$ and
${\mathbf{z}}_{t}^{2}$ and action ${\mathbf{a}}_{t}$, a latent dynamics model
$h$ is view-consistent, if for any time step $t$,

$\begin{gathered}d\big{(}h({\mathbf{z}}_{t}^{1},{\mathbf{a}}),h({\mathbf{z}}_{t}^{2},{\mathbf{a}})\big{)}=0,\end{gathered}$
(1)

where $d$ is a distance metric in the representation space.

Essentially, a view-consistent DM holds the same dynamics characteristics over
view representations given a state. By enforcing view-consistency on the DM,
the RL agent learns an encoder that extracts transition-relevant information.
We posit that learning such a DM will accelerate representation learning in
RL. To this end, we will restrict the DM with view-consistency. Designing
details are discussed in the following section.

Figure 3: Performance evaluation on DMC-100k, DMC-500k and Atari-100k. We run
each task with 10 different random seeds. First line. Aggregate metrics,
including Mean and Interquartile-mean (IQM) score of VCD and other state-of-
the-art baselines with $95\%$ Confidence Intervals (CIs). The CIs are
estimated using the percentile bootstrap with stratified sampling. Second
Line. Performance Profile reflects the score-distribution across all runs,
which is also more robust to outliers. Shaded regions show pointwise $95\%$
condifence bands based on percentile bootstrap with stratified sampling. The
area under the performance profile corresponds to the mean.

###  3.3 Implementation Details

#### Online and target architecture.

We consider a one-step transition
$\left(({\mathbf{v}}_{t}^{1},{\mathbf{v}}_{t}^{2}),{\mathbf{a}}_{t},r_{t},{\mathbf{v}}_{t+1}\right)$
in an MMDP. Following Grill et al. (2020); Schwarzer et al. (2020), we adopt
two lines of nerworks: (i) online networks that include online encoder
$f_{o}$, online DM $h_{o}$ and online projector $g_{o}$ parameterized by
$\theta_{o}$; and (ii) their counterpart in the target line parameterized by
$\theta_{m}$. The online (target) encoder maps view ${\mathbf{v}}^{1}_{t}$
(${\mathbf{v}}^{2}_{t}$) into view representation ${\mathbf{z}}_{t}^{1}$
($\bar{{\mathbf{z}}}_{t}^{2}$), followed by an online (a target) DM that
outputs prediction of one-step state representation
$\hat{{\mathbf{x}}}_{t+1}^{1}$ ($\bar{{\mathbf{x}}}_{t+1}^{2}$), given action
${\mathbf{a}}_{t}$. Then, the online (target) projector maps predictions into
projection space as
$\hat{{\mathbf{y}}}_{t+1}^{1}=g_{o}({\mathbf{x}}_{t+1}^{1})$ and
$\bar{{\mathbf{y}}}_{t+1}^{2}=g_{m}(\bar{{\mathbf{x}}}_{t+1}^{2})$.
Introducing projection space empirically improves the overall performance.
Meanwhile, only online parameters $\theta_{o}$ are updated via gradient
descent during the training, and the target line network parameters
$\theta_{m}$ are updated with exponential moving average (EMA):

$\theta_{m}\leftarrow(1-\tau)\theta_{m}+\tau\theta_{o},$

where $\tau\in[0,1]$ is the EMA coefficient.

#### Auxiliary task.

We design an auxiliary task for self-supervised representation learning along
with policy learning. It contains two parts: a prediction loss
$\mathcal{L}_{pre}$ and a view-consistency loss $\mathcal{L}_{con}$.
Practically, we apply an prediction head $q_{pre}$ to the online projection
that gives a prediction towards the future view representation
$\tilde{{\mathbf{y}}}_{t+1}=g_{m}(f_{m}({\mathbf{v}}_{t+1}))$:

$\displaystyle\mathcal{L}_{pre}=2-2\frac{q_{pre}(\hat{{\mathbf{y}}}_{t+1}^{1})}{\|q_{pre}(\hat{{\mathbf{y}}}_{t+1}^{1})\|_{2}}\frac{\tilde{{\mathbf{y}}}_{t+1}}{\|\tilde{{\mathbf{y}}}_{t+1}\|_{2}}.$

For view-consistency loss, we apply another predictor $q_{con}$ to the
projection:

$\displaystyle\mathcal{L}_{con}=2-2\frac{q_{con}(\hat{{\mathbf{y}}}_{t+1}^{1})}{\|q_{con}(\hat{{\mathbf{y}}}_{t+1}^{1})\|_{2}}\frac{\bar{{\mathbf{y}}}_{t+1}^{2}}{\|\bar{{\mathbf{y}}}_{t+1}^{2}\|_{2}}.$

#### Training objective.

Composing the RL and auxiliary objectives gives the overall training
objective:

$\mathcal{L}_{total}=\mathcal{L}_{rl}+\underbrace{\mathcal{L}_{pre}+\lambda\mathcal{L}_{con}}_{\text{Auxiliary
Loss}},$ (2)

where $\lambda$ steers the weight of the view-consistency loss. The auxiliary
loss updates all online network parameters $\theta_{o}$. Note that any RL
algorithm (corresponds to $\mathcal{L}_{rl}$) can be the candidate in the
policy learning stage.

Figure 4: Ablation studies. (a). Aggreagte performance of VCD with different
weighting factors ($\lambda$) of view-consistency loss. (b). Efficacy of
different VCD modules, including vanilla VCD, VCD with a view-consistency-
agnostic encoder (VCD-CNE), VCD with a prediction-agnostic encoder (VCD-PNE).
(c). Performance of VCD and its variants including vanilla VCD, VCD with
contrastive loss (VCD-Cont) and VCD without projectors (VCD w/o Proj), where
diamonds are sample-means and circles are scores of individual runs. (d).
Effect of different EMA coefficients for target networks. All ablation
experiments are conducted on six tasks on DMC-100k with 5 seed runs.

##  4 Experiments

We evaluate VCD on diverse visual controls tasks. We first outline the
experiment setup, including environments and evaluation metrics in Section
4.1. Then we present the training details in Section 4.2 and compare VCD with
other state-of-the-arts in Section 4.3. Finally, we conduct ablation studies
to validate the efficacy of different VCD modules in Section 4.4.

###  4.1 Setup

#### DMControl.

First, we evaluate the performance of VCD on six continuous control tasks in
the DeepMind Control Suit (DMControl) Tassa et al. (2018), which is a widely-
adopted benchmark for measuring the data efficiency of an RL algorithm. These
tasks are of different traits and provide RL agents with image-based states.
Following previous works Laskin et al. (2020a), we measure the performance of
VCD at 100k and 500k environment steps during the training stage (referred to
as DMC-100k and DMC-500k, respectively).222The environment step is defined as
the number of environment transitions instead of learning updates (e.g., 250
learning updates with an action-repeat of $4$ corresponds to $1000$
environment steps). DMC-100k investigates the data-efficiency, while DMC-500k
evaluates the asymptotic long-horizon performance of an RL algorithm. The
score range for each task is $[0,1000]$, which we normalize to $[0,1]$.

#### Atari.

Second, we test VCD on the the Atari-100k benchmark which consists of 26
discrete image-based controls tasks. The agent is allowed to play with $100$k
environment steps or two hours of game-playing compared to the standard
benchmark of $500$k environment steps (or 39 days of experience). The
performance is measure by human-normalized score (HNS) computed by
$(s_{a}-s_{r})/(s_{h}-s_{r})$, where $s_{a}$, $s_{r}$ and $s_{h}$ denote agent
score, score of random play and score of human play, respectively.

#### Evaluation Metric.

A fair comparison for deep RL has always been a thorny problem. Point-estimate
methods like mean and median score adopted by most previous works are not
suitable in the few-run cases and are not robust to outliers (extremely high
(or low) score given a (mis-) fortunate seed). Recent work Agarwal et al.
(2021) investigates the biases in those conventional evaluation metrics.
Following their recommendation, we make three attempts here, seeking for fair
comparison of VCD and its predecessors:  (1) We run each task on both
environments with 10 different seeds;  (2) We report the Mean and
Interquartile-mean (IQM)333IQM discards the bottom and top $25\%$ of the runs
and calculates the mean score of the remaining $50\%$ runs. score with
Percentile Confidence Intervals as an aggregate metric across all tasks and
seeds;  (3) We present the Performance Profile444Performance Profile
approximates the Score Distribution $X$ as the fraction of runs above a
certain normalized score $\rho$ across $N$ task and $M$ seeds, i.e.,
$\hat{F}_{X}(\rho)=\frac{1}{M}\sum_{m=1}^{M}\frac{1}{N}\sum_{n=1}^{N}\mathds{1}\left[x_{m,n}>\rho\right]$.
with $95\%$ confidence bound to approximate the run-score distributions;  All
these metrics are calculated with the open-source library rliable.

###  4.2 Training Details

We largely follow previous training settings to avoid the potential impact of
varying network architectures or hyperparameters. Specifically, our VCD is
built on top of SAC Haarnoja et al. (2018) on DMControl. We adopt the encoder
and actor-critic networks from Laskin et al. (2020a) and set DM, projectors,
and predictors as MLPs. The views are generated from random crop on the image-
based states. For Atari, we follow Schwarzer et al. (2020) for DQN Mnih et al.
(2015) hyperparameters and network architectures, where random shift is
applied for view-generation. The main difference is adding a target DM and an
independent predictor for computing view-consistency loss. Besides, we set the
EMA coefficient $\tau$ for the target networks as $0.05$ and $0$ on DMControl
and Atari, respectively. The weight of view-consistency loss $\lambda$ is set
to $0.5$ for both environments (except for Walker_walk in DMControl). We will
provide an open-source implementation of VCD once accepted.

###  4.3 Performance Comparison

#### Comparison on DMControl.

The left two columns in Figure 3 compare VCD with other SOTA model-based and
model-free algorithms on DMControl. For data-limited regime of DMC-100k, VCD
reaches an IQM of $0.80$ and Mean of $0.75$, which is $9.5\%$ and $8.7\%$
higher than the previous best result from DrQ (an IQM of $0.73$ and Mean of
$0.69$). Also, the variance of VCD is much smaller than most previous methods,
which again indicates the benefit of VCD in improving data-efficiency.
Similarly, the performance profile of DMC-100k, which summaries the run-score
distribution of each method, shows that $75\%$ runs of VCD are above
$\rho=0.6$ normalized score, compared to $63\%$ of previous best method (DrQ).
A fraction of $53\%$ of runs are above $0.8$ normalized score, which is $12\%$
higher than its best predecessor DrQ ($41\%$).

In the asymptotic optimal regime of DMC-500k, VCD is competitive
($IQM=0.93,Mean=0.89$) against the previous best method DrQ
($IQM=0.92,Mean=0.88$) but with lower variance, meaning that VCD would also
boost and stabilize the long-term performance of RL agents. This is also
reflected in the performance profile of DMC-500k.

#### Comparison on Atari-100k

The rightmost column of Figure 3 compares VCD with the other six
representatives of influential algorithms. For HNS, VCD reaches an IQM of
$0.38$ and Mean of $0.65$, which is $13\%$ and $5,7\%$ higher than the
previous best method SPR (an IQM of $0.33$ and Mean of $0.61$). As for the
performance profile across all runs, VCD mainly gains improvements on those
mediocre tasks for RL agents (with HNS ranging from $0.2$ to $0.6$) compared
with previous best method.

###  4.4 Ablation Studies

#### View-consistency loss is critical.

In Eqn. 2, we leave the weighting of the view-consistency loss as a
hyperparameter $\lambda$ that steers the loss-balancing. From Figure 4(a), we
can see that a positive $\lambda$ introduces a certain degree of view-
consistency to the learnt dynamics and helps improve the data-efficiency
w.r.t. all evaluation metrics. Notably, the best performance is achieved with
a medium $\lambda$ at around $0.5$. Such a “concavity” indicates that too
small $\lambda$ brings weak supervision signals and overwhelmingly emphasizing
view-consistency will diminish the effect of other learning objectives.

To investigate view-consistency loss further, we also explore a multi-step
version of VCD. It predicts state representations $K$-steps into the future
given input ${\mathbf{z}}_{t}$ and a sequence of actions
$({\mathbf{a}}_{t},...,{\mathbf{a}}_{t+K-1})$. Table 1 shows that multi-step
(two-step here) prediction helps when there’s no view-consistency, while
incorporating multi-step prediction hampers the performance of VCD. It implies
that the augmentation of view-consistency already enables DM to learn
sufficiently predictive representation within only a single step. Instead, too
large $K$ may unbalance the multiple losses that incur performance
degeneration, a similar trend followed by $\lambda=0$ as well.

Table 1: Ablations on multi-step VCD. Performance evaluation for method 1 (Original VCD) and 2 (VCD without view-consistency) with different Predition Steps on DMC-100k with $5$ random seeds. |  Weight $\lambda$ |  Pred. Steps $K$ | Median | IQM | Mean
---|---|---|---|---|---
1 | $0$ | $1$ | $0.665$ | $0.527$ | $0.618$
2 |  | $2$ | $0.726$ | $\mathbf{0.736}$ | $\mathbf{0.701}$
3 |  | $3$ | $\mathbf{0.731}$ | $0.685$ | $0.650$
4 | $1$ | $1$ | $\mathbf{0.792}$ | $\mathbf{0.777}$ | $\mathbf{0.731}$
5 |  | $2$ | $0.736$ | $0.707$ | $0.726$
6 |  | $3$ | $0.679$ | $0.589$ | $0.588$

#### How does View-Consistency work?

In the training process, the gradients from view-consistency loss
$\mathcal{L}_{con}$ would simultaneously optimize the encoder and DM. A
natural question arises: how does view-consistency help the representation
learning? Is it because a view-consistent DM captures more “realistic”
dynamics in the latent space? Or is it because the encoder extracts more
predictive information to improve the state representation? To studies this,
we design another two schemes with other settings of VCD unchanged.  (1) VCD-
PNEwhere the encoder is not updated by the prediction loss
$\mathcal{L}_{pre}$.  (2) VCD-CNEwhere the encoder is not updated by the view-
consistency loss $\mathcal{L}_{con}$.

Figure 4(b) shows that both VCD and VCD-CNE outperform the base scheme (VCD
with $\lambda=0$) w.r.t. mean scores, while VCD-PNE incurs performance drop.
We also present relative gains of these schemes in Table 2. The results imply
that not only that view-consistency loss contributes to training a better DM
(with a relative gain of $+12.1\%$), but the main gain attributes to boosting
the representative power of encoder (with a relative gain of $+23.1\%$).
Meanwhile, the performance drop of VCD-PNE demonstrates that learning a
precise DM is also of significance, without which adding view-consistency may
slightly impoverish the performance (with a relative gain of $-16.8\%$). These
findings indicate that view-consistency aids the simultaneous training of DM
and encoder to achieve great performance.

Table 2: Ablations on VCD-Variants. Relative gains of mean scores of VCD and its variants on DMC-100k with $5$ random seeds. Scheme |  Optimized by $\mathcal{L}_{pre}$ |  Optimized by $\mathcal{L}_{con}$ | Relative Gain
---|---|---|---
Encoder | Dynamics | Encoder | Dynamics
VCD | ✓ | ✓ | ✓ | ✓ | $\mathbf{+23.1\%}$
VCD-CNE | ✓ | ✓ |  | ✓ | $\mathbf{+12.1\%}$
VCD-PNE |  | ✓ | ✓ | ✓ | $-16.8\%$
Base | ✓ | ✓ |  |  | $0$

#### What about constrastive loss?

We also implement a contrastive version of VCD (VCD-Cont), where the losses in
the auxiliary task are substituted with an InfoNCE loss following Laskin et
al. (2020a); Chen et al. (2020). Figure 4(c) show that VCD achieve better (or
at least comparable) performance than its contrastive counterpart VCD-Cont. We
attribute this to the bias of negative samples in the contrastive loss, as
investigated by those debiased approaches Chuang et al. (2020). Meanwhile, we
observe that VCD w/o Proj significantly outperforms original VCD, i.e.,
learning representation in a projection space is more preferable than learning
directly in the embedding space following an encoder Grill et al. (2020).

In Table 3, we also find that providing distinct predictors for the prediction
and consistency branch in VCD boosts performance. We attribute this to the
fact that the (gradient) information of different SSL tasks may inference each
other when the number of predictors becomes a bottleneck.

Table 3: Ablations on predictors. Performance of VCD with different number of Predictors on DMC-100k over 5 seeds.  | Num. of Predictors | Median | IQM | Mean
---|---|---|---|---
1 | $0$ | $0.663$ | $0.646$ | $0.641$
2 | $1$ | $0.720$ | $0.736$ | $0.713$
3 | $2$ | 0.792 | 0.777 | 0.731

#### EMA matters.

We consider EMA an important factor for the final performance due to the two-
stream network design. Figure 4(d) indicates both updating target networks too
slowly or frequently hampers model learning, while a mediocre $\tau$ around
$0.05$ yields better performance. This observation is in line with the
observations in Grill et al. (2020) that a moderate EMA factor is the most
suitable in SSL auxiliary tasks.

##  5 Conclusion

In this paper, we investigate the problem of data inefficiency for deep RL
agents from a view representation perspective. We introduce a new framework,
MMDP, to characterize the interaction between the dynamics model and view
representations. Based on that, we proposed View-Consistent Dynamics to
accelerate the representation learning in RL, which can be built as an
auxiliary task on top of any deep RL algorithms. Finally, we validate its
efficacy on diverse visual control tasks with rigorous statistical metrics.

## References

  * Agarwal et al. [2021] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare.  Deep reinforcement learning at the edge of the statistical precipice.  In NeurIPS, 2021.
  * Anand et al. [2019] Ankesh Anand, Evan Racah, Sherjil Ozair, Yoshua Bengio, Marc-Alexandre Côté, and R Devon Hjelm.  Unsupervised state representation learning in atari.  In NeurIPS, 2019.
  * Bellemare et al. [2013] Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling.  The arcade learning environment: An evaluation platform for general agents.  Journal of Artificial Intelligence Research, 2013.
  * Chen et al. [2020] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In ICML, 2020.
  * Chuang et al. [2020] Ching-Yao Chuang, Joshua Robinson, Lin Yen-Chen, Antonio Torralba, and Stefanie Jegelka.  Debiased contrastive learning.  In NeurIPS, 2020.
  * Du et al. [2019] Simon Du, Akshay Krishnamurthy, Nan Jiang, Alekh Agarwal, Miroslav Dudik, and John Langford.  Provably efficient rl with rich observations via latent state decoding.  In ICML, 2019.
  * Dulac-Arnold et al. [2019] Gabriel Dulac-Arnold, Daniel Mankowitz, and Todd Hester.  Challenges of real-world reinforcement learning.  In ICML, 2019.
  * Grill et al. [2020] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al.  Bootstrap your own latent: A new approach to self-supervised learning.  In NeurIPS, 2020.
  * Haarnoja et al. [2018] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  In ICML, 2018.
  * Hafner et al. [2019] Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson.  Learning latent dynamics for planning from pixels.  In ICML, 2019.
  * Hafner et al. [2020] Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi.  Dream to control: Learning behaviors by latent imagination.  In ICLR, 2020.
  * Jaderberg et al. [2017] Max Jaderberg, Volodymyr Mnih, Wojciech M. Czarnecki, Tom Schaul, Joel Z. Leibo, David Silver, and Koray Kavukcuoglu.  Reinforcement learning with unsupervised auxiliary tasks.  In ICLR, 2017.
  * Kalashnikov et al. [2018] Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al.  Scalable deep reinforcement learning for vision-based robotic manipulation.  In CoRL, 2018.
  * Lake et al. [2016] Brenden M. Lake, Tomer D. Ullman, Joshua B. Tenenbaum, and Samuel J. Gershman.  Building machines that learn and think like people.  Behavioral and Brain Sciences, 2016.
  * Laskin et al. [2020a] Michael Laskin, Aravind Srinivas, and Pieter Abbeel.  Curl: Contrastive unsupervised representations for reinforcement learning.  In ICML, 2020.
  * Laskin et al. [2020b] Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas.  Reinforcement learning with augmented data.  In NeurIPS, 2020.
  * Lee et al. [2020a] Alex Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine.  Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model.  In NeurIPS, 2020.
  * Lee et al. [2020b] Kuang-Huei Lee, Ian Fischer, Anthony Liu, Yijie Guo, Honglak Lee, John Canny, and Sergio Guadarrama.  Predictive information accelerates learning in rl.  In NeurIPS, 2020.
  * Mnih et al. [2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charlie Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis.  Human-level control through deep reinforcement learning.  Nature, 2015.
  * Schwarzer et al. [2020] Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman.  Data-efficient reinforcement learning with self-predictive representations.  In ICLR, 2020.
  * Shalev-Shwartz et al. [2016] Shai Shalev-Shwartz, Shaked Shammah, and Amnon Shashua.  Safe, multi-agent, reinforcement learning for autonomous driving.  ArXiv, 2016.
  * Stooke et al. [2021] Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin.  Decoupling representation learning from reinforcement learning.  In ICML, 2021.
  * Tassa et al. [2018] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al.  Deepmind control suite.  ArXiv, preprint, 2018.
  * van den Oord et al. [2018] Aäron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  ArXiv, preprint, 2018.
  * Yarats et al. [2020] Denis Yarats, Ilya Kostrikov, and Rob Fergus.  Image augmentation is all you need: Regularizing deep reinforcement learning from pixels.  In ICLR, 2020.
  * Yarats et al. [2021] Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus.  Improving sample efficiency in model-free reinforcement learning from images.  In AAAI, 2021.
  * Yu et al. [2021] Tao Yu, Cuiling Lan, Wenjun Zeng, Mingxiao Feng, and Zhibo Chen.  Playvirtual: Augmenting cycle-consistent virtual trajectories for reinforcement learning.  In NeurIPS, 2021.


Extended Dynamical Mean Field Theory
for Correlated Electron Models

Haoyu Hu1,2, Lei Chen1, Qimiao Si1

1Department of Physics and Astronomy, Rice Center for Quantum Materials, Rice
University, Houston, TX 77005, USA

2 Donostia International Physics Center, P. Manuel de Lardizabal 4, 20018
Donostia-San Sebastian, Spain

An overarching question in strongly correlated electron systems is how the
landscape of quantum phases emerges from electron correlations. The method of
extended dynamical mean field theory (EDMFT) has been developed for clean
lattice models of the correlated electrons. For such models, not only onsite
Hubbard-like interactions are important, but so are intersite interactions.
Importantly, the EDMFT method treats the interplay between the onsite and
intersite interactions dynamically. It was initially formulated for models of
the two-band Anderson-lattice type with intersite interactions, as well as for
the one-band Hubbard type with intersite Heisenberg-like terms that are often
called Hubbard-Heisenberg models. In the case of Kondo lattice models, the
EDMFT method incorporates a dynamical competition between the local Kondo and
intersite Ruderman-Kittel-Kasuya-Yosida (RKKY) interactions. In these models,
the EDMFT-based analyses led to the notion of Kondo destruction, which has
played a central role in the understanding of quantum critical heavy fermion
metals. In this article, we summarize the EDMFT method, and survey its
applications, particularly for Kondo/Anderson lattice models. We also discuss
the prospect for further developing the EDMFT method, as well as for applying
it to address the correlation physics in a variety of new settings. Among the
latter are the orbital-selective Mott physics that arises both in iron-based
superconductors and in frustrated bulk systems with topological flat bands.

##  I Introduction

A wide variety of bad metals form the bedrock of strongly correlated electron
systems. Often, a bad metal is operationally defined in terms of the
electrical resistivity at room temperature reaches or exceeds the Mott-Ioffe-
Regel limit Pas21.1 ; Hus04.1 . This limit is defined in terms of the mean
free path $\ell$, corresponding to $k_{F}\ell/2\pi\approx 1$ for each Fermi
surface, which signifies a proximity to a Mott localized state Hus04.1 . A
complementary criterion is based on a sizeable reduction of the optical Drude
weight from its non-interacting-electron expectation Dre02.1 ; Qaz09.1 ;
Si16.1 . Heavy fermion metals and metals that lie in the vicinity of a Mott
insulator exemplify bad metals.

Quantum many-electron systems are traditionally studied in terms of a
perturbative expansion of the interaction Nozieres18 . The assumption is that
the noninteracting electrons are the building blocks for low-energy physics. A
standard model is the one-band Hubbard model, which contains an electron band
of width $D$ and an onsite interaction – the Hubbard interaction – of strength
$U$. The Fermi liquid description applies in an order-by-order perturbative
treatment of $U/D$ to infinite orders. Here, coherent quasiparticles appear as
a pole in the single-particle Green’s function $G({\bf k},\omega)$. When $U/D$
becomes of order unity, non-perturbative effects may develop. The most famous
case is the development of a Mott insulator for a half-filled band Imada1998 ,
in which the electron correlations drive the quasiparticle weight to zero.
Concurrently, a gap develops in the single-particle and charge excitation
spectrum; the ground state is no longer a metal and is no longer adiabatically
connected to its noninteracting counterpart.

In general, when the electron correlations reach and exceed the electron
bandwidth, as is the case in bad metals, other degrees of freedom emerge as a
part of the building blocks for the low-energy physics. A typical case
involves spins, as seen from the notion of local moment formation of a
correlated and deep orbital in a metallic matrix Anderson1961 . As a result,
models of strongly correlated electrons may involve both local moments and
itinerant electrons, as exemplified by heavy fermion metals Pas21.1 ;
KirchnerRMP ; StewartRMP ; Coleman-Nature ; Si_Science10 . A similar set of
building blocks develop once a Mott insulator Imada1998  is coupled to a
metallic band.

When expressed in terms of the electron degrees of freedom, the correlation
physics describes the effect of Coulomb repulsion between the electrons
GeorgesRMP ; Vollhardt2019 . However, if represented in terms of the emergent
building blocks such as local moments, the competition between different types
of effective interactions becomes particularly explicit. A case in point is a
Kondo lattice Hamiltonian, which contains an antiferromagnetic spin-exchange
(Kondo) interaction between the local moments and spins of the conduction
electrons on the one hand, and the RKKY interaction between the local moments
on the other.

The EDMFT method was introduced to dynamically study this type of competition
Si.96 ; SmithSi-edmft ; Chitra2001 . Consider the case of the Kondo lattice
model. Here, the local Kondo interaction favors the formation of a Kondo
singlet between the local moments and spins of the conduction electrons.
Whereas the RKKY interaction, which often is also antiferromagnetic, promotes
singlet formation among the local moments. To properly account for the
amplified quantum fluctuations that develop when the two types of effects
would produce comparable energy scales, the dynamical interplay between the
two interactions is crucial; this interplay is captured by the EDMFT method.
This is to be contrasted with the standard dynamical mean field theory
GeorgesRMP ; Vollhardt2019 , in which the local interactions such as the Kondo
couplings are treated in a dynamical way, while the intersite interactions
such as the RKKY couplings are handled in terms of a static Hartree-Fock
approximation.

The EDMFT method has played a central role in the elucidation of quantum phase
transitions of heavy fermion metals Hu-QCM-review2022 . Studies based on such
a method have led to the advancement of the concept of Kondo destruction Si-
Nature ; Si2003 ; Si1999 ; Colemanetal ; senthil2004a . Qualitatively, the
Kondo physics underlies the formation of (heavy) quasiparticles in such
systems. The quasiparticles develop through the formation of the Kondo
singlet, and appear in the form of a Kondo-driven composite fermion. These
processes are captured by the local self-energy of the conduction electrons.
As such, studying the dynamical competition between the RKKY and Kondo
interactions provides a way of describing the reduction – and eventual
destruction – of the Kondo singlet in the ground state and, by extension, the
quasiparticles. In other words, the EDMFT approach is ideally suited to access
how (heavy) quasiparticles are lost.

The concept of Kondo destruction has influenced the development of quantum
critical metals in a profound way, particularly on the destruction of
quasiparticles and dynamical Planckian ($\hbar\omega/k_{\rm B}T$) scaling at
the quantum critical point (QCP), and a sudden jump of a “large” to “small”
Fermi surface across the QCP Hu-QCM-review2022 .

The present article is devoted to three topics:

  * •

We describe the dynamical equations associated with the EDMFT method.

  * •

We illustrate the application of the EDMFT approach by focusing on the Kondo
lattice models and summarizing i) the methodological aspect of the approach
and ii) the results as pertaining to the heavy fermion quantum criticality.
Further analyses that have been motivated by the result of such calculations
are touched upon, especially in the form of a global phase diagram.

  * •

The prospect for further progress along this general direction is discussed.

##  II EDMFT Approach

We illustrate the EDMFT approach in terms of the single-band Hubbard-
Heisenberg-type model Si.96 ; SmithSi-edmft ; Chitra2001 ; ZhuGrempelSi :

$\displaystyle
H_{\text{U-v}}=\sum_{i}Un_{i\uparrow}n_{i\downarrow}+\sum_{\langle
ij\rangle,\sigma}t_{ij}c_{i\sigma}^{\dagger}c_{j\sigma}+{1\over
2}\sum_{\langle ij\rangle}\,v_{ij}\,:n_{i}::n_{j}:$ (1)

The first two terms specify the Hubbard model for a spin-$\frac{1}{2}$ band.
The onsite Hubbard interaction is $U$ and the hopping matrix is $t_{ij}$,
whose Fourier transform corresponds to the band dispersion $\varepsilon_{\bf
k}$. The third term describes an intersite density-density ($v_{ij}$)
interaction (a spin-exchange interaction, $J_{ij}$, can also be added, as
originally done; see below), with $n_{i}$ being the density operator of the
$c-$electrons and $:n:\equiv n-\langle n\rangle$ representing its normal-
ordered form. For concreteness, we limit the intersite interactions to the
nearest-neighbor ($\langle ij\rangle$) contributions, but this can be readily
generalized.

Figure 1:  The single-site, two-site, and three-site diagrams for the
Luttinger Ward potential in the extended DMFT SmithSi-edmft ; Si2003 . A
shaded circle contains all the on-site diagrams fully-dressed by the fermion
Green’s function (solid lines) as shown in the first diagram. The dashed lines
further represent the intersite interactions; we have denoted them by
$J_{ij}$, but they could also be interactions in other channels, such as
$v_{ij}$ of Eq. (1).

The EDMFT approach amounts to the summation of an infinite series of diagrams
as outlined in Fig. 1. The approach is systematic and conserving. Moreover,
the EDMFT equations are generated by an effective action function of the Baym-
Kadanoff type.

The EDMFT approach incorporates a local self-energy, $\Sigma$, for the single-
electron Green’s function $G$ and a related irreducible quantity in the
density channel, $M$, which is defined in terms of a cumulant that is
$v_{ij}$-irreducible Si.96 ; for notational convenience, it has been referred
to as a (density) self-energy. The self-energies determine the dynamical
density susceptibility and single-particle Green’s function as follows:

$\displaystyle\chi({\bf q},\omega)$ $\displaystyle=$
$\displaystyle\frac{1}{M(\omega)+v_{{\bf q}}}\,,$ (2)

together with

$\displaystyle G({\bf k},\epsilon)$ $\displaystyle=$
$\displaystyle\frac{1}{\epsilon+\mu-\epsilon_{\bf k}-\Sigma(\epsilon)}.$ (3)

These self-energies can be calculated from a local action, which can
equivalently be expressed in terms of a local Hamiltonian:

$\displaystyle{\cal H}_{\text{loc,U-v}}=Un_{\uparrow}n_{\downarrow}+\sum_{{\bf
k},\sigma}E_{p}~{}c_{{\bf k}\sigma}^{\dagger}~{}c_{{\bf
k}\sigma}+\;g\sum_{{\bf p}}:n:\left(\phi_{{\bf p}}+\phi_{-{\bf
p}}^{\;\dagger}\right)$ (4)

Here the dispersion of the bosonic bath, along with its fermionic bath
counterpart, are self-consistently determined. The set of nonlinear equations
can be expressed as follows. The bath dispersion defines Weiss fields,
$\chi_{0}$ and $G_{0}$, as follows:

$\displaystyle g^{2}\sum_{{\bf p}}{2w_{{\bf p}}\over{(i\nu_{m})^{2}-w_{{\bf
p}}^{2}}}$ $\displaystyle=$ $\displaystyle-\chi_{0}^{-1}(i\nu_{m}),$
$\displaystyle\sum_{{\bf k}}{1\over{i\omega_{n}-E_{\bf k}}}$ $\displaystyle=$
$\displaystyle G_{0}(i\omega_{n}),$

where $\nu_{m}$ and $\omega_{n}$ are bosonic and fermionic Matsubara
frequencies, respectively. The Dyson equations for the density and electron
self-energies take the following forms:

$\displaystyle M(\omega)=\chi_{0}^{-1}(\omega)+1/\chi_{\text{loc}}(\omega)\,,$
(6)

and, as usual,

$\displaystyle\Sigma(\omega)=G_{0}^{-1}(\omega)-1/G_{\text{loc}}(\omega)\,.$
(7)

The self-consistency equations can then be simply written to capture the
translational invariance:

$\displaystyle\chi_{\text{loc}}(\omega)$ $\displaystyle=$
$\displaystyle\sum_{\bf q}\chi({\bf q},\omega),$ $\displaystyle
G_{\text{loc}}(\omega)$ $\displaystyle=$ $\displaystyle\sum_{\bf k}G({\bf
k},\omega)\;.$

It can be inferred from Eq. 2 that, for the intersite interactions $v_{ij}$ or
$I_{ij}$ (for later reference), the following quantity,

$\displaystyle\rho_{\rm I}(x)=\sum_{\bf q}\delta(x-I_{\bf q})\,.$ (9)

comes into the self-consistency equation. In practice two particular forms
have been used. In one form, $\rho_{I}(x)$ has a constant onset at the lower
edge of its support. In another, it has a square-root onset.

We stress that the dynamical susceptibility is defined in terms of connected
correlation functions. As such, the density operator appears in the equations
in the normal-ordered form. This point is also important when states with a
conventional long-range order (such as charge or spin orders) are considered.
We will write down the corresponding equations below, in the context of
Anderson/Kondo lattice models. The importance of this normal ordering in the
EDMFT formulation was stressed from the beginning Si.96 ; SmithSi-edmft ;
ZhuGrempelSi . When the normal ordering is not taken into account, the ordered
state would yield an action that (unphysically) scales as $\beta^{2}$
ChowdhuryRMP .

##  III Kondo lattice model: EDMFT equations and solution methods

The Kondo lattice model represents the limit of the Anderson lattice model
when the $f$-electron level lies sufficiently below the Fermi energy, and when
the interaction $U$ for the $f$-electrons is sufficiently large Hewson . It
features two competing interactions. The Kondo coupling/hybridization favors
the formation of a Kondo singlet. Whereas the RKKY interaction leads to the
development of quantum magnetism; the typical case corresponds to
antiferromagnetic (AF) RKKY interactions, which promotes the formation of
inter-moment spin singlets.

The vantage point of the Kondo effect provides a particularly clear physical
picture. It is the development of the Kondo singlet in the ground state that
supports heavy quasiparticles in the excitation spectrum. The dynamical
competition brought on by the RKKY interactions is detrimental to the
quasiparticles. This has led to the notion of Kondo destruction Pas21.1 ;
StewartRMP ; Coleman-Nature ; Si_Science10 ; KirchnerRMP ; Si-Nature , which
has played a central role in the elucidation of quantum critical heavy fermion
systems.

In this section, we summarize the basic equations of the Kondo lattice model
within the EDMFT approach, and discuss the corresponding solutions for the
phases and quantum phase transitions.

###  III.1 Kondo lattice Hamiltonian

The Kondo lattice Hamiltonian takes the following form:

$\displaystyle H_{\rm
KL}=\sum_{ij}t_{ij}c^{\dagger}_{i\sigma}c^{\phantom{\dagger}}_{j\sigma}+\sum_{ij}I_{ij}{\bf
S}_{i}\cdot{\bf S}_{j}+\sum_{i}J_{K}{\bf S}_{i}\cdot
c^{\dagger}_{i}\frac{\vec{\sigma}}{2}c^{\phantom{\dagger}}_{i}.$ (10)

It contains a conduction-electron band, $c_{i\sigma}$, with hopping matrix
$t_{ij}$, and, correspondingly, band dispersion
$\varepsilon^{\phantom{\dagger}}_{\bf k}$. At each site $i$, the spin of the
conduction electrons, ${\bf s}_{c,i}=(1/2)c_{i}^{\dagger}\vec{\sigma}c_{i}$,
is coupled to the spin of the local moment, $\bf{S}_{i}$, by a Kondo
interaction, namely an AF Kondo exchange interaction $J_{K}$. Here
$\vec{\sigma}$ represents the Pauli matrices. Meanwhile, the local moments are
coupled to each other via an RKKY interaction $I_{ij}$.

###  III.2 EDMFT equations

For the Kondo lattice Hamiltonian, the EDMFT approach calculates local
correlation function in terms of the following local action Si-Nature ; Si2003
; ZhuGrempelSi ; Glossop07 ; Zhu07 :

$\displaystyle{\cal S}_{\text{loc}}$ $\displaystyle=$ $\displaystyle{\cal
S}_{\text{top}}+\int_{0}^{\beta}d\tau~{}J_{K}~{}{\bf S}\cdot{\bf s}_{c}$ (11)
$\displaystyle-\int_{0}^{\beta}d\tau\int_{0}^{\beta}d\tau^{\prime}\left[\sum_{\sigma}c_{\sigma}^{\dagger}(\tau)G_{0}^{-1}(\tau-\tau^{\prime})c_{\sigma}(\tau^{\prime})\right.$
$\displaystyle\left.+\;{1\over 2}:{\bf
S}:(\tau)\cdot\chi_{0}^{-1}(\tau-\tau^{\prime}):{\bf
S}:(\tau^{\prime})\rule[-8.61108pt]{0.0pt}{17.22217pt}\right]+\int_{0}^{\beta}d\tau
h_{\text{loc}}\,S^{z}\,.$

Here, $\beta=1/k_{B}T$, ${\cal S}_{\text{top}}$ describes the Berry phase of
the local moment, while $G_{0}^{-1}$ and $\chi_{0}^{-1}$ are the dynamical
Weiss fields determined self-consistently by Eq. LABEL:self-consistent.
Finally, $h_{loc}$ is the static Weiss field determined by an additional self-
consistent equation:

$\displaystyle h_{loc}=-I_{\textbf{Q}}\langle S^{z}\rangle\,.$ (12)

The retarded interactions of the spin fields have been written with normal
ordering. Equivalently, it could be expressed without using normal ordering:

$\displaystyle-\int_{0}^{\beta}d\tau\int_{0}^{\beta}d\tau^{\prime}\frac{1}{2}:{\bf
S}:(\tau)\cdot\chi_{0}^{-1}(\tau-\tau^{\prime}):{\bf S}:(\tau^{\prime})$ (13)
$\displaystyle=$ $\displaystyle-\frac{1}{2}{\bf
S}(\tau)\cdot\chi_{0}^{-1}(\tau-\tau^{\prime}){\bf
S}(\tau^{\prime})+\int_{0}^{\beta}d\tau\tilde{h}_{loc}S^{z}(\tau)+const$

$\tilde{h}_{loc}=\int_{0}^{\beta}\chi_{0}^{-1}(\tau)m^{z}d\tau$ is the
effective magnetic field from normal ordering and $m^{z}=\langle
S^{z}\rangle$. This action is equivalent to the following effective local
Hamiltonian:

$\displaystyle H_{BFK}$ $\displaystyle=$ $\displaystyle\sum_{{\bf k}}E_{{\bf
k}}c^{\dagger}_{{\bf k}\sigma}c^{\phantom{\dagger}}_{{\bf k}\sigma}+\sum_{{\bf
p}}\omega_{{\bf p}}\bm{\phi}_{-{\bf p}}^{{\dagger}}\bm{\phi}_{{\bf p}}$ (14)
$\displaystyle+J_{K}\bm{S}\cdot\frac{c_{0}^{\dagger}\bm{\sigma}c_{0}}{2}+g:\bm{S}:\cdot\sum_{{\bf
p}}(\bm{\phi}_{{\bf p}}^{{\dagger}}+\bm{\phi}_{-{\bf
p}}^{{\dagger}})+h_{loc}S^{z}$

where $E_{{\bf k}}$ and $\omega_{\bf p}$ characterize the dispersion of the
fermionic fields ($c_{{\bf k},\sigma}$) and bosonic fields ($\bm{\phi}_{{\bf
p}}$), and $J_{K}$ ($g$) is the coupling strength between the impurity and
fermionic fields (bosonic fields). The relation between the dispersions and
Weiss fields are given in Eq. LABEL:eq:weiss_defs.

###  III.3 Variants of the Kondo lattice model

The Kondo lattice Hamiltonian, Eq. 10, has an SU(2) symmetry. In heavy-fermion
systems, the spin-orbit coupling can reduce the spin symmetry down to either
easy plane (xy-anisotropic) or easy axis (Ising-anisotropic). This can be
treated by making the interactions to depend on the spin component
$\alpha=x,y,z$. Because the flow towards the Kondo fixed point in systems with
anisotropic Kondo interactions restores the SU(2) symmetry, in practice it
suffices to incorporate the spin-anisotropy in the RKKY interaction, namely to
replace the term $\sum_{ij,\alpha=x,y,z}I_{ij}{\bf S}_{i}\cdot{\bf S}_{j}$ by
$\sum_{ij}I_{ij}^{\alpha}S^{\alpha}_{i}\,S^{\alpha}_{j}$. Correspondingly, in
the Bose-Fermi Kondo model, both the dispersion of the bosonic bath and the
Bose-Kondo coupling $g$ depends on the component $\alpha=x,y,z$.

For large but finite values of the Hubbard interaction $U$ and $f$-orbital-
energy level separation from the Fermi level, the $f$ electrons behave as
correlated electrons instead of a pure quantum spin. The system in this case
is described by an Anderson lattice model with explicit RKKY interactions:

$\displaystyle H_{\rm
AL}=\sum_{ij}t_{ij}c^{\dagger}_{i\sigma}c^{\phantom{\dagger}}_{j\sigma}+\sum_{ij}I_{ij}{\bf
S}_{i}\cdot{\bf
S}_{j}+\sum_{i\sigma}V(f_{i\sigma}^{\dagger}c_{i\sigma}+c_{i\sigma}^{\dagger}f_{i\sigma})+\sum_{i}(Un_{i\uparrow}^{f}n_{i\downarrow}^{f}+\epsilon_{d}n_{i}^{f})\,.$
(15)

Here, $f_{\sigma}^{\dagger}$ creates a correlated electron with spin $\sigma$,
$n_{i\sigma}^{f}=f_{i\sigma}^{\dagger}f_{i\sigma}$ and ${\bf
S}_{i}=\frac{f^{\dagger}_{i}\bm{\sigma}f_{i}}{2}$ are the density operator and
spin operators of the $f$ electrons, $\epsilon_{d}$ is the local energy scale,
and $V$ is the hybridization strength. Via EDMFT, the local correlators of the
Anderson lattice model are determined by a Bose-Fermi Anderson model with the
following action

$\displaystyle S_{\text{loc,A}}=$
$\displaystyle-\int_{0}^{\beta}d\tau\int_{0}^{\beta}d\tau^{\prime}\sum_{\sigma}\left[c_{\sigma}^{\dagger}(\tau)G_{0}^{-1}(\tau-\tau^{\prime})c_{\sigma}(\tau^{\prime})+f_{\sigma}^{\dagger}(\tau)(\partial_{\tau}-\epsilon_{d})f_{\sigma}(\tau)\right]$
(16)
$\displaystyle+\int_{0}^{\beta}d\tau\left[Un_{\uparrow}^{f}(\tau)n_{\downarrow}^{f}(\tau)+V\sum_{\sigma}(c_{\sigma}^{\dagger}(\tau)f_{\sigma}(\tau)+\text{h.c.})+h_{\text{loc}}\,S^{z}(\tau)\right]$
$\displaystyle-\frac{1}{2}\int_{0}^{\beta}d\tau\int_{0}^{\beta}d\tau^{\prime}:{\bf
S}:(\tau)\cdot\chi_{0}^{-1}(\tau-\tau^{\prime}):{\bf S}:(\tau^{\prime})$

The corresponding Hamitltonian reads

$\displaystyle H_{BFA}$ $\displaystyle=$ $\displaystyle\sum_{{\bf k}}E_{{\bf
k}}c^{\dagger}_{{\bf k}\sigma}c^{\phantom{\dagger}}_{{\bf k}\sigma}+\sum_{{\bf
p}}\omega_{{\bf p}}\bm{\phi}_{-{\bf p}}^{{\dagger}}\bm{\phi}_{{\bf
p}}+h_{loc}S^{z}+Un_{\uparrow}^{f}n_{\downarrow}^{f}+\epsilon_{d}(n_{\uparrow}^{f}+n_{\downarrow}^{f})$
(17) $\displaystyle+V\sum_{{\bf k},\sigma}(c_{{\bf
k}\sigma}^{\dagger}f_{\sigma}+\text{h.c.})+g:\bm{S}:\cdot\sum_{{\bf
p}}(\bm{\phi}_{{\bf p}}^{{\dagger}}+\bm{\phi}_{-{\bf p}}^{{\dagger}})$

where the relation between bath dispersions and bath functions are given in
Eq. LABEL:eq:weiss_defs.

###  III.4 Bose-Fermi Kondo model

Figure 2:  The Kondo fixed point and its destruction of the Bose-Fermi Kondo
model, described in terms of an RG flow based on an expansion in $\epsilon$
Si2003  (that is taken to be $>0$).

The Bose-Fermi Kondo model contains a Kondo fixed point and, for a suitable
form of the bosonic-bath spectrum, its destruciton. This was first seen in an
RG analysis based on an $\epsilon-$expansion in the Ising-anisotropic Bose-
Fermi Kondo model Si.96 , which was subsequently generalized to the
SU(2)-symmetric version of the model SmithSi_EPL1999 ; Sengupta . Here,
$\epsilon$ is defined in terms of the power-law spectrum of the bosonic bath:

$\displaystyle\rho_{b}(\omega)\equiv\sum_{{\bf p}}\delta(\omega-w_{\bf
p})\propto|\omega|^{1-\epsilon}~{}~{}~{}~{}~{}~{}~{}~{}\text{for}~{}|\omega|<\Lambda\,.$
(18)

When $\epsilon=0$, the bosnic bath has an ohmic spectrum. For $\epsilon>0$,
the spectrum is sub-ohmic. The RG analysis was extended to two and higher
loops ZhuSi ; ZarandDemler .

To one loop, the RG equations are

$\displaystyle\beta(J_{K})=J_{K}(J_{K}-g^{2})\quad,\quad\beta(g)=g(\epsilon/2-g^{2})\,.$
(19)

They yield a Kondo destruction (KD) quantum critical point as shown in Fig. 2.
The KD quantum critical point describes a quantum phase transition between a
Kondo-screened phase with $J_{K}\neq 0$ and a Kondo-destroyed phase with
$J_{K}=0$.

The Kondo fixed point features a spin-singlet ground state. An infrared Kondo
energy scale is nonzero, which makes the local spin susceptibility to take a
Pauli form. As the system approaches the separatrix of the RG flow, which is
controlled by the critical KD fixed point, this infrared Kondo energy scale
collapses to zero, and the local spin susceptibility diverges in the zero-
frequency limit at $T=0$. On the other side of the separatrix, the RG flow is
towards a fixed point with zero Kondo coupling. Nonetheless, dynamical Kondo
effect still takes place here when the bare Kondo coupling is nonzero.

In the SU(2) BFKM, a very recent study has analytically treated the model
based on a large-$S$ expansion Hu22.fix , where $S$ is the spin size. This
analysis finds a slew of new fixed points for small $\epsilon$, beyond the
ones that were identified in the aforementioned $\epsilon$-expansion RG
method. With increasing $\epsilon$, the analysis identifies a sequence of
fixed-point annihilation, which are consistent with – and extends – the
results of quantum Monte Carlo calculations Cai-CTQMC19.1 .

###  III.5 Solution methods

To analyze the self-consistent equations of the EDMFT approach, one needs to
solve the Bose-Fermi Kondo/Anderson models. We have already discussed the
$\epsilon$-expansion RG method. This analytical approach is accompanied by
several numerical methods.

One is the numerical renormalization group (NRG) method Glossop.05 ; Glossop07
. Within the framework of NRG, we first preform a logarithmic discretization
of both the fermionic bath and bosoinc bath and map both baths into semi-
infinite chains where the local degrees of freedom are coupled to the first
site. Subsequently, the model is solved by diagonalizing the Hamiltonian
iteratively. In each step, we add new sites to both chains.

Another is the continuous-time Monte Carlo method. Both the SU(2)-symmetric
Anderson model Cai-CTQMC19.1  and the Anderson model in the Ising limit
pixley2011 ; pixley2013 . can be solved via the quantum Monte Carlo method.
For the model in the Ising limit, we first perform the unitary (Firsov-Lang)
transformation pixley2011 ; pixley2013 ; Werner-2010
$f\rightarrow\tilde{f}_{\sigma}=f_{\sigma}e^{-g/2\sigma\Phi^{z}}$, where
$\Phi^{z}$ is a linear function of the bosonic fields corresponding to the
bosonic bath. Then we can solve the model by directly expanding the
hybridization term
$V(c_{\sigma}^{\dagger}\tilde{f}_{\sigma}+\tilde{f}^{\dagger}_{\sigma}c_{\sigma})$
and sampling the configuration via the Monte Carlo method. As for the
SU(2)-symmetric Anderson model, besides expanding the hybridization term, we
also need to expand the perpendicular coupling Cai-CTQMC19.1 ; Otsuki-2013
between spin impurity and bosonic fields $gS^{x/y}\phi^{x/y}$.

##  IV Kondo lattice model: Kondo destruction

###  IV.1 Phase diagram

The EDMFT analysis led to a continuous zero-temperature transition between a
Kondo-driven paramagnetic phase and an AF order, as illustrated in Fig. 3(a),
for an Ising-anisotropic Kondo lattice Zhu07 . A $E_{\text{loc}}^{\ast}$ scale
is defined in terms of the local responses of the effective Bose-Fermi Kondo
model. Qualitatively, a vanishing $E_{\text{loc}}^{\ast}$ means that the Bose-
Fermi Kondo model has reached the critical Kondo-destruction fixed point – the
red fixed point shown in the RG flow of Fig. 2.

a b

Figure 3:  (a) The phase diagram for the Ising-anisotropic Kondo lattice model
as determined by the EDMFT approach. The inverse static AF susceptibility as a
function of $I/T_{K}^{0}$ from the paramagnetic (trial $h_{loc}=0$) and AF
(trial $h_{loc}\neq 0$) solutions Zhu2007 . (b) The staggered magnetization
(solid circle) and the Kondo-destruction energy scale $E_{loc}^{*}$ (open
diamonds) as a function of $I/T_{K}^{0}$ Zhu2003PRL .

###  IV.2 Kondo destruction

For the Ising-anisotropic Kondo lattice model, the scale
$E_{\text{loc}}^{\ast}$ as a function of the tuning parameter $\delta$,
defined as the ratio of the RKKY interaction ($I$) to the bare Kondo scale
($T_{\text{K}}^{0}$) is shown in Fig. 3(b) GrempelSi ; ZhuGrempelSi . The
scale $E_{\text{loc}}^{\ast}$ is found to vanish at the QCP, signifying that
the QCP is of the Kondo-destruction type. This feature has recently been shown
to develop in the SU(2) Kondo lattice model Hu20.1x .

Kondo destruction leads to the development of singular spin dynamics with a
fractional exponent at the QCP. This is illustrated in Fig. 4(a) for the
Ising-anisotropic Kondo lattice model SiGrempelZhu  and in Fig. 4(b) for the
SU(2)-symmetry Kondo lattice model Hu20.1x .

a b

Figure 4:  Frequency dependence of the antiferromagnetic lattice spin
susceptibility for (a) Ising-anistropic Kondo lattice at the QCP SiGrempelZhu
and (b) SU(2) Anderson lattice in the paramagnetic side Hu20.1x .  Figure 5:
Dynamical local spin susceptibility for SU(2) Anderson lattice model in the
magnetic ordered side Hu20.1x .

###  IV.3 Dynamical Kondo effect

Kondo destruction in Kondo lattice models originates from the dynamical
competition between the Kondo and RKKY exchange interactions. The exact
marginality of the Kondo coupling inside the AF order phase has been
demonstrated based on an RG analysis within a quantum nonlinear sigma model
analysis Yamamoto07 ; Yamamoto10 . Indeed, the dynamical Kondo effect allows
the gain of a Kondo exchange energy in the absence of static Kondo singlet
formation, which is important to help stabilize a Kondo-destroyed phase and
allows for the quantum phase transition from Kondo phase to be second order.
The dynamical Kondo effects is particularly pronounced in the Kondo destroyed
phase, in which the amplitude of the static Kondo-singlet vanishes. Figure. 5
shows the local dynamical spin susceptibility, $\chi_{loc}(w_{n})$, as a
function of frequency in the AF phase Hu20.1x . The monotonic increase of
$\chi_{loc}(w_{n})$ as $I/T_{K}^{0}$ is tuned towards the QCP reflects the
growth of the dynamical Kondo effect. The match of the local dynamics from
both sides at the quantum critical point demonstrates the second-order nature
of the zero-temperature transition. The dynamical Kondo effect also implies
that the quasiparticles near the small Fermi surface in the Kondo-destroyed
phase have an enhanced mass, which is important to understand the enhancement
of the Sommerfeld coefficient, cyclotron mass, and quadratic-in-$T$
coefficient of the electrical resistivity in the Kondo-destroyed phase
implicated in YbRh2Si2 (Ref. Gegenwart02 ) and related systems.

##  V Kondo lattice Hamiltonian: Singular charge response

In addition to the spin dynamics, Kondo destruction quantum criticality also
involves the localization-delocalization of the $f$-electrons at the QCP. This
partial Mott transition is accompanied by a sharp large-to-small Fermi surface
reconstruction across the phase transition. Thus, the charge degree of freedom
represents an integral part of the quantum criticality. A recent terahertz
spectroscopy measurement in the quantum critical heavy fermion metal YbRh2Si2
have discovered a singular charge response Prochaska2020 . This observation is
inconsistent with conventional SDW QCP, where only the order parameter should
be singular and the charge correlations are expected to be smooth. The EDMFT
analysis is able to capture the singular charge response at the KD QCP
Ang19PRL . As shown in Fig. 6(a), the charge-carrying hybridization B-field at
the KD QCP follows dynamical Planckian scaling. This result is complemented by
the corresponding calculation in a Bose-Fermi Kondo model within a dynamical
large-$N$ approach, where the index $N$ appears in the spin channel, as shown
in Fig. 6(b).

a b

Figure 6:  (a) Singular charge response of the hybridization $B$-field in the
lattice model at the KD quantum critical point, calculated by the EDMFT
method. (b) Dynamical Planckian scaling of the charge response at the KD
quantum critical point in the spectral function of the $B$-field in the
dynamical large-$N$ limit. Both figures from Ref. Ang19PRL  (see also Ref.
Zhu.04 ).

##  VI Further developments

We now turn to discussing the prospect for further developing the EDMFT
method, as well as for applying it to new settings for correlation physics.

###  VI.1 Cluster version of EDMFT

We first summarize the cluster extended dynamical mean field theory (CEDMFT)
CEDMFT15 . We first divide a $d$ dimensional lattice with size $L$ into
several clusters, where each cluster has $N_{c}(=(L_{c})^{d})$ sites with size
$L_{c}$. We therefore use ${\bf\tilde{{x}}}$ to label the position of cluster
and use ${\bf X}$ to label the position of site within the cluster.
Correspondingly, we let ${\bf K}$ (or ${\bf P}$) and ${\bf\tilde{k}}$ (or
${\bf\tilde{p}}$) to label the reciprocal spaces corresponding to the position
vectors ${\bf X}$ and ${\bf\tilde{{x}}}$ respectively. Therefore, the original
momentum vector ${\bf k}$ can be written as ${\bf k}={\bf K}+{\bf\tilde{k}}$.
Then, via dynamical cluster approximation, we are able to map the original
lattice model to a self-consistent impurity model, where the impurities are
defined on a cluster. In the case of Kondo lattice model defined in Eq. 10, we
obtain the following impurity model

$\displaystyle S_{imp}$ $\displaystyle=$ $\displaystyle S_{c}+S_{\phi}+S_{I}$
$\displaystyle S_{c}$ $\displaystyle=$
$\displaystyle-\int_{0}^{\beta}\int_{0}^{\beta}\sum_{{\bf K},\sigma}c_{{\bf
K},\sigma}^{\dagger}(\tau)G^{-1}_{0}(\tau-\tau^{\prime},{\bf K})c_{{\bf
K},\sigma}(\tau^{\prime})d\tau d\tau^{\prime}$ $\displaystyle S_{\phi}$
$\displaystyle=$
$\displaystyle-\frac{1}{2}\int_{0}^{\beta}\int_{0}^{\beta}\sum_{{\bf
P},\mu}S^{\mu}_{{\bf P}}(\tau)\chi^{\mu,-1}_{0}(\tau-\tau^{\prime},{\bf
P})S^{\mu}_{-{\bf P}}(\tau^{\prime})d\tau d\tau^{\prime}$ $\displaystyle
S_{I}$ $\displaystyle=$ $\displaystyle\int_{0}^{\beta}\sum_{{\bf P},{\bf
K}}\frac{J_{K}}{\sqrt{N_{c}}}{\bf S}_{{\bf P}}(\tau)\cdot\frac{c_{{\bf K}+{\bf
P}}^{\dagger}(\tau)\bm{\sigma}c_{{\bf K}}(\tau)}{2}d\tau
d\tau+\int_{0}^{\beta}\sum_{{\bf P}}\bar{I}_{{\bf P}}{\bf S}_{{\bf
P}}(\tau){\bf S}_{-{\bf P}}(\tau)d\tau$ (20)
$\displaystyle+\int_{0}^{\beta}\sum_{{\bf P},\mu}h^{\mu}_{{\bf P}}{\bf
S}^{\mu}_{{\bf P}}(\tau)d\tau$

where the intra-cluster interactions are defined as $\bar{I}_{{\bf
P}}=\frac{(L_{c})^{d}}{L^{d}}\sum_{{\bf\tilde{p}}}I_{{\bf P}+{\bf\tilde{p}}}$
with $I_{{\bf p}}$ the Fourier transformation of RKKY interactions $I_{ij}$.
$h_{{\bf P}}^{\mu}$ corresponds to an effective magnetic field, which
characterize the development of long-range magnetic order. $G_{0}(\tau,{\bf
K})$ and $\chi_{0}^{\mu}(\tau,{\bf P})$ are the fermionic and bosonic bath
functions respectively. The case with $N_{c}=2$ is illustrated in Fig. 7.

Figure 7:  Illustration of the two impurity Bose -Fermi Anderson model that
arises from the CEDMFT approach.

Via the Dyson equations, the self-energies of the single-particle Green’s
function and spin-spin correlation functions take the following forms

$\displaystyle M^{\mu}(i\omega_{n},{\bf P})=\chi_{0}^{\mu,-1}(i\omega_{n},{\bf
P})-\bar{I}_{{\bf P}}+\chi^{\mu,-1}_{loc}(i\omega_{n},{\bf P})$
$\displaystyle\Sigma(i\omega_{b},{\bf K})=G_{0}^{-1}(i\omega_{n},{\bf
K})-G^{-1}_{loc}(i\omega_{n},{\bf K})$ (21)

where the local correlation functions are defined as

$\displaystyle\chi_{loc}^{\mu}(\tau,{\bf P})=\langle T_{\tau}:S^{\mu}_{{\bf
P}}:(\tau):S^{\mu}_{-{\bf P}}:(0)\rangle_{S_{imp}}$ $\displaystyle
G_{loc}(\tau,{\bf K})=-\langle T_{\tau}c_{{\bf K},\sigma}(\tau)c_{{\bf
K},\sigma}^{\dagger}(0)\rangle_{S_{imp}}\,.$ (22)

where we introduce the normal ordering of spin fields $:S^{\mu}_{{\bf
P}}:=:S^{\mu}_{{\bf P}}:-m^{\mu}_{{\bf P}}$ with the magnetization
$m^{\mu}_{{\bf P}}=\langle S^{\mu}_{{\bf P}}\rangle_{S_{imp}}$. The bath
functions and the effective magnetic fields are then defined self-consistently
by requiring

$\displaystyle\chi_{loc}^{\mu}(i\omega_{n},{\bf
P})=\bigg{(}\frac{L_{c}}{L}\bigg{)}^{d}\sum_{{\bf\tilde{p}}}\frac{1}{I_{{\bf
P}+{\bf\tilde{p}}}+M^{\mu}(i\omega_{n},{\bf P})}$ $\displaystyle
G_{loc}(i\omega_{n},{\bf
K})=\bigg{(}\frac{L_{c}}{L}\bigg{)}^{d}\sum_{{\bf\tilde{k}}}\frac{1}{i\omega_{n}-\epsilon_{{\bf
K}+{\bf\tilde{k}}}-\Sigma_{c}(i\omega_{n},{\bf K})}$ $\displaystyle h_{{\bf
P}}^{\mu}=-m^{\mu}_{-{\bf P}}\bigg{(}\chi_{0}^{\mu,-1}(i\omega_{n}=0,{\bf
P})+I_{{\bf P}}^{\mu}-\bar{I}_{{\bf P}}^{\mu}\bigg{)}$ (23)

We note that, by adding the $h_{{\bf P}}^{\mu}$ contributions, we have normal-
ordered the retarded interactions induced by the bosonic bath
$\chi_{0}^{\mu}(\tau,{\bf P})$.

The original lattice model are then solved by repeatedly solving the cluster
model in Eq. 20 via quantum Monte Carlo until the self-consistent relations in
Eq. 23 are satisfied. Comparing to the single-impurity formula, we are able to
partially capture the momentum dependency of the self energies and describe
the inter-site correlations in CEDMFT. The inter-site correlations would allow
attractive interactions between two electrons from different sites and, thus,
provide a route towards unconventional superconductivity. Therefore, CEDMFT
provides a platform to study the unconventional superconductivity induced by
electron correlations. This approach has allowed the study of how Kondo
destruction quantum criticality drives unconventional and high-$T_{c}$
superconductivity Hu2021.sc-x .

###  VI.2 Models with disordered exchange interactions

The EDMFT approach has some similarities with the dynamical mean field theory
of random spin systems. The quantum Sherrington-Kirkpatrick model was
considered by Bray and Moore in Ref. Bray80 , where the spin-exchange
interactions are infinite-ranged, random and have a Gaussian distribution
whose mean vanishes and whose variance is $J^{2}/N_{s}$ where $N_{s}$ is the
size of the system. A disorder averaging using the replica trick, as usual,
gives rise to four-spin interactions, with each spin carrying a replica index.
Because the exchange couplings on different bonds are not correlated, the
four-spin interaction corresponds to two spins at two different times $\tau$
and $\tau^{\prime}$, from any site, interacting with two other spins, at the
same times $\tau$ and $\tau^{\prime}$ and from every other site. Taking the
limit of $N_{s}\rightarrow\infty$ yields a single-site problem with retarded
spin-spin interactions. In the paramagnetic phase, the local correlators are
determined by an effective local problem whose form is similar to the spin
part of Eq. (11), with a self-consistency equation that corresponds to a semi-
circular $\rho_{\rm I}(x)$. When the spin symmetry is generalized from SU(2)
to SU(N) and the limit of large-$N$ is taken, a spin liquid ground state is
realized SachYe93 .

More recently, a $t-J$ model with such a random-Gaussian $J$ interactions
between the spins of the fermions has been studied. The saddle-point equations
are similar to the EDMFT equations as reviewed in this article, with a semi-
circular $\rho_{I}(x)$. In this case, while the asymptotic low-temperature
dynamics will ultimately be similar to what happens in an SDW QCP, at
intermediate temperatures and energies the results Joshi2020 ; Dumitrescu2022
; ChowdhuryRMP  are similar to what we have surveyed.

###  VI.3 Application to electronic structure calculations: the GW$+$EDMFT
approach

The EDMFT approach has been combined with ab initio methods for electronic-
structure calculations, incorporating non-perturbative effects in the GW-
method (GW+DMFT). In the GW approach gw_origin , the self-energies are derived
by calculating the one-loop diagrams involving the screened Coulomb
interaction, and are momentum- and frequency-dependent. The EDMFT method
provides a non-perturbative approach for the local self-energies. In the
GW+EDMFT approach PhysRevLett.92.196402 ; PhysRevMaterials.1.043803 ;
PhysRevLett.90.086402 ; kutepov2021spatial , the two methods are combined
where the local part of the GW self-energies are replaced by the EDMFT-derived
local self-energies. Thus, the GW+EDMFT approach provides a non-perturbative
and efficient method to perform electronic structure calculations from first
principles.

###  VI.4 EDMFT approach in new settings

The EDMFT method is uniquely suited to elucidate the correlation physics in a
variety of new settings. Here we describe two issues as examples.

One issue concerns the orbital-selective Mott phase, in which certain orbitals
are Mott localized while the others remain itinerant, which is of considerable
current interest in the field of iron-based superconductors Si16.1 . Such an
orbital-selective Mott phase in the presence of kinetic inter-orbital
hybridization has been demonstrated using certain auxiliary spin method but
has yet to be shown in methods based on the dynamical mean field theory.
Recent renormalization-group analysis Hu-chen-dehyb22.x  motivates an EDMFT-
based examination of the interplay between the inter-orbital hybridization and
intersite spin-exchange interactions, both of which are important to the
physics of the iron-based superconductors.

Another issue deals with frustrated bulk systems with topological flat bands
that are coupled to wide bands. Recent work has illustrated how an effective
orbital-selective Mott transition can take place when an Anderson/Kondo
lattice model is realized as an effective description Hu-flat22.x . The EDMFT
approach is expected to shed considerable new light on this emerging topic.

##  VII Summary and Outlook

We have provided an overview of the extended dynamical mean field theory for
correlated electron models. The method is especially suited for settings where
the competing interactions arise from local and intersite terms, respectively.
Our emphasis has been on models for which the method was initially formulated
and developed, namely models of the multi-band Anderson-lattice type and those
of the one-band Hubbard-Heisenberg kind.

A setting in which this method has been most extensively used is the
Kondo/Anderson lattice models for heavy fermion systems, where the key issue
is the dynamical competition between the local Kondo and intersite RKKY
interactions. In elucidating how the amplitude of the Kondo singlet in the
ground state and the weight of heavy quasiparticles are critically destroyed,
these studies underline the notion of Kondo destruction. The associated
physics includes a loss of quasiparticles and dynamical Planckian scaling at
the quantum critical point and a large-to-small Fermi surface jump across the
quantum critical point Hu-QCM-review2022 .

We have also discussed the prospect for further developing the method,
including in the form that is suitable to study unconventional
superconductivity. In the same spirit, we note that new settings have continue
to arise for which the approach of the extended dynamical mean field theory is
poised to clarify new physics. We have described two such issues. One concerns
the orbital-selective Mott phase in the iron-based superconductors, and the
other about the correlation physics of coupled topological flat-wide bands in
kinetically-frustrated bulk systems. Given the versatility that the EDMFT
approach has shown in elucidating the dynamical interplay between local and
spatial correlations, we expect that other settings will also develop in which
the approach will provide new insights into the underlying correlation
physics.

###### Acknowledgements.

We would like to thank the late D. R. Grempel, A. Cai, M. T. Glossop, P.
Goswami, A. Kandala, K. Ingersent, S. Kirchner, E. M. Nica, J. H. Pixley, J. L
Smith, J.-X. Zhu, L. J. Zhu and, particularly, S. Paschen and F. Steglich, for
collaborations on work pertaining to the subject of this article. This work
has been supported in part by the NSF Grant No. DMR-2220603 (Q.S.) the Air
Force Office of Scientific Research under Grant No. FA9550-21-1-0356 (H.H.),
the Robert A. Welch Foundation Grant No. C-1411 (L.C.), and the European
Research Council (ERC) under the European Union’s Horizon 2020 research and
innovation program (Grant Agreement No. 101020833) (H.H).

## References

  * (1) Paschen, S. & Si, Q.  Quantum phases driven by strong correlations.  _Nat. Rev. Phys._ 3, 9 (2021).
  * (2) Hussey, N. E., Takenaka, K. & Takagi, H.  Universality of the Mott–Ioffe–Regel limit in metals.  _Philos. Mag._ 84, 2847 (2004).
  * (3) Dressel, M., Kasper, N., Petukhov, K., Peligrad, D. N., Gorshunov, B., Jourdan, M., Huth, M. & Adrian, H.  Correlation gap in the heavy-fermion antiferromagnet ${\mathrm{UPd}}_{2}{\mathrm{Al}}_{3}$.  _Phys. Rev. B_ 66, 035110 (2002).
  * (4) Qazilbash, M. M., Hamlin, J. J., Baumbach, R. E., Zhang, L., Singh, D. J., Maple, M. B. & Basov, D. N.  Electronic correlations in the iron pnictides.  _Nat. Phys._ 5, 647 (2009).
  * (5) Si, Q., Yu, R. & Abrahams, E.  High-temperature superconductivity in iron pnictides and chalcogenides.  _Nat. Rev. Mater._ 1, 16017 (2016).
  * (6) Nozieres, P.  _Theory of interacting Fermi systems_ (CRC Press, 2018).
  * (7) Imada, M., Fujimori, A. & Tokura, Y.  Metal-insulator transitions.  _Rev. Mod. Phys._ 70, 1039–1263 (1998).
  * (8) Anderson, P. W.  Localized magnetic states in metals.  _Phys. Rev._ 124, 41–53 (1961).
  * (9) Kirchner, S., Paschen, S., Chen, Q., Wirth, S., Feng, D., Thompson, J. D. & Si, Q.  Colloquium: Heavy-electron quantum criticality and single-particle spectroscopy.  _Rev. Mod. Phys._ 92, 011002 (2020).
  * (10) Stewart, G. R.  Non-Fermi-liquid behavior in $d$\- and $f$-electron metals.  _Rev. Mod. Phys._ 73, 797–855 (2001).
  * (11) Coleman, P. & Schofield, A. J.  Quantum criticality.  _Nature_ 433, 226–229 (2005).
  * (12) Si, Q. & Steglich, F.  Heavy fermions and quantum phase transitions.  _Science_ 329, 1161–1166 (2010).
  * (13) Georges, A., Kotliar, G., Krauth, W. & Rozenberg, M.  Dynamical mean-field theory of strongly correlated fermion systems and the limit of infinite dimensions.  _Rev. Mod. Phys._ 68, 13–125 (1996).
  * (14) Vollhardt, D.  _Dynamical Mean-Field Theory of Strongly Correlated Electron Systems_.
  * (15) Si, Q. & Smith, J. L.  Kosterlitz-Thouless transition and short range spatial correlations in an extended Hubbard model.  _Phys. Rev. Lett._ 77, 3391–3394 (1996).
  * (16) Smith, J. L. & Si, Q.  Spatial correlations in dynamical mean-field theory.  _Phys. Rev. B_ 61, 5184–5193 (2000).
  * (17) Chitra, R. & Kotliar, G.  Effective-action approach to strongly correlated fermion systems.  _Phys. Rev. B_ 63, 115110 (2001).
  * (18) Hu, H., Chen, L. & Si, Q.  Quantum critical metals: Dynamical Planckian scaling and loss of quasiparticles.  _arXiv preprint arXiv: 2210.yyyzz_ (2022).
  * (19) Si, Q., Rabello, S., Ingersent, K. & Smith, J.  Locally critical quantum phase transitions in strongly correlated metals.  _Nature_ 413, 804–808 (2001).
  * (20) Si, Q., Rabello, S., Ingersent, K. & Smith, J. L.  Local fluctuations in quantum critical metals.  _Phys. Rev. B_ 68, 115103 (2003).
  * (21) Si, Q., Smith, J. L. & Ingersent, K.  Quantum critical behavior in Kondo systems.  _Int. J. Mod. Phys. B_ 13, 2331–2342 (1999).
  * (22) Coleman, P., Pépin, C., Si, Q. & Ramazashvili, R.  How do Fermi liquids get heavy and die?  _J. Phys. Cond. Matt._ 13, R723 (2001).
  * (23) Senthil, T., Vojta, M. & Sachdev, S.  Weak magnetism and non-Fermi liquids near heavy-fermion critical points.  _Phys. Rev. B_ 69, 035111 (2004).
  * (24) Zhu, J., Grempel, D. & Si, Q.  Continuous quantum phase transition in a Kondo lattice model.  _Phys. Rev. Lett._ 91, 156404 (2003).
  * (25) Chowdhury, D., Georges, A., Parcollet, O. & Sachdev, S.  Sachdev-Ye-Kitaev models and beyond: Window into non-Fermi liquids.  _Rev. Mod. Phys._ 94, 035004 (2022).
  * (26) Hewson, A. C.  _The Kondo Problem to Heavy Fermions_ (Cambridge University Press, Cambridge, 1993).
  * (27) Glossop, M. & Ingersent, K.  Magnetic quantum phase transition in an anisotropic Kondo lattice.  _Phys. Rev. Lett._ 99, 227203 (2007).
  * (28) Zhu, J.-X., Kirchner, S., Bulla, R. & Si, Q.  Zero-temperature magnetic transition in an easy-axis Kondo lattice model.  _Phys. Rev. Lett._ 99, 227204 (2007).
  * (29) Smith, J. L. & Si, Q.  Non-Fermi liquids in the two-band extended Hubbard model.  _Europhys. Lett._ 45, 228 (1999).
  * (30) Sengupta, A. M.  Spin in a fluctuating field: The Bose(+Fermi) Kondo models.  _Phys. Rev. B_ 61, 4041–4043 (2000).
  * (31) Zhu, L. & Si, Q.  Critical local moment fluctuations in the Bose-Fermi Kondo model.  _Phys. Rev. B_ 66, 024426 (2002).
  * (32) Zaránd, G. & Demler, E.  Quantum phase transitions in the Bose-Fermi Kondo model.  _Phys. Rev. B_ 66, 024427 (2002).
  * (33) Hu, H. & Si, Q.  Kondo destruction and fixed-point annihilation in a Bose-Fermi Kondo model.  _arXiv preprint arXiv:2207.08744_ (2022).
  * (34) Cai, A. & Si, Q.  Bose-Fermi Anderson model with SU(2) symmetry: Continuous-time quantum Monte Carlo study.  _Phys. Rev. B_ 100, 014439 (2019).
  * (35) Glossop, M. T. & Ingersent, K.  Numerical renormalization-group study of the Bose-Fermi Kondo model.  _Phys. Rev. Lett._ 95, 067202 (2005).
  * (36) Pixley, J., Kirchner, S., Glossop, M. & Si, Q.  Continuous-time Monte Carlo study of the pseudogap Bose-Fermi Kondo model.  In _Journal of Physics: Conference Series_ , vol. 273, 012050 (IOP Publishing, 2011).
  * (37) Pixley, J., Kirchner, S., Ingersent, K. & Si, Q.  Quantum criticality in the pseudogap Bose-Fermi Anderson and Kondo models: Interplay between fermion-and boson-induced Kondo destruction.  _Phys. Rev. B_ 88, 245111 (2013).
  * (38) Werner, P. & Millis, A. J.  Dynamical screening in correlated electron materials.  _Phys. Rev. Lett._ 104, 146401 (2010).
  * (39) Otsuki, J.  Spin-boson coupling in continuous-time quantum Monte Carlo.  _Phys. Rev. B_ 87, 125102 (2013).
  * (40) Zhu, J.-X., Kirchner, S., Bulla, R. & Si, Q.  Zero-temperature magnetic transition in an easy-axis Kondo lattice model.  _Phys. Rev. Lett._ 99, 227204 (2007).
  * (41) Zhu, J.-X., Grempel, D. R. & Si, Q.  Continuous quantum phase transition in a Kondo lattice model.  _Phys. Rev. Lett._ 91, 156404 (2003).
  * (42) Grempel, D. & Si, Q.  Locally critical point in an anisotropic Kondo lattice.  _Phys. Rev. Lett._ 91, 026401 (2003).
  * (43) Hu, H., Cai, A. & Si, Q.  Quantum criticality and dynamical Kondo effect in an SU(2) Anderson lattice model.  _arXiv:2004.04679_ (2020).
  * (44) Si, Q., Zhu, J.-X. & Grempel, D. R.  Magnetic quantum phase transitions in Kondo lattices.  _J. Phys.: Condens. Matter_ 17, R1025–R1040 (2005).
  * (45) Yamamoto, S. J. & Si, Q.  Fermi surface and antiferromagnetism in the Kondo lattice: an asymptotically exact solution in $d>1$ dimensions.  _Phys. Rev. Lett._ 99, 016401 (2007).
  * (46) Yamamoto, S. J. & Si, Q.  Renormalization group for mixed fermion-boson system.  _Phys. Rev. B_ 81, 205106(1)–(14) (2010).
  * (47) Gegenwart, P., Custers, J., Geibel, C., Neumaier, K., Tayama, T., Tenya, K., Trovarelli, O. & Steglich, F.  Magnetic-field induced quantum critical point in $\mathrm{Y}\mathrm{b}\mathrm{R}{\mathrm{h}}_{\mathrm{2}}\mathrm{S}{\mathrm{i}}_{\mathrm{2}}$.  _Phys. Rev. Lett._ 89, 056402 (2002).
  * (48) Prochaska, L., Li, X., MacFarland, D. C., Andrews, A. M., Bonta, M., Bianco, E. F., Yazdi, S., Schrenk, W., Detz, H., Limbeck, A., Si, Q., Ringe, E., Strasser, G., Kono, J. & Paschen, S.  Singular charge fluctuations at a magnetic quantum critical point.  _Science_ 367, 285–288 (2020).
  * (49) Cai, A., Yu, Z., Hu, H., Kirchner, S. & Si, Q.  Dynamical scaling of charge and spin responses at a Kondo destruction quantum critical point.  _Phys. Rev. Lett._ 124, 027205 (2020).
  * (50) Zhu, L., Kirchner, S., Si, Q. & Georges, A.  Quantum critical properties of the Bose-Fermi Kondo model in a large-$N$ limit.  _Phys. Rev. Lett._ 93, 267201 (2004).
  * (51) Pixley, J., Cai, A. & Si, Q.  Cluster extended dynamical mean-field approach and unconventional superconductivity.  _Physical Review B_ 91, 125127 (2015).
  * (52) Hu, H., Cai, A., Chen, L., Deng, L., Pixley, J. H., Ingersent, K. & Si, Q.  Unconventional superconductivity from Fermi surface fluctuations in strongly correlated metals.  _arXiv:2109.13224_ (2021).
  * (53) Bray, A. J. & Moore, M. A.  Replica theory of quantum spin glasses.  _Journal of Physics C: Solid State Physics_ 13, L655 (1980).
  * (54) Sachdev, S. & Ye, J.  Gapless spin-fluid ground state in a random quantum heisenberg magnet.  _Phys. Rev. Lett._ 70, 3339–3342 (1993).
  * (55) Joshi, D. G., Li, C., Tarnopolsky, G., Georges, A. & Sachdev, S.  Deconfined critical point in a doped random quantum heisenberg magnet.  _Phys. Rev. X_ 10, 021033 (2020).
  * (56) Dumitrescu, P. T., Wentzell, N., Georges, A. & Parcollet, O.  Planckian metal at a doping-induced quantum critical point.  _Phys. Rev. B_ 105, L180404 (2022).
  * (57) Hedin, L.  New method for calculating the one-particle Green’s function with application to the electron-gas problem.  _Phys. Rev._ 139, A796–A823 (1965).
  * (58) Sun, P. & Kotliar, G.  Many-body approximation scheme beyond GW.  _Phys. Rev. Lett._ 92, 196402 (2004).
  * (59) Nilsson, F., Boehnke, L., Werner, P. & Aryasetiawan, F.  Multitier self-consistent $GW+\text{EDMFT}$.  _Phys. Rev. Materials_ 1, 043803 (2017).
  * (60) Biermann, S., Aryasetiawan, F. & Georges, A.  First-principles approach to the electronic structure of strongly correlated systems: Combining the $GW$ approximation and dynamical mean-field theory.  _Phys. Rev. Lett._ 90, 086402 (2003).
  * (61) Kutepov, A. L.  Spatial non-locality of electronic correlations beyond $GW$ approximation.  _J. Condens. Matter Phys_ 33, 485601 (2021).
  * (62) Hu, H., Chen, L., Zhu, J.-X., Yu, R. & Si, Q.  Orbital-selective Mott phase as a dehybridization fixed point.  _arXiv preprint arXiv:2203.06140_ (2022).
  * (63) Hu, H. & Si, Q.  Coupled topological flat and wide bands: Quasiparticle formation and destruction.  _arXiv preprint arXiv:2209.10396_ (2022).