# Spin-triplet superconductivity at the onset of isospin order in biased
bilayer graphene

Zhiyu Dong1, Andrey V. Chubukov2, Leonid Levitov  Department of Physics,
Massachusetts Institute of Technology, Cambridge, MA 02139, USA
2W. I. Fine Theoretical Physics Institute, University of Minnesota,
Minneapolis, MN 55455, USA

###### Abstract

The quest for unconventional superconductivity governed by Coulomb repulsion
between electrons rather than phonon attraction received new momentum with the
advent of moiré graphene. Initially, delineating the phonon and Coulomb-
repulsion-based pairing mechanisms has proven to be a challenging task,
however the situation has changed after recent discovery of superconductivity
in non-twisted graphene bilayers and trilayers. Superconductivity occurring at
the phase boundaries of spin and valley polarized orders calls for non-phonon
scenarios, yet the specific pairing mechanisms remain to be understood. Here
we analyze a striking example — superconductivity in graphene bilayers
occurring at the onset of valley-polarized order induced by a magnetic field.
We describe an attraction-from-repulsion mechanism for pairing mediated by a
quantum-critical mode, which fully explains the observed phenomenology. While
it is usually notoriously difficult to infer the pairing mechanism from the
observed superconducting phases, this case presents a rare exception, allowing
for a fairly unambiguous identification of the origin of the pairing glue. A
combination of factors such as the location of superconducting phase at the
onset of isospin-polarized phase, a threshold in a magnetic field, above which
superconductivity occurs, and its resilience at high magnetic fields paints a
clear picture of a triplet superconductivity driven by quantum-critical
fluctuations.

Superconductivity (SC) in moiré graphene, occurring in proximity to other
correlated electronic ordersBistritzer and MacDonald (2011); Cao et al.
(2018); Lu et al. (2019); Andrei and MacDonald (2020); Saito et al. (2020);
Polshyn et al. (2019); Oh et al. (2021); Cao et al. (2021); Jaoui et al.
(2022), attracts intense interest as a possible instance of an unconventional
(Coulomb-repulsion-driven) pairing mechanism. However, so far no consistent
picture has emerged. In some cases SC appears to be associated with correlated
orders Cao et al. (2021); Jaoui et al. (2022), suggesting exotic pairing
scenariosYou and Vishwanath (2019); Kozii et al. (2019); Chichinadze et al.
(2020); Khalaf et al. (2021), whereas other experiments report on SC that can
be isolated (and thus decoupled) from other ordered phases Saito et al. (2019,
2020). Presently, there is no clear verdict on the pairing mechanism, as some
studies point to the irrelevance of electron interactions for the pairing
Stepanov et al. (2020); Polshyn et al. (2019), supporting the conventional
phonon mechanism Wu et al. (2018); Lian et al. (2019); Wu et al. (2019),
whereas others suggest these interactions do matterLiu et al. (2021). System
complexity, such as the peculiar form of moiré flatband electron
wavefunctionBistritzer and MacDonald (2011); Tarnopolsky et al. (2019) and
multiple kinds of moiré-related disorder (e.g. twist-angle disorder, strain,
buckling, and so on), make this debate difficult to settle.

Helpfully, however, recent work unveiled two non-moiré systems that host
superconducting orders intertwined with correlated electronic orders — the
field-biased rhombohedral trilayer grapheneZhou et al. (2021a, b) (RTG) and
Bernal bilayer grapheneZhou et al. (2021c); de la Barrera et al. (2021) (BBG).
These systems present a distinct advantage for studying strongly-correlated
physics owing to the simplicity of their bandstructure, tunable by an external
transverse electric field, and exceptional cleanness due to the absence of
strain. Here we focus on BBG, where a particularly interesting superconducting
order has been observed Zhou et al. (2021c). BBG is a system with a simple
band structure consisting of two bands McCann and Koshino (2013); McCann and
Fal’ko (2006), whose wavefunctions are predominantly constructed from the
orbitals in one of the two layers: one from the A sublattice in the upper
layer, the other from the B sublattice in the bottom layer. In the absence of
transverse field, the two bands have quadratic dispersion and touch at the
high-symmetry points $K$ and $K^{\prime}$. In unbiased BBG, theory anticipates
various correlated ordered states Nandkishore and Levitov (2010a, b); Vafek
and Yang (2010); Jung et al. (2011); MacDonald et al. (2012); Zhang and
MacDonald (2012); Cvetkovic et al. (2012); Throckmorton and Das Sarma (2014);
Min et al. (2008); Nilsson et al. (2006), some of which have been identified
in experimentsMartin et al. (2010); Weitz et al. (2010); Mayorov et al.
(2011); Veligura et al. (2012); Velasco et al. (2012); Bao et al. (2012);
Freitag et al. (2012).

Recent renewal of interest in BBG is triggered by the experimental realization
of a flatband regime in this system, where new exotic orders are foundZhou et
al. (2021c); de la Barrera et al. (2021); Seiler et al. (2021). This new
regime is accessible through applying a transverse electric field to open a
band gap at charge neutrality McCann and Fal’ko (2006); McCann (2006). Under
an applied field band dispersion changes from quadratic to quartic, which
flattens out as the field grows. This leads to properties completely distinct
from those of unbiased bilayer graphene Nandkishore and Levitov (2010a, b);
Vafek and Yang (2010); Jung et al. (2011); MacDonald et al. (2012); Zhang and
MacDonald (2012); Cvetkovic et al. (2012); Throckmorton and Das Sarma (2014);
Min et al. (2008); Nilsson et al. (2006). Specifically, as seen in
experimentsZhou et al. (2021c); de la Barrera et al. (2021); Seiler et al.
(2021), at a low carrier density BBG exhibits an isospin instability, where
electrons populate only one or two isospin sub-bands [phases PIP1 and PIP2 in
Fig.1 a), where isospin refers to spin and $K/K^{\prime}$ valley degrees of
freedom]. This behavior leads to a cascade of phase transitions between states
with different polarization, resembling those in moiré grapheneSaito et al.
(2021); Zondiner et al. (2020); Rozen et al. (2021); Choi et al. (2021);
Pierce et al. (2021) and in rhombohedral trilayer grapheneZhou et al. (2021a).
On top of this cascade of isospin orders, other symmetry-breaking orders are
predicted theoretically, e.g. the momentum space polarization wherein all
carriers are shifted into one, two or three pockets at the band minima
produced by the trigonal warping effectsJung et al. (2015); Dong et al.
(2021).

Further, both BBG and RTG host superconductivity Zhou et al. (2021a, c). In
both BBG and RTG the superconducting phases occur at phase boundaries between
different isospin-ordered states, forming narrow sleeves extending along the
phase boundaries. There are interesting differences between these SC phases.
In RTG, there are two SC phases in the hole-doped regimeZhou et al. (2021c),
which show a conventional suppression under an applied magnetic field. To the
contrary, in BBG, there is only one SC phase (the cyan area in Fig.1 c) which
arises at the phase boundary between isospin-ordered and isospin-disordered
phases in the presence of an in-plane magnetic field. At zero magnetic field,
superconductivity is suppressed, giving way to a correlated state with an
insulator-like temperature dependence of resistivity. This state, of a yet
unknown origin, is seen as a red dome in panels b) and c) in Fig.1. In the
literature, several candidate mechanisms for SC in RTG have been proposed
Ghazaryan et al. (2021); Chatterjee et al. (2021); You and Vishwanath (2022),
however the origin of SC and the adjacent correlated phase in BBG have not yet
been understood.

##  I Exotic superconductivity in Bernal bilayer graphene

Figure 1: a) Schematic of the experimental isospin-order phase diagram
[adapted from Ref.Zhou et al. (2021c)]. In phases PIP1 and PIP2 only one or
two isospin-polarized sub-bands are populated. Here, we focus on the isospin
polarization in each phase, ignoring other differences between unpolarized and
partially polarized states that are irrelevant for our discussion. b)
Predicted phase diagram for superconductivity governed by critical mode at the
phase boundary between isospin-valley-polarized and unpolarized phases.
Superconducting order is spin-triplet and is induced by a finite magnetic
field that creates spin imbalance as shown in the insets. The pairing
interaction in the triplet channel is a repulsion at $B=0$, turning into an
effective attraction at a finite $B$. c) The experimental phase diagram for
superconductivity in bilayer grapheneZhou et al. (2021c) strongly resembles
the phase diagram predicted in b) [see text].

Perhaps the most mysterious and intriguing phenomenon observed in BBG systems
is the magnetic-field-induced SC, which will be the subject of this study.
Ref.Zhou et al. (2021c) reports on a unique dependence of SC on parallel
magnetic field, suggesting a pairing mechanism entirely different from those
studied in moiré You and Vishwanath (2019); Kozii et al. (2019); Chichinadze
et al. (2020); Khalaf et al. (2021) and RTG systems Zhou et al. (2021a);
Ghazaryan et al. (2021); Chou et al. (2022). Specifically, it is found that SC
occurs only when $B_{\parallel}$ exceeds a finite threshold (see Fig.1c). This
is quite unlike textbook SC, which is suppressed by a $B$ field. Since
$B_{\parallel}$ only couples to spin when applied in-plane, the $B$-induced SC
indicates that spin imbalance is essential for pairing. Moreover, SC is found
to persist in a high field, surviving well above the Pauli limit. The
resilience of SC in a $B$ field unambiguously points to a spin-triplet
pairing, and thus an unconventional pairing mechanism.

Some clues for pairing mechanism are revealed by several features, shared by
BBG and RTG. In both systems, SC tracks the boundary between isospin ordered
and disordered phases (PIP2 and Sym12 phases in the notations of Zhou et al.
(2021c)). This boundary remains sharp in the presence of a $B$ field Zhou et
al. (2021a, c). This suggests pairing mediated by a critical isospin mode – a
scenario, in which both superconductivity and the Stoner instability
responsible for isospin order arise from electron-electron repulsion. A
pairing of this type would place BBG into the class of systems with quantum-
critical SC Klein et al. (2020); Oganesyan et al. (2001); Lederer et al.
(2015); Tremblay (2013); Chubukov et al. (2020); Berg et al. (2019); Lee
(2018); Efetov et al. (2013).

Yet, this scenario, encounters a crucial obstacle when applied to BBG system.
In previously studied instances, soft quantum-critical modes generate an
effective e-e attraction either by scattering Cooper pairs between different
Fermi surfaces or different hotspots on a single Fermi surfaceMazin et al.
(2008); Mazin and Schmalian (2009) by exchange-type interactions Scalapino et
al. (1966). This does not work for BBG because Cooper pairs in graphene are
formed by electrons in valleys $K$ and $K^{\prime}$, and the size of a Fermi
surface in each valley is much smaller than the reciprocal lattice vector.
This excludes the usual pair hopping scenarios and makes the exchange-type
processes negligible (see Ref.SM  for more detail). As the consequence, the
soft-mode-mediated interaction, while being strong, is repulsive in graphene
systems. Therefore, understanding this unusual superconductivity requires a
new mechanism that converts a strong repulsion, mediated by the soft modes,
into an attraction.

Here we propose such mechanism for BBG. This mechanism ties together all
ingredients mentioned above: a quantum-critical mode, a repulsive Coulomb
coupling, a broken spin degeneracy, and pairing in spin-triplet channel. We
find that the pairing interaction acquires new properties at a finite spin
imbalance, which lead to a SC instability. First, for spins opposite to $B$,
the pairing interaction at Stoner transition, while remaining repulsive at all
bosonic frequencies $\nu$, acquires a non-monotonic frequency dependence.
Namely, it drops sharply at small $\nu$ and passes through a maximum at
intermediate $\nu$ (see Fig.3). This behavior is “universal” in the sense that
the existence of such non-monotonicity is independent of the details in the
bandstructure. This non-monotonicity enables pairing with a superconducting
gap $\Delta(\omega)$ changing sign between small and large $\omega$ Morel and
Anderson (1962); McMillan (1968); Bogoljubov et al. (1958); Coleman (2015);
Chubukov et al. (2019). We show that in our case this mechanism leads to spin-
triplet, valley-singlet $s-$wave pairing. Second, for spins along $B$, the
same fully dressed pairing interaction, mediated by a quantum-critical mode,
acquires field-induced dependence on soft-mode momenta. This generates finite
interaction components in non-$s-$wave channels. We show that in our case this
leads to pairing in spin- and valley-triplet, spatially-odd $p$-wave channel.
This is essentially a field-induced Kohn-Luttinger (KL) mechanism Kohn and
Luttinger (1965) in 2D.

We emphasize that the attraction for both spin components comes from the same
effective interaction mediated by soft isospin excitations, i.e., the two
attractive channels are intertwined. Either pairing mechanism explains all the
qualitative features in the experimentally measured phase diagram, Fig.1 c).
First, superconductivity peaks at the phase boundary. Second,
superconductivity has a threshold in $B$ field as one needs a finite $B$ to
overcome initial repulsion. At $B$ below the threshold we predict a strong
repulsive interaction. This interaction is expected to produce a correlated
insulator state, in line with experiment [Fig.1 c)].

One more aspect of the observed phase diagram that supports this scenario is
the dependence of the onset of the PIP2 order on the in-plane $B$ field. In
experiments, the $B$ field merely pushes the onset of the order to lower
carrier densities, producing a phase boundary with a constant slope marked by
dashed line in Fig.1 c). As we will see, this behavior follows directly from
our model (see (8) below and accompanying discussion). We note that in Fig.1
c), the $x$ axis is drawn from high carrier density to low carrier density,
and must be reversed when compared to panel b). The experimental and
theoretical slopes of the phase boundary match both in sign and in value,
lending further support to theory.

##  II The pairing interaction due to quantum-critical modes

We now proceed with the analysis. We model the interacting electrons in BLG
using the conventional Hubbard Hamiltonian:

$H=\sum_{i}\epsilon_{i}({\boldsymbol{p}})\psi_{i{\boldsymbol{p}}}^{\dagger}\psi_{i{\boldsymbol{p}}}+\sum_{ii^{\prime}}\frac{V_{0}}{2}\psi_{i{\boldsymbol{p}}+{\boldsymbol{q}}}^{\dagger}\psi_{i^{\prime}{\boldsymbol{p}}^{\prime}-{\boldsymbol{q}}}^{\dagger}\psi_{i^{\prime}{\boldsymbol{p}}^{\prime}}\psi_{i{\boldsymbol{p}}},$
(1)

where
$i,i^{\prime}=K\uparrow,K\downarrow,K^{\prime}\uparrow,K^{\prime}\downarrow$
are isospin indices, $V_{0}$ is a short-range repulsion. The band structure
$\epsilon({\boldsymbol{p}})$ details will be inessential in either mechanism
we are going to describe in next two sections. Specifically, as we will see
later, the role of $\epsilon({\boldsymbol{p}})$ in our first mechanism can be
described by one parameter, and only affects the result quantitatively,
whereas in the second mechanism, a realistic $\epsilon({\boldsymbol{p}})$ is
only a small twist on the case of parabolic dispersion which we will use as a
minimal model. This small twist leads to a finite field threshold, but does
not invalidate the scenario. Since they are inessential, we will keep it
general for now and specify them later when needed.

Figure 2:  a) The RPA diagrams describing the valley-polarization instability.
Here $v,s$ are valley and spin indices. b) Diagrams describing the effective
pairing interaction between electrons in valleys $K$ and $K^{\prime}$ mediated
by quantum-critical modes. These processes give a divergent enhancement to
forward scattering near the valley-polarization instability (see text). The
analytical expression for the effective interaction is given in (2). c) A
diagrammatic description of the renormalized susceptibility.

A conventional RPA-type analysis of interaction-induced particle-hole
instabilities shows that in a field the system develops an intra-valley spin
or charge $q=0$ order, which changes sign between the two valleys (this is
what we termed as an isospin valley-polarized order). The condition for the
instability is $1+V_{0}\Pi_{0,s}(0,0)=0$, where $s=\uparrow$ or $\downarrow$
with respect to $B$, and $\Pi_{0,s}(\nu,q)$ is a bare polarization bubble (see
Fig.2). Here $\nu$ and $q$ denote Matsubara frequency and momentum transfer,
and $\Pi_{0,s}(0,0)$, as defined, is negative. Below we assume, following the
experiments, that $|\Pi_{0,\uparrow}(0,0)|>|\Pi_{0,\downarrow}(0,0)|$, hence
the first instability at a non-zero $B$ is for fermions with spin parallel to
$B$.

Figure 3:  Frequency-dependent pairing interaction $S({\bar{\nu}})$, Eq. (6),
describing the universal contribution of a quantum-critical mode. The
nonmonotonic frequency dependence, which is a generic property of
$S({\bar{\nu}})$ for all values of the stiffness $K$, allows for a repulsive
interaction to generate an attractive effective pairing interaction. The value
at $\nu=0$ is finite at $K>0$ and zero at $K=0$. In the latter case
superconducting $T_{c}$ is non-zero for any $\delta_{\downarrow}$, the
distance to critical point for spin-down fermions.

The pairing interaction involves fermions with momenta ${\bf k}$ and $-{\bf
k}$, which in our case belong to different valleys $K$ and $K^{\prime}$. At
the lowest order, the pairing interaction is just $V_{0}$, however near the
onset of an isospin order dressing of the pairing interaction by particle-hole
bubbles is essential. We argue SM  that the relevant diagrams are the ones
shown in Fig.2, where each arrow represents the electron’s Green’s function
$G_{s}(\omega,{\boldsymbol{p}})=1/\left(i\omega-\epsilon_{s}({\boldsymbol{p}})\right)$.
The resulting effective interaction can be written as

$\displaystyle\Gamma_{ss}(\nu,q)=\gamma^{2}_{s}(\nu,q)V(\nu,q),~{}~{}\gamma_{s}(\nu,q)=\frac{1}{1+V_{0}\Pi_{0,s}(\nu,q)},$
$\displaystyle\quad
V(\nu,q)=\frac{V_{0}}{1-2V_{0}\sum_{s^{\prime}}\Pi_{0,s^{\prime}}(\nu,q)\gamma_{s^{\prime}}(\nu,q)}.$
(2)

Here and below we approach SC from a non-polarized state and assume valley
symmetry. Near the onset of valley polarization,
$V_{0}\Pi_{0,s}(\nu,q=0)\approx-1$, and at small momentum transfer $q\ll
k_{F}$ and small $\nu<v_{F}q$ the vertex-correction factors
$\gamma_{s}(\nu,{\boldsymbol{q}})$ take the familiar form:

$\gamma_{s}(\nu,{\boldsymbol{q}})\sim\frac{1}{\frac{|\nu|}{v_{F}q}+Kq^{2}+\delta_{s}},$
(3)

where $\delta_{s}$ is a distance to valley-polarization phase boundary for
fermions with a given spin projection. The stiffness $K$ and Fermi velocity
$v_{F}$, taken here to be spin-independent, are determined by band dispersion
$\epsilon({\boldsymbol{p}})$. At zero $B$,
$\delta_{\uparrow}=\delta_{\downarrow}$, the effective interaction is

$\Gamma_{ss}(\nu,q)=\frac{V_{0}\gamma(\nu,{\boldsymbol{q}})^{2}}{1-4V_{0}\Pi_{0}(\nu,q)\gamma(\nu,{\boldsymbol{q}})}\approx\frac{1}{4}\frac{V_{0}}{\frac{|\nu|}{v_{F}q}+Kq^{2}+\delta}.$

The sign of this interaction is a repulsion, in distinction to that found for
pairing mediated by a critical $q=0$ mode for QHE composite fermions Bonesteel
et al. (1996) and for a nematic QCPKlein et al. (2020); Oganesyan et al.
(2001); Lederer et al. (2015). In these systems electrons with ${\bf k}$ and
${\bf-k}$ live on the same Fermi surface and interact through an exchange
processes. In our case, such a process is forbidden as it requires a fermion
to scatter from one valley to the other. However, a nonzero $B$ field lifts
spin degeneracy and generates new frequency and momentum scales below which
the interaction $\Gamma_{ss}(\nu,q)$ gives rise to pairing, as we now
demonstrate.

Figure 4:  Critical temperature $T_{c}$ vs. the distance to criticality
$\delta_{\downarrow}$ for down spins, a parameter controlled by magnetic
field. For $K=0$, $T_{c}$ is nonvanishing for all $\delta$ (see text). For
$K\neq 0$, to the contrary, each curve starts at a finite threshold value
$\delta_{\downarrow}>0$. As $K$ grows, the threshold value first grows and
then decreases, reflecting the behavior of $S(\nu)$ at small $\nu$ shown in
Fig.3.

##  III  Valley-singlet $s$-wave pairing

A possibility of an $s-$wave pairing from nominally repulsive (positive)
interaction has been discussed several times in the literature both some time
ago Morel and Anderson (1962); McMillan (1968); Bogoljubov et al. (1958);
Coleman (2015) and recently Ruhman and Lee (2016); Chubukov et al. (2019);
Pimenov and Chubukov (2022). The idea is that if the interaction either has a
non-monotonic frequency dependence or is reduced at small frequencies, there
may emerge an effective attraction for a gap function that changes sign
between small and large frequencies, in analogy with how a nominally repulsive
interaction may become attractive in a non-$s-$wave spatial channel where a
gap changes sign between different regions on the Fermi surface. In our case,
at zero field the average interaction $\overline{\Gamma}_{ss}(\nu)$, which is
defined as $\Gamma_{ss}(\nu,{\boldsymbol{q}})$ averaged over momentum
transfers ${\boldsymbol{q}}$ on the Fermi surface, is a monotonically
decreasing function of frequency $\nu$, and a solution with sign-changing
$\Delta(\omega)$ is impossible. At a finite $B$, realistic bandstructure
calculations in Ref.Zhou et al. (2021c) show that the valley order sets first
for majority spin (spins along $B$), and near the onset of the order,
$\delta_{\uparrow}\ll\delta_{\downarrow}$ and
$\gamma_{\uparrow}\gg\gamma_{\downarrow}$. In this situation, the average
interaction for spin-up electrons $\overline{\Gamma}_{\uparrow\uparrow}(\nu)$
still monotonically decreases with $\nu$, but the average interaction for
spin-down electrons $\overline{\Gamma}_{\downarrow\downarrow}(\nu)$ becomes
non-monotonic. This happens because $\Gamma_{\downarrow\downarrow}(\nu,q)$ has
a linear rather than inverse linear dependence on
$1+V_{0}\Pi_{\uparrow}(\nu,q)$. In explicit form, the averaged interaction
$\overline{\Gamma}_{\downarrow\downarrow}(\nu)$ is

$\overline{\Gamma}_{\downarrow\downarrow}(\nu)=V_{0}\int_{0}^{k_{F}}\frac{dq}{2\pi}\frac{\frac{|\nu|}{v_{F}|q|}+Kq^{2}}{\left(\frac{|\nu|}{v_{F}|q|}+Kq^{2}+\delta_{\downarrow}\right)^{2}}$
(4)

The result can be cast into the scaling form
$\overline{\Gamma}_{\downarrow\downarrow}(\nu)=V_{0}(k_{F}/2\pi\delta_{\downarrow})S\left(\nu/\nu_{0},\nu_{1}/\nu_{0}\right)$,
where $\nu_{0}=2E_{F}\delta_{\downarrow}$, $\nu_{1}=2E_{F}(Kk^{2}_{F})$, and
$E_{F}=v_{F}k_{F}/2$. To see the consequence, consider first the case
$\nu_{1}\ll\nu_{0}$. The function
$S(x,0)=x(\log{\frac{1+x}{x}}-\frac{1}{1+x})$. is manifestly non-monotonic: it
is linear in $x$ at small $x$, passes through a maximum at $x\sim 0.5$, and
drops at higher $x$. We emphasize that the non-monotonic behavior is fully
induced by $B$, which splits $\delta_{\uparrow}$ and $\delta_{\downarrow}$.
The non-monotonic dependence holds if we increase the ratio $\nu_{1}/\nu_{0}$
as we show in Fig.3.

The gap equation for the pairing of spin-down fermions, mediated by
$\overline{\Gamma}_{\downarrow\downarrow}(\nu)$, is

$\Delta(\omega)=-\frac{T_{c}}{2v_{F}}\sum_{\omega^{\prime}=\pi
T_{c}(2n+1)}\frac{\Delta(\omega^{\prime})\overline{\Gamma}_{\downarrow\downarrow}(\omega-\omega^{\prime})}{|\omega^{\prime}|},$
(5)

The overall minus sign reflects that the interaction is repulsive. The gap
equation takes a universal form when expressed in terms dimensionless
${\bar{T}}_{c}=T_{c}/\nu_{0}$ and ${\bar{\omega}}=\omega/\nu_{0}$:

$\Delta({\bar{\omega}})=-\lambda\pi{\bar{T}}_{c}\sum_{\bar{\omega}^{\prime}=\pi{\bar{T}}_{c}(2n+1)}\frac{\Delta({\bar{\omega}}^{\prime})}{|\bar{\omega}^{\prime}|}S\left({\bar{\omega}}-{\bar{\omega}}^{\prime},0\right)$
(6)

where $\lambda=k_{F}V_{0}/(4\pi^{2}v_{F}\delta_{\downarrow})$. Because
$S\left({\bar{\omega}}-{\bar{\omega}}^{\prime},0\right)$ is strongly peaked at
$|{\bar{\omega}}-{\bar{\omega}}^{\prime}|={\bar{\nu}}_{*}\approx 0.5$, one can
change the overall sign in (6) by searching for gap functions which change
sign under ${\bar{\omega}}\to{\bar{\omega}}+{\bar{\nu}}_{*}$. At small
$\lambda$, analytical consideration yields
$T_{c}\propto\omega_{0}e^{-1/\lambda^{2}}$ (Ref.Chubukov et al. (2019)). At
$\lambda\leq 1$, $T_{c}\sim\omega_{0}$, but with a numerically small
prefactor. At larger $\lambda$, the prefactor increases and at $\lambda\gg 1$
(i.e., at small $\delta_{\downarrow}$, $T_{c}\sim\lambda\nu_{0}\sim E_{F}$.
For full consideration, at large $\lambda$ one also has to include fermionic
self-energy. In supplement, we estimate the effect of self-energy generated by
the pairing interaction, and find that the resulting suppression of $T_{c}$ is
acceptable. However, we emphasize that the self-energy comes from intravalley
interaction, which is different from the intervalley pairing interaction. They
only coincide with each other in mean-field theory. In reality, the self-
energy does not have to diverge when pairing interaction diverge. Therefore,
it is safe to ignore the role of self-energy. We show $T_{c}$, obtained by
numerical solution of Eq. (6), in Fig.4. We set $k_{F}V_{0}/(2\pi v_{F})=1$,
as required for a Stoner instability, and set $E_{F}=10\rm{meV}$. We see that
at $\nu_{1}=0$, $T_{c}$ monotonically increases with decreasing
$\delta_{\downarrow}$ and at small $\delta_{\downarrow}$ saturates at roughly
$1\rm{K}$.

For a more realistic case of $\nu_{0}\sim\nu_{1}$, the momentum-averaged
$\overline{\Gamma}_{\downarrow\downarrow}(\nu)$ tends to a finite value at
$\nu=0$, leading to a smaller $T_{c}$ and also setting a threshold on
$\delta_{\downarrow}$ as an $s-$wave pairing by a frequency-dependent
repulsion is a threshold phenomenon Ruhman and Lee (2016); Chubukov et al.
(2019); Pimenov and Chubukov (2022), and at a small $\delta_{\downarrow}$ the
non-monotonicity of ${\overline{\Gamma}}_{\downarrow\downarrow}(\nu)$ is too
weak to give rise to a pairing when the self-energy is included. At larger
$\delta_{\downarrow}$ $T_{c}$ also drops because the coupling $\lambda$ gets
smaller. This gives rise to a dome-like dependence of $T_{c}$ on
$\delta_{\downarrow}$ at a given $Kk_{F}^{2}$. For $Kk_{F}^{2}=10^{-2}$ we
obtained at $T_{c}\sim 35\rm{mK}$ at optimal $\delta_{\downarrow}=7\times
10^{-3}$. This value is in line with experimental $T_{c}$.

##  IV Valley-triplet $p$-wave pairing.

We now argue that at a finite $B$, the effective interaction, mediated by soft
isospin fluctuations, also gives rise to an attraction in another spin-triplet
channel, this time valley triplet and spatially odd. The mechanism here is
field-induced KL effect in 2D – the development of attractive spatial
component(s) due to screening of a purely repulsive bare pairing interaction
by particle-hole polarization bubbles. This pairing comes from momentum
transfers $q\sim 2k_{F}$, when there is no good theoretical reason to restrict
with only diagrams in Fig. (2). We assume without proof that Eq. (2) is still
valid, at least by order of magnitude, when $q\sim 2k_{F}$.

To understand field-induced KL effect, assume momentarily parabolic
$\epsilon_{i}(p)$ near $K$ and $K^{\prime}$ and consider static interaction
$\Gamma_{ss}(0,q)$. The free-fermion polarization in 2D is
$\Pi_{s}(0,q)=-(m/2\pi)$ for $q<2k_{F,s}$ and
$-(m/2\pi)\left(1-\sqrt{4k^{2}_{F,s}/q^{2}}\right)$ for $q>k_{F,s}$. Relevant
$q$ for SC are below $2k_{F,s}$ for a given spin projection. At $B=0$,
$k_{F,s}=k_{F}$ is the same for up- and down-spins. In this situation,
$\Pi_{s}(0,q)=-m/(2\pi)$ for relevant $q$, and the effective interaction
$\Gamma_{ss}(0,q)$ has only an $s$-wave repulsive component, like the bare
$V_{0}$. This is commonly known as the absence of KL effect in 2D for a
parabolic dispersion baranov1992. The situation changes at a finite $B$ as now
the effective interaction for fermions with spin-up partly comes from fermions
with spin-down and vice versa. Because the Fermi momentum $k_{F,\uparrow}$ is
larger than $k_{F,\downarrow}$, there is a range
$2k_{F,\downarrow}<q<2k_{F,\uparrow}$, where the interaction
$\Gamma_{\uparrow\uparrow}(0,q)$ for spin-up fermions at momentum transfer on
their Fermi surface acquires a momentum-dependence via the momentum dependence
of $\Pi_{\downarrow}(0,q)$. There is no such effect for
$\Gamma_{\downarrow\downarrow}(0,q)$ at $q<2k_{F,\downarrow}$.

Once $\Gamma_{\uparrow\uparrow}(0,q)$ becomes momentum-dependent, one can
search for spatially-odd solutions $\Delta(\theta)$, subject to $\int
d\theta\Delta(\theta)=0$ and $\Delta(\theta+\pi)=-\Delta(\theta)$, where
$\theta$ is an angle along the Fermi surface counted from, e.g., $k_{x}$
direction. These gap functions are necessary valley-triplets. The analysis of
the pairing instability is rather standard and we just present the result. We
find that the $q-$dependence of the interaction gives rise to an attraction
for spatially-odd $\Delta(\theta)$. At at a small field the gap equation is
approximately local in $\theta$, and yields $T_{c}\sim
E_{F}e^{-1/\lambda_{KL}}$, where

$\lambda_{KL}=\frac{mV_{0}}{8\pi^{2}\delta^{2}}\frac{\mu_{B}B}{E_{F}}$ (7)

We see that the field-induced $\lambda_{KL}$ is positive. At small $B$,
$T_{c}$ increases exponentially with the field. At a larger $B$, the prefactor
gets smaller as the number of down-spin fermions decreases. As a result,
$T_{c}$ has a dome-like shape as a function of $B$. We also note that Eq. (7)
is valid when $\lambda_{KL}<1$. At larger coupling, the coupling gets
renormalized by fermionic self-energy and eventually saturates. The analysis
can be straightforwardly extended to the physically relevant case
$\delta_{\uparrow}\ll\delta_{\downarrow}$, which one can model by non-equal
DOS for up and down spins. We found that Eq. (7) holds, but $\delta^{2}$ in
(7) has to be replaced by $\delta^{2}_{\downarrow}$. One can also move away
from parabolic dispersion and include the $q$-dependence of $\Pi(0,q)$ at
$q<2k_{F}$. Similarly to valley-singlet case, this will (i) decrease $T_{c}$
and (ii) set a finite threshold on a field as field-induced attraction has to
compete with a repulsive bare interaction in valley-triplet channel. As a
result, $T_{c}$ as a function of $B$ displays a dome-like behavior above a
finite threshold, much like for valley-singlet $s-$wave pairing.

##  V Relation to experiments

We now discuss several items related to experiments. First, in our analysis,
particularly of valley-singlet SC, we assumed that isospin order sets up first
for spin-up fermions. To see that our understanding of the phase transition is
correct, below we calculate the slope of the phase boundary, and comparing it
with experiment. According to our model, instability happens only in majority
spin. Therefore, at the phase transition, the density of carriers in the
majority spin is a fixed value, while the density of carrier in minority spin
depends on $B$ field. Specifically, we expect that the phase transition shifts
towards lower total carrier density under increasing $B$. The shift of carrier
density is linear in magnetic field:

$\frac{dn_{*}}{dB}=2\mu_{B}\nu_{0},$ (8)

where $n_{*}$ is the total carrier density at the phase boundary, $\mu_{B}$ is
the Bohr magneton, $\nu_{0}$ is the density of states per isospin, the factor
of $2$ arises from $K$/$K^{\prime}$ valley degeneracy. Plugging in the value
of the density of states obtained numerically in Ref.Zhou et al. (2021c), we
find the slope is $\frac{dn_{*}}{dB}=5\times
10^{-4}\rm{nm}^{-2}\rm{meV}^{-1}$, which matches the slope extracted from
Fig.1c).

Second, the two pairing scenarios that we discussed yield dome-shaped
$T_{c}(B)$ with a threshold on $B$, but differ in which spin components pair:
valley-singlet pairing involves spins opposite to the field, while valley-
triplet pairing involves spins along the field. One way to test which spin
components are involved in SC is to measure the DC voltage drop when injecting
a spin-polarized current into the system. If we inject electrons from a
ferromagnetic material which is polarized by the same in-plane magnetic field
as in the BLG, then our theory predicts that for valley-singlet pairing this
spin-polarized current should give a finite DC voltage drop even $T_{c}$
because the SC only occurs in the Fermi sea of the opposite spin polarization,
while for valley-triplet pairing a DC voltage drop should disappear below
$T_{c}$.

Third, valley-singlet pairing arises from small-momentum part scattering, and
should be sensitive to screening. When a metallic gate is brought closer to
the sample, the Coulomb interaction will be suppressed. As a result, the width
(in terms of density) of the SC phase should narrow when a proximal metal gate
is applied, because the same value of $T_{c}$ can be achieved only by getting
closer to the phase boundary. For valley-triplet pairing, this effect is
smaller as the pairing is not confined to small-momentum scattering.

Finally, can the quantum-critical mode contribute to resistivity through
carrier scattering by thermal fluctuations? This may seem plausible at a first
glance, yet in our scenario a strong effective interaction, mediated by a soft
boson, holds for small momentum transfers $q\ll k_{F}$ due to the proximity to
the $q=0$ isospin order. As a result, despite thermal fluctuations being
strong, forward scattering does not produce a relaxation of current and thus
does not contribute to resistivity Maslov et al. (2011). This is in line with
the experiment where $T$ dependence of resistivity shows no signature of
critical fluctuations near the phase transitionZhou et al. (2021c).

We therefore conclude that all the unique aspects of the observed
superconductivity are successfully explained by the attraction-from-repulsion-
based pairing scenario. Furthermore, this mechanism is ‘natural’ as it arises
from the strong electron-electron interactions that drive the adjacent
isospin-polarized electron orders. As such, it constitutes a unique verifiable
instance of exotic pairing. Supported by experiments, it sheds light on the
origin of spin-triplet superconductivity in BBG and is applicable to a variety
of other systems of interest.

## References

  * Bistritzer and MacDonald (2011) R. Bistritzer and A. H. MacDonald, Proceedings of the National Academy of Sciences 108, 12233 (2011), ISSN 0027-8424, eprint https://www.pnas.org/content/108/30/12233.full.pdf, URL https://www.pnas.org/content/108/30/12233.
  * Cao et al. (2018) Y. Cao, V. Fatemi, S. Fang, K. Watanabe, T. Taniguchi, E. Kaxiras, and P. Jarillo-Herrero, Nature 556, 43–50 (2018), ISSN 1476-4687, URL http://dx.doi.org/10.1038/nature26160.
  * Lu et al. (2019) X. Lu, P. Stepanov, W. Yang, M. Xie, M. A. Aamir, I. Das, C. Urgell, K. Watanabe, T. Taniguchi, G. Zhang, et al., Nature 574, 653 (2019).
  * Andrei and MacDonald (2020) E. Y. Andrei and A. H. MacDonald, Nature materials 19, 1265 (2020).
  * Saito et al. (2020) Y. Saito, J. Ge, K. Watanabe, T. Taniguchi, and A. F. Young, Nature Physics 16, 926 (2020).
  * Polshyn et al. (2019) H. Polshyn, M. Yankowitz, S. Chen, Y. Zhang, K. Watanabe, T. Taniguchi, C. R. Dean, and A. F. Young, Nature Physics 15, 1011 (2019).
  * Oh et al. (2021) M. Oh, K. P. Nuckolls, D. Wong, R. L. Lee, X. Liu, K. Watanabe, T. Taniguchi, and A. Yazdani, Nature 600, 240 (2021), URL https://doi.org/10.1038%2Fs41586-021-04121-x.
  * Cao et al. (2021) Y. Cao, D. Rodan-Legrain, J. M. Park, N. F. Q. Yuan, K. Watanabe, T. Taniguchi, R. M. Fernandes, L. Fu, and P. Jarillo-Herrero, Science 372, 264 (2021), URL https://doi.org/10.1126%2Fscience.abc2836.
  * Jaoui et al. (2022) A. Jaoui, I. Das, G. Di Battista, J. Díez-Mérida, X. Lu, K. Watanabe, T. Taniguchi, H. Ishizuka, L. Levitov, and D. K. Efetov, Nature Physics pp. 1–6 (2022).
  * You and Vishwanath (2019) Y.-Z. You and A. Vishwanath, npj Quantum Materials 4, 1 (2019).
  * Kozii et al. (2019) V. Kozii, H. Isobe, J. W. F. Venderbos, and L. Fu, Phys. Rev. B 99, 144507 (2019), URL https://link.aps.org/doi/10.1103/PhysRevB.99.144507.
  * Chichinadze et al. (2020) D. V. Chichinadze, L. Classen, and A. V. Chubukov, Phys. Rev. B 101, 224513 (2020), URL https://link.aps.org/doi/10.1103/PhysRevB.101.224513.
  * Khalaf et al. (2021) E. Khalaf, S. Chatterjee, N. Bultinck, M. P. Zaletel, and A. Vishwanath, Science advances 7, eabf5299 (2021).
  * Saito et al. (2019) Y. Saito, J. Ge, K. Watanabe, T. Taniguchi, and A. F. Young, arXiv preprint arXiv:1911.13302 (2019).
  * Stepanov et al. (2020) P. Stepanov, I. Das, X. Lu, A. Fahimniya, K. Watanabe, T. Taniguchi, F. H. Koppens, J. Lischner, L. Levitov, and D. K. Efetov, Nature 583, 375 (2020).
  * Wu et al. (2018) F. Wu, A. H. MacDonald, and I. Martin, Phys. Rev. Lett. 121, 257001 (2018), URL https://link.aps.org/doi/10.1103/PhysRevLett.121.257001.
  * Lian et al. (2019) B. Lian, Z. Wang, and B. A. Bernevig, Phys. Rev. Lett. 122, 257002 (2019), URL https://link.aps.org/doi/10.1103/PhysRevLett.122.257002.
  * Wu et al. (2019) F. Wu, E. Hwang, and S. Das Sarma, Phys. Rev. B 99, 165112 (2019), URL https://link.aps.org/doi/10.1103/PhysRevB.99.165112.
  * Liu et al. (2021) X. Liu, Z. Wang, K. Watanabe, T. Taniguchi, O. Vafek, and J. Li, Science 371, 1261 (2021).
  * Tarnopolsky et al. (2019) G. Tarnopolsky, A. J. Kruchkov, and A. Vishwanath, Phys. Rev. Lett. 122, 106405 (2019), URL https://link.aps.org/doi/10.1103/PhysRevLett.122.106405.
  * Zhou et al. (2021a) H. Zhou, T. Xie, A. Ghazaryan, T. Holder, J. R. Ehrets, E. M. Spanton, T. Taniguchi, K. Watanabe, E. Berg, M. Serbyn, et al., Nature 598, 429–433 (2021a), ISSN 1476-4687, URL http://dx.doi.org/10.1038/s41586-021-03938-w.
  * Zhou et al. (2021b) H. Zhou, T. Xie, T. Taniguchi, K. Watanabe, and A. F. Young, Nature 598, 434 (2021b).
  * Zhou et al. (2021c) H. Zhou, Y. Saito, L. Cohen, W. Huynh, C. L. Patterson, F. Yang, T. Taniguchi, K. Watanabe, and A. F. Young, arXiv preprint arXiv:2110.11317 (2021c).
  * de la Barrera et al. (2021) S. C. de la Barrera, S. Aronson, Z. Zheng, K. Watanabe, T. Taniguchi, Q. Ma, P. Jarillo-Herrero, and R. Ashoori, arXiv preprint arXiv:2110.13907 (2021).
  * McCann and Koshino (2013) E. McCann and M. Koshino, Reports on Progress in Physics 76, 056503 (2013), URL https://doi.org/10.1088/0034-4885/76/5/056503.
  * McCann and Fal’ko (2006) E. McCann and V. I. Fal’ko, Phys. Rev. Lett. 96, 086805 (2006), URL https://link.aps.org/doi/10.1103/PhysRevLett.96.086805.
  * Nandkishore and Levitov (2010a) R. Nandkishore and L. Levitov, Phys. Rev. Lett. 104, 156803 (2010a), URL https://link.aps.org/doi/10.1103/PhysRevLett.104.156803.
  * Nandkishore and Levitov (2010b) R. Nandkishore and L. Levitov, Phys. Rev. B 82, 115124 (2010b), URL https://link.aps.org/doi/10.1103/PhysRevB.82.115124.
  * Vafek and Yang (2010) O. Vafek and K. Yang, Phys. Rev. B 81, 041401 (2010), URL https://link.aps.org/doi/10.1103/PhysRevB.81.041401.
  * Jung et al. (2011) J. Jung, F. Zhang, and A. H. MacDonald, Phys. Rev. B 83, 115408 (2011), URL https://link.aps.org/doi/10.1103/PhysRevB.83.115408.
  * MacDonald et al. (2012) A. H. MacDonald, J. Jung, and F. Zhang, Physica Scripta 2012, 014012 (2012).
  * Zhang and MacDonald (2012) F. Zhang and A. H. MacDonald, Phys. Rev. Lett. 108, 186804 (2012), URL https://link.aps.org/doi/10.1103/PhysRevLett.108.186804.
  * Cvetkovic et al. (2012) V. Cvetkovic, R. E. Throckmorton, and O. Vafek, Phys. Rev. B 86, 075467 (2012), URL https://link.aps.org/doi/10.1103/PhysRevB.86.075467.
  * Throckmorton and Das Sarma (2014) R. E. Throckmorton and S. Das Sarma, Phys. Rev. B 90, 205407 (2014), URL https://link.aps.org/doi/10.1103/PhysRevB.90.205407.
  * Min et al. (2008) H. Min, G. Borghi, M. Polini, and A. H. MacDonald, Phys. Rev. B 77, 041407 (2008), URL https://link.aps.org/doi/10.1103/PhysRevB.77.041407.
  * Nilsson et al. (2006) J. Nilsson, A. H. Castro Neto, N. M. R. Peres, and F. Guinea, Phys. Rev. B 73, 214418 (2006), URL https://link.aps.org/doi/10.1103/PhysRevB.73.214418.
  * Martin et al. (2010) J. Martin, B. E. Feldman, R. T. Weitz, M. T. Allen, and A. Yacoby, Phys. Rev. Lett. 105, 256806 (2010), URL https://link.aps.org/doi/10.1103/PhysRevLett.105.256806.
  * Weitz et al. (2010) R. T. Weitz, M. T. Allen, B. E. Feldman, J. Martin, and A. Yacoby, Science 330, 812 (2010).
  * Mayorov et al. (2011) A. S. Mayorov, D. C. Elias, M. Mucha-Kruczynski, R. V. Gorbachev, T. Tudorovskiy, A. Zhukov, S. V. Morozov, M. I. Katsnelson, null null, A. K. Geim, et al., Science 333, 860 (2011), eprint https://www.science.org/doi/pdf/10.1126/science.1208683, URL https://www.science.org/doi/abs/10.1126/science.1208683.
  * Veligura et al. (2012) A. Veligura, H. J. van Elferen, N. Tombros, J. C. Maan, U. Zeitler, and B. J. van Wees, Phys. Rev. B 85, 155412 (2012), URL https://link.aps.org/doi/10.1103/PhysRevB.85.155412.
  * Velasco et al. (2012) J. Velasco, L. Jing, W. Bao, Y. Lee, P. Kratz, V. Aji, M. Bockrath, C. Lau, C. Varma, R. Stillwell, et al., Nature nanotechnology 7, 156 (2012).
  * Bao et al. (2012) W. Bao, J. Velasco, F. Zhang, L. Jing, B. Standley, D. Smirnov, M. Bockrath, A. H. MacDonald, and C. N. Lau, Proceedings of the National Academy of Sciences 109, 10802 (2012), eprint https://www.pnas.org/doi/pdf/10.1073/pnas.1205978109, URL https://www.pnas.org/doi/abs/10.1073/pnas.1205978109.
  * Freitag et al. (2012) F. Freitag, J. Trbovic, M. Weiss, and C. Schönenberger, Phys. Rev. Lett. 108, 076602 (2012), URL https://link.aps.org/doi/10.1103/PhysRevLett.108.076602.
  * Seiler et al. (2021) A. M. Seiler, F. R. Geisenhof, F. Winterer, K. Watanabe, T. Taniguchi, T. Xu, F. Zhang, and R. T. Weitz, arXiv preprint arXiv:2111.06413 (2021).
  * McCann (2006) E. McCann, Phys. Rev. B 74, 161403 (2006), URL https://link.aps.org/doi/10.1103/PhysRevB.74.161403.
  * Saito et al. (2021) Y. Saito, F. Yang, J. Ge, X. Liu, T. Taniguchi, K. Watanabe, J. Li, E. Berg, and A. F. Young, Nature 592, 220 (2021).
  * Zondiner et al. (2020) U. Zondiner, A. Rozen, D. Rodan-Legrain, Y. Cao, R. Queiroz, T. Taniguchi, K. Watanabe, Y. Oreg, F. von Oppen, A. Stern, et al., Nature 582, 203 (2020).
  * Rozen et al. (2021) A. Rozen, J. M. Park, U. Zondiner, Y. Cao, D. Rodan-Legrain, T. Taniguchi, K. Watanabe, Y. Oreg, A. Stern, E. Berg, et al., Nature 592, 214 (2021).
  * Choi et al. (2021) Y. Choi, H. Kim, Y. Peng, A. Thomson, C. Lewandowski, R. Polski, Y. Zhang, H. S. Arora, K. Watanabe, T. Taniguchi, et al., Nature 589, 536 (2021).
  * Pierce et al. (2021) A. T. Pierce, Y. Xie, J. M. Park, E. Khalaf, S. H. Lee, Y. Cao, D. E. Parker, P. R. Forrester, S. Chen, K. Watanabe, et al., Nature Physics 17, 1210 (2021).
  * Jung et al. (2015) J. Jung, M. Polini, and A. H. MacDonald, Phys. Rev. B 91, 155423 (2015), URL https://link.aps.org/doi/10.1103/PhysRevB.91.155423.
  * Dong et al. (2021) Z. Dong, M. Davydova, M. Ogunnaike, and L. Levitov, arXiv preprint arXiv:2110.15254 (2021).
  * Ghazaryan et al. (2021) A. Ghazaryan, T. Holder, M. Serbyn, and E. Berg, Phys. Rev. Lett. 127, 247001 (2021), URL https://link.aps.org/doi/10.1103/PhysRevLett.127.247001.
  * Chatterjee et al. (2021) S. Chatterjee, T. Wang, E. Berg, and M. P. Zaletel, arXiv preprint arXiv:2109.00002 (2021).
  * You and Vishwanath (2022) Y.-Z. You and A. Vishwanath, Phys. Rev. B 105, 134524 (2022), URL https://link.aps.org/doi/10.1103/PhysRevB.105.134524.
  * Chou et al. (2022) Y.-Z. Chou, F. Wu, J. D. Sau, and S. Das Sarma, Phys. Rev. B 105, L100503 (2022), URL https://link.aps.org/doi/10.1103/PhysRevB.105.L100503.
  * Klein et al. (2020) A. Klein, A. V. Chubukov, Y. Schattner, and E. Berg, Phys. Rev. X 10, 031053 (2020), URL https://link.aps.org/doi/10.1103/PhysRevX.10.031053.
  * Oganesyan et al. (2001) V. Oganesyan, S. A. Kivelson, and E. Fradkin, Phys. Rev. B 64, 195109 (2001), URL https://link.aps.org/doi/10.1103/PhysRevB.64.195109.
  * Lederer et al. (2015) S. Lederer, Y. Schattner, E. Berg, and S. A. Kivelson, Phys. Rev. Lett. 114, 097001 (2015), URL https://link.aps.org/doi/10.1103/PhysRevLett.114.097001.
  * Tremblay (2013) A.-M. Tremblay, Emergent Phenomena in Correlated Matter Modeling and Simulation 3 (2013).
  * Chubukov et al. (2020) A. V. Chubukov, A. Abanov, Y. Wang, and Y.-M. Wu, Annals of Physics p. 168142 (2020), ISSN 0003-4916, URL http://www.sciencedirect.com/science/article/pii/S0003491620300750.
  * Berg et al. (2019) E. Berg, S. Lederer, Y. Schattner, and S. Trebst, Annual Review of Condensed Matter Physics 10, 63 (2019), URL https://doi.org/10.1146/annurev-conmatphys-031218-013339.
  * Lee (2018) S.-S. Lee, Annual Review of Condensed Matter Physics 9, 227 (2018), URL https://doi.org/10.1146/annurev-conmatphys-031016-025531.
  * Efetov et al. (2013) K. B. Efetov, H. Meier, and C. Pepin, Nature Physics 9, 442 (2013).
  * Mazin et al. (2008) I. I. Mazin, D. J. Singh, M. D. Johannes, and M. H. Du, Phys. Rev. Lett. 101, 057003 (2008), URL https://link.aps.org/doi/10.1103/PhysRevLett.101.057003.
  * Mazin and Schmalian (2009) I. Mazin and J. Schmalian, Physica C: Superconductivity 469, 614 (2009), ISSN 0921-4534, superconductivity in Iron-Pnictides, URL https://www.sciencedirect.com/science/article/pii/S0921453409001002.
  * Scalapino et al. (1966) D. J. Scalapino, J. R. Schrieffer, and J. W. Wilkins, Phys. Rev. 148, 263 (1966), URL https://link.aps.org/doi/10.1103/PhysRev.148.263.
  * (68) _Supplemental material [url will be inserted by publisher]._
  * Morel and Anderson (1962) P. Morel and P. W. Anderson, Phys. Rev. 125, 1263 (1962), URL https://link.aps.org/doi/10.1103/PhysRev.125.1263.
  * McMillan (1968) W. L. McMillan, Phys. Rev. 167, 331 (1968), URL https://link.aps.org/doi/10.1103/PhysRev.167.331.
  * Bogoljubov et al. (1958) N. N. Bogoljubov, V. V. Tolmachov, and D. V. Širkov, Fortschritte der Physik 6, 605 (1958), eprint https://onlinelibrary.wiley.com/doi/pdf/10.1002/prop.19580061102, URL https://onlinelibrary.wiley.com/doi/abs/10.1002/prop.19580061102.
  * Coleman (2015) P. Coleman, _Introduction to Many-Body Physics_ (Cambridge University Press, 2015), ISBN 9781139020916, URL https://doi.org/10.1017/cbo9781139020916.
  * Chubukov et al. (2019) A. Chubukov, N. V. Prokof’ev, and B. V. Svistunov, Phys. Rev. B 100, 064513 (2019), URL https://link.aps.org/doi/10.1103/PhysRevB.100.064513.
  * Kohn and Luttinger (1965) W. Kohn and J. M. Luttinger, Phys. Rev. Lett. 15, 524 (1965), URL https://link.aps.org/doi/10.1103/PhysRevLett.15.524.
  * Bonesteel et al. (1996) N. E. Bonesteel, I. A. McDonald, and C. Nayak, Phys. Rev. Lett. 77, 3009 (1996), URL https://link.aps.org/doi/10.1103/PhysRevLett.77.3009.
  * Ruhman and Lee (2016) J. Ruhman and P. A. Lee, Phys. Rev. B 94, 224515 (2016), URL https://link.aps.org/doi/10.1103/PhysRevB.94.224515.
  * Pimenov and Chubukov (2022) D. Pimenov and A. V. Chubukov, npj Quantum Materials 7, 1 (2022).
  * Maslov et al. (2011) D. L. Maslov, V. I. Yudson, and A. V. Chubukov, Phys. Rev. Lett. 106, 106403 (2011), URL https://link.aps.org/doi/10.1103/PhysRevLett.106.106403.
  * Chubukov and Wölfle (2014) A. V. Chubukov and P. Wölfle, Phys. Rev. B 89, 045108 (2014), URL https://link.aps.org/doi/10.1103/PhysRevB.89.045108.

##  Appendix A Diagrammatic derivation of the effective interaction.

It is instructive to analyze how the diagrammatic series, which we use to
obtain the effective pairing interaction $\Gamma_{ss}(\nu,q)$, emerge in
order-by-order expansion in $V_{0}$.

By general rules, the vertex function $\Gamma_{ss^{\prime}}(\nu,q)$ is the
fully dressed antisymmetrized interaction, irreducible in a particular
channel. For pairing we need irreducible interaction between fermions with
momenta $(k,-k;p,-p)$. The specifics of our case is that (i) pairing involves
one fermion near $K$ and one near $K^{\prime}$, and (ii) the scattering from
$K$ to $K^{\prime}$ is weak in graphene and can be neglected. One can check
that in this situation the contributions to $\Gamma_{ss}(\nu,q)$ from
antisymmetrization vanish, and we can restrict with just dressed interaction
$V_{0}$. As we are interested in pairing in a $B$ field, we focus on the
vertex function with equal spin projections, $\Gamma_{ss}(\nu,q)$.

To first order in $V_{0}$ the vertex function is just a constant $V_{0}$. To
second-order we have three topologically different sets of diagrams, shown in
Fig.5 (they are often called KL diagrams). The diagrams of the first two sets
(bubble and ”wine glass” diagrams) are expressed via the polarization bubble
at momenta $q=({\bf k}-{\bf p})$, the last ”exchange” diagram is expressed via
the polarization bubble at momenta $q_{+}={\bf k}+{\bf p}$. We are primarily
interested in the dressed interaction at small momentum transfer as we expect
that this interaction gets enhanced near the onset of $q=0$ isospin order.
Accordingly, at each order of expansion in $V_{0}$ we will only use diagrams,
which contain polarization bubbles with $q$ and neglect diagrams, which
contain bubbles with $q_{+}$ (see Ref.Chubukov and Wölfle (2014) for similar
consideration). The corresponding diagrams at 3-loop order are shown in Fig.6
They contain diagrams with zero, one, and two bubbles, and one diagram with
interaction line inserted into the bubble.

This structure suggests the way to sum up relevant diagrams by separating them
into sub-classes with different number of bubbles, and within each subclass
inserting all possible interaction lines into the bubbles. A small
experimentation shows that this leads to diagrammatic series shown in Fig.2.
Diagrams without bubbles sum up into $V_{0}\gamma^{2}_{s}(\nu,q)$, where
$\gamma_{s}(\nu,q)$ is given by a ladder series of $V_{0}\Pi_{s}(\nu,q)$. An
insertion of a bare bubble brings the factor
$-2V_{0}\left(\Pi_{\uparrow}(\nu,q)+\Pi_{\downarrow}(\nu,q)\right)$, where
$-1$ is due to a loop and the factor $2$ comes about because intermediate
fermions can be near $K$ or near $K^{\prime}$ no matter where external
fermions are. The insertion of ladder series of interactions into each bubble
further multiples $\Pi_{s}(\nu,q)$ by $\gamma_{s}(\nu,q)$. The end result is
Eq. (2) in the main text.

Figure 5: Two types of second-order diagrams that are enhanced near the onset
of $q=0$ Stoner order. a) bubble diagram b) “wine glass” diagram c) “exchange”
diagram

Figure 6: Diagrams at 3-loop order. a) diagrams with 3 bubbles b) diagrams
with 0,1 or 2, which are subleading near the Stoner transition.

##  Appendix B Self-energy

In this section, we elaborate on the effect of self-energy correction, and
show that it is unimportant. Here, we focus on the case of $\nu_{1}=0$. The
self-energy for spin-down electrons is given by

$\displaystyle\Sigma_{\downarrow\downarrow}(\omega)$
$\displaystyle=\int\frac{d\nu}{2\pi}\int\frac{dq_{\perp}}{2\pi}\frac{\overline{\Gamma}_{\downarrow\downarrow}(\nu,q)}{i(\omega+\nu)-v_{F}q_{\perp}}$
(11) $\displaystyle=\frac{\lambda}{2}\int\frac{d\nu}{2\pi}{\rm
sgn}(\nu+\omega)S(\nu/\nu_{0})$
$\displaystyle=\lambda\nu_{0}\int_{0}^{{\bar{\omega}}}dxS(x)$

To see the relevant scale of $\omega$ in pairing problem, we look at the gap
equation without self-energy correction (6). We rewrite it as follows:

$\Delta(n)=-\frac{\lambda}{2}\sum_{n^{\prime}}\frac{\Delta(n^{\prime})}{|2n^{\prime}+1|}S\left(2\pi{\bar{T}}_{c}\left(n-n^{\prime}\right),0\right)$
(12)

where we have rewritten $\bar{\omega}=\pi(2n+1){\bar{T}_{c}}$,
$\bar{\omega}^{\prime}=\pi(2n^{\prime}+1){\bar{T}_{c}}$. As shown by numerics
in the main text, the critical temperature is
$T_{c}\sim\frac{1}{2}\lambda\nu_{0}\times 10^{-2}$. For extremely large value
of $\lambda\gg 10^{2}$ (i.e. extremely small $\delta_{\downarrow}$), the
relevant $n,n^{\prime}\sim O(1)$. Replacing $S(x)$ with its asymptotic form
$\frac{1}{2x}$, we find numerically that the wavefunction stop changing sign
at $n=4$. The self-energy relevant for this pairing problem should be
evaluated at $\omega=9\pi T_{c}$:

$\Sigma_{\downarrow\downarrow}(\omega\sim 9\pi
T_{c})\sim\lambda\nu_{0}\ln(\lambda)$ (13)

where logarithm comes from integrating $S(x)$ which scales as $\frac{1}{2x}$
at large $x$. Then, we find the effective coupling scales with $\lambda$ as

$\tilde{\lambda}=\frac{\lambda}{1+\kappa\ln\lambda},\quad\kappa\sim\frac{100}{9\pi}=3.5$
(14)

The denominator is only marginally relevant at large $\lambda$, thus does not
suppress the $T_{c}$ substantially.

For a not-so-large $\lambda$ value ($\lambda<10^{2}$), the relevant value of
$n,n^{\prime}$ the equation above is
$n,n^{\prime}\lesssim\nu_{0}/T_{c}=10^{2}/\lambda$. The self-energy relevant
for this pairing problem should be evaluated at $\omega\lesssim\nu_{0}$:

$\Sigma_{\downarrow\downarrow}(\omega\sim\nu_{0})\sim 0.2\lambda\nu_{0}$ (15)

where the numerical factor comes from integrating $S(x)$ below the turning
pint $x\sim 0.5$. We find the effective coupling scales with $\lambda$ as

$\tilde{\lambda}=\frac{\lambda}{1+0.2\lambda}.$ (16)

In this case, carrying out the simulation as Fig.4 of main text, we get
$T_{c}\sim 10^{-4}$, which is much smaller than the value in Fig.4, but is
still of an acceptable orders of magnitude.


# LLM as an Art Director (LaDi): Using LLM’s to improve Text-to-Media
Generators

Allen Roush
Plai Labs
<EMAIL_ADDRESS>
&Emil Zakirov
Plai Labs
<EMAIL_ADDRESS>
&Artemiy Shirokov
Plai Labs
<EMAIL_ADDRESS>
&Polina Lunina
Plai Labs
<EMAIL_ADDRESS>
&Jack Gane
Plai Labs
<EMAIL_ADDRESS>
&Alexander Duffy
Plai Labs
<EMAIL_ADDRESS>
&Charlie Basil
Plai Labs
<EMAIL_ADDRESS>
&Aber Whitcomb
CTO, Plai Labs
<EMAIL_ADDRESS>
&Jim Benedetto
Plai Labs
<EMAIL_ADDRESS>
&Chris DeWolfe
CEO, Plai Labs
<EMAIL_ADDRESS>
###### Abstract

Recent advancements in text-to-image generation have revolutionized numerous
fields, including art and cinema, by automating the generation of high-
quality, context-aware images and video. However, the utility of these
technologies is often limited by the inadequacy of text prompts in guiding the
generator to produce artistically coherent and subject-relevant images. In
this paper, We describe the techniques that can be used to make Large Language
Models (LLMs) act as Art Directors that enhance image and video generation. We
describe our unified system for this called "LaDi". We explore how LaDi
integrates multiple techniques for augmenting the capabilities of text-to-
image generators (T2Is) and text-to-video generators (T2Vs), with a focus on
constrained decoding, intelligent prompting, fine-tuning, and retrieval. LaDi
and these techniques are being used today in apps and platforms developed by
Plai Labs.

##  1 Introduction

###  1.1 Background

The landscape of image generation has experienced a paradigm shift owing to
the emergence of advanced text-to-image models like Stable Diffusion and
DALL-E. These models have proven to be indispensable in a variety of
applications, ranging from art and cinema to digital marketing and virtual
simulations. However, a persistent limitation hindering their utility is the
quality of text prompts employed to guide image generation. While the raw
power of text-to-image generators is undeniably remarkable, the absence of
sophisticated prompting techniques often results in images that are
technically impressive but lack subject relevance or artistic coherence.

The crux of the issue lies in the text prompts themselves. While text-to-image
models boast impressive technical prowess, their output is only as good as the
input prompt. Vague, ambiguous, or poorly composed prompts often yield images
that, while visually striking, are irrelevant or nonsensical. The art of
prompting is complex; striking the right balance of detail versus brevity
while conveying subtle attributes like tone, composition, and style is non-
trivial even for humans.

In light of these challenges, this paper introduces a groundbreaking framework
called "LaDi" in which Large Language Models (LLMs) serve as Art Directors for
text-to-image generators. Instead of relying on simple, direct text prompts
like "a two-story pink house," our approach leverages the nuanced
understanding of LLMs to construct more elaborate, descriptive prompts that
could translate to "a two-story Victorian house bathed in the rosy hues of a
sunset, complete with a garden in full bloom." Such intelligent prompting
brings a level of depth and specificity that has been largely missing in
automated image generation.

We extend this concept to include constrained decoding and specialized tool
usage, effectively setting parameters that the image must conform to.
Constrained decoding allows for the enforcement of certain conditions—such as
maintaining a consistent color scheme or ensuring the presence of specific
elements—thus making the generated image both predictable and relevant.

In this paper, we describe the techniques that we discovered at Plai Labs to
create an effective LLM based "Art Director" called LaDi. LaDi is used
throughout the social media platform "PlaiDay" and will be used in all future
services.

###  1.2 Text-to-Image Models

The growing capability of text-to-image generation models has engendered a
profound impact across multiple domains, from content creation and digital art
to medical imaging and beyond. One of the most notable advancements in this
sphere has been the development of Stable Diffusion from [17], a framework
grounded in latent diffusion models (LDMs) crafted by the CompVis group at LMU
Munich. By integrating variational autoencoders (VAEs), U-Net architectures,
and text encoders, Stable Diffusion provides a robust yet efficient solution
to the challenge of generating contextually rich and visually captivating
images.

We choose Stable Diffusion as our main T2I model because of its open-source
nature and huge amount of custom community content/extensions. We believe that
with this custom content, Stable Diffusion is superior to competitor models.

While Stable Diffusion offers increased computational efficiency for training
and generation, it still faces challenges, particularly in the realm of text
prompting. Prompting is an art in itself—providing the model with just the
right amount of context to generate images that are not only visually
appealing but also semantically meaningful. The utilization of a fixed,
pretrained CLIP ViT-L/14 text encoder aids in transforming text prompts to an
embedding space; however, the process is far from trivial. Effectively
managing the balance between under-specification and over-specification in
text prompts becomes a crucial factor in achieving desired outputs.

Moreover, Stable Diffusion has its limitations, such as issues with
degradation and inaccuracies under certain conditions, complexities in
adapting the model for novel use-cases, and susceptibility to algorithmic
biases. Despite these challenges, its relative lightweight nature—in terms of
computational resources—and its flexibility make it a cornerstone in the
current landscape of text-to-image generation models.

###  1.3 Text-to-Video Models

Text-to-video synthesis has seen remarkable progress in recent years, with
models capable of generating increasingly high-quality and controllable video
content from text prompts. Several key innovations have catalyzed these
advancements.

[23] introduced ModelScopeT2V which uses spatio-temporal blocks to capture
complex dependencies between frames. By training on both image-text and video-
text datasets, ModelScopeT2V enhances semantic diversity. It achieves state-
of-the-art performance on established benchmarks, serving as an impactful
baseline.

[5] introduced Imagen Video which demonstrates the effectiveness of cascaded
diffusion models for high-definition video generation. It combines a text
encoder, base video model, and interleaved spatial and temporal super-
resolution models, totaling 11.6B parameters. The model exhibits strong
temporal consistency, while inheriting Imagen’s image generation capabilities.

[21] introduced Make-A-Video which pioneers video generation without paired
video-text data. It creatively adapts image-to-image diffusion models for
frame generation and employs noise conditioning to enable video-length
sampling. Despite no video supervision, Make-A-Video produces videos aligned
with text prompts.

These models have expanded the horizons of controllable video synthesis.
Looking ahead, potential areas of improvement include longer video durations,
3D scene consistency, and multimodal control. An exciting new frontier is the
combination of language, audio, and video in unified generative models. Text-
to-video generation remains an active area of research.

AnimateDiff, introduced by [4], facilitates the conversion of any personalized
T2I model into an animation generator without requiring model-specific tuning.
Built upon motion modeling modules trained on extensive video datasets,
AnimateDiff can be integrated into existing T2I models to produce animated
clips that are both temporally smooth and visually consistent. This framework
succeeds in democratizing animation generation just as prior T2I models have
done for static image generation.

AnimateDiff was originally released with two "motion modules", which are
models that take an input sequence of images and generates a video with motion
connecting the images. Subsequently, many community contributed motion moduels
have been released. These include "Motion Loras" analogous to Stable Diffusion
"Lora" models (low rank adapters) which modify an existing motion module.

AnimateDiff became the dominant form of AI video generation at the end of 2023
because of its speed, support for controlnet models (which make consistent
video possible), and because users realized they could create any length video
by generating more images and using a sliding window input context.

Despite its revolutionary capabilities, AnimateDiff is not without its
challenges. One of the key difficulties lies in the act of prompting—how do we
accurately instruct the model to generate content that not only captures the
nuances of the text but also transitions smoothly in the animation? General
text-to-video generation techniques have often found it challenging to manage
the sensitivity of hyperparameters, the collection of personalized video, and
the computational resources required, making the task of generating high-
quality animated content from text not as straightforward.

##  2 Prior Work

Using language models to assist with generating prompts for image generators
is not a new idea. In this section, we look at prior work related to
connecting LLMs to T2I models.

[24] introduced a 1.8 million Stable Diffusion prompt dataset called
DiffusionDB. Such a dataset is extremely valuable for fine-tuning a language
model to generate prompts. [12] introduced a benchmark dataset called the
"Text-to-image Ambiguity Benchmark". They show that their framework for prompt
disambiguation generates more faithful images which align more to user
preferences. [19] study the aspects of creative image generation which are
unique, and propose criteria that dataset authors should think about when
building text-to-image generation benchmarks.

[6] look at techniques for improving visual story generation and contribute a
dataset called "Visual Writing Prompts". Their work looks at the idea of using
images to guide multi modal language models for story generation. [2]
introduce a new way to leverage language models for captioning image datasets.
They show that this is effective in the context of visual question answering.
[14] leveraged GPT-3 guided CLIP for solving the SemEval 2023 Shared task of
Visual Word Sense Disambiguation.

[1] describe a technique for aligning any LLM with a vision dataset. This is
useful for preserving an LLMs effectiveness while augmenting it with visual
understanding. [9] look at prompt based low resource learning of visual
learning tasks. They show that their technique is more effective than
traditional approaches to few-shot visual learning, and results in tiny
models.

[16] Show that DALLE-2 has serious issues with concept mapping in its
generations. They show significant inductive biases are introduced as a result
of DALLE-2 interpreting words with multiple senses simultaneously.

##  3 Techniques

###  3.1 Language Model Choice

The single most important factor in creating an effective Art Director is the
underlying LLM. Commercial LLMs, such as GPT-4[13], Anthropic Claude, and
Google Bard are still more effective than their locally hosted counterparts
out of the box. The huge variety of techniques and settings which are
available with locally hosted models make them far more flexible and
controllable than closed source alternatives.

Because language models are advancing so quickly, we advocate continuous
experimentation. We’ve found success with the Mistral[8] and Llama[22] models
along with variants like Vicuna[15]. We also find that for situations when we
need an LLM to make decisions based on image data, that leveraging visual
aligned language models like Minigpt-4[25] to caption or describe images is
powerful. We also make the following observations: All things being equal,
larger parameter count models are preferred to smaller variants, models
trained for longer and on more (and better quality) tokens perform
increasingly well, and current quantization techniques can reduce LLM output
quality more strongly than their benchmarks suggest.

###  3.2 Retrieval Augmented Generation

[11] introduced a technique called Retrieval Augmented Generation (RAG).
Retrieval Augmented Generation is a methodology that combines the power of
large-scale retrieval and language models for a particular task. RAG leverages
an external vector database or a corpus to find relevant contextual
information before the generation process. This information is appended to the
language models prompt.

RAG can help improve the quality of images generated by models like Dall-E by
providing them with more contextual information derived from a large corpus of
data. For instance, if the prompt involves generating an image of "a
futuristic cityscape," RAG can pull data on what elements constitute
’futuristic’ and ’cityscape,’ thereby guiding the model to produce a richer,
more accurate image.

RAG is extremely powerful for implementing in-context few-shot learning by
leveraging an external prompt database of known good prompts. Instructing the
language model in what a good prompt looks like, and then appending 5 known
good prompts as examples leads to quality output prompts and correspondingly
beautiful images.

###  3.3 Fine-Tuning

Low Rank Adapters (Lora), introduced by [7] is a technique used to fine-tune
large pre-trained models efficiently. Instead of fine-tuning all parameters,
Lora modifies a low-rank factorized adapter layer, which is significantly
smaller in size but still highly expressive. In the context of Stable
Diffusion, Lora enables the model to be fine-tuned for specific artistic
styles or to generate images of objects/persons not present in the initial
training set. The fine-tuning is accomplished with less computational cost
while retaining the original model’s features. Other competing techniques for
fine-tuning the image models include Dreambooth[18], Hypernetworks, Textual
Inversion[3], and traditional fine-tuning.

Lora’s low-rank adapter layers also provide benefits in language models. When
a language model needs to adapt to specific domain jargon or idiomatic
expressions, Lora facilitates this with minimal adjustments to the original
parameters, ensuring computational efficiency.

LaDi leverages many fine-tuned Lora models for its constituent Stable
Diffusion, AnimateDiff, and Large Language Models. We find that they are
extremely valuable due to their compositional nature. Many of them can be used
at once and they can be merged together as needed.

###  3.4 Prompt Crafting

One of the most crucial components for optimizing the quality of images
generated by text-to-image models like Stable Diffusion is the textual prompt.
In this section, we delve into the building blocks of an effective prompt,
elucidating how each element contributes to the model’s ability to generate
high-fidelity, contextually relevant, and aesthetically pleasing images.

A competent prompt is often characterized by its specificity and detail,
elements that can be parsed through an extensive checklist of keyword
categories. These categories include:

  * •

Subject: The central focus of the image.

  * •

Medium: The artistic medium.

  * •

Style: The artistic style or aesthetic.

  * •

Artist: Names of artists to emulate.

  * •

Website: Websites known for a particular style or quality.

  * •

Resolution: Keywords indicating the sharpness or detail.

  * •

Color: Dominant or thematic colors.

  * •

Lighting: The type and quality of lighting.

  * •

Additional Details: Other elements to enhance the image.

Each category serves as a guide for enhancing the expressiveness of a prompt.

Let’s consider generating an image of a "steampunk explorer in a mysterious
jungle."

  * •

Subject: "A steampunk explorer with mechanical limbs, wearing leather and
gears, standing amidst a mysterious, dense jungle with exotic flora and
fauna."

  * •

Medium: "Digital painting"

  * •

Style: "Neo-Victorian, adventure-themed"

  * •

Artist: "Inspired by the styles of Vincent Van Gogh and M.C Escher"

  * •

Website: "DeviantArt"

  * •

Resolution: "Highly detailed, with sharp focus"

  * •

Additional Details: "With a backdrop of an ancient temple and steampunk
airships in the sky"

  * •

Color: "Muted earth tones with bursts of vibrant color"

  * •

Lighting: "Dappled sunlight filtering through dense foliage"

Final Prompt: "A steampunk explorer with mechanical limbs, wearing leather and
gears, standing amidst a mysterious, dense jungle with exotic flora and fauna,
digital painting, neo-Victorian, adventure-themed, inspired by the styles of
H.R. Giger and Brian Froud, DeviantArt, highly detailed with sharp focus, with
a backdrop of an ancient temple and steampunk airships in the sky, muted earth
tones with bursts of vibrant color, dappled sunlight filtering through dense
foliage."

While a prompt doesn’t necessarily require every category to be effective, a
comprehensive approach can substantially elevate the quality of generated
images. Experimentation remains crucial; different combinations yield
different results, and users are encouraged to iterate on their prompts to
achieve desired outcomes.

Through effective prompting, Stable Diffusion and similar models can create
not just accurate representations but truly artistic works that are complex,
nuanced, and emotionally evocative.

We tailor LaDi to follow instructions for crafting prompts according to this
checklist. We also write instructions which show its LLM how to
emphasize/upweight important tokens and de-emphasize/downweight unimportant
concepts.

####  3.4.1 Effective Prompting for Text-to-Video

The techniques we have described thus far focus primarily on enhancing image
generation with Stable Diffusion. However, our framework for leveraging large
language models as Art Directors can extend to other modalities such as text-
to-video generation.

One of the most compelling features of text-to-video techniques is prompt
traveling, which facilitates smooth visual transitions between different
prompts over a sequence of frames. This allows for the creation of animated
narratives by progressively modifying elements of the scene. However, a key
challenge lies in crafting prompts that can seamlessly morph into one another
without visual jarring or incoherence.

LaDi can be readily adapted to generate prompts tailored specifically for
prompt traveling in text-to-video models. We optimize the language model to
construct prompts with significant lexical and semantic overlap, which aids in
the interpolation process. The prompts are structured to feature consistent
core subjects while modifying descriptive elements incrementally between
keyframes.

For instance, consider generating a video depicting a spaceship traveling
through varied cosmic scenery. The LLM Art Director may produce the following
prompts for different keyframes:

  * •

Frame 1: "A sleek silver spaceship gliding through a nebula of purple and blue
hues."

  * •

Frame 15: "The sleek silver spaceship gliding past an asteroid field with
giant space rocks."

  * •

Frame 30: "The same sleek silver spaceship approaching a planet with red and
orange rings."

Note the consistent use of "sleek silver spaceship" across prompts. By keeping
core subjects intact while smoothly modifying details like the backdrop, text-
to-video models can readily interpolate between these prompts to animate the
space travel sequence.

The language model is fine-tuned using paired samples of prompts that morph in
this coherent fashion. It also learns to identify optimal points for prompt
changes based on narrative structure. Final videos may contain several
seamless prompt transitions driven entirely by AI-generated text prompts.

Our techniques allow leveraging the knowledge and compositional skills of LLMs
to unlock the full potential of text-to-video models combined with prompt
traveling. The Art Director provides a robust solution to controlling text-to-
video synthesis for both practical and creative applications.

###  3.5 Classifier-Free Guidance (CFG) and Negative Prompts

Incorporating the concept of Negative Prompts allows users to specify
undesirable features explicitly. For instance, if a user does not want an
image to contain "low resolution" or "amateur drawing," these terms are
included as negative prompts to guide the generation process.

Stable Diffusion has always had direct support for negative prompts, but up
until recently, regular LLMs, such as the LLM generating our Stable Diffusion
prompts, did not support negative prompting. [20] introduced the concepts of
Classifier-Free Guidance and Negative Prompts, techniques known to the image
generation community.

CFG works by appending carefully constructed demonstrative examples that
exemplify unwanted behaviors. For instance, to avoid generating violent
content, sample negative demonstrations like "A man punching another man"
could be appended. The LLM learns from these examples during decoding.

We implement CFG by curating labeled datasets of positive and negative prompt
demonstrations. The LLM is fine-tuned to differentiate between these examples.
During prompt generation, negative demonstrations are dynamically selected via
retrieval and concatenated to the context.

This guides the LLM to avoid undesirable attributes in its outputs. CFG allows
imposing far more complex constraints compared to simply blacklisting certain
tokens. Entire concepts like "poor grammar" or "excessive verbosity" can be
discouraged by tailoring demonstrations.

We also experiment with mixing CFG and explicit negative prompts for greater
control. Certain immutable tokens like profanities can be blacklisted, while
broader attributes are shaped via CFG demonstrations. We find this hybrid
approach prevents "leakage" where constraints are circumvented by synonymous
words.

LaDi leverages CFG across multiple dimensions relevant to text-to-image
generation. We curate demonstrations to discourage prompts with attributes
like:

  * •

Incoherent scene descriptions

  * •

Ambiguous or contradictory details

  * •

Overly verbose or rambling sentences

  * •

Poor artistic composition

  * •

Grammar and spelling errors

By combining CFG and negative prompts, we refine the LaDi’s generated prompts
to minimize artistic flaws and maximize creative potential. Images produced
using prompts crafted with these techniques exhibit significant gains in
coherence, relevance, and aesthetic quality.

###  3.6 Grammar Based Sampling

While Large-Language Models have demonstrated robust few-shot learning
capabilities across a plethora of tasks, generating structured outputs remains
a challenging endeavor. This limitation becomes particularly prominent when
attempting to craft precise text prompts for text-to-image generators like
Stable Diffusion. To tackle this problem, we leverage formal grammar-based
sampling during the decoding process to create highly effective prompts. By
enriching the decoding process with formal grammar constraints, we aim to
generate sequences that conform to predefined structures essential for
specialized tasks like Stable Diffusion prompt generation.

We construct a Context-Free Grammar (CFG) for LaDi tailored for generating
Stable Diffusion prompts. The grammar aims to specify the elements of a scene,
their attributes, and relationships, as they are pivotal in directing Stable
Diffusion towards generating more accurate and relevant images. This CFG is
long and beyond the scope of the paper, but we showcase an example CFG below.

Let us define a sample formal grammar $G$, consisting of:

  * •

A set of non-terminal symbols
$N=\\{S,\text{Element},\text{Attribute},\text{Relation}\\}$

  * •

A set of terminal symbols
$T=\\{\text{"cat"},\text{"dog"},\text{"sitting"},\text{"jumping"},\text{"next
to"},\text{"above"}\\}$

  * •

A set of production rules $P$

  * •

A start symbol $S$

The production rules $P$ can be expressed as:

$\displaystyle 1.\quad S\rightarrow\text{Element}$ $\displaystyle 2.\quad
S\rightarrow\text{Element}\,\text{Attribute}$ $\displaystyle 3.\quad
S\rightarrow\text{Element}\,\text{Attribute}\,\text{Relation}\,\text{Element}$
$\displaystyle 4.\quad\text{Element}\rightarrow\text{"cat"}\,|\,\text{"dog"}$
$\displaystyle
5.\quad\text{Attribute}\rightarrow\text{"sitting"}\,|\,\text{"jumping"}$
$\displaystyle 6.\quad\text{Relation}\rightarrow\text{"next
to"}\,|\,\text{"above"}$

Our framework employs an incremental parser that leverages the CFG rules
during the decoding process. At each decoding step, the parser consults the
grammar to ascertain which tokens could serve as valid continuations. This
narrows down the sequence space to only those that adhere to the CFG rules.

For example, if the partially generated sequence is "cat sitting", the parser
will consult the CFG and realize that the only valid continuation would
involve a token from the set "next to", "above" as specified by the Relation
non-terminal. This way, prompts like "cat sitting jumping" would be
invalidated by the framework, ensuring higher quality prompts.

With this CFG, a sample valid string (or prompt) generated can be "cat sitting
next to dog", which Stable Diffusion can use to create an image of a sitting
cat next to a dog.

###  3.7 Constrained Beam Search

Constrained Beam Search is a technique that allows us to direct the text
generation process according to certain predetermined conditions or tokens,
ensuring that the generated output adheres to these specified sequence level
constraints. Given that textual prompts directly influence the visual
attributes of images generated by Stable Diffusion, Constrained Beam Search
can be extremely useful in achieving a precise kind of visual output.

Huggingface has made this form of Constrained Decoding available for their
models for awhile. Beside the default "Disjunctive" and "Phrasal" constraints,
users can define their own kinds of constraints.

To make our approach more innovative, we introduce two unique types of
constraints: ColorPatterns and SubjectAnchor.

  * •

ColorPatterns: This constraint ensures that a specific color pattern or
palette is mentioned in the text prompt, thereby directly influencing the
color scheme of the generated image. For instance, if we constrain the text
prompt to include "sunrise hues," Stable Diffusion is more likely to generate
an image with colors resembling an actual sunrise.

  * •

SubjectAnchor: This constraint ensures that the central subject of the image
remains consistent across multiple generations. If the SubjectAnchor is set to
"eagle," for example, the image will always feature an eagle as the central
subject, regardless of other varying elements.

Suppose we want to generate a series of landscape images that not only
transition through the four seasons but also feature a consistent element,
such as a particular tree or mountain, across the images.

  * •

Constrained Prompt: The constrained prompt could be initialized with a
SubjectAnchor as "oak tree" and a ColorPattern that varies according to the
season (e.g., "winter hues" for a snowy scene or "fall colors" for an autumn
setting).

  * •

Constrained Beam Search Steps:

    1. 1.

Step 1: Along with typical high-probability tokens, we insert "oak tree" and
"winter hues" into the generation candidates.

    2. 2.

Step 2: We maintain the SubjectAnchor ("oak tree") while changing the
ColorPattern to "spring blossoms" for a new iteration, ensuring that while the
scene changes to reflect spring, the oak tree remains consistent in the frame.

  * •

Banking and Selection: The constrained tokens are sorted into banks depending
on their steps towards fulfilling the constraint. A round-robin selection
ensures that while the constraints are met, the output prompt is also
semantically coherent.

  * •

Resulting Prompts: Possible output prompts might include "An oak tree
surrounded by a winter wonderland," followed by "An oak tree amidst a field of
spring blossoms."

Constrained Beam Search is one form of constraint, but LaDi leverages other
kinds as well to further improve the creativity of its prompts.

###  3.8 Language Model Probabilistic Programming

Despite the versatility of large language models in various NLP tasks,
generating controlled outputs remains a significant challenge. To mitigate
this issue, we leverage an inference-time approach called Sequential Monte
Carlo (SMC) steering introduced by [10]. This method relies on a novel
probabilistic programming library, LLaMPPL, for steering LLMs to produce
controlled and coherent prompts for image generators.

SMC steering treats language generation tasks as posterior inference problems
in discrete probabilistic sequence models. By replacing the standard decoding
process with Sequential Monte Carlo inference, it becomes feasible to impose
token or sequence level syntactic and semantic constraints on the outputs of
LLMs.

LLaMPPL allows users to write probabilistic programs that combine symbolic
program logic, probabilistic conditioning, and calls to LLMs. It uses a
specialized SMC inference algorithm to solve these tasks, offering an elegant
and robust mechanism to steer language models.

Consider a task where the aim is to generate an art piece that depicts a
surreal landscape, but under the constraint that the landscape should contain
neither water bodies nor any red hues. Generating such an image with Stable
Diffusion can be challenging if the text prompt is not carefully managed. By
using LLaMPPL and SMC steering, these constraints can be methodically
enforced.

We give sample code for leveraging the "Huggingface Probabilistic Programming
Language" library to follow SMC steering below. This sample code is modified
from examples provided on the projects github repo.

⬇

1from hfppl import Model, LMContext, TokenCategorical, CachedCausalLM

2

3# Sample forbidden tokens

4FORBIDDEN_WORDS = [’water’, ’lake’, ’ocean’, ’river’, ’sea’, ’red’]

5

6# A LLaMPPL model subclasses the Model class

7class MyModel(Model):

8

9 # The __init__ method is used to process arguments

10 # and initialize instance variables.

11 def __init__(self, lm, prompt):

12 super().__init__()

13

14 # A stateful context object for the LLM,

15 # initialized with the prompt

16 self.context = LMContext(lm, prompt)

17 self.lm = lm

18

19 # The forbidden tokens

20 self.forbidden_tokens =

21 [i for (i, v) in enumerate(lm.vocab)

22 if any(word in v for word in FORBIDDEN_WORDS)]

23

24 # The step method is used to

25 # perform a single ’step’ of generation.

26 # This might be a single token, a single phrase,

27 # or any other division.

28 # Here, we generate one token at a time.

29 async def step(self):

30 # Sample a token from the LLM

31 # We use ‘await‘ so that

32 # LLaMPPL can automatically batch language model calls.

33 token = await self.sample(self.context.next_token(),

34 proposal=self.proposal())

35

36

37 self.condition(token.token_id not in self.forbidden_tokens)

38

39 # Check for EOS or end of sentence

40 if token.token_id == self.lm.tokenizer.eos_token_id:

41 # Finish generation

42 self.finish()

43

44 # Helper method to define a custom proposal

45 def proposal(self):

46 logits = self.context.next_token_logprobs.copy()

47 logits[self.forbidden_tokens] = -float(’inf’)

48 return TokenCategorical(self.lm, logits)

49

50

51 def immutable_properties(self):

52 return set([’forbidden_tokens’])

53

54# Example usage

55# Initialize the model with CachedCausalLM

56# (or any other compatible LM) and the initial prompt

57lm = CachedCausalLM(...) # Initialize your language model here

58prompt = "Create␣a␣surreal␣landscape␣with"

59model = MyModel(lm, prompt)

60

61# Run the generation loop

62await model.run()

In this example, the prompt "Create a surreal landscape with" is initiated,
but the SMC-steered LLM is conditioned to avoid generating any tokens
associated with "water" or "red." The generated text prompts can then be fed
into Stable Diffusion, ensuring that the resulting images adhere to the
specified artistic constraints. Because SMC modifies future tokens to be more
likely to follow constraints than traditional filter-based decoding alone,
quality of LLM output is significantly above more naive approaches to
constrained text generation.

##  4 Conclusion

In this paper, we have explored a variety of techniques for enhancing text-to-
image generators such as Stable Diffusion with intelligent prompting and
constrained decoding powered by large language models. Our approaches
demonstrate how LLMs can serve as "Art Directors" that guide image generation
towards greater relevance, coherence, and artistic merit. We cumulated these
approaches into a powerful Art Director called LaDi.

We described the methods used in Ladi including retrieval augmented generation
to incorporate external contextual information, fine-tuning with adapters to
specialize models for specific domains, crafting detailed prompt checklists,
using negative prompts and classifier-free guidance, grammar-based sampling to
generate structured prompts, constrained beam search to enforce sequence
constraints, and SMC steering with probabilistic programming to methodically
direct text generation.

Our techniques aim to elevate text-to-image generation beyond impressive but
often inaccurate visuals, towards images that are precise artistic
representations of intended concepts and aesthetics. With LLMs as Art
Directors, limitations stemming from weak prompts can be mitigated. The
solutions presented form a comprehensive framework for users across industries
to produce tailored, high-fidelity images that align with creative and
practical needs.

There remain opportunities for improvement, including integrating additional
modalities beyond text for directing image synthesis, experimenting with other
text decoding and image diffusion algorithms, and using more sophisticated
models as they are released.

## References

  * [1] Wenliang Dai et al.  “Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation”  In _Findings of the Association for Computational Linguistics: ACL 2022_ Dublin, Ireland: Association for Computational Linguistics, 2022, pp. 2383–2395  DOI: 10.18653/v1/2022.findings-acl.187
  * [2] Yifan Du et al.  “Zero-shot Visual Question Answering with Language Model Feedback”  In _Findings of the Association for Computational Linguistics: ACL 2023_ Toronto, Canada: Association for Computational Linguistics, 2023, pp. 9268–9281  DOI: 10.18653/v1/2023.findings-acl.590
  * [3] Rinon Gal et al.  “An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion”, 2022  arXiv:2208.01618 [cs.CV]
  * [4] Yuwei Guo et al.  “AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning”, 2023  arXiv:2307.04725 [cs.CV]
  * [5] Jonathan Ho et al.  “Imagen Video: High Definition Video Generation with Diffusion Models”, 2022  arXiv:2210.02303 [cs.CV]
  * [6] Xudong Hong et al.  “Visual Writing Prompts: Character-Grounded Story Generation with Curated Image Sequences”  In _Transactions of the Association for Computational Linguistics_ 11 Cambridge, MA: MIT Press, 2023, pp. 565–581  DOI: 10.1162/tacl_a_00553
  * [7] Edward J. Hu et al.  “LoRA: Low-Rank Adaptation of Large Language Models”, 2021  arXiv:2106.09685 [cs.CL]
  * [8] Albert Q. Jiang et al.  “Mistral 7B”, 2023  arXiv:2310.06825 [cs.CL]
  * [9] Woojeong Jin et al.  “A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Models”  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ Dublin, Ireland: Association for Computational Linguistics, 2022, pp. 2763–2775  DOI: 10.18653/v1/2022.acl-long.197
  * [10] Alexander K. Lew, Tan Zhi-Xuan, Gabriel Grand and Vikash K. Mansinghka  “Sequential Monte Carlo Steering of Large Language Models using Probabilistic Programs”, 2023  arXiv:2306.03081 [cs.AI]
  * [11] Patrick Lewis et al.  “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks”, 2021  arXiv:2005.11401 [cs.CL]
  * [12] Ninareh Mehrabi et al.  “Resolving Ambiguities in Text-to-Image Generative Models”  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ Toronto, Canada: Association for Computational Linguistics, 2023, pp. 14367–14388  DOI: 10.18653/v1/2023.acl-long.804
  * [13] OpenAI  “GPT-4 Technical Report”, 2023  arXiv:2303.08774 [cs.CL]
  * [14] Xiaomeng Pan, Zhousi Chen and Mamoru Komachi  “Query Generation Using GPT-3 for CLIP-Based Word Sense Disambiguation for Image Retrieval”  In _Proceedings of the 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023)_ Toronto, Canada: Association for Computational Linguistics, 2023, pp. 417–422  DOI: 10.18653/v1/2023.starsem-1.36
  * [15] Baolin Peng et al.  “Instruction Tuning with GPT-4”, 2023  arXiv:2304.03277 [cs.CL]
  * [16] Royi Rassin, Shauli Ravfogel and Yoav Goldberg  “DALLE-2 is Seeing Double: Flaws in Word-to-Concept Mapping in Text2Image Models”  In _Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP_ Abu Dhabi, United Arab Emirates (Hybrid): Association for Computational Linguistics, 2022, pp. 335–345  DOI: 10.18653/v1/2022.blackboxnlp-1.28
  * [17] Robin Rombach et al.  “High-Resolution Image Synthesis with Latent Diffusion Models”, 2022  arXiv:2112.10752 [cs.CV]
  * [18] Nataniel Ruiz et al.  “DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation”, 2023  arXiv:2208.12242 [cs.CV]
  * [19] Irene Russo  “Creative Text-to-Image Generation: Suggestions for a Benchmark”  In _Proceedings of the 2nd International Workshop on Natural Language Processing for Digital Humanities_ Taipei, Taiwan: Association for Computational Linguistics, 2022, pp. 145–154  URL: https://aclanthology.org/2022.nlp4dh-1.18
  * [20] Guillaume Sanchez et al.  “Stay on topic with Classifier-Free Guidance”, 2023  arXiv:2306.17806 [cs.CL]
  * [21] Uriel Singer et al.  “Make-A-Video: Text-to-Video Generation without Text-Video Data”, 2022  arXiv:2209.14792 [cs.CV]
  * [22] Hugo Touvron et al.  “LLaMA: Open and Efficient Foundation Language Models”, 2023  arXiv:2302.13971 [cs.CL]
  * [23] Jiuniu Wang et al.  “ModelScope Text-to-Video Technical Report”, 2023  arXiv:2308.06571 [cs.CV]
  * [24] Zijie J. Wang et al.  “DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models”  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ Toronto, Canada: Association for Computational Linguistics, 2023, pp. 893–911  DOI: 10.18653/v1/2023.acl-long.51
  * [25] Deyao Zhu et al.  “MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models”  In _arXiv preprint arXiv:2304.10592_ , 2023