#  Tensor product random matrix theory

Alexander Altland  Institut für Theoretische Physik, Universität zu Köln,
Zülpicher Str. 77, 50937 Cologne, Germany  Joaquim Telles de Miranda  Centro
Brasileiro de Pesquisas Físicas, Rua Xavier Sigaud 150, 22290-180, Rio de
Janeiro, Brazil  Tobias Micklitz  Centro Brasileiro de Pesquisas Físicas, Rua
Xavier Sigaud 150, 22290-180, Rio de Janeiro, Brazil

###### Abstract

The evolution of complex correlated quantum systems such as random circuit
networks is governed by the dynamical buildup of both entanglement and
entropy. We here introduce a real-time field theory approach — essentially a
fusion of the $G\Sigma$-functional of the SYK-model and the field theory of
disordered systems — enigneered to microscopically describe the full range of
such crossover dynamics: from initial product states to a maximum entropy
ergodic state. To showcase this approach in the simplest nontrivial setting,
we consider a tensor product of coupled random matrices, and compare to exact
diagonalization.

###### pacs:

05.45.Mt, 72.15.Rn, 71.30.+h

Introduction:—When addressing the physics of complex many-body systems, an
early choice needs to be made: One option is to focus on individually resolved
quasi-particle trajectories, which in concrete terms means working with second
quantized representations, many-body field integrals, and related concepts.
The alternative is to consider the more complex trajectory of the system at
large, as described by a single exponentially high-dimensional Hamiltonian
matrix. Depending on the context, the preferred choice is usually evident, use
cases including large portfolio of many-body field theory approaches to
condensed matter systems, or the modelling of complex nuclei in terms of high
dimensional random matrix theories, respectively. However, since the advent of
synthetic quantum matter, we are increasingly witnessing applications in
twilight zones, where the appropriate approach is not so clear. For example,
starting from initial product states of qudits or ‘quasi-particles’, circuit
dynamics addresses the buildup of complexity, eventually leading to a fully
entangled state. In the latter, individual states can no longer be
meaningfully resolved and global representations, e.g., in terms of a high-
dimensional random matrices, may be more appropriate. The crux of the matter
is that precisely the crossover dynamics leading from individually resolved
quasi-particle worldlines to the single complex trajectory of a many-body
state is hard to capture in terms of the analytical toolkits of field- or
path-integration, which may be a reason for the relative scarcity of such
methods in the theory of quantum circuits. (For powerful phenomenological,
semiclassical, or statistical mechanics inspired approaches, see, e.g., Refs.
Chan et al. (2018a); Nahum et al. (2017); Vasseur et al. (2019); Roy et al.
(2019); Chan et al. (2018b); Garratt and Chalker (2021); Liao and Galitski
(2022).)

In this paper we define a random matrix model displaying such type of
crossover dynamics in the simplest possible setting. The model will serve as a
test bed for a new path integral approach whose central ingredient is an
interaction vertex possessing the mathematical structure of a Hadamard
product. We will discuss how the Hadamard vertex catalyzes the progressive
locking of initially separate subsystem trajectories to a configuration
representing a maximally entangled ergodic final state. As a representative of
various observables witnessing this dynamics we will consider the spectral
form factor, whose time dependence we will compute analytically and compare to
exact diagonalization. We conclude with a discussion of the extensibility of
the approach exemplified here to more complex circuit arrays.

Figure 1:  Schematic representation of closed scattering sequences in $A$ and
$B$ Hilbert space traversed in retarded and advanced time order (outer and
inner rings). For constructive interference, the sequences must contain the
same scatterers but can be traversed at a relative time delay, $\tau_{A}$ and
$\tau_{B}$, indicated by dots on the retarded and advanced contours. Insets:
In- and out-scattering induced by $H_{A}$, and temporal structure of the
Gaussian propagator Eq. (3).

The model comprises two $N_{X}$ dimensional qudits, $X=A,B$ governed by the
Hamiltonians $H_{A}\otimes\mathds{1}_{B}$ and $\mathds{1}_{A}\otimes H_{B}$,
with $H_{X}$ drawn from a Gaussian random matrix ensemble. The two sectors are
entangled via the, likewise random $N_{A}N_{B}$-dimensional Hamiltonian
$H_{AB}$, defining the full Hamiltonian $H=H_{A}+H_{B}+H_{AB}$, where we
suppress trivial factors $\otimes\mathds{1}$ for notational simplicity. We
begin our analysis of this model with a qualitative discussion in terms of
Feynman path amplitudes. This picture will provide an intuitive basis for the
subsequent quantitative formulation in terms of a real-time matrix path
integral.

Qualitative discussion: In the absence of coupling, the spectrum of our model
comprises all $N_{A}N_{B}$ different sums $\epsilon_{Ai}+\epsilon_{Bj}$. While
these levels can come arbitrarily close (no RMT like level repulsion) they
remain statistically correlated (no Poisson statistics). To understand these
correlations, it is best to switch to time space, and consider the spectral
form factor, which, prior to normalization, is defined as
$K(t)=\langle\mathrm{tr}(G^{+}(t))\mathrm{tr}(G^{-}(-t))\rangle$ in terms of
the retarded and advanced Green functions $G^{\pm}(t)=\mp i\Theta\left(\pm
t\right)\exp(-iHt)$. Consider first a single ergodic system, such as the
$A$-factor of our tensor product structure. Representing the trace as
$\mathrm{tr}(\dots)=\sum_{\mu}\langle\mu|\dots|\mu\rangle$, the form factor
becomes a double sum over self-returning scattering paths traversed in time
$t$, as indicated in Fig. 1. To leading approximation in $N_{A}^{-1}$, only
paths traversing the same sequence of scattering events in retarded and
advanced chronology contribute upon averaging over the ensemble. However, the
anchor points at which the sequences start in Hilbert space at time $t=0$ can
be independently chosen for the two amplitudes, and it is this ‘sliding’
degree of freedom which is responsible for the linearity $K(t)\sim t$ Haake
(2018). Turning to the full problem, the trace assumes the form
$\mathrm{tr}(\dots)=\sum_{\mu\nu}\langle\mu\nu|\dots|\mu\nu\rangle$, where we
use the shorthand notation
$\langle\mu\nu|\equiv\langle\mu|\otimes\langle\nu|$, and the second factor
labels the states of $B$. On this basis, we now compare the two alternative
approaches mentioned above, paraphrased as second vs. first quantized for
brevity.

Second quantized: Consider the states
$|\mu\nu\rangle=a_{\mu}^{\dagger}b_{\nu}^{\dagger}|0\rangle$ represented in
terms of single particle bosonic operators generating the subsystem states.
Beginning with the uncoupled system, $H_{AB}=0$, the factorization of the
trace
$\mathrm{tr}(G^{\pm}(t))=\mathrm{tr}(G^{\pm}_{A}(t))\mathrm{tr}(G^{\pm}_{B}(t))$
is manifest in this language. It extends to the form factor, and so $K(t)\sim
t^{2}$ in this limit. However, upon switching on the interaction, we encounter
the _nonlinear_ operator (summation convention)
$H_{AB}=a_{\mu}^{\dagger}b_{\nu}^{\dagger}H_{\mu\nu,\mu^{\prime}\nu^{\prime}}a_{\mu^{\prime}}b_{\nu^{\prime}}$,
expected to send the full system into an ergodic regime. Defining the
variances of our random Hamiltonians as
$N_{A}\langle|H_{A\mu\mu^{\prime}}|^{2}\rangle=N_{B}\langle|H_{B\nu\nu^{\prime}}|^{2}\rangle=\lambda^{2}$,
and
$N_{A}N_{B}\langle|H_{AB\mu\nu,\mu^{\prime}\nu^{\prime}}|^{2}\rangle=\Lambda^{2}$,
a straightforward Golden Rule estimate suggests a coupling rate
$\gamma\approx\Lambda^{2}/2\lambda$, and hence we expect $K(t)\sim t$ for time
scales $t>\gamma^{-1}$. However, the quantitative description of this
crossover requires the solution of an ‘interacting’ problem, which is
nontrivial in this language.

First quantized: In this representation, $H_{\mu\nu,\mu^{\prime}\nu^{\prime}}$
is a matrix of dimension $N_{A}N_{B}$, and the derivation of a linear form
factor $K(t)\sim t$ follows standard protocols (for example, by interpreting
the construction indicated in Fig. 1 in terms of scattering paths in tensor
product space.) However, it is no longer straightforward to describe the
crossover to $K(t)\sim t^{2}$ upon _lowering_ $\Lambda$; extracting the
‘disentangling’ into separate subsystem paths in $\mu$\- and $\nu$-space and
the emergence of two independent sliding symmetries from the global matrix
representation is difficult. We observe that neither approch appears to be
well suited to address the middle ground between the two limiting regimes.

Field integral approach: In the following, we follow a strategy inspired in
equal parts by field theories of disordered systems Efetov (1997), and the
$G\Sigma$-approach to the SYK model (for a review, see Ref. Rosenhaus (2019)),
to construct a theory tailored to describe the form factor at all time scales.
Referring to the supplemental material for details, we start in second
quantization and next turn to a real time field integral representation,
$K(t)=\langle\prod_{s=+,-}a^{s}_{\mu,t}b^{s}_{\nu,t}\bar{a}^{s}_{\mu,0}\bar{b}^{s}_{\nu,0}\rangle$,
where $a=\\{a_{\mu,t}^{s}\\}$ and $b=\\{b_{\nu,t}^{s}\\}$ are now complex
commuting field variables, and the causal index ($s=\pm$) distinguishes
between fields propagating forward and backward in time. The functional
average $\langle\dots\rangle=\int D(a,b)\exp(iS[a,b])(\dots)$ is over the
action $S[a,b]=S_{0}[a,b]+S_{\textrm{int}}[a,b]$, with
$S_{0}[a,b]=\int_{0}^{t}dt\sum_{x=a,b}\bar{x}\tau_{3}(i\partial_{t}-H_{X}+i0\tau_{3})x$,
and $S_{\textrm{int}}[a,b]=\int dt\,\bar{a}\bar{b}\,\tau_{3}H_{AB}ab$, where
Hilbert space indices are suppressed for clarity, and $\tau_{3}$ is a Pauli
matrix in contour space. The (Itô) time discretization of the functional
integral implies $\langle 1\rangle=1$, i.e. it yields real time Green
functions without the need for explicit normalization Kamenev (2011).

Averaging over the distribution of the three involved random matrices produces
an effective action containing products of up to eight field operators. They
all occur in combinations summed over Hilbert space indices, suggesting the
introduction of effective Hubbard-Stratonovich variables
$(G_{X})_{tt^{\prime}}^{ss^{\prime}}\equiv-iN_{X}^{-1}\,x_{\mu
t}^{s}\bar{x}_{\mu t^{\prime}}^{s^{\prime}}(-)^{s^{\prime}}$. We effect this
locking via Lagrange multipliers $(\Sigma_{X})_{tt^{\prime}}^{ss^{\prime}}$,
and arrive at the representation $\langle\dots\rangle=\int
D(G,\Sigma)\,\exp(iS[G,\Sigma])$, where $S=S_{A}+S_{B}+S_{AB}$, with

$\displaystyle
S_{X}[G,\Sigma]=iN_{X}\mathrm{tr}\left(\ln(i\partial_{t}-\Sigma_{X})+G_{X}\Sigma_{X}+\frac{\lambda^{2}}{2}G_{X}^{2}\right),$
$\displaystyle
S_{AB}[G]=i\frac{\Lambda^{2}N_{A}N_{B}}{2}\mathrm{tr}((G_{A}\odot
G_{B}\tau_{3})^{2}).$ (1)

Here, the traces imply summation over all indices, $\mathrm{tr}(AB)=\int
dtdt^{\prime}A^{ss^{\prime}}_{tt^{\prime}}B^{s^{\prime}s}_{t^{\prime}t}$, and
we encounter the Hadamard- or element-wise product defined as $(A\odot
B)^{ss^{\prime}}_{tt^{\prime}}=A^{ss^{\prime}}_{tt^{\prime}}B^{ss^{\prime}}_{tt^{\prime}}$.
In the same notation, the correlation function assumes the form
$K(t)=(N_{A}N_{B})^{2}\langle(G_{A}\odot G_{B})^{++}_{t,0}(G_{A}\odot
G_{B})^{--}_{0,t}\rangle$.

Stationary phase: The introduction of the collective $(G,\Sigma)$-variables is
rewarded by the appearance of the factors $N_{X}$ upfront the action, inviting
a stationary phase approach. For simplicity, we assume that the coupling
$H_{AB}$ is sufficiently weak to not significantly affect the mean-field Green
functions of our system. Temporarily neglecting the Hadamard vertex, a
variation of the action in $G$ and $\Sigma$ (we suppress the label $X=A,B$ for
readability) then leads to the equations $G=(i\partial_{t}-\Sigma)^{-1}$ and
$\Sigma=\lambda^{2}G$, which are solved by
$\bar{G}_{tt^{\prime}}=-i\tau_{3}\Theta(\tau_{3}(t-t^{\prime}))\exp(-\lambda|t-t^{\prime}|)$:
the average Green functions and the self energies are rapidly decaying
functions in time, respecting causality. Neglecting $\partial_{t}$ in
comparison to $\Sigma$, the saddle point equations are approximately invariant
under temporally slow rotations, $\bar{G}\to T\bar{G}T^{-1}$,
$T=\\{T_{tt^{\prime}}^{ss^{\prime}}\\}$. (Note that the rotational symmetry
breaking in causal space, $\bar{G}\propto\tau_{3}$, identifies the $T$’s as
Goldstone mode fluctuations, with $\partial_{t}$ an ‘explicit symmetry
breaking’.) With the notation $Q=T\tau_{3}T^{-1}$, the expansion of the action
to leading order in derivatives $\partial_{t}$ reads as
$S[Q_{A},Q_{B}]=\sum_{X}S_{\rho_{X}}[Q_{X}]$, where

$\displaystyle S_{\rho}[Q]=i\pi\rho\int
dtdt^{\prime}\,(\partial_{t}+\partial_{t^{\prime}})\mathrm{tr}(\tau_{3}Q_{tt^{\prime}}),$
(2)

is the real time representation of the matrix actions known to produce the
random matrix form factor of systems with spectral density
$\rho_{X}=N_{X}/(\pi\lambda)$ Efetov (1997); Kamenev and Andreev (1999), and
the correlation function assumes the form
$K(t)=\rho_{A}^{2}\rho_{B}^{2}\prod_{X}\langle
Q^{++}_{Xt,0}Q^{--}_{X0,t}\rangle$. Its factorization implies that at this
level we obtain the product form factor of the decoupled theory.

However, what is not so evident is how for finite $\Lambda$ the Hadamard
coupling vertex will effect a collapse to a single Goldstone mode $Q$ required
to describe an emerging ergodic phase. Naively, one might speculate that its
proportionality to $G_{A}\odot G_{B}\sim Q_{A}\odot Q_{B}$ enforces a locking
$Q_{A}=Q_{B}=Q$ of the fluctuation fields. However, this is not how it works.
It is straightforward to verify that the vertex does not vanish on such field
configurations, implying that sought-after modes must be realized differently.
The crossover problem articulated in the beginning now assumes a very concrete
form: our path integral must possess a single Goldstone mode with a uniquely
specified action. However, it is not obvious where this mode hides.

Figure 2:  Correlation function in the presence of interactions. Top: For
scattering path traversing Fock space in generic, asynchronous order, only the
two-body generalization of out-scattering or self energy damping is
operational. Bottom: for time-synchronized path, or equal-time scattering
sequences in tensor space, in-scattering cancels against out-scattering,
stabilizing an undamped ergodic model.

Perturbation theory: In order to get traction with this problem, and establish
contact to the above Feynman path picture, we turn to a more concrete level
and expand the Goldstone mode fluctuations in generators chosen to anti-
commute with the saddle point $\sim\tau_{3}$,
$T=\exp\left(B\tau_{+}-B^{\dagger}\tau_{-}\right)$, where
$\tau^{\pm}=\frac{1}{2}(\tau_{1}\pm i\tau_{2})$ and
$B=\\{B_{tt^{\prime}}\\}$Kamenev and Andreev (1999). The quadratic expansion
of the decoupled theories assumes the form $S[B]=2i\pi\rho\int
dtdt^{\prime}B_{tt^{\prime}}(\partial_{t}+\partial_{t^{\prime}})B^{\dagger}_{t^{\prime}t}$,
implying that the Wick contraction of matrices in this free quadratic theory
is given by

$\displaystyle\langle
B_{t_{1+}t_{1-}}B^{\dagger}_{t_{2-}t_{2+}}\rangle=\Delta\delta(\Delta
t_{1}-\Delta t_{2})\Theta(t_{2}-t_{1}),$ (3)

with $\Delta=1/(2\pi\rho)$, $\Delta t=t_{+}-t_{-}$ and $t=(t_{+}+t_{-})/2$,
and the $\delta$-function broadened on the minimal timescale resolved by the
$B$-theory, $\lambda^{-1}$. Physically, these modes describe the pair
propagation of states indicated in the lower inset of Fig. 1, where the
temporal constraints imply equal propagation time of the retarded and advanced
amplitude, $t_{2+}-t_{1+}=t_{2-}-t_{1-}$, and causality $t_{2s}>t_{1s}$.
Otherwise, the $B$-propagator is structureless, reflecting ergodicity.
Substitution of the expansion $G^{++}_{t,0}\to\textrm{const.}+\int d\tau
B_{t,\tau}B^{\dagger}_{\tau,0}$ and application of the contraction yields each
of the correlation functions as a product of two modes (Fig. 1), leading to
$K(t)=\prod_{X}\left(\rho_{X}^{2}\int_{0}^{t}d\tau_{X}\cdot 1\right)\propto
t^{2}$, i.e. the product of two form factors, individually proportional to
$t$. The influence of the Hadamard vertex becomes transparent once we subject
it, too, to the generator expansion. Referring to the supplemental material,
the latter starts at fourth order,
$S_{AB}=S_{\textrm{o}+}+S_{\textrm{o}-}+S_{\textrm{i}}$ with

$\displaystyle S_{\textrm{o}+}[B]$ $\displaystyle=c_{\textrm{o}}\int
dt_{+}\prod_{X}\int dt_{X}\,B_{Xt_{+}t_{X-}}B_{Xt_{X-}t_{+}}^{\dagger},$ (4)
$\displaystyle S_{\textrm{i}}[B]$ $\displaystyle=-c_{\textrm{i}}\int
dt_{+}dt_{-}B_{At_{+}t_{-}}B_{Bt_{+}t_{-}}B^{\dagger}_{At_{-}t_{+}}B^{\dagger}_{Bt_{-}t_{+}},$

indicating that (a) we are expanding around a proper saddle point and (b) it
does not induce a trivial ‘mass’ as a quadratic term would. Here,
$c_{\textrm{o}}=2iN_{A}N_{B}\Lambda^{2}/\lambda^{3}$,
$c_{\textrm{i}}=16iN_{A}N_{B}\Lambda^{2}/\lambda^{4}$, and the action
$S_{\textrm{o}-}$ is obtained from $S_{\textrm{o}+}$ by an exchange
$B\leftrightarrow B^{\dagger}$. These expressions describe the two-body
analogs of the elementary one-body self-energy and vertex, o and i indicated
in the inset of Fig. 1. Specifically, $S_{\textrm{o},\pm}$ represents a
damping term correlating the previously independent pair amplitudes in $A$ and
$B$ Hilbert space through ‘out-’ scattering processes, cf. Fig. 2 top (for the
discussion of a similar damping mechanism for SYK-like models or coupled
circuits, cf. Refs.Garratt and Chalker (2021); Chan et al. (2018a); Liao and
Galitski (2022)). Coupling only to retarded $(A+,B+)$, respectively, advanced
$(A-,B-)$ amplitudes, it leaves the time differences $\tau_{A}=t_{A+}-t_{A-}$
and $\tau_{B}=t_{B+}-t_{B-}$ unconstrained. By contrast, the competing (note
the opposite sign) ‘in-’ scattering vertex does couple advanced and retarded
amplitudes, where the instantaneous of the scattering between the $A$ and $B$
sector requires $\tau_{A}=\tau_{B}$, cf. Fig. 2, bottom.

Including these vertices into the computation of the correlation function, we
obtain the situation depicted in Fig. 2. Prior to interactions (cf. Fig. 1),
the correlation function $K(t)$ implies the independent integration over two
time arguments $\tau_{X}$, $X=A,B$, parameterizing the temporal delay between
retarded and advanced scattering paths. Upon switching on interactions,
generic of these amplitudes suffer out-scattering (top figure), and hence get
exponentially damped. However, for synchronous paths
$\tau_{A}\approx\tau_{B}\equiv\tau$ in-scattering balances the out-scattering
and an undamped contribution ensues (bottom figure). The integral over the
single parameter $\tau$ then produces the proportionality $K(t)\sim t$
indicative of ergodicity. The projection onto synchronous paths implies that
we are effectively summing over trajectories in tensor product space
$t\mapsto\ket{\mu\nu}(t)$, rather than over products of ‘single particle
trajectories’ $t\mapsto\ket{\mu(t)}$ and $t\mapsto\ket{\nu(t)}$.

A straightforward summation over interaction vertex insertions of arbitrary
order leads to the result

$\displaystyle K(t)=\frac{c}{N_{A}N_{B}}\left((t\delta)^{2}e^{-2\gamma
t}+t\delta\right),$ (5)

where $\gamma\approx\Lambda^{2}/(2\lambda)$, the coupling rate,
$\delta\sim\lambda$ is set by the width of the spectrum, and $c$ a numerical
factor of order ${\cal O}(1)$. The first and second term in Eq. (5) are the
exponentially damped contribution of generic asynchronous pairs of single
qudit paths, and that of undamped tensor product trajectories, respectively.

Figure 3:  Comparison of the spectral form factor Eq. (5) (red) and numerical
simulations (grey), for $\Lambda/\lambda=0,0.1$ and $2$. Dashed and dash-
dotted lines in the insets indicate quadratic, respectively, linear profiles.
For the main plot we use three fitting parameters, resulting in good agreement
(see main text for further discussion).

Numerical validation: Before discussing how the analysis above can be pushed
beyond the level of perturbation theory, let us discuss how it fares in
comparison to numerics. Fig. 3 shows the analytical prediction Eq. (5) and
exact diagonalization results obtained by sampling over 500 realizations of
the matrix model with dimensions $N_{A}=N_{B}=130$. The comparison involves
three fit parameters, $(a,b,c)=(1.08,0.44,2.11)$ defined through the
proportionalities, $\gamma=a\Lambda^{2}/2\lambda$, $\delta=b\lambda$, and the
normalization in Eq. (5), respectively. The reason why the values $a,b$ are
left unspecified is that our definition of the form factor effectively
integrates over all states in the spectrum, whereas the analytical calculation
is based on approximations valid in the band center. While we have not been
ambitious to push the computation to the higher level of refinement required
for a parameter-free comparison, the non-monotonous profile of the form factor
predicted by Eq. (5) at intermediate values of the coupling is quantitatively
confirmed.

Generalizations: It is straightforward to generalize the above construction
beyond perturbation theory, and in this way identify the Goldstone mode action
governing the system in the long time limit: Starting from the matrix
representation of observables as
$\psi_{At}\psi_{Bt}\bar{\psi}_{At^{\prime}}\bar{\psi}_{Bt^{\prime}}\dots\to(Q_{A}\tau_{3}\odot
Q_{B}\tau_{3})_{tt^{\prime}}\dots$, we imagine each of the constituent
Goldstone modes expanded in generators $W=B\tau_{+}-B^{\dagger}\tau_{-}$,
leading to expressions like $(W_{A}^{n}\odot W_{B}^{n})_{tt^{\prime}}$ at
$n$th order in the expansion. We now observe that each individual Wick
contraction applied in the reduction of such expansions must be subject to the
equal-time difference condition, $W_{At,t+\tau_{A}}W_{Bt,t+\tau_{B}}\to
W_{At,t+\tau}W_{Bt,t+\tau}=(W_{A}\odot W_{B})_{t,t+\tau}$, to escape the
damping by out-terms. The recursive application of this con dition, implies
the collapse $(W_{A}^{n}\odot W_{B}^{n})_{tt^{\prime}}\to((W_{A}\odot
W_{B})^{n})_{tt^{\prime}}$. Finally, the individual generators appearing in
this Hadamard product must be contracted subject to the rule Eq. (3). With Eq.
(3), and the idempotency $\Theta^{2}(t-t^{\prime})=\Theta(t-t^{\prime})$, we
find that the product of $A$\- and $B$-contractions is equivalent to a single
one, $W_{A}\odot W_{B}\to W$, governed by the action Eq. (2), where
$Q=T\tau_{3}T^{-1}$ and $T=\exp(W)$, with the two-body density of states
$\rho\propto N_{A}N_{B}$. In this way, we demonstrate how at time scales
exceeding the coupling time, $t\sim\lambda/\Lambda^{2}$ the effective theory
of an ergodic regime in a Hilbert space of dimension $N_{A}N_{B}$ emerges.

Summary and discussion: We have constructed a path integral describing the
evolution of a tensor product of two weakly coupled random qudits. For times
shorter than a Golden Rule coupling time, the latter evolve as nearly
independent, but individually ergodic subsystems, subject to weak damping due
to the inter-qudit relaxation. However, a crucial feature of the integral is
that it selects temporally synchronous paths which are exempt of damping and
eventually combine to define the ergodic phase of the coupled system.
Alongside the semiclassical picture, these contributions afford a non-
perturbative description in terms of a Goldstone mode, $Q$, representing the
universal ‘causal symmetry breaking principle’ indicative of ergodic quantum
chaos. Turning back to the tension between first and second quantization
raised in the beginning, we note that had we approached the problem in the
former that Goldstone mode would have been visible from the beginning Efetov
(1997). However, in this formulation the opposite short time regime, requiring
the emergence of two independent modes representing the subsystems looks
inaccessible (or at least we have not managed to describe it within this
framework.)

The exploration of a minimal problem prompts considerations regarding the
extensibility to more intricate cases, such as systems in the SYK class —
where the challenge of bridging between short time quasiparticle evolution
Rosenhaus (2019) and long time ergodicity Altland and Bagrets (2018) remains
unresolved — , or networks of coupled circuits. While these must be considered
on a case-by-case basis, a few overarching observations can be made. First,
interacting models subject to statistically independent random coefficients
generically lead to path integrals with Hadamard vertices. (A case in point is
the $G\Sigma$-action of the SYK model, which is governed by an operator
$G^{\odot 4}$ in the notation of the present paper.) Second, we have seen that
the emergence of a universal ergodic Goldstone mode via this operator followed
from a cancellation of self energy (‘out’) and vertex (‘in’) scattering
processes, which in turn is a robust feature of unitarity. On this basis, we
are optimistic that the concepts introduced here, will become instrumental in
the analytical description of more complex use cases of entanglement dynamics.

Acknowledgements:— A.A. acknowledges partial support from the Deutsche
Forschungsgemeinschaft (DFG) through the Cluster of Excellence Matter and
Light for Quantum Computing (ML4Q) EXC 2004/1 390534769 and the CRC network TR
183 (project grant 277101999) as part of project A03. TM acknowledges
financial support by Brazilian agencies CNPq and FAPERJ, and J.T.M. financial
support by Brazilian agency CAPES. Data and materials availability: Processed
data and python script used to generate Fig. 3 are available in Zenodo with
identifier 10.5281/zenodo.10949792 Altland et al. (2024).

## References

  * Chan et al. (2018a) A. Chan, A. De Luca, and J. Chalker, Physical Review Letters 121, 060601 (2018a), ISSN 0031-9007, 1079-7114, URL https://link.aps.org/doi/10.1103/PhysRevLett.121.060601.
  * Nahum et al. (2017) A. Nahum, J. Ruhman, S. Vijay, and J. Haah, Physical Review X 7, 031016 (2017), publisher: American Physical Society, URL https://link.aps.org/doi/10.1103/PhysRevX.7.031016.
  * Vasseur et al. (2019) R. Vasseur, A. C. Potter, Y.-Z. You, and A. W. W. Ludwig, Physical Review B 100, 134203 (2019), publisher: American Physical Society, URL https://link.aps.org/doi/10.1103/PhysRevB.100.134203.
  * Roy et al. (2019) S. Roy, J. T. Chalker, and D. E. Logan, Physical Review B 99, 104206 (2019), publisher: American Physical Society, URL https://link.aps.org/doi/10.1103/PhysRevB.99.104206.
  * Chan et al. (2018b) A. Chan, A. De Luca, and J. T. Chalker, Phys. Rev. X 8, 041019 (2018b), URL https://link.aps.org/doi/10.1103/PhysRevX.8.041019.
  * Garratt and Chalker (2021) S. J. Garratt and J. T. Chalker, Phys. Rev. X 11, 021051 (2021), URL https://link.aps.org/doi/10.1103/PhysRevX.11.021051.
  * Liao and Galitski (2022) Y. Liao and V. Galitski, Physical Review Research 4, L012037 (2022), publisher: American Physical Society, URL https://link.aps.org/doi/10.1103/PhysRevResearch.4.L012037.
  * Haake (2018) F. Haake, _Quantum signatures of chaos, 4th edition_ (Springer-Verlag, Berlin, 2018).
  * Efetov (1997) K. B. Efetov, _Sypersymmetry in Disorder and Chaos_ (Cambridge University Press, Cambridge, 1997).
  * Rosenhaus (2019) V. Rosenhaus, Journal of Physics A: Mathematical and Theoretical 52, 323001 (2019), ISSN 1751-8121, publisher: IOP Publishing, URL https://dx.doi.org/10.1088/1751-8121/ab2ce1.
  * Kamenev (2011) A. Kamenev, _Field Theory of Non-Equilibrium Systems_ (Cambridge University Press, 2011).
  * Kamenev and Andreev (1999) A. Kamenev and A. Andreev, Physical Review B 60, 2218 (1999), ISSN 0163-1829, URL http://link.aps.org/doi/10.1103/PhysRevB.60.2218.
  * Altland and Bagrets (2018) A. Altland and D. Bagrets, Nuclear Physics B 930 (2018), ISSN 05503213.
  * Altland et al. (2024) A. Altland, T. Micklitz, and J. Telles de Miranda, _Data and code for ”Tensor product random matrix theory”_ (2024), URL https://doi.org/10.5281/zenodo.10949793.
  * Berke et al. (2022) C. Berke, E. Varvelis, S. Trebst, A. Altland, and D. P. DiVincenzo, Nature Communications 13, 2495 (2022), ISSN 2041-1723, number: 1 Publisher: Nature Publishing Group, URL https://www.nature.com/articles/s41467-022-29940-y.
  * Altland and Simons (2023) A. Altland and B. Simons, _Condensed Matter Field Theory_ (Cambridge University Press, 2023), ISBN 978-1-108-49460-1.

##  I Supplementary Material

In this supplemental material we provide details of the real time path
integral construction for the tensor product random matrix model.

###  I.1 Real time field integral

We here provide some details of the real time path integral construction of
our tensor model. Starting from the second quantized representation of the
trace over the time evolution operator,
$\mathrm{tr}(G^{\pm}(t))=\bra{0}a_{\mu}b^{\nu}\exp(\pm
iHt-0t)a_{\mu}^{\dagger}b_{\nu}^{\dagger}\ket{0}$, trotterization in terms of
a standard bosonic coherent state representationAltland and Simons (2023),
leads to the representation of the correlation function detailed in the main
text, where the causal index $s$ distinguishes between the two contour
orientations. (For comparison, note that we might just as well have worked
with an algebra of creation operators $\\{X_{\mu\nu}\\}$ as
$\mathrm{tr}(G^{\pm}(t))=\bra{0}X_{\mu\nu}\exp(\pm
iHt-0t)X_{\mu\nu}^{\dagger}\ket{0}$, and coherent states associated to these.
In this representation, we would have ended up with a Gaussian path integral
for the matrix Hamiltonian $H=\\{H_{\mu\nu,\mu^{\prime}\nu^{\prime}}\\}$
(first quantization). However, we have not succeeded to extract the evolution
along approximately independent trajectories in $A$ and $B$ space at early
times from this representation.)

Doing the Gaussian average over $H_{A},H_{B}$ and $H_{AB}$, we arrive at a
path integral with the action

$\displaystyle S[\psi]$
$\displaystyle=\sum_{x=a,b}\bar{x}\tau_{3}(i\partial_{t}+i0\tau_{3})x+\frac{i\lambda^{2}N_{X}}{2}\mathrm{tr}(\Xi_{X}^{2})$
$\displaystyle\qquad+\frac{i\Lambda^{2}N_{A}N_{B}}{2}\mathrm{tr}\left(((\Xi_{A}\odot\Xi_{B})\tau_{3})^{2}\right),$

where
$(\Xi_{A})^{ss^{\prime}}_{tt^{\prime}}=-iN_{A}^{-1}a^{s}_{\mu,t}\bar{a}^{s^{\prime}}_{\mu,t^{\prime}}(-)^{s^{\prime}}$,
$(\Xi_{B})^{ss^{\prime}}_{tt^{\prime}}=-iN_{B}^{-1}b^{s}_{\nu,t}\bar{b}^{s^{\prime}}_{\nu,t^{\prime}}(-)^{s^{\prime}}$.
Here, the traces include summation over all indices, i.e.
$\mathrm{tr}(XY)=\int
dtdt^{\prime}\sum_{ss^{\prime}}X^{ss^{\prime}}_{tt^{\prime}}Y^{s^{\prime}s}_{t^{\prime}t}$,
and similarly $\bar{x}Xx\equiv\int
dtdt^{\prime}\,\bar{x}_{t}X_{tt^{\prime}}x_{t^{\prime}}$. The explicit
representation of the Hadamard product reads as
$((\Xi_{A}\odot\Xi_{B})\tau_{3})^{ss^{\prime}}_{tt^{\prime}}=\Xi^{ss^{\prime}}_{Att^{\prime}}\Xi^{ss^{\prime}}_{Btt^{\prime}}(-)^{s^{\prime}}$.
The dependence of the action on bilinears summed over Hilbert space indices
suggests the introduction of collective variables, $\Xi\equiv G$ via a
Lagrange multiplier locking: $\delta(\Xi_{X}-G_{X})=\int
D\Sigma_{X}\,\exp(-\mathrm{tr}(N_{X}\Sigma_{X}(G_{X}-\Xi_{X}))$. Noting that
$\mathrm{tr}(N_{X}\Sigma_{X}\Xi_{X})=\exp(i\bar{x}\Sigma_{X}\tau_{3}x)$. With
the free action given by
$\sum_{x}\bar{x}\tau_{3}(i\partial_{t}+i0\tau_{3}-\Sigma_{X})x$, the Gaussian
integration over $x=a,b$ produces the ‘$G\Sigma$’ action ( Tensor product
random matrix theory) structurally similar to the action describing the SYK
model Rosenhaus (2019).

###  I.2 Stationary phase action

We consider the stationary phase ansatz $\bar{G}\to
T\bar{G}T^{-1}\approx-\frac{i}{\lambda}T\tau_{3}T^{-1}$, where we used that,
for slowly varying matrices $T$, $(T\bar{G}T^{-1})_{tt^{\prime}}=\int
d\tilde{t}T_{tu}\bar{G}_{u-u^{\prime}}T^{-1}_{u^{\prime}t^{\prime}}\approx-\frac{i}{\lambda}(T\tau_{3}T^{-1})_{tt^{\prime}}\equiv-\frac{i}{\lambda}Q_{tt^{\prime}}$,
on account of the temporal short-rangedness of $\bar{G}$. Likewise,
$\Sigma=-i\lambda Q$. Entering with this representation into the action, the
quadratic contributions $\mathrm{tr}(G_{X}^{2})$ and $\mathrm{tr}(G\Sigma)$
decouple from the Goldstone modes (because $Q^{2}=\mathds{1}$), while the
expansion of the ‘${\rm tr}\ln$’ in $\partial_{t}$ acting on the Goldstone
modes yields

$\displaystyle iN_{X}\mathrm{tr}\ln(i\partial_{t}+i\lambda
Q)=iN_{X}\mathrm{tr}\ln(i\partial_{t}+i\lambda\tau_{3}+iT^{-1}[\partial_{t},T])$
$\displaystyle\qquad\approx
i\frac{N_{X}}{\lambda}\mathrm{tr}(\tau_{3}T^{-1}[\partial_{t},T]).$

Noting that $N_{X}/\lambda=\pi\rho_{X}$ determines the spectral density of the
factor systems at the band center, and turning to an explicit integral
representation of the trace, we obtain Eq. (2).

###  I.3 $N_{X}^{-1}$-expansion

The effect of the Hadamard vertex is best studied within in the framework of a
perturbative expansion of the Goldstone modes in generators, $T=\exp(W)$,
$W=B\tau_{+}-B^{\dagger}\tau_{-}$, which, in view of the proportionality of
the action to $N_{X}$ is an $N_{X}^{-1}$ expansion. Considering its structure
in causal space, $\mathrm{tr}((G_{A}\odot
G_{B})^{2})=\sum_{ss^{\prime}}\mathrm{tr}(G_{A}\odot
G_{B})^{ss^{\prime}}(G_{A}\odot G_{B})^{s^{\prime}s}$, we realize that we get
two structurally (and as we discuss in the main text physically) different
contribution of lowest, quartic order in $B$: in the terms $s=s^{\prime}$ of
equal causality we expand one of the two factors $(G_{A}\odot
G_{B})^{ss}\to\bar{G}^{s}_{A}\odot\bar{G}^{s}_{B}$ to zeroth order in the
Goldstone mode generators, and the other to quartic order, e.g., $(G_{A}\odot
G_{B})^{ss}\to-(\lambda_{A}\lambda_{B})^{-1}(B_{A}B_{A}^{\dagger})\odot(B_{B}B_{B}^{\dagger})$.
We note that the element-wise Hadamard product does not admit other terms at
quartic order. Substituting the time representation of $\bar{G}_{X}$, we
arrive at the first of Eqs. (4). Turning to the causality mixing channel,
$(G_{A}\odot G_{B})^{+-}(G_{A}\odot G_{B})^{-+s}$, the expansion
$G_{X}^{+-}\to i\lambda_{X}B$ and $G_{X}^{-+}\to i\lambda_{X}B^{\dagger}$
immediately leads to the second line of Eq. (4). In the same way, we obtain
the Gaussian expansion of the free action stated above Eq. (3) from the
Goldstone mode action Eq. (2). In the calculation of numerical coefficients of
the coupling contants we assume that $B_{t,t^{\prime}}$ is a slow function in
$t+t^{\prime}$ and independent of $t-t^{\prime}$, which e.g. gives rise to the
Fermi’s Golden Rule coupling rate $\gamma=2\pi\rho(0)\Lambda^{2}$, with
$\rho(0)$ the density of states at energy $\epsilon=0$. A more accurate
calculation accounting for all time-dependencies would rather lead to an
energy-averaged density of states, but details are beyond the scope of this
work.


# Fine color guidance in diffusion models and its application to image
compression at extremely low bitrates

Tom Bordin
INRIA
Rennes, France  Thomas Maugey  INRIA
Rennes France

###### Abstract

This study addresses the challenge of, without training or fine-tuning,
controlling the global color aspect of images generated with a diffusion
model. We rewrite the guidance equations to ensure that the outputs are closer
to a known color map, and this without hindering the quality of the
generation. Our method leads to new guidance equations. We show in the color
guidance context that, the scaling of the guidance should not decrease but
remains high throughout the diffusion process. In a second contribution, our
guidance is applied in a compression framework, we combine both semantic and
general color information on the image to decode the images at low cost. We
show that our method is effective at improving fidelity and realism of
compressed images at extremely low bit rates, when compared to other classical
or more semantic oriented approaches.

##  I Introduction

Image generation has made significant progress, mainly due to the emergence of
diffusion models[1]. Those generative methods rely heavily on computations and
data in order to produce the highest image quality. Therefore, when using the
most recent networks for other applications, it can be interesting to avoid
retraining or fine-tuning it but rather adapt the models. The current popular
and widely available conditioning, done through text or image, is often not
enough for any applications that require finer control. Getting control on the
outputs without retraining thus becomes a challenge. One of the difficulty of
controlling image generation lies in the fact that most recent models are
trained to work in the latent space of a fixed VAE, as proposed in [2], those
models are called Latent Diffusion Models(LDMs). By reducing the dimension
considered, the model can be shrunk, and the generation process can be sped
up, but this comes at the expense of interpretability and control. The
solutions for controlling outputs in pixel space are not always relevant for
LDMs, while unfortunately most current state-of-the-art models are trained in
this configuration.

Figure 1: Two images generated with a Diffusion Model, conditioned by the same
semantics (rocky mountain and cloudy sky). Contrary to the left image, the
right image is obtained by guiding the diffusion model towards a general color
aspect (top right).

In this paper, we focus on a specific type of control on LDMs, we aim at
generating an image that is coherent with a given general color aspect which
is given by a color map, we illustrate the concept in Fig. 1, adding control
to an already semantic conditional generation. Control through color maps can
be a useful tool for image generation, especially in the context of
compression, where, a really coarse color map can be used as a base to
generate details. There already exist some options available to control color
for diffusion in the pixel space, such as generating only the high frequencies
using a decomposition of the image proposed by the authors of [3], or using a
specific scheduler for the diffusion model [4]. However, to the best of our
knowledge, a good control of the color in LDM without training, has not been
presented. Some works propose training methods to add control to diffusion
models, by learning the directions in the latent space associated with
semantic meaning[5] or by adding a condition as input[6]. However, we do not
want to rely on training or fine-tuning a new model on the specific task of
color map conditioning, as this would not take advantage of the most recent
models that could be trained in the future. We want to focus on a training-
free control that is adaptable to any diffusion model, and in particular to
LDMs. In SdEdit[7], the authors propose a method that provides some control
over the color of the output using cartoon-like color maps, but this control
comes at the cost of a tradeoff between the quality of the generation and the
fidelity to the color condition. Exploiting classifier guidance[8] which
requires training, in [9], the authors propose a universal guidance, with a
training-free method to control any diffusion models for any condition. In the
specific case of color guidance, we noticed that among the existing methods,
only a part of the information used for control is preserved when guiding the
generation.

We propose a new formulation of the guidance specific to color map control. We
rewrite the equations inspired by universal-guidance[9], and show that, for
color maps, contrary to the general case, the scaling of the guidance term
should not decrease during diffusion. We call our method fine color guidance,
we show that we make use of most of the information provided by the color map,
without affecting the quality of the generated images.

Since the distortion perception tradeoff highlighted by Blau.et al[10], and
more particularly at extremely low bitrates [11], compression has been taking
a step back from the MSE as a reconstruction criterion and focusing on
semantic descriptors[12, 13]. One way of depicting an image is through a high-
level color description, giving only a global aspect of what it contains
(e.g., a blue spot on the right, green in the middle, blue on the top, etc.).
Controlling the color map in the generation of images becomes interesting in
that case. Inspired by the framework CoCliCo based on a CLIP and color
semantic proposed in [14], we propose to apply our fine-color guidance to
improve on image compression. We compare the compression framework with
another recent generative approach[15] and a classical encoder VVC[16] by
targeting similar bitrates.

After a brief reminder of how diffusion models work, we present the
formulation of our problem in section II. We then compare existing color
control solutions in section III. We present our proposed method for fine-
guidance in section IV and its application to image compression in an ultra-
low bit-rate framework in section V. Finally, in section VI, we compare our
fine color guidance with other existing methods and their impact on
compression.

##  II Controlling colors in diffusion models

Our goal is to add control, without requiring training, to any image diffusion
model by using color information. Since the latest models work in the latent
space of a VAE, control should also be available in these contexts. First, let
us recall how diffusion models work.

###  II-A Diffusion models

Diffusion models stand as denoisers, starting from a random gaussian noise,
iteratively converging towards a clean signal. With the popularity of
diffusion models, the need for fast generation and fast training has
encouraged the emergence of latent diffusion models[2], diffusing in the
latent space of a fixed VAE. In the following, to be more concise, we consider
that pixel space diffusion is a specific case of latent diffusion where the
VAE is the identity, see Fig. 2.

Figure 2: Illustration of the latent diffusion models. Diffusion in the domain
of pixels becomes a particular case where there is no encoder or decoders
$E_{ldm}=D_{ldm}=I$

####  II-A1 Noise model

Diffusion models learn to invert an iterative noise process on the signal. The
model denoises on a series of timesteps $[0,T]$ with scaling noise controlled
by the parameters $\\{\alpha_{t}\\}_{t=0}^{T}$. For a given signal
$\bm{z}_{0}$ and at a given timestep $t$, the added noise follows the
schedule:

$\displaystyle\bm{z}_{t}=\sqrt{\alpha_{t}}\bm{z}_{0}+\sqrt{1-\alpha_{t}}\epsilon$
(1) $\displaystyle\text{where \hskip
6.0pt}\epsilon\sim\mathcal{N}(\bm{0},\bm{I})$

The mean of $\bm{z}_{t}$ is progressively shifting towards $0$ while the
amount of noise is increasing with $t$. Going from the value of the signal
$\bm{z}_{0}$ at $t=0$ to a normal distribution at $t=T$. The diffusion model
reverse this noise by estimating the value of $\epsilon$ at each step.

####  II-A2 Standard diffusion models

The diffusion model learns to predict the noise $\epsilon$ added to
$\bm{z}_{t}$ in (1) from the observation of the pair $(\bm{z}_{t},t)$. The
training loss for the timestep $t$ can be written as:

$\mathcal{L}_{t}(\theta)=\mathbb{E}_{q(\bm{x}_{0});\epsilon\sim\mathcal{N}(\bm{0},\bm{I})}\left[\left\|\epsilon-\epsilon_{\theta}(\bm{x}_{t}(\bm{x}_{0},\epsilon),t)\right\|^{2}\right]$
(2)

When generating an image with a diffusion model, the noising process is
reversed. Starting from a random gaussian noise as $\bm{z}_{T}$, we
iteratively estimate the next sample $\bm{z}_{t-1}$ until reaching $t=0$.
Different methods – called schedulers – can be used to estimate
$\bm{z}_{t-1}$, for instance in denoising diffusion implicit model (DDIM)[17]:

$\bm{z}_{t-1}=\sqrt{\alpha_{t-1}}\underbrace{\left(\frac{\bm{z}_{t}-\sqrt{1-\alpha_{t}}\epsilon_{\theta}(\bm{z}_{t},t)}{\sqrt{\alpha_{t}}}\right)}_{\text{"predicted
$\bm{z}_{0}:\hat{\bm{z}}_{0}(\bm{z}_{t},t)$"}}+\underbrace{\sqrt{1-\alpha_{t-1}}\epsilon_{\theta}(\bm{z}_{t},t)}_{\text{"direction
pointing to $\bm{z}_{t}$"}}$

(3)

We are able to estimate a predicted signal $\hat{\bm{z}}_{0}(\bm{z}_{t},t)$ at
each step of the generation process, refining the prediction as the process is
repeated. Reaching the timestep $0$, we get a clean signal $\bm{z}_{0}$, which
can de be decoded by the VAE to obtain a generated image $\bm{x}_{0}$.

###  II-B Formulation

Using an already trained and fixed diffusion model $\epsilon_{\theta}$, that
can generate images $\bm{x}_{0}(\epsilon_{\theta})$. We would like, through
the use of a color map $\bm{c}$, to control the outputs of the model in order
such that the generated image has the global appearance of the color map. We
also would like that this control should not come at the expense of the
generative power of the model, hindering the realism or quality of images.
Control should be achieved without retraining, keeping the parameters $\theta$
of the model intact, but modifying the noise estimation, noted as
$\tilde{\epsilon}_{\theta}(\bm{c})=\tilde{\epsilon}_{\theta}(\epsilon_{\theta},\bm{c})$
to relax the notations. The problem can then be formulated as:

$\displaystyle\min_{\tilde{\epsilon}_{\theta}}d(\bm{c},\bm{x}_{0}(\tilde{\epsilon}_{\theta}(\bm{c})))\text{
s.t. }$ (4)
$\displaystyle\Psi(\bm{x}_{0}(\epsilon_{\theta}))=\Psi(\bm{x}_{0}(\tilde{\epsilon}_{\theta}(\bm{c})))$

where $d$ measures the consistency between a color map and an image, and
$\Psi$ measures realism. In other words, the control function should minimize
the error of color map while keeping the generative power. Since the control
is done during the diffusion process, the estimation of the correction has to
be assessed using noisy observation of the latent image $\bm{z}_{t}$, this
additional level of abstraction makes it difficult to have an explicit
function.

##  III Existing solutions

Several solutions already address the problem previously formulated, however,
none of them satisfies all the conditions: no-training, functioning for LDMs
and minimizing the error. We present the pros and cons of several methods in
our context and motivate the need for a specific method for color guidance,
especially in the case of latent diffusion models.

(a) Standard diffusion

(b) Enforced conditioning

(c) Initialized diffusion

(d) Guided diffusion

Figure 3: Illustration of different method to control the diffusion process.
The region of possible sample is illustrated in blue. The black arrow shows a
possible diffusion schedule. The first figure shows a standard diffusion
process with no control. Our goal is to focus the output region as close as
possible of $\bm{z}_{0}|\bm{c}$ as possible.

###  III-A Conditional diffusion models

A conditional diffusion model follows the same principle as a standard
diffusion model, the main difference is that the prediction of the noise is
done over $(\bm{z}_{t},t,\bm{c})$. The condition, $\bm{c}$ is then given as
input of the diffusion model at each timestep, with parameters $\theta$
trained to take $\bm{c}$ into account. Conditional models are widely popular
for text-to-image generation or image-to-image generation application,
however, to get a conditional diffusion model, it requires either retraining a
whole network or fine-tuning an already trained one on the desired condition.
Which is something we would like to avoid in our framework.

###  III-B Enforcing colors in diffusion model

Given a diffusion model $\epsilon_{\theta}$, it is sometime possible to
enforce a condition on the output of the model, ensuring that the output
exactly follows our condition. For color map conditioning, it is the case, but
only when the diffusion process is done in the pixel space, which means
trained to directly generate images from noise without the auto-encoder. Then,
for a given image $\bm{z}_{0}$, we can separate it as the sum of its lower and
higher frequencies uniquely $\bm{z}_{0}=l(\bm{z}_{0})+h(\bm{z}_{0})$ for
specified functions $l,h$, for instance, authors in [3] propose to use a
wavelet decomposition. It is then possible to enforce the value of the low (or
high frequencies) of the generated image by replacing at each step the
generated low frequencies by our target $\bm{c}$. With
$\hat{\bm{z}}_{0}(\bm{z}_{t},t)$ the predicted $\bm{z}_{0}$ in (3), we edit
the value at each step, replacing it by:

$\hat{\bm{z}}_{0}(\bm{z}_{t},t,\bm{c})=\bm{c}+h(\hat{\bm{z}}_{0}(\bm{z}_{t},t))$
(5)

At each step, the low frequencies of the sample are the one we enforce, and
only the higher frequencies are the one generated. Enforced diffusion is
illustrated in Fig. 3(b), where vertical movement in the arrow indicates the
correction of the low frequencies. This method is similar to the inpainting
process described in [18], where a mask is used to do the partitioning of
$\bm{z}_{0}$ rather than a DCT. This method can be used when the constraint on
$\bm{c}$ needs to be strong and when the diffusion is done in the pixel space.
This does not allow a trade-off and to scale the proximity with the condition.
Moreover, with the popularity of diffusion models, the speed of their training
and sampling time has been a major concern and most of the models are now
processing in the latent space of VAE to reduce the dimension, then, the same
partitioning is no longer available.

###  III-C Initialized DM

Another method to control the generation of the diffusion model is
initialization. First introduced in [7], the diffusion process can be
initialized with information on the signal instead of using a random noise.
With this method, we influence the diffusion process by giving partial
information on the signal we desire to get. In practice, it is done by
starting the generation at a later timestep $0<\tau<T$ with information on the
color degraded at the corresponding scaling of noise:

$\bm{z}_{\tau}=\sqrt{\alpha_{\tau}}E_{ldm}(\bm{c})+\sqrt{1-\alpha_{\tau}}\epsilon$
(6)

Depending on the quantity of information we want to preserve, we can adjust
the starting timestep to add more or less noise. This method is quite simple,
does not require training, and does even shorten the generation time, however,
the generation process is not controlled towards the output and can still
drift away from the initialization as illustrated in the diagram Fig. 3(c).
Also, authors in [7] shows the presence of a tradeoff between the fidelity to
the condition $\bm{c}$ and the quality of the generated signal. This tradeoff
depends on the starting timestep $\tau$. Indeed, the higher the timesteps, the
more noise we add and the further we get from $\bm{c}$ and inversely.

###  III-D Guidance for diffusion model

Instead of initializing as in III-C, or replacing as in III-B, authors in [8]
propose to guide the diffusion at each iteration towards an image that fits
the color map condition. Moreover, in [9] authors propose a method they call
universal to guide the output towards a condition without any training.

First introduced as classifier guidance, to condition on a set of classes,
guidance can be extended to any type of condition. The only requirement is to
be able to estimate the value of $\nabla_{\bm{z}_{t}}\log
p_{t}(\bm{c}|\bm{z}_{t})$, the gradient of the log likelihood of the condition
$\bm{c}$ given a noisy observation $\bm{z}_{t}$. While conditional models
(III-A) can learn to estimate this term separately through classifier-free
guidance[19], classifier guidance uses a trained classifier to do so. A guided
diffusion model process is illustrated in Fig. 3(d), the guidance correction
is applied after each timestep, iteratively getting closer to the condition.

The term for guidance comes from the parallel between the diffusion model
$\epsilon_{\theta}$ with energy based models. Indeed, if $\bm{f}_{\theta}$ is
an energy based model trained to predict the log likelihood $\log p(\bm{x})$,
the corresponding training loss is formulated as:

$\mathcal{L}_{\sigma}(\theta)=\mathbb{E}_{\bm{x}\sim
q(\bm{x})\epsilon\mathcal{N}(\bm{0},\bm{I})}\left[\left\|\frac{\epsilon}{\sigma}+\nabla_{\bm{x}}\bm{f}_{\theta}(\bm{x}+\sigma\epsilon)\right\|^{2}\right]$
(7)

Putting the training losses of $\epsilon_{\theta}$ (2) and $f_{\theta}$ (7) in
parallel, we can conclude that:

$\epsilon_{\theta}(\bm{z}_{t},t)=-\sqrt{1-\alpha_{t}}\nabla_{\bm{z}_{t}}\log
p_{t}(\bm{z}_{t})$ (8)

And using the Bayes formula:

$\nabla_{\bm{z}}\log(p(\bm{z}|\bm{c}))=\nabla_{\bm{z}}\log(p(\bm{z}))-\nabla_{\bm{z}}\log(p(\bm{c}|\bm{z}))$
(9)

From which we can then get the classifier guidance formula [20] to control the
generation:

$\tilde{\epsilon}_{\theta}(\bm{z}_{t},t,\bm{c})=\epsilon_{\theta}(\bm{z}_{t},t)-\sqrt{1-\alpha_{t}}\nabla_{\bm{z}_{t}}\log
p_{t}(\bm{c}|\bm{z}_{t})$ (10)

The impact of the guidance is often scaled by a scaling factor $s$,
$s\cdot\sqrt{1-\alpha_{t}}\nabla_{\bm{z}_{t}}\log p_{t}(\bm{c}|\bm{z}_{t})$.
Increasing the scaling factor then focuses the outputs more towards fidelity
to the condition, at the cost of the diversity of generation. To simplify and
shorten notations, we will note $G(\cdot,\cdot,\cdot)$ the guidance correction
operator:

$G(\bm{z}_{t},t,\bm{c})=-\sqrt{1-\alpha_{t}}\nabla_{\bm{z}_{t}}\log
p_{t}(\bm{c}|\bm{z}_{t})$ (11)

Authors in [5] propose to measure the log likelihood using the predicted
signal $\hat{\bm{z}}_{0}(\bm{z}_{t},t)$ (3) rather than the noisy signal
$\bm{z}_{t}$. In [9] the authors propose to go further and to approximate the
log likelihood $\log p_{t}(\bm{c}|\hat{\bm{z}}_{0}(\bm{z}_{t},t))$ by a
distance between the estimated condition on the predicted signal
$\hat{\bm{c}}_{t}=\bm{c}(\hat{\bm{z}}_{0}(\bm{z}_{t},t))$, and the given
condition. In the case of color guidance, with their formulation, the
correction becomes:

$G(\bm{z}_{t},t,\bm{c})=s\sqrt{1-\alpha_{t}}\nabla_{\bm{z}_{t}}\|\bm{c}-\hat{\bm{c}}_{t}\|^{2}$
(12)

where the scale $s$ is specific to colors and needs to be determined
empirically. Their approximation comes from the intuition that if the
estimated condition is close to the given condition, then the probability that
the signal follows this condition should be high. However, this formulation
highly varies with the type of condition we give, and obtaining the right
value for $\lambda$ is detrimental to guide the model.

##  IV Proposed fine color guidance for diffusion models

Inspired by the formulation of universal guidance(12), we propose to study how
the estimation of guidance could be formulated in the specific context of
color conditioning. We first model the color map before unrolling the formula
for guidance.

###  IV-A Modeling the color map

The color map $\bm{c}$ represents coarse information on the color of an image
$\bm{x}$. For our problem, we define the color map as the low frequencies of
the image. Using the 2d-DCT, $\bm{x}=\bm{V}\bm{D}\bm{V}^{T}$, we extract the
$m$ lowest frequencies in the DCT. In other words:

$\bm{c}=\bm{A}\bm{x}$ (13)

is a $m$ dimension vector. With $\bm{A}$ a being thresholding operator on the
frequencies in the DCT basis, which can be written as:

$\bm{A}=\bm{V}_{[m,n]}^{g}\bm{I}_{m}\bm{V}^{T}_{[m,n]}$ (14)

where $\bm{V}_{[m,n]}$ stands for the sub-matrix containing the $m$ first rows
and $n$ first columns of $\bm{V}$, and where $\bm{V}_{[m,n]}^{g}$ is the
generalized inverse of $\bm{V}_{[n,m]}$. In the following, we consider that
$\bm{c}$ is a fixed color map of size $m$.

###  IV-B Formulation

The formulation, previously defined in section II in equation (4), can now be
adapted and simplified. With the notation of guidance defined in (10) and
(11), the formulation becomes:

$\displaystyle\min_{G}||\bm{c}-\bm{A}\bm{x}_{0}(\tilde{\epsilon}_{\theta}(\bm{c}))||_{2}^{2}\text{
s.t. }$ (15)
$\displaystyle\Psi(\bm{x}_{0}(\epsilon_{\theta}))=\Psi(\bm{x}_{0}(\tilde{\epsilon}_{\theta}(\bm{c})))$

we measure the distance between the conditioning color map and the color map
of the output with a simple $L\texttwosuperior$ norm. We add a guidance term
$G$ to the diffusion model to modify the noise estimation
$\tilde{\epsilon}_{\theta}$. The problem lies in writing the explicit
formulation of the guidance function $G$ optimally.

###  IV-C Unrolling classifier guidance equations

The guidance of a diffusion model relies on the classifier guidance expression
in (10). As proposed in [5], we use the predicted signal for the estimation in
(3) rather than the noisy signal.

####  IV-C1 The predicted signal

$\hat{\bm{z}}_{0}(\bm{z}_{t},t)$
We know from (1) that for a given sample $\bm{z}_{0}$, the noisy samples
$\bm{z}_{t}$ at timestep $t$ follow a multivariate normal distribution:

$\bm{z}_{t}\sim\mathcal{N}\left(\sqrt{\alpha_{t}}\bm{z}_{0},(1-\alpha_{t}){\bm{I}}\right)$
(16)

Moreover, at any timestep $t$, we can build a prediction $\hat{\bm{z}}_{0}$ of
the value of $\bm{z}_{0}$. Using the expression in (3):

$\hat{\bm{z}}_{0}(\bm{z}_{t},t)=\frac{\bm{z}_{t}-\sqrt{1-\alpha_{t}}\epsilon_{\theta}(\bm{z}_{t},t)}{\sqrt{\alpha_{t}}}$
(17)

If the diffusion model was ideal, then $\epsilon_{\theta}(\bm{z}_{t},t)$ would
predict exactly the noise added $\epsilon$ and we would have
$\hat{\bm{z}}_{0}(\bm{z}_{t},t)=\bm{z}_{0}$. In practice, however, noisy
signals can overlap for different images, especially for higher value of $t$
with a complete overlap at $T$. The model will not be able to correctly
predict $\epsilon$ because the solution is not unique. We can thus consider
that the prediction error $\Delta\epsilon_{t}$ is non-zero. We can then
rewrite $\hat{\bm{z}}_{0}(\bm{z}_{t},t)$:

$\displaystyle\hat{\bm{z}}_{0}(\bm{z}_{t},t)$
$\displaystyle=\frac{\sqrt{\alpha_{t}}\bm{z}_{0}-\sqrt{1-\alpha_{t}}(\epsilon_{\theta}(\bm{z}_{t},t)-\epsilon)}{\sqrt{\alpha_{t}}}$
$\displaystyle=\bm{z}_{0}-\frac{\sqrt{1-\alpha_{t}}}{\sqrt{\alpha_{t}}}\Delta\epsilon_{t}$
(18)

We measured the distribution of the error for a given diffusion model, and we
observed that depending on the value of $t$:

$\Delta\epsilon_{t}\sim\mathcal{N}(0,\overline{\lambda}_{t}^{2}{\bm{I}})$ (19)

with $\overline{\lambda}_{t}$ measured empirically over a set of images, and
probably specific to the model. Thus, the predicted sample
$\hat{\bm{z}}_{0}(\bm{z}_{t},t)$ follows a gaussian distribution centered on
the real value $\bm{z}_{0}$:

$\hat{\bm{z}}_{0}(\bm{z}_{t},t)\sim\mathcal{N}\left(\bm{z}_{0},\overline{\lambda}_{t}^{2}\frac{1-\alpha_{t}}{\alpha_{t}}{\bf
I}\right)$ (20)

We now have the distribution of the predicted signal given the noisy
observation. Let us now move on to the estimation of the density probability:
$p_{t}(\bm{c},\hat{\bm{z}}_{0}(\bm{z}_{t},t))$.

###  IV-D Guidance for pixel space diffusion

In the pixel space (i.e $\bm{z}=E_{ldm}(\bm{x})=\bm{x}$), at the timestep $t$,
we can evaluate $\hat{\bm{c}}_{t}$ the estimated color map over the predicted
image $\hat{\bm{x}}_{0}(\bm{x}_{t},t)$, in other words, the predicted color
map:

$\displaystyle\hat{\bm{c}}_{t}(\bm{x}_{t},t)$
$\displaystyle=\bm{A}\hat{\bm{x}}_{0}(\bm{x}_{t},t)$
$\displaystyle=\bm{c}+\overline{\lambda}_{t}\frac{\sqrt{1-\alpha_{t}}}{\sqrt{\alpha_{t}}}\bm{A}\epsilon$
(21)

The distribution of the estimated color map is then also a gaussian centered
on the targeted color map $\bm{c}$. Which can also be written as:

$\bm{c}|\bm{x}_{t}\sim\mathcal{N}\left(\hat{\bm{c}}_{t},\overline{\lambda}_{t}^{2}\frac{1-\alpha_{t}}{\alpha_{t}}{\bm{I}_{m}}\right)$
(22)

We now have an explicit formula of the distribution of $\bm{c}|\bm{x}_{t}$. We
also get the value of $p_{t}(\bm{c}|\bm{x}_{t})$, the probability density of a
multivariate distribution, that we were looking for:

$p_{t}(\bm{c}|\bm{x}_{t})=\frac{1}{\sqrt{(2\pi)^{m}}}\exp\left(-\frac{1}{2\overline{\lambda}_{t}}\frac{\sqrt{\alpha_{t}}}{\sqrt{1-\alpha_{t}}}\|\bm{c}-\hat{\bm{c}}_{t}\|^{2}_{2}\right)$
(23)

We can then use (23) in the guidance term in (11). The guidance correction
term for a color map in a pixel space diffusion model then becomes:

$G(\bm{x}_{t},t,\bm{c})=\frac{\sqrt{\alpha_{t}}}{2\overline{\lambda}_{t}}\nabla_{\bm{x}_{t}}\|\bm{c}-\hat{\bm{c}}_{t}\|^{2}_{2}$
(24)

We can see that the correction term to guide the color in the diffusion model
is similar to the one proposed in universal guidance (12), as both uses a
$L^{2}$ norm. Though, there are still substantial differences. Firstly, the
coefficient in front of the norm scales differently with $t$. Indeed, in (12),
$\sqrt{1-\alpha_{t}}$ decreases, guiding less as the diffusion progresses,
whereas we observed that $\frac{\sqrt{\alpha_{t}}}{2\overline{\lambda}_{t}}$
preserve the importance of guidance along the process. Moreover, once
$\overline{\lambda}_{t}$ is determined, there is no need for tuning any
parameter in the loss. Guidance in pixel space brings fewer guarantees than an
enforced approach such as [3]. However, since it takes less advantage of the
image in pixel space, it can be adapted to latent diffusion models and be
applied to the state-of-the-art models.

###  IV-E Guidance for latent diffusion

While in the pixel space we had access to an estimated image, here, we only
have an estimated latent vector. However, we can still approximate an image as
$\hat{\bm{x}}_{0}=D_{ldm}(\hat{\bm{z}}_{0})$, the decoded predicted latent.
Similarly to what we had in the pixel space, we would like to know the
distribution of the predicted images depending on the timestep. To do so,
since the distribution of $\hat{\bm{z}}_{0}(\bm{z}_{t},t)$ at each timestep is
a multivariate normal, we look at the response of the decoder $D_{ldm}$ when
gaussian noise is added to the latent vector. In other words, we observe the
behavior of the decoder to $\bm{z}_{\lambda}\sim\mathcal{N}(\bm{z},\lambda I)$
for a wide range of $\lambda$ and images $\bm{z}$. We found that for small
increase in noise, the decoded image is not far from the original one, while
still recovering gaussian noise at the output. Moreover, we model the amount
of noise after the decoding as a linear function of $\lambda$, estimating the
parameters $\overline{a},\overline{b}$ such as:

$D_{ldm}(\bm{z}_{\lambda})\sim\mathcal{N}\left(D_{ldm}(\bm{z})+\overline{a}\lambda,\overline{b}^{2}\lambda^{2}\bm{I}\right)$
(25)

Adding noise in the latent space thus provokes a shift of the mean in the
decoded image. The decoded image distribution becomes:

$\hat{\bm{x}}_{0}(\bm{x}_{t},t)\sim\mathcal{N}\left(\bm{x}_{0}+\overline{a}\overline{\lambda}_{t}\frac{\sqrt{1-\alpha_{t}}}{\sqrt{\alpha_{t}}},\overline{b}^{2}\overline{\lambda}_{t}^{2}\frac{1-\alpha_{t}}{\alpha_{t}}I\right)$
(26)

From here, the computations are similar to what we had in the pixel space,
just taking into account the shift of the mean, the predicted color map
becomes:

$\displaystyle\hat{\bm{c}}_{t}(\bm{x}_{t},t)$
$\displaystyle=\bm{A}\hat{\bm{x}}_{0}(\bm{x}_{t},t)$
$\displaystyle=\bm{c}+\overline{a}\overline{\lambda}_{t}\frac{\sqrt{1-\alpha_{t}}}{\sqrt{\alpha_{t}}}\bm{A}\mathbbm{1}_{hw}+\overline{b}\overline{\lambda}_{t}\frac{\sqrt{1-\alpha_{t}}}{\sqrt{\alpha_{t}}}\bm{A}\epsilon$
(27)

and the distribution of the estimated color map becomes:

$\hat{\bm{c}}_{t}(\bm{x}_{t},t)\sim\mathcal{N}\left(\bm{c}+\overline{a}\overline{\lambda}_{t}\frac{\sqrt{1-\alpha_{t}}}{\sqrt{\alpha_{t}}}\bm{A}\mathbbm{1}_{hw},\overline{b}\overline{\lambda}_{t}\frac{\sqrt{1-\alpha_{t}}}{\sqrt{\alpha_{t}}}{\bf
I}\right)$ (28)

The guidance correction term using a color map in a LDM then is:

$None$

(29)

The guidance correction term is similar to the pixel space (24). The scaling
is slightly different, and we take into account the shifting of the mean
produced by the decoder. For an ideal pair of encoder and decoder, we then
have $(\bar{a},\bar{b})=(0,1)$, which results in the pixel space guidance.
When compared to universal guidance (12) we can see that the scaling
coefficient in front of the gradient still differs from the one they proposed.
The value of the guidance should not decrease with the value of $t$ in
$\sqrt{1-\alpha_{t}}$ but rather follow the ratio
$\sqrt{\alpha_{t}}/\overline{\lambda_{t}}$ which remains high along $t$ for
the diffusion model observed. We present, in Fig. 4, the comparison between
the two guidance schedule, contrary to universal-guidance, the value does not
reach $0$ towards the last steps. In other words, the theory states that one
should guide the generation with the color along the diffusion process and not
only at the start. We note that this observation is specific to our
formulation of the color maps, but could also be generalized to other
conditions computed as a matrix product.

Figure 4: Scaling of the gradient in guidance. In universal-guidance, the
scaling decreases to $0$, we observed that theoretically, guidance for color
should follow another guidance schedule.

##  V Application to compression

The most recent diffusion models can now generate a high variety of images
with impressive quality [21]. When applied to generative compression, they can
either be used at high bitrates to generate small details and avoid artifacts
[22], and at lower bitrates, they can be used as decoder using semantic
information for conditional generation [23, 13, 24]. Our definition of color
map IV-A, particularly aligns with image compression purposes. Here, we
propose to follow the pipeline of CoCliCo[14] and we represent an image as a
combination of semantic and color information. However, to best harness the
information on the color that is given, we propose to rely on our formulation
of guidance to condition the generation.

###  V-A Compression framework

The image is encoded into a vector pair $\sigma=(\sigma_{s},\sigma_{c})$ of
semantic and color information, it is decoded using a generative decoder
conditioned on the semantic, and guided to correspond to the color. The goal
of the compression becomes to generate images with high realism while
preserving the maximum semantic information contained in $\sigma$ at a given,
extremely low, bitrate.

Figure 5: Encoding-decoding pipeline of our framework. Our guidance term $G$
is added over the CoCliCo framework based on CLIP and color semantic. We
provide a color correction during generation.

We call $\Phi$ the semantic projection of an image. The range of the function
$\Phi$ then defines a semantic space. We measure the loss of semantic as a
distance $d$ in this semantic space. Using this definition, noting
$\mathcal{E},\mathcal{D}$ the encoder and decoder
${\bm{x}}_{\rm{out}}=\mathcal{D}(\mathcal{E}(\bm{x}_{\rm{in}}))$, we can then
formulate the problem of image compression as:

$\displaystyle\min_{\mathcal{D}}d(\Phi(\bm{x}_{\rm{in}}),\Phi(\bm{x}_{\rm{out}}))\text{
s.t. }$ (30) $\displaystyle\Psi(\bm{x}_{\rm{out}})<\Psi_{min}\text{ and
}R(\mathcal{E}(\bm{x}_{\rm{in}}))<R_{min}$

where $R$ is the function of the rate and $\Psi$ measures the realism of an
image. Similarly to the CoCliCo[14] framework, we want to minimize the loss of
semantic while preserving high realism at a given rate. In our context, the
decoder is a fixed function given by the diffusion model that can be
influenced by the guide $G$. The rate is measured on the encoded image, i.e,
on the color map and the semantic. Seeing the general color aspect as semantic
information, the distance $d$ defined here in (30) encompasses the color
metric defined in section II in (4).

###  V-B Representation of the semantic

The image is encoded in two parts: the semantic $\sigma_{s}$ and the color
$\sigma_{c}$. We already described the choice and motivation for the color
map, but we haven’t discussed the semantic yet. As proposed in the CoCliCo
framework [14], we choose to use the foundation model CLIP[25] to encode the
semantic of an image. This choice is motivated firstly by the capacity of
abstraction of this model but also by its popularity. Indeed, as CLIP is used
to encode images and text in the same projection space, this model has been an
advantageous choice for text representation in recent text-to-image models,
which lead to a large amount of available trained models.

In Fig. 5, we illustrate the proposed encoding-decoding pipeline. On one hand,
$\sigma_{s}$ is extracted using CLIP as a semantic encoder $\mathcal{E}_{s}$,
and on the other the color map $\sigma_{c}$ is computed using frequency
decomposition. Both the CLIP vector and the color map are quantified as
illustrated in Fig. 5 to reduce the rate. The color map is encoded as an
$m\times m$ image in YUV 4:2:0 format with a uniform color palette over
$b_{c}$ bits per channel. The CLIP latent vector is clamped then uniformly
quantized on $b_{s}$ bits per dimension as:

$\displaystyle\sigma_{c}^{q}=\lfloor\frac{\sigma_{c}}{b_{s}}\rfloor
q+\frac{b_{s}}{2}$ (31)

With a clip vector of dimension $768$, the rate of an image then becomes:

$\displaystyle R(\sigma^{q})$
$\displaystyle=R(\sigma_{c}^{q})+R(\sigma_{s}^{q})$
$\displaystyle=-\log_{2}(b_{s})*768+1.5*b_{c}*m^{2}$ (32)

where the 1.5 coefficient comes from the YUV 4:2:0 format.

###  V-C Decoding the semantic with a guided diffusion model

The diffusion model used for decoding thus takes two conditions as inputs: a
learned semantic conditioning on $\sigma_{s}$, and a guided color conditioning
on $\sigma_{c}$. The conditional diffusion model $\epsilon_{\theta}$ predicts
the noise $\epsilon$ from the triplet $(\bm{z}_{t},t,\sigma_{s})$ as described
in section III-A. Our contribution lies in the correction term using the color
map guidance term (24) or (29) respectively for image and latent diffusion
during the generative decoding of images. We then have the following the noise
estimation function, based on a CLIP model:

$\tilde{\epsilon}_{\theta}(\bm{z}_{t},t,\sigma_{s},\sigma_{c})=\epsilon_{\theta}(\bm{z}_{t},t,\sigma_{s})+G(\bm{z}_{t},t,\sigma_{c})$
(33)

##  VI Evaluation

We compare in this section several methods to control colors, previously
described. We show that our guidance can be applied to pixel space diffusion,
but more importantly, it also performs well when applied to latent diffusion
models, outperforming current available methods on the latter. We then apply
our formulation of guidance in the context of compression in the CoCliCo
scheme and show how it compares to other extremely low-bitrates and semantic
focused methods.

###  VI-A Performance of color control

####  VI-A1 Guidance in the pixel space

We showcase the guidance for pixel space diffusion. We use an unconditional
diffusion model presented in [8] and pretrained for the generation of
$512\times 512$ images. This model does not use semantic information on the
image and does not always produce realistic images, but is sufficient to
appreciate the impact of color guidance. We show in Fig. 6 some images
generated using our guidance terms presented in equation (24) and universal
guidance (12). We can see that the generated images follow the given color
maps, with more accuracy than when using universal guidance.

Figure 6: Color guidance applied to pixel space diffusion models. Figure 7:
Visual comparison between the different methods to control colors. Figure 8:
Color map guidance at different resolutions. Figure 9: Compression with
$25\times 25$ resolution color maps. Figure 10: Images decoded using 25x25
resolution color maps and quantized CLIP latent for encoding. Figure 11:
Compression with $16\times 16$ resolution color maps.

####  VI-A2 Guidance in the latent space

For latent diffusion guidance, we use another model to compare the different
methods to control colors. The model we use is a LDM conditioned on CLIP
latent vectors as inputs for text-to-image synthesis. We repurposed it for
image-to-image synthesis using the CLIP image encoder rather than the text
encoder which refers to $\mathcal{E}_{clip}$ in Fig. 5. The weights of the
model are available on diffusers [26]. We use the scheduler DPMSolver[27], for
faster generation, to step from $\bm{z}_{t}$ to $\bm{z}_{t-1}$, with $50$
timesteps for generation. We use a dataset composed of images from the
Landscape dataset[28] and the Kodak dataset[29], images were cropped to fit in
$512\times 512$, but our method can easily be adapted to any image resolution.

We compare our fine color guidance with universal guidance applied to color
and initialized diffusion from [7]. We use a color guidance scale of $1$ for
our method, using only the determined parameters. In the case of universal
guidance, we compute the best coefficient $\lambda$ in their formula (12) by
using our formulation of guidance, we verify experimentally its coherence. For
initialized diffusion, we start the generation at 55% of the total timesteps
to maximize the information on the color map given without impairing the
generative quality too much, as explained in [7]. We show the performance of
the different methods in Fig. 7. The images generated using fine guidance
systematically follow closely the given color map. On the other hand, the two
other methods drift away from the original image, losing some information
given by the color map. With universal guidance being slightly better than the
initialized diffusion. With a quantitative evaluation presented in Table I, we
confirm those results. Using the MSE between the original and generated image
as criteria, we notice that our method preserves most of the information. We
also confirm using image quality assessment metric (IQA) that, while
initialized diffusion comes at the cost of the quality of images, using
universal guidance or fine guidance does not degrade the image. The quality
score of the generated images is similar to natural clean images of the
dataset. For more information on the quality of generated images, FID of such
models can be found in the paper of the latent diffusion models [2], but were
not measured here because of the size of the dataset considered.

TABLE I: Evaluation of color control | Color fidelity $\downarrow$ | Image realism $\uparrow$
---|---|---
| MSE | CLIPIQA[30] | DBCNN[31]
color map | 0.009 | - | -
Initialized[7] | 0.080 | 0.49 | 0.48
Universal[9] | 0.031 | 0.69 | 0.66
Ours | 0.015 | 0.66 | 0.63

Fine guidance can be applied to different color map resolutions. We show in
Fig. 8 a comparison with universal guidance, on $16\times 16$ and $8\times 8$
color maps. With less information, guidance constraints are relaxed, and the
difference between methods is less noticeable. We can see that the universal
guidance does not use the additional information provided by a higher
resolution color maps, indeed, the images generated using $8\times 8$ color
maps are quite similar to the ones generated using higher resolutions, this
affirmation is later confirmed in the next section. We can also see, from the
lower resolution generated images, that the semantic information becomes
crucial when the color maps are not precise enough, such as Eiffel Tower color
map in $8\times 8$. We can no longer see the semantic information from the
color map only, and semantic is required to get an image close to the
original.

####  VI-A3 Discussion

Initializing the diffusion is a simple method that does not require a
computation of gradients, however, this is a method that loses both in realism
of generation and fidelity to the color map because of the presence of a
tradeoff between both. Our formulation of guidance is similar to the
formulation of universal guidance, as both coincidentally requires computing
similar gradients. The main difference between the two methods lies in the
scaling of the loss. While in universal, the importance of the guide decreases
after each step, we found that to optimally guide the scaling should not
decrease in $\sqrt{1-\alpha_{t}}$ but remain high instead because of the ratio
$\sqrt{\alpha_{t}}/\bar{\lambda_{t}}$, the generated results confirm this
observation.

###  VI-B Fine guidance for compression

We apply fine color guidance in the compression framework previously described
in Section V. The color map and the CLIP latent are quantized to reach lower
bitrate. The clip latent is quantified using $1$ bit per dimension, and the
color maps’ palette are quantified over $5$ bits per channel as in the CoCliCo
pipeline. We use the fine color guidance to preserve most of the information
that is sent inside the color map. We compare our method to state-of-the-art
compression methods, notably, VVC. But we also compare with other generative
approach aiming at semantic compression rather than at the pixel-value
fidelity, such as Text+sketch[15]. We consider the generation of images in
patch of $512\times 512$.

We evaluate our method on two configurations. One using only a $16\times 16$
color map resolution, targeting a bitrate similar to the lowest VVC
quantization parameter. And another one, keeping more information on the
colors, using a $25\times 25$ color map, targeting a bitrate similar to
text+sketch. For the $25\times 25$ configuration, we compare our method to
Text+Sketch, their choice of representation uses similar in bitrate. For VVC,
we use the intra coder(v1.6) with Vvenc implementation [16], and we target
similar bitrate. Images in Fig. 9 showcase the different compression
framework, because the focus of VVC is an optimization of PSNR, the quality of
the reconstructed image is lacking. Images reconstructed using a text and
sketch semantic representation have a distinctly different aspect because of
the incoherence of color, sometimes mistaking the sky for something else in
the reconstruction. We show that furthermore in Fig. 10 that images generated
using our framework can have a high quality of texture on top of following the
color map. This offers clear advantages over classical methods like VVC.

With the other configuration in $16\times 16$, showed in Fig. 11, at those
rates, the classical approach cannot always preserve information of the color
as seen in the first and fifth row. The reconstructed images are still
faithful in terms of semantic, rather losing fidelity to the input in terms of
color and thus PSNR.

TABLE II: Compression at extremely low bitrates, *$=$proposed approach |  $\Psi$ $\downarrow$ |  $\Phi$ $\uparrow$ |  $R$ $\downarrow$
---|---|---|---
| MSE | CLIP[25] | CLIPIQA[30] | DBCNN[31] | bits
Original images | - | 0.00 | 0.70 | 0.63 | -
VVC (qp 63) | 0.011 | 0.41 | 0.20 | 0.20 | 3220
CLIP+Initialized($16\times 16$) | 0.091 | 0.21 | 0.56 | 0.50 | 2688
CLIP+Universal($16\times 16$) | 0.034 | 0.18 | 0.64 | 0.60 | 2688
CLIP+Fine guidance*($16\times 16$) | 0.021 | 0.18 | 0.62 | 0.58 | 2688
VVC (qp 55) | 0.007 | 0.30 | 0.19 | 0.26 | 6640
Text+Sketch[9] | 0.150 | 0.19 | 0.73 | 0.61 | 6638
CLIP+Initialized($25\times 25$) | 0.035 | 0.22 | 0.55 | 0.50 | 5838
CLIP+Universal($25\times 25$) | 0.033 | 0.20 | 0.64 | 0.60 | 5838
CLIP+Fine guidance*($25\times 25$) | 0.016 | 0.16 | 0.64 | 0.60 | 5838

A quantitative evaluation of the compression framework is presented in Table
II. We evaluate on the two criteria presented in our formulation in (30)
comparing on lower or equal bitrates. The first criterion, $\Psi$, is the
preservation of the given semantic. We measure the preservation of the general
color aspect using the MSE, and preservation of a more textual aspect of the
semantic using CLIP sine similarity. Over the color preservation, we place
right behind VVC on both configurations. Note that if the MSE is used as a
color preservation metric here, the objective of VVC and ours do not align,
indeed, we do not aim at pixel fidelity only. Over the semantic aspect, our
method scores higher than the others. The second criterion is the realism,
$\Phi$, used for the evaluation of the guidance, we also use IQA metrics, and
while text+sketch scores higher on this criterion, this is mainly due to the
use of a different model rather than a performance issue caused by our
guidance. Indeed, we reach a realism score which is close to the images from
the dataset. IQA metrics do not necessarily reflect perfectly the notion of
realism, which is hard to grasp, however, we noted that it makes a good
distinction between images with a lot of artifacts, such as VVC, and clean
images. The realism of generated images can also be appreciated in Fig. 10. On
both configurations, our method manages to preserve the semantic and color
information while keeping low bitrates. Our fine guidance also allows us to
use higher resolution color maps without wasting the bitrate invested in it.
Indeed, as observed in Fig. 8, while our methods shows an important difference
in terms of MSE when increasing the resolution of the color map, the score
barely changes for other guidance methods. We show in the figure and table
that the resolution of the color maps is an interesting parameter to vary. We
present in the table the rate per image rather than a bit per pixel, to be
independent of the diffusion model output capacity since our description of an
image does not depend on its size.

The MSE is here used as a metric of color fidelity and not as an objective to
optimize at all cost. The goal of guidance is to get closer to the information
brought by the color map. The use of our fine color guidance in compression
brings fidelity to the color map, it can be applied on the latest diffusion
model, even when trained in the latent space of a VAE.

##  VII Conclusion

We proposed a method for color guidance in diffusion models, more
specifically, for latent diffusion models that does not require any training.
This method outperforms other training-free methods and can be applied to any
latent or pixel space diffusion model. We can then advantageously use it on
the latest diffusions models to control the output. We showcased the use of
our method in a context of compression at extremely low bitrates, showing that
a low resolution color map can be a good descriptor to add to the semantic. We
show, in the context of the CoCliCo coding pipeline, that the information
given by the color map is better preserved using fine color guidance, avoiding
the waste of bitrate.

##  VIII Acknowledgements

This work was funded by the French National Research Agency (_MADARE_ ,
Project-ANR-21-CE48-0002).

##

## References

  * [1] Y. Song and S. Ermon, “Generative modeling by estimating gradients of the data distribution,” _Advances in neural information processing systems_ , vol. 32, 2019.
  * [2] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-Resolution Image Synthesis with Latent Diffusion Models,” Apr. 2022, arXiv:2112.10752 [cs]. [Online]. Available: http://arxiv.org/abs/2112.10752
  * [3] D. You, A. Floros, and P. L. Dragotti, “Indigo: An inn-guided probabilistic diffusion algorithm for inverse problems,” _arXiv preprint arXiv:2306.02949_ , 2023.
  * [4] B. Kawar, M. Elad, S. Ermon, and J. Song, “Denoising diffusion restoration models,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 23 593–23 606, 2022.
  * [5] M. Kwon, J. Jeong, and Y. Uh, “Diffusion models already have a semantic latent space,” _arXiv preprint arXiv:2210.10960_ , 2022.
  * [6] L. Zhang, A. Rao, and M. Agrawala, “Adding conditional control to text-to-image diffusion models,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023, pp. 3836–3847.
  * [7] C. Meng, Y. He, Y. Song, J. Song, J. Wu, J.-Y. Zhu, and S. Ermon, “Sdedit: Guided image synthesis and editing with stochastic differential equations,” _arXiv preprint arXiv:2108.01073_ , 2021.
  * [8] P. Dhariwal and A. Nichol, “Diffusion Models Beat GANs on Image Synthesis,” in _Advances in Neural Information Processing Systems_ , vol. 34. Curran Associates, Inc., 2021, pp. 8780–8794. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2021/hash/49ad23d1ec9fa4bd8d77d02681df5cfa-Abstract.html
  * [9] A. Bansal, H.-M. Chu, A. Schwarzschild, S. Sengupta, M. Goldblum, J. Geiping, and T. Goldstein, “Universal guidance for diffusion models,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 843–852.
  * [10] Y. Blau and T. Michaeli, “The perception-distortion tradeoff,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 6228–6237.
  * [11] ——, “Rethinking lossy compression: The rate-distortion-perception tradeoff,” in _International Conference on Machine Learning_. PMLR, 2019, pp. 675–685.
  * [12] F. Pezone, O. Musa, G. Caire, and S. Barbarossa, “Semantic-preserving image coding based on conditional diffusion models,” _arXiv preprint arXiv:2310.15737_ , 2023.
  * [13] T. Bordin and T. Maugey, “Semantic based generative compression of images for extremely low bitrates,” in _2023 IEEE 25th International Workshop on Multimedia Signal Processing (MMSP)_. IEEE, 2023, pp. 1–6.
  * [14] T. Bordin, T. Bachard, and T. Maugey, “Coclico: Extremely low bitrate image compression based on clip semantic and tiny color map,” in _2024 IEEE 37th International Workshop of Picture Coding Symposium (PCS)_. IEEE, 2024.
  * [15] E. Lei, Y. B. Uslu, H. Hassani, and S. S. Bidokhti, “Text+ sketch: Image compression at ultra low rates,” _arXiv preprint arXiv:2307.01944_ , 2023.
  * [16] A. Wieckowski, J. Brandenburg, T. Hinz, C. Bartnik, V. George, G. Hege, C. Helmrich, A. Henkel, C. Lehmann, C. Stoffers, I. Zupancic, B. Bross, and D. Marpe, “Vvenc: An open and optimized vvc encoder implementation,” in _Proc. IEEE International Conference on Multimedia Expo Workshops (ICMEW)_ , pp. 1–2.
  * [17] J. Song, C. Meng, and S. Ermon, “Denoising diffusion implicit models,” _arXiv preprint arXiv:2010.02502_ , 2020.
  * [18] A. Lugmayr, M. Danelljan, A. Romero, F. Yu, R. Timofte, and L. Van Gool, “Repaint: Inpainting using denoising diffusion probabilistic models,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 11 461–11 471.
  * [19] J. Ho and T. Salimans, “Classifier-free diffusion guidance,” _arXiv preprint arXiv:2207.12598_ , 2022.
  * [20] Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, “Score-based generative modeling through stochastic differential equations,” _arXiv preprint arXiv:2011.13456_ , 2020.
  * [21] J. Betker, G. Goh, L. Jing, T. Brooks, J. Wang, L. Li, L. Ouyang, J. Zhuang, J. Lee, Y. Guo _et al._ , “Improving image generation with better captions,” _Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf_ , vol. 2, no. 3, p. 8, 2023.
  * [22] E. Agustsson, D. Minnen, G. Toderici, and F. Mentzer, “Multi-realism image compression with a conditional generator,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 22 324–22 333.
  * [23] Z. Pan, X. Zhou, and H. Tian, “Extreme generative image compression by learning text embedding from diffusion models,” _arXiv preprint arXiv:2211.07793_ , 2022.
  * [24] E. Grassucci, S. Barbarossa, and D. Comminiello, “Generative semantic communication: Diffusion models beyond bit recovery,” _arXiv preprint arXiv:2306.04321_ , 2023.
  * [25] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark _et al._ , “Learning transferable visual models from natural language supervision,” in _International conference on machine learning_. PMLR, 2021, pp. 8748–8763.
  * [26] P. von Platen, S. Patil, A. Lozhkov, P. Cuenca, N. Lambert, K. Rasul, M. Davaadorj, and T. Wolf, “Diffusers: State-of-the-art diffusion models,” https://github.com/huggingface/diffusers, 2022.
  * [27] C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, and J. Zhu, “Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models,” _arXiv preprint arXiv:2211.01095_ , 2022.
  * [28] P. Esser, R. Rombach, and B. Ommer, “Taming transformers for high-resolution image synthesis,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2021, pp. 12 873–12 883.
  * [29] “Kodak: The kodak color image dataset,” http://r0k.us/graphics/kodak/.
  * [30] J. Wang, K. C. Chan, and C. C. Loy, “Exploring clip for assessing the look and feel of images,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 37, no. 2, 2023, pp. 2555–2563.
  * [31] W. Zhang, K. Ma, J. Yan, D. Deng, and Z. Wang, “Blind image quality assessment using a deep bilinear convolutional neural network,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 30, no. 1, pp. 36–47, 2018.