# ENOS: Energy-Aware Network Operator Search for Hybrid Digital and Compute-
in-Memory DNN Accelerators

Shamma Nasrin, Ahish Shylendra, Yuti Kadakia, Nick Iliev, Wilfred Gomes, Theja
Tulabandhula, and Amit Ranjan Trivedi

###### Abstract

This work proposes a novel Energy-Aware Network Operator Search (ENOS)
approach to address the energy-accuracy trade-offs of a deep neural network
(DNN) accelerator. In recent years, novel inference operators such as binary
weight, multiplication-free, and deep shift have been proposed to improve the
computational efficiency of a DNN. Augmenting the operators, their
corresponding novel computing modes such as compute-in-memory and XOR networks
have also been explored. However, simplification of DNN operators invariably
comes at the cost of lower accuracy, especially on complex processing tasks.
While the prior works explore an end-to-end DNN processing with the same
operator and computing mode, our proposed ENOS framework allows an optimal
layer-wise integration of inference operators and computing modes to achieve
the desired balance of energy and accuracy. The search in ENOS is formulated
as a continuous optimization problem, solvable using typical gradient descent
methods, thereby scalable to larger DNNs with minimal increase in training
cost. We characterize ENOS under two settings. In the first setting, for
digital accelerators, we discuss ENOS on multiply-accumulate (MAC) cores that
can be reconfigured to different operators. ENOS training methods with single
and bi-level optimization objectives are discussed and compared. We also
discuss a sequential operator assignment strategy in ENOS that only learns the
assignment for one layer in one training step, enabling greater flexibility in
converging towards the optimal operator allocations. Furthermore, following
Bayesian principles, a sampling-based variational mode of ENOS is also
presented. ENOS is characterized on popular DNNs ShuffleNet and SqueezeNet on
CIFAR10 and CIFAR100. Compared to the conventional uni-operator approaches,
under the same energy budget, ENOS improves accuracy by 10–20%. In the second
setting, for a hybrid digital and compute-in-memory accelerator, we
characterize ENOS to assign both layer-wise computing mode (high precision
digital or low precision analog compute-in-memory) as well as operator while
staying within the total compute-in-memory budget. Under varying
configurations of hybrid accelerator, ENOS can leverage higher energy
efficiency of compute-in-memory operations to reduce the operating energy of
DNNs by 5$\times$ while suffering $<$1% reduction in accuracy.
Characterization results using ENOS show interesting insights, such as
amenability of different filters to using low complexity operators, minimizing
the energy of inference while maintaining high prediction accuracy.

###### Index Terms:

Operator search, DNN, Multiplication-Free, Binary

##  I Introduction

The proliferation of deep learning in embedded computing space has aggravated
computational efficiency challenges of deep neural networks (DNNs).
Consequently, DNN architectures are going through a dramatic evolution to
improve their prediction capacity within limited computing and storage
resources. While so far, DNNs were hand-designed by domain experts,
increasingly, more attention is being dedicated to automating their
architecture search. This sub-field of machine learning, called Neural
Architecture Search (NAS), has seen rapid growth. Among recent NAS approaches,
NASnet[1] learns the best convolutional architecture on a smaller dataset
using reinforcement learning and then transfers the learned architecture to a
larger dataset. In Efficient Neural Architecture Search (ENAS) [2], a
controller is trained using policy gradient to find an optimal subgraph within
a large computational graph. Unlike prior reinforcement learning-based
approaches, which learn over a discrete search space and thereby require
thousands of GPU hours to compute, in [3], architecture search was solved
using gradient descent by formulating the search space to be continuous. Using
[3], the architecture search could be performed within a few GPU hours on
modern DNNs.

In parallel, considerable efforts have also focused on optimizing DNN’s
correlation operators. Scalar multiplication of input and weight vectors is
the most prevalent correlation operator in DNNs. Since DNN’s processing
involves evaluating thousands of correlations among high-dimensional
input/weight vectors, the operator dictates the overall computational
efficiency of a DNN. Therefore, prior works have explored alternate, more
computationally efficient correlation operators. [4, 5] introduced
multiplication-free operation where high-precision multiplication between
input and weight is replaced by 1-bit multiplication and multi-bit addition.
[6] introduced bit-wise shift operators where weights are approximated by a
power of two, i.e., 2n [$n\in\mathbb{Z}^{+}$], replacing multiplications with
shift operations. [7, 8] quantize inputs and weights to one-bit and uses XNOR
instead of multiplications. The new correlation operators further benefit from
the custom-designed computing modes. For example, multiplication-free and
binary weights operators are quite suited for compute-in-memory [4, 9, 10].
Deep-shift operator in [6] can be implemented using digital shifters.
Operators in [7, 8] can be efficiently implemented using XOR logic gates.
Combining the operators with an optimal computing mode im proves the energy
efficiency of DNNs considerably.

Figure 1: ENOS searches for an optimal layer-wise operator and computing mode
assignment to maximize energy efficiency of inference in a DNN while adhering
to application and computing platform characteristics such as technology node
and compute-in-memory budget.

Our novel framework on energy-aware network operator search (ENOS)
synergistically integrates the above two efforts, namely, neural network
design automation and exploration of more computationally efficient learning
operators. ENOS automatically combines various correlation operators and their
computing modes layer-wise for a DNN so that its energy-accuracy trade-off can
be optimally balanced. For example, unlike the current approaches, which
consider end-to-end DNN processing using a single correlation operator, in
Figure 1, ENOS can optimally integrate typical, binary weight, and
multiplication-free operators among DNN layers along with their computing
modes, such as compute-in-memory or digital. The automated synthesis in ENOS
can be made to inherently depend on application and hardware characteristics
as well. For example, ENOS can account for a limited compute-in-memory budget
in hybrid digital and compute-in-memory accelerators. Similarly, ENOS can
recognize task complexity to determine the filter-wise operator sequence in a
DNN architecture optimally. Furthermore, inspired by [3], ENOS learns in a
continuous search space using standard gradient descent-based routines (such
as ADAM [11]) so that its training is scalable to complex and large DNNs.

In Sec. II, we discuss the overview of ENOS. In Sec. III, we discuss ENOS on
digital platforms while showing results using single-level, bi-level, and
sequential optimization strategies. In the section, we also discuss ENOS’s
variational perspective and demonstrate its performance by sampling filter-
wise operator assignments from a learned variational distribution. In Sec. IV,
we consider ENOS’s application on hybrid accelerators that combine both (low
precision) compute-in-memory and (high precision) digital modules. For hybrid
accelerators, ENOS is augmented with Lagrangian constraints that prepare the
operator and computing mode assignment by considering the limited compute-in-
memory budget. Sec. V concludes.

Figure 2: Learning architecture of Single and Bi-Level ENOS: In the first
step, at each layer, candidate operators’ outputs are combined using linear
interpolation with layer-wise operator weight parameters, $\alpha$. The
operator weights are also learned through back-propagation. In the second
step, only the operators with the highest weight are retained across the
network. The corresponding weights for the entire network are relearned. Since
the cost function in both steps is continuous, typical learning tools such as
ADAM can be utilized.

##  II Framework of Energy-Aware Network Operator Search (ENOS)

Prior works have explored novel deep learning operators such as binarized
weights and inputs [7], binary-connect [8], multiplication-free [4], and deep-
shift[6] to improve the computational efficiency of deep learning within
limited computing and storage resources. However, computational advantages of
simplified operators come at the cost of accuracy degradation. Moreover, the
efficiency of novel operators also strongly depends on hardware and task
specifications. For example, deep networks based on binarized weights and
inputs can be very efficiently implemented using a network of XOR logic gates
[12, 13]. Despite showing a competitive accuracy on datasets such as MNIST
[14], the operator has limited accuracy on more complex datasets such as
ImageNet[15] compared to the typical case, i.e., full precision scalar product
operator. Therefore, a unique opportunity exists where these operators can be
optimally combined, so that their computational efficiency can be harnessed
while maintaining high prediction accuracy. Moreover, such optimal layer-wise
fusion of diverse network operators should also consider the characteristics
of the underlying hardware, energy constraints, and task specifications.
Addressing this challenge, we present a systematic framework for such
automated operator allocation.

###  II-A Single-level and Bi-level ENOS

Figure 2 shows the framework of ENOS which is inspired from [3]. For the
ensuing discussion, at each layer, we consider an optimal operator choice
among typical, binary connect, and multiplication-free operators. However, our
approach is generalizable to any number of and different operator choices. At
each layer, input feature map is processed against all three operators across
three parallel paths. The output of all three operators are combined using an
affine interpolation using the corresponding weight factors ($\alpha$), where
the weight factors itself are learning parameters. The overall cost function
of DNN is, thus, defined as

$\mathcal{L}\textsubscript{net}=\underbrace{\mathcal{L}\textsubscript{acc}(\theta,\alpha)}_{\text{Accuracy}}+\lambda\underbrace{\sum_{i=1}^{N}N_{OP,i}\sum_{j=1}^{M}\text{softmax}(\alpha_{ij})E_{OP,j}}_{\text{Regularizer
for DNN accelerator energy}}$ (1a)
$\theta^{*},\alpha^{*}=\text{argmin}(\mathcal{L}\textsubscript{net})$ (1b)

Here, the first term of $\mathcal{L}\textsubscript{net}$, i.e.,
$\mathcal{L}\textsubscript{acc}(\theta,\alpha)$ minimizes with the higher
prediction accuracy while the second term minimizes with the net lower energy
of operation. $i$ indexes over network layers, while $j$ indexes over operator
choices for each layer. $\theta$ are the network weight parameters. $\alpha$
are the layer-wise operator weight parameters. $\lambda$ is a user-defined
hyper-parameter that considers energy-accuracy trade-off. $N_{OP,i}$ is the
number of multiply-accumulate (MAC) operations at layer i. $E_{OP,j}$ is
average energy for an operator j for unit operation.

Without the following non-linear activation (such as ReLU), the operation of
input ($\mathbf{x}$) and weight ($\mathbf{w}$) vectors with the three chosen
operators, typical, multiplication-free, and binary, are as following,
respectively,

$f_{T}(\mathbf{x},\mathbf{w})=\sum x_{i}\cdot w_{i}$ (2a)
$f_{M}(\mathbf{x},\mathbf{w})=\sum\text{sign}(x_{i})\cdot\text{abs}(w_{i})+\text{sign}(w_{i})\cdot\text{abs}(x_{i})$
(2b) $f_{B}(\mathbf{x},\mathbf{w})=\sum x_{i}\cdot binarize(w_{i})$ (2c)

Here, $\cdot$ is an element-wise multiplication operator, $+$ is element-wise
addition operator, and $\sum$ is vector sum operator. $\text{sign}()$ operator
is $\pm 1$ and $\text{abs}()$ operator produces absolute unsigned value.

Although the binary operator in Eq. (2c) is non-differentiable, prior works
[16, 17, 18] have presented approaches such as straight-through estimators to
circumvent the problem. Similarly, sign() and abs() functions are non-
differentiable in Eq. (2b). Prior work [19] approximates the derivative of
abs() with a steep hyperbolic tangent and the derivative of sign() with a
steep Gaussian function to address this. Therefore, with the added
modifications, the net loss function can still be optimized using typical
gradient descent approaches such as ADAM [20]. In the first step of a baseline
approach to ENOS, following the above loss formulation, a candidate locally
optimal parameter set $\theta^{*}$ and $\alpha^{*}$ can be extracted using
gradient descent. Then, in the second step, a new design of DNN is considered
where only the operator with the highest weight factor ($\alpha_{j}$) is
considered at each layer. The corresponding weights for new DNN are then
relearned.

In addition to this single-level baseline optimization approach to
simultaneously extract optimal parameter set $\theta^{*}$ and $\alpha^{*}$, we
also consider a bi-level optimization procedure for step-1 of the learning
procedure. Algorithm 1 describes the bi-level optimization procedure, which
trains the network hierarchically. On the top level, we update the weight
factors $\alpha$ by descending the validation loss
$\Delta\mathcal{L}\textsubscript{net}_{Val}$ defined on a dataset separate
from training. Keeping the learned $\alpha$, we descend
$\Delta\mathcal{L}\textsubscript{net}_{Train}$ to learn the DNN weights
$\theta$. Likewise, we iterate through the validation and training data set
sequentially to learn the final $\alpha$.

Goal: Extract trained $\theta^{*}$ and $\alpha^{*}$ to reduce the loss for
Step-1. Loss function:
$\mathcal{L}\textsubscript{net}=\mathcal{L}\textsubscript{acc}(\theta,\alpha)+\lambda\sum_{i=1}^{N}N_{OP,i}\sum_{j=1}^{M}\text{softmax}(\alpha_{ij})E_{OP,j}$

Given: Network weights: $\theta$; Operator weights: $\alpha$.

while  _not converged do_ do

1\. Update $\alpha$ by descending the loss
$\Delta\mathcal{L}\textsubscript{net}_{Val}(\theta^{t},\alpha)$

2\. Update $\theta$ descending the loss
$\Delta\mathcal{L}\textsubscript{net}_{Train}(\theta,\alpha^{t})$;

end while

For Step-2, choose the operator with the highest weight factor ($\alpha_{j}$)
at each layer. Relearn the network weights.

Algorithm 1 Bi-Level ENOS Figure 3: Layer wise operator mapping for SqueezeNet
network for Cifar10 and Cifar100 data set. The higher hyper parameter
$\lambda$ in equation 1 prioritizes energy efficient operators. The table
compares the energy of SqueezeNet with different operator combination.

Figure 3 shows an exemplary synthesis of SqueezeNet using ENOS on CIFAR10 and
CIFAR100 datasets. More details on ENOS parameters such as extraction of
operator energy ($E_{OP,j}$), implementation hardware, and comparison among
different optimization objectives are discussed later. Rows in the table show
optimal operator assignments under varying $\lambda$. As discussed earlier,
increasing $\lambda$ allows trading-off accuracy against DNN accelerator’s
operating energy. For CIFAR10, compared to typical (T) operator on all layers,
optimally integrating different operators on different layers allows $\sim$20%
lower accelerator energy in ENOS at only $\sim$1.5% accuracy reduction. The
lower rows in the figure show even more aggressive energy scaling while
suffering moderate accuracy degradation. The application of ENOS on the
network also offers many interesting insights. With increasing $\lambda$ to
improve energy efficiency, operator assignments begin progressively changing
only in the middle convolution layers from typical to multiplication-free and
then to binary. Meanwhile, operator assignments for the initial and last
convolution layers remain typical. This characterization illustrates that
feature extraction in the middle layers in the network is more amenable to
simplification and accelerator energy saving without a considerable effect on
the overall network’s output accuracy.

###  II-B Sequential ENOS

In the previous approach, concurrent searching of operators for all layers is
susceptible to overfitting. To address this concern, we also investigated a
sequential mode of operator search. Under the sequential search, operator
assignments are selected only for a single layer in one iteration while
keeping the other layers’ operator choices open. Specifically, a layer “$i$”
is assigned the operator choice “$j$” if it has the highest weight factor
$\alpha_{ij}$ among all layers and operator choices. All operator choices are
considered in the following training iteration for the remaining unassigned
network layers, and their corresponding operator weights are relearned. The
iterations continue until all layer-wise operator assignments have been found.
Algorithm 2 describes such sequential operator search procedures in ENOS.

Goal: Sequentially select the operator for network layers.

Loss function:
$\mathcal{L}\textsubscript{net}=\mathcal{L}\textsubscript{acc}(\theta,\alpha)+\lambda\sum_{i=1}^{N}N_{OP,i}\sum_{j=1}^{M}\text{softmax}(\alpha_{ij})E_{OP,j}$

Given: Network weights: $\theta$; Operator weights: $\alpha$.

for  _layer $l$=1 to $L$_ do

while  _not converged_ do

Update $\alpha$, $\theta$ by descending the loss on training set
$\Delta\mathcal{L}\textsubscript{net}_{Train}(\theta,\alpha)$

end while

1\. Find the highest network weight factor ($\alpha_{ij}$).

2\. Assign operator $j$ to layer $i$ and prevent further optimization of this
choice.

end for

Relearn the network weights.

Algorithm 2 Sequential ENOS Figure 4: Evolution of operator assignments on the
layers of SqueezeNet under sequential optimization in ENOS.

Figure 4 shows the evolution of operator assignments on SqeezeNet under
sequential ENOS. We kept the operator for the first convolution layer (Conv1)
as typical (T) since, under bi-level ENOS, the layer preferred standard
operator even under aggressive $\lambda$ scaling. Among the remaining Fire
layers, Fire7 is first assigned binary (B) operator since it showed the
highest propensity towards B operating under accelerator energy scaling
constraints among all the other layers. The operator choices remained open for
the other layers. Subsequently, Fire6 is assigned B operator, and Fire1 is
assigned T operator at the last cycle. Compared to single-level and bi-level
ENOS, the training workload for sequential ENOS is substantially more. Note
that the number of training iterations in sequential ENOS scale proportionally
to the number of network layers.

###  II-C Variational ENOS

The optimal operator assignment problem in ENOS can also be treated under a
variational framework. Specifically, softmax($\alpha_{ij}$) can be seen as the
parameters of a multinomial distribution that represents the operator choices.
Unlike the previous optimization procedures where a layer “$i$” is assigned
the operator “$j$” if $\alpha_{ij}$ is the maximum, the assignments under the
variational setting is made by treating softmax($\alpha_{ij}$) as the
probability of the optimal operator assignment. Here, a population of $N$
candidate networks can be prepared by sampling the optimal operator choice at
each layer “$i$” based on the corresponding softmax($\alpha_{ij}$),
representing the probability of the operator choice “$j$” on the layer.
Subsequently, the best performing network from the candidate networks can be
selected by testing it on the validation set. Algorithm 3 describes this
procedure formally. We note in passing that while one could define a formal
variational objective, lower bound it and maximize that with respect to the
parameters of the operator choice distribution using the reparametrization
trick, we found that our simpler strategy already provides tangible
improvements and is closer to the algorithmic templates of single-level, bi-
level and sequential strategies.

Goal: Learn multinomial distribution parameters representing optimal operator
choices and operator assignments.

Loss:
$\mathcal{L}\textsubscript{net}=\mathcal{L}\textsubscript{acc}(\theta,\alpha)+\lambda\sum_{i=1}^{N}N_{OP,i}\sum_{j=1}^{M}\text{softmax}(\alpha_{ij})E_{OP,j}$

Given: Network weights: $\theta$; Operator weights: $\alpha$.

Step-1: learn $\theta^{*}$ and $\alpha^{*}$

while  _not converged_ do

Update $\alpha$, $\theta$ by descending the loss
$\Delta\mathcal{L}\textsubscript{net}_{Train}(\theta,\alpha)$

end while

Step-2: Create a population of candidate networks

for  _samples $n$=1 to N_ do

for  _layer $l$=1 to L_ do

Assign operator based on multinomial distribution parameters
softmax($\alpha_{ij}$).

end for

Relearn network weights

end for

Step-3: Find the optimal operator assignment

for  _samples $n$=1 to N_ do

Characterize the network on validation set.

end for

Assign the most accurate network as the optimal choice.

Algorithm 3 Variational ENOS Figure 5: Candidate networks under variational
treatment of ENOS. For different layers, the operators are sampled based on
the learned multinomial distribution parameters.

Figure 5 shows the operator assignments on various candidate networks under
variational ENOS. Interestingly, under the considered $\lambda$, i.e., DNN
accelerator’s energy constraints, layers Fire4 and Fire6 show less entropy and
mostly opt for T and MF operator respectively. Meanwhile, layers such as Fire2
and Fire7 show much higher entropy in their operator assignment. Layer-wise
operator assignment among various candidate networks are also jointly
correlated as the network searches for the best operator combinations to
maximize accuracy under accelerator energy constraints.

##  III ENOS-based DNN Mapping for Digital Accelerators

###  III-A Digital MAC unit with reconfigurable operators

This section discusses the energy efficiency benefits of ENOS-based DNN
mapping considering a digital spatial accelerator. Spatial digital
accelerators are prevalent for high-performance DNN [21, 22, 23]. A spatial
accelerator combines many parallel multiply-accumulate (MAC) cores using a
network-on-chip (NOC). DNN workload is distributed among the parallel cores so
that the entire inference flow can be processed with the least latency.
Considering ENOS-based DNN mapping on a spatial accelerator, Figure 6 shows a
MAC unit that can be reconfigured among the three operators that we considered
above. A spatial DNN accelerator can combine such reconfigurable MAC cores to
flexibly implement ENOS-based layer-wise operator integration under various
energy-accuracy constraints.

In Figure 6(a), datapaths for different operators are associatively color-
coded: datapaths for typical, multiplication-free & binary operators are
highlighted with green, red and blue, respectively. Although training for the
network weights is performed considering a floating-point precision, an 8-bit
fixed precision operation is considered during inference. Similar to prior
work [24, 25, 26], 8-bit fixed-precision inference suffers minimal accuracy
degradation but can considerably simplify the hardware implementation and
improve computations’ energy efficiency. Datapath to implement typical
operator includes an 8-bit multiplier (to obtain the product of 8-bit input
activation and weight) and a 20-bit adder (to accumulate the 16-bit outputs of
multiplier continually). Meanwhile, the multiplication-free operator requires
just an adder/subtractor block as it essentially implements
addition/subtraction of input activation (IACT) with corresponding weight (W),
where IACT and W are both 8-bits. Unlike typical and multiplication-free
operators, weights in the binary operator are just one-bit while IACT is
represented using 8-bits. Hence, the binary operator adds/subtracts
consecutive input activation depending on corresponding weight bits,
therefore, it needs an adder/subtractor block. We use a 12-bit adder with
multiplication-free and binary operators since they require a lower dynamic
range of IACT than typical operator. In the considered MAC unit, the adder’s
dynamic range can be configured to either 20-bit or 12-bit based on a switch.

The design of the above MAC core was synthesized using Cadence RC Compiler.
Predictive technology models (PTM) for 15nm CMOS technology [27] were used for
the synthesis at nominal supply voltage. Timing-closure of the MAC core was
obtained considering a 100 MHz clock. Table I shows the energy of a unit
operation for all three operators. Typical operator is the most
computationally expensive whereas the binary operator requires $\sim$9$\times$
lower compute energy than the typical operator.

Figure 6: Digital MAC unit with reconfigurable operators. The datapaths for different operators are color coded: datapaths for typical, multiplication-free & binary operators are highlighted with green, red & blue, respectively. TABLE I: Per operation energy of different operators with 8-bit digital implementation at 1 GHz operating frequency. Operator | Components | Energy/op. (fJ)
---|---|---
Typical | 8-bit Multiplier + 20-bit Adder | 295.7
Multiplication-Free | 12-bit Adder | 64
Binary | 12-bit Adder | 32
Figure 7: Energy vs. Accuracy on CIFAR10 and CIFAR100 for (a) SqueezeNet and
(b) ShuffleNet under single, bi-level, sequential modes of ENOS.

###  III-B Simulation results and discussions

We study ENOS on popular networks SqueezeNet[28] and ShuffleNet[29]. Note that
unlike the predecessor DNNs such as VGG16 [30] and GoogleNet [31], SqueezeNet
and ShuffleNet are already optimized for computational efficiency and edge
computing, therefore are harder test-cases for further downscaling of their
computing energy. The networks were trained on CIFAR10 and CIFAR100 to search
the optimal operator assignments in a layer-wise manner. Figure 7 compares
single-level, bi-level, and sequential operator search modes of ENOS.
Different DNN configurations are estimated based on the operator energies in
Table I and the number of operations per operator to process the input. For
better clarity, the figure shows normalized energies of different
configurations under the baseline case where all operations are done using the
typical (T) operator.

In Figure 7, bi-level ENOS outperforms single-level ENOS under the same energy
constraints. For single-level optimization, learning network weights $\theta$
and operator weights $\alpha$ on the same dataset, i.e., training set, makes
the training vulnerable to over-fitting. Meanwhile, bi-level optimization
learns $\theta$ on the training set and $\alpha$ on the validation set,
thereby improves inductive biases by considering more data. Especially, bi-
level optimization shows significant improvement in accuracy on a more
challenging datasets, i.e., on CIFAR100 compared to CIFAR10.

Figure 7 also compares single-level and bi-level ENOS against sequential ENOS.
Compared to the concurrent operator assignment modes (single and bi-level),
sequential ENOS significantly improves the network synthesis. For example, for
SqeezeNet on CIFAR100 [Figure 7(a)], sequential ENOS results in a network
configuration that has $>$10% accuracy than single and bi-level ENOS when the
energy budget is only 40% of the baseline all typical operator case. Single
and bi-level ENOS modes can only gracefully scale down energy to 60% of the
baseline case; meanwhile, sequential ENOS incurs only $\sim$5% accuracy loss
when the accelerator energy is constrained to be one fifth of the baseline
case. Similarly, for ShuffleNet on CIFAR100 [Figure 7(b)], sequential ENOS
suffers a minimal degradation in energy even when the network energy is scaled
down to about 30% of the baseline case. Here as well, single and bi-level
modes of ENOS cannot scale down the network energy to such lower levels.
Nonetheless, due to its sequential mode of operator search, sequential ENOS is
much more computationally expensive than single and bi-level modes.
Significantly, under sequential ENOS, the training iterations grow in
sequential ENOS as the number of layers increases.

Energy-accuracy characteristics of various ENOS modes are also compared to
traditional uni-operator cases. DNN accelerator’s energy scaling using only
binary (B) or multiplication-free (MF) operators in the traditional uni-
operator approach considerably degrades the prediction accuracy.
Comparatively, with ENOS, optimal integration of various operators allows
scaling down accelerator energy to the half without a significant loss in
accuracy on all tested cases. If the energy constraints of DNN accelerators
are moderate, single and bi-level ENOS have a matching performance to
sequential ENOS, albeit, with much less training cost. However, under
considerable energy constraints, sequential ENOS surpasses other optimization
modes.

Figure 8: Energy vs. Accuracy on CIFAR10 and CIFAR100 for (a) SqueezeNet and
(b) ShuffleNet under variational mode of ENOS and comparison with sequential
optimization mode. Figure 9: (a) Hybrid DNN accelerator integrating compute-
in-memory and digital processing units. (b) Compute-in-memory array with 8-T
SRAM cell, Flash-ADC, and digital Shift-ADD unit. (c) Peripherals of compute-
in-memory array, rail-to-rail clocked comparator and priority encoder, to
digitize output.

Figure 8 shows the characterization of ENOS under variational optimization. In
the figure, color-coded clusters of accuracy-energy values are shown
corresponding to the candidate networks synthesized by variational ENOS under
various $\lambda$ [Algorithm 3]. As we discussed before, the candidate
networks are synthesized by treating operator weight softmax($\alpha_{ij}$) as
the probability of optimal operator as sampling from a multinomial
distribution of operators. Clusters with various $\lambda$ settings correspond
to various energy constraints when determining the optimal operator
assignments. The blue curve in the figure shows the accuracy-energy trajectory
corresponding to the best sample in each cluster, i.e., the one with the
highest accuracy.

Interestingly, in the figure, variational ENOS shows a quite matching
performance to sequential ENOS. Under extreme energy constraints, variational
ENOS degrades more gracefully and provides better accuracy in the synthesized
networks than sequential ENOS. For example, in ShuffleNet, even when the
operating energy constraints are $\sim$20% of the baseline all typical
operator case, variational ENOS maintains a graceful degradation. Meanwhile,
the accuracy of sequential ENOS drops considerably. Compared to sequential
ENOS, variational ENOS is also more computationally efficient. In sequential
ENOS, the number of training iterations grows with the number of deep learning
layers since each iteration determines the operator for one layer. In
variational ENOS, the entire network is trained only once, similar to ENOS’s
single-level optimization method. Subsequently, many candidate networks are
characterized on the validation set, which is a much less workload than
training itself. Therefore, even with many candidate samples, variational ENOS
maintains high computational efficiency over sequential ENOS.

##  IV ENOS-based DNN Mapping on Hybrid Digital and Compute-in-Memory
Accelerators

###  IV-A Hybrid Digital and Compute-in-Memory Accelerator

To downscale DNN’s processing energy even further, mutually-suited hardware
modules to computationally efficient learning operators are being
investigated. In particular, compute-in-memory (CiM) is becoming a dominant
approach in emerging DNN accelerators [32]. While a crucial bottleneck in
traditional digital DNN accelerators is limited bandwidth between processor
and memory, CiM minimizes weight and operand movements by collapsing DNN’s
parameter storage and computations within the same physical structure. Even
more, multiplication-free and binary operators are especially more suited for
CiM[4, 33] and allow further reduction in their operating energy. However,
since typical CiM units require mixed-signal data conversion circuits
(ADC/DAC) integrated within memory modules, CiM’s energy and area efficiency
is optimal only at a lower precision. Therefore, it is typical to explore CiM-
based DNN acceleration under low precision [32, 34, 9].

TABLE II: Per operation energy of different operators with 4-bit compute-in-memory (CIM) implementation at 1 GHz operating frequency. Operator | Typical | Multiplication-free | Binary
---|---|---|---
Energy/op. | 51.78fJ | 12.95fJ | 6.47fJ
Figure 10: Energy vs. Accuracy of various candidate networks on CIFAR10 and
CIFAR100 for (a) SqueezeNet and (b) ShuffleNet under variational mode of ENOS
and hybrid digital and compute-in-memory acceleration.

Considering the complementary advantages and challenges of digital and CiM for
deep learning, in Figure 9(a), we consider a hybrid DNN accelerator which
combines both processing modes and operating precision, higher precision
digital units and lower precision mixed-signal compute-in-memory units, and
uses ENOS to map DNN layers between them adaptively. In Figure 9(b), the
considered CiM unit is composed of 8-T SRAM cells [shown in the inset of the
figure] and uses bitplane-wise processing that avoids DAC as shown in [34]. In
the bitplane-wise processing, bitplanes of the same significance input and
weight vectors are processed in one cycle. Each SRAM cell has three access
ports–RL, PL, and CL–dedicated for CiM operations. Row-lines (RL) connect
horizontally in Figure 9(b) to select the respective row of the array. Product
lines (PL) and column lines (CL) connect vertically, PL to the right and CL to
the left of a cell. Within memory product is computed on PL while CL applies
input patterns.

The CiM module’s operation begins with precharging PL and applying input to CL
in one half of a clock cycle. Next, RL is activated to compute within cell
product bit (zero or one) in one half of a clock cycle. Then, PLs are summed
using MUXes in Figure 9(b). Finally, the sum-line output is digitized using a
Flash-ADC. The output of Flash-ADC in consecutive cycles is accumulated using
a Shift-ADD unit to determine the net product sum of multi-bit input and
weight precision. For Flash-ADC, we use a clocked rail-to-rail comparator as
shown in Figure 9(c). A similar in-memory compute of DNN was discussed in
details in [35].

Table II shows the energy of various operators in CiM mode. The energy of
various operators are extracted by SPICE simulations using predictive
technology models (PTMs) of 15 nm CMOS technology. As we discussed before, due
to integrated mixed-signal units, CiM implementations are suitable for low
precision operation. Therefore, a 4-bit operation of operators is considered
here. Compared to digital processing, CiM-based operations reduce the
necessary processing energy by $\sim$5-6$\times$.

###  IV-B ENOS for Hybrid Accelerators

For such hybrid acceleration, ENOS searches for mixed-precision, mixed-
operator, as well as mixed-mode operation of DNN layers to optimize accuracy-
energy trade-off. Since, under footprint constraints, a hybrid accelerator may
only support limited CiM capacity, the loss function is modified using a
Lagrangian to integrate the constraint as

$\mathcal{L}\textsubscript{net}=\mathcal{L}\textsubscript{acc}(\theta,\alpha)+\mathcal{L}\textsubscript{en}(\alpha)+\gamma\Big{(}B-\sum_{i=1}^{N}\sum_{j=1}^{M}\text{softmax}(\alpha_{ij})\times
A_{j}\times N_{W,i}\Big{)}^{2}$ (3)

Here, $\mathcal{L}\textsubscript{acc}$ corresponds to the first term in the
right-hand side of (2), and $\mathcal{L}\textsubscript{en}$ corresponds to the
energy-dependent second term in (2). $\gamma$ is the Lagrangian parameter that
enforces the total weight mapping on CiM units equal to $B$. $i$ is an index
over DNN layers, and $j$ is an index over the tuple of operator and processing
mode. $\alpha_{i,j}$ is the learned weight of the operator and processing mode
tuple. $A_{j}$ is the area cost of per weight for operator $j$. For example,
the area-cost per weight in binary operator is 1-bit but in multiplication-
free operator is 4-bit in the considered 4-bit implementation in Table II.

In our previous discussion, variational ENOS was found to be more efficient
over other learning modes. Therefore, variational ENOS is also extended for
hybrid accelerators. Since Lagrangian-based constrained optimization doesn’t
guarantee meeting the equality constraints, each candidate network is tested
if CiM constraints are satisfied and invalid samples are filtered ouot. Figure
10 shows the exemplary clusters of accuracy-energy for various candidate DNNs,
color-coded with the corresponding CiM budget, under variational ENOS for
hybrid accelerators. The samples that violate CiM constraints are marked with
cross symbol ($\times$). Eligible candidate networks that meet CiM budget are
selected and characterized on the validation set by relearning their
corresponding weights. The best candidate network with the highest accuracy on
validation set is chosen as the final output.

Figure 11: Energy vs. Accuracy on CIFAR10 and CIFAR100 for (a) SqueezeNet and
(b) ShuffleNet under varying compute-in-memory budget. Figure 12: Energy vs.
Accuracy on CIFAR10 and CIFAR100 for (a) SqueezeNet and (b) ShuffleNet under
varying compute-in-memory budget.

###  IV-C Simulation Results and Discussions

Figure 11 shows the energy vs. accuracy trends of variational ENOS under
hybrid acceleration. We consider many hybrid accelerator configurations of
varying CiM budget. CiM budget of an accelerator is expressed in terms of the
percentage of weights that can be operated with CiM mode [using modules in
Figure 9] while the remaining weights need to be operated digitally [using
modules in Figure 6]. An accelerator with higher CiM capability can perform
most of the operation in highly energy efficient CiM mode, but also requires
more area to implement since CiM units are not amenable to multiplexing unlike
digital units. In Figure 12, with increasing the CiM budget, ENOS adaptively
allocates more CiM units to exploit their higher energy efficiency while
staying within the allocated CiM budget. Interestingly, compared to the
digital accelerator’s variational ENOS in Figure 8, variational ENOS on the
hybrid accelerator can rather exploit the flexibility of selecting operating
mode, rather than operator choices alone, to allow a significant reduction in
operating energy with only a minimal accuracy loss.

##  V Conclusions

This paper presents a novel class of efficient energy-aware operator search
algorithms (ENOS) for a wide variety of deep neural networks. ENOS can match
the state-of-the-art performance metrics for neural networks on image
classification with remarkable energy efficiency improvement, and can be
applied as-is to other domains/applications such as Transformer based language
models. Additionally, we present a reconfigurable MAC core that implements the
chosen optimal operators by carefully considering the energy-accuracy trade-
offs. In the future, the layer-level operator search approach presented here
can be expanded to neuron-level operator search. The ENOS approach can also be
widened to incorporate other similar goals in multiple resource constrained
settings beyond energy-efficiency.

## References

  * [1] B. Zoph and Q. V. Le, “Neural architecture search with reinforcement learning,” _CoRR_ , vol. abs/1611.01578, 2016. [Online]. Available: http://arxiv.org/abs/1611.01578
  * [2] H. Pham, M. Y. Guan, B. Zoph, Q. V. Le, and J. Dean, “Efficient neural architecture search via parameter sharing,” _CoRR_ , vol. abs/1802.03268, 2018. [Online]. Available: http://arxiv.org/abs/1802.03268
  * [3] H. Liu, K. Simonyan, and Y. Yang, “DARTS: differentiable architecture search,” _CoRR_ , vol. abs/1806.09055, 2018. [Online]. Available: http://arxiv.org/abs/1806.09055
  * [4] S. Nasrin, D. Badawi, A. E. Cetin, W. Gomes, and A. R. Trivedi, “Mf-net: Compute-in-memory sram for multibit precision inference using memory-immersed data conversion and multiplication-free operators,” _arXiv preprint_ , 2020\.
  * [5] S. Nasrin, P. Shukla, S. Jaisimha, and A. R. Trivedi, “Compute-in-memory upside down: A learning operator co-design perspective for scalability,” _IEEE Design Automation and Test in Europe (DATE)_ , 2021.
  * [6] M. Elhoushi, F. Shafiq, Y. H. Tian, J. Y. Li, and Z. Chen, “Deepshift: Towards multiplication-less neural networks,” _CoRR_ , vol. abs/1905.13298, 2019\. [Online]. Available: http://arxiv.org/abs/1905.13298
  * [7] M. Courbariaux, I. Hubara, D. Soudry, R. El-Yaniv, and Y. Bengio, “Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1,” _arXiv preprint arXiv:1602.02830_ , 2016.
  * [8] M. Courbariaux, Y. Bengio, and J. David, “Binaryconnect: Training deep neural networks with binary weights during propagations,” _CoRR_ , vol. abs/1511.00363, 2015. [Online]. Available: http://arxiv.org/abs/1511.00363
  * [9] S. Nasrin, S. Ramakrishna, T. Tulabandhula, and A. R. Trivedi, “Supported-binarynet: Bitcell array-based weight supports for dynamic accuracy-energy trade-offs in sram-based binarized neural network,” in _2020 IEEE International Symposium on Circuits and Systems (ISCAS)_. IEEE, 2020, pp. 1–5.
  * [10] P. Shukla, A. Shylendra, T. Tulabandhula, and A. R. Trivedi, “Mc2ram: Markov chain monte carlo sampling in sram for fast bayesian inference,” in _2020 IEEE International Symposium on Circuits and Systems (ISCAS)_. IEEE, 2020, pp. 1–5.
  * [11] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [12] T. Simons and D.-J. Lee, “A review of binarized neural networks,” _Electronics_ , vol. 8, no. 6, p. 661, 2019.
  * [13] M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi, “Xnor-net: Imagenet classification using binary convolutional neural networks,” in _European conference on computer vision_. Springer, 2016, pp. 525–542.
  * [14] Y. LeCun, C. Cortes, and C. Burges, “Mnist handwritten digit database,” _ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist_ , vol. 2, 2010.
  * [15] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “ImageNet: A Large-Scale Hierarchical Image Database,” in _CVPR09_ , 2009.
  * [16] Y. Bengio, N. Léonard, and A. Courville, “Estimating or propagating gradients through stochastic neurons for conditional computation,” _arXiv preprint arXiv:1308.3432_ , 2013.
  * [17] H. Yang, M. Fritzsche, C. Bartz, and C. Meinel, “Bmxnet: An open-source binary neural network implementation based on mxnet,” in _Proceedings of the 25th ACM international conference on Multimedia_ , 2017, pp. 1209–1212.
  * [18] J. W. Peters and M. Welling, “Probabilistic binary neural networks,” _arXiv preprint arXiv:1809.03368_ , 2018.
  * [19] C. E. Akbaş, A. Bozkurt, A. E. Çetin, R. Çetin-Atalay, and A. Üner, “Multiplication-free neural networks,” in _2015 23nd Signal Processing and Communications Applications Conference (SIU)_. IEEE, 2015, pp. 2416–2418.
  * [20] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” 2017.
  * [21] Y.-H. Chen, T. Krishna, J. S. Emer, and V. Sze, “Eyeriss: An energy-efficient reconfigurable accelerator for deep convolutional neural networks,” _IEEE journal of solid-state circuits_ , vol. 52, no. 1, pp. 127–138, 2016\.
  * [22] N. Iliev and A. R. Trivedi, “Low latency cmos hardware acceleration for fully connected layers in deep neural networks,” _arXiv preprint arXiv:2011.12839_ , 2020.
  * [23] B. Kim, S. Lee, A. R. Trivedi, and W. J. Song, “Energy-efficient acceleration of deep neural networks on realtime-constrained embedded edge devices,” _IEEE Access_ , vol. 8, pp. 216 259–216 270, 2020.
  * [24] J. Su, X. Si, Y. Chou, T. Chang, W. Huang, Y. Tu, R. Liu, P. Lu, T. Liu, J. Wang, Z. Zhang, H. Jiang, S. Huang, C. Lo, R. Liu, C. Hsieh, K. Tang, S. Sheu, S. Li, H. Lee, S. Chang, S. Yu, and M. Chang, “15.2 a 28nm 64kb inference-training two-way transpose multibit 6t sram compute-in-memory macro for ai edge chips,” in _2020 IEEE International Solid- State Circuits Conference - (ISSCC)_ , 2020, pp. 240–242.
  * [25] J. Yue, Z. Yuan, X. Feng, Y. He, Z. Zhang, X. Si, R. Liu, M. Chang, X. Li, H. Yang, and Y. Liu, “14.3 a 65nm computing-in-memory-based cnn processor with 2.9-to-35.8tops/w system energy efficiency using dynamic-sparsity performance-scaling architecture and energy-efficient inter/intra-macro data reuse,” in _2020 IEEE International Solid- State Circuits Conference - (ISSCC)_ , 2020, pp. 234–236.
  * [26] A. Sayal, S. Fathima, S. T. Nibhanupudi, and J. P. Kulkarni, “Compac: Compressed time-domain, pooling-aware convolution cnn engine with reduced data movement for energy-efficient ai computing,” _IEEE Journal of Solid-State Circuits_ , pp. 1–1, 2020.
  * [27] S. Sinha, G. Yeric, V. Chandra, B. Cline, and Y. Cao, “Exploring sub-20nm finfet design with predictive technology models,” in _DAC Design Automation Conference 2012_. IEEE, 2012, pp. 283–288.
  * [28] F. N. Iandola, S. Han, M. W. Moskewicz, K. Ashraf, W. J. Dally, and K. Keutzer, “Squeezenet: Alexnet-level accuracy with 50x fewer parameters and ¡0.5mb model size,” 2016.
  * [29] X. Zhang, X. Zhou, M. Lin, and J. Sun, “Shufflenet: An extremely efficient convolutional neural network for mobile devices,” 2017.
  * [30] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _arXiv preprint arXiv:1409.1556_ , 2014.
  * [31] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 1–9.
  * [32] N. Verma, H. Jia, H. Valavi, Y. Tang, M. Ozatay, L.-Y. Chen, B. Zhang, and P. Deaville, “In-memory computing: Advances and prospects,” _IEEE Solid-State Circuits Magazine_ , vol. 11, no. 3, pp. 43–55, 2019.
  * [33] H. Valavi, P. J. Ramadge, E. Nestler, and N. Verma, “A mixed-signal binarized convolutional-neural-network accelerator integrating dense weight storage and multiplication for reduced data movement,” in _2018 IEEE Symposium on VLSI Circuits_. IEEE, 2018, pp. 141–142.
  * [34] A. Biswas and A. P. Chandrakasan, “Conv-sram: An energy-efficient sram with in-memory dot-product computation for low-power convolutional neural networks,” _IEEE Journal of Solid-State Circuits_ , vol. 54, no. 1, pp. 217–230, 2018.
  * [35] S. Nasrin, D. Badawi, A. E. Cetin, W. Gomes, and A. R. Trivedi, “Mf-net: Compute-in-memory sram for multibit precision inference using memory-immersed data conversion and multiplication-free operators,” _arXiv preprint arXiv:2102.00035_ , 2021.


# MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation

Omer Bar-Tal  Lior Yariv  Yaron Lipman  Tali Dekel

###### Abstract

Recent advances in text-to-image generation with diffusion models present
transformative capabilities in image quality. However, user controllability of
the generated image, and fast adaptation to new tasks still remains an open
challenge, currently mostly addressed by costly and long re-training and fine-
tuning or ad-hoc adaptations to specific image generation tasks. In this work,
we present _MultiDiffusion_ , a unified framework that enables versatile and
controllable image generation, using a pre-trained text-to-image diffusion
model, without any further training or finetuning. At the center of our
approach is a new generation process, based on an optimization task that binds
together multiple diffusion generation processes with a shared set of
parameters or constraints. We show that MultiDiffusion can be readily applied
to generate high quality and diverse images that adhere to user-provided
controls, such as desired aspect ratio (e.g., panorama), and spatial guiding
signals, ranging from tight segmentation masks to bounding boxes.

Machine Learning, ICML

figuret

Figure 1: MultiDiffusion enables flexible text-to-image generation, unifying
multiple controls over the generated content, including desired aspect ratio,
or simple spatial guiding signals such as rough region-based text-prompts.

##  1 Introduction

Text-to-image generative models have emerged as a “disruptive technology”,
demonstrating unprecedented capabilities in synthesizing high-quality and
diverse images from text prompts, where diffusion models are currently
established as state-of-the-art (Saharia et al., 2022b; Ramesh et al., 2022;
Rombach et al., 2022; Croitoru et al., 2022). While this progress holds a
great promise in changing the way we can create digital content, deploying
text-to-image models to real-world applications remains challenging due to the
difficulty to provide users with intuitive control over the generated content.
Currently, controllability over diffusion models is achieved in one of two
ways: (i) training a model from scratch or finetuning a given diffusion model
for the task at hand (e.g., inpainting, layout-to-image training, etc. (Wang
et al., 2022a; Ramesh et al., 2022; Rombach et al., 2022; Nichol et al., 2021;
Avrahami et al., 2022b; Brooks et al., 2022; Wang et al., 2022b)). With the
ever-increasing scale of models and training data, this approach often
requires _extensive compute_ and _long development period_ , even in a
finetuning setting. (ii) Reuse a pre-trained model and add some controlled
generation capability. Previously, these methods have concentrated on specific
tasks and designed a tailored methodology (e.g., replacing objects in an
image, manipulating style, or controlling layout (Tumanyan et al., 2022; Hertz
et al., 2022; Avrahami et al., 2022a)). ††Project page is available at
https://multidiffusion.github.io.

The goal of this work is to design _MultiDiffusion_ , a new unified framework
that significantly increases the flexibility in adapting a pre-trained
(reference) diffusion model to controlled image generation. The basic idea
behind the MultiDiffusion is to define a _new generation process_ that is
composed of several reference diffusion generation processes binded together
with a set of shared parameters or constraints. In more detail, the reference
diffusion model is applied to different regions in the generated image,
predicting a denoising sampling step for each. In turn, the MultiDiffusion
takes a global denoising sampling step reconciling all these different steps
via least squares optimal solution.

For example, consider the task of generating an image at arbitrary aspect
ratio given a reference diffusion model trained on square images (Fig. 2). At
each denoising step, the MultiDiffusion fuses the denoising directions,
provided by the reference model, from _all_ the square crops, and strives to
follow them all as closely as possible, constrained by the fact that nearby
crops share common pixels. Intuitively, we encourage each crop to be a real
sample from the reference model. Note that while each crop might pull to a
different denoising direction, our framework yields a unified denoising step,
hence produces high-quality and seamless images.

With MultiDiffusion, we are able to harness a reference pre-trained text-to-
image model to different applications including synthesizing images at desired
resolution or aspect ratio, or synthesizing images using rough region-based
text prompts, as seen in Fig. 1. Notably, our framework allows to solve these
tasks _simultaneously_ , using a common generation process. Comparing to
relevant baselines, we found that our approach is able to produce state-of-
the-art controlled generation quality even compared to methods that are
specifically trained for these tasks. Furthermore, our method works
efficiently, without introducing computational overhead.

Figure 2: MultiDiffusion: a new generation process, $\Psi$, is defined over a
pre-trained reference model $\Phi$. Starting from a noise image $J_{T}$, at
each generation step, we solve an optimization task whose objective is that
_each_ crop $F_{i}(J_{t})$ will follow as closely as possible its denoised
version $\Phi(F_{i}(J_{t}))$. Note that while each denoising step
$\Phi(F_{i}(J_{t}))$ may pull to a different direction, our process fuses
these inconsistent directions into a _global_ denoising step $\Phi(J_{t})$,
resulting in a high-quality seamless image.

##  2 Related Work

#### Diffusion Models

Diffusion models (Sohl-Dickstein et al., 2015; Croitoru et al., 2022; Dhariwal
& Nichol, 2021; Ho et al., 2020; Nichol & Dhariwal, 2021) are a class of
generative probabilistic models that aim to approximate a data distribution
$q$, and are easy to sample from. Specifically, these models take a Gaussian
noise input $I_{T}\sim\mathcal{N}(0,I)$, and through a series of gradual
denoising steps, transform it into a sample $I_{0}$, that should be
distributed according to $q$. The number of denoising steps, and the
parameterization of the transformation varies among different works (Sohl-
Dickstein et al., 2015; Ho et al., 2020; Song et al., 2020; Lu et al., 2022a,
b; Liu et al., 2022). Recently, Diffusion Models have emerged as state-of-the-
art generators due to their success in learning complex distributions and
generating diverse high quality samples. These models have been successfully
used in various domains, including images (Dhariwal & Nichol, 2021; Nichol &
Dhariwal, 2021; Saharia et al., 2022b; Ramesh et al., 2022; Rombach et al.,
2022), video (Ho et al., 2022; Singer et al., 2022), 3D scenes (Múller et al.,
2022), and motion sequences (Yuan et al., 2022; Tevet et al., 2022).

#### Controllable generation with diffusion models

Diffusion models can be trained with guiding input channels (e.g., semantic
layout, category label) and successfully perform conditional image generation
(Ramesh et al., 2021; Saharia et al., 2022c, a; Wang et al., 2022a; Preechakul
et al., 2022; Ho & Salimans, 2022). The most prominent example of conditional
diffusion models is recent text-to-image diffusion models, which have
demonstrated groundbreaking synthesis capabilities (Nichol et al., 2021;
Saharia et al., 2022b; Ramesh et al., 2022; Nichol et al., 2021; Rombach et
al., 2022; Sheynin et al., 2022). However, these models provide only little
control over the generated content, which is mainly achieved through the input
text. Recently, a surge of methods have been proposed to gain wider and better
user controllability. Existing methods can be roughly divided into two main
approaches: (i) methods that incorporate explicit control by using additional
guiding signals to the model (Avrahami et al., 2022b; Rombach et al., 2022;
Brooks et al., 2022). However, these works require costly extensive training
on curated datasets. (ii) On the other side of the spectrum, numerous methods
proposed to implicitly control the generated content by manipulating the
generation process of a pre-trained model (Kwon & Ye, 2022; Meng et al., 2021;
Tumanyan et al., 2022; Hertz et al., 2022; Avrahami et al., 2022c; Choi et
al., 2021; Mokady et al., 2022; Couairon et al., 2022; Kong et al., 2023; Kwon
et al., 2022) or by performing lightweight model finetuning (Ruiz et al.,
2022; Kawar et al., 2022; Kim et al., 2022; Valevski et al., 2022). Avarahami
et al. designed image inpainting methods (Avrahami et al., 2022a, c) that do
not require finetuning. Recent works (Tumanyan et al., 2022; Hertz et al.,
2022) rely on architectural properties and insights about the internal
features of the pretrained model, and tailor image editing techniques
accordingly. Our work also manipulates the generation process of a pretrained
diffusion model, and does not require any training or finetuning. However, in
contrast to existing works that target a specific application, without a well
defined objective, we propose a more general approach that allows us to unify
different user control inputs in a more principled manner.

##  3 Method

We consider a pre-trained diffusion model, which serves as a reference model:

$\Phi:{\mathcal{I}}\times{\mathcal{Y}}\rightarrow{\mathcal{I}}$

working in image space ${\mathcal{I}}=\mathbb{R}^{H\times W\times C}$ and
condition space ${\mathcal{Y}}$, e.g., $y\in{\mathcal{Y}}$ is a text prompt.
Initializing $I_{T}\sim P_{\mathcal{I}}$, where $P_{\mathcal{I}}$ represents
the distribution of Gaussian i.i.d. pixel values, and setting a condition
$y\in{\mathcal{Y}}$, the diffusion model builds a sequence of images,

$\begin{array}[]{lcr}I_{T},I_{T-1},\ldots,I_{0}&\text{s.t.}&I_{t-1}=\Phi(I_{t}|y)\end{array}$
(1)

gradually transforming the noisy image $I_{T}$ into a clean image $I_{0}$.

#### MultiDiffusion.

Our goal is to leverage $\Phi$ to generate images in a potentially different
image space ${\mathcal{J}}=\mathbb{R}^{H^{\prime}\times W^{\prime}\times C}$
and condition space ${\mathcal{Z}}$, without any training or finetuning. To do
so, we define a _MultiDiffusion process_ , defined by a function, called
_MultiDiffuser_ ,

$\Psi:{\mathcal{J}}\times{\mathcal{Z}}\rightarrow{\mathcal{J}}$

The MultiDiffusion, similarly to a diffusion process, starts with some initial
noisy input $J_{T}\sim P_{\mathcal{J}}$, where $P_{\mathcal{J}}$ is a noise
distribution over ${\mathcal{J}}$, and produces a series of images

$\begin{array}[]{lcr}J_{T},J_{T-1},\ldots,J_{0}&\text{s.t.}&J_{t-1}=\Psi(J_{t}|z)\end{array}$
(2)

Our key idea is to define $\Psi$ to be _as-consistent-as-possible_ with
$\Phi$. More specifically, we define a set of mappings between the target and
reference image spaces $F_{i}:{\mathcal{J}}\rightarrow{\mathcal{I}}$, and a
corresponding set of mappings between the condition spaces:
$\lambda_{i}:{\mathcal{Z}}\rightarrow{\mathcal{Y}}$ where
$i\in[n]=\left\\{1,\ldots,n\right\\}$. These mappings are application
depended, as will be described later in Sec. 4. Our goal is to make every
MultiDiffuser step $J_{t-1}=\Psi(J_{t}|z)$ follow as closely as possible
$\Phi(I^{i}_{t}|y_{i})$, $i\in[n]$, i.e., the denoising steps of $\Phi$ when
applied to the images and conditions:

$I^{i}_{t}=F_{i}(J_{t}),\quad y_{i}=\lambda_{i}(z)$

Formally, our new process is given by solving the following optimization
problem:

$\displaystyle\Psi(J_{t}|z)=$
$\displaystyle\operatorname*{arg\,min}_{J\in{\mathcal{J}}}\ \
{\mathcal{L}}_{{\scriptscriptstyle\text{FTD}}}(J|J_{t},z)$ (3)

${\mathcal{L}}_{{\scriptscriptstyle\text{FTD}}}(J|J_{t},z)=\sum_{i=1}^{n}\Big{\|}W_{i}\otimes\Big{[}F_{i}(J)-\Phi(I^{i}_{t}|y_{i})\Big{]}\Big{\|}^{2}$
(4)

where $W_{i}\in\mathbb{R}^{H\times W}_{+}$ are per pixel weights and $\otimes$
is the Hadamard product. Intuitively, the FTD loss reconciles, in the least-
squares sense, the different denoising sampling steps,
$\Phi(I^{i}_{t}|y_{i})$, suggested on different regions, $F_{i}(J_{t})$, of
the generated image $J_{t}$. Fig. 2 illustrates one step of the MultiDiffuser;
Algorithm 2 recaps the MultiDiffusion sampling process.

#### Closed-form formula.

In the applications demonstrated in this paper $F_{i}$ consist of direct pixel
samples (e.g., taking a crop out of image $J_{t}$). In this case, Eq. 4 is a
quadratic Least-Squares (LS) where each pixel of the minimizer $J$ is a
weighted average of all its diffusion sample updates, i.e.,

$\Psi(J_{t}|z)=\sum_{i=1}^{n}\frac{F_{i}^{-1}(W_{i})}{\sum_{j=1}^{n}F_{j}^{-1}(W_{j})}\otimes
F_{i}^{-1}(\Phi(I^{i}_{t}|y_{i}))$ (5)

#### Properties of MultiDiffusion.

The main motivation for the definition of $\Psi$ in Eq. 3 comes from the
following observation: If we choose a probability distribution
$P_{\mathcal{J}}$ such that

$F_{i}(J_{T})\sim P_{\mathcal{I}},\qquad\forall i\in[n]$ (6)

and compute $J_{t-1}=\Psi(J_{t}|z)$, as defined in Eq. 3, where we reach a
zero FTC loss,
${\mathcal{L}}_{\scriptscriptstyle\text{FTD}}(J_{t-1}|J_{t},z)=0$, then:

$I^{i}_{t-1}=F_{i}(J_{t})=\Phi(I^{i}_{t}|y_{i})$

That is, $I^{i}_{t}$, for all $i\in[n]$, is a diffusion sequence and thus
$I^{i}_{0}$ is distributed according to the distribution defined by $\Phi$
over the image space ${\mathcal{I}}$. We summarize

######  Proposition 3.1.

If $P_{\mathcal{J}}$ is a distribution over ${\mathcal{J}}$ satisfying Eq. 6,
and the FTD cost (Eq. 4) is minimized to zero in Eq. 3 for all steps
$T,T-1,\ldots,0$, then the images $I^{i}_{t}=F_{i}(J_{t})$ reproduce a $\Phi$
diffusion path. In particular $F_{i}(J_{0})$, $i\in[n]$ are distributed
identically to samples from the reference diffusion model $\Phi$.

The implications of this proposition are far reaching: using a single
reference diffusion process we can flexibly adapt to different image
generation scenarios without the need to retrain the model, while still being
consistent with the reference diffusion model. Next, we instantiate this
framework outlining several application of the Follow-the-Diffusion-Paths
approach.

Input : ;

$\Phi$ $\triangleright$ pre-trained Diffusion Model
;

$\\{F_{i}\\}_{i=1}^{n}$ $\triangleright$ image space mappings
;

$\\{y_{i}\\}_{i=1}^{n}$ $\triangleright$ text-prompts conditioning
;

$\\{W_{i}\\}_{i=1}^{n}$ $\triangleright$ per-pixel weights

$J_{T}\sim P_{\mathcal{J}}$ $\triangleright$ noise initialization
for  _$t=T,...,1$_ do

;

$I_{t-1}^{i}\leftarrow\Phi(F_{i}(J_{t}),y_{i})\;\;\forall i\in[n]$
$\triangleright$ diffusion updates
$J_{t-1}\leftarrow\texttt{MultiDiffuser}(\left\\{I^{i}_{t-1}\right\\}_{i=1}^{n})$
$\triangleright$ Eq. 5

Output : $J_{0}$

Algorithm 1 MultiDiffusion sampling.

##  4 Applications

###  4.1 Panorama

As a first instantiation we use our framework to define a diffusion model in
an image space ${\mathcal{J}}$ with $H^{\prime}\geq H$, $W^{\prime}\geq H$
directly from a trained model $\Phi$ working in image space ${\mathcal{I}}$.
Let ${\mathcal{Z}}={\mathcal{Y}}$ (namely, generating a panoramic image for a
given text-prompt), $F_{i}(J)\in{\mathcal{I}}$ is an $H\times W$ crop of image
$J$, and $z=\lambda_{i}(z)$. We consider $n$ such crops that cover the
original images $J$. Setting $W_{i}=\mathbf{1}$, we get

$\displaystyle\Psi(J_{t},z)=$
$\displaystyle\operatorname*{arg\,min}_{J\in{\mathcal{J}}}\ \
\sum_{i=1}^{n}\left\|F_{i}(J)-\Phi(F_{i}(J),z)\right\|^{2}$ (7)

that is a least-squares problem, the solution of which is calculated
analytically according to Eq. 5. See the Appendix B.1 for implementation
details.

---
(a) Generation with per-crop independent diffusion paths.
(b) Generation with fused diffusion paths using MultiDiffusion.
Figure 3: Independent diffusion paths vs. MultiDiffusion. (a) Panoramic image
generated by applying the reference model on four crops independently; as
expected, there is no coherency between the crops. (b) Starting from the same
noise, our generation process steers these initial diffusion paths into a
consistent and high quality image.

As discussed in Sec. 3, _MultiDiffusion_ reconciles multiple diffusion paths
provided by the reference model $\Phi$. We illustrate this property in Fig. 3,
where we consider a panorama of $H\times 4W$. Fig. 3(a) shows the generation
result when independently applying $\Phi$ on four non-overlapping crops. As
expected, there is no coherency between the crops since this amounts to four
random samples from the model. Starting from the same initial noise, our
generation process (Eq. 7), allows us to fuse these initially-unrelated
diffusion paths, and steer the generation into a high-quality, coherent
panorama (b).

###  4.2 Region-based text-to-image-generation

Given a set of region-masks $\\{M_{i}\\}_{i=1}^{n}\subset\\{0,1\\}^{H\times
W}$ and a corresponding set of text-prompts
$\\{y_{i}\\}_{i=1}^{n}\subset{\mathcal{Y}}^{n}$, our goal is to generate a
high-quality image $I\in{\mathcal{I}}$ that depicts the desired content in
each region. That is, the image segment $I\otimes M_{i}$ should manifest
$y_{i}$. Going back to our formulation (Eq. 2), the _MultiDiffusion_ process
is defined over the condition space ${\mathcal{Z}}={\mathcal{Y}}^{n}$, i.e.,
$z=(y_{1},\ldots,y_{n})$, and the target image space
${\mathcal{J}}={\mathcal{I}}$ is identical to the reference one:

$\Psi:{\mathcal{I}}\times{\mathcal{Y}}^{n}\rightarrow{\mathcal{I}}$

Furthermore, the region selection maps are defined as $F_{i}(I)=I$, the pixel
weights are set according to the masks, $W_{i}=M_{i}$, and the $\Psi$ step is
defined as the solution to the least-squares problem:

$\displaystyle\Psi(J_{t},z)=$
$\displaystyle\operatorname*{arg\,min}_{J\in{\mathcal{I}}}\ \
\sum_{i=1}^{n}\Big{\|}M_{i}\otimes\Big{[}J-\Phi(J_{t}|y_{i})\Big{]}\Big{\|}^{2}$
(8)

The solution to this LS problem is calculated analytically. At each step we
apply the pretrained diffusion w.r.t. each of the given prompts, resulting in
multiple diffusion directions $\Phi(J_{t}|y_{i})$. We encourage each pixel in
$J_{t}$ to follow the (averaged) directions associated with the regions
$M_{i}$ containing it (Eq. 5).

#### Fidelity to tight masks

We further support obtaining high-fidelity to tight masks if provided by the
user (see Fig. 5). We noticed that the layout is being determined early on in
the diffusion process, and thus we strive to encourage $\Phi(J_{t}|y_{i})$ to
focus on the region $M_{i}$ early on in the process in order to match the
desired layout, and to consider the full context in the image next, to achieve
an harmonized result. We integrate time dependency in the maps $F_{i}$,
introducing a bootstrapping phase. That is,

$F_{i}(J_{t},t)=\begin{cases}J_{t},&\text{if $t\leq T_{init}$}\\\ M_{i}\otimes
J_{t}+(1-M_{i})\otimes S_{t},&\text{otherwise}\end{cases}$ (9)

Where $T_{init}$ is the bootstrapping stopping step parameter, and $S_{t}$ is
a random image with a constant color, which serves as background (see Appendix
B.2 for implementation details).

We demonstrate the efficiency of our bootstrapping approach in Sec. 5.2. We
set $T_{init}$ to be $20\%$ of the generation process (i.e., $T_{init}=800$).

Figure 4: Text-to-Panorama comparison to Blended Latent Diffusion (BLD)
(Avrahami et al., 2022a) and Stable Inpainting (SI) (Rombach et al., 2022).
Our framework produces seamless and diverse content whereas the baselines
either contain repetitive content, visible seams or artifacts.

##  5 Results

We thoroughly evaluate our method when applied to each task as discussed in
Sec. 4. In all experiments, we used Stable Diffusion (Rombach et al., 2022),
where the diffusion process is defined over a latent space
${\mathcal{I}}=\mathbb{R}^{64\times 64\times 4}$, and a decoder is trained to
reconstruct natural images in higher resolution $[0,1]^{512\times 512\times
3}$. Similarly, the MultiDiffusion process, $\Psi$ is defined in the latent
space ${\mathcal{J}}=\mathbb{R}^{H`\times W`\times 4}$ and using the decoder
we produce the results in the target image space $[0,1]^{8H^{\prime}\times
8W^{\prime}\times 3}$. We use the public implementation of Stable Diffusion by
HuggingFace (von Platen et al., 2022), with the v2 pre-trained model.

###  5.1 Panorama Generation

To evaluate our method on the task of text-to-panorama generation (Sec. 4.1),
we generated a diverse set of $512\times 4608$ panoramas, $\times 9$ wider
than the original training resolution. Since there is no direct method for
generating images at arbitrary aspect ratio from text, we compare to the
following two baselines: (i) Blended Latent Diffusion (BLD) (Avrahami et al.,
2022a) (combined with Stable Diffusion (Rombach et al., 2022)), and Stable
Inpainting (SI) (Rombach et al., 2022), which has been finetuned on large-
scale data for inpainting. For both baselines, the panoramic image is
generated gradually, starting from a central image (sampled by $\Phi$ given
the input text), and extrapolated progressively to the right and left.

Fig. 4 shows sample generation results by our method compared to the above
baselines. As seen, both baselines often exhibit visible seams and
discontinuities between overlapping crops, as well as degradation in visual
quality as moving away from the center pivotal image; this is expected due to
the iterative generation process. BLD often generates repetitive content
(e.g., skiers example), where SI results in noticeable visual difference
between the left and right parts of the image. In contrast, our framework
_simultaneously_ “samples” the panoramic image by combining the diffusion
paths of _all_ crops, resulting in seamless and high quality images.
Additional comparisons are in the Appendix 10.

| FID $\downarrow$ | CLIP-score$\uparrow$ | CLIP-aesthetic$\uparrow$
---|---|---|---
Stable Diffusion | $6.05\pm 3.1$ | $0.27$ | $6.36$
SI | $45.5\pm 14.5$ | $0.26$ | $5.76$
BLD | $18.4\pm 7.4$ | $\bf{0.27}$ | $6.02$
Ours | $\bf{10.3\pm 4.8}$ | $\bf{0.27}$ | $\bf{6.36}$
Table 1:  Panorama generation evaluation. We report FID, CLIP text-image
score, and CLIP aesthetic scores for of our method compared to the baselines.
See more details in Section. 5.1.

To quantify these observations, we use the Frechet Inception Distance (FID)
(Parmar et al., 2022) to measure the distance between the distribution of
$512\times 512$ crops from the panoramic images to the distribution of images
generated by the reference model $\Phi$. That is, for a given text prompt, we
sample $N$ different $512\times 512$ images from $\Phi$, and consider them as
our reference dataset. For the baselines and our method, we generated $N$
panoramic images, and randomly sampled a $512\times 512$ crop from each sample
to serves as the generated dataset and computed the FID accordingly.

To further assess the quality of our results, we evaluated two CLIP-based
scores: (i) text-image CLIP score (Radford et al., 2021) measured by the
cosine similarity between the text prompt and the image embeddings, and (ii)
CLIP aesthetic (Schuhmann et al., 2022) measured by a linear estimator on top
of CLIP predicting the aesthetic quality of the images.

We used $N=2000$ samples and repeated this evaluation for $8$ different text-
conditioning. Table 1 reports the mean and standard-deviation of FID and CLIP
scores for our method and the baselines. We additionally report the scores for
an independent set of samples images from $\Phi$, which serves as a baseline.
As seen, our method outperforms the existing baselines in all metrics.

Figure 5: Region-based text-to-image generation. The input segmentation maps
with the corresponding region text descriptions are shown above each example.
Below: Make-A-Scene (Gafni et al., 2022)and SpaText (Avrahami et al., 2022b) –
trained specifically for this task on a large-scale segmentation-text-image
dataset; Blended Latent Diffusion (BLD) (Avrahami et al., 2022a), and our
results.

###  5.2 Region-based Text-to-Image Generation

Our region-based formulation (Sec. 4.2) allows novice users greater
flexibility in their content creation, by lifting the burden of creating
accurate tight masks. As can be seen in Fig. 1, Fig. 7 and Fig. 8, our method
generates diverse high-quality samples that comply with text description,
given only bounding boxes region guidance. As seen in Fig. 7, by starting our
generation from a different input noise, we can generate diverse samples,
depicting objects in different scales and appearances, all following the same
spatial controls. Notably, since we integrate the controls from all regions
into a unified generation process, our method can generate complex scene
effects (e.g., background blur, shadows or reflections) which are coherently
immerse in the scene. More results are included in the Appendix.

We compare our region-based framework with Make-A-Scene (Gafni et al., 2022)
and the concurrent work SpaText (Avrahami et al., 2022b). Both baselines
perform large-scale training specifically for this task. Note that these
models are not publicly available, thus we qualitatively compare to their
provided examples.

Additionally, we consider an adaptation of BLD (Avrahami et al., 2022a) as a
baseline. Similarly to Sec. 5.1, this is done by applying their method in an
auto-regressive manner by first generating the background, and sequentially
generating each of the foreground objects.

As seen in Fig. 5, our framework produces consistent images that adhere to the
spatial constraints, and are qualitatively on par with (Avrahami et al.,
2022b). The auto-regressive approach based on BLD (Avrahami et al., 2022a)
often results in incoherent images and an unnatural scene. (e.g., misplaced
sink in “bathroom” example). Additional comparisons to the baselines are in
the Appendix.

| IoU $\uparrow$
---|---
COCO dataset | $0.43\pm 0.09$
SI | $0.16\pm 0.10$
BLD | $0.17\pm 0.11$
Ours w/o bootstrapping | $0.18\pm 0.10$
Ours | $\bf{0.26\pm 0.12}$
Table 2: Region-based generation evaluation of the COCO dataset. We evaluate
Intersection over Union (IoU), see Sec. 5.2 for details. Figure 6:
Bootstrapping ablation. Without bootstrapping (middle), our method
successfully generates the glass in _some_ location inside the mask (left).
With our bootstrapping mechanism (right), we achieve high-fidelity to the
provided tight mask. See Sec 4.2 for details.

To quantitatively evaluate our performance, we use the COCO dataset (Lin et
al., 2014), which contains images with global text caption and instance masks
for each object in the image. We apply our method on a subset from the
validation set, obtained by filtering examples which consists of 2 to 4
foreground objects, excluding people, and masks that occupy less than $5\%$ of
the image. This results in $1K$ diverse samples. Following (Avrahami et al.,
2022b), we use the ground truth labels to provide a text prompt for each
foreground region, i.e., “a {label}”, and use the full image caption as the
prompt describing the background.

We evaluate the results with an off-the-shelf segmentation model (Cheng et
al., 2022) on the generated images, and measure the Intersection over Union
(IoU) w.r.t. to the ground-truth segmentation. Table 2 reports the performance
for our method and the baselines described above. As an upper bound, we also
report the IoU w.r.t. the original images in the set. Note that our method
outperforms the existing baselines SI (Rombach et al., 2022) and BLD (Avrahami
et al., 2022a). We additional provide qualitative examples are included in the
Appendix.

Finally, we present an ablation of our bootstrapping stage ( Eq. 9):
qualitatively in Fig. 6, and quantitatively in Table 2. Note that without
bootstrapping, our framework still generates the desired object within the
mask region, however, the bootstrapping stage makes it tighter to the given
mask.

Figure 7: Diverse samples generated by our framework, given rough scene layout
guidance (left). All images depict sensible composition, scene effects and
relative size of objects. Figure 8: Rough masks. Sample results of our region-
based generation approach (see Sec. 4.2). Our method can work with rough
masks, the can intuitively be obtained by novice users.

##  6 Discussion and Conclusions

Controllable generation is one of the major pending challenges with text-to-
image diffusion models. We proposed to tackle this challenge from a
fundamentally new direction – defining a new generation process on top of a
pre-trained and fixed diffusion model. This approach has several key
advantages over previous works: (i) it does not require any further training
or finetuning, (ii) it can be applied to various different generation tasks,
and (iii) our generation process yields an optimization task which can be
solved in closed form for many tasks, hence can be computed efficiently, while
ensuring convergence to the global optimum of our objective. As for
limitations, our method heavily relies on the generative prior of the
reference diffusion model, i.e., the quality of our results depends on the
diffusion paths provided by the model. Thus, when a “bad” path is chosen by
the reference model (e.g., bad seed, or biased text-prompt), our results will
be affected as well. In some cases, we can mitigate it by introducing more
constraints into our framework (bootstrapping in Sec. 4.2), or prompt-
engineering (Fig. 9). We thoroughly evaluated our framework, demonstrating
state-of-the-art results even compared to methods that are tailored-trained
for specific tasks.

Figure 9: Our method heavily relies on the prior of the reference diffusion
model. Left: our method when applied to a vertical panorama. The reference
diffusion model is biased towards adding a waterfall in each viewing crop,
resulting with an unnatural scene. Right: we can try to overcome this by
adding a region-based constraint.

We believe that our work can trigger further future research in harnessing the
power of a pre-trained diffusion model in more principled manner. One way
forward, for example, is to generalize the MultiDiffusion with a more general
optimization problem,

$\displaystyle\Psi(J_{t}|z)=$
$\displaystyle\operatorname*{arg\,min}_{J\in{\mathcal{J}}}\ \
{\mathcal{L}}_{{\scriptscriptstyle\text{FTD}}}(J|J_{t},z)+{\mathcal{L}}_{0}(J,J_{t},z)$
(10) $\displaystyle\qquad\ \text{s.t.}\ \ \ J\in{\mathcal{C}}(J_{t},z)$

where ${\mathcal{L}}_{0}$ is a cost function and ${\mathcal{C}}$ is a set of
(hard) constrains that control the MultiDiffusion process by incorporating
other priors and/or design constraints. This approach provides a further of
freedom in designing MultiDiffusion processes.

##  7 Acknowledgments

LY is supported by a grant from Israel CHE Program for Data Science Research
Centers and the Minerva Stiftung. OB is supported by the Israeli Science
Foundation (grant 2303/20). The research was supported also in part by a
research grant from the Carolito Stiftung (WAIC). We thank Michal Geyer and
Dolev Ofri-Amar for proofreading the paper.

## References

  * Avrahami et al. (2022a) Avrahami, O., Fried, O., and Lischinski, D.  Blended latent diffusion.  _arXiv preprint arXiv:2206.02779_ , 2022a.
  * Avrahami et al. (2022b) Avrahami, O., Hayes, T., Gafni, O., Gupta, S., Taigman, Y., Parikh, D., Lischinski, D., Fried, O., and Yin, X.  Spatext: Spatio-textual representation for controllable image generation.  _arXiv preprint arXiv:2211.14305_ , 2022b.
  * Avrahami et al. (2022c) Avrahami, O., Lischinski, D., and Fried, O.  Blended diffusion for text-driven editing of natural images.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 18208–18218, 2022c.
  * Brooks et al. (2022) Brooks, T., Holynski, A., and Efros, A. A.  Instructpix2pix: Learning to follow image editing instructions.  November 2022.
  * Cheng et al. (2022) Cheng, B., Misra, I., Schwing, A. G., Kirillov, A., and Girdhar, R.  Masked-attention mask transformer for universal image segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022.
  * Choi et al. (2021) Choi, J., Kim, S., Jeong, Y., Gwon, Y., and Yoon, S.  Ilvr: Conditioning method for denoising diffusion probabilistic models.  _arXiv preprint arXiv:2108.02938_ , 2021.
  * Couairon et al. (2022) Couairon, G., Verbeek, J., Schwenk, H., and Cord, M.  Diffedit: Diffusion-based semantic image editing with mask guidance.  _arXiv preprint arXiv:2210.11427_ , 2022.
  * Croitoru et al. (2022) Croitoru, F.-A., Hondru, V., Ionescu, R. T., and Shah, M.  Diffusion models in vision: A survey.  _arXiv preprint arXiv:2209.04747_ , 2022.
  * Dhariwal & Nichol (2021) Dhariwal, P. and Nichol, A.  Diffusion models beat gans on image synthesis.  _Advances in Neural Information Processing Systems_ , 2021.
  * Gafni et al. (2022) Gafni, O., Polyak, A., Ashual, O., Sheynin, S., Parikh, D., and Taigman, Y.  Make-a-scene: Scene-based text-to-image generation with human priors.  In _European Conference on Computer Vision (ECCV)_ , 2022.
  * Hertz et al. (2022) Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., and Cohen-Or, D.  Prompt-to-prompt image editing with cross attention control.  _arXiv preprint arXiv:2208.01626_ , 2022.
  * Ho & Salimans (2022) Ho, J. and Salimans, T.  Classifier-free diffusion guidance.  _arXiv preprint arXiv:2207.12598_ , 2022.
  * Ho et al. (2020) Ho, J., Jain, A., and Abbeel, P.  Denoising diffusion probabilistic models.  _Advances in Neural Information Processing Systems_ , 2020.
  * Ho et al. (2022) Ho, J., Chan, W., Saharia, C., Whang, J., Gao, R., Gritsenko, A., Kingma, D. P., Poole, B., Norouzi, M., Fleet, D. J., et al.  Imagen video: High definition video generation with diffusion models.  _arXiv preprint arXiv:2210.02303_ , 2022.
  * Kawar et al. (2022) Kawar, B., Zada, S., Lang, O., Tov, O., Chang, H., Dekel, T., Mosseri, I., and Irani, M.  Imagic: Text-based real image editing with diffusion models.  _arXiv preprint arXiv:2210.09276_ , 2022.
  * Kim et al. (2022) Kim, G., Kwon, T., and Ye, J. C.  Diffusionclip: Text-guided diffusion models for robust image manipulation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 2426–2435, 2022.
  * Kong et al. (2023) Kong, C., Jeon, D., Kwon, O., and Kwak, N.  Leveraging off-the-shelf diffusion model for multi-attribute fashion image manipulation.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2023.
  * Kwon & Ye (2022) Kwon, G. and Ye, J. C.  Diffusion-based image translation using disentangled style and content representation.  _arXiv preprint arXiv:2209.15264_ , 2022.
  * Kwon et al. (2022) Kwon, M., Jeong, J., and Uh, Y.  Diffusion models already have a semantic latent space.  _arXiv preprint arXiv:2210.10960_ , 2022.
  * Lin et al. (2014) Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L.  Microsoft coco: Common objects in context.  In _European conference on computer vision_ , pp. 740–755. Springer, 2014.
  * Liu et al. (2022) Liu, L., Ren, Y., Lin, Z., and Zhao, Z.  Pseudo numerical methods for diffusion models on manifolds.  _arXiv preprint arXiv:2202.09778_ , 2022.
  * Lu et al. (2022a) Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C., and Zhu, J.  Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps.  _arXiv preprint arXiv:2206.00927_ , 2022a.
  * Lu et al. (2022b) Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C., and Zhu, J.  Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models.  _arXiv preprint arXiv:2211.01095_ , 2022b.
  * Meng et al. (2021) Meng, C., Song, Y., Song, J., Wu, J., Zhu, J.-Y., and Ermon, S.  Sdedit: Image synthesis and editing with stochastic differential equations.  _arXiv preprint arXiv:2108.01073_ , 2021.
  * Mokady et al. (2022) Mokady, R., Hertz, A., Aberman, K., Pritch, Y., and Cohen-Or, D.  Null-text inversion for editing real images using guided diffusion models.  _arXiv preprint arXiv:2211.09794_ , 2022.
  * Múller et al. (2022) Múller, N., , Siddiqui, Y., Porzi, L., Rota Buló, S., Kontschieder, P., and Nießner, M.  Diffrf: Rendering-guided 3d radiance field diffusion.  arxiv, 2022.
  * Nichol et al. (2021) Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., and Chen, M.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  _arXiv preprint arXiv:2112.10741_ , 2021.
  * Nichol & Dhariwal (2021) Nichol, A. Q. and Dhariwal, P.  Improved denoising diffusion probabilistic models.  In _International Conference on Machine Learning_ , pp. 8162–8171. PMLR, 2021.
  * Parmar et al. (2022) Parmar, G., Zhang, R., and Zhu, J.-Y.  On aliased resizing and surprising subtleties in gan evaluation.  In _CVPR_ , 2022.
  * Preechakul et al. (2022) Preechakul, K., Chatthee, N., Wizadwongsa, S., and Suwajanakorn, S.  Diffusion autoencoders: Toward a meaningful and decodable representation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 10619–10629, 2022.
  * Radford et al. (2021) Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.  Learning transferable visual models from natural language supervision.  In _International Conference on Machine Learning_ , pp. 8748–8763. PMLR, 2021.
  * Ramesh et al. (2021) Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., and Sutskever, I.  Zero-shot text-to-image generation.  In _International Conference on Machine Learning_ , pp. 8821–8831. PMLR, 2021.
  * Ramesh et al. (2022) Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ , 2022.
  * Rombach et al. (2022) Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 10684–10695, 2022.
  * Ruiz et al. (2022) Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., and Aberman, K.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  _arXiv preprint arXiv:2208.12242_ , 2022.
  * Saharia et al. (2022a) Saharia, C., Chan, W., Chang, H., Lee, C., Ho, J., Salimans, T., Fleet, D., and Norouzi, M.  Palette: Image-to-image diffusion models.  In _ACM SIGGRAPH 2022 Conference Proceedings_ , 2022a.
  * Saharia et al. (2022b) Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E., Ghasemipour, S. K. S., Ayan, B. K., Mahdavi, S. S., Lopes, R. G., et al.  Photorealistic text-to-image diffusion models with deep language understanding.  _arXiv preprint arXiv:2205.11487_ , 2022b.
  * Saharia et al. (2022c) Saharia, C., Ho, J., Chan, W., Salimans, T., Fleet, D. J., and Norouzi, M.  Image super-resolution via iterative refinement.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2022c.
  * Schuhmann et al. (2022) Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., Schramowski, P., Kundurthy, S., Crowson, K., Schmidt, L., Kaczmarczyk, R., and Jitsev, J.  Laion-5b: An open large-scale dataset for training next generation image-text models.  _ArXiv_ , abs/2210.08402, 2022.
  * Sheynin et al. (2022) Sheynin, S., Ashual, O., Polyak, A., Singer, U., Gafni, O., Nachmani, E., and Taigman, Y.  Knn-diffusion: Image generation via large-scale retrieval.  _arXiv preprint arXiv:2204.02849_ , 2022.
  * Singer et al. (2022) Singer, U., Polyak, A., Hayes, T., Yin, X., An, J., Zhang, S., Hu, Q., Yang, H., Ashual, O., Gafni, O., Parikh, D., Gupta, S., and Taigman, Y.  Make-a-video: Text-to-video generation without text-video data, 2022.
  * Sohl-Dickstein et al. (2015) Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S.  Deep unsupervised learning using nonequilibrium thermodynamics.  In _International Conference on Machine Learning_ , pp. 2256–2265. PMLR, 2015.
  * Song et al. (2020) Song, J., Meng, C., and Ermon, S.  Denoising diffusion implicit models.  _arXiv preprint arXiv:2010.02502_ , 2020.
  * Tevet et al. (2022) Tevet, G., Raab, S., Gordon, B., Shafir, Y., Bermano, A. H., and Cohen-Or, D.  Human motion diffusion model.  _arXiv preprint arXiv:2209.14916_ , 2022.
  * Tumanyan et al. (2022) Tumanyan, N., Geyer, M., Bagon, S., and Dekel, T.  Plug-and-play diffusion features for text-driven image-to-image translation.  _arXiv preprint arXiv:2211.12572_ , 2022.
  * Valevski et al. (2022) Valevski, D., Kalman, M., Matias, Y., and Leviathan, Y.  Unitune: Text-driven image editing by fine tuning an image generation model on a single image.  _arXiv preprint arXiv:2210.09477_ , 2022.
  * von Platen et al. (2022) von Platen, P., Patil, S., Lozhkov, A., Cuenca, P., Lambert, N., Rasul, K., Davaadorj, M., and Wolf, T.  Diffusers: State-of-the-art diffusion models.  https://github.com/huggingface/diffusers, 2022.
  * Wang et al. (2022a) Wang, T., Zhang, T., Zhang, B., Ouyang, H., Chen, D., Chen, Q., and Wen, F.  Pretraining is all you need for image-to-image translation.  In _arXiv_ , 2022a.
  * Wang et al. (2022b) Wang, W., Bao, J., Zhou, W., Chen, D., Chen, D., Yuan, L., and Li, H.  Semantic image synthesis via diffusion models.  _arXiv preprint arXiv:2207.00050_ , 2022b.
  * Yuan et al. (2022) Yuan, Y., Song, J., Iqbal, U., Vahdat, A., and Kautz, J.  Physdiff: Physics-guided human motion diffusion model.  _arXiv preprint arXiv:2212.02500_ , 2022.

##  Appendix A Additional Results

In the following section we provide additional results and comparisons for the
applications shown in the main paper.

###  A.1 Panorama Generation

We provide additional results and qualitative comparisons for the task of
text-to-panorama (Sec. 5.1). Fig. 10 depicts additional comparisons of our
method vs Stable Inpainting (SI) (Rombach et al., 2022) and Blended Latent
Diffusion (BLD) (Avrahami et al., 2022a). We also show vertical panorama
result in Fig. 12 left.

###  A.2 Region-based Text-to-Image Generation

We provide additional qualitative results and comparisons for the task of
region-based generation (Sec.4.2) in Fig. 12 and Fig. 14.

###  A.3 Region-based Text-to-Image Generation on COCO

We include sample results and comparison on the subset from the validation set
of COCO in Fig. 13. See more details about this experiment in Sec. 5.2.

Figure 10: Text-to-Panorama additional results and comparisons to Sec. 5.1.
Figure 11: Additional results of our method on generation from rough scene
layouts ( Sec. 5.2). For each spatial layout, and for each text prompt, we
show different samples from our method.  Figure 12: Left: vertical panoramic
image ($1024\times 512$) generated by our method. Right: additional generation
results using a combination of rough and tight regions; for each layout, we
present diverse generated samples. Figure 13: Sample results from COCO
validation set by BLD, SI and our method. See more details in Sec. 5.2.
Figure 14: Additional qualitatively comparison to Make-A-Scene (Gafni et al.,
2022), Blended-Latent-Diffusion (Avrahami et al., 2022a), Spa-Text (Avrahami
et al., 2022b), and ours. See Sec. 4.2, and Sec. 5.2

##  Appendix B Additional Implementation Details.

###  B.1 Panorama (Sec. 4.1)

In the case of panorama generation, our maps $F_{i}$ are defined as fixed-size
crops from the full panorama. Specifically, for a panorama with spatial
resolution $H^{\prime}\times W^{\prime}$, we consider overlapping crops of
size $H\times W$ where $H=W=64$ defined in the Stable Diffusion latent space
(which translates to size $512\times 512$ in RGB space). Our maps
$F_{i},...,F_{n}$ provide crops with a sliding window of size
$\texttt{step}=8$ in the latent space (64 pixels in RGB space). In particular,
$n={\frac{H^{\prime}-64}{\texttt{step}}\cdot{\frac{W^{\prime}-64}{\texttt{step}}}}$.
We summarize,

Input : ;

$\Phi$ $\triangleright$ pre-trained Diffusion Model
;

$H^{\prime},W^{\prime}$ $\triangleright$ resolution of the desired panorama
;

$\\{F_{i}\\}_{i=1}^{n}$ $\triangleright$ mappings defining crops from the
panorama
;

$y$ $\triangleright$ conditioned text-prompt

$J_{T}\sim\mathcal{N}(0,I)\;\;J_{T}\in\mathcal{R}^{H^{\prime}\times
W^{\prime}\times C}$ $\triangleright$ noise initialization
for  _$t=T,...,1$_ do

;

$I_{t}^{i}\leftarrow F_{i}(J_{t})\;\;\forall i\in[n]$ $\triangleright$ take
crops from the panorama
$I_{t-1}^{i}\leftarrow\Phi(I_{t}^{i},y)\;\;\forall i\in[n]$ $\triangleright$
per-crop diffusion updates
$J_{t-1}\leftarrow\texttt{MultiDiffuser}(\left\\{I^{i}_{t-1}\right\\}_{i=1}^{n})$
$\triangleright$ Eq. 5

$\texttt{Panorama}\leftarrow\mathcal{D}(J_{0})$ $\triangleright$ Decode the
panorama to RGB space
Output : Panorama

Algorithm 2 MultiDiffusion sampling - Panorama.

Note that we can compute the per-crop diffusion updates in parallel (i.e., in
a batch), resulting in total of ${\frac{{T\cdot n}}{b}}$ calls to the
reference diffusion $\Phi$, where $b$ denotes the batch size.

###  B.2 Bootstrapping (Sec. 4.2)

In case the user desires to maintain high fidelity to tight masks (see Fig.
4), we introduce a bootstrapping phase to our maps $F_{i}$ (see Eq. 9).
Specifically, we pre-compute each $S_{t}$ as follows: we randomize an image
$I\in[0,1]^{512x512x3}$ with a random constant RGB value, and encode it to
Stable Diffusion latent space $S=\mathcal{E}(I)$, where $\mathcal{E}$ is the
pre-trained encoder provided by the Stable Diffusion framework. Finally, we
obtain $S_{t}$ by noising $S$ to the noise level of time-step $t$. That is,
$S_{t}\sim\mathcal{N}$ where ${(\mu_{t}\cdot S,\sigma_{t}^{2})}$, $\mu_{t}$
and $\sigma_{t}$ are the diffusion noise schedulers (Ho et al., 2020).