# Compositional 3D Scene Generation using Locally Conditioned Diffusion

Ryan Po
Stanford University
<EMAIL_ADDRESS>Gordon Wetzstein
Stanford University
<EMAIL_ADDRESS>
###### Abstract

Designing complex 3D scenes has been a tedious, manual process requiring
domain expertise. Emerging text-to-3D generative models show great promise for
making this task more intuitive, but existing approaches are limited to
object-level generation. We introduce locally conditioned diffusion as an
approach to compositional scene diffusion, providing control over semantic
parts using text prompts and bounding boxes while ensuring seamless
transitions between these parts. We demonstrate a score distillation
sampling–based text-to-3D synthesis pipeline that enables compositional 3D
scene generation at a higher fidelity than relevant baselines.

††Project page: http://www.ryanpo.com/comp3d

##  1 Introduction

Figure 1: Results of our method. Given user-input bounding boxes with
corresponding text prompts, our method is able to generate high-quality 3D
scenes that adhere to the desired layout with seamless transitions. Our
locally conditioned diffusion method blends multiple objects into a single
coherent scene, while simultaneously providing control over the size and
position of individual scene components. Text prompts for bottom row (from
left): (1) “a lighthouse” and (2) “a beach”; (1) “the Sydney Opera House” and
(2) “a desert”; (1) “a cherry blossom tree” and (2) “a lake”; (1) “a small
castle” and (2) “a field of red flowers”. Videos of our results can be found
in the supplementary materials.

Traditionally, 3D scene modelling has been a time-consuming process exclusive
to those with domain expertise. While a large bank of 3D assets exists in the
public domain, it is quite rare to find a 3D scene that fits the user’s exact
specifications. For this reason, 3D designers often spend hours to days
modelling individual 3D assets and composing them together into a scene. To
bridge the gap between expert 3D designers and the average person, 3D
generation should be made simple and intuitive while maintaining control over
its elements (e.g., size and position of individual objects).

Recent work on 3D generative models has made progress towards making 3D scene
modelling more accessible. 3D-aware generative adversarial networks (GANs)
[21, 50, 29, 10, 31, 23, 5, 44, 27, 19, 38, 4, 13, 54, 33, 53] have shown
promising results for 3D object synthesis, demonstrating elementary progress
towards composing generated objects into scene [32, 51]. However, GANs are
specific to an object category, limiting the diversity of results and making
scene-level text-to-3D generation challenging. In contrast, text-to-3D
generation [35, 22, 48] using diffusion models can generate 3D assets from a
wide variety of categories via text prompts [1, 39, 40]. Existing work
leverages strong 2D image diffusion priors trained on internet-scale data,
using a single text prompt to apply a global conditioning on rendered views of
a differentiable scene representation. Such methods can generate high-quality
object-centric generations but struggle to generate scenes with multiple
distinct elements. Global conditioning also limits controllability, as user
input is constrained to a single text prompt, providing no control over the
layout of the generated scene.

We introduce locally conditioned diffusion, a method for compositional text-
to-image generation using diffusion models. Taking an input segmentation mask
with corresponding text prompts, our method selectively applies conditional
diffusion steps to specified regions of the image, generating outputs that
adhere to the user specified composition. We also achieve compositional text-
to-3D scene generation by applying our method to a score distillation
sampling–based text-to-3D generation pipeline. Our proposed method takes 3D
bounding boxes and text prompts as input and generates coherent 3D scenes
while providing control over size and positioning of individual assets.
Specifically, our contributions are the following:

  * •

We introduce locally conditioned diffusion, a method that allows greater
compositional control over existing 2D diffusion models.

  * •

We introduce a method for compositional 3D synthesis by applying locally
conditioned diffusion to a score distillation sampling–based 3D generative
pipeline.

  * •

We introduce key camera pose sampling strategies, crucial for compositional 3D
generation.

##  2 Related work

#### 2D diffusion models.

Advances in large-scale 2D diffusion models trained on internet-scale data [7,
30, 39, 36, 37, 40, 43] have allowed generation of high-quality images that
stay accurate to complex text prompts. While text-conditioned diffusion models
excel at reproducing the semantics of a prompt, compositional information is
usually ignored. Variants of existing methods [39] instead condition their
models with semantic bounding boxes. This change allows greater control over
the composition of the generated image. However, bounding-box-conditioned
models must be trained with annotated image data [3]. These datasets are often
much more limited in size, which restricts the diversity of the resulting
diffusion model. Our locally conditioned diffusion approach leverages pre-
trained text-conditioned 2D diffusion models to generate high-quality images
with better compositional control without restricting the complexity of user-
provided text-prompts.

Figure 2: Overview of our method. We generate text-to-3D content using a score
distillation sampling–based pipeline. A latent diffusion prior is used to
optimize a Voxel NeRF representation of the 3D scene. The latent diffusion
prior is conditioned on a bounding box rendering of the scene, where a noise
estimation on the image is formed for every input text prompt, and denoising
steps are applied based on the segmentation mask provided by the bounding box
rendering.

#### Compositional image generation.

Recent work found that Energy-Based Models (EBMs) [8, 9, 20, 11, 12] tend to
struggle with composing multiple concepts into a single image [8, 25]. Noting
that EBMs and diffusion models are functionally similar, recent work improves
the expressivity of diffusion by borrowing theory from EBMs. For example, [25]
achieves this by composing gradients from denoisers conditioned on separate
text-prompts in a manner similar to classifier-free guidance as proposed by
[7]. Existing work, such as Composable-Diffusion [25], however, apply
composition to the entire image, offering no control over the position and
size of different concepts. Our locally conditioned diffusion approach
selectively applies denoising steps over user-defined regions, providing
increased compositional control for image synthesis while ensuring seamless
transitions.

#### Text-to-3D diffusion models.

Recent advances in 2D diffusion models have motivated a class of methods for
performing text-to-3D synthesis. Existing methods leverage 2D diffusion models
trained on internet-scale data to achieve text-to-3D synthesis. Notably,
DreamFusion [35] with Imagen [40], Score Jacobian Chaining (SJC) [48] with
StableDiffusion [39] and Magic3D [22] with eDiff-I [1] and StableDiffusion
[39]. Previous methods [35, 48, 22, 28] perform 3D synthesis by denoising
rendered views of a differentiable 3D representation. This process is coined
Score Distillation Sampling (SDS) by the authors of DreamFusion [35].
Intuitively, SDS ensures that all rendered views of the 3D representation
resemble an image generated by the text-conditioned 2D diffusion model.
Current methods are able to generate high quality 3D assets from complex text
prompts. However, they are unable to create 3D scenes with specific
compositions. Our proposed method enables explicit control over size and
position of scene components.

##  3 Diffusion preliminaries

Recent work has shown that diffusion models can achieve state-of-the-art
quality for image generation tasks [7]. Specifically, Denoising Diffusion
Probabilistic Models (DDPMs) implement image synthesis as a denoising process.
DDPMs begin from sampled Gaussian noise $x_{T}$ and apply $T$ denoising steps
to create a final image $x_{0}$. The forward diffusion process $q$ is modelled
as a Markov chain that gradually adds Gaussian noise to a ground truth image
according to a predetermined variance schedule
$\beta_{1},\beta_{2},\dots,\beta_{T}$

$q(x_{t}|x_{t-1})=\mathcal{N}\left(x_{t};\sqrt{1-\beta_{t}}x_{t-1},\beta_{t}\mathbf{I}\right)$
(1)

The goal of DDPMs is to train a diffusion model to revert the forward process.
Specifically, a function approximator $\boldsymbol{\epsilon}_{\phi}$ is
trained to predict the noise $\boldsymbol{\epsilon}$ contained in a noisy
image $x_{t}$ at step $t$. $\boldsymbol{\epsilon}_{\phi}$ is typically
represented as a convolutional neural network characterised by its parameters
$\phi$. Most successful models [7, 15, 41] train their models using a
simplified variant of the variational lower bound on the data distribution:

$\mathcal{L}_{\textrm{DDPM}}\\!=\\!\mathbb{E}_{t,x,\boldsymbol{\epsilon}}\left[\left\|\boldsymbol{\epsilon}-\boldsymbol{\epsilon}_{\phi}\left(x_{t},t\right)\right\|^{2}\right]\\!\\!$
(2)

with $t$ uniformly sampled from $\\{1,\dots,T\\}$. The resulting update step
for obtaining a sample for $x_{t-1}$ from $x_{t}$ is then

$x_{t-1}=x_{t}-\frac{1-\alpha_{t}}{\sqrt{1-\bar{\alpha}_{t}}}\epsilon_{\phi}(x_{t},t)+\frac{1-\bar{\alpha}_{t-1}}{1-\bar{\alpha}_{t}}\beta_{t}\mathcal{N}(0,\mathbf{I})$
(3)

where $\bar{\alpha}_{t}=\prod_{s=1}^{t}\alpha_{s}$, $\alpha_{t}=1-\beta_{t}$.

Text-to-image diffusion models build upon the above theory to introduce
conditional diffusion processes using classifier-free guidance [14]. Given a
condition $y$, usually represented as a text prompt, a diffusion model
$\boldsymbol{\epsilon}_{\phi}(x_{t},t,y)$ is trained to predict noise in an
image as shown in Eq. 2. During training, conditioning $y$ is randomly dropped
out, leaving the diffusion model to predict noise without it. At inference,
noise prediction is instead represented by:

$\hat{\boldsymbol{\epsilon}}_{\phi}(x_{t},t,y)=\boldsymbol{\epsilon}_{\phi}(x_{t},t,\emptyset)+s\Bigl{(}\boldsymbol{\epsilon}_{\phi}(x_{t},t,y)-\boldsymbol{\epsilon}_{\phi}(x_{t},t,\emptyset)\Bigr{)}$
(4)

Where $s$ is a user-defined constant controlling the degree of guidance and
$\boldsymbol{\epsilon}(x_{t},t,\emptyset)$ represents the noise prediction
without conditioning.

Figure 3: 2D locally conditioned diffusion results. Given coarse segmentation
masks as input, our method is able to generate images that follow the
specified layout while ensuring seamless transitions. Results in the first row
are generated using GLIDE [30], while the second and third rows show results
generated using StableDiffusion [39]. Figure 4: Baseline comparisons. Left to
right: (i) SJC results using a single text prompt, (ii) SJC generating each
scene component independently, (iii) SJC combined with Composable-Diffusion
[25], and (iv) our method with corresponding bounding boxes and text prompts.
Generations for each row use the text prompts listed on the right. Results in
the first column are generated by combining individual text prompts with the
connecting phrase “in the middle of”, e.g. “a lighthouse in the middle of a
beach”. Our method successfully composes different objects into a coherent
scene while following the user input bounding boxes.

##  4 Locally conditioned diffusion

We introduce locally conditioned diffusion as a method for providing better
control over the composition of images generated by text-conditioned diffusion
models. The key insight of our method is that we can selectively apply
denoising steps conditioned on different text prompts to specific regions of
an image.

Given a set of text prompts $\\{y_{1},\dots,y_{P}\\}$, classifier-free
guidance [14] provides a method for predicting denoising steps conditioned on
$y_{i}$:

$\hat{\boldsymbol{\epsilon}}_{\phi}(x_{t},t,y_{i})=\boldsymbol{\epsilon}_{\phi}(x_{t},t,\emptyset)+s\Bigl{(}\boldsymbol{\epsilon}_{\phi}(x_{t},t,y_{i})-\boldsymbol{\epsilon}_{\phi}(x_{t},t,\emptyset)\Bigr{)}$
(5)

Using a user-defined semantic segmentation mask $m$, where each pixel $m[j]$
has integer value $[1,P]$, the overall noise prediction can then be
represented by selectively applying noise predictions to each labelled image
patch:

$\hat{\boldsymbol{\epsilon}}_{\phi}(x_{t},t,y_{1:P},m)=\sum_{i=1}^{P}\mathds{1}_{i}(m)\odot\hat{\boldsymbol{\epsilon}}_{\phi}(x_{t},t,y_{i})$
(6)

Where $\mathds{1}_{i}(m)$ is the indicator image with equivalent
dimensionality as $m$ and

$\mathds{1}_{i}(m)[j]=\begin{cases}1,&\text{if }m[j]=i\\\
0,&\text{otherwise}\end{cases}$ (7)

The proposed locally conditioned diffusion method is summarized in Algorithm
1.

Algorithm 1 Locally conditioned diffusion

Diffusion models $\hat{\boldsymbol{\epsilon}}_{\phi}(x_{t},t,y_{i})$, guidance
scale $s$, semantic mask $m$

$x_{T}\sim\mathcal{N}(0,I)$ $\triangleright$ Initialize Gaussian noise image

for $t=T,\dots,1$ do

$\epsilon_{i}\leftarrow\hat{\boldsymbol{\epsilon}}_{\phi}(x_{t},t,y_{i})$
$\triangleright$ Individual noise predictions

$\epsilon\leftarrow\hat{\boldsymbol{\epsilon}}_{\phi}(x_{t},t,\emptyset)$
$\triangleright$ Unconditional noise prediction

$\epsilon_{\textrm{sem}}\leftarrow\sum_{i=1}^{P}\mathds{1}_{i}(m)\odot
s(\epsilon_{i}-\epsilon)$ $\triangleright$ Combine noise predictions

$x_{t-1}=\textrm{Update}(x_{t},\epsilon_{\textrm{sem}})$ $\triangleright$
Apply denoising step

end for

Although a large proportion of noise predictions are not used, in practice
only one diffusion model $\boldsymbol{\epsilon}_{\phi}$ is queried. All calls
to the model for each unique text-conditioning $y_{i}$ can be batched together
for increased efficiency.

Our locally conditioned diffusion method generates high-fidelity 2D images
that adhere to the given semantic segmentation masks. Note that, while each
segment of the image is locally conditioned, there are no visible seams in the
resulting image and transitions between differently labelled regions are
smooth, as shown in Fig. 3 (see Sec. 6 for more details).

##  5 Compositional 3D synthesis

To make compositional text-to-3D synthesis as simple as possible, our method
takes 3D bounding boxes with corresponding text prompts as input. The goal of
our method is to generate 3D scenes that contain objects specified by the text
prompts while adhering to the specific composition provided by the input
bounding boxes. In this section, we describe our method and how we apply
locally conditioned diffusion in 2D to enable controllable generation in 3D.

#### Text-to-3D with Score Distillation Sampling.

Our method builds off existing SDS-based text-to-3D methods [22, 35, 48]. SDS-
based methods leverage a 3D scene representation parameterized by $\theta$ is
differentiably rendered at a sampled camera pose, generating a noised image
$g(\theta)$ which is passed into an image diffusion prior. Our method builds
off SJC [48], therefore we follow their pipeline, using a Voxel NeRF [6, 24,
45, 47, 52] representation and a volumetric renderer. The image diffusion
prior provides the gradient direction to update scene parameters $\theta$.

$\nabla_{\theta}\mathcal{L}_{\textrm{SDS}}(\phi,g(\theta))=\mathbb{E}_{t,\epsilon}\Bigl{[}w(t)(\hat{\epsilon}(x_{t},y,t)-\epsilon)\frac{\partial
x}{\partial\theta}\Bigr{]}$ (8)

This process is repeated for randomly sampled camera poses, as the text-
conditioned image diffusion prior pushes each rendered image towards high
density regions in the data distribution. Intuitively, SDS ensures images
rendered from all camera poses resembles an image generated by the text-
conditioned diffusion prior.

#### Bounding-box-guided text-to-3D synthesis.

To achieve text-to-3D scene generations that adhere to user input bounding
boxes, our method takes the standard SDS-based pipeline and conditions the
image diffusion prior with renderings of the input bounding boxes.
Specifically, our method works as follows. First, a random camera pose is
sampled and a volume rendering of the 3D scene model is generated, we call
this image $x_{t}$. Using the same camera pose, a rendering of the bounding
boxes is also generated, we call this image $m$. This image is a segmentation
mask, where each pixel contains an integer value corresponding to a user input
text prompt. The volume rendering is then passed in to the image diffusion
prior which provides the necessary gradients for optimizing the 3D scene
representation. However, instead of conditioning the image diffusion prior on
a single text prompt, we generate denoising steps for all text prompts with
corresponding bounding boxes visible from the sampled camera pose. We then
selectively apply these denoising steps to the image based on the segmentation
mask $m$, and backpropagate the gradients to the 3D scene as usual. This is
equivalent to applying the noise estimator described in Eq. 6 to the SDS
gradient updates.

$\nabla_{\theta}\mathcal{L}_{\textrm{SDS}}(\phi,g(\theta),m)=\mathbb{E}_{t,\epsilon}\Bigl{[}w(t)(\hat{\boldsymbol{\epsilon}}_{\phi}(x_{t},t,y_{1:P},m)-\epsilon)\frac{\partial
x}{\partial\theta}\Bigr{]}$ (9)

While previous SDS-based text-to-3D methods ensure all rendered views of the
3D scene lie in the high probability density regions in the image prior
conditioned on a single text prompt, our method ensures that all rendered
views also align with the rendered bounding box segmentation masks. An
overview of our method is provided in Fig. 2.

#### Object-centric camera pose sampling.

As discussed in prior work [35, 22, 48], high classifier-free guidance weights
are crucial for SDS methods to work. While image generation methods typically
use guidance weights in the range of [2, 50], methods such as DreamFusion use
guidance weights up to 100 [35]. Using a high guidance scale leads to mode-
seeking properties which is desirable in the context of SDS-based generation.
However, mode-seeking properties in image diffusion priors have the tendency
of generating images with the object at the center of the image. When applying
high guidance weights to locally conditioned diffusion, it is possible for the
resulting image to ignore semantic regions that are off center, since mode-
seeking behaviour of the diffusion model expects the object described by the
text prompt to be at the center of the image, while the semantic mask only
applies gradients from off-centered regions. In the context of our method,
this mode-seeking behavior causes off-centered bounding box regions to become
empty.

We combat this effect using object-centric camera pose sampling. While
existing works [35, 22, 48] sample camera poses that are always pointed at the
origin of the 3D scene model, in our method, we randomly sample camera poses
that point at the center of each object bounding box instead. This means that
during optimization of the 3D scene, each bounding box region will have the
chance at appearing at the center of the image diffusion prior.

#### Locally conditioned diffusion with latent diffusion models.

Existing SDS-based methods, such as DreamFusion [35] and Magic3D [22],
leverage image diffusion priors in their method 111In Magic3D, a latent
diffusion prior is also used, but the gradient of the encoder in the latent
diffusion model is provided to convert gradient updates in the latent space
back to the image space.. While SJC [48] uses a very similar methodology,
their method actual employs a latent diffusion prior in the form of
StableDiffusion [39]. Therefore, volume renderings of the 3D scene lies in the
latent space instead of the image space. Note that previous work [34] has
shown that the latent space is essentially a downsampled version of the image
space, meaning we are still able to apply locally conditioned diffusion to the
latent space.

Figure 5: Size and position control. Our method provides size and position
control of individual scene components through user-defined bounding boxes.
Our method provides fine-grained control over scene composition while ensuring
each components blends seamlessly into the overall scene.

##  6 Experiments

We show qualitative results on compositional text-to-2D and text-to-3D
generation. For 3D results, we mainly compare against SJC [48] as it is the
best-performing publicly-available text-to-3D method. We also implemented a
version of SJC that leverages Composable-Diffusion [25] as an additional
baseline.

###  6.1 Compositional 2D results

#### Implementation details.

We apply our locally conditioned diffusion method to existing text-conditioned
diffusion models: GLIDE [30] and StableDiffusion [39]. We use pre-trained
models provided by the authors of each respective paper to implement locally
conditioned diffusion. Each image sample takes 10–15 seconds to generate on an
NVIDIA A100 GPU, where duration varies according to the number of distinct
semantic regions provided. Note that sampling time increases sub-linearly with
respect to number of regions/prompts, this is because calls to the same model
for each text-conditioning can be done in a single batch.

#### Qualitative results.

We provide qualitative examples in Fig. 3. Our method is able to generate
high-fidelity images that adhere to the input semantic masks and text prompts.
Note that our method does not aim at generating images that follow the exact
boundaries of the input semantic masks, instead it strives to achieve seamless
transitions between different semantic regions. A key advantage of locally
conditioned diffusion is that it is agnostic to the network architecture. We
demonstrate this by showing that our method works on two popular text-to-image
diffusion models GLIDE [30] and StableDiffusion [39].

###  6.2 Compositional 3D results

#### Implementation details.

Our compositional text-to-3D method builds upon the SJC [48] codebase.
Following SJC, we use a Voxel NeRF to represent the 3D scene model and
StableDiffusion [39] as the diffusion prior for SDS-based generation. The
Voxel NeRF representing the 3D scene is set to a resolution of $100^{3}$. This
configuration uses $\approx 10$ GB of GPU memory. The original SJC method uses
an emptiness loss scheduler to improve the quality of generated scenes. Our
method also leverages this emptiness loss; please refer to the original SJC
[48] for more details.

#### Qualitative results.

We provide qualitative examples of compositional text-to-3D generations with
bounding box guidance in Fig. 1. Notice that our method is able to generate
coherent 3D scenes using simple bounding boxes with corresponding text
prompts. Our method generates results that adhere to the input bounding boxes,
allowing users to edit the size and position of individual scene components
before generation. Fig. 5 shows generated results of the same scene prompts
with differing bounding box sizes and positions. Note that our method is able
to adapt to the user’s input and generate scenes with varying compositions.

#### Baseline comparisons.

We compare our method to different variants of SJC [48]. Namely, (i) SJC
generations using a single prompt for the entire scene, (ii) individual SJC
generations for each scene component, and (iii) an implementation of
Composable-Diffusion [25] combined with SJC. Although DreamFusion [35] and
Magic3D [22] have also shown to generate high-quality results, both models
leverage image diffusion priors (Imagen [40] and eDiff-I [1]) that are not
publicly available. However, it is important to note that our method can
theoretically be applied to any SDS-based method. This can be achieved by
replacing the image diffusion model in DreamFusion [35] and Magic3D [22] with
the locally conditioned method described above.

We provide qualitative results for our method and each baseline in Fig. 4. In
our experiments we attempt to compose two scene components into a coherent
scene. Specifically, we choose an object-centric prompt that describes
individual objects, paired with a scene-centric prompt that describes a
background or an environment.

We observe that SJC fails to capture certain scene components when composing
multiple scene components into a single prompt. Our method is able to capture
individual scene components while blending them seamlessly into a coherent
scene.

For object-centric prompts, SJC is able to create high-quality 3D generations.
However, scene-centric prompts such as “a desert” or “a beach” end up
generating dense volumes that resemble the text-prompt when rendered from
different angles, but fail to reconstruct reasonable geometry. By defining
bounding boxes for each scene component, our method provides coarse guidance
for the geometry of the scene, this helps generate results with fewer
“floater” artifacts.

Figure 6: Seamless transitions. Our method is able to smoothly transition
between scene components in different bounding boxes. In this example, we can
see the reflection of the cherry blossom tree in the lake.

One option for compositional scene generation is to generate each scene
component individually and then combine them manually afterwards. However,
blending scene components together in a seamless manner takes considerable
effort. Our method is able to blend individual objects with scene-level
detail. As shown in Fig. 6, although the cherry blossom tree and the
reflective purple lake correspond to different bounding box regions, our
method is able to generate reflections of the tree in the water. Such effects
would not be present if each scene component were generated individually and
then manually combined.

We also compare our method to a composable implementation of SJC using
Composable-Diffusion [25]. However, this method fails to generate reasonable
3D scenes.

#### Quantitative results.

Following prior work [35, 16], we evaluate the CLIP R-Precision, the accuracy
of retrieving the correct input caption from a set of distractors using CLIP
[36], of our compositional method. Tab. 1 reports CLIP R-Precision values for
rendered views of scenes shown in Fig. 4 using our compositional method and
SJC with a single prompt. Our method outperforms the baseline across all
evaluation methods.

Table 1: CLIP R-Precision comparisons. | R-Precision $\uparrow$
---|---
Method | B/32 | B/16 | L/14
Single Prompt (SJC) | 27..8 | 31.5 | 28.53
Composed (Ours) | 38.6 | 54.3 | 29.8

#### Ablations.

We found that object-centric camera pose sampling is essential for successful
composition of multiple scene components. This is especially true for bounding
boxes further away from the origin. We compare generations with and without
object-centric pose sampling in Fig. 7. Note that our method tends to ignore
certain scene components without object-centric sampling.

Figure 7: Ablation over object-centric sampling. Without object-centric
sampling, our method fails fully capture off-centered scene components.

#### Speed evaluation.

Unless stated otherwise, all results were generated by running our method for
10000 denoising iterations with a learning rate of 0.05 on a single NVIDIA RTX
A6000. Note that scenes with a higher number of distinct text prompts require
a longer period of time to generate. Using SJC, generating scene components
individually causes generation time to scale linearly with number of prompts.
In contrast, our method can compose the same number of prompts in a shorter
amount of time, as calls to the same diffusion prior conditioned on different
text-prompts can be batched together. Table 2 shows generation times for SJC
and our method for 3000 denoising iterations.

Table 2: Generation times using SJC [48] for individual prompts and composing multiple prompts using our method. |  | # of prompts |
---|---|---|---
Method | 1 | 2 | 3
Individual (SJC) | 8 mins | 16 mins | 24 mins
Composed (Ours) | 8 mins | 12 mins | 15 mins

##  7 Discussion and Conclusions

Creating coherent 3D scenes is a challenging task that requires 3D design
expertise and plenty of manual labor. Our method introduces a basic interface
for creating 3D scenes without any knowledge of 3D design. Simply define
bounding boxes for the desired scene components and fill in text prompts for
what to generate in those regions.

#### Limitations and future work.

Although text-to-3D methods using SDS [35, 48, 22] have shown promising
results, speed is still a limiting factor. While advances in image-diffusion-
model sampling [26, 18, 42, 49, 46] have enabled the generation of high-
quality results in dozens of denoising steps, SDS method still require
thousands of iterations before a 3D scene can be learned. SDS-based methods
are also limited by their reliance on unusually high guidance scales [35]. A
high guidance scale promotes mode-seeking, but leads to low diversity in the
generated results. Concurrent works [2, 17] have shown other methods for
controlling text-to-image diffusion synthesis with coarse segmentation masks.
However, these methods require running a diffusion prior on multiple image
patches before forming a single image, greatly increasing time needed to
generate a single denoising step. In theory, these works could be applied in
combination with our method, albeit with greatly increased time needed to
generate a single 3D scene.

#### Ethical considerations.

Generative models, such as ours, can potentially be used for spreading
disinformation. Such misuses pose a societal threat and the authors of this
paper do not condone such behavior. Since our method leverages StableDiffusion
[39] as an image prior, it may also inherit any biases and limitations found
in the 2D diffusion model.

#### Conclusion.

Text-to-3D synthesis has recently seen promising advances. However, these
methods mostly specialize in object-centric generations. Our method is an
exciting step forward for 3D scene generation. Designing a 3D scene with
multiple components no longer requires 3D modeling expertise. Instead, by
defining a few bounding boxes and text prompts, our method can generate
coherent 3D scenes that fit the input specifications.

## Acknowledgements

We thank Alex Bergman and Cindy Nguyen for valuable discussions and feedback
on drafts. This project was in part supported by the Samsung GRO program. Ryan
Po was supported by a Stanford Graduate Fellowship.

## References

  * [1] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, Tero Karras, and Ming-Yu Liu.  ediff-i: Text-to-image diffusion models with an ensemble of expert denoisers.  ArXiv, abs/2211.01324, 2022.
  * [2] Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel.  Multidiffusion: Fusing diffusion paths for controlled image generation.  ArXiv, abs/2302.08113, 2023.
  * [3] Holger Caesar, Jasper R. R. Uijlings, and Vittorio Ferrari.  Coco-stuff: Thing and stuff classes in context.  2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1209–1218, 2016.
  * [4] Eric R. Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J. Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, and Gordon Wetzstein.  Efficient geometry-aware 3d generative adversarial networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16123–16133, June 2022.
  * [5] Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein.  pi-GAN: Periodic implicit generative adversarial networks for 3D-aware image synthesis.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [6] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su.  Tensorf: Tensorial radiance fields.  In European Conference on Computer Vision, 2022.
  * [7] Prafulla Dhariwal and Alex Nichol.  Diffusion models beat gans on image synthesis.  ArXiv, abs/2105.05233, 2021.
  * [8] Yilun Du, Shuang Li, and Igor Mordatch.  Compositional visual generation with energy based models.  In Neural Information Processing Systems, 2020.
  * [9] Yilun Du and Igor Mordatch.  Implicit generation and generalization in energy-based models.  ArXiv, abs/1903.08689, 2019.
  * [10] Matheus Gadelha, Subhransu Maji, and Rui Wang.  3d shape induction from 2d views of multiple objects.  In 2017 International Conference on 3D Vision (3DV), pages 402–411, 2017.
  * [11] Ruiqi Gao, Yang Song, Ben Poole, Ying Nian Wu, and Diederik P. Kingma.  Learning energy-based models by diffusion recovery likelihood.  ArXiv, abs/2012.08125, 2020.
  * [12] Will Grathwohl, Kuan-Chieh Wang, Jörn-Henrik Jacobsen, David Kristjanson Duvenaud, and Richard S. Zemel.  Learning the stein discrepancy for training and evaluating energy-based models without sampling.  In International Conference on Machine Learning, 2020.
  * [13] Jiatao Gu, Lingjie Liu, Peng Wang, and Christian Theobalt.  StyleNeRF: A style-based 3D-aware generator for high-resolution image synthesis.  arXiv preprint arXiv:2110.08985, 2021.
  * [14] Jonathan Ho.  Classifier-free diffusion guidance.  ArXiv, abs/2207.12598, 2022.
  * [15] Jonathan Ho, Ajay Jain, and P. Abbeel.  Denoising diffusion probabilistic models.  ArXiv, abs/2006.11239, 2020.
  * [16] Ajay Jain, Ben Mildenhall, Jonathan T. Barron, P. Abbeel, and Ben Poole.  Zero-shot text-guided object generation with dream fields.  2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 857–866, 2021.
  * [17] Álvaro Barbero Jiménez.  Mixture of diffusers for scene composition and high resolution image generation.  ArXiv, abs/2302.02412, 2023.
  * [18] Zhifeng Kong and Wei Ping.  On fast sampling of diffusion probabilistic models.  ArXiv, abs/2106.00132, 2021.
  * [19] Adam R Kosiorek, Heiko Strathmann, Daniel Zoran, Pol Moreno, Rosalia Schneider, Sona Mokra, and Danilo Jimenez Rezende.  Nerf-vae: A geometry aware 3d scene generative model.  In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 5742–5752. PMLR, 18–24 Jul 2021.
  * [20] Yann LeCun, Sumit Chopra, Raia Hadsell, Aurelio Ranzato, and Fu Jie Huang.  A tutorial on energy-based learning.  2006\.
  * [21] Yiyi Liao, Katja Schwarz, Lars Mescheder, and Andreas Geiger.  Towards unsupervised learning of generative models for 3d controllable image synthesis.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [22] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin.  Magic3d: High-resolution text-to-3d content creation.  ArXiv, abs/2211.10440, 2022.
  * [23] Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt.  Neural sparse voxel fields.  In Advances in Neural Information Processing Systems (NeurIPS), 2020\.
  * [24] Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt.  Neural sparse voxel fields.  ArXiv, abs/2007.11571, 2020.
  * [25] Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B. Tenenbaum.  Compositional visual generation with composable diffusion models.  ArXiv, abs/2206.01714, 2022.
  * [26] Zhaoyang Lyu, Xu Xudong, Ceyuan Yang, Dahua Lin, and Bo Dai.  Accelerating diffusion models via early stop of the diffusion process.  ArXiv, abs/2205.12524, 2022.
  * [27] Quan Meng, Anpei Chen, Haimin Luo, Minye Wu, Hao Su, Lan Xu, Xuming He, and Jingyi Yu.  Gnerf: Gan-based neural radiance field without posed camera.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 6351–6361, October 2021.
  * [28] Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or.  Latent-nerf for shape-guided generation of 3d shapes and textures.  ArXiv, abs/2211.07600, 2022.
  * [29] Thu Nguyen-Phuoc, Chuan Li, Lucas Theis, Christian Richardt, and Yong-Liang Yang.  Hologan: Unsupervised learning of 3d representations from natural images.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
  * [30] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  In International Conference on Machine Learning, 2021.
  * [31] Michael Niemeyer and Andreas Geiger.  Giraffe: Representing scenes as compositional generative neural feature fields.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11453–11464, June 2021.
  * [32] Michael Niemeyer and Andreas Geiger.  GIRAFFE: Representing scenes as compositional generative neural feature fields.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [33] Roy Or-El, Xuan Luo, Mengyi Shan, Eli Shechtman, Jeong Joon Park, and Ira Kemelmacher-Shlizerman.  Stylesdf: High-resolution 3d-consistent image and geometry generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13503–13513, June 2022.
  * [34] Dong Huk Park, Grace Luo, C. M. Toste, Samaneh Azadi, Xihui Liu, Maka Karalashvili, Anna Rohrbach, and Trevor Darrell.  Shape-guided diffusion with inside-outside attention.  ArXiv, abs/2212.00210, 2022.
  * [35] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall.  Dreamfusion: Text-to-3d using 2d diffusion.  ArXiv, abs/2209.14988, 2022.
  * [36] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In International Conference on Machine Learning, 2021.
  * [37] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  ArXiv, abs/2204.06125, 2022.
  * [38] Daniel Rebain, Mark Matthews, Kwang Moo Yi, Dmitry Lagun, and Andrea Tagliasacchi.  Lolnerf: Learn from one look.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1558–1567, June 2022.
  * [39] Robin Rombach, A. Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10674–10685, 2021.
  * [40] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, Seyedeh Sara Mahdavi, Raphael Gontijo Lopes, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi.  Photorealistic text-to-image diffusion models with deep language understanding.  ArXiv, abs/2205.11487, 2022.
  * [41] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J. Fleet, and Mohammad Norouzi.  Image super-resolution via iterative refinement.  IEEE transactions on pattern analysis and machine intelligence, PP, 2021.
  * [42] Tim Salimans and Jonathan Ho.  Progressive distillation for fast sampling of diffusion models.  ArXiv, abs/2202.00512, 2022.
  * [43] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev.  Laion-5b: An open large-scale dataset for training next generation image-text models.  ArXiv, abs/2210.08402, 2022.
  * [44] Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger.  GRAF: Generative radiance fields for 3D-aware image synthesis.  In Advances in Neural Information Processing Systems (NeurIPS), 2020\.
  * [45] Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, and Michael Zollhöfer.  Deepvoxels: Learning persistent 3d feature embeddings.  2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2432–2441, 2018.
  * [46] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  ArXiv, abs/2010.02502, 2020.
  * [47] Cheng Sun, Min Sun, and Hwann-Tzong Chen.  Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction.  2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5449–5459, 2021.
  * [48] Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A. Yeh, and Greg Shakhnarovich.  Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation.  ArXiv, abs/2212.00774, 2022.
  * [49] Daniel Watson, Jonathan Ho, Mohammad Norouzi, and William Chan.  Learning to efficiently sample from diffusion probabilistic models.  ArXiv, abs/2106.03802, 2021.
  * [50] Jiajun Wu, Chengkai Zhang, Tianfan Xue, Bill Freeman, and Josh Tenenbaum.  Learning a probabilistic latent space of object shapes via 3d generative-adversarial modeling.  In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
  * [51] Yinghao Xu, Menglei Chai, Zifan Shi, Sida Peng, Ivan Skorokhodov, Aliaksandr Siarohin, Ceyuan Yang, Yujun Shen, Hsin-Ying Lee, Bolei Zhou, and S. Tulyakov.  Discoscene: Spatially disentangled generative radiance fields for controllable 3d-aware scene synthesis.  ArXiv, abs/2212.11984, 2022.
  * [52] Alex Yu, Sara Fridovich-Keil, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa.  Plenoxels: Radiance fields without neural networks.  2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5491–5500, 2021.
  * [53] Xin-Yang Zheng, Yang Liu, Peng-Shuai Wang, and Xin Tong.  Sdf-stylegan: Implicit sdf-based stylegan for 3d shape generation.  CoRR, abs/2206.12055, 2022.
  * [54] Peng Zhou, Lingxi Xie, Bingbing Ni, and Qi Tian.  CIPS-3D: A 3D-Aware Generator of GANs Based on Conditionally-Independent Pixel Synthesis.  arXiv preprint arXiv:2110.09788, 2021.


$\left(\Sigma_{X,X}^{e}\right)_{e\in\mathcal{E}_{\text{train}}}$.

By transversality, we know that $W$ cannot intersect $\text{LR}(m,d,k)$ if
$\dim(W)+\dim\left(\text{LR}(m,d,k)\right)-\dim\left(\mathbb{R}^{m\times
d}\right)<0$. By a dimensional argument (see [Lee03], example 5.30), it
follows that
$\operatorname{codim}(\text{LR}(m,d,k))=\dim\left(\mathbb{R}^{m\times
d}\right)-\dim\left(\text{LR}(m,d,k)\right)=(m-k)(d-k)$. Therefore, if $k<d-r$
and $m>\frac{d}{r}+d-r$, it follows that

$\displaystyle\dim(W)+\dim\left(\text{LR}(m,d,k)\right)-\dim\left(\mathbb{R}^{m\times
d}\right)$
$\displaystyle=\dim(W)-\operatorname{codim}\left(\text{LR}(m,d,k)\right)$
$\displaystyle\leq d-(m-k)(d-k)$ $\displaystyle\leq d-(m-(d-r))(d-(d-r))$
$\displaystyle=d-r(m-d+r)$
$\displaystyle<d-r\left(\left(\frac{d}{r}+d-r\right)-d+r\right)$
$\displaystyle=d-d=0.$

Therefore, $W\cap\text{LR}(m,d,k)=\emptyset$ under these conditions, finishing
the proof. ∎

###  B.8 Proof of Theorem A.3

###### Proof.

We assume that $\tilde{\Phi}\in\mathcal{F}$ leads to an invariant predictor
$(w\circ\tilde{\Phi})$ across $\mathcal{E}_{\text{train}}$, with $w$ linear,
and hence we denote $v\in\mathbb{R}^{a}$. Using the same notation as in
Appendix A, for any $\Phi$ we denote $c^{e}(\Phi)\in\mathbb{R}^{p}$ as the
optimal linear least squares regressor on top of $\Phi$ on environment
$e\in\mathcal{E}_{\text{train}}$ minus $w$. Furthermore, we denote by
$F(\Phi)\in\mathbb{R}^{p|\mathcal{E}_{\text{train}}|}$ the concatenation of
these $c^{e}(\Phi)$. Since $(w\circ\tilde{\Phi})$ is an invariant predictor
across $\mathcal{E}_{\text{train}}$, we have that $F(\tilde{\Phi})=0$. Since
$F$ is defined from a manifold of dimension $q$ to
$\mathbb{R}^{p|\mathcal{E}_{\text{train}}|}$, with
$\frac{q}{p}<|\mathcal{E}_{\text{train}}|$, the problem $F(\Phi)=0$ is
strictly overdetermined. In combinatioin with $\nabla_{\Phi}F(\Phi)$ being
full rank, this means that the set of $\Phi$ such that $F(\Phi)=0$ is at most
a set of isolated points, and by Appendix A, this means the solution is
unique. Since both $\tilde{\Phi}$ and $\Phi^{*}$ are solutions, this means
$\tilde{\Phi}=\Phi^{*}$, finishing the proof. ∎

##  Appendix C Failure Cases of Domain Adaptation

Domain adaptation [Ben+07] considers labeled data from a source environment
$e_{\text{train}}$ and unlabeled data from a target environment
$e_{\text{test}}$ with the goal of training a classifier that works well on
$e_{\text{test}}$. Many domain adaptation techniques, including the popular
Adversarial Domain Adaptation [Gan+16, ADA], proceed by learning a feature
representation $\Phi$ such that (i) the input marginals
$P(\Phi(X^{e_{\text{train}}}))=P(\Phi(X^{e_{\text{test}}}))$, and (ii) the
classifier $w$ on top of $\Phi$ predicts well the labeled data from
$e_{\text{train}}$. Thus, are domain adaptation techniques applicable to
finding invariances across multiple environments?

One shall proceed cautiously, as there are important caveats. For instance,
consider a binary classification problem, where the only difference between
environments is that $P(Y^{e_{\text{train}}}=1)=\frac{1}{2}$, but
$P(Y^{e_{\text{test}}}=1)=\frac{9}{10}$. Using these data and the domain
adaptation recipe outlined above, we build a classifier $w\circ\Phi$. Since
domain adaptation enforces
$P(\Phi(X^{e_{\text{train}}}))=P(\Phi(X^{e_{\text{test}}}))$, it consequently
enforces $P(\hat{Y}^{e_{\text{train}}})=P(\hat{Y}^{e_{\text{test}}})$, where
$\hat{Y}^{e}=w(\Phi(X^{e}))$, for all
$e\in\\{e_{\text{train}},e_{\text{test}}\\}$. Then, the classification
accuracy will be at most 20%. This is worse than random guessing, in a problem
where simply training on the source domain leads to a classifier that
generalizes to the target domain.

Following on this example, we could think of applying conditional domain
adaptation techniques [Li+18, C-ADA]. These enforce one invariance
$P(\Phi(X^{e})|Y^{e})=P(\Phi(X^{e^{\prime}})|Y^{e^{\prime}})$ per value of
$Y^{e}$, and all pairs $e,e^{\prime}\in\mathcal{E}_{\text{train}}$. Using
Bayes rule, it follows that C-ADA enforces a stronger condition than invariant
prediction when $P(Y^{e})=P(Y^{e^{\prime}})$. However, there are general
problems where the invariant predictor cannot be identified by C-ADA.

To see this, consider a discrete input feature $X^{e}\sim P(X^{e})$, and a
binary target $Y^{e}=F(X^{e})\oplus\text{Bernoulli}(p)$. This model represents
a generic binary classification problem with label noise. Since the
distribution $P(X^{e})$ is the only moving part across environments, the
trivial representation $\Phi(x)=x$ elicits an invariant prediction rule.
Assuming that the discrete variable $X^{e}$ takes $n$ values, we can summarize
$P(X^{e})$ as the probability $n$-vector $p^{x,e}$. Then, $\Phi(X^{e})$ is
also discrete, and we can summarize its distribution as the probability vector
$p^{\phi,e}=A_{\phi}p^{x,e}$, where $A_{\phi}$ is a matrix of zeros and ones.
By Bayes rule,

$\displaystyle\pi^{\phi,e}:=P(\Phi(X^{e})|Y^{e}=1)=\frac{P(Y^{e}=1|\Phi(X^{e}))\odot
p^{\phi,e}}{\langle
P(Y^{e}=1|\Phi(X^{e})),p^{\phi,e}\rangle}=\frac{\left(A_{\Phi}\left(v\odot
p^{x,e}\right)\right)\odot(A_{\Phi}p^{x,e})}{\langle\left(A_{\Phi}\left(v\odot
p_{x,e}\right)\right),A_{\Phi}p^{x,e}\rangle},$

where $\odot$ is the entry-wise multiplication, $\langle,\rangle$ is the dot
product, and $v:=P(Y^{e}=1|X^{e})$ does not depend on $e$. Unfortunately for
C-ADA, it can be shown that the set
$\Pi_{\phi}:=\\{(p^{x,e},p^{x,e^{\prime}}):\pi^{\phi,e}=\pi^{\phi,e^{\prime}}\\}$
has measure zero. Since the union of sets with zero measure has zero measure,
and there exists only a finite amount of possible $A_{\phi}$, the set
$\Pi_{\phi}$ has measure zero for _any_ $\Phi$. In conclusion and almost
surely, C-ADA disregards any non-zero data representation eliciting an
invariant prediction rule, regardless of the fact that the trivial
representation $\Phi(x)=x$ achieves such goal.

As a general remark, domain adaptation is often justified using the bound
[Ben+07]:

$\text{Error}^{e_{\text{test}}}(w\circ\Phi)\leq\text{Error}^{e_{\text{train}}}(w\circ\Phi)+\text{Distance}(\Phi(X^{e_{\text{train}}}),\Phi(X^{e_{\text{test}}}))+\lambda^{\star}.$

Here, $\lambda^{\star}$ is the error of the optimal classifier in our
hypothesis class, operating on top of $\Phi$, summed over the two domains.
Crucially, $\lambda^{\star}$ is often disregarded as a constant, justifying
the DA goals (i, ii) outlined above. But, $\lambda^{\star}$ depends on the
data representation $\Phi$, instantiating a third trade-off that it is often
ignored. For a more in depth analysis of this issue, we recommend [JSR19].

##  Appendix D Meta-Learning

Meta-learning is at the moment of writing an umbrella term encompassing many
things: using data to learn an optimizer [And+16] or architectures [ZL17] for
future learning, generalizing to new classes other than those seen during
training [SSZ17], self-modifying code [SZS98], increasing speed of adaptation
to a new dataset from the same task [FAL17], and many other topics [TP98].

The philosophy generally attached to meta-learning is particularly appealing:
phrasing the above OOD problems as variants of assuming that the training and
test distributions come from a meta-distribution, and try to reuse as much as
possible the technical machinery that works so well in i.i.d. problems [Sil18,
FAL17]. Furthermore, in the last few decades models with less inductive bias
and feature engineering have overwhelmingly surpassed the performance of more
hand-crafted approaches in complicated sensory tasks, with variants of neural
networks now being state of the art in an increasingly large family of
language, vision, and speech problems. Therefore, could using less inductive
bias about the distributions at hand not only make things easier by reusing
the i.i.d. machinery, but ultimately increase performance?

There is no yes or no answer to this question. As we have seen, different OOD
tasks have different structure. ERM and other approaches make sense in some
OOD tasks, and catastrophically fail in others. However, it is our conjecture
that for an important class of AI related problems, the meta-learning dream of
simply reusing the i.i.d. machinery is futile, and in this chapter we try to
elaborate on why.

###  D.1 A Bayesian Meta-Learner

In order to understand how meta-learning and other ideas may fall short, it is
quite useful to see what the ‘right thing’ to do is, and quantitatively study
how we deviate from that. We can describe a version of the right thing by
proceeding in a fully Bayesian way. Obviously, the output of this Bayesian
derivation will be completely intractable from a computational point of view,
so from an algorithmic standpoint it will be useless, but from an
understanding viewpoint it will help us see precisely how we are deviating
from the Bayes-optimal solution.

We utilize the notation and setup of [FAL17] slightly rewritten, which for
completeness we now detail. An important assumption that we’ll be dealing
with, following from that of [FAL17], is that the different training and test
environments come i.i.d from a meta-distribution $e\sim\mathscr{P}(e)$. This
is obviously a strong limitation of meta-learning for many real life settings,
since it is likely that new environments are not independent from previous
ones. For instance, in a nonstationary environment such as an agent observing
new states, the previous environments will lead to actions that will impact
the new states the robot will be in, and hence the new environments. As well,
previous users’ data will alter the decisions taken by services like
recomendation systems or search engines, and hence alter the future users’
reactions. Nonetheless, we need a starting point for this technical
discussion, and this disclaimer aside we could imagine problems where this
assumption is closer to the truth. For instance, we could obtain datasets from
different hospitals with data independently collected across them, and want to
generalize predictions to a new hospital selected in the same manner. In
essence, while this assumption is limiting, it will make the following
analysis clearer (and not any less valid), thus helping elucidate other
important downsides of meta-learning approaches.

Proceeding in a full Bayesian manner, let us assume a meta-prior
$P(\mathscr{P})$ that encompasses all our prior knowledge about the meta-
distribution $\mathscr{P}$. Following [FAL17] we assume access to datasets
$\\{S^{j}\\}_{j=1,\cdots,|\mathcal{E}_{\text{train}}|}$, and potentially empty
sample sets from the test distribution $S^{\text{test}}$. Here,
$S^{j}=\\{x^{(i),j},y^{(i),j}\\}_{i=1}^{n_{j}}$ are sampled i.i.d. from
$P(x,y|e_{j})$, the data distribution for environment $e_{j}$. Given a new
test datapoint $x^{*}$, our goal will be to estimate
$P(y^{*}|x^{*},S^{1},\dots,S^{|\mathcal{E}_{\text{train}}|},S^{\text{test}})$.
The expectation of this distribution is the Bayes-optimal prediction for
losses such as cross-entropy or MSE.

For simplicity we assume that $P(x|e)$ is the same for all environments
$e\in\mathcal{E}_{\text{all}}$, and that all the distributions involved admit
nonzero densities. This is not strictly necessary for the analysis, but will
help us avoid cumbersome calculations that bring nothing to the table. This
then means that $P(y,x|e)=P(y|x,e)P(x)$. By Bayes rule, we can see

$P(y^{*}|x^{*},S^{1},\dots,S^{|\mathcal{E}_{\text{train}}|},S^{\text{test}})=\int_{\mathcal{E}_{\text{all}}}P(y^{*}|x^{*},e^{\text{test}})P(e^{\text{test}}|S^{1},\dots,S^{{|\mathcal{E}_{\text{train}}|}},S^{\text{test}})\operatorname{d}\\!{e}^{\text{test}}$
(D.1)

Now, $P(y^{*}|x^{*},e^{\text{test}})$ is a trivial part, since the
conditioning assumes we know what the test environment is. In more traditional
Bayesian notation, this is written $P(y^{*}|x^{*},\theta)$, just running the
forward pass once we know the parameters of the test environment. The
difficulty of calculating (D.1) lies in sampling or estimating
$P(e^{\text{test}}|S^{1},\dots,S^{{|\mathcal{E}_{\text{train}}|}},S^{\text{test}})$,
i.e. what the likely forward pass is for the test samples given the previous
environments’ and limited test data.
$S^{1},\dots,S^{{|\mathcal{E}_{\text{train}}|}}$ will give us information
about $\mathscr{P}$ and thus what things are likely across most environments,
and $S^{\text{test}}$ will help us nail down things that are specific for the
test distribution.

More concretely,

$\displaystyle P(e^{\text{test}}|S^{1},$
$\displaystyle\dots,S^{{|\mathcal{E}_{\text{train}}|}},S^{\text{test}})$
$\displaystyle=\int_{\mathcal{P}(\mathcal{E}_{\text{all}})}P(e^{\text{test}}|\mathscr{P},S^{1},\dots,S^{{|\mathcal{E}_{\text{train}}|}},S^{\text{test}})P(\mathscr{P}|S^{1},\dots,S^{{|\mathcal{E}_{\text{train}}|}},S^{\text{test}})\operatorname{d}\\!{\mathscr{P}}$
$\displaystyle=\int_{\mathcal{P}(\mathcal{E}_{\text{all}})}P(e^{\text{test}}|\mathscr{P},S^{\text{test}})P(\mathscr{P}|S^{e_{1}},\dots,S^{{|\mathcal{E}_{\text{train}}|}},S^{\text{test}})\operatorname{d}\\!{\mathscr{P}}$

Now, the sampling term $P(e^{\text{test}}|\mathscr{P},S^{\text{test}})$
describes how to nail down to things that are specific to $e^{\text{test}}$
once we already know $\mathscr{P}$ (i.e. the things that are likely for all
environments), and $S^{\text{test}}$ which tells us additional information to
the specifics of the test environment. By Bayes rule, this can be written as
$P(e^{\text{test}}|\mathscr{P},S^{\text{test}})=\frac{P(S^{\text{test}}|e^{\text{test}})\mathscr{P}(e^{\text{test}}))}{P(S^{\text{test}}|\mathscr{P})}$.
This is the standard decomposition of the posterior into likelihood, prior,
and marginal. In more traditional Bayesian language, this would be written
down as $P(\theta|S)=\frac{P(S|\theta)P(\theta)}{P(S)}$.

Since this last problem of approximating the posterior once we have the prior
$\mathscr{P}$ has been studied in the entire literature of Bayesian
optimization and is essentially a standard i.i.d. problem, we will focus on
the term that’s more relevant to us: estimating
$P(\mathscr{P}|S^{1},\dots,S^{{|\mathcal{E}_{\text{train}}|}},S^{\text{test}})$.
This term describes what things are shared with high probability across all
environments, given training data from multiple environments. An example of
this would be given data from multiple environments, recovering the shared
causal structure between them. This is the term that all of chapter 3
implicitly focused on, and the term that enables zero-shot out of distribution
generalization. One could interpret section 3.3 as assumptions on the meta-
prior $P(\mathscr{P})$ such that $P(\mathscr{P})$ assigns probability only to
those $\mathscr{P}$ whose environments $e\sim\mathscr{P}$ all share a low
error invariant predictor. This invariant predictor can be different for
different $\mathscr{P}$’s, but for each $\mathscr{P}$ there is a shared
invariant predictor across its supported environments.

Rewriting $S^{\text{test}}$ as $S^{0}$, we have

$\displaystyle P(\mathscr{P}|S^{0},\dots,S^{{|\mathcal{E}_{\text{train}}|}})$
$\displaystyle=\frac{P(S^{0},\dots,S^{{|\mathcal{E}_{\text{train}}|}}|\mathscr{P})P(\mathscr{P})}{P(S^{0},\dots,S^{{|\mathcal{E}_{\text{train}}|}})}$
$\displaystyle=\frac{P(\mathscr{P})\prod_{j=0}^{|\mathcal{E}_{\text{train}}|}P(S^{j}|\mathscr{P})}{P(S^{0},\dots,S^{{|\mathcal{E}_{\text{train}}|}})}$

We will be interested in maximum a posteriori (MAP) estimates for
$\mathscr{P}$, since these ones will describe the most likely commonalities
shared across environments with high probability, and as briefly mentioned in
the previous paragraph, enable OOD. Since the term in the denominator doesn’t
depend on $\mathscr{P}$, this term will be phrased as a constant $C$. Taking
logarithms, we finally obtain the posterior probability of $\mathscr{P}$

$\displaystyle\log
P(\mathscr{P}|S^{0},\dots,S^{{|\mathcal{E}_{\text{train}}|}})$
$\displaystyle=\log C+\log
P(\mathscr{P})+\sum_{j=0}^{|\mathcal{E}_{\text{train}}|}\log
P(S^{j}|\mathscr{P})$ $\displaystyle=\log C+\log
P(\mathscr{P})+\sum_{j=0}^{|\mathcal{E}_{\text{train}}|}\log\int_{\mathcal{E}_{\text{all}}}\prod_{i=1}^{n_{j}}P(x^{(i),j},y^{(i),j}|e_{j})\mathscr{P}(e_{j})\operatorname{d}\\!{e_{j}}$
$\displaystyle\geq\log C+\log
P(\mathscr{P})+\sum_{j=0}^{|\mathcal{E}_{\text{train}}|}\int_{\mathcal{E}_{\text{all}}}\log\prod_{i=1}^{n_{j}}P(x^{(i),j},y^{(i),j}|e_{j})\mathscr{P}(e_{j})\operatorname{d}\\!{e_{j}}$
(D.2) $\displaystyle=\log C+\log
P(\mathscr{P})+\sum_{j=0}^{|\mathcal{E}_{\text{train}}|}\sum_{i=1}^{n_{j}}\mathbb{E}_{e\sim\mathscr{P}(e)}\left[\log
P(x^{(i),j},y^{(i),j}|e)\right]$ (D.3)

Thus, we have arrived to (D.3) which is the addition of three terms: a
constant, the meta-prior over $\mathscr{P}$, and the likelihood of the data
under $\mathscr{P}$. If the inequality (D.2) was an equality, we would have
obtained a guarantee that MAP estimation for $\mathscr{P}$ (the left hand
side) is equivalent to maximizing the likelihood of the data under
$\mathscr{P}$ and its meta-prior probability. This is almost identical to the
result on the i.i.d. case, in which MAP estimation over conditional
distributions is equivalent to maximizing log-likelihood and a prior. Since
using a flat prior and optimizing likelihood (i.e. empirical risk
minimization) can work so well in the i.i.d. setup, we might wonder whether we
can do the same in (D.3). There are however a few differences from the i.i.d.
setup, which will show why reussing the i.i.d. machinnery and just optimizing
the likelihood term of (D.3) may not be such a good idea.

The first difference in this meta-i.i.d. derivation is the variational bound
in equation (D.2), which is due to Jensen’s inequality. Since Jensen’s
inequality is an equality if and only if the random variable inside the
expectation is deterministic, we can see that this is not the case, and indeed
the inequality is strict unless we are in the pathological case where the
meta-prior is concentrated on a delta (which would be assuming the test
distribution is the same as the training one). This means that if our goal is
to estimate the $\mathscr{P}$ with the highest probability under the data,
optimizing the likelihood and the meta-prior can be suboptimal (which is
different than in the i.i.d. setting). However, it is a good exercise that we
leave for the reader to prove that this bound becomes an equality after
normalizing and letting $n_{j}\to\infty$. This means that if we assume that we
have enough samples from each environment but not a large number of
environments (as assumed in chapter 3), MAP estimation over $\mathscr{P}$ is
approximately equivalent to optimizing the data likelihood and the meta-prior.
Then, what could be the difference with the i.i.d. setup where just optimizing
likelihood can yield such good results? The answer lies in the meta-prior.

In the typical i.i.d. setting we have

$\log P(e|S)=\log C+\log P(S|e)+\log P(e)$ (D.4)

or in more traditional notation $\log P(\theta|S)=\log C+\log P(S|\theta)+\log
P(\theta)$. Successful approaches for tasks like language modelling and object
classification often require three simple things: carefully restricting the
architecture (i.e. the structure of $P(S|\theta)$), using SGD and clever
initialization schemes, and optimizing the likelihood term $\log P(S|\theta)$.
Restricting the architecture, SGD and initialization schemes can easily be
phrased as using particular priors $P(\theta)$ [GBC16, CS18]. The combination
of these priors and optimizing the likelihood leads to good in-distribution
generalization for the mentioned problems with large amounts of data.

Restricting the architecture and optimizers in the case of equation (D.3)
would mean having $P(\mathscr{P})$ supported on those $\mathscr{P}(e)$ that
assign probability to environments $e$ with conditional distributions
$P(y|x,e)$ that are of these kinds of architectures. However, it doesn’t say
how these different environments should relate to each other. To make this
more concrete, in a problem with two variables $X,Y$ where we want to find
whether $X$ causes $Y$ or $Y$ causes $X$, we need two environments to figure
this out by looking at whether $P(Y|X)$ or $P(X|Y)$ are invariant. This means
having a meta-prior $P(\mathscr{P})$ where each $\mathscr{P}$ has only one of
those conditionals invariant across supported environments. Put it another
way, restricting the architectures and using SGD is akin to defining a shared
support of the possible $\mathscr{P}$, while OOD problems like cause and
effect require $P(\mathscr{P})$ such that we can identify the stable
properties over environments in $\mathscr{P}$ from a few of its meta-samples.
In essence, we need meta-priors $P(\mathscr{P})$ that give us
_identifiability_ over what properties are stable over all
$\text{Supp}(\mathscr{P})$, not just sets that contain
$\text{Supp}(\mathscr{P})$. To stress this important point once more:
architectural and optimization priors state a set $A$ such that
$\text{Supp}(\mathscr{P})\subseteq A$ with high probability under the meta-
prior $P(\mathscr{P})$. We want more. We want, given training environments,
that there is an identifiable predictor with low error across all
$e\in\text{Supp}(\mathscr{P})$. Since this predictor can be different for
different $\mathscr{P}$’s, it is not enough to specify a common superset of
the supports
$\cup_{\mathscr{P}\in\text{Supp}(P(\mathscr{P}))}\text{Supp}(\mathscr{P})\subseteq
A$.

These differences in possible meta-priors wouldn’t matter given sufficient
number of environments (at least for the meta-i.i.d. case we are assuming in
this chapter), since the role of the meta-prior dissapears as the likelihood
dominates. However, we are interested in the nonasymptotic case where the
number of environments is reasonably small. As seen in chapter 3, we are
sometimes able to generalize with two environments. This is in contrast to the
typical i.i.d. case where we have millions of samples. We have a lot less
meta-samples in the meta-i.i.d. setup than we have samples in traditional
i.i.d. problems. Luckily in most tasks, and in particular in those of section
2.2, there is a shared structure of the environments, which can be boiled down
as a specific meta-prior. For instance, in example 2.1 a meta-prior that
allows us to solve this problem is one that supports $\mathscr{P}$ for which
with high probability given
$e_{1},\dots,e_{|\mathcal{E}_{\text{train}}|}\sim\mathscr{P}$ there is a
single low-error invariant predictor across
$\\{e_{1},\dots,e_{|\mathcal{E}_{\text{train}}|}\\}$ that is invariant across
$\text{Supp}(\mathscr{P})$.

MAML [FAL17] goes one step forward of simply restricting the family of
possible $P(Y|X,e)$. In addition to this, it implicitly has a meta-prior that
assumes a low error solution is obtainable by few steps of SGD on all
environments by a common initialization point. This already starts to look at
things which are shared across environments and identifiability, since we can
discard possible test environments when those would be far away from common
initialization points of training environments. As such, it has a very non-
flat meta-prior. Indeed, this is likely a reasonable meta-prior for some
problems. However, in example 2.1 the initialization point would make the
conditional of $P(Y|X_{2})$ lie in the middle of $[\beta(e=1),\beta(e=2)]$,
and thus far from that of the test environment. It is clear that invariant
prediction is not identifiable by MAML or other methods that look for lowest-
error classifiers close to a common initialization.

A final work that deserves mention in this chapter is that of [Ben+19]. There,
the authors show that speed of adaptation from one environment to another is
enough to identify the causal directions given two environments. This work
shows that meta-priors like assuming shared causal structure across
environments make optimal OOD predictors algorithmically identifiable by the
meta-learning flavored idea of speed of adaptation. In this work, as in
chapter 3, this meta-prior is crucial. However, the algorithmic means to
exploit this prior are different, and more research needs to be done so that
we can figure out how to efficiently leverage this and other shared
structures.

As a final comment, we would like to mention that we find figuring out how to
merge learning invariances across training environments with finetuning when
some limited data is available from the test domain to be a particularly
promising direction for future work.