Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

In particular, we demonstrate that INFERNO learns interpretable object poses, which we can modify and render to alter the pose of an object in a scene.

The review comment to be evaluated is:

Strengths:

+ The paper is clearly written and easy to follow.
+ It is novel and interesting to see object-centric learning models to infer 3D poses.
+ Validating the usefulness in downstream visual reasoning task is a plus.

Weakness:

My main conern is in the insufficient experiments that do not support the main contributions, i.e., the experiments do not demonstrate the benefits of the proposed 3D, structured object representations.

As for representations being structured (i.e., disentangled shape, appearance and poses):
- For disentanglement of shape and appearance, there is no experiments demonstrating this point. For example, this can be easily shown by swapping appearances of two objects in the scene while keeping their shapes unchanged, and vice versa.
- For disentanglement of 3D pose, the experiments include scaling and translation. However, there is no comparison to 2D baseline methods, which are able to do scaling and translation (at least the experiments on moving to the "right").

As for representations being 3D, there is no comparisons to show the benefits of 3D representation over 2D. In segmentation experiments, the results are worse than 2D baselines. In the visual reasoning task, the performance is worse than Aloe. Perhaps Aloe is better because it is designed for that task, but there is neither comparisons to 2D general object-centric baselines (such as original slot attention) or unstructured baselines.

Other minor weaknesses include:
- No ablation study on the model components to show the benefits of proposed technical contributions, such as the effect of regressing pose vs. not regressing pose (I suppose this should affect background decomposition because that is the only inductive bias that separates background slot from foreground slots).
- It would be good to demonstrate on datasets other than CLEVR.

The evidence to be evaluated is:The goal of our method is to infer object-centric 3D scene representations from single 2D views.

Each object is composed of three tensors \(o_{i}=(o_{i}^{shape},o_{i}^{app},o_{i}^{pose})\).

The object location \(o_{i}^{pose}\in\mathbb{R}^{4\times 4}\) is an affine matrix that describes the object 3D pose (i.e. scale, translation and rotation) in the scene.

We define a rendering function \(g_{\gamma}\) that takes as input a scene representation and generates a 2D view of that scene \(\hat{x}=g_{\gamma}(f_{\theta}(x))\).

\[\gamma^{*},\theta^{*}=\operatorname*{arg\,min}_{\gamma,\theta}p(X|\gamma,\theta) =\operatorname*{arg\,min}_{\gamma,\theta}\frac{1}{N}\sum_{i\leq N}(x_{i}-g_{ \gamma}(f_{\theta}(x_{i})))^{2} \tag{1}\]

We represent 3D objects as Neural Radiance Fields (NeRFs) (Mildenhall et al., 2020) with a similar setup as that of GIRAFFE (Niemeyer and Geiger, 2021). A NeRF is a function \(g_{\tau}\) that defines a 3D shape implicitly. It takes as input a 3D location \(\mathbf{l}=(x,y,z)\) and a 2D viewing direction \(\mathbf{d}=(\psi,\phi)\) and outputs an occupancy value \(\sigma\) and a color value \(\mathbf{a}=(r,g,b)\).

Each object specific shape and appearance is defined by the shape and appearance latent variables, which correspond to the object attributes \(o^{shape},o^{app}\).

The pose of an object \(o_{i}\) in the scene is determined by the affine transformation matrix \(o_{i}^{pose}\). We denote the coordinate system of the NeRF function of an object as the object space, and the coordinate system of the scene (and the background NeRF) as scene space. Given an object pose, we can convert points from the scene space to the object space by applying the \(o^{pose}\) transformation matrix on those points, and we can transform points from object space to scene space by computing the inverse of the object pose matrix.

We map these slots to our structured scene representation through small fully-connected networks that operate on individual objects. More concretely, we map object slots to their 3D pose in the scene through a 2-layer MLP. To infer the object shape and appearance tensors we also use 2-layer MLPs, but we make them conditional on the predicted object pose through conditional normalization (Dumoulin et al., 2018).

Their factored representations are amenable to object manipulations.

Our model uses a variant of Slot Attention to decide on which part of a 2D view should each object in our scene attend to, but we infer 3D-aware representations for each object.

Contrary to most previous approaches which segment 2D shapes, our method infers object-centric scene representations in 3D space.

First, we demonstrate the interpretability of the scene representations through manipulating scenes and verifying the corresponding effects in the rendered outputs.

Our model infers 3D object-centric scene representations from single 2D views. These representations disentangle the 3D appearance and pose of objects, which allows for semantic manipulations of the scene not possible otherwise. These manipulations can be validated by rendering the modified scene representations.

In INFERNO, we generate novel scenes by inferring representations for ground-truth images and then manipulating them.

We also investigate different interpretable manipulations of scene representations and visualize the effects in the corresponding output renderings.

We validate that our model is able to render out-of-distribution scenes not corresponding to training examples, such as scenes having 1 or 6 objects, and verify that the pose manipulations have semantically coherent effects.

Additionally, our model can perform inference and manipulations on existing scenes. We use that capability to generate novel scenes by manipulating existing ones.

In Figure 2 we show some examples of the scene manipulations possible with our model. Given a scene representation, we can remove or add objects, rearrange object poses, translate the objects to new locations or change the object scales. While some of these manipulations can be performed with regular object-centric models, modifications to the scale and location of the objects are hard to implement without explicitly modeling 3D object pose.

Our model is able to discover objects in a scene without annotations, and the inferred scene representations are interpretable and amenable to manipulations.

Our model is composed of five main modules: encoder, slot attention, slot to object mapping, NeRF decoder, neural network upscaler.

For each object slot, we first map the object to its pose parameters. We use a 2-layer MLP with LayerNorm to map a slot to its pose.

Once we have inferred the object poses, we infer object shapes and appearances. These are inferred individually for each object using a common 2-layer MLP. The MLPs are conditional on the object pose using Conditional Layer Normalization, that makes the learned parameters of LayerNorm be a function of a condition. Specifically, we map the 7-dimensional pose tensor to LayerNorm parameters with a single linear layer with no activation or normalization.

With the object shape and appearance tensors we can render them following GRAF (Schwarz et al., 2020).

Note that, to render objects according to their pose, we query their NeRF MLP in a canonical object space by transforming input coordinates in scene space to object space using the object pose.

* _Forward:_ We manipulate the pose vector of one object in the substraction scene and move it forward on the Z axis.

* _Right:_ We manipulate the pose vector of one object in the substraction scene and move it forward on the X axis.

This transformations modifies a scene by replacing the object shape and appearance vectors with those of an object from another scene.

We show: i) each object rendered individually, ii) object identity swaps with other scenes, iii) object translations along one axis, iv) translations in diagonal (two axis), and v) objects moving in a circle.

While our model is trained without ground-truth camera poses and with single views of scenes, it is able to generalize to small camera pose modifications and render novel views of a scene

Input: Figure 2: Manipulations on CLEVR: we show some examples of the manipulations we perform to CLEVR2345 images, including object removal and addition, changing the scale of an object, and object translation. Our model can perform these transformations because it disentangles object pose and appearance.

Input: Figure 1: Model Overview: We propose INFERNO, a model that infers and renders object-centric 3D scene representations. 1 Our model first decomposes an input observation into multiple object slots. 2 For each slot we infer a structured 3D representation. 3 The shape and appearance determine canonical objects rendered through NeRFs. 4 Objects are transformed and located in the overall scene according to their pose. 5 We combine objects and background and render a low-resolution scene given a camera location. 6 The input is reconstructed by upscaling the low resolution scene.

Input: Figure 6: Novel view synthesis on CLEVR2345. For each scene, we move the camera ±15◦ on the azimuth axis. Additionally, we zoom in the scene twice. While our model is trained with a fixed default camera pose and single scene views, it is able to generalize to small camera pose modifications and render novel views of a scene.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on the evidence, the claim "we demonstrate that INFERNO learns interpretable object poses, which we can modify and render to alter the pose of an object in a scene" appears to be well-supported. The figures show clear examples of pose manipulations including translation (moving objects right/forward), scaling, and rotation through camera viewpoint changes. The technical details explain how poses are represented as affine matrices and modified. However, the reviewer points out there's no comparison to 2D baselines to demonstrate unique benefits of 3D poses. While the claim itself is accurate about the capability, it could be more precise about limitations.

<score>2</score>

Justification:
1. Supporting evidence:
- Figures demonstrate successful pose modifications including translation, scaling, and rotation
- Technical implementation details showing how poses are represented and manipulated
- Model architecture clearly shows pose representation and transformation components

2. Reasons for score of 2 (slightly overstated):
- While the claim is fundamentally accurate about the capability, it could acknowledge:
  * Lack of comparative evaluation vs 2D methods that can also do some pose modifications
  * Limited to specific types of pose changes (translation, scale, rotation)
  * Only tested on CLEVR dataset
- The reviewer's concern about insufficient experiments comparing to 2D baselines is valid but doesn't invalidate the core claim about the capability itself

3. Direct citations:
"These representations disentangle the 3D appearance and pose of objects, which allows for semantic manipulations of the scene not possible otherwise."
"We use a 2-layer MLP with LayerNorm to map a slot to its pose."

The claim would be better stated with some qualifiers about the scope and limitations of the pose manipulation capabilities.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

* We propose a model able to infer and render 3D scene representations composed of multiple objects, each of them modeled by an implicit function and explicitly localized in the scene.

The evidence to be evaluated is:The goal of our method is to infer object-centric 3D scene representations from single 2D views. Given an image \(x\in\mathbb{R}^{H\times W\times 3}\), we learn an inference function \(f_{\theta}\) that maps images to scene representations \(s=f_{\theta}(x)=(o_{1},o_{2},...,o_{K},o_{bg},c)\). Scenes are composed of \(K\) objects \(o_{i}\), a background object \(o_{bg}\) and a camera location \(c\). Each object is composed of three tensors \(o_{i}=(o_{i}^{shape},o_{i}^{app},o_{i}^{pose})\). The object shape \(o_{i}^{shape}\in\mathbb{R}^{D_{shape}}\) and object appearance \(o^{app}\in\mathbb{R}^{D_{shape}}\) are tensors that respectively describe the 3D shape occupancy and color of an object with an implicit function. The object location \(o_{i}^{pose}\in\mathbb{R}^{4\times 4}\) is an affine matrix that describes the object 3D pose (i.e. scale, translation and rotation) in the scene.

We define a rendering function \(g_{\gamma}\) that takes as input a scene representation and generates a 2D view of that scene \(\hat{x}=g_{\gamma}(f_{\theta}(x))\).

In the following sections we describe in more detail our inference mechanism, our rendering pipeline and their implementation. We represent 3D objects as Neural Radiance Fields (NeRFs) (Mildenhall et al., 2020) with a similar setup as that of GIRAFFE (Niemeyer and Geiger, 2021). A NeRF is a function \(g_{\tau}\) that defines a 3D shape implicitly. It takes as input a 3D location \(\mathbf{l}=(x,y,z)\) and a 2D viewing direction \(\mathbf{d}=(\psi,\phi)\) and outputs an occupancy value \(\sigma\) and a color value \(\mathbf{a}=(r,g,b)\).

To represent multiple 3D shapes with the same NeRF function, we can augment it with latent variables that determine which shape is being modeled (Schwarz et al., 2020).

In INFERNO, we share a single parametrization of a NeRF function across all objects. Each object specific shape and appearance is defined by the shape and appearance latent variables, which correspond to the object attributes \(o^{shape},o^{app}\).

The pose of an object \(o_{i}\) in the scene is determined by the affine transformation matrix \(o_{i}^{pose}\). We denote the coordinate system of the NeRF function of an object as the object space, and the coordinate system of the scene (and the background NeRF) as scene space. Given an object pose, we can convert points from the scene space to the object space by applying the \(o^{pose}\) transformation matrix on those points, and we can transform points from object space to scene space by computing the inverse of the object pose matrix. To render a scene, we cast rays from each pixel in the 2D plane defined by a given camera to the 3D scene. We evaluate NeRFs at different points along a given ray, and integrate their occupancy and color outputs to determine pixel values. Rays might traverse multiple object NeRFs in addition to the background NeRF. To determine the occupancy and color of points described by multiple NeRFs, we first query each NeRF at those particular points. To query the object NeRFs, we first need to transform the points from scene space to the particular object space. Then, we compose the results of each NeRF with a pooling function \(C\), which in our case is a weighted average: \[C(l,\mathbf{d})=(\sigma=\sum_{i=1}^{N}\sigma_{i},\frac{1}{\sigma}\sum_{i=1}^{N }\sigma_{i}\mathbf{a}_{i}) \tag{4}\]

Given the rendering pipeline, the goal of our method is to infer representations that reconstruct a given scene. Our inference mechanism computes image features through a neural network encoder and then extracts \(K\) object and a background slots. These slots are then mapped to our structured scene representation through learned neural networks. To extract image features for each object and background slots, we use Slot Attention (Locatello et al., 2020). Slot Attention is a mechanism that maps a set of \(K\) entities, called slots, to image features without annotations. It extracts image features \(I\in\mathbb{R}^{H\times W\times D}\) from a given input using a resolution-preserving convolutional encoder. These features are then attended to by a set of \(K\) randomly sampled slots \(\pi_{j}\sim\mathcal{N}(\mu,\sigma)\) of dimension \(D\), where \(\mu\) and \(\sigma\) are learnable parameters.

Slots attend spatial chunks of the input features \(I\) through soft-attention \(u=TQ^{T}\), where \(T=k(I)\) and \(Q=q(\pi)\) are the embeddings of the inputs and slots respectively. The attention weights are normalized through a softmax operating on the slots axis, which makes slots compete among themselves and discourages multiple slots from attending the same input region \(w=softmax(u)\). The weighted average of \(I\) according to the attention weights is then computed and fed to a GRU network, to update each slot value: \(\pi_{j}=GRU(w_{j}*I,\pi_{j})\ \forall j\in{1,K}\). Multiple rounds of soft attention are performed to iteratively refine the slots.

Object slots are unstructured tensors that result from aggregating image features. We map these slots to our structured scene representation through small fully-connected networks that operate on individual objects. More concretely, we map object slots to their 3D pose in the scene through a 2-layer MLP. To infer the object shape and appearance tensors we also use 2-layer MLPs, but we make them conditional on the predicted object pose through conditional normalization (Dumoulin et al., 2018).

Recently, the use of functions that implicitly model 3D volumes has gained popularity (Park et al., 2019; Mescheder et al., 2019; Sitzmann et al., 2019; Sitzmann et al., 2020; Schiorek et al., 2021; Pumarola et al., 2021; Yu et al., 2021). Implicit representations have better scaling properties, as usually the output resolution does not directly affect the dimensionality of the learned function. NeRFs (Mildenhall et al., 2020) generate scenes by learning a function that outputs the occupancy and color of points in a scene when viewed from a particular direction. By casting rays through a plane and aggregating the output values NeRFs can generate 2D views of an implicitly modeled 3D scenes. NeRFs have obtained superior reconstructions compared to other implicit methods, and our model uses NeRFs to represent multiple objects and the background of a scene.

GIRAFFE (Niemeyer and Geiger, 2021) is a GAN-based method that represents multiple objects in a scene using NeRFs as part of their generator.

However, we focus on recovering scene representations from existing images, while GIRAFFE does not have an inference mechanism. ObjSURF (Stelzner et al., 2021) and UORF (Yu et al., 2021) infer scene representations composed of multiple objects, each object represented with a differently instantiated NeRF.

Recently, there is a line of work on object-centric generative models of scenes (Kosiorek et al., 2021; Burgess et al., 2019; Locatello et al., 2020; Lin et al., 2020; Greff et al., 2019). These models learn to generate scenes as a composition of multiple objects and a background.

Slot-Attention (Locatello et al., 2020) maps a set of entities, called slots, to image features through multiple rounds of soft attention. The slots compete among themselves to attend to features, making each slot attend to a region of the input image. When driven with alpha-compositing decoder, slot attention learns to segment objects in a scene. Our model uses a variant of Slot Attention to decide on which part of a 2D view should each object in our scene attend to, but we infer 3D-aware representations for each object.

Contrary to most previous approaches which segment 2D shapes, our method infers object-centric scene representations in 3D space.

First, we demonstrate the interpretability of the scene representations through manipulating scenes and verifying the corresponding effects in the rendered outputs. Then we show that it learns to identify and segment the objects in a scene without supervision.

Our model infers 3D object-centric scene representations from single 2D views. These representations disentangle the 3D appearance and pose of objects, which allows for semantic manipulations of the scene not possible otherwise.

In INFERNO, we generate novel scenes by inferring representations for ground-truth images and then manipulating them.

We also investigate different interpretable manipulations of scene representations and visualize the effects in the corresponding output renderings.

We validate that our model is able to render out-of-distribution scenes not corresponding to training examples, such as scenes having 1 or 6 objects, and verify that the pose manipulations have semantically coherent effects.

Additionally, our model can perform inference and manipulations on existing scenes.

Our model is able to generate novel scenes with additional objects or with altered object shapes and appearances, with better FID than GIRAFFE.

Given a scene representation, we can remove or add objects, rearrange object poses, translate the objects to new locations or change the object scales.

Note that this setup evaluates 2D segmentation masks, although our model naturally provides 3D segmentations.

However, our model learns to segment objects in 3D, while the other baselines extract 2D segmentation masks. Figure 3 shows some examples of the discovered object masks. We can see that the model learns to segment different objects and properly discards object slots when the number of objects in the scene is lower than needed.

The scenes show multiple objects that move over time, one being the snitch object.

Each object in our representation is given as an input element to the transformer. We add a learned positional encoding to the object representations based on their frame index. Objects in the same frame have the same positional encoding.

This result suggests that the object-centric representation learned by INFERNO is useful for the visual reasoning task.

This result supports the advantage of 3D-aware representation for solving the CATER task. We propose INFERNO, a model for inferring object-centric 3D scene representations. Our model is able to discover objects in a scene without annotations, and the inferred scene representations are interpretable and amenable to manipulations.

Our model is composed of five main modules: encoder, slot attention, slot to object mapping, NeRF decoder, neural network upscaler.

We employ Slot Attention to map image features to object slots. Slots are sampled randomly from a Gaussian distribution with learned parameters. We use different distributions for the background slot and the object slots.

For each object slot, we first map the object to its pose parameters. We use a 2-layer MLP with LayerNorm to map a slot to its pose.

Once we have inferred the object poses, we infer object shapes and appearances. These are inferred individually for each object using a common 2-layer MLP. The MLPs are conditional on the object pose using Conditional Layer Normalization, that makes the learned parameters of LayerNorm be a function of a condition. Specifically, we map the 7-dimensional pose tensor to LayerNorm parameters with a single linear layer with no activation or normalization. Shapes and appearances are defined by the output of the MLPs, which produce two 128-dimensional tensors. For the background object we only infer shape and appearance, and define its pose to be that of the scene cube. The shape and appearance of the background are inferred through another 2-layer MLP with ReLU and LayerNorm.

With the object shape and appearance tensors we can render them following GRAF (Schwarz et al., 2020). We use one NeRF for the objects and one NeRF MLP for the background.

Note that, to render objects according to their pose, we query their NeRF MLP in a canonical object space by transforming input coordinates in scene space to object space using the object pose.

For generating manipulated scenes we consider the CLEVR2345 dataset (Niemeyer and Geiger, 2021). Images in the CLEVR2345 dataset contain from 2 to 5 objects.

We use 5 objects and rely on the model to not use additional slots if the scene shows less than 5 objects.

We use 6 objects and rely on the model to not use additional slots when needed.

We first pretrain our INFERNO to reconstruct individual frames from the CATER dataset. We bootstrap a model trained on CLEVR6 for object discovery for 400k iterations and train it on the CATER dataset for an additional 100k iterations to speed-up training, as the iteration time of a model with 10 objects is larger.

We show: i) each object rendered individually, ii) object identity swaps with other scenes, iii) object translations along one axis, iv) translations in diagonal (two axis), and v) objects moving in a circle.

While our model is trained without ground-truth camera poses and with single views of scenes, it is able to generalize to small camera pose modifications and render novel views of a scene

While these generations have reduced details due to their resolution, they clearly show the different objects and their location in the scene.

In this section we show the occupancy maps for each rendered object. The occupancy maps are obtained by integrating rays going from the image plane to the scene according to the NeRF density, but without taking into account the RGB output. We observe that each object is rendering a part of the scene corresponding to a single object instance.

To perform unsupervised object discovery we use the outputs of slot attention, since they do not require any upsampling and thus can provide more accurate segmentations.

Input: Figure 3: Object Discovery on CLEVR6: INFERNO identifies the different objects in a scene without supervision. For each input image, we show which regions of the input are attended by each object as well as the background. We include an example of a failed segmentation in the last row, where one object slot (4) is trying to represent multiple objects at the same time.

Input: Figure 2: Manipulations on CLEVR: we show some examples of the manipulations we perform to CLEVR2345 images, including object removal and addition, changing the scale of an object, and object translation. Our model can perform these transformations because it disentangles object pose and appearance.

Input: Figure 9: Object occupancy maps on CLEVR2345. For each input scene we show the reconstructed image as well as each individual object occupancy map. Occupancy maps are obtained by integrating the density outputs of the NeRF function along rays going from the image plane to the scene. We observe that each object is rendering a part of the scene corresponding to a single object instance. Outputs of the NeRF function have low resolution and are upscaled by a neural network. The last row shows an example of a bad grouping of two objects in the scene.

Input: Figure 1: Model Overview: We propose INFERNO, a model that infers and renders object-centric 3D scene representations. 1 Our model first decomposes an input observation into multiple object slots. 2 For each slot we infer a structured 3D representation. 3 The shape and appearance determine canonical objects rendered through NeRFs. 4 Objects are transformed and located in the overall scene according to their pose. 5 We combine objects and background and render a low-resolution scene given a camera location. 6 The input is reconstructed by upscaling the low resolution scene.

Input: Figure 6: Novel view synthesis on CLEVR2345. For each scene, we move the camera ±15◦ on the azimuth axis. Additionally, we zoom in the scene twice. While our model is trained with a fixed default camera pose and single scene views, it is able to generalize to small camera pose modifications and render novel views of a scene.

Input: Figure 7: NeRF outputs on CLEVR2345. For each input scene we show the reconstructed image as well as each the low resolution output of the NeRF function. This output is then upscaled with a neural network to obtain the reconstructed scene. While NeRF output lack full detail, they correctly depict each individual object and their position in the scene.

Input: Figure 5: Additional object additions and removals on CLEVR2345. For each scene, we show images with one randomly added object, and with 1-3 random objects removed. Some of these images show out-of-distribution samples with a number of objects not seen during training (2-5 objects).

Input: Figure 8: Shape and appearance disentanglement on CLEVR2345. For each input scene we alter the appearance vector of an object in the scene. First, we show the reconstructed scene. Then, for one of the objects in the scene, we replace its appearance vector by that of another object in the scene, while keeping its shape vector fixed. We perform this operation two times. We observe that, while the shading and color of the object changes as we change the appearance vector, its shape remains constant.

Input: Figure 4: Additional reconstructions on CLEVR2345.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on the comprehensive evidence provided, including detailed model architecture diagrams, mathematical formulations, and extensive experimental results, the claim appears to be well-stated and accurately reflects the model's capabilities. The evidence demonstrates that:

1. The model successfully infers object-centric representations through slot attention mechanisms (Figure 1, Figure 3)
2. Each object is explicitly modeled using implicit functions (NeRFs) as shown in occupancy maps (Figure 9)
3. Objects are explicitly localized in 3D space through pose parameters (demonstrated in manipulation experiments in Figure 2)
4. The model can render coherent scene reconstructions and novel views (Figures 4-8)

The claim avoids overstating capabilities and matches what is demonstrated in the technical details and results.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

* We propose a model able to infer and render 3D scene representations composed of multiple objects, each of them modeled by an implicit function and explicitly localized in the scene.

The review comment to be evaluated is:

Strengths:

1. It is interesting to use NeRFs to represent objects in a scene, which encourages the understanding of 2D visual scene taking account into its 3D structure.
2. Explicitly modeling the 3D structure of 2D scene may help better deal with occlusion, ambiguity from similar appearance which can be a big challenge in 2D scene understanding.
3. It looks great to show how learned object features can be used for downstream tasks.

Weaknesses:
1. The technical novelty is somewhat limited. The paper mainly combines the Slot-attention and NeRF. Personally, it is hard to identify the exciting point(s).

2. The motivation of this work seems not well justified. It is claimed that the model is proposed to infer 3D representation for objects that are interpretable and amenable. But the experiments presented with such 3D representation cannot show its superiority and it is
hard to see the necessity of 3D object representation in this case considering the task presented can also be fulfilled with 2D object representation. In terms of the interpretability and manipulability of latent code, this paper presents objects removal/addition, object scaling and object translation. All of the above operations can actually be done just by explicitly modeling 2D object pose such as AIR (NIPS'16 paper). In terms of scene understanding in the form of object segmentation, this model cannot beat Slot Attention. In terms of downstream tasks, this model is not compared with other object-centric models. Thus, based on your current experiments, the motivation for learning such 3D representation for objects is not well justified.

3. Is it appropriate to use NeRF? If I understand correctly, NeRF learns the radiance field of a scene with 2D images rendered from different viewpoints. This model tries to recover the object radiance field from a single view of the objects. But the model training
objective can actually be fulfilled by just learning a single-view of the scene since there is only one image for each scene to generate reconstruction loss.

4. From my understanding, the 3D rendering model in this paper seems more similar to a decoder in VAE-wise architecture. It could be more convincing to provide some novel view synthesis (not novel scene) to claim it as a NeRF. For example, you may consider
object rotation.

5. All experiment datasets are a bit too simple. It is unclear how good/bad the proposed model will perform on realistic 3D scenes such as LLFF dataset or ScanNet. Otherwise, it is hard to see the real impact of this proposed method.

The evidence to be evaluated is:The goal of our method is to infer object-centric 3D scene representations from single 2D views. Given an image \(x\in\mathbb{R}^{H\times W\times 3}\), we learn an inference function \(f_{\theta}\) that maps images to scene representations \(s=f_{\theta}(x)=(o_{1},o_{2},...,o_{K},o_{bg},c)\). Scenes are composed of \(K\) objects \(o_{i}\), a background object \(o_{bg}\) and a camera location \(c\). Each object is composed of three tensors \(o_{i}=(o_{i}^{shape},o_{i}^{app},o_{i}^{pose})\). The object shape \(o_{i}^{shape}\in\mathbb{R}^{D_{shape}}\) and object appearance \(o^{app}\in\mathbb{R}^{D_{shape}}\) are tensors that respectively describe the 3D shape occupancy and color of an object with an implicit function. The object location \(o_{i}^{pose}\in\mathbb{R}^{4\times 4}\) is an affine matrix that describes the object 3D pose (i.e. scale, translation and rotation) in the scene.

We define a rendering function \(g_{\gamma}\) that takes as input a scene representation and generates a 2D view of that scene \(\hat{x}=g_{\gamma}(f_{\theta}(x))\).

In the following sections we describe in more detail our inference mechanism, our rendering pipeline and their implementation. We represent 3D objects as Neural Radiance Fields (NeRFs) (Mildenhall et al., 2020) with a similar setup as that of GIRAFFE (Niemeyer and Geiger, 2021). A NeRF is a function \(g_{\tau}\) that defines a 3D shape implicitly. It takes as input a 3D location \(\mathbf{l}=(x,y,z)\) and a 2D viewing direction \(\mathbf{d}=(\psi,\phi)\) and outputs an occupancy value \(\sigma\) and a color value \(\mathbf{a}=(r,g,b)\).

To represent multiple 3D shapes with the same NeRF function, we can augment it with latent variables that determine which shape is being modeled (Schwarz et al., 2020).

In INFERNO, we share a single parametrization of a NeRF function across all objects. Each object specific shape and appearance is defined by the shape and appearance latent variables, which correspond to the object attributes \(o^{shape},o^{app}\).

The pose of an object \(o_{i}\) in the scene is determined by the affine transformation matrix \(o_{i}^{pose}\). We denote the coordinate system of the NeRF function of an object as the object space, and the coordinate system of the scene (and the background NeRF) as scene space. Given an object pose, we can convert points from the scene space to the object space by applying the \(o^{pose}\) transformation matrix on those points, and we can transform points from object space to scene space by computing the inverse of the object pose matrix. To render a scene, we cast rays from each pixel in the 2D plane defined by a given camera to the 3D scene. We evaluate NeRFs at different points along a given ray, and integrate their occupancy and color outputs to determine pixel values. Rays might traverse multiple object NeRFs in addition to the background NeRF. To determine the occupancy and color of points described by multiple NeRFs, we first query each NeRF at those particular points. To query the object NeRFs, we first need to transform the points from scene space to the particular object space. Then, we compose the results of each NeRF with a pooling function \(C\), which in our case is a weighted average: \[C(l,\mathbf{d})=(\sigma=\sum_{i=1}^{N}\sigma_{i},\frac{1}{\sigma}\sum_{i=1}^{N }\sigma_{i}\mathbf{a}_{i}) \tag{4}\]

Given the rendering pipeline, the goal of our method is to infer representations that reconstruct a given scene. Our inference mechanism computes image features through a neural network encoder and then extracts \(K\) object and a background slots. These slots are then mapped to our structured scene representation through learned neural networks. To extract image features for each object and background slots, we use Slot Attention (Locatello et al., 2020). Slot Attention is a mechanism that maps a set of \(K\) entities, called slots, to image features without annotations. It extracts image features \(I\in\mathbb{R}^{H\times W\times D}\) from a given input using a resolution-preserving convolutional encoder. These features are then attended to by a set of \(K\) randomly sampled slots \(\pi_{j}\sim\mathcal{N}(\mu,\sigma)\) of dimension \(D\), where \(\mu\) and \(\sigma\) are learnable parameters.

Slots attend spatial chunks of the input features \(I\) through soft-attention \(u=TQ^{T}\), where \(T=k(I)\) and \(Q=q(\pi)\) are the embeddings of the inputs and slots respectively. The attention weights are normalized through a softmax operating on the slots axis, which makes slots compete among themselves and discourages multiple slots from attending the same input region \(w=softmax(u)\). The weighted average of \(I\) according to the attention weights is then computed and fed to a GRU network, to update each slot value: \(\pi_{j}=GRU(w_{j}*I,\pi_{j})\ \forall j\in{1,K}\). Multiple rounds of soft attention are performed to iteratively refine the slots.

Object slots are unstructured tensors that result from aggregating image features. We map these slots to our structured scene representation through small fully-connected networks that operate on individual objects. More concretely, we map object slots to their 3D pose in the scene through a 2-layer MLP. To infer the object shape and appearance tensors we also use 2-layer MLPs, but we make them conditional on the predicted object pose through conditional normalization (Dumoulin et al., 2018).

Recently, the use of functions that implicitly model 3D volumes has gained popularity (Park et al., 2019; Mescheder et al., 2019; Sitzmann et al., 2019; Sitzmann et al., 2020; Schiorek et al., 2021; Pumarola et al., 2021; Yu et al., 2021). Implicit representations have better scaling properties, as usually the output resolution does not directly affect the dimensionality of the learned function. NeRFs (Mildenhall et al., 2020) generate scenes by learning a function that outputs the occupancy and color of points in a scene when viewed from a particular direction. By casting rays through a plane and aggregating the output values NeRFs can generate 2D views of an implicitly modeled 3D scenes. NeRFs have obtained superior reconstructions compared to other implicit methods, and our model uses NeRFs to represent multiple objects and the background of a scene.

GIRAFFE (Niemeyer and Geiger, 2021) is a GAN-based method that represents multiple objects in a scene using NeRFs as part of their generator.

However, we focus on recovering scene representations from existing images, while GIRAFFE does not have an inference mechanism. ObjSURF (Stelzner et al., 2021) and UORF (Yu et al., 2021) infer scene representations composed of multiple objects, each object represented with a differently instantiated NeRF.

Recently, there is a line of work on object-centric generative models of scenes (Kosiorek et al., 2021; Burgess et al., 2019; Locatello et al., 2020; Lin et al., 2020; Greff et al., 2019). These models learn to generate scenes as a composition of multiple objects and a background.

Slot-Attention (Locatello et al., 2020) maps a set of entities, called slots, to image features through multiple rounds of soft attention. The slots compete among themselves to attend to features, making each slot attend to a region of the input image. When driven with alpha-compositing decoder, slot attention learns to segment objects in a scene. Our model uses a variant of Slot Attention to decide on which part of a 2D view should each object in our scene attend to, but we infer 3D-aware representations for each object.

Contrary to most previous approaches which segment 2D shapes, our method infers object-centric scene representations in 3D space.

First, we demonstrate the interpretability of the scene representations through manipulating scenes and verifying the corresponding effects in the rendered outputs. Then we show that it learns to identify and segment the objects in a scene without supervision.

Our model infers 3D object-centric scene representations from single 2D views. These representations disentangle the 3D appearance and pose of objects, which allows for semantic manipulations of the scene not possible otherwise.

In INFERNO, we generate novel scenes by inferring representations for ground-truth images and then manipulating them.

We also investigate different interpretable manipulations of scene representations and visualize the effects in the corresponding output renderings.

We validate that our model is able to render out-of-distribution scenes not corresponding to training examples, such as scenes having 1 or 6 objects, and verify that the pose manipulations have semantically coherent effects.

Additionally, our model can perform inference and manipulations on existing scenes.

Our model is able to generate novel scenes with additional objects or with altered object shapes and appearances, with better FID than GIRAFFE.

Given a scene representation, we can remove or add objects, rearrange object poses, translate the objects to new locations or change the object scales.

Note that this setup evaluates 2D segmentation masks, although our model naturally provides 3D segmentations.

However, our model learns to segment objects in 3D, while the other baselines extract 2D segmentation masks. Figure 3 shows some examples of the discovered object masks. We can see that the model learns to segment different objects and properly discards object slots when the number of objects in the scene is lower than needed.

The scenes show multiple objects that move over time, one being the snitch object.

Each object in our representation is given as an input element to the transformer. We add a learned positional encoding to the object representations based on their frame index. Objects in the same frame have the same positional encoding.

This result suggests that the object-centric representation learned by INFERNO is useful for the visual reasoning task.

This result supports the advantage of 3D-aware representation for solving the CATER task. We propose INFERNO, a model for inferring object-centric 3D scene representations. Our model is able to discover objects in a scene without annotations, and the inferred scene representations are interpretable and amenable to manipulations.

Our model is composed of five main modules: encoder, slot attention, slot to object mapping, NeRF decoder, neural network upscaler.

We employ Slot Attention to map image features to object slots. Slots are sampled randomly from a Gaussian distribution with learned parameters. We use different distributions for the background slot and the object slots.

For each object slot, we first map the object to its pose parameters. We use a 2-layer MLP with LayerNorm to map a slot to its pose.

Once we have inferred the object poses, we infer object shapes and appearances. These are inferred individually for each object using a common 2-layer MLP. The MLPs are conditional on the object pose using Conditional Layer Normalization, that makes the learned parameters of LayerNorm be a function of a condition. Specifically, we map the 7-dimensional pose tensor to LayerNorm parameters with a single linear layer with no activation or normalization. Shapes and appearances are defined by the output of the MLPs, which produce two 128-dimensional tensors. For the background object we only infer shape and appearance, and define its pose to be that of the scene cube. The shape and appearance of the background are inferred through another 2-layer MLP with ReLU and LayerNorm.

With the object shape and appearance tensors we can render them following GRAF (Schwarz et al., 2020). We use one NeRF for the objects and one NeRF MLP for the background.

Note that, to render objects according to their pose, we query their NeRF MLP in a canonical object space by transforming input coordinates in scene space to object space using the object pose.

For generating manipulated scenes we consider the CLEVR2345 dataset (Niemeyer and Geiger, 2021). Images in the CLEVR2345 dataset contain from 2 to 5 objects.

We use 5 objects and rely on the model to not use additional slots if the scene shows less than 5 objects.

We use 6 objects and rely on the model to not use additional slots when needed.

We first pretrain our INFERNO to reconstruct individual frames from the CATER dataset. We bootstrap a model trained on CLEVR6 for object discovery for 400k iterations and train it on the CATER dataset for an additional 100k iterations to speed-up training, as the iteration time of a model with 10 objects is larger.

We show: i) each object rendered individually, ii) object identity swaps with other scenes, iii) object translations along one axis, iv) translations in diagonal (two axis), and v) objects moving in a circle.

While our model is trained without ground-truth camera poses and with single views of scenes, it is able to generalize to small camera pose modifications and render novel views of a scene

While these generations have reduced details due to their resolution, they clearly show the different objects and their location in the scene.

In this section we show the occupancy maps for each rendered object. The occupancy maps are obtained by integrating rays going from the image plane to the scene according to the NeRF density, but without taking into account the RGB output. We observe that each object is rendering a part of the scene corresponding to a single object instance.

To perform unsupervised object discovery we use the outputs of slot attention, since they do not require any upsampling and thus can provide more accurate segmentations.

Input: Figure 3: Object Discovery on CLEVR6: INFERNO identifies the different objects in a scene without supervision. For each input image, we show which regions of the input are attended by each object as well as the background. We include an example of a failed segmentation in the last row, where one object slot (4) is trying to represent multiple objects at the same time.

Input: Figure 2: Manipulations on CLEVR: we show some examples of the manipulations we perform to CLEVR2345 images, including object removal and addition, changing the scale of an object, and object translation. Our model can perform these transformations because it disentangles object pose and appearance.

Input: Figure 9: Object occupancy maps on CLEVR2345. For each input scene we show the reconstructed image as well as each individual object occupancy map. Occupancy maps are obtained by integrating the density outputs of the NeRF function along rays going from the image plane to the scene. We observe that each object is rendering a part of the scene corresponding to a single object instance. Outputs of the NeRF function have low resolution and are upscaled by a neural network. The last row shows an example of a bad grouping of two objects in the scene.

Input: Figure 1: Model Overview: We propose INFERNO, a model that infers and renders object-centric 3D scene representations. 1 Our model first decomposes an input observation into multiple object slots. 2 For each slot we infer a structured 3D representation. 3 The shape and appearance determine canonical objects rendered through NeRFs. 4 Objects are transformed and located in the overall scene according to their pose. 5 We combine objects and background and render a low-resolution scene given a camera location. 6 The input is reconstructed by upscaling the low resolution scene.

Input: Figure 6: Novel view synthesis on CLEVR2345. For each scene, we move the camera ±15◦ on the azimuth axis. Additionally, we zoom in the scene twice. While our model is trained with a fixed default camera pose and single scene views, it is able to generalize to small camera pose modifications and render novel views of a scene.

Input: Figure 7: NeRF outputs on CLEVR2345. For each input scene we show the reconstructed image as well as each the low resolution output of the NeRF function. This output is then upscaled with a neural network to obtain the reconstructed scene. While NeRF output lack full detail, they correctly depict each individual object and their position in the scene.

Input: Figure 5: Additional object additions and removals on CLEVR2345. For each scene, we show images with one randomly added object, and with 1-3 random objects removed. Some of these images show out-of-distribution samples with a number of objects not seen during training (2-5 objects).

Input: Figure 8: Shape and appearance disentanglement on CLEVR2345. For each input scene we alter the appearance vector of an object in the scene. First, we show the reconstructed scene. Then, for one of the objects in the scene, we replace its appearance vector by that of another object in the scene, while keeping its shape vector fixed. We perform this operation two times. We observe that, while the shading and color of the object changes as we change the appearance vector, its shape remains constant.

Input: Figure 4: Additional reconstructions on CLEVR2345.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I've carefully reviewed the claim against the extensive evidence provided. The claim states the model can "infer and render 3D scene representations composed of multiple objects, each modeled by an implicit function and explicitly localized in the scene."

The evidence strongly supports this claim through:
- Detailed architectural descriptions showing how objects are modeled using NeRF implicit functions
- Multiple figures demonstrating successful object-centric scene decomposition and rendering
- Examples of object manipulations (removal, addition, translation, scaling) that validate 3D understanding
- Novel view synthesis results showing generalization despite single-view training

The claim appears well-stated and appropriately scoped to the model's demonstrated capabilities, without overstating the results. The limitations are also openly acknowledged in the evidence.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

* We propose a model able to infer and render 3D scene representations composed of multiple objects, each of them modeled by an implicit function and explicitly localized in the scene.

The review comment to be evaluated is:

Strengths:

* an interesting combination of slot attention and NeRF mixture models.
* paper is nicely presented
* results are nice, though see below

Weaknesses:

While I like the paper in general I feel there are several points missing which are probably worth addressing to:

* The objects are transformed using the pose - but that means the appearance is transformed as well - this could be a problem if some of the appearance is pose dependent, for example shading will change, or reflections. Can the authors comment how this is handled here? my guess is that the post rendering network (the upscaling one) takes some of this burden but that's probably not ideal in terms of representation.

* How does the model perform when scenes are rendered from other viewpoints? if the resulting representation is indeed 3D it should be no problem to render the scene from other viewpoints - is that the case here? if not, why? (again, my guess is the upscaling network may be problematic here as well)

* I see that the masks used for estimating segmentation performance (Figure 3) are from the slot attention mechanism - what happens if you use the masks as decoded from the slots (i.e. render the densitiy of each slot separately). This would be more convincing to demonstrate these are "3D" segmentation.

* I feel the experimental validation is quite limited - using only a single dataset (even if it's two variants) is not very convincing as to the generality of the method. How does it work on other datasets? more/less complex.

* I would be happy to see more discussion of the post rendering / upscaling network - this seems like a cruicial part and is not discussed enough.

The evidence to be evaluated is:The goal of our method is to infer object-centric 3D scene representations from single 2D views. Given an image \(x\in\mathbb{R}^{H\times W\times 3}\), we learn an inference function \(f_{\theta}\) that maps images to scene representations \(s=f_{\theta}(x)=(o_{1},o_{2},...,o_{K},o_{bg},c)\). Scenes are composed of \(K\) objects \(o_{i}\), a background object \(o_{bg}\) and a camera location \(c\). Each object is composed of three tensors \(o_{i}=(o_{i}^{shape},o_{i}^{app},o_{i}^{pose})\). The object shape \(o_{i}^{shape}\in\mathbb{R}^{D_{shape}}\) and object appearance \(o^{app}\in\mathbb{R}^{D_{shape}}\) are tensors that respectively describe the 3D shape occupancy and color of an object with an implicit function. The object location \(o_{i}^{pose}\in\mathbb{R}^{4\times 4}\) is an affine matrix that describes the object 3D pose (i.e. scale, translation and rotation) in the scene.

We define a rendering function \(g_{\gamma}\) that takes as input a scene representation and generates a 2D view of that scene \(\hat{x}=g_{\gamma}(f_{\theta}(x))\).

In the following sections we describe in more detail our inference mechanism, our rendering pipeline and their implementation. We represent 3D objects as Neural Radiance Fields (NeRFs) (Mildenhall et al., 2020) with a similar setup as that of GIRAFFE (Niemeyer and Geiger, 2021). A NeRF is a function \(g_{\tau}\) that defines a 3D shape implicitly. It takes as input a 3D location \(\mathbf{l}=(x,y,z)\) and a 2D viewing direction \(\mathbf{d}=(\psi,\phi)\) and outputs an occupancy value \(\sigma\) and a color value \(\mathbf{a}=(r,g,b)\).

To represent multiple 3D shapes with the same NeRF function, we can augment it with latent variables that determine which shape is being modeled (Schwarz et al., 2020).

In INFERNO, we share a single parametrization of a NeRF function across all objects. Each object specific shape and appearance is defined by the shape and appearance latent variables, which correspond to the object attributes \(o^{shape},o^{app}\).

The pose of an object \(o_{i}\) in the scene is determined by the affine transformation matrix \(o_{i}^{pose}\). We denote the coordinate system of the NeRF function of an object as the object space, and the coordinate system of the scene (and the background NeRF) as scene space. Given an object pose, we can convert points from the scene space to the object space by applying the \(o^{pose}\) transformation matrix on those points, and we can transform points from object space to scene space by computing the inverse of the object pose matrix. To render a scene, we cast rays from each pixel in the 2D plane defined by a given camera to the 3D scene. We evaluate NeRFs at different points along a given ray, and integrate their occupancy and color outputs to determine pixel values. Rays might traverse multiple object NeRFs in addition to the background NeRF. To determine the occupancy and color of points described by multiple NeRFs, we first query each NeRF at those particular points. To query the object NeRFs, we first need to transform the points from scene space to the particular object space. Then, we compose the results of each NeRF with a pooling function \(C\), which in our case is a weighted average: \[C(l,\mathbf{d})=(\sigma=\sum_{i=1}^{N}\sigma_{i},\frac{1}{\sigma}\sum_{i=1}^{N }\sigma_{i}\mathbf{a}_{i}) \tag{4}\]

Given the rendering pipeline, the goal of our method is to infer representations that reconstruct a given scene. Our inference mechanism computes image features through a neural network encoder and then extracts \(K\) object and a background slots. These slots are then mapped to our structured scene representation through learned neural networks. To extract image features for each object and background slots, we use Slot Attention (Locatello et al., 2020). Slot Attention is a mechanism that maps a set of \(K\) entities, called slots, to image features without annotations. It extracts image features \(I\in\mathbb{R}^{H\times W\times D}\) from a given input using a resolution-preserving convolutional encoder. These features are then attended to by a set of \(K\) randomly sampled slots \(\pi_{j}\sim\mathcal{N}(\mu,\sigma)\) of dimension \(D\), where \(\mu\) and \(\sigma\) are learnable parameters.

Slots attend spatial chunks of the input features \(I\) through soft-attention \(u=TQ^{T}\), where \(T=k(I)\) and \(Q=q(\pi)\) are the embeddings of the inputs and slots respectively. The attention weights are normalized through a softmax operating on the slots axis, which makes slots compete among themselves and discourages multiple slots from attending the same input region \(w=softmax(u)\). The weighted average of \(I\) according to the attention weights is then computed and fed to a GRU network, to update each slot value: \(\pi_{j}=GRU(w_{j}*I,\pi_{j})\ \forall j\in{1,K}\). Multiple rounds of soft attention are performed to iteratively refine the slots.

Object slots are unstructured tensors that result from aggregating image features. We map these slots to our structured scene representation through small fully-connected networks that operate on individual objects. More concretely, we map object slots to their 3D pose in the scene through a 2-layer MLP. To infer the object shape and appearance tensors we also use 2-layer MLPs, but we make them conditional on the predicted object pose through conditional normalization (Dumoulin et al., 2018).

Recently, the use of functions that implicitly model 3D volumes has gained popularity (Park et al., 2019; Mescheder et al., 2019; Sitzmann et al., 2019; Sitzmann et al., 2020; Schiorek et al., 2021; Pumarola et al., 2021; Yu et al., 2021). Implicit representations have better scaling properties, as usually the output resolution does not directly affect the dimensionality of the learned function. NeRFs (Mildenhall et al., 2020) generate scenes by learning a function that outputs the occupancy and color of points in a scene when viewed from a particular direction. By casting rays through a plane and aggregating the output values NeRFs can generate 2D views of an implicitly modeled 3D scenes. NeRFs have obtained superior reconstructions compared to other implicit methods, and our model uses NeRFs to represent multiple objects and the background of a scene.

GIRAFFE (Niemeyer and Geiger, 2021) is a GAN-based method that represents multiple objects in a scene using NeRFs as part of their generator.

However, we focus on recovering scene representations from existing images, while GIRAFFE does not have an inference mechanism. ObjSURF (Stelzner et al., 2021) and UORF (Yu et al., 2021) infer scene representations composed of multiple objects, each object represented with a differently instantiated NeRF.

Recently, there is a line of work on object-centric generative models of scenes (Kosiorek et al., 2021; Burgess et al., 2019; Locatello et al., 2020; Lin et al., 2020; Greff et al., 2019). These models learn to generate scenes as a composition of multiple objects and a background.

Slot-Attention (Locatello et al., 2020) maps a set of entities, called slots, to image features through multiple rounds of soft attention. The slots compete among themselves to attend to features, making each slot attend to a region of the input image. When driven with alpha-compositing decoder, slot attention learns to segment objects in a scene. Our model uses a variant of Slot Attention to decide on which part of a 2D view should each object in our scene attend to, but we infer 3D-aware representations for each object.

Contrary to most previous approaches which segment 2D shapes, our method infers object-centric scene representations in 3D space.

First, we demonstrate the interpretability of the scene representations through manipulating scenes and verifying the corresponding effects in the rendered outputs. Then we show that it learns to identify and segment the objects in a scene without supervision.

Our model infers 3D object-centric scene representations from single 2D views. These representations disentangle the 3D appearance and pose of objects, which allows for semantic manipulations of the scene not possible otherwise.

In INFERNO, we generate novel scenes by inferring representations for ground-truth images and then manipulating them.

We also investigate different interpretable manipulations of scene representations and visualize the effects in the corresponding output renderings.

We validate that our model is able to render out-of-distribution scenes not corresponding to training examples, such as scenes having 1 or 6 objects, and verify that the pose manipulations have semantically coherent effects.

Additionally, our model can perform inference and manipulations on existing scenes.

Our model is able to generate novel scenes with additional objects or with altered object shapes and appearances, with better FID than GIRAFFE.

Given a scene representation, we can remove or add objects, rearrange object poses, translate the objects to new locations or change the object scales.

Note that this setup evaluates 2D segmentation masks, although our model naturally provides 3D segmentations.

However, our model learns to segment objects in 3D, while the other baselines extract 2D segmentation masks. Figure 3 shows some examples of the discovered object masks. We can see that the model learns to segment different objects and properly discards object slots when the number of objects in the scene is lower than needed.

The scenes show multiple objects that move over time, one being the snitch object.

Each object in our representation is given as an input element to the transformer. We add a learned positional encoding to the object representations based on their frame index. Objects in the same frame have the same positional encoding.

This result suggests that the object-centric representation learned by INFERNO is useful for the visual reasoning task.

This result supports the advantage of 3D-aware representation for solving the CATER task. We propose INFERNO, a model for inferring object-centric 3D scene representations. Our model is able to discover objects in a scene without annotations, and the inferred scene representations are interpretable and amenable to manipulations.

Our model is composed of five main modules: encoder, slot attention, slot to object mapping, NeRF decoder, neural network upscaler.

We employ Slot Attention to map image features to object slots. Slots are sampled randomly from a Gaussian distribution with learned parameters. We use different distributions for the background slot and the object slots.

For each object slot, we first map the object to its pose parameters. We use a 2-layer MLP with LayerNorm to map a slot to its pose.

Once we have inferred the object poses, we infer object shapes and appearances. These are inferred individually for each object using a common 2-layer MLP. The MLPs are conditional on the object pose using Conditional Layer Normalization, that makes the learned parameters of LayerNorm be a function of a condition. Specifically, we map the 7-dimensional pose tensor to LayerNorm parameters with a single linear layer with no activation or normalization. Shapes and appearances are defined by the output of the MLPs, which produce two 128-dimensional tensors. For the background object we only infer shape and appearance, and define its pose to be that of the scene cube. The shape and appearance of the background are inferred through another 2-layer MLP with ReLU and LayerNorm.

With the object shape and appearance tensors we can render them following GRAF (Schwarz et al., 2020). We use one NeRF for the objects and one NeRF MLP for the background.

Note that, to render objects according to their pose, we query their NeRF MLP in a canonical object space by transforming input coordinates in scene space to object space using the object pose.

For generating manipulated scenes we consider the CLEVR2345 dataset (Niemeyer and Geiger, 2021). Images in the CLEVR2345 dataset contain from 2 to 5 objects.

We use 5 objects and rely on the model to not use additional slots if the scene shows less than 5 objects.

We use 6 objects and rely on the model to not use additional slots when needed.

We first pretrain our INFERNO to reconstruct individual frames from the CATER dataset. We bootstrap a model trained on CLEVR6 for object discovery for 400k iterations and train it on the CATER dataset for an additional 100k iterations to speed-up training, as the iteration time of a model with 10 objects is larger.

We show: i) each object rendered individually, ii) object identity swaps with other scenes, iii) object translations along one axis, iv) translations in diagonal (two axis), and v) objects moving in a circle.

While our model is trained without ground-truth camera poses and with single views of scenes, it is able to generalize to small camera pose modifications and render novel views of a scene

While these generations have reduced details due to their resolution, they clearly show the different objects and their location in the scene.

In this section we show the occupancy maps for each rendered object. The occupancy maps are obtained by integrating rays going from the image plane to the scene according to the NeRF density, but without taking into account the RGB output. We observe that each object is rendering a part of the scene corresponding to a single object instance.

To perform unsupervised object discovery we use the outputs of slot attention, since they do not require any upsampling and thus can provide more accurate segmentations.

Input: Figure 3: Object Discovery on CLEVR6: INFERNO identifies the different objects in a scene without supervision. For each input image, we show which regions of the input are attended by each object as well as the background. We include an example of a failed segmentation in the last row, where one object slot (4) is trying to represent multiple objects at the same time.

Input: Figure 2: Manipulations on CLEVR: we show some examples of the manipulations we perform to CLEVR2345 images, including object removal and addition, changing the scale of an object, and object translation. Our model can perform these transformations because it disentangles object pose and appearance.

Input: Figure 9: Object occupancy maps on CLEVR2345. For each input scene we show the reconstructed image as well as each individual object occupancy map. Occupancy maps are obtained by integrating the density outputs of the NeRF function along rays going from the image plane to the scene. We observe that each object is rendering a part of the scene corresponding to a single object instance. Outputs of the NeRF function have low resolution and are upscaled by a neural network. The last row shows an example of a bad grouping of two objects in the scene.

Input: Figure 1: Model Overview: We propose INFERNO, a model that infers and renders object-centric 3D scene representations. 1 Our model first decomposes an input observation into multiple object slots. 2 For each slot we infer a structured 3D representation. 3 The shape and appearance determine canonical objects rendered through NeRFs. 4 Objects are transformed and located in the overall scene according to their pose. 5 We combine objects and background and render a low-resolution scene given a camera location. 6 The input is reconstructed by upscaling the low resolution scene.

Input: Figure 6: Novel view synthesis on CLEVR2345. For each scene, we move the camera ±15◦ on the azimuth axis. Additionally, we zoom in the scene twice. While our model is trained with a fixed default camera pose and single scene views, it is able to generalize to small camera pose modifications and render novel views of a scene.

Input: Figure 7: NeRF outputs on CLEVR2345. For each input scene we show the reconstructed image as well as each the low resolution output of the NeRF function. This output is then upscaled with a neural network to obtain the reconstructed scene. While NeRF output lack full detail, they correctly depict each individual object and their position in the scene.

Input: Figure 5: Additional object additions and removals on CLEVR2345. For each scene, we show images with one randomly added object, and with 1-3 random objects removed. Some of these images show out-of-distribution samples with a number of objects not seen during training (2-5 objects).

Input: Figure 8: Shape and appearance disentanglement on CLEVR2345. For each input scene we alter the appearance vector of an object in the scene. First, we show the reconstructed scene. Then, for one of the objects in the scene, we replace its appearance vector by that of another object in the scene, while keeping its shape vector fixed. We perform this operation two times. We observe that, while the shading and color of the object changes as we change the appearance vector, its shape remains constant.

Input: Figure 4: Additional reconstructions on CLEVR2345.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the extensive evidence, I assess the claim as well-stated. The evidence strongly supports that the model:

1. Successfully infers object-centric representations through unsupervised segmentation (shown in Figure 3's attention masks)
2. Renders 3D scenes by combining individual object NeRFs (demonstrated in Figures 7 and 9's occupancy maps)
3. Explicitly localizes objects through pose parameters (validated by manipulation results in Figure 2 showing object removal, addition, translation)
4. Shows consistent 3D understanding through novel view synthesis (Figure 6)

The claim carefully avoids overstating capabilities and accurately reflects what the model achieves based on the comprehensive evidence provided.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

* We propose a model able to infer and render 3D scene representations composed of multiple objects, each of them modeled by an implicit function and explicitly localized in the scene.

The review comment to be evaluated is:

The paper tackles a very challenging problem, is well-written and easy to follow, and has a good set of experiments for testing object-centric representations. However, my main concern is that the experiments did not focus on testing the '3D-ness' of the inferred representations.
- For scene segmentation, reconstruction, and manipulation, each scene is tested only on a single camera viewpoint. It is unclear whether the learned representations can capture the object appearance from viewpoints different from the input view. Previous work [1, 2] demonstrates this through novel view synthesis.
- The results on the CATER snitch localization task did not strongly support that the '3D' representations learned by the proposed model are more beneficial than 2D representations. One reason might be that this task relies more on long-term reasoning than on 3D reasoning. The authors may want to consider some other tasks that can clearly show the benefit of having 3D representations, like the ones used in [2]. Also, instead of comparing to Aloe, a more direct comparison would be to compare with Slot Attention combined with the same 12-layer transformer used for the proposed model.

**MINOR COMMENTS**
- The rendering function is denoted $g_\gamma$ in Equation 1 but $g_\tau$ in Section 2.1.
- From the paragraph directly below Figure 1, it seems the model can infer the camera location $c$. However, this was not demonstrated in any of the experiments.
- The object NeRFs are shared, which may only be able to deal with similar objects. The paper could be strengthened by experiments on more complex scenes, like the ones used in [1].
- What is the reference frame for the pose vectors? It is mentioned in Appendix B.1 that the manipulations are done by moving the objects forward on the Z or X axes. How are the XYZ axes defined? I would imagine that under different camera viewpoints, the same manipulation would have different effects, but the results in Figure 2 seem quite consistent.

[1] [Decomposing 3D Scenes into Objects via Unsupervised Volume Segmentation](https://arxiv.org/abs/2104.01148).

[2] [ROOTS: Object-Centric Representation and Rendering of 3D Scenes](https://arxiv.org/abs/2006.06130).

The evidence to be evaluated is:The goal of our method is to infer object-centric 3D scene representations from single 2D views. Given an image \(x\in\mathbb{R}^{H\times W\times 3}\), we learn an inference function \(f_{\theta}\) that maps images to scene representations \(s=f_{\theta}(x)=(o_{1},o_{2},...,o_{K},o_{bg},c)\). Scenes are composed of \(K\) objects \(o_{i}\), a background object \(o_{bg}\) and a camera location \(c\). Each object is composed of three tensors \(o_{i}=(o_{i}^{shape},o_{i}^{app},o_{i}^{pose})\). The object shape \(o_{i}^{shape}\in\mathbb{R}^{D_{shape}}\) and object appearance \(o^{app}\in\mathbb{R}^{D_{shape}}\) are tensors that respectively describe the 3D shape occupancy and color of an object with an implicit function. The object location \(o_{i}^{pose}\in\mathbb{R}^{4\times 4}\) is an affine matrix that describes the object 3D pose (i.e. scale, translation and rotation) in the scene.

We define a rendering function \(g_{\gamma}\) that takes as input a scene representation and generates a 2D view of that scene \(\hat{x}=g_{\gamma}(f_{\theta}(x))\).

In the following sections we describe in more detail our inference mechanism, our rendering pipeline and their implementation. We represent 3D objects as Neural Radiance Fields (NeRFs) (Mildenhall et al., 2020) with a similar setup as that of GIRAFFE (Niemeyer and Geiger, 2021). A NeRF is a function \(g_{\tau}\) that defines a 3D shape implicitly. It takes as input a 3D location \(\mathbf{l}=(x,y,z)\) and a 2D viewing direction \(\mathbf{d}=(\psi,\phi)\) and outputs an occupancy value \(\sigma\) and a color value \(\mathbf{a}=(r,g,b)\).

To represent multiple 3D shapes with the same NeRF function, we can augment it with latent variables that determine which shape is being modeled (Schwarz et al., 2020).

In INFERNO, we share a single parametrization of a NeRF function across all objects. Each object specific shape and appearance is defined by the shape and appearance latent variables, which correspond to the object attributes \(o^{shape},o^{app}\).

The pose of an object \(o_{i}\) in the scene is determined by the affine transformation matrix \(o_{i}^{pose}\). We denote the coordinate system of the NeRF function of an object as the object space, and the coordinate system of the scene (and the background NeRF) as scene space. Given an object pose, we can convert points from the scene space to the object space by applying the \(o^{pose}\) transformation matrix on those points, and we can transform points from object space to scene space by computing the inverse of the object pose matrix. To render a scene, we cast rays from each pixel in the 2D plane defined by a given camera to the 3D scene. We evaluate NeRFs at different points along a given ray, and integrate their occupancy and color outputs to determine pixel values. Rays might traverse multiple object NeRFs in addition to the background NeRF. To determine the occupancy and color of points described by multiple NeRFs, we first query each NeRF at those particular points. To query the object NeRFs, we first need to transform the points from scene space to the particular object space. Then, we compose the results of each NeRF with a pooling function \(C\), which in our case is a weighted average: \[C(l,\mathbf{d})=(\sigma=\sum_{i=1}^{N}\sigma_{i},\frac{1}{\sigma}\sum_{i=1}^{N }\sigma_{i}\mathbf{a}_{i}) \tag{4}\]

Given the rendering pipeline, the goal of our method is to infer representations that reconstruct a given scene. Our inference mechanism computes image features through a neural network encoder and then extracts \(K\) object and a background slots. These slots are then mapped to our structured scene representation through learned neural networks. To extract image features for each object and background slots, we use Slot Attention (Locatello et al., 2020). Slot Attention is a mechanism that maps a set of \(K\) entities, called slots, to image features without annotations. It extracts image features \(I\in\mathbb{R}^{H\times W\times D}\) from a given input using a resolution-preserving convolutional encoder. These features are then attended to by a set of \(K\) randomly sampled slots \(\pi_{j}\sim\mathcal{N}(\mu,\sigma)\) of dimension \(D\), where \(\mu\) and \(\sigma\) are learnable parameters.

Slots attend spatial chunks of the input features \(I\) through soft-attention \(u=TQ^{T}\), where \(T=k(I)\) and \(Q=q(\pi)\) are the embeddings of the inputs and slots respectively. The attention weights are normalized through a softmax operating on the slots axis, which makes slots compete among themselves and discourages multiple slots from attending the same input region \(w=softmax(u)\). The weighted average of \(I\) according to the attention weights is then computed and fed to a GRU network, to update each slot value: \(\pi_{j}=GRU(w_{j}*I,\pi_{j})\ \forall j\in{1,K}\). Multiple rounds of soft attention are performed to iteratively refine the slots.

Object slots are unstructured tensors that result from aggregating image features. We map these slots to our structured scene representation through small fully-connected networks that operate on individual objects. More concretely, we map object slots to their 3D pose in the scene through a 2-layer MLP. To infer the object shape and appearance tensors we also use 2-layer MLPs, but we make them conditional on the predicted object pose through conditional normalization (Dumoulin et al., 2018).

Recently, the use of functions that implicitly model 3D volumes has gained popularity (Park et al., 2019; Mescheder et al., 2019; Sitzmann et al., 2019; Sitzmann et al., 2020; Schiorek et al., 2021; Pumarola et al., 2021; Yu et al., 2021). Implicit representations have better scaling properties, as usually the output resolution does not directly affect the dimensionality of the learned function. NeRFs (Mildenhall et al., 2020) generate scenes by learning a function that outputs the occupancy and color of points in a scene when viewed from a particular direction. By casting rays through a plane and aggregating the output values NeRFs can generate 2D views of an implicitly modeled 3D scenes. NeRFs have obtained superior reconstructions compared to other implicit methods, and our model uses NeRFs to represent multiple objects and the background of a scene.

GIRAFFE (Niemeyer and Geiger, 2021) is a GAN-based method that represents multiple objects in a scene using NeRFs as part of their generator.

However, we focus on recovering scene representations from existing images, while GIRAFFE does not have an inference mechanism. ObjSURF (Stelzner et al., 2021) and UORF (Yu et al., 2021) infer scene representations composed of multiple objects, each object represented with a differently instantiated NeRF.

Recently, there is a line of work on object-centric generative models of scenes (Kosiorek et al., 2021; Burgess et al., 2019; Locatello et al., 2020; Lin et al., 2020; Greff et al., 2019). These models learn to generate scenes as a composition of multiple objects and a background.

Slot-Attention (Locatello et al., 2020) maps a set of entities, called slots, to image features through multiple rounds of soft attention. The slots compete among themselves to attend to features, making each slot attend to a region of the input image. When driven with alpha-compositing decoder, slot attention learns to segment objects in a scene. Our model uses a variant of Slot Attention to decide on which part of a 2D view should each object in our scene attend to, but we infer 3D-aware representations for each object.

Contrary to most previous approaches which segment 2D shapes, our method infers object-centric scene representations in 3D space.

First, we demonstrate the interpretability of the scene representations through manipulating scenes and verifying the corresponding effects in the rendered outputs. Then we show that it learns to identify and segment the objects in a scene without supervision.

Our model infers 3D object-centric scene representations from single 2D views. These representations disentangle the 3D appearance and pose of objects, which allows for semantic manipulations of the scene not possible otherwise.

In INFERNO, we generate novel scenes by inferring representations for ground-truth images and then manipulating them.

We also investigate different interpretable manipulations of scene representations and visualize the effects in the corresponding output renderings.

We validate that our model is able to render out-of-distribution scenes not corresponding to training examples, such as scenes having 1 or 6 objects, and verify that the pose manipulations have semantically coherent effects.

Additionally, our model can perform inference and manipulations on existing scenes.

Our model is able to generate novel scenes with additional objects or with altered object shapes and appearances, with better FID than GIRAFFE.

Given a scene representation, we can remove or add objects, rearrange object poses, translate the objects to new locations or change the object scales.

Note that this setup evaluates 2D segmentation masks, although our model naturally provides 3D segmentations.

However, our model learns to segment objects in 3D, while the other baselines extract 2D segmentation masks. Figure 3 shows some examples of the discovered object masks. We can see that the model learns to segment different objects and properly discards object slots when the number of objects in the scene is lower than needed.

The scenes show multiple objects that move over time, one being the snitch object.

Each object in our representation is given as an input element to the transformer. We add a learned positional encoding to the object representations based on their frame index. Objects in the same frame have the same positional encoding.

This result suggests that the object-centric representation learned by INFERNO is useful for the visual reasoning task.

This result supports the advantage of 3D-aware representation for solving the CATER task. We propose INFERNO, a model for inferring object-centric 3D scene representations. Our model is able to discover objects in a scene without annotations, and the inferred scene representations are interpretable and amenable to manipulations.

Our model is composed of five main modules: encoder, slot attention, slot to object mapping, NeRF decoder, neural network upscaler.

We employ Slot Attention to map image features to object slots. Slots are sampled randomly from a Gaussian distribution with learned parameters. We use different distributions for the background slot and the object slots.

For each object slot, we first map the object to its pose parameters. We use a 2-layer MLP with LayerNorm to map a slot to its pose.

Once we have inferred the object poses, we infer object shapes and appearances. These are inferred individually for each object using a common 2-layer MLP. The MLPs are conditional on the object pose using Conditional Layer Normalization, that makes the learned parameters of LayerNorm be a function of a condition. Specifically, we map the 7-dimensional pose tensor to LayerNorm parameters with a single linear layer with no activation or normalization. Shapes and appearances are defined by the output of the MLPs, which produce two 128-dimensional tensors. For the background object we only infer shape and appearance, and define its pose to be that of the scene cube. The shape and appearance of the background are inferred through another 2-layer MLP with ReLU and LayerNorm.

With the object shape and appearance tensors we can render them following GRAF (Schwarz et al., 2020). We use one NeRF for the objects and one NeRF MLP for the background.

Note that, to render objects according to their pose, we query their NeRF MLP in a canonical object space by transforming input coordinates in scene space to object space using the object pose.

For generating manipulated scenes we consider the CLEVR2345 dataset (Niemeyer and Geiger, 2021). Images in the CLEVR2345 dataset contain from 2 to 5 objects.

We use 5 objects and rely on the model to not use additional slots if the scene shows less than 5 objects.

We use 6 objects and rely on the model to not use additional slots when needed.

We first pretrain our INFERNO to reconstruct individual frames from the CATER dataset. We bootstrap a model trained on CLEVR6 for object discovery for 400k iterations and train it on the CATER dataset for an additional 100k iterations to speed-up training, as the iteration time of a model with 10 objects is larger.

We show: i) each object rendered individually, ii) object identity swaps with other scenes, iii) object translations along one axis, iv) translations in diagonal (two axis), and v) objects moving in a circle.

While our model is trained without ground-truth camera poses and with single views of scenes, it is able to generalize to small camera pose modifications and render novel views of a scene

While these generations have reduced details due to their resolution, they clearly show the different objects and their location in the scene.

In this section we show the occupancy maps for each rendered object. The occupancy maps are obtained by integrating rays going from the image plane to the scene according to the NeRF density, but without taking into account the RGB output. We observe that each object is rendering a part of the scene corresponding to a single object instance.

To perform unsupervised object discovery we use the outputs of slot attention, since they do not require any upsampling and thus can provide more accurate segmentations.

Input: Figure 3: Object Discovery on CLEVR6: INFERNO identifies the different objects in a scene without supervision. For each input image, we show which regions of the input are attended by each object as well as the background. We include an example of a failed segmentation in the last row, where one object slot (4) is trying to represent multiple objects at the same time.

Input: Figure 2: Manipulations on CLEVR: we show some examples of the manipulations we perform to CLEVR2345 images, including object removal and addition, changing the scale of an object, and object translation. Our model can perform these transformations because it disentangles object pose and appearance.

Input: Figure 9: Object occupancy maps on CLEVR2345. For each input scene we show the reconstructed image as well as each individual object occupancy map. Occupancy maps are obtained by integrating the density outputs of the NeRF function along rays going from the image plane to the scene. We observe that each object is rendering a part of the scene corresponding to a single object instance. Outputs of the NeRF function have low resolution and are upscaled by a neural network. The last row shows an example of a bad grouping of two objects in the scene.

Input: Figure 1: Model Overview: We propose INFERNO, a model that infers and renders object-centric 3D scene representations. 1 Our model first decomposes an input observation into multiple object slots. 2 For each slot we infer a structured 3D representation. 3 The shape and appearance determine canonical objects rendered through NeRFs. 4 Objects are transformed and located in the overall scene according to their pose. 5 We combine objects and background and render a low-resolution scene given a camera location. 6 The input is reconstructed by upscaling the low resolution scene.

Input: Figure 6: Novel view synthesis on CLEVR2345. For each scene, we move the camera ±15◦ on the azimuth axis. Additionally, we zoom in the scene twice. While our model is trained with a fixed default camera pose and single scene views, it is able to generalize to small camera pose modifications and render novel views of a scene.

Input: Figure 7: NeRF outputs on CLEVR2345. For each input scene we show the reconstructed image as well as each the low resolution output of the NeRF function. This output is then upscaled with a neural network to obtain the reconstructed scene. While NeRF output lack full detail, they correctly depict each individual object and their position in the scene.

Input: Figure 5: Additional object additions and removals on CLEVR2345. For each scene, we show images with one randomly added object, and with 1-3 random objects removed. Some of these images show out-of-distribution samples with a number of objects not seen during training (2-5 objects).

Input: Figure 8: Shape and appearance disentanglement on CLEVR2345. For each input scene we alter the appearance vector of an object in the scene. First, we show the reconstructed scene. Then, for one of the objects in the scene, we replace its appearance vector by that of another object in the scene, while keeping its shape vector fixed. We perform this operation two times. We observe that, while the shading and color of the object changes as we change the appearance vector, its shape remains constant.

Input: Figure 4: Additional reconstructions on CLEVR2345.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
1. The evidence provides substantial support for this claim. The model successfully:
- Decomposes scenes into individual objects and background through slot attention (Fig 3)
- Infers 3D object properties including shape, appearance, and pose (Fig 1 architecture)
- Renders objects individually and composites them into scenes (Fig 7, 9)
- Supports manipulation of object properties while maintaining scene coherence (Fig 2)

However, the reviewer raises valid concerns that 3D capabilities were not fully validated through novel view synthesis tests. While Fig 6 shows some view synthesis ability, testing was limited to small angle changes. The claim is slightly overstated in implying fully general 3D capabilities.

<score>2</score>