Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We present a method to map 2D image observations of a scene to a persistent 3D scene representation, enabling novel view synthesis and disentangled representation of the movable and immovable components of the scene.

The review comment to be evaluated is:

Strengths include:
* The self-supervised training removes the need for GT annotation, e.g., object bounding boxes, for the supported downstream tasks.
* The decomposition of static and movable object feature spaces looks like quite an elegant solution to utilize the inductive bias in multi-view videos while enabling capabilities such as segmentation and object-level scene editing.
* The results overall look promising. Single-view/few-view novel view synthesis is a very challenging task, and the comparisons with PixelNeRF and uORF clearly demonstrated the advantages in synthesis quality. The analysis of adding more views (Fig.10) is strong evidence that the learned 3D representation can figure out how 2D views are related to, and hence help improve, 3D reconstruction.

Weaknesses include:
* There is no study of the generalization capability of the model. It seems likely that the learned models can overfit the limited appearances in the dataset (a few types of shapes/objects with mostly uniform colors).
* Related to the last point, the evaluation is completely done on synthesized datasets, and it's not clear how well the model can learn real-world data with much more diverse appearances. Since the method does not require GT annotations, some evaluation on real-world data does not seem too difficult.
* The are some quality issues with the presented view synthesis results. For example, while unobserved regions are understandably blurry, why are observed regions also not sharp? We can see from Fig.10 that adding more views does not help the quality of observed regions any further.
* Training requires multi-view videos. I can't find clarification on whether these need to be from static cameras. Nothing in the model appears to have such a requirement, but if so, that's a strong limitation. Moreover, why can't the model be trained on frames from a single moving camera?
* It's not clearly described how scene editing is done. It says that's enabled by "directly editing the neural groundplan". Does that mean editing the (movable object) feature space? The feature space is not sparse (i.e. locations without objects have non-zero features), so how does cropping in such space work, and why can that make sense?

The evidence to be evaluated is:Several works have explored learning neural scene representations for downstream tasks in 3D. Emerging neural scene representations enable reconstruction of geometry and appearance from images as well as high-quality novel view synthesis via differentiable rendering.

Hybrid discrete-continuous neural scene representations offer a compromise by factorizing a dense 3D field into several lower-dimensional representations that are used to condition an MLP (Chan et al., 2022; Chen et al., 2022).

We similarly use axis-aligned 2D grids of features for self-supervised scene representation via neural rendering, but reconstruct them directly from few or a single 2D image observations.

In contrast, we present a self-supervised conditional groundplan representation, learned only from images via neural rendering. While we show that our self-supervised representation can be used for rich inference tasks using simple heuristics, our method may be extended for more challenging tasks using the techniques developed in prior work.

Our work is related to prior work on learning to disentangle dynamic objects and static background. Some prior work leverages object motion across video frames to learn separate representations for movable foreground and static background in 2D (Kasten et al., 2021; Ye et al., 2022; Bao et al., 2022), while other recent work can also learn 3D representations (Yuan et al., 2021; Tschernezki et al., 2021). Our approach is similar in using object motion as cue for disentanglement and multi-view as cue for 3D reconstruction, but uses it as supervision to train an encoder-based approach that enables reconstruction from a _single_ image instead of scene-specific disentanglement from multiple video frames.

Prior work has aimed to infer object-centric representations directly from images, with objects either represented as localized object-centric patches (Lin et al., 2020; Eslami et al., 2016; Crawford and Pineau, 2019; Kosiorek et al., 2018; Jiang et al., 2019) or scene mixture components (Engelcke et al., 2020; Burgess et al., 2019; Greff et al., 2019, 2016; 2017; Du et al., 2021), with the slot attention module (Locatello et al., 2020) increasingly driving object-centric inference. Resulting object representations may be decoded into object-centric 3D representations and composed for novel view synthesis (Yu et al., 2022; Smith et al., 2022; Elich et al., 2022; Chen et al., 2021; Bear et al., 2020; Zakharov et al., 2020; Zekharov et al., 2020; Beker et al., 2020; Du et al., 2021).

In this work, we demonstrate that a representation factorized into static and movable 3D regions can serve as a powerful backbone for object discovery.

In this section, we describe the process of inferring a neural ground plan given one or several image observations in a feed-forward manner, as well as subsequent novel view synthesis.

A neural groundplan is a 2D grid of features aligned with the ground plane of the 3D scene, which we define to be the \(xz-\)plane. A 3D point is decoded by projecting it onto the groundplan and retrieving the corresponding feature vector using bilinear interpolation. This feature is then concatenated with the vertical \(y\)-coordinate of the query point and decoded into radiance and density values via a fully connected network, enabling novel view synthesis using volume rendering (Mildenhall et al., 2020).

Inferring a neural groundplan from one or several images proceeds in three steps: (1) feature extraction, (2) feature unprojection, (3) pillar aggregation. Given a single image \(\mathbf{I}\), we first extract per-pixel features via a CNN encoder to yield a feature tensor \(\mathbf{F}\). We define the camera as the world origin and center the neural groundplan accordingly, approximately aligned with the ground level. The image features are unprojected to a 3D feature volume \(\mathbf{v}\) in contracted world space using the inverse of the contract function defined earlier. We extract the feature at a contracted 3D point \(\mathbf{x}^{\prime}\) in \(\mathbf{v}\) as \(\mathbf{v}(\mathbf{x}^{\prime})=\mathbf{F}[\pi(C^{-1}(\mathbf{x}^{\prime}))]\), where \(C^{-1}(\mathbf{x}^{\prime})\) first maps the contracted point to linear world space and \(\pi(\cdot)\) projects it onto the image plane of the context view using camera extrinsics and intrinsics.

We first use a coordinate-encoding MLP \(D(\cdot)\) to transform the volume as \(\mathbf{f}(\mathbf{x}^{\prime})=D(\mathbf{v}(\mathbf{x}^{\prime}),\mathbf{x}_ {\mathbf{c}},\mathbf{d})\), where \(\mathbf{x}_{\mathbf{c}}\) denotes the 3D point in linear camera coordinates of the context camera, and \(\mathbf{d}\) denotes the ray direction from the camera center to that point. Since all features along a camera ray are identical in \(\mathbf{v}\), coordinate encoding is used to add the depth information to the features.

We can render images from novel camera views via differentiable volume rendering (DeVries et al., 2021; Lombardi et al., 2019; Mildenhall et al., 2020).

We now describe training on multi-view video to learn to disentangle static and dynamic components of the scene.

We leverage the fact that objects move in the given multi-view videos as the training signal.

We feed this groundplan into a fully convolutional 2D network, which disentangles it into two separate groundplans containing _static_ and _dynamic_ features. The per-frame static groundplans are mean-pooled to obtain a single, time-independent static groundplan.

To render a scene using the disentangled static and dynamic groundplans, we first decode query points using both groundplans, yielding _two_ sets of (density, color) values for each point. We use the compositing operation proposed by Yuan et al. (2021) to compose the contribution from static and dynamic components along the ray. Given the color and density for static \((\mathbf{c}^{S},\sigma^{S})\) and dynamic \((\mathbf{c}^{D},\sigma^{D})\) parts, the density of the combined scene is calculated as \(\sigma^{S}+\sigma^{D}\). The color at the sampled point is computed as a weighted linear combination \(w^{S}\mathbf{c}^{S}+w^{D}\mathbf{c}^{D}\), where \(w^{S}=(1-\exp(-\delta\sigma^{S}))/(1-\exp(-\delta(\sigma^{S}+\sigma^{D}))\), \(w^{D}=(1-\exp(-\delta\sigma^{D}))/(1-\exp(-\delta(\sigma^{S}+\sigma^{D}))\), and \(\delta\) is the distance between adjacent samples on the camera ray.

We train our model on multi-view video, where multi-view information is used to learn the 3D structure, while motion is used to disentangle the static and dynamic components in the scene.

We use the input images to infer static and dynamic groundplans, and use them to render out per-frame query views.

Our formulation yields a model that maps a single image to two radiance fields, parameterizing static and dynamic 3D regions respectively.

We now perform a search for connected components in the dynamic neural groundplan to perform 3D instance-level segmentation, monocular 3D bounding box prediction, and the extraction of object-centric 3D representations. Specifically, given a dynamic groundplan, we first sample points in a 3D grid around the groundplan origin and decode their densities. We now perform conventional connected-component labeling in the groundplan space using accumulated density values, identifying the disconnected dynamic objects. We perform 2D instance-level segmentation for a queried viewpoint using volume rendering based on the densities expressed by the dynamic groundplan and assigning a color to the points corresponding to each of the identified objects, see Fig. 1 for an example.

Note that our approach is compatible with prior work leveraging slot attention (Locatello et al., 2020; Kipf et al., 2021) and other inference modules, which can be run on the disentangled dynamic groundplan which, in contrast to image space, enables shift-equivariant processing free from perspective distortion and encodes 3D structure.

We demonstrate that our method infers a 3D scene representation from a single image while disentangling static and dynamic components of the scene into static and dynamic groundplans respectively. We then show that connected components analysis suffices to leverage the densities in the dynamic groundplan for instance-level segmentation, bounding box prediction, and scene editing.

We present novel views rendered from groundplans inferred from a single image from CLEVR and CoSY. For single-shot 3D reconstruction and novel view synthesis, we compare against PixelNeRF (Yu et al., 2020), a state-of-the-art single-image 3D reconstruction method, and uORF (Yu et al., 2022), state-of-the-art unsupervised object-centric 3D reconstruction method.

Fig. 4 provides a qualitative comparison to PixelNeRF and uORF in terms of single-image 3D novel view synthesis on both CoSY and CLEVR. Note that our model produces novel views with plausible completions of parts of the scene that are unobserved in the context image such as the back-side of objects.

Given only a single image, our method computes separate static and dynamic groundplans that can be used to individually render the static and movable parts of the scene respectively. Fig. 5 shows results on single-image reconstruction of static and movable scene elements.

The separate reconstruction of movable scene components in the dynamic groundplan enables object detection via instance-level segmentation and bounding box prediction.

Instance-level segmentation, dynamic-static disentanglement, and 3D bounding boxes enable straight-forward 3D editing, such as translation, rotation, deletion, and insertion of individual objects in the scene.

As the dynamic groundplan does not encode static scene regions such as the street below cars, cars can easily be moved from one scene to another.

Our paper demonstrates self-supervised learning of 3D scene representations that are disentangled into movable and immovable scene elements. By leveraging multi-view video at training time, we can reconstruct disentangled 3D representations from a single image observation at test time. We show the potential of neural ground plans as a representation that may enable data-efficient solutions to downstream processing of the 3D scene, such as completion, instance-level segmentation, 3D bounding box prediction, and 3D scene editing. We hope that our paper will inspire future work on the use of self-supervised neural scene representations for general scene understanding tasks.

* Yang et al. (2021a) Bangbang Yang, Yinda Zhang, Yinghao Xu, Yijin Li, Han Zhou, Hujun Bao, Guofeng Zhang, and Zhaopeng Cui. Learning object-compositional neural radiance field for editable scene rendering. In _Proc. ICCV_, pp. 13779-13788, October 2021a.

* Yang et al. (2022) Bangbang Yang, Yinda Zhang, Yijin Li, Zhaopeng Cui, Sean Fanello, Hujun Bao, and Guofeng Zhang. Neural rendering in a room: Amodal 3d understanding and free-viewpoint rendering for the closed scene composed of pre-captured objects.

Multiview neural surface reconstruction by disentangling geometry and appearance.

pixelnerf: Neural radiance fields from one or few images.

Unsupervised discovery of object radiance fields.

Star: Self-supervised tracking and reconstruction of rigid objects in motion with neural rendering.

For a \(128\times 128\) resolution image, this results in a feature grid of \(64\times 64\) resolution with \(512\) features per point. We further use a shallow CNN to reduce the number of feature channels, and to aggregate information along the y-axis. This step helps us reduce the memory footprint for further processing in 3D. This feature aggregation CNN has two convolutional hidden layers with 256 hidden units and a stride of 2 along the height of the feature tensor. The output is a \(16\times 64\) resolution feature grid with \(128\) features per point.

**Unprojection**. We populate a coarse 3D feature volume of shape \(64\times 16\times 64\) with 128-dimensional latents for points in the camera frustum by bilinearly sampling the aggregated encoder feature tensor. This volume is processed as explained in the main paper: a coordinate encoding MLP transforms each feature into a new \(128\) dimensional feature vector. This MLP has two hidden layers with 128 hidden units each and a ReLU activation after each hidden layer.

In case of multi-view inputs, we average the unprojected latents for all 3D points in the canonical 3D coordinate system for all views. This results, like in the monocular case, in a 3D volume of \(128\times 64\times 16\times 64\) (\(CH\times X\times Y\times Z\)).

We use a shallow CNN with 4 hidden convolutional layers to disentangle the entangled groundplan of shape \(128\times 64\times 64\).

This shallow CNN outputs groundplan of shape \(256\times 256\times 256\), in which the first 128 channels of the feature tensor are attributed as the static groundplan and the rest of the 128 channels are used as the dynamic groundplan for that timestep.

Our method performs project of a 3D point on the image plane to get the corresponding feature tensor. This is performed using the intrinsic matrix of the camera. For a given 3D point in camera coordinates \(\mathbf{x_{c}}\) and intrinsic matrix \(\mathbf{K}\), the resulting 2D point on the image plane is computed as \(\mathbf{Kx}\). For projecting a 3D query point \(\mathbf{x}\) on the groundplan, we use grid sample using \((x,z)\)-coordinates of the 3D query points \(\mathbf{x}\).

Our model supports multi-view, as well as monocular reconstruction. We start our training only using multi-view reconstruction with 5 input views for the first 200 epochs, and then switch to a variable mode where the model is given a varying number of input views ranging between 1-5 views. Thiscurriculum approach helps the network to first learn the relevant scene priors, before learning to complete the 3D structure of the scene.

Given the localization in the orthographic bird's-eye view, we can find the height of the objects by computing the depth within each region. This information gives us a 3D bounding box around each of the localized object. The instance level segmentation is produced via volume rendering, by overriding the RGB values within the detected region to a chosen color for the object.

Scene editing is performed by editing the dynamic groundplan. Once the different objects have been localized in the dynamic groundplan, we can edit the dynamic groundplan to carry out object deletion, insertion, and rearrangement. Localizing an object gives us its features in the spatial region of the groundplan used for rendering. To delete the object, we replace the features in this regions with features from the dynamic groundplan that encode zero density. For inserting an object at a given location, we find the corresponding \((x,z)\) location in the dynamic groundplan and set the features at that location to the features corresponding to the object. Rearrangement of objects can be seen as a combination of deletion and insertion where we first delete the object from the existing location, followed by inserting the object at the new location in a possibly new orientation. To perform rotation of an object, we rotate the patch of features that correspond to the object to be rotated.

We generate scenes using the default configuration from (Johnson et al., 2017) (BSD License1), using the rubber material and default object sizes. We render the scene with 6 different cameras, all at the same fixed distance from the origin as the camera used in CLEVR, but with azimuth angles increasing at 60 degree increments. Objects in the CLEVR dataset are captured at 2 timesteps, and are simulated to move a distance of 0.25 to 0.75 meters between the two timesteps.

We generate 15 different configurations of the city using CityEngine, with variations in building shapes, heights, and materials. This city layout is further processed in Blender using Python to add cameras, trees, bus stops, and moving cars. Cameras are sampled on hemispheres of radii in range [4-6m] a maximum height of 4m viewing the center of the circular base of the hemisphere located at randomly chosen points. We sample 15 cameras for each of the randomly chosen centers, with the varying radius of the hemisphere for each camera. The field-of-view for all sampled cameras is 60 degrees, with a symmetric sensor size of 32mm, resulting in a focal length of 110.85 in pixel space.

In addition to the rendering capabilities of CoSY, we add the ability to move cars over timesteps. Given the location and direction of the movement (direction where the car is facing), we change the location of the car over 10 timesteps to a sampled total translation from range 2-4 m.

Fig. 9 presents the output renderings for various target camera viewpoints for the given input image. We observe that as the target view shifts away from the context view, our method renders the geometry with appropriate texture of the car in the view, but outputs a blurry background as expected from a non-generative model. Fig. 10 provides results of street-scale scenes reconstructed using an increasing number of input images from different camera viewpoints. Our method successfully integrates information across observations into a single, multi-view consistent representation. The background reconstructions improve with more input views, as a result of less uncertainty for the occluded regions.

In contrast to single-scene methods, our method needs to fit not only the differentiable rendering in GPU memory, but also the whole inference pipeline - CNNs, 3D lifting, groundplans, etc.

Fig. 11 provides rendered images from edited groundplans where the cars are rotated at different angles.

The dynamic groundplan can be further used to perform motion interpolation in 3D, using optical flow for groundplan interpolation prediction using off-the-shelf, state-of-the-art optical flow and frame interpolation methods.

Firstly, we extracted dynamic groundplan for the samples over different timesteps by passing the multi-view observation through our groundplan generation pipeline. Two dynamic groundplans at different timesteps were given as input to RIFE, and an intermediate timestep was queried.

All losses discussed in the main paper were applied on the rendered novel views generated using the output dynamic floorplans on the intermediate timesteps. In Fig. 12, we show the rendered output of our model for the intermediate timesteps given the leftmost and rightmost frames as input. The proposed method succeeds at inferring the correct object motion, and enables novel view synthesis through space and time.

Input: Figure 1: Given a single image, our model infers separate 3D representations for static and dynamic scene elements, enabling high-quality novel view synthesis with plausible completion, unsupervised instance-level segmentation, 3D bounding box prediction, 3D scene editing, and extraction of objectcentric 3D representations. Our model is trained self-supervised using unlabeled multi-view videos.

Input: Figure 10: Resolving uncertainty with increasing number of context views Our model can take a variable number of viewpoints of a scene as input. We find that the model gets a good prior on the object from a single view but struggles to reconstruct high-quality details in the unobserved regions of the background. This uncertainty gets resolved with more input views.

Input: Figure 11: Scene editing with rotated objects. We present images of scenes with edited groundplans where cars are rotated at various angles in the ground plane.

Input: Figure 12: Temporal 3D scene interpolation for novel view synthesis in space and time.

Input: Figure 4: Qualitative comparisons. Comparison for novel-view synthesis given a single context view with PixelNeRF (Yu et al., 2020) and uORF (Yu et al., 2022).

Input: Figure 5: Single-image reconstruction, disentanglement of static and dynamic objects, and novel view synthesis. Given a single input image, our method can disentangle the observed scene into static and object components based on what the model observed as not-moving and moving in the training data. In these examples, the cars are isolated in the object component as the model was training on video data of cars moving on the road.

Input: Figure 10: Resolving uncertainty with increasing number of context views Our model can take a variable number of viewpoints of a scene as input. We find that the model gets a good prior on the object from a single view but struggles to reconstruct high-quality details in the unobserved regions of the background. This uncertainty gets resolved with more input views.

Input: Figure 6: Object Localization. Given a single input image, we can use our inferred 3D representation of the scene to compute (a) bird’s-eye view rendering, (b) object localization in the bird’s-eye view space, (c) instance-level segmentation, and (d) 3D bounding box prediction.

Input: Figure 7: Qualitative Comparison for instance-level segmentation. Qualitative comparison of instance-level segmentation result with uORF (Yu et al., 2022).

Input: Figure 2: Groundplan inference. Given a context image, we first extract a set of CNN features. We unproject the features into 3D and re-sample them at “pillars” on top of the location of groundplan vertices. Pillars are aggregated into groundplan features using a softmax-weighted sum. The resulting 2D grid of features is decomposed into separate dynamic and static groundplans by a 2D CNN. The coordinate-encoding MLP is not visualized in this figure. Please refer to Sec. 3 for details.

Input: Figure 11: Scene editing with rotated objects. We present images of scenes with edited groundplans where cars are rotated at various angles in the ground plane.

Input: Figure 9: Visual quality of views for a given input image.

Input: Figure 8: Object-centric representations and scene editing. The proposed static-dynamic neural groundplans enables object discovery using connected components labeling on the density of objects encoded in the dynamic groundplan (left). This enables straight-forward scene editing such as deletion, addition, or rigid-body transformation via directly editing the neural groundplan (right).

Input: Figure 12: Temporal 3D scene interpolation for novel view synthesis in space and time.

Input: Figure 3: Learning Static-Dynamic Disentanglement. Given multiple frames of a video, we extract per-frame, compactified, static and dynamic groundplans according to Fig. 2. Static groundplans are pooled into a time-invariant groundplan. We then composite per-frame dynamic and static time-invariant groundplans via differentiable volume rendering. Our model is supervised only via a re-rendering loss on video frames. We encourage the model to explain as much of the scene density as possible with the static groundplan via a sparsity loss on per-frame dynamic volume rendering densities. The surface loss is not visualized here.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
1. The claim appears to be well-stated and accurately reflects what the evidence demonstrates. The paper provides extensive technical details and visual results showing how their method:
- Maps 2D images to 3D scene representations (demonstrated in Figs 1,4,5,9,10)
- Enables novel view synthesis (shown across multiple figures with comparison to baselines)
- Successfully disentangles static and dynamic components (evidenced in Figs 5,6,8 showing separate reconstruction of movable objects and static background)

The evidence comprehensively supports each component of the claim without exaggeration. The reviewer comments also validate the key technical contributions while noting reasonable limitations.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We present a method to map 2D image observations of a scene to a persistent 3D scene representation, enabling novel view synthesis and disentangled representation of the movable and immovable components of the scene.

The review comment to be evaluated is:

Strength：
1. The conditional neural groundplans proposed are intriguing. This allows efficient processing of scene appearance and geometry directly in 3D, through the self-supervised learning of conditional neural groundplans, a hybrid discrete continuous representation of 3D neural scenes.
2. The proposed disentanglement of static background and movable foreground objects is novel. This approach uses object motion for training an encoder-based system that can reconstruct scenes from a single image instead of decomposing video frames based on scene specifics.
3. The proposed method is promising for understanding 3D from a single image. The dynamic groundplan encodes 3D geometry, allowing us to segment and animate objects in 3D using single images and 3D bounding boxes.

Weaknesses：
1. The biggest weakness is that only results on simple synthetic data are presented. Currently, there are a number of well-annotated 3D driving scenes available. An example is KITTI 360 (https://www.cvlibs.net/datasets/kitti-360/). Could the authors explain why results on the real datasets are not presented? Is it because of some specific annotation (such as BEV correspondence or the number of views) needed by the method? Otherwise, the generalizability on natural scenes and the practical value of the method are questioned.
2. I think there are some ambiguity behind the proposed Static-Dynamic Disentanglement. A definition of Static and Dynamic should be made clear in the proposed framework. A car remaining still in the sequence will be considered as Static. Considering a situation when two moving and static cars looks similar or exactly the same, how can we distinguish from a single image if a car is static or dynamic? The method should mention the above cases as an assumption, or making a clear definition of being Static and Dynamic.
3. Groundplans implicitly assume that objects are aligned on a plane. However, in real life, it is common that scenes/objects are not placed on a flat ground plane (e.g., a curved plane). The ability of persistent 3D scene representation is a little overclaimed, if we consider the implicit assumption behind groundplans.

The evidence to be evaluated is:Several works have explored learning neural scene representations for downstream tasks in 3D. Emerging neural scene representations enable reconstruction of geometry and appearance from images as well as high-quality novel view synthesis via differentiable rendering.

Hybrid discrete-continuous neural scene representations offer a compromise by factorizing a dense 3D field into several lower-dimensional representations that are used to condition an MLP (Chan et al., 2022; Chen et al., 2022).

We similarly use axis-aligned 2D grids of features for self-supervised scene representation via neural rendering, but reconstruct them directly from few or a single 2D image observations.

In contrast, we present a self-supervised conditional groundplan representation, learned only from images via neural rendering. While we show that our self-supervised representation can be used for rich inference tasks using simple heuristics, our method may be extended for more challenging tasks using the techniques developed in prior work.

Our work is related to prior work on learning to disentangle dynamic objects and static background. Some prior work leverages object motion across video frames to learn separate representations for movable foreground and static background in 2D (Kasten et al., 2021; Ye et al., 2022; Bao et al., 2022), while other recent work can also learn 3D representations (Yuan et al., 2021; Tschernezki et al., 2021). Our approach is similar in using object motion as cue for disentanglement and multi-view as cue for 3D reconstruction, but uses it as supervision to train an encoder-based approach that enables reconstruction from a _single_ image instead of scene-specific disentanglement from multiple video frames.

Prior work has aimed to infer object-centric representations directly from images, with objects either represented as localized object-centric patches (Lin et al., 2020; Eslami et al., 2016; Crawford and Pineau, 2019; Kosiorek et al., 2018; Jiang et al., 2019) or scene mixture components (Engelcke et al., 2020; Burgess et al., 2019; Greff et al., 2019, 2016; 2017; Du et al., 2021), with the slot attention module (Locatello et al., 2020) increasingly driving object-centric inference. Resulting object representations may be decoded into object-centric 3D representations and composed for novel view synthesis (Yu et al., 2022; Smith et al., 2022; Elich et al., 2022; Chen et al., 2021; Bear et al., 2020; Zakharov et al., 2020; Zekharov et al., 2020; Beker et al., 2020; Du et al., 2021).

In this work, we demonstrate that a representation factorized into static and movable 3D regions can serve as a powerful backbone for object discovery.

In this section, we describe the process of inferring a neural ground plan given one or several image observations in a feed-forward manner, as well as subsequent novel view synthesis.

A neural groundplan is a 2D grid of features aligned with the ground plane of the 3D scene, which we define to be the \(xz-\)plane. A 3D point is decoded by projecting it onto the groundplan and retrieving the corresponding feature vector using bilinear interpolation. This feature is then concatenated with the vertical \(y\)-coordinate of the query point and decoded into radiance and density values via a fully connected network, enabling novel view synthesis using volume rendering (Mildenhall et al., 2020).

Inferring a neural groundplan from one or several images proceeds in three steps: (1) feature extraction, (2) feature unprojection, (3) pillar aggregation. Given a single image \(\mathbf{I}\), we first extract per-pixel features via a CNN encoder to yield a feature tensor \(\mathbf{F}\). We define the camera as the world origin and center the neural groundplan accordingly, approximately aligned with the ground level. The image features are unprojected to a 3D feature volume \(\mathbf{v}\) in contracted world space using the inverse of the contract function defined earlier. We extract the feature at a contracted 3D point \(\mathbf{x}^{\prime}\) in \(\mathbf{v}\) as \(\mathbf{v}(\mathbf{x}^{\prime})=\mathbf{F}[\pi(C^{-1}(\mathbf{x}^{\prime}))]\), where \(C^{-1}(\mathbf{x}^{\prime})\) first maps the contracted point to linear world space and \(\pi(\cdot)\) projects it onto the image plane of the context view using camera extrinsics and intrinsics.

We first use a coordinate-encoding MLP \(D(\cdot)\) to transform the volume as \(\mathbf{f}(\mathbf{x}^{\prime})=D(\mathbf{v}(\mathbf{x}^{\prime}),\mathbf{x}_ {\mathbf{c}},\mathbf{d})\), where \(\mathbf{x}_{\mathbf{c}}\) denotes the 3D point in linear camera coordinates of the context camera, and \(\mathbf{d}\) denotes the ray direction from the camera center to that point. Since all features along a camera ray are identical in \(\mathbf{v}\), coordinate encoding is used to add the depth information to the features.

We can render images from novel camera views via differentiable volume rendering (DeVries et al., 2021; Lombardi et al., 2019; Mildenhall et al., 2020).

We now describe training on multi-view video to learn to disentangle static and dynamic components of the scene.

We leverage the fact that objects move in the given multi-view videos as the training signal.

We feed this groundplan into a fully convolutional 2D network, which disentangles it into two separate groundplans containing _static_ and _dynamic_ features. The per-frame static groundplans are mean-pooled to obtain a single, time-independent static groundplan.

To render a scene using the disentangled static and dynamic groundplans, we first decode query points using both groundplans, yielding _two_ sets of (density, color) values for each point. We use the compositing operation proposed by Yuan et al. (2021) to compose the contribution from static and dynamic components along the ray. Given the color and density for static \((\mathbf{c}^{S},\sigma^{S})\) and dynamic \((\mathbf{c}^{D},\sigma^{D})\) parts, the density of the combined scene is calculated as \(\sigma^{S}+\sigma^{D}\). The color at the sampled point is computed as a weighted linear combination \(w^{S}\mathbf{c}^{S}+w^{D}\mathbf{c}^{D}\), where \(w^{S}=(1-\exp(-\delta\sigma^{S}))/(1-\exp(-\delta(\sigma^{S}+\sigma^{D}))\), \(w^{D}=(1-\exp(-\delta\sigma^{D}))/(1-\exp(-\delta(\sigma^{S}+\sigma^{D}))\), and \(\delta\) is the distance between adjacent samples on the camera ray.

We train our model on multi-view video, where multi-view information is used to learn the 3D structure, while motion is used to disentangle the static and dynamic components in the scene.

We use the input images to infer static and dynamic groundplans, and use them to render out per-frame query views.

Our formulation yields a model that maps a single image to two radiance fields, parameterizing static and dynamic 3D regions respectively.

We now perform a search for connected components in the dynamic neural groundplan to perform 3D instance-level segmentation, monocular 3D bounding box prediction, and the extraction of object-centric 3D representations. Specifically, given a dynamic groundplan, we first sample points in a 3D grid around the groundplan origin and decode their densities. We now perform conventional connected-component labeling in the groundplan space using accumulated density values, identifying the disconnected dynamic objects. We perform 2D instance-level segmentation for a queried viewpoint using volume rendering based on the densities expressed by the dynamic groundplan and assigning a color to the points corresponding to each of the identified objects, see Fig. 1 for an example.

Note that our approach is compatible with prior work leveraging slot attention (Locatello et al., 2020; Kipf et al., 2021) and other inference modules, which can be run on the disentangled dynamic groundplan which, in contrast to image space, enables shift-equivariant processing free from perspective distortion and encodes 3D structure.

We demonstrate that our method infers a 3D scene representation from a single image while disentangling static and dynamic components of the scene into static and dynamic groundplans respectively. We then show that connected components analysis suffices to leverage the densities in the dynamic groundplan for instance-level segmentation, bounding box prediction, and scene editing.

We present novel views rendered from groundplans inferred from a single image from CLEVR and CoSY. For single-shot 3D reconstruction and novel view synthesis, we compare against PixelNeRF (Yu et al., 2020), a state-of-the-art single-image 3D reconstruction method, and uORF (Yu et al., 2022), state-of-the-art unsupervised object-centric 3D reconstruction method.

Fig. 4 provides a qualitative comparison to PixelNeRF and uORF in terms of single-image 3D novel view synthesis on both CoSY and CLEVR. Note that our model produces novel views with plausible completions of parts of the scene that are unobserved in the context image such as the back-side of objects.

Given only a single image, our method computes separate static and dynamic groundplans that can be used to individually render the static and movable parts of the scene respectively. Fig. 5 shows results on single-image reconstruction of static and movable scene elements.

The separate reconstruction of movable scene components in the dynamic groundplan enables object detection via instance-level segmentation and bounding box prediction.

Instance-level segmentation, dynamic-static disentanglement, and 3D bounding boxes enable straight-forward 3D editing, such as translation, rotation, deletion, and insertion of individual objects in the scene.

As the dynamic groundplan does not encode static scene regions such as the street below cars, cars can easily be moved from one scene to another.

Our paper demonstrates self-supervised learning of 3D scene representations that are disentangled into movable and immovable scene elements. By leveraging multi-view video at training time, we can reconstruct disentangled 3D representations from a single image observation at test time. We show the potential of neural ground plans as a representation that may enable data-efficient solutions to downstream processing of the 3D scene, such as completion, instance-level segmentation, 3D bounding box prediction, and 3D scene editing. We hope that our paper will inspire future work on the use of self-supervised neural scene representations for general scene understanding tasks.

* Yang et al. (2021a) Bangbang Yang, Yinda Zhang, Yinghao Xu, Yijin Li, Han Zhou, Hujun Bao, Guofeng Zhang, and Zhaopeng Cui. Learning object-compositional neural radiance field for editable scene rendering. In _Proc. ICCV_, pp. 13779-13788, October 2021a.

* Yang et al. (2022) Bangbang Yang, Yinda Zhang, Yijin Li, Zhaopeng Cui, Sean Fanello, Hujun Bao, and Guofeng Zhang. Neural rendering in a room: Amodal 3d understanding and free-viewpoint rendering for the closed scene composed of pre-captured objects.

Multiview neural surface reconstruction by disentangling geometry and appearance.

pixelnerf: Neural radiance fields from one or few images.

Unsupervised discovery of object radiance fields.

Star: Self-supervised tracking and reconstruction of rigid objects in motion with neural rendering.

For a \(128\times 128\) resolution image, this results in a feature grid of \(64\times 64\) resolution with \(512\) features per point. We further use a shallow CNN to reduce the number of feature channels, and to aggregate information along the y-axis. This step helps us reduce the memory footprint for further processing in 3D. This feature aggregation CNN has two convolutional hidden layers with 256 hidden units and a stride of 2 along the height of the feature tensor. The output is a \(16\times 64\) resolution feature grid with \(128\) features per point.

**Unprojection**. We populate a coarse 3D feature volume of shape \(64\times 16\times 64\) with 128-dimensional latents for points in the camera frustum by bilinearly sampling the aggregated encoder feature tensor. This volume is processed as explained in the main paper: a coordinate encoding MLP transforms each feature into a new \(128\) dimensional feature vector. This MLP has two hidden layers with 128 hidden units each and a ReLU activation after each hidden layer.

In case of multi-view inputs, we average the unprojected latents for all 3D points in the canonical 3D coordinate system for all views. This results, like in the monocular case, in a 3D volume of \(128\times 64\times 16\times 64\) (\(CH\times X\times Y\times Z\)).

We use a shallow CNN with 4 hidden convolutional layers to disentangle the entangled groundplan of shape \(128\times 64\times 64\).

This shallow CNN outputs groundplan of shape \(256\times 256\times 256\), in which the first 128 channels of the feature tensor are attributed as the static groundplan and the rest of the 128 channels are used as the dynamic groundplan for that timestep.

Our method performs project of a 3D point on the image plane to get the corresponding feature tensor. This is performed using the intrinsic matrix of the camera. For a given 3D point in camera coordinates \(\mathbf{x_{c}}\) and intrinsic matrix \(\mathbf{K}\), the resulting 2D point on the image plane is computed as \(\mathbf{Kx}\). For projecting a 3D query point \(\mathbf{x}\) on the groundplan, we use grid sample using \((x,z)\)-coordinates of the 3D query points \(\mathbf{x}\).

Our model supports multi-view, as well as monocular reconstruction. We start our training only using multi-view reconstruction with 5 input views for the first 200 epochs, and then switch to a variable mode where the model is given a varying number of input views ranging between 1-5 views. Thiscurriculum approach helps the network to first learn the relevant scene priors, before learning to complete the 3D structure of the scene.

Given the localization in the orthographic bird's-eye view, we can find the height of the objects by computing the depth within each region. This information gives us a 3D bounding box around each of the localized object. The instance level segmentation is produced via volume rendering, by overriding the RGB values within the detected region to a chosen color for the object.

Scene editing is performed by editing the dynamic groundplan. Once the different objects have been localized in the dynamic groundplan, we can edit the dynamic groundplan to carry out object deletion, insertion, and rearrangement. Localizing an object gives us its features in the spatial region of the groundplan used for rendering. To delete the object, we replace the features in this regions with features from the dynamic groundplan that encode zero density. For inserting an object at a given location, we find the corresponding \((x,z)\) location in the dynamic groundplan and set the features at that location to the features corresponding to the object. Rearrangement of objects can be seen as a combination of deletion and insertion where we first delete the object from the existing location, followed by inserting the object at the new location in a possibly new orientation. To perform rotation of an object, we rotate the patch of features that correspond to the object to be rotated.

We generate scenes using the default configuration from (Johnson et al., 2017) (BSD License1), using the rubber material and default object sizes. We render the scene with 6 different cameras, all at the same fixed distance from the origin as the camera used in CLEVR, but with azimuth angles increasing at 60 degree increments. Objects in the CLEVR dataset are captured at 2 timesteps, and are simulated to move a distance of 0.25 to 0.75 meters between the two timesteps.

We generate 15 different configurations of the city using CityEngine, with variations in building shapes, heights, and materials. This city layout is further processed in Blender using Python to add cameras, trees, bus stops, and moving cars. Cameras are sampled on hemispheres of radii in range [4-6m] a maximum height of 4m viewing the center of the circular base of the hemisphere located at randomly chosen points. We sample 15 cameras for each of the randomly chosen centers, with the varying radius of the hemisphere for each camera. The field-of-view for all sampled cameras is 60 degrees, with a symmetric sensor size of 32mm, resulting in a focal length of 110.85 in pixel space.

In addition to the rendering capabilities of CoSY, we add the ability to move cars over timesteps. Given the location and direction of the movement (direction where the car is facing), we change the location of the car over 10 timesteps to a sampled total translation from range 2-4 m.

Fig. 9 presents the output renderings for various target camera viewpoints for the given input image. We observe that as the target view shifts away from the context view, our method renders the geometry with appropriate texture of the car in the view, but outputs a blurry background as expected from a non-generative model. Fig. 10 provides results of street-scale scenes reconstructed using an increasing number of input images from different camera viewpoints. Our method successfully integrates information across observations into a single, multi-view consistent representation. The background reconstructions improve with more input views, as a result of less uncertainty for the occluded regions.

In contrast to single-scene methods, our method needs to fit not only the differentiable rendering in GPU memory, but also the whole inference pipeline - CNNs, 3D lifting, groundplans, etc.

Fig. 11 provides rendered images from edited groundplans where the cars are rotated at different angles.

The dynamic groundplan can be further used to perform motion interpolation in 3D, using optical flow for groundplan interpolation prediction using off-the-shelf, state-of-the-art optical flow and frame interpolation methods.

Firstly, we extracted dynamic groundplan for the samples over different timesteps by passing the multi-view observation through our groundplan generation pipeline. Two dynamic groundplans at different timesteps were given as input to RIFE, and an intermediate timestep was queried.

All losses discussed in the main paper were applied on the rendered novel views generated using the output dynamic floorplans on the intermediate timesteps. In Fig. 12, we show the rendered output of our model for the intermediate timesteps given the leftmost and rightmost frames as input. The proposed method succeeds at inferring the correct object motion, and enables novel view synthesis through space and time.

Input: Figure 1: Given a single image, our model infers separate 3D representations for static and dynamic scene elements, enabling high-quality novel view synthesis with plausible completion, unsupervised instance-level segmentation, 3D bounding box prediction, 3D scene editing, and extraction of objectcentric 3D representations. Our model is trained self-supervised using unlabeled multi-view videos.

Input: Figure 10: Resolving uncertainty with increasing number of context views Our model can take a variable number of viewpoints of a scene as input. We find that the model gets a good prior on the object from a single view but struggles to reconstruct high-quality details in the unobserved regions of the background. This uncertainty gets resolved with more input views.

Input: Figure 11: Scene editing with rotated objects. We present images of scenes with edited groundplans where cars are rotated at various angles in the ground plane.

Input: Figure 12: Temporal 3D scene interpolation for novel view synthesis in space and time.

Input: Figure 4: Qualitative comparisons. Comparison for novel-view synthesis given a single context view with PixelNeRF (Yu et al., 2020) and uORF (Yu et al., 2022).

Input: Figure 5: Single-image reconstruction, disentanglement of static and dynamic objects, and novel view synthesis. Given a single input image, our method can disentangle the observed scene into static and object components based on what the model observed as not-moving and moving in the training data. In these examples, the cars are isolated in the object component as the model was training on video data of cars moving on the road.

Input: Figure 10: Resolving uncertainty with increasing number of context views Our model can take a variable number of viewpoints of a scene as input. We find that the model gets a good prior on the object from a single view but struggles to reconstruct high-quality details in the unobserved regions of the background. This uncertainty gets resolved with more input views.

Input: Figure 6: Object Localization. Given a single input image, we can use our inferred 3D representation of the scene to compute (a) bird’s-eye view rendering, (b) object localization in the bird’s-eye view space, (c) instance-level segmentation, and (d) 3D bounding box prediction.

Input: Figure 7: Qualitative Comparison for instance-level segmentation. Qualitative comparison of instance-level segmentation result with uORF (Yu et al., 2022).

Input: Figure 2: Groundplan inference. Given a context image, we first extract a set of CNN features. We unproject the features into 3D and re-sample them at “pillars” on top of the location of groundplan vertices. Pillars are aggregated into groundplan features using a softmax-weighted sum. The resulting 2D grid of features is decomposed into separate dynamic and static groundplans by a 2D CNN. The coordinate-encoding MLP is not visualized in this figure. Please refer to Sec. 3 for details.

Input: Figure 11: Scene editing with rotated objects. We present images of scenes with edited groundplans where cars are rotated at various angles in the ground plane.

Input: Figure 9: Visual quality of views for a given input image.

Input: Figure 8: Object-centric representations and scene editing. The proposed static-dynamic neural groundplans enables object discovery using connected components labeling on the density of objects encoded in the dynamic groundplan (left). This enables straight-forward scene editing such as deletion, addition, or rigid-body transformation via directly editing the neural groundplan (right).

Input: Figure 12: Temporal 3D scene interpolation for novel view synthesis in space and time.

Input: Figure 3: Learning Static-Dynamic Disentanglement. Given multiple frames of a video, we extract per-frame, compactified, static and dynamic groundplans according to Fig. 2. Static groundplans are pooled into a time-invariant groundplan. We then composite per-frame dynamic and static time-invariant groundplans via differentiable volume rendering. Our model is supervised only via a re-rendering loss on video frames. We encourage the model to explain as much of the scene density as possible with the static groundplan via a sparsity loss on per-frame dynamic volume rendering densities. The surface loss is not visualized here.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on the extensive evidence provided, I'll evaluate the claim: "We present a method to map 2D image observations of a scene to a persistent 3D scene representation, enabling novel view synthesis and disentangled representation of the movable and immovable components of the scene."

The evidence strongly supports this claim through:
- Detailed technical implementation showing successful mapping from 2D to 3D via neural groundplans
- Multiple figures demonstrating effective novel view synthesis
- Clear examples of disentangling static and dynamic scene elements
- Results showing successful separation of movable objects (cars) from immovable background

However, there are some limitations noted in the reviewer comments:
- Only tested on synthetic data
- Assumes objects aligned to ground plane
- Potential ambiguity in static/dynamic classification

While these are real limitations, they don't invalidate the core claim, which is carefully scoped to describe what the method accomplishes without overstating its capabilities or generalizability.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Motivated by the bird's-eye-view (BEV) representation commonly used in vision and robotics, we propose _conditional neural groundplans_, ground-aligned 2D feature grids, as persistent and memory-efficient scene representations.

The evidence to be evaluated is:Several works have explored learning neural scene representations for downstream tasks in 3D. Emerging neural scene representations enable reconstruction of geometry and appearance from images as well as high-quality novel view synthesis via differentiable rendering.

In particular, neural groundplans and axis-aligned 2D grids enable high-quality unconditional generation of 3D scenes (DeVries et al., 2021; Chan et al., 2022) as well as reconstruction of 3D geometry from pointclouds (Peng et al., 2020). We similarly use axis-aligned 2D grids of features for self-supervised scene representation via neural rendering, but reconstruct them directly from few or a single 2D image observations.

Bird's-eye view has been explored as a 3D representation in vision and robotics, particularly for autonomous driving applications. Prior work uses ground-plane 2D grids as representations for object detection and segmentation (Saha et al., 2022; Harley et al., 2022; Philion and Fidler, 2020; Reiher et al., 2020; Roddick et al., 2019), layout generation and completion (Cao and de Charette, 2022; Jeong et al., 2022; Mani et al., 2020; Yang et al., 2021), and next-frame prediction (Hu et al., 2021; Zieba et al., 2020). The bird's-eye view is generated either directly without 3D inductive biases (Mani et al., 2020), or similar to our proposed approach, by using 3D geometry-driven inductive biases such as unprojection into a volume (Harley et al., 2022; Chen et al., 2022; Roddick et al., 2019), or by generating a 3D point cloud (Philion and Fidler, 2020; Hu et al., 2021).

In contrast, we present a self-supervised conditional groundplan representation, learned only from images via neural rendering.

In this section, we describe the process of inferring a neural ground plan given one or several image observations in a feed-forward manner, as well as subsequent novel view synthesis.

A neural groundplan is a 2D grid of features aligned with the ground plane of the 3D scene, which we define to be the \(xz-\)plane. A 3D point is decoded by projecting it onto the groundplan and retrieving the corresponding feature vector using bilinear interpolation. This feature is then concatenated with the vertical \(y\)-coordinate of the query point and decoded into radiance and density values via a fully connected network, enabling novel view synthesis using volume rendering (Mildenhall et al., 2020).

Points \(\mathbf{x}\) within a radius \(r_{\text{inner}}\) around the groundplan origin remain unaffected, but points outside this radius are contracted. For any 3D point \(\mathbf{x}\), the contracted 3D coordinate can be computed as \(\mathbf{x}^{\prime}=C(\mathbf{x})=((1+k)-k/||\mathbf{u}||)(\mathbf{u}/|| \mathbf{u}||)r_{\text{inner}}\), where \(\mathbf{u}=\mathbf{x}/r_{\text{inner}}\), and \(k\) is a hyperparameter which controls the size of the contracted region. Note that \(C\) is invertible, such that \(\mathbf{x}=C^{-1}(\mathbf{x}^{\prime})\) is a function that takes a 3D point in contracted space \(\mathbf{x}^{\prime}\) to the original 3D point \(\mathbf{x}\) in linear space.

Inferring a neural groundplan from one or several images proceeds in three steps: (1) feature extraction, (2) feature unprojection, (3) pillar aggregation. Given a single image \(\mathbf{I}\), we first extract per-pixel features via a CNN encoder to yield a feature tensor \(\mathbf{F}\). We define the camera as the world origin and center the neural groundplan accordingly, approximately aligned with the ground level. The image features are unprojected to a 3D feature volume \(\mathbf{v}\) in contracted world space using the inverse of the contract function defined earlier. We extract the feature at a contracted 3D point \(\mathbf{x}^{\prime}\) in \(\mathbf{v}\) as \(\mathbf{v}(\mathbf{x}^{\prime})=\mathbf{F}[\pi(C^{-1}(\mathbf{x}^{\prime}))]\), where \(C^{-1}(\mathbf{x}^{\prime})\) first maps the contracted point to linear world space and \(\pi(\cdot)\) projects it onto the image plane of the context view using camera extrinsics and intrinsics. At any vertex of the groundplan, the discretized \(y\)-coordinates of the volume form a "pillar". Next, we aggregate each pillar into a point to create the 2D groundplan. We first use a coordinate-encoding MLP \(D(\cdot)\) to transform the volume as \(\mathbf{f}(\mathbf{x}^{\prime})=D(\mathbf{v}(\mathbf{x}^{\prime}),\mathbf{x}_ {\mathbf{c}},\mathbf{d})\), where \(\mathbf{x}_{\mathbf{c}}\) denotes the 3D point in linear camera coordinates of the context camera, and \(\mathbf{d}\) denotes the ray direction from the camera center to that point. Since all features along a camera ray are identical in \(\mathbf{v}\), coordinate encoding is used to add the depth information to the features. In the case of multi-view input images, the volumes corresponding to each input view are mean pooled. Associated to each 2D vertex of the groundplan is now a set of features \(\{\mathbf{f}_{i}\}_{i=1}^{N}\), where \(N\) is the number of samples along the \(y\)-dimension. We use a "pillar-aggregation" MLP to compute softmax scores as \(\alpha_{i}=P(\mathbf{f}_{i},\mathbf{x}_{i})\), where \(P(\cdot)\) denotes the MLP and \(\mathbf{x}_{i}\) is the linear coordinate of the \(i\)-th point on the pillar. Finally, the features are aggregated by computing the weighted sum of the features, \(\mathbf{g}=\sum_{i}\alpha_{i}\mathbf{f}_{i}\).

For each frame, we infer an _entangled_ neural groundplan as described in the previous section. Features in this entangled neural groundplan parameterize both static and dynamic features of the scene, for instance, a car as well as the road below it. We feed this groundplan into a fully convolutional 2D network, which disentangles it into two separate groundplans containing _static_ and _dynamic_ features.

Our formulation yields a model that maps a single image to two radiance fields, parameterizing static and dynamic 3D regions respectively.

We now perform a search for connected components in the dynamic neural groundplan to perform 3D instance-level segmentation, monocular 3D bounding box prediction, and the extraction of object-centric 3D representations. Specifically, given a dynamic groundplan, we first sample points in a 3D grid around the groundplan origin and decode their densities. We now perform conventional connected-component labeling in the groundplan space using accumulated density values, identifying the disconnected dynamic objects. We perform 2D instance-level segmentation for a queried viewpoint using volume rendering based on the densities expressed by the dynamic groundplan and assigning a color to the points corresponding to each of the identified objects, see Fig. 1 for an example.

Note that our approach is compatible with prior work leveraging slot attention (Locatello et al., 2020; Kipf et al., 2021) and other inference modules, which can be run on the disentangled dynamic groundplan which, in contrast to image space, enables shift-equivariant processing free from perspective distortion and encodes 3D structure.

We demonstrate that our method infers a 3D scene representation from a single image while disentangling static and dynamic components of the scene into static and dynamic groundplans respectively.

We present novel views rendered from groundplans inferred from a single image from CLEVR and CoSY.

Given only a single image, our method computes separate static and dynamic groundplans that can be used to individually render the static and movable parts of the scene respectively.

Note that cars are reliably encoded by the dynamic groundplan, and our method inpaints regions occluded in the input view.

The separate reconstruction of movable scene components in the dynamic groundplan enables object detection via instance-level segmentation and bounding box prediction. Fig. 6 presents the instance-level segmentation and 3D bounding box prediction results of the proposed 3D object discovery via connected component discovery using the density inferred using the dynamic groundplan from the bird's-eye view.

As the dynamic groundplan does not encode static scene regions such as the street below cars, cars can easily be moved from one scene to another.

We show the potential of neural ground plans as a representation that may enable data-efficient solutions to downstream processing of the 3D scene, such as completion, instance-level segmentation, 3D bounding box prediction, and 3D scene editing. We hope that our paper will inspire future work on the use of self-supervised neural scene representations for general scene understanding tasks.

Following PixelNeRF, we use the first 4 convolutional blocks of the ResNet34 architecture to create a feature grid. For a \(128\times 128\) resolution image, this results in a feature grid of \(64\times 64\) resolution with \(512\) features per point. We further use a shallow CNN to reduce the number of feature channels, and to aggregate information along the y-axis. This step helps us reduce the memory footprint for further processing in 3D. This feature aggregation CNN has two convolutional hidden layers with 256 hidden units and a stride of 2 along the height of the feature tensor. The output is a \(16\times 64\) resolution feature grid with \(128\) features per point.

To construct an entangled groundplan, we aggregate the 3D feature volume into a 2D groundplan using a pillar aggregation MLP. Consider a pillar of features orthogonal to the groundplan for a particular point \((x,z)\), in our case that is \(128\times 16\). The MLP takes in each latent with its corresponding 3D coordinate in the world coordinate system. It outputs the softmax scores for each of these latents which can be used to sum these latents along the pillar into a single 128-dimensional latent, as explained in the main paper.

We use a shallow CNN with 4 hidden convolutional layers to disentangle the entangled groundplan of shape \(128\times 64\times 64\).

This shallow CNN outputs groundplan of shape \(256\times 256\times 256\), in which the first 128 channels of the feature tensor are attributed as the static groundplan and the rest of the 128 channels are used as the dynamic groundplan for that timestep.

For projecting a 3D query point \(\mathbf{x}\) on the groundplan, we use grid sample using \((x,z)\)-coordinates of the 3D query points \(\mathbf{x}\).

For localization of objects, we consider the dynamic groundplan and render it from an orthographic bird's-eye view. Along with the bird's-eye view RGB image, we also generate the occupancy map (per-pixel accumulated alpha) as shown in the manuscript in Fig. 5. Since the groundplans have a spatial extent of \(256\times 256\), the phplic occupancy map is rendered at the same resolution.

Given the localization in the orthographic bird's-eye view, we can find the height of the objects by computing the depth within each region.

These predictions can be made more accurate, for example, by (1) further tuning these hyperparameters, (2) sampling the orthographic density map at a higher resolution, (3) increasing the resolution of the groundplans at training time, and (4) training with more samples per camera ray. Scene editing is performed by editing the dynamic groundplan. Once the different objects have been localized in the dynamic groundplan, we can edit the dynamic groundplan to carry out object deletion, insertion, and rearrangement. Localizing an object gives us its features in the spatial region of the groundplan used for rendering. To delete the object, we replace the features in this regions with features from the dynamic groundplan that encode zero density. For inserting an object at a given location, we find the corresponding \((x,z)\) location in the dynamic groundplan and set the features at that location to the features corresponding to the object. Rearrangement of objects can be seen as a combination of deletion and insertion where we first delete the object from the existing location, followed by inserting the object at the new location in a possibly new orientation.

Our method successfully integrates information across observations into a single, multi-view consistent representation. The background reconstructions improve with more input views, as a result of less uncertainty for the occluded regions.

In contrast to single-scene methods, our method needs to fit not only the differentiable rendering in GPU memory, but also the whole inference pipeline - CNNs, 3D lifting, groundplans, etc. This limits the resolution of the groundplans (a car is expressed by \(\sim 6\) latents) as well as the number of volume rendering samples.

Fig. 11 provides rendered images from edited groundplans where the cars are rotated at different angles. Note that the representation only allows for rotations in the \(xz\)-plane. The dynamic groundplan can be further used to perform motion interpolation in 3D, using optical flow for groundplan interpolation prediction using off-the-shelf, state-of-the-art optical flow and frame interpolation methods.

We first trained our model on the GQN dataset and used the output dynamic groundplans to train the frame interpolation method.

Firstly, we extracted dynamic groundplan for the samples over different timesteps by passing the multi-view observation through our groundplan generation pipeline. Two dynamic groundplans at different timesteps were given as input to RIFE, and an intermediate timestep was queried.

Input: Figure 6: Object Localization. Given a single input image, we can use our inferred 3D representation of the scene to compute (a) bird’s-eye view rendering, (b) object localization in the bird’s-eye view space, (c) instance-level segmentation, and (d) 3D bounding box prediction.

Input: Figure 7: Qualitative Comparison for instance-level segmentation. Qualitative comparison of instance-level segmentation result with uORF (Yu et al., 2022).

Input: Figure 2: Groundplan inference. Given a context image, we first extract a set of CNN features. We unproject the features into 3D and re-sample them at “pillars” on top of the location of groundplan vertices. Pillars are aggregated into groundplan features using a softmax-weighted sum. The resulting 2D grid of features is decomposed into separate dynamic and static groundplans by a 2D CNN. The coordinate-encoding MLP is not visualized in this figure. Please refer to Sec. 3 for details.

Input: Figure 11: Scene editing with rotated objects. We present images of scenes with edited groundplans where cars are rotated at various angles in the ground plane.

Input: Figure 8: Object-centric representations and scene editing. The proposed static-dynamic neural groundplans enables object discovery using connected components labeling on the density of objects encoded in the dynamic groundplan (left). This enables straight-forward scene editing such as deletion, addition, or rigid-body transformation via directly editing the neural groundplan (right).

Input: Figure 3: Learning Static-Dynamic Disentanglement. Given multiple frames of a video, we extract per-frame, compactified, static and dynamic groundplans according to Fig. 2. Static groundplans are pooled into a time-invariant groundplan. We then composite per-frame dynamic and static time-invariant groundplans via differentiable volume rendering. Our model is supervised only via a re-rendering loss on video frames. We encourage the model to explain as much of the scene density as possible with the static groundplan via a sparsity loss on per-frame dynamic volume rendering densities. The surface loss is not visualized here.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find it to be well-stated and accurately reflects the capabilities and limitations of the proposed conditional neural groundplans approach. The evidence thoroughly demonstrates how the method:
- Uses bird's-eye-view representation through 2D feature grids
- Shows persistence through static-dynamic disentanglement
- Demonstrates memory efficiency through its compact 2D representation
- Functions effectively as a scene representation for various tasks

The extensive technical details, implementation specifics, and experimental results directly support the key aspects mentioned in the claim without overstating capabilities. The authors are also transparent about limitations like resolution constraints and memory requirements.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Motivated by the bird's-eye-view (BEV) representation commonly used in vision and robotics, we propose _conditional neural groundplans_, ground-aligned 2D feature grids, as persistent and memory-efficient scene representations.

The review comment to be evaluated is:

Strength:

1. The authors tackle a challenging yet valuable problem, disentanglement of the movable and immovable components of the scene from a single image. They propose a neural scene representation called conditional neural groundplans and a self-supervised framework to disentangle static background and movable foreground objects given only a single image. The framework enables a variety of downstream tasks, such as extraction of object-centric 3D representations, novel view synthesis, instance-level segmentation, 3D bounding box prediction, and scene editing. The direction would interest a wide range of communities, including 3D scene understanding, 3D neural representation, disentangled representation learning, and novel view synthesis.
2. The proposed 3D persistent scene representation learning framework elegantly leverages ideas from BEV representation, neural representation learning, volume rendering, and compositing operation (Yuan et al., 2021). Moreover, the proposed method utilizes multi-view videos (in a self-supervised manner) at training time to learn to separately reconstruct static and movable components of the scene from a single image at test time. The authors present a self-supervised neural scene representation framework for general scene understanding tasks. The reviewer found the work valuable to the community.
3. The proposed 3D scene representations enable novel view synthesis and disentangle movable and immovable components of the scene via diverse qualitative evaluation on two datasets, CLEVR and CoSY. In particular, the qualitative results shown on CoSY are impressive, as shown in the manuscript and supplementary website.

Weaknesses:

1. Persistent scene representation: By sufficient qualitative analysis, the proposed 3D scene representations enable novel view synthesis and disentangle movable and immovable elements for different scene understanding tasks. One failure case shown on the top row of p.17 is that the framework cannot effectively synthesize the viewing perspective behind the box with a green door. The framework seems unable to capture the background correctly, yielding incorrect synthesis. Moreover, the box's location seems incorrect from the images on the left. The reviewer wonder if the performance can be improved with additional observations.
2. The authors propose to have two separate groundplans neural representations for static and dynamic, respectively. As we know from the literature (e.g., Yuan et al., 2021), it is possible to keep separate 3D neural representations for novel view synthesis. It seems reasonable to tackle the goal of this paper, i.e., achieve decent performance on a variety of tasks from a single image, with separate 3D neural representations based on the proposed framework. It would be great for the authors to elaborate on the technical challenge of this direction. That could motivate the community to tackle the direction collectively.
3. Differentiable rendering based on neural groundplans feature: the reviewer found it less intuitive to understand why the differentiable rendering would work using the groundplan features g_x. Specifically, g_x is obtained based on the weighted sum of the features along the y-axis. The groundplan feature g_x contains information in a particular pillar. Why is it a better feature for volume rendering? Empirically, the design choice renders better quality than STOA methods (i.e., PixelNeRF and uORF). Could the authors please elaborate more on this design choice?
4. The design choice of BEV representation: The authors utilize a particular operation to form BEV representations, i.e., the weighted sum of the features. However, there are other approaches, e.g., Roddick and Cipolla, 2020, Philion and Fidler, 2020 and Saha et al., 2022. It would be valuable for the audience to learn the difference between these operations for representation learning.
5. The reviewer is interested in the qualitative performance of novel view synthesis without using the contracted coordinate. With the analysis, readers would be able to visualize the difference.

The evidence to be evaluated is:Several works have explored learning neural scene representations for downstream tasks in 3D. Emerging neural scene representations enable reconstruction of geometry and appearance from images as well as high-quality novel view synthesis via differentiable rendering.

In particular, neural groundplans and axis-aligned 2D grids enable high-quality unconditional generation of 3D scenes (DeVries et al., 2021; Chan et al., 2022) as well as reconstruction of 3D geometry from pointclouds (Peng et al., 2020). We similarly use axis-aligned 2D grids of features for self-supervised scene representation via neural rendering, but reconstruct them directly from few or a single 2D image observations.

Bird's-eye view has been explored as a 3D representation in vision and robotics, particularly for autonomous driving applications. Prior work uses ground-plane 2D grids as representations for object detection and segmentation (Saha et al., 2022; Harley et al., 2022; Philion and Fidler, 2020; Reiher et al., 2020; Roddick et al., 2019), layout generation and completion (Cao and de Charette, 2022; Jeong et al., 2022; Mani et al., 2020; Yang et al., 2021), and next-frame prediction (Hu et al., 2021; Zieba et al., 2020). The bird's-eye view is generated either directly without 3D inductive biases (Mani et al., 2020), or similar to our proposed approach, by using 3D geometry-driven inductive biases such as unprojection into a volume (Harley et al., 2022; Chen et al., 2022; Roddick et al., 2019), or by generating a 3D point cloud (Philion and Fidler, 2020; Hu et al., 2021).

In contrast, we present a self-supervised conditional groundplan representation, learned only from images via neural rendering.

In this section, we describe the process of inferring a neural ground plan given one or several image observations in a feed-forward manner, as well as subsequent novel view synthesis.

A neural groundplan is a 2D grid of features aligned with the ground plane of the 3D scene, which we define to be the \(xz-\)plane. A 3D point is decoded by projecting it onto the groundplan and retrieving the corresponding feature vector using bilinear interpolation. This feature is then concatenated with the vertical \(y\)-coordinate of the query point and decoded into radiance and density values via a fully connected network, enabling novel view synthesis using volume rendering (Mildenhall et al., 2020).

Points \(\mathbf{x}\) within a radius \(r_{\text{inner}}\) around the groundplan origin remain unaffected, but points outside this radius are contracted. For any 3D point \(\mathbf{x}\), the contracted 3D coordinate can be computed as \(\mathbf{x}^{\prime}=C(\mathbf{x})=((1+k)-k/||\mathbf{u}||)(\mathbf{u}/|| \mathbf{u}||)r_{\text{inner}}\), where \(\mathbf{u}=\mathbf{x}/r_{\text{inner}}\), and \(k\) is a hyperparameter which controls the size of the contracted region. Note that \(C\) is invertible, such that \(\mathbf{x}=C^{-1}(\mathbf{x}^{\prime})\) is a function that takes a 3D point in contracted space \(\mathbf{x}^{\prime}\) to the original 3D point \(\mathbf{x}\) in linear space.

Inferring a neural groundplan from one or several images proceeds in three steps: (1) feature extraction, (2) feature unprojection, (3) pillar aggregation. Given a single image \(\mathbf{I}\), we first extract per-pixel features via a CNN encoder to yield a feature tensor \(\mathbf{F}\). We define the camera as the world origin and center the neural groundplan accordingly, approximately aligned with the ground level. The image features are unprojected to a 3D feature volume \(\mathbf{v}\) in contracted world space using the inverse of the contract function defined earlier. We extract the feature at a contracted 3D point \(\mathbf{x}^{\prime}\) in \(\mathbf{v}\) as \(\mathbf{v}(\mathbf{x}^{\prime})=\mathbf{F}[\pi(C^{-1}(\mathbf{x}^{\prime}))]\), where \(C^{-1}(\mathbf{x}^{\prime})\) first maps the contracted point to linear world space and \(\pi(\cdot)\) projects it onto the image plane of the context view using camera extrinsics and intrinsics. At any vertex of the groundplan, the discretized \(y\)-coordinates of the volume form a "pillar". Next, we aggregate each pillar into a point to create the 2D groundplan. We first use a coordinate-encoding MLP \(D(\cdot)\) to transform the volume as \(\mathbf{f}(\mathbf{x}^{\prime})=D(\mathbf{v}(\mathbf{x}^{\prime}),\mathbf{x}_ {\mathbf{c}},\mathbf{d})\), where \(\mathbf{x}_{\mathbf{c}}\) denotes the 3D point in linear camera coordinates of the context camera, and \(\mathbf{d}\) denotes the ray direction from the camera center to that point. Since all features along a camera ray are identical in \(\mathbf{v}\), coordinate encoding is used to add the depth information to the features. In the case of multi-view input images, the volumes corresponding to each input view are mean pooled. Associated to each 2D vertex of the groundplan is now a set of features \(\{\mathbf{f}_{i}\}_{i=1}^{N}\), where \(N\) is the number of samples along the \(y\)-dimension. We use a "pillar-aggregation" MLP to compute softmax scores as \(\alpha_{i}=P(\mathbf{f}_{i},\mathbf{x}_{i})\), where \(P(\cdot)\) denotes the MLP and \(\mathbf{x}_{i}\) is the linear coordinate of the \(i\)-th point on the pillar. Finally, the features are aggregated by computing the weighted sum of the features, \(\mathbf{g}=\sum_{i}\alpha_{i}\mathbf{f}_{i}\).

For each frame, we infer an _entangled_ neural groundplan as described in the previous section. Features in this entangled neural groundplan parameterize both static and dynamic features of the scene, for instance, a car as well as the road below it. We feed this groundplan into a fully convolutional 2D network, which disentangles it into two separate groundplans containing _static_ and _dynamic_ features.

Our formulation yields a model that maps a single image to two radiance fields, parameterizing static and dynamic 3D regions respectively.

We now perform a search for connected components in the dynamic neural groundplan to perform 3D instance-level segmentation, monocular 3D bounding box prediction, and the extraction of object-centric 3D representations. Specifically, given a dynamic groundplan, we first sample points in a 3D grid around the groundplan origin and decode their densities. We now perform conventional connected-component labeling in the groundplan space using accumulated density values, identifying the disconnected dynamic objects. We perform 2D instance-level segmentation for a queried viewpoint using volume rendering based on the densities expressed by the dynamic groundplan and assigning a color to the points corresponding to each of the identified objects, see Fig. 1 for an example.

Note that our approach is compatible with prior work leveraging slot attention (Locatello et al., 2020; Kipf et al., 2021) and other inference modules, which can be run on the disentangled dynamic groundplan which, in contrast to image space, enables shift-equivariant processing free from perspective distortion and encodes 3D structure.

We demonstrate that our method infers a 3D scene representation from a single image while disentangling static and dynamic components of the scene into static and dynamic groundplans respectively.

We present novel views rendered from groundplans inferred from a single image from CLEVR and CoSY.

Given only a single image, our method computes separate static and dynamic groundplans that can be used to individually render the static and movable parts of the scene respectively.

Note that cars are reliably encoded by the dynamic groundplan, and our method inpaints regions occluded in the input view.

The separate reconstruction of movable scene components in the dynamic groundplan enables object detection via instance-level segmentation and bounding box prediction. Fig. 6 presents the instance-level segmentation and 3D bounding box prediction results of the proposed 3D object discovery via connected component discovery using the density inferred using the dynamic groundplan from the bird's-eye view.

As the dynamic groundplan does not encode static scene regions such as the street below cars, cars can easily be moved from one scene to another.

We show the potential of neural ground plans as a representation that may enable data-efficient solutions to downstream processing of the 3D scene, such as completion, instance-level segmentation, 3D bounding box prediction, and 3D scene editing. We hope that our paper will inspire future work on the use of self-supervised neural scene representations for general scene understanding tasks.

Following PixelNeRF, we use the first 4 convolutional blocks of the ResNet34 architecture to create a feature grid. For a \(128\times 128\) resolution image, this results in a feature grid of \(64\times 64\) resolution with \(512\) features per point. We further use a shallow CNN to reduce the number of feature channels, and to aggregate information along the y-axis. This step helps us reduce the memory footprint for further processing in 3D. This feature aggregation CNN has two convolutional hidden layers with 256 hidden units and a stride of 2 along the height of the feature tensor. The output is a \(16\times 64\) resolution feature grid with \(128\) features per point.

To construct an entangled groundplan, we aggregate the 3D feature volume into a 2D groundplan using a pillar aggregation MLP. Consider a pillar of features orthogonal to the groundplan for a particular point \((x,z)\), in our case that is \(128\times 16\). The MLP takes in each latent with its corresponding 3D coordinate in the world coordinate system. It outputs the softmax scores for each of these latents which can be used to sum these latents along the pillar into a single 128-dimensional latent, as explained in the main paper.

We use a shallow CNN with 4 hidden convolutional layers to disentangle the entangled groundplan of shape \(128\times 64\times 64\).

This shallow CNN outputs groundplan of shape \(256\times 256\times 256\), in which the first 128 channels of the feature tensor are attributed as the static groundplan and the rest of the 128 channels are used as the dynamic groundplan for that timestep.

For projecting a 3D query point \(\mathbf{x}\) on the groundplan, we use grid sample using \((x,z)\)-coordinates of the 3D query points \(\mathbf{x}\).

For localization of objects, we consider the dynamic groundplan and render it from an orthographic bird's-eye view. Along with the bird's-eye view RGB image, we also generate the occupancy map (per-pixel accumulated alpha) as shown in the manuscript in Fig. 5. Since the groundplans have a spatial extent of \(256\times 256\), the phplic occupancy map is rendered at the same resolution.

Given the localization in the orthographic bird's-eye view, we can find the height of the objects by computing the depth within each region.

These predictions can be made more accurate, for example, by (1) further tuning these hyperparameters, (2) sampling the orthographic density map at a higher resolution, (3) increasing the resolution of the groundplans at training time, and (4) training with more samples per camera ray. Scene editing is performed by editing the dynamic groundplan. Once the different objects have been localized in the dynamic groundplan, we can edit the dynamic groundplan to carry out object deletion, insertion, and rearrangement. Localizing an object gives us its features in the spatial region of the groundplan used for rendering. To delete the object, we replace the features in this regions with features from the dynamic groundplan that encode zero density. For inserting an object at a given location, we find the corresponding \((x,z)\) location in the dynamic groundplan and set the features at that location to the features corresponding to the object. Rearrangement of objects can be seen as a combination of deletion and insertion where we first delete the object from the existing location, followed by inserting the object at the new location in a possibly new orientation.

Our method successfully integrates information across observations into a single, multi-view consistent representation. The background reconstructions improve with more input views, as a result of less uncertainty for the occluded regions.

In contrast to single-scene methods, our method needs to fit not only the differentiable rendering in GPU memory, but also the whole inference pipeline - CNNs, 3D lifting, groundplans, etc. This limits the resolution of the groundplans (a car is expressed by \(\sim 6\) latents) as well as the number of volume rendering samples.

Fig. 11 provides rendered images from edited groundplans where the cars are rotated at different angles. Note that the representation only allows for rotations in the \(xz\)-plane. The dynamic groundplan can be further used to perform motion interpolation in 3D, using optical flow for groundplan interpolation prediction using off-the-shelf, state-of-the-art optical flow and frame interpolation methods.

We first trained our model on the GQN dataset and used the output dynamic groundplans to train the frame interpolation method.

Firstly, we extracted dynamic groundplan for the samples over different timesteps by passing the multi-view observation through our groundplan generation pipeline. Two dynamic groundplans at different timesteps were given as input to RIFE, and an intermediate timestep was queried.

Input: Figure 6: Object Localization. Given a single input image, we can use our inferred 3D representation of the scene to compute (a) bird’s-eye view rendering, (b) object localization in the bird’s-eye view space, (c) instance-level segmentation, and (d) 3D bounding box prediction.

Input: Figure 7: Qualitative Comparison for instance-level segmentation. Qualitative comparison of instance-level segmentation result with uORF (Yu et al., 2022).

Input: Figure 2: Groundplan inference. Given a context image, we first extract a set of CNN features. We unproject the features into 3D and re-sample them at “pillars” on top of the location of groundplan vertices. Pillars are aggregated into groundplan features using a softmax-weighted sum. The resulting 2D grid of features is decomposed into separate dynamic and static groundplans by a 2D CNN. The coordinate-encoding MLP is not visualized in this figure. Please refer to Sec. 3 for details.

Input: Figure 11: Scene editing with rotated objects. We present images of scenes with edited groundplans where cars are rotated at various angles in the ground plane.

Input: Figure 8: Object-centric representations and scene editing. The proposed static-dynamic neural groundplans enables object discovery using connected components labeling on the density of objects encoded in the dynamic groundplan (left). This enables straight-forward scene editing such as deletion, addition, or rigid-body transformation via directly editing the neural groundplan (right).

Input: Figure 3: Learning Static-Dynamic Disentanglement. Given multiple frames of a video, we extract per-frame, compactified, static and dynamic groundplans according to Fig. 2. Static groundplans are pooled into a time-invariant groundplan. We then composite per-frame dynamic and static time-invariant groundplans via differentiable volume rendering. Our model is supervised only via a re-rendering loss on video frames. We encourage the model to explain as much of the scene density as possible with the static groundplan via a sparsity loss on per-frame dynamic volume rendering densities. The surface loss is not visualized here.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate the claim: "Motivated by the bird's-eye-view (BEV) representation commonly used in vision and robotics, we propose conditional neural groundplans, ground-aligned 2D feature grids, as persistent and memory-efficient scene representations."

Based on the extensive evidence provided, this claim appears well-stated (score of 1) for several reasons:
- The evidence demonstrates that BEV representations are indeed common in vision/robotics, with multiple citations showing its established use
- The ground-aligned 2D feature grids are thoroughly detailed in implementation
- The persistence and memory efficiency are supported by technical details about the architecture and demonstrated through various applications (scene editing, object detection, etc.)
- The reviewer comments affirm the technical soundness and value of the approach

The claim avoids overstatement by staying focused on the core technical contribution without making grandiose claims about capabilities or impact.

<score>1</score>