Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Neural groundplans are a hybrid discrete-continuous 3D neural scene representation (Chan et al., 2022; Peng et al., 2020; Philion and Fidler, 2020; Roddick et al., 2019; Mani et al., 2020) and enable 3D queries by projecting a 3D point onto the groundplan, retrieving the respective feature, and decoding it via an MLP into a full 3D scene.

The evidence to be evaluated is:Hybrid discrete-continuous neural scene representations offer a compromise by factorizing a dense 3D field into several lower-dimensional representations that are used to condition an MLP (Chan et al., 2022; Chen et al., 2022). In particular, neural groundplans and axis-aligned 2D grids enable high-quality unconditional generation of 3D scenes (DeVries et al., 2021; Chan et al., 2022) as well as reconstruction of 3D geometry from pointclouds (Peng et al., 2020).

Prior work uses ground-plane 2D grids as representations for object detection and segmentation (Saha et al., 2022; Harley et al., 2022; Philion and Fidler, 2020; Reiher et al., 2020; Roddick et al., 2019), layout generation and completion (Cao and de Charette, 2022; Jeong et al., 2022; Mani et al., 2020; Yang et al., 2021), and next-frame prediction (Hu et al., 2021; Zieba et al., 2020). The bird's-eye view is generated either directly without 3D inductive biases (Mani et al., 2020), or similar to our proposed approach, by using 3D geometry-driven inductive biases such as unprojection into a volume (Harley et al., 2022; Chen et al., 2022; Roddick et al., 2019), or by generating a 3D point cloud (Philion and Fidler, 2020; Hu et al., 2021).

In contrast, we present a self-supervised conditional groundplan representation, learned only from images via neural rendering.

A neural groundplan is a 2D grid of features aligned with the ground plane of the 3D scene, which we define to be the \(xz-\)plane. A 3D point is decoded by projecting it onto the groundplan and retrieving the corresponding feature vector using bilinear interpolation. This feature is then concatenated with the vertical \(y\)-coordinate of the query point and decoded into radiance and density values via a fully connected network, enabling novel view synthesis using volume rendering (Mildenhall et al., 2020).

Inferring a neural groundplan from one or several images proceeds in three steps: (1) feature extraction, (2) feature unprojection, (3) pillar aggregation. Given a single image \(\mathbf{I}\), we first extract per-pixel features via a CNN encoder to yield a feature tensor \(\mathbf{F}\). We define the camera as the world origin and center the neural groundplan accordingly, approximately aligned with the ground level. The image features are unprojected to a 3D feature volume \(\mathbf{v}\) in contracted world space using the inverse of the contract function defined earlier. We extract the feature at a contracted 3D point \(\mathbf{x}^{\prime}\) in \(\mathbf{v}\) as \(\mathbf{v}(\mathbf{x}^{\prime})=\mathbf{F}[\pi(C^{-1}(\mathbf{x}^{\prime}))]\), where \(C^{-1}(\mathbf{x}^{\prime})\) first maps the contracted point to linear world space and \(\pi(\cdot)\) projects it onto the image plane of the context view using camera extrinsics and intrinsics.

Next, we aggregate each pillar into a point to create the 2D groundplan.

Associated to each 2D vertex of the groundplan is now a set of features \(\{\mathbf{f}_{i}\}_{i=1}^{N}\), where \(N\) is the number of samples along the \(y\)-dimension. We use a "pillar-aggregation" MLP to compute softmax scores as \(\alpha_{i}=P(\mathbf{f}_{i},\mathbf{x}_{i})\), where \(P(\cdot)\) denotes the MLP and \(\mathbf{x}_{i}\) is the linear coordinate of the \(i\)-th point on the pillar. Finally, the features are aggregated by computing the weighted sum of the features, \(\mathbf{g}=\sum_{i}\alpha_{i}\mathbf{f}_{i}\).

For each frame, we infer an _entangled_ neural groundplan as described in the previous section. Features in this entangled neural groundplan parameterize both static and dynamic features of the scene, for instance, a car as well as the road below it. We feed this groundplan into a fully convolutional 2D network, which disentangles it into two separate groundplans containing _static_ and _dynamic_ features.

To render a scene using the disentangled static and dynamic groundplans, we first decode query points using both groundplans, yielding _two_ sets of (density, color) values for each point.

We use the input images to infer static and dynamic groundplans, and use them to render out per-frame query views.

Our formulation yields a model that maps a single image to two radiance fields, parameterizing static and dynamic 3D regions respectively.

We now perform a search for connected components in the dynamic neural groundplan to perform 3D instance-level segmentation, monocular 3D bounding box prediction, and the extraction of object-centric 3D representations. Specifically, given a dynamic groundplan, we first sample points in a 3D grid around the groundplan origin and decode their densities. We now perform conventional connected-component labeling in the groundplan space using accumulated density values, identifying the disconnected dynamic objects.

Note that our approach is compatible with prior work leveraging slot attention (Locatello et al., 2020; Kipf et al., 2021) and other inference modules, which can be run on the disentangled dynamic groundplan which, in contrast to image space, enables shift-equivariant processing free from perspective distortion and encodes 3D structure.

We demonstrate that our method infers a 3D scene representation from a single image while disentangling static and dynamic components of the scene into static and dynamic groundplans respectively.

Given only a single image, our method computes separate static and dynamic groundplans that can be used to individually render the static and movable parts of the scene respectively.

The separate reconstruction of movable scene components in the dynamic groundplan enables object detection via instance-level segmentation and bounding box prediction. Fig. 6 presents the instance-level segmentation and 3D bounding box prediction results of the proposed 3D object discovery via connected component discovery using the density inferred using the dynamic groundplan from the bird's-eye view.

Objects can be rotated by arbitrary angles by simple bilinear interpolation of the groundplan features (refer to Appendix B.3). As the dynamic groundplan does not encode static scene regions such as the street below cars, cars can easily be moved from one scene to another.

Future work may further explore the use of more sophisticated downstream processing of the groundplan to enable, for instance, prior-based inference of object-centric representations (Locatello et al., 2020).

We show the potential of neural ground plans as a representation that may enable data-efficient solutions to downstream processing of the 3D scene, such as completion, instance-level segmentation, 3D bounding box prediction, and 3D scene editing.

We populate a coarse 3D feature volume of shape \(64\times 16\times 64\) with 128-dimensional latents for points in the camera frustum by bilinearly sampling the aggregated encoder feature tensor. This volume is processed as explained in the main paper: a coordinate encoding MLP transforms each feature into a new \(128\) dimensional feature vector. This MLP has two hidden layers with 128 hidden units each and a ReLU activation after each hidden layer.

To construct an entangled groundplan, we aggregate the 3D feature volume into a 2D groundplan using a pillar aggregation MLP. Consider a pillar of features orthogonal to the groundplan for a particular point \((x,z)\), in our case that is \(128\times 16\). The MLP takes in each latent with its corresponding 3D coordinate in the world coordinate system. It outputs the softmax scores for each of these latents which can be used to sum these latents along the pillar into a single 128-dimensional latent, as explained in the main paper.

This shallow CNN outputs groundplan of shape \(256\times 256\times 256\), in which the first 128 channels of the feature tensor are attributed as the static groundplan and the rest of the 128 channels are used as the dynamic groundplan for that timestep.

Our method performs project of a 3D point on the image plane to get the corresponding feature tensor.

For projecting a 3D query point \(\mathbf{x}\) on the groundplan, we use grid sample using \((x,z)\)-coordinates of the 3D query points \(\mathbf{x}\).

For localization of objects, we consider the dynamic groundplan and render it from an orthographic bird's-eye view. Along with the bird's-eye view RGB image, we also generate the occupancy map (per-pixel accumulated alpha) as shown in the manuscript in Fig. 5.

Given the localization in the orthographic bird's-eye view, we can find the height of the objects by computing the depth within each region. This information gives us a 3D bounding box around each of the localized object.

Scene editing is performed by editing the dynamic groundplan. Once the different objects have been localized in the dynamic groundplan, we can edit the dynamic groundplan to carry out object deletion, insertion, and rearrangement. Localizing an object gives us its features in the spatial region of the groundplan used for rendering. To delete the object, we replace the features in this regions with features from the dynamic groundplan that encode zero density. For inserting an object at a given location, we find the corresponding \((x,z)\) location in the dynamic groundplan and set the features at that location to the features corresponding to the object. Rearrangement of objects can be seen as a combination of deletion and insertion where we first delete the object from the existing location, followed by inserting the object at the new location in a possibly new orientation. To perform rotation of an object, we rotate the patch of features that correspond to the object to be rotated.

In contrast to single-scene methods, our method needs to fit not only the differentiable rendering in GPU memory, but also the whole inference pipeline - CNNs, 3D lifting, groundplans, etc. This limits the resolution of the groundplans (a car is expressed by \(\sim 6\) latents) as well as the number of volume rendering samples.

Fig. 11 provides rendered images from edited groundplans where the cars are rotated at different angles. Note that the representation only allows for rotations in the \(xz\)-plane.

We first trained our model on the GQN dataset and used the output dynamic groundplans to train the frame interpolation method.

Firstly, we extracted dynamic groundplan for the samples over different timesteps by passing the multi-view observation through our groundplan generation pipeline. Two dynamic groundplans at different timesteps were given as input to RIFE, and an intermediate timestep was queried.

Input: Figure 2: Groundplan inference. Given a context image, we first extract a set of CNN features. We unproject the features into 3D and re-sample them at “pillars” on top of the location of groundplan vertices. Pillars are aggregated into groundplan features using a softmax-weighted sum. The resulting 2D grid of features is decomposed into separate dynamic and static groundplans by a 2D CNN. The coordinate-encoding MLP is not visualized in this figure. Please refer to Sec. 3 for details.

Input: Figure 11: Scene editing with rotated objects. We present images of scenes with edited groundplans where cars are rotated at various angles in the ground plane.

Input: Figure 3: Learning Static-Dynamic Disentanglement. Given multiple frames of a video, we extract per-frame, compactified, static and dynamic groundplans according to Fig. 2. Static groundplans are pooled into a time-invariant groundplan. We then composite per-frame dynamic and static time-invariant groundplans via differentiable volume rendering. Our model is supervised only via a re-rendering loss on video frames. We encourage the model to explain as much of the scene density as possible with the static groundplan via a sparsity loss on per-frame dynamic volume rendering densities. The surface loss is not visualized here.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find that it is generally well-stated but has some minor overstatement. The evidence confirms that neural groundplans are indeed hybrid discrete-continuous 3D representations that enable 3D queries through projection and MLP decoding. However, the evidence shows more complexity in the process, including feature extraction, unprojection, and pillar aggregation steps not mentioned in the claim. The claim also doesn't acknowledge limitations like resolution constraints and restricted rotations in the xz-plane only.

The evidence supports the core technical description but simplifies the full pipeline somewhat. The cited papers are accurately referenced, though the claim could better reflect the nuanced implementation details shown in the evidence.

<score>2</score>

Justification:
1. The core mechanism is accurately described and supported by detailed evidence, especially in the technical implementation sections.
2. The evidence shows additional complexity in the process (e.g., "feature extraction, feature unprojection, pillar aggregation") not captured in the simplified claim.
3. Implementation limitations are not acknowledged (e.g., "limits the resolution of the groundplans" and "only allows for rotations in the xz-plane").
4. The cited papers are appropriately referenced and support the basic concept.
5. While slightly simplified, the claim doesn't substantially exaggerate capabilities or results.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Neural groundplans are a hybrid discrete-continuous 3D neural scene representation (Chan et al., 2022; Peng et al., 2020; Philion and Fidler, 2020; Roddick et al., 2019; Mani et al., 2020) and enable 3D queries by projecting a 3D point onto the groundplan, retrieving the respective feature, and decoding it via an MLP into a full 3D scene.

The review comment to be evaluated is:

Strength:

1. The authors tackle a challenging yet valuable problem, disentanglement of the movable and immovable components of the scene from a single image. They propose a neural scene representation called conditional neural groundplans and a self-supervised framework to disentangle static background and movable foreground objects given only a single image. The framework enables a variety of downstream tasks, such as extraction of object-centric 3D representations, novel view synthesis, instance-level segmentation, 3D bounding box prediction, and scene editing. The direction would interest a wide range of communities, including 3D scene understanding, 3D neural representation, disentangled representation learning, and novel view synthesis.
2. The proposed 3D persistent scene representation learning framework elegantly leverages ideas from BEV representation, neural representation learning, volume rendering, and compositing operation (Yuan et al., 2021). Moreover, the proposed method utilizes multi-view videos (in a self-supervised manner) at training time to learn to separately reconstruct static and movable components of the scene from a single image at test time. The authors present a self-supervised neural scene representation framework for general scene understanding tasks. The reviewer found the work valuable to the community.
3. The proposed 3D scene representations enable novel view synthesis and disentangle movable and immovable components of the scene via diverse qualitative evaluation on two datasets, CLEVR and CoSY. In particular, the qualitative results shown on CoSY are impressive, as shown in the manuscript and supplementary website.

Weaknesses:

1. Persistent scene representation: By sufficient qualitative analysis, the proposed 3D scene representations enable novel view synthesis and disentangle movable and immovable elements for different scene understanding tasks. One failure case shown on the top row of p.17 is that the framework cannot effectively synthesize the viewing perspective behind the box with a green door. The framework seems unable to capture the background correctly, yielding incorrect synthesis. Moreover, the box's location seems incorrect from the images on the left. The reviewer wonder if the performance can be improved with additional observations.
2. The authors propose to have two separate groundplans neural representations for static and dynamic, respectively. As we know from the literature (e.g., Yuan et al., 2021), it is possible to keep separate 3D neural representations for novel view synthesis. It seems reasonable to tackle the goal of this paper, i.e., achieve decent performance on a variety of tasks from a single image, with separate 3D neural representations based on the proposed framework. It would be great for the authors to elaborate on the technical challenge of this direction. That could motivate the community to tackle the direction collectively.
3. Differentiable rendering based on neural groundplans feature: the reviewer found it less intuitive to understand why the differentiable rendering would work using the groundplan features g_x. Specifically, g_x is obtained based on the weighted sum of the features along the y-axis. The groundplan feature g_x contains information in a particular pillar. Why is it a better feature for volume rendering? Empirically, the design choice renders better quality than STOA methods (i.e., PixelNeRF and uORF). Could the authors please elaborate more on this design choice?
4. The design choice of BEV representation: The authors utilize a particular operation to form BEV representations, i.e., the weighted sum of the features. However, there are other approaches, e.g., Roddick and Cipolla, 2020, Philion and Fidler, 2020 and Saha et al., 2022. It would be valuable for the audience to learn the difference between these operations for representation learning.
5. The reviewer is interested in the qualitative performance of novel view synthesis without using the contracted coordinate. With the analysis, readers would be able to visualize the difference.

The evidence to be evaluated is:Hybrid discrete-continuous neural scene representations offer a compromise by factorizing a dense 3D field into several lower-dimensional representations that are used to condition an MLP (Chan et al., 2022; Chen et al., 2022). In particular, neural groundplans and axis-aligned 2D grids enable high-quality unconditional generation of 3D scenes (DeVries et al., 2021; Chan et al., 2022) as well as reconstruction of 3D geometry from pointclouds (Peng et al., 2020).

Prior work uses ground-plane 2D grids as representations for object detection and segmentation (Saha et al., 2022; Harley et al., 2022; Philion and Fidler, 2020; Reiher et al., 2020; Roddick et al., 2019), layout generation and completion (Cao and de Charette, 2022; Jeong et al., 2022; Mani et al., 2020; Yang et al., 2021), and next-frame prediction (Hu et al., 2021; Zieba et al., 2020). The bird's-eye view is generated either directly without 3D inductive biases (Mani et al., 2020), or similar to our proposed approach, by using 3D geometry-driven inductive biases such as unprojection into a volume (Harley et al., 2022; Chen et al., 2022; Roddick et al., 2019), or by generating a 3D point cloud (Philion and Fidler, 2020; Hu et al., 2021).

In contrast, we present a self-supervised conditional groundplan representation, learned only from images via neural rendering.

A neural groundplan is a 2D grid of features aligned with the ground plane of the 3D scene, which we define to be the \(xz-\)plane. A 3D point is decoded by projecting it onto the groundplan and retrieving the corresponding feature vector using bilinear interpolation. This feature is then concatenated with the vertical \(y\)-coordinate of the query point and decoded into radiance and density values via a fully connected network, enabling novel view synthesis using volume rendering (Mildenhall et al., 2020).

Inferring a neural groundplan from one or several images proceeds in three steps: (1) feature extraction, (2) feature unprojection, (3) pillar aggregation. Given a single image \(\mathbf{I}\), we first extract per-pixel features via a CNN encoder to yield a feature tensor \(\mathbf{F}\). We define the camera as the world origin and center the neural groundplan accordingly, approximately aligned with the ground level. The image features are unprojected to a 3D feature volume \(\mathbf{v}\) in contracted world space using the inverse of the contract function defined earlier. We extract the feature at a contracted 3D point \(\mathbf{x}^{\prime}\) in \(\mathbf{v}\) as \(\mathbf{v}(\mathbf{x}^{\prime})=\mathbf{F}[\pi(C^{-1}(\mathbf{x}^{\prime}))]\), where \(C^{-1}(\mathbf{x}^{\prime})\) first maps the contracted point to linear world space and \(\pi(\cdot)\) projects it onto the image plane of the context view using camera extrinsics and intrinsics.

Next, we aggregate each pillar into a point to create the 2D groundplan.

Associated to each 2D vertex of the groundplan is now a set of features \(\{\mathbf{f}_{i}\}_{i=1}^{N}\), where \(N\) is the number of samples along the \(y\)-dimension. We use a "pillar-aggregation" MLP to compute softmax scores as \(\alpha_{i}=P(\mathbf{f}_{i},\mathbf{x}_{i})\), where \(P(\cdot)\) denotes the MLP and \(\mathbf{x}_{i}\) is the linear coordinate of the \(i\)-th point on the pillar. Finally, the features are aggregated by computing the weighted sum of the features, \(\mathbf{g}=\sum_{i}\alpha_{i}\mathbf{f}_{i}\).

For each frame, we infer an _entangled_ neural groundplan as described in the previous section. Features in this entangled neural groundplan parameterize both static and dynamic features of the scene, for instance, a car as well as the road below it. We feed this groundplan into a fully convolutional 2D network, which disentangles it into two separate groundplans containing _static_ and _dynamic_ features.

To render a scene using the disentangled static and dynamic groundplans, we first decode query points using both groundplans, yielding _two_ sets of (density, color) values for each point.

We use the input images to infer static and dynamic groundplans, and use them to render out per-frame query views.

Our formulation yields a model that maps a single image to two radiance fields, parameterizing static and dynamic 3D regions respectively.

We now perform a search for connected components in the dynamic neural groundplan to perform 3D instance-level segmentation, monocular 3D bounding box prediction, and the extraction of object-centric 3D representations. Specifically, given a dynamic groundplan, we first sample points in a 3D grid around the groundplan origin and decode their densities. We now perform conventional connected-component labeling in the groundplan space using accumulated density values, identifying the disconnected dynamic objects.

Note that our approach is compatible with prior work leveraging slot attention (Locatello et al., 2020; Kipf et al., 2021) and other inference modules, which can be run on the disentangled dynamic groundplan which, in contrast to image space, enables shift-equivariant processing free from perspective distortion and encodes 3D structure.

We demonstrate that our method infers a 3D scene representation from a single image while disentangling static and dynamic components of the scene into static and dynamic groundplans respectively.

Given only a single image, our method computes separate static and dynamic groundplans that can be used to individually render the static and movable parts of the scene respectively.

The separate reconstruction of movable scene components in the dynamic groundplan enables object detection via instance-level segmentation and bounding box prediction. Fig. 6 presents the instance-level segmentation and 3D bounding box prediction results of the proposed 3D object discovery via connected component discovery using the density inferred using the dynamic groundplan from the bird's-eye view.

Objects can be rotated by arbitrary angles by simple bilinear interpolation of the groundplan features (refer to Appendix B.3). As the dynamic groundplan does not encode static scene regions such as the street below cars, cars can easily be moved from one scene to another.

Future work may further explore the use of more sophisticated downstream processing of the groundplan to enable, for instance, prior-based inference of object-centric representations (Locatello et al., 2020).

We show the potential of neural ground plans as a representation that may enable data-efficient solutions to downstream processing of the 3D scene, such as completion, instance-level segmentation, 3D bounding box prediction, and 3D scene editing.

We populate a coarse 3D feature volume of shape \(64\times 16\times 64\) with 128-dimensional latents for points in the camera frustum by bilinearly sampling the aggregated encoder feature tensor. This volume is processed as explained in the main paper: a coordinate encoding MLP transforms each feature into a new \(128\) dimensional feature vector. This MLP has two hidden layers with 128 hidden units each and a ReLU activation after each hidden layer.

To construct an entangled groundplan, we aggregate the 3D feature volume into a 2D groundplan using a pillar aggregation MLP. Consider a pillar of features orthogonal to the groundplan for a particular point \((x,z)\), in our case that is \(128\times 16\). The MLP takes in each latent with its corresponding 3D coordinate in the world coordinate system. It outputs the softmax scores for each of these latents which can be used to sum these latents along the pillar into a single 128-dimensional latent, as explained in the main paper.

This shallow CNN outputs groundplan of shape \(256\times 256\times 256\), in which the first 128 channels of the feature tensor are attributed as the static groundplan and the rest of the 128 channels are used as the dynamic groundplan for that timestep.

Our method performs project of a 3D point on the image plane to get the corresponding feature tensor.

For projecting a 3D query point \(\mathbf{x}\) on the groundplan, we use grid sample using \((x,z)\)-coordinates of the 3D query points \(\mathbf{x}\).

For localization of objects, we consider the dynamic groundplan and render it from an orthographic bird's-eye view. Along with the bird's-eye view RGB image, we also generate the occupancy map (per-pixel accumulated alpha) as shown in the manuscript in Fig. 5.

Given the localization in the orthographic bird's-eye view, we can find the height of the objects by computing the depth within each region. This information gives us a 3D bounding box around each of the localized object.

Scene editing is performed by editing the dynamic groundplan. Once the different objects have been localized in the dynamic groundplan, we can edit the dynamic groundplan to carry out object deletion, insertion, and rearrangement. Localizing an object gives us its features in the spatial region of the groundplan used for rendering. To delete the object, we replace the features in this regions with features from the dynamic groundplan that encode zero density. For inserting an object at a given location, we find the corresponding \((x,z)\) location in the dynamic groundplan and set the features at that location to the features corresponding to the object. Rearrangement of objects can be seen as a combination of deletion and insertion where we first delete the object from the existing location, followed by inserting the object at the new location in a possibly new orientation. To perform rotation of an object, we rotate the patch of features that correspond to the object to be rotated.

In contrast to single-scene methods, our method needs to fit not only the differentiable rendering in GPU memory, but also the whole inference pipeline - CNNs, 3D lifting, groundplans, etc. This limits the resolution of the groundplans (a car is expressed by \(\sim 6\) latents) as well as the number of volume rendering samples.

Fig. 11 provides rendered images from edited groundplans where the cars are rotated at different angles. Note that the representation only allows for rotations in the \(xz\)-plane.

We first trained our model on the GQN dataset and used the output dynamic groundplans to train the frame interpolation method.

Firstly, we extracted dynamic groundplan for the samples over different timesteps by passing the multi-view observation through our groundplan generation pipeline. Two dynamic groundplans at different timesteps were given as input to RIFE, and an intermediate timestep was queried.

Input: Figure 2: Groundplan inference. Given a context image, we first extract a set of CNN features. We unproject the features into 3D and re-sample them at “pillars” on top of the location of groundplan vertices. Pillars are aggregated into groundplan features using a softmax-weighted sum. The resulting 2D grid of features is decomposed into separate dynamic and static groundplans by a 2D CNN. The coordinate-encoding MLP is not visualized in this figure. Please refer to Sec. 3 for details.

Input: Figure 11: Scene editing with rotated objects. We present images of scenes with edited groundplans where cars are rotated at various angles in the ground plane.

Input: Figure 3: Learning Static-Dynamic Disentanglement. Given multiple frames of a video, we extract per-frame, compactified, static and dynamic groundplans according to Fig. 2. Static groundplans are pooled into a time-invariant groundplan. We then composite per-frame dynamic and static time-invariant groundplans via differentiable volume rendering. Our model is supervised only via a re-rendering loss on video frames. We encourage the model to explain as much of the scene density as possible with the static groundplan via a sparsity loss on per-frame dynamic volume rendering densities. The surface loss is not visualized here.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find that the claim accurately describes neural groundplans as a hybrid discrete-continuous 3D scene representation. The evidence thoroughly demonstrates how neural groundplans work by projecting 3D points onto the groundplan, retrieving features through bilinear interpolation, and decoding via an MLP. The technical details provided in both text and figures (particularly Figure 2) directly support each component mentioned in the claim. The cited papers are also correctly referenced. The claim is neither understated nor overstated, but rather presents a precise technical description aligned with the evidence.

<score>1</score>

Justification:
1. The evidence precisely confirms the hybrid nature through descriptions of both discrete (2D grid) and continuous (MLP decoding) components
2. The mechanics of 3D queries are explicitly detailed in the evidence: "A 3D point is decoded by projecting it onto the groundplan and retrieving the corresponding feature vector using bilinear interpolation"
3. Figure 2 provides a clear visual demonstration of the process described in the claim
4. The evidence supports the final MLP decoding step: "This feature is then concatenated with the vertical y-coordinate of the query point and decoded into radiance and density values via a fully connected network"
5. The claim accurately cites relevant prior work that aligns with the evidence presented


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Neural groundplans are a hybrid discrete-continuous 3D neural scene representation (Chan et al., 2022; Peng et al., 2020; Philion and Fidler, 2020; Roddick et al., 2019; Mani et al., 2020) and enable 3D queries by projecting a 3D point onto the groundplan, retrieving the respective feature, and decoding it via an MLP into a full 3D scene.

The review comment to be evaluated is:

====> Strength
1. The authors introduced the bird’s eye-view (BEV) representation as a scene representation, which is novel in the area of neural rendering.
2. The disentanglement of the static and moving component of the scene adds more contribution to the proposed method. Although it is not new to automatically separate these components in the neural rendering community, using a vanilla 2D CNN to separate the feature representation is new to the reviewer.
3. The authors show that using a simple heuristic method, they can generate instance segmentation and bounding boxes for the dynamic components in the scene.
4. Good qualitative results are given.

====> Weakness
1. The details of the neural groundplans is confusing:

    i. In page 4, there are multiple $x$ such as $x'$, $x$, $x_c$, $x_i$. Their shape should be clarified. Whether they belong to 2D or 3D point, and also their coordinate system (i.e., the image frame coordinate or the world coordinate, and how to transform between these coordinates using the camera intrinsic)

    ii. For equation like $v(x ′ ) = F(π(C^{-1}(x ′ )))$, $f(x ′ ) = ...$, the parentheses here denote indexing the feature volume $v$ and feature tensor $F$. However, it is easy to confuse them with the functions like $\pi()$ and $D()$. The authors should consider using other formulation.

    iii.The $\pi(\cdot)$ is not clear. What is the meaning of it and how to learn $\pi(\cdot)$, is it learned per image I?

    iv. ``$g_x$ are the groundplan features for the point $x$ computed by projecting the coordinates onto the groundplane...'' How is the projection conducted? Is it projected by the camera extrinsic？

    v. In page 5, the authors say ``The per-frame static groundplans are pooled to obtain a single, time-independent static groundplan.'' How is the pooling process performed? The authors need to detail it.

2. The task setting is not clear. The authors should consider formulate the problem setting in a separated subsection (e.g., train one model for multiple scenes, all images are calibrated, the view density per scene, etc.)

3. It seems the compared baselines like pixelNerf and uORF are not proposed for the topic of separating dynamic and static representation. It would be fairer to compare with ``Neuraldiff: Segmenting 3d objects that move in egocentric videos.'' that used different components for the moving and static objects.

The evidence to be evaluated is:Hybrid discrete-continuous neural scene representations offer a compromise by factorizing a dense 3D field into several lower-dimensional representations that are used to condition an MLP (Chan et al., 2022; Chen et al., 2022). In particular, neural groundplans and axis-aligned 2D grids enable high-quality unconditional generation of 3D scenes (DeVries et al., 2021; Chan et al., 2022) as well as reconstruction of 3D geometry from pointclouds (Peng et al., 2020).

Prior work uses ground-plane 2D grids as representations for object detection and segmentation (Saha et al., 2022; Harley et al., 2022; Philion and Fidler, 2020; Reiher et al., 2020; Roddick et al., 2019), layout generation and completion (Cao and de Charette, 2022; Jeong et al., 2022; Mani et al., 2020; Yang et al., 2021), and next-frame prediction (Hu et al., 2021; Zieba et al., 2020). The bird's-eye view is generated either directly without 3D inductive biases (Mani et al., 2020), or similar to our proposed approach, by using 3D geometry-driven inductive biases such as unprojection into a volume (Harley et al., 2022; Chen et al., 2022; Roddick et al., 2019), or by generating a 3D point cloud (Philion and Fidler, 2020; Hu et al., 2021).

In contrast, we present a self-supervised conditional groundplan representation, learned only from images via neural rendering.

A neural groundplan is a 2D grid of features aligned with the ground plane of the 3D scene, which we define to be the \(xz-\)plane. A 3D point is decoded by projecting it onto the groundplan and retrieving the corresponding feature vector using bilinear interpolation. This feature is then concatenated with the vertical \(y\)-coordinate of the query point and decoded into radiance and density values via a fully connected network, enabling novel view synthesis using volume rendering (Mildenhall et al., 2020).

Inferring a neural groundplan from one or several images proceeds in three steps: (1) feature extraction, (2) feature unprojection, (3) pillar aggregation. Given a single image \(\mathbf{I}\), we first extract per-pixel features via a CNN encoder to yield a feature tensor \(\mathbf{F}\). We define the camera as the world origin and center the neural groundplan accordingly, approximately aligned with the ground level. The image features are unprojected to a 3D feature volume \(\mathbf{v}\) in contracted world space using the inverse of the contract function defined earlier. We extract the feature at a contracted 3D point \(\mathbf{x}^{\prime}\) in \(\mathbf{v}\) as \(\mathbf{v}(\mathbf{x}^{\prime})=\mathbf{F}[\pi(C^{-1}(\mathbf{x}^{\prime}))]\), where \(C^{-1}(\mathbf{x}^{\prime})\) first maps the contracted point to linear world space and \(\pi(\cdot)\) projects it onto the image plane of the context view using camera extrinsics and intrinsics.

Next, we aggregate each pillar into a point to create the 2D groundplan.

Associated to each 2D vertex of the groundplan is now a set of features \(\{\mathbf{f}_{i}\}_{i=1}^{N}\), where \(N\) is the number of samples along the \(y\)-dimension. We use a "pillar-aggregation" MLP to compute softmax scores as \(\alpha_{i}=P(\mathbf{f}_{i},\mathbf{x}_{i})\), where \(P(\cdot)\) denotes the MLP and \(\mathbf{x}_{i}\) is the linear coordinate of the \(i\)-th point on the pillar. Finally, the features are aggregated by computing the weighted sum of the features, \(\mathbf{g}=\sum_{i}\alpha_{i}\mathbf{f}_{i}\).

For each frame, we infer an _entangled_ neural groundplan as described in the previous section. Features in this entangled neural groundplan parameterize both static and dynamic features of the scene, for instance, a car as well as the road below it. We feed this groundplan into a fully convolutional 2D network, which disentangles it into two separate groundplans containing _static_ and _dynamic_ features.

To render a scene using the disentangled static and dynamic groundplans, we first decode query points using both groundplans, yielding _two_ sets of (density, color) values for each point.

We use the input images to infer static and dynamic groundplans, and use them to render out per-frame query views.

Our formulation yields a model that maps a single image to two radiance fields, parameterizing static and dynamic 3D regions respectively.

We now perform a search for connected components in the dynamic neural groundplan to perform 3D instance-level segmentation, monocular 3D bounding box prediction, and the extraction of object-centric 3D representations. Specifically, given a dynamic groundplan, we first sample points in a 3D grid around the groundplan origin and decode their densities. We now perform conventional connected-component labeling in the groundplan space using accumulated density values, identifying the disconnected dynamic objects.

Note that our approach is compatible with prior work leveraging slot attention (Locatello et al., 2020; Kipf et al., 2021) and other inference modules, which can be run on the disentangled dynamic groundplan which, in contrast to image space, enables shift-equivariant processing free from perspective distortion and encodes 3D structure.

We demonstrate that our method infers a 3D scene representation from a single image while disentangling static and dynamic components of the scene into static and dynamic groundplans respectively.

Given only a single image, our method computes separate static and dynamic groundplans that can be used to individually render the static and movable parts of the scene respectively.

The separate reconstruction of movable scene components in the dynamic groundplan enables object detection via instance-level segmentation and bounding box prediction. Fig. 6 presents the instance-level segmentation and 3D bounding box prediction results of the proposed 3D object discovery via connected component discovery using the density inferred using the dynamic groundplan from the bird's-eye view.

Objects can be rotated by arbitrary angles by simple bilinear interpolation of the groundplan features (refer to Appendix B.3). As the dynamic groundplan does not encode static scene regions such as the street below cars, cars can easily be moved from one scene to another.

Future work may further explore the use of more sophisticated downstream processing of the groundplan to enable, for instance, prior-based inference of object-centric representations (Locatello et al., 2020).

We show the potential of neural ground plans as a representation that may enable data-efficient solutions to downstream processing of the 3D scene, such as completion, instance-level segmentation, 3D bounding box prediction, and 3D scene editing.

We populate a coarse 3D feature volume of shape \(64\times 16\times 64\) with 128-dimensional latents for points in the camera frustum by bilinearly sampling the aggregated encoder feature tensor. This volume is processed as explained in the main paper: a coordinate encoding MLP transforms each feature into a new \(128\) dimensional feature vector. This MLP has two hidden layers with 128 hidden units each and a ReLU activation after each hidden layer.

To construct an entangled groundplan, we aggregate the 3D feature volume into a 2D groundplan using a pillar aggregation MLP. Consider a pillar of features orthogonal to the groundplan for a particular point \((x,z)\), in our case that is \(128\times 16\). The MLP takes in each latent with its corresponding 3D coordinate in the world coordinate system. It outputs the softmax scores for each of these latents which can be used to sum these latents along the pillar into a single 128-dimensional latent, as explained in the main paper.

This shallow CNN outputs groundplan of shape \(256\times 256\times 256\), in which the first 128 channels of the feature tensor are attributed as the static groundplan and the rest of the 128 channels are used as the dynamic groundplan for that timestep.

Our method performs project of a 3D point on the image plane to get the corresponding feature tensor.

For projecting a 3D query point \(\mathbf{x}\) on the groundplan, we use grid sample using \((x,z)\)-coordinates of the 3D query points \(\mathbf{x}\).

For localization of objects, we consider the dynamic groundplan and render it from an orthographic bird's-eye view. Along with the bird's-eye view RGB image, we also generate the occupancy map (per-pixel accumulated alpha) as shown in the manuscript in Fig. 5.

Given the localization in the orthographic bird's-eye view, we can find the height of the objects by computing the depth within each region. This information gives us a 3D bounding box around each of the localized object.

Scene editing is performed by editing the dynamic groundplan. Once the different objects have been localized in the dynamic groundplan, we can edit the dynamic groundplan to carry out object deletion, insertion, and rearrangement. Localizing an object gives us its features in the spatial region of the groundplan used for rendering. To delete the object, we replace the features in this regions with features from the dynamic groundplan that encode zero density. For inserting an object at a given location, we find the corresponding \((x,z)\) location in the dynamic groundplan and set the features at that location to the features corresponding to the object. Rearrangement of objects can be seen as a combination of deletion and insertion where we first delete the object from the existing location, followed by inserting the object at the new location in a possibly new orientation. To perform rotation of an object, we rotate the patch of features that correspond to the object to be rotated.

In contrast to single-scene methods, our method needs to fit not only the differentiable rendering in GPU memory, but also the whole inference pipeline - CNNs, 3D lifting, groundplans, etc. This limits the resolution of the groundplans (a car is expressed by \(\sim 6\) latents) as well as the number of volume rendering samples.

Fig. 11 provides rendered images from edited groundplans where the cars are rotated at different angles. Note that the representation only allows for rotations in the \(xz\)-plane.

We first trained our model on the GQN dataset and used the output dynamic groundplans to train the frame interpolation method.

Firstly, we extracted dynamic groundplan for the samples over different timesteps by passing the multi-view observation through our groundplan generation pipeline. Two dynamic groundplans at different timesteps were given as input to RIFE, and an intermediate timestep was queried.

Input: Figure 2: Groundplan inference. Given a context image, we first extract a set of CNN features. We unproject the features into 3D and re-sample them at “pillars” on top of the location of groundplan vertices. Pillars are aggregated into groundplan features using a softmax-weighted sum. The resulting 2D grid of features is decomposed into separate dynamic and static groundplans by a 2D CNN. The coordinate-encoding MLP is not visualized in this figure. Please refer to Sec. 3 for details.

Input: Figure 11: Scene editing with rotated objects. We present images of scenes with edited groundplans where cars are rotated at various angles in the ground plane.

Input: Figure 3: Learning Static-Dynamic Disentanglement. Given multiple frames of a video, we extract per-frame, compactified, static and dynamic groundplans according to Fig. 2. Static groundplans are pooled into a time-invariant groundplan. We then composite per-frame dynamic and static time-invariant groundplans via differentiable volume rendering. Our model is supervised only via a re-rendering loss on video frames. We encourage the model to explain as much of the scene density as possible with the static groundplan via a sparsity loss on per-frame dynamic volume rendering densities. The surface loss is not visualized here.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence and reviewer comments, I'll evaluate the claim about neural groundplans.

Summary: The claim is generally well-stated but has some minor overstatements. The evidence thoroughly supports that neural groundplans are indeed a hybrid discrete-continuous 3D representation and demonstrates the 3D query mechanism through projection and MLP decoding. However, the cited papers don't all specifically discuss neural groundplans in the same way - some focus on different aspects of BEV representations. The reviewer points out some unclear technical details about the projection and coordinate systems, suggesting the mechanism isn't as straightforward as the claim implies.

<score>2</score>

Justification:
1. The evidence strongly supports the hybrid discrete-continuous nature through detailed explanations of the 2D grid structure and MLP decoding process.
2. The projection and querying mechanism is documented in detail in the evidence: "A 3D point is decoded by projecting it onto the groundplan and retrieving the corresponding feature vector using bilinear interpolation."
3. The reviewer highlights some ambiguity in the technical details of projection and coordinate systems, suggesting slight overstatement in how clearly defined the mechanism is.
4. Not all cited papers specifically discuss neural groundplans - some focus on related but distinct BEV representations.
5. The implementation details in the evidence show more complexity than the claim suggests, with multiple steps including feature extraction, unprojection, and pillar aggregation.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Neural groundplans are a hybrid discrete-continuous 3D neural scene representation (Chan et al., 2022; Peng et al., 2020; Philion and Fidler, 2020; Roddick et al., 2019; Mani et al., 2020) and enable 3D queries by projecting a 3D point onto the groundplan, retrieving the respective feature, and decoding it via an MLP into a full 3D scene.

The review comment to be evaluated is:

Strengths include:
* The self-supervised training removes the need for GT annotation, e.g., object bounding boxes, for the supported downstream tasks.
* The decomposition of static and movable object feature spaces looks like quite an elegant solution to utilize the inductive bias in multi-view videos while enabling capabilities such as segmentation and object-level scene editing.
* The results overall look promising. Single-view/few-view novel view synthesis is a very challenging task, and the comparisons with PixelNeRF and uORF clearly demonstrated the advantages in synthesis quality. The analysis of adding more views (Fig.10) is strong evidence that the learned 3D representation can figure out how 2D views are related to, and hence help improve, 3D reconstruction.

Weaknesses include:
* There is no study of the generalization capability of the model. It seems likely that the learned models can overfit the limited appearances in the dataset (a few types of shapes/objects with mostly uniform colors).
* Related to the last point, the evaluation is completely done on synthesized datasets, and it's not clear how well the model can learn real-world data with much more diverse appearances. Since the method does not require GT annotations, some evaluation on real-world data does not seem too difficult.
* The are some quality issues with the presented view synthesis results. For example, while unobserved regions are understandably blurry, why are observed regions also not sharp? We can see from Fig.10 that adding more views does not help the quality of observed regions any further.
* Training requires multi-view videos. I can't find clarification on whether these need to be from static cameras. Nothing in the model appears to have such a requirement, but if so, that's a strong limitation. Moreover, why can't the model be trained on frames from a single moving camera?
* It's not clearly described how scene editing is done. It says that's enabled by "directly editing the neural groundplan". Does that mean editing the (movable object) feature space? The feature space is not sparse (i.e. locations without objects have non-zero features), so how does cropping in such space work, and why can that make sense?

The evidence to be evaluated is:Hybrid discrete-continuous neural scene representations offer a compromise by factorizing a dense 3D field into several lower-dimensional representations that are used to condition an MLP (Chan et al., 2022; Chen et al., 2022). In particular, neural groundplans and axis-aligned 2D grids enable high-quality unconditional generation of 3D scenes (DeVries et al., 2021; Chan et al., 2022) as well as reconstruction of 3D geometry from pointclouds (Peng et al., 2020).

Prior work uses ground-plane 2D grids as representations for object detection and segmentation (Saha et al., 2022; Harley et al., 2022; Philion and Fidler, 2020; Reiher et al., 2020; Roddick et al., 2019), layout generation and completion (Cao and de Charette, 2022; Jeong et al., 2022; Mani et al., 2020; Yang et al., 2021), and next-frame prediction (Hu et al., 2021; Zieba et al., 2020). The bird's-eye view is generated either directly without 3D inductive biases (Mani et al., 2020), or similar to our proposed approach, by using 3D geometry-driven inductive biases such as unprojection into a volume (Harley et al., 2022; Chen et al., 2022; Roddick et al., 2019), or by generating a 3D point cloud (Philion and Fidler, 2020; Hu et al., 2021).

In contrast, we present a self-supervised conditional groundplan representation, learned only from images via neural rendering.

A neural groundplan is a 2D grid of features aligned with the ground plane of the 3D scene, which we define to be the \(xz-\)plane. A 3D point is decoded by projecting it onto the groundplan and retrieving the corresponding feature vector using bilinear interpolation. This feature is then concatenated with the vertical \(y\)-coordinate of the query point and decoded into radiance and density values via a fully connected network, enabling novel view synthesis using volume rendering (Mildenhall et al., 2020).

Inferring a neural groundplan from one or several images proceeds in three steps: (1) feature extraction, (2) feature unprojection, (3) pillar aggregation. Given a single image \(\mathbf{I}\), we first extract per-pixel features via a CNN encoder to yield a feature tensor \(\mathbf{F}\). We define the camera as the world origin and center the neural groundplan accordingly, approximately aligned with the ground level. The image features are unprojected to a 3D feature volume \(\mathbf{v}\) in contracted world space using the inverse of the contract function defined earlier. We extract the feature at a contracted 3D point \(\mathbf{x}^{\prime}\) in \(\mathbf{v}\) as \(\mathbf{v}(\mathbf{x}^{\prime})=\mathbf{F}[\pi(C^{-1}(\mathbf{x}^{\prime}))]\), where \(C^{-1}(\mathbf{x}^{\prime})\) first maps the contracted point to linear world space and \(\pi(\cdot)\) projects it onto the image plane of the context view using camera extrinsics and intrinsics.

Next, we aggregate each pillar into a point to create the 2D groundplan.

Associated to each 2D vertex of the groundplan is now a set of features \(\{\mathbf{f}_{i}\}_{i=1}^{N}\), where \(N\) is the number of samples along the \(y\)-dimension. We use a "pillar-aggregation" MLP to compute softmax scores as \(\alpha_{i}=P(\mathbf{f}_{i},\mathbf{x}_{i})\), where \(P(\cdot)\) denotes the MLP and \(\mathbf{x}_{i}\) is the linear coordinate of the \(i\)-th point on the pillar. Finally, the features are aggregated by computing the weighted sum of the features, \(\mathbf{g}=\sum_{i}\alpha_{i}\mathbf{f}_{i}\).

For each frame, we infer an _entangled_ neural groundplan as described in the previous section. Features in this entangled neural groundplan parameterize both static and dynamic features of the scene, for instance, a car as well as the road below it. We feed this groundplan into a fully convolutional 2D network, which disentangles it into two separate groundplans containing _static_ and _dynamic_ features.

To render a scene using the disentangled static and dynamic groundplans, we first decode query points using both groundplans, yielding _two_ sets of (density, color) values for each point.

We use the input images to infer static and dynamic groundplans, and use them to render out per-frame query views.

Our formulation yields a model that maps a single image to two radiance fields, parameterizing static and dynamic 3D regions respectively.

We now perform a search for connected components in the dynamic neural groundplan to perform 3D instance-level segmentation, monocular 3D bounding box prediction, and the extraction of object-centric 3D representations. Specifically, given a dynamic groundplan, we first sample points in a 3D grid around the groundplan origin and decode their densities. We now perform conventional connected-component labeling in the groundplan space using accumulated density values, identifying the disconnected dynamic objects.

Note that our approach is compatible with prior work leveraging slot attention (Locatello et al., 2020; Kipf et al., 2021) and other inference modules, which can be run on the disentangled dynamic groundplan which, in contrast to image space, enables shift-equivariant processing free from perspective distortion and encodes 3D structure.

We demonstrate that our method infers a 3D scene representation from a single image while disentangling static and dynamic components of the scene into static and dynamic groundplans respectively.

Given only a single image, our method computes separate static and dynamic groundplans that can be used to individually render the static and movable parts of the scene respectively.

The separate reconstruction of movable scene components in the dynamic groundplan enables object detection via instance-level segmentation and bounding box prediction. Fig. 6 presents the instance-level segmentation and 3D bounding box prediction results of the proposed 3D object discovery via connected component discovery using the density inferred using the dynamic groundplan from the bird's-eye view.

Objects can be rotated by arbitrary angles by simple bilinear interpolation of the groundplan features (refer to Appendix B.3). As the dynamic groundplan does not encode static scene regions such as the street below cars, cars can easily be moved from one scene to another.

Future work may further explore the use of more sophisticated downstream processing of the groundplan to enable, for instance, prior-based inference of object-centric representations (Locatello et al., 2020).

We show the potential of neural ground plans as a representation that may enable data-efficient solutions to downstream processing of the 3D scene, such as completion, instance-level segmentation, 3D bounding box prediction, and 3D scene editing.

We populate a coarse 3D feature volume of shape \(64\times 16\times 64\) with 128-dimensional latents for points in the camera frustum by bilinearly sampling the aggregated encoder feature tensor. This volume is processed as explained in the main paper: a coordinate encoding MLP transforms each feature into a new \(128\) dimensional feature vector. This MLP has two hidden layers with 128 hidden units each and a ReLU activation after each hidden layer.

To construct an entangled groundplan, we aggregate the 3D feature volume into a 2D groundplan using a pillar aggregation MLP. Consider a pillar of features orthogonal to the groundplan for a particular point \((x,z)\), in our case that is \(128\times 16\). The MLP takes in each latent with its corresponding 3D coordinate in the world coordinate system. It outputs the softmax scores for each of these latents which can be used to sum these latents along the pillar into a single 128-dimensional latent, as explained in the main paper.

This shallow CNN outputs groundplan of shape \(256\times 256\times 256\), in which the first 128 channels of the feature tensor are attributed as the static groundplan and the rest of the 128 channels are used as the dynamic groundplan for that timestep.

Our method performs project of a 3D point on the image plane to get the corresponding feature tensor.

For projecting a 3D query point \(\mathbf{x}\) on the groundplan, we use grid sample using \((x,z)\)-coordinates of the 3D query points \(\mathbf{x}\).

For localization of objects, we consider the dynamic groundplan and render it from an orthographic bird's-eye view. Along with the bird's-eye view RGB image, we also generate the occupancy map (per-pixel accumulated alpha) as shown in the manuscript in Fig. 5.

Given the localization in the orthographic bird's-eye view, we can find the height of the objects by computing the depth within each region. This information gives us a 3D bounding box around each of the localized object.

Scene editing is performed by editing the dynamic groundplan. Once the different objects have been localized in the dynamic groundplan, we can edit the dynamic groundplan to carry out object deletion, insertion, and rearrangement. Localizing an object gives us its features in the spatial region of the groundplan used for rendering. To delete the object, we replace the features in this regions with features from the dynamic groundplan that encode zero density. For inserting an object at a given location, we find the corresponding \((x,z)\) location in the dynamic groundplan and set the features at that location to the features corresponding to the object. Rearrangement of objects can be seen as a combination of deletion and insertion where we first delete the object from the existing location, followed by inserting the object at the new location in a possibly new orientation. To perform rotation of an object, we rotate the patch of features that correspond to the object to be rotated.

In contrast to single-scene methods, our method needs to fit not only the differentiable rendering in GPU memory, but also the whole inference pipeline - CNNs, 3D lifting, groundplans, etc. This limits the resolution of the groundplans (a car is expressed by \(\sim 6\) latents) as well as the number of volume rendering samples.

Fig. 11 provides rendered images from edited groundplans where the cars are rotated at different angles. Note that the representation only allows for rotations in the \(xz\)-plane.

We first trained our model on the GQN dataset and used the output dynamic groundplans to train the frame interpolation method.

Firstly, we extracted dynamic groundplan for the samples over different timesteps by passing the multi-view observation through our groundplan generation pipeline. Two dynamic groundplans at different timesteps were given as input to RIFE, and an intermediate timestep was queried.

Input: Figure 2: Groundplan inference. Given a context image, we first extract a set of CNN features. We unproject the features into 3D and re-sample them at “pillars” on top of the location of groundplan vertices. Pillars are aggregated into groundplan features using a softmax-weighted sum. The resulting 2D grid of features is decomposed into separate dynamic and static groundplans by a 2D CNN. The coordinate-encoding MLP is not visualized in this figure. Please refer to Sec. 3 for details.

Input: Figure 11: Scene editing with rotated objects. We present images of scenes with edited groundplans where cars are rotated at various angles in the ground plane.

Input: Figure 3: Learning Static-Dynamic Disentanglement. Given multiple frames of a video, we extract per-frame, compactified, static and dynamic groundplans according to Fig. 2. Static groundplans are pooled into a time-invariant groundplan. We then composite per-frame dynamic and static time-invariant groundplans via differentiable volume rendering. Our model is supervised only via a re-rendering loss on video frames. We encourage the model to explain as much of the scene density as possible with the static groundplan via a sparsity loss on per-frame dynamic volume rendering densities. The surface loss is not visualized here.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, I find that the claim about neural groundplans is well-stated and accurately reflects the technical implementation described in the evidence. The evidence thoroughly details how neural groundplans work through:
- Projecting 3D points onto a 2D grid aligned with the ground plane
- Using bilinear interpolation to retrieve features
- Decoding via an MLP into 3D scene representations

The evidence directly supports each component mentioned in the claim without exaggeration. The review comments, while noting some limitations, do not contradict the basic mechanical description provided in the claim.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Neural groundplans are a hybrid discrete-continuous 3D neural scene representation (Chan et al., 2022; Peng et al., 2020; Philion and Fidler, 2020; Roddick et al., 2019; Mani et al., 2020) and enable 3D queries by projecting a 3D point onto the groundplan, retrieving the respective feature, and decoding it via an MLP into a full 3D scene.

The review comment to be evaluated is:

Strength：
1. The conditional neural groundplans proposed are intriguing. This allows efficient processing of scene appearance and geometry directly in 3D, through the self-supervised learning of conditional neural groundplans, a hybrid discrete continuous representation of 3D neural scenes.
2. The proposed disentanglement of static background and movable foreground objects is novel. This approach uses object motion for training an encoder-based system that can reconstruct scenes from a single image instead of decomposing video frames based on scene specifics.
3. The proposed method is promising for understanding 3D from a single image. The dynamic groundplan encodes 3D geometry, allowing us to segment and animate objects in 3D using single images and 3D bounding boxes.

Weaknesses：
1. The biggest weakness is that only results on simple synthetic data are presented. Currently, there are a number of well-annotated 3D driving scenes available. An example is KITTI 360 (https://www.cvlibs.net/datasets/kitti-360/). Could the authors explain why results on the real datasets are not presented? Is it because of some specific annotation (such as BEV correspondence or the number of views) needed by the method? Otherwise, the generalizability on natural scenes and the practical value of the method are questioned.
2. I think there are some ambiguity behind the proposed Static-Dynamic Disentanglement. A definition of Static and Dynamic should be made clear in the proposed framework. A car remaining still in the sequence will be considered as Static. Considering a situation when two moving and static cars looks similar or exactly the same, how can we distinguish from a single image if a car is static or dynamic? The method should mention the above cases as an assumption, or making a clear definition of being Static and Dynamic.
3. Groundplans implicitly assume that objects are aligned on a plane. However, in real life, it is common that scenes/objects are not placed on a flat ground plane (e.g., a curved plane). The ability of persistent 3D scene representation is a little overclaimed, if we consider the implicit assumption behind groundplans.

The evidence to be evaluated is:Hybrid discrete-continuous neural scene representations offer a compromise by factorizing a dense 3D field into several lower-dimensional representations that are used to condition an MLP (Chan et al., 2022; Chen et al., 2022). In particular, neural groundplans and axis-aligned 2D grids enable high-quality unconditional generation of 3D scenes (DeVries et al., 2021; Chan et al., 2022) as well as reconstruction of 3D geometry from pointclouds (Peng et al., 2020).

Prior work uses ground-plane 2D grids as representations for object detection and segmentation (Saha et al., 2022; Harley et al., 2022; Philion and Fidler, 2020; Reiher et al., 2020; Roddick et al., 2019), layout generation and completion (Cao and de Charette, 2022; Jeong et al., 2022; Mani et al., 2020; Yang et al., 2021), and next-frame prediction (Hu et al., 2021; Zieba et al., 2020). The bird's-eye view is generated either directly without 3D inductive biases (Mani et al., 2020), or similar to our proposed approach, by using 3D geometry-driven inductive biases such as unprojection into a volume (Harley et al., 2022; Chen et al., 2022; Roddick et al., 2019), or by generating a 3D point cloud (Philion and Fidler, 2020; Hu et al., 2021).

In contrast, we present a self-supervised conditional groundplan representation, learned only from images via neural rendering.

A neural groundplan is a 2D grid of features aligned with the ground plane of the 3D scene, which we define to be the \(xz-\)plane. A 3D point is decoded by projecting it onto the groundplan and retrieving the corresponding feature vector using bilinear interpolation. This feature is then concatenated with the vertical \(y\)-coordinate of the query point and decoded into radiance and density values via a fully connected network, enabling novel view synthesis using volume rendering (Mildenhall et al., 2020).

Inferring a neural groundplan from one or several images proceeds in three steps: (1) feature extraction, (2) feature unprojection, (3) pillar aggregation. Given a single image \(\mathbf{I}\), we first extract per-pixel features via a CNN encoder to yield a feature tensor \(\mathbf{F}\). We define the camera as the world origin and center the neural groundplan accordingly, approximately aligned with the ground level. The image features are unprojected to a 3D feature volume \(\mathbf{v}\) in contracted world space using the inverse of the contract function defined earlier. We extract the feature at a contracted 3D point \(\mathbf{x}^{\prime}\) in \(\mathbf{v}\) as \(\mathbf{v}(\mathbf{x}^{\prime})=\mathbf{F}[\pi(C^{-1}(\mathbf{x}^{\prime}))]\), where \(C^{-1}(\mathbf{x}^{\prime})\) first maps the contracted point to linear world space and \(\pi(\cdot)\) projects it onto the image plane of the context view using camera extrinsics and intrinsics.

Next, we aggregate each pillar into a point to create the 2D groundplan.

Associated to each 2D vertex of the groundplan is now a set of features \(\{\mathbf{f}_{i}\}_{i=1}^{N}\), where \(N\) is the number of samples along the \(y\)-dimension. We use a "pillar-aggregation" MLP to compute softmax scores as \(\alpha_{i}=P(\mathbf{f}_{i},\mathbf{x}_{i})\), where \(P(\cdot)\) denotes the MLP and \(\mathbf{x}_{i}\) is the linear coordinate of the \(i\)-th point on the pillar. Finally, the features are aggregated by computing the weighted sum of the features, \(\mathbf{g}=\sum_{i}\alpha_{i}\mathbf{f}_{i}\).

For each frame, we infer an _entangled_ neural groundplan as described in the previous section. Features in this entangled neural groundplan parameterize both static and dynamic features of the scene, for instance, a car as well as the road below it. We feed this groundplan into a fully convolutional 2D network, which disentangles it into two separate groundplans containing _static_ and _dynamic_ features.

To render a scene using the disentangled static and dynamic groundplans, we first decode query points using both groundplans, yielding _two_ sets of (density, color) values for each point.

We use the input images to infer static and dynamic groundplans, and use them to render out per-frame query views.

Our formulation yields a model that maps a single image to two radiance fields, parameterizing static and dynamic 3D regions respectively.

We now perform a search for connected components in the dynamic neural groundplan to perform 3D instance-level segmentation, monocular 3D bounding box prediction, and the extraction of object-centric 3D representations. Specifically, given a dynamic groundplan, we first sample points in a 3D grid around the groundplan origin and decode their densities. We now perform conventional connected-component labeling in the groundplan space using accumulated density values, identifying the disconnected dynamic objects.

Note that our approach is compatible with prior work leveraging slot attention (Locatello et al., 2020; Kipf et al., 2021) and other inference modules, which can be run on the disentangled dynamic groundplan which, in contrast to image space, enables shift-equivariant processing free from perspective distortion and encodes 3D structure.

We demonstrate that our method infers a 3D scene representation from a single image while disentangling static and dynamic components of the scene into static and dynamic groundplans respectively.

Given only a single image, our method computes separate static and dynamic groundplans that can be used to individually render the static and movable parts of the scene respectively.

The separate reconstruction of movable scene components in the dynamic groundplan enables object detection via instance-level segmentation and bounding box prediction. Fig. 6 presents the instance-level segmentation and 3D bounding box prediction results of the proposed 3D object discovery via connected component discovery using the density inferred using the dynamic groundplan from the bird's-eye view.

Objects can be rotated by arbitrary angles by simple bilinear interpolation of the groundplan features (refer to Appendix B.3). As the dynamic groundplan does not encode static scene regions such as the street below cars, cars can easily be moved from one scene to another.

Future work may further explore the use of more sophisticated downstream processing of the groundplan to enable, for instance, prior-based inference of object-centric representations (Locatello et al., 2020).

We show the potential of neural ground plans as a representation that may enable data-efficient solutions to downstream processing of the 3D scene, such as completion, instance-level segmentation, 3D bounding box prediction, and 3D scene editing.

We populate a coarse 3D feature volume of shape \(64\times 16\times 64\) with 128-dimensional latents for points in the camera frustum by bilinearly sampling the aggregated encoder feature tensor. This volume is processed as explained in the main paper: a coordinate encoding MLP transforms each feature into a new \(128\) dimensional feature vector. This MLP has two hidden layers with 128 hidden units each and a ReLU activation after each hidden layer.

To construct an entangled groundplan, we aggregate the 3D feature volume into a 2D groundplan using a pillar aggregation MLP. Consider a pillar of features orthogonal to the groundplan for a particular point \((x,z)\), in our case that is \(128\times 16\). The MLP takes in each latent with its corresponding 3D coordinate in the world coordinate system. It outputs the softmax scores for each of these latents which can be used to sum these latents along the pillar into a single 128-dimensional latent, as explained in the main paper.

This shallow CNN outputs groundplan of shape \(256\times 256\times 256\), in which the first 128 channels of the feature tensor are attributed as the static groundplan and the rest of the 128 channels are used as the dynamic groundplan for that timestep.

Our method performs project of a 3D point on the image plane to get the corresponding feature tensor.

For projecting a 3D query point \(\mathbf{x}\) on the groundplan, we use grid sample using \((x,z)\)-coordinates of the 3D query points \(\mathbf{x}\).

For localization of objects, we consider the dynamic groundplan and render it from an orthographic bird's-eye view. Along with the bird's-eye view RGB image, we also generate the occupancy map (per-pixel accumulated alpha) as shown in the manuscript in Fig. 5.

Given the localization in the orthographic bird's-eye view, we can find the height of the objects by computing the depth within each region. This information gives us a 3D bounding box around each of the localized object.

Scene editing is performed by editing the dynamic groundplan. Once the different objects have been localized in the dynamic groundplan, we can edit the dynamic groundplan to carry out object deletion, insertion, and rearrangement. Localizing an object gives us its features in the spatial region of the groundplan used for rendering. To delete the object, we replace the features in this regions with features from the dynamic groundplan that encode zero density. For inserting an object at a given location, we find the corresponding \((x,z)\) location in the dynamic groundplan and set the features at that location to the features corresponding to the object. Rearrangement of objects can be seen as a combination of deletion and insertion where we first delete the object from the existing location, followed by inserting the object at the new location in a possibly new orientation. To perform rotation of an object, we rotate the patch of features that correspond to the object to be rotated.

In contrast to single-scene methods, our method needs to fit not only the differentiable rendering in GPU memory, but also the whole inference pipeline - CNNs, 3D lifting, groundplans, etc. This limits the resolution of the groundplans (a car is expressed by \(\sim 6\) latents) as well as the number of volume rendering samples.

Fig. 11 provides rendered images from edited groundplans where the cars are rotated at different angles. Note that the representation only allows for rotations in the \(xz\)-plane.

We first trained our model on the GQN dataset and used the output dynamic groundplans to train the frame interpolation method.

Firstly, we extracted dynamic groundplan for the samples over different timesteps by passing the multi-view observation through our groundplan generation pipeline. Two dynamic groundplans at different timesteps were given as input to RIFE, and an intermediate timestep was queried.

Input: Figure 2: Groundplan inference. Given a context image, we first extract a set of CNN features. We unproject the features into 3D and re-sample them at “pillars” on top of the location of groundplan vertices. Pillars are aggregated into groundplan features using a softmax-weighted sum. The resulting 2D grid of features is decomposed into separate dynamic and static groundplans by a 2D CNN. The coordinate-encoding MLP is not visualized in this figure. Please refer to Sec. 3 for details.

Input: Figure 11: Scene editing with rotated objects. We present images of scenes with edited groundplans where cars are rotated at various angles in the ground plane.

Input: Figure 3: Learning Static-Dynamic Disentanglement. Given multiple frames of a video, we extract per-frame, compactified, static and dynamic groundplans according to Fig. 2. Static groundplans are pooled into a time-invariant groundplan. We then composite per-frame dynamic and static time-invariant groundplans via differentiable volume rendering. Our model is supervised only via a re-rendering loss on video frames. We encourage the model to explain as much of the scene density as possible with the static groundplan via a sparsity loss on per-frame dynamic volume rendering densities. The surface loss is not visualized here.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on my review of the evidence and claim, the statement accurately describes the core functionality of neural groundplans but slightly understates their complexity. The evidence shows that neural groundplans are indeed hybrid discrete-continuous 3D representations that enable 3D queries through projection and MLP decoding. However, the claim omits key details about the multi-step process involving feature extraction, unprojection, and pillar aggregation. The evidence from figures and technical descriptions demonstrates a more sophisticated pipeline than suggested. The claim is generally well-stated but could be more complete.

<score>2</score>

Justification:
1. The claim correctly captures the fundamental nature of neural groundplans as shown in the evidence: "A neural groundplan is a 2D grid of features aligned with the ground plane of the 3D scene"

2. The evidence supports the 3D query mechanism described: "A 3D point is decoded by projecting it onto the groundplan and retrieving the corresponding feature vector using bilinear interpolation"

3. However, the claim simplifies the complex three-step inference process detailed in the evidence: "(1) feature extraction, (2) feature unprojection, (3) pillar aggregation"

4. The reviewer comments about groundplan limitations on curved surfaces and real-world scenarios are valid but don't directly impact this specific claim's accuracy

5. The process involves more sophisticated elements like "pillar-aggregation MLP", "softmax scores", and feature aggregation that aren't mentioned in the claim but don't make it incorrect, just less complete


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

By compactifying 3D space with a nonlinear mapping, neural groundplans can encode unbounded 3D scenes in a bounded region.

The evidence to be evaluated is:In particular, neural groundplans and axis-aligned 2D grids enable high-quality unconditional generation of 3D scenes (DeVries et al., 2021; Chan et al., 2022) as well as reconstruction of 3D geometry from pointclouds (Peng et al., 2020).

The bird's-eye view is generated either directly without 3D inductive biases (Mani et al., 2020), or similar to our proposed approach, by using 3D geometry-driven inductive biases such as unprojection into a volume (Harley et al., 2022; Chen et al., 2022; Roddick et al., 2019), or by generating a 3D point cloud (Philion and Fidler, 2020; Hu et al., 2021).

In contrast, we present a self-supervised conditional groundplan representation, learned only from images via neural rendering.

Our approach is similar in using object motion as cue for disentanglement and multi-view as cue for 3D reconstruction, but uses it as supervision to train an encoder-based approach that enables reconstruction from a _single_ image instead of scene-specific disentanglement from multiple video frames.

Prior work has aimed to infer object-centric representations directly from images, with objects either represented as localized object-centric patches (Lin et al., 2020; Eslami et al., 2016; Crawford and Pineau, 2019; Kosiorek et al., 2018; Jiang et al., 2019) or scene mixture components (Engelcke et al., 2020; Burgess et al., 2019; Greff et al., 2019, 2016; 2017; Du et al., 2021), with the slot attention module (Locatello et al., 2020) increasingly driving object-centric inference. Resulting object representations may be decoded into object-centric 3D representations and composed for novel view synthesis (Yu et al., 2022; Smith et al., 2022; Elich et al., 2022; Chen et al., 2021; Bear et al., 2020; Zakharov et al., 2020; Zekharov et al., 2020; Beker et al., 2020; Du et al., 2021).

In this work, we demonstrate that a representation factorized into static and movable 3D regions can serve as a powerful backbone for object discovery.

In this definition, however, it is only possible to decode 3D points that lie within the boundaries of the neural groundplan, which precludes reconstruction and representation of unbounded scenes. We thus compactify \(\mathbb{R}^{3}\) by implementing a non-linear coordinate re-mapping as proposed by Barron et al. (2021). Points \(\mathbf{x}\) within a radius \(r_{\text{inner}}\) around the groundplan origin remain unaffected, but points outside this radius are contracted. For any 3D point \(\mathbf{x}\), the contracted 3D coordinate can be computed as \(\mathbf{x}^{\prime}=C(\mathbf{x})=((1+k)-k/||\mathbf{u}||)(\mathbf{u}/|| \mathbf{u}||)r_{\text{inner}}\), where \(\mathbf{u}=\mathbf{x}/r_{\text{inner}}\), and \(k\) is a hyperparameter which controls the size of the contracted region. Note that \(C\) is invertible, such that \(\mathbf{x}=C^{-1}(\mathbf{x}^{\prime})\) is a function that takes a 3D point in contracted space \(\mathbf{x}^{\prime}\) to the original 3D point \(\mathbf{x}\) in linear space.

We first use a coordinate-encoding MLP \(D(\cdot)\) to transform the volume as \(\mathbf{f}(\mathbf{x}^{\prime})=D(\mathbf{v}(\mathbf{x}^{\prime}),\mathbf{x}_ {\mathbf{c}},\mathbf{d})\), where \(\mathbf{x}_{\mathbf{c}}\) denotes the 3D point in linear camera coordinates of the context camera, and \(\mathbf{d}\) denotes the ray direction from the camera center to that point. Since all features along a camera ray are identical in \(\mathbf{v}\), coordinate encoding is used to add the depth information to the features.

Our formulation yields a model that maps a single image to two radiance fields, parameterizing static and dynamic 3D regions respectively.

We now perform a search for connected components in the dynamic neural groundplan to perform 3D instance-level segmentation, monocular 3D bounding box prediction, and the extraction of object-centric 3D representations. Specifically, given a dynamic groundplan, we first sample points in a 3D grid around the groundplan origin and decode their densities. We now perform conventional connected-component labeling in the groundplan space using accumulated density values, identifying the disconnected dynamic objects.

Note that our approach is compatible with prior work leveraging slot attention (Locatello et al., 2020; Kipf et al., 2021) and other inference modules, which can be run on the disentangled dynamic groundplan which, in contrast to image space, enables shift-equivariant processing free from perspective distortion and encodes 3D structure.

We demonstrate that our method infers a 3D scene representation from a single image while disentangling static and dynamic components of the scene into static and dynamic groundplans respectively. We then show that connected components analysis suffices to leverage the densities in the dynamic groundplan for instance-level segmentation, bounding box prediction, and scene editing.

The separate reconstruction of movable scene components in the dynamic groundplan enables object detection via instance-level segmentation and bounding box prediction. Fig. 6 presents the instance-level segmentation and 3D bounding box prediction results of the proposed 3D object discovery via connected component discovery using the density inferred using the dynamic groundplan from the bird's-eye view.

As the dynamic groundplan does not encode static scene regions such as the street below cars, cars can easily be moved from one scene to another.

We show the potential of neural ground plans as a representation that may enable data-efficient solutions to downstream processing of the 3D scene, such as completion, instance-level segmentation, 3D bounding box prediction, and 3D scene editing.

This results, like in the monocular case, in a 3D volume of \(128\times 64\times 16\times 64\) (\(CH\times X\times Y\times Z\)).

To construct an entangled groundplan, we aggregate the 3D feature volume into a 2D groundplan using a pillar aggregation MLP. Consider a pillar of features orthogonal to the groundplan for a particular point \((x,z)\), in our case that is \(128\times 16\). The MLP takes in each latent with its corresponding 3D coordinate in the world coordinate system. It outputs the softmax scores for each of these latents which can be used to sum these latents along the pillar into a single 128-dimensional latent, as explained in the main paper.

This shallow CNN outputs groundplan of shape \(256\times 256\times 256\), in which the first 128 channels of the feature tensor are attributed as the static groundplan and the rest of the 128 channels are used as the dynamic groundplan for that timestep.

Our method performs project of a 3D point on the image plane to get the corresponding feature tensor. This is performed using the intrinsic matrix of the camera. For a given 3D point in camera coordinates \(\mathbf{x_{c}}\) and intrinsic matrix \(\mathbf{K}\), the resulting 2D point on the image plane is computed as \(\mathbf{Kx}\). For projecting a 3D query point \(\mathbf{x}\) on the groundplan, we use grid sample using \((x,z)\)-coordinates of the 3D query points \(\mathbf{x}\).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate the claim: "By compactifying 3D space with a nonlinear mapping, neural groundplans can encode unbounded 3D scenes in a bounded region."

1. Evaluation Summary:
The claim is well-stated and directly supported by the evidence. The paper clearly describes the compactification process using a non-linear coordinate re-mapping method from Barron et al. (2021), with specific mathematical details. The evidence shows that points within radius r_inner remain unaffected while points outside are contracted, and importantly, the transformation is invertible (C^(-1)). The implementation details and mathematical formulation are thoroughly explained, demonstrating how unbounded scenes can be represented in a bounded region.

2. <score>1</score>

Justification:
- The claim directly reflects the technical implementation described in the evidence without exaggeration
- The evidence provides explicit mathematical formulation: "Points x within a radius r_inner around the groundplan origin remain unaffected, but points outside this radius are contracted"
- The invertibility of the transformation is clearly stated: "C is invertible, such that x=C^(-1)(x') is a function that takes a 3D point in contracted space x' to the original 3D point x in linear space"
- The practical implementation details are provided, including specific dimensions (128×64×16×64) and processing steps
- The evidence demonstrates actual applications of this approach in scene representation, object detection, and editing