Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We also seek to address the multi-agent emergent communications by proposing a collaborative learning framework that enables the agents to generate and understand emergent language and solve task.

The review comment to be evaluated is:

The paper is not clearly written and some sections are hard to understand. The related works do capture how the framework proposes new challenges that are not already captured in previous work. Besides a similar work that also uses communication between agents [2] perform a similar analysis and comparisons to that work would be help in highlighting the difference with the proposed framework.

Besides the scope of the proposed framework is limited as it can only handle artificial messages that have a much smaller length, vocabulary, and sources of variation. In contrast, previous work have already proposed similar benchmarks that involve real images with natural language descriptions [1].

[1] de Vries et al 2018. Talk the Walk: Navigating New York City through Grounded Dialogue.

[2] Singh et al. 2019. Learning when to Communicate at Scale in Multiagent Cooperative and Competitive Tasks.

The evidence to be evaluated is:Navigation tasks with vision and natural language information have attracted much attention in the last few years. Generally, this task involves humans, an artificial agent, and the environment. The agent communicates with humans in natural language. The agent may receive requests or instructions and ask for more guidance from the human. It can navigate and interact with the environment to get more information or complete corresponding task requested by the human.

By putting the VLN task into multi-agent settings, the problems listed above can be alleviated or solved. To our knowledge, talk-the-walk de Vries et al. (2018) is the only work applying the VLN task with multiple agents and is very similar to our task settings. However, talk-the-walk only focuses on the localization part, where the guide finds where the tourist is. But ignore the guidance part, in which the guide makes a route plan and gives guidance to the tourist, and the tourist makes corresponding actions. Talk-the-walk models this part with a random walk and only implements a minimal baseline, making the VLN task with multi-agent settings incomplete. **Emergent Language** Emergent language is an unplanned language that comes up in interactions between the students, and/or the teacher and the students. From the perspective of agents' relationships, most of the works are focused on fully cooperative tasks like referential games Lazaridou et al. (2018) adapted from the Lewis signaling game. In the referential game, there are two roles: speaker and listener. The speaker has to give instructions to the listener so that it can select the target picture from the candidates successfully. Some navigation tasks are proposed Kalinowska et al. (2022); Kajic et al. (2020), however, those works can only be applied in limited environments, with 2 and 4 cases respectively. Some semi-cooperative tasks are also proposed, like the negotiation game Cao et al. (2018), in which two agents have to negotiate to get more scores for themselves. There are also some fully-competitive cases like a circular-based sender-receiver game Noukhovitch et al. (2021) where the sender tries to make the receiver choose the position closer to its own target position rather than the receiver's. Most of those part of works are studying single message pass, with one agent only having a language generation part, and the other one having a language understanding module. Still, there are some works putting attention to the multi-turn form Cao et al. (2018); Evtimova et al. (2017). But the former study focuses on a semi-cooperative case rather than a fully cooperative one like us, while the latter one uses the bag of words, and generates 0 or 1, with 1 indicating this token is in the generated sentence.

This task is inspired by a real-world scenario where two agents with unequal abilities, agents A and B, cooperate to help agent A find the target place. As pictured in Figure 1, agent A can only see the objects around it, while agent B has the floor plan and the target place but doesn't know where A is. To win this game, they have to cooperate to help A reach the target place with limited rounds.

Agent B needs to find where A is, according to the message given by A describing the objects around it.

Having the position of A and the target place, agent B has to find the shortest path for A to get there.

Agent B gives instructions to A, helping it follow the route plan.

Formally, this navigation task is constructed as follows: There are houses, containing rooms \(R=\{r_{0},r_{1},...,r_{N}\}\) each. Every time, one house is randomly chosen from those \(K\) candidates. Then two rooms are drawn from the selected house, and one of them is the target room \(r_{o}\), and the other is the start room \(r_{s}\). Every room has several gates \(D_{i}=\{d_{ij},j\in Neighbor(i)\}\). There is a graph for each house, to model the relationship between rooms. We use \(G=(R,D)\) to denote a house, where \(R\) is the vertex set, and \(D\) is the edge set. There are two players called agent A, and agent B parameterized by \(\theta_{A},\theta_{B}\) separately.

* In the localization task, agent A takes in the room's vector where it is and gives out the messages of step t consisting of \(M\) tokens \(m_{t}=\{v_{0},v_{1},...,v_{M}\},v_{i}\in V\). While agent B collects history dialogue \(h_{t}=(m_{0},m_{1},..,m_{t})\) and tries to guess where A is.

* In the route plan task, agent B uses the room graph \(G\), the guessed position of A \(\arg\max(\theta_{B}(h_{t},G))\), and the target position \(r_{o}\) to get the next step A should take \(\arg\max(\theta_{A}(h_{t+1},D))\).

Agent A encodes the target room information \(r_{o}\) including objects and gates and then uses the encoded vector to generate corresponding emergent language with its language generation module \(e_{s}^{o}=GRU(MLP(r_{o}))\). With the message coming from agent A in this turn, and the history sentences from A, agent B makes its guess by making use of the topological structure of rooms \(\theta_{B}(h_{t},G)\).

Similarly, the model for the guidance task also includes an encoder, a language generation, and a language understanding module used for giving instructions to agent A about its next step. Considering the non-differentiability of the discrete messages passing between agents, those models are trained with REINFORCE algorithm.

Finally, we put those sub-tasks together to complete the navigation task.

With the limitations of the message length and vocabulary size in discrete communication scenarios, discrete VAE may communicate less useful information compared with our model. We also show the case when agents can pass continuous messages, giving us a ceiling performance when the communication channel is not restricted.

We suppose that good language tends to give similar expressions describing a similar object. So we use topographic similarity \(\rho\) as our metric, which is the Spearman correlation between the message similarity sequence and the room vector similarity sequence. And our method outperforms the discrete VAE largely.

Then, we cluster those rooms with the generated message. Labeled these rooms with their objects' color, shape and direction separately, we get the corresponding purity. And the purity score can tell us the information that the emergent language encoded. From the table, we can tell that compared with discrete VAE, our model focus more on the direction and less on color and shape which is reasonable because there are only 3 choices for color and shape which makes it less useful for disguising the target room from others, comparing with the direction with 8 possible value. We also investigate factors that may affect task performance 6. Countering with our intuitions that, as the sentence length grows, the model has a much more powerful expression ability, so the performance should grow, but localization accuracy grows first and then drop. When the sentence length is enough for expression, much more tokens may increase the difficulty for the model to tackle. The shortest route length between the start positions and the target ones can also affect the task performance, but it does not follow an exponential drop when the shortest route length grows. In this work, we propose a navigation task in an emergent communication scenario.

However, compared with the traditional VLN task, the application of multi-agent communication in the VLN task makes the communication policy more flexible and can generate a more complex

* Kajic et al. (2020) Ivana Kajic, Eser Aygun, and Doina Precup. Learning to cooperate: Emergent communication in multi-agent navigation. _arXiv preprint arXiv:2004.01097_, 2020.

* Lazaridou et al. (2018) Angeliki Lazaridou, Karl Moritz Hermann, Karl Tuyls, and Stephen Clark. Emergence of linguistic communication from referential games with symbolic and pixel input, 2018. URL [https://arxiv.org/abs/1804.03984](https://arxiv.org/abs/1804.03984). * Narayan-Chen et al. (2019) Anjali Narayan-Chen, Prashant Jayannavar, and Julia Hockenmaier. Collaborative dialogue in Minecraft. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pp. 5405-5415, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1537. URL [https://aclanthology.org/P19-1537](https://aclanthology.org/P19-1537).

Input: Figure 1: The navigation task involves three components: the tourist (agent A), the guide (agent B), and the environment. Agent A and B have unequal abilities. While agent A can only see its surroundings and describe them to B, agent B has the global perspective, can make a route plan, and give instructions to A. Agent A and B cooperate to lead A to the target place.

Input: Figure 3: Model Structure. There are three sub-modules here, including localization, route plan, and guidance. In the localization part, by calculating the similarity score between messages given by A and room feature embeddings incorporating structural information, agent B tries to find where A is. Then we use the Floyd algorithm to find the shortest path between the guessed place and the target one. And in the guidance module, A could find its path with history dialogue with B and its own observations.

Input: Figure 2: Navigation Case. The navigation task consists of three sub-tasks: Localization, Route Plan, and Guidance. At every turn, agent A first gives a description of its surroundings in the room it is located. With the current messages and the history messages given by A, agent B tries to guess where A is using the topological structure of the rooms. Then, based on this guess, agent B makes a route plan for A to get to the target place, and gives an instruction to A describing the next step A should take.

Input: Figure 6: The effect of (a) sentence length and (b) the shortest route length between the start and target places on the task performance

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find the claim to be somewhat overstated. While the paper does present a collaborative learning framework for emergent communication between agents, there are several limitations noted in both the review comments and evidence:

1. The framework is limited to artificial messages with smaller length and vocabulary compared to existing work with natural language
2. Similar work already exists (e.g., talk-the-walk) that the paper builds upon
3. The experiments show mixed results - performance decreases with longer sentences which contradicts claims about communication capabilities
4. The review points out unclear writing and insufficient comparison to related work

The evidence shows the framework does enable basic emergent communication between agents, but the claim overstates its capabilities and novelty.

<score>4</score>

Justification:
- The claim presents the framework as a novel solution but doesn't acknowledge significant limitations in message complexity compared to existing work with natural language, as noted in review comment
- Evidence shows mixed results - longer messages actually hurt performance in some cases (Fig 6a), contradicting implied benefits
- Review highlights similar existing work [1][2] that isn't adequately differentiated
- Phrase "enable the agents to generate and understand emergent language and solve task" overstates capabilities given the artificial/limited nature of the messages
- Evidence shows task performance declines with route length (Fig 6b), suggesting limitations in scalability not acknowledged in claim


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Results show that the proposed framework achieves competing performance in both accuracy of language understanding and in task success rate.

The evidence to be evaluated is:We also show the case when agents can pass continuous messages, giving us a ceiling performance when the communication channel is not restricted. To evaluate the generated language, we use topographic similarity and purity. We suppose that good language tends to give similar expressions describing a similar object. So we use topographic similarity \(\rho\) as our metric, which is the Spearman correlation between the message similarity sequence and the room vector similarity sequence. And our method outperforms the discrete VAE largely.

Input: Figure 6: The effect of (a) sentence length and (b) the shortest route length between the start and target places on the task performance

Input: Figure 4: The change of accuracy over epoch in different tasks. (a) localization task. (b) navigation task

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate the claim "Results show that the proposed framework achieves competing performance in both accuracy of language understanding and in task success rate."

Analysis:
The evidence shows mixed performance that doesn't fully support characterizing the results as "competing":

1. For localization, accuracy reaches ~85% after many epochs
2. For navigation, accuracy fluctuates around 80-85%
3. Performance decreases with longer route lengths
4. Language understanding is only evaluated via topographic similarity, with limited comparative analysis
5. The term "competing" implies comparison with other methods, but the evidence only shows comparison to a discrete VAE baseline

The claim somewhat overstates the strength of the results by using "competing" without comprehensive benchmarking against other approaches. While the framework shows reasonable performance, the characterization could be more precise about the specific achievements and limitations.

<score>3</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Results show that the proposed framework achieves competing performance in both accuracy of language understanding and in task success rate.

The review comment to be evaluated is:

Strengths:
* The proposed setting is novel.

Weaknesses:
* While novel, the proposed setting is poorly motivated. Why studying this setting is significant? What would it enable?
* The description of the environment is very nebulous. Especially, the complexity of the visual and language components are highly under-specified. How is the input observation represented (image or feature-based)? How many object categories? How large is the agent's vocabulary? How are the environment graph generated? It is thus difficult to comprehend the significance of solving this environment.
* The plots are missing confidence intervals. Each experiments are not conducted multiple times with different random seeds. This seriously weakens the empirical claims.

The evidence to be evaluated is:We also show the case when agents can pass continuous messages, giving us a ceiling performance when the communication channel is not restricted. To evaluate the generated language, we use topographic similarity and purity. We suppose that good language tends to give similar expressions describing a similar object. So we use topographic similarity \(\rho\) as our metric, which is the Spearman correlation between the message similarity sequence and the room vector similarity sequence. And our method outperforms the discrete VAE largely.

Input: Figure 6: The effect of (a) sentence length and (b) the shortest route length between the start and target places on the task performance

Input: Figure 4: The change of accuracy over epoch in different tasks. (a) localization task. (b) navigation task

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence:

The claim that the framework "achieves competing performance" is significantly overstated based on the available evidence. While the figures show reasonable accuracy levels (around 80-85% for various tasks), several critical issues exist:

1. As noted in the review comments, there are no confidence intervals or multiple runs with different random seeds, making it impossible to assess statistical significance.

2. The term "competing performance" implies comparison with other methods, but the evidence only shows comparison with a "discrete VAE" baseline.

3. The environment's complexity and specifications are poorly defined, making it difficult to evaluate the significance of the performance.

4. The graphs show considerable variation in performance, particularly in the navigation task, suggesting potential instability.

The claim needs more rigorous experimental validation and clearer benchmarking against competing methods to support such a statement.

<score>4</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Results show that the proposed framework achieves competing performance in both accuracy of language understanding and in task success rate.

The review comment to be evaluated is:

While this paper goes in a good direction by moving away from the Lewis game in the language emergence communication setting, it falls short as there is little scientific analysis. The authors mainly designed an artificial navigation task, handcrafted an observation space and reward, designed a neural network, trained it, and reported the results. Those results are not put into perspective with a detailed researched perspective. Indeed, the point of Emecom in ML is not to create artificial tasks for the sake of solving it, but to design tasks that could help understand how language evolves with a human intervention as possible (or being generic).

I am listing here a few leads to improve the paper in the long run:
 - The introduction should frame the research problem into the past literature, ask a scientific question, and explain how the paper help answers it. Here, there is no citation; the VLM task is only said to be more natural (no reference to the embodiment, situated learning, or going beyond the signaling game)
 - The related work is extremely short. 18 citations in total for a paper about language emergence, and the navigation task is just inherently incomplete. On a positive note, the most relevant citations kajic/Kanilowska/DeVries are present
- The paper is 7 pages long, and page 3 is barely empty
- The model is designed to solve the task, e.g., the graph networks are specific to the observation space. I would recommend either picking a minimal architecture or reproducing the architecture that is classic in VLM.
- having 4 losses in an RL task is generally a bad sign as it is a symptom of human intervention. cf The bitter lesson by Sutton
 - Results are not compared to any baselines (VAE vs your network is not really a baseline...), and simply reporting the success rate is not sufficient. Examples of suitable ablation include: the correlation between language and environment complexity. Qualitative experiments to translate language and change in the environment (e.g., how changing the graph impacts the language), what are the external constraints that shape language, reward ablation, etc.

I would recommend the authors to re-work their paper and present it to a workshop to have further guidance. The initial idea is good, but the realization lacks scientific maturity.

The evidence to be evaluated is:We also show the case when agents can pass continuous messages, giving us a ceiling performance when the communication channel is not restricted. To evaluate the generated language, we use topographic similarity and purity. We suppose that good language tends to give similar expressions describing a similar object. So we use topographic similarity \(\rho\) as our metric, which is the Spearman correlation between the message similarity sequence and the room vector similarity sequence. And our method outperforms the discrete VAE largely.

Input: Figure 6: The effect of (a) sentence length and (b) the shortest route length between the start and target places on the task performance

Input: Figure 4: The change of accuracy over epoch in different tasks. (a) localization task. (b) navigation task

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence:

The claim that "the framework achieves competing performance" appears overstated given the available evidence. While the graphs show reasonable accuracy levels (around 80-85%), several key issues exist:

1. No comparative baselines are shown except for a VAE comparison mentioned briefly
2. The reviewer correctly points out the lack of scientific analysis and proper benchmarking
3. The performance graphs show significant volatility, particularly in the navigation task
4. The sentence length and route length effects suggest performance limitations under certain conditions
5. Without comparative metrics against other frameworks, claiming "competing performance" is not well-supported

The evidence shows the framework works, but claiming it "achieves competing performance" implies comparison with other methods, which is not demonstrated in the provided evidence.

<score>4</score>

Justification:
- The claim implies competitive comparison but lacks proper baselines and comparative analysis
- The reviewer specifically notes "Results are not compared to any baselines" and that "simply reporting the success rate is not sufficient"
- The graphs show decent but volatile performance metrics
- Performance limitations are evident in the sentence/route length analyses
- The term "competing performance" suggests benchmarking that isn't present in the evidence


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Results show that the proposed framework achieves competing performance in both accuracy of language understanding and in task success rate.

The review comment to be evaluated is:

The paper is not clearly written and some sections are hard to understand. The related works do capture how the framework proposes new challenges that are not already captured in previous work. Besides a similar work that also uses communication between agents [2] perform a similar analysis and comparisons to that work would be help in highlighting the difference with the proposed framework.

Besides the scope of the proposed framework is limited as it can only handle artificial messages that have a much smaller length, vocabulary, and sources of variation. In contrast, previous work have already proposed similar benchmarks that involve real images with natural language descriptions [1].

[1] de Vries et al 2018. Talk the Walk: Navigating New York City through Grounded Dialogue.

[2] Singh et al. 2019. Learning when to Communicate at Scale in Multiagent Cooperative and Competitive Tasks.

The evidence to be evaluated is:We also show the case when agents can pass continuous messages, giving us a ceiling performance when the communication channel is not restricted. To evaluate the generated language, we use topographic similarity and purity. We suppose that good language tends to give similar expressions describing a similar object. So we use topographic similarity \(\rho\) as our metric, which is the Spearman correlation between the message similarity sequence and the room vector similarity sequence. And our method outperforms the discrete VAE largely.

Input: Figure 6: The effect of (a) sentence length and (b) the shortest route length between the start and target places on the task performance

Input: Figure 4: The change of accuracy over epoch in different tasks. (a) localization task. (b) navigation task

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find the statement "achieves competing performance" to be somewhat overstated. While the evidence shows reasonable performance metrics (accuracy reaching ~0.8-0.85 for both localization and navigation tasks), there are several limitations:

1. The review comments highlight that the framework only handles artificial messages with limited length/vocabulary compared to previous work with natural language.
2. No direct performance comparisons are shown against other frameworks, making it impossible to verify if it truly "competes" with alternatives.
3. The graphs show decent but not exceptional performance, with some instability in the navigation task.
4. The reviewer notes missing comparisons to similar work [2] that would help validate competitive claims.

A more measured claim would acknowledge these limitations and specify the exact performance metrics achieved rather than making broad competitive assertions.

<score>4</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

\(\nabla Sim\) performs differentiable physics simulation and rendering at independent and adjustable rates, allowing us to trade computation for accuracy by rendering fewer frames than dynamics updates.

The evidence to be evaluated is:Each of our experiments comprises an _environment_\(\mathcal{E}\) that applies a particular set of physical forces and/or constraints, a (differentiable) _loss function_\(\mathcal{L}\) that implicitly specifies an objective, and an _initial guess_\(\theta_{0}\) of the physical state of the simulation. The goal is to recover optimal physics parameters \(\theta^{*}\) that minimize \(\mathcal{L}\), by backpropagating through the simulator.

Inference with \(\nabla\)_Sim_ is done by guessing an initial mass (uniformly random in the range \([2,12]kg/m^{3}\)),rolling a _differentiable_ simulation using this guess, comparing the rendered out video with the true video (pixelwise mean-squared error - MSE), and performing gradient descent updates.

We compare \(\nabla\)_Sim_ with three other baselines: PyBullet + REINFORCE (Ehsani et al., 2020; Wu et al., 2015), diff. physics only (requiring 3D supervision), and a ConvLSTM baseline adopted from Xu et al. (2019) but with a stronger backbone. The _DiffPhysics_ baseline is a strict subset of \(\nabla\)_Sim_, it only inolves the differentiable physics engine.

Nevertheless, \(\nabla\)_Sim_ is able to very precisely estimate mass from video, to a absolute relative error of 9.01e-5, nearly two orders of magnitude better than the ConvLSTM baseline.

We conduct a series of experiments to investigate the ability of \(\nabla Sim\) to recover physical parameters of deformable solids and thin-shell solids (cloth).

We found our method is able to accurately recover the parameters of \(100\) instances of deformable objects (cloth, balls, beams) as reported in Table 3 and Fig. 3. Since \(\nabla Sim\) is a complex combination of differentiable non-linear components, we analyze the loss landscape to verify the validity of gradients through the system.

\(\nabla Sim\) is able to _solve_ this environment within three iterations of gradient descent, by minimizing a pixelwise MSE between the last frame of the rendered video and the goal image as shown in Fig. 5 (lower left). In our second test, we formulate a more challenging 3D control problem (control-fem) where the goal is to actuate a soft-body FEM object (a _geear_) consisting of \(1152\) tetrahedral elements to move to a target position as shown in Fig. 5 (center).

We also train a privileged baseline (_diffphysics_) that uses strong supervision and minimizes the MSE between the target position and the precise 3D location of the center-of-mass (COM) of the FEM model at each time step (i.e. a _dense_ reward). We test both _diffphysics_ and \(\nabla Sim\) against a naive baseline that generates random activations and plot convergence behaviors in Fig. 5(a). While _diffphysics_ appears to be a strong performer on this task, it is important to note that it uses explicit 3D supervision at each timestep (i.e. \(30\) FPS). In contrast, \(\nabla Sim\) uses a _single image_ as an implicit target, and yet manages to achieve the goal state, albeit taking a longer number of iterations.

Being a _white box_ method, the performance of \(\nabla\)_Sim_ relies on the choice of dynamics and rendering models employed. An immediate question that arises is "_how would the performance of \(\nabla\)_Sim_ be impacted (if at all) by such modeling choices._" We conduct multiple experiments targeted at investigating modelling errors and summarize them in Table 4 (left).

Table 4 (right) shows simulation rates for the forward and backward passes of each module. We report forward and backward pass rates separately for the differentiable physics (DP) and the differentiable rendering (DR) modules. The time complexity of \(\nabla\mathit{Sim}\) is a function of the number of tetrahedrons and/or triangles. We illustrate the arguably more complex case of deformable object simulation for varying numbers of tetrahedra (ranging from \(100\) to \(10000\)). Even in the case of \(10000\) tetrahedra--enough to contruct complex mesh models of multiple moving objects--\(\nabla\mathit{Sim}\) enables faster-than-real-time simulation (\(1500\) steps/second). We presented \(\nabla\mathit{Sim}\), a versatile differentiable simulator that enables system identification from videos by differentiating through physical processes governing dyanmics and image formation.

However, we instead consider our physics engine as a differentiable operation that provides an implicit relationship between a state vector \(\mathbf{s}^{-}=\mathbf{s}(t)\) at the start of a time step, and the updated state at the end of the time step \(\mathbf{s}^{+}=\mathbf{s}(t+\Delta t)\). An arbitrary discrete time integration scheme can be then be abstracted as the function \(\mathbf{g}(\mathbf{s}^{-},\mathbf{s}^{+},\theta)=\mathbf{0}\), relating the initial and final system state and the model parameters \(\theta\). By the implicit function theorem, if we can specify a loss function \(l\) at the output of the simulator, we can compute \(\frac{\partial l}{\partial\mathbf{s}^{-}}\) as \(\mathbf{c}^{T}\frac{\partial\mathbf{g}}{\partial\mathbf{s}^{-}}\), where \(\mathbf{c}\) is the solution to the linear system \(\frac{\partial\mathbf{g}}{\partial\mathbf{s}^{+}}^{T}\mathbf{c}=-\frac{ \partial l}{\partial\mathbf{s}^{+}}^{T}\), and likewise for the model parameters \(\theta\). While the partial derivatives \(\frac{\partial\mathbf{g}}{\partial\mathbf{s}^{-}},\frac{\partial\mathbf{g}}{ \partial\mathbf{s}^{+}},\frac{\partial\mathbf{g}}{\partial\mathbf{s}}\) can be computed by graph-based automatic differentiation frameworks Paszke et al. (2019); Abadi et al. (2015); Bradbury et al. (2018), program transformation approaches such as DiffTaichi, and Google Tangent Hu et al. (2020); van Merrienboer et al. (2018) are particularly well-suited to simulation code. We use an embedded subset of Python syntax, which computes the adjoint of each simulation kernel at runtime, and generates C++/CUDA Kirk et al. (2007) code. Kernels are wrapped as custom autograd operations on PyTorch tensors, which allows users to focus on the definition of physical models, and leverage the PyTorch tape-based autodiff to track the overall program flow. While this formulation is general enough to represent explicit, multi-step, or fully implicit time-integration schemes, we employ semi-implicit Euler integration, which is the preferred integration scheme for most simulators Erez et al. (2015).

\(\nabla\)_Sim_ integrates several functional blocks, many of which contain nonlinear operations. Furthermore, we employ a pixelwise mean-squared error (MSE) loss function for estimating physical parameters from video. To demonstrate whether the gradients obtained from \(\nabla\)_Sim_ are relevant for the task of physical parameter estimation, in Figure 2 of the main paper, we present an analysis of the MSE loss landscape for mass estimation.

Most of the models presented in \(\nabla Sim\) can be trained and evaluated on modern laptops equipped with graphics processing units (GPUs). We find that, on a laptop with an Intel i7 processor and a GeForce GTX 1060 GPU, parameter estimation experiments for rigid/nonrigid bodies can be run in under \(5\)-\(20\) minutes per object on CPU and in under \(1\) minute on the GPU.

Input: Table 3: Parameter estimation of deformable objects: We estimate per-particle masses and material properties (for solid def. objects) and per-particle velocities for cloth. In the case of cloth, there is a perceivable performance drop in diffphysics, as the center of mass of a cloth is often outside the body, which results in ambiguity.

Input: Figure 2: ∇Sim: Given video observations of an evolving physical system (e), we randomly initialize scene object properties (a) and evolve them over time using a differentiable physics engine (b), which generates states. Our renderer (c) processes states, object vertices and global rendering parameters to produce image frames for computing our loss. We backprop through this computation graph to estimate physical attributes and controls. Existing methods rely solely on differentiable physics engines and require supervision in state-space (f), while ∇Sim only needs image-space supervision (g).

Input: Table 4: (Left) Impact of imperfect models: The accuracy of physical parameters estimated by ∇Sim is impacted by the choice of dynamics and graphics (rendering) models. We find that the system is more sensitive to the choice of dynamics models than to the rendering engine used. (Right) Timing analysis: We report runtime in simulation steps / second (Hz). ∇Sim is significantly faster than real-time, even for complex geometries.

Input: Figure 12: Impact of the length of a video sequence on the loss landscape. Notice how the loss landscape is much steeper for smaller videos (e.g., MSE of first and last frames). Nonetheless, all cases have a smooth loss landscape with the same unique minimum.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find the claim is generally well-stated but has minor overstatements. The evidence shows ∇Sim can indeed perform differentiable physics simulation and rendering, with Table 4 demonstrating separate forward/backward pass rates for physics and rendering modules. The evidence confirms adjustable rates (e.g., "MSE on every 5th frame" in Fig 12) and performance tradeoffs. However, the claim could be more precise about the exact nature of the computation-accuracy tradeoff, which isn't extensively quantified in the evidence. The system's capabilities are demonstrated but slightly oversimplified in the claim.

<score>2</score>

Justification:
1. The evidence strongly supports the core technical capability through:
- Table 4 showing separate simulation/rendering rates
- Fig 12 demonstrating different frame sampling approaches
- Timing analysis showing faster-than-realtime performance (1500 steps/sec)

2. Minor overstatement concerns:
- The claim implies a clear computation-accuracy tradeoff that isn't thoroughly quantified in the evidence
- While frame sampling is shown possible (Fig 12), the specific impacts on accuracy vs computation aren't extensively documented
- The independence between physics and rendering rates is demonstrated but its practical implications aren't fully explored

The score of 2 reflects that while the claim is mostly accurate, it could be more precise in characterizing the tradeoffs involved.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

\(\nabla Sim\) performs differentiable physics simulation and rendering at independent and adjustable rates, allowing us to trade computation for accuracy by rendering fewer frames than dynamics updates.

The review comment to be evaluated is:

This work focuses on the problem of estimating object physical properties from video sequences.

The proposed framework combines differentiable physical simulations and differentiable rendering to map physical parameters into images differentiably. This paradigm is then used to recover physical parameters from image sequences by means of gradient based optimisation.

Validation of the proposed method is carried through two main synthetic applications, parameter identification and visuomotor control.

Although the proposed approach still requires 3D ground truth information to yield reliable estimates, it is and encouraging step towards unsupervised physics understanding from image/video data.

Positive:

-Crucially and differently from previous attempts, the proposed approach does not require 3D supervision - except for geometry and appearance of the static scene (i.e. at t=0).

-Approach is clever, simple and yields interpretable representation

-First step towards physics understanding from videos

Negative:

- I would improve the quality of the visualisations and plots in the paper (e.g. I found Figure 6 impossible to read)

-  How to differentiate through the physical simulator was not obvious to me. I would have appreciated a more detailed explanation of how that is done in practice for one of the physical problems studied in the paper to be included in the main manuscript, in an effort to make the paper more readable.

The evidence to be evaluated is:Each of our experiments comprises an _environment_\(\mathcal{E}\) that applies a particular set of physical forces and/or constraints, a (differentiable) _loss function_\(\mathcal{L}\) that implicitly specifies an objective, and an _initial guess_\(\theta_{0}\) of the physical state of the simulation. The goal is to recover optimal physics parameters \(\theta^{*}\) that minimize \(\mathcal{L}\), by backpropagating through the simulator.

Inference with \(\nabla\)_Sim_ is done by guessing an initial mass (uniformly random in the range \([2,12]kg/m^{3}\)),rolling a _differentiable_ simulation using this guess, comparing the rendered out video with the true video (pixelwise mean-squared error - MSE), and performing gradient descent updates.

We compare \(\nabla\)_Sim_ with three other baselines: PyBullet + REINFORCE (Ehsani et al., 2020; Wu et al., 2015), diff. physics only (requiring 3D supervision), and a ConvLSTM baseline adopted from Xu et al. (2019) but with a stronger backbone. The _DiffPhysics_ baseline is a strict subset of \(\nabla\)_Sim_, it only inolves the differentiable physics engine.

Nevertheless, \(\nabla\)_Sim_ is able to very precisely estimate mass from video, to a absolute relative error of 9.01e-5, nearly two orders of magnitude better than the ConvLSTM baseline.

We conduct a series of experiments to investigate the ability of \(\nabla Sim\) to recover physical parameters of deformable solids and thin-shell solids (cloth).

We found our method is able to accurately recover the parameters of \(100\) instances of deformable objects (cloth, balls, beams) as reported in Table 3 and Fig. 3. Since \(\nabla Sim\) is a complex combination of differentiable non-linear components, we analyze the loss landscape to verify the validity of gradients through the system.

\(\nabla Sim\) is able to _solve_ this environment within three iterations of gradient descent, by minimizing a pixelwise MSE between the last frame of the rendered video and the goal image as shown in Fig. 5 (lower left). In our second test, we formulate a more challenging 3D control problem (control-fem) where the goal is to actuate a soft-body FEM object (a _geear_) consisting of \(1152\) tetrahedral elements to move to a target position as shown in Fig. 5 (center).

We also train a privileged baseline (_diffphysics_) that uses strong supervision and minimizes the MSE between the target position and the precise 3D location of the center-of-mass (COM) of the FEM model at each time step (i.e. a _dense_ reward). We test both _diffphysics_ and \(\nabla Sim\) against a naive baseline that generates random activations and plot convergence behaviors in Fig. 5(a). While _diffphysics_ appears to be a strong performer on this task, it is important to note that it uses explicit 3D supervision at each timestep (i.e. \(30\) FPS). In contrast, \(\nabla Sim\) uses a _single image_ as an implicit target, and yet manages to achieve the goal state, albeit taking a longer number of iterations.

Being a _white box_ method, the performance of \(\nabla\)_Sim_ relies on the choice of dynamics and rendering models employed. An immediate question that arises is "_how would the performance of \(\nabla\)_Sim_ be impacted (if at all) by such modeling choices._" We conduct multiple experiments targeted at investigating modelling errors and summarize them in Table 4 (left).

Table 4 (right) shows simulation rates for the forward and backward passes of each module. We report forward and backward pass rates separately for the differentiable physics (DP) and the differentiable rendering (DR) modules. The time complexity of \(\nabla\mathit{Sim}\) is a function of the number of tetrahedrons and/or triangles. We illustrate the arguably more complex case of deformable object simulation for varying numbers of tetrahedra (ranging from \(100\) to \(10000\)). Even in the case of \(10000\) tetrahedra--enough to contruct complex mesh models of multiple moving objects--\(\nabla\mathit{Sim}\) enables faster-than-real-time simulation (\(1500\) steps/second). We presented \(\nabla\mathit{Sim}\), a versatile differentiable simulator that enables system identification from videos by differentiating through physical processes governing dyanmics and image formation.

However, we instead consider our physics engine as a differentiable operation that provides an implicit relationship between a state vector \(\mathbf{s}^{-}=\mathbf{s}(t)\) at the start of a time step, and the updated state at the end of the time step \(\mathbf{s}^{+}=\mathbf{s}(t+\Delta t)\). An arbitrary discrete time integration scheme can be then be abstracted as the function \(\mathbf{g}(\mathbf{s}^{-},\mathbf{s}^{+},\theta)=\mathbf{0}\), relating the initial and final system state and the model parameters \(\theta\). By the implicit function theorem, if we can specify a loss function \(l\) at the output of the simulator, we can compute \(\frac{\partial l}{\partial\mathbf{s}^{-}}\) as \(\mathbf{c}^{T}\frac{\partial\mathbf{g}}{\partial\mathbf{s}^{-}}\), where \(\mathbf{c}\) is the solution to the linear system \(\frac{\partial\mathbf{g}}{\partial\mathbf{s}^{+}}^{T}\mathbf{c}=-\frac{ \partial l}{\partial\mathbf{s}^{+}}^{T}\), and likewise for the model parameters \(\theta\). While the partial derivatives \(\frac{\partial\mathbf{g}}{\partial\mathbf{s}^{-}},\frac{\partial\mathbf{g}}{ \partial\mathbf{s}^{+}},\frac{\partial\mathbf{g}}{\partial\mathbf{s}}\) can be computed by graph-based automatic differentiation frameworks Paszke et al. (2019); Abadi et al. (2015); Bradbury et al. (2018), program transformation approaches such as DiffTaichi, and Google Tangent Hu et al. (2020); van Merrienboer et al. (2018) are particularly well-suited to simulation code. We use an embedded subset of Python syntax, which computes the adjoint of each simulation kernel at runtime, and generates C++/CUDA Kirk et al. (2007) code. Kernels are wrapped as custom autograd operations on PyTorch tensors, which allows users to focus on the definition of physical models, and leverage the PyTorch tape-based autodiff to track the overall program flow. While this formulation is general enough to represent explicit, multi-step, or fully implicit time-integration schemes, we employ semi-implicit Euler integration, which is the preferred integration scheme for most simulators Erez et al. (2015).

\(\nabla\)_Sim_ integrates several functional blocks, many of which contain nonlinear operations. Furthermore, we employ a pixelwise mean-squared error (MSE) loss function for estimating physical parameters from video. To demonstrate whether the gradients obtained from \(\nabla\)_Sim_ are relevant for the task of physical parameter estimation, in Figure 2 of the main paper, we present an analysis of the MSE loss landscape for mass estimation.

Most of the models presented in \(\nabla Sim\) can be trained and evaluated on modern laptops equipped with graphics processing units (GPUs). We find that, on a laptop with an Intel i7 processor and a GeForce GTX 1060 GPU, parameter estimation experiments for rigid/nonrigid bodies can be run in under \(5\)-\(20\) minutes per object on CPU and in under \(1\) minute on the GPU.

Input: Table 3: Parameter estimation of deformable objects: We estimate per-particle masses and material properties (for solid def. objects) and per-particle velocities for cloth. In the case of cloth, there is a perceivable performance drop in diffphysics, as the center of mass of a cloth is often outside the body, which results in ambiguity.

Input: Figure 2: ∇Sim: Given video observations of an evolving physical system (e), we randomly initialize scene object properties (a) and evolve them over time using a differentiable physics engine (b), which generates states. Our renderer (c) processes states, object vertices and global rendering parameters to produce image frames for computing our loss. We backprop through this computation graph to estimate physical attributes and controls. Existing methods rely solely on differentiable physics engines and require supervision in state-space (f), while ∇Sim only needs image-space supervision (g).

Input: Table 4: (Left) Impact of imperfect models: The accuracy of physical parameters estimated by ∇Sim is impacted by the choice of dynamics and graphics (rendering) models. We find that the system is more sensitive to the choice of dynamics models than to the rendering engine used. (Right) Timing analysis: We report runtime in simulation steps / second (Hz). ∇Sim is significantly faster than real-time, even for complex geometries.

Input: Figure 12: Impact of the length of a video sequence on the loss landscape. Notice how the loss landscape is much steeper for smaller videos (e.g., MSE of first and last frames). Nonetheless, all cases have a smooth loss landscape with the same unique minimum.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me evaluate the claim that "∇Sim performs differentiable physics simulation and rendering at independent and adjustable rates, allowing us to trade computation for accuracy by rendering fewer frames than dynamics updates."

1. Summary of evaluation:
The evidence strongly supports this claim through:
- Table 4 showing separate forward/backward pass rates for physics (DP) and rendering (DR) modules
- Performance metrics demonstrating faster-than-real-time simulation (1500 steps/second) even with 10,000 tetrahedra
- Figure 12 showing results with different frame sampling rates (every frame vs every 5th frame)
- Detailed technical explanation of the system architecture separating physics and rendering components

The claim is well-stated and accurately reflects the system's capabilities without exaggeration. The evidence provides concrete performance metrics and demonstrates the independent operation of physics and rendering components.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

\(\nabla Sim\) performs differentiable physics simulation and rendering at independent and adjustable rates, allowing us to trade computation for accuracy by rendering fewer frames than dynamics updates.

The review comment to be evaluated is:

This work presents a fully differentiable physics simulation coupled with neural rendering such that input video can be used to estimate object properties or find control policies to move those objects by trying to generate the same video at the output.

The paper is well motivated by presenting a natural progression of ideas from this literature and it does a thorough job discussing related work. The paper is light on details in section 3 and it is necessary to refer to the appendix to get a complete picture. Overall, the technical contribution is solid and thus worth accepting the paper even if the validation is with relatively simpler experiments since they are sufficient to motivate this direction to be further researched. Below are a few comments to aid in improving the current work:

- All experiments use what I am guessing are input (desired) videos from the same pipeline and then later hiding some parameters (to be learned). While this is a good validation the learning done here is still 'in distribution'. It would be useful to see if video (even simplistic) from a different simulator or simplified from a real world video could be applied. To what extent is this possible and are there any fundamental limitations that prevent this at the moment?

- Analysis is mostly with one object in an empty scene. Are there technical limitations to handling realistic scenes where there are multiple objects and those objects interact with each other as well the environment? How does this affect performance wrt forward and backward pass timings? With such experiments, it would be helpful to understand if the released code can be easily extended to such (more complex) settings or if someone would need to start a new implementation from scratch.

- The scale on the loss landscape is quite small, '0.4 pixelwise mse'. How good does the initial guess need to be to stay in the range, do the curves in fig 3 continue the trend beyond these values for larger error?

- Reality gap: while this is discuss in reference to visual appearances, since the current experiments deal with synthetic scenes, the more relevant topic to discuss is the reality gap wrt physics and object motions. Experiments designed to study this would boost confidence in this approach.

Other comments:

- How much does the performance depend on good initial guess?

- Currently a single impulse is used to set things in motion, can this be extended to handle more continuous actions?

- Presenting qualitative results for baselines would be helpful

- Some baselines not clearly explained: average, random, ConvLSTM

- How does performance scale with the length of the video?

The evidence to be evaluated is:Each of our experiments comprises an _environment_\(\mathcal{E}\) that applies a particular set of physical forces and/or constraints, a (differentiable) _loss function_\(\mathcal{L}\) that implicitly specifies an objective, and an _initial guess_\(\theta_{0}\) of the physical state of the simulation. The goal is to recover optimal physics parameters \(\theta^{*}\) that minimize \(\mathcal{L}\), by backpropagating through the simulator.

Inference with \(\nabla\)_Sim_ is done by guessing an initial mass (uniformly random in the range \([2,12]kg/m^{3}\)),rolling a _differentiable_ simulation using this guess, comparing the rendered out video with the true video (pixelwise mean-squared error - MSE), and performing gradient descent updates.

We compare \(\nabla\)_Sim_ with three other baselines: PyBullet + REINFORCE (Ehsani et al., 2020; Wu et al., 2015), diff. physics only (requiring 3D supervision), and a ConvLSTM baseline adopted from Xu et al. (2019) but with a stronger backbone. The _DiffPhysics_ baseline is a strict subset of \(\nabla\)_Sim_, it only inolves the differentiable physics engine.

Nevertheless, \(\nabla\)_Sim_ is able to very precisely estimate mass from video, to a absolute relative error of 9.01e-5, nearly two orders of magnitude better than the ConvLSTM baseline.

We conduct a series of experiments to investigate the ability of \(\nabla Sim\) to recover physical parameters of deformable solids and thin-shell solids (cloth).

We found our method is able to accurately recover the parameters of \(100\) instances of deformable objects (cloth, balls, beams) as reported in Table 3 and Fig. 3. Since \(\nabla Sim\) is a complex combination of differentiable non-linear components, we analyze the loss landscape to verify the validity of gradients through the system.

\(\nabla Sim\) is able to _solve_ this environment within three iterations of gradient descent, by minimizing a pixelwise MSE between the last frame of the rendered video and the goal image as shown in Fig. 5 (lower left). In our second test, we formulate a more challenging 3D control problem (control-fem) where the goal is to actuate a soft-body FEM object (a _geear_) consisting of \(1152\) tetrahedral elements to move to a target position as shown in Fig. 5 (center).

We also train a privileged baseline (_diffphysics_) that uses strong supervision and minimizes the MSE between the target position and the precise 3D location of the center-of-mass (COM) of the FEM model at each time step (i.e. a _dense_ reward). We test both _diffphysics_ and \(\nabla Sim\) against a naive baseline that generates random activations and plot convergence behaviors in Fig. 5(a). While _diffphysics_ appears to be a strong performer on this task, it is important to note that it uses explicit 3D supervision at each timestep (i.e. \(30\) FPS). In contrast, \(\nabla Sim\) uses a _single image_ as an implicit target, and yet manages to achieve the goal state, albeit taking a longer number of iterations.

Being a _white box_ method, the performance of \(\nabla\)_Sim_ relies on the choice of dynamics and rendering models employed. An immediate question that arises is "_how would the performance of \(\nabla\)_Sim_ be impacted (if at all) by such modeling choices._" We conduct multiple experiments targeted at investigating modelling errors and summarize them in Table 4 (left).

Table 4 (right) shows simulation rates for the forward and backward passes of each module. We report forward and backward pass rates separately for the differentiable physics (DP) and the differentiable rendering (DR) modules. The time complexity of \(\nabla\mathit{Sim}\) is a function of the number of tetrahedrons and/or triangles. We illustrate the arguably more complex case of deformable object simulation for varying numbers of tetrahedra (ranging from \(100\) to \(10000\)). Even in the case of \(10000\) tetrahedra--enough to contruct complex mesh models of multiple moving objects--\(\nabla\mathit{Sim}\) enables faster-than-real-time simulation (\(1500\) steps/second). We presented \(\nabla\mathit{Sim}\), a versatile differentiable simulator that enables system identification from videos by differentiating through physical processes governing dyanmics and image formation.

However, we instead consider our physics engine as a differentiable operation that provides an implicit relationship between a state vector \(\mathbf{s}^{-}=\mathbf{s}(t)\) at the start of a time step, and the updated state at the end of the time step \(\mathbf{s}^{+}=\mathbf{s}(t+\Delta t)\). An arbitrary discrete time integration scheme can be then be abstracted as the function \(\mathbf{g}(\mathbf{s}^{-},\mathbf{s}^{+},\theta)=\mathbf{0}\), relating the initial and final system state and the model parameters \(\theta\). By the implicit function theorem, if we can specify a loss function \(l\) at the output of the simulator, we can compute \(\frac{\partial l}{\partial\mathbf{s}^{-}}\) as \(\mathbf{c}^{T}\frac{\partial\mathbf{g}}{\partial\mathbf{s}^{-}}\), where \(\mathbf{c}\) is the solution to the linear system \(\frac{\partial\mathbf{g}}{\partial\mathbf{s}^{+}}^{T}\mathbf{c}=-\frac{ \partial l}{\partial\mathbf{s}^{+}}^{T}\), and likewise for the model parameters \(\theta\). While the partial derivatives \(\frac{\partial\mathbf{g}}{\partial\mathbf{s}^{-}},\frac{\partial\mathbf{g}}{ \partial\mathbf{s}^{+}},\frac{\partial\mathbf{g}}{\partial\mathbf{s}}\) can be computed by graph-based automatic differentiation frameworks Paszke et al. (2019); Abadi et al. (2015); Bradbury et al. (2018), program transformation approaches such as DiffTaichi, and Google Tangent Hu et al. (2020); van Merrienboer et al. (2018) are particularly well-suited to simulation code. We use an embedded subset of Python syntax, which computes the adjoint of each simulation kernel at runtime, and generates C++/CUDA Kirk et al. (2007) code. Kernels are wrapped as custom autograd operations on PyTorch tensors, which allows users to focus on the definition of physical models, and leverage the PyTorch tape-based autodiff to track the overall program flow. While this formulation is general enough to represent explicit, multi-step, or fully implicit time-integration schemes, we employ semi-implicit Euler integration, which is the preferred integration scheme for most simulators Erez et al. (2015).

\(\nabla\)_Sim_ integrates several functional blocks, many of which contain nonlinear operations. Furthermore, we employ a pixelwise mean-squared error (MSE) loss function for estimating physical parameters from video. To demonstrate whether the gradients obtained from \(\nabla\)_Sim_ are relevant for the task of physical parameter estimation, in Figure 2 of the main paper, we present an analysis of the MSE loss landscape for mass estimation.

Most of the models presented in \(\nabla Sim\) can be trained and evaluated on modern laptops equipped with graphics processing units (GPUs). We find that, on a laptop with an Intel i7 processor and a GeForce GTX 1060 GPU, parameter estimation experiments for rigid/nonrigid bodies can be run in under \(5\)-\(20\) minutes per object on CPU and in under \(1\) minute on the GPU.

Input: Table 3: Parameter estimation of deformable objects: We estimate per-particle masses and material properties (for solid def. objects) and per-particle velocities for cloth. In the case of cloth, there is a perceivable performance drop in diffphysics, as the center of mass of a cloth is often outside the body, which results in ambiguity.

Input: Figure 2: ∇Sim: Given video observations of an evolving physical system (e), we randomly initialize scene object properties (a) and evolve them over time using a differentiable physics engine (b), which generates states. Our renderer (c) processes states, object vertices and global rendering parameters to produce image frames for computing our loss. We backprop through this computation graph to estimate physical attributes and controls. Existing methods rely solely on differentiable physics engines and require supervision in state-space (f), while ∇Sim only needs image-space supervision (g).

Input: Table 4: (Left) Impact of imperfect models: The accuracy of physical parameters estimated by ∇Sim is impacted by the choice of dynamics and graphics (rendering) models. We find that the system is more sensitive to the choice of dynamics models than to the rendering engine used. (Right) Timing analysis: We report runtime in simulation steps / second (Hz). ∇Sim is significantly faster than real-time, even for complex geometries.

Input: Figure 12: Impact of the length of a video sequence on the loss landscape. Notice how the loss landscape is much steeper for smaller videos (e.g., MSE of first and last frames). Nonetheless, all cases have a smooth loss landscape with the same unique minimum.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate the claim: "∇Sim performs differentiable physics simulation and rendering at independent and adjustable rates, allowing us to trade computation for accuracy by rendering fewer frames than dynamics updates."

Based on the evidence:

1. Table 4 demonstrates that ∇Sim can indeed perform physics simulation and rendering at different rates, showing separate forward/backward pass timings for differentiable physics (DP) and differentiable rendering (DR).

2. Figure 12 provides direct evidence of the trade-off between computation and accuracy by showing how using different frame sampling rates (every frame vs every 5th frame vs first/last frames) affects the loss landscape while maintaining the same minimum.

3. The timing analysis shows the system operating at varying rates for different complexities (from 100 to 10000 tetrahedra).

The claim appears to be well-stated (score 1) as it makes a specific technical assertion that is directly supported by quantitative evidence without exaggeration.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

\(\nabla Sim\) performs differentiable physics simulation and rendering at independent and adjustable rates, allowing us to trade computation for accuracy by rendering fewer frames than dynamics updates.

The review comment to be evaluated is:

- Summary

This paper presents a framework for performing both differentiable physics simulations and differentiable rendering. This fully differentiable simulation and rendering pipeline is then employed to perform system identification tasks, directly from video frames, being able to match or outperform both visual-based and state-based baselines. Moreover, the potential of this framework to be applied for visuomotor control is also demonstrated.

- Pros

This method unified advances in the differentiation of both physics simulation and rendering.

The experimental results demonstrate a good ability to perform system identification for diverse parameters and control directly from videos.
The ability to identify parameters or direct control tasks directly from images is useful, since it reduces the need for direct supervision/annotation in the form of state information.

The presented simulator supports a variety of "domains", such as rigid and deformable body dynamics, cloth simulation, and these are efficient enough to be run faster than real time (at least for simple tasks).

- Cons

Overall, the proposed method is mostly a unification of pre-existing techniques from different fields, such as differentiable rigid and deformable body dynamics, differentiable rendering.

The paper itself admits that a limitation of this method is that it currently "has limited capability to handle contact-rich motion that introduces a large number of discontinuities", which limits its applicability to real-world scenes. It cannot also currently handle joints. All of these would be important for possible robotic applications, for example.

The tasks demonstrated in the experiments are simple, and issues from model mismatch does not seem to have been thoroughly evaluated (see comments below for more).

- Reasons for score

[Edit: Score updated, see discussion below]

Overall, given the "pros" described above, notably the interesting results achieved for system identification and control directly from video frames by combining differentiable physics and rendering into a single framework, I recommend this paper for acceptance. Given some of the concerns raised in the "cons" and in more detail in the comments below, I for now will score this paper as a little above the acceptance threshold.

- Additional comments

The scenarios used for the system identification and control tasks are fairly simple, with usually only a single object and few contact points.
Was the ground truth for the scenarios in the system identification tasks generated using gradsim itself? If so, isn't it unfair that it is compared to other models (e.g., pybullet), for which there would be model mismatch? (While not mismatch would be present for gradsim)

Along the same direction, the experiments present a section on "Impact of imperfect dynamics and rendering models". It would also be interesting to see a quantification of the impact of model mismatch (possibly both while using the same renderer, i.e. only dynamics mismatch, or also different renderers)

In the experiments section, it is said that "Inference ... is done by picking an initial guess of the mass (at random)". From what distribution is this random initial guess picked from? What are these starting guesses in relation to the true parameters?

The section on "Impact of shading and texture cues" seems a little too short, which renders it hard to understand in detail what is going on.

The evidence to be evaluated is:Each of our experiments comprises an _environment_\(\mathcal{E}\) that applies a particular set of physical forces and/or constraints, a (differentiable) _loss function_\(\mathcal{L}\) that implicitly specifies an objective, and an _initial guess_\(\theta_{0}\) of the physical state of the simulation. The goal is to recover optimal physics parameters \(\theta^{*}\) that minimize \(\mathcal{L}\), by backpropagating through the simulator.

Inference with \(\nabla\)_Sim_ is done by guessing an initial mass (uniformly random in the range \([2,12]kg/m^{3}\)),rolling a _differentiable_ simulation using this guess, comparing the rendered out video with the true video (pixelwise mean-squared error - MSE), and performing gradient descent updates.

We compare \(\nabla\)_Sim_ with three other baselines: PyBullet + REINFORCE (Ehsani et al., 2020; Wu et al., 2015), diff. physics only (requiring 3D supervision), and a ConvLSTM baseline adopted from Xu et al. (2019) but with a stronger backbone. The _DiffPhysics_ baseline is a strict subset of \(\nabla\)_Sim_, it only inolves the differentiable physics engine.

Nevertheless, \(\nabla\)_Sim_ is able to very precisely estimate mass from video, to a absolute relative error of 9.01e-5, nearly two orders of magnitude better than the ConvLSTM baseline.

We conduct a series of experiments to investigate the ability of \(\nabla Sim\) to recover physical parameters of deformable solids and thin-shell solids (cloth).

We found our method is able to accurately recover the parameters of \(100\) instances of deformable objects (cloth, balls, beams) as reported in Table 3 and Fig. 3. Since \(\nabla Sim\) is a complex combination of differentiable non-linear components, we analyze the loss landscape to verify the validity of gradients through the system.

\(\nabla Sim\) is able to _solve_ this environment within three iterations of gradient descent, by minimizing a pixelwise MSE between the last frame of the rendered video and the goal image as shown in Fig. 5 (lower left). In our second test, we formulate a more challenging 3D control problem (control-fem) where the goal is to actuate a soft-body FEM object (a _geear_) consisting of \(1152\) tetrahedral elements to move to a target position as shown in Fig. 5 (center).

We also train a privileged baseline (_diffphysics_) that uses strong supervision and minimizes the MSE between the target position and the precise 3D location of the center-of-mass (COM) of the FEM model at each time step (i.e. a _dense_ reward). We test both _diffphysics_ and \(\nabla Sim\) against a naive baseline that generates random activations and plot convergence behaviors in Fig. 5(a). While _diffphysics_ appears to be a strong performer on this task, it is important to note that it uses explicit 3D supervision at each timestep (i.e. \(30\) FPS). In contrast, \(\nabla Sim\) uses a _single image_ as an implicit target, and yet manages to achieve the goal state, albeit taking a longer number of iterations.

Being a _white box_ method, the performance of \(\nabla\)_Sim_ relies on the choice of dynamics and rendering models employed. An immediate question that arises is "_how would the performance of \(\nabla\)_Sim_ be impacted (if at all) by such modeling choices._" We conduct multiple experiments targeted at investigating modelling errors and summarize them in Table 4 (left).

Table 4 (right) shows simulation rates for the forward and backward passes of each module. We report forward and backward pass rates separately for the differentiable physics (DP) and the differentiable rendering (DR) modules. The time complexity of \(\nabla\mathit{Sim}\) is a function of the number of tetrahedrons and/or triangles. We illustrate the arguably more complex case of deformable object simulation for varying numbers of tetrahedra (ranging from \(100\) to \(10000\)). Even in the case of \(10000\) tetrahedra--enough to contruct complex mesh models of multiple moving objects--\(\nabla\mathit{Sim}\) enables faster-than-real-time simulation (\(1500\) steps/second). We presented \(\nabla\mathit{Sim}\), a versatile differentiable simulator that enables system identification from videos by differentiating through physical processes governing dyanmics and image formation.

However, we instead consider our physics engine as a differentiable operation that provides an implicit relationship between a state vector \(\mathbf{s}^{-}=\mathbf{s}(t)\) at the start of a time step, and the updated state at the end of the time step \(\mathbf{s}^{+}=\mathbf{s}(t+\Delta t)\). An arbitrary discrete time integration scheme can be then be abstracted as the function \(\mathbf{g}(\mathbf{s}^{-},\mathbf{s}^{+},\theta)=\mathbf{0}\), relating the initial and final system state and the model parameters \(\theta\). By the implicit function theorem, if we can specify a loss function \(l\) at the output of the simulator, we can compute \(\frac{\partial l}{\partial\mathbf{s}^{-}}\) as \(\mathbf{c}^{T}\frac{\partial\mathbf{g}}{\partial\mathbf{s}^{-}}\), where \(\mathbf{c}\) is the solution to the linear system \(\frac{\partial\mathbf{g}}{\partial\mathbf{s}^{+}}^{T}\mathbf{c}=-\frac{ \partial l}{\partial\mathbf{s}^{+}}^{T}\), and likewise for the model parameters \(\theta\). While the partial derivatives \(\frac{\partial\mathbf{g}}{\partial\mathbf{s}^{-}},\frac{\partial\mathbf{g}}{ \partial\mathbf{s}^{+}},\frac{\partial\mathbf{g}}{\partial\mathbf{s}}\) can be computed by graph-based automatic differentiation frameworks Paszke et al. (2019); Abadi et al. (2015); Bradbury et al. (2018), program transformation approaches such as DiffTaichi, and Google Tangent Hu et al. (2020); van Merrienboer et al. (2018) are particularly well-suited to simulation code. We use an embedded subset of Python syntax, which computes the adjoint of each simulation kernel at runtime, and generates C++/CUDA Kirk et al. (2007) code. Kernels are wrapped as custom autograd operations on PyTorch tensors, which allows users to focus on the definition of physical models, and leverage the PyTorch tape-based autodiff to track the overall program flow. While this formulation is general enough to represent explicit, multi-step, or fully implicit time-integration schemes, we employ semi-implicit Euler integration, which is the preferred integration scheme for most simulators Erez et al. (2015).

\(\nabla\)_Sim_ integrates several functional blocks, many of which contain nonlinear operations. Furthermore, we employ a pixelwise mean-squared error (MSE) loss function for estimating physical parameters from video. To demonstrate whether the gradients obtained from \(\nabla\)_Sim_ are relevant for the task of physical parameter estimation, in Figure 2 of the main paper, we present an analysis of the MSE loss landscape for mass estimation.

Most of the models presented in \(\nabla Sim\) can be trained and evaluated on modern laptops equipped with graphics processing units (GPUs). We find that, on a laptop with an Intel i7 processor and a GeForce GTX 1060 GPU, parameter estimation experiments for rigid/nonrigid bodies can be run in under \(5\)-\(20\) minutes per object on CPU and in under \(1\) minute on the GPU.

Input: Table 3: Parameter estimation of deformable objects: We estimate per-particle masses and material properties (for solid def. objects) and per-particle velocities for cloth. In the case of cloth, there is a perceivable performance drop in diffphysics, as the center of mass of a cloth is often outside the body, which results in ambiguity.

Input: Figure 2: ∇Sim: Given video observations of an evolving physical system (e), we randomly initialize scene object properties (a) and evolve them over time using a differentiable physics engine (b), which generates states. Our renderer (c) processes states, object vertices and global rendering parameters to produce image frames for computing our loss. We backprop through this computation graph to estimate physical attributes and controls. Existing methods rely solely on differentiable physics engines and require supervision in state-space (f), while ∇Sim only needs image-space supervision (g).

Input: Table 4: (Left) Impact of imperfect models: The accuracy of physical parameters estimated by ∇Sim is impacted by the choice of dynamics and graphics (rendering) models. We find that the system is more sensitive to the choice of dynamics models than to the rendering engine used. (Right) Timing analysis: We report runtime in simulation steps / second (Hz). ∇Sim is significantly faster than real-time, even for complex geometries.

Input: Figure 12: Impact of the length of a video sequence on the loss landscape. Notice how the loss landscape is much steeper for smaller videos (e.g., MSE of first and last frames). Nonetheless, all cases have a smooth loss landscape with the same unique minimum.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate the claim: "∇Sim performs differentiable physics simulation and rendering at independent and adjustable rates, allowing us to trade computation for accuracy by rendering fewer frames than dynamics updates."

1. Summary of evaluation:
The evidence strongly supports this claim. Table 4 (right) shows separate forward and backward pass rates for differentiable physics (DP) and differentiable rendering (DR) modules, demonstrating their independence. The timing analysis reveals that for various tetrahedral counts, both simulation and rendering operate at different rates. Figure 12 further validates this by showing how the system performs with different frame sampling rates ("MSE on every frame" vs "MSE on every 5th frame"), demonstrating the ability to trade computation for accuracy.

<score>1</score>

2. Detailed Justification:
- Table 4 provides concrete evidence of independent rates, showing distinct Hz values for physics and rendering
- The timing analysis demonstrates faster-than-real-time performance (1500 steps/second) even with complex geometries
- Figure 12 empirically shows system functionality with varying frame rates
- The review comments do not dispute this technical capability
- The claim is precise and well-grounded in the presented evidence without exaggeration