Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Compared with methods with agents speaking the natural language, ours is cheaper with no expensive annotation, making it a more practical way for agents to communicate.

The review comment to be evaluated is:

While this paper goes in a good direction by moving away from the Lewis game in the language emergence communication setting, it falls short as there is little scientific analysis. The authors mainly designed an artificial navigation task, handcrafted an observation space and reward, designed a neural network, trained it, and reported the results. Those results are not put into perspective with a detailed researched perspective. Indeed, the point of Emecom in ML is not to create artificial tasks for the sake of solving it, but to design tasks that could help understand how language evolves with a human intervention as possible (or being generic).

I am listing here a few leads to improve the paper in the long run:
 - The introduction should frame the research problem into the past literature, ask a scientific question, and explain how the paper help answers it. Here, there is no citation; the VLM task is only said to be more natural (no reference to the embodiment, situated learning, or going beyond the signaling game)
 - The related work is extremely short. 18 citations in total for a paper about language emergence, and the navigation task is just inherently incomplete. On a positive note, the most relevant citations kajic/Kanilowska/DeVries are present
- The paper is 7 pages long, and page 3 is barely empty
- The model is designed to solve the task, e.g., the graph networks are specific to the observation space. I would recommend either picking a minimal architecture or reproducing the architecture that is classic in VLM.
- having 4 losses in an RL task is generally a bad sign as it is a symptom of human intervention. cf The bitter lesson by Sutton
 - Results are not compared to any baselines (VAE vs your network is not really a baseline...), and simply reporting the success rate is not sufficient. Examples of suitable ablation include: the correlation between language and environment complexity. Qualitative experiments to translate language and change in the environment (e.g., how changing the graph impacts the language), what are the external constraints that shape language, reward ablation, etc.

I would recommend the authors to re-work their paper and present it to a workshop to have further guidance. The initial idea is good, but the realization lacks scientific maturity.

The evidence to be evaluated is:With the limitations of the message length and vocabulary size in discrete communication scenarios, discrete VAE may communicate less useful information compared with our model.

And our method outperforms the discrete VAE largely.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on the provided evidence and review comment, I'll assess the claim "Compared with methods with agents speaking the natural language, ours is cheaper with no expensive annotation, making it a more practical way for agents to communicate."

This claim appears significantly overstated for several reasons:

1. The evidence only compares the method to discrete VAE, not to methods using natural language.
2. The reviewer points out significant limitations:
   - Lack of proper scientific analysis and baselines
   - Minimal related work citations (only 18 total)
   - No detailed comparison with existing methods
   - The task is artificial and lacks broader perspective
3. The claim about being "cheaper" and "more practical" isn't supported by comparative cost analysis or practicality metrics.
4. The evidence doesn't address annotation costs or implementation complexity relative to other methods.

<score>5</score>

The claim significantly overreaches beyond what the evidence supports. The available evidence only shows better performance compared to discrete VAE, while the claim makes broad statements about cost and practicality compared to all natural language methods without supporting evidence. The reviewer's comments further highlight the lack of comprehensive analysis and comparisons needed to support such claims.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Compared with methods with agents speaking the natural language, ours is cheaper with no expensive annotation, making it a more practical way for agents to communicate.

The review comment to be evaluated is:

The paper is not clearly written and some sections are hard to understand. The related works do capture how the framework proposes new challenges that are not already captured in previous work. Besides a similar work that also uses communication between agents [2] perform a similar analysis and comparisons to that work would be help in highlighting the difference with the proposed framework.

Besides the scope of the proposed framework is limited as it can only handle artificial messages that have a much smaller length, vocabulary, and sources of variation. In contrast, previous work have already proposed similar benchmarks that involve real images with natural language descriptions [1].

[1] de Vries et al 2018. Talk the Walk: Navigating New York City through Grounded Dialogue.

[2] Singh et al. 2019. Learning when to Communicate at Scale in Multiagent Cooperative and Competitive Tasks.

The evidence to be evaluated is:With the limitations of the message length and vocabulary size in discrete communication scenarios, discrete VAE may communicate less useful information compared with our model.

And our method outperforms the discrete VAE largely.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll assess the claim based on the available evidence and review comments:

The claim appears significantly overstated for several reasons:

1. The claim states their method is "cheaper with no expensive annotation" but provides no cost comparison evidence with other methods.

2. The reviewer points out that the framework has significant limitations - it can only handle artificial messages with smaller length, vocabulary, and variation compared to previous work that handles real images and natural language.

3. The evidence provided is very limited and only compares to discrete VAE, not to methods using natural language as claimed.

4. The reviewer notes similar work exists ([2]) that hasn't been adequately compared against.

The claim makes broad statements about practicality and cost advantages without sufficient supporting evidence, while ignoring significant limitations highlighted by the reviewer.

<score>5</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

As far as we know, We are the first to propose a VLN-like task in a two-agent cooperation scenario.

The evidence to be evaluated is:By putting the VLN task into multi-agent settings, the problems listed above can be alleviated or solved. To our knowledge, talk-the-walk de Vries et al. (2018) is the only work applying the VLN task with multiple agents and is very similar to our task settings. However, talk-the-walk only focuses on the localization part, where the guide finds where the tourist is. But ignore the guidance part, in which the guide makes a route plan and gives guidance to the tourist, and the tourist makes corresponding actions. Talk-the-walk models this part with a random walk and only implements a minimal baseline, making the VLN task with multi-agent settings incomplete.

From the perspective of agents' relationships, most of the works are focused on fully cooperative tasks like referential games Lazaridou et al. (2018) adapted from the Lewis signaling game.

Some navigation tasks are proposed Kalinowska et al. (2022); Kajic et al. (2020), however, those works can only be applied in limited environments, with 2 and 4 cases respectively.

Most of those part of works are studying single message pass, with one agent only having a language generation part, and the other one having a language understanding module.

But the former study focuses on a semi-cooperative case rather than a fully cooperative one like us, while the latter one uses the bag of words, and generates 0 or 1, with 1 indicating this token is in the generated sentence.

This task is inspired by a real-world scenario where two agents with unequal abilities, agents A and B, cooperate to help agent A find the target place. As pictured in Figure 1, agent A can only see the objects around it, while agent B has the floor plan and the target place but doesn't know where A is. To win this game, they have to cooperate to help A reach the target place with limited rounds.

In this work, we propose a navigation task in an emergent communication scenario. This task is adapted from the vision-language navigation task proposed in human-machine communication. However, compared with the traditional VLN task, the application of multi-agent communication in the VLN task makes the communication policy more flexible and can generate a more complex

Input: Figure 1: The navigation task involves three components: the tourist (agent A), the guide (agent B), and the environment. Agent A and B have unequal abilities. While agent A can only see its surroundings and describe them to B, agent B has the global perspective, can make a route plan, and give instructions to A. Agent A and B cooperate to lead A to the target place.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on my review of the evidence, the claim "We are the first to propose a VLN-like task in a two-agent cooperation scenario" appears to be overstated. The evidence explicitly mentions "talk-the-walk" by de Vries et al. (2018) as a previous work that applied VLN tasks with multiple agents in a similar setting. While the evidence argues that talk-the-walk focused only on localization and not guidance, it still represents a prior implementation of VLN in a multi-agent scenario. The authors' work may be novel in its comprehensive approach to both localization and guidance, but claiming to be "first" overlooks this existing precedent.

<score>4</score>

Justification:
1. Direct contradiction in evidence: "To our knowledge, talk-the-walk de Vries et al. (2018) is the only work applying the VLN task with multiple agents"
2. The claim would be more accurate if qualified to specify their novel contributions (comprehensive approach to both localization and guidance) rather than claiming to be first overall
3. The evidence shows they built upon and expanded existing work rather than being first to propose the concept
4. Other navigation tasks in multi-agent settings are also mentioned (Kalinowska et al., 2022; Kajic et al., 2020), though in limited environments


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

As far as we know, We are the first to propose a VLN-like task in a two-agent cooperation scenario.

The review comment to be evaluated is:

Strengths:
* The proposed setting is novel.

Weaknesses:
* While novel, the proposed setting is poorly motivated. Why studying this setting is significant? What would it enable?
* The description of the environment is very nebulous. Especially, the complexity of the visual and language components are highly under-specified. How is the input observation represented (image or feature-based)? How many object categories? How large is the agent's vocabulary? How are the environment graph generated? It is thus difficult to comprehend the significance of solving this environment.
* The plots are missing confidence intervals. Each experiments are not conducted multiple times with different random seeds. This seriously weakens the empirical claims.

The evidence to be evaluated is:By putting the VLN task into multi-agent settings, the problems listed above can be alleviated or solved. To our knowledge, talk-the-walk de Vries et al. (2018) is the only work applying the VLN task with multiple agents and is very similar to our task settings. However, talk-the-walk only focuses on the localization part, where the guide finds where the tourist is. But ignore the guidance part, in which the guide makes a route plan and gives guidance to the tourist, and the tourist makes corresponding actions. Talk-the-walk models this part with a random walk and only implements a minimal baseline, making the VLN task with multi-agent settings incomplete.

From the perspective of agents' relationships, most of the works are focused on fully cooperative tasks like referential games Lazaridou et al. (2018) adapted from the Lewis signaling game.

Some navigation tasks are proposed Kalinowska et al. (2022); Kajic et al. (2020), however, those works can only be applied in limited environments, with 2 and 4 cases respectively.

Most of those part of works are studying single message pass, with one agent only having a language generation part, and the other one having a language understanding module.

But the former study focuses on a semi-cooperative case rather than a fully cooperative one like us, while the latter one uses the bag of words, and generates 0 or 1, with 1 indicating this token is in the generated sentence.

This task is inspired by a real-world scenario where two agents with unequal abilities, agents A and B, cooperate to help agent A find the target place. As pictured in Figure 1, agent A can only see the objects around it, while agent B has the floor plan and the target place but doesn't know where A is. To win this game, they have to cooperate to help A reach the target place with limited rounds.

In this work, we propose a navigation task in an emergent communication scenario. This task is adapted from the vision-language navigation task proposed in human-machine communication. However, compared with the traditional VLN task, the application of multi-agent communication in the VLN task makes the communication policy more flexible and can generate a more complex

Input: Figure 1: The navigation task involves three components: the tourist (agent A), the guide (agent B), and the environment. Agent A and B have unequal abilities. While agent A can only see its surroundings and describe them to B, agent B has the global perspective, can make a route plan, and give instructions to A. Agent A and B cooperate to lead A to the target place.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on the evidence provided, the claim "We are the first to propose a VLN-like task in a two-agent cooperation scenario" appears to be overstated. The evidence explicitly mentions "talk-the-walk" by de Vries et al. (2018) as a prior work that applies VLN in multi-agent settings with similar task configurations. While the authors introduce novel elements and expanded capabilities compared to talk-the-walk (which focused mainly on localization), they are not the first to propose such a scenario. The review comments also highlight the novelty but point out significant limitations in motivation and specifications.

<score>4</score>

Justification:
1. Direct contradiction: The evidence states "talk-the-walk de Vries et al. (2018) is the only work applying the VLN task with multiple agents and is very similar to our task settings."

2. Key differences:
- Talk-the-walk focused on localization while this work expands to include guidance
- This work implements a more complete version of the multi-agent VLN task
- The authors introduce novel elements but build upon existing work

3. The review comments acknowledge novelty but highlight significant limitations in:
- Task motivation and significance
- Environment specifications
- Empirical validation

The claim would be more accurate if it acknowledged building upon existing work while highlighting its novel contributions and expansions to the task framework.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

As far as we know, We are the first to propose a VLN-like task in a two-agent cooperation scenario.

The review comment to be evaluated is:

While this paper goes in a good direction by moving away from the Lewis game in the language emergence communication setting, it falls short as there is little scientific analysis. The authors mainly designed an artificial navigation task, handcrafted an observation space and reward, designed a neural network, trained it, and reported the results. Those results are not put into perspective with a detailed researched perspective. Indeed, the point of Emecom in ML is not to create artificial tasks for the sake of solving it, but to design tasks that could help understand how language evolves with a human intervention as possible (or being generic).

I am listing here a few leads to improve the paper in the long run:
 - The introduction should frame the research problem into the past literature, ask a scientific question, and explain how the paper help answers it. Here, there is no citation; the VLM task is only said to be more natural (no reference to the embodiment, situated learning, or going beyond the signaling game)
 - The related work is extremely short. 18 citations in total for a paper about language emergence, and the navigation task is just inherently incomplete. On a positive note, the most relevant citations kajic/Kanilowska/DeVries are present
- The paper is 7 pages long, and page 3 is barely empty
- The model is designed to solve the task, e.g., the graph networks are specific to the observation space. I would recommend either picking a minimal architecture or reproducing the architecture that is classic in VLM.
- having 4 losses in an RL task is generally a bad sign as it is a symptom of human intervention. cf The bitter lesson by Sutton
 - Results are not compared to any baselines (VAE vs your network is not really a baseline...), and simply reporting the success rate is not sufficient. Examples of suitable ablation include: the correlation between language and environment complexity. Qualitative experiments to translate language and change in the environment (e.g., how changing the graph impacts the language), what are the external constraints that shape language, reward ablation, etc.

I would recommend the authors to re-work their paper and present it to a workshop to have further guidance. The initial idea is good, but the realization lacks scientific maturity.

The evidence to be evaluated is:By putting the VLN task into multi-agent settings, the problems listed above can be alleviated or solved. To our knowledge, talk-the-walk de Vries et al. (2018) is the only work applying the VLN task with multiple agents and is very similar to our task settings. However, talk-the-walk only focuses on the localization part, where the guide finds where the tourist is. But ignore the guidance part, in which the guide makes a route plan and gives guidance to the tourist, and the tourist makes corresponding actions. Talk-the-walk models this part with a random walk and only implements a minimal baseline, making the VLN task with multi-agent settings incomplete.

From the perspective of agents' relationships, most of the works are focused on fully cooperative tasks like referential games Lazaridou et al. (2018) adapted from the Lewis signaling game.

Some navigation tasks are proposed Kalinowska et al. (2022); Kajic et al. (2020), however, those works can only be applied in limited environments, with 2 and 4 cases respectively.

Most of those part of works are studying single message pass, with one agent only having a language generation part, and the other one having a language understanding module.

But the former study focuses on a semi-cooperative case rather than a fully cooperative one like us, while the latter one uses the bag of words, and generates 0 or 1, with 1 indicating this token is in the generated sentence.

This task is inspired by a real-world scenario where two agents with unequal abilities, agents A and B, cooperate to help agent A find the target place. As pictured in Figure 1, agent A can only see the objects around it, while agent B has the floor plan and the target place but doesn't know where A is. To win this game, they have to cooperate to help A reach the target place with limited rounds.

In this work, we propose a navigation task in an emergent communication scenario. This task is adapted from the vision-language navigation task proposed in human-machine communication. However, compared with the traditional VLN task, the application of multi-agent communication in the VLN task makes the communication policy more flexible and can generate a more complex

Input: Figure 1: The navigation task involves three components: the tourist (agent A), the guide (agent B), and the environment. Agent A and B have unequal abilities. While agent A can only see its surroundings and describe them to B, agent B has the global perspective, can make a route plan, and give instructions to A. Agent A and B cooperate to lead A to the target place.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on the evidence and review comments, the claim "We are the first to propose a VLN-like task in a two-agent cooperation scenario" appears to be overstated. The evidence directly mentions "talk-the-walk" by de Vries et al. (2018) as a similar work applying VLN in multi-agent settings. While the authors argue their approach is more complete by including guidance aspects that talk-the-walk ignored, they cannot claim to be "first." Additionally, the reviewer points out insufficient literature review and citations, suggesting there may be other relevant prior work. The claim would be more accurate if it specified their novel contributions rather than claiming to be first.

<score>4</score>

Justification:
1. Direct contradiction: Evidence mentions "talk-the-walk" as prior work in VLN multi-agent settings
2. Review highlights extremely limited citations (18 total) suggesting incomplete literature review
3. The claim ignores existing similar work and fails to acknowledge limitations
4. Would be better stated as "We propose a more complete VLN-like task in two-agent cooperation by incorporating both localization and guidance aspects"
5. Citations like Kajic et al. and Kalinowska et al. show prior work in navigation tasks with multiple agents, though in limited environments


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

As far as we know, We are the first to propose a VLN-like task in a two-agent cooperation scenario.

The review comment to be evaluated is:

The paper is not clearly written and some sections are hard to understand. The related works do capture how the framework proposes new challenges that are not already captured in previous work. Besides a similar work that also uses communication between agents [2] perform a similar analysis and comparisons to that work would be help in highlighting the difference with the proposed framework.

Besides the scope of the proposed framework is limited as it can only handle artificial messages that have a much smaller length, vocabulary, and sources of variation. In contrast, previous work have already proposed similar benchmarks that involve real images with natural language descriptions [1].

[1] de Vries et al 2018. Talk the Walk: Navigating New York City through Grounded Dialogue.

[2] Singh et al. 2019. Learning when to Communicate at Scale in Multiagent Cooperative and Competitive Tasks.

The evidence to be evaluated is:By putting the VLN task into multi-agent settings, the problems listed above can be alleviated or solved. To our knowledge, talk-the-walk de Vries et al. (2018) is the only work applying the VLN task with multiple agents and is very similar to our task settings. However, talk-the-walk only focuses on the localization part, where the guide finds where the tourist is. But ignore the guidance part, in which the guide makes a route plan and gives guidance to the tourist, and the tourist makes corresponding actions. Talk-the-walk models this part with a random walk and only implements a minimal baseline, making the VLN task with multi-agent settings incomplete.

From the perspective of agents' relationships, most of the works are focused on fully cooperative tasks like referential games Lazaridou et al. (2018) adapted from the Lewis signaling game.

Some navigation tasks are proposed Kalinowska et al. (2022); Kajic et al. (2020), however, those works can only be applied in limited environments, with 2 and 4 cases respectively.

Most of those part of works are studying single message pass, with one agent only having a language generation part, and the other one having a language understanding module.

But the former study focuses on a semi-cooperative case rather than a fully cooperative one like us, while the latter one uses the bag of words, and generates 0 or 1, with 1 indicating this token is in the generated sentence.

This task is inspired by a real-world scenario where two agents with unequal abilities, agents A and B, cooperate to help agent A find the target place. As pictured in Figure 1, agent A can only see the objects around it, while agent B has the floor plan and the target place but doesn't know where A is. To win this game, they have to cooperate to help A reach the target place with limited rounds.

In this work, we propose a navigation task in an emergent communication scenario. This task is adapted from the vision-language navigation task proposed in human-machine communication. However, compared with the traditional VLN task, the application of multi-agent communication in the VLN task makes the communication policy more flexible and can generate a more complex

Input: Figure 1: The navigation task involves three components: the tourist (agent A), the guide (agent B), and the environment. Agent A and B have unequal abilities. While agent A can only see its surroundings and describe them to B, agent B has the global perspective, can make a route plan, and give instructions to A. Agent A and B cooperate to lead A to the target place.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on the review and evidence provided, the claim "We are the first to propose a VLN-like task in a two-agent cooperation scenario" appears to be overstated. The evidence and reviewer comments directly reference previous work in this area, particularly de Vries et al. (2018) "Talk the Walk" which also implements a VLN task with multiple agents in a cooperative setting. While the authors' approach may have novel aspects in terms of the specific implementation and focus on guidance, claiming to be "first" is not accurate given the existing literature.

<score>4</score>

Justification:
1. Direct contradiction: The evidence itself acknowledges "talk-the-walk de Vries et al. (2018) is the only work applying the VLN task with multiple agents and is very similar to our task settings."

2. Reviewer points out similar existing work:
- Talk the Walk benchmark (de Vries et al. 2018)
- Singh et al. (2019) work on multi-agent communication

3. More appropriate claim wording should:
- Acknowledge previous work explicitly
- Highlight specific novel contributions/differences
- Focus on improvements or extensions rather than claiming "first"

4. While the authors' implementation may have unique aspects (focusing on both localization and guidance), this doesn't support claiming to be the first to propose such a task structure.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We release a new dataset of 3000 scenarios that involve such visual and language navigation.

The evidence to be evaluated is:We construct the dataset with 6 rooms for each navigation task. And every object in this dataset has 3 optional colors and 3 optional shapes. Also, some features not relevant to this task are added, like the gate color which we suppose to be the same among all rooms in the same house.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me evaluate the claim against the provided evidence:

The claim states there are 3000 scenarios, but the evidence only describes the dataset structure (6 rooms per task, 3 colors and 3 shapes per object) without explicitly stating the total number of scenarios. While these parameters could potentially generate 3000 scenarios through combinations, the exact calculation or verification is not provided in the evidence. This makes the specific claim of "3000 scenarios" unverifiable from the given evidence alone.

Additionally, while the evidence mentions various dataset characteristics (rooms, object properties, gate colors), it doesn't confirm the total dataset size. The claim makes a precise numerical statement that isn't directly supported by the presented evidence.

<score>4</score>

Justification:
- The claim makes a specific numerical assertion (3000 scenarios) that cannot be directly verified from the provided evidence
- While the dataset structure is described, the total number of scenarios is not explicitly stated or proven
- The evidence provides details about dataset parameters but doesn't connect these to demonstrate how the total of 3000 scenarios is achieved
- The claim would be better stated with either supporting calculations or more cautious wording about the dataset size


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We release a new dataset of 3000 scenarios that involve such visual and language navigation.

The review comment to be evaluated is:

Strengths:
* The proposed setting is novel.

Weaknesses:
* While novel, the proposed setting is poorly motivated. Why studying this setting is significant? What would it enable?
* The description of the environment is very nebulous. Especially, the complexity of the visual and language components are highly under-specified. How is the input observation represented (image or feature-based)? How many object categories? How large is the agent's vocabulary? How are the environment graph generated? It is thus difficult to comprehend the significance of solving this environment.
* The plots are missing confidence intervals. Each experiments are not conducted multiple times with different random seeds. This seriously weakens the empirical claims.

The evidence to be evaluated is:We construct the dataset with 6 rooms for each navigation task. And every object in this dataset has 3 optional colors and 3 optional shapes. Also, some features not relevant to this task are added, like the gate color which we suppose to be the same among all rooms in the same house.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me evaluate the claim based on the provided evidence and review comments:

Summary:
The claim states the release of a 3000-scenario dataset for visual and language navigation, but the evidence provides very limited details about the dataset's composition and complexity. While it mentions 6 rooms per task and object variations (3 colors, 3 shapes), crucial details about visual representations, vocabulary size, and environment generation are missing, as highlighted in the review comments. The claim of "3000 scenarios" lacks sufficient substantiation in the provided evidence, making it difficult to verify the scale and comprehensiveness of the dataset.

<score>4</score>

Justification:
1. The claim makes a definitive statement about 3000 scenarios without providing clear evidence of this number in the dataset details
2. The evidence only specifies basic parameters (6 rooms, 3 colors, 3 shapes) without explaining:
   - How these combine to create 3000 scenarios
   - Visual input representation
   - Language complexity and vocabulary
   - Environment generation methods
3. The reviewer correctly points out that the environment description is "very nebulous" and "highly under-specified"
4. The lack of technical details makes it impossible to fully validate the claimed scale and complexity of the dataset


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We release a new dataset of 3000 scenarios that involve such visual and language navigation.

The review comment to be evaluated is:

While this paper goes in a good direction by moving away from the Lewis game in the language emergence communication setting, it falls short as there is little scientific analysis. The authors mainly designed an artificial navigation task, handcrafted an observation space and reward, designed a neural network, trained it, and reported the results. Those results are not put into perspective with a detailed researched perspective. Indeed, the point of Emecom in ML is not to create artificial tasks for the sake of solving it, but to design tasks that could help understand how language evolves with a human intervention as possible (or being generic).

I am listing here a few leads to improve the paper in the long run:
 - The introduction should frame the research problem into the past literature, ask a scientific question, and explain how the paper help answers it. Here, there is no citation; the VLM task is only said to be more natural (no reference to the embodiment, situated learning, or going beyond the signaling game)
 - The related work is extremely short. 18 citations in total for a paper about language emergence, and the navigation task is just inherently incomplete. On a positive note, the most relevant citations kajic/Kanilowska/DeVries are present
- The paper is 7 pages long, and page 3 is barely empty
- The model is designed to solve the task, e.g., the graph networks are specific to the observation space. I would recommend either picking a minimal architecture or reproducing the architecture that is classic in VLM.
- having 4 losses in an RL task is generally a bad sign as it is a symptom of human intervention. cf The bitter lesson by Sutton
 - Results are not compared to any baselines (VAE vs your network is not really a baseline...), and simply reporting the success rate is not sufficient. Examples of suitable ablation include: the correlation between language and environment complexity. Qualitative experiments to translate language and change in the environment (e.g., how changing the graph impacts the language), what are the external constraints that shape language, reward ablation, etc.

I would recommend the authors to re-work their paper and present it to a workshop to have further guidance. The initial idea is good, but the realization lacks scientific maturity.

The evidence to be evaluated is:We construct the dataset with 6 rooms for each navigation task. And every object in this dataset has 3 optional colors and 3 optional shapes. Also, some features not relevant to this task are added, like the gate color which we suppose to be the same among all rooms in the same house.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me evaluate the claim that "We release a new dataset of 3000 scenarios that involve such visual and language navigation."

Based on the provided evidence and review comments:

1. The evidence only mentions the structure (6 rooms per task, 3 colors and 3 shapes per object) but doesn't explicitly state the total number of scenarios (3000).

2. The reviewer points out several concerns about scientific rigor and lack of detailed analysis, though doesn't specifically dispute the dataset size.

3. There's a significant gap between the claim of 3000 scenarios and the evidence provided. While the dataset parameters are described, there's no clear calculation or verification of how the 3000 number was derived.

This claim appears overstated given the available evidence. While the dataset structure is described, the specific claim of 3000 scenarios is not substantiated in the provided evidence.

<score>4</score>

Justification:
- The claim makes a specific numerical assertion (3000 scenarios) without supporting evidence
- While the dataset structure is described (6 rooms, 3 colors, 3 shapes), there's no clear link to how the 3000 figure was calculated
- The reviewer's comments about lack of scientific rigor and detailed analysis further suggest potential overstatement
- The claim would be better stated by either providing explicit evidence for the 3000 figure or by describing the dataset structure without making specific numerical claims about total scenarios


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We release a new dataset of 3000 scenarios that involve such visual and language navigation.

The review comment to be evaluated is:

The paper is not clearly written and some sections are hard to understand. The related works do capture how the framework proposes new challenges that are not already captured in previous work. Besides a similar work that also uses communication between agents [2] perform a similar analysis and comparisons to that work would be help in highlighting the difference with the proposed framework.

Besides the scope of the proposed framework is limited as it can only handle artificial messages that have a much smaller length, vocabulary, and sources of variation. In contrast, previous work have already proposed similar benchmarks that involve real images with natural language descriptions [1].

[1] de Vries et al 2018. Talk the Walk: Navigating New York City through Grounded Dialogue.

[2] Singh et al. 2019. Learning when to Communicate at Scale in Multiagent Cooperative and Competitive Tasks.

The evidence to be evaluated is:We construct the dataset with 6 rooms for each navigation task. And every object in this dataset has 3 optional colors and 3 optional shapes. Also, some features not relevant to this task are added, like the gate color which we suppose to be the same among all rooms in the same house.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me evaluate the claim based on the provided evidence and review comments:

The claim states "We release a new dataset of 3000 scenarios" but the evidence only describes the structure of rooms and objects without explicitly stating the total number of scenarios. The review comments point out limitations in the framework's scope, particularly regarding artificial messages compared to real-world datasets. Additionally, the review highlights that similar work exists, suggesting the novelty may be overstated.

The evidence provides details about:
- 6 rooms per navigation task
- 3 color options and 3 shape options for objects
- Additional features like gate color

However, there is no direct evidence supporting the specific number "3000 scenarios" mentioned in the claim. This appears to be an overstatement given the available evidence.

<score>4</score>

Justification:
1. The claim makes a specific numerical assertion (3000 scenarios) without supporting evidence
2. The review comments suggest the framework is limited compared to existing work
3. The evidence describes the dataset structure but doesn't validate the quantity claimed
4. The novelty may be overstated given similar existing work mentioned in the review
5. The evidence provided is insufficient to support the specific quantity claimed


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We also seek to address the multi-agent emergent communications by proposing a collaborative learning framework that enables the agents to generate and understand emergent language and solve task.

The evidence to be evaluated is:Navigation tasks with vision and natural language information have attracted much attention in the last few years. Generally, this task involves humans, an artificial agent, and the environment. The agent communicates with humans in natural language. The agent may receive requests or instructions and ask for more guidance from the human. It can navigate and interact with the environment to get more information or complete corresponding task requested by the human.

By putting the VLN task into multi-agent settings, the problems listed above can be alleviated or solved. To our knowledge, talk-the-walk de Vries et al. (2018) is the only work applying the VLN task with multiple agents and is very similar to our task settings. However, talk-the-walk only focuses on the localization part, where the guide finds where the tourist is. But ignore the guidance part, in which the guide makes a route plan and gives guidance to the tourist, and the tourist makes corresponding actions. Talk-the-walk models this part with a random walk and only implements a minimal baseline, making the VLN task with multi-agent settings incomplete. **Emergent Language** Emergent language is an unplanned language that comes up in interactions between the students, and/or the teacher and the students. From the perspective of agents' relationships, most of the works are focused on fully cooperative tasks like referential games Lazaridou et al. (2018) adapted from the Lewis signaling game. In the referential game, there are two roles: speaker and listener. The speaker has to give instructions to the listener so that it can select the target picture from the candidates successfully. Some navigation tasks are proposed Kalinowska et al. (2022); Kajic et al. (2020), however, those works can only be applied in limited environments, with 2 and 4 cases respectively. Some semi-cooperative tasks are also proposed, like the negotiation game Cao et al. (2018), in which two agents have to negotiate to get more scores for themselves. There are also some fully-competitive cases like a circular-based sender-receiver game Noukhovitch et al. (2021) where the sender tries to make the receiver choose the position closer to its own target position rather than the receiver's. Most of those part of works are studying single message pass, with one agent only having a language generation part, and the other one having a language understanding module. Still, there are some works putting attention to the multi-turn form Cao et al. (2018); Evtimova et al. (2017). But the former study focuses on a semi-cooperative case rather than a fully cooperative one like us, while the latter one uses the bag of words, and generates 0 or 1, with 1 indicating this token is in the generated sentence.

This task is inspired by a real-world scenario where two agents with unequal abilities, agents A and B, cooperate to help agent A find the target place. As pictured in Figure 1, agent A can only see the objects around it, while agent B has the floor plan and the target place but doesn't know where A is. To win this game, they have to cooperate to help A reach the target place with limited rounds.

Agent B needs to find where A is, according to the message given by A describing the objects around it.

Having the position of A and the target place, agent B has to find the shortest path for A to get there.

Agent B gives instructions to A, helping it follow the route plan.

Formally, this navigation task is constructed as follows: There are houses, containing rooms \(R=\{r_{0},r_{1},...,r_{N}\}\) each. Every time, one house is randomly chosen from those \(K\) candidates. Then two rooms are drawn from the selected house, and one of them is the target room \(r_{o}\), and the other is the start room \(r_{s}\). Every room has several gates \(D_{i}=\{d_{ij},j\in Neighbor(i)\}\). There is a graph for each house, to model the relationship between rooms. We use \(G=(R,D)\) to denote a house, where \(R\) is the vertex set, and \(D\) is the edge set. There are two players called agent A, and agent B parameterized by \(\theta_{A},\theta_{B}\) separately.

* In the localization task, agent A takes in the room's vector where it is and gives out the messages of step t consisting of \(M\) tokens \(m_{t}=\{v_{0},v_{1},...,v_{M}\},v_{i}\in V\). While agent B collects history dialogue \(h_{t}=(m_{0},m_{1},..,m_{t})\) and tries to guess where A is.

* In the route plan task, agent B uses the room graph \(G\), the guessed position of A \(\arg\max(\theta_{B}(h_{t},G))\), and the target position \(r_{o}\) to get the next step A should take \(\arg\max(\theta_{A}(h_{t+1},D))\).

Agent A encodes the target room information \(r_{o}\) including objects and gates and then uses the encoded vector to generate corresponding emergent language with its language generation module \(e_{s}^{o}=GRU(MLP(r_{o}))\). With the message coming from agent A in this turn, and the history sentences from A, agent B makes its guess by making use of the topological structure of rooms \(\theta_{B}(h_{t},G)\).

Similarly, the model for the guidance task also includes an encoder, a language generation, and a language understanding module used for giving instructions to agent A about its next step. Considering the non-differentiability of the discrete messages passing between agents, those models are trained with REINFORCE algorithm.

Finally, we put those sub-tasks together to complete the navigation task.

With the limitations of the message length and vocabulary size in discrete communication scenarios, discrete VAE may communicate less useful information compared with our model. We also show the case when agents can pass continuous messages, giving us a ceiling performance when the communication channel is not restricted.

We suppose that good language tends to give similar expressions describing a similar object. So we use topographic similarity \(\rho\) as our metric, which is the Spearman correlation between the message similarity sequence and the room vector similarity sequence. And our method outperforms the discrete VAE largely.

Then, we cluster those rooms with the generated message. Labeled these rooms with their objects' color, shape and direction separately, we get the corresponding purity. And the purity score can tell us the information that the emergent language encoded. From the table, we can tell that compared with discrete VAE, our model focus more on the direction and less on color and shape which is reasonable because there are only 3 choices for color and shape which makes it less useful for disguising the target room from others, comparing with the direction with 8 possible value. We also investigate factors that may affect task performance 6. Countering with our intuitions that, as the sentence length grows, the model has a much more powerful expression ability, so the performance should grow, but localization accuracy grows first and then drop. When the sentence length is enough for expression, much more tokens may increase the difficulty for the model to tackle. The shortest route length between the start positions and the target ones can also affect the task performance, but it does not follow an exponential drop when the shortest route length grows. In this work, we propose a navigation task in an emergent communication scenario.

However, compared with the traditional VLN task, the application of multi-agent communication in the VLN task makes the communication policy more flexible and can generate a more complex

* Kajic et al. (2020) Ivana Kajic, Eser Aygun, and Doina Precup. Learning to cooperate: Emergent communication in multi-agent navigation. _arXiv preprint arXiv:2004.01097_, 2020.

* Lazaridou et al. (2018) Angeliki Lazaridou, Karl Moritz Hermann, Karl Tuyls, and Stephen Clark. Emergence of linguistic communication from referential games with symbolic and pixel input, 2018. URL [https://arxiv.org/abs/1804.03984](https://arxiv.org/abs/1804.03984). * Narayan-Chen et al. (2019) Anjali Narayan-Chen, Prashant Jayannavar, and Julia Hockenmaier. Collaborative dialogue in Minecraft. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pp. 5405-5415, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1537. URL [https://aclanthology.org/P19-1537](https://aclanthology.org/P19-1537).

Input: Figure 1: The navigation task involves three components: the tourist (agent A), the guide (agent B), and the environment. Agent A and B have unequal abilities. While agent A can only see its surroundings and describe them to B, agent B has the global perspective, can make a route plan, and give instructions to A. Agent A and B cooperate to lead A to the target place.

Input: Figure 3: Model Structure. There are three sub-modules here, including localization, route plan, and guidance. In the localization part, by calculating the similarity score between messages given by A and room feature embeddings incorporating structural information, agent B tries to find where A is. Then we use the Floyd algorithm to find the shortest path between the guessed place and the target one. And in the guidance module, A could find its path with history dialogue with B and its own observations.

Input: Figure 2: Navigation Case. The navigation task consists of three sub-tasks: Localization, Route Plan, and Guidance. At every turn, agent A first gives a description of its surroundings in the room it is located. With the current messages and the history messages given by A, agent B tries to guess where A is using the topological structure of the rooms. Then, based on this guess, agent B makes a route plan for A to get to the target place, and gives an instruction to A describing the next step A should take.

Input: Figure 6: The effect of (a) sentence length and (b) the shortest route length between the start and target places on the task performance

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find the claim to be moderately overstated. While the evidence demonstrates that the authors did develop a collaborative framework for multi-agent communication in a navigation task, several aspects are overstated:

1. The claim suggests a general solution for multi-agent emergent communications, but the evidence shows a specific implementation for a two-agent navigation task.

2. The phrase "enables the agents to generate and understand emergent language" implies a broader capability than what's shown in the evidence, which demonstrates a specific implementation with limitations (e.g., sentence length affecting performance).

3. The evidence shows mixed results - while some aspects work well, there are clear limitations and performance trade-offs (as shown in Figure 6).

4. The framework focuses primarily on navigation tasks rather than general emergent communication problems.

<score>3</score>

Justification:
- The evidence supports the development of a collaborative framework, but its scope and capabilities are more limited than the claim suggests
- Results show both successes and limitations (e.g., sentence length effects on performance)
- The implementation is specific to navigation tasks rather than general emergent communication
- The framework works for a specific two-agent scenario rather than general multi-agent settings
- The evidence demonstrates concrete results but also reveals constraints and performance trade-offs not acknowledged in the broad claim


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We also seek to address the multi-agent emergent communications by proposing a collaborative learning framework that enables the agents to generate and understand emergent language and solve task.

The review comment to be evaluated is:

Strengths:
* The proposed setting is novel.

Weaknesses:
* While novel, the proposed setting is poorly motivated. Why studying this setting is significant? What would it enable?
* The description of the environment is very nebulous. Especially, the complexity of the visual and language components are highly under-specified. How is the input observation represented (image or feature-based)? How many object categories? How large is the agent's vocabulary? How are the environment graph generated? It is thus difficult to comprehend the significance of solving this environment.
* The plots are missing confidence intervals. Each experiments are not conducted multiple times with different random seeds. This seriously weakens the empirical claims.

The evidence to be evaluated is:Navigation tasks with vision and natural language information have attracted much attention in the last few years. Generally, this task involves humans, an artificial agent, and the environment. The agent communicates with humans in natural language. The agent may receive requests or instructions and ask for more guidance from the human. It can navigate and interact with the environment to get more information or complete corresponding task requested by the human.

By putting the VLN task into multi-agent settings, the problems listed above can be alleviated or solved. To our knowledge, talk-the-walk de Vries et al. (2018) is the only work applying the VLN task with multiple agents and is very similar to our task settings. However, talk-the-walk only focuses on the localization part, where the guide finds where the tourist is. But ignore the guidance part, in which the guide makes a route plan and gives guidance to the tourist, and the tourist makes corresponding actions. Talk-the-walk models this part with a random walk and only implements a minimal baseline, making the VLN task with multi-agent settings incomplete. **Emergent Language** Emergent language is an unplanned language that comes up in interactions between the students, and/or the teacher and the students. From the perspective of agents' relationships, most of the works are focused on fully cooperative tasks like referential games Lazaridou et al. (2018) adapted from the Lewis signaling game. In the referential game, there are two roles: speaker and listener. The speaker has to give instructions to the listener so that it can select the target picture from the candidates successfully. Some navigation tasks are proposed Kalinowska et al. (2022); Kajic et al. (2020), however, those works can only be applied in limited environments, with 2 and 4 cases respectively. Some semi-cooperative tasks are also proposed, like the negotiation game Cao et al. (2018), in which two agents have to negotiate to get more scores for themselves. There are also some fully-competitive cases like a circular-based sender-receiver game Noukhovitch et al. (2021) where the sender tries to make the receiver choose the position closer to its own target position rather than the receiver's. Most of those part of works are studying single message pass, with one agent only having a language generation part, and the other one having a language understanding module. Still, there are some works putting attention to the multi-turn form Cao et al. (2018); Evtimova et al. (2017). But the former study focuses on a semi-cooperative case rather than a fully cooperative one like us, while the latter one uses the bag of words, and generates 0 or 1, with 1 indicating this token is in the generated sentence.

This task is inspired by a real-world scenario where two agents with unequal abilities, agents A and B, cooperate to help agent A find the target place. As pictured in Figure 1, agent A can only see the objects around it, while agent B has the floor plan and the target place but doesn't know where A is. To win this game, they have to cooperate to help A reach the target place with limited rounds.

Agent B needs to find where A is, according to the message given by A describing the objects around it.

Having the position of A and the target place, agent B has to find the shortest path for A to get there.

Agent B gives instructions to A, helping it follow the route plan.

Formally, this navigation task is constructed as follows: There are houses, containing rooms \(R=\{r_{0},r_{1},...,r_{N}\}\) each. Every time, one house is randomly chosen from those \(K\) candidates. Then two rooms are drawn from the selected house, and one of them is the target room \(r_{o}\), and the other is the start room \(r_{s}\). Every room has several gates \(D_{i}=\{d_{ij},j\in Neighbor(i)\}\). There is a graph for each house, to model the relationship between rooms. We use \(G=(R,D)\) to denote a house, where \(R\) is the vertex set, and \(D\) is the edge set. There are two players called agent A, and agent B parameterized by \(\theta_{A},\theta_{B}\) separately.

* In the localization task, agent A takes in the room's vector where it is and gives out the messages of step t consisting of \(M\) tokens \(m_{t}=\{v_{0},v_{1},...,v_{M}\},v_{i}\in V\). While agent B collects history dialogue \(h_{t}=(m_{0},m_{1},..,m_{t})\) and tries to guess where A is.

* In the route plan task, agent B uses the room graph \(G\), the guessed position of A \(\arg\max(\theta_{B}(h_{t},G))\), and the target position \(r_{o}\) to get the next step A should take \(\arg\max(\theta_{A}(h_{t+1},D))\).

Agent A encodes the target room information \(r_{o}\) including objects and gates and then uses the encoded vector to generate corresponding emergent language with its language generation module \(e_{s}^{o}=GRU(MLP(r_{o}))\). With the message coming from agent A in this turn, and the history sentences from A, agent B makes its guess by making use of the topological structure of rooms \(\theta_{B}(h_{t},G)\).

Similarly, the model for the guidance task also includes an encoder, a language generation, and a language understanding module used for giving instructions to agent A about its next step. Considering the non-differentiability of the discrete messages passing between agents, those models are trained with REINFORCE algorithm.

Finally, we put those sub-tasks together to complete the navigation task.

With the limitations of the message length and vocabulary size in discrete communication scenarios, discrete VAE may communicate less useful information compared with our model. We also show the case when agents can pass continuous messages, giving us a ceiling performance when the communication channel is not restricted.

We suppose that good language tends to give similar expressions describing a similar object. So we use topographic similarity \(\rho\) as our metric, which is the Spearman correlation between the message similarity sequence and the room vector similarity sequence. And our method outperforms the discrete VAE largely.

Then, we cluster those rooms with the generated message. Labeled these rooms with their objects' color, shape and direction separately, we get the corresponding purity. And the purity score can tell us the information that the emergent language encoded. From the table, we can tell that compared with discrete VAE, our model focus more on the direction and less on color and shape which is reasonable because there are only 3 choices for color and shape which makes it less useful for disguising the target room from others, comparing with the direction with 8 possible value. We also investigate factors that may affect task performance 6. Countering with our intuitions that, as the sentence length grows, the model has a much more powerful expression ability, so the performance should grow, but localization accuracy grows first and then drop. When the sentence length is enough for expression, much more tokens may increase the difficulty for the model to tackle. The shortest route length between the start positions and the target ones can also affect the task performance, but it does not follow an exponential drop when the shortest route length grows. In this work, we propose a navigation task in an emergent communication scenario.

However, compared with the traditional VLN task, the application of multi-agent communication in the VLN task makes the communication policy more flexible and can generate a more complex

* Kajic et al. (2020) Ivana Kajic, Eser Aygun, and Doina Precup. Learning to cooperate: Emergent communication in multi-agent navigation. _arXiv preprint arXiv:2004.01097_, 2020.

* Lazaridou et al. (2018) Angeliki Lazaridou, Karl Moritz Hermann, Karl Tuyls, and Stephen Clark. Emergence of linguistic communication from referential games with symbolic and pixel input, 2018. URL [https://arxiv.org/abs/1804.03984](https://arxiv.org/abs/1804.03984). * Narayan-Chen et al. (2019) Anjali Narayan-Chen, Prashant Jayannavar, and Julia Hockenmaier. Collaborative dialogue in Minecraft. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pp. 5405-5415, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1537. URL [https://aclanthology.org/P19-1537](https://aclanthology.org/P19-1537).

Input: Figure 1: The navigation task involves three components: the tourist (agent A), the guide (agent B), and the environment. Agent A and B have unequal abilities. While agent A can only see its surroundings and describe them to B, agent B has the global perspective, can make a route plan, and give instructions to A. Agent A and B cooperate to lead A to the target place.

Input: Figure 3: Model Structure. There are three sub-modules here, including localization, route plan, and guidance. In the localization part, by calculating the similarity score between messages given by A and room feature embeddings incorporating structural information, agent B tries to find where A is. Then we use the Floyd algorithm to find the shortest path between the guessed place and the target one. And in the guidance module, A could find its path with history dialogue with B and its own observations.

Input: Figure 2: Navigation Case. The navigation task consists of three sub-tasks: Localization, Route Plan, and Guidance. At every turn, agent A first gives a description of its surroundings in the room it is located. With the current messages and the history messages given by A, agent B tries to guess where A is using the topological structure of the rooms. Then, based on this guess, agent B makes a route plan for A to get to the target place, and gives an instruction to A describing the next step A should take.

Input: Figure 6: The effect of (a) sentence length and (b) the shortest route length between the start and target places on the task performance

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence carefully, I find the claim to be moderately overstated. While the paper does propose a collaborative learning framework for multi-agent communication, several key issues highlighted in the review weaken the claim's strength:

1. The environment specifications are underspecified (vocabulary size, object categories, etc.)
2. Lack of confidence intervals and multiple random seed experiments weakens empirical validation
3. The evidence shows limitations in performance with increased sentence length
4. The motivation and significance of the setting are poorly justified

While the framework does enable agents to generate and understand emergent language, claiming it "solves the task" is an overstatement given these limitations and experimental weaknesses.

<score>4</score>

Justification:
1. The claim uses definitive language ("solves task") despite experimental limitations noted in the review
2. Evidence shows mixed results - while some communication emerges, performance varies significantly with sentence length (Fig 6)
3. Missing critical experimental details and statistical validation make it difficult to fully support the claim's strength
4. The environment's loose specification makes it unclear how well the solution generalizes


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We also seek to address the multi-agent emergent communications by proposing a collaborative learning framework that enables the agents to generate and understand emergent language and solve task.

The review comment to be evaluated is:

While this paper goes in a good direction by moving away from the Lewis game in the language emergence communication setting, it falls short as there is little scientific analysis. The authors mainly designed an artificial navigation task, handcrafted an observation space and reward, designed a neural network, trained it, and reported the results. Those results are not put into perspective with a detailed researched perspective. Indeed, the point of Emecom in ML is not to create artificial tasks for the sake of solving it, but to design tasks that could help understand how language evolves with a human intervention as possible (or being generic).

I am listing here a few leads to improve the paper in the long run:
 - The introduction should frame the research problem into the past literature, ask a scientific question, and explain how the paper help answers it. Here, there is no citation; the VLM task is only said to be more natural (no reference to the embodiment, situated learning, or going beyond the signaling game)
 - The related work is extremely short. 18 citations in total for a paper about language emergence, and the navigation task is just inherently incomplete. On a positive note, the most relevant citations kajic/Kanilowska/DeVries are present
- The paper is 7 pages long, and page 3 is barely empty
- The model is designed to solve the task, e.g., the graph networks are specific to the observation space. I would recommend either picking a minimal architecture or reproducing the architecture that is classic in VLM.
- having 4 losses in an RL task is generally a bad sign as it is a symptom of human intervention. cf The bitter lesson by Sutton
 - Results are not compared to any baselines (VAE vs your network is not really a baseline...), and simply reporting the success rate is not sufficient. Examples of suitable ablation include: the correlation between language and environment complexity. Qualitative experiments to translate language and change in the environment (e.g., how changing the graph impacts the language), what are the external constraints that shape language, reward ablation, etc.

I would recommend the authors to re-work their paper and present it to a workshop to have further guidance. The initial idea is good, but the realization lacks scientific maturity.

The evidence to be evaluated is:Navigation tasks with vision and natural language information have attracted much attention in the last few years. Generally, this task involves humans, an artificial agent, and the environment. The agent communicates with humans in natural language. The agent may receive requests or instructions and ask for more guidance from the human. It can navigate and interact with the environment to get more information or complete corresponding task requested by the human.

By putting the VLN task into multi-agent settings, the problems listed above can be alleviated or solved. To our knowledge, talk-the-walk de Vries et al. (2018) is the only work applying the VLN task with multiple agents and is very similar to our task settings. However, talk-the-walk only focuses on the localization part, where the guide finds where the tourist is. But ignore the guidance part, in which the guide makes a route plan and gives guidance to the tourist, and the tourist makes corresponding actions. Talk-the-walk models this part with a random walk and only implements a minimal baseline, making the VLN task with multi-agent settings incomplete. **Emergent Language** Emergent language is an unplanned language that comes up in interactions between the students, and/or the teacher and the students. From the perspective of agents' relationships, most of the works are focused on fully cooperative tasks like referential games Lazaridou et al. (2018) adapted from the Lewis signaling game. In the referential game, there are two roles: speaker and listener. The speaker has to give instructions to the listener so that it can select the target picture from the candidates successfully. Some navigation tasks are proposed Kalinowska et al. (2022); Kajic et al. (2020), however, those works can only be applied in limited environments, with 2 and 4 cases respectively. Some semi-cooperative tasks are also proposed, like the negotiation game Cao et al. (2018), in which two agents have to negotiate to get more scores for themselves. There are also some fully-competitive cases like a circular-based sender-receiver game Noukhovitch et al. (2021) where the sender tries to make the receiver choose the position closer to its own target position rather than the receiver's. Most of those part of works are studying single message pass, with one agent only having a language generation part, and the other one having a language understanding module. Still, there are some works putting attention to the multi-turn form Cao et al. (2018); Evtimova et al. (2017). But the former study focuses on a semi-cooperative case rather than a fully cooperative one like us, while the latter one uses the bag of words, and generates 0 or 1, with 1 indicating this token is in the generated sentence.

This task is inspired by a real-world scenario where two agents with unequal abilities, agents A and B, cooperate to help agent A find the target place. As pictured in Figure 1, agent A can only see the objects around it, while agent B has the floor plan and the target place but doesn't know where A is. To win this game, they have to cooperate to help A reach the target place with limited rounds.

Agent B needs to find where A is, according to the message given by A describing the objects around it.

Having the position of A and the target place, agent B has to find the shortest path for A to get there.

Agent B gives instructions to A, helping it follow the route plan.

Formally, this navigation task is constructed as follows: There are houses, containing rooms \(R=\{r_{0},r_{1},...,r_{N}\}\) each. Every time, one house is randomly chosen from those \(K\) candidates. Then two rooms are drawn from the selected house, and one of them is the target room \(r_{o}\), and the other is the start room \(r_{s}\). Every room has several gates \(D_{i}=\{d_{ij},j\in Neighbor(i)\}\). There is a graph for each house, to model the relationship between rooms. We use \(G=(R,D)\) to denote a house, where \(R\) is the vertex set, and \(D\) is the edge set. There are two players called agent A, and agent B parameterized by \(\theta_{A},\theta_{B}\) separately.

* In the localization task, agent A takes in the room's vector where it is and gives out the messages of step t consisting of \(M\) tokens \(m_{t}=\{v_{0},v_{1},...,v_{M}\},v_{i}\in V\). While agent B collects history dialogue \(h_{t}=(m_{0},m_{1},..,m_{t})\) and tries to guess where A is.

* In the route plan task, agent B uses the room graph \(G\), the guessed position of A \(\arg\max(\theta_{B}(h_{t},G))\), and the target position \(r_{o}\) to get the next step A should take \(\arg\max(\theta_{A}(h_{t+1},D))\).

Agent A encodes the target room information \(r_{o}\) including objects and gates and then uses the encoded vector to generate corresponding emergent language with its language generation module \(e_{s}^{o}=GRU(MLP(r_{o}))\). With the message coming from agent A in this turn, and the history sentences from A, agent B makes its guess by making use of the topological structure of rooms \(\theta_{B}(h_{t},G)\).

Similarly, the model for the guidance task also includes an encoder, a language generation, and a language understanding module used for giving instructions to agent A about its next step. Considering the non-differentiability of the discrete messages passing between agents, those models are trained with REINFORCE algorithm.

Finally, we put those sub-tasks together to complete the navigation task.

With the limitations of the message length and vocabulary size in discrete communication scenarios, discrete VAE may communicate less useful information compared with our model. We also show the case when agents can pass continuous messages, giving us a ceiling performance when the communication channel is not restricted.

We suppose that good language tends to give similar expressions describing a similar object. So we use topographic similarity \(\rho\) as our metric, which is the Spearman correlation between the message similarity sequence and the room vector similarity sequence. And our method outperforms the discrete VAE largely.

Then, we cluster those rooms with the generated message. Labeled these rooms with their objects' color, shape and direction separately, we get the corresponding purity. And the purity score can tell us the information that the emergent language encoded. From the table, we can tell that compared with discrete VAE, our model focus more on the direction and less on color and shape which is reasonable because there are only 3 choices for color and shape which makes it less useful for disguising the target room from others, comparing with the direction with 8 possible value. We also investigate factors that may affect task performance 6. Countering with our intuitions that, as the sentence length grows, the model has a much more powerful expression ability, so the performance should grow, but localization accuracy grows first and then drop. When the sentence length is enough for expression, much more tokens may increase the difficulty for the model to tackle. The shortest route length between the start positions and the target ones can also affect the task performance, but it does not follow an exponential drop when the shortest route length grows. In this work, we propose a navigation task in an emergent communication scenario.

However, compared with the traditional VLN task, the application of multi-agent communication in the VLN task makes the communication policy more flexible and can generate a more complex

* Kajic et al. (2020) Ivana Kajic, Eser Aygun, and Doina Precup. Learning to cooperate: Emergent communication in multi-agent navigation. _arXiv preprint arXiv:2004.01097_, 2020.

* Lazaridou et al. (2018) Angeliki Lazaridou, Karl Moritz Hermann, Karl Tuyls, and Stephen Clark. Emergence of linguistic communication from referential games with symbolic and pixel input, 2018. URL [https://arxiv.org/abs/1804.03984](https://arxiv.org/abs/1804.03984). * Narayan-Chen et al. (2019) Anjali Narayan-Chen, Prashant Jayannavar, and Julia Hockenmaier. Collaborative dialogue in Minecraft. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pp. 5405-5415, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1537. URL [https://aclanthology.org/P19-1537](https://aclanthology.org/P19-1537).

Input: Figure 1: The navigation task involves three components: the tourist (agent A), the guide (agent B), and the environment. Agent A and B have unequal abilities. While agent A can only see its surroundings and describe them to B, agent B has the global perspective, can make a route plan, and give instructions to A. Agent A and B cooperate to lead A to the target place.

Input: Figure 3: Model Structure. There are three sub-modules here, including localization, route plan, and guidance. In the localization part, by calculating the similarity score between messages given by A and room feature embeddings incorporating structural information, agent B tries to find where A is. Then we use the Floyd algorithm to find the shortest path between the guessed place and the target one. And in the guidance module, A could find its path with history dialogue with B and its own observations.

Input: Figure 2: Navigation Case. The navigation task consists of three sub-tasks: Localization, Route Plan, and Guidance. At every turn, agent A first gives a description of its surroundings in the room it is located. With the current messages and the history messages given by A, agent B tries to guess where A is using the topological structure of the rooms. Then, based on this guess, agent B makes a route plan for A to get to the target place, and gives an instruction to A describing the next step A should take.

Input: Figure 6: The effect of (a) sentence length and (b) the shortest route length between the start and target places on the task performance

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on my analysis of the evidence and review comments, this claim appears to be overstated. While the paper does present a collaborative learning framework for emergent communication between agents, several key issues highlighted in the review suggest the claim overstates the contribution:

1. Limited scientific analysis and lack of proper baselines
2. Highly specific/handcrafted task design rather than generalizable insights about language evolution
3. Multiple losses suggesting heavy human intervention rather than true emergence
4. Insufficient comparison to existing work (only 18 citations in a language emergence paper)
5. Lack of detailed analysis of how language actually emerges and evolves

The evidence shows they implemented a navigation task with emergent communication, but falls short of rigorously demonstrating how their framework enables genuine language emergence and understanding.

<score>4</score>