Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

More notably, GAPS maintains good performance in even more impoverished annotation settings, where only single instances of novel objects are annotated.

The review comment to be evaluated is:

Strengths:
- The paper introduces the copy-paste data augmentation strategy to the few-shot incremental semantic segmentation, which seems to be a novel combination.
- The proposed guided sample selection method for memory construction and data synthesis improves the performance of several baseline methods.
- The paper is most clearly written and easy to follow.

Weaknesses:
- While the proposed data augmentation strategy helps few-shot learning, its technical novelty in the incremental semantic segmentation (ISS) is rather limited, as it is an add-on module and has to rely on the base ISS methods.
- The construction of the memory-replay buffer has several limitations and will affect the quality of the data augmentation in ISS:
1) Only the base stage has fully-annotated images that can be used for the target images in the copy-paste operation, and all the images from the incremental stages are partially labeled, which are unusable for that purpose. As a result, when the incremental learning evolves, the memory has to add samples of new classes and remove some base-stage images due to the capacity limit. Eventually, the memory buffer would be less useful.
2) The diversity-guided selection procedure in Sec 3.2 considers each class separately in sampling. However, for semantic segmentation, it is common that each image has regions of multiple classes. It is unclear how such a method can balance different classes. This is also a problem for the class-frequency-guided selection, as each image can have multiple classes. Another issue is that this method did not take into account the region sizes as it only considers the image-level sampling factor.
- There are several concerns on the experimental evaluation, which make the conclusion less convincing:
1) Both benchmarks are mainly used in the few-shot segmentation literature, which are a bit small in terms of data set size and number of classes. Similar to SSUL and other ISS works, the method should be evaluated on more diverse ISS task settings and larger dataset, such as ADE.
2) For the few-shot side, while this work adopts a different evaluation protocol (as described in Sec. 4.2), it would be more informative if it also uses the protocol in Cermelli et al 2021 for a fair comparison. Moreover, unlike the standard ISS, the training process of the few-shot setting typically has a large variance due to different training samples annotated. Thus it should report both the mean and standard deviation in the final results.
3) The gaps in the ablation study are relatively small. It would more convincing if the standard deviation can be provided. It is unclear how significant such improvements are in the few-shot learning setting.

The evidence to be evaluated is:However, existing work relies on rich annotations and tends to fail when only a limited amount of data is available. In contrast, our work enables incremental segmentation learning with _few data_ via a guided copy-paste process, which demonstrates promising performance under the few-shot and more impoverished single-instance setting.

During the _incremental learning stage_, a sequence of tasks \(\{D_{1},D_{2},\dots\}\) with novel categories is presented to the model, where \(D_{j}=\{(x_{i},y_{i})|x_{i}\in\mathcal{X},y_{i}\in\mathcal{Y}^{\mathcal{C}_{j}}\}\) and \(\mathcal{C}_{j}\) is a set of classes for task \(D_{j}\). In few-shot learning, the size of the training sets for the novel tasks is small, _i.e_., \(|D_{j}|\ll|\mathcal{D}_{0}|\).

To enforce the partially-annotated image setting, we follow Cermelli _et al_. and assume that only novel classes are annotated, _i.e_., \(\mathcal{C}_{i}\cap C_{j}=\varnothing\) for all \(i\neq j\).

It is a _generic and model-agnostic_ data synthesis framework that generates fully-labeled scenes from partially-annotated images of novel objects as a preprocessor to the underlying segmentation model.

During the incremental learning stage, fully-labeled samples are synthesized by copying from the masked novel objects in \(D_{1},\dots,D_{j}\) and pasting onto base exemplars from the replay buffer \(\hat{D}_{0}\).

In principle, GAPS is model-agnostic, which means that it can work with many incremental segmentation models as a diverse data source to improve their performance.

**GAPS consistently increases performance under few-shot settings.** Methods combined with our proposed data source, GAPS, consistently outperform their un-augmented counterpart on both the base and novel categories performance. It is worth noting that GAPS substantially boosts the performance of methods that originally require fully-annotated training images (i.e., fine-tuning and PIFS), despite using only partially-annotated images now. Even for methods that do not carry out end-to-end training and update only the classifier (i.e., SSUL), GAPS still steadily increases performance on novel categories.

**Frequency-guided probabilistic synthesis boosts results on novel classes**.

Though the semantic background shift proposed by Cermelli et al. (2020) relaxes the requirement to provide full segmentation annotations, it still requires _all novel instances_ in images to be annotated, which can be time-consuming to obtain in cluttered scenes and hinder potential applications. Here we consider a more challenging task setting, which we term _single-instance incremental learning_. Namely, for training images provided in incremental learning stages, if there are multiple instances of a novel class in the image, we assume that only _one instance_ will be annotated. To simulate this setting, we use the instance-level segmentation annotation provided by the COCO dataset to enforce only annotation of one novel instance in every image is available to the model.

We can observe that the pseudo-labeling baseline, \(\text{PIFS}(\mathcal{L}_{2})^{\dagger}\), yields substantially worse performance when the model receives single-instance annotations.

On the contrary, \(\text{PIFS}(\mathcal{L}_{2})\)+GAPS shows only a minor performance decrease with single instances. This highlights the potential of copy-paste synthesis as an alternative to the existing pseudo-labeling paradigm to cope with the more realistic single-instance setting.

In this paper, we demonstrate how judicious use of copy-paste dramatically boosts the performance of incremental segmentation methods under the few-shot setting and enables learning with partially-annotated images.

Finally, the ability to learn with as few as one annotated instance in an image raises several intriguing possibilities. For example, integrating our work with learning-based interactive segmentation will enable human operators to continually and adaptively teach novel classes and correct failed predictions. This workflow has many interesting applications such as robot teleoperation where sparse annotations are preferable. Learning with weaker annotations, like bounding boxes or single clicks, and even self-supervision, is also an interesting direction to explore.

In our implementation, the probability to synthesize novel objects is simply given by \(1/M\), where \(M\) is the sum of total number of base classes and seen novel classes.

In the result, we can observe that our method (vRFS) achieves the best result in terms of the novel IoU and harmonic mean of base and novel IoU.

We perform an additional experiment on COCO-20-1 where a long few-shot incremental learning sequence (60 base; 20 novel classes with 20 incremental steps; 5-shot) is used.

PIFS(\(\mathcal{L}_{2}\)), augmented with GAPS, demonstrates more robustness to forgetting with many incremental learning steps than standard PIFS. To validate our approach on the ADE20k dataset (Zhou et al., 2017, 2019), we ran a suite of experiments where we follow the ADE100-5 (11 steps) setup discussed in (Cha et al., 2021) and modify it as 5-shot learning. We tested PIFS(\(\mathcal{L}_{2}\)) and PIFS(\(\mathcal{L}_{2}\))+GAPS on this setting. The results are given in Table 9. We observe a steady increase in both the base IoU and novel IoU in this challenging task setting on methods augmented with GAPS.

Our PIFS(\(\mathcal{L}_{2}\))+GAPS significantly outperforms both baselines.

We conducted an additional experiment on the COCO-20-1 dataset under the 5-shot setting, where we vary the number of fully-annotated images in the memory-replay buffer to investigate the per formance of our method. As can be seen from Fig. 4, even with as few as 100 fully-annotated base images, GAPS still consistently boosts the performance of the underlying PIFS(\(\mathcal{L}_{2}\)) method.

Fig. 5 shows qualitative results of PIFS(\(\mathcal{L}_{2}\)), PIFS(\(\mathcal{L}_{2}\))+NPS (naive copy-paste), and PIFS(\(\mathcal{L}_{2}\))+GAPS on the PASCAL-5-1 split dataset under the 1-shot setting using images on the left column.

On the other hand, the model trained using GAPS not only yields good result on recently learned classes, but also maintains promising performance on classes adapted earlier.

On the other hand, PIFS(\(\mathcal{L}_{2}\))+GAPS is able to maintain good performance on learned classes as it adapts to new classes.

Input: Table 9: Performance of PIFS(L2) and PIFS(L2)+GAPS on the ADE100-5 disjoint 5-shot setting. Mean and standard deviation over 5 runs are reported.

Input: Figure 1: Our proposed method utilizes guided copy-paste augmentation to synthesize diverse training data, using as few as one single novel instance for training. For example, the model encounters an image of many motorcycles, which is novel to the model. As a result, the model incorrectly assigns learned bicycle labels to these pixels and therefore needs to be updated. Our proposed method can adapt to the novel motorcycle class with an annotation of a single motorcycle, which can be efficiently annotated; whereas previous work (Cermelli et al., 2020; 2021) require time-consuming annotation of all instances of motorcycles or even the entire image. Best view in color.

Input: Figure 7: Multi-step qualitative results of our method (GAPS) on the PASCAL-5-1 split under 1- shot setting. (Best view in color.)

Input: Figure 2: Overview of GAPS. During the incremental learning stage, GAPS takes in as few as one annotated instance of a single image. It is more probable for GAPS to select a scene contextually similar to the provided image from memory-replay buffer D̂0. The image is then probabilistically pasted to generate synthetic fully-labeled scenes. Note that GAPS is model-agnostic, and here we use PIFS (Cermelli et al., 2021) as an example for the underlying segmentation model to illustrate how GAPS is applied as a pre-processor. Best seen in color.

Input: Figure 5: Qualitative results on the PASCAL-5-1 split under 1-shot setting. Sequential adaptation to five classes are performed in the order indicated by the blue arrow on the left. Visualization is done after all adaptation steps are completed. Observe, particularly in rows 1, 3, and 4, that our method mitigates catastrophic forgetting and improves performance on both base and novel classes compared to PIFS and PIFS+NPS. (Best viewed in color)

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on my review of the evidence, the claim "GAPS maintains good performance in even more impoverished annotation settings, where only single instances of novel objects are annotated" appears well-supported but slightly overstated. While the evidence shows GAPS performs better than baselines with single-instance annotations, showing "only minor performance decrease," the limited experimental validation and reviewer concerns about small gaps in ablation studies and lack of standard deviation reporting suggest some uncertainty in the strength of these results. The evidence demonstrates good but not definitive performance maintenance.

<score>2</score>

Justification:
1. Supporting evidence:
- Results show "only a minor performance decrease with single instances" compared to baseline
- Experiments demonstrate effectiveness with as few as 100 fully-annotated base images
- Performance visualization in Fig. 5 and Fig. 7 shows maintained segmentation quality

2. Tempering factors:
- Reviewer notes "gaps in the ablation study are relatively small" and lack of standard deviation reporting
- Limited benchmark datasets used (noted by reviewer)
- Results focus mainly on comparison to baselines rather than absolute performance metrics

3. The claim uses measured language ("maintains good performance") rather than stronger claims about superiority, which aligns with the evidence presented. However, broader validation across more diverse datasets would strengthen the conclusion.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We propose a new task adapted from the vision-language navigation(VLN) task coming from the human-machine communication area where the guide should communicate with the tourist to give guidance and help it find the target location.

The evidence to be evaluated is:Navigation tasks with vision and natural language information have attracted much attention in the last few years. Generally, this task involves humans, an artificial agent, and the environment. The agent communicates with humans in natural language. The agent may receive requests or instructions and ask for more guidance from the human. It can navigate and interact with the environment to get more information or complete corresponding task requested by the human. While humans should give requests or guidance to help the agent complete the task. Generally, the goal of these VLN tasks is to help the agent find the target place known only by the human side.

Narayan-Chen et al. (2019) put the task into the Minecraft game and instruct the agent to complete building tasks. From the view of language format, we can divide those tasks into three categories Gu et al. (2022):* Initial instructions: The agent is given instructions for the whole navigation episode.

However, this kind of work only gives the instructions at the beginning, so multi-turn communication can not be generated.

* Oracle guidance: In the process of navigation, the agent can request additional natural language guidance to get more information Chi et al. (2019).

By putting the VLN task into multi-agent settings, the problems listed above can be alleviated or solved. To our knowledge, talk-the-walk de Vries et al. (2018) is the only work applying the VLN task with multiple agents and is very similar to our task settings. However, talk-the-walk only focuses on the localization part, where the guide finds where the tourist is. But ignore the guidance part, in which the guide makes a route plan and gives guidance to the tourist, and the tourist makes corresponding actions. Talk-the-walk models this part with a random walk and only implements a minimal baseline, making the VLN task with multi-agent settings incomplete.

In the referential game, there are two roles: speaker and listener. The speaker has to give instructions to the listener so that it can select the target picture from the candidates successfully. Some navigation tasks are proposed Kalinowska et al. (2022); Kajic et al. (2020), however, those works can only be applied in limited environments, with 2 and 4 cases respectively.

Most of those part of works are studying single message pass, with one agent only having a language generation part, and the other one having a language understanding module. Still, there are some works putting attention to the multi-turn form Cao et al. (2018); Evtimova et al. (2017). But the former study focuses on a semi-cooperative case rather than a fully cooperative one like us, while the latter one uses the bag of words, and generates 0 or 1, with 1 indicating this token is in the generated sentence.

This task is inspired by a real-world scenario where two agents with unequal abilities, agents A and B, cooperate to help agent A find the target place. As pictured in Figure 1, agent A can only see the objects around it, while agent B has the floor plan and the target place but doesn't know where A is. To win this game, they have to cooperate to help A reach the target place with limited rounds.

Here, we divide the navigation task into three steps: First, localization. Agent B needs to find where A is, according to the message given by A describing the objects around it. Second, route plan. Having the position of A and the target place, agent B has to find the shortest path for A to get there. Third, guidance. Agent B gives instructions to A, helping it follow the route plan.

Formally, this navigation task is constructed as follows: There are houses, containing rooms \(R=\{r_{0},r_{1},...,r_{N}\}\) each. Every time, one house is randomly chosen from those \(K\) candidates. Then two rooms are drawn from the selected house, and one of them is the target room \(r_{o}\), and the other is the start room \(r_{s}\). Every room has several gates \(D_{i}=\{d_{ij},j\in Neighbor(i)\}\). There is a graph for each house, to model the relationship between rooms. We use \(G=(R,D)\) to denote a house, where \(R\) is the vertex set, and \(D\) is the edge set. There are two players called agent A, and agent B parameterized by \(\theta_{A},\theta_{B}\) separately.

* The only difference between guidance and localization task is that the guidance task switches the role of agents A and B, and agent A does not require the graph \(G\). A case is shown in Figure 2, where Agent A starts from room 0 and wants to get to room 5.

Agent A encodes the target room information \(r_{o}\) including objects and gates and then uses the encoded vector to generate corresponding emergent language with its language generation module \(e_{s}^{o}=GRU(MLP(r_{o}))\). With the message coming from agent A in this turn, and the history sentences from A, agent B makes its guess by making use of the topological structure of rooms \(\theta_{B}(h_{t},G)\).

And B encodes the history sentences with another GRU and computes the similarity between this encoded history \(e_{m}\), with each node representation \(e_{s}^{i}\) calculated above to find the most possible position of agent A. Similarly, the model for the guidance task also includes an encoder, a language generation, and a language understanding module used for giving instructions to agent A about its next step.

Finally, we put those sub-tasks together to complete the navigation task. In the training process, the input of the localization part is the surroundings of agent A, and the output of it is the guessing position made by agent B. However, the input of the route plan part in the next timestep is the actual position of A, instead of the guessing one, preventing the error from localization influencing other modules. While in the testing process, the input of the route plan module comes from the last localization module. We construct the dataset with 6 rooms for each navigation task. And every object in this dataset has 3 optional colors and 3 optional shapes.

In this work, we propose a navigation task in an emergent communication scenario. This task is adapted from the vision-language navigation task proposed in human-machine communication.

Touchdown: Natural language navigation and spatial reasoning in visual street environments. 2018. doi: 10.48550/ARXIV.1811.12354. URL [https://arxiv.org/abs/1811.12354](https://arxiv.org/abs/1811.12354).

Just ask:an interactive learning framework for vision and language navigation, 2019. URL [https://arxiv.org/abs/1912.00915](https://arxiv.org/abs/1912.00915).

Talk the walk: Navigating new york city through grounded dialogue, 2018. URL [https://arxiv.org/abs/1807.03367](https://arxiv.org/abs/1807.03367).

Vision-and-language navigation: A survey of tasks, methods, and future directions. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, 2022. doi: 10.18653/v1/2022.acl-long.524. URL [https://doi.org/10.18653%2Fv1%2F2022.acl-long.524](https://doi.org/10.18653%2Fv1%2F2022.acl-long.524).

Beyond the nav-graph: Vision-and-language navigation in continuous environments, 2020. URL [https://arxiv.org/abs/2004.02857](https://arxiv.org/abs/2004.02857).

* Lazaridou et al. (2018) Angeliki Lazaridou, Karl Moritz Hermann, Karl Tuyls, and Stephen Clark. Emergence of linguistic communication from referential games with symbolic and pixel input, 2018. URL [https://arxiv.org/abs/1804.03984](https://arxiv.org/abs/1804.03984). * Narayan-Chen et al. (2019) Anjali Narayan-Chen, Prashant Jayannavar, and Julia Hockenmaier. Collaborative dialogue in Minecraft. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pp. 5405-5415, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1537. URL [https://aclanthology.org/P19-1537](https://aclanthology.org/P19-1537).

* Thomason et al. (2019) Jesse Thomason, Michael Murray, Maya Cakmak, and Luke Zettlemoyer. Vision-and-dialog navigation, 2019. URL [https://arxiv.org/abs/1907.04957](https://arxiv.org/abs/1907.04957).

Input: Figure 1: The navigation task involves three components: the tourist (agent A), the guide (agent B), and the environment. Agent A and B have unequal abilities. While agent A can only see its surroundings and describe them to B, agent B has the global perspective, can make a route plan, and give instructions to A. Agent A and B cooperate to lead A to the target place.

Input: Figure 2: Navigation Case. The navigation task consists of three sub-tasks: Localization, Route Plan, and Guidance. At every turn, agent A first gives a description of its surroundings in the room it is located. With the current messages and the history messages given by A, agent B tries to guess where A is using the topological structure of the rooms. Then, based on this guess, agent B makes a route plan for A to get to the target place, and gives an instruction to A describing the next step A should take.

Input: Figure 3: Model Structure. There are three sub-modules here, including localization, route plan, and guidance. In the localization part, by calculating the similarity score between messages given by A and room feature embeddings incorporating structural information, agent B tries to find where A is. Then we use the Floyd algorithm to find the shortest path between the guessed place and the target one. And in the guidance module, A could find its path with history dialogue with B and its own observations.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find this claim to be well-stated and accurately reflects what is presented in the evidence. The authors propose adapting the VLN task to a multi-agent setting where a guide and tourist communicate to reach a target location. The evidence thoroughly describes the task setup, methodology, and implementation details through text and figures. The novelty is justified by showing how their approach differs from existing work (like talk-the-walk) by incorporating both localization and guidance components. The claim avoids exaggeration and clearly states what is being proposed.

<score>1</score>

Justification:
1. The evidence provides comprehensive details about how the task works, supported by clear diagrams (Figures 1-3) showing the interaction between agents and environment.

2. The authors acknowledge prior work (e.g., talk-the-walk) while clearly differentiating their contribution: "talk-the-walk only focuses on the localization part... But ignore the guidance part"

3. The claim is precise and modest, simply stating they are proposing an adapted task without making grand claims about performance or impact.

4. The formalization and implementation details provided in the evidence fully support the basic claim being made.

5. No overstatement of capabilities or results - the claim sticks to describing what is actually being proposed.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We propose a new task adapted from the vision-language navigation(VLN) task coming from the human-machine communication area where the guide should communicate with the tourist to give guidance and help it find the target location.

The review comment to be evaluated is:

Strengths:
* The proposed setting is novel.

Weaknesses:
* While novel, the proposed setting is poorly motivated. Why studying this setting is significant? What would it enable?
* The description of the environment is very nebulous. Especially, the complexity of the visual and language components are highly under-specified. How is the input observation represented (image or feature-based)? How many object categories? How large is the agent's vocabulary? How are the environment graph generated? It is thus difficult to comprehend the significance of solving this environment.
* The plots are missing confidence intervals. Each experiments are not conducted multiple times with different random seeds. This seriously weakens the empirical claims.

The evidence to be evaluated is:Navigation tasks with vision and natural language information have attracted much attention in the last few years. Generally, this task involves humans, an artificial agent, and the environment. The agent communicates with humans in natural language. The agent may receive requests or instructions and ask for more guidance from the human. It can navigate and interact with the environment to get more information or complete corresponding task requested by the human. While humans should give requests or guidance to help the agent complete the task. Generally, the goal of these VLN tasks is to help the agent find the target place known only by the human side.

Narayan-Chen et al. (2019) put the task into the Minecraft game and instruct the agent to complete building tasks. From the view of language format, we can divide those tasks into three categories Gu et al. (2022):* Initial instructions: The agent is given instructions for the whole navigation episode.

However, this kind of work only gives the instructions at the beginning, so multi-turn communication can not be generated.

* Oracle guidance: In the process of navigation, the agent can request additional natural language guidance to get more information Chi et al. (2019).

By putting the VLN task into multi-agent settings, the problems listed above can be alleviated or solved. To our knowledge, talk-the-walk de Vries et al. (2018) is the only work applying the VLN task with multiple agents and is very similar to our task settings. However, talk-the-walk only focuses on the localization part, where the guide finds where the tourist is. But ignore the guidance part, in which the guide makes a route plan and gives guidance to the tourist, and the tourist makes corresponding actions. Talk-the-walk models this part with a random walk and only implements a minimal baseline, making the VLN task with multi-agent settings incomplete.

In the referential game, there are two roles: speaker and listener. The speaker has to give instructions to the listener so that it can select the target picture from the candidates successfully. Some navigation tasks are proposed Kalinowska et al. (2022); Kajic et al. (2020), however, those works can only be applied in limited environments, with 2 and 4 cases respectively.

Most of those part of works are studying single message pass, with one agent only having a language generation part, and the other one having a language understanding module. Still, there are some works putting attention to the multi-turn form Cao et al. (2018); Evtimova et al. (2017). But the former study focuses on a semi-cooperative case rather than a fully cooperative one like us, while the latter one uses the bag of words, and generates 0 or 1, with 1 indicating this token is in the generated sentence.

This task is inspired by a real-world scenario where two agents with unequal abilities, agents A and B, cooperate to help agent A find the target place. As pictured in Figure 1, agent A can only see the objects around it, while agent B has the floor plan and the target place but doesn't know where A is. To win this game, they have to cooperate to help A reach the target place with limited rounds.

Here, we divide the navigation task into three steps: First, localization. Agent B needs to find where A is, according to the message given by A describing the objects around it. Second, route plan. Having the position of A and the target place, agent B has to find the shortest path for A to get there. Third, guidance. Agent B gives instructions to A, helping it follow the route plan.

Formally, this navigation task is constructed as follows: There are houses, containing rooms \(R=\{r_{0},r_{1},...,r_{N}\}\) each. Every time, one house is randomly chosen from those \(K\) candidates. Then two rooms are drawn from the selected house, and one of them is the target room \(r_{o}\), and the other is the start room \(r_{s}\). Every room has several gates \(D_{i}=\{d_{ij},j\in Neighbor(i)\}\). There is a graph for each house, to model the relationship between rooms. We use \(G=(R,D)\) to denote a house, where \(R\) is the vertex set, and \(D\) is the edge set. There are two players called agent A, and agent B parameterized by \(\theta_{A},\theta_{B}\) separately.

* The only difference between guidance and localization task is that the guidance task switches the role of agents A and B, and agent A does not require the graph \(G\). A case is shown in Figure 2, where Agent A starts from room 0 and wants to get to room 5.

Agent A encodes the target room information \(r_{o}\) including objects and gates and then uses the encoded vector to generate corresponding emergent language with its language generation module \(e_{s}^{o}=GRU(MLP(r_{o}))\). With the message coming from agent A in this turn, and the history sentences from A, agent B makes its guess by making use of the topological structure of rooms \(\theta_{B}(h_{t},G)\).

And B encodes the history sentences with another GRU and computes the similarity between this encoded history \(e_{m}\), with each node representation \(e_{s}^{i}\) calculated above to find the most possible position of agent A. Similarly, the model for the guidance task also includes an encoder, a language generation, and a language understanding module used for giving instructions to agent A about its next step.

Finally, we put those sub-tasks together to complete the navigation task. In the training process, the input of the localization part is the surroundings of agent A, and the output of it is the guessing position made by agent B. However, the input of the route plan part in the next timestep is the actual position of A, instead of the guessing one, preventing the error from localization influencing other modules. While in the testing process, the input of the route plan module comes from the last localization module. We construct the dataset with 6 rooms for each navigation task. And every object in this dataset has 3 optional colors and 3 optional shapes.

In this work, we propose a navigation task in an emergent communication scenario. This task is adapted from the vision-language navigation task proposed in human-machine communication.

Touchdown: Natural language navigation and spatial reasoning in visual street environments. 2018. doi: 10.48550/ARXIV.1811.12354. URL [https://arxiv.org/abs/1811.12354](https://arxiv.org/abs/1811.12354).

Just ask:an interactive learning framework for vision and language navigation, 2019. URL [https://arxiv.org/abs/1912.00915](https://arxiv.org/abs/1912.00915).

Talk the walk: Navigating new york city through grounded dialogue, 2018. URL [https://arxiv.org/abs/1807.03367](https://arxiv.org/abs/1807.03367).

Vision-and-language navigation: A survey of tasks, methods, and future directions. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, 2022. doi: 10.18653/v1/2022.acl-long.524. URL [https://doi.org/10.18653%2Fv1%2F2022.acl-long.524](https://doi.org/10.18653%2Fv1%2F2022.acl-long.524).

Beyond the nav-graph: Vision-and-language navigation in continuous environments, 2020. URL [https://arxiv.org/abs/2004.02857](https://arxiv.org/abs/2004.02857).

* Lazaridou et al. (2018) Angeliki Lazaridou, Karl Moritz Hermann, Karl Tuyls, and Stephen Clark. Emergence of linguistic communication from referential games with symbolic and pixel input, 2018. URL [https://arxiv.org/abs/1804.03984](https://arxiv.org/abs/1804.03984). * Narayan-Chen et al. (2019) Anjali Narayan-Chen, Prashant Jayannavar, and Julia Hockenmaier. Collaborative dialogue in Minecraft. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pp. 5405-5415, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1537. URL [https://aclanthology.org/P19-1537](https://aclanthology.org/P19-1537).

* Thomason et al. (2019) Jesse Thomason, Michael Murray, Maya Cakmak, and Luke Zettlemoyer. Vision-and-dialog navigation, 2019. URL [https://arxiv.org/abs/1907.04957](https://arxiv.org/abs/1907.04957).

Input: Figure 1: The navigation task involves three components: the tourist (agent A), the guide (agent B), and the environment. Agent A and B have unequal abilities. While agent A can only see its surroundings and describe them to B, agent B has the global perspective, can make a route plan, and give instructions to A. Agent A and B cooperate to lead A to the target place.

Input: Figure 2: Navigation Case. The navigation task consists of three sub-tasks: Localization, Route Plan, and Guidance. At every turn, agent A first gives a description of its surroundings in the room it is located. With the current messages and the history messages given by A, agent B tries to guess where A is using the topological structure of the rooms. Then, based on this guess, agent B makes a route plan for A to get to the target place, and gives an instruction to A describing the next step A should take.

Input: Figure 3: Model Structure. There are three sub-modules here, including localization, route plan, and guidance. In the localization part, by calculating the similarity score between messages given by A and room feature embeddings incorporating structural information, agent B tries to find where A is. Then we use the Floyd algorithm to find the shortest path between the guessed place and the target one. And in the guidance module, A could find its path with history dialogue with B and its own observations.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on my review of the claim, evidence, and reviewer comments, I find the claim to be moderately overstated. While the authors do present a novel adaptation of the VLN task for human-machine communication, several key issues make the claim overstated:

1. The environment and task complexity are insufficiently specified, as noted in the reviewer comments. The evidence shows a very simplified setup (6 rooms, 3 colors, 3 shapes) without clear justification.

2. The significance and motivation for studying this particular setting are not well established in the evidence.

3. While the authors describe the technical setup in detail, they don't provide experimental results or comparative analysis to support the effectiveness of their approach.

4. The reviewer points out missing confidence intervals and lack of multiple experimental runs, which weakens the empirical support.

<score>4</score>

Justification: While the authors do propose a new task adaptation, the claim overstates its contribution by not acknowledging significant limitations and lacking proper empirical validation. The evidence primarily focuses on describing the technical setup rather than demonstrating its effectiveness or significance. This aligns with the reviewer's concerns about poor motivation and under-specified components.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We propose a new task adapted from the vision-language navigation(VLN) task coming from the human-machine communication area where the guide should communicate with the tourist to give guidance and help it find the target location.

The review comment to be evaluated is:

While this paper goes in a good direction by moving away from the Lewis game in the language emergence communication setting, it falls short as there is little scientific analysis. The authors mainly designed an artificial navigation task, handcrafted an observation space and reward, designed a neural network, trained it, and reported the results. Those results are not put into perspective with a detailed researched perspective. Indeed, the point of Emecom in ML is not to create artificial tasks for the sake of solving it, but to design tasks that could help understand how language evolves with a human intervention as possible (or being generic).

I am listing here a few leads to improve the paper in the long run:
 - The introduction should frame the research problem into the past literature, ask a scientific question, and explain how the paper help answers it. Here, there is no citation; the VLM task is only said to be more natural (no reference to the embodiment, situated learning, or going beyond the signaling game)
 - The related work is extremely short. 18 citations in total for a paper about language emergence, and the navigation task is just inherently incomplete. On a positive note, the most relevant citations kajic/Kanilowska/DeVries are present
- The paper is 7 pages long, and page 3 is barely empty
- The model is designed to solve the task, e.g., the graph networks are specific to the observation space. I would recommend either picking a minimal architecture or reproducing the architecture that is classic in VLM.
- having 4 losses in an RL task is generally a bad sign as it is a symptom of human intervention. cf The bitter lesson by Sutton
 - Results are not compared to any baselines (VAE vs your network is not really a baseline...), and simply reporting the success rate is not sufficient. Examples of suitable ablation include: the correlation between language and environment complexity. Qualitative experiments to translate language and change in the environment (e.g., how changing the graph impacts the language), what are the external constraints that shape language, reward ablation, etc.

I would recommend the authors to re-work their paper and present it to a workshop to have further guidance. The initial idea is good, but the realization lacks scientific maturity.

The evidence to be evaluated is:Navigation tasks with vision and natural language information have attracted much attention in the last few years. Generally, this task involves humans, an artificial agent, and the environment. The agent communicates with humans in natural language. The agent may receive requests or instructions and ask for more guidance from the human. It can navigate and interact with the environment to get more information or complete corresponding task requested by the human. While humans should give requests or guidance to help the agent complete the task. Generally, the goal of these VLN tasks is to help the agent find the target place known only by the human side.

Narayan-Chen et al. (2019) put the task into the Minecraft game and instruct the agent to complete building tasks. From the view of language format, we can divide those tasks into three categories Gu et al. (2022):* Initial instructions: The agent is given instructions for the whole navigation episode.

However, this kind of work only gives the instructions at the beginning, so multi-turn communication can not be generated.

* Oracle guidance: In the process of navigation, the agent can request additional natural language guidance to get more information Chi et al. (2019).

By putting the VLN task into multi-agent settings, the problems listed above can be alleviated or solved. To our knowledge, talk-the-walk de Vries et al. (2018) is the only work applying the VLN task with multiple agents and is very similar to our task settings. However, talk-the-walk only focuses on the localization part, where the guide finds where the tourist is. But ignore the guidance part, in which the guide makes a route plan and gives guidance to the tourist, and the tourist makes corresponding actions. Talk-the-walk models this part with a random walk and only implements a minimal baseline, making the VLN task with multi-agent settings incomplete.

In the referential game, there are two roles: speaker and listener. The speaker has to give instructions to the listener so that it can select the target picture from the candidates successfully. Some navigation tasks are proposed Kalinowska et al. (2022); Kajic et al. (2020), however, those works can only be applied in limited environments, with 2 and 4 cases respectively.

Most of those part of works are studying single message pass, with one agent only having a language generation part, and the other one having a language understanding module. Still, there are some works putting attention to the multi-turn form Cao et al. (2018); Evtimova et al. (2017). But the former study focuses on a semi-cooperative case rather than a fully cooperative one like us, while the latter one uses the bag of words, and generates 0 or 1, with 1 indicating this token is in the generated sentence.

This task is inspired by a real-world scenario where two agents with unequal abilities, agents A and B, cooperate to help agent A find the target place. As pictured in Figure 1, agent A can only see the objects around it, while agent B has the floor plan and the target place but doesn't know where A is. To win this game, they have to cooperate to help A reach the target place with limited rounds.

Here, we divide the navigation task into three steps: First, localization. Agent B needs to find where A is, according to the message given by A describing the objects around it. Second, route plan. Having the position of A and the target place, agent B has to find the shortest path for A to get there. Third, guidance. Agent B gives instructions to A, helping it follow the route plan.

Formally, this navigation task is constructed as follows: There are houses, containing rooms \(R=\{r_{0},r_{1},...,r_{N}\}\) each. Every time, one house is randomly chosen from those \(K\) candidates. Then two rooms are drawn from the selected house, and one of them is the target room \(r_{o}\), and the other is the start room \(r_{s}\). Every room has several gates \(D_{i}=\{d_{ij},j\in Neighbor(i)\}\). There is a graph for each house, to model the relationship between rooms. We use \(G=(R,D)\) to denote a house, where \(R\) is the vertex set, and \(D\) is the edge set. There are two players called agent A, and agent B parameterized by \(\theta_{A},\theta_{B}\) separately.

* The only difference between guidance and localization task is that the guidance task switches the role of agents A and B, and agent A does not require the graph \(G\). A case is shown in Figure 2, where Agent A starts from room 0 and wants to get to room 5.

Agent A encodes the target room information \(r_{o}\) including objects and gates and then uses the encoded vector to generate corresponding emergent language with its language generation module \(e_{s}^{o}=GRU(MLP(r_{o}))\). With the message coming from agent A in this turn, and the history sentences from A, agent B makes its guess by making use of the topological structure of rooms \(\theta_{B}(h_{t},G)\).

And B encodes the history sentences with another GRU and computes the similarity between this encoded history \(e_{m}\), with each node representation \(e_{s}^{i}\) calculated above to find the most possible position of agent A. Similarly, the model for the guidance task also includes an encoder, a language generation, and a language understanding module used for giving instructions to agent A about its next step.

Finally, we put those sub-tasks together to complete the navigation task. In the training process, the input of the localization part is the surroundings of agent A, and the output of it is the guessing position made by agent B. However, the input of the route plan part in the next timestep is the actual position of A, instead of the guessing one, preventing the error from localization influencing other modules. While in the testing process, the input of the route plan module comes from the last localization module. We construct the dataset with 6 rooms for each navigation task. And every object in this dataset has 3 optional colors and 3 optional shapes.

In this work, we propose a navigation task in an emergent communication scenario. This task is adapted from the vision-language navigation task proposed in human-machine communication.

Touchdown: Natural language navigation and spatial reasoning in visual street environments. 2018. doi: 10.48550/ARXIV.1811.12354. URL [https://arxiv.org/abs/1811.12354](https://arxiv.org/abs/1811.12354).

Just ask:an interactive learning framework for vision and language navigation, 2019. URL [https://arxiv.org/abs/1912.00915](https://arxiv.org/abs/1912.00915).

Talk the walk: Navigating new york city through grounded dialogue, 2018. URL [https://arxiv.org/abs/1807.03367](https://arxiv.org/abs/1807.03367).

Vision-and-language navigation: A survey of tasks, methods, and future directions. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, 2022. doi: 10.18653/v1/2022.acl-long.524. URL [https://doi.org/10.18653%2Fv1%2F2022.acl-long.524](https://doi.org/10.18653%2Fv1%2F2022.acl-long.524).

Beyond the nav-graph: Vision-and-language navigation in continuous environments, 2020. URL [https://arxiv.org/abs/2004.02857](https://arxiv.org/abs/2004.02857).

* Lazaridou et al. (2018) Angeliki Lazaridou, Karl Moritz Hermann, Karl Tuyls, and Stephen Clark. Emergence of linguistic communication from referential games with symbolic and pixel input, 2018. URL [https://arxiv.org/abs/1804.03984](https://arxiv.org/abs/1804.03984). * Narayan-Chen et al. (2019) Anjali Narayan-Chen, Prashant Jayannavar, and Julia Hockenmaier. Collaborative dialogue in Minecraft. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pp. 5405-5415, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1537. URL [https://aclanthology.org/P19-1537](https://aclanthology.org/P19-1537).

* Thomason et al. (2019) Jesse Thomason, Michael Murray, Maya Cakmak, and Luke Zettlemoyer. Vision-and-dialog navigation, 2019. URL [https://arxiv.org/abs/1907.04957](https://arxiv.org/abs/1907.04957).

Input: Figure 1: The navigation task involves three components: the tourist (agent A), the guide (agent B), and the environment. Agent A and B have unequal abilities. While agent A can only see its surroundings and describe them to B, agent B has the global perspective, can make a route plan, and give instructions to A. Agent A and B cooperate to lead A to the target place.

Input: Figure 2: Navigation Case. The navigation task consists of three sub-tasks: Localization, Route Plan, and Guidance. At every turn, agent A first gives a description of its surroundings in the room it is located. With the current messages and the history messages given by A, agent B tries to guess where A is using the topological structure of the rooms. Then, based on this guess, agent B makes a route plan for A to get to the target place, and gives an instruction to A describing the next step A should take.

Input: Figure 3: Model Structure. There are three sub-modules here, including localization, route plan, and guidance. In the localization part, by calculating the similarity score between messages given by A and room feature embeddings incorporating structural information, agent B tries to find where A is. Then we use the Floyd algorithm to find the shortest path between the guessed place and the target one. And in the guidance module, A could find its path with history dialogue with B and its own observations.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim, evidence, and reviewer comments, I'll evaluate whether the claim about proposing a new VLN-adapted task is well-stated or overstated.

Summary of evaluation:
The claim appears moderately overstated based on several factors:
1. The task is more of an adaptation of existing VLN work rather than entirely new, as acknowledged in the evidence citing similar work like "talk-the-walk"
2. The reviewer correctly points out limited scientific analysis and lack of proper baselines
3. The evidence shows the task is quite simplified (only 6 rooms, limited object variations) compared to real VLN scenarios
4. While the implementation is described, there's insufficient analysis of how this advances understanding of language emergence

<score>4</score>

Justification:
1. The evidence shows this is an adaptation rather than a truly novel task - particularly the passage stating "This task is adapted from the vision-language navigation task proposed in human-machine communication"

2. The reviewer highlights valid concerns about scientific rigor, noting "little scientific analysis" and that "Results are not compared to any baselines"

3. The implementation details reveal significant simplifications: "We construct the dataset with 6 rooms for each navigation task. And every object in this dataset has 3 optional colors and 3 optional shapes"

4. While the paper describes the mechanics of the task, it doesn't sufficiently justify how this advances the field beyond existing work, as noted in the review comment "the point of Emecom in ML is not to create artificial tasks for the sake of solving it"

Given these points, rating the claim as significantly overstated (4 out of 5) seems appropriate, as it presents an incremental adaptation as a novel task without sufficient scientific validation or comparative analysis.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We propose a new task adapted from the vision-language navigation(VLN) task coming from the human-machine communication area where the guide should communicate with the tourist to give guidance and help it find the target location.

The review comment to be evaluated is:

The paper is not clearly written and some sections are hard to understand. The related works do capture how the framework proposes new challenges that are not already captured in previous work. Besides a similar work that also uses communication between agents [2] perform a similar analysis and comparisons to that work would be help in highlighting the difference with the proposed framework.

Besides the scope of the proposed framework is limited as it can only handle artificial messages that have a much smaller length, vocabulary, and sources of variation. In contrast, previous work have already proposed similar benchmarks that involve real images with natural language descriptions [1].

[1] de Vries et al 2018. Talk the Walk: Navigating New York City through Grounded Dialogue.

[2] Singh et al. 2019. Learning when to Communicate at Scale in Multiagent Cooperative and Competitive Tasks.

The evidence to be evaluated is:Navigation tasks with vision and natural language information have attracted much attention in the last few years. Generally, this task involves humans, an artificial agent, and the environment. The agent communicates with humans in natural language. The agent may receive requests or instructions and ask for more guidance from the human. It can navigate and interact with the environment to get more information or complete corresponding task requested by the human. While humans should give requests or guidance to help the agent complete the task. Generally, the goal of these VLN tasks is to help the agent find the target place known only by the human side.

Narayan-Chen et al. (2019) put the task into the Minecraft game and instruct the agent to complete building tasks. From the view of language format, we can divide those tasks into three categories Gu et al. (2022):* Initial instructions: The agent is given instructions for the whole navigation episode.

However, this kind of work only gives the instructions at the beginning, so multi-turn communication can not be generated.

* Oracle guidance: In the process of navigation, the agent can request additional natural language guidance to get more information Chi et al. (2019).

By putting the VLN task into multi-agent settings, the problems listed above can be alleviated or solved. To our knowledge, talk-the-walk de Vries et al. (2018) is the only work applying the VLN task with multiple agents and is very similar to our task settings. However, talk-the-walk only focuses on the localization part, where the guide finds where the tourist is. But ignore the guidance part, in which the guide makes a route plan and gives guidance to the tourist, and the tourist makes corresponding actions. Talk-the-walk models this part with a random walk and only implements a minimal baseline, making the VLN task with multi-agent settings incomplete.

In the referential game, there are two roles: speaker and listener. The speaker has to give instructions to the listener so that it can select the target picture from the candidates successfully. Some navigation tasks are proposed Kalinowska et al. (2022); Kajic et al. (2020), however, those works can only be applied in limited environments, with 2 and 4 cases respectively.

Most of those part of works are studying single message pass, with one agent only having a language generation part, and the other one having a language understanding module. Still, there are some works putting attention to the multi-turn form Cao et al. (2018); Evtimova et al. (2017). But the former study focuses on a semi-cooperative case rather than a fully cooperative one like us, while the latter one uses the bag of words, and generates 0 or 1, with 1 indicating this token is in the generated sentence.

This task is inspired by a real-world scenario where two agents with unequal abilities, agents A and B, cooperate to help agent A find the target place. As pictured in Figure 1, agent A can only see the objects around it, while agent B has the floor plan and the target place but doesn't know where A is. To win this game, they have to cooperate to help A reach the target place with limited rounds.

Here, we divide the navigation task into three steps: First, localization. Agent B needs to find where A is, according to the message given by A describing the objects around it. Second, route plan. Having the position of A and the target place, agent B has to find the shortest path for A to get there. Third, guidance. Agent B gives instructions to A, helping it follow the route plan.

Formally, this navigation task is constructed as follows: There are houses, containing rooms \(R=\{r_{0},r_{1},...,r_{N}\}\) each. Every time, one house is randomly chosen from those \(K\) candidates. Then two rooms are drawn from the selected house, and one of them is the target room \(r_{o}\), and the other is the start room \(r_{s}\). Every room has several gates \(D_{i}=\{d_{ij},j\in Neighbor(i)\}\). There is a graph for each house, to model the relationship between rooms. We use \(G=(R,D)\) to denote a house, where \(R\) is the vertex set, and \(D\) is the edge set. There are two players called agent A, and agent B parameterized by \(\theta_{A},\theta_{B}\) separately.

* The only difference between guidance and localization task is that the guidance task switches the role of agents A and B, and agent A does not require the graph \(G\). A case is shown in Figure 2, where Agent A starts from room 0 and wants to get to room 5.

Agent A encodes the target room information \(r_{o}\) including objects and gates and then uses the encoded vector to generate corresponding emergent language with its language generation module \(e_{s}^{o}=GRU(MLP(r_{o}))\). With the message coming from agent A in this turn, and the history sentences from A, agent B makes its guess by making use of the topological structure of rooms \(\theta_{B}(h_{t},G)\).

And B encodes the history sentences with another GRU and computes the similarity between this encoded history \(e_{m}\), with each node representation \(e_{s}^{i}\) calculated above to find the most possible position of agent A. Similarly, the model for the guidance task also includes an encoder, a language generation, and a language understanding module used for giving instructions to agent A about its next step.

Finally, we put those sub-tasks together to complete the navigation task. In the training process, the input of the localization part is the surroundings of agent A, and the output of it is the guessing position made by agent B. However, the input of the route plan part in the next timestep is the actual position of A, instead of the guessing one, preventing the error from localization influencing other modules. While in the testing process, the input of the route plan module comes from the last localization module. We construct the dataset with 6 rooms for each navigation task. And every object in this dataset has 3 optional colors and 3 optional shapes.

In this work, we propose a navigation task in an emergent communication scenario. This task is adapted from the vision-language navigation task proposed in human-machine communication.

Touchdown: Natural language navigation and spatial reasoning in visual street environments. 2018. doi: 10.48550/ARXIV.1811.12354. URL [https://arxiv.org/abs/1811.12354](https://arxiv.org/abs/1811.12354).

Just ask:an interactive learning framework for vision and language navigation, 2019. URL [https://arxiv.org/abs/1912.00915](https://arxiv.org/abs/1912.00915).

Talk the walk: Navigating new york city through grounded dialogue, 2018. URL [https://arxiv.org/abs/1807.03367](https://arxiv.org/abs/1807.03367).

Vision-and-language navigation: A survey of tasks, methods, and future directions. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, 2022. doi: 10.18653/v1/2022.acl-long.524. URL [https://doi.org/10.18653%2Fv1%2F2022.acl-long.524](https://doi.org/10.18653%2Fv1%2F2022.acl-long.524).

Beyond the nav-graph: Vision-and-language navigation in continuous environments, 2020. URL [https://arxiv.org/abs/2004.02857](https://arxiv.org/abs/2004.02857).

* Lazaridou et al. (2018) Angeliki Lazaridou, Karl Moritz Hermann, Karl Tuyls, and Stephen Clark. Emergence of linguistic communication from referential games with symbolic and pixel input, 2018. URL [https://arxiv.org/abs/1804.03984](https://arxiv.org/abs/1804.03984). * Narayan-Chen et al. (2019) Anjali Narayan-Chen, Prashant Jayannavar, and Julia Hockenmaier. Collaborative dialogue in Minecraft. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pp. 5405-5415, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1537. URL [https://aclanthology.org/P19-1537](https://aclanthology.org/P19-1537).

* Thomason et al. (2019) Jesse Thomason, Michael Murray, Maya Cakmak, and Luke Zettlemoyer. Vision-and-dialog navigation, 2019. URL [https://arxiv.org/abs/1907.04957](https://arxiv.org/abs/1907.04957).

Input: Figure 1: The navigation task involves three components: the tourist (agent A), the guide (agent B), and the environment. Agent A and B have unequal abilities. While agent A can only see its surroundings and describe them to B, agent B has the global perspective, can make a route plan, and give instructions to A. Agent A and B cooperate to lead A to the target place.

Input: Figure 2: Navigation Case. The navigation task consists of three sub-tasks: Localization, Route Plan, and Guidance. At every turn, agent A first gives a description of its surroundings in the room it is located. With the current messages and the history messages given by A, agent B tries to guess where A is using the topological structure of the rooms. Then, based on this guess, agent B makes a route plan for A to get to the target place, and gives an instruction to A describing the next step A should take.

Input: Figure 3: Model Structure. There are three sub-modules here, including localization, route plan, and guidance. In the localization part, by calculating the similarity score between messages given by A and room feature embeddings incorporating structural information, agent B tries to find where A is. Then we use the Floyd algorithm to find the shortest path between the guessed place and the target one. And in the guidance module, A could find its path with history dialogue with B and its own observations.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, this appears to be a moderately overstated claim. While the authors do propose a navigation task involving agent communication, several key points make the claim overstated:

1. As noted in the review comments, similar work already exists (Talk the Walk) with real images and natural language, making the "new task" claim less novel.

2. The evidence shows significant limitations - the environment is simplified (6 rooms, limited object attributes) and uses artificial messages rather than natural language.

3. The review rightly points out that similar multi-agent communication frameworks already exist, weakening the novelty claim.

4. The task is more accurately described as an adaptation/modification of existing VLN frameworks rather than a wholly new task.

<score>4</score>

Justification:

1. The review comments highlight pre-existing similar work, particularly [1] and [2], indicating less novelty than claimed.

2. Evidence from methodology section shows significant simplifications: "We construct the dataset with 6 rooms... every object in this dataset has 3 optional colors and 3 optional shapes."

3. The authors themselves acknowledge similarity to existing work: "talk-the-walk... is very similar to our task settings."

4. The artificial nature of the communication (versus natural language in prior work) represents a limitation rather than advancement.

The claim would be better stated by acknowledging these limitations and positioning it as an adaptation or simplified version of existing VLN tasks for studying specific aspects of agent communication.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Besides, we suppose that the guide does not know the initial position of the tourist, so the guide not only has to give guidance to the tourist but also confirm the location of it.

The evidence to be evaluated is:Generally, this task involves humans, an artificial agent, and the environment.

The agent may receive requests or instructions and ask for more guidance from the human. It can navigate and interact with the environment to get more information or complete corresponding task requested by the human.

Generally, the goal of these VLN tasks is to help the agent find the target place known only by the human side.

However, this kind of work only gives the instructions at the beginning, so multi-turn communication can not be generated.

* Oracle guidance: In the process of navigation, the agent can request additional natural language guidance to get more information Chi et al. (2019). But in most cases, the oracle can only respond with simple words like "turn left" and "turn right".

However, talk-the-walk only focuses on the localization part, where the guide finds where the tourist is. But ignore the guidance part, in which the guide makes a route plan and gives guidance to the tourist, and the tourist makes corresponding actions.

As pictured in Figure 1, agent A can only see the objects around it, while agent B has the floor plan and the target place but doesn't know where A is.

Agent B needs to find where A is, according to the message given by A describing the objects around it.

* In the localization task, agent A takes in the room's vector where it is and gives out the messages of step t consisting of \(M\) tokens \(m_{t}=\{v_{0},v_{1},...,v_{M}\},v_{i}\in V\). While agent B collects history dialogue \(h_{t}=(m_{0},m_{1},..,m_{t})\) and tries to guess where A is.

* The only difference between guidance and localization task is that the guidance task switches the role of agents A and B, and agent A does not require the graph \(G\).

With the message coming from agent A in this turn, and the history sentences from A, agent B makes its guess by making use of the topological structure of rooms \(\theta_{B}(h_{t},G)\).

And B encodes the history sentences with another GRU and computes the similarity between this encoded history \(e_{m}\), with each node representation \(e_{s}^{i}\) calculated above to find the most possible position of agent A. Similarly, the model for the guidance task also includes an encoder, a language generation, and a language understanding module used for giving instructions to agent A about its next step.

In the training process, the input of the localization part is the surroundings of agent A, and the output of it is the guessing position made by agent B. However, the input of the route plan part in the next timestep is the actual position of A, instead of the guessing one, preventing the error from localization influencing other modules.

Input: Figure 1: The navigation task involves three components: the tourist (agent A), the guide (agent B), and the environment. Agent A and B have unequal abilities. While agent A can only see its surroundings and describe them to B, agent B has the global perspective, can make a route plan, and give instructions to A. Agent A and B cooperate to lead A to the target place.

Input: Figure 3: Model Structure. There are three sub-modules here, including localization, route plan, and guidance. In the localization part, by calculating the similarity score between messages given by A and room feature embeddings incorporating structural information, agent B tries to find where A is. Then we use the Floyd algorithm to find the shortest path between the guessed place and the target one. And in the guidance module, A could find its path with history dialogue with B and its own observations.

Input: Figure 2: Navigation Case. The navigation task consists of three sub-tasks: Localization, Route Plan, and Guidance. At every turn, agent A first gives a description of its surroundings in the room it is located. With the current messages and the history messages given by A, agent B tries to guess where A is using the topological structure of the rooms. Then, based on this guess, agent B makes a route plan for A to get to the target place, and gives an instruction to A describing the next step A should take.

Input: Figure 6: The effect of (a) sentence length and (b) the shortest route length between the start and target places on the task performance

Input: Figure 4: The change of accuracy over epoch in different tasks. (a) localization task. (b) navigation task

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate the claim: "Besides, we suppose that the guide does not know the initial position of the tourist, so the guide not only has to give guidance to the tourist but also confirm the location of it."

This claim is well-stated and accurately reflects the evidence provided. The evidence strongly supports this through:

1. Figure 2 explicitly shows the localization task where Agent B must first guess Agent A's location before planning a route
2. The text explicitly states "agent B has the floor plan and the target place but doesn't know where A is"
3. The model structure in Figure 3 shows dedicated localization and guidance modules
4. The accuracy graphs in Figure 4 show separate tracking of localization and navigation performance

The claim makes no exaggerated assertions and directly aligns with both the architectural design and implementation details provided in the evidence.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Besides, we suppose that the guide does not know the initial position of the tourist, so the guide not only has to give guidance to the tourist but also confirm the location of it.

The review comment to be evaluated is:

Strengths:
* The proposed setting is novel.

Weaknesses:
* While novel, the proposed setting is poorly motivated. Why studying this setting is significant? What would it enable?
* The description of the environment is very nebulous. Especially, the complexity of the visual and language components are highly under-specified. How is the input observation represented (image or feature-based)? How many object categories? How large is the agent's vocabulary? How are the environment graph generated? It is thus difficult to comprehend the significance of solving this environment.
* The plots are missing confidence intervals. Each experiments are not conducted multiple times with different random seeds. This seriously weakens the empirical claims.

The evidence to be evaluated is:Generally, this task involves humans, an artificial agent, and the environment.

The agent may receive requests or instructions and ask for more guidance from the human. It can navigate and interact with the environment to get more information or complete corresponding task requested by the human.

Generally, the goal of these VLN tasks is to help the agent find the target place known only by the human side.

However, this kind of work only gives the instructions at the beginning, so multi-turn communication can not be generated.

* Oracle guidance: In the process of navigation, the agent can request additional natural language guidance to get more information Chi et al. (2019). But in most cases, the oracle can only respond with simple words like "turn left" and "turn right".

However, talk-the-walk only focuses on the localization part, where the guide finds where the tourist is. But ignore the guidance part, in which the guide makes a route plan and gives guidance to the tourist, and the tourist makes corresponding actions.

As pictured in Figure 1, agent A can only see the objects around it, while agent B has the floor plan and the target place but doesn't know where A is.

Agent B needs to find where A is, according to the message given by A describing the objects around it.

* In the localization task, agent A takes in the room's vector where it is and gives out the messages of step t consisting of \(M\) tokens \(m_{t}=\{v_{0},v_{1},...,v_{M}\},v_{i}\in V\). While agent B collects history dialogue \(h_{t}=(m_{0},m_{1},..,m_{t})\) and tries to guess where A is.

* The only difference between guidance and localization task is that the guidance task switches the role of agents A and B, and agent A does not require the graph \(G\).

With the message coming from agent A in this turn, and the history sentences from A, agent B makes its guess by making use of the topological structure of rooms \(\theta_{B}(h_{t},G)\).

And B encodes the history sentences with another GRU and computes the similarity between this encoded history \(e_{m}\), with each node representation \(e_{s}^{i}\) calculated above to find the most possible position of agent A. Similarly, the model for the guidance task also includes an encoder, a language generation, and a language understanding module used for giving instructions to agent A about its next step.

In the training process, the input of the localization part is the surroundings of agent A, and the output of it is the guessing position made by agent B. However, the input of the route plan part in the next timestep is the actual position of A, instead of the guessing one, preventing the error from localization influencing other modules.

Input: Figure 1: The navigation task involves three components: the tourist (agent A), the guide (agent B), and the environment. Agent A and B have unequal abilities. While agent A can only see its surroundings and describe them to B, agent B has the global perspective, can make a route plan, and give instructions to A. Agent A and B cooperate to lead A to the target place.

Input: Figure 3: Model Structure. There are three sub-modules here, including localization, route plan, and guidance. In the localization part, by calculating the similarity score between messages given by A and room feature embeddings incorporating structural information, agent B tries to find where A is. Then we use the Floyd algorithm to find the shortest path between the guessed place and the target one. And in the guidance module, A could find its path with history dialogue with B and its own observations.

Input: Figure 2: Navigation Case. The navigation task consists of three sub-tasks: Localization, Route Plan, and Guidance. At every turn, agent A first gives a description of its surroundings in the room it is located. With the current messages and the history messages given by A, agent B tries to guess where A is using the topological structure of the rooms. Then, based on this guess, agent B makes a route plan for A to get to the target place, and gives an instruction to A describing the next step A should take.

Input: Figure 6: The effect of (a) sentence length and (b) the shortest route length between the start and target places on the task performance

Input: Figure 4: The change of accuracy over epoch in different tasks. (a) localization task. (b) navigation task

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim that "the guide does not know the initial position of the tourist, so the guide not only has to give guidance but also confirm the location" appears to be well-stated and accurately reflects the evidence presented. The evidence clearly shows:

1. The model architecture (Figure 3) explicitly includes a localization module
2. The description explains that Agent B must guess Agent A's location based on Agent A's descriptions
3. Performance metrics (Figure 4) specifically track localization accuracy
4. The reviewer comments, while critical of other aspects, don't dispute this fundamental setup

The claim stays within the bounds of what's demonstrated and doesn't overstate the functionality or performance.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Besides, we suppose that the guide does not know the initial position of the tourist, so the guide not only has to give guidance to the tourist but also confirm the location of it.

The review comment to be evaluated is:

While this paper goes in a good direction by moving away from the Lewis game in the language emergence communication setting, it falls short as there is little scientific analysis. The authors mainly designed an artificial navigation task, handcrafted an observation space and reward, designed a neural network, trained it, and reported the results. Those results are not put into perspective with a detailed researched perspective. Indeed, the point of Emecom in ML is not to create artificial tasks for the sake of solving it, but to design tasks that could help understand how language evolves with a human intervention as possible (or being generic).

I am listing here a few leads to improve the paper in the long run:
 - The introduction should frame the research problem into the past literature, ask a scientific question, and explain how the paper help answers it. Here, there is no citation; the VLM task is only said to be more natural (no reference to the embodiment, situated learning, or going beyond the signaling game)
 - The related work is extremely short. 18 citations in total for a paper about language emergence, and the navigation task is just inherently incomplete. On a positive note, the most relevant citations kajic/Kanilowska/DeVries are present
- The paper is 7 pages long, and page 3 is barely empty
- The model is designed to solve the task, e.g., the graph networks are specific to the observation space. I would recommend either picking a minimal architecture or reproducing the architecture that is classic in VLM.
- having 4 losses in an RL task is generally a bad sign as it is a symptom of human intervention. cf The bitter lesson by Sutton
 - Results are not compared to any baselines (VAE vs your network is not really a baseline...), and simply reporting the success rate is not sufficient. Examples of suitable ablation include: the correlation between language and environment complexity. Qualitative experiments to translate language and change in the environment (e.g., how changing the graph impacts the language), what are the external constraints that shape language, reward ablation, etc.

I would recommend the authors to re-work their paper and present it to a workshop to have further guidance. The initial idea is good, but the realization lacks scientific maturity.

The evidence to be evaluated is:Generally, this task involves humans, an artificial agent, and the environment.

The agent may receive requests or instructions and ask for more guidance from the human. It can navigate and interact with the environment to get more information or complete corresponding task requested by the human.

Generally, the goal of these VLN tasks is to help the agent find the target place known only by the human side.

However, this kind of work only gives the instructions at the beginning, so multi-turn communication can not be generated.

* Oracle guidance: In the process of navigation, the agent can request additional natural language guidance to get more information Chi et al. (2019). But in most cases, the oracle can only respond with simple words like "turn left" and "turn right".

However, talk-the-walk only focuses on the localization part, where the guide finds where the tourist is. But ignore the guidance part, in which the guide makes a route plan and gives guidance to the tourist, and the tourist makes corresponding actions.

As pictured in Figure 1, agent A can only see the objects around it, while agent B has the floor plan and the target place but doesn't know where A is.

Agent B needs to find where A is, according to the message given by A describing the objects around it.

* In the localization task, agent A takes in the room's vector where it is and gives out the messages of step t consisting of \(M\) tokens \(m_{t}=\{v_{0},v_{1},...,v_{M}\},v_{i}\in V\). While agent B collects history dialogue \(h_{t}=(m_{0},m_{1},..,m_{t})\) and tries to guess where A is.

* The only difference between guidance and localization task is that the guidance task switches the role of agents A and B, and agent A does not require the graph \(G\).

With the message coming from agent A in this turn, and the history sentences from A, agent B makes its guess by making use of the topological structure of rooms \(\theta_{B}(h_{t},G)\).

And B encodes the history sentences with another GRU and computes the similarity between this encoded history \(e_{m}\), with each node representation \(e_{s}^{i}\) calculated above to find the most possible position of agent A. Similarly, the model for the guidance task also includes an encoder, a language generation, and a language understanding module used for giving instructions to agent A about its next step.

In the training process, the input of the localization part is the surroundings of agent A, and the output of it is the guessing position made by agent B. However, the input of the route plan part in the next timestep is the actual position of A, instead of the guessing one, preventing the error from localization influencing other modules.

Input: Figure 1: The navigation task involves three components: the tourist (agent A), the guide (agent B), and the environment. Agent A and B have unequal abilities. While agent A can only see its surroundings and describe them to B, agent B has the global perspective, can make a route plan, and give instructions to A. Agent A and B cooperate to lead A to the target place.

Input: Figure 3: Model Structure. There are three sub-modules here, including localization, route plan, and guidance. In the localization part, by calculating the similarity score between messages given by A and room feature embeddings incorporating structural information, agent B tries to find where A is. Then we use the Floyd algorithm to find the shortest path between the guessed place and the target one. And in the guidance module, A could find its path with history dialogue with B and its own observations.

Input: Figure 2: Navigation Case. The navigation task consists of three sub-tasks: Localization, Route Plan, and Guidance. At every turn, agent A first gives a description of its surroundings in the room it is located. With the current messages and the history messages given by A, agent B tries to guess where A is using the topological structure of the rooms. Then, based on this guess, agent B makes a route plan for A to get to the target place, and gives an instruction to A describing the next step A should take.

Input: Figure 6: The effect of (a) sentence length and (b) the shortest route length between the start and target places on the task performance

Input: Figure 4: The change of accuracy over epoch in different tasks. (a) localization task. (b) navigation task

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find the claim is moderately well-stated with some minor overstatement. While the evidence clearly shows that the guide (Agent B) does need to localize the tourist (Agent A) through communication before providing guidance, the phrasing "has to" is slightly too strong given that:

1. The evidence shows this is one component of their designed system, not necessarily a requirement for all guide-tourist interactions
2. The localization module runs in parallel with other components rather than strictly preceding guidance
3. The review points out that this is an artificial task with handcrafted observations/rewards

The claim generally aligns with the model structure and task design shown in the figures, but could be more precisely worded.

<score>2</score>