Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Given this wide spectrum of endpoints, a recombination module takes care of addressing consistency in the scene among agents.

The review comment to be evaluated is:

### Strong Points

The idea of performing re-estimation based on the computed heat map end-points
seems novel to me. This idea seems like a natural next step to improve multi agent motion
prediction. The fact that you first get the end-point and then reason about the scene is
intuitively sounding and seems to be a good solution for the problem.

The experiments on interact are a nice direction to be taken by the field.

The visualizations of the heat-maps and the different predictions was helpful.

### Weak Points

#### I could not find a clear distinction with literature
While reading the related work section I found hard to contrast the ideas of the paper with the ones proposed on the literature.
The difference between this method and GOHOME was  not clear to me in this sense.
From what I understood the main contribution here  is the modality combination based on the collision
free end-point sampling. However, this was not obvious for me when reading the paper.

#### The implemented baselines seemed to have performed way worse than expected.
For example, both [1] and [7] are methods designed for *joint* prediction and scene level prediction. This is contradicted
by the observations made by the authors on page 8, quote: " ... This problem is aggravated in the joint training case, since the modality selected is
the same for all agents in a training sample....". Both [1] and [3] fairly confidently claim producing multimodal joint predictions.
Either the version implemented by the authors does not correctly match the other works in the literature or there are specific findings
on the limitations existent in other methods.
Also, the paper lack comparison with latent variable based [2][5][[6] multi modal prediction methods. They seem to perform
better when the issue is diverse prediction. Definitely the mode does not tend to be the same for all predictions in those cases.

#### A single dataset evalution might not be enough for the community to access the value of this paper.

 Motion prediction is currently a very competitive scene. Even though I appreciate the idea on focusing reporting results on multi-agents
and joint prediction tasks, it is important to also evaluate the prediction on more standard benchmarks such as argoverse.
With results on argoverse, me and the rest of the community can more quickly access the the capabilities of the proposed
 heat map prediction pipeline, on producing consistent ego-agent motion prediction. I understand that this would not be the main
results, but it would be helpful.

##### Computational cost is apparently not shown.

I could not find the computational cost of running and training the proposed method on the paper. It seems that even
with the optimizations made on the heat map estimation the training time and inference time should be high.
Those kinds of motion prediction models have the necessity to further scale with way more data. Thus, highly
complex methods can be hard to train. There is also the case of potential future execution in an embedded hardware.

### Specific Questions

I am not  sure about the scene modality vectors ( Section 3.2.2), are they like seed parameters which are going to be used
to incorporate the joint consistent scene level ? I couldn't really find any sort of detailed explanation on that.

Why the lack of ADE for scene level metrics or using similar metrics as [3] ?

[1] Jiquan Ngiam, Benjamin Caine, Vijay Vasudevan, Zhengdong Zhang, Hao-Tien Lewis Chiang,
Jeffrey Ling, Rebecca Roelofs, Alex Bewley, Chenxi Liu, Ashish Venugopal, et al. Scene
transformer: A unified multi-task model for behavior prediction and planning. arXiv preprint
arXiv:2106.08417, 2021
[2] Tang, Charlie, and Russ R. Salakhutdinov. "Multiple futures prediction." Advances in Neural Information Processing Systems 32 (2019): 15424-15434.
[3] Li, Lingyun Luke, et al. "End-to-end contextual perception and prediction with interaction transformer." 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2020.
[4] Yuan, Ye, et al. "AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting." arXiv preprint arXiv:2103.14023 (2021).
[5] Girgis, Roger, et al. "Autobots: Latent Variable Sequential Set Transformers." arXiv preprint arXiv:2104.00563 (2021).
[6] Casas, Sergio, et al. "Implicit latent variable model for scene-consistent motion forecasting." Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28,
[7] Ming Liang, Bin Yang, Rui Hu, Yun Chen, Renjie Liao, Song Feng, and Raquel Urtasun. Learning
lane graph representations for motion forecasting. In ECCV, 2020

The evidence to be evaluated is:LLVM (Casas et al., 2020) uses scene latent representations conditioned on all agents to generate scene-consistent samples, but its variational inference does not provide a confidence score for each modality, hence LookOut (Cui et al., 2021) proposes a scenario scoring function and a diverse sampler to improve sample efficiency.

Finally, none of these previous methods considers the problems of scene consistency for multi-agent prediction.

Finally, we recombine the sampled endpoints into scene-consistent modalities across agents and build the full trajectories for each agent.

Furthermore, we add a novel scene-consistency module that recombines the marginal outputs into a joint prediction.

Having a heatmap as output grants us the good property of letting us decide to privilege coverage in order to give a wide spectrum of candidates to the scene recombination process.

The difficulty in multimodal multi-agent prediction resides in having coherent modalities between each agent. Since the modalities are considered scene-wise, each agent's most probable modality has to match with the most probable prediction of the other agents, and so on. Moreover, these modalities must not collide with each other, as they should represent realistic scenarios.

To achieve this, our main hypothesis lays in the fact that good trajectory proposals are already present in the marginal predictions, but they need to be coherently aligned among agents to achieve a set of overall consistent scene predictions.

Each scene modality \(l\) would select a different association between the agents.

Since argmax is non-differentiable, we employ a soft argmax as a weighted linear combination of the agent modalities \(p_{k}^{a}\) using a softmax on the \(c_{l}^{k,a}\) scores: \[p_{l}^{a}=\sum_{k}softmax(c_{l}^{k,a})p_{k}^{a}\] The recombination module is trained with the scene-level minimum displacement error, where for each modality inside a single scene all predicted agent displacement losses are averaged before taking the minimum (winner-takes-all loss) over the \(L\) scene modalities.

For consistent scene multi-agent prediction, we report the joint scene-level metrics, where the average operation over the agents is done before the minimum operator. In this formulation, the minimum is taken over scene modalities, meaning that only the best scene (joint over agents) modality is taken into account:

In other words, the marginal metrics pick their optimal solutions in a pool of \(k\) to the power of \(a\) predicted solutions, while the joint metrics restrict this pool to only \(k\) possibilities, making it a much more complex problem.

**ILVM (Casas et al., 2020)** uses variational inference to learn a latent representation of the scene conditioned of each agent with a Scene Interaction Module, and decodes it with a similar Scene Interaction Module.

Even though the sampling is independent for each agent, the latent space is generated conditionally on all the agents. **SceneTransformer (Ngiam et al., 2021)** duplicates the agent encoding with the number of modalities required and adds a one-hot encoding specific to each modality. They then apply a shared transformer architecture on all modalities to encode intra-modality interactions between agents and generate the final trajectories.

Compared to these baselines, THOMAS can be seen as an hybrid sampling method that takes the result of deterministic sampling as input and learns to recombine it into a more coherent solution. With regard to the joint algorithmic sampling that only tackled collisions but has little to no effect on actual consistency, as highlighted by the big drop in collision rate from \(7.2\%\) to \(2.6\%\) but a similar joint SMR, THOMAS actually brings a lot of consistency in the multi-agent prediction and drops the joint SMR from \(14.8\%\) to \(11.8\%\).

While the \(Before\) model tries to spread the modalities for all agents to minimize marginal miss-rate, the recombined model presents much less spread compared to the original model, maintaining a multimodal behavior only in presence of very different possible agent intentions such as different possible exits or turn choices.

We have presented THOMAS, a recombination module that can be added after any trajectory prediction module outputing multi-modal predictions. By design, THOMAS allows to generate scene-consistent modalities across all agents by making the scene modalities select coherent agent modalities and restricting the modality budget on the agents that truly need it. We show significant performance increase when adding the THOMAS module compared to the vanilla model and achieve state-of-the-art results compared to already existing methods tackling scene-consistent predictions.

We select for each scene a reference agent on which the scene will be centered and oriented according to its heading at prediction time. At training time, this reference agent is chosen randomly across all agents.

We designed a deterministic sampling algorithm based on the heatmaps generated in previous section in order to sample endpoints for each agent in a collision aware manner.

We then not only set to zero the heatmap values of this agent heatmap \(\mathcal{I}^{a^{\prime}}_{k^{\prime}}\) around the sampled location so not to sample it in the next modalities \(k^{\prime}\), but we also set to zero the same area on the heatmaps \(\mathcal{I}^{a^{\prime}}_{k}\) of the other agents \(a^{\prime}\) on the same modality \(k\), so that these other agents cannot be sampled at the same position for this modality. This way, we try to enforce collision-free endpoints, and expect that considering collisions brings logic to improve the overall consistency of the predictions.

It is to be noted that the DenseTNT solution explicitely checks for collisions in the search for its proposed predictions, which we don't, hence their \(0\%\) collision rate (SCR) and its direct impact on consistent collision-free joint Miss Rate (cSMR).

Input: Figure 1: Illustration of the THOMAS multi-agent prediction pipeline

Input: Figure 9: Qualitative examples of heatmap output from our multi-agent model. All the heatmaps from one scene are the results from one single forward pass in our model predicting all agents at once. We use matching colors for the agent history, current location and their future predicted heatmap (best viewed in color).

Input: Figure 5: Qualitative examples of recombination model assembling collision-free modalities together compared to initial colliding modalities. For each example we display the general context with highlighted agents and area of interest, then two zooms in on the agents, one displaying the initial best modality before recombination in dashed orange and all the other available modalities in grey. The second zooms shows the best modality after recombination in full line orange.

Input: Figure 6: Qualitative examples of recombination model selecting fewer but more pronounced and impactful agent modalities compared to initial colliding modalities. For each example we display on the left the vanilla model modalities with dashed lines, with the initial best modality in dashed orange and all the other available modalities in grey. On the right we display the selected modalities after recombination, where the model focuses on the most likely occurrence in most agents.

Input: Figure 4: Illustration of THOMAS methods for generation of scene-consistent agent modalities

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, I find the claim "Given this wide spectrum of endpoints, a recombination module takes care of addressing consistency in the scene among agents" to be fairly well-stated. The evidence demonstrates that THOMAS implements a recombination module that specifically addresses scene consistency by selecting coherent agent modalities and restricting modality budgets. The figures and technical details show how the module recombines endpoints to create collision-free, consistent predictions across agents. The reviewer's comments also support this, noting the idea of re-estimation based on heat map endpoints seems novel and intuitive. While slightly simplified, the claim accurately represents the core functionality without major exaggeration.

<score>2</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Given this wide spectrum of endpoints, a recombination module takes care of addressing consistency in the scene among agents.

The review comment to be evaluated is:

Strengths:
1. The paper is well written and easy to follow.
2. Good performance was achieved on the Interaction multi-agent prediction challenge.

Weaknesses:
1. The method largely follows previous works, with little technical contributions. And even for these limited technical novelties, many of them are not well supported with experimemts.
In the end of introduction section, the authors summarize three main contributions:

i) an efficient graph-based model. However, the implementation mainly follows LaneGCN and GOHOME.

ii) a collision-aware endpoint sampling. The algorithm mainly follows GOHOME, with a small modification in that when we sample an endpoint for actor a modality k, we not only mask out the same region for other modalities of actor a, but also mask out for other actors of modality k. However, this modification doesn't make sense to me, as we still have a scene-consistent modality recombination module afterwards, therefore the second mask-out won't make much difference here. I also don't see an ablation study for this specific change (ie, do not mask-out for other actors of modality k, which means exactly follows the original algorithm in GOHOME). I want to see this ablation for the last three rows in Table 3.

iii) a novel recombination model that could obtain scene-consistent trajectories across the agents. To me this is the core novelty of the paper. From the last two rows in Table 3 we can see that with the proposed recombination module joint metrics improve a lot. However, the marginal metrics degrade a lot (e.g., miss rate more than doubled). The only explanation is that with the proposed module, the same modality (also the sub-optimal one) of an actor is chosen for multiple times in scene-level multi-modalities. To me this may not be what we want: scene-level consistency, but sacrifices actor-level precision a lot. Further analysis is needed here to convince me that this trade-off is worthwhile.

2. Another technical novelty that's not mentioned is the hierarchical iterative refinement of endpoints from the heatmap. However, no ablation study is done on this. I'd like to see a speed-performance comparison with original endpoint estimation model (as in GOHOME).

3. Also, the overall runtime efficiency of the method is not given.

4. GOHOME is the closest approach but I don't see it as a baseline in the experiments.

The evidence to be evaluated is:LLVM (Casas et al., 2020) uses scene latent representations conditioned on all agents to generate scene-consistent samples, but its variational inference does not provide a confidence score for each modality, hence LookOut (Cui et al., 2021) proposes a scenario scoring function and a diverse sampler to improve sample efficiency.

Finally, none of these previous methods considers the problems of scene consistency for multi-agent prediction.

Finally, we recombine the sampled endpoints into scene-consistent modalities across agents and build the full trajectories for each agent.

Furthermore, we add a novel scene-consistency module that recombines the marginal outputs into a joint prediction.

Having a heatmap as output grants us the good property of letting us decide to privilege coverage in order to give a wide spectrum of candidates to the scene recombination process.

The difficulty in multimodal multi-agent prediction resides in having coherent modalities between each agent. Since the modalities are considered scene-wise, each agent's most probable modality has to match with the most probable prediction of the other agents, and so on. Moreover, these modalities must not collide with each other, as they should represent realistic scenarios.

To achieve this, our main hypothesis lays in the fact that good trajectory proposals are already present in the marginal predictions, but they need to be coherently aligned among agents to achieve a set of overall consistent scene predictions.

Each scene modality \(l\) would select a different association between the agents.

Since argmax is non-differentiable, we employ a soft argmax as a weighted linear combination of the agent modalities \(p_{k}^{a}\) using a softmax on the \(c_{l}^{k,a}\) scores: \[p_{l}^{a}=\sum_{k}softmax(c_{l}^{k,a})p_{k}^{a}\] The recombination module is trained with the scene-level minimum displacement error, where for each modality inside a single scene all predicted agent displacement losses are averaged before taking the minimum (winner-takes-all loss) over the \(L\) scene modalities.

For consistent scene multi-agent prediction, we report the joint scene-level metrics, where the average operation over the agents is done before the minimum operator. In this formulation, the minimum is taken over scene modalities, meaning that only the best scene (joint over agents) modality is taken into account:

In other words, the marginal metrics pick their optimal solutions in a pool of \(k\) to the power of \(a\) predicted solutions, while the joint metrics restrict this pool to only \(k\) possibilities, making it a much more complex problem.

**ILVM (Casas et al., 2020)** uses variational inference to learn a latent representation of the scene conditioned of each agent with a Scene Interaction Module, and decodes it with a similar Scene Interaction Module.

Even though the sampling is independent for each agent, the latent space is generated conditionally on all the agents. **SceneTransformer (Ngiam et al., 2021)** duplicates the agent encoding with the number of modalities required and adds a one-hot encoding specific to each modality. They then apply a shared transformer architecture on all modalities to encode intra-modality interactions between agents and generate the final trajectories.

Compared to these baselines, THOMAS can be seen as an hybrid sampling method that takes the result of deterministic sampling as input and learns to recombine it into a more coherent solution. With regard to the joint algorithmic sampling that only tackled collisions but has little to no effect on actual consistency, as highlighted by the big drop in collision rate from \(7.2\%\) to \(2.6\%\) but a similar joint SMR, THOMAS actually brings a lot of consistency in the multi-agent prediction and drops the joint SMR from \(14.8\%\) to \(11.8\%\).

While the \(Before\) model tries to spread the modalities for all agents to minimize marginal miss-rate, the recombined model presents much less spread compared to the original model, maintaining a multimodal behavior only in presence of very different possible agent intentions such as different possible exits or turn choices.

We have presented THOMAS, a recombination module that can be added after any trajectory prediction module outputing multi-modal predictions. By design, THOMAS allows to generate scene-consistent modalities across all agents by making the scene modalities select coherent agent modalities and restricting the modality budget on the agents that truly need it. We show significant performance increase when adding the THOMAS module compared to the vanilla model and achieve state-of-the-art results compared to already existing methods tackling scene-consistent predictions.

We select for each scene a reference agent on which the scene will be centered and oriented according to its heading at prediction time. At training time, this reference agent is chosen randomly across all agents.

We designed a deterministic sampling algorithm based on the heatmaps generated in previous section in order to sample endpoints for each agent in a collision aware manner.

We then not only set to zero the heatmap values of this agent heatmap \(\mathcal{I}^{a^{\prime}}_{k^{\prime}}\) around the sampled location so not to sample it in the next modalities \(k^{\prime}\), but we also set to zero the same area on the heatmaps \(\mathcal{I}^{a^{\prime}}_{k}\) of the other agents \(a^{\prime}\) on the same modality \(k\), so that these other agents cannot be sampled at the same position for this modality. This way, we try to enforce collision-free endpoints, and expect that considering collisions brings logic to improve the overall consistency of the predictions.

It is to be noted that the DenseTNT solution explicitely checks for collisions in the search for its proposed predictions, which we don't, hence their \(0\%\) collision rate (SCR) and its direct impact on consistent collision-free joint Miss Rate (cSMR).

Input: Figure 1: Illustration of the THOMAS multi-agent prediction pipeline

Input: Figure 9: Qualitative examples of heatmap output from our multi-agent model. All the heatmaps from one scene are the results from one single forward pass in our model predicting all agents at once. We use matching colors for the agent history, current location and their future predicted heatmap (best viewed in color).

Input: Figure 5: Qualitative examples of recombination model assembling collision-free modalities together compared to initial colliding modalities. For each example we display the general context with highlighted agents and area of interest, then two zooms in on the agents, one displaying the initial best modality before recombination in dashed orange and all the other available modalities in grey. The second zooms shows the best modality after recombination in full line orange.

Input: Figure 6: Qualitative examples of recombination model selecting fewer but more pronounced and impactful agent modalities compared to initial colliding modalities. For each example we display on the left the vanilla model modalities with dashed lines, with the initial best modality in dashed orange and all the other available modalities in grey. On the right we display the selected modalities after recombination, where the model focuses on the most likely occurrence in most agents.

Input: Figure 4: Illustration of THOMAS methods for generation of scene-consistent agent modalities

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim "Given this wide spectrum of endpoints, a recombination module takes care of addressing consistency in the scene among agents" appears to be well-stated and accurately reflects what the evidence shows. The paper demonstrates through figures and detailed explanations how the recombination module works to ensure scene consistency across agents. The evidence shows that:

1. The module explicitly handles scene-level consistency through a novel recombination approach
2. Visual evidence in Figures 5 and 6 demonstrates the module successfully addressing consistency
3. Quantitative results show improvements in joint metrics and collision reduction
4. The reviewer's comments, while critical of some aspects, acknowledge this as the "core novelty"

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Given this wide spectrum of endpoints, a recombination module takes care of addressing consistency in the scene among agents.

The review comment to be evaluated is:

- The overall idea of iterative hierarchical heatmap refinement is interesting and appears to be novel. The experiments demonstrate the ideas validity, although there is definitely room for further strengthening.

- The related work is thorough, one quick comment here is to also reference the following work in the discussion of methods which can predict multiple agents at the same time (beginning of the last paragraph before Section 3): "MATS: An Interpretable Trajectory Forecasting Representation for Planning and Control" by B. Ivanovic, A. Elhafsi, G. Rosman, A. Gaidon, M. Pavone in CoRL 2020.

- Acronyms should be explained when they are first used. For instance, it might not be obvious what a UGRU is in Section 3.1.1. What does MR optimization mean in Section 3.1.3?

- Do not use contractions in scientific writing (e.g., change "don't" to "do not").

- A thorough round of editing is necessary, there seem to be writing errors that materially change the meaning of sentences. For instance, "The final output is therefore no strictly a re-ordering..." Should this have said "not strictly"? Same question for "which is JointMR where colliding modalities are also counted as misses even if they are closer than the defined threshold." Should this have just said "where colliding modalities are also counted as misses?" It is not very clear what these metrics mean here.

- The novelty with respect to the original GOHOME work needs to be made clearer, there seem to be many reused components and it is difficult to tell which parts are new in this work.

- Related to the above point, it is a bit difficult to understand this work's core contributions in Sections 3.2.1 and 3.2.2, also because the writing makes this difficult (e.g., a spurious use of "However" makes it difficult to determine if a statement is being negated or not).

- There are some key insights missing from the experimental section, which will be expanded on below.

- Ultimately, the method is only evaluated on one dataset, and it is difficult to determine if its performance would generalize to other datasets (for instance, what about nuScenes, Lyft, Waymo Open, Argoverse, etc?).

The evidence to be evaluated is:LLVM (Casas et al., 2020) uses scene latent representations conditioned on all agents to generate scene-consistent samples, but its variational inference does not provide a confidence score for each modality, hence LookOut (Cui et al., 2021) proposes a scenario scoring function and a diverse sampler to improve sample efficiency.

Finally, none of these previous methods considers the problems of scene consistency for multi-agent prediction.

Finally, we recombine the sampled endpoints into scene-consistent modalities across agents and build the full trajectories for each agent.

Furthermore, we add a novel scene-consistency module that recombines the marginal outputs into a joint prediction.

Having a heatmap as output grants us the good property of letting us decide to privilege coverage in order to give a wide spectrum of candidates to the scene recombination process.

The difficulty in multimodal multi-agent prediction resides in having coherent modalities between each agent. Since the modalities are considered scene-wise, each agent's most probable modality has to match with the most probable prediction of the other agents, and so on. Moreover, these modalities must not collide with each other, as they should represent realistic scenarios.

To achieve this, our main hypothesis lays in the fact that good trajectory proposals are already present in the marginal predictions, but they need to be coherently aligned among agents to achieve a set of overall consistent scene predictions.

Each scene modality \(l\) would select a different association between the agents.

Since argmax is non-differentiable, we employ a soft argmax as a weighted linear combination of the agent modalities \(p_{k}^{a}\) using a softmax on the \(c_{l}^{k,a}\) scores: \[p_{l}^{a}=\sum_{k}softmax(c_{l}^{k,a})p_{k}^{a}\] The recombination module is trained with the scene-level minimum displacement error, where for each modality inside a single scene all predicted agent displacement losses are averaged before taking the minimum (winner-takes-all loss) over the \(L\) scene modalities.

For consistent scene multi-agent prediction, we report the joint scene-level metrics, where the average operation over the agents is done before the minimum operator. In this formulation, the minimum is taken over scene modalities, meaning that only the best scene (joint over agents) modality is taken into account:

In other words, the marginal metrics pick their optimal solutions in a pool of \(k\) to the power of \(a\) predicted solutions, while the joint metrics restrict this pool to only \(k\) possibilities, making it a much more complex problem.

**ILVM (Casas et al., 2020)** uses variational inference to learn a latent representation of the scene conditioned of each agent with a Scene Interaction Module, and decodes it with a similar Scene Interaction Module.

Even though the sampling is independent for each agent, the latent space is generated conditionally on all the agents. **SceneTransformer (Ngiam et al., 2021)** duplicates the agent encoding with the number of modalities required and adds a one-hot encoding specific to each modality. They then apply a shared transformer architecture on all modalities to encode intra-modality interactions between agents and generate the final trajectories.

Compared to these baselines, THOMAS can be seen as an hybrid sampling method that takes the result of deterministic sampling as input and learns to recombine it into a more coherent solution. With regard to the joint algorithmic sampling that only tackled collisions but has little to no effect on actual consistency, as highlighted by the big drop in collision rate from \(7.2\%\) to \(2.6\%\) but a similar joint SMR, THOMAS actually brings a lot of consistency in the multi-agent prediction and drops the joint SMR from \(14.8\%\) to \(11.8\%\).

While the \(Before\) model tries to spread the modalities for all agents to minimize marginal miss-rate, the recombined model presents much less spread compared to the original model, maintaining a multimodal behavior only in presence of very different possible agent intentions such as different possible exits or turn choices.

We have presented THOMAS, a recombination module that can be added after any trajectory prediction module outputing multi-modal predictions. By design, THOMAS allows to generate scene-consistent modalities across all agents by making the scene modalities select coherent agent modalities and restricting the modality budget on the agents that truly need it. We show significant performance increase when adding the THOMAS module compared to the vanilla model and achieve state-of-the-art results compared to already existing methods tackling scene-consistent predictions.

We select for each scene a reference agent on which the scene will be centered and oriented according to its heading at prediction time. At training time, this reference agent is chosen randomly across all agents.

We designed a deterministic sampling algorithm based on the heatmaps generated in previous section in order to sample endpoints for each agent in a collision aware manner.

We then not only set to zero the heatmap values of this agent heatmap \(\mathcal{I}^{a^{\prime}}_{k^{\prime}}\) around the sampled location so not to sample it in the next modalities \(k^{\prime}\), but we also set to zero the same area on the heatmaps \(\mathcal{I}^{a^{\prime}}_{k}\) of the other agents \(a^{\prime}\) on the same modality \(k\), so that these other agents cannot be sampled at the same position for this modality. This way, we try to enforce collision-free endpoints, and expect that considering collisions brings logic to improve the overall consistency of the predictions.

It is to be noted that the DenseTNT solution explicitely checks for collisions in the search for its proposed predictions, which we don't, hence their \(0\%\) collision rate (SCR) and its direct impact on consistent collision-free joint Miss Rate (cSMR).

Input: Figure 1: Illustration of the THOMAS multi-agent prediction pipeline

Input: Figure 9: Qualitative examples of heatmap output from our multi-agent model. All the heatmaps from one scene are the results from one single forward pass in our model predicting all agents at once. We use matching colors for the agent history, current location and their future predicted heatmap (best viewed in color).

Input: Figure 5: Qualitative examples of recombination model assembling collision-free modalities together compared to initial colliding modalities. For each example we display the general context with highlighted agents and area of interest, then two zooms in on the agents, one displaying the initial best modality before recombination in dashed orange and all the other available modalities in grey. The second zooms shows the best modality after recombination in full line orange.

Input: Figure 6: Qualitative examples of recombination model selecting fewer but more pronounced and impactful agent modalities compared to initial colliding modalities. For each example we display on the left the vanilla model modalities with dashed lines, with the initial best modality in dashed orange and all the other available modalities in grey. On the right we display the selected modalities after recombination, where the model focuses on the most likely occurrence in most agents.

Input: Figure 4: Illustration of THOMAS methods for generation of scene-consistent agent modalities

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate the claim: "Given this wide spectrum of endpoints, a recombination module takes care of addressing consistency in the scene among agents."

The evidence strongly supports this claim through:
- Detailed explanation of the recombination module's functionality (Figs 1, 4)
- Demonstration of how it handles scene consistency (Figs 5, 6)
- Quantitative improvements in joint scene metrics (reduction in joint SMR from 14.8% to 11.8%)
- Clear visual examples showing how the module recombines modalities to achieve collision-free, consistent predictions
- Mathematical formulation of how the recombination works

The claim is neither understated nor overstated - it accurately describes the module's core function without exaggerating its capabilities.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We propose a learnable trajectory recombination model that takes as input a set of predicted trajectories for each agent and outputs its consistent reordered recombination.

The evidence to be evaluated is:Finally, we recombine the sampled endpoints into scene-consistent modalities across agents and build the full trajectories for each agent.

Furthermore, we add a novel scene-consistency module that recombines the marginal outputs into a joint prediction.

Initially, the prediction output of the model can be defined as marginal, as all \(A\) agents have been predicted and sampled independently in order to get \(K\) possible endpoint each. Our goal is to output a set of scene predictions \(\mathcal{J}=(L,A)\) from the marginal prediction \(\mathcal{M}=(A,K)\), where each scene modality \(l\) belonging to \(\mathcal{J}\) is an association of endpoints \(p_{l}^{a}\) for each agent \(a\). To achieve this, our main hypothesis lays in the fact that good trajectory proposals are already present in the marginal predictions, but they need to be coherently aligned among agents to achieve a set of overall consistent scene predictions. For a given agent \(a\), the modality selected for the scene \(l\) would be a combination of this agent available marginal modalities \(p_{k}^{a}\in\mathcal{M}\). Each scene modality \(l\) would select a different association between the agents.

The model learns \(L\) scene modality embeddings \(S_{l}\) of \(D\) features each. \(K\times A\) agent modality vectors \(A_{k}^{a}\) are also derived from each agent modality endpoint.

Since argmax is non-differentiable, we employ a soft argmax as a weighted linear combination of the agent modalities \(p_{k}^{a}\) using a softmax on the \(c_{l}^{k,a}\) scores: \[p_{l}^{a}=\sum_{k}softmax(c_{l}^{k,a})p_{k}^{a}\] The recombination module is trained with the scene-level minimum displacement error, where for each modality inside a single scene all predicted agent displacement losses are averaged before taking the minimum (winner-takes-all loss) over the \(L\) scene modalities.

Compared to these baselines, THOMAS can be seen as an hybrid sampling method that takes the result of deterministic sampling as input and learns to recombine it into a more coherent solution.

While the \(Before\) model tries to spread the modalities for all agents to minimize marginal miss-rate, the recombined model presents much less spread compared to the original model, maintaining a multimodal behavior only in presence of very different possible agent intentions such as different possible exits or turn choices. For most other agents, almost all modalities are located at the same position, that is, the one deemed the most likely by the model.

We have presented THOMAS, a recombination module that can be added after any trajectory prediction module outputing multi-modal predictions. By design, THOMAS allows to generate scene-consistent modalities across all agents by making the scene modalities select coherent agent modalities and restricting the modality budget on the agents that truly need it.

The initial encoded \([A,D]\) tensor is obtained with a GOHOME encoder, multiplied across the \(F\) futures and concatenated with a modality-specific one-hot encoding as in Ngiam et al. (2021) to obtain the \([F,A,D]\) tensor. We then apply two layers of agent self-attention similar to the original paper, before decoding the endpoints through a MLP.

This way, we try to enforce collision-free endpoints, and expect that considering collisions brings logic to improve the overall consistency of the predictions.

Input: Figure 1: Illustration of the THOMAS multi-agent prediction pipeline

Input: Figure 5: Qualitative examples of recombination model assembling collision-free modalities together compared to initial colliding modalities. For each example we display the general context with highlighted agents and area of interest, then two zooms in on the agents, one displaying the initial best modality before recombination in dashed orange and all the other available modalities in grey. The second zooms shows the best modality after recombination in full line orange.

Input: Figure 6: Qualitative examples of recombination model selecting fewer but more pronounced and impactful agent modalities compared to initial colliding modalities. For each example we display on the left the vanilla model modalities with dashed lines, with the initial best modality in dashed orange and all the other available modalities in grey. On the right we display the selected modalities after recombination, where the model focuses on the most likely occurrence in most agents.

Input: Figure 4: Illustration of THOMAS methods for generation of scene-consistent agent modalities

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find that the claim is generally well-stated but has minor overstatement issues. The evidence demonstrates that the model does perform trajectory recombination using predicted trajectories as input, as claimed. The figures (particularly Fig. 1, 4, and 5) clearly show the recombination process and its results. However, the word "learnable" in the claim could be slightly overstated since the evidence doesn't explicitly demonstrate the learning process, though it does mention training with "scene-level minimum displacement error." The model architecture and mathematical formulation provided support the basic recombination functionality.

<score>2</score>

Justification:
1. The evidence strongly supports the core claim about trajectory recombination, with clear mathematical formulation and visual examples showing the process.
2. The implementation details shown in figures and equations validate that the model takes "predicted trajectories" as input and outputs "recombinations."
3. The term "learnable" is supported by mentions of training and soft argmax for differentiability, though not extensively proven.
4. The consistency aspect is well-supported by figures showing collision-free and coherent recombinations.
5. Minor overstatement in emphasizing "learnable" without extensive evidence of the learning process itself.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We propose a learnable trajectory recombination model that takes as input a set of predicted trajectories for each agent and outputs its consistent reordered recombination.

The review comment to be evaluated is:

### Strengths:
- __Originality__: the two main contributions outlined in the summary of the paper are fairly distinctive from previous works.
  - The hierarchical heatmap prediction tackles one of the main issues of heatmap representations, which is their heavy memory consumption.
  - The learnable recombination method is general enough to be suitable for any marginal prediction model, which makes me think it would have a solid impact on the community.
- Introduction and related work are succinct and touch upon the main relevant points to the future prediction task.
- Figures are useful to understand the method as well as to evaluate the performance qualitatively.

### Weaknesses:
- __Writing ambiguity and lack of details__: I am not sure the write-up is polished enough for submission to ICLR. I outline a few reasons below:
  - Throughout the paper, there is no mention or description of the loss functions utilized, and barely any mention of the overall training process (with the exception of a sentence in 3.1.3). In the appendix, the authors describe how the heatmap output is optimized, but there is no mention about how the learnable recombination module is trained, which is one of the main contributions of the paper.
  - Because of this lack of description of the overall training process and loss functions, it is unclear if in the experiments the authors utilize 2 separate THOMAS models - one for marginal prediction and one for joint prediction. For instance, are the marginal metrics and joint metrics in Table 1 coming from a single model, or two separately trained models? A sentence that contributes to this confusion is in 3.1.3, where the authors mention that they decode K end points using the same MR optimization algorithm as Gilles et al. (2021b). However, in 3.2.1 the authors enhance this algorithm with collision-free sampling. Please clarify this in the rebuttal.
  - The paper criticizes generative models in the second paragraph of the related work for not providing a probability value for each prediction, which makes me expect the method to provide such scores. However, there is no mention about it in the approach or experiments sections.
  - In Table 3, the Marg and Joint objectives are not defined.
- It is unclear how some pieces tie together. My main concern is with respect to collision-free end point sampling (3.2.1) and recombination (3.2.2). The heuristic in 3.2.1 seems to assume that the modalities are "already ordered" since the algorithm basically avoids sampling close end points across actors for a given modality k. However, the recombination can perform arbitrary linear combinations across modes, thus not respecting this. Is the idea that only one of the two methods are used to perform inference/training on a model? If so, I would suggest moving 3.2.1 to the experiments as a baseline/ablation, and I definitely would not claim it as a contribution (particularly since it's just a small tweak on top of a previously proposed algorithm).
- Experiments are not as thorough as I would expect. This is understandable for the joint prediction setting as there are very few baselines and the dataset/benchmarks on this task have just started. In fact, I appreciate the effort of the authors of re-implementing ILVM and SceneTransformer. However, with respect to marginal prediction, there are a lot of methods and benchmarks out there. Another aspect that would have been great to get more insight on is the hierarchical heatmap output. What's the tradeoff between computation (both runtime and memory) and accuracy, with respect to dense heatmap approaches? How was the current 3-layer hierarchy chosen?

### Additional feedback:
In this section I provide additional feedback for rebuttal, but please address first the main points above.
- Re: training details in the appendix. I find it surprising that the method is trained with random sampling of up to 8 agents, given that for joint prediction there might be other agents that are particularly relevant to the task. Was this better than lowering the batch size? If so, is it because it is rare that a scene contains > 8 actors in the INTERACTION dataset?
- The equation that describes the weighted linear combination of agent modalities seems incomplete. As far as I understand, there should be a weighted sum in the equation, using the scores coming out of the softmax.
- Related work is missing some relevant works. In the last paragraph, joint autoregressive methods should be mentioned (e.g., PRECOG[A] - see references at the bottom) as they were the first to model the interactions across agents at the output level. With respect to generative models and their random sampling and lack of confidence scores per modality, LookOut[B] is a recent method that addresses both these concerns with a learnable diverse sampler and scene scoring. The authors can treat this as concurrent work since it was just published at ICCV, but please include it in the discussion as it is highly related.
- Is the notation of the joint metrics coming from the Interpret challenge? If not, I would suggest unifying this with the names proposed originally in ILVM (e.g., minJointFDE --> minSFDE, CrossCollisionRate --> SCR).
- While the sparse and hierarchical heatmaps make sense to mitigate their high resource consumption, I am a bit concerned about the fact that endpoints are only sampled from the finest resolution, as this might not contain more unlikely but safety-critical modalities. Overall, the heuristic refinement of the most likely grid cells seems contradictory with the endpoint sampling objective of maximizing coverage.
- The writing would benefit from more intuitions. For instance, the method samples heuristically from the heatmaps to maximize coverage, but why is this important to the overall self-driving task?
- I personally did not get much out of figure 4, perhaps adding the shapes of the tensors could be helpful.
- I would avoid calling the combination method an "ordering", as the method is more general than that.
- How come the CrossCollisionRate is still 2.6% when using the collision-free end point sampling? I would expect it to be zero.

### Additional references:
- [A] Rhinehart, Nicholas, et al. "Precog: Prediction conditioned on goals in visual multi-agent settings." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.
- [B] Cui, Alexander, et al. "LookOut: Diverse multi-future prediction and planning for self-driving." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

The evidence to be evaluated is:Finally, we recombine the sampled endpoints into scene-consistent modalities across agents and build the full trajectories for each agent.

Furthermore, we add a novel scene-consistency module that recombines the marginal outputs into a joint prediction.

Initially, the prediction output of the model can be defined as marginal, as all \(A\) agents have been predicted and sampled independently in order to get \(K\) possible endpoint each. Our goal is to output a set of scene predictions \(\mathcal{J}=(L,A)\) from the marginal prediction \(\mathcal{M}=(A,K)\), where each scene modality \(l\) belonging to \(\mathcal{J}\) is an association of endpoints \(p_{l}^{a}\) for each agent \(a\). To achieve this, our main hypothesis lays in the fact that good trajectory proposals are already present in the marginal predictions, but they need to be coherently aligned among agents to achieve a set of overall consistent scene predictions. For a given agent \(a\), the modality selected for the scene \(l\) would be a combination of this agent available marginal modalities \(p_{k}^{a}\in\mathcal{M}\). Each scene modality \(l\) would select a different association between the agents.

The model learns \(L\) scene modality embeddings \(S_{l}\) of \(D\) features each. \(K\times A\) agent modality vectors \(A_{k}^{a}\) are also derived from each agent modality endpoint.

Since argmax is non-differentiable, we employ a soft argmax as a weighted linear combination of the agent modalities \(p_{k}^{a}\) using a softmax on the \(c_{l}^{k,a}\) scores: \[p_{l}^{a}=\sum_{k}softmax(c_{l}^{k,a})p_{k}^{a}\] The recombination module is trained with the scene-level minimum displacement error, where for each modality inside a single scene all predicted agent displacement losses are averaged before taking the minimum (winner-takes-all loss) over the \(L\) scene modalities.

Compared to these baselines, THOMAS can be seen as an hybrid sampling method that takes the result of deterministic sampling as input and learns to recombine it into a more coherent solution.

While the \(Before\) model tries to spread the modalities for all agents to minimize marginal miss-rate, the recombined model presents much less spread compared to the original model, maintaining a multimodal behavior only in presence of very different possible agent intentions such as different possible exits or turn choices. For most other agents, almost all modalities are located at the same position, that is, the one deemed the most likely by the model.

We have presented THOMAS, a recombination module that can be added after any trajectory prediction module outputing multi-modal predictions. By design, THOMAS allows to generate scene-consistent modalities across all agents by making the scene modalities select coherent agent modalities and restricting the modality budget on the agents that truly need it.

The initial encoded \([A,D]\) tensor is obtained with a GOHOME encoder, multiplied across the \(F\) futures and concatenated with a modality-specific one-hot encoding as in Ngiam et al. (2021) to obtain the \([F,A,D]\) tensor. We then apply two layers of agent self-attention similar to the original paper, before decoding the endpoints through a MLP.

This way, we try to enforce collision-free endpoints, and expect that considering collisions brings logic to improve the overall consistency of the predictions.

Input: Figure 1: Illustration of the THOMAS multi-agent prediction pipeline

Input: Figure 5: Qualitative examples of recombination model assembling collision-free modalities together compared to initial colliding modalities. For each example we display the general context with highlighted agents and area of interest, then two zooms in on the agents, one displaying the initial best modality before recombination in dashed orange and all the other available modalities in grey. The second zooms shows the best modality after recombination in full line orange.

Input: Figure 6: Qualitative examples of recombination model selecting fewer but more pronounced and impactful agent modalities compared to initial colliding modalities. For each example we display on the left the vanilla model modalities with dashed lines, with the initial best modality in dashed orange and all the other available modalities in grey. On the right we display the selected modalities after recombination, where the model focuses on the most likely occurrence in most agents.

Input: Figure 4: Illustration of THOMAS methods for generation of scene-consistent agent modalities

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
The evidence provides substantial details about a trajectory recombination model that takes predicted trajectories as input and outputs reordered recombinations, supporting the core claim. The model architecture, mathematical formulation, and visual examples demonstrate how it combines agent modalities through cross-attention and linear recombination to create scene-consistent predictions. However, based on the reviewer comments, there are significant gaps in describing the training process and loss functions for the recombination module, making the claim slightly overstated in terms of its completeness. The method appears to work but lacks critical implementation details.

<score>2</score>

Justification:
1. The evidence clearly shows the model architecture and demonstrates results through figures
2. Mathematical formulation is provided for the recombination process using soft argmax and weighted linear combinations
3. Qualitative results in Figures 5 and 6 validate the recombination capabilities
4. However, reviewer highlights important missing details about:
- Training process for the recombination module
- Loss functions used
- Unclear relationship between collision-free sampling and recombination
- Ambiguity about whether separate models are used for marginal vs joint predictions

While the core claim is supported, these gaps in crucial technical details make it slightly overstated in its current form.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We propose a learnable trajectory recombination model that takes as input a set of predicted trajectories for each agent and outputs its consistent reordered recombination.

The review comment to be evaluated is:

### Strong Points

The idea of performing re-estimation based on the computed heat map end-points
seems novel to me. This idea seems like a natural next step to improve multi agent motion
prediction. The fact that you first get the end-point and then reason about the scene is
intuitively sounding and seems to be a good solution for the problem.

The experiments on interact are a nice direction to be taken by the field.

The visualizations of the heat-maps and the different predictions was helpful.

### Weak Points

#### I could not find a clear distinction with literature
While reading the related work section I found hard to contrast the ideas of the paper with the ones proposed on the literature.
The difference between this method and GOHOME was  not clear to me in this sense.
From what I understood the main contribution here  is the modality combination based on the collision
free end-point sampling. However, this was not obvious for me when reading the paper.

#### The implemented baselines seemed to have performed way worse than expected.
For example, both [1] and [7] are methods designed for *joint* prediction and scene level prediction. This is contradicted
by the observations made by the authors on page 8, quote: " ... This problem is aggravated in the joint training case, since the modality selected is
the same for all agents in a training sample....". Both [1] and [3] fairly confidently claim producing multimodal joint predictions.
Either the version implemented by the authors does not correctly match the other works in the literature or there are specific findings
on the limitations existent in other methods.
Also, the paper lack comparison with latent variable based [2][5][[6] multi modal prediction methods. They seem to perform
better when the issue is diverse prediction. Definitely the mode does not tend to be the same for all predictions in those cases.

#### A single dataset evalution might not be enough for the community to access the value of this paper.

 Motion prediction is currently a very competitive scene. Even though I appreciate the idea on focusing reporting results on multi-agents
and joint prediction tasks, it is important to also evaluate the prediction on more standard benchmarks such as argoverse.
With results on argoverse, me and the rest of the community can more quickly access the the capabilities of the proposed
 heat map prediction pipeline, on producing consistent ego-agent motion prediction. I understand that this would not be the main
results, but it would be helpful.

##### Computational cost is apparently not shown.

I could not find the computational cost of running and training the proposed method on the paper. It seems that even
with the optimizations made on the heat map estimation the training time and inference time should be high.
Those kinds of motion prediction models have the necessity to further scale with way more data. Thus, highly
complex methods can be hard to train. There is also the case of potential future execution in an embedded hardware.

### Specific Questions

I am not  sure about the scene modality vectors ( Section 3.2.2), are they like seed parameters which are going to be used
to incorporate the joint consistent scene level ? I couldn't really find any sort of detailed explanation on that.

Why the lack of ADE for scene level metrics or using similar metrics as [3] ?

[1] Jiquan Ngiam, Benjamin Caine, Vijay Vasudevan, Zhengdong Zhang, Hao-Tien Lewis Chiang,
Jeffrey Ling, Rebecca Roelofs, Alex Bewley, Chenxi Liu, Ashish Venugopal, et al. Scene
transformer: A unified multi-task model for behavior prediction and planning. arXiv preprint
arXiv:2106.08417, 2021
[2] Tang, Charlie, and Russ R. Salakhutdinov. "Multiple futures prediction." Advances in Neural Information Processing Systems 32 (2019): 15424-15434.
[3] Li, Lingyun Luke, et al. "End-to-end contextual perception and prediction with interaction transformer." 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2020.
[4] Yuan, Ye, et al. "AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting." arXiv preprint arXiv:2103.14023 (2021).
[5] Girgis, Roger, et al. "Autobots: Latent Variable Sequential Set Transformers." arXiv preprint arXiv:2104.00563 (2021).
[6] Casas, Sergio, et al. "Implicit latent variable model for scene-consistent motion forecasting." Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28,
[7] Ming Liang, Bin Yang, Rui Hu, Yun Chen, Renjie Liao, Song Feng, and Raquel Urtasun. Learning
lane graph representations for motion forecasting. In ECCV, 2020

The evidence to be evaluated is:Finally, we recombine the sampled endpoints into scene-consistent modalities across agents and build the full trajectories for each agent.

Furthermore, we add a novel scene-consistency module that recombines the marginal outputs into a joint prediction.

Initially, the prediction output of the model can be defined as marginal, as all \(A\) agents have been predicted and sampled independently in order to get \(K\) possible endpoint each. Our goal is to output a set of scene predictions \(\mathcal{J}=(L,A)\) from the marginal prediction \(\mathcal{M}=(A,K)\), where each scene modality \(l\) belonging to \(\mathcal{J}\) is an association of endpoints \(p_{l}^{a}\) for each agent \(a\). To achieve this, our main hypothesis lays in the fact that good trajectory proposals are already present in the marginal predictions, but they need to be coherently aligned among agents to achieve a set of overall consistent scene predictions. For a given agent \(a\), the modality selected for the scene \(l\) would be a combination of this agent available marginal modalities \(p_{k}^{a}\in\mathcal{M}\). Each scene modality \(l\) would select a different association between the agents.

The model learns \(L\) scene modality embeddings \(S_{l}\) of \(D\) features each. \(K\times A\) agent modality vectors \(A_{k}^{a}\) are also derived from each agent modality endpoint.

Since argmax is non-differentiable, we employ a soft argmax as a weighted linear combination of the agent modalities \(p_{k}^{a}\) using a softmax on the \(c_{l}^{k,a}\) scores: \[p_{l}^{a}=\sum_{k}softmax(c_{l}^{k,a})p_{k}^{a}\] The recombination module is trained with the scene-level minimum displacement error, where for each modality inside a single scene all predicted agent displacement losses are averaged before taking the minimum (winner-takes-all loss) over the \(L\) scene modalities.

Compared to these baselines, THOMAS can be seen as an hybrid sampling method that takes the result of deterministic sampling as input and learns to recombine it into a more coherent solution.

While the \(Before\) model tries to spread the modalities for all agents to minimize marginal miss-rate, the recombined model presents much less spread compared to the original model, maintaining a multimodal behavior only in presence of very different possible agent intentions such as different possible exits or turn choices. For most other agents, almost all modalities are located at the same position, that is, the one deemed the most likely by the model.

We have presented THOMAS, a recombination module that can be added after any trajectory prediction module outputing multi-modal predictions. By design, THOMAS allows to generate scene-consistent modalities across all agents by making the scene modalities select coherent agent modalities and restricting the modality budget on the agents that truly need it.

The initial encoded \([A,D]\) tensor is obtained with a GOHOME encoder, multiplied across the \(F\) futures and concatenated with a modality-specific one-hot encoding as in Ngiam et al. (2021) to obtain the \([F,A,D]\) tensor. We then apply two layers of agent self-attention similar to the original paper, before decoding the endpoints through a MLP.

This way, we try to enforce collision-free endpoints, and expect that considering collisions brings logic to improve the overall consistency of the predictions.

Input: Figure 1: Illustration of the THOMAS multi-agent prediction pipeline

Input: Figure 5: Qualitative examples of recombination model assembling collision-free modalities together compared to initial colliding modalities. For each example we display the general context with highlighted agents and area of interest, then two zooms in on the agents, one displaying the initial best modality before recombination in dashed orange and all the other available modalities in grey. The second zooms shows the best modality after recombination in full line orange.

Input: Figure 6: Qualitative examples of recombination model selecting fewer but more pronounced and impactful agent modalities compared to initial colliding modalities. For each example we display on the left the vanilla model modalities with dashed lines, with the initial best modality in dashed orange and all the other available modalities in grey. On the right we display the selected modalities after recombination, where the model focuses on the most likely occurrence in most agents.

Input: Figure 4: Illustration of THOMAS methods for generation of scene-consistent agent modalities

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim about a "learnable trajectory recombination model" appears to be well-stated and accurately reflects the presented methodology. The evidence thoroughly demonstrates:
- A clear technical description of how the recombination works through scene modality embeddings and agent modality vectors
- Visual examples in Figures 4-6 showing the recombination process and its effectiveness
- Concrete mathematical formulation of the recombination approach
- Validation through qualitative results showing improved collision-free and consistent trajectories
- Appropriate acknowledgment of limitations and design choices

The claim avoids exaggeration and is directly supported by the detailed technical evidence provided.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We propose a learnable trajectory recombination model that takes as input a set of predicted trajectories for each agent and outputs its consistent reordered recombination.

The review comment to be evaluated is:

Strengths:
1. The paper is well written and easy to follow.
2. Good performance was achieved on the Interaction multi-agent prediction challenge.

Weaknesses:
1. The method largely follows previous works, with little technical contributions. And even for these limited technical novelties, many of them are not well supported with experimemts.
In the end of introduction section, the authors summarize three main contributions:

i) an efficient graph-based model. However, the implementation mainly follows LaneGCN and GOHOME.

ii) a collision-aware endpoint sampling. The algorithm mainly follows GOHOME, with a small modification in that when we sample an endpoint for actor a modality k, we not only mask out the same region for other modalities of actor a, but also mask out for other actors of modality k. However, this modification doesn't make sense to me, as we still have a scene-consistent modality recombination module afterwards, therefore the second mask-out won't make much difference here. I also don't see an ablation study for this specific change (ie, do not mask-out for other actors of modality k, which means exactly follows the original algorithm in GOHOME). I want to see this ablation for the last three rows in Table 3.

iii) a novel recombination model that could obtain scene-consistent trajectories across the agents. To me this is the core novelty of the paper. From the last two rows in Table 3 we can see that with the proposed recombination module joint metrics improve a lot. However, the marginal metrics degrade a lot (e.g., miss rate more than doubled). The only explanation is that with the proposed module, the same modality (also the sub-optimal one) of an actor is chosen for multiple times in scene-level multi-modalities. To me this may not be what we want: scene-level consistency, but sacrifices actor-level precision a lot. Further analysis is needed here to convince me that this trade-off is worthwhile.

2. Another technical novelty that's not mentioned is the hierarchical iterative refinement of endpoints from the heatmap. However, no ablation study is done on this. I'd like to see a speed-performance comparison with original endpoint estimation model (as in GOHOME).

3. Also, the overall runtime efficiency of the method is not given.

4. GOHOME is the closest approach but I don't see it as a baseline in the experiments.

The evidence to be evaluated is:Finally, we recombine the sampled endpoints into scene-consistent modalities across agents and build the full trajectories for each agent.

Furthermore, we add a novel scene-consistency module that recombines the marginal outputs into a joint prediction.

Initially, the prediction output of the model can be defined as marginal, as all \(A\) agents have been predicted and sampled independently in order to get \(K\) possible endpoint each. Our goal is to output a set of scene predictions \(\mathcal{J}=(L,A)\) from the marginal prediction \(\mathcal{M}=(A,K)\), where each scene modality \(l\) belonging to \(\mathcal{J}\) is an association of endpoints \(p_{l}^{a}\) for each agent \(a\). To achieve this, our main hypothesis lays in the fact that good trajectory proposals are already present in the marginal predictions, but they need to be coherently aligned among agents to achieve a set of overall consistent scene predictions. For a given agent \(a\), the modality selected for the scene \(l\) would be a combination of this agent available marginal modalities \(p_{k}^{a}\in\mathcal{M}\). Each scene modality \(l\) would select a different association between the agents.

The model learns \(L\) scene modality embeddings \(S_{l}\) of \(D\) features each. \(K\times A\) agent modality vectors \(A_{k}^{a}\) are also derived from each agent modality endpoint.

Since argmax is non-differentiable, we employ a soft argmax as a weighted linear combination of the agent modalities \(p_{k}^{a}\) using a softmax on the \(c_{l}^{k,a}\) scores: \[p_{l}^{a}=\sum_{k}softmax(c_{l}^{k,a})p_{k}^{a}\] The recombination module is trained with the scene-level minimum displacement error, where for each modality inside a single scene all predicted agent displacement losses are averaged before taking the minimum (winner-takes-all loss) over the \(L\) scene modalities.

Compared to these baselines, THOMAS can be seen as an hybrid sampling method that takes the result of deterministic sampling as input and learns to recombine it into a more coherent solution.

While the \(Before\) model tries to spread the modalities for all agents to minimize marginal miss-rate, the recombined model presents much less spread compared to the original model, maintaining a multimodal behavior only in presence of very different possible agent intentions such as different possible exits or turn choices. For most other agents, almost all modalities are located at the same position, that is, the one deemed the most likely by the model.

We have presented THOMAS, a recombination module that can be added after any trajectory prediction module outputing multi-modal predictions. By design, THOMAS allows to generate scene-consistent modalities across all agents by making the scene modalities select coherent agent modalities and restricting the modality budget on the agents that truly need it.

The initial encoded \([A,D]\) tensor is obtained with a GOHOME encoder, multiplied across the \(F\) futures and concatenated with a modality-specific one-hot encoding as in Ngiam et al. (2021) to obtain the \([F,A,D]\) tensor. We then apply two layers of agent self-attention similar to the original paper, before decoding the endpoints through a MLP.

This way, we try to enforce collision-free endpoints, and expect that considering collisions brings logic to improve the overall consistency of the predictions.

Input: Figure 1: Illustration of the THOMAS multi-agent prediction pipeline

Input: Figure 5: Qualitative examples of recombination model assembling collision-free modalities together compared to initial colliding modalities. For each example we display the general context with highlighted agents and area of interest, then two zooms in on the agents, one displaying the initial best modality before recombination in dashed orange and all the other available modalities in grey. The second zooms shows the best modality after recombination in full line orange.

Input: Figure 6: Qualitative examples of recombination model selecting fewer but more pronounced and impactful agent modalities compared to initial colliding modalities. For each example we display on the left the vanilla model modalities with dashed lines, with the initial best modality in dashed orange and all the other available modalities in grey. On the right we display the selected modalities after recombination, where the model focuses on the most likely occurrence in most agents.

Input: Figure 4: Illustration of THOMAS methods for generation of scene-consistent agent modalities

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on my review of the claim "We propose a learnable trajectory recombination model that takes as input a set of predicted trajectories for each agent and outputs its consistent reordered recombination," I find this to be a reasonably well-stated claim that is supported by the evidence provided. The paper thoroughly describes the recombination module's architecture and functioning, demonstrates its implementation through figures, and shows qualitative results. However, the reviewer raises valid concerns about limited technical novelty and potential tradeoffs between scene-level consistency and actor-level precision that aren't fully acknowledged in the claim.

<score>2</score>

Justification:
1. Supporting evidence:
- The paper provides detailed mathematical formulation of the recombination process
- Figures 4-6 clearly illustrate the recombination methodology and results
- The evidence shows implementation details and working examples

2. Minor overstatement concerns:
- The reviewer points out that "the method largely follows previous works"
- There are tradeoffs in performance (improved joint metrics but degraded marginal metrics) that aren't mentioned in the claim
- Some technical aspects lack ablation studies

3. The claim could be more precise by:
- Acknowledging the foundation in previous work
- Mentioning the specific focus on scene-consistency
- Noting the tradeoffs involved

While the claim describes what the model does, it would benefit from more context about its relationship to existing work and its specific advantages and limitations. However, these are relatively minor issues that don't significantly detract from the claim's overall accuracy.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We propose a learnable trajectory recombination model that takes as input a set of predicted trajectories for each agent and outputs its consistent reordered recombination.

The review comment to be evaluated is:

- The overall idea of iterative hierarchical heatmap refinement is interesting and appears to be novel. The experiments demonstrate the ideas validity, although there is definitely room for further strengthening.

- The related work is thorough, one quick comment here is to also reference the following work in the discussion of methods which can predict multiple agents at the same time (beginning of the last paragraph before Section 3): "MATS: An Interpretable Trajectory Forecasting Representation for Planning and Control" by B. Ivanovic, A. Elhafsi, G. Rosman, A. Gaidon, M. Pavone in CoRL 2020.

- Acronyms should be explained when they are first used. For instance, it might not be obvious what a UGRU is in Section 3.1.1. What does MR optimization mean in Section 3.1.3?

- Do not use contractions in scientific writing (e.g., change "don't" to "do not").

- A thorough round of editing is necessary, there seem to be writing errors that materially change the meaning of sentences. For instance, "The final output is therefore no strictly a re-ordering..." Should this have said "not strictly"? Same question for "which is JointMR where colliding modalities are also counted as misses even if they are closer than the defined threshold." Should this have just said "where colliding modalities are also counted as misses?" It is not very clear what these metrics mean here.

- The novelty with respect to the original GOHOME work needs to be made clearer, there seem to be many reused components and it is difficult to tell which parts are new in this work.

- Related to the above point, it is a bit difficult to understand this work's core contributions in Sections 3.2.1 and 3.2.2, also because the writing makes this difficult (e.g., a spurious use of "However" makes it difficult to determine if a statement is being negated or not).

- There are some key insights missing from the experimental section, which will be expanded on below.

- Ultimately, the method is only evaluated on one dataset, and it is difficult to determine if its performance would generalize to other datasets (for instance, what about nuScenes, Lyft, Waymo Open, Argoverse, etc?).

The evidence to be evaluated is:Finally, we recombine the sampled endpoints into scene-consistent modalities across agents and build the full trajectories for each agent.

Furthermore, we add a novel scene-consistency module that recombines the marginal outputs into a joint prediction.

Initially, the prediction output of the model can be defined as marginal, as all \(A\) agents have been predicted and sampled independently in order to get \(K\) possible endpoint each. Our goal is to output a set of scene predictions \(\mathcal{J}=(L,A)\) from the marginal prediction \(\mathcal{M}=(A,K)\), where each scene modality \(l\) belonging to \(\mathcal{J}\) is an association of endpoints \(p_{l}^{a}\) for each agent \(a\). To achieve this, our main hypothesis lays in the fact that good trajectory proposals are already present in the marginal predictions, but they need to be coherently aligned among agents to achieve a set of overall consistent scene predictions. For a given agent \(a\), the modality selected for the scene \(l\) would be a combination of this agent available marginal modalities \(p_{k}^{a}\in\mathcal{M}\). Each scene modality \(l\) would select a different association between the agents.

The model learns \(L\) scene modality embeddings \(S_{l}\) of \(D\) features each. \(K\times A\) agent modality vectors \(A_{k}^{a}\) are also derived from each agent modality endpoint.

Since argmax is non-differentiable, we employ a soft argmax as a weighted linear combination of the agent modalities \(p_{k}^{a}\) using a softmax on the \(c_{l}^{k,a}\) scores: \[p_{l}^{a}=\sum_{k}softmax(c_{l}^{k,a})p_{k}^{a}\] The recombination module is trained with the scene-level minimum displacement error, where for each modality inside a single scene all predicted agent displacement losses are averaged before taking the minimum (winner-takes-all loss) over the \(L\) scene modalities.

Compared to these baselines, THOMAS can be seen as an hybrid sampling method that takes the result of deterministic sampling as input and learns to recombine it into a more coherent solution.

While the \(Before\) model tries to spread the modalities for all agents to minimize marginal miss-rate, the recombined model presents much less spread compared to the original model, maintaining a multimodal behavior only in presence of very different possible agent intentions such as different possible exits or turn choices. For most other agents, almost all modalities are located at the same position, that is, the one deemed the most likely by the model.

We have presented THOMAS, a recombination module that can be added after any trajectory prediction module outputing multi-modal predictions. By design, THOMAS allows to generate scene-consistent modalities across all agents by making the scene modalities select coherent agent modalities and restricting the modality budget on the agents that truly need it.

The initial encoded \([A,D]\) tensor is obtained with a GOHOME encoder, multiplied across the \(F\) futures and concatenated with a modality-specific one-hot encoding as in Ngiam et al. (2021) to obtain the \([F,A,D]\) tensor. We then apply two layers of agent self-attention similar to the original paper, before decoding the endpoints through a MLP.

This way, we try to enforce collision-free endpoints, and expect that considering collisions brings logic to improve the overall consistency of the predictions.

Input: Figure 1: Illustration of the THOMAS multi-agent prediction pipeline

Input: Figure 5: Qualitative examples of recombination model assembling collision-free modalities together compared to initial colliding modalities. For each example we display the general context with highlighted agents and area of interest, then two zooms in on the agents, one displaying the initial best modality before recombination in dashed orange and all the other available modalities in grey. The second zooms shows the best modality after recombination in full line orange.

Input: Figure 6: Qualitative examples of recombination model selecting fewer but more pronounced and impactful agent modalities compared to initial colliding modalities. For each example we display on the left the vanilla model modalities with dashed lines, with the initial best modality in dashed orange and all the other available modalities in grey. On the right we display the selected modalities after recombination, where the model focuses on the most likely occurrence in most agents.

Input: Figure 4: Illustration of THOMAS methods for generation of scene-consistent agent modalities

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on my review of the evidence and review comments:

The claim about proposing a learnable trajectory recombination model is generally well-supported by the detailed technical evidence provided, including mathematical formulations, architectural details, and visualizations demonstrating how the model recombines trajectories. The figures clearly show the recombination process and its effects. However, the review comments raise valid concerns about clarity of novelty compared to GOHOME work, limited dataset evaluation, and some unclear technical details. The claim itself is relatively straightforward and doesn't overstate capabilities, though greater precision in describing novelty would help.

<score>2</score>

Justification:
1. Strong evidence supports the basic mechanism through:
- Detailed mathematical formulation of the recombination process
- Clear visualization of the pipeline in Figure 1
- Qualitative examples in Figures 5-6 demonstrating the practical effects
- Technical description of scene modality embeddings and training process

2. Minor overstatement concerns:
- Reviewer notes unclear differentiation from GOHOME work
- Limited evaluation on only one dataset raises generalizability questions
- Some technical details (like UGRU, MR optimization) need better explanation

The claim itself is measured and specific to the technical contribution, avoiding grand statements about performance or broad applicability, earning a score closer to well-stated than overstated.