Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We present a unified model architecture for simultaneous agent future heatmap estimation, in which we leverage hierarchical and sparse image generation for fast and memory-efficient inference.

The review comment to be evaluated is:

### Strong Points

The idea of performing re-estimation based on the computed heat map end-points
seems novel to me. This idea seems like a natural next step to improve multi agent motion
prediction. The fact that you first get the end-point and then reason about the scene is
intuitively sounding and seems to be a good solution for the problem.

The experiments on interact are a nice direction to be taken by the field.

The visualizations of the heat-maps and the different predictions was helpful.

### Weak Points

#### I could not find a clear distinction with literature
While reading the related work section I found hard to contrast the ideas of the paper with the ones proposed on the literature.
The difference between this method and GOHOME was  not clear to me in this sense.
From what I understood the main contribution here  is the modality combination based on the collision
free end-point sampling. However, this was not obvious for me when reading the paper.

#### The implemented baselines seemed to have performed way worse than expected.
For example, both [1] and [7] are methods designed for *joint* prediction and scene level prediction. This is contradicted
by the observations made by the authors on page 8, quote: " ... This problem is aggravated in the joint training case, since the modality selected is
the same for all agents in a training sample....". Both [1] and [3] fairly confidently claim producing multimodal joint predictions.
Either the version implemented by the authors does not correctly match the other works in the literature or there are specific findings
on the limitations existent in other methods.
Also, the paper lack comparison with latent variable based [2][5][[6] multi modal prediction methods. They seem to perform
better when the issue is diverse prediction. Definitely the mode does not tend to be the same for all predictions in those cases.

#### A single dataset evalution might not be enough for the community to access the value of this paper.

 Motion prediction is currently a very competitive scene. Even though I appreciate the idea on focusing reporting results on multi-agents
and joint prediction tasks, it is important to also evaluate the prediction on more standard benchmarks such as argoverse.
With results on argoverse, me and the rest of the community can more quickly access the the capabilities of the proposed
 heat map prediction pipeline, on producing consistent ego-agent motion prediction. I understand that this would not be the main
results, but it would be helpful.

##### Computational cost is apparently not shown.

I could not find the computational cost of running and training the proposed method on the paper. It seems that even
with the optimizations made on the heat map estimation the training time and inference time should be high.
Those kinds of motion prediction models have the necessity to further scale with way more data. Thus, highly
complex methods can be hard to train. There is also the case of potential future execution in an embedded hardware.

### Specific Questions

I am not  sure about the scene modality vectors ( Section 3.2.2), are they like seed parameters which are going to be used
to incorporate the joint consistent scene level ? I couldn't really find any sort of detailed explanation on that.

Why the lack of ADE for scene level metrics or using similar metrics as [3] ?

[1] Jiquan Ngiam, Benjamin Caine, Vijay Vasudevan, Zhengdong Zhang, Hao-Tien Lewis Chiang,
Jeffrey Ling, Rebecca Roelofs, Alex Bewley, Chenxi Liu, Ashish Venugopal, et al. Scene
transformer: A unified multi-task model for behavior prediction and planning. arXiv preprint
arXiv:2106.08417, 2021
[2] Tang, Charlie, and Russ R. Salakhutdinov. "Multiple futures prediction." Advances in Neural Information Processing Systems 32 (2019): 15424-15434.
[3] Li, Lingyun Luke, et al. "End-to-end contextual perception and prediction with interaction transformer." 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2020.
[4] Yuan, Ye, et al. "AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting." arXiv preprint arXiv:2103.14023 (2021).
[5] Girgis, Roger, et al. "Autobots: Latent Variable Sequential Set Transformers." arXiv preprint arXiv:2104.00563 (2021).
[6] Casas, Sergio, et al. "Implicit latent variable model for scene-consistent motion forecasting." Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28,
[7] Ming Liang, Bin Yang, Rui Hu, Yun Chen, Renjie Liao, Song Feng, and Raquel Urtasun. Learning
lane graph representations for motion forecasting. In ECCV, 2020

The evidence to be evaluated is:The approach most related to this paper is GOHOME (Gilles et al., 2021), which uses a similar graph encoder and then leverage lane rasters to generate a probability heatmap in a sparse manner.

Our goal is to predict the future \(T\) timesteps of \(A\) agents using their past history made of \(H\) timesteps and the HD-Map context.

We then decode a future probability heatmap for each agent in the scene, which we sample heuristically to maximize coverage. Finally, we recombine the sampled endpoints into scene-consistent modalities across agents and build the full trajectories for each agent. Our pipeline shares the same graph encoder, sampling algorithm and full trajectory generation as GOHOME (Gilles et al., 2021), but uses a novel efficient hierarchical heatmap process that enables to scale to simultaneous multi-agent prediction.

Our aim here is to decode each agent encoding into a heatmap representing its future probability distribution at prediction horizon \(T\). Since we create this heatmap for each agent in the scene, the decoding process has to be fast so that it can be applied to a great number of agents in parallel. We use hierarchical predictions at various levels of resolutions so that the decoder has the possibility of predicting over the full surroundings of the agent but learns to refine with more precision only in places where the agent will end up with high probability. This hierarchical process is illustrated in Fig. 3. Starting from an initial full dense grid probability at low resolution \(R_{0}\times R_{0}\) by pixels, we iteratively refine the resolution by a fixed factor \(f\) until we reach the desired final resolution \(R_{final}\times R_{final}\). At each iteration \(i\), we select only the \(N_{i}\) highest ranking grid points from the previous iteration, and upsample only these points to the \(R_{i}\times R_{i}=\nicefrac{{R_{i-1}}}{{f}}\times\nicefrac{{R_{i-1}}}{{f}}\). At each step, the grid points features are computed by a 2-layer MLP applied on the point coordinates, these features are then concatenated to the agent encoding and passed through a linear layer, and finally enriched by a 2-layer cross-attention on the graph lane features, before applying a linear layer with sigmoid to get the probability. For a given \(W\) output range, this hierarchical process allows the model to only operate on \(\nicefrac{{W}}{{R_{0}}}\times\nicefrac{{W}}{{R_{0}}}+\sum_{i}N_{i}\times f^{2}\) grid points instead of the \(\nicefrac{{W}}{{R_{final}}}\times\nicefrac{{W}}{{R_{final}}}\) available. In practice, for a final output range of 192 meters with desired \(R_{final}=0.5m\) resolution, we start with an initial resolution of \(R_{0}=8m\) and use two iterations of \((N_{1},N_{2})=(16,64)\) points each and an upscaling factor \(f=4\). This way, we compute only 1856 grid points from the 147 456 available, with no performance loss.

Initially, the prediction output of the model can be defined as marginal, as all \(A\) agents have been predicted and sampled independently in order to get \(K\) possible endpoint each. Our goal is to output a set of scene predictions \(\mathcal{J}=(L,A)\) from the marginal prediction \(\mathcal{M}=(A,K)\), where each scene modality \(l\) belonging to \(\mathcal{J}\) is an association of endpoints \(p_{l}^{a}\) for each agent \(a\). To achieve this, our main hypothesis lays in the fact that good trajectory proposals are already present in the marginal predictions, but they need to be coherently aligned among agents to achieve a set of overall consistent scene predictions. For a given agent \(a\), the modality selected for the scene \(l\) would be a combination of this agent available marginal modalities \(p_{k}^{a}\in\mathcal{M}\).

The model learns \(L\) scene modality embeddings \(S_{l}\) of \(D\) features each. \(K\times A\) agent modality vectors \(A_{k}^{a}\) are also derived from each agent modality endpoint.

We assess the speed gain of our proposed hierarchical decoder compared to the lane rasters of GOHOME Gilles et al. (2021). In Tab. 3. We report training time for 16 epochs with a batchsize of 16, and inference time for 32 and 128 simultaneous agents. For additional comparison, the other existing dense prediction method DenseTNT (Gu et al., 2021) reports an inference speed of 100ms per sample for their model.

We have presented THOMAS, a recombination module that can be added after any trajectory prediction module outputing multi-modal predictions. By design, THOMAS allows to generate scene-consistent modalities across all agents by making the scene modalities select coherent agent modalities and restricting the modality budget on the agents that truly need it.

The initial encoded \([A,D]\) tensor is obtained with a GOHOME encoder, multiplied across the \(F\) futures and concatenated with a modality-specific one-hot encoding as in Ngiam et al. (2021) to obtain the \([F,A,D]\) tensor. We then apply two layers of agent self-attention similar to the original paper, before decoding the endpoints through a MLP. We designed a deterministic sampling algorithm based on the heatmaps generated in previous section in order to sample endpoints for each agent in a collision aware manner. We use the same sampling algorithm as Gilles et al. (2021) based on MR optimization, but add a sequential iteration over the agents for each modalities. For a single modality \(k\), we predict the possible endpoint of a first agent \(a\) by taking the maximum accumulated predicted probability under an area of radius \(r\). We then not only set to zero the heatmap values of this agent heatmap \(\mathcal{I}^{a^{\prime}}_{k^{\prime}}\) around the sampled location so not to sample it in the next modalities \(k^{\prime}\), but we also set to zero the same area on the heatmaps \(\mathcal{I}^{a^{\prime}}_{k}\) of the other agents \(a^{\prime}\) on the same modality \(k\), so that these other agents cannot be sampled at the same position for this modality. This way, we try to enforce collision-free endpoints, and expect that considering collisions brings logic to improve the overall consistency of the predictions.

We also display the trade-off between inference speed and coverage from hierarchical refinement in Fig. 7, evaluated on the Interpret multi-agent dataset with marginal MissRate\({}_{6}\). The curve is obtained setting the number \(N\) of upsampled points at the last refinement iteration from 2 to 128.

Input: Figure 7: Curve of MissRate6 with regard to inference time with varying number of points upsampled at the last hierarchical refinement iteration.

Input: Figure 2: Model architecture for multi-agent prediction with shared backbone

Input: Figure 3: Hierarchical iterative refinement of the grid probabilities. First, the full grid is evaluated at a very low resolution, then the highest cells are up-sampled and evaluated at a higher resolution, until final resolution is reached. We highlight in grey the restricted area considered for refinement at each step.

Input: Figure 8: Detailed illustration of our heatmap generator model

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim appears to be well-stated based on several key factors:

1. The paper clearly demonstrates a hierarchical heatmap process that reduces computational complexity from 147,456 to 1,856 grid points "with no performance loss"

2. Figure 7 provides empirical evidence showing the speed-accuracy trade-off of their approach

3. The detailed architecture in Figure 8 and technical descriptions support the unified model claim

4. The review comments acknowledge the novelty of the heat map-based re-estimation approach

The only minor concern is that computational costs aren't fully detailed, as noted in the review. However, this doesn't significantly detract from the core claim about the unified architecture and efficient inference.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We present a unified model architecture for simultaneous agent future heatmap estimation, in which we leverage hierarchical and sparse image generation for fast and memory-efficient inference.

The review comment to be evaluated is:

Strengths:
1. The paper is well written and easy to follow.
2. Good performance was achieved on the Interaction multi-agent prediction challenge.

Weaknesses:
1. The method largely follows previous works, with little technical contributions. And even for these limited technical novelties, many of them are not well supported with experimemts.
In the end of introduction section, the authors summarize three main contributions:

i) an efficient graph-based model. However, the implementation mainly follows LaneGCN and GOHOME.

ii) a collision-aware endpoint sampling. The algorithm mainly follows GOHOME, with a small modification in that when we sample an endpoint for actor a modality k, we not only mask out the same region for other modalities of actor a, but also mask out for other actors of modality k. However, this modification doesn't make sense to me, as we still have a scene-consistent modality recombination module afterwards, therefore the second mask-out won't make much difference here. I also don't see an ablation study for this specific change (ie, do not mask-out for other actors of modality k, which means exactly follows the original algorithm in GOHOME). I want to see this ablation for the last three rows in Table 3.

iii) a novel recombination model that could obtain scene-consistent trajectories across the agents. To me this is the core novelty of the paper. From the last two rows in Table 3 we can see that with the proposed recombination module joint metrics improve a lot. However, the marginal metrics degrade a lot (e.g., miss rate more than doubled). The only explanation is that with the proposed module, the same modality (also the sub-optimal one) of an actor is chosen for multiple times in scene-level multi-modalities. To me this may not be what we want: scene-level consistency, but sacrifices actor-level precision a lot. Further analysis is needed here to convince me that this trade-off is worthwhile.

2. Another technical novelty that's not mentioned is the hierarchical iterative refinement of endpoints from the heatmap. However, no ablation study is done on this. I'd like to see a speed-performance comparison with original endpoint estimation model (as in GOHOME).

3. Also, the overall runtime efficiency of the method is not given.

4. GOHOME is the closest approach but I don't see it as a baseline in the experiments.

The evidence to be evaluated is:The approach most related to this paper is GOHOME (Gilles et al., 2021), which uses a similar graph encoder and then leverage lane rasters to generate a probability heatmap in a sparse manner.

Our goal is to predict the future \(T\) timesteps of \(A\) agents using their past history made of \(H\) timesteps and the HD-Map context.

We then decode a future probability heatmap for each agent in the scene, which we sample heuristically to maximize coverage. Finally, we recombine the sampled endpoints into scene-consistent modalities across agents and build the full trajectories for each agent. Our pipeline shares the same graph encoder, sampling algorithm and full trajectory generation as GOHOME (Gilles et al., 2021), but uses a novel efficient hierarchical heatmap process that enables to scale to simultaneous multi-agent prediction.

Our aim here is to decode each agent encoding into a heatmap representing its future probability distribution at prediction horizon \(T\). Since we create this heatmap for each agent in the scene, the decoding process has to be fast so that it can be applied to a great number of agents in parallel. We use hierarchical predictions at various levels of resolutions so that the decoder has the possibility of predicting over the full surroundings of the agent but learns to refine with more precision only in places where the agent will end up with high probability. This hierarchical process is illustrated in Fig. 3. Starting from an initial full dense grid probability at low resolution \(R_{0}\times R_{0}\) by pixels, we iteratively refine the resolution by a fixed factor \(f\) until we reach the desired final resolution \(R_{final}\times R_{final}\). At each iteration \(i\), we select only the \(N_{i}\) highest ranking grid points from the previous iteration, and upsample only these points to the \(R_{i}\times R_{i}=\nicefrac{{R_{i-1}}}{{f}}\times\nicefrac{{R_{i-1}}}{{f}}\). At each step, the grid points features are computed by a 2-layer MLP applied on the point coordinates, these features are then concatenated to the agent encoding and passed through a linear layer, and finally enriched by a 2-layer cross-attention on the graph lane features, before applying a linear layer with sigmoid to get the probability. For a given \(W\) output range, this hierarchical process allows the model to only operate on \(\nicefrac{{W}}{{R_{0}}}\times\nicefrac{{W}}{{R_{0}}}+\sum_{i}N_{i}\times f^{2}\) grid points instead of the \(\nicefrac{{W}}{{R_{final}}}\times\nicefrac{{W}}{{R_{final}}}\) available. In practice, for a final output range of 192 meters with desired \(R_{final}=0.5m\) resolution, we start with an initial resolution of \(R_{0}=8m\) and use two iterations of \((N_{1},N_{2})=(16,64)\) points each and an upscaling factor \(f=4\). This way, we compute only 1856 grid points from the 147 456 available, with no performance loss.

Initially, the prediction output of the model can be defined as marginal, as all \(A\) agents have been predicted and sampled independently in order to get \(K\) possible endpoint each. Our goal is to output a set of scene predictions \(\mathcal{J}=(L,A)\) from the marginal prediction \(\mathcal{M}=(A,K)\), where each scene modality \(l\) belonging to \(\mathcal{J}\) is an association of endpoints \(p_{l}^{a}\) for each agent \(a\). To achieve this, our main hypothesis lays in the fact that good trajectory proposals are already present in the marginal predictions, but they need to be coherently aligned among agents to achieve a set of overall consistent scene predictions. For a given agent \(a\), the modality selected for the scene \(l\) would be a combination of this agent available marginal modalities \(p_{k}^{a}\in\mathcal{M}\).

The model learns \(L\) scene modality embeddings \(S_{l}\) of \(D\) features each. \(K\times A\) agent modality vectors \(A_{k}^{a}\) are also derived from each agent modality endpoint.

We assess the speed gain of our proposed hierarchical decoder compared to the lane rasters of GOHOME Gilles et al. (2021). In Tab. 3. We report training time for 16 epochs with a batchsize of 16, and inference time for 32 and 128 simultaneous agents. For additional comparison, the other existing dense prediction method DenseTNT (Gu et al., 2021) reports an inference speed of 100ms per sample for their model.

We have presented THOMAS, a recombination module that can be added after any trajectory prediction module outputing multi-modal predictions. By design, THOMAS allows to generate scene-consistent modalities across all agents by making the scene modalities select coherent agent modalities and restricting the modality budget on the agents that truly need it.

The initial encoded \([A,D]\) tensor is obtained with a GOHOME encoder, multiplied across the \(F\) futures and concatenated with a modality-specific one-hot encoding as in Ngiam et al. (2021) to obtain the \([F,A,D]\) tensor. We then apply two layers of agent self-attention similar to the original paper, before decoding the endpoints through a MLP. We designed a deterministic sampling algorithm based on the heatmaps generated in previous section in order to sample endpoints for each agent in a collision aware manner. We use the same sampling algorithm as Gilles et al. (2021) based on MR optimization, but add a sequential iteration over the agents for each modalities. For a single modality \(k\), we predict the possible endpoint of a first agent \(a\) by taking the maximum accumulated predicted probability under an area of radius \(r\). We then not only set to zero the heatmap values of this agent heatmap \(\mathcal{I}^{a^{\prime}}_{k^{\prime}}\) around the sampled location so not to sample it in the next modalities \(k^{\prime}\), but we also set to zero the same area on the heatmaps \(\mathcal{I}^{a^{\prime}}_{k}\) of the other agents \(a^{\prime}\) on the same modality \(k\), so that these other agents cannot be sampled at the same position for this modality. This way, we try to enforce collision-free endpoints, and expect that considering collisions brings logic to improve the overall consistency of the predictions.

We also display the trade-off between inference speed and coverage from hierarchical refinement in Fig. 7, evaluated on the Interpret multi-agent dataset with marginal MissRate\({}_{6}\). The curve is obtained setting the number \(N\) of upsampled points at the last refinement iteration from 2 to 128.

Input: Figure 7: Curve of MissRate6 with regard to inference time with varying number of points upsampled at the last hierarchical refinement iteration.

Input: Figure 2: Model architecture for multi-agent prediction with shared backbone

Input: Figure 3: Hierarchical iterative refinement of the grid probabilities. First, the full grid is evaluated at a very low resolution, then the highest cells are up-sampled and evaluated at a higher resolution, until final resolution is reached. We highlight in grey the restricted area considered for refinement at each step.

Input: Figure 8: Detailed illustration of our heatmap generator model

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find that the claim is moderately overstated. While the paper does present a hierarchical and sparse image generation approach for agent future heatmap estimation, several key issues emerge:

1. The reviewer points out that many technical elements follow previous works (LaneGCN and GOHOME) with limited novel contributions
2. The evidence shows the hierarchical approach builds heavily on GOHOME's framework
3. The speed-memory efficiency claims lack comprehensive benchmarking
4. The evidence demonstrates some trade-offs in performance that aren't acknowledged in the claim

The claim somewhat overstates the unified and novel aspects while understating the incremental nature of the contributions relative to prior work.

<score>3</score>

Justification:

1. The evidence shows significant borrowing from previous work: "Our pipeline shares the same graph encoder, sampling algorithm and full trajectory generation as GOHOME"

2. While the hierarchical decoder shows efficiency gains (computing 1,856 vs 147,456 grid points), comprehensive runtime comparisons against baselines are limited, making it difficult to fully validate the efficiency claims

3. The review notes several missing experimental validations:
- No ablation study for hierarchical iterative refinement
- Overall runtime efficiency not provided
- Missing comparisons to GOHOME baseline

4. The "unified model architecture" claim is overstated given that many core components are directly adapted from previous work with incremental modifications

5. Some technical changes have unclear benefits, as noted in the review regarding the collision-aware sampling modifications

The score of 3 reflects that while the approach does demonstrate some efficiency improvements through hierarchical prediction, the claim overstates the unified and novel aspects while understating its incremental nature relative to prior work.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We present a unified model architecture for simultaneous agent future heatmap estimation, in which we leverage hierarchical and sparse image generation for fast and memory-efficient inference.

The review comment to be evaluated is:

- The overall idea of iterative hierarchical heatmap refinement is interesting and appears to be novel. The experiments demonstrate the ideas validity, although there is definitely room for further strengthening.

- The related work is thorough, one quick comment here is to also reference the following work in the discussion of methods which can predict multiple agents at the same time (beginning of the last paragraph before Section 3): "MATS: An Interpretable Trajectory Forecasting Representation for Planning and Control" by B. Ivanovic, A. Elhafsi, G. Rosman, A. Gaidon, M. Pavone in CoRL 2020.

- Acronyms should be explained when they are first used. For instance, it might not be obvious what a UGRU is in Section 3.1.1. What does MR optimization mean in Section 3.1.3?

- Do not use contractions in scientific writing (e.g., change "don't" to "do not").

- A thorough round of editing is necessary, there seem to be writing errors that materially change the meaning of sentences. For instance, "The final output is therefore no strictly a re-ordering..." Should this have said "not strictly"? Same question for "which is JointMR where colliding modalities are also counted as misses even if they are closer than the defined threshold." Should this have just said "where colliding modalities are also counted as misses?" It is not very clear what these metrics mean here.

- The novelty with respect to the original GOHOME work needs to be made clearer, there seem to be many reused components and it is difficult to tell which parts are new in this work.

- Related to the above point, it is a bit difficult to understand this work's core contributions in Sections 3.2.1 and 3.2.2, also because the writing makes this difficult (e.g., a spurious use of "However" makes it difficult to determine if a statement is being negated or not).

- There are some key insights missing from the experimental section, which will be expanded on below.

- Ultimately, the method is only evaluated on one dataset, and it is difficult to determine if its performance would generalize to other datasets (for instance, what about nuScenes, Lyft, Waymo Open, Argoverse, etc?).

The evidence to be evaluated is:The approach most related to this paper is GOHOME (Gilles et al., 2021), which uses a similar graph encoder and then leverage lane rasters to generate a probability heatmap in a sparse manner.

Our goal is to predict the future \(T\) timesteps of \(A\) agents using their past history made of \(H\) timesteps and the HD-Map context.

We then decode a future probability heatmap for each agent in the scene, which we sample heuristically to maximize coverage. Finally, we recombine the sampled endpoints into scene-consistent modalities across agents and build the full trajectories for each agent. Our pipeline shares the same graph encoder, sampling algorithm and full trajectory generation as GOHOME (Gilles et al., 2021), but uses a novel efficient hierarchical heatmap process that enables to scale to simultaneous multi-agent prediction.

Our aim here is to decode each agent encoding into a heatmap representing its future probability distribution at prediction horizon \(T\). Since we create this heatmap for each agent in the scene, the decoding process has to be fast so that it can be applied to a great number of agents in parallel. We use hierarchical predictions at various levels of resolutions so that the decoder has the possibility of predicting over the full surroundings of the agent but learns to refine with more precision only in places where the agent will end up with high probability. This hierarchical process is illustrated in Fig. 3. Starting from an initial full dense grid probability at low resolution \(R_{0}\times R_{0}\) by pixels, we iteratively refine the resolution by a fixed factor \(f\) until we reach the desired final resolution \(R_{final}\times R_{final}\). At each iteration \(i\), we select only the \(N_{i}\) highest ranking grid points from the previous iteration, and upsample only these points to the \(R_{i}\times R_{i}=\nicefrac{{R_{i-1}}}{{f}}\times\nicefrac{{R_{i-1}}}{{f}}\). At each step, the grid points features are computed by a 2-layer MLP applied on the point coordinates, these features are then concatenated to the agent encoding and passed through a linear layer, and finally enriched by a 2-layer cross-attention on the graph lane features, before applying a linear layer with sigmoid to get the probability. For a given \(W\) output range, this hierarchical process allows the model to only operate on \(\nicefrac{{W}}{{R_{0}}}\times\nicefrac{{W}}{{R_{0}}}+\sum_{i}N_{i}\times f^{2}\) grid points instead of the \(\nicefrac{{W}}{{R_{final}}}\times\nicefrac{{W}}{{R_{final}}}\) available. In practice, for a final output range of 192 meters with desired \(R_{final}=0.5m\) resolution, we start with an initial resolution of \(R_{0}=8m\) and use two iterations of \((N_{1},N_{2})=(16,64)\) points each and an upscaling factor \(f=4\). This way, we compute only 1856 grid points from the 147 456 available, with no performance loss.

Initially, the prediction output of the model can be defined as marginal, as all \(A\) agents have been predicted and sampled independently in order to get \(K\) possible endpoint each. Our goal is to output a set of scene predictions \(\mathcal{J}=(L,A)\) from the marginal prediction \(\mathcal{M}=(A,K)\), where each scene modality \(l\) belonging to \(\mathcal{J}\) is an association of endpoints \(p_{l}^{a}\) for each agent \(a\). To achieve this, our main hypothesis lays in the fact that good trajectory proposals are already present in the marginal predictions, but they need to be coherently aligned among agents to achieve a set of overall consistent scene predictions. For a given agent \(a\), the modality selected for the scene \(l\) would be a combination of this agent available marginal modalities \(p_{k}^{a}\in\mathcal{M}\).

The model learns \(L\) scene modality embeddings \(S_{l}\) of \(D\) features each. \(K\times A\) agent modality vectors \(A_{k}^{a}\) are also derived from each agent modality endpoint.

We assess the speed gain of our proposed hierarchical decoder compared to the lane rasters of GOHOME Gilles et al. (2021). In Tab. 3. We report training time for 16 epochs with a batchsize of 16, and inference time for 32 and 128 simultaneous agents. For additional comparison, the other existing dense prediction method DenseTNT (Gu et al., 2021) reports an inference speed of 100ms per sample for their model.

We have presented THOMAS, a recombination module that can be added after any trajectory prediction module outputing multi-modal predictions. By design, THOMAS allows to generate scene-consistent modalities across all agents by making the scene modalities select coherent agent modalities and restricting the modality budget on the agents that truly need it.

The initial encoded \([A,D]\) tensor is obtained with a GOHOME encoder, multiplied across the \(F\) futures and concatenated with a modality-specific one-hot encoding as in Ngiam et al. (2021) to obtain the \([F,A,D]\) tensor. We then apply two layers of agent self-attention similar to the original paper, before decoding the endpoints through a MLP. We designed a deterministic sampling algorithm based on the heatmaps generated in previous section in order to sample endpoints for each agent in a collision aware manner. We use the same sampling algorithm as Gilles et al. (2021) based on MR optimization, but add a sequential iteration over the agents for each modalities. For a single modality \(k\), we predict the possible endpoint of a first agent \(a\) by taking the maximum accumulated predicted probability under an area of radius \(r\). We then not only set to zero the heatmap values of this agent heatmap \(\mathcal{I}^{a^{\prime}}_{k^{\prime}}\) around the sampled location so not to sample it in the next modalities \(k^{\prime}\), but we also set to zero the same area on the heatmaps \(\mathcal{I}^{a^{\prime}}_{k}\) of the other agents \(a^{\prime}\) on the same modality \(k\), so that these other agents cannot be sampled at the same position for this modality. This way, we try to enforce collision-free endpoints, and expect that considering collisions brings logic to improve the overall consistency of the predictions.

We also display the trade-off between inference speed and coverage from hierarchical refinement in Fig. 7, evaluated on the Interpret multi-agent dataset with marginal MissRate\({}_{6}\). The curve is obtained setting the number \(N\) of upsampled points at the last refinement iteration from 2 to 128.

Input: Figure 7: Curve of MissRate6 with regard to inference time with varying number of points upsampled at the last hierarchical refinement iteration.

Input: Figure 2: Model architecture for multi-agent prediction with shared backbone

Input: Figure 3: Hierarchical iterative refinement of the grid probabilities. First, the full grid is evaluated at a very low resolution, then the highest cells are up-sampled and evaluated at a higher resolution, until final resolution is reached. We highlight in grey the restricted area considered for refinement at each step.

Input: Figure 8: Detailed illustration of our heatmap generator model

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim about presenting "a unified model architecture for simultaneous agent future heatmap estimation" with hierarchical and sparse image generation is generally well-supported. The evidence provides detailed technical explanations of the hierarchical process, demonstrates computational efficiency (reducing 147,456 points to 1,856), and includes comprehensive architectural diagrams (Figures 2, 3, and 8). The speed-memory trade-off is quantitatively shown in Figure 7. However, reviewer comments note limited dataset evaluation and some unclear novelty relative to GOHOME, suggesting slight overstatement.

<score>2</score>

Justification:
1. Strengths:
- Detailed technical implementation is provided with clear figures
- Quantitative evidence of efficiency (1,856 vs 147,456 grid points)
- Performance-speed tradeoff demonstrated in Figure 7

2. Minor overstatement concerns:
- Reviewer notes evaluation on only one dataset, questioning generalizability
- Some components reused from GOHOME with unclear novelty differentiation
- Writing clarity issues noted by reviewer regarding core contributions

The score of 2 indicates the claim is mostly well-stated with only minor overstatement issues that don't significantly impact the core technical claims.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Our THOMAS model encodes the past trajectories of all the agents present in the scene, as well as the HD-Map lanelet graph, and then predicts for each agent a sparse heatmap representing the future probability distribution at a fixed timestep in the future, using hierarchical refinement for very efficient decoding.

The evidence to be evaluated is:Then, benefiting from the latest progresses in computer vision, convolutional layers can easily be applied to bird-view rasters of the map context (Lee et al., 2017; Tang & Salakhutdinov, 2019; Cui et al., 2019; Hong et al., 2019; Salzmann et al., 2020; Chai et al., 2020; Gilles et al., 2021). A more compact representation closer to the trajectory space can encode surrounding HD-Maps (usually formalized as connected lanelets), using Graph Neural Networks (Gao et al., 2020; Liang et al., 2020; Zeng et al., 2021; Gilles et al., 2021).

The approach most related to this paper is GOHOME (Gilles et al., 2021), which uses a similar graph encoder and then leverage lane rasters to generate a probability heatmap in a sparse manner.

Our goal is to predict the future \(T\) timesteps of \(A\) agents using their past history made of \(H\) timesteps and the HD-Map context.

We first encode each agent trajectory and the HD-Map context graph into a common representation. We then decode a future probability heatmap for each agent in the scene, which we sample heuristically to maximize coverage.

Our pipeline shares the same graph encoder, sampling algorithm and full trajectory generation as GOHOME (Gilles et al., 2021), but uses a novel efficient hierarchical heatmap process that enables to scale to simultaneous multi-agent prediction.

The agent trajectories are encoded though \(TrajEncoder\) using a 1D CNN followed by a UGRU recurrent layer (Rozenberg et al., 2021), and the HD-Map is encoded as a lanelet graph using a GNN \(GraphEncoder\) made of graph convolutions.

The final result is an encoding \(F_{a}\) for each agent, where history, context and interactions have been summarized. This encoding \(F_{a}\) is used in the next decoder operations and is also stored to be potentially used in modality recombination described in Sec. 3.2.

Our aim here is to decode each agent encoding into a heatmap representing its future probability distribution at prediction horizon \(T\).

We use hierarchical predictions at various levels of resolutions so that the decoder has the possibility of predicting over the full surroundings of the agent but learns to refine with more precision only in places where the agent will end up with high probability.

Starting from an initial full dense grid probability at low resolution \(R_{0}\times R_{0}\) by pixels, we iteratively refine the resolution by a fixed factor \(f\) until we reach the desired final resolution \(R_{final}\times R_{final}\). At each iteration \(i\), we select only the \(N_{i}\) highest ranking grid points from the previous iteration, and upsample only these points to the \(R_{i}\times R_{i}=\nicefrac{{R_{i-1}}}{{f}}\times\nicefrac{{R_{i-1}}}{{f}}\). At each step, the grid points features are computed by a 2-layer MLP applied on the point coordinates, these features are then concatenated to the agent encoding and passed through a linear layer, and finally enriched by a 2-layer cross-attention on the graph lane features, before applying a linear layer with sigmoid to get the probability. For a given \(W\) output range, this hierarchical process allows the model to only operate on \(\nicefrac{{W}}{{R_{0}}}\times\nicefrac{{W}}{{R_{0}}}+\sum_{i}N_{i}\times f^{2}\) grid points instead of the \(\nicefrac{{W}}{{R_{final}}}\times\nicefrac{{W}}{{R_{final}}}\) available. In practice, for a final output range of 192 meters with desired \(R_{final}=0.5m\) resolution, we start with an initial resolution of \(R_{0}=8m\) and use two iterations of \((N_{1},N_{2})=(16,64)\) points each and an upscaling factor \(f=4\). This way, we compute only 1856 grid points from the 147 456 available, with no performance loss. The heatmap is trained on each resolution level using as pixel-wise focal loss as in Gilles et al. (2021), detailed in Appendix A.3, with the ground truth being a Gaussian centered at the target agent future position. Having a heatmap as output grants us the good property of letting us decide to privilege coverage in order to give a wide spectrum of candidates to the scene recombination process. From each heatmap, we therefore decode \(K\) end points using the same MissRate optimization algorithm as Gilles et al. (2021). We then generate the full trajectories for each end point using also the same model, a fully-connected MLP. The MLP takes the car history and predicted endpoint as input and produces \(T\) 2D-coordinates representing the full future trajectory.

Initially, the prediction output of the model can be defined as marginal, as all \(A\) agents have been predicted and sampled independently in order to get \(K\) possible endpoint each. Our goal is to output a set of scene predictions \(\mathcal{J}=(L,A)\) from the marginal prediction \(\mathcal{M}=(A,K)\), where each scene modality \(l\) belonging to \(\mathcal{J}\) is an association of endpoints \(p_{l}^{a}\) for each agent \(a\).

The model learns \(L\) scene modality embeddings \(S_{l}\) of \(D\) features each. \(K\times A\) agent modality vectors \(A_{k}^{a}\) are also derived from each agent modality endpoint.

We show significant performance increase when adding the THOMAS module compared to the vanilla model and achieve state-of-the-art results compared to already existing methods tackling scene-consistent predictions.

We use the provided dataset HD-Maps to create the context graphs. As in Liang et al. (2020), we use four relations:\(\{predecessor,successor,left,right\}\) obtained from the lane graph connectivity.

For each agent, we use their trajectory history made of position, yaw and speed in the past second sampled at 10Hz.

The initial paper applies a transformer architecture on a \([F,A,T,D]\) tensor where \(F\) is the potential modality dimension, \(A\) the agent dimension and \(T\) the time dimension, with \(D\) the feature embedding, with factorized self-attention to the agent and time dimensions separately, so that agents can look at each-other inside a specific scene modality.

The initial encoded \([A,D]\) tensor is obtained with a GOHOME encoder, multiplied across the \(F\) futures and concatenated with a modality-specific one-hot encoding as in Ngiam et al. (2021) to obtain the \([F,A,D]\) tensor.

We also display the trade-off between inference speed and coverage from hierarchical refinement in Fig. 7, evaluated on the Interpret multi-agent dataset with marginal MissRate\({}_{6}\). The curve is obtained setting the number \(N\) of upsampled points at the last refinement iteration from 2 to 128.

Input: Figure 7: Curve of MissRate6 with regard to inference time with varying number of points upsampled at the last hierarchical refinement iteration.

Input: Figure 1: Illustration of the THOMAS multi-agent prediction pipeline

Input: Figure 2: Model architecture for multi-agent prediction with shared backbone

Input: Figure 9: Qualitative examples of heatmap output from our multi-agent model. All the heatmaps from one scene are the results from one single forward pass in our model predicting all agents at once. We use matching colors for the agent history, current location and their future predicted heatmap (best viewed in color).

Input: Figure 3: Hierarchical iterative refinement of the grid probabilities. First, the full grid is evaluated at a very low resolution, then the highest cells are up-sampled and evaluated at a higher resolution, until final resolution is reached. We highlight in grey the restricted area considered for refinement at each step.

Input: Figure 8: Detailed illustration of our heatmap generator model

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find that the claim is well-stated and accurately reflects what is demonstrated in the detailed technical descriptions, architecture diagrams, and results. The evidence thoroughly documents how THOMAS encodes past trajectories and HD-Map data through specific neural network architectures, and demonstrates the hierarchical refinement process for efficient heatmap decoding through clear visualizations and explanations. The performance trade-off curve in Fig. 7 specifically validates the efficiency claims. The model components and processes are described precisely without exaggeration.

<score>1</score>

Justification:
1. The claim about encoding trajectories is supported by detailed descriptions of the TrajEncoder and GraphEncoder components, with specific architecture details shown in Fig. 8.

2. The HD-Map lanelet graph encoding is verified through explicit documentation of the four relation types used and the graph neural network architecture.

3. The hierarchical refinement process for efficient decoding is thoroughly demonstrated through:
- Detailed mathematical formulation showing how refinement reduces computation from 147,456 to 1,856 grid points
- Clear visualization of the process in Fig. 3
- Empirical validation via the speed-accuracy trade-off curve in Fig. 7
- Qualitative results showing the heatmap outputs in Fig. 9

4. The claim stays within the bounds of what is proven by the evidence, avoiding any unsupported generalizations or exaggerated performance claims.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Our THOMAS model encodes the past trajectories of all the agents present in the scene, as well as the HD-Map lanelet graph, and then predicts for each agent a sparse heatmap representing the future probability distribution at a fixed timestep in the future, using hierarchical refinement for very efficient decoding.

The review comment to be evaluated is:

### Strengths:
- __Originality__: the two main contributions outlined in the summary of the paper are fairly distinctive from previous works.
  - The hierarchical heatmap prediction tackles one of the main issues of heatmap representations, which is their heavy memory consumption.
  - The learnable recombination method is general enough to be suitable for any marginal prediction model, which makes me think it would have a solid impact on the community.
- Introduction and related work are succinct and touch upon the main relevant points to the future prediction task.
- Figures are useful to understand the method as well as to evaluate the performance qualitatively.

### Weaknesses:
- __Writing ambiguity and lack of details__: I am not sure the write-up is polished enough for submission to ICLR. I outline a few reasons below:
  - Throughout the paper, there is no mention or description of the loss functions utilized, and barely any mention of the overall training process (with the exception of a sentence in 3.1.3). In the appendix, the authors describe how the heatmap output is optimized, but there is no mention about how the learnable recombination module is trained, which is one of the main contributions of the paper.
  - Because of this lack of description of the overall training process and loss functions, it is unclear if in the experiments the authors utilize 2 separate THOMAS models - one for marginal prediction and one for joint prediction. For instance, are the marginal metrics and joint metrics in Table 1 coming from a single model, or two separately trained models? A sentence that contributes to this confusion is in 3.1.3, where the authors mention that they decode K end points using the same MR optimization algorithm as Gilles et al. (2021b). However, in 3.2.1 the authors enhance this algorithm with collision-free sampling. Please clarify this in the rebuttal.
  - The paper criticizes generative models in the second paragraph of the related work for not providing a probability value for each prediction, which makes me expect the method to provide such scores. However, there is no mention about it in the approach or experiments sections.
  - In Table 3, the Marg and Joint objectives are not defined.
- It is unclear how some pieces tie together. My main concern is with respect to collision-free end point sampling (3.2.1) and recombination (3.2.2). The heuristic in 3.2.1 seems to assume that the modalities are "already ordered" since the algorithm basically avoids sampling close end points across actors for a given modality k. However, the recombination can perform arbitrary linear combinations across modes, thus not respecting this. Is the idea that only one of the two methods are used to perform inference/training on a model? If so, I would suggest moving 3.2.1 to the experiments as a baseline/ablation, and I definitely would not claim it as a contribution (particularly since it's just a small tweak on top of a previously proposed algorithm).
- Experiments are not as thorough as I would expect. This is understandable for the joint prediction setting as there are very few baselines and the dataset/benchmarks on this task have just started. In fact, I appreciate the effort of the authors of re-implementing ILVM and SceneTransformer. However, with respect to marginal prediction, there are a lot of methods and benchmarks out there. Another aspect that would have been great to get more insight on is the hierarchical heatmap output. What's the tradeoff between computation (both runtime and memory) and accuracy, with respect to dense heatmap approaches? How was the current 3-layer hierarchy chosen?

### Additional feedback:
In this section I provide additional feedback for rebuttal, but please address first the main points above.
- Re: training details in the appendix. I find it surprising that the method is trained with random sampling of up to 8 agents, given that for joint prediction there might be other agents that are particularly relevant to the task. Was this better than lowering the batch size? If so, is it because it is rare that a scene contains > 8 actors in the INTERACTION dataset?
- The equation that describes the weighted linear combination of agent modalities seems incomplete. As far as I understand, there should be a weighted sum in the equation, using the scores coming out of the softmax.
- Related work is missing some relevant works. In the last paragraph, joint autoregressive methods should be mentioned (e.g., PRECOG[A] - see references at the bottom) as they were the first to model the interactions across agents at the output level. With respect to generative models and their random sampling and lack of confidence scores per modality, LookOut[B] is a recent method that addresses both these concerns with a learnable diverse sampler and scene scoring. The authors can treat this as concurrent work since it was just published at ICCV, but please include it in the discussion as it is highly related.
- Is the notation of the joint metrics coming from the Interpret challenge? If not, I would suggest unifying this with the names proposed originally in ILVM (e.g., minJointFDE --> minSFDE, CrossCollisionRate --> SCR).
- While the sparse and hierarchical heatmaps make sense to mitigate their high resource consumption, I am a bit concerned about the fact that endpoints are only sampled from the finest resolution, as this might not contain more unlikely but safety-critical modalities. Overall, the heuristic refinement of the most likely grid cells seems contradictory with the endpoint sampling objective of maximizing coverage.
- The writing would benefit from more intuitions. For instance, the method samples heuristically from the heatmaps to maximize coverage, but why is this important to the overall self-driving task?
- I personally did not get much out of figure 4, perhaps adding the shapes of the tensors could be helpful.
- I would avoid calling the combination method an "ordering", as the method is more general than that.
- How come the CrossCollisionRate is still 2.6% when using the collision-free end point sampling? I would expect it to be zero.

### Additional references:
- [A] Rhinehart, Nicholas, et al. "Precog: Prediction conditioned on goals in visual multi-agent settings." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.
- [B] Cui, Alexander, et al. "LookOut: Diverse multi-future prediction and planning for self-driving." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

The evidence to be evaluated is:Then, benefiting from the latest progresses in computer vision, convolutional layers can easily be applied to bird-view rasters of the map context (Lee et al., 2017; Tang & Salakhutdinov, 2019; Cui et al., 2019; Hong et al., 2019; Salzmann et al., 2020; Chai et al., 2020; Gilles et al., 2021). A more compact representation closer to the trajectory space can encode surrounding HD-Maps (usually formalized as connected lanelets), using Graph Neural Networks (Gao et al., 2020; Liang et al., 2020; Zeng et al., 2021; Gilles et al., 2021).

The approach most related to this paper is GOHOME (Gilles et al., 2021), which uses a similar graph encoder and then leverage lane rasters to generate a probability heatmap in a sparse manner.

Our goal is to predict the future \(T\) timesteps of \(A\) agents using their past history made of \(H\) timesteps and the HD-Map context.

We first encode each agent trajectory and the HD-Map context graph into a common representation. We then decode a future probability heatmap for each agent in the scene, which we sample heuristically to maximize coverage.

Our pipeline shares the same graph encoder, sampling algorithm and full trajectory generation as GOHOME (Gilles et al., 2021), but uses a novel efficient hierarchical heatmap process that enables to scale to simultaneous multi-agent prediction.

The agent trajectories are encoded though \(TrajEncoder\) using a 1D CNN followed by a UGRU recurrent layer (Rozenberg et al., 2021), and the HD-Map is encoded as a lanelet graph using a GNN \(GraphEncoder\) made of graph convolutions.

The final result is an encoding \(F_{a}\) for each agent, where history, context and interactions have been summarized. This encoding \(F_{a}\) is used in the next decoder operations and is also stored to be potentially used in modality recombination described in Sec. 3.2.

Our aim here is to decode each agent encoding into a heatmap representing its future probability distribution at prediction horizon \(T\).

We use hierarchical predictions at various levels of resolutions so that the decoder has the possibility of predicting over the full surroundings of the agent but learns to refine with more precision only in places where the agent will end up with high probability.

Starting from an initial full dense grid probability at low resolution \(R_{0}\times R_{0}\) by pixels, we iteratively refine the resolution by a fixed factor \(f\) until we reach the desired final resolution \(R_{final}\times R_{final}\). At each iteration \(i\), we select only the \(N_{i}\) highest ranking grid points from the previous iteration, and upsample only these points to the \(R_{i}\times R_{i}=\nicefrac{{R_{i-1}}}{{f}}\times\nicefrac{{R_{i-1}}}{{f}}\). At each step, the grid points features are computed by a 2-layer MLP applied on the point coordinates, these features are then concatenated to the agent encoding and passed through a linear layer, and finally enriched by a 2-layer cross-attention on the graph lane features, before applying a linear layer with sigmoid to get the probability. For a given \(W\) output range, this hierarchical process allows the model to only operate on \(\nicefrac{{W}}{{R_{0}}}\times\nicefrac{{W}}{{R_{0}}}+\sum_{i}N_{i}\times f^{2}\) grid points instead of the \(\nicefrac{{W}}{{R_{final}}}\times\nicefrac{{W}}{{R_{final}}}\) available. In practice, for a final output range of 192 meters with desired \(R_{final}=0.5m\) resolution, we start with an initial resolution of \(R_{0}=8m\) and use two iterations of \((N_{1},N_{2})=(16,64)\) points each and an upscaling factor \(f=4\). This way, we compute only 1856 grid points from the 147 456 available, with no performance loss. The heatmap is trained on each resolution level using as pixel-wise focal loss as in Gilles et al. (2021), detailed in Appendix A.3, with the ground truth being a Gaussian centered at the target agent future position. Having a heatmap as output grants us the good property of letting us decide to privilege coverage in order to give a wide spectrum of candidates to the scene recombination process. From each heatmap, we therefore decode \(K\) end points using the same MissRate optimization algorithm as Gilles et al. (2021). We then generate the full trajectories for each end point using also the same model, a fully-connected MLP. The MLP takes the car history and predicted endpoint as input and produces \(T\) 2D-coordinates representing the full future trajectory.

Initially, the prediction output of the model can be defined as marginal, as all \(A\) agents have been predicted and sampled independently in order to get \(K\) possible endpoint each. Our goal is to output a set of scene predictions \(\mathcal{J}=(L,A)\) from the marginal prediction \(\mathcal{M}=(A,K)\), where each scene modality \(l\) belonging to \(\mathcal{J}\) is an association of endpoints \(p_{l}^{a}\) for each agent \(a\).

The model learns \(L\) scene modality embeddings \(S_{l}\) of \(D\) features each. \(K\times A\) agent modality vectors \(A_{k}^{a}\) are also derived from each agent modality endpoint.

We show significant performance increase when adding the THOMAS module compared to the vanilla model and achieve state-of-the-art results compared to already existing methods tackling scene-consistent predictions.

We use the provided dataset HD-Maps to create the context graphs. As in Liang et al. (2020), we use four relations:\(\{predecessor,successor,left,right\}\) obtained from the lane graph connectivity.

For each agent, we use their trajectory history made of position, yaw and speed in the past second sampled at 10Hz.

The initial paper applies a transformer architecture on a \([F,A,T,D]\) tensor where \(F\) is the potential modality dimension, \(A\) the agent dimension and \(T\) the time dimension, with \(D\) the feature embedding, with factorized self-attention to the agent and time dimensions separately, so that agents can look at each-other inside a specific scene modality.

The initial encoded \([A,D]\) tensor is obtained with a GOHOME encoder, multiplied across the \(F\) futures and concatenated with a modality-specific one-hot encoding as in Ngiam et al. (2021) to obtain the \([F,A,D]\) tensor.

We also display the trade-off between inference speed and coverage from hierarchical refinement in Fig. 7, evaluated on the Interpret multi-agent dataset with marginal MissRate\({}_{6}\). The curve is obtained setting the number \(N\) of upsampled points at the last refinement iteration from 2 to 128.

Input: Figure 7: Curve of MissRate6 with regard to inference time with varying number of points upsampled at the last hierarchical refinement iteration.

Input: Figure 1: Illustration of the THOMAS multi-agent prediction pipeline

Input: Figure 2: Model architecture for multi-agent prediction with shared backbone

Input: Figure 9: Qualitative examples of heatmap output from our multi-agent model. All the heatmaps from one scene are the results from one single forward pass in our model predicting all agents at once. We use matching colors for the agent history, current location and their future predicted heatmap (best viewed in color).

Input: Figure 3: Hierarchical iterative refinement of the grid probabilities. First, the full grid is evaluated at a very low resolution, then the highest cells are up-sampled and evaluated at a higher resolution, until final resolution is reached. We highlight in grey the restricted area considered for refinement at each step.

Input: Figure 8: Detailed illustration of our heatmap generator model

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim appears to be well-supported with detailed technical descriptions and figures showing the model's architecture, hierarchical refinement process, and qualitative results. The evidence demonstrates how THOMAS encodes agent trajectories and HD-Map data through specific neural network components (TrajEncoder, GraphEncoder), and validates the hierarchical heatmap prediction approach through both architectural details and experimental results (Fig. 7 shows speed-coverage tradeoffs). The model components and prediction process align closely with what's stated in the claim without exaggeration.

<score>1</score>

Justification:
1. The evidence provides comprehensive technical details validating each component mentioned in the claim:
- Past trajectory encoding via TrajEncoder and GraphEncoder
- HD-Map lanelet graph encoding using GNN
- Hierarchical heatmap prediction with specific implementation details (R0=8m, N1=16, N2=64, f=4)
- Efficiency claims supported by concrete numbers (1856 vs 147,456 grid points)

2. Figure validation:
- Fig. 2 shows the full pipeline architecture
- Fig. 3 illustrates the hierarchical refinement process
- Fig. 7 quantifies the efficiency benefits
- Fig. 9 demonstrates qualitative results

3. The review comment, while noting some writing ambiguities, acknowledges the paper's technical strengths, particularly regarding the hierarchical heatmap prediction and its efficiency.

4. The claim stays focused on core model capabilities without overstating performance or making unsupported generalizations.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Our THOMAS model encodes the past trajectories of all the agents present in the scene, as well as the HD-Map lanelet graph, and then predicts for each agent a sparse heatmap representing the future probability distribution at a fixed timestep in the future, using hierarchical refinement for very efficient decoding.

The review comment to be evaluated is:

### Strong Points

The idea of performing re-estimation based on the computed heat map end-points
seems novel to me. This idea seems like a natural next step to improve multi agent motion
prediction. The fact that you first get the end-point and then reason about the scene is
intuitively sounding and seems to be a good solution for the problem.

The experiments on interact are a nice direction to be taken by the field.

The visualizations of the heat-maps and the different predictions was helpful.

### Weak Points

#### I could not find a clear distinction with literature
While reading the related work section I found hard to contrast the ideas of the paper with the ones proposed on the literature.
The difference between this method and GOHOME was  not clear to me in this sense.
From what I understood the main contribution here  is the modality combination based on the collision
free end-point sampling. However, this was not obvious for me when reading the paper.

#### The implemented baselines seemed to have performed way worse than expected.
For example, both [1] and [7] are methods designed for *joint* prediction and scene level prediction. This is contradicted
by the observations made by the authors on page 8, quote: " ... This problem is aggravated in the joint training case, since the modality selected is
the same for all agents in a training sample....". Both [1] and [3] fairly confidently claim producing multimodal joint predictions.
Either the version implemented by the authors does not correctly match the other works in the literature or there are specific findings
on the limitations existent in other methods.
Also, the paper lack comparison with latent variable based [2][5][[6] multi modal prediction methods. They seem to perform
better when the issue is diverse prediction. Definitely the mode does not tend to be the same for all predictions in those cases.

#### A single dataset evalution might not be enough for the community to access the value of this paper.

 Motion prediction is currently a very competitive scene. Even though I appreciate the idea on focusing reporting results on multi-agents
and joint prediction tasks, it is important to also evaluate the prediction on more standard benchmarks such as argoverse.
With results on argoverse, me and the rest of the community can more quickly access the the capabilities of the proposed
 heat map prediction pipeline, on producing consistent ego-agent motion prediction. I understand that this would not be the main
results, but it would be helpful.

##### Computational cost is apparently not shown.

I could not find the computational cost of running and training the proposed method on the paper. It seems that even
with the optimizations made on the heat map estimation the training time and inference time should be high.
Those kinds of motion prediction models have the necessity to further scale with way more data. Thus, highly
complex methods can be hard to train. There is also the case of potential future execution in an embedded hardware.

### Specific Questions

I am not  sure about the scene modality vectors ( Section 3.2.2), are they like seed parameters which are going to be used
to incorporate the joint consistent scene level ? I couldn't really find any sort of detailed explanation on that.

Why the lack of ADE for scene level metrics or using similar metrics as [3] ?

[1] Jiquan Ngiam, Benjamin Caine, Vijay Vasudevan, Zhengdong Zhang, Hao-Tien Lewis Chiang,
Jeffrey Ling, Rebecca Roelofs, Alex Bewley, Chenxi Liu, Ashish Venugopal, et al. Scene
transformer: A unified multi-task model for behavior prediction and planning. arXiv preprint
arXiv:2106.08417, 2021
[2] Tang, Charlie, and Russ R. Salakhutdinov. "Multiple futures prediction." Advances in Neural Information Processing Systems 32 (2019): 15424-15434.
[3] Li, Lingyun Luke, et al. "End-to-end contextual perception and prediction with interaction transformer." 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2020.
[4] Yuan, Ye, et al. "AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting." arXiv preprint arXiv:2103.14023 (2021).
[5] Girgis, Roger, et al. "Autobots: Latent Variable Sequential Set Transformers." arXiv preprint arXiv:2104.00563 (2021).
[6] Casas, Sergio, et al. "Implicit latent variable model for scene-consistent motion forecasting." Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28,
[7] Ming Liang, Bin Yang, Rui Hu, Yun Chen, Renjie Liao, Song Feng, and Raquel Urtasun. Learning
lane graph representations for motion forecasting. In ECCV, 2020

The evidence to be evaluated is:Then, benefiting from the latest progresses in computer vision, convolutional layers can easily be applied to bird-view rasters of the map context (Lee et al., 2017; Tang & Salakhutdinov, 2019; Cui et al., 2019; Hong et al., 2019; Salzmann et al., 2020; Chai et al., 2020; Gilles et al., 2021). A more compact representation closer to the trajectory space can encode surrounding HD-Maps (usually formalized as connected lanelets), using Graph Neural Networks (Gao et al., 2020; Liang et al., 2020; Zeng et al., 2021; Gilles et al., 2021).

The approach most related to this paper is GOHOME (Gilles et al., 2021), which uses a similar graph encoder and then leverage lane rasters to generate a probability heatmap in a sparse manner.

Our goal is to predict the future \(T\) timesteps of \(A\) agents using their past history made of \(H\) timesteps and the HD-Map context.

We first encode each agent trajectory and the HD-Map context graph into a common representation. We then decode a future probability heatmap for each agent in the scene, which we sample heuristically to maximize coverage.

Our pipeline shares the same graph encoder, sampling algorithm and full trajectory generation as GOHOME (Gilles et al., 2021), but uses a novel efficient hierarchical heatmap process that enables to scale to simultaneous multi-agent prediction.

The agent trajectories are encoded though \(TrajEncoder\) using a 1D CNN followed by a UGRU recurrent layer (Rozenberg et al., 2021), and the HD-Map is encoded as a lanelet graph using a GNN \(GraphEncoder\) made of graph convolutions.

The final result is an encoding \(F_{a}\) for each agent, where history, context and interactions have been summarized. This encoding \(F_{a}\) is used in the next decoder operations and is also stored to be potentially used in modality recombination described in Sec. 3.2.

Our aim here is to decode each agent encoding into a heatmap representing its future probability distribution at prediction horizon \(T\).

We use hierarchical predictions at various levels of resolutions so that the decoder has the possibility of predicting over the full surroundings of the agent but learns to refine with more precision only in places where the agent will end up with high probability.

Starting from an initial full dense grid probability at low resolution \(R_{0}\times R_{0}\) by pixels, we iteratively refine the resolution by a fixed factor \(f\) until we reach the desired final resolution \(R_{final}\times R_{final}\). At each iteration \(i\), we select only the \(N_{i}\) highest ranking grid points from the previous iteration, and upsample only these points to the \(R_{i}\times R_{i}=\nicefrac{{R_{i-1}}}{{f}}\times\nicefrac{{R_{i-1}}}{{f}}\). At each step, the grid points features are computed by a 2-layer MLP applied on the point coordinates, these features are then concatenated to the agent encoding and passed through a linear layer, and finally enriched by a 2-layer cross-attention on the graph lane features, before applying a linear layer with sigmoid to get the probability. For a given \(W\) output range, this hierarchical process allows the model to only operate on \(\nicefrac{{W}}{{R_{0}}}\times\nicefrac{{W}}{{R_{0}}}+\sum_{i}N_{i}\times f^{2}\) grid points instead of the \(\nicefrac{{W}}{{R_{final}}}\times\nicefrac{{W}}{{R_{final}}}\) available. In practice, for a final output range of 192 meters with desired \(R_{final}=0.5m\) resolution, we start with an initial resolution of \(R_{0}=8m\) and use two iterations of \((N_{1},N_{2})=(16,64)\) points each and an upscaling factor \(f=4\). This way, we compute only 1856 grid points from the 147 456 available, with no performance loss. The heatmap is trained on each resolution level using as pixel-wise focal loss as in Gilles et al. (2021), detailed in Appendix A.3, with the ground truth being a Gaussian centered at the target agent future position. Having a heatmap as output grants us the good property of letting us decide to privilege coverage in order to give a wide spectrum of candidates to the scene recombination process. From each heatmap, we therefore decode \(K\) end points using the same MissRate optimization algorithm as Gilles et al. (2021). We then generate the full trajectories for each end point using also the same model, a fully-connected MLP. The MLP takes the car history and predicted endpoint as input and produces \(T\) 2D-coordinates representing the full future trajectory.

Initially, the prediction output of the model can be defined as marginal, as all \(A\) agents have been predicted and sampled independently in order to get \(K\) possible endpoint each. Our goal is to output a set of scene predictions \(\mathcal{J}=(L,A)\) from the marginal prediction \(\mathcal{M}=(A,K)\), where each scene modality \(l\) belonging to \(\mathcal{J}\) is an association of endpoints \(p_{l}^{a}\) for each agent \(a\).

The model learns \(L\) scene modality embeddings \(S_{l}\) of \(D\) features each. \(K\times A\) agent modality vectors \(A_{k}^{a}\) are also derived from each agent modality endpoint.

We show significant performance increase when adding the THOMAS module compared to the vanilla model and achieve state-of-the-art results compared to already existing methods tackling scene-consistent predictions.

We use the provided dataset HD-Maps to create the context graphs. As in Liang et al. (2020), we use four relations:\(\{predecessor,successor,left,right\}\) obtained from the lane graph connectivity.

For each agent, we use their trajectory history made of position, yaw and speed in the past second sampled at 10Hz.

The initial paper applies a transformer architecture on a \([F,A,T,D]\) tensor where \(F\) is the potential modality dimension, \(A\) the agent dimension and \(T\) the time dimension, with \(D\) the feature embedding, with factorized self-attention to the agent and time dimensions separately, so that agents can look at each-other inside a specific scene modality.

The initial encoded \([A,D]\) tensor is obtained with a GOHOME encoder, multiplied across the \(F\) futures and concatenated with a modality-specific one-hot encoding as in Ngiam et al. (2021) to obtain the \([F,A,D]\) tensor.

We also display the trade-off between inference speed and coverage from hierarchical refinement in Fig. 7, evaluated on the Interpret multi-agent dataset with marginal MissRate\({}_{6}\). The curve is obtained setting the number \(N\) of upsampled points at the last refinement iteration from 2 to 128.

Input: Figure 7: Curve of MissRate6 with regard to inference time with varying number of points upsampled at the last hierarchical refinement iteration.

Input: Figure 1: Illustration of the THOMAS multi-agent prediction pipeline

Input: Figure 2: Model architecture for multi-agent prediction with shared backbone

Input: Figure 9: Qualitative examples of heatmap output from our multi-agent model. All the heatmaps from one scene are the results from one single forward pass in our model predicting all agents at once. We use matching colors for the agent history, current location and their future predicted heatmap (best viewed in color).

Input: Figure 3: Hierarchical iterative refinement of the grid probabilities. First, the full grid is evaluated at a very low resolution, then the highest cells are up-sampled and evaluated at a higher resolution, until final resolution is reached. We highlight in grey the restricted area considered for refinement at each step.

Input: Figure 8: Detailed illustration of our heatmap generator model

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find the claim to be generally well-stated but with minor overstatement. The evidence thoroughly documents the THOMAS model's architecture, including trajectory encoding, HD-Map integration, and hierarchical heatmap prediction. The efficiency claim is supported by Figure 7, showing reasonable inference times, and Figure 3 demonstrating the hierarchical refinement process. However, the claim slightly overstates the efficiency by not acknowledging computational limitations raised in the review comments. The core technical description is accurate but could better qualify the efficiency claims.

<score>2</score>

Justification:
1. The claim accurately describes the core model architecture and prediction approach, supported by detailed technical evidence in the text and figures.
2. The hierarchical refinement process is well-documented with empirical evidence showing computation reduction from "147,456 to 1,856 grid points."
3. The reviewer points out missing computational cost analysis and potential scaling challenges, suggesting the efficiency claims need more qualification.
4. Figures 2, 3, 8 and 9 provide clear visual evidence supporting the model's technical implementation as described.
5. While slightly optimistic about efficiency, the claim stays reasonably grounded in the demonstrated capabilities.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Our THOMAS model encodes the past trajectories of all the agents present in the scene, as well as the HD-Map lanelet graph, and then predicts for each agent a sparse heatmap representing the future probability distribution at a fixed timestep in the future, using hierarchical refinement for very efficient decoding.

The review comment to be evaluated is:

Strengths:
1. The paper is well written and easy to follow.
2. Good performance was achieved on the Interaction multi-agent prediction challenge.

Weaknesses:
1. The method largely follows previous works, with little technical contributions. And even for these limited technical novelties, many of them are not well supported with experimemts.
In the end of introduction section, the authors summarize three main contributions:

i) an efficient graph-based model. However, the implementation mainly follows LaneGCN and GOHOME.

ii) a collision-aware endpoint sampling. The algorithm mainly follows GOHOME, with a small modification in that when we sample an endpoint for actor a modality k, we not only mask out the same region for other modalities of actor a, but also mask out for other actors of modality k. However, this modification doesn't make sense to me, as we still have a scene-consistent modality recombination module afterwards, therefore the second mask-out won't make much difference here. I also don't see an ablation study for this specific change (ie, do not mask-out for other actors of modality k, which means exactly follows the original algorithm in GOHOME). I want to see this ablation for the last three rows in Table 3.

iii) a novel recombination model that could obtain scene-consistent trajectories across the agents. To me this is the core novelty of the paper. From the last two rows in Table 3 we can see that with the proposed recombination module joint metrics improve a lot. However, the marginal metrics degrade a lot (e.g., miss rate more than doubled). The only explanation is that with the proposed module, the same modality (also the sub-optimal one) of an actor is chosen for multiple times in scene-level multi-modalities. To me this may not be what we want: scene-level consistency, but sacrifices actor-level precision a lot. Further analysis is needed here to convince me that this trade-off is worthwhile.

2. Another technical novelty that's not mentioned is the hierarchical iterative refinement of endpoints from the heatmap. However, no ablation study is done on this. I'd like to see a speed-performance comparison with original endpoint estimation model (as in GOHOME).

3. Also, the overall runtime efficiency of the method is not given.

4. GOHOME is the closest approach but I don't see it as a baseline in the experiments.

The evidence to be evaluated is:Then, benefiting from the latest progresses in computer vision, convolutional layers can easily be applied to bird-view rasters of the map context (Lee et al., 2017; Tang & Salakhutdinov, 2019; Cui et al., 2019; Hong et al., 2019; Salzmann et al., 2020; Chai et al., 2020; Gilles et al., 2021). A more compact representation closer to the trajectory space can encode surrounding HD-Maps (usually formalized as connected lanelets), using Graph Neural Networks (Gao et al., 2020; Liang et al., 2020; Zeng et al., 2021; Gilles et al., 2021).

The approach most related to this paper is GOHOME (Gilles et al., 2021), which uses a similar graph encoder and then leverage lane rasters to generate a probability heatmap in a sparse manner.

Our goal is to predict the future \(T\) timesteps of \(A\) agents using their past history made of \(H\) timesteps and the HD-Map context.

We first encode each agent trajectory and the HD-Map context graph into a common representation. We then decode a future probability heatmap for each agent in the scene, which we sample heuristically to maximize coverage.

Our pipeline shares the same graph encoder, sampling algorithm and full trajectory generation as GOHOME (Gilles et al., 2021), but uses a novel efficient hierarchical heatmap process that enables to scale to simultaneous multi-agent prediction.

The agent trajectories are encoded though \(TrajEncoder\) using a 1D CNN followed by a UGRU recurrent layer (Rozenberg et al., 2021), and the HD-Map is encoded as a lanelet graph using a GNN \(GraphEncoder\) made of graph convolutions.

The final result is an encoding \(F_{a}\) for each agent, where history, context and interactions have been summarized. This encoding \(F_{a}\) is used in the next decoder operations and is also stored to be potentially used in modality recombination described in Sec. 3.2.

Our aim here is to decode each agent encoding into a heatmap representing its future probability distribution at prediction horizon \(T\).

We use hierarchical predictions at various levels of resolutions so that the decoder has the possibility of predicting over the full surroundings of the agent but learns to refine with more precision only in places where the agent will end up with high probability.

Starting from an initial full dense grid probability at low resolution \(R_{0}\times R_{0}\) by pixels, we iteratively refine the resolution by a fixed factor \(f\) until we reach the desired final resolution \(R_{final}\times R_{final}\). At each iteration \(i\), we select only the \(N_{i}\) highest ranking grid points from the previous iteration, and upsample only these points to the \(R_{i}\times R_{i}=\nicefrac{{R_{i-1}}}{{f}}\times\nicefrac{{R_{i-1}}}{{f}}\). At each step, the grid points features are computed by a 2-layer MLP applied on the point coordinates, these features are then concatenated to the agent encoding and passed through a linear layer, and finally enriched by a 2-layer cross-attention on the graph lane features, before applying a linear layer with sigmoid to get the probability. For a given \(W\) output range, this hierarchical process allows the model to only operate on \(\nicefrac{{W}}{{R_{0}}}\times\nicefrac{{W}}{{R_{0}}}+\sum_{i}N_{i}\times f^{2}\) grid points instead of the \(\nicefrac{{W}}{{R_{final}}}\times\nicefrac{{W}}{{R_{final}}}\) available. In practice, for a final output range of 192 meters with desired \(R_{final}=0.5m\) resolution, we start with an initial resolution of \(R_{0}=8m\) and use two iterations of \((N_{1},N_{2})=(16,64)\) points each and an upscaling factor \(f=4\). This way, we compute only 1856 grid points from the 147 456 available, with no performance loss. The heatmap is trained on each resolution level using as pixel-wise focal loss as in Gilles et al. (2021), detailed in Appendix A.3, with the ground truth being a Gaussian centered at the target agent future position. Having a heatmap as output grants us the good property of letting us decide to privilege coverage in order to give a wide spectrum of candidates to the scene recombination process. From each heatmap, we therefore decode \(K\) end points using the same MissRate optimization algorithm as Gilles et al. (2021). We then generate the full trajectories for each end point using also the same model, a fully-connected MLP. The MLP takes the car history and predicted endpoint as input and produces \(T\) 2D-coordinates representing the full future trajectory.

Initially, the prediction output of the model can be defined as marginal, as all \(A\) agents have been predicted and sampled independently in order to get \(K\) possible endpoint each. Our goal is to output a set of scene predictions \(\mathcal{J}=(L,A)\) from the marginal prediction \(\mathcal{M}=(A,K)\), where each scene modality \(l\) belonging to \(\mathcal{J}\) is an association of endpoints \(p_{l}^{a}\) for each agent \(a\).

The model learns \(L\) scene modality embeddings \(S_{l}\) of \(D\) features each. \(K\times A\) agent modality vectors \(A_{k}^{a}\) are also derived from each agent modality endpoint.

We show significant performance increase when adding the THOMAS module compared to the vanilla model and achieve state-of-the-art results compared to already existing methods tackling scene-consistent predictions.

We use the provided dataset HD-Maps to create the context graphs. As in Liang et al. (2020), we use four relations:\(\{predecessor,successor,left,right\}\) obtained from the lane graph connectivity.

For each agent, we use their trajectory history made of position, yaw and speed in the past second sampled at 10Hz.

The initial paper applies a transformer architecture on a \([F,A,T,D]\) tensor where \(F\) is the potential modality dimension, \(A\) the agent dimension and \(T\) the time dimension, with \(D\) the feature embedding, with factorized self-attention to the agent and time dimensions separately, so that agents can look at each-other inside a specific scene modality.

The initial encoded \([A,D]\) tensor is obtained with a GOHOME encoder, multiplied across the \(F\) futures and concatenated with a modality-specific one-hot encoding as in Ngiam et al. (2021) to obtain the \([F,A,D]\) tensor.

We also display the trade-off between inference speed and coverage from hierarchical refinement in Fig. 7, evaluated on the Interpret multi-agent dataset with marginal MissRate\({}_{6}\). The curve is obtained setting the number \(N\) of upsampled points at the last refinement iteration from 2 to 128.

Input: Figure 7: Curve of MissRate6 with regard to inference time with varying number of points upsampled at the last hierarchical refinement iteration.

Input: Figure 1: Illustration of the THOMAS multi-agent prediction pipeline

Input: Figure 2: Model architecture for multi-agent prediction with shared backbone

Input: Figure 9: Qualitative examples of heatmap output from our multi-agent model. All the heatmaps from one scene are the results from one single forward pass in our model predicting all agents at once. We use matching colors for the agent history, current location and their future predicted heatmap (best viewed in color).

Input: Figure 3: Hierarchical iterative refinement of the grid probabilities. First, the full grid is evaluated at a very low resolution, then the highest cells are up-sampled and evaluated at a higher resolution, until final resolution is reached. We highlight in grey the restricted area considered for refinement at each step.

Input: Figure 8: Detailed illustration of our heatmap generator model

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, the claim appears slightly overstated. While the evidence shows the model does encode agent trajectories and HD-Map data to predict probability heatmaps using hierarchical refinement, some key considerations make it less than fully well-stated:

1. The model heavily builds upon GOHOME's architecture rather than being entirely novel (explicitly stated in evidence)
2. The reviewer points out limited technical novelty and missing ablation studies
3. The hierarchical refinement, while efficient, lacks comprehensive performance comparisons
4. The evidence shows trade-offs in performance that aren't acknowledged in the claim

The claim would be more accurate if it acknowledged building upon existing work and noted the specific improvements/trade-offs rather than presenting it as a wholly novel approach.

<score>3</score>