Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

In addition, we find that this underestimation behaviour (iv) is weakened, but not eliminated by greater amounts of training data, and (v) is exacerbated for target distributions with lower entropy.

The review comment to be evaluated is:

**Interesting background**. The connections between productivity, low-probability sequences, and the limitations of perplexity were clearly written and interesting.

**Well-executed experiments and methodology**. The idea of using a language model as a ground-truth distribution was interesting and well-suited for the analysis done here. The experiments were easy to follow and the analysis was clear.

**Interesting findings**. The main findings related to underestimation, comparison of training-set and test-set dynamics during training, and dependence on training data were interesting and not obvious. The finding that perturbed, unnatural sequence received unusually high probabilities (while test samples receive unusually low probabilities) [Figure 4] was especially interesting.

**Clarity**. The paper was well-written and enjoyable to read. The authors articulated how their work fits in with related work, and the concepts (e.g. the LNRE zone), methodology, and results are clearly written. Well done.

**Additional experiments**: A good addition would be measuring the impact of model size with models larger than the target distribution. What would GPT2-XL's or GPT-3's probability assignments look like on a target distribution from GPT2-medium?

The evidence to be evaluated is:To understand the training dynamics underlying the previously reported underestimation, we compute model probability estimates on subsets of \(\mathcal{D}_{\text{train}}\) and \(\mathcal{D}_{\text{test}}\) at the end of each training iteration \(i\).

A plausible explanation for the model's underestimation behaviour on unseen test sequences is therefore that the language model has not seen enough samples from the target distribution. Here we explore how estimation error varies as a function of the amount of training data. We train GPT2-medium, GPT2-large and an LSTM model in the online "Ideal World" setting (Nakkiran et al., 2020) by sampling, at the beginning of each training iteration, a fresh set of 500,000 sequences from \(p_{L}\), and training \(M\) on this sample. Doing so for 60 iterations, we obtain LMs which have been trained on 30 million sequences. We compute model estimation error on \(\mathcal{D}_{\text{test}}\) at the end of each iteration \(i\).

The estimation curves in Figure 4 suggest that while increasing the amount of data in training initially leads to lower estimation error, this reduction eventually asymptotes.

In the previous section, we saw that even when increasing the amount of training data, \(p_{M}\) consistently underestimates the probabilities of sequences sampled from the tail of \(p_{L}\).

In the previous section, we saw that \(p_{M}\) placed excess probability mass on areas in \(\Sigma^{*}\) with low-probability under \(p_{L}\).

By increasing the value of \(T\), we increase the entropy of the distributions over next tokens, which in turn, spreads probability mass across a larger number of sequences in \(\Sigma^{*}\). We define four artificial languages with varying \(T\) and train GPT2-medium on an ancestral sample of 1M sequences from each of these artificial languages. Figure 6 visualizes model estimation error by true sequence probability for each model. We find that models trained on languages with increased entropy perform comparatively better than models trained on low entropy languages. Estimation error for models trained on languages with greater \(T\) is less severe, and this holds throughout nearly all target sequence probabilities.

Our results indicate that neural LMs systematically under-represent sequences in the tail of the target distribution, even when increasing the amount of the training data.

Comparing model performance on target distributions with varying properties, we find that neural LMs tend to provide more accurate approximations of distributions with greater entropy.

Therefore, it is possible that some systematic difference between natural language distributions and our ground-truth distributions may affect our results to a certain degree. That being said, our experiments in Section 4.5 suggest that it may actually be easier for neural LMs to learn less structured distributions, and we expect the task of recovering a ground-truth distribution to be made easier when the target distribution and LM are in the same model class. Nevertheless, future work should seek to conduct similar experiments using ground-truth distributions with more explicit structure.

Similarly to our experiments in Section 4.4, we find that the LM underestimates the probability of the majority of sequences, and does so more severely for less probable sequences.

Figure 7 center visualizes mean model estimation error as a function of target sequence probability. We find that \(p_{M}\) overestimates the majority of ill-formed sequences, indicating that these findings hold when the ground-truth distribution is defined using a pretrained model.

In Section 4.3, we explored how estimation error varies as a function of the amount of training data, finding that while increased data weakens estimation error, the underestimation behaviour persists.

Computing estimation error on a set of unseen test sequences, we find, analogously to our experiments on models trained from scratch, that pre-trained models underestimate the probability of the majority of sequences sampled from the target distribution, and do so more severely for rarer sequences (we visualize this in Figure 9). Furthermore, in Figure 10, we increase the amount of fine-tuning data substantially, and we plot test sequence estimation behaviour at the end of each epoch for a pre-trained GPT2-medium model. Once again, increased training data lessens but does eliminate the underestimation behaviour.

Input: Figure 4: GPT2-medium, GPT2-large and LSTM trained on 30M sequences sampled from pL. Main plots: Mean estimation error on test sequences as a function of the number of sequences seen in training. Inset plots: Relative change in mean estimation error of test sequences as a function of the number of sequences seen in training. In both cases, each line denotes estimation behaviour for a 50th of the test sequences; darker lines represent less probable sequences.

Input: Figure 9: Test sequence probability estimates given by pretrained neural LMs fine-tuned on 1M sequences sampled from pL. Three left-most figures: The joint histograms of sequence probability estimates. The dotted line denotes the cases in which the model’s estimates perfectly align with the target probability; shading to the right of this line denotes underestimation. Right-most figure: Mean sequence estimation error by target sequence probability.

Input: Figure 10: Pre-trained GPT2-medium fine-tuned on 30M sequences sampled from pL. Main plots: Mean estimation error on test sequences as a function of the number of sequences seen in finetuning. Inset plots: Relative change in mean estimation error of test sequences as a function of the number of sequences seen in fine-tuning. In both cases, each line denotes estimation behaviour for a 50th of the test sequences; darker lines represent less probable sequences.

Input: Figure 7: Left: Joint histogram of sequence probability estimates for test sequences. Center: Mean model estimation error by true sequence probability for test sequences. Right: Mean model estimation error by true sequence probability for sequences randomly sampled from Σ∗.

Input: Figure 6: Model estimation error on test sequences as a function of target sequence probability for three different artificial languages. Each line visualizes estimation error for a GPT2-medium model trained to model a language with a specific softmax temperature parameter T .

Input: Figure 3: Mean model estimation error by training epoch (GPT2-medium). Each line denotes the mean estimation error for a 50th of the sequences; darker lines represent less probable sequences. The shaded area denotes the area in which validation cross-entropy reaches a minimum.

Input: Figure 8: GPT2-medium and GPT2-large trained on 30M sequences sampled from pL with T = 1.00. Main plots: Mean estimation error on test sequences as a function of the number of sequences seen in training. Inset plots: Relative change in mean estimation error of test sequences as a function of the number of sequences seen in training. In both cases, each line denotes estimation behaviour for a 50th of the test sequences; darker lines represent less probable sequences.

Input: Figure 2: Test sequence probability estimates given by neural LMs. Three left-most figures: The joint histograms of sequence probability estimates. The dotted line denotes the cases in which the model’s estimates perfectly align with the target probability; shading to the right of this line denotes underestimation. Right-most figure: Mean sequence estimation error by target sequence probability.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim about underestimation behavior being "weakened but not eliminated by greater amounts of training data" and being "exacerbated for target distributions with lower entropy" is well-supported by the data. Figures 4, 8, and 10 clearly show that while increased training data reduces estimation error, the underestimation persists even with 30M sequences. Figure 6 directly demonstrates that models trained on languages with lower entropy (lower T values) show more severe estimation errors. The reviewer comments affirm these findings as "interesting and not obvious." The claim carefully reflects the evidence without overstating.

<score>1</score>

Justification:
1. The claim about training data is supported by multiple experiments showing persistent underestimation despite increased data:
- "while increasing the amount of data in training initially leads to lower estimation error, this reduction eventually asymptotes" (with Figures 4, 8, 10 demonstrating this)
- "increased data weakens estimation error, the underestimation behaviour persists"

2. The entropy claim is directly supported by:
- Figure 6 showing more severe estimation errors for lower T values (lower entropy)
- "models trained on languages with increased entropy perform comparatively better than models trained on low entropy languages"
- "Estimation error for models trained on languages with greater T is less severe"

The claim uses measured language ("weakened, but not eliminated") that accurately reflects the evidence without exaggeration.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We validate the decoder on 43,000 NeuroVault images, successfully decoding 65 out of 81 concepts in a zero-shot setting.

The evidence to be evaluated is:Thanks to the rise of large-scale databases of fMRI brain images such as NeuroVault (Gorgolewski et al., 2015) or OpenNeuro (Gorgolewski et al., 2017), decoding can now be performed across multiple studies (Mensch et al., 2017; Walters et al., 2022).

Menuet et al. (2022) leverages the Cognitive Atlas (Poldrack et al., 2011) to improve the label quality of NeuroVault images, enabling the decoding of a large set of concepts.

To solve the minimization problem in eq 1, we leverage a set of statistical maps taken from NeuroVault, an online repository of fMRI maps, from which we extract peak coordinates.

Once trained, Peaks2Image can be applied to any set of peak coordinates from neuroscientific publications. This leads to a combination of both continuous brain images and extensively annotated studies, which opens the possibility to decode a larger set of cognitive concepts.

The figure is divided between the learning of the reconstruction process from peaks (A), the application of the reconstruction to neuroscientific studies (B), and the decoding of brain images with a model trained from those reconstructions in a zero-shot setting.

We use these representations to decode cognitive processes in a zero-shot multi-label setting.

We evaluate the decoding performance on the NeuroVault database (#4438 excluded).

The model is trained from the sole neuroscientific studies, without supervision from any NeuroVault sample or its labels.

In the following experiments, we predict only \(l{=}81\) terms from the 6308 in the tf-idf vocabulary.

We leverage the NeuroVault database.

We consider the remaining 43k brain maps from 2376 different collections to evaluate the decoding performances.

For validation, we restrict the analysis to shared terms between the tags of NeuroVault images and the vocabulary from the tf-idf. Among the 240 tags from the NeuroVault test collections, 189 are present in the vocabulary. We focus on those 189 tags without adding any additional mappings between the left-out tags and words from the vocabulary. Additionally, we filter out terms that contain less than 50 positive examples in the test set to prevent any non-significant effects, keeping 81 terms to decode.

We evaluate the reconstruction produced by Peaks2Image on all NeuroVault except collection #4438 which represents a testing set of around 43k images.

Peaks2Image significantly outperforms linear models. As shown in the next section, although the correlations obtained with the different methods are comparable, the Peaks2Image reconstruction yields a much better performance on the downstream decoding task.

We validate the relevance of the reconstructions by decoding NeuroVault images using reconstructed neuroscientific studies as training set.

We exclude collection #4438 that has been used as training set in the reconstruction task, and evaluate the decoding of all remaining images from NeuroVault.

This results in 43k images, totalling around 200 different cognitive processes. We evaluate only cognitive processes with at least 50 positive examples to avoid results with a too small sample. This reduces the evaluation to 81 terms. The limitation of terms is induced by the evaluation against NeuroVault, but Peaks2Image could extend the prediction to any word included in the tf-idf vocabulary. In Fig. 3, we summarize the decoding performance by aggregating over Cognitive Atlas categories. Peaks2Image performs well over all the main concept families of Cognitive Atlas.

On terms that are shared with NeuroSynth (Yarkoni et al., 2011), we show that Peaks2Image outperforms the former, while largely expanding the number of cognitive processes decoded. We noted an effect of NNoD being able to decode some terms unseen during training.

We introduced Peaks2Image, a model for reconstructing brain images from peak activity coordinates. By reconstructing dense spatial representations, Peaks2Image bridges the gap between neuroscientific literature and brain image databases. Compared to images-based models, it allows to decode a much broader set of cognitive processes by leveraging the rich textual information of neuroscientific studies, along with the reconstructions.

While Peaks2Image could extend to any word contained in the neuroscientific literature, evaluation was reduced to a limited number of concepts due to the lack of extensively annotated brain images.

It shows that one can outrun the limitations of large-scale poor labeling quality by performing zero-shot decoding.

We also apply Peaks2Image to NeuroVault images that are associated to a set of cognitive terms. We compute the average of those reconstructions to show that Peaks2Image generates brain images that are consistent with expected patterns from the literature (fig 7).

In figure 8, we report the per-label decoding performance along with the variance of the results. Most terms are decoded above chance.

Input: Figure 8: Zero-shot decoding performance on NeuroVault: we use Peaks2Image to produce dense brain images for neuroscientific studies. We train a decoder from the generated data on a broad set of cognitive terms. We evaluate the decoding performance on NeuroVault data. Peaks2Image succesfully decodes a large part of those terms in a zero-shot setting. Peaks2Image could extend to any word from the studies’ vocabulary, but could be evaluated on the NeuroVault annotations only.

Input: Figure 7: Reconstructions of NeuroVault samples: we show the average sparse Gaussian representation used in standard meta-analysis procedures (left) along with the Peaks2Image reconstruction (right) for NeuroVault samples containing certain terms. The reconstruction yields more extended networks (audition, face, language production) and sometimes biases the image away from the initial map (sadness). In some rare cases (visual) it focuses the activity patterns.

Input: Figure 3: Zero-shot decoding performance on NeuroVault: evaluation is performed on all NeuroVault except collection #4438. We exclude categories with less than 50 positive examples for stability. Peaks2Image decodes better than chance 65 out of 81 cognitive processes over 43k images (AUC > 0.55).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence carefully, I find the claim "We validate the decoder on 43,000 NeuroVault images, successfully decoding 65 out of 81 concepts in a zero-shot setting" to be well-stated (1). The evidence directly supports this through:

1. Figure 3 explicitly shows decoding performance across categories, stating "Peaks2Image decodes better than chance 65 out of 81 cognitive processes over 43k images (AUC > 0.55)"
2. The methodology is clearly explained, including the data filtering process and evaluation criteria
3. The limitations are acknowledged and the results are presented with appropriate context
4. Figure 8 provides detailed performance metrics for each concept

The claim accurately reflects the evidence without exaggeration.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We validate the decoder on 43,000 NeuroVault images, successfully decoding 65 out of 81 concepts in a zero-shot setting.

The review comment to be evaluated is:

Strengths
1. The authors proposed an interesting workflow. If I understand it correctly, the authors are using the following priors: (1) fMRI brain maps is tightly correlated to the underlying cognitive states, and (2) distribution of peak locations in fMRI brain maps is highly correlated to the overall global pattern in fMRI brain maps. With these priors, the authors proposed to train a neural network model to predict the global fMRI pattern from sparse locations of peaks, and with that trained model they synthesized a paired dataset of (cognitive descriptions, global fMRI patterns) using the (cognitive descriptions, sparse locations of peaks) from the neuroscientific literatures. Finally, they trained a separate decoder with the (cognitive descriptions, global fMRI patterns) pairs to decode the former from the latter. The decoder is the ultimate piece of mastery that can describe the underlying cognitive states of an arbitrary fMRI image.

Weaknesses
1. In my humble opinion, the abstract may need to be better articulated. I was very disoriented and confused by the abstract until I finished reading the introduction section. Specifically, I was confused by (1) what the two complementary sources of data are, (2) why they are complementary, and (3) what is the proposed method trying to accomplish. For (1) and (2), even though the two sources of data are both explicitly mentioned, I did not have enough context to parse their relationship. In contrary, the introduction section has a very natural flow and is painless to follow.
2. It is a bit misleading that when the authors mentioned the word “image” they did not necessarily mean “image”, but rather the “1x1024 vector representation in the DeFuMo space of the image”. I believe it might be helpful to provide a clarifying statement somewhere early in the paper to avoid potential misunderstanding.
3. Figure 1 can be further polished. (1) Figure 1A did not clearly show what data format is provided and what are derived. It might be better if that information is illustrated more explicitly. (2) A lot of things are represented as “blue thin rectangle”, yet some of them are modules/operations while others are entities. It may be better to replace some of them. For example, “Reconstruction” and “decoder” can be replaced by sketches of the neural networks, and “Labels” can be replaced by some clipart object, etc. (3) The caption mentioned that in Figure 1B, the brain map is reconstructed using the trained model, so it would be better to show the sketch of model in an appropriate place and indicate that its weights are frozen.
4. The reconstruction performance (Section 5.1, Figure 2) is quite poor. a correlation of 0.3 does not seem too promising.

Minor Things
1. Is there some typo in the following sentence? Page 5, 3rd paragraph. “When the percentile corresponds to a To avoid selecting studies with a null tf-idf value, we replace the percentile threshold with a strictly positive tf-idf value criteria.”

The evidence to be evaluated is:Thanks to the rise of large-scale databases of fMRI brain images such as NeuroVault (Gorgolewski et al., 2015) or OpenNeuro (Gorgolewski et al., 2017), decoding can now be performed across multiple studies (Mensch et al., 2017; Walters et al., 2022).

Menuet et al. (2022) leverages the Cognitive Atlas (Poldrack et al., 2011) to improve the label quality of NeuroVault images, enabling the decoding of a large set of concepts.

To solve the minimization problem in eq 1, we leverage a set of statistical maps taken from NeuroVault, an online repository of fMRI maps, from which we extract peak coordinates.

Once trained, Peaks2Image can be applied to any set of peak coordinates from neuroscientific publications. This leads to a combination of both continuous brain images and extensively annotated studies, which opens the possibility to decode a larger set of cognitive concepts.

The figure is divided between the learning of the reconstruction process from peaks (A), the application of the reconstruction to neuroscientific studies (B), and the decoding of brain images with a model trained from those reconstructions in a zero-shot setting.

We use these representations to decode cognitive processes in a zero-shot multi-label setting.

We evaluate the decoding performance on the NeuroVault database (#4438 excluded).

The model is trained from the sole neuroscientific studies, without supervision from any NeuroVault sample or its labels.

In the following experiments, we predict only \(l{=}81\) terms from the 6308 in the tf-idf vocabulary.

We leverage the NeuroVault database.

We consider the remaining 43k brain maps from 2376 different collections to evaluate the decoding performances.

For validation, we restrict the analysis to shared terms between the tags of NeuroVault images and the vocabulary from the tf-idf. Among the 240 tags from the NeuroVault test collections, 189 are present in the vocabulary. We focus on those 189 tags without adding any additional mappings between the left-out tags and words from the vocabulary. Additionally, we filter out terms that contain less than 50 positive examples in the test set to prevent any non-significant effects, keeping 81 terms to decode.

We evaluate the reconstruction produced by Peaks2Image on all NeuroVault except collection #4438 which represents a testing set of around 43k images.

Peaks2Image significantly outperforms linear models. As shown in the next section, although the correlations obtained with the different methods are comparable, the Peaks2Image reconstruction yields a much better performance on the downstream decoding task.

We validate the relevance of the reconstructions by decoding NeuroVault images using reconstructed neuroscientific studies as training set.

We exclude collection #4438 that has been used as training set in the reconstruction task, and evaluate the decoding of all remaining images from NeuroVault.

This results in 43k images, totalling around 200 different cognitive processes. We evaluate only cognitive processes with at least 50 positive examples to avoid results with a too small sample. This reduces the evaluation to 81 terms. The limitation of terms is induced by the evaluation against NeuroVault, but Peaks2Image could extend the prediction to any word included in the tf-idf vocabulary. In Fig. 3, we summarize the decoding performance by aggregating over Cognitive Atlas categories. Peaks2Image performs well over all the main concept families of Cognitive Atlas.

On terms that are shared with NeuroSynth (Yarkoni et al., 2011), we show that Peaks2Image outperforms the former, while largely expanding the number of cognitive processes decoded. We noted an effect of NNoD being able to decode some terms unseen during training.

We introduced Peaks2Image, a model for reconstructing brain images from peak activity coordinates. By reconstructing dense spatial representations, Peaks2Image bridges the gap between neuroscientific literature and brain image databases. Compared to images-based models, it allows to decode a much broader set of cognitive processes by leveraging the rich textual information of neuroscientific studies, along with the reconstructions.

While Peaks2Image could extend to any word contained in the neuroscientific literature, evaluation was reduced to a limited number of concepts due to the lack of extensively annotated brain images.

It shows that one can outrun the limitations of large-scale poor labeling quality by performing zero-shot decoding.

We also apply Peaks2Image to NeuroVault images that are associated to a set of cognitive terms. We compute the average of those reconstructions to show that Peaks2Image generates brain images that are consistent with expected patterns from the literature (fig 7).

In figure 8, we report the per-label decoding performance along with the variance of the results. Most terms are decoded above chance.

Input: Figure 8: Zero-shot decoding performance on NeuroVault: we use Peaks2Image to produce dense brain images for neuroscientific studies. We train a decoder from the generated data on a broad set of cognitive terms. We evaluate the decoding performance on NeuroVault data. Peaks2Image succesfully decodes a large part of those terms in a zero-shot setting. Peaks2Image could extend to any word from the studies’ vocabulary, but could be evaluated on the NeuroVault annotations only.

Input: Figure 7: Reconstructions of NeuroVault samples: we show the average sparse Gaussian representation used in standard meta-analysis procedures (left) along with the Peaks2Image reconstruction (right) for NeuroVault samples containing certain terms. The reconstruction yields more extended networks (audition, face, language production) and sometimes biases the image away from the initial map (sadness). In some rare cases (visual) it focuses the activity patterns.

Input: Figure 3: Zero-shot decoding performance on NeuroVault: evaluation is performed on all NeuroVault except collection #4438. We exclude categories with less than 50 positive examples for stability. Peaks2Image decodes better than chance 65 out of 81 cognitive processes over 43k images (AUC > 0.55).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim "We validate the decoder on 43,000 NeuroVault images, successfully decoding 65 out of 81 concepts in a zero-shot setting" appears to be well-stated and accurately reflects the data presented. The evidence shows:

- Clear validation on ~43k NeuroVault images (excluding collection #4438)
- Figure 3 explicitly states "Peaks2Image decodes better than chance 65 out of 81 cognitive processes over 43k images (AUC > 0.55)"
- Figure 8 provides detailed performance metrics per concept
- The evaluation criteria (AUC > 0.55 as "better than chance") is clearly defined

The claim directly corresponds to the results without exaggeration or overstatement.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We validate the decoder on 43,000 NeuroVault images, successfully decoding 65 out of 81 concepts in a zero-shot setting.

The review comment to be evaluated is:

This paper contains several controversial claims.

The paper uses Nilearn to estimate peaks and then uses these peaks to learn a model from data. However, there is no evaluation of the error associated with the selection of these peaks. Further, what evidence shows that you are more likely to see neural responses in these peaks than examining the whole brain data?

Another concern is the proposed formulation. As an example, what is the definition of the function f() in Equation 1? To learn f*, what constraints are considered? The same question can be asked for D in Equation 2. Due to the fact that D without any constraint may increase the chance of overfitting.

Empirical studies are another concern. The proposed method only compares with some classic machine learning approaches, such as Kernel Ridge or Space Gaussian. There are several groups of state-of-the-art techniques that aim at similar types of analysis and should be compared with the proposed approach, e.g., HTFA (Manning 2018), BRSA (Cai 2016), and DRD (Wu 2019).

In addition to the aforementioned concerns, the proposed approach should clarify some related concepts for multi-subject and multi-resource analysis. It seems that this study uses multi-subject data. However, I could not find anywhere that discusses how the author(s) functionally align brain responses from different subjects. Alternatively, Neuro Vault is a multi-resource repository where data are provided from different machines and environments. I also did not find any information on how the proposed method deals with bach effects to learn a model from multi-site data. You may provide additional information in a preprocessing/preparation section.

The evidence to be evaluated is:Thanks to the rise of large-scale databases of fMRI brain images such as NeuroVault (Gorgolewski et al., 2015) or OpenNeuro (Gorgolewski et al., 2017), decoding can now be performed across multiple studies (Mensch et al., 2017; Walters et al., 2022).

Menuet et al. (2022) leverages the Cognitive Atlas (Poldrack et al., 2011) to improve the label quality of NeuroVault images, enabling the decoding of a large set of concepts.

To solve the minimization problem in eq 1, we leverage a set of statistical maps taken from NeuroVault, an online repository of fMRI maps, from which we extract peak coordinates.

Once trained, Peaks2Image can be applied to any set of peak coordinates from neuroscientific publications. This leads to a combination of both continuous brain images and extensively annotated studies, which opens the possibility to decode a larger set of cognitive concepts.

The figure is divided between the learning of the reconstruction process from peaks (A), the application of the reconstruction to neuroscientific studies (B), and the decoding of brain images with a model trained from those reconstructions in a zero-shot setting.

We use these representations to decode cognitive processes in a zero-shot multi-label setting.

We evaluate the decoding performance on the NeuroVault database (#4438 excluded).

The model is trained from the sole neuroscientific studies, without supervision from any NeuroVault sample or its labels.

In the following experiments, we predict only \(l{=}81\) terms from the 6308 in the tf-idf vocabulary.

We leverage the NeuroVault database.

We consider the remaining 43k brain maps from 2376 different collections to evaluate the decoding performances.

For validation, we restrict the analysis to shared terms between the tags of NeuroVault images and the vocabulary from the tf-idf. Among the 240 tags from the NeuroVault test collections, 189 are present in the vocabulary. We focus on those 189 tags without adding any additional mappings between the left-out tags and words from the vocabulary. Additionally, we filter out terms that contain less than 50 positive examples in the test set to prevent any non-significant effects, keeping 81 terms to decode.

We evaluate the reconstruction produced by Peaks2Image on all NeuroVault except collection #4438 which represents a testing set of around 43k images.

Peaks2Image significantly outperforms linear models. As shown in the next section, although the correlations obtained with the different methods are comparable, the Peaks2Image reconstruction yields a much better performance on the downstream decoding task.

We validate the relevance of the reconstructions by decoding NeuroVault images using reconstructed neuroscientific studies as training set.

We exclude collection #4438 that has been used as training set in the reconstruction task, and evaluate the decoding of all remaining images from NeuroVault.

This results in 43k images, totalling around 200 different cognitive processes. We evaluate only cognitive processes with at least 50 positive examples to avoid results with a too small sample. This reduces the evaluation to 81 terms. The limitation of terms is induced by the evaluation against NeuroVault, but Peaks2Image could extend the prediction to any word included in the tf-idf vocabulary. In Fig. 3, we summarize the decoding performance by aggregating over Cognitive Atlas categories. Peaks2Image performs well over all the main concept families of Cognitive Atlas.

On terms that are shared with NeuroSynth (Yarkoni et al., 2011), we show that Peaks2Image outperforms the former, while largely expanding the number of cognitive processes decoded. We noted an effect of NNoD being able to decode some terms unseen during training.

We introduced Peaks2Image, a model for reconstructing brain images from peak activity coordinates. By reconstructing dense spatial representations, Peaks2Image bridges the gap between neuroscientific literature and brain image databases. Compared to images-based models, it allows to decode a much broader set of cognitive processes by leveraging the rich textual information of neuroscientific studies, along with the reconstructions.

While Peaks2Image could extend to any word contained in the neuroscientific literature, evaluation was reduced to a limited number of concepts due to the lack of extensively annotated brain images.

It shows that one can outrun the limitations of large-scale poor labeling quality by performing zero-shot decoding.

We also apply Peaks2Image to NeuroVault images that are associated to a set of cognitive terms. We compute the average of those reconstructions to show that Peaks2Image generates brain images that are consistent with expected patterns from the literature (fig 7).

In figure 8, we report the per-label decoding performance along with the variance of the results. Most terms are decoded above chance.

Input: Figure 8: Zero-shot decoding performance on NeuroVault: we use Peaks2Image to produce dense brain images for neuroscientific studies. We train a decoder from the generated data on a broad set of cognitive terms. We evaluate the decoding performance on NeuroVault data. Peaks2Image succesfully decodes a large part of those terms in a zero-shot setting. Peaks2Image could extend to any word from the studies’ vocabulary, but could be evaluated on the NeuroVault annotations only.

Input: Figure 7: Reconstructions of NeuroVault samples: we show the average sparse Gaussian representation used in standard meta-analysis procedures (left) along with the Peaks2Image reconstruction (right) for NeuroVault samples containing certain terms. The reconstruction yields more extended networks (audition, face, language production) and sometimes biases the image away from the initial map (sadness). In some rare cases (visual) it focuses the activity patterns.

Input: Figure 3: Zero-shot decoding performance on NeuroVault: evaluation is performed on all NeuroVault except collection #4438. We exclude categories with less than 50 positive examples for stability. Peaks2Image decodes better than chance 65 out of 81 cognitive processes over 43k images (AUC > 0.55).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After careful review, the claim "We validate the decoder on 43,000 NeuroVault images, successfully decoding 65 out of 81 concepts in a zero-shot setting" appears to be well-stated and accurately reflects the evidence. The evidence shows:

- Clear documentation of testing on ~43k NeuroVault images (excluding collection #4438)
- Explicit reporting of decoding 65 out of 81 cognitive processes above chance (AUC > 0.55)
- Detailed figures showing performance across categories (Fig 3) and individual terms (Fig 8)
- Appropriate acknowledgment of limitations and evaluation constraints

While the reviewer raises valid methodological concerns about preprocessing and comparisons, these don't directly challenge the specific numerical claims made about decoder validation results.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We validate the decoder on 43,000 NeuroVault images, successfully decoding 65 out of 81 concepts in a zero-shot setting.

The review comment to be evaluated is:

Strength:
1.This paper is well-written and easy to read. The definition of tasks and problems are very clear.
2.For a decoding task, the idea that fully utilize the rich descriptions of the studied cognitive tasks in scientific report is novel.
Weaknesses
1.My major concern is the necessity of image reconstruction from peaks. As illustrated in the paper, the authors train a reconstruction network to convert the peak coordinates into images. In my opinion, this ill-posed problem is too hard to be reasonable. Reconstructing the fMRI from only a few peak coordinates (less than 100 according to the paper) seems impossible to recover the details of images. Thus, the reconstruction is unnecessary. If the peak coordinates contain enough information for the following decoding tasks, reconstruction would introduce inevitable noise. If the peak coordinates are not enough, reconstruction still can’t help the decoding since extra information comes from the ‘guessing’ of the trained network. Can the authors give more explanation about this?
2.Also, the experiments do not verify that image reconstruction can help in improving the decoding performance. The ablation study should compare with model using peak coordinates as input. So, I am wondering about the performance of model that uses peak coordinates as input.
3.Experimental results are all shown in figures, which is good for analysis. However, giving some extra tables with exact values can be better.

The evidence to be evaluated is:Thanks to the rise of large-scale databases of fMRI brain images such as NeuroVault (Gorgolewski et al., 2015) or OpenNeuro (Gorgolewski et al., 2017), decoding can now be performed across multiple studies (Mensch et al., 2017; Walters et al., 2022).

Menuet et al. (2022) leverages the Cognitive Atlas (Poldrack et al., 2011) to improve the label quality of NeuroVault images, enabling the decoding of a large set of concepts.

To solve the minimization problem in eq 1, we leverage a set of statistical maps taken from NeuroVault, an online repository of fMRI maps, from which we extract peak coordinates.

Once trained, Peaks2Image can be applied to any set of peak coordinates from neuroscientific publications. This leads to a combination of both continuous brain images and extensively annotated studies, which opens the possibility to decode a larger set of cognitive concepts.

The figure is divided between the learning of the reconstruction process from peaks (A), the application of the reconstruction to neuroscientific studies (B), and the decoding of brain images with a model trained from those reconstructions in a zero-shot setting.

We use these representations to decode cognitive processes in a zero-shot multi-label setting.

We evaluate the decoding performance on the NeuroVault database (#4438 excluded).

The model is trained from the sole neuroscientific studies, without supervision from any NeuroVault sample or its labels.

In the following experiments, we predict only \(l{=}81\) terms from the 6308 in the tf-idf vocabulary.

We leverage the NeuroVault database.

We consider the remaining 43k brain maps from 2376 different collections to evaluate the decoding performances.

For validation, we restrict the analysis to shared terms between the tags of NeuroVault images and the vocabulary from the tf-idf. Among the 240 tags from the NeuroVault test collections, 189 are present in the vocabulary. We focus on those 189 tags without adding any additional mappings between the left-out tags and words from the vocabulary. Additionally, we filter out terms that contain less than 50 positive examples in the test set to prevent any non-significant effects, keeping 81 terms to decode.

We evaluate the reconstruction produced by Peaks2Image on all NeuroVault except collection #4438 which represents a testing set of around 43k images.

Peaks2Image significantly outperforms linear models. As shown in the next section, although the correlations obtained with the different methods are comparable, the Peaks2Image reconstruction yields a much better performance on the downstream decoding task.

We validate the relevance of the reconstructions by decoding NeuroVault images using reconstructed neuroscientific studies as training set.

We exclude collection #4438 that has been used as training set in the reconstruction task, and evaluate the decoding of all remaining images from NeuroVault.

This results in 43k images, totalling around 200 different cognitive processes. We evaluate only cognitive processes with at least 50 positive examples to avoid results with a too small sample. This reduces the evaluation to 81 terms. The limitation of terms is induced by the evaluation against NeuroVault, but Peaks2Image could extend the prediction to any word included in the tf-idf vocabulary. In Fig. 3, we summarize the decoding performance by aggregating over Cognitive Atlas categories. Peaks2Image performs well over all the main concept families of Cognitive Atlas.

On terms that are shared with NeuroSynth (Yarkoni et al., 2011), we show that Peaks2Image outperforms the former, while largely expanding the number of cognitive processes decoded. We noted an effect of NNoD being able to decode some terms unseen during training.

We introduced Peaks2Image, a model for reconstructing brain images from peak activity coordinates. By reconstructing dense spatial representations, Peaks2Image bridges the gap between neuroscientific literature and brain image databases. Compared to images-based models, it allows to decode a much broader set of cognitive processes by leveraging the rich textual information of neuroscientific studies, along with the reconstructions.

While Peaks2Image could extend to any word contained in the neuroscientific literature, evaluation was reduced to a limited number of concepts due to the lack of extensively annotated brain images.

It shows that one can outrun the limitations of large-scale poor labeling quality by performing zero-shot decoding.

We also apply Peaks2Image to NeuroVault images that are associated to a set of cognitive terms. We compute the average of those reconstructions to show that Peaks2Image generates brain images that are consistent with expected patterns from the literature (fig 7).

In figure 8, we report the per-label decoding performance along with the variance of the results. Most terms are decoded above chance.

Input: Figure 8: Zero-shot decoding performance on NeuroVault: we use Peaks2Image to produce dense brain images for neuroscientific studies. We train a decoder from the generated data on a broad set of cognitive terms. We evaluate the decoding performance on NeuroVault data. Peaks2Image succesfully decodes a large part of those terms in a zero-shot setting. Peaks2Image could extend to any word from the studies’ vocabulary, but could be evaluated on the NeuroVault annotations only.

Input: Figure 7: Reconstructions of NeuroVault samples: we show the average sparse Gaussian representation used in standard meta-analysis procedures (left) along with the Peaks2Image reconstruction (right) for NeuroVault samples containing certain terms. The reconstruction yields more extended networks (audition, face, language production) and sometimes biases the image away from the initial map (sadness). In some rare cases (visual) it focuses the activity patterns.

Input: Figure 3: Zero-shot decoding performance on NeuroVault: evaluation is performed on all NeuroVault except collection #4438. We exclude categories with less than 50 positive examples for stability. Peaks2Image decodes better than chance 65 out of 81 cognitive processes over 43k images (AUC > 0.55).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim that "We validate the decoder on 43,000 NeuroVault images, successfully decoding 65 out of 81 concepts in a zero-shot setting" appears to be well-stated and accurately reflects the evidence. Figure 3 directly supports this by showing decoding performance on ~43k NeuroVault images (excluding collection #4438) with 65 out of 81 cognitive processes decoded above chance (AUC > 0.55). The claim clearly states the scope, sample size, and results without exaggeration. The review comments do not contradict this specific claim about decoding performance.

<score>1</score>

Justification:
1. The claim directly matches quantitative evidence shown in Figure 3 and explained in the text
2. No exaggeration of results - clear presentation of numbers (65/81 concepts)
3. Appropriate context provided (zero-shot setting, validation on NeuroVault)
4. Sample size accurately stated (43,000 images)
5. While reviewer had concerns about image reconstruction, those comments don't impact the validity of this specific claim about decoding performance


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

In this work, Peaks2Image learns to discriminate dozens of cognitive terms on the largest available image repository, NeuroVault.

The evidence to be evaluated is:Dockes et al. (2020) broadened the spectrum of terms analyzed by mapping rare concepts to more common cognitive terms. Ngo et al. (2021) leveraged language models (Beltagy et al., 2019) to capture term relationships semantically, leading to the encoding of any query.

Thanks to the rise of large-scale databases of fMRI brain images such as NeuroVault (Gorgolewski et al., 2015) or OpenNeuro (Gorgolewski et al., 2017), decoding can now be performed across multiple studies (Mensch et al., 2017; Walters et al., 2022).

Menuet et al. (2022) leverages the Cognitive Atlas (Poldrack et al., 2011) to improve the label quality of NeuroVault images, enabling the decoding of a large set of concepts.

In particular, we found no attempt at merging rich textual information from neuroscientific studies with the dense spatial information of brain images. We focus on this particular aspect, proving that obtaining image representations for neuroscientific studies helps overcome the limitations of CBMA by generating a corpus with both rich textual annotations and dense spatial representations.

To solve the minimization problem in eq 1, we leverage a set of statistical maps taken from NeuroVault, an online repository of fMRI maps, from which we extract peak coordinates.

This leads to a combination of both continuous brain images and extensively annotated studies, which opens the possibility to decode a larger set of cognitive concepts.

Peaks2Image consists of a neural network that reconstructs brain images from activity peaks.

To train this reconstruction process, we use around 6k images from the #4438 NeuroVault collection ([https://neurovault.org/collections/4438](https://neurovault.org/collections/4438)).

By doing this on the most comprehensive public literature corpus, we reconstruct 13,000 brain images from the peak activation tables. We use these representations to decode cognitive processes in a zero-shot multi-label setting.

We evaluate the decoding performance on the NeuroVault database (#4438 excluded).

We leverage a 3-layer dense neural network to predict the set of binary labels \(\mathbf{L}=\{0,1\}_{1}\times\cdots\times\{0,1\}_{l}\), \(l\) denoting the total number of classes, from the input image \(\mathbf{X}\in\mathbb{R}^{d}\) compressed in DiFuMo space.

Peaks2Image could extend to a set of cognitive terms as large as the vocabulary size without any additional cost. We leverage the NeuroVault database. We exclude a subset of images according to a set of rules such as having incoherent range of values (indicating that an image does not contain Z or T statistics), duplicated images or images with missing metadata to keep around 50k brain maps for our task.

We consider the remaining 43k brain maps from 2376 different collections to evaluate the decoding performances.

For validation, we restrict the analysis to shared terms between the tags of NeuroVault images and the vocabulary from the tf-idf. Among the 240 tags from the NeuroVault test collections, 189 are present in the vocabulary. We focus on those 189 tags without adding any additional mappings between the left-out tags and words from the vocabulary. Additionally, we filter out terms that contain less than 50 positive examples in the test set to prevent any non-significant effects, keeping 81 terms to decode.

Images were fetched from NeuroVault using the Nileam library (Abraham et al., 2014). We resample all images to the standard MNI152 template and apply the MNI152 brain mask.

We generate pairs of matching (peaks, image) pairs from the collection (#4438) from NeuroVault images only.

We use Nileam's method for peak extraction, which first separatesthe images into statistically active clusters and retrieves peak coordinates from each cluster. As these images represent statistical maps, equivalent to p-values of voxel-level tests, we keep peaks with a p-value lower than \(10^{-3}\), and with a minimal distance between peaks within the same cluster of 8mm. These are standard parameters from peak extraction in brain imaging contexts.

From a given set of peaks, we build a sparse Gaussian representation by placing Gaussian kernels at each peak position with a 9mm full-width at half-maximum, following what is done in Dockes et al. (2020). Similarly to the NeuroVault images, we extract the DiFuMo representations of the sparse Gaussian maps using Nilearn.

We evaluate the reconstruction produced by Peaks2Image on all NeuroVault except collection #4438 which represents a testing set of around 43k images.

We apply label enrichment to the original image labels. This includes regular expressions to map different terminologies for a same concept, and a hierarchy-based inference of labels based on the Cognitive Atlas ontology. For a given term, we assign parent terms from the ontology used in (Menuet et al., 2022). This results in 43k images, totalling around 200 different cognitive processes.

The limitation of terms is induced by the evaluation against NeuroVault, but Peaks2Image could extend the prediction to any word included in the tf-idf vocabulary.

Peaks2Image performs well over all the main concept families of Cognitive Atlas.

On terms that are shared with NeuroSynth (Yarkoni et al., 2011), we show that Peaks2Image outperforms the former, while largely expanding the number of cognitive processes decoded.

By reconstructing dense spatial representations, Peaks2Image bridges the gap between neuroscientific literature and brain image databases. Compared to images-based models, it allows to decode a much broader set of cognitive processes by leveraging the rich textual information of neuroscientific studies, along with the reconstructions. Most importantly, the decoding performance is reached without any labeling requirements on the images side, avoiding a time-consuming and error-prone task.

While Peaks2Image could extend to any word contained in the neuroscientific literature, evaluation was reduced to a limited number of concepts due to the lack of extensively annotated brain images.

Peaks2Image paves the way towards better generation of large sets of images for neuroscientific studies.

We also apply Peaks2Image to NeuroVault images that are associated to a set of cognitive terms. We compute the average of those reconstructions to show that Peaks2Image generates brain images that are consistent with expected patterns from the literature (fig 7).

In figure 8, we report the per-label decoding performance along with the variance of the results. Most terms are decoded above chance.

Input: Figure 8: Zero-shot decoding performance on NeuroVault: we use Peaks2Image to produce dense brain images for neuroscientific studies. We train a decoder from the generated data on a broad set of cognitive terms. We evaluate the decoding performance on NeuroVault data. Peaks2Image succesfully decodes a large part of those terms in a zero-shot setting. Peaks2Image could extend to any word from the studies’ vocabulary, but could be evaluated on the NeuroVault annotations only.

Input: Figure 7: Reconstructions of NeuroVault samples: we show the average sparse Gaussian representation used in standard meta-analysis procedures (left) along with the Peaks2Image reconstruction (right) for NeuroVault samples containing certain terms. The reconstruction yields more extended networks (audition, face, language production) and sometimes biases the image away from the initial map (sadness). In some rare cases (visual) it focuses the activity patterns.

Input: Figure 3: Zero-shot decoding performance on NeuroVault: evaluation is performed on all NeuroVault except collection #4438. We exclude categories with less than 50 positive examples for stability. Peaks2Image decodes better than chance 65 out of 81 cognitive processes over 43k images (AUC > 0.55).

Input: Figure 5: AUC performance of our reconstructions vs NeuroVault samples: we compare the performance of the NNoD architecture on NeuroVault (except collection #4438 used for the reconstruction training) in two settings: training on NeuroVault data (collection 4438) or training on reconstructions with percentile-inferred labels. NeuroVault training outperforms when it contains the term considered, but Peaks2Image provides a better guess when no such training data are available.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find the claim "Peaks2Image learns to discriminate dozens of cognitive terms on the largest available image repository, NeuroVault" to be generally well-stated but slightly overstated.

The evidence shows that:
- Out of 189 possible tags, analysis was restricted to 81 terms (those with >50 positive examples)
- Of these 81 terms, Peaks2Image decoded 65 terms above chance level (AUC > 0.55)
- Figure 3 shows varying performance across cognitive domains
- Some limitations are acknowledged regarding evaluation being reduced to a limited number of concepts

While the model does discriminate dozens of terms (65), describing this as occurring on "the largest available repository" without qualifying the substantial filtering of terms could be seen as slightly overstating the scope.

<score>2</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

In this work, Peaks2Image learns to discriminate dozens of cognitive terms on the largest available image repository, NeuroVault.

The review comment to be evaluated is:

Strengths
1. The authors proposed an interesting workflow. If I understand it correctly, the authors are using the following priors: (1) fMRI brain maps is tightly correlated to the underlying cognitive states, and (2) distribution of peak locations in fMRI brain maps is highly correlated to the overall global pattern in fMRI brain maps. With these priors, the authors proposed to train a neural network model to predict the global fMRI pattern from sparse locations of peaks, and with that trained model they synthesized a paired dataset of (cognitive descriptions, global fMRI patterns) using the (cognitive descriptions, sparse locations of peaks) from the neuroscientific literatures. Finally, they trained a separate decoder with the (cognitive descriptions, global fMRI patterns) pairs to decode the former from the latter. The decoder is the ultimate piece of mastery that can describe the underlying cognitive states of an arbitrary fMRI image.

Weaknesses
1. In my humble opinion, the abstract may need to be better articulated. I was very disoriented and confused by the abstract until I finished reading the introduction section. Specifically, I was confused by (1) what the two complementary sources of data are, (2) why they are complementary, and (3) what is the proposed method trying to accomplish. For (1) and (2), even though the two sources of data are both explicitly mentioned, I did not have enough context to parse their relationship. In contrary, the introduction section has a very natural flow and is painless to follow.
2. It is a bit misleading that when the authors mentioned the word “image” they did not necessarily mean “image”, but rather the “1x1024 vector representation in the DeFuMo space of the image”. I believe it might be helpful to provide a clarifying statement somewhere early in the paper to avoid potential misunderstanding.
3. Figure 1 can be further polished. (1) Figure 1A did not clearly show what data format is provided and what are derived. It might be better if that information is illustrated more explicitly. (2) A lot of things are represented as “blue thin rectangle”, yet some of them are modules/operations while others are entities. It may be better to replace some of them. For example, “Reconstruction” and “decoder” can be replaced by sketches of the neural networks, and “Labels” can be replaced by some clipart object, etc. (3) The caption mentioned that in Figure 1B, the brain map is reconstructed using the trained model, so it would be better to show the sketch of model in an appropriate place and indicate that its weights are frozen.
4. The reconstruction performance (Section 5.1, Figure 2) is quite poor. a correlation of 0.3 does not seem too promising.

Minor Things
1. Is there some typo in the following sentence? Page 5, 3rd paragraph. “When the percentile corresponds to a To avoid selecting studies with a null tf-idf value, we replace the percentile threshold with a strictly positive tf-idf value criteria.”

The evidence to be evaluated is:Dockes et al. (2020) broadened the spectrum of terms analyzed by mapping rare concepts to more common cognitive terms. Ngo et al. (2021) leveraged language models (Beltagy et al., 2019) to capture term relationships semantically, leading to the encoding of any query.

Thanks to the rise of large-scale databases of fMRI brain images such as NeuroVault (Gorgolewski et al., 2015) or OpenNeuro (Gorgolewski et al., 2017), decoding can now be performed across multiple studies (Mensch et al., 2017; Walters et al., 2022).

Menuet et al. (2022) leverages the Cognitive Atlas (Poldrack et al., 2011) to improve the label quality of NeuroVault images, enabling the decoding of a large set of concepts.

In particular, we found no attempt at merging rich textual information from neuroscientific studies with the dense spatial information of brain images. We focus on this particular aspect, proving that obtaining image representations for neuroscientific studies helps overcome the limitations of CBMA by generating a corpus with both rich textual annotations and dense spatial representations.

To solve the minimization problem in eq 1, we leverage a set of statistical maps taken from NeuroVault, an online repository of fMRI maps, from which we extract peak coordinates.

This leads to a combination of both continuous brain images and extensively annotated studies, which opens the possibility to decode a larger set of cognitive concepts.

Peaks2Image consists of a neural network that reconstructs brain images from activity peaks.

To train this reconstruction process, we use around 6k images from the #4438 NeuroVault collection ([https://neurovault.org/collections/4438](https://neurovault.org/collections/4438)).

By doing this on the most comprehensive public literature corpus, we reconstruct 13,000 brain images from the peak activation tables. We use these representations to decode cognitive processes in a zero-shot multi-label setting.

We evaluate the decoding performance on the NeuroVault database (#4438 excluded).

We leverage a 3-layer dense neural network to predict the set of binary labels \(\mathbf{L}=\{0,1\}_{1}\times\cdots\times\{0,1\}_{l}\), \(l\) denoting the total number of classes, from the input image \(\mathbf{X}\in\mathbb{R}^{d}\) compressed in DiFuMo space.

Peaks2Image could extend to a set of cognitive terms as large as the vocabulary size without any additional cost. We leverage the NeuroVault database. We exclude a subset of images according to a set of rules such as having incoherent range of values (indicating that an image does not contain Z or T statistics), duplicated images or images with missing metadata to keep around 50k brain maps for our task.

We consider the remaining 43k brain maps from 2376 different collections to evaluate the decoding performances.

For validation, we restrict the analysis to shared terms between the tags of NeuroVault images and the vocabulary from the tf-idf. Among the 240 tags from the NeuroVault test collections, 189 are present in the vocabulary. We focus on those 189 tags without adding any additional mappings between the left-out tags and words from the vocabulary. Additionally, we filter out terms that contain less than 50 positive examples in the test set to prevent any non-significant effects, keeping 81 terms to decode.

Images were fetched from NeuroVault using the Nileam library (Abraham et al., 2014). We resample all images to the standard MNI152 template and apply the MNI152 brain mask.

We generate pairs of matching (peaks, image) pairs from the collection (#4438) from NeuroVault images only.

We use Nileam's method for peak extraction, which first separatesthe images into statistically active clusters and retrieves peak coordinates from each cluster. As these images represent statistical maps, equivalent to p-values of voxel-level tests, we keep peaks with a p-value lower than \(10^{-3}\), and with a minimal distance between peaks within the same cluster of 8mm. These are standard parameters from peak extraction in brain imaging contexts.

From a given set of peaks, we build a sparse Gaussian representation by placing Gaussian kernels at each peak position with a 9mm full-width at half-maximum, following what is done in Dockes et al. (2020). Similarly to the NeuroVault images, we extract the DiFuMo representations of the sparse Gaussian maps using Nilearn.

We evaluate the reconstruction produced by Peaks2Image on all NeuroVault except collection #4438 which represents a testing set of around 43k images.

We apply label enrichment to the original image labels. This includes regular expressions to map different terminologies for a same concept, and a hierarchy-based inference of labels based on the Cognitive Atlas ontology. For a given term, we assign parent terms from the ontology used in (Menuet et al., 2022). This results in 43k images, totalling around 200 different cognitive processes.

The limitation of terms is induced by the evaluation against NeuroVault, but Peaks2Image could extend the prediction to any word included in the tf-idf vocabulary.

Peaks2Image performs well over all the main concept families of Cognitive Atlas.

On terms that are shared with NeuroSynth (Yarkoni et al., 2011), we show that Peaks2Image outperforms the former, while largely expanding the number of cognitive processes decoded.

By reconstructing dense spatial representations, Peaks2Image bridges the gap between neuroscientific literature and brain image databases. Compared to images-based models, it allows to decode a much broader set of cognitive processes by leveraging the rich textual information of neuroscientific studies, along with the reconstructions. Most importantly, the decoding performance is reached without any labeling requirements on the images side, avoiding a time-consuming and error-prone task.

While Peaks2Image could extend to any word contained in the neuroscientific literature, evaluation was reduced to a limited number of concepts due to the lack of extensively annotated brain images.

Peaks2Image paves the way towards better generation of large sets of images for neuroscientific studies.

We also apply Peaks2Image to NeuroVault images that are associated to a set of cognitive terms. We compute the average of those reconstructions to show that Peaks2Image generates brain images that are consistent with expected patterns from the literature (fig 7).

In figure 8, we report the per-label decoding performance along with the variance of the results. Most terms are decoded above chance.

Input: Figure 8: Zero-shot decoding performance on NeuroVault: we use Peaks2Image to produce dense brain images for neuroscientific studies. We train a decoder from the generated data on a broad set of cognitive terms. We evaluate the decoding performance on NeuroVault data. Peaks2Image succesfully decodes a large part of those terms in a zero-shot setting. Peaks2Image could extend to any word from the studies’ vocabulary, but could be evaluated on the NeuroVault annotations only.

Input: Figure 7: Reconstructions of NeuroVault samples: we show the average sparse Gaussian representation used in standard meta-analysis procedures (left) along with the Peaks2Image reconstruction (right) for NeuroVault samples containing certain terms. The reconstruction yields more extended networks (audition, face, language production) and sometimes biases the image away from the initial map (sadness). In some rare cases (visual) it focuses the activity patterns.

Input: Figure 3: Zero-shot decoding performance on NeuroVault: evaluation is performed on all NeuroVault except collection #4438. We exclude categories with less than 50 positive examples for stability. Peaks2Image decodes better than chance 65 out of 81 cognitive processes over 43k images (AUC > 0.55).

Input: Figure 5: AUC performance of our reconstructions vs NeuroVault samples: we compare the performance of the NNoD architecture on NeuroVault (except collection #4438 used for the reconstruction training) in two settings: training on NeuroVault data (collection 4438) or training on reconstructions with percentile-inferred labels. NeuroVault training outperforms when it contains the term considered, but Peaks2Image provides a better guess when no such training data are available.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, this appears to be somewhat overstated. While Peaks2Image does demonstrate decoding capabilities across multiple cognitive terms (81 terms shown in Figure 8), calling it "dozens" understates the actual scope. The evidence shows that out of these 81 terms, 65 were decoded above chance level (AUC > 0.55). However, labeling the dataset as "largest available" is unsupported in the evidence - while NeuroVault is referenced extensively, no explicit comparison of repository sizes is provided to justify this superlative claim.

The review comments don't specifically address the claim about size or number of terms. Looking at the evidence against the claim components:
- "dozens of terms": Somewhat understated given 65+ terms
- "largest available repository": Unsupported claim
- "learns to discriminate": Generally supported by performance data

<score>3</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

In this work, Peaks2Image learns to discriminate dozens of cognitive terms on the largest available image repository, NeuroVault.

The review comment to be evaluated is:

This paper contains several controversial claims.

The paper uses Nilearn to estimate peaks and then uses these peaks to learn a model from data. However, there is no evaluation of the error associated with the selection of these peaks. Further, what evidence shows that you are more likely to see neural responses in these peaks than examining the whole brain data?

Another concern is the proposed formulation. As an example, what is the definition of the function f() in Equation 1? To learn f*, what constraints are considered? The same question can be asked for D in Equation 2. Due to the fact that D without any constraint may increase the chance of overfitting.

Empirical studies are another concern. The proposed method only compares with some classic machine learning approaches, such as Kernel Ridge or Space Gaussian. There are several groups of state-of-the-art techniques that aim at similar types of analysis and should be compared with the proposed approach, e.g., HTFA (Manning 2018), BRSA (Cai 2016), and DRD (Wu 2019).

In addition to the aforementioned concerns, the proposed approach should clarify some related concepts for multi-subject and multi-resource analysis. It seems that this study uses multi-subject data. However, I could not find anywhere that discusses how the author(s) functionally align brain responses from different subjects. Alternatively, Neuro Vault is a multi-resource repository where data are provided from different machines and environments. I also did not find any information on how the proposed method deals with bach effects to learn a model from multi-site data. You may provide additional information in a preprocessing/preparation section.

The evidence to be evaluated is:Dockes et al. (2020) broadened the spectrum of terms analyzed by mapping rare concepts to more common cognitive terms. Ngo et al. (2021) leveraged language models (Beltagy et al., 2019) to capture term relationships semantically, leading to the encoding of any query.

Thanks to the rise of large-scale databases of fMRI brain images such as NeuroVault (Gorgolewski et al., 2015) or OpenNeuro (Gorgolewski et al., 2017), decoding can now be performed across multiple studies (Mensch et al., 2017; Walters et al., 2022).

Menuet et al. (2022) leverages the Cognitive Atlas (Poldrack et al., 2011) to improve the label quality of NeuroVault images, enabling the decoding of a large set of concepts.

In particular, we found no attempt at merging rich textual information from neuroscientific studies with the dense spatial information of brain images. We focus on this particular aspect, proving that obtaining image representations for neuroscientific studies helps overcome the limitations of CBMA by generating a corpus with both rich textual annotations and dense spatial representations.

To solve the minimization problem in eq 1, we leverage a set of statistical maps taken from NeuroVault, an online repository of fMRI maps, from which we extract peak coordinates.

This leads to a combination of both continuous brain images and extensively annotated studies, which opens the possibility to decode a larger set of cognitive concepts.

Peaks2Image consists of a neural network that reconstructs brain images from activity peaks.

To train this reconstruction process, we use around 6k images from the #4438 NeuroVault collection ([https://neurovault.org/collections/4438](https://neurovault.org/collections/4438)).

By doing this on the most comprehensive public literature corpus, we reconstruct 13,000 brain images from the peak activation tables. We use these representations to decode cognitive processes in a zero-shot multi-label setting.

We evaluate the decoding performance on the NeuroVault database (#4438 excluded).

We leverage a 3-layer dense neural network to predict the set of binary labels \(\mathbf{L}=\{0,1\}_{1}\times\cdots\times\{0,1\}_{l}\), \(l\) denoting the total number of classes, from the input image \(\mathbf{X}\in\mathbb{R}^{d}\) compressed in DiFuMo space.

Peaks2Image could extend to a set of cognitive terms as large as the vocabulary size without any additional cost. We leverage the NeuroVault database. We exclude a subset of images according to a set of rules such as having incoherent range of values (indicating that an image does not contain Z or T statistics), duplicated images or images with missing metadata to keep around 50k brain maps for our task.

We consider the remaining 43k brain maps from 2376 different collections to evaluate the decoding performances.

For validation, we restrict the analysis to shared terms between the tags of NeuroVault images and the vocabulary from the tf-idf. Among the 240 tags from the NeuroVault test collections, 189 are present in the vocabulary. We focus on those 189 tags without adding any additional mappings between the left-out tags and words from the vocabulary. Additionally, we filter out terms that contain less than 50 positive examples in the test set to prevent any non-significant effects, keeping 81 terms to decode.

Images were fetched from NeuroVault using the Nileam library (Abraham et al., 2014). We resample all images to the standard MNI152 template and apply the MNI152 brain mask.

We generate pairs of matching (peaks, image) pairs from the collection (#4438) from NeuroVault images only.

We use Nileam's method for peak extraction, which first separatesthe images into statistically active clusters and retrieves peak coordinates from each cluster. As these images represent statistical maps, equivalent to p-values of voxel-level tests, we keep peaks with a p-value lower than \(10^{-3}\), and with a minimal distance between peaks within the same cluster of 8mm. These are standard parameters from peak extraction in brain imaging contexts.

From a given set of peaks, we build a sparse Gaussian representation by placing Gaussian kernels at each peak position with a 9mm full-width at half-maximum, following what is done in Dockes et al. (2020). Similarly to the NeuroVault images, we extract the DiFuMo representations of the sparse Gaussian maps using Nilearn.

We evaluate the reconstruction produced by Peaks2Image on all NeuroVault except collection #4438 which represents a testing set of around 43k images.

We apply label enrichment to the original image labels. This includes regular expressions to map different terminologies for a same concept, and a hierarchy-based inference of labels based on the Cognitive Atlas ontology. For a given term, we assign parent terms from the ontology used in (Menuet et al., 2022). This results in 43k images, totalling around 200 different cognitive processes.

The limitation of terms is induced by the evaluation against NeuroVault, but Peaks2Image could extend the prediction to any word included in the tf-idf vocabulary.

Peaks2Image performs well over all the main concept families of Cognitive Atlas.

On terms that are shared with NeuroSynth (Yarkoni et al., 2011), we show that Peaks2Image outperforms the former, while largely expanding the number of cognitive processes decoded.

By reconstructing dense spatial representations, Peaks2Image bridges the gap between neuroscientific literature and brain image databases. Compared to images-based models, it allows to decode a much broader set of cognitive processes by leveraging the rich textual information of neuroscientific studies, along with the reconstructions. Most importantly, the decoding performance is reached without any labeling requirements on the images side, avoiding a time-consuming and error-prone task.

While Peaks2Image could extend to any word contained in the neuroscientific literature, evaluation was reduced to a limited number of concepts due to the lack of extensively annotated brain images.

Peaks2Image paves the way towards better generation of large sets of images for neuroscientific studies.

We also apply Peaks2Image to NeuroVault images that are associated to a set of cognitive terms. We compute the average of those reconstructions to show that Peaks2Image generates brain images that are consistent with expected patterns from the literature (fig 7).

In figure 8, we report the per-label decoding performance along with the variance of the results. Most terms are decoded above chance.

Input: Figure 8: Zero-shot decoding performance on NeuroVault: we use Peaks2Image to produce dense brain images for neuroscientific studies. We train a decoder from the generated data on a broad set of cognitive terms. We evaluate the decoding performance on NeuroVault data. Peaks2Image succesfully decodes a large part of those terms in a zero-shot setting. Peaks2Image could extend to any word from the studies’ vocabulary, but could be evaluated on the NeuroVault annotations only.

Input: Figure 7: Reconstructions of NeuroVault samples: we show the average sparse Gaussian representation used in standard meta-analysis procedures (left) along with the Peaks2Image reconstruction (right) for NeuroVault samples containing certain terms. The reconstruction yields more extended networks (audition, face, language production) and sometimes biases the image away from the initial map (sadness). In some rare cases (visual) it focuses the activity patterns.

Input: Figure 3: Zero-shot decoding performance on NeuroVault: evaluation is performed on all NeuroVault except collection #4438. We exclude categories with less than 50 positive examples for stability. Peaks2Image decodes better than chance 65 out of 81 cognitive processes over 43k images (AUC > 0.55).

Input: Figure 5: AUC performance of our reconstructions vs NeuroVault samples: we compare the performance of the NNoD architecture on NeuroVault (except collection #4438 used for the reconstruction training) in two settings: training on NeuroVault data (collection 4438) or training on reconstructions with percentile-inferred labels. NeuroVault training outperforms when it contains the term considered, but Peaks2Image provides a better guess when no such training data are available.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I assess that it is moderately overstated. While Peaks2Image does demonstrate ability to decode cognitive terms from NeuroVault data, several key limitations exist:

1. The paper evaluates only 81 terms (not "dozens") after filtering for terms with >50 positive examples
2. The reviewer raises valid concerns about lack of validation for peak selection methods and handling of multi-subject/multi-site data
3. Figures show variable performance across terms, with some performing poorly
4. Limited comparison to state-of-the-art methods as noted in review
5. The claim doesn't acknowledge these important limitations

<score>4</score>

The claim would be more accurate if it specified the actual number of reliably decoded terms and acknowledged the methodological limitations raised in the review. The evidence shows the method works for some terms but has significant constraints that are not reflected in the broad wording of the claim.