Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Using those reconstructions, we train a decoder using tf-idf features as labels, leading to a much broader set of decoded terms than current image-based studies.

The review comment to be evaluated is:

Strengths
1. The authors proposed an interesting workflow. If I understand it correctly, the authors are using the following priors: (1) fMRI brain maps is tightly correlated to the underlying cognitive states, and (2) distribution of peak locations in fMRI brain maps is highly correlated to the overall global pattern in fMRI brain maps. With these priors, the authors proposed to train a neural network model to predict the global fMRI pattern from sparse locations of peaks, and with that trained model they synthesized a paired dataset of (cognitive descriptions, global fMRI patterns) using the (cognitive descriptions, sparse locations of peaks) from the neuroscientific literatures. Finally, they trained a separate decoder with the (cognitive descriptions, global fMRI patterns) pairs to decode the former from the latter. The decoder is the ultimate piece of mastery that can describe the underlying cognitive states of an arbitrary fMRI image.

Weaknesses
1. In my humble opinion, the abstract may need to be better articulated. I was very disoriented and confused by the abstract until I finished reading the introduction section. Specifically, I was confused by (1) what the two complementary sources of data are, (2) why they are complementary, and (3) what is the proposed method trying to accomplish. For (1) and (2), even though the two sources of data are both explicitly mentioned, I did not have enough context to parse their relationship. In contrary, the introduction section has a very natural flow and is painless to follow.
2. It is a bit misleading that when the authors mentioned the word “image” they did not necessarily mean “image”, but rather the “1x1024 vector representation in the DeFuMo space of the image”. I believe it might be helpful to provide a clarifying statement somewhere early in the paper to avoid potential misunderstanding.
3. Figure 1 can be further polished. (1) Figure 1A did not clearly show what data format is provided and what are derived. It might be better if that information is illustrated more explicitly. (2) A lot of things are represented as “blue thin rectangle”, yet some of them are modules/operations while others are entities. It may be better to replace some of them. For example, “Reconstruction” and “decoder” can be replaced by sketches of the neural networks, and “Labels” can be replaced by some clipart object, etc. (3) The caption mentioned that in Figure 1B, the brain map is reconstructed using the trained model, so it would be better to show the sketch of model in an appropriate place and indicate that its weights are frozen.
4. The reconstruction performance (Section 5.1, Figure 2) is quite poor. a correlation of 0.3 does not seem too promising.

Minor Things
1. Is there some typo in the following sentence? Page 5, 3rd paragraph. “When the percentile corresponds to a To avoid selecting studies with a null tf-idf value, we replace the percentile threshold with a strictly positive tf-idf value criteria.”

The evidence to be evaluated is:Dockes et al. (2020) broadened the spectrum of terms analyzed by mapping rare concepts to more common cognitive terms. Ngo et al. (2021) leveraged language models (Beltagy et al., 2019) to capture term relationships semantically, leading to the encoding of any query.

Menuet et al. (2022) leverages the Cognitive Atlas (Poldrack et al., 2011) to improve the label quality of NeuroVault images, enabling the decoding of a large set of concepts.

Although there has been interest in enriching brain images to increase the amount of data (Zhuang et al., 2019), to generate or reconstruct fMRI images (Manning et al., 2018; Cai et al., 2016), there has been limited interest in generating reconstructions of statistical brain maps from peaks (Gorgolewski et al., 2019), probably due to the daunting data dimensionality. In particular, we found no attempt at merging rich textual information from neuroscientific studies with the dense spatial information of brain images. We focus on this particular aspect, proving that obtaining image representations for neuroscientific studies helps overcome the limitations of CBMA by generating a corpus with both rich textual annotations and dense spatial representations.

We aim to reconstruct brain images from those few datapoints. This reconstruction problem is an inverse problem between the set of peaks and the image from which they were extracted. From a given set of \(l\) peaks \(\mathbf{P_{y}}\in\mathbb{R}^{3\times l}\) extracted from an image \(\mathbf{y}\in\mathbb{R}^{m}\) with \(m\) voxels, we learn the transformation \(f:\mathbb{R}^{3\times l}\mapsto\mathbb{R}^{m}\) by minimization of the \(\mathbf{L_{2}}\) error over the original image: \[f^{*}=\operatorname*{arg\,min}_{f}\mathbf{E_{y}}\|f(\mathbf{P_{y}})-\mathbf{ y}\|_{2}^{2} \tag{1}\]

Once trained, Peaks2Image can be applied to any set of peak coordinates from neuroscientific publications. This leads to a combination of both continuous brain images and extensively annotated studies, which opens the possibility to decode a larger set of cognitive concepts.

The figure is divided between the learning of the reconstruction process from peaks (A), the application of the reconstruction to neuroscientific studies (B), and the decoding of brain images with a model trained from those reconstructions in a zero-shot setting.

We use the Dictionary of Functional Modes (DiFuMo) probabilistic atlas (Dadi et al., 2020), that efficiently reduces the \(m=50,000\) voxel values to \(k=1024\) components. DiFuMo has been trained on thousands of resting-state and task fMRI with non-negativity and sparsity constraints, making this atlas relevant to represent population-level information.

Mapping voxels \(\mathbf{x}\in\mathbb{R}^{m}\) to coefficients \(\alpha\in R^{k}\) along DiFuMo dictionary \(\mathbf{D}\in\mathbb{R}^{m\times k}\) of dimension \(k\) is a mere linear regression problem following equation 2.

To train this reconstruction process, we use around 6k images from the #4438 NeuroVault collection ([https://neurovault.org/collections/4438](https://neurovault.org/collections/4438)). The reconstructed images are obtained from the predicted DiFuMo components by linear combinations of the dictionary components: \(\hat{\mathbf{y}}=\mathbf{D}f^{*}(\alpha(\mathbf{x}_{\mathbf{y}}))\).

By doing this on the most comprehensive public literature corpus, we reconstruct 13,000 brain images from the peak activation tables.

We extract the term frequency-inverse document frequency (tf-idf) (Salton and Buckley, 1988) from the studies' text. The tf-idf reflects the relevance of a specific term to a study by giving higher power to terms that occurs repetitively in few studies. Based on it, we assign labels to studies, by setting a threshold at the 95th percentile of the tf-idf. This ensures that the few studies that are most relevant to each term get positive labels.

We leverage a 3-layer dense neural network to predict the set of binary labels \(\mathbf{L}=\{0,1\}_{1}\times\cdots\times\{0,1\}_{l}\), \(l\) denoting the total number of classes, from the input image \(\mathbf{X}\in\mathbb{R}^{d}\) compressed in DiFuMo space.

In the following experiments, we predict only \(l{=}81\) terms from the 6308 in the tf-idf vocabulary. Peaks2Image could extend to a set of cognitive terms as large as the vocabulary size without any additional cost.

It also transforms each study text into tf-idf features across a vocabulary of 6308 terms. For validation, we restrict the analysis to shared terms between the tags of NeuroVault images and the vocabulary from the tf-idf. Among the 240 tags from the NeuroVault test collections, 189 are present in the vocabulary. We focus on those 189 tags without adding any additional mappings between the left-out tags and words from the vocabulary. Additionally, we filter out terms that contain less than 50 positive examples in the test set to prevent any non-significant effects, keeping 81 terms to decode.

The decoder model is an MLP composed of one hidden layer of 300 units and ReLU activation. It is trained for 200 epochs on the reconstructed neuroscientific studies using the Adam optimizer with a learning rate of 3e-3 on the binary cross-entropy loss.

As shown in the next section, although the correlations obtained with the different methods are comparable, the Peaks2Image reconstruction yields a much better performance on the downstream decoding task.

We validate the relevance of the reconstructions by decoding NeuroVault images using reconstructed neuroscientific studies as training set. The problem is a multi-label classification, which we evaluate by computing the ROC AUC over each label. We exclude collection #4438 that has been used as training set in the reconstruction task, and evaluate the decoding of all remaining images from NeuroVault. We apply label enrichment to the original image labels. This includes regular expressions to map different terminologies for a same concept, and a hierarchy-based inference of labels based on the Cognitive Atlas ontology. For a given term, we assign parent terms from the ontology used in (Menuet et al., 2022). This results in 43k images, totalling around 200 different cognitive processes.

The limitation of terms is induced by the evaluation against NeuroVault, but Peaks2Image could extend the prediction to any word included in the tf-idf vocabulary.

We show that, even though the correlation of sparse Gaussian representations remains close to Peaks2Image reconstruction, Peaks2Image largely outperforms the sparse Gaussian model on the decoding task.

On terms that are shared with NeuroSynth (Yarkoni et al., 2011), we show that Peaks2Image outperforms the former, while largely expanding the number of cognitive processes decoded.

Compared to images-based models, it allows to decode a much broader set of cognitive processes by leveraging the rich textual information of neuroscientific studies, along with the reconstructions. Most importantly, the decoding performance is reached without any labeling requirements on the images side, avoiding a time-consuming and error-prone task.

While Peaks2Image could extend to any word contained in the neuroscientific literature, evaluation was reduced to a limited number of concepts due to the lack of extensively annotated brain images. Exploration of more powerful language models as in Ngo et al. (2021) is a future direction to better benefit from the continuous brain images generated for the neuroscientific studies.

Peaks2Image paves the way towards better generation of large sets of images for neuroscientific studies. It shows that one can outrun the limitations of large-scale poor labeling quality by performing zero-shot decoding.

We also apply Peaks2Image to NeuroVault images that are associated to a set of cognitive terms.

In figure 8, we report the per-label decoding performance along with the variance of the results. Most terms are decoded above chance.

Input: Figure 8: Zero-shot decoding performance on NeuroVault: we use Peaks2Image to produce dense brain images for neuroscientific studies. We train a decoder from the generated data on a broad set of cognitive terms. We evaluate the decoding performance on NeuroVault data. Peaks2Image succesfully decodes a large part of those terms in a zero-shot setting. Peaks2Image could extend to any word from the studies’ vocabulary, but could be evaluated on the NeuroVault annotations only.

Input: Figure 7: Reconstructions of NeuroVault samples: we show the average sparse Gaussian representation used in standard meta-analysis procedures (left) along with the Peaks2Image reconstruction (right) for NeuroVault samples containing certain terms. The reconstruction yields more extended networks (audition, face, language production) and sometimes biases the image away from the initial map (sadness). In some rare cases (visual) it focuses the activity patterns.

Input: Figure 1: Reconstructing Brain Activation Maps from Activation Peaks (A) We learn the reconstruction process from NeuroVault collection #4438. After selection and preprocessing of the images, we extract the peaks that are above a statistical threshold. From those peaks, we build an intermediate representation by placing a Gaussian kernel at each peak as in Dockès et al. (2020); Ngo et al. (2021). We compress the representation using the DiFuMo atlas. We regress the original DiFuMo components of each image by leveraging a 3-layer multi-layer perceptron (MLP). (B) Once the reconstruction model is trained, we build intermediate Gaussian-kernel representations for the scraped studies, and reconstruct a brain map for each study using the above model, supporting further image-level computations such as decoding. From the text, we extract the tf-idf values for each word, and infer labels on a percentile-based threshold. (C) We perform decoding of NeuroVault images by training on the newly acquired representations of neuroscientific studies. We decode a broad set of cognitive processes benefiting from the large vocabulary.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim "Using those reconstructions, we train a decoder using tf-idf features as labels, leading to a much broader set of decoded terms than current image-based studies" is well-supported by the evidence. The paper demonstrates:
- Training on 6,308 tf-idf vocabulary terms, though evaluated on 81 terms due to validation constraints
- Ability to extend to any word in the vocabulary without additional cost
- Outperformance compared to NeuroSynth while expanding decoded processes
- Zero-shot decoding of multiple cognitive terms as shown in Figure 8
- Clear evidence of broader term coverage through reconstructions

The claim accurately reflects the capabilities and limitations without overstating the results. The review comments do not contradict this specific claim.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Using those reconstructions, we train a decoder using tf-idf features as labels, leading to a much broader set of decoded terms than current image-based studies.

The review comment to be evaluated is:

This paper contains several controversial claims.

The paper uses Nilearn to estimate peaks and then uses these peaks to learn a model from data. However, there is no evaluation of the error associated with the selection of these peaks. Further, what evidence shows that you are more likely to see neural responses in these peaks than examining the whole brain data?

Another concern is the proposed formulation. As an example, what is the definition of the function f() in Equation 1? To learn f*, what constraints are considered? The same question can be asked for D in Equation 2. Due to the fact that D without any constraint may increase the chance of overfitting.

Empirical studies are another concern. The proposed method only compares with some classic machine learning approaches, such as Kernel Ridge or Space Gaussian. There are several groups of state-of-the-art techniques that aim at similar types of analysis and should be compared with the proposed approach, e.g., HTFA (Manning 2018), BRSA (Cai 2016), and DRD (Wu 2019).

In addition to the aforementioned concerns, the proposed approach should clarify some related concepts for multi-subject and multi-resource analysis. It seems that this study uses multi-subject data. However, I could not find anywhere that discusses how the author(s) functionally align brain responses from different subjects. Alternatively, Neuro Vault is a multi-resource repository where data are provided from different machines and environments. I also did not find any information on how the proposed method deals with bach effects to learn a model from multi-site data. You may provide additional information in a preprocessing/preparation section.

The evidence to be evaluated is:Dockes et al. (2020) broadened the spectrum of terms analyzed by mapping rare concepts to more common cognitive terms. Ngo et al. (2021) leveraged language models (Beltagy et al., 2019) to capture term relationships semantically, leading to the encoding of any query.

Menuet et al. (2022) leverages the Cognitive Atlas (Poldrack et al., 2011) to improve the label quality of NeuroVault images, enabling the decoding of a large set of concepts.

Although there has been interest in enriching brain images to increase the amount of data (Zhuang et al., 2019), to generate or reconstruct fMRI images (Manning et al., 2018; Cai et al., 2016), there has been limited interest in generating reconstructions of statistical brain maps from peaks (Gorgolewski et al., 2019), probably due to the daunting data dimensionality. In particular, we found no attempt at merging rich textual information from neuroscientific studies with the dense spatial information of brain images. We focus on this particular aspect, proving that obtaining image representations for neuroscientific studies helps overcome the limitations of CBMA by generating a corpus with both rich textual annotations and dense spatial representations.

We aim to reconstruct brain images from those few datapoints. This reconstruction problem is an inverse problem between the set of peaks and the image from which they were extracted. From a given set of \(l\) peaks \(\mathbf{P_{y}}\in\mathbb{R}^{3\times l}\) extracted from an image \(\mathbf{y}\in\mathbb{R}^{m}\) with \(m\) voxels, we learn the transformation \(f:\mathbb{R}^{3\times l}\mapsto\mathbb{R}^{m}\) by minimization of the \(\mathbf{L_{2}}\) error over the original image: \[f^{*}=\operatorname*{arg\,min}_{f}\mathbf{E_{y}}\|f(\mathbf{P_{y}})-\mathbf{ y}\|_{2}^{2} \tag{1}\]

Once trained, Peaks2Image can be applied to any set of peak coordinates from neuroscientific publications. This leads to a combination of both continuous brain images and extensively annotated studies, which opens the possibility to decode a larger set of cognitive concepts.

The figure is divided between the learning of the reconstruction process from peaks (A), the application of the reconstruction to neuroscientific studies (B), and the decoding of brain images with a model trained from those reconstructions in a zero-shot setting.

We use the Dictionary of Functional Modes (DiFuMo) probabilistic atlas (Dadi et al., 2020), that efficiently reduces the \(m=50,000\) voxel values to \(k=1024\) components. DiFuMo has been trained on thousands of resting-state and task fMRI with non-negativity and sparsity constraints, making this atlas relevant to represent population-level information.

Mapping voxels \(\mathbf{x}\in\mathbb{R}^{m}\) to coefficients \(\alpha\in R^{k}\) along DiFuMo dictionary \(\mathbf{D}\in\mathbb{R}^{m\times k}\) of dimension \(k\) is a mere linear regression problem following equation 2.

To train this reconstruction process, we use around 6k images from the #4438 NeuroVault collection ([https://neurovault.org/collections/4438](https://neurovault.org/collections/4438)). The reconstructed images are obtained from the predicted DiFuMo components by linear combinations of the dictionary components: \(\hat{\mathbf{y}}=\mathbf{D}f^{*}(\alpha(\mathbf{x}_{\mathbf{y}}))\).

By doing this on the most comprehensive public literature corpus, we reconstruct 13,000 brain images from the peak activation tables.

We extract the term frequency-inverse document frequency (tf-idf) (Salton and Buckley, 1988) from the studies' text. The tf-idf reflects the relevance of a specific term to a study by giving higher power to terms that occurs repetitively in few studies. Based on it, we assign labels to studies, by setting a threshold at the 95th percentile of the tf-idf. This ensures that the few studies that are most relevant to each term get positive labels.

We leverage a 3-layer dense neural network to predict the set of binary labels \(\mathbf{L}=\{0,1\}_{1}\times\cdots\times\{0,1\}_{l}\), \(l\) denoting the total number of classes, from the input image \(\mathbf{X}\in\mathbb{R}^{d}\) compressed in DiFuMo space.

In the following experiments, we predict only \(l{=}81\) terms from the 6308 in the tf-idf vocabulary. Peaks2Image could extend to a set of cognitive terms as large as the vocabulary size without any additional cost.

It also transforms each study text into tf-idf features across a vocabulary of 6308 terms. For validation, we restrict the analysis to shared terms between the tags of NeuroVault images and the vocabulary from the tf-idf. Among the 240 tags from the NeuroVault test collections, 189 are present in the vocabulary. We focus on those 189 tags without adding any additional mappings between the left-out tags and words from the vocabulary. Additionally, we filter out terms that contain less than 50 positive examples in the test set to prevent any non-significant effects, keeping 81 terms to decode.

The decoder model is an MLP composed of one hidden layer of 300 units and ReLU activation. It is trained for 200 epochs on the reconstructed neuroscientific studies using the Adam optimizer with a learning rate of 3e-3 on the binary cross-entropy loss.

As shown in the next section, although the correlations obtained with the different methods are comparable, the Peaks2Image reconstruction yields a much better performance on the downstream decoding task.

We validate the relevance of the reconstructions by decoding NeuroVault images using reconstructed neuroscientific studies as training set. The problem is a multi-label classification, which we evaluate by computing the ROC AUC over each label. We exclude collection #4438 that has been used as training set in the reconstruction task, and evaluate the decoding of all remaining images from NeuroVault. We apply label enrichment to the original image labels. This includes regular expressions to map different terminologies for a same concept, and a hierarchy-based inference of labels based on the Cognitive Atlas ontology. For a given term, we assign parent terms from the ontology used in (Menuet et al., 2022). This results in 43k images, totalling around 200 different cognitive processes.

The limitation of terms is induced by the evaluation against NeuroVault, but Peaks2Image could extend the prediction to any word included in the tf-idf vocabulary.

We show that, even though the correlation of sparse Gaussian representations remains close to Peaks2Image reconstruction, Peaks2Image largely outperforms the sparse Gaussian model on the decoding task.

On terms that are shared with NeuroSynth (Yarkoni et al., 2011), we show that Peaks2Image outperforms the former, while largely expanding the number of cognitive processes decoded.

Compared to images-based models, it allows to decode a much broader set of cognitive processes by leveraging the rich textual information of neuroscientific studies, along with the reconstructions. Most importantly, the decoding performance is reached without any labeling requirements on the images side, avoiding a time-consuming and error-prone task.

While Peaks2Image could extend to any word contained in the neuroscientific literature, evaluation was reduced to a limited number of concepts due to the lack of extensively annotated brain images. Exploration of more powerful language models as in Ngo et al. (2021) is a future direction to better benefit from the continuous brain images generated for the neuroscientific studies.

Peaks2Image paves the way towards better generation of large sets of images for neuroscientific studies. It shows that one can outrun the limitations of large-scale poor labeling quality by performing zero-shot decoding.

We also apply Peaks2Image to NeuroVault images that are associated to a set of cognitive terms.

In figure 8, we report the per-label decoding performance along with the variance of the results. Most terms are decoded above chance.

Input: Figure 8: Zero-shot decoding performance on NeuroVault: we use Peaks2Image to produce dense brain images for neuroscientific studies. We train a decoder from the generated data on a broad set of cognitive terms. We evaluate the decoding performance on NeuroVault data. Peaks2Image succesfully decodes a large part of those terms in a zero-shot setting. Peaks2Image could extend to any word from the studies’ vocabulary, but could be evaluated on the NeuroVault annotations only.

Input: Figure 7: Reconstructions of NeuroVault samples: we show the average sparse Gaussian representation used in standard meta-analysis procedures (left) along with the Peaks2Image reconstruction (right) for NeuroVault samples containing certain terms. The reconstruction yields more extended networks (audition, face, language production) and sometimes biases the image away from the initial map (sadness). In some rare cases (visual) it focuses the activity patterns.

Input: Figure 1: Reconstructing Brain Activation Maps from Activation Peaks (A) We learn the reconstruction process from NeuroVault collection #4438. After selection and preprocessing of the images, we extract the peaks that are above a statistical threshold. From those peaks, we build an intermediate representation by placing a Gaussian kernel at each peak as in Dockès et al. (2020); Ngo et al. (2021). We compress the representation using the DiFuMo atlas. We regress the original DiFuMo components of each image by leveraging a 3-layer multi-layer perceptron (MLP). (B) Once the reconstruction model is trained, we build intermediate Gaussian-kernel representations for the scraped studies, and reconstruct a brain map for each study using the above model, supporting further image-level computations such as decoding. From the text, we extract the tf-idf values for each word, and infer labels on a percentile-based threshold. (C) We perform decoding of NeuroVault images by training on the newly acquired representations of neuroscientific studies. We decode a broad set of cognitive processes benefiting from the large vocabulary.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim "Using those reconstructions, we train a decoder using tf-idf features as labels, leading to a much broader set of decoded terms than current image-based studies" appears to be well-supported but slightly overstated. While the evidence shows they decode 81 terms out of 6308 vocabulary terms and achieve above-chance performance for most terms, several limitations exist: only terms with >50 positive examples were included, and evaluation was limited due to NeuroVault annotation constraints. The review also raises valid concerns about missing comparisons to state-of-the-art techniques and potential overfitting issues.

<score>3</score>

Justification:
1. The evidence confirms tf-idf features were used as labels and the method can technically extend to a broader vocabulary, but practical validation was limited to 81 terms.

2. Supporting evidence:
- "For validation, we restrict the analysis to shared terms...keeping 81 terms to decode"
- "Peaks2Image could extend to any word contained in the neuroscientific literature, evaluation was reduced to a limited number of concepts"

3. Limitations/concerns from evidence and review:
- Missing comparisons to relevant state-of-the-art techniques (HTFA, BRSA, DRD)
- Potential overfitting issues due to unconstrained D in equation 2
- Limited practical validation despite theoretical capability for broader decoding
- No discussion of multi-subject alignment and batch effects

4. While the method shows promise for broader term decoding, the actual demonstrated scope is more limited than the claim suggests, making it somewhat overstated.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Using those reconstructions, we train a decoder using tf-idf features as labels, leading to a much broader set of decoded terms than current image-based studies.

The review comment to be evaluated is:

Strength:
1.This paper is well-written and easy to read. The definition of tasks and problems are very clear.
2.For a decoding task, the idea that fully utilize the rich descriptions of the studied cognitive tasks in scientific report is novel.
Weaknesses
1.My major concern is the necessity of image reconstruction from peaks. As illustrated in the paper, the authors train a reconstruction network to convert the peak coordinates into images. In my opinion, this ill-posed problem is too hard to be reasonable. Reconstructing the fMRI from only a few peak coordinates (less than 100 according to the paper) seems impossible to recover the details of images. Thus, the reconstruction is unnecessary. If the peak coordinates contain enough information for the following decoding tasks, reconstruction would introduce inevitable noise. If the peak coordinates are not enough, reconstruction still can’t help the decoding since extra information comes from the ‘guessing’ of the trained network. Can the authors give more explanation about this?
2.Also, the experiments do not verify that image reconstruction can help in improving the decoding performance. The ablation study should compare with model using peak coordinates as input. So, I am wondering about the performance of model that uses peak coordinates as input.
3.Experimental results are all shown in figures, which is good for analysis. However, giving some extra tables with exact values can be better.

The evidence to be evaluated is:Dockes et al. (2020) broadened the spectrum of terms analyzed by mapping rare concepts to more common cognitive terms. Ngo et al. (2021) leveraged language models (Beltagy et al., 2019) to capture term relationships semantically, leading to the encoding of any query.

Menuet et al. (2022) leverages the Cognitive Atlas (Poldrack et al., 2011) to improve the label quality of NeuroVault images, enabling the decoding of a large set of concepts.

Although there has been interest in enriching brain images to increase the amount of data (Zhuang et al., 2019), to generate or reconstruct fMRI images (Manning et al., 2018; Cai et al., 2016), there has been limited interest in generating reconstructions of statistical brain maps from peaks (Gorgolewski et al., 2019), probably due to the daunting data dimensionality. In particular, we found no attempt at merging rich textual information from neuroscientific studies with the dense spatial information of brain images. We focus on this particular aspect, proving that obtaining image representations for neuroscientific studies helps overcome the limitations of CBMA by generating a corpus with both rich textual annotations and dense spatial representations.

We aim to reconstruct brain images from those few datapoints. This reconstruction problem is an inverse problem between the set of peaks and the image from which they were extracted. From a given set of \(l\) peaks \(\mathbf{P_{y}}\in\mathbb{R}^{3\times l}\) extracted from an image \(\mathbf{y}\in\mathbb{R}^{m}\) with \(m\) voxels, we learn the transformation \(f:\mathbb{R}^{3\times l}\mapsto\mathbb{R}^{m}\) by minimization of the \(\mathbf{L_{2}}\) error over the original image: \[f^{*}=\operatorname*{arg\,min}_{f}\mathbf{E_{y}}\|f(\mathbf{P_{y}})-\mathbf{ y}\|_{2}^{2} \tag{1}\]

Once trained, Peaks2Image can be applied to any set of peak coordinates from neuroscientific publications. This leads to a combination of both continuous brain images and extensively annotated studies, which opens the possibility to decode a larger set of cognitive concepts.

The figure is divided between the learning of the reconstruction process from peaks (A), the application of the reconstruction to neuroscientific studies (B), and the decoding of brain images with a model trained from those reconstructions in a zero-shot setting.

We use the Dictionary of Functional Modes (DiFuMo) probabilistic atlas (Dadi et al., 2020), that efficiently reduces the \(m=50,000\) voxel values to \(k=1024\) components. DiFuMo has been trained on thousands of resting-state and task fMRI with non-negativity and sparsity constraints, making this atlas relevant to represent population-level information.

Mapping voxels \(\mathbf{x}\in\mathbb{R}^{m}\) to coefficients \(\alpha\in R^{k}\) along DiFuMo dictionary \(\mathbf{D}\in\mathbb{R}^{m\times k}\) of dimension \(k\) is a mere linear regression problem following equation 2.

To train this reconstruction process, we use around 6k images from the #4438 NeuroVault collection ([https://neurovault.org/collections/4438](https://neurovault.org/collections/4438)). The reconstructed images are obtained from the predicted DiFuMo components by linear combinations of the dictionary components: \(\hat{\mathbf{y}}=\mathbf{D}f^{*}(\alpha(\mathbf{x}_{\mathbf{y}}))\).

By doing this on the most comprehensive public literature corpus, we reconstruct 13,000 brain images from the peak activation tables.

We extract the term frequency-inverse document frequency (tf-idf) (Salton and Buckley, 1988) from the studies' text. The tf-idf reflects the relevance of a specific term to a study by giving higher power to terms that occurs repetitively in few studies. Based on it, we assign labels to studies, by setting a threshold at the 95th percentile of the tf-idf. This ensures that the few studies that are most relevant to each term get positive labels.

We leverage a 3-layer dense neural network to predict the set of binary labels \(\mathbf{L}=\{0,1\}_{1}\times\cdots\times\{0,1\}_{l}\), \(l\) denoting the total number of classes, from the input image \(\mathbf{X}\in\mathbb{R}^{d}\) compressed in DiFuMo space.

In the following experiments, we predict only \(l{=}81\) terms from the 6308 in the tf-idf vocabulary. Peaks2Image could extend to a set of cognitive terms as large as the vocabulary size without any additional cost.

It also transforms each study text into tf-idf features across a vocabulary of 6308 terms. For validation, we restrict the analysis to shared terms between the tags of NeuroVault images and the vocabulary from the tf-idf. Among the 240 tags from the NeuroVault test collections, 189 are present in the vocabulary. We focus on those 189 tags without adding any additional mappings between the left-out tags and words from the vocabulary. Additionally, we filter out terms that contain less than 50 positive examples in the test set to prevent any non-significant effects, keeping 81 terms to decode.

The decoder model is an MLP composed of one hidden layer of 300 units and ReLU activation. It is trained for 200 epochs on the reconstructed neuroscientific studies using the Adam optimizer with a learning rate of 3e-3 on the binary cross-entropy loss.

As shown in the next section, although the correlations obtained with the different methods are comparable, the Peaks2Image reconstruction yields a much better performance on the downstream decoding task.

We validate the relevance of the reconstructions by decoding NeuroVault images using reconstructed neuroscientific studies as training set. The problem is a multi-label classification, which we evaluate by computing the ROC AUC over each label. We exclude collection #4438 that has been used as training set in the reconstruction task, and evaluate the decoding of all remaining images from NeuroVault. We apply label enrichment to the original image labels. This includes regular expressions to map different terminologies for a same concept, and a hierarchy-based inference of labels based on the Cognitive Atlas ontology. For a given term, we assign parent terms from the ontology used in (Menuet et al., 2022). This results in 43k images, totalling around 200 different cognitive processes.

The limitation of terms is induced by the evaluation against NeuroVault, but Peaks2Image could extend the prediction to any word included in the tf-idf vocabulary.

We show that, even though the correlation of sparse Gaussian representations remains close to Peaks2Image reconstruction, Peaks2Image largely outperforms the sparse Gaussian model on the decoding task.

On terms that are shared with NeuroSynth (Yarkoni et al., 2011), we show that Peaks2Image outperforms the former, while largely expanding the number of cognitive processes decoded.

Compared to images-based models, it allows to decode a much broader set of cognitive processes by leveraging the rich textual information of neuroscientific studies, along with the reconstructions. Most importantly, the decoding performance is reached without any labeling requirements on the images side, avoiding a time-consuming and error-prone task.

While Peaks2Image could extend to any word contained in the neuroscientific literature, evaluation was reduced to a limited number of concepts due to the lack of extensively annotated brain images. Exploration of more powerful language models as in Ngo et al. (2021) is a future direction to better benefit from the continuous brain images generated for the neuroscientific studies.

Peaks2Image paves the way towards better generation of large sets of images for neuroscientific studies. It shows that one can outrun the limitations of large-scale poor labeling quality by performing zero-shot decoding.

We also apply Peaks2Image to NeuroVault images that are associated to a set of cognitive terms.

In figure 8, we report the per-label decoding performance along with the variance of the results. Most terms are decoded above chance.

Input: Figure 8: Zero-shot decoding performance on NeuroVault: we use Peaks2Image to produce dense brain images for neuroscientific studies. We train a decoder from the generated data on a broad set of cognitive terms. We evaluate the decoding performance on NeuroVault data. Peaks2Image succesfully decodes a large part of those terms in a zero-shot setting. Peaks2Image could extend to any word from the studies’ vocabulary, but could be evaluated on the NeuroVault annotations only.

Input: Figure 7: Reconstructions of NeuroVault samples: we show the average sparse Gaussian representation used in standard meta-analysis procedures (left) along with the Peaks2Image reconstruction (right) for NeuroVault samples containing certain terms. The reconstruction yields more extended networks (audition, face, language production) and sometimes biases the image away from the initial map (sadness). In some rare cases (visual) it focuses the activity patterns.

Input: Figure 1: Reconstructing Brain Activation Maps from Activation Peaks (A) We learn the reconstruction process from NeuroVault collection #4438. After selection and preprocessing of the images, we extract the peaks that are above a statistical threshold. From those peaks, we build an intermediate representation by placing a Gaussian kernel at each peak as in Dockès et al. (2020); Ngo et al. (2021). We compress the representation using the DiFuMo atlas. We regress the original DiFuMo components of each image by leveraging a 3-layer multi-layer perceptron (MLP). (B) Once the reconstruction model is trained, we build intermediate Gaussian-kernel representations for the scraped studies, and reconstruct a brain map for each study using the above model, supporting further image-level computations such as decoding. From the text, we extract the tf-idf values for each word, and infer labels on a percentile-based threshold. (C) We perform decoding of NeuroVault images by training on the newly acquired representations of neuroscientific studies. We decode a broad set of cognitive processes benefiting from the large vocabulary.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on my review of the evidence, the claim that they "train a decoder using tf-idf features as labels, leading to a much broader set of decoded terms than current image-based studies" is partially supported but somewhat overstated. While they do use tf-idf features for labeling and demonstrate decoding across 81 terms (out of a possible 6,308 vocabulary terms), the evaluation was actually limited to shared terms between NeuroVault tags and their vocabulary. The potential for broader decoding is discussed theoretically but not fully demonstrated empirically in the evidence provided.

The reviewer also raises valid concerns about the necessity and effectiveness of the image reconstruction step, which is central to their method. While the authors show some promising results, the claim would be better stated with more qualification about the current limitations and actual demonstrated scope.

<score>3</score>

Justification:
1. The evidence shows they did use tf-idf features as labels and achieved decoding across 81 terms
2. The claim about "much broader set" is only partially demonstrated:
- Of 240 NeuroVault tags, only 189 were in their vocabulary
- They further filtered to 81 terms requiring 50+ positive examples
- While they suggest potential for 6,308 terms, this wasn't empirically validated
3. Key limitations noted in evidence:
- "evaluation was reduced to a limited number of concepts due to the lack of extensively annotated brain images"
- The reviewer raises valid concerns about reconstruction effectiveness
4. The authors themselves acknowledge these limitations in their discussion

A more measured claim would better reflect the demonstrated capabilities while acknowledging current constraints.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

The main goal of this research is to determine a person's country-of-origin using English text written in less controlled environments, employing Machine Learning (ML) and Natural Language Processing (NLP) techniques.

The evidence to be evaluated is:The work of Koppel et al. (2005) is one of the first ethno-nationality identification work where it involves categorizing users based on their native language employing Support Vector Machines (SVMs) on various stylistic features focused on identification of common. Authors achieved 0.802 accuracy using five chosen languages (i.e., Czech, French, Bulgarian, Russian and Spanish) from International Corpus of Learner English (ICLE)(Granger, 2014).

Bykh & Meurers (2012) proposed use of recurring n-grams on three different classes (word based, POS based, Open-Class-POS-based) as features for training SVMs. Out of other SVM implementations LIBLINEAR produced best results. For this study, random data from seven native languages selected from the ICLE corpus. The highest performance was obtained for word-level n-grams with an accuracy of 0.8971. Gebre et al. (2013) employs linear SVM, logistic regression and perceptron (as baseline) for the native language identification (NLI) and achieved accuracy of 0.814 for eleven languages of 'The Test of English as a Foreign Language' (TOEFL) data set(Blanchard et al., 2013). Features used includes; word n-grams, POS n-grams, character n-grams and spelling errors. TOEFL11 has become standard benchmark in NLI tasks since its introduction for the NLI Shared Task 2013.

Cimino & Dell'Orletta (2017) utilizes a novel stacked classifier approach where linear logistic regression based sentence feature classifier is stacked with a SVM based document feature classifier with standard lexical, stylistic and syntactic features.

Best results of the NLI shared task 2017 reported for this approach with F1-score of 0.8818 for the TOEFL11 dataset. Kulmizev et al. (2017) introduced 'Groningen' system for the NLI Shared Task 2017 which out performs employing linear SVM for character 1-9 n-grams with the F1-score of 0.8756.

Goutte & Leger (2017) explored use of voting ensemble SVM models with character, word and POS n-grams.

Markov et al. (2018) has used punctuation-based features with POS n-grams for his experiments and accuracies of 0.4883 (TOEFL11) and 0.6948 (ICLE) reported for best performing settings.

Malmasi & Dras (2018) employed a supervised multi-class classification approach and incorporated several corpora, including; TOEFL11, EF Cambridge Open Language Database Corpus (EFCAM-DAT), ASK Corpus (Andresprakskorpus, Second Language Corpus), and Jinan Chinese Learner Corpus. The features extracted were; word/lemma n-grams, character n-grams, function word n-grams, POS n-grams, dependencies, CFG rules, adaptor Grammars and TSG fragments.

On the other hand, identifying the nation of the English writer opposed to his native language will be equally beneficial for author profiling as well. Thus, we Supervised learning approach was chosen as it is suitable in classifying text documents into classes more accurately if the classes are known and the data set is labelled (Slotte, 2018). Hence, according to the literature most promising ML algorithms for the text classification such as; Support Vector Machines with Stochastic Gradient Descent Optimizer (linear SVM with SGD), Multinomial Naive Bayes (MNB), Decision Tree (DT) and Random Forest (an ensemble approach) have been employed. For this research mostly Scikit-learn(Pedregosa et al., 2012) has been used for pre-processing, feature extraction and classification tasks. Scikit-learn is a Python module for machine learning which provides state-of-the-art implementations of many well-known machine learning algorithms, and maintains an easy-to-use interface(Pedregosa et al., 2012).

The main assumptions of this research are authors from the same ethno-nationality share the same linguistic features in their writing and will often have an influence on the way they express themselves in writings (Jain et al., 2017).

Q3. Which machine learning techniques can gainfully employ the extracted data to identify country-of-origin of English writers?_ The main dataset employed in this study is International Corpus of English (ICE) (Greenbaum and Nelson, 1996; Kirk and Nelson, 2018) which comprises of several corpora from different countries. This data-set was built with the intention of providing a resource to conduct comparative studies of English used in different countries where English is the native language or the second language.

Only written texts have been considered for this study. Written texts were gathered from many areas, such as; student writings, letters, academic writing, news reports, instructional writing, persuasive writing and creative writing (Kirk and Nelson, 2018). The authors and speakers of the texts are aged 18 or above, educated through the medium of English, and were either born in the country in whose corpus they are included or spent the majority of their lives there, or moved there at an early age, and received their education through the medium of English in the country concerned. Written English corpora from Sri Lanka, India, Philippines, Singapore, Canada, Hong Kong, Nigeria, Ireland, Jamaica and USA have been collected and used for this research. Each country is consisted of \(\approx\)200 text documents (\(\approx\)2,000 words per document) (refer Figure 1). As the initial step, 30% (testing data) of the total data of each country corpus was kept as unseen or un-touched data. The remaining 70% of data (training data) was analyzed in order to identify the pre-processing requisites. Raw text-files with specific markups are used as input. These files are cleaned as per the ICE-Corpus markup guide Nelson (2002). In addition,'strip_accents' parameter for the TF-IDF vectorizer and 'Unicode data normalize NFKD' were used. Moreover, URLs, XML tags, e-mail addresses, censored data, line feeds were removed. The data set contained HTML entity encodings and those were decoded(refer Figure 1). In order to avoid country names, nationality, currency and popular cities being trained as features, all the country specific nationality, cities and country names are removed as a pre-processing step. The training data set were used for model building, parameter tuning, training and selection. This labelled training data was tokenized using the TF-IDF vectorizer. Vectorized output was fed to ML classifier to train the model. Further, k-fold (k=3) cross validation technique was used on the training data. Testing data set was kept hold to feed and evaluate the classification model in later stages. Models were built based on two classification strategies;

Flat classification refers to a single classifier at the root level as the decision point as depicted in Figure 2(Top). This classifier will handle the all classes as per its classification approaches such as one-vs-rest, one-vs-one_etc. In this study 'one-vs-all' approach has been followed. Hence, the dataset was trained for 10 classes (countries) together.

Sub-category classification employs N number of classifiers as depicted in Figure 2(Bottom Left) and Figure 2(Bottom Right). For each sub-category specialized in solving a subset of the problem in which each classifier is trained. Each sub-category was trained separately. As depicted, some are binary (marked with black boxes in Figure 2) and some are multi-class classification models (marked with grey boxes in Figure 2).

A combination of both 'hold-out validation' and k-fold cross-validation was used in order to reduce biasness for training and testing data set (Slotte, 2018). Hence, on the 70% of the training data set, 3-fold cross validation technique was used for model building, training and selection and 30% of the data was kept for hold-out validation to validate the machine learning model.

For each ML algorithm, word level (1-1,2,3,4) n-grams were tested. Further, performance accuracy, F1-score, precision and recall were calculated to select the best performing ML model.

The best ML model obtained for the flat classification strategy is linear SVM with SGD optimizer trained with word (1,1) uni-gram model. Furthermore, overall balanced accuracy of 0.7620 and macro average F1-Score of 0.76 was obtained.

Accuracies of different sub-categories have varied in between 0.6224\(\sim\)1.000 and F1-score ranges between 0.49\(\sim\)1.00. Based on the Kachru's concentric theory, lowest F1-Score of 0.71 is for "Outer vs Expanded" sub-category. On the other hand, 'Expanded' sub-category (Hong Kong vs Jamaica) has the highest F1-score of 0.92. In geo-graphical distribution-based model, lowest F1-score of 0.49 is for "Non-Asia" sub-category, while 'South Asian countries' sub-category holding the highest F1-score of 1.0.

Assessing the generalizability of the ML models is crucial. Over-fitting leads to poor generalizability of the model. Hence, in order to detect and prevent over-fitting, 'hold-out validation' and k-fold cross-validation used.

Some of the features are driven by the cultural and geo-graphical differences of the particular country. For instance, Ireland: northern, christmas, groundwater, queen. and Singapore: business, batik, population. Nigeria: donut(for do not), federal, god.etc. **Q2: Which machine learning techniques can gainfully employ the extracted data to identify country-of-origin of English writers?** Comparison of ML algorithms based on the selected best parameters are analysed on top of the test data and results are depicted in Figure 7.

With various accuracy levels the literature has proven that the ethno-nationality of a person can be identified using their written English texts and this area of research has lot of practical applications and usage.

Therefore, those identified limitations have re-framed this research to obtain solutions to the identified research problem and defined research questions.

We are deeply grateful for the International English of Corpus (ICE) project owners and maintainers of the respective countries for making available this corpus to the research community without any license fee.

The Contributors of the respective ICE Corpus countries have informed about the data collection procedures and objectives.

However, author-profiling is gaining pace and the authors of this paper admire the true potential of such categorizations.

Input: Figure 2: (Top) Flat classification strategy setup. (Bottom Left) Sub-category classification setup based on the Kachru’s concentric circles. (Bottom Right) Sub-category classification setup based on the geographical distribution

Input: Figure 1: (Left) Workflow of the research methodology. (Right) Document & word distribution of ICE corpus

Input: Figure 7: Performance comparison of each ML technique

Input: Figure 3: Identified sub-categories based on Kachru’s Theory and Geographical distribution

Input: Figure 4: (Left) Test Results for the Flat Classification Strategy. (Right) Confusion matrix for the Flat Classification Strategy

Input: Figure 5: (Left) Test results of the sub-category classification strategy based on selected ML model. (Right) Learning curve for the Flat Classification to assess the generalisability

Input: Figure 6: Most significant features of the flat classification

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me evaluate the claim against the evidence:

The claim states that the main goal is to determine country-of-origin using English text from less controlled environments with ML/NLP techniques. Based on the evidence provided, this appears to be a well-stated claim for several reasons:

1. The evidence shows detailed methodology using ML/NLP techniques (SVM, Decision Trees, etc.) specifically for country-of-origin classification
2. The data comes from ICE corpus containing various types of written texts (student writings, letters, academic writing, etc.) representing less controlled environments
3. The research demonstrates practical implementation with reasonable results (accuracies between 0.62-1.00 for different classification approaches)
4. The limitations and challenges are acknowledged in the methodology

The claim accurately reflects the scope and nature of the research without exaggeration.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

The main goal of this research is to determine a person's country-of-origin using English text written in less controlled environments, employing Machine Learning (ML) and Natural Language Processing (NLP) techniques.

The review comment to be evaluated is:

This paper aims to determine the country of origin of English language writers. It uses documents from the International corpus of English (ICE) and employs various learning algorithms and text-based features. The findings are that linear SVM with unigram features works best, that some countries are better distinguished than others, and that some features are more informative than others.

The main premise of this work is that previous work in this space used controlled settings such as English learners texts or standardized examples, while this work examines more naturalistic settings. This is a reasonable motivation. However, besides that, the contributions of this work are very limited: it uses standard algorithms and features, there isn't much analysis of the results, and so the reader does not learn much new from this study. It also doesn't seem like a very good fit for ICLR.

The evidence to be evaluated is:The work of Koppel et al. (2005) is one of the first ethno-nationality identification work where it involves categorizing users based on their native language employing Support Vector Machines (SVMs) on various stylistic features focused on identification of common. Authors achieved 0.802 accuracy using five chosen languages (i.e., Czech, French, Bulgarian, Russian and Spanish) from International Corpus of Learner English (ICLE)(Granger, 2014).

Bykh & Meurers (2012) proposed use of recurring n-grams on three different classes (word based, POS based, Open-Class-POS-based) as features for training SVMs. Out of other SVM implementations LIBLINEAR produced best results. For this study, random data from seven native languages selected from the ICLE corpus. The highest performance was obtained for word-level n-grams with an accuracy of 0.8971. Gebre et al. (2013) employs linear SVM, logistic regression and perceptron (as baseline) for the native language identification (NLI) and achieved accuracy of 0.814 for eleven languages of 'The Test of English as a Foreign Language' (TOEFL) data set(Blanchard et al., 2013). Features used includes; word n-grams, POS n-grams, character n-grams and spelling errors. TOEFL11 has become standard benchmark in NLI tasks since its introduction for the NLI Shared Task 2013.

Cimino & Dell'Orletta (2017) utilizes a novel stacked classifier approach where linear logistic regression based sentence feature classifier is stacked with a SVM based document feature classifier with standard lexical, stylistic and syntactic features.

Best results of the NLI shared task 2017 reported for this approach with F1-score of 0.8818 for the TOEFL11 dataset. Kulmizev et al. (2017) introduced 'Groningen' system for the NLI Shared Task 2017 which out performs employing linear SVM for character 1-9 n-grams with the F1-score of 0.8756.

Goutte & Leger (2017) explored use of voting ensemble SVM models with character, word and POS n-grams.

Markov et al. (2018) has used punctuation-based features with POS n-grams for his experiments and accuracies of 0.4883 (TOEFL11) and 0.6948 (ICLE) reported for best performing settings.

Malmasi & Dras (2018) employed a supervised multi-class classification approach and incorporated several corpora, including; TOEFL11, EF Cambridge Open Language Database Corpus (EFCAM-DAT), ASK Corpus (Andresprakskorpus, Second Language Corpus), and Jinan Chinese Learner Corpus. The features extracted were; word/lemma n-grams, character n-grams, function word n-grams, POS n-grams, dependencies, CFG rules, adaptor Grammars and TSG fragments.

On the other hand, identifying the nation of the English writer opposed to his native language will be equally beneficial for author profiling as well. Thus, we Supervised learning approach was chosen as it is suitable in classifying text documents into classes more accurately if the classes are known and the data set is labelled (Slotte, 2018). Hence, according to the literature most promising ML algorithms for the text classification such as; Support Vector Machines with Stochastic Gradient Descent Optimizer (linear SVM with SGD), Multinomial Naive Bayes (MNB), Decision Tree (DT) and Random Forest (an ensemble approach) have been employed. For this research mostly Scikit-learn(Pedregosa et al., 2012) has been used for pre-processing, feature extraction and classification tasks. Scikit-learn is a Python module for machine learning which provides state-of-the-art implementations of many well-known machine learning algorithms, and maintains an easy-to-use interface(Pedregosa et al., 2012).

The main assumptions of this research are authors from the same ethno-nationality share the same linguistic features in their writing and will often have an influence on the way they express themselves in writings (Jain et al., 2017).

Q3. Which machine learning techniques can gainfully employ the extracted data to identify country-of-origin of English writers?_ The main dataset employed in this study is International Corpus of English (ICE) (Greenbaum and Nelson, 1996; Kirk and Nelson, 2018) which comprises of several corpora from different countries. This data-set was built with the intention of providing a resource to conduct comparative studies of English used in different countries where English is the native language or the second language.

Only written texts have been considered for this study. Written texts were gathered from many areas, such as; student writings, letters, academic writing, news reports, instructional writing, persuasive writing and creative writing (Kirk and Nelson, 2018). The authors and speakers of the texts are aged 18 or above, educated through the medium of English, and were either born in the country in whose corpus they are included or spent the majority of their lives there, or moved there at an early age, and received their education through the medium of English in the country concerned. Written English corpora from Sri Lanka, India, Philippines, Singapore, Canada, Hong Kong, Nigeria, Ireland, Jamaica and USA have been collected and used for this research. Each country is consisted of \(\approx\)200 text documents (\(\approx\)2,000 words per document) (refer Figure 1). As the initial step, 30% (testing data) of the total data of each country corpus was kept as unseen or un-touched data. The remaining 70% of data (training data) was analyzed in order to identify the pre-processing requisites. Raw text-files with specific markups are used as input. These files are cleaned as per the ICE-Corpus markup guide Nelson (2002). In addition,'strip_accents' parameter for the TF-IDF vectorizer and 'Unicode data normalize NFKD' were used. Moreover, URLs, XML tags, e-mail addresses, censored data, line feeds were removed. The data set contained HTML entity encodings and those were decoded(refer Figure 1). In order to avoid country names, nationality, currency and popular cities being trained as features, all the country specific nationality, cities and country names are removed as a pre-processing step. The training data set were used for model building, parameter tuning, training and selection. This labelled training data was tokenized using the TF-IDF vectorizer. Vectorized output was fed to ML classifier to train the model. Further, k-fold (k=3) cross validation technique was used on the training data. Testing data set was kept hold to feed and evaluate the classification model in later stages. Models were built based on two classification strategies;

Flat classification refers to a single classifier at the root level as the decision point as depicted in Figure 2(Top). This classifier will handle the all classes as per its classification approaches such as one-vs-rest, one-vs-one_etc. In this study 'one-vs-all' approach has been followed. Hence, the dataset was trained for 10 classes (countries) together.

Sub-category classification employs N number of classifiers as depicted in Figure 2(Bottom Left) and Figure 2(Bottom Right). For each sub-category specialized in solving a subset of the problem in which each classifier is trained. Each sub-category was trained separately. As depicted, some are binary (marked with black boxes in Figure 2) and some are multi-class classification models (marked with grey boxes in Figure 2).

A combination of both 'hold-out validation' and k-fold cross-validation was used in order to reduce biasness for training and testing data set (Slotte, 2018). Hence, on the 70% of the training data set, 3-fold cross validation technique was used for model building, training and selection and 30% of the data was kept for hold-out validation to validate the machine learning model.

For each ML algorithm, word level (1-1,2,3,4) n-grams were tested. Further, performance accuracy, F1-score, precision and recall were calculated to select the best performing ML model.

The best ML model obtained for the flat classification strategy is linear SVM with SGD optimizer trained with word (1,1) uni-gram model. Furthermore, overall balanced accuracy of 0.7620 and macro average F1-Score of 0.76 was obtained.

Accuracies of different sub-categories have varied in between 0.6224\(\sim\)1.000 and F1-score ranges between 0.49\(\sim\)1.00. Based on the Kachru's concentric theory, lowest F1-Score of 0.71 is for "Outer vs Expanded" sub-category. On the other hand, 'Expanded' sub-category (Hong Kong vs Jamaica) has the highest F1-score of 0.92. In geo-graphical distribution-based model, lowest F1-score of 0.49 is for "Non-Asia" sub-category, while 'South Asian countries' sub-category holding the highest F1-score of 1.0.

Assessing the generalizability of the ML models is crucial. Over-fitting leads to poor generalizability of the model. Hence, in order to detect and prevent over-fitting, 'hold-out validation' and k-fold cross-validation used.

Some of the features are driven by the cultural and geo-graphical differences of the particular country. For instance, Ireland: northern, christmas, groundwater, queen. and Singapore: business, batik, population. Nigeria: donut(for do not), federal, god.etc. **Q2: Which machine learning techniques can gainfully employ the extracted data to identify country-of-origin of English writers?** Comparison of ML algorithms based on the selected best parameters are analysed on top of the test data and results are depicted in Figure 7.

With various accuracy levels the literature has proven that the ethno-nationality of a person can be identified using their written English texts and this area of research has lot of practical applications and usage.

Therefore, those identified limitations have re-framed this research to obtain solutions to the identified research problem and defined research questions.

We are deeply grateful for the International English of Corpus (ICE) project owners and maintainers of the respective countries for making available this corpus to the research community without any license fee.

The Contributors of the respective ICE Corpus countries have informed about the data collection procedures and objectives.

However, author-profiling is gaining pace and the authors of this paper admire the true potential of such categorizations.

Input: Figure 2: (Top) Flat classification strategy setup. (Bottom Left) Sub-category classification setup based on the Kachru’s concentric circles. (Bottom Right) Sub-category classification setup based on the geographical distribution

Input: Figure 1: (Left) Workflow of the research methodology. (Right) Document & word distribution of ICE corpus

Input: Figure 7: Performance comparison of each ML technique

Input: Figure 3: Identified sub-categories based on Kachru’s Theory and Geographical distribution

Input: Figure 4: (Left) Test Results for the Flat Classification Strategy. (Right) Confusion matrix for the Flat Classification Strategy

Input: Figure 5: (Left) Test results of the sub-category classification strategy based on selected ML model. (Right) Learning curve for the Flat Classification to assess the generalisability

Input: Figure 6: Most significant features of the flat classification

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, I find this claim to be well-stated. The evidence demonstrates that the research indeed focuses on determining country-of-origin from English text in less controlled environments, specifically using the International Corpus of English (ICE) which contains naturalistic written texts from various contexts (academic, news, letters, etc.). The methodology clearly employs ML and NLP techniques, including SVMs, Naive Bayes, and Decision Trees, with detailed feature extraction and classification strategies. The claim accurately represents the research scope without exaggerating its capabilities or findings.

<score>1</score>

Justification:
1. The evidence directly supports the goal through:
- Use of ICE corpus containing natural written texts from multiple contexts
- Clear implementation of ML/NLP techniques (SVM, Naive Bayes, etc.)
- Detailed methodology showing feature extraction and classification
- Results demonstrating the approach's effectiveness across different countries

2. No exaggeration present - the claim simply states the main goal without overpromising or overstating capabilities.

3. The reviewer's comment, while critical of the paper's broader contributions, actually supports that this was indeed the main goal and motivation of the work.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

The main goal of this research is to determine a person's country-of-origin using English text written in less controlled environments, employing Machine Learning (ML) and Natural Language Processing (NLP) techniques.

The review comment to be evaluated is:

I think the motivation for studying ethnonationality is a bit weak. The main argument is forensic linguistics and marketing. Both motivations are, in my view, ethically questionable, but the study of ethnonationality also has direct applications for *improving* the fairness of NLP systems.

I like the idea of moving away from TOEFL/ICLE datasets for native language identification - in theory. I previously did research on social media data, and the task suddenly becomes a lot harder. The challenge leaving the TOEFL/ICLE domain is that suddenly a lot of open class words become give-aways of the target. People in Jamaica talk about Jamaica. People in Australia talk about Melbourne, surfing, and kangaroos. In the context of forensic linguists, Australians are not gonna talk about surfing or kangaroos, especially not if they want to hide where they’re from. For this reason, it is common methodology when moving out of a controlled setting (like TOEFL/ICLE), to remove open class words or rely only on character bigram or trigram statistics. The authors fail to do so.

Other problems: a) The protocol is weak; split data temporally rather than randomly to make sure your results will generalize. b) The description of state of the art is out-dated, e.g., see [0-2].

[0] https://aclanthology.org/2020.coling-main.159.pdf
[1] https://aclanthology.org/2020.icon-main.16.pdf
[2] https://arxiv.org/pdf/2010.01869.pdf

The evidence to be evaluated is:The work of Koppel et al. (2005) is one of the first ethno-nationality identification work where it involves categorizing users based on their native language employing Support Vector Machines (SVMs) on various stylistic features focused on identification of common. Authors achieved 0.802 accuracy using five chosen languages (i.e., Czech, French, Bulgarian, Russian and Spanish) from International Corpus of Learner English (ICLE)(Granger, 2014).

Bykh & Meurers (2012) proposed use of recurring n-grams on three different classes (word based, POS based, Open-Class-POS-based) as features for training SVMs. Out of other SVM implementations LIBLINEAR produced best results. For this study, random data from seven native languages selected from the ICLE corpus. The highest performance was obtained for word-level n-grams with an accuracy of 0.8971. Gebre et al. (2013) employs linear SVM, logistic regression and perceptron (as baseline) for the native language identification (NLI) and achieved accuracy of 0.814 for eleven languages of 'The Test of English as a Foreign Language' (TOEFL) data set(Blanchard et al., 2013). Features used includes; word n-grams, POS n-grams, character n-grams and spelling errors. TOEFL11 has become standard benchmark in NLI tasks since its introduction for the NLI Shared Task 2013.

Cimino & Dell'Orletta (2017) utilizes a novel stacked classifier approach where linear logistic regression based sentence feature classifier is stacked with a SVM based document feature classifier with standard lexical, stylistic and syntactic features.

Best results of the NLI shared task 2017 reported for this approach with F1-score of 0.8818 for the TOEFL11 dataset. Kulmizev et al. (2017) introduced 'Groningen' system for the NLI Shared Task 2017 which out performs employing linear SVM for character 1-9 n-grams with the F1-score of 0.8756.

Goutte & Leger (2017) explored use of voting ensemble SVM models with character, word and POS n-grams.

Markov et al. (2018) has used punctuation-based features with POS n-grams for his experiments and accuracies of 0.4883 (TOEFL11) and 0.6948 (ICLE) reported for best performing settings.

Malmasi & Dras (2018) employed a supervised multi-class classification approach and incorporated several corpora, including; TOEFL11, EF Cambridge Open Language Database Corpus (EFCAM-DAT), ASK Corpus (Andresprakskorpus, Second Language Corpus), and Jinan Chinese Learner Corpus. The features extracted were; word/lemma n-grams, character n-grams, function word n-grams, POS n-grams, dependencies, CFG rules, adaptor Grammars and TSG fragments.

On the other hand, identifying the nation of the English writer opposed to his native language will be equally beneficial for author profiling as well. Thus, we Supervised learning approach was chosen as it is suitable in classifying text documents into classes more accurately if the classes are known and the data set is labelled (Slotte, 2018). Hence, according to the literature most promising ML algorithms for the text classification such as; Support Vector Machines with Stochastic Gradient Descent Optimizer (linear SVM with SGD), Multinomial Naive Bayes (MNB), Decision Tree (DT) and Random Forest (an ensemble approach) have been employed. For this research mostly Scikit-learn(Pedregosa et al., 2012) has been used for pre-processing, feature extraction and classification tasks. Scikit-learn is a Python module for machine learning which provides state-of-the-art implementations of many well-known machine learning algorithms, and maintains an easy-to-use interface(Pedregosa et al., 2012).

The main assumptions of this research are authors from the same ethno-nationality share the same linguistic features in their writing and will often have an influence on the way they express themselves in writings (Jain et al., 2017).

Q3. Which machine learning techniques can gainfully employ the extracted data to identify country-of-origin of English writers?_ The main dataset employed in this study is International Corpus of English (ICE) (Greenbaum and Nelson, 1996; Kirk and Nelson, 2018) which comprises of several corpora from different countries. This data-set was built with the intention of providing a resource to conduct comparative studies of English used in different countries where English is the native language or the second language.

Only written texts have been considered for this study. Written texts were gathered from many areas, such as; student writings, letters, academic writing, news reports, instructional writing, persuasive writing and creative writing (Kirk and Nelson, 2018). The authors and speakers of the texts are aged 18 or above, educated through the medium of English, and were either born in the country in whose corpus they are included or spent the majority of their lives there, or moved there at an early age, and received their education through the medium of English in the country concerned. Written English corpora from Sri Lanka, India, Philippines, Singapore, Canada, Hong Kong, Nigeria, Ireland, Jamaica and USA have been collected and used for this research. Each country is consisted of \(\approx\)200 text documents (\(\approx\)2,000 words per document) (refer Figure 1). As the initial step, 30% (testing data) of the total data of each country corpus was kept as unseen or un-touched data. The remaining 70% of data (training data) was analyzed in order to identify the pre-processing requisites. Raw text-files with specific markups are used as input. These files are cleaned as per the ICE-Corpus markup guide Nelson (2002). In addition,'strip_accents' parameter for the TF-IDF vectorizer and 'Unicode data normalize NFKD' were used. Moreover, URLs, XML tags, e-mail addresses, censored data, line feeds were removed. The data set contained HTML entity encodings and those were decoded(refer Figure 1). In order to avoid country names, nationality, currency and popular cities being trained as features, all the country specific nationality, cities and country names are removed as a pre-processing step. The training data set were used for model building, parameter tuning, training and selection. This labelled training data was tokenized using the TF-IDF vectorizer. Vectorized output was fed to ML classifier to train the model. Further, k-fold (k=3) cross validation technique was used on the training data. Testing data set was kept hold to feed and evaluate the classification model in later stages. Models were built based on two classification strategies;

Flat classification refers to a single classifier at the root level as the decision point as depicted in Figure 2(Top). This classifier will handle the all classes as per its classification approaches such as one-vs-rest, one-vs-one_etc. In this study 'one-vs-all' approach has been followed. Hence, the dataset was trained for 10 classes (countries) together.

Sub-category classification employs N number of classifiers as depicted in Figure 2(Bottom Left) and Figure 2(Bottom Right). For each sub-category specialized in solving a subset of the problem in which each classifier is trained. Each sub-category was trained separately. As depicted, some are binary (marked with black boxes in Figure 2) and some are multi-class classification models (marked with grey boxes in Figure 2).

A combination of both 'hold-out validation' and k-fold cross-validation was used in order to reduce biasness for training and testing data set (Slotte, 2018). Hence, on the 70% of the training data set, 3-fold cross validation technique was used for model building, training and selection and 30% of the data was kept for hold-out validation to validate the machine learning model.

For each ML algorithm, word level (1-1,2,3,4) n-grams were tested. Further, performance accuracy, F1-score, precision and recall were calculated to select the best performing ML model.

The best ML model obtained for the flat classification strategy is linear SVM with SGD optimizer trained with word (1,1) uni-gram model. Furthermore, overall balanced accuracy of 0.7620 and macro average F1-Score of 0.76 was obtained.

Accuracies of different sub-categories have varied in between 0.6224\(\sim\)1.000 and F1-score ranges between 0.49\(\sim\)1.00. Based on the Kachru's concentric theory, lowest F1-Score of 0.71 is for "Outer vs Expanded" sub-category. On the other hand, 'Expanded' sub-category (Hong Kong vs Jamaica) has the highest F1-score of 0.92. In geo-graphical distribution-based model, lowest F1-score of 0.49 is for "Non-Asia" sub-category, while 'South Asian countries' sub-category holding the highest F1-score of 1.0.

Assessing the generalizability of the ML models is crucial. Over-fitting leads to poor generalizability of the model. Hence, in order to detect and prevent over-fitting, 'hold-out validation' and k-fold cross-validation used.

Some of the features are driven by the cultural and geo-graphical differences of the particular country. For instance, Ireland: northern, christmas, groundwater, queen. and Singapore: business, batik, population. Nigeria: donut(for do not), federal, god.etc. **Q2: Which machine learning techniques can gainfully employ the extracted data to identify country-of-origin of English writers?** Comparison of ML algorithms based on the selected best parameters are analysed on top of the test data and results are depicted in Figure 7.

With various accuracy levels the literature has proven that the ethno-nationality of a person can be identified using their written English texts and this area of research has lot of practical applications and usage.

Therefore, those identified limitations have re-framed this research to obtain solutions to the identified research problem and defined research questions.

We are deeply grateful for the International English of Corpus (ICE) project owners and maintainers of the respective countries for making available this corpus to the research community without any license fee.

The Contributors of the respective ICE Corpus countries have informed about the data collection procedures and objectives.

However, author-profiling is gaining pace and the authors of this paper admire the true potential of such categorizations.

Input: Figure 2: (Top) Flat classification strategy setup. (Bottom Left) Sub-category classification setup based on the Kachru’s concentric circles. (Bottom Right) Sub-category classification setup based on the geographical distribution

Input: Figure 1: (Left) Workflow of the research methodology. (Right) Document & word distribution of ICE corpus

Input: Figure 7: Performance comparison of each ML technique

Input: Figure 3: Identified sub-categories based on Kachru’s Theory and Geographical distribution

Input: Figure 4: (Left) Test Results for the Flat Classification Strategy. (Right) Confusion matrix for the Flat Classification Strategy

Input: Figure 5: (Left) Test results of the sub-category classification strategy based on selected ML model. (Right) Learning curve for the Flat Classification to assess the generalisability

Input: Figure 6: Most significant features of the flat classification

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on my review of the evidence and review comment, I assess that this claim is moderately overstated. While the evidence demonstrates the research does aim to identify country-of-origin using ML/NLP techniques, there are important limitations not acknowledged in the claim:

1. The reviewer correctly points out that open class words (location-specific content) significantly influence results, making the task easier but less meaningful for real-world applications.

2. The methodology has weaknesses in temporal data splitting and outdated baseline comparisons.

3. The claim doesn't acknowledge the controlled nature of the ICE corpus used, which affects generalizability to "less controlled environments."

These limitations suggest the claim overstates the research's scope and applicability to real-world scenarios.

<score>4</score>