Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

* Our approach achieve superior classification and localization results against several competitive baselines, on the NIH chest X-ray benchmark.

The review comment to be evaluated is:

Strengths
+ The paper is fairly well written and presents a well-studied literature review.
+ The paper proposes an end-to-end transformer-based disease classification and localization model incorporating radiomic feature knowledge with cross-attention. The idea of bounding box generation and radiomic feature extraction in a feedback loop module is new and interesting.
+ Experimental setting including the ablation study for the effect of thresholds on attention maps and the effect of contrastive learning is good.

Weaknesses
- Exposition of results is not good. CheXT is compared against some weak baselines. A more appropriate CNN baseline should be something that includes similar radiomic feature extraction and bounding box generation [1]. Also, [1] is not cited in Related Work.
- The paper claims significant performance improvement by the CheXT model although no evidence (statistical test) is provided to support the claim.
-  Classification performance drops for some of the disease classes (effusion, pneumonia, pneumothorax) after incorporating the radiomics branch. It looks like radiomics branch is hurting the performance for these classes. Is there any impact of label distribution?
- In Fig. 3, two images look different. Why so?
- The training of CheXT is not necessarily semi-supervised as every image has at least disease class label.
- It is not clear how to obtain f_i and f_r; y, y' are not defined Eqn. (1).
- It would be more viable to report both classification and localization performances on the same 880 images. Is there any correlation between e.g., classification AUC and localization IoU?
- It is not clear how the model could improve generalization. There is no generalization assessment of the model for out-of-distribution cases. For example, using other datasets such as, CheXpert and MIMIC-CXR.
- In 4.1, the paper mentions the train/test/val splits, but Table 1 reports results after 3-fold cross-validation. They don't look consistent.
- In 4.4.2: "when the IoU is set to 0.1" should be threshold of IoU. "For Cardiomegaly, Infiltration, and Pneumonia." incomplete sentence.
- It is not clear how higher values of \lambda implies lower weight to contrastive loss in Eqn. (3). How does the model perform without the contrastive loss at all (\lambda=0)?
- There is no mention of the execution speed, computing resource, and failure cases (if any) for the proposed model.
- There are certainly other Transformer-based chest X-ray models such as [2].

1. https://arxiv.org/abs/2011.12506
2. https://arxiv.org/abs/2104.07235

The evidence to be evaluated is:We conducted 3-fold cross-validation (Table 1). We compared CheXT with reference models, which have published state-of-the-art performance of disease classification on the NIH Chest X-ray dataset. We used Area under the Receiver Operating Characteristics (AUC) to estimate the performance of our model (FawcettTom, 2006). A higher AUC score implies a better classification model.

From the table, we can find that CheXT outperformed the baselines in the majority of diseases.

Compared to all baselines, CheXT achieves a mean AUC score of 0.839 using a dual-branch shallow Transformer across the 8 different diseases, which is 0.011 higher than the SOTA (uses DenseNet-121) (Rajpurkar et al., 2017) on disease classification. Specifically, our results significantly outperformed the best baseline models by 0.13, 0.09, and 0.02 in AUC for detecting Infiltration, Nodule, and Mass higher respectively.

The NIH Chest X-ray dataset has 880 images labeled by radiologists with the bounding box information. We have used this dataset to evaluate the performance of CheXT for abnormality localization.

Table 2 presents our evaluation results on all 880 images. We used (Wang et al., 2017) as our baseline to compare our localization results since it has the same experimental settings. For localization, we evaluated our detected regular rectangular regions against the annotated bounding boxes, using intersection over union ratio (IoU). Our localization results are only calculated for 880 images which have ground truth annotation for 8 diseases.

CheXT average performance across 8 diseases is significantly better than the baseline under all IoU thresholds. When the thresholds of IoU is set to 0.1, CheXT outperforms the baseline (Wang et al., 2017) in Cardiomegaly, infiltration, Mass, and Pneumonia. Even with higher thresholds, our model performs superior to baseline. For example, when evaluated at T(IoU) = 0.5, our "Cardiomegaly" accuracy is still 32%, while the reference model achieves only 18%.

More importantly, we also add the ViT as our additional baseline here, the results show that the radiomics branch and BYOA module can help the model learn more important regions, which also improves the model's interpretability.

Experimental results demonstrate that our method outperforms the state-of-the-art algorithms, especially for the disease localization task, where our method can generate more accurate bounding boxes. \begin{table}
\begin{tabular}{c l c c c c c c c c c} \hline \hline
**T(IoU)** & Model & \multicolumn{2}{c}{Alelectasis} & \multicolumn{2}{c}{Cardiomegaly} & \multicolumn{2}{c}{Effusion} & \multicolumn{2}{c}{Infiltration} & \multicolumn{1}{c}{Mass} & \multicolumn{1}{c}{Nodule} & \multicolumn{1}{c}{Precmonia} & \multicolumn{1}{c}{Precumbotox} & \multicolumn{1}{c}{**Mean**} \\ \hline
0.1 & (Wang et al.,2017) & 0.69 & 0.94 & 0.66 & 0.71 & 0.40 & 0.14 & 0.63 & 0.38 & 0.569 \\  & ViT & 0.58 & 0.91 & 0.61 & 0.77 & 0.44 & 0.11 & 0.75 & 0.25 & 0.553 \\  & CheXT & 0.61 & 0.95 & 0.65 & 0.82 & 0.50 & 0.13 & 0.79 & 0.28 & **0.591** \\ \hline
0.2 & (Wang et al.,2017) & 0.47 & 0.68 & 0.45 & 0.48 & 0.26 & 0.05 & 0.35 & 0.23 & 0.371 \\  & ViT & 0.38 & 0.85 & 0.39 & 0.55 & 0.24 & 0.01 & 0.51 & 0.15 & 0.385 \\  & CheXT & 0.41 & 0.91 & 0.41 & 0.59 & 0.26 & 0.05 & 0.57 & 0.19 & **0.424** \\ \hline
0.3 & (Wang et al.,2017) & 0.24 & 0.46 & 0.30 & 0.28 & 0.15 & 0.04 & 0.17 & 0.13 & 0.221 \\  & ViT & 0.20 & 0.45 & 0.19 & 0.32 & 0.06 & 0.00 & 0.21 & 0.02 & 0.181 \\  & CheXT & 0.28 & 0.79 & 0.22 & 0.38 & 0.12 & 0.01 & 0.41 & 0.05 & **0.283** \\ \hline
0.4 & (Wang et al.,2017) & 0.09 & 0.28 & 0.20 & 0.12 & 0.07 & 0.01 & 0.08 & 0.07 & 0.115 \\  & ViT & 0.10 & 0.21 & 0.03 & 0.05 & 0.02 & 0.00 & 0.04 & 0.00 & 0.056 \\  & CheXT & 0.17 & 0.54 & 0.13 & 0.18 & 0.07 & 0.01 & 0.26 & 0.02 & **0.173** \\ \hline
0.5 & (Wang et al.,2017) & 0.05 & 0.18 & 0.11 & 0.07 & 0.01 & 0.01 & 0.03 & 0.03 & 0.061 \\  & ViT & 0.05 & 0.15 & 0.01 & 0.04 & 0.02 & 0.00 & 0.03 & 0.00 & 0.034 \\  & CheXT & 0.08 & 0.32 & 0.05 & 0.09 & 0.05 & 0.00 & 0.12 & 0.01 & **0.090** \\ \hline
0.6 & (Wang et al.,2017) & 0.02 & 0.08 & 0.05 & 0.02 & 0.00 & 0.01 & 0.02 & 0.03 & 0.029 \\  & ViT & 0.01 & 0.03 & 0.01 & 0.01 & 0.01 & 0.00 & 0.01 & 0.00 & 0.010 \\  & CheXT & 0.02 & 0.15 & 0.03 & 0.04 & 0.03 & 0.00 & 0.06 & 0.00 & **0.041** \\ \hline
0.7 & (Wang et al.,2017) & 0.01 & 0.03 & 0.02 & 0.00 & 0.00 & 0.00 & 0.01 & 0.02 & 0.011 \\  & ViT & 0.00 & 0.00 & 0.00 & 0.01 & 0.00 & 0.00 & 0.00 & 0.00 & 0.001 \\  & CheXT & 0.01 & 0.04 & 0.01 & 0.02 & 0.01 & 0.00 & 0.03 & 0.00 & **0.015** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Disease localization under varying IoU on the NIH Chest X-ray dataset. Please note that since our model doesn’t use any ground truth bounding box information, to fairly evaluate the performance of our model, we only consider the previous methods’ results under the same setting,

Input: Table 1: Comparison with the baseline models for AUC of each class and average AUC. For each column, bold values denote the best results.

Input: Table 2: Disease localization under varying IoU on the NIH Chest X-ray dataset. Please note that since our model doesn’t use any ground truth bounding box information, to fairly evaluate the performance of our model, we only consider the previous methods’ results under the same setting,

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence and claim, the statement appears to be somewhat overstated. While CheXT does show improvements in some areas, several key points suggest the claim needs more nuanced presentation:

1. Classification: The mean AUC improvement is modest (0.839 vs 0.828, only 0.011 higher than SOTA)
2. Some disease classes show decreased performance (noted in review comments for effusion, pneumonia, pneumothorax)
3. Statistical significance tests are missing (highlighted in reviews)
4. Localization: While showing better average performance across IoU thresholds, improvements vary considerably by disease type
5. Limited comparison against contemporary transformer-based models
6. Lack of generalization assessment on other datasets

The claim would be more accurate if it acknowledged these limitations and specified where improvements were achieved rather than claiming broad "superior" results.

<score>4</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

* Our approach achieve superior classification and localization results against several competitive baselines, on the NIH chest X-ray benchmark.

The review comment to be evaluated is:

1) Radiomics is a term that most people in the ICLR community are not familiar with. What is it exactily? Is it a set of radiological/image features or bio-markers in medical images? Or something else? The authors should consider explaining it in plain language or ML language.

2) Prior multi-modal feature fusion work in the chest x-ray domain should be discussed, examples below.

    - Moradi, Mehdi, Ali Madani, Yaniv Gur, Yufan Guo, and Tanveer Syeda-Mahmood. "Bimodal network architectures for automatic generation of image annotation from text." arXiv preprint arXiv:1809.01610 (2018).

    - Wang, Xiaosong, Yifan Peng, Le Lu, Zhiyong Lu, and Ronald M. Summers. "TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-rays." arXiv e-prints (2018): arXiv-1801.

    - Xue, Yuan, and Xiaolei Huang. "Improved disease classification in chest x-rays with transferred features from report generation." In International Conference on Information Processing in Medical Imaging, pp. 125-138. Springer, Cham, 2019.

    - Liao, Ruizhi, Daniel Moyer, Miriam Cha, Keegan Quigley, Seth Berkowitz, Steven Horng, Polina Golland, and William M. Wells. "Multimodal Representation Learning via Maximization of Local Mutual Information." arXiv preprint arXiv:2103.04537 (2021).

3) I don't see "prior knowledge" and how it's utilized in this work. Isn't the so called prior knowledge in this paper essentially extracted from the data (chest x-ray images)?

4) What are the patch sizes? How are they determined?

5) There are 14 pathology labels in the NIH chest x-ray dataset. Why/how did the authors select the 8 classes out of the 14?

6) In Table 1, make the best results bold instead of in red.

7) Are the AUC values generated on the test set? If so, I'm concerned about overfitting by over tuning the hyper-parameters.

The evidence to be evaluated is:We conducted 3-fold cross-validation (Table 1). We compared CheXT with reference models, which have published state-of-the-art performance of disease classification on the NIH Chest X-ray dataset. We used Area under the Receiver Operating Characteristics (AUC) to estimate the performance of our model (FawcettTom, 2006). A higher AUC score implies a better classification model.

From the table, we can find that CheXT outperformed the baselines in the majority of diseases.

Compared to all baselines, CheXT achieves a mean AUC score of 0.839 using a dual-branch shallow Transformer across the 8 different diseases, which is 0.011 higher than the SOTA (uses DenseNet-121) (Rajpurkar et al., 2017) on disease classification. Specifically, our results significantly outperformed the best baseline models by 0.13, 0.09, and 0.02 in AUC for detecting Infiltration, Nodule, and Mass higher respectively.

The NIH Chest X-ray dataset has 880 images labeled by radiologists with the bounding box information. We have used this dataset to evaluate the performance of CheXT for abnormality localization.

Table 2 presents our evaluation results on all 880 images. We used (Wang et al., 2017) as our baseline to compare our localization results since it has the same experimental settings. For localization, we evaluated our detected regular rectangular regions against the annotated bounding boxes, using intersection over union ratio (IoU). Our localization results are only calculated for 880 images which have ground truth annotation for 8 diseases.

CheXT average performance across 8 diseases is significantly better than the baseline under all IoU thresholds. When the thresholds of IoU is set to 0.1, CheXT outperforms the baseline (Wang et al., 2017) in Cardiomegaly, infiltration, Mass, and Pneumonia. Even with higher thresholds, our model performs superior to baseline. For example, when evaluated at T(IoU) = 0.5, our "Cardiomegaly" accuracy is still 32%, while the reference model achieves only 18%.

More importantly, we also add the ViT as our additional baseline here, the results show that the radiomics branch and BYOA module can help the model learn more important regions, which also improves the model's interpretability.

Experimental results demonstrate that our method outperforms the state-of-the-art algorithms, especially for the disease localization task, where our method can generate more accurate bounding boxes. \begin{table}
\begin{tabular}{c l c c c c c c c c c} \hline \hline
**T(IoU)** & Model & \multicolumn{2}{c}{Alelectasis} & \multicolumn{2}{c}{Cardiomegaly} & \multicolumn{2}{c}{Effusion} & \multicolumn{2}{c}{Infiltration} & \multicolumn{1}{c}{Mass} & \multicolumn{1}{c}{Nodule} & \multicolumn{1}{c}{Precmonia} & \multicolumn{1}{c}{Precumbotox} & \multicolumn{1}{c}{**Mean**} \\ \hline
0.1 & (Wang et al.,2017) & 0.69 & 0.94 & 0.66 & 0.71 & 0.40 & 0.14 & 0.63 & 0.38 & 0.569 \\  & ViT & 0.58 & 0.91 & 0.61 & 0.77 & 0.44 & 0.11 & 0.75 & 0.25 & 0.553 \\  & CheXT & 0.61 & 0.95 & 0.65 & 0.82 & 0.50 & 0.13 & 0.79 & 0.28 & **0.591** \\ \hline
0.2 & (Wang et al.,2017) & 0.47 & 0.68 & 0.45 & 0.48 & 0.26 & 0.05 & 0.35 & 0.23 & 0.371 \\  & ViT & 0.38 & 0.85 & 0.39 & 0.55 & 0.24 & 0.01 & 0.51 & 0.15 & 0.385 \\  & CheXT & 0.41 & 0.91 & 0.41 & 0.59 & 0.26 & 0.05 & 0.57 & 0.19 & **0.424** \\ \hline
0.3 & (Wang et al.,2017) & 0.24 & 0.46 & 0.30 & 0.28 & 0.15 & 0.04 & 0.17 & 0.13 & 0.221 \\  & ViT & 0.20 & 0.45 & 0.19 & 0.32 & 0.06 & 0.00 & 0.21 & 0.02 & 0.181 \\  & CheXT & 0.28 & 0.79 & 0.22 & 0.38 & 0.12 & 0.01 & 0.41 & 0.05 & **0.283** \\ \hline
0.4 & (Wang et al.,2017) & 0.09 & 0.28 & 0.20 & 0.12 & 0.07 & 0.01 & 0.08 & 0.07 & 0.115 \\  & ViT & 0.10 & 0.21 & 0.03 & 0.05 & 0.02 & 0.00 & 0.04 & 0.00 & 0.056 \\  & CheXT & 0.17 & 0.54 & 0.13 & 0.18 & 0.07 & 0.01 & 0.26 & 0.02 & **0.173** \\ \hline
0.5 & (Wang et al.,2017) & 0.05 & 0.18 & 0.11 & 0.07 & 0.01 & 0.01 & 0.03 & 0.03 & 0.061 \\  & ViT & 0.05 & 0.15 & 0.01 & 0.04 & 0.02 & 0.00 & 0.03 & 0.00 & 0.034 \\  & CheXT & 0.08 & 0.32 & 0.05 & 0.09 & 0.05 & 0.00 & 0.12 & 0.01 & **0.090** \\ \hline
0.6 & (Wang et al.,2017) & 0.02 & 0.08 & 0.05 & 0.02 & 0.00 & 0.01 & 0.02 & 0.03 & 0.029 \\  & ViT & 0.01 & 0.03 & 0.01 & 0.01 & 0.01 & 0.00 & 0.01 & 0.00 & 0.010 \\  & CheXT & 0.02 & 0.15 & 0.03 & 0.04 & 0.03 & 0.00 & 0.06 & 0.00 & **0.041** \\ \hline
0.7 & (Wang et al.,2017) & 0.01 & 0.03 & 0.02 & 0.00 & 0.00 & 0.00 & 0.01 & 0.02 & 0.011 \\  & ViT & 0.00 & 0.00 & 0.00 & 0.01 & 0.00 & 0.00 & 0.00 & 0.00 & 0.001 \\  & CheXT & 0.01 & 0.04 & 0.01 & 0.02 & 0.01 & 0.00 & 0.03 & 0.00 & **0.015** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Disease localization under varying IoU on the NIH Chest X-ray dataset. Please note that since our model doesn’t use any ground truth bounding box information, to fairly evaluate the performance of our model, we only consider the previous methods’ results under the same setting,

Input: Table 1: Comparison with the baseline models for AUC of each class and average AUC. For each column, bold values denote the best results.

Input: Table 2: Disease localization under varying IoU on the NIH Chest X-ray dataset. Please note that since our model doesn’t use any ground truth bounding box information, to fairly evaluate the performance of our model, we only consider the previous methods’ results under the same setting,

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate the claim "Our approach achieve superior classification and localization results against several competitive baselines, on the NIH chest X-ray benchmark."

The evidence partially supports this claim:

Classification: Table 1 shows CheXT achieves a mean AUC of 0.839, which is marginally better (0.011 higher) than some baselines like Han et al. 2020 (0.828). While superior, the improvement is modest.

Localization: Table 2 demonstrates CheXT consistently outperforms the baseline Wang et al. 2017 and ViT across different IoU thresholds, with clear numerical advantages in mean scores. The improvement is more substantial for localization.

While the claim is generally supported by evidence, the word "superior" slightly overstates the classification improvements which are incremental. However, it accurately reflects the localization results. The claim would be better stated with more specific quantification of the improvements.

<score>2</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

* Our approach achieve superior classification and localization results against several competitive baselines, on the NIH chest X-ray benchmark.

The review comment to be evaluated is:

Strength:
- The proposed approach and considering radiomic features along with learnt features is valuable and very relevant to the community.

Main Concerns and comments to improve the paper:
- The paper does not consider evaluation of the method on more publicly available datasets and the study is only limited to NIH-14 Chest X-ray data.
- Baselines and state of the art results on the test data: One important baseline is to make the prediction using only radiomic features.  Also,  [Han’20]  has reported much higher performance on NIH-14 and for the same task which they have not been included in the paper.
Improvement over the CNN baseline on classification: We see 0.011 improvement in mean AUC over Rajpurkar’17 as one of the baseline which is not significant.
- Details of the comparison setups are missing. I am wondering if these numbers are collected from the reported values by authors or they ran every SOTA against their data division (less likely since there is no error bound reported for SOTAs). And if they are-implemented or re-evaluated each method what are the details and consideration for a fair comparison and reproducing the results.
- Main contribution of the paper is to activate usage or radiomic features within the blackbox  structure of deep networks, however, there is no discussion or insight around the meaning and importance of multiple radiomic features that have been used. There are also works that include prior knowledge for Chest X-ray classification including [Han’20]. These features either unlock a better performance (which seems not being really successful in) or demystify the learnt feature space in a way that is pleasant for human interpreters. The paper does not provide such results.

Suggestion to improve the work:
- It would be useful to add some more explanation around radiomics term, feature extraction procedure and meaning of it, since this is not a common term for the machine learning community.
- Consider boosting the results and observation using a few more test data such as MIMIC and CheXpert that have the similar label space as NIH-14 and publicly being used in most of the related papers.
- The above mentioned datasets also can be used for evaluation of the out-of-domain generalization of the method which has not been discussed in the paper.
- Experimental setup and evaluation toward the SOTA can improve significantly. For the comparison purpose I would consider producing a statistical error range with either k-fold cross validation or boot-strapping for both CheXT and SOTAs. I would personally see a proper statistical comparison between a single best SOTA with the proposed method more valuable than having multiple models without any statistical validation.
- Adding interpretation around radiomic feature meaning, how the proposed method unlocked one or more of the 3 listed limitations of CNN methods that has been discussed in the Introduction.
- It is worth also discussing failure cases and the reasoning behind lower performance for some of the conditions such as effusion and pneumonia.

[Han'20] Han, Yan, Chongyan Chen, Liyan Tang, Mingquan Lin, Ajay Jaiswal, Song Wang, Ahmed Tewfik, George Shih, Ying Ding, and Yifan Peng. "Using Radiomics as Prior Knowledge for Thorax Disease Classification and Localization in Chest X-rays." arXiv preprint arXiv:2011.12506 (2020).

The evidence to be evaluated is:We conducted 3-fold cross-validation (Table 1). We compared CheXT with reference models, which have published state-of-the-art performance of disease classification on the NIH Chest X-ray dataset. We used Area under the Receiver Operating Characteristics (AUC) to estimate the performance of our model (FawcettTom, 2006). A higher AUC score implies a better classification model.

From the table, we can find that CheXT outperformed the baselines in the majority of diseases.

Compared to all baselines, CheXT achieves a mean AUC score of 0.839 using a dual-branch shallow Transformer across the 8 different diseases, which is 0.011 higher than the SOTA (uses DenseNet-121) (Rajpurkar et al., 2017) on disease classification. Specifically, our results significantly outperformed the best baseline models by 0.13, 0.09, and 0.02 in AUC for detecting Infiltration, Nodule, and Mass higher respectively.

The NIH Chest X-ray dataset has 880 images labeled by radiologists with the bounding box information. We have used this dataset to evaluate the performance of CheXT for abnormality localization.

Table 2 presents our evaluation results on all 880 images. We used (Wang et al., 2017) as our baseline to compare our localization results since it has the same experimental settings. For localization, we evaluated our detected regular rectangular regions against the annotated bounding boxes, using intersection over union ratio (IoU). Our localization results are only calculated for 880 images which have ground truth annotation for 8 diseases.

CheXT average performance across 8 diseases is significantly better than the baseline under all IoU thresholds. When the thresholds of IoU is set to 0.1, CheXT outperforms the baseline (Wang et al., 2017) in Cardiomegaly, infiltration, Mass, and Pneumonia. Even with higher thresholds, our model performs superior to baseline. For example, when evaluated at T(IoU) = 0.5, our "Cardiomegaly" accuracy is still 32%, while the reference model achieves only 18%.

More importantly, we also add the ViT as our additional baseline here, the results show that the radiomics branch and BYOA module can help the model learn more important regions, which also improves the model's interpretability.

Experimental results demonstrate that our method outperforms the state-of-the-art algorithms, especially for the disease localization task, where our method can generate more accurate bounding boxes. \begin{table}
\begin{tabular}{c l c c c c c c c c c} \hline \hline
**T(IoU)** & Model & \multicolumn{2}{c}{Alelectasis} & \multicolumn{2}{c}{Cardiomegaly} & \multicolumn{2}{c}{Effusion} & \multicolumn{2}{c}{Infiltration} & \multicolumn{1}{c}{Mass} & \multicolumn{1}{c}{Nodule} & \multicolumn{1}{c}{Precmonia} & \multicolumn{1}{c}{Precumbotox} & \multicolumn{1}{c}{**Mean**} \\ \hline
0.1 & (Wang et al.,2017) & 0.69 & 0.94 & 0.66 & 0.71 & 0.40 & 0.14 & 0.63 & 0.38 & 0.569 \\  & ViT & 0.58 & 0.91 & 0.61 & 0.77 & 0.44 & 0.11 & 0.75 & 0.25 & 0.553 \\  & CheXT & 0.61 & 0.95 & 0.65 & 0.82 & 0.50 & 0.13 & 0.79 & 0.28 & **0.591** \\ \hline
0.2 & (Wang et al.,2017) & 0.47 & 0.68 & 0.45 & 0.48 & 0.26 & 0.05 & 0.35 & 0.23 & 0.371 \\  & ViT & 0.38 & 0.85 & 0.39 & 0.55 & 0.24 & 0.01 & 0.51 & 0.15 & 0.385 \\  & CheXT & 0.41 & 0.91 & 0.41 & 0.59 & 0.26 & 0.05 & 0.57 & 0.19 & **0.424** \\ \hline
0.3 & (Wang et al.,2017) & 0.24 & 0.46 & 0.30 & 0.28 & 0.15 & 0.04 & 0.17 & 0.13 & 0.221 \\  & ViT & 0.20 & 0.45 & 0.19 & 0.32 & 0.06 & 0.00 & 0.21 & 0.02 & 0.181 \\  & CheXT & 0.28 & 0.79 & 0.22 & 0.38 & 0.12 & 0.01 & 0.41 & 0.05 & **0.283** \\ \hline
0.4 & (Wang et al.,2017) & 0.09 & 0.28 & 0.20 & 0.12 & 0.07 & 0.01 & 0.08 & 0.07 & 0.115 \\  & ViT & 0.10 & 0.21 & 0.03 & 0.05 & 0.02 & 0.00 & 0.04 & 0.00 & 0.056 \\  & CheXT & 0.17 & 0.54 & 0.13 & 0.18 & 0.07 & 0.01 & 0.26 & 0.02 & **0.173** \\ \hline
0.5 & (Wang et al.,2017) & 0.05 & 0.18 & 0.11 & 0.07 & 0.01 & 0.01 & 0.03 & 0.03 & 0.061 \\  & ViT & 0.05 & 0.15 & 0.01 & 0.04 & 0.02 & 0.00 & 0.03 & 0.00 & 0.034 \\  & CheXT & 0.08 & 0.32 & 0.05 & 0.09 & 0.05 & 0.00 & 0.12 & 0.01 & **0.090** \\ \hline
0.6 & (Wang et al.,2017) & 0.02 & 0.08 & 0.05 & 0.02 & 0.00 & 0.01 & 0.02 & 0.03 & 0.029 \\  & ViT & 0.01 & 0.03 & 0.01 & 0.01 & 0.01 & 0.00 & 0.01 & 0.00 & 0.010 \\  & CheXT & 0.02 & 0.15 & 0.03 & 0.04 & 0.03 & 0.00 & 0.06 & 0.00 & **0.041** \\ \hline
0.7 & (Wang et al.,2017) & 0.01 & 0.03 & 0.02 & 0.00 & 0.00 & 0.00 & 0.01 & 0.02 & 0.011 \\  & ViT & 0.00 & 0.00 & 0.00 & 0.01 & 0.00 & 0.00 & 0.00 & 0.00 & 0.001 \\  & CheXT & 0.01 & 0.04 & 0.01 & 0.02 & 0.01 & 0.00 & 0.03 & 0.00 & **0.015** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Disease localization under varying IoU on the NIH Chest X-ray dataset. Please note that since our model doesn’t use any ground truth bounding box information, to fairly evaluate the performance of our model, we only consider the previous methods’ results under the same setting,

Input: Table 1: Comparison with the baseline models for AUC of each class and average AUC. For each column, bold values denote the best results.

Input: Table 2: Disease localization under varying IoU on the NIH Chest X-ray dataset. Please note that since our model doesn’t use any ground truth bounding box information, to fairly evaluate the performance of our model, we only consider the previous methods’ results under the same setting,

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find the claim to be moderately overstated. While the results do show improvements in some areas, several key factors make the claim of "superior" results questionable:

1. Classification improvement is minimal (0.011 higher mean AUC) as noted in the review
2. Results are limited to only NIH dataset, lacking broader validation
3. Some key baselines are missing (e.g., Han'20)
4. Statistical validation and error bounds are lacking for most comparisons
5. While localization results show improvements, they're modest at higher IoU thresholds

The evidence supports showing improvements but calling them "superior" overstates the magnitude and comprehensiveness of the advantages demonstrated.

<score>4</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Notably, compared to existing approaches, CheXT generates more accurate disease localization for extracting radiomic features, by a significant average margin of 3.6% over different IoU thresholds.

The evidence to be evaluated is:Specifically, we introduce a dual-branch cross-attention Transformer where the first (primary) branch operates the image part, while the second (complementary) branch handles the radiomic features. To resolve the "chicken-and-egg" dilemma in extracting reliable radiomic features without bounding boxes, we have designed a novel _Bootstrap Your Own Attention_ (**BYOA**) module, using feedbacks to learn region localization for extracting radiomic features.

We experimentally observed PS-ViT outperforms ViT and other variants in our framework because it generates higher-quality and more structure-aware attention maps, which are crucial for estimating the pathology localization during training.

The NIH dataset also includes high-quality bounding box annotations for 880 images by radiologists. We separated these 880 images from our entire dataset, and they are used only to evaluate disease localization. A significant difference between our method and existing baseline methods (Liu et al., 2019; Li et al., 2018) is that our method does not require any training data related to the bounding box while others use some percentage of these images for training.

During our experiments, we found that CheXT performs worse when very large bounding boxes are generated. This observation meets our expectations since radiomic features are very subtle to bounding boxes. Specifically, the extracted radiomic features are more robust if more focal and accurate bounding boxes are given.

The NIH Chest X-ray dataset has 880 images labeled by radiologists with the bounding box information. We have used this dataset to evaluate the performance of CheXT for abnormality localization.

Table 2 presents our evaluation results on all 880 images. We used (Wang et al., 2017) as our baseline to compare our localization results since it has the same experimental settings. For localization, we evaluated our detected regular rectangular regions against the annotated bounding boxes, using intersection over union ratio (IoU).

We compared disease localization accuracy under varying IoU, with baselines having similar settings as CheXT (Table 2).

CheXT average performance across 8 diseases is significantly better than the baseline under all IoU thresholds. When the thresholds of IoU is set to 0.1, CheXT outperforms the baseline (Wang et al., 2017) in Cardiomegaly, infiltration, Mass, and Pneumonia. Even with higher thresholds, our model performs superior to baseline. For example, when evaluated at T(IoU) = 0.5, our "Cardiomegaly" accuracy is still 32%, while the reference model achieves only 18%.

More importantly, we also add the ViT as our additional baseline here, the results show that the radiomics branch and BYOA module can help the model learn more important regions, which also improves the model's interpretability.

Our approach differs from previous studies in the choice of universal modeling transformer, the use of radiomic features as prior knowledge, and a feedback loop for image and radiomic features to mutually interact with each other.

Experimental results demonstrate that our method outperforms the state-of-the-art algorithms, especially for the disease localization task, where our method can generate more accurate bounding boxes. \begin{table}
\begin{tabular}{c l c c c c c c c c c} \hline \hline
**T(IoU)** & Model & \multicolumn{2}{c}{Alelectasis} & \multicolumn{2}{c}{Cardiomegaly} & \multicolumn{2}{c}{Effusion} & \multicolumn{2}{c}{Infiltration} & \multicolumn{1}{c}{Mass} & \multicolumn{1}{c}{Nodule} & \multicolumn{1}{c}{Precmonia} & \multicolumn{1}{c}{Precumbotox} & \multicolumn{1}{c}{**Mean**} \\ \hline
0.1 & (Wang et al.,2017) & 0.69 & 0.94 & 0.66 & 0.71 & 0.40 & 0.14 & 0.63 & 0.38 & 0.569 \\  & ViT & 0.58 & 0.91 & 0.61 & 0.77 & 0.44 & 0.11 & 0.75 & 0.25 & 0.553 \\  & CheXT & 0.61 & 0.95 & 0.65 & 0.82 & 0.50 & 0.13 & 0.79 & 0.28 & **0.591** \\ \hline
0.2 & (Wang et al.,2017) & 0.47 & 0.68 & 0.45 & 0.48 & 0.26 & 0.05 & 0.35 & 0.23 & 0.371 \\  & ViT & 0.38 & 0.85 & 0.39 & 0.55 & 0.24 & 0.01 & 0.51 & 0.15 & 0.385 \\  & CheXT & 0.41 & 0.91 & 0.41 & 0.59 & 0.26 & 0.05 & 0.57 & 0.19 & **0.424** \\ \hline
0.3 & (Wang et al.,2017) & 0.24 & 0.46 & 0.30 & 0.28 & 0.15 & 0.04 & 0.17 & 0.13 & 0.221 \\  & ViT & 0.20 & 0.45 & 0.19 & 0.32 & 0.06 & 0.00 & 0.21 & 0.02 & 0.181 \\  & CheXT & 0.28 & 0.79 & 0.22 & 0.38 & 0.12 & 0.01 & 0.41 & 0.05 & **0.283** \\ \hline
0.4 & (Wang et al.,2017) & 0.09 & 0.28 & 0.20 & 0.12 & 0.07 & 0.01 & 0.08 & 0.07 & 0.115 \\  & ViT & 0.10 & 0.21 & 0.03 & 0.05 & 0.02 & 0.00 & 0.04 & 0.00 & 0.056 \\  & CheXT & 0.17 & 0.54 & 0.13 & 0.18 & 0.07 & 0.01 & 0.26 & 0.02 & **0.173** \\ \hline
0.5 & (Wang et al.,2017) & 0.05 & 0.18 & 0.11 & 0.07 & 0.01 & 0.01 & 0.03 & 0.03 & 0.061 \\  & ViT & 0.05 & 0.15 & 0.01 & 0.04 & 0.02 & 0.00 & 0.03 & 0.00 & 0.034 \\  & CheXT & 0.08 & 0.32 & 0.05 & 0.09 & 0.05 & 0.00 & 0.12 & 0.01 & **0.090** \\ \hline
0.6 & (Wang et al.,2017) & 0.02 & 0.08 & 0.05 & 0.02 & 0.00 & 0.01 & 0.02 & 0.03 & 0.029 \\  & ViT & 0.01 & 0.03 & 0.01 & 0.01 & 0.01 & 0.00 & 0.01 & 0.00 & 0.010 \\  & CheXT & 0.02 & 0.15 & 0.03 & 0.04 & 0.03 & 0.00 & 0.06 & 0.00 & **0.041** \\ \hline
0.7 & (Wang et al.,2017) & 0.01 & 0.03 & 0.02 & 0.00 & 0.00 & 0.00 & 0.01 & 0.02 & 0.011 \\  & ViT & 0.00 & 0.00 & 0.00 & 0.01 & 0.00 & 0.00 & 0.00 & 0.00 & 0.001 \\  & CheXT & 0.01 & 0.04 & 0.01 & 0.02 & 0.01 & 0.00 & 0.03 & 0.00 & **0.015** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Disease localization under varying IoU on the NIH Chest X-ray dataset. Please note that since our model doesn’t use any ground truth bounding box information, to fairly evaluate the performance of our model, we only consider the previous methods’ results under the same setting,

Input: Table 2: Disease localization under varying IoU on the NIH Chest X-ray dataset. Please note that since our model doesn’t use any ground truth bounding box information, to fairly evaluate the performance of our model, we only consider the previous methods’ results under the same setting,

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After careful review, the claim stating "CheXT generates more accurate disease localization for extracting radiomic features, by a significant average margin of 3.6% over different IoU thresholds" appears overstated. While Table 2 shows CheXT does outperform baselines across IoU thresholds, the margin varies significantly. For example, at IoU 0.1, the improvement is only 2.2% (0.591 vs 0.569), while at IoU 0.4, it's 5.8% (0.173 vs 0.115). The claim presents an average improvement without acknowledging this variation or discussing performance limitations at higher IoU thresholds where improvements are minimal.

<score>4</score>

Justification:
1. The claim overstates by using "significant average margin" without acknowledging performance variations across thresholds
2. The evidence shows inconsistent margins of improvement - some thresholds show minimal gains
3. No statistical significance tests are presented to support the term "significant"
4. The evidence acknowledges limitations ("CheXT performs worse when very large bounding boxes are generated") which aren't reflected in the claim
5. The performance drops substantially at higher IoU thresholds (>0.5) for all methods, including CheXT, which isn't addressed in the claim


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Notably, compared to existing approaches, CheXT generates more accurate disease localization for extracting radiomic features, by a significant average margin of 3.6% over different IoU thresholds.

The review comment to be evaluated is:

Strengths
+ The paper is fairly well written and presents a well-studied literature review.
+ The paper proposes an end-to-end transformer-based disease classification and localization model incorporating radiomic feature knowledge with cross-attention. The idea of bounding box generation and radiomic feature extraction in a feedback loop module is new and interesting.
+ Experimental setting including the ablation study for the effect of thresholds on attention maps and the effect of contrastive learning is good.

Weaknesses
- Exposition of results is not good. CheXT is compared against some weak baselines. A more appropriate CNN baseline should be something that includes similar radiomic feature extraction and bounding box generation [1]. Also, [1] is not cited in Related Work.
- The paper claims significant performance improvement by the CheXT model although no evidence (statistical test) is provided to support the claim.
-  Classification performance drops for some of the disease classes (effusion, pneumonia, pneumothorax) after incorporating the radiomics branch. It looks like radiomics branch is hurting the performance for these classes. Is there any impact of label distribution?
- In Fig. 3, two images look different. Why so?
- The training of CheXT is not necessarily semi-supervised as every image has at least disease class label.
- It is not clear how to obtain f_i and f_r; y, y' are not defined Eqn. (1).
- It would be more viable to report both classification and localization performances on the same 880 images. Is there any correlation between e.g., classification AUC and localization IoU?
- It is not clear how the model could improve generalization. There is no generalization assessment of the model for out-of-distribution cases. For example, using other datasets such as, CheXpert and MIMIC-CXR.
- In 4.1, the paper mentions the train/test/val splits, but Table 1 reports results after 3-fold cross-validation. They don't look consistent.
- In 4.4.2: "when the IoU is set to 0.1" should be threshold of IoU. "For Cardiomegaly, Infiltration, and Pneumonia." incomplete sentence.
- It is not clear how higher values of \lambda implies lower weight to contrastive loss in Eqn. (3). How does the model perform without the contrastive loss at all (\lambda=0)?
- There is no mention of the execution speed, computing resource, and failure cases (if any) for the proposed model.
- There are certainly other Transformer-based chest X-ray models such as [2].

1. https://arxiv.org/abs/2011.12506
2. https://arxiv.org/abs/2104.07235

The evidence to be evaluated is:Specifically, we introduce a dual-branch cross-attention Transformer where the first (primary) branch operates the image part, while the second (complementary) branch handles the radiomic features. To resolve the "chicken-and-egg" dilemma in extracting reliable radiomic features without bounding boxes, we have designed a novel _Bootstrap Your Own Attention_ (**BYOA**) module, using feedbacks to learn region localization for extracting radiomic features.

We experimentally observed PS-ViT outperforms ViT and other variants in our framework because it generates higher-quality and more structure-aware attention maps, which are crucial for estimating the pathology localization during training.

The NIH dataset also includes high-quality bounding box annotations for 880 images by radiologists. We separated these 880 images from our entire dataset, and they are used only to evaluate disease localization. A significant difference between our method and existing baseline methods (Liu et al., 2019; Li et al., 2018) is that our method does not require any training data related to the bounding box while others use some percentage of these images for training.

During our experiments, we found that CheXT performs worse when very large bounding boxes are generated. This observation meets our expectations since radiomic features are very subtle to bounding boxes. Specifically, the extracted radiomic features are more robust if more focal and accurate bounding boxes are given.

The NIH Chest X-ray dataset has 880 images labeled by radiologists with the bounding box information. We have used this dataset to evaluate the performance of CheXT for abnormality localization.

Table 2 presents our evaluation results on all 880 images. We used (Wang et al., 2017) as our baseline to compare our localization results since it has the same experimental settings. For localization, we evaluated our detected regular rectangular regions against the annotated bounding boxes, using intersection over union ratio (IoU).

We compared disease localization accuracy under varying IoU, with baselines having similar settings as CheXT (Table 2).

CheXT average performance across 8 diseases is significantly better than the baseline under all IoU thresholds. When the thresholds of IoU is set to 0.1, CheXT outperforms the baseline (Wang et al., 2017) in Cardiomegaly, infiltration, Mass, and Pneumonia. Even with higher thresholds, our model performs superior to baseline. For example, when evaluated at T(IoU) = 0.5, our "Cardiomegaly" accuracy is still 32%, while the reference model achieves only 18%.

More importantly, we also add the ViT as our additional baseline here, the results show that the radiomics branch and BYOA module can help the model learn more important regions, which also improves the model's interpretability.

Our approach differs from previous studies in the choice of universal modeling transformer, the use of radiomic features as prior knowledge, and a feedback loop for image and radiomic features to mutually interact with each other.

Experimental results demonstrate that our method outperforms the state-of-the-art algorithms, especially for the disease localization task, where our method can generate more accurate bounding boxes. \begin{table}
\begin{tabular}{c l c c c c c c c c c} \hline \hline
**T(IoU)** & Model & \multicolumn{2}{c}{Alelectasis} & \multicolumn{2}{c}{Cardiomegaly} & \multicolumn{2}{c}{Effusion} & \multicolumn{2}{c}{Infiltration} & \multicolumn{1}{c}{Mass} & \multicolumn{1}{c}{Nodule} & \multicolumn{1}{c}{Precmonia} & \multicolumn{1}{c}{Precumbotox} & \multicolumn{1}{c}{**Mean**} \\ \hline
0.1 & (Wang et al.,2017) & 0.69 & 0.94 & 0.66 & 0.71 & 0.40 & 0.14 & 0.63 & 0.38 & 0.569 \\  & ViT & 0.58 & 0.91 & 0.61 & 0.77 & 0.44 & 0.11 & 0.75 & 0.25 & 0.553 \\  & CheXT & 0.61 & 0.95 & 0.65 & 0.82 & 0.50 & 0.13 & 0.79 & 0.28 & **0.591** \\ \hline
0.2 & (Wang et al.,2017) & 0.47 & 0.68 & 0.45 & 0.48 & 0.26 & 0.05 & 0.35 & 0.23 & 0.371 \\  & ViT & 0.38 & 0.85 & 0.39 & 0.55 & 0.24 & 0.01 & 0.51 & 0.15 & 0.385 \\  & CheXT & 0.41 & 0.91 & 0.41 & 0.59 & 0.26 & 0.05 & 0.57 & 0.19 & **0.424** \\ \hline
0.3 & (Wang et al.,2017) & 0.24 & 0.46 & 0.30 & 0.28 & 0.15 & 0.04 & 0.17 & 0.13 & 0.221 \\  & ViT & 0.20 & 0.45 & 0.19 & 0.32 & 0.06 & 0.00 & 0.21 & 0.02 & 0.181 \\  & CheXT & 0.28 & 0.79 & 0.22 & 0.38 & 0.12 & 0.01 & 0.41 & 0.05 & **0.283** \\ \hline
0.4 & (Wang et al.,2017) & 0.09 & 0.28 & 0.20 & 0.12 & 0.07 & 0.01 & 0.08 & 0.07 & 0.115 \\  & ViT & 0.10 & 0.21 & 0.03 & 0.05 & 0.02 & 0.00 & 0.04 & 0.00 & 0.056 \\  & CheXT & 0.17 & 0.54 & 0.13 & 0.18 & 0.07 & 0.01 & 0.26 & 0.02 & **0.173** \\ \hline
0.5 & (Wang et al.,2017) & 0.05 & 0.18 & 0.11 & 0.07 & 0.01 & 0.01 & 0.03 & 0.03 & 0.061 \\  & ViT & 0.05 & 0.15 & 0.01 & 0.04 & 0.02 & 0.00 & 0.03 & 0.00 & 0.034 \\  & CheXT & 0.08 & 0.32 & 0.05 & 0.09 & 0.05 & 0.00 & 0.12 & 0.01 & **0.090** \\ \hline
0.6 & (Wang et al.,2017) & 0.02 & 0.08 & 0.05 & 0.02 & 0.00 & 0.01 & 0.02 & 0.03 & 0.029 \\  & ViT & 0.01 & 0.03 & 0.01 & 0.01 & 0.01 & 0.00 & 0.01 & 0.00 & 0.010 \\  & CheXT & 0.02 & 0.15 & 0.03 & 0.04 & 0.03 & 0.00 & 0.06 & 0.00 & **0.041** \\ \hline
0.7 & (Wang et al.,2017) & 0.01 & 0.03 & 0.02 & 0.00 & 0.00 & 0.00 & 0.01 & 0.02 & 0.011 \\  & ViT & 0.00 & 0.00 & 0.00 & 0.01 & 0.00 & 0.00 & 0.00 & 0.00 & 0.001 \\  & CheXT & 0.01 & 0.04 & 0.01 & 0.02 & 0.01 & 0.00 & 0.03 & 0.00 & **0.015** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Disease localization under varying IoU on the NIH Chest X-ray dataset. Please note that since our model doesn’t use any ground truth bounding box information, to fairly evaluate the performance of our model, we only consider the previous methods’ results under the same setting,

Input: Table 2: Disease localization under varying IoU on the NIH Chest X-ray dataset. Please note that since our model doesn’t use any ground truth bounding box information, to fairly evaluate the performance of our model, we only consider the previous methods’ results under the same setting,

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, the statement "CheXT generates more accurate disease localization by a significant average margin of 3.6%" appears overstated. While Table 2 shows CheXT does outperform baselines across different IoU thresholds, the margins vary considerably. For example, at IoU=0.1, CheXT's mean score is 0.591 vs 0.569 for Wang et al. (only 2.2% difference). The claim presents the improvement as uniformly significant without acknowledging this variation or providing statistical significance tests, which was specifically noted as a weakness in the review comments.

<score>4</score>

Justification:
1. The claim overstates the consistency and magnitude of improvement without statistical validation
2. The review explicitly criticizes that "no evidence (statistical test) is provided to support the claim"
3. The 3.6% margin is not clearly demonstrated in the provided data
4. While CheXT does show improvements, they vary by condition and threshold
5. The paper compares against "weak baselines" according to reviewer feedback
6. The claim would be better stated with more precise language about where and when improvements occur, with appropriate statistical support


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Notably, compared to existing approaches, CheXT generates more accurate disease localization for extracting radiomic features, by a significant average margin of 3.6% over different IoU thresholds.

The review comment to be evaluated is:

1) Radiomics is a term that most people in the ICLR community are not familiar with. What is it exactily? Is it a set of radiological/image features or bio-markers in medical images? Or something else? The authors should consider explaining it in plain language or ML language.

2) Prior multi-modal feature fusion work in the chest x-ray domain should be discussed, examples below.

    - Moradi, Mehdi, Ali Madani, Yaniv Gur, Yufan Guo, and Tanveer Syeda-Mahmood. "Bimodal network architectures for automatic generation of image annotation from text." arXiv preprint arXiv:1809.01610 (2018).

    - Wang, Xiaosong, Yifan Peng, Le Lu, Zhiyong Lu, and Ronald M. Summers. "TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-rays." arXiv e-prints (2018): arXiv-1801.

    - Xue, Yuan, and Xiaolei Huang. "Improved disease classification in chest x-rays with transferred features from report generation." In International Conference on Information Processing in Medical Imaging, pp. 125-138. Springer, Cham, 2019.

    - Liao, Ruizhi, Daniel Moyer, Miriam Cha, Keegan Quigley, Seth Berkowitz, Steven Horng, Polina Golland, and William M. Wells. "Multimodal Representation Learning via Maximization of Local Mutual Information." arXiv preprint arXiv:2103.04537 (2021).

3) I don't see "prior knowledge" and how it's utilized in this work. Isn't the so called prior knowledge in this paper essentially extracted from the data (chest x-ray images)?

4) What are the patch sizes? How are they determined?

5) There are 14 pathology labels in the NIH chest x-ray dataset. Why/how did the authors select the 8 classes out of the 14?

6) In Table 1, make the best results bold instead of in red.

7) Are the AUC values generated on the test set? If so, I'm concerned about overfitting by over tuning the hyper-parameters.

The evidence to be evaluated is:Specifically, we introduce a dual-branch cross-attention Transformer where the first (primary) branch operates the image part, while the second (complementary) branch handles the radiomic features. To resolve the "chicken-and-egg" dilemma in extracting reliable radiomic features without bounding boxes, we have designed a novel _Bootstrap Your Own Attention_ (**BYOA**) module, using feedbacks to learn region localization for extracting radiomic features.

We experimentally observed PS-ViT outperforms ViT and other variants in our framework because it generates higher-quality and more structure-aware attention maps, which are crucial for estimating the pathology localization during training.

The NIH dataset also includes high-quality bounding box annotations for 880 images by radiologists. We separated these 880 images from our entire dataset, and they are used only to evaluate disease localization. A significant difference between our method and existing baseline methods (Liu et al., 2019; Li et al., 2018) is that our method does not require any training data related to the bounding box while others use some percentage of these images for training.

During our experiments, we found that CheXT performs worse when very large bounding boxes are generated. This observation meets our expectations since radiomic features are very subtle to bounding boxes. Specifically, the extracted radiomic features are more robust if more focal and accurate bounding boxes are given.

The NIH Chest X-ray dataset has 880 images labeled by radiologists with the bounding box information. We have used this dataset to evaluate the performance of CheXT for abnormality localization.

Table 2 presents our evaluation results on all 880 images. We used (Wang et al., 2017) as our baseline to compare our localization results since it has the same experimental settings. For localization, we evaluated our detected regular rectangular regions against the annotated bounding boxes, using intersection over union ratio (IoU).

We compared disease localization accuracy under varying IoU, with baselines having similar settings as CheXT (Table 2).

CheXT average performance across 8 diseases is significantly better than the baseline under all IoU thresholds. When the thresholds of IoU is set to 0.1, CheXT outperforms the baseline (Wang et al., 2017) in Cardiomegaly, infiltration, Mass, and Pneumonia. Even with higher thresholds, our model performs superior to baseline. For example, when evaluated at T(IoU) = 0.5, our "Cardiomegaly" accuracy is still 32%, while the reference model achieves only 18%.

More importantly, we also add the ViT as our additional baseline here, the results show that the radiomics branch and BYOA module can help the model learn more important regions, which also improves the model's interpretability.

Our approach differs from previous studies in the choice of universal modeling transformer, the use of radiomic features as prior knowledge, and a feedback loop for image and radiomic features to mutually interact with each other.

Experimental results demonstrate that our method outperforms the state-of-the-art algorithms, especially for the disease localization task, where our method can generate more accurate bounding boxes. \begin{table}
\begin{tabular}{c l c c c c c c c c c} \hline \hline
**T(IoU)** & Model & \multicolumn{2}{c}{Alelectasis} & \multicolumn{2}{c}{Cardiomegaly} & \multicolumn{2}{c}{Effusion} & \multicolumn{2}{c}{Infiltration} & \multicolumn{1}{c}{Mass} & \multicolumn{1}{c}{Nodule} & \multicolumn{1}{c}{Precmonia} & \multicolumn{1}{c}{Precumbotox} & \multicolumn{1}{c}{**Mean**} \\ \hline
0.1 & (Wang et al.,2017) & 0.69 & 0.94 & 0.66 & 0.71 & 0.40 & 0.14 & 0.63 & 0.38 & 0.569 \\  & ViT & 0.58 & 0.91 & 0.61 & 0.77 & 0.44 & 0.11 & 0.75 & 0.25 & 0.553 \\  & CheXT & 0.61 & 0.95 & 0.65 & 0.82 & 0.50 & 0.13 & 0.79 & 0.28 & **0.591** \\ \hline
0.2 & (Wang et al.,2017) & 0.47 & 0.68 & 0.45 & 0.48 & 0.26 & 0.05 & 0.35 & 0.23 & 0.371 \\  & ViT & 0.38 & 0.85 & 0.39 & 0.55 & 0.24 & 0.01 & 0.51 & 0.15 & 0.385 \\  & CheXT & 0.41 & 0.91 & 0.41 & 0.59 & 0.26 & 0.05 & 0.57 & 0.19 & **0.424** \\ \hline
0.3 & (Wang et al.,2017) & 0.24 & 0.46 & 0.30 & 0.28 & 0.15 & 0.04 & 0.17 & 0.13 & 0.221 \\  & ViT & 0.20 & 0.45 & 0.19 & 0.32 & 0.06 & 0.00 & 0.21 & 0.02 & 0.181 \\  & CheXT & 0.28 & 0.79 & 0.22 & 0.38 & 0.12 & 0.01 & 0.41 & 0.05 & **0.283** \\ \hline
0.4 & (Wang et al.,2017) & 0.09 & 0.28 & 0.20 & 0.12 & 0.07 & 0.01 & 0.08 & 0.07 & 0.115 \\  & ViT & 0.10 & 0.21 & 0.03 & 0.05 & 0.02 & 0.00 & 0.04 & 0.00 & 0.056 \\  & CheXT & 0.17 & 0.54 & 0.13 & 0.18 & 0.07 & 0.01 & 0.26 & 0.02 & **0.173** \\ \hline
0.5 & (Wang et al.,2017) & 0.05 & 0.18 & 0.11 & 0.07 & 0.01 & 0.01 & 0.03 & 0.03 & 0.061 \\  & ViT & 0.05 & 0.15 & 0.01 & 0.04 & 0.02 & 0.00 & 0.03 & 0.00 & 0.034 \\  & CheXT & 0.08 & 0.32 & 0.05 & 0.09 & 0.05 & 0.00 & 0.12 & 0.01 & **0.090** \\ \hline
0.6 & (Wang et al.,2017) & 0.02 & 0.08 & 0.05 & 0.02 & 0.00 & 0.01 & 0.02 & 0.03 & 0.029 \\  & ViT & 0.01 & 0.03 & 0.01 & 0.01 & 0.01 & 0.00 & 0.01 & 0.00 & 0.010 \\  & CheXT & 0.02 & 0.15 & 0.03 & 0.04 & 0.03 & 0.00 & 0.06 & 0.00 & **0.041** \\ \hline
0.7 & (Wang et al.,2017) & 0.01 & 0.03 & 0.02 & 0.00 & 0.00 & 0.00 & 0.01 & 0.02 & 0.011 \\  & ViT & 0.00 & 0.00 & 0.00 & 0.01 & 0.00 & 0.00 & 0.00 & 0.00 & 0.001 \\  & CheXT & 0.01 & 0.04 & 0.01 & 0.02 & 0.01 & 0.00 & 0.03 & 0.00 & **0.015** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Disease localization under varying IoU on the NIH Chest X-ray dataset. Please note that since our model doesn’t use any ground truth bounding box information, to fairly evaluate the performance of our model, we only consider the previous methods’ results under the same setting,

Input: Table 2: Disease localization under varying IoU on the NIH Chest X-ray dataset. Please note that since our model doesn’t use any ground truth bounding box information, to fairly evaluate the performance of our model, we only consider the previous methods’ results under the same setting,

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find the stated "significant average margin of 3.6%" claim needs careful analysis. Looking at Table 2, CheXT does show consistent improvements over baselines across different IoU thresholds, but the margins vary. For example:

At IoU 0.1: CheXT (0.591) vs Wang (0.569) = 2.2% improvement
At IoU 0.3: CheXT (0.283) vs Wang (0.221) = 6.2% improvement
At IoU 0.5: CheXT (0.090) vs Wang (0.061) = 2.9% improvement

While CheXT does show better performance, claiming a "significant average margin of 3.6%" appears slightly overstated without providing the detailed breakdown of how this average was calculated across different IoU thresholds. The improvements are real but vary considerably across thresholds.

<score>2</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Notably, compared to existing approaches, CheXT generates more accurate disease localization for extracting radiomic features, by a significant average margin of 3.6% over different IoU thresholds.

The review comment to be evaluated is:

Strength:
- The proposed approach and considering radiomic features along with learnt features is valuable and very relevant to the community.

Main Concerns and comments to improve the paper:
- The paper does not consider evaluation of the method on more publicly available datasets and the study is only limited to NIH-14 Chest X-ray data.
- Baselines and state of the art results on the test data: One important baseline is to make the prediction using only radiomic features.  Also,  [Han’20]  has reported much higher performance on NIH-14 and for the same task which they have not been included in the paper.
Improvement over the CNN baseline on classification: We see 0.011 improvement in mean AUC over Rajpurkar’17 as one of the baseline which is not significant.
- Details of the comparison setups are missing. I am wondering if these numbers are collected from the reported values by authors or they ran every SOTA against their data division (less likely since there is no error bound reported for SOTAs). And if they are-implemented or re-evaluated each method what are the details and consideration for a fair comparison and reproducing the results.
- Main contribution of the paper is to activate usage or radiomic features within the blackbox  structure of deep networks, however, there is no discussion or insight around the meaning and importance of multiple radiomic features that have been used. There are also works that include prior knowledge for Chest X-ray classification including [Han’20]. These features either unlock a better performance (which seems not being really successful in) or demystify the learnt feature space in a way that is pleasant for human interpreters. The paper does not provide such results.

Suggestion to improve the work:
- It would be useful to add some more explanation around radiomics term, feature extraction procedure and meaning of it, since this is not a common term for the machine learning community.
- Consider boosting the results and observation using a few more test data such as MIMIC and CheXpert that have the similar label space as NIH-14 and publicly being used in most of the related papers.
- The above mentioned datasets also can be used for evaluation of the out-of-domain generalization of the method which has not been discussed in the paper.
- Experimental setup and evaluation toward the SOTA can improve significantly. For the comparison purpose I would consider producing a statistical error range with either k-fold cross validation or boot-strapping for both CheXT and SOTAs. I would personally see a proper statistical comparison between a single best SOTA with the proposed method more valuable than having multiple models without any statistical validation.
- Adding interpretation around radiomic feature meaning, how the proposed method unlocked one or more of the 3 listed limitations of CNN methods that has been discussed in the Introduction.
- It is worth also discussing failure cases and the reasoning behind lower performance for some of the conditions such as effusion and pneumonia.

[Han'20] Han, Yan, Chongyan Chen, Liyan Tang, Mingquan Lin, Ajay Jaiswal, Song Wang, Ahmed Tewfik, George Shih, Ying Ding, and Yifan Peng. "Using Radiomics as Prior Knowledge for Thorax Disease Classification and Localization in Chest X-rays." arXiv preprint arXiv:2011.12506 (2020).

The evidence to be evaluated is:Specifically, we introduce a dual-branch cross-attention Transformer where the first (primary) branch operates the image part, while the second (complementary) branch handles the radiomic features. To resolve the "chicken-and-egg" dilemma in extracting reliable radiomic features without bounding boxes, we have designed a novel _Bootstrap Your Own Attention_ (**BYOA**) module, using feedbacks to learn region localization for extracting radiomic features.

We experimentally observed PS-ViT outperforms ViT and other variants in our framework because it generates higher-quality and more structure-aware attention maps, which are crucial for estimating the pathology localization during training.

The NIH dataset also includes high-quality bounding box annotations for 880 images by radiologists. We separated these 880 images from our entire dataset, and they are used only to evaluate disease localization. A significant difference between our method and existing baseline methods (Liu et al., 2019; Li et al., 2018) is that our method does not require any training data related to the bounding box while others use some percentage of these images for training.

During our experiments, we found that CheXT performs worse when very large bounding boxes are generated. This observation meets our expectations since radiomic features are very subtle to bounding boxes. Specifically, the extracted radiomic features are more robust if more focal and accurate bounding boxes are given.

The NIH Chest X-ray dataset has 880 images labeled by radiologists with the bounding box information. We have used this dataset to evaluate the performance of CheXT for abnormality localization.

Table 2 presents our evaluation results on all 880 images. We used (Wang et al., 2017) as our baseline to compare our localization results since it has the same experimental settings. For localization, we evaluated our detected regular rectangular regions against the annotated bounding boxes, using intersection over union ratio (IoU).

We compared disease localization accuracy under varying IoU, with baselines having similar settings as CheXT (Table 2).

CheXT average performance across 8 diseases is significantly better than the baseline under all IoU thresholds. When the thresholds of IoU is set to 0.1, CheXT outperforms the baseline (Wang et al., 2017) in Cardiomegaly, infiltration, Mass, and Pneumonia. Even with higher thresholds, our model performs superior to baseline. For example, when evaluated at T(IoU) = 0.5, our "Cardiomegaly" accuracy is still 32%, while the reference model achieves only 18%.

More importantly, we also add the ViT as our additional baseline here, the results show that the radiomics branch and BYOA module can help the model learn more important regions, which also improves the model's interpretability.

Our approach differs from previous studies in the choice of universal modeling transformer, the use of radiomic features as prior knowledge, and a feedback loop for image and radiomic features to mutually interact with each other.

Experimental results demonstrate that our method outperforms the state-of-the-art algorithms, especially for the disease localization task, where our method can generate more accurate bounding boxes. \begin{table}
\begin{tabular}{c l c c c c c c c c c} \hline \hline
**T(IoU)** & Model & \multicolumn{2}{c}{Alelectasis} & \multicolumn{2}{c}{Cardiomegaly} & \multicolumn{2}{c}{Effusion} & \multicolumn{2}{c}{Infiltration} & \multicolumn{1}{c}{Mass} & \multicolumn{1}{c}{Nodule} & \multicolumn{1}{c}{Precmonia} & \multicolumn{1}{c}{Precumbotox} & \multicolumn{1}{c}{**Mean**} \\ \hline
0.1 & (Wang et al.,2017) & 0.69 & 0.94 & 0.66 & 0.71 & 0.40 & 0.14 & 0.63 & 0.38 & 0.569 \\  & ViT & 0.58 & 0.91 & 0.61 & 0.77 & 0.44 & 0.11 & 0.75 & 0.25 & 0.553 \\  & CheXT & 0.61 & 0.95 & 0.65 & 0.82 & 0.50 & 0.13 & 0.79 & 0.28 & **0.591** \\ \hline
0.2 & (Wang et al.,2017) & 0.47 & 0.68 & 0.45 & 0.48 & 0.26 & 0.05 & 0.35 & 0.23 & 0.371 \\  & ViT & 0.38 & 0.85 & 0.39 & 0.55 & 0.24 & 0.01 & 0.51 & 0.15 & 0.385 \\  & CheXT & 0.41 & 0.91 & 0.41 & 0.59 & 0.26 & 0.05 & 0.57 & 0.19 & **0.424** \\ \hline
0.3 & (Wang et al.,2017) & 0.24 & 0.46 & 0.30 & 0.28 & 0.15 & 0.04 & 0.17 & 0.13 & 0.221 \\  & ViT & 0.20 & 0.45 & 0.19 & 0.32 & 0.06 & 0.00 & 0.21 & 0.02 & 0.181 \\  & CheXT & 0.28 & 0.79 & 0.22 & 0.38 & 0.12 & 0.01 & 0.41 & 0.05 & **0.283** \\ \hline
0.4 & (Wang et al.,2017) & 0.09 & 0.28 & 0.20 & 0.12 & 0.07 & 0.01 & 0.08 & 0.07 & 0.115 \\  & ViT & 0.10 & 0.21 & 0.03 & 0.05 & 0.02 & 0.00 & 0.04 & 0.00 & 0.056 \\  & CheXT & 0.17 & 0.54 & 0.13 & 0.18 & 0.07 & 0.01 & 0.26 & 0.02 & **0.173** \\ \hline
0.5 & (Wang et al.,2017) & 0.05 & 0.18 & 0.11 & 0.07 & 0.01 & 0.01 & 0.03 & 0.03 & 0.061 \\  & ViT & 0.05 & 0.15 & 0.01 & 0.04 & 0.02 & 0.00 & 0.03 & 0.00 & 0.034 \\  & CheXT & 0.08 & 0.32 & 0.05 & 0.09 & 0.05 & 0.00 & 0.12 & 0.01 & **0.090** \\ \hline
0.6 & (Wang et al.,2017) & 0.02 & 0.08 & 0.05 & 0.02 & 0.00 & 0.01 & 0.02 & 0.03 & 0.029 \\  & ViT & 0.01 & 0.03 & 0.01 & 0.01 & 0.01 & 0.00 & 0.01 & 0.00 & 0.010 \\  & CheXT & 0.02 & 0.15 & 0.03 & 0.04 & 0.03 & 0.00 & 0.06 & 0.00 & **0.041** \\ \hline
0.7 & (Wang et al.,2017) & 0.01 & 0.03 & 0.02 & 0.00 & 0.00 & 0.00 & 0.01 & 0.02 & 0.011 \\  & ViT & 0.00 & 0.00 & 0.00 & 0.01 & 0.00 & 0.00 & 0.00 & 0.00 & 0.001 \\  & CheXT & 0.01 & 0.04 & 0.01 & 0.02 & 0.01 & 0.00 & 0.03 & 0.00 & **0.015** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Disease localization under varying IoU on the NIH Chest X-ray dataset. Please note that since our model doesn’t use any ground truth bounding box information, to fairly evaluate the performance of our model, we only consider the previous methods’ results under the same setting,

Input: Table 2: Disease localization under varying IoU on the NIH Chest X-ray dataset. Please note that since our model doesn’t use any ground truth bounding box information, to fairly evaluate the performance of our model, we only consider the previous methods’ results under the same setting,

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on the evidence provided and review comments, I'll evaluate the claim: "Notably, compared to existing approaches, CheXT generates more accurate disease localization for extracting radiomic features, by a significant average margin of 3.6% over different IoU thresholds."

Looking at Table 2, while CheXT does show improvements over baselines across different IoU thresholds, the claim appears somewhat overstated for several reasons:

1. The improvement varies across different IoU thresholds and diseases, with some showing minimal gains
2. The review points out missing statistical validation and error bounds
3. Limited comparison to only two baselines (Wang et al. 2017 and ViT)
4. The reviewer notes that Han'20 reported higher performance but wasn't included
5. Testing was limited to only NIH-14 dataset without cross-dataset validation

The "significant average margin of 3.6%" appears to be technically accurate based on the mean values shown, but characterizing it as "significant" without statistical validation and considering the limitations makes this somewhat overstated.

<score>3</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Inspired by its recent success, this paper proposes **CheXT**, the first Transformer-based chest X-ray model.

The evidence to be evaluated is:Recently, Vision Transformer (ViT) (Dosovitskiy et al., 2020) achieved state-of-the-art classification on ImageNet by directly applying Transformers with global self-attention to full-sized images. Inspired by the promising performance of ViT, researchers have recently applied the idea to medical images.

(Park et al., 2021) utilized a hybrid framework of CNN and Transformer for Covid-19 prediction.

An overview of CheXT is illustrated in Figure 2. In the following subsections, we will first present Cross-attention Vision Transformer (CrossViT), a recent two-branch ViT backbone on which CheXT is built, and then describe our many unique improvements customized for Chest X-ray analysis. ViT first converts an image into a sequence of patch tokens by dividing the image with certain patch size and linearly projecting each patch into tokens. A special token (CLS) is added in front of the sequence, as in the original BERT (Devlin et al., 2018). Then, all tokens are passed through stacked Transformer encoders. Finally, the hidden state corresponding to the CLS token is used as the aggregate sequence representation for image classification. A Transformer encoder is composed of a sequence of blocks where each block contains multiheaded self-attention with a feed-forward network.

In CheXT, we extend their idea by treating image itself as one "view" and the radiomic feature extracted from the same image as another "view" (Figure 2). The two views are the jointly learned by interacting through cross-attention. Transformer serves as the modality-agnostic backbone for both. Specifically, we introduce a dual-branch cross-attention Transformer where the first (primary) branch operates the image part, while the second (complementary) branch handles the radiomic features. To resolve the "chicken-and-egg" dilemma in extracting reliable radiomic features without bounding boxes, we have designed a novel _Bootstrap Your Own Attention_ (**BYOA**) module, using feedbacks to learn region localization for extracting radiomic features. A simple yet effective module is also utilized to fuse information between the branches. In the subsequent sections, we will describe the two branches, the BOYA module, and the fusion module.

In our framework, we aim to make the hidden features of the CLS similar to the radiomic features to learn the localization of pathologies in the chest X-rays. For this branch, we use the vanilla Transformer (Liu et al., 2017) as the radiomic features encoder.

As shown in Figure 2, CheXT is trained using the linear combination of the supervised focal classification and unsupervised contrastive losses. For the supervised classification, considering that the chest X-ray dataset is usually highly imbalanced, we adopt the focal loss (Pasupa et al., 2020). For unsupervised contrastive learning, we use the cross-view contrastive loss (Chen et al., 2020).

We feed the output of the CLS tokens of two branches \(I_{cls}\) and \(R_{cls}\) to a simple linear classifier.

Not only we compared CheXT with previous CNN-based SOTA models, but also several Transformer-based models.

Compared to all baselines, CheXT achieves a mean AUC score of 0.839 using a dual-branch shallow Transformer across the 8 different diseases, which is 0.011 higher than the SOTA (uses DenseNet-121) (Rajpurkar et al., 2017) on disease classification. Specifically, our results significantly outperformed the best baseline models by 0.13, 0.09, and 0.02 in AUC for detecting Infiltration, Nodule, and Mass higher respectively. Besides, compared to the Transformer-based models, the key difference is that we utilize the extracted radiomic features for disease prediction, which improves the accuracy and adds the model's interpretability due to the utilization of self-designed handcrafted radiomic features.

In a calculated ratio, they help CheXT to learn both disease-level and patient-level discriminative visual features.

In this paper, we propose an end-to-end knowledge-guided cross-attention Transformer, named CheXT that can jointly model abnormality classification and localization without the supervision of localization annotation of chest X-rays. Our approach differs from previous studies in the choice of universal modeling transformer, the use of radiomic features as prior knowledge, and a feedback loop for image and radiomic features to mutually interact with each other.

Input: Figure 2: Overview of our proposed CheXT. It contains two branches, the Image branch and the Radiomics branch, to process the image and radiomic features (generated by the BYOA module shown in Figure 3), respectively. The output tokens are then fused by an efficient module via cross attention of the CLS tokens. Finally, the output of two CLS tokens (Icls andRcls) are used for disease classification. We minimize the classification errors via a focal loss. Since we train with both labeled and unlabeled images, we leverage a contrastive learning strategy. Specifically, we generate the image view zi = gi(Icls) by a projection head gi. Similarily, we generate the radiomic view zr = gr(Rcls) by another projection head gr. We maximize the agreement between zi and zr via a contrastive loss (NT-Xent). Of note, the contrastive loss is only active during the training.

Input: Figure 3: Overview of our BYOA module. For the input chest X-rays, we look at the self-attention of the CLS token of the Image branch on the heads of the final output of the cross attention module. Then we apply a threshold (0.1) (which means we only keep the top 10% active pixels in the whole attention map) to generate the bounding boxes. Then with the generated bounding boxes, we use the Pyradiomic tool as the radiomic extractor to extract the radiomic features.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find the claim "CheXT is the first Transformer-based chest X-ray model" to be overstated. The evidence shows that prior Transformer applications existed in medical imaging, specifically mentioning Park et al. (2021) who "utilized a hybrid framework of CNN and Transformer for Covid-19 prediction." While CheXT offers novel contributions through its dual-branch architecture and BYOA module, claiming to be the "first" Transformer-based chest X-ray model ignores previous work in the field. The evidence does support CheXT being innovative, but not being first.

<score>4</score>

Justification:
1. Direct contradiction: Evidence mentions prior Transformer use in medical imaging, specifically Park et al. (2021)'s work
2. Novel elements: While CheXT introduces new features (dual-branch architecture, BYOA module, radiomic features), these make it innovative but not "first"
3. The evidence supports CheXT's performance improvements ("0.011 higher than the SOTA") and unique approach, but this doesn't justify the "first" claim
4. The paper's own language is more measured in other sections, describing it as different from previous studies rather than being first: "Our approach differs from previous studies in the choice of universal modeling transformer..."


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Inspired by its recent success, this paper proposes **CheXT**, the first Transformer-based chest X-ray model.

The review comment to be evaluated is:

Strengths
+ The paper is fairly well written and presents a well-studied literature review.
+ The paper proposes an end-to-end transformer-based disease classification and localization model incorporating radiomic feature knowledge with cross-attention. The idea of bounding box generation and radiomic feature extraction in a feedback loop module is new and interesting.
+ Experimental setting including the ablation study for the effect of thresholds on attention maps and the effect of contrastive learning is good.

Weaknesses
- Exposition of results is not good. CheXT is compared against some weak baselines. A more appropriate CNN baseline should be something that includes similar radiomic feature extraction and bounding box generation [1]. Also, [1] is not cited in Related Work.
- The paper claims significant performance improvement by the CheXT model although no evidence (statistical test) is provided to support the claim.
-  Classification performance drops for some of the disease classes (effusion, pneumonia, pneumothorax) after incorporating the radiomics branch. It looks like radiomics branch is hurting the performance for these classes. Is there any impact of label distribution?
- In Fig. 3, two images look different. Why so?
- The training of CheXT is not necessarily semi-supervised as every image has at least disease class label.
- It is not clear how to obtain f_i and f_r; y, y' are not defined Eqn. (1).
- It would be more viable to report both classification and localization performances on the same 880 images. Is there any correlation between e.g., classification AUC and localization IoU?
- It is not clear how the model could improve generalization. There is no generalization assessment of the model for out-of-distribution cases. For example, using other datasets such as, CheXpert and MIMIC-CXR.
- In 4.1, the paper mentions the train/test/val splits, but Table 1 reports results after 3-fold cross-validation. They don't look consistent.
- In 4.4.2: "when the IoU is set to 0.1" should be threshold of IoU. "For Cardiomegaly, Infiltration, and Pneumonia." incomplete sentence.
- It is not clear how higher values of \lambda implies lower weight to contrastive loss in Eqn. (3). How does the model perform without the contrastive loss at all (\lambda=0)?
- There is no mention of the execution speed, computing resource, and failure cases (if any) for the proposed model.
- There are certainly other Transformer-based chest X-ray models such as [2].

1. https://arxiv.org/abs/2011.12506
2. https://arxiv.org/abs/2104.07235

The evidence to be evaluated is:Recently, Vision Transformer (ViT) (Dosovitskiy et al., 2020) achieved state-of-the-art classification on ImageNet by directly applying Transformers with global self-attention to full-sized images. Inspired by the promising performance of ViT, researchers have recently applied the idea to medical images.

(Park et al., 2021) utilized a hybrid framework of CNN and Transformer for Covid-19 prediction.

An overview of CheXT is illustrated in Figure 2. In the following subsections, we will first present Cross-attention Vision Transformer (CrossViT), a recent two-branch ViT backbone on which CheXT is built, and then describe our many unique improvements customized for Chest X-ray analysis. ViT first converts an image into a sequence of patch tokens by dividing the image with certain patch size and linearly projecting each patch into tokens. A special token (CLS) is added in front of the sequence, as in the original BERT (Devlin et al., 2018). Then, all tokens are passed through stacked Transformer encoders. Finally, the hidden state corresponding to the CLS token is used as the aggregate sequence representation for image classification. A Transformer encoder is composed of a sequence of blocks where each block contains multiheaded self-attention with a feed-forward network.

In CheXT, we extend their idea by treating image itself as one "view" and the radiomic feature extracted from the same image as another "view" (Figure 2). The two views are the jointly learned by interacting through cross-attention. Transformer serves as the modality-agnostic backbone for both. Specifically, we introduce a dual-branch cross-attention Transformer where the first (primary) branch operates the image part, while the second (complementary) branch handles the radiomic features. To resolve the "chicken-and-egg" dilemma in extracting reliable radiomic features without bounding boxes, we have designed a novel _Bootstrap Your Own Attention_ (**BYOA**) module, using feedbacks to learn region localization for extracting radiomic features. A simple yet effective module is also utilized to fuse information between the branches. In the subsequent sections, we will describe the two branches, the BOYA module, and the fusion module.

In our framework, we aim to make the hidden features of the CLS similar to the radiomic features to learn the localization of pathologies in the chest X-rays. For this branch, we use the vanilla Transformer (Liu et al., 2017) as the radiomic features encoder.

As shown in Figure 2, CheXT is trained using the linear combination of the supervised focal classification and unsupervised contrastive losses. For the supervised classification, considering that the chest X-ray dataset is usually highly imbalanced, we adopt the focal loss (Pasupa et al., 2020). For unsupervised contrastive learning, we use the cross-view contrastive loss (Chen et al., 2020).

We feed the output of the CLS tokens of two branches \(I_{cls}\) and \(R_{cls}\) to a simple linear classifier.

Not only we compared CheXT with previous CNN-based SOTA models, but also several Transformer-based models.

Compared to all baselines, CheXT achieves a mean AUC score of 0.839 using a dual-branch shallow Transformer across the 8 different diseases, which is 0.011 higher than the SOTA (uses DenseNet-121) (Rajpurkar et al., 2017) on disease classification. Specifically, our results significantly outperformed the best baseline models by 0.13, 0.09, and 0.02 in AUC for detecting Infiltration, Nodule, and Mass higher respectively. Besides, compared to the Transformer-based models, the key difference is that we utilize the extracted radiomic features for disease prediction, which improves the accuracy and adds the model's interpretability due to the utilization of self-designed handcrafted radiomic features.

In a calculated ratio, they help CheXT to learn both disease-level and patient-level discriminative visual features.

In this paper, we propose an end-to-end knowledge-guided cross-attention Transformer, named CheXT that can jointly model abnormality classification and localization without the supervision of localization annotation of chest X-rays. Our approach differs from previous studies in the choice of universal modeling transformer, the use of radiomic features as prior knowledge, and a feedback loop for image and radiomic features to mutually interact with each other.

Input: Figure 2: Overview of our proposed CheXT. It contains two branches, the Image branch and the Radiomics branch, to process the image and radiomic features (generated by the BYOA module shown in Figure 3), respectively. The output tokens are then fused by an efficient module via cross attention of the CLS tokens. Finally, the output of two CLS tokens (Icls andRcls) are used for disease classification. We minimize the classification errors via a focal loss. Since we train with both labeled and unlabeled images, we leverage a contrastive learning strategy. Specifically, we generate the image view zi = gi(Icls) by a projection head gi. Similarily, we generate the radiomic view zr = gr(Rcls) by another projection head gr. We maximize the agreement between zi and zr via a contrastive loss (NT-Xent). Of note, the contrastive loss is only active during the training.

Input: Figure 3: Overview of our BYOA module. For the input chest X-rays, we look at the self-attention of the CLS token of the Image branch on the heads of the final output of the cross attention module. Then we apply a threshold (0.1) (which means we only keep the top 10% active pixels in the whole attention map) to generate the bounding boxes. Then with the generated bounding boxes, we use the Pyradiomic tool as the radiomic extractor to extract the radiomic features.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me assess the claim "this paper proposes CheXT, the first Transformer-based chest X-ray model" based on the provided evidence and review comments.

The claim appears significantly overstated based on two key points:

1. The review comment explicitly points out: "There are certainly other Transformer-based chest X-ray models such as [2]" and provides a citation to a prior work (https://arxiv.org/abs/2104.07235).

2. The evidence itself acknowledges prior Transformer applications in medical imaging, noting "researchers have recently applied the idea to medical images" and cites Park et al. (2021) who "utilized a hybrid framework of CNN and Transformer for Covid-19 prediction."

While CheXT appears to be novel in its specific approach using radiomic features and cross-attention, claiming it as the "first" Transformer-based chest X-ray model is demonstrably incorrect given the existence of prior work.

<score>5</score>