Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Specifically, CheXT consists of an image branch and a radiomics branch, interacted by cross-attention layers.

The evidence to be evaluated is:In the following subsections, we will first present Cross-attention Vision Transformer (CrossViT), a recent two-branch ViT backbone on which CheXT is built, and then describe our many unique improvements customized for Chest X-ray analysis.

In CheXT, we extend their idea by treating image itself as one "view" and the radiomic feature extracted from the same image as another "view" (Figure 2). The two views are the jointly learned by interacting through cross-attention. Transformer serves as the modality-agnostic backbone for both. Specifically, we introduce a dual-branch cross-attention Transformer where the first (primary) branch operates the image part, while the second (complementary) branch handles the radiomic features.

On the other hand, image features would benefit from the guidance from radiomic features that encode important domain knowledge. The learning of image and radiomic features are fully entangled, forming a challenging "chicken-and-egg" loop.

BYOA contains two components: attention map generation and radiomic feature extraction.

Given the original images and generated bounding boxes, we used the Pyradiomic tool to extract radiomic features (Van Griethuysen et al., 2017). **Cross-Attention Fusion Module.** This fusion involves the CLS token of one branch and patch tokens of the other branch. As the CLS token is the aggregate representation of the branch, this interaction helps include information at different scale.

We feed the output of the CLS tokens of two branches \(I_{cls}\) and \(R_{cls}\) to a simple linear classifier.

The difference is that we maximize agreement between two feature views extracted from different input formats, one the image and the other radiomics. Given an anchor chest X-ray in a minibatch, the positive sample will be its radiomic feature view, and the negative samples will be other chest X-rays (either image or radiomics).

We build our image branch encoder based on PS-ViT (Yue et al., 2021) and apply their default hyperparameters for training.

For the radiomic branch encoder, since the radiomic features are already informative features, we use a more shallow Transformer (1-layers) to encode the radiomic features. And we add one more cross-attention layer to fuse these two format features.

Besides, compared to the Transformer-based models, the key difference is that we utilize the extracted radiomic features for disease prediction, which improves the accuracy and adds the model's interpretability due to the utilization of self-designed handcrafted radiomic features.

More importantly, we also add the ViT as our additional baseline here, the results show that the radiomics branch and BYOA module can help the model learn more important regions, which also improves the model's interpretability.

In this paper, we propose an end-to-end knowledge-guided cross-attention Transformer, named CheXT that can jointly model abnormality classification and localization without the supervision of localization annotation of chest X-rays. Our approach differs from previous studies in the choice of universal modeling transformer, the use of radiomic features as prior knowledge, and a feedback loop for image and radiomic features to mutually interact with each other.

Input: Figure 2: Overview of our proposed CheXT. It contains two branches, the Image branch and the Radiomics branch, to process the image and radiomic features (generated by the BYOA module shown in Figure 3), respectively. The output tokens are then fused by an efficient module via cross attention of the CLS tokens. Finally, the output of two CLS tokens (Icls andRcls) are used for disease classification. We minimize the classification errors via a focal loss. Since we train with both labeled and unlabeled images, we leverage a contrastive learning strategy. Specifically, we generate the image view zi = gi(Icls) by a projection head gi. Similarily, we generate the radiomic view zr = gr(Rcls) by another projection head gr. We maximize the agreement between zi and zr via a contrastive loss (NT-Xent). Of note, the contrastive loss is only active during the training.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, the statement "CheXT consists of an image branch and a radiomics branch, interacted by cross-attention layers" is well-stated and accurately reflects the evidence without exaggeration.

The evidence clearly shows:
1. The architecture contains two distinct branches (image and radiomics)
2. These branches interact through cross-attention, as shown in both the text ("two views are jointly learned by interacting through cross-attention") and Figure 2
3. The cross-attention fusion module is explicitly described, involving CLS token interaction between branches

The claim is precise, technically accurate, and avoids overstating the complexity or capabilities of the system. It aligns perfectly with both the textual descriptions and the architectural diagram provided.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Specifically, CheXT consists of an image branch and a radiomics branch, interacted by cross-attention layers.

The review comment to be evaluated is:

Strengths
+ The paper is fairly well written and presents a well-studied literature review.
+ The paper proposes an end-to-end transformer-based disease classification and localization model incorporating radiomic feature knowledge with cross-attention. The idea of bounding box generation and radiomic feature extraction in a feedback loop module is new and interesting.
+ Experimental setting including the ablation study for the effect of thresholds on attention maps and the effect of contrastive learning is good.

Weaknesses
- Exposition of results is not good. CheXT is compared against some weak baselines. A more appropriate CNN baseline should be something that includes similar radiomic feature extraction and bounding box generation [1]. Also, [1] is not cited in Related Work.
- The paper claims significant performance improvement by the CheXT model although no evidence (statistical test) is provided to support the claim.
-  Classification performance drops for some of the disease classes (effusion, pneumonia, pneumothorax) after incorporating the radiomics branch. It looks like radiomics branch is hurting the performance for these classes. Is there any impact of label distribution?
- In Fig. 3, two images look different. Why so?
- The training of CheXT is not necessarily semi-supervised as every image has at least disease class label.
- It is not clear how to obtain f_i and f_r; y, y' are not defined Eqn. (1).
- It would be more viable to report both classification and localization performances on the same 880 images. Is there any correlation between e.g., classification AUC and localization IoU?
- It is not clear how the model could improve generalization. There is no generalization assessment of the model for out-of-distribution cases. For example, using other datasets such as, CheXpert and MIMIC-CXR.
- In 4.1, the paper mentions the train/test/val splits, but Table 1 reports results after 3-fold cross-validation. They don't look consistent.
- In 4.4.2: "when the IoU is set to 0.1" should be threshold of IoU. "For Cardiomegaly, Infiltration, and Pneumonia." incomplete sentence.
- It is not clear how higher values of \lambda implies lower weight to contrastive loss in Eqn. (3). How does the model perform without the contrastive loss at all (\lambda=0)?
- There is no mention of the execution speed, computing resource, and failure cases (if any) for the proposed model.
- There are certainly other Transformer-based chest X-ray models such as [2].

1. https://arxiv.org/abs/2011.12506
2. https://arxiv.org/abs/2104.07235

The evidence to be evaluated is:In the following subsections, we will first present Cross-attention Vision Transformer (CrossViT), a recent two-branch ViT backbone on which CheXT is built, and then describe our many unique improvements customized for Chest X-ray analysis.

In CheXT, we extend their idea by treating image itself as one "view" and the radiomic feature extracted from the same image as another "view" (Figure 2). The two views are the jointly learned by interacting through cross-attention. Transformer serves as the modality-agnostic backbone for both. Specifically, we introduce a dual-branch cross-attention Transformer where the first (primary) branch operates the image part, while the second (complementary) branch handles the radiomic features.

On the other hand, image features would benefit from the guidance from radiomic features that encode important domain knowledge. The learning of image and radiomic features are fully entangled, forming a challenging "chicken-and-egg" loop.

BYOA contains two components: attention map generation and radiomic feature extraction.

Given the original images and generated bounding boxes, we used the Pyradiomic tool to extract radiomic features (Van Griethuysen et al., 2017). **Cross-Attention Fusion Module.** This fusion involves the CLS token of one branch and patch tokens of the other branch. As the CLS token is the aggregate representation of the branch, this interaction helps include information at different scale.

We feed the output of the CLS tokens of two branches \(I_{cls}\) and \(R_{cls}\) to a simple linear classifier.

The difference is that we maximize agreement between two feature views extracted from different input formats, one the image and the other radiomics. Given an anchor chest X-ray in a minibatch, the positive sample will be its radiomic feature view, and the negative samples will be other chest X-rays (either image or radiomics).

We build our image branch encoder based on PS-ViT (Yue et al., 2021) and apply their default hyperparameters for training.

For the radiomic branch encoder, since the radiomic features are already informative features, we use a more shallow Transformer (1-layers) to encode the radiomic features. And we add one more cross-attention layer to fuse these two format features.

Besides, compared to the Transformer-based models, the key difference is that we utilize the extracted radiomic features for disease prediction, which improves the accuracy and adds the model's interpretability due to the utilization of self-designed handcrafted radiomic features.

More importantly, we also add the ViT as our additional baseline here, the results show that the radiomics branch and BYOA module can help the model learn more important regions, which also improves the model's interpretability.

In this paper, we propose an end-to-end knowledge-guided cross-attention Transformer, named CheXT that can jointly model abnormality classification and localization without the supervision of localization annotation of chest X-rays. Our approach differs from previous studies in the choice of universal modeling transformer, the use of radiomic features as prior knowledge, and a feedback loop for image and radiomic features to mutually interact with each other.

Input: Figure 2: Overview of our proposed CheXT. It contains two branches, the Image branch and the Radiomics branch, to process the image and radiomic features (generated by the BYOA module shown in Figure 3), respectively. The output tokens are then fused by an efficient module via cross attention of the CLS tokens. Finally, the output of two CLS tokens (Icls andRcls) are used for disease classification. We minimize the classification errors via a focal loss. Since we train with both labeled and unlabeled images, we leverage a contrastive learning strategy. Specifically, we generate the image view zi = gi(Icls) by a projection head gi. Similarily, we generate the radiomic view zr = gr(Rcls) by another projection head gr. We maximize the agreement between zi and zr via a contrastive loss (NT-Xent). Of note, the contrastive loss is only active during the training.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim, evidence, and review comments, I'll evaluate whether "CheXT consists of an image branch and a radiomics branch, interacted by cross-attention layers" is well-stated or overstated.

The evidence strongly supports this claim through:
1. Clear architectural details shown in Figure 2 depicting both branches
2. Explicit descriptions of the cross-attention fusion module
3. Detailed explanations of how the two branches interact
4. Supporting evidence of implementation details for both branches

The claim is factual, precise, and directly supported by the evidence without exaggeration. It makes no overreaching statements about performance or capabilities beyond what is demonstrated. The review comments, while critical of other aspects, don't dispute this architectural claim.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Specifically, CheXT consists of an image branch and a radiomics branch, interacted by cross-attention layers.

The review comment to be evaluated is:

1) Radiomics is a term that most people in the ICLR community are not familiar with. What is it exactily? Is it a set of radiological/image features or bio-markers in medical images? Or something else? The authors should consider explaining it in plain language or ML language.

2) Prior multi-modal feature fusion work in the chest x-ray domain should be discussed, examples below.

    - Moradi, Mehdi, Ali Madani, Yaniv Gur, Yufan Guo, and Tanveer Syeda-Mahmood. "Bimodal network architectures for automatic generation of image annotation from text." arXiv preprint arXiv:1809.01610 (2018).

    - Wang, Xiaosong, Yifan Peng, Le Lu, Zhiyong Lu, and Ronald M. Summers. "TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-rays." arXiv e-prints (2018): arXiv-1801.

    - Xue, Yuan, and Xiaolei Huang. "Improved disease classification in chest x-rays with transferred features from report generation." In International Conference on Information Processing in Medical Imaging, pp. 125-138. Springer, Cham, 2019.

    - Liao, Ruizhi, Daniel Moyer, Miriam Cha, Keegan Quigley, Seth Berkowitz, Steven Horng, Polina Golland, and William M. Wells. "Multimodal Representation Learning via Maximization of Local Mutual Information." arXiv preprint arXiv:2103.04537 (2021).

3) I don't see "prior knowledge" and how it's utilized in this work. Isn't the so called prior knowledge in this paper essentially extracted from the data (chest x-ray images)?

4) What are the patch sizes? How are they determined?

5) There are 14 pathology labels in the NIH chest x-ray dataset. Why/how did the authors select the 8 classes out of the 14?

6) In Table 1, make the best results bold instead of in red.

7) Are the AUC values generated on the test set? If so, I'm concerned about overfitting by over tuning the hyper-parameters.

The evidence to be evaluated is:In the following subsections, we will first present Cross-attention Vision Transformer (CrossViT), a recent two-branch ViT backbone on which CheXT is built, and then describe our many unique improvements customized for Chest X-ray analysis.

In CheXT, we extend their idea by treating image itself as one "view" and the radiomic feature extracted from the same image as another "view" (Figure 2). The two views are the jointly learned by interacting through cross-attention. Transformer serves as the modality-agnostic backbone for both. Specifically, we introduce a dual-branch cross-attention Transformer where the first (primary) branch operates the image part, while the second (complementary) branch handles the radiomic features.

On the other hand, image features would benefit from the guidance from radiomic features that encode important domain knowledge. The learning of image and radiomic features are fully entangled, forming a challenging "chicken-and-egg" loop.

BYOA contains two components: attention map generation and radiomic feature extraction.

Given the original images and generated bounding boxes, we used the Pyradiomic tool to extract radiomic features (Van Griethuysen et al., 2017). **Cross-Attention Fusion Module.** This fusion involves the CLS token of one branch and patch tokens of the other branch. As the CLS token is the aggregate representation of the branch, this interaction helps include information at different scale.

We feed the output of the CLS tokens of two branches \(I_{cls}\) and \(R_{cls}\) to a simple linear classifier.

The difference is that we maximize agreement between two feature views extracted from different input formats, one the image and the other radiomics. Given an anchor chest X-ray in a minibatch, the positive sample will be its radiomic feature view, and the negative samples will be other chest X-rays (either image or radiomics).

We build our image branch encoder based on PS-ViT (Yue et al., 2021) and apply their default hyperparameters for training.

For the radiomic branch encoder, since the radiomic features are already informative features, we use a more shallow Transformer (1-layers) to encode the radiomic features. And we add one more cross-attention layer to fuse these two format features.

Besides, compared to the Transformer-based models, the key difference is that we utilize the extracted radiomic features for disease prediction, which improves the accuracy and adds the model's interpretability due to the utilization of self-designed handcrafted radiomic features.

More importantly, we also add the ViT as our additional baseline here, the results show that the radiomics branch and BYOA module can help the model learn more important regions, which also improves the model's interpretability.

In this paper, we propose an end-to-end knowledge-guided cross-attention Transformer, named CheXT that can jointly model abnormality classification and localization without the supervision of localization annotation of chest X-rays. Our approach differs from previous studies in the choice of universal modeling transformer, the use of radiomic features as prior knowledge, and a feedback loop for image and radiomic features to mutually interact with each other.

Input: Figure 2: Overview of our proposed CheXT. It contains two branches, the Image branch and the Radiomics branch, to process the image and radiomic features (generated by the BYOA module shown in Figure 3), respectively. The output tokens are then fused by an efficient module via cross attention of the CLS tokens. Finally, the output of two CLS tokens (Icls andRcls) are used for disease classification. We minimize the classification errors via a focal loss. Since we train with both labeled and unlabeled images, we leverage a contrastive learning strategy. Specifically, we generate the image view zi = gi(Icls) by a projection head gi. Similarily, we generate the radiomic view zr = gr(Rcls) by another projection head gr. We maximize the agreement between zi and zr via a contrastive loss (NT-Xent). Of note, the contrastive loss is only active during the training.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me evaluate the claim "CheXT consists of an image branch and a radiomics branch, interacted by cross-attention layers" based on the provided evidence:

The evidence strongly supports this claim through:
1. Figure 2 clearly shows the two-branch architecture with image and radiomics branches
2. Multiple text excerpts confirm the dual-branch structure and cross-attention interaction:
- "CheXT, we extend their idea by treating image itself as one 'view' and the radiomic feature extracted from the same image as another 'view'"
- "a dual-branch cross-attention Transformer where the first (primary) branch operates the image part, while the second (complementary) branch handles the radiomic features"
- The cross-attention fusion is explicitly described in the evidence

This claim is well-stated and accurately reflects the fundamental architecture without exaggeration or overstatement.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Specifically, CheXT consists of an image branch and a radiomics branch, interacted by cross-attention layers.

The review comment to be evaluated is:

Strength:
- The proposed approach and considering radiomic features along with learnt features is valuable and very relevant to the community.

Main Concerns and comments to improve the paper:
- The paper does not consider evaluation of the method on more publicly available datasets and the study is only limited to NIH-14 Chest X-ray data.
- Baselines and state of the art results on the test data: One important baseline is to make the prediction using only radiomic features.  Also,  [Han’20]  has reported much higher performance on NIH-14 and for the same task which they have not been included in the paper.
Improvement over the CNN baseline on classification: We see 0.011 improvement in mean AUC over Rajpurkar’17 as one of the baseline which is not significant.
- Details of the comparison setups are missing. I am wondering if these numbers are collected from the reported values by authors or they ran every SOTA against their data division (less likely since there is no error bound reported for SOTAs). And if they are-implemented or re-evaluated each method what are the details and consideration for a fair comparison and reproducing the results.
- Main contribution of the paper is to activate usage or radiomic features within the blackbox  structure of deep networks, however, there is no discussion or insight around the meaning and importance of multiple radiomic features that have been used. There are also works that include prior knowledge for Chest X-ray classification including [Han’20]. These features either unlock a better performance (which seems not being really successful in) or demystify the learnt feature space in a way that is pleasant for human interpreters. The paper does not provide such results.

Suggestion to improve the work:
- It would be useful to add some more explanation around radiomics term, feature extraction procedure and meaning of it, since this is not a common term for the machine learning community.
- Consider boosting the results and observation using a few more test data such as MIMIC and CheXpert that have the similar label space as NIH-14 and publicly being used in most of the related papers.
- The above mentioned datasets also can be used for evaluation of the out-of-domain generalization of the method which has not been discussed in the paper.
- Experimental setup and evaluation toward the SOTA can improve significantly. For the comparison purpose I would consider producing a statistical error range with either k-fold cross validation or boot-strapping for both CheXT and SOTAs. I would personally see a proper statistical comparison between a single best SOTA with the proposed method more valuable than having multiple models without any statistical validation.
- Adding interpretation around radiomic feature meaning, how the proposed method unlocked one or more of the 3 listed limitations of CNN methods that has been discussed in the Introduction.
- It is worth also discussing failure cases and the reasoning behind lower performance for some of the conditions such as effusion and pneumonia.

[Han'20] Han, Yan, Chongyan Chen, Liyan Tang, Mingquan Lin, Ajay Jaiswal, Song Wang, Ahmed Tewfik, George Shih, Ying Ding, and Yifan Peng. "Using Radiomics as Prior Knowledge for Thorax Disease Classification and Localization in Chest X-rays." arXiv preprint arXiv:2011.12506 (2020).

The evidence to be evaluated is:In the following subsections, we will first present Cross-attention Vision Transformer (CrossViT), a recent two-branch ViT backbone on which CheXT is built, and then describe our many unique improvements customized for Chest X-ray analysis.

In CheXT, we extend their idea by treating image itself as one "view" and the radiomic feature extracted from the same image as another "view" (Figure 2). The two views are the jointly learned by interacting through cross-attention. Transformer serves as the modality-agnostic backbone for both. Specifically, we introduce a dual-branch cross-attention Transformer where the first (primary) branch operates the image part, while the second (complementary) branch handles the radiomic features.

On the other hand, image features would benefit from the guidance from radiomic features that encode important domain knowledge. The learning of image and radiomic features are fully entangled, forming a challenging "chicken-and-egg" loop.

BYOA contains two components: attention map generation and radiomic feature extraction.

Given the original images and generated bounding boxes, we used the Pyradiomic tool to extract radiomic features (Van Griethuysen et al., 2017). **Cross-Attention Fusion Module.** This fusion involves the CLS token of one branch and patch tokens of the other branch. As the CLS token is the aggregate representation of the branch, this interaction helps include information at different scale.

We feed the output of the CLS tokens of two branches \(I_{cls}\) and \(R_{cls}\) to a simple linear classifier.

The difference is that we maximize agreement between two feature views extracted from different input formats, one the image and the other radiomics. Given an anchor chest X-ray in a minibatch, the positive sample will be its radiomic feature view, and the negative samples will be other chest X-rays (either image or radiomics).

We build our image branch encoder based on PS-ViT (Yue et al., 2021) and apply their default hyperparameters for training.

For the radiomic branch encoder, since the radiomic features are already informative features, we use a more shallow Transformer (1-layers) to encode the radiomic features. And we add one more cross-attention layer to fuse these two format features.

Besides, compared to the Transformer-based models, the key difference is that we utilize the extracted radiomic features for disease prediction, which improves the accuracy and adds the model's interpretability due to the utilization of self-designed handcrafted radiomic features.

More importantly, we also add the ViT as our additional baseline here, the results show that the radiomics branch and BYOA module can help the model learn more important regions, which also improves the model's interpretability.

In this paper, we propose an end-to-end knowledge-guided cross-attention Transformer, named CheXT that can jointly model abnormality classification and localization without the supervision of localization annotation of chest X-rays. Our approach differs from previous studies in the choice of universal modeling transformer, the use of radiomic features as prior knowledge, and a feedback loop for image and radiomic features to mutually interact with each other.

Input: Figure 2: Overview of our proposed CheXT. It contains two branches, the Image branch and the Radiomics branch, to process the image and radiomic features (generated by the BYOA module shown in Figure 3), respectively. The output tokens are then fused by an efficient module via cross attention of the CLS tokens. Finally, the output of two CLS tokens (Icls andRcls) are used for disease classification. We minimize the classification errors via a focal loss. Since we train with both labeled and unlabeled images, we leverage a contrastive learning strategy. Specifically, we generate the image view zi = gi(Icls) by a projection head gi. Similarily, we generate the radiomic view zr = gr(Rcls) by another projection head gr. We maximize the agreement between zi and zr via a contrastive loss (NT-Xent). Of note, the contrastive loss is only active during the training.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find that the claim "CheXT consists of an image branch and a radiomics branch, interacted by cross-attention layers" is well-stated and accurately reflects the evidence. The provided figure and text clearly show the dual-branch architecture with image and radiomics branches connected via cross-attention. The evidence specifically describes how "the two views are jointly learned by interacting through cross-attention" and details the "Cross-Attention Fusion Module" that connects the CLS tokens between branches. The claim is factual and doesn't overstate the architecture's components or capabilities.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

During training, the image branch leverages its learned attention to estimate pathology localization, which is then utilized to extract radiomic features from images in the radiomics branch.

The evidence to be evaluated is:We believe that the joint utilization of radiomics and interpretable learning techniques in our framework can further advance accurate yet interpretable learning in the medical image domain.

For example, (Oktay et al., 2018; Wang et al., 2019; Chen et al., 2021) used the attention mechanism to boost the performance of medical image segmentation.

(Han et al., 2021) applied pre-extracted radiomic features to guide pneumonia detection from chest X-ray images.

In CheXT, we extend their idea by treating image itself as one "view" and the radiomic feature extracted from the same image as another "view" (Figure 2). The two views are the jointly learned by interacting through cross-attention.

To resolve the "chicken-and-egg" dilemma in extracting reliable radiomic features without bounding boxes, we have designed a novel _Bootstrap Your Own Attention_ (**BYOA**) module, using feedbacks to learn region localization for extracting radiomic features.

The primary image branch uses a Progress-Sampling ViT (PS-ViT) (Yue et al., 2021) as its backbone. Unlike the vanilla ViT that splits images into fixed-size tokens, PS-ViT utilizes an iterative and progressive sampling strategy to locate discriminative regions and avoid over-partition object structures. We experimentally observed PS-ViT outperforms ViT and other variants in our framework because it generates higher-quality and more structure-aware attention maps, which are crucial for estimating the pathology localization during training.

In our framework, we aim to make the hidden features of the CLS similar to the radiomic features to learn the localization of pathologies in the chest X-rays. For this branch, we use the vanilla Transformer (Liu et al., 2017) as the radiomic features encoder.

Our main _roadblock_ is how to generate robust radiomic features without pathology localization. On one hand, radiomic features are dependent on and highly sensitive to the local image regions of interest, for which we have no bounding box annotation. On the other hand, image features would benefit from the guidance from radiomic features that encode important domain knowledge. The learning of image and radiomic features are fully entangled, forming a challenging "chicken-and-egg" loop. To address this issue, we design **BYOA** to constitute an end-to-end feedback loop that can bootstrap accurate pathology localization from image data, without any bounding box used for training (Figure 3). BYOA contains two components: attention map generation and radiomic feature extraction.

Here, we have two CLS tokens from two branches, but the attention maps only come from the Image branch. Then we apply a threshold on the self-attention maps to generate bounding boxes for the extraction of radiomic features.

Specifically, the threshold is designed as how much active pixels to keep in the generated attention map. The more active pixels we keep, the larger the generated bounding box is.

Given the original images and generated bounding boxes, we used the Pyradiomic tool to extract radiomic features (Van Griethuysen et al., 2017).

The difference is that we maximize agreement between two feature views extracted from different input formats, one the image and the other radiomics. Given an anchor chest X-ray in a minibatch, the positive sample will be its radiomic feature view, and the negative samples will be other chest X-rays (either image or radiomics).

Since for chest x-rays, there exists two labels, disease class labels and pathology bounding box annotations. In our case, we only have access to the disease labels, however, bounding box annotations are more important than the disease class labels for chest x-rays. Here, when we say "semi-supervised", we refer to the bounding box annotations, not disease class labels.

Higher values of \(T\) imply smaller bounding boxes to extract radiomic features. During our experiments, we found that CheXT performs worse when very large bounding boxes are generated. This observation meets our expectations since radiomic features are very subtle to bounding boxes. Specifically, the extracted radiomic features are more robust if more focal and accurate bounding boxes are given.

In a calculated ratio, they help CheXT to learn both disease-level and patient-level discriminative visual features.

We have used this dataset to evaluate the performance of CheXT for abnormality localization.

For localization, we evaluated our detected regular rectangular regions against the annotated bounding boxes, using intersection over union ratio (IoU).

More importantly, we also add the ViT as our additional baseline here, the results show that the radiomics branch and BYOA module can help the model learn more important regions, which also improves the model's interpretability.

Experimental results demonstrate that our method outperforms the state-of-the-art algorithms, especially for the disease localization task, where our method can generate more accurate bounding boxes.

Input: Figure 2: Overview of our proposed CheXT. It contains two branches, the Image branch and the Radiomics branch, to process the image and radiomic features (generated by the BYOA module shown in Figure 3), respectively. The output tokens are then fused by an efficient module via cross attention of the CLS tokens. Finally, the output of two CLS tokens (Icls andRcls) are used for disease classification. We minimize the classification errors via a focal loss. Since we train with both labeled and unlabeled images, we leverage a contrastive learning strategy. Specifically, we generate the image view zi = gi(Icls) by a projection head gi. Similarily, we generate the radiomic view zr = gr(Rcls) by another projection head gr. We maximize the agreement between zi and zr via a contrastive loss (NT-Xent). Of note, the contrastive loss is only active during the training.

Input: Figure 3: Overview of our BYOA module. For the input chest X-rays, we look at the self-attention of the CLS token of the Image branch on the heads of the final output of the cross attention module. Then we apply a threshold (0.1) (which means we only keep the top 10% active pixels in the whole attention map) to generate the bounding boxes. Then with the generated bounding boxes, we use the Pyradiomic tool as the radiomic extractor to extract the radiomic features.

Input: Figure 1: General overview of our CheXT framework for Cardiopulmonary Disease Classification and Localization from chest X-rays. CheXT takes the chest X-ray image as the input and outputs a heatmap for pathology localization, based on which the bounding box could be obtained. Radiomic features are further extracted from the bounded region and are fed to predict the disease.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find that the claim is relatively well-stated but has some minor overstatement issues.

The evidence shows that the image branch does use attention mechanisms to identify regions of interest (through PS-ViT and BYOA), which are then used for radiomic feature extraction. However, the term "learned attention" may slightly overstate the certainty of the process, as the system uses a threshold-based approach (keeping top 10% active pixels) rather than truly "learned" localization. The framework does demonstrate this general workflow, but the actual mechanism is more mechanical than the claim suggests.

The evidence strongly supports the basic workflow described, including figures showing the process (especially Figure 3), but the language could be more precise about the technical implementation.

<score>2</score>

Justification:
1. The basic claim about the workflow is supported by multiple pieces of evidence, including:
- "BYOA contains two components: attention map generation and radiomic feature extraction"
- "we apply a threshold on the self-attention maps to generate bounding boxes for the extraction of radiomic features"
- Figures 2 and 3 clearly illustrate this process

2. Minor overstatement issues:
- The term "learned attention" implies more sophistication than the actual threshold-based approach
- The process is more mechanical than truly learned, using "threshold (0.1) (which means we only keep the top 10% active pixels in the whole attention map)"
- The system's limitations and dependencies are not acknowledged in the claim


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

During training, the image branch leverages its learned attention to estimate pathology localization, which is then utilized to extract radiomic features from images in the radiomics branch.

The review comment to be evaluated is:

Strengths
+ The paper is fairly well written and presents a well-studied literature review.
+ The paper proposes an end-to-end transformer-based disease classification and localization model incorporating radiomic feature knowledge with cross-attention. The idea of bounding box generation and radiomic feature extraction in a feedback loop module is new and interesting.
+ Experimental setting including the ablation study for the effect of thresholds on attention maps and the effect of contrastive learning is good.

Weaknesses
- Exposition of results is not good. CheXT is compared against some weak baselines. A more appropriate CNN baseline should be something that includes similar radiomic feature extraction and bounding box generation [1]. Also, [1] is not cited in Related Work.
- The paper claims significant performance improvement by the CheXT model although no evidence (statistical test) is provided to support the claim.
-  Classification performance drops for some of the disease classes (effusion, pneumonia, pneumothorax) after incorporating the radiomics branch. It looks like radiomics branch is hurting the performance for these classes. Is there any impact of label distribution?
- In Fig. 3, two images look different. Why so?
- The training of CheXT is not necessarily semi-supervised as every image has at least disease class label.
- It is not clear how to obtain f_i and f_r; y, y' are not defined Eqn. (1).
- It would be more viable to report both classification and localization performances on the same 880 images. Is there any correlation between e.g., classification AUC and localization IoU?
- It is not clear how the model could improve generalization. There is no generalization assessment of the model for out-of-distribution cases. For example, using other datasets such as, CheXpert and MIMIC-CXR.
- In 4.1, the paper mentions the train/test/val splits, but Table 1 reports results after 3-fold cross-validation. They don't look consistent.
- In 4.4.2: "when the IoU is set to 0.1" should be threshold of IoU. "For Cardiomegaly, Infiltration, and Pneumonia." incomplete sentence.
- It is not clear how higher values of \lambda implies lower weight to contrastive loss in Eqn. (3). How does the model perform without the contrastive loss at all (\lambda=0)?
- There is no mention of the execution speed, computing resource, and failure cases (if any) for the proposed model.
- There are certainly other Transformer-based chest X-ray models such as [2].

1. https://arxiv.org/abs/2011.12506
2. https://arxiv.org/abs/2104.07235

The evidence to be evaluated is:We believe that the joint utilization of radiomics and interpretable learning techniques in our framework can further advance accurate yet interpretable learning in the medical image domain.

For example, (Oktay et al., 2018; Wang et al., 2019; Chen et al., 2021) used the attention mechanism to boost the performance of medical image segmentation.

(Han et al., 2021) applied pre-extracted radiomic features to guide pneumonia detection from chest X-ray images.

In CheXT, we extend their idea by treating image itself as one "view" and the radiomic feature extracted from the same image as another "view" (Figure 2). The two views are the jointly learned by interacting through cross-attention.

To resolve the "chicken-and-egg" dilemma in extracting reliable radiomic features without bounding boxes, we have designed a novel _Bootstrap Your Own Attention_ (**BYOA**) module, using feedbacks to learn region localization for extracting radiomic features.

The primary image branch uses a Progress-Sampling ViT (PS-ViT) (Yue et al., 2021) as its backbone. Unlike the vanilla ViT that splits images into fixed-size tokens, PS-ViT utilizes an iterative and progressive sampling strategy to locate discriminative regions and avoid over-partition object structures. We experimentally observed PS-ViT outperforms ViT and other variants in our framework because it generates higher-quality and more structure-aware attention maps, which are crucial for estimating the pathology localization during training.

In our framework, we aim to make the hidden features of the CLS similar to the radiomic features to learn the localization of pathologies in the chest X-rays. For this branch, we use the vanilla Transformer (Liu et al., 2017) as the radiomic features encoder.

Our main _roadblock_ is how to generate robust radiomic features without pathology localization. On one hand, radiomic features are dependent on and highly sensitive to the local image regions of interest, for which we have no bounding box annotation. On the other hand, image features would benefit from the guidance from radiomic features that encode important domain knowledge. The learning of image and radiomic features are fully entangled, forming a challenging "chicken-and-egg" loop. To address this issue, we design **BYOA** to constitute an end-to-end feedback loop that can bootstrap accurate pathology localization from image data, without any bounding box used for training (Figure 3). BYOA contains two components: attention map generation and radiomic feature extraction.

Here, we have two CLS tokens from two branches, but the attention maps only come from the Image branch. Then we apply a threshold on the self-attention maps to generate bounding boxes for the extraction of radiomic features.

Specifically, the threshold is designed as how much active pixels to keep in the generated attention map. The more active pixels we keep, the larger the generated bounding box is.

Given the original images and generated bounding boxes, we used the Pyradiomic tool to extract radiomic features (Van Griethuysen et al., 2017).

The difference is that we maximize agreement between two feature views extracted from different input formats, one the image and the other radiomics. Given an anchor chest X-ray in a minibatch, the positive sample will be its radiomic feature view, and the negative samples will be other chest X-rays (either image or radiomics).

Since for chest x-rays, there exists two labels, disease class labels and pathology bounding box annotations. In our case, we only have access to the disease labels, however, bounding box annotations are more important than the disease class labels for chest x-rays. Here, when we say "semi-supervised", we refer to the bounding box annotations, not disease class labels.

Higher values of \(T\) imply smaller bounding boxes to extract radiomic features. During our experiments, we found that CheXT performs worse when very large bounding boxes are generated. This observation meets our expectations since radiomic features are very subtle to bounding boxes. Specifically, the extracted radiomic features are more robust if more focal and accurate bounding boxes are given.

In a calculated ratio, they help CheXT to learn both disease-level and patient-level discriminative visual features.

We have used this dataset to evaluate the performance of CheXT for abnormality localization.

For localization, we evaluated our detected regular rectangular regions against the annotated bounding boxes, using intersection over union ratio (IoU).

More importantly, we also add the ViT as our additional baseline here, the results show that the radiomics branch and BYOA module can help the model learn more important regions, which also improves the model's interpretability.

Experimental results demonstrate that our method outperforms the state-of-the-art algorithms, especially for the disease localization task, where our method can generate more accurate bounding boxes.

Input: Figure 2: Overview of our proposed CheXT. It contains two branches, the Image branch and the Radiomics branch, to process the image and radiomic features (generated by the BYOA module shown in Figure 3), respectively. The output tokens are then fused by an efficient module via cross attention of the CLS tokens. Finally, the output of two CLS tokens (Icls andRcls) are used for disease classification. We minimize the classification errors via a focal loss. Since we train with both labeled and unlabeled images, we leverage a contrastive learning strategy. Specifically, we generate the image view zi = gi(Icls) by a projection head gi. Similarily, we generate the radiomic view zr = gr(Rcls) by another projection head gr. We maximize the agreement between zi and zr via a contrastive loss (NT-Xent). Of note, the contrastive loss is only active during the training.

Input: Figure 3: Overview of our BYOA module. For the input chest X-rays, we look at the self-attention of the CLS token of the Image branch on the heads of the final output of the cross attention module. Then we apply a threshold (0.1) (which means we only keep the top 10% active pixels in the whole attention map) to generate the bounding boxes. Then with the generated bounding boxes, we use the Pyradiomic tool as the radiomic extractor to extract the radiomic features.

Input: Figure 1: General overview of our CheXT framework for Cardiopulmonary Disease Classification and Localization from chest X-rays. CheXT takes the chest X-ray image as the input and outputs a heatmap for pathology localization, based on which the bounding box could be obtained. Radiomic features are further extracted from the bounded region and are fed to predict the disease.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me assess the claim: "During training, the image branch leverages its learned attention to estimate pathology localization, which is then utilized to extract radiomic features from the radiomics branch."

Summary of evaluation:
Based on the evidence provided, including Figures 2 and 3 and accompanying text explanations, this claim is generally well-stated. The framework clearly shows how the image branch's attention maps are used to generate bounding boxes for pathology localization, which are then used to extract radiomic features. The evidence demonstrates this process through detailed diagrams and explanations of the BYOA module's operation. However, there is one minor imprecision - the claim suggests the radiomics branch does the extraction, when actually the BYOA module handles the radiomic feature extraction.

<score>2</score>

Justification:
1. The evidence strongly supports most aspects of the claim:
- Figures 2 and 3 clearly illustrate the feedback loop where attention maps from the image branch guide localization
- The text explicitly states "we apply a threshold on the self-attention maps to generate bounding boxes for the extraction of radiomic features"
- The BYOA module's role is well-documented in handling the radiomic feature extraction

2. Minor overstatement issues:
- The claim implies the radiomics branch extracts features, but the evidence shows the BYOA module performs this function
- The review comments note some concerns about the model's generalization and statistical validation, though these don't directly impact this specific claim's validity

The claim is largely accurate but contains a small technical imprecision in attributing feature extraction to the wrong component, warranting a score of 2 (slightly overstated) rather than 1 (perfectly stated).


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

During training, the image branch leverages its learned attention to estimate pathology localization, which is then utilized to extract radiomic features from images in the radiomics branch.

The review comment to be evaluated is:

1) Radiomics is a term that most people in the ICLR community are not familiar with. What is it exactily? Is it a set of radiological/image features or bio-markers in medical images? Or something else? The authors should consider explaining it in plain language or ML language.

2) Prior multi-modal feature fusion work in the chest x-ray domain should be discussed, examples below.

    - Moradi, Mehdi, Ali Madani, Yaniv Gur, Yufan Guo, and Tanveer Syeda-Mahmood. "Bimodal network architectures for automatic generation of image annotation from text." arXiv preprint arXiv:1809.01610 (2018).

    - Wang, Xiaosong, Yifan Peng, Le Lu, Zhiyong Lu, and Ronald M. Summers. "TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-rays." arXiv e-prints (2018): arXiv-1801.

    - Xue, Yuan, and Xiaolei Huang. "Improved disease classification in chest x-rays with transferred features from report generation." In International Conference on Information Processing in Medical Imaging, pp. 125-138. Springer, Cham, 2019.

    - Liao, Ruizhi, Daniel Moyer, Miriam Cha, Keegan Quigley, Seth Berkowitz, Steven Horng, Polina Golland, and William M. Wells. "Multimodal Representation Learning via Maximization of Local Mutual Information." arXiv preprint arXiv:2103.04537 (2021).

3) I don't see "prior knowledge" and how it's utilized in this work. Isn't the so called prior knowledge in this paper essentially extracted from the data (chest x-ray images)?

4) What are the patch sizes? How are they determined?

5) There are 14 pathology labels in the NIH chest x-ray dataset. Why/how did the authors select the 8 classes out of the 14?

6) In Table 1, make the best results bold instead of in red.

7) Are the AUC values generated on the test set? If so, I'm concerned about overfitting by over tuning the hyper-parameters.

The evidence to be evaluated is:We believe that the joint utilization of radiomics and interpretable learning techniques in our framework can further advance accurate yet interpretable learning in the medical image domain.

For example, (Oktay et al., 2018; Wang et al., 2019; Chen et al., 2021) used the attention mechanism to boost the performance of medical image segmentation.

(Han et al., 2021) applied pre-extracted radiomic features to guide pneumonia detection from chest X-ray images.

In CheXT, we extend their idea by treating image itself as one "view" and the radiomic feature extracted from the same image as another "view" (Figure 2). The two views are the jointly learned by interacting through cross-attention.

To resolve the "chicken-and-egg" dilemma in extracting reliable radiomic features without bounding boxes, we have designed a novel _Bootstrap Your Own Attention_ (**BYOA**) module, using feedbacks to learn region localization for extracting radiomic features.

The primary image branch uses a Progress-Sampling ViT (PS-ViT) (Yue et al., 2021) as its backbone. Unlike the vanilla ViT that splits images into fixed-size tokens, PS-ViT utilizes an iterative and progressive sampling strategy to locate discriminative regions and avoid over-partition object structures. We experimentally observed PS-ViT outperforms ViT and other variants in our framework because it generates higher-quality and more structure-aware attention maps, which are crucial for estimating the pathology localization during training.

In our framework, we aim to make the hidden features of the CLS similar to the radiomic features to learn the localization of pathologies in the chest X-rays. For this branch, we use the vanilla Transformer (Liu et al., 2017) as the radiomic features encoder.

Our main _roadblock_ is how to generate robust radiomic features without pathology localization. On one hand, radiomic features are dependent on and highly sensitive to the local image regions of interest, for which we have no bounding box annotation. On the other hand, image features would benefit from the guidance from radiomic features that encode important domain knowledge. The learning of image and radiomic features are fully entangled, forming a challenging "chicken-and-egg" loop. To address this issue, we design **BYOA** to constitute an end-to-end feedback loop that can bootstrap accurate pathology localization from image data, without any bounding box used for training (Figure 3). BYOA contains two components: attention map generation and radiomic feature extraction.

Here, we have two CLS tokens from two branches, but the attention maps only come from the Image branch. Then we apply a threshold on the self-attention maps to generate bounding boxes for the extraction of radiomic features.

Specifically, the threshold is designed as how much active pixels to keep in the generated attention map. The more active pixels we keep, the larger the generated bounding box is.

Given the original images and generated bounding boxes, we used the Pyradiomic tool to extract radiomic features (Van Griethuysen et al., 2017).

The difference is that we maximize agreement between two feature views extracted from different input formats, one the image and the other radiomics. Given an anchor chest X-ray in a minibatch, the positive sample will be its radiomic feature view, and the negative samples will be other chest X-rays (either image or radiomics).

Since for chest x-rays, there exists two labels, disease class labels and pathology bounding box annotations. In our case, we only have access to the disease labels, however, bounding box annotations are more important than the disease class labels for chest x-rays. Here, when we say "semi-supervised", we refer to the bounding box annotations, not disease class labels.

Higher values of \(T\) imply smaller bounding boxes to extract radiomic features. During our experiments, we found that CheXT performs worse when very large bounding boxes are generated. This observation meets our expectations since radiomic features are very subtle to bounding boxes. Specifically, the extracted radiomic features are more robust if more focal and accurate bounding boxes are given.

In a calculated ratio, they help CheXT to learn both disease-level and patient-level discriminative visual features.

We have used this dataset to evaluate the performance of CheXT for abnormality localization.

For localization, we evaluated our detected regular rectangular regions against the annotated bounding boxes, using intersection over union ratio (IoU).

More importantly, we also add the ViT as our additional baseline here, the results show that the radiomics branch and BYOA module can help the model learn more important regions, which also improves the model's interpretability.

Experimental results demonstrate that our method outperforms the state-of-the-art algorithms, especially for the disease localization task, where our method can generate more accurate bounding boxes.

Input: Figure 2: Overview of our proposed CheXT. It contains two branches, the Image branch and the Radiomics branch, to process the image and radiomic features (generated by the BYOA module shown in Figure 3), respectively. The output tokens are then fused by an efficient module via cross attention of the CLS tokens. Finally, the output of two CLS tokens (Icls andRcls) are used for disease classification. We minimize the classification errors via a focal loss. Since we train with both labeled and unlabeled images, we leverage a contrastive learning strategy. Specifically, we generate the image view zi = gi(Icls) by a projection head gi. Similarily, we generate the radiomic view zr = gr(Rcls) by another projection head gr. We maximize the agreement between zi and zr via a contrastive loss (NT-Xent). Of note, the contrastive loss is only active during the training.

Input: Figure 3: Overview of our BYOA module. For the input chest X-rays, we look at the self-attention of the CLS token of the Image branch on the heads of the final output of the cross attention module. Then we apply a threshold (0.1) (which means we only keep the top 10% active pixels in the whole attention map) to generate the bounding boxes. Then with the generated bounding boxes, we use the Pyradiomic tool as the radiomic extractor to extract the radiomic features.

Input: Figure 1: General overview of our CheXT framework for Cardiopulmonary Disease Classification and Localization from chest X-rays. CheXT takes the chest X-ray image as the input and outputs a heatmap for pathology localization, based on which the bounding box could be obtained. Radiomic features are further extracted from the bounded region and are fed to predict the disease.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find this claim to be moderately overstated. While the evidence shows that the image branch does use attention mechanisms to generate localization information (via attention maps and thresholding), which is then used for radiomic feature extraction, the term "leverages its learned attention" implies a more sophisticated process than what is actually demonstrated. The evidence shows a relatively straightforward thresholding approach (keeping top 10% active pixels) rather than truly "learned" attention. Additionally, the feedback loop described in BYOA suggests this is an iterative process rather than a direct leveraging of learned attention.

The included review comments don't directly challenge this specific claim, though comment #4 about patch sizes indicates some implementation details are missing.

<score>3</score>

Justification:
1. The evidence shows the process is more mechanical than "learned":
- "we apply a threshold (0.1) (which means we only keep the top 10% active pixels in the whole attention map) to generate the bounding boxes"
- This is a fixed threshold rather than learned attention

2. The process is presented as part of a feedback loop rather than direct leveraging:
- "we design BYOA to constitute an end-to-end feedback loop that can bootstrap accurate pathology localization"
- "The learning of image and radiomic features are fully entangled"

3. While the core mechanism described in the claim exists, the wording overstates its sophistication and directness.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

During training, the image branch leverages its learned attention to estimate pathology localization, which is then utilized to extract radiomic features from images in the radiomics branch.

The review comment to be evaluated is:

Strength:
- The proposed approach and considering radiomic features along with learnt features is valuable and very relevant to the community.

Main Concerns and comments to improve the paper:
- The paper does not consider evaluation of the method on more publicly available datasets and the study is only limited to NIH-14 Chest X-ray data.
- Baselines and state of the art results on the test data: One important baseline is to make the prediction using only radiomic features.  Also,  [Han’20]  has reported much higher performance on NIH-14 and for the same task which they have not been included in the paper.
Improvement over the CNN baseline on classification: We see 0.011 improvement in mean AUC over Rajpurkar’17 as one of the baseline which is not significant.
- Details of the comparison setups are missing. I am wondering if these numbers are collected from the reported values by authors or they ran every SOTA against their data division (less likely since there is no error bound reported for SOTAs). And if they are-implemented or re-evaluated each method what are the details and consideration for a fair comparison and reproducing the results.
- Main contribution of the paper is to activate usage or radiomic features within the blackbox  structure of deep networks, however, there is no discussion or insight around the meaning and importance of multiple radiomic features that have been used. There are also works that include prior knowledge for Chest X-ray classification including [Han’20]. These features either unlock a better performance (which seems not being really successful in) or demystify the learnt feature space in a way that is pleasant for human interpreters. The paper does not provide such results.

Suggestion to improve the work:
- It would be useful to add some more explanation around radiomics term, feature extraction procedure and meaning of it, since this is not a common term for the machine learning community.
- Consider boosting the results and observation using a few more test data such as MIMIC and CheXpert that have the similar label space as NIH-14 and publicly being used in most of the related papers.
- The above mentioned datasets also can be used for evaluation of the out-of-domain generalization of the method which has not been discussed in the paper.
- Experimental setup and evaluation toward the SOTA can improve significantly. For the comparison purpose I would consider producing a statistical error range with either k-fold cross validation or boot-strapping for both CheXT and SOTAs. I would personally see a proper statistical comparison between a single best SOTA with the proposed method more valuable than having multiple models without any statistical validation.
- Adding interpretation around radiomic feature meaning, how the proposed method unlocked one or more of the 3 listed limitations of CNN methods that has been discussed in the Introduction.
- It is worth also discussing failure cases and the reasoning behind lower performance for some of the conditions such as effusion and pneumonia.

[Han'20] Han, Yan, Chongyan Chen, Liyan Tang, Mingquan Lin, Ajay Jaiswal, Song Wang, Ahmed Tewfik, George Shih, Ying Ding, and Yifan Peng. "Using Radiomics as Prior Knowledge for Thorax Disease Classification and Localization in Chest X-rays." arXiv preprint arXiv:2011.12506 (2020).

The evidence to be evaluated is:We believe that the joint utilization of radiomics and interpretable learning techniques in our framework can further advance accurate yet interpretable learning in the medical image domain.

For example, (Oktay et al., 2018; Wang et al., 2019; Chen et al., 2021) used the attention mechanism to boost the performance of medical image segmentation.

(Han et al., 2021) applied pre-extracted radiomic features to guide pneumonia detection from chest X-ray images.

In CheXT, we extend their idea by treating image itself as one "view" and the radiomic feature extracted from the same image as another "view" (Figure 2). The two views are the jointly learned by interacting through cross-attention.

To resolve the "chicken-and-egg" dilemma in extracting reliable radiomic features without bounding boxes, we have designed a novel _Bootstrap Your Own Attention_ (**BYOA**) module, using feedbacks to learn region localization for extracting radiomic features.

The primary image branch uses a Progress-Sampling ViT (PS-ViT) (Yue et al., 2021) as its backbone. Unlike the vanilla ViT that splits images into fixed-size tokens, PS-ViT utilizes an iterative and progressive sampling strategy to locate discriminative regions and avoid over-partition object structures. We experimentally observed PS-ViT outperforms ViT and other variants in our framework because it generates higher-quality and more structure-aware attention maps, which are crucial for estimating the pathology localization during training.

In our framework, we aim to make the hidden features of the CLS similar to the radiomic features to learn the localization of pathologies in the chest X-rays. For this branch, we use the vanilla Transformer (Liu et al., 2017) as the radiomic features encoder.

Our main _roadblock_ is how to generate robust radiomic features without pathology localization. On one hand, radiomic features are dependent on and highly sensitive to the local image regions of interest, for which we have no bounding box annotation. On the other hand, image features would benefit from the guidance from radiomic features that encode important domain knowledge. The learning of image and radiomic features are fully entangled, forming a challenging "chicken-and-egg" loop. To address this issue, we design **BYOA** to constitute an end-to-end feedback loop that can bootstrap accurate pathology localization from image data, without any bounding box used for training (Figure 3). BYOA contains two components: attention map generation and radiomic feature extraction.

Here, we have two CLS tokens from two branches, but the attention maps only come from the Image branch. Then we apply a threshold on the self-attention maps to generate bounding boxes for the extraction of radiomic features.

Specifically, the threshold is designed as how much active pixels to keep in the generated attention map. The more active pixels we keep, the larger the generated bounding box is.

Given the original images and generated bounding boxes, we used the Pyradiomic tool to extract radiomic features (Van Griethuysen et al., 2017).

The difference is that we maximize agreement between two feature views extracted from different input formats, one the image and the other radiomics. Given an anchor chest X-ray in a minibatch, the positive sample will be its radiomic feature view, and the negative samples will be other chest X-rays (either image or radiomics).

Since for chest x-rays, there exists two labels, disease class labels and pathology bounding box annotations. In our case, we only have access to the disease labels, however, bounding box annotations are more important than the disease class labels for chest x-rays. Here, when we say "semi-supervised", we refer to the bounding box annotations, not disease class labels.

Higher values of \(T\) imply smaller bounding boxes to extract radiomic features. During our experiments, we found that CheXT performs worse when very large bounding boxes are generated. This observation meets our expectations since radiomic features are very subtle to bounding boxes. Specifically, the extracted radiomic features are more robust if more focal and accurate bounding boxes are given.

In a calculated ratio, they help CheXT to learn both disease-level and patient-level discriminative visual features.

We have used this dataset to evaluate the performance of CheXT for abnormality localization.

For localization, we evaluated our detected regular rectangular regions against the annotated bounding boxes, using intersection over union ratio (IoU).

More importantly, we also add the ViT as our additional baseline here, the results show that the radiomics branch and BYOA module can help the model learn more important regions, which also improves the model's interpretability.

Experimental results demonstrate that our method outperforms the state-of-the-art algorithms, especially for the disease localization task, where our method can generate more accurate bounding boxes.

Input: Figure 2: Overview of our proposed CheXT. It contains two branches, the Image branch and the Radiomics branch, to process the image and radiomic features (generated by the BYOA module shown in Figure 3), respectively. The output tokens are then fused by an efficient module via cross attention of the CLS tokens. Finally, the output of two CLS tokens (Icls andRcls) are used for disease classification. We minimize the classification errors via a focal loss. Since we train with both labeled and unlabeled images, we leverage a contrastive learning strategy. Specifically, we generate the image view zi = gi(Icls) by a projection head gi. Similarily, we generate the radiomic view zr = gr(Rcls) by another projection head gr. We maximize the agreement between zi and zr via a contrastive loss (NT-Xent). Of note, the contrastive loss is only active during the training.

Input: Figure 3: Overview of our BYOA module. For the input chest X-rays, we look at the self-attention of the CLS token of the Image branch on the heads of the final output of the cross attention module. Then we apply a threshold (0.1) (which means we only keep the top 10% active pixels in the whole attention map) to generate the bounding boxes. Then with the generated bounding boxes, we use the Pyradiomic tool as the radiomic extractor to extract the radiomic features.

Input: Figure 1: General overview of our CheXT framework for Cardiopulmonary Disease Classification and Localization from chest X-rays. CheXT takes the chest X-ray image as the input and outputs a heatmap for pathology localization, based on which the bounding box could be obtained. Radiomic features are further extracted from the bounded region and are fed to predict the disease.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on the evidence provided, I'll evaluate the claim: "During training, the image branch leverages its learned attention to estimate pathology localization, which is then utilized to extract radiomic features from images in the radiomics branch."

The evidence strongly supports this claim through:
- Clear demonstration in Figures 2 and 3 showing the feedback loop where attention maps from the image branch generate bounding boxes for radiomic feature extraction
- Detailed explanation of the BYOA module that uses attention maps with a threshold to generate bounding boxes
- Explicit description of using Pyradiomics tool to extract features from the bounded regions

The claim appears to be well-stated (Score 1) as it accurately describes the technical process without exaggeration and is fully supported by both textual and visual evidence. The reviewer comments don't contradict this specific technical claim about the model's architecture and process flow.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

In this work, we propose a novel framework called Lifelong User Representation Model (LURM) to model user behaviors since registration.

The evidence to be evaluated is:Previous works have shown that considering long-term historical behavior sequences for user modeling can significantly improve the performance of different tasks

Ren _et al._ proposed a hierarchical periodic memory network for lifelong sequential modeling. They built a personalized memorization for each user, which remembers both intrinsic user tastes and multi-facet user interests with the learned while compressed memory. Pi _et al._ decoupled the user modeling from the whole CTR prediction system to tackle the challenge of the storage cost and the system latency. Specifically, they proposed a user interest center module for real-time inference and a memory-based network that can be implemented incrementally. Pi _et al._ also designed a search-based interest model (SIM) with a cascaded two-stage search paradigm to capture the diverse long-term interest with target item.

In this work, we are committed to learning universal user representation through truly lifelong user behavior sequences with arbitrary length. For this purpose, we propose a framework named Lifelong User Representation Model (LURM) which consists of two cascaded sub-models: Bag of Interests (BoI) and Self-supervised Multi-anchor Encoder Network (SMEN).

In order to model extremely long lifelong user behavior sequence, we propose to aggregate the content of items under user purchases, clicks, or other behaviors at a certain granularity.

Therefore, we propose to get a BoI feature sequence by applying the BoI model at each time period of the whole lifelong time. Then, a Self-supervised Multi-anchor Encoder Network (SMEN) is designed to learn compressed user representations from the sequence of BoI features. The lifelong time \(\mathbf{T}\) is divided into \(N\) parts at a fixed time interval, i.e. \(\mathbf{T}=\{\mathbf{t_{1}},\mathbf{t_{2}},...,\mathbf{t_{N}}\}\) and \(\mathbf{t_{i}}\) denotes the \(i\)-th time period.

In this way, a sequence of BoI features \(\mathbf{B}=\{\mathbf{b_{t_{1}}},\mathbf{b_{t_{2}}},...,\mathbf{b_{t_{N}}}\}\) can be obtained, where \(\mathbf{b_{t_{i}}}\) denotes the BoI feature corresponding to the \(i\)-th time period \(\mathbf{t_{i}}\). After that, SMEN is used to map \(\mathbf{B}\) to low-dimensional user representations. As shown in Fig. 1 (b), SMEN consists of a multi-anchor module, a time aggregation module, a multi-scale aggregation module and a contrastive learning module.

The data of user behavior, which is highly unstructured and complex, implies different preferences of the user. To capture diverse aspects of user preferences, a novel multi-anchor module is proposed. Specifically, suppose there are \(M\) anchors, and each of them indicates a certain preference of users. Let \(\mathbf{b}\) be a BoI feature, the module converts \(\mathbf{b}\) to \(M\) low-dimensional representations as shown in Fig. 2. Each representation is computed as \[\mathbf{r^{i}}=ReLU(\mathbf{\alpha_{i}^{T}}f(\mathbf{b}))=ReLU(\sum_{j}^{D}\alpha_{ij}f(b_ {j}))=ReLU(\sum_{j}^{D}\alpha_{ij}b_{j}\mathbf{W_{j}^{e}}), \tag{5}\] where \(f(b_{j})=b_{j}\mathbf{W_{j}^{e}}\) is the 'interest' embedding function, \(b_{j}\) is the \(j\)-th element of \(\mathbf{b}\), \(\mathbf{W^{e}}=\left(\mathbf{W_{1}^{e}},\mathbf{W_{2}^{e}},...,\mathbf{W_{D}^{e}}\right)^{T}\) is the embedding matrix. And \(\alpha_{ij}\) is the attention weight between the \(i\)-th anchor and the \(j\)-th 'interest', which measures the portion assigned to the \(i\)-th preference from the \(j\)-th behavior 'interest'. The weight \(\alpha_{ij}\) is defined as \[\alpha_{ij}=\frac{\exp(\mathbf{W_{i}^{a}}\mathbf{k_{j}})}{\sum_{l}\exp(\mathbf{W_{l}^{a}} \mathbf{k_{j}})}, \tag{6}\]

Specially, experiments prove that SMEN can achieve almost lossless dimensionality reduction mainly due to the multi-anchor module. Through the multi-anchor module, a sequence of representation groups \(\mathbf{\mathcal{R}}=\{\mathbf{R_{t_{1}}},\mathbf{R_{t_{2}}},...,\mathbf{R_{t_{N}}}\}\) for each user is obtained, where \(\mathbf{R_{t_{i}}}=\{\mathbf{r_{t_{i}}^{1}},\mathbf{r_{t_{i}}^{2}},...,\mathbf{r_{t_{i}}^{M}}\}\) is a representation group generated by multi-anchor module corresponding to the BoI feature \(\mathbf{b_{t_{i}}}\) in time period \(\mathbf{t_{i}}\). In the time aggregation module, each sequence \(\mathbf{R^{i}}=\{\mathbf{r_{t_{i}}^{i}},\mathbf{r_{t_{i}}^{i}},...,\mathbf{r_{t_{N}}^{i}}\}\) is aggregated separately to yield a new representation \(\mathbf{\tilde{r}^{i}}\in\mathbb{R}^{H}\). Thus the variable-size representation sequence \(\mathbf{\mathcal{R}}\) can be transformed into \(M\) fixed-size representations \(\mathbf{\tilde{R}}=\big{\{}\mathbf{\tilde{r}^{1}},\mathbf{\tilde{r}^{2}},...,\mathbf{\tilde{r }^{M}}\big{\}}\).

Compared to average and max pooling, RNNs are more appropriate to capture variations over time.

To address this trade-off, multi-scale aggregation module is designed. The module captures diverse patterns from user behavior by aggregating several representations generated at different granularities. Generally, user behavior in this work is aggregated monthly and yearly, i.e. \(\mathbf{T}=\{\mathbf{t_{1}},\mathbf{t_{2}},...,\mathbf{t_{N}}\}=\big{\{}\mathbf{t_{1}^{\prime}},\bm {t_{2}^{\prime}},...,\mathbf{t_{N^{\prime}}^{\prime}}\big{\}}\) (e.g., if the length of time period is 5 years, \(\mathbf{N}\) is 60 for monthly granularity, and \(\mathbf{N^{\prime}}\) is 5 for yearly granularity). Thus two sequences of BoI feature \(\mathbf{B}=\{\mathbf{b_{t_{1}}},\mathbf{b_{t_{2}}},...,\mathbf{b_{t_{N}}}\}\) and \(\mathbf{B^{\prime}}=\big{\{}\mathbf{b_{t_{1}^{\prime}}},\mathbf{b_{t_{2}^{\prime}}},...,\bm {b_{t_{N^{\prime}}^{\prime}}}\big{\}}\) at different scales correspondingly are obtained. Then, two groups of representations \(\mathbf{\tilde{R}}=\big{\{}\mathbf{\tilde{r}^{1}},\mathbf{\tilde{r}^{2}},...,\mathbf{\tilde{r }^{M}}\big{\}}\) and \(\mathbf{\tilde{R^{\prime}}}=\big{\{}\mathbf{\tilde{r^{\prime}}^{1}},\mathbf{\tilde{r^{ \prime}}^{2}},...,\mathbf{\tilde{r^{\prime}}^{M}}\big{\}}\) are obtained after the multi-anchor module and time aggregation module with parameter sharing.

\[\mathbf{\hat{r}^{i}}=s^{i}([\mathbf{\tilde{r}^{i}},\mathbf{\tilde{r^{\prime}}^{i}}])\odot \mathbf{\tilde{r}^{i}}+(1-s^{i}([\mathbf{\tilde{r}^{i}},\mathbf{\tilde{r^{\prime}}^{i}}]) )\odot\mathbf{\tilde{r^{\prime}}^{i}}, \tag{7}\] where \(\odot\) is the Hadamard product, \([\cdot]\) is a operation concatenating vectors along the last dimension, and \(s^{i}(\cdot)\) is the switch function which is implemented as a fully-connected layer followed by a sigmoid function. The switch function controls the fusion between BoI feature sequences of different scales. The output of this module \(\mathbf{\hat{R}}=[\mathbf{\hat{r}^{1}},\mathbf{\hat{r}^{2}},...,\mathbf{\hat{r}^{M}}]\) is served as the final user representation. As mentioned in 3.1.1, contrastive loss is also used to learn user representation in this network. In order to obtain a unified vector for contrastive learning, a nonlinear projection head is added, i.e. \(h(\cdot)\), as used in SimCLR Chen et al. (2020). The function \(h(\cdot)\) is computed as \[\mathbf{v} =ReLU([\mathbf{W_{1}^{h_{1}}}\mathbf{\hat{r}^{1}},\mathbf{W_{2}^{h_{1}}}\mathbf{ \hat{r}^{2}},...,\mathbf{W_{M}^{h_{1}}}\mathbf{\hat{r}^{M}}]) \tag{8}\] \[h(\mathbf{v}) =\mathbf{W^{h_{2}}}ReLU(\mathbf{v}+mix(\mathbf{v},\mathbf{W^{h_{2}}}).\] The function \(mix(\mathbf{v},\mathbf{W^{h_{3}}})\) is to allow interaction between different preferences, and is implemented as \[mix(\mathbf{v},\mathbf{W^{h_{3}}})=r_{2}(\mathbf{W^{h_{3}}}r_{1}(\mathbf{v})), \tag{9}\]

We set the latent dimensions of word/item embedding and all hidden layers to 128 in LURM. In item embedding module, the window size \(\beta\) was set to 5 days, and the temperature \(\tau\) was 0.1. In order to retain details as many as possible, the size of the 'interest' vocabulary was set to \(10^{5}\), and the number of anchors \(M\) was set to 10 in our experiments. So the final user embedding dimension was 1280. In addition, the input of month-granularity and year-granularity were used.

The first one uses the high-dimensional output of BoI module as input in the downstream tasks, while the second uses the output of LURM. It can be seen that LURM consistently outperforms other unsupervised methods, e.g., about 3.23%/1.49% average improvements than SUMN and 4.41%/2.09% average improvements than PUTM in terms of AUC and ACC respectively. Even compared with supervised methods, LURM outperforms TextCNN and HAN about 8.26%/3.47% and 3.82%/2.04% respectively. In particular, it can be observed that LURM has a significant improvement over BoI.

When using the same input, LURM(short) achieved better results than other methods on both user profiling prediction tasks and category preference identification tasks. When using longer input, we can obtain more improvements, e.g., about 15.55%/17.55% average improvement on user profiling prediction tasks, and 4.19%/2.91% average improvement on category preference identification tasks compared with SUMN. From Table 2 and Table 3, we have observed that LURM has achieved better performance compared to other self-supervised and supervised methods. Its success can be attributed to lifelong user behavior modeling and lossless dimensionality reduction of SMEN. Fig. 3 shows the performance of LURM on several downstream tasks of industrial dataset when using inputs of different lengths. It can be seen that the richer the behavior, the better the performance on user profiling prediction tasks. Compared to using only behaviors of two months, an average improvement of 9.75%/16.03% can be achieved when using behaviors of 5 years on these tasks. While the benefits of using richer behavioral data become small for category preference identification tasks. This is because these tasks are more focused on the short-term interests of users, and it can be observed that the best performance can be obtained when input with a length of one year is used on these tasks.

It can be seen that using multi-scale aggregation module can achieve significant improvements on user profiling prediction tasks, while the benefits disappear on category preference tasks.

In this work, a novel framework named LURM is proposed to model lifelong user behaviors with any length. With the ability to model lifelong user behaviors, our method shows promising results on different downstream tasks and datasets. \begin{table}
\begin{tabular}{c c c c c} \hline Method & Books of Literature & Games of Sports & Outdoor-hunting & Musical Instrument \\ \hline TextCNN & 73.847/2.83 & 61.00/69.92 & 66.06/72.76 & 68.28/71.81 \\ HAN & 79.12/77.49 & 66.62/70.01 & 71.19/73.42 & 70.01/72.12 \\ \hline TF-IDF & 78.62/77.21 & 65.69/68.69 & 71.90/72.19 & 69.16/68.62 \\ Doc2Vec & 71.21/68.26 & 66.75/66.85 & 71.58/66.36 & 70.29/67.37 \\ \hline PTUM & 78.66/77.91 & 66.06/68.97 & 70.34/73.01 & 69.50/72.92 \\ SUMN & 79.57/77.73 & 66.83/70.63 & 72.27/73.44 & 70.61/73.43 \\ \hline BoI & 76.29/75.75 & 59.34/67.58 & 67.80/71.28 & 66.36/69.88 \\ LURM & **82.55/79.13** & **68.02/72.43** & **77.59/75.19** & **74.04/74.43** \\ \hline \end{tabular}
\end{table}
Table 2: Comparison in terms of AUC(%)/ACC(%) on the Amazon dataset. Several category preference identification tasks are evaluated. The best score is bold.

 \begin{table}
\begin{tabular}{c c c c c c} \hline \hline Method & Age & Baby Age & Clothing & Shoe & Coffee \\ \hline TextCNN & 86.40/61.02 & 72.19/66.01 & 76.64/76.13 & 84.33/80.24 & 80.49/79.46 \\ TF-IDF & 87.73/61.75 & 73.08/67.26 & 77.61/78.01 & 85.04/81.15 & 81.75/80.78 \\ SUMN & 85.35/60.61 & 74.20/67.63 & 73.86/74.16 & 83.47/79.63 & 79.89/79.42 \\ \hline BoI(short) & 89.57/63.99 & 82.23/70.22 & 78.93/78.85 & 85.53/81.68 & 82.61/81.41 \\ LURM(short) & 88.48/61.99 & 82.71/69.29 & 78.37/78.57 & 85.15/81.18 & 81.94/80.78 \\ BoI & **96.48**/**80.19** & 93.29/83.57 & **80.83**/**79.33** & **86.69**/**82.16** & **83.78**/**81.74** \\ LURM & 96.09/78.43 & **94.59**/**84.91** & 80.37/79.10 & 86.30/81.65 & 83.11/81.18 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Comparison in terms of AUC/ACC on Industrial dataset. Two profiling prediction tasks and three category preference identification tasks are evaluated. The best score is bold.

Input: Table 2: Comparison in terms of AUC(%)/ACC(%) on the Amazon dataset. Several category preference identification tasks are evaluated. The best score is bold.

Input: Table 3: Comparison in terms of AUC/ACC on Industrial dataset. Two profiling prediction tasks and three category preference identification tasks are evaluated. The best score is bold.

Input: Figure 1: Illustration of our Lifelong User Representation Model (LURM) for user understanding. LURM consists of two sub-model: (a) Bag of Interests (BoI) is used to aggregate user behavior data at ‘interest’ granularity, and is composed of an item embedding module and a large scale clustering module. We apply the BoI model on lifelong behavior data to generate multi-scale BoI feature sequences, (b) Self-supervised Multi-anchor Encoder Network (SMEN) is used to learn compressed user representations from BoI features sequences, and is composed of a multi-anchor module, a time aggregation module, a multi-scale aggregation module and a contrastive learning module.

Input: Figure 3: Comparison of LURM with inputs of different lengths on several downstream tasks. The two figures show the results in terms of AUC and ACC respectively.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim "we propose a novel framework called Lifelong User Representation Model (LURM) to model user behaviors since registration" appears to be well-stated. The evidence thoroughly documents the LURM framework's components (BoI and SMEN), methodology, and implementation details. Experimental results demonstrate LURM's effectiveness compared to baseline methods, showing improvements in both user profiling and category preference tasks. The framework's ability to handle variable-length behavioral data from registration is clearly explained and supported by empirical evidence. The claim accurately represents what is presented without exaggeration.

<score>1</score>

Justification:
1. The evidence provides comprehensive technical details of the LURM framework components and architecture (as shown in Figure 1)
2. Experimental results in Tables 2 and 3 quantitatively demonstrate LURM's performance advantages
3. Figure 3 validates the framework's capability to handle different input lengths of behavioral data
4. The methodology for processing lifelong user behavior sequences is clearly explained with mathematical formulations
5. Claims are measured and supported by comparative analysis against baseline methods
6. Limitations and task-specific performance variations are acknowledged (e.g., diminishing returns for category preference tasks with longer historical data)