– Under Review 2022 Full Paper – MIDL 2022 submission Yu-Jen ChenContributed
equally1<EMAIL_ADDRESS>
Wei-Hsiang Shen2<EMAIL_ADDRESS>
Hao-Wei Chung1<EMAIL_ADDRESS>
Jing-Hao Chiu1<EMAIL_ADDRESS>
Da-Cheng Juan1<EMAIL_ADDRESS>
Tsung-Ying Ho3<EMAIL_ADDRESS>
Chi-Tung Cheng4<EMAIL_ADDRESS>
Meng-Lin Li2<EMAIL_ADDRESS>
Tsung-Yi Ho1<EMAIL_ADDRESS>
1 Department of Computer Science, National Tsing Hua Unviserity, Taiwan
2 Department of Electrical Engineering, National Tsing Hua Unviserity, Taiwan
3 Department of Nuclear Medicine, Chang Gung Memorial Hospital, Linkou, Taiwan
4 Department of Trauma and Emergency Surgery, Chang Gung Memorial Hospital,
Linkou, Taiwan

# Representative Image Feature Extraction via Contrastive Learning Pretraining
for Chest X-ray Report Generation

###### Abstract

Medical report generation is a challenging task since it is time-consuming and
requires expertise from experienced radiologists. The goal of medical report
generation is to accurately capture and describe the image findings. Previous
works pretrain their visual encoding neural networks with large datasets in
different domains, which cannot learn general visual representation in the
specific medical domain. In this work, we propose a medical report generation
framework that uses a contrastive learning approach to pretrain the visual
encoder and requires no additional meta information. In addition, we adopt
lung segmentation as an augmentation method in the contrastive learning
framework. This segmentation guides the network to focus on encoding the
visual feature within the lung region. Experimental results show that the
proposed framework improves the performance and the quality of the generated
medical reports both quantitatively and qualitatively.

###### keywords:

Radiology, Chest X-ray, Contrastive Learning, Lung Masking, Medical Report
Generation

††editors: Under Review for MIDL 2022

##  1 Introduction

Medical Images, such as X-rays, have been widely used for supporting clinical
diagnosis and diseases treatment. Radiologists are responsible for delivering
the interpretation and diagnosis based on their medical expertise and
providing the medical reports as reference. However, medical report generation
is time-consuming and requires medical expertise from experienced
radiologists, which may not be abundant in rural area. Thus, automatic medical
report generation is essential in reducing workload, facilitating clinical
workflow, and providing additional diagnostic insights.

Most of the current state-of-the-art methods are based on encoder-decoder
frameworks [Shin et al.(2016)Shin, Roberts, Lu, Demner-Fushman, Yao, and
Summers]. Many of them [Jing et al.(2017)Jing, Xie, and Xing, Li et
al.(2018)Li, Liang, Hu, and Xing, Chen et al.(2020c)Chen, Song, Chang, and
Wan] utilize pretrained encoders to extract image features and design decoders
to generate the corresponding reports. Although these works design new
architectures which result in significant results, their encoders remain to be
pretrained with datasets of different domains, where some medical
characteristics cannot be fully learned and captured, and lead to limited
performance improvement.

In this work, we proposed a medical report generating framework by leveraging
the power of Contrastive Learning (CL) and lung segmentation without using
extra datasets and additional metadata such as keywords in our model. The
encoder is first pretrained with CL using images only and then the entire
network is trained end-to-end under a supervised setting (images and reports).
Experiments showed that the representation trained by the CL framework results
in better performance than other commonly used pretraining settings. In
addition, in the CL model, we propose a new augmentation method, lung
segmentation, which helps the model attend to the lung region of the given
images. The lung segmentation augmentation guides the model to focus on lung
regions and generate more detailed and accurate descriptions around the lung
regions. In summary, we demonstrate that CL with lung segmentation improves
the performance and the quality of the generated medical reports. Moreover,
applying the CL pretrained encoder to state-of-the-art methods also boosts
their performance.

##  2 Related Work

###  2.1 Chest X-ray Report Generation

Most existing report generating approaches [Yuan et al.(2019)Yuan, Liao, Luo,
and Luo, Syeda-Mahmood et al.(2020)Syeda-Mahmood, Wong, Gur, Wu, Jadhav,
Kashyap, Karargyris, Pillai, Sharma, Syed, et al., Xue et al.(2018)Xue, Xu,
Rodney Long, Xue, Antani, Thoma, and Huang, Li et al.(2020)Li, Cao, and Zhu]
follow the traditional encoder-decoder architecture to design their model.
[Jing et al.(2017)Jing, Xie, and Xing] proposed a powerful model combining the
co-attention mechanism and the multi-level LSTM to generate a longer paragraph
of reports. The co-attention mechanism will combine the visual feature and the
paragraph information to achieve better performance. [Li et al.(2018)Li,
Liang, Hu, and Xing] harnessed the reinforcement-learning strategy to update
the HRGR-Agent via sentence-level and word-level rewards and maintained a high
accuracy in report diagnosis. Finally, [Chen et al.(2020c)Chen, Song, Chang,
and Wan] proposed a memory-based Transformer model to explore the correlation
between similar images in the dataset, and such model can report necessary
medical terms as well as a meaningful attention map. However, these works
focus on designing novel decoders and pretrain their encoders with datasets of
domains not related to the application under concern. The medical
characteristic cannot be fully learned and captured with such encoder
settings. In comparison, our results show that our training framework provides
better-encoded features that are applicable to state-of-the-art methods.

###  2.2 Contrastive Learning (CL)

Contrastive learning has shown great promise to learn image representation via
reflecting the difference between image features and boosting the performance
of various image-related tasks. In fundamental CL frameworks, such as SimCLR
[Chen et al.(2020b)Chen, Kornblith, Norouzi, and Hinton], MoCo [He et
al.(2020)He, Fan, Wu, Xie, and Girshick], and PIRL [Misra and Maaten(2020)],
the feature extractor is trained by minimizing the contrastive loss of the two
positive related images and repelling the two negative related images. In the
medical domain, several previous works [Chen et al.(2020a)Chen, Zhang, Lin,
Chen, and Lu, Liu et al.(2021)Liu, Yin, Wu, Ge, Zhang, and Sun, Liu et
al.(2019)Liu, Zhao, Fei, Zhang, Wang, and Yu] also explore the capability that
the CL framework achieve performance improvement for medical images.

##  3 Methodology

Most existing automatic chest X-ray report generation approaches follow an
encoder-decoder framework, which consists of an image encoder and a report
decoder. The image encoder $E(\cdot)=I\rightarrow V$ encodes the input image
$I$ into the visual representation $V$ in latent space. The report decoder
$D(\cdot)=V\rightarrow R$ is used to generate the report $R$ from the given
image representation $V$. ResNet is often used as the encoder $E$ and
Transformer is often used as the decoder $D$. The encoder-decoder framework is
trained in a supervised manner in which paired chest X-ray images and their
corresponding medical reports are required.

In this work, we explore the importance of effective visual representation for
chest X-ray report generation. Our image encoder network $E$ is pretrained
with contrastive learning techniques that incorporate lung segmentation as one
of the augmentation methods.

###  3.1 Contrastive Learning (CL) Pretraining

In this work, different from most existing approaches whose image encoders are
pretrained with large datasets with noisy labels, we adopt the Contrastive
Learning (CL) approaches SimCLR [Chen et al.(2020b)Chen, Kornblith, Norouzi,
and Hinton] and MoCo [He et al.(2020)He, Fan, Wu, Xie, and Girshick] to
pretrain the image encoder.

Figure 1: An illustration of the fundamental concept for CL. The left figure
shows how positive and negative image pairs are created. The figure on the
right shows the details of the network, which is a combination of an encoder
and a projection network.

CL is to minimize the image embedding distance between “positive” image pairs,
and maximize the distance between “negative” image pairs. A positive image
pair is created by applying two different augmentations to a single image, and
a negative image pair is two augmented images that come from two different
source images. In this work for X-ray images, the augmentation techniques used
are random resized crop and random horizontal flip.

The augmented images are then passed to an encoder neural network $f(\cdot)$,
and encoded to their representation vectors $r=f(x)\in R^{d}$, where $d$ is
the dimension of the representation vector, as shown in Fig. 1. Then the
representation vectors are projected to a fewer dimension space $z=g(r)\in
R^{k}$ by a project network $g(\cdot)$, which is a neural network of two
fully-connected layers.

Finally, for a minibatch of $N$ images, let $i\in I\equiv\\{1,2,...,2N\\}$ be
the index of the augmented sample in the minibatch, the contrastive loss
function is formulated as follows:

$\large L=-\sum_{i\in I}\frac{\exp(z_{i}\cdot z_{j(i)}/\tau)}{\sum_{k\in
B(i)}\exp(z_{i}\cdot z_{k}/\tau)}$ (1)

where $z_{i}=g(f(x))\in R^{k}$ is the projected vector of the image $x$, the
$\cdot$ operator is the inner product, index $j(i)$ is the positive image of
anchor $i$, $\tau=0.5$ is the temperature parameter, and $B(i)\equiv
I\setminus\\{i\\}$. The numerator of the loss calculates all the vector
similarity of positive pairs, and the denominator is the summation of
similarity of all the image pairs (except the positive pairs) inside the
minibatch.

###  3.2 Lung Segmentation Contrastive Learning

Figure 2: An example of positive pair with the lung masking augmentation
added. We can see that the images in a positive pair contain lung image and
whole image.

In addition to conventional image augmentation techniques, we explore
segmentation methods as augmentation methods to support the model to generate
medical findings related to the segmented region. We use a trained lung
segmentation model111https://github.com/rani700/xray as an augmentation
function, which guides the network by focusing on the lung region. The
segmentation model is randomly applied with the probability of 0.5 and the
binary segmentation mask crops out the lung region in the image, as shown in
Fig. 2. Since the contrastive learning setting guides the network to minimize
the distance between images with and without lung segmentation crop-off, the
network learns to focus less on the non-lung region and generates more
detailed features of the lung region. Thus, the report decoder receives lung-
specific features and generates reports associated with tissue or abnormality
of the lung or rib.

##  4 Experimental Settings

###  4.1 Dataset

In this work, we implement report generation on two datasets, Indiana
University Chest X-ray Collection (IU X-ray) dataset [Demner-Fushman et
al.(2016)Demner-Fushman, Kohli, Rosenman, Shooshan, Rodriguez, Antani, Thoma,
and McDonald] which contains 7,470 pairs of frontal and lateral chest X-rays
and reports, and the Medical Information Mart for Intensive Care - Chest X-ray
(MIMIC-CXR) dataset [Johnson et al.(2019)Johnson, Pollard, Greenbaum, Lungren,
Deng, Peng, Lu, Mark, Berkowitz, and Horng], which contains 377,110 images and
227,835 radiographic reports. We duplicated the data-preprocess of [Chen et
al.(2020c)Chen, Song, Chang, and Wan] and followed their dataset split on both
dataset, which is widely adopted by many related works [Jing et al.(2017)Jing,
Xie, and Xing, Chen et al.(2020c)Chen, Song, Chang, and Wan, Rennie et
al.(2017)Rennie, Marcheret, Mroueh, Ross, and Goel, Lu et al.(2017)Lu, Xiong,
Parikh, and Socher, Li et al.(2018)Li, Liang, Hu, and Xing]. In detail, IU
X-ray dataset is divided by 7:1:2 division of the entire dataset into
training/validation/testing sets, as for MIMIC-CXR, we use the official split.

The experiments are evaluated with quantitative NLP metrics that measure the
similarity between the predicted and target sentence sequence. The metrics
used include BLEU [Papineni et al.(2002)Papineni, Roukos, Ward, and Zhu],
METEOR [Denkowski and Lavie(2014)], and ROGUE-L [Lin(2004)]. BLEU measures the
similarity between the two sentences by calculating the number of word matches
in a weighted manner. These matches are independent of position. METEOR is
similar to BLEU but uses the harmonic mean of unigram precision and recall,
with recall weighted higher than precision. ROGUE-L takes into account the
structural similarity at the sentence level and identifies the longest co-
occurring n-grams in the sequence.

###  4.2 Implementation Details

We adopt the ResNet-50 architecture as the backbone of the encoder in all
experiments. At the pretraining phase for the CL approaches (SimCLR and MoCo),
all the images in the training set are used, and the model is trained for 500
epochs with the batch size as 64. As for the decoder training (fine-tuning
step), the images and the corresponded report in the training set are used,
and the entire framework is trained for 200 epochs with the batch size set as
64. Adam optimizer with the learning rate set as 0.001 and decay to 0.00001
followed by the cosine scheduler. We also use the weight decay of 0.000001 to
avoid the overfitting problem. The validation set and the test set are used
only for evaluation.

##  5 Results and Discussion

###  5.1 Comparison across Different Encoder Settings

In this section, we compared pretraining methods for the encoder to show the
feasibility of the proposed framework. The compared pretraining methods
include using AutoEncoder (AE) pretraining [Baldi(2012)], Imagenet dataset
(Imagenet) pretraining, Multi-Label Classification (MLC) pretraining [Bi and
Kwok(2013)], without pretraining (scratch), MoCo pretraining, SimCLR
pretraining, and SimCLR pretraining with lung segmentation included for the
encoder. The labels used in the MLC pretraining are the MeSH (Medical Subject
Headings) tag extracted from the reports (the MIMIC-CXR dataset did not
provide MeSH tags for the reports). In this section, we discuss using
Transformer, LSTM, and GRU as the decoder architecture. The pretrained weight
of the encoder is assigned and trained end-to-end with the decoder.

Dataset | Pretraining Method | B-1 | B-2 | B-3 | B-4 | M | R-L
---|---|---|---|---|---|---|---
IU X-ray | Scratch | 0.404 | 0.249 | 0.172 | 0.126 | 0.161 | 0.320
AE | 0.417 | 0.273 | 0.195 | 0.148 | 0.176 | 0.361
Imagenet | 0.420 | 0.256 | 0.175 | 0.127 | 0.167 | 0.326
MLC | 0.423 | 0.270 | 0.190 | 0.141 | 0.176 | 0.359
MoCo | 0.426 | 0.259 | 0.175 | 0.126 | 0.168 | 0.325
SimCLR | 0.456 | 0.289 | 0.202 | 0.150 | 0.176 | 0.362
SimCLR (w/ Lung Seg) | 0.448 | 0.282 | 0.195 | 0.141 | 0.176 | 0.340
MIMIC-CXR | Scratch | 0.211 | 0.132 | 0.090 | 0.064 | 0.105 | 0.250
AE | 0.316 | 0.186 | 0.119 | 0.081 | 0.119 | 0.243
Imagenet | 0.270 | 0.166 | 0.109 | 0.076 | 0.114 | 0.255
MoCo | 0.334 | 0.203 | 0.133 | 0.091 | 0.129 | 0.261
SimCLR | 0.343 | 0.208 | 0.137 | 0.095 | 0.132 | 0.260
SimCLR (w/ Lung Seg) | 0.325 | 0.198 | 0.130 | 0.090 | 0.129 | 0.265
Table 1: Quantitative comparison among different encoders when using
Transformer as decoder. B-n, M, and R-L are short for BLEU-n, METEOR, and
ROUGE-L, respectively. Figure 3: Qualitative comparison among different
encoders when using Transformer as the decoder. The descriptions marked with
red underline represent to match the ground truth. AE and GT stand for the
Auto-encoder pretraining and the ground truth, respectively.

####  5.1.1 Transformer Quantitative Comparison

As shown in Table 1, in the IU X-ray dataset, the one without pretraining
leads to the worst results, which is expected since the encoder has no prior
knowledge of the image. Afterward, due to the expensive pixel-wise image
reconstruction, it is difficult for AE to learn the details of images,
especially medical images. As for the weight pretrained by Imagenet, though it
contains significant information of the nature images, there is a domain gap
from chest X-ray images. In MLC pretraining, keywords are extracted from the
report, and the model is pretrained with the supervised multi-label
classification task. As the encoder has learned little prior knowledge of the
reports, we observe that the results are comparable with CL pretraining.
Ultimately, CL (SimCLR and MoCo) pretraining outperforms all other pretraining
strategies in almost all the metrics. For example, the performance boost at
BLEU-1 score compared with other pretraining strategies are up to 9.0%, and
2.25% for SimCLR and MoCo, respectively.

As for the MIMIC-CXR dataset, as shown in Table 1, the improvement trend is
similar to that of IU X-ray dataset. CL (SimCLR and MoCo) pretraining
outperforms all other pretraining strategies in all the metrics. The
performance boost at BLEU-1 score compared with other pretraining strategies
is up to 32.0%, and 28.0% for SimCLR and MoCo, respectively.

####  5.1.2 Transformer Qualitative Comparison

Fig. 3 shows the ground truth and the predicted description by different
encoders. In case (a), the AE pretrained model generates the description which
did not match the ground truth, and the Imagenet pretrained model only
describes that the lung is clear, whereas, with CL model, it points out the
heart condition and mediastinum condition. In case (b), only CL pretrained
model points out “hyperexpanded lungs” and “negative for acute bone
abnormality”, whereas the Imagenet pretrained model and the AutoEncoder
predict some common information. This shows that the CL pretrained encoder
generates precise descriptions when using the Transformer decoder.

Dataset | Pretraining Method | B-1 | B-2 | B-3 | B-4 | M | R-L
---|---|---|---|---|---|---|---
IU X-ray | Scratch | 0.299 | 0.185 | 0.132 | 0.100 | 0.134 | 0.324
AE | 0.389 | 0.233 | 0.157 | 0.111 | 0.154 | 0.317
Imagenet | 0.397 | 0.246 | 0.170 | 0.126 | 0.160 | 0.329
MLC | 0.410 | 0.250 | 0.170 | 0.123 | 0.164 | 0.325
MoCo | 0.425 | 0.260 | 0.177 | 0.128 | 0.169 | 0.326
SimCLR | 0.426 | 0.265 | 0.181 | 0.131 | 0.168 | 0.326
| SimCLR (w/ Lung Seg) | 0.431 | 0.268 | 0.185 | 0.137 | 0.172 | 0.332
MIMC-CXR | Scratch | 0.312 | 0.193 | 0.130 | 0.092 | 0.128 | 0.272
AE | 0.282 | 0.174 | 0.117 | 0.083 | 0.119 | 0.264
Imagenet | 0.309 | 0.193 | 0.131 | 0.094 | 0.130 | 0.276
MoCo | 0.309 | 0.191 | 0.128 | 0.091 | 0.127 | 0.270
SimCLR | 0.330 | 0.206 | 0.139 | 0.099 | 0.134 | 0.278
| SimCLR (w/ Lung Seg) | 0.325 | 0.203 | 0.138 | 0.095 | 0.134 | 0.278
Table 2: Quantitative comparison among different encoders when using LSTM as
decoder. B-n, M, and R-L are short for BLEU-n, METEOR, and ROUGE-L,
respectively. Figure 4: Qualitative Comparison among Different Encoders when
using LSTM as the decoder. The descriptions marked with red underline
represent ground truth matching. AE and GT stand for the Auto-encoder
preatraining and the ground truth, respectively.

####  5.1.3 LSTM Quantitative Comparison

Table 2 shows that SimCLR pretraining improves the BLEU-1 and BLEU-4 score up
to 42% and 31%, respectively, compared to other pretraining strategies on the
IU X-ray dataset. Moreover, in the MIMIC-CXR dataset, CL pretraining
outperforms other pretraining strategies in all metrics.

####  5.1.4 LSTM Qualitative Comparison

Fig. 4 shows the ground truth and the predicted description by different
encoders when using LSTM as a decoder. In case (a), the AutoEncoder pretrained
model only matches the mediastinum description with GT, and the Imagenet
pretrained model describes the lung parenchyma and pleura condition, whereas,
with CL model, it points out all of them with the bone condition additionally.
In case (b), only the CL pretrained model points out “low lung volumes”,
whereas the Imagenet pretrained model only predicts that the lung stays
normal. This shows that the CL encoder aids the model in generating precise
descriptions.

####  5.1.5 GRU Quantitative Comparison

Table 3 shows that in the IU X-ray dataset, SimCLR and MoCo pretraining boost
the BLEU-1 score up to 7.75% and 7.74%, respectively, compared with other
pretraining strategies. As for the MIMIC-CXR dataset, SimCLR pretraining
achieves the highest score in all evaluate metrics, about 5.5% higher than
without pretraining the encoder (Scratch).

####  5.1.6 GRU Qualitative Comparison

Fig. 5 shows the ground truth and the predicted description by different
encoders when using GRU as the decoder. In case (a), the Auto-Encoder
pretrained model did not describe the bone condition, and the Imagenet
pretrained model only describes the lung condition, whereas the CL pretrained
model points out both of them. In case (b), only CL pretrained model point out
“heart is mildly enlarged” and the lung had no problem, whereas the Imagenet
pretrained model only predicts that the lung stays normal, and the AutoEncoder
predicts others false information. Qualitatively, the CL encoder aids the
model in generating precise descriptions while using the GRU decoder than
other pretraining approaches.

Dataset | Pretraining Method | B-1 | B-2 | B-3 | B-4 | M | R-L
---|---|---|---|---|---|---|---
IU X-ray | Scratch | 0.351 | 0.221 | 0.156 | 0.117 | 0.153 | 0.333
AE | 0.389 | 0.241 | 0.166 | 0.121 | 0.161 | 0.332
Imagenet | 0.417 | 0.254 | 0.173 | 0.124 | 0.165 | 0.328
MLC | 0.412 | 0.263 | 0.189 | 0.145 | 0.169 | 0.341
MoCo | 0.424 | 0.259 | 0.174 | 0.124 | 0.169 | 0.331
SimCLR | 0.423 | 0.259 | 0.175 | 0.126 | 0.166 | 0.327
SimCLR (w/ Lung Seg) | 0.415 | 0.255 | 0.174 | 0.127 | 0.165 | 0.331
MIMC-CXR | Scratch | 0.309 | 0.193 | 0.130 | 0.093 | 0.128 | 0.274
AE | 0.299 | 0.184 | 0.124 | 0.088 | 0.123 | 0.270
Imagenet | 0.313 | 0.196 | 0.132 | 0.095 | 0.129 | 0.275
MoCo | 0.309 | 0.190 | 0.127 | 0.090 | 0.126 | 0.269
SimCLR | 0.327 | 0.204 | 0.137 | 0.098 | 0.134 | 0.278
SimCLR (w/ Lung Seg) | 0.335 | 0.209 | 0.140 | 0.100 | 0.136 | 0.279
Table 3: Quantitative comparison among different encoders when using GRU as
decoder. B-n, M, and R-L are short for BLEU-n, METEOR, and ROUGE-L,
respectively. Figure 5: Qualitative comparison among different encoders when
using GRU as the decoder. The descriptions marked with red underline represent
to match the ground truth. AE and GT stand for the Auto-encoder preatraining
and the ground truth, respectively.

###  5.2 Comparison with Previous Studies

Dataset | Method | B-1 | B-2 | B-3 | B-4 | R-L
---|---|---|---|---|---|---
IU X-ray |  Att2in† [Rennie et al.(2017)Rennie, Marcheret, Mroueh, Ross, and Goel] | 0.399 | 0.249 | 0.172 | 0.126 | 0.321
AdaAtt† [Lu et al.(2017)Lu, Xiong, Parikh, and Socher] | 0.436 | 0.288 | 0.203 | 0.150 | 0.354
HRGR-Agent† [Li et al.(2018)Li, Liang, Hu, and Xing] | 0.438 | 0.298 | 0.208 | 0.151 | 0.322
CoAtt† [Jing et al.(2017)Jing, Xie, and Xing] | 0.455 | 0.288 | 0.205 | 0.154 | 0.369
Transformer [Vaswani et al.(2017)Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, and Polosukhin] | 0.420 | 0.256 | 0.175 | 0.127 | 0.326
Ours (SimCLR pretraining + Transformer) | 0.456 | 0.289 | 0.202 | 0.150 | 0.362
R2Gen (Res50) [Chen et al.(2020c)Chen, Song, Chang, and Wan] | 0.455 | 0.293 | 0.210 | 0.160 | 0.360
Ours (MoCo pretraining + R2Gen) | 0.466 | 0.303 | 0.218 | 0.165 | 0.361
MIMIC-CXR |  AdaAt† [Lu et al.(2017)Lu, Xiong, Parikh, and Socher] | 0.299 | 0.185 | 0.124 | 0.088 | 0.266
Att2in† [Rennie et al.(2017)Rennie, Marcheret, Mroueh, Ross, and Goel] | 0.325 | 0.203 | 0.136 | 0.096 | 0.276
Transformer [Vaswani et al.(2017)Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, and Polosukhin] | 0.314 | 0.192 | 0.127 | 0.090 | 0.265
Ours (SimCLR pretraining + Transformer) | 0.343 | 0.208 | 0.137 | 0.095 | 0.260
R2Gen (Res50) [Chen et al.(2020c)Chen, Song, Chang, and Wan] | 0.354 | 0.236 | 0.164 | 0.118 | 0.305
| Ours (MoCo pretraining + R2Gen) | 0.359 | 0.239 | 0.167 | 0.121 | 0.304
Table 4: Performance comparison with previous studies on the IU X-Ray dataset
and MIMIC-CXR dataset. B-n and R-L are short for BLEU-n and ROUGE-L,
respectively. † indicates the results are quoted from the published paper.
Note that we reproduce R2Gen [Chen et al.(2020c)Chen, Song, Chang, and Wan] in
both datasets with ResNet-50 architecture followed by their recommended
setting.

We compare our model with existing studies on the test set of both IU X-ray
and MIMIC-CXR datasets in Table 4. Results show that the proposed method
improves the performance with large margins without any extra information,
such as pretraining on other datasets or keyword extraction.

In addition, to show the feasibility of the CL pretrained encoder, we applied
our MoCo pretrained encoder to the state-of-the-art model (R2Gen). Table 4
shows the BLEU-1 score improved from 0.455 and 0.354 (R2Gen) to 0.466 and
0.359 (MoCo pretraining + R2Gen) when MoCo pretrained feature is used in IU
X-ray and MIMIC-CXR dataset, respectively. In conclusion, results show that CL
pretrained encoder gives representative image embeddings that can improve the
performance of state-of-the-art methods.

###  5.3 Quantitative Comparison between Encoders with and without Lung
Segmentation

In this section, we compare the effectiveness of lung segmentation
augmentation. We use F1 scores on lung-related MeSH tags to evaluate whether
lung segmentation gives more accurate descriptions regarding lung regions, as
shown in Table 5. The results show that lung segmentation would improve the
accuracy of lung-related tags. This is because the encoder focuses more on
lung regions and generates lung-specific features for the decoder. However,
since the encoder cannot utilize features in the entire image, the overall
performance of the generated descriptions would drop slightly, as shown in
Table 1.

Keyword | CL w/ Lung Segmentation | CL w/o Lung Segmentation
---|---|---
pneumothorax | 0.808 | 0.752
volume | 0.191 | 0.116
effusion | 0.789 | 0.772
calcification | 0.120 | 0.030
Table 5: F1 score of lung-related keywords appeared in experiments w/ and w/o
adding lung segmentation.

###  5.4 Qualitative Comparison between Encoders with and without Lung
Segmentation

In this section, we discuss the qualitative comparison between encoders
pretrained by using and not using lung segmentation-based augmentation. Fig. 6
shows the ground truth and the predicted descriptions by different encoders.
In case (a), the CL model only describes the clearness of the lungs, whereas
with lung segmentation, the model can also point out low lung volumes which
match the ground truth description. In case (b) and (c), the CL model falsely
predicts an expanded (inflated) lungs. In contrast, the lung segmentation
model and the ground truth describe a “low lung volume”. This shows that lung
segmentation aids the model in generating accurate descriptions, especially on
the lung area.

Figure 6: Qualitative comparison for whether lung segmentation is adopted for
CL pretraining. With lung segmentation, the model can generate accurate
descriptions about the lung, whereas the naive CL sometimes generates false
descriptions about the lung volumes. GT stands for the ground truth
description.

##  6 Conclusion

In this work, we showed that the contrastive learning method provides high-
quality image representation for chest X-ray report generation. Unlike most
existing approaches, which pretrained the encoder with the dataset of
different domains, our method successfully extracted the feature from the
target dataset, which there is no domain bias. Furthermore, with lung
segmentation included, the network generates more detailed and accurate
descriptions regarding the lung regions. Experimental results on two datasets
have shown that our framework improved the performance with a large margin
from other pretraining strategies.

## References

  * [Baldi(2012)] Pierre Baldi.  Autoencoders, unsupervised learning, and deep architectures.  In _Proceedings of ICML workshop on unsupervised and transfer learning_ , pages 37–49. JMLR Workshop and Conference Proceedings, 2012.
  * [Bi and Kwok(2013)] Wei Bi and James Kwok.  Efficient multi-label classification with many labels.  In _International conference on machine learning_ , pages 405–413. PMLR, 2013.
  * [Chen et al.(2020a)Chen, Zhang, Lin, Chen, and Lu] Bingzhi Chen, Zheng Zhang, Jianyong Lin, Yi Chen, and Guangming Lu.  Two-stream collaborative network for multi-label chest x-ray image classification with lung segmentation.  _Pattern Recognition Letters_ , 135:221–227, 2020a.
  * [Chen et al.(2020b)Chen, Kornblith, Norouzi, and Hinton] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In _International conference on machine learning_ , pages 1597–1607. PMLR, 2020b.
  * [Chen et al.(2020c)Chen, Song, Chang, and Wan] Zhihong Chen, Yan Song, Tsung-Hui Chang, and Xiang Wan.  Generating radiology reports via memory-driven transformer.  _arXiv preprint arXiv:2010.16056_ , 2020c.
  * [Demner-Fushman et al.(2016)Demner-Fushman, Kohli, Rosenman, Shooshan, Rodriguez, Antani, Thoma, and McDonald] Dina Demner-Fushman, Marc D Kohli, Marc B Rosenman, Sonya E Shooshan, Laritza Rodriguez, Sameer Antani, George R Thoma, and Clement J McDonald.  Preparing a collection of radiology examinations for distribution and retrieval.  _Journal of the American Medical Informatics Association_ , 23(2):304–310, 2016.
  * [Denkowski and Lavie(2014)] Michael Denkowski and Alon Lavie.  Meteor universal: Language specific translation evaluation for any target language.  In _Proceedings of the EACL 2014 Workshop on Statistical Machine Translation_ , 2014.
  * [He et al.(2020)He, Fan, Wu, Xie, and Girshick] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9729–9738, 2020.
  * [Jing et al.(2017)Jing, Xie, and Xing] Baoyu Jing, Pengtao Xie, and Eric Xing.  On the automatic generation of medical imaging reports.  _arXiv preprint arXiv:1711.08195_ , 2017.
  * [Johnson et al.(2019)Johnson, Pollard, Greenbaum, Lungren, Deng, Peng, Lu, Mark, Berkowitz, and Horng] Alistair EW Johnson, Tom J Pollard, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng, Yifan Peng, Zhiyong Lu, Roger G Mark, Seth J Berkowitz, and Steven Horng.  Mimic-cxr-jpg, a large publicly available database of labeled chest radiographs.  _arXiv preprint arXiv:1901.07042_ , 2019.
  * [Li et al.(2018)Li, Liang, Hu, and Xing] Christy Y Li, Xiaodan Liang, Zhiting Hu, and Eric P Xing.  Hybrid retrieval-generation reinforced agent for medical image report generation.  _arXiv preprint arXiv:1805.08298_ , 2018.
  * [Li et al.(2020)Li, Cao, and Zhu] Xin Li, Rui Cao, and Dongxiao Zhu.  Vispi: Automatic visual perception and interpretation of chest x-rays.  In _Medical Imaging with Deep Learning_ , 2020.  URL https://openreview.net/forum?id=43NPhWwbek.
  * [Lin(2004)] Chin-Yew Lin.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain, July 2004. Association for Computational Linguistics.  URL https://aclanthology.org/W04-1013.
  * [Liu et al.(2021)Liu, Yin, Wu, Ge, Zhang, and Sun] Fenglin Liu, Changchang Yin, Xian Wu, Shen Ge, Ping Zhang, and Xu Sun.  Contrastive attention for automatic chest x-ray report generation.  _arXiv preprint arXiv:2106.06965_ , 2021.
  * [Liu et al.(2019)Liu, Zhao, Fei, Zhang, Wang, and Yu] Jingyu Liu, Gangming Zhao, Yu Fei, Ming Zhang, Yizhou Wang, and Yizhou Yu.  Align, attend and locate: Chest x-ray diagnosis via contrast induced attention network with limited supervision.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 10632–10641, 2019.
  * [Lu et al.(2017)Lu, Xiong, Parikh, and Socher] Jiasen Lu, Caiming Xiong, Devi Parikh, and Richard Socher.  Knowing when to look: Adaptive attention via a visual sentinel for image captioning.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 375–383, 2017.
  * [Misra and Maaten(2020)] Ishan Misra and Laurens van der Maaten.  Self-supervised learning of pretext-invariant representations.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2020.
  * [Papineni et al.(2002)Papineni, Roukos, Ward, and Zhu] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  Bleu: A method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting on Association for Computational Linguistics_ , ACL ’02, page 311–318, USA, 2002. Association for Computational Linguistics.  10.3115/1073083.1073135.  URL https://doi.org/10.3115/1073083.1073135.
  * [Rennie et al.(2017)Rennie, Marcheret, Mroueh, Ross, and Goel] Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel.  Self-critical sequence training for image captioning.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 7008–7024, 2017.
  * [Shin et al.(2016)Shin, Roberts, Lu, Demner-Fushman, Yao, and Summers] Hoo-Chang Shin, Kirk Roberts, Le Lu, Dina Demner-Fushman, Jianhua Yao, and Ronald M Summers.  Learning to read chest x-rays: Recurrent neural cascade model for automated image annotation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2497–2506, 2016.
  * [Syeda-Mahmood et al.(2020)Syeda-Mahmood, Wong, Gur, Wu, Jadhav, Kashyap, Karargyris, Pillai, Sharma, Syed, et al.] Tanveer Syeda-Mahmood, Ken CL Wong, Yaniv Gur, Joy T Wu, Ashutosh Jadhav, Satyananda Kashyap, Alexandros Karargyris, Anup Pillai, Arjun Sharma, Ali Bin Syed, et al.  Chest x-ray report generation through fine-grained label learning.  In _International Conference on Medical Image Computing and Computer-Assisted Intervention_ , pages 561–571. Springer, 2020.
  * [Vaswani et al.(2017)Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, and Polosukhin] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _Advances in neural information processing systems_ , pages 5998–6008, 2017.
  * [Xue et al.(2018)Xue, Xu, Rodney Long, Xue, Antani, Thoma, and Huang] Yuan Xue, Tao Xu, L Rodney Long, Zhiyun Xue, Sameer Antani, George R Thoma, and Xiaolei Huang.  Multimodal recurrent model with attention for automated radiology report generation.  In _International Conference on Medical Image Computing and Computer-Assisted Intervention_ , pages 457–466. Springer, 2018.
  * [Yuan et al.(2019)Yuan, Liao, Luo, and Luo] Jianbo Yuan, Haofu Liao, Rui Luo, and Jiebo Luo.  Automatic radiology report generation based on multi-view image fusion and medical concept enrichment.  In _International Conference on Medical Image Computing and Computer-Assisted Intervention_ , pages 721–729. Springer, 2019.

## Acknowledgement

We thank the National Center for High-performance Computing (NCHC) for
providing computational and storage resources.


Higgs Physics at HL-LHC

Michaela Mlynarikova, on behalf of the ATLAS and CMS
Collaborations111Copyright 2023 CERN for the benefit of the ATLAS and CMS
Collaborations. Reproduction of this article or parts of it is allowed as
specified in the CC-BY-4.0 license.

CERN

> The large dataset of about 3000 fb-1 that will be collected by both the
> ATLAS and CMS experiments at the High Luminosity LHC (HL-LHC) will be used
> to measure Higgs boson properties in detail. Studies based on current
> analyses have been carried out to understand the expected precision and
> limitations of these measurements. The large dataset will also allow for
> better sensitivity to Higgs boson pair production processes and the Higgs
> boson self-coupling. This proceeding presents the prospects for the precise
> measurements of the Higgs boson properties and Higgs boson pair production
> with the ATLAS and CMS detectors at the HL-LHC.

> PRESENTED AT
>
>
>
>
> DIS2023: XXX International Workshop on Deep-Inelastic Scattering and Related
> Subjects,
> Michigan State University, USA, 27-31 March 2023
>

##  1 Introduction to the HL-LHC

On July 4, 2012, the ATLAS [1] and CMS [2] collaborations at the Large Hadron
Collider (LHC) made a groundbreaking announcement, revealing the discovery of
a Higgs boson with a mass of 125 GeV [3, 4]. In the Standard Model [5, 6, 7,
8, 9, 10] (SM) the couplings of the Higgs boson are defined once its mass is
known. Following the Higgs boson discovery, both collaborations started an
extensive programme of measurements of its properties, which is currently
ongoing using data from LHC Run 2. These measurements, including its mass [11,
12], spin and parity [13, 14], as well as production and decay rates [15, 16],
have found no significant deviations from the SM expectations. While the
hypothesis of a purely pseudo-scalar Higgs boson has been already ruled out in
certain bosonic couplings [17, 18], $CP$ violation might still occur at tree
level in fermion couplings [19, 20, 21]. Numerous alternative theories beyond
the SM (BSM) exist, each making different predictions about the properties of
one or more Higgs bosons. Therefore, precise measurements in the Higgs sector
are of utmost importance for the future programme of particle physics, as they
allow us to test how well the measured values of the Higgs boson properties
align with the SM predictions with increased precision. To increase the
precision of the Higgs properties measurements, it will be necessary to
collect more data.

Addressing this challenge, an extensive upgrade program has been implemented
for both the LHC and the experiments in several phases. The High-Luminosity
LHC (HL-LHC) is expected to deliver 3000 to 4000 fb-1 of data, 10 times more
than the integrated luminosity of the LHC Runs 1–3 combined. The HL-LHC
presents a very challenging environment to the existing experiments at the LHC
due to a peak luminosity of $5-7\times
10^{-34}~{}\mathrm{cm}^{-2}\mathrm{s}^{-1}$ and an average number of $pp$
interactions per bunch crossing (pile-up) of 140 to 200.

##  2 Detector upgrades for HL-LHC

In order to achieve the physics objectives established for the HL-LHC,
significant upgrades are required for both the ATLAS and CMS detectors. These
upgrades are crucial for maintaining or enhancing the performance and physics
acceptance of the detectors. While a comprehensive description of all planned
upgrades is beyond the scope of this text, a summary of selected upgrades is
provided below.

###  2.1 The ATLAS detector

The ATLAS detector is carrying out an upgrade program to accommodate the
higher occupancies and data rates expected at the HL-LHC. These upgrades
encompass various components of the detector system. The inner tracking
detector will be entirely replaced with a new all-silicon Inner Tracker (ITk).
This upgrade will expand the coverage of the tracking detector up to a
pseudorapidity of $|\eta|=4$, and enabling more precise measurements of
particle trajectories. New front-end electronics and a new read-out system in
the calorimeters will allow for the read-out of higher resolution objects at
the lowest trigger level at an increased rate. In the muon detector system,
additional inner barrel chambers will be installed to extend the coverage at
the trigger level. Furthermore, a new detector known as the high-granularity
timing detector will be installed in the forward region. This detector will
play a crucial role in distinguishing jets originating from different proton-
proton interactions, thereby enhancing the precision of event reconstruction
and analysis. The Trigger and Data Acquisition (TDAQ) system will be
extensively redesigned. This redesign aims to provide hardware tracking with
full granularity detector data at a selection rate of 1 MHz, while the
software-based selection output will be limited to 10 kHz. This upgrade will
improve the overall efficiency and performance of the trigger system.

###  2.2 The CMS detector

Similarly, the CMS detector is undergoing significant upgrades to meet the
challenges posed by the HL-LHC environment. These include a replacement of the
silicon strip and pixel detectors to increase the granularity while reducing
material as well as increasing the coverage to $|\eta|=3.8$. Upgrades in the
calorimeter system include the replacement of the endcap electromagnetic and
hadron calorimeters with a new high-granularity sampling detector and the
front-end electronics in the electromagnetic barrel enabling trigger readout
at 40 MHz. One of the key enhancements is the extension of new muon chambers
in the forward region, expanding the muon coverage up to $|\eta|=3$. To
improve the particle-flow performance, a minimum ionizing particle timing
detector will be installed in the central region of the CMS detector. Lastly,
also the TDAQ system will undergo upgrades for hardware tracking at 40 MHz
with a software selection output of 7.5 kHz.

##  3 Physics prospects at the HL-LHC

Higgs boson physics will play a crucial role in the extensive physics program
at the HL-LHC. Key measurements will focus on the Higgs boson couplings and
self-coupling, Higgs boson differential distributions, rare Higgs boson decays
and searches for heavy Higgs particles. All these investigations will shed
light on the properties and behavior of the Higgs boson. Moreover, there is
substantial support for a diverse program encompassing SM physics, flavor
physics, as well as searches for the BSM physics. These research areas
complement the Higgs boson studies and contribute to a comprehensive
understanding of fundamental particles and their interactions. In the
following sections, we will provide concise descriptions of selected
projections that have been presented. For more comprehensive details, readers
are encouraged to refer to the respective references. These performance
assessments and projections have been conducted as contributions to the CERN
Yellow Report from European Strategy for Particle Physics (2019) [22], and
feature in the Snowmass White Paper (2022) [23].

###  3.1 HL-LHC projection strategies

All HL-LHC projections presented in this document rely on the ATLAS and CMS
analyses results which utilizes LHC Run-2 $pp$ collisions at $\sqrt{s}=13$
TeV. For the extrapolations, a dataset of 3000 fb-1 of $pp$ collisions at
$\sqrt{s}=14$ TeV was considered. Several scenarios have been considered for
the systematic uncertainties estimates. In a baseline scenario, often referred
to as ”YR18 systematics uncertainties” scenario, most of the experimental
uncertainties are scaled down with the square root of the integrated
luminosity. Statistical uncertainties in measurements are reduced by a factor
of $1/\sqrt{\mathcal{L}}$, where $\mathcal{L}$ is the expected HL-LHC
integrated luminosity divided by that of the reference Run-2 analysis.
Uncertainties related to the limited number of simulated events are neglected,
assuming that sufficiently large simulation samples will be available at that
time. We anticipate improvements in theoretical predictions, and thus the
theoretical uncertainties are halved compared to current values. The
uncertainty in the integrated luminosity of the data sample is taken as 1%,
reflecting expectations of a precise understanding of calibration methods and
the utilization of the enhanced capabilities of the upgraded detectors.
Uncertainties associated with analysis methods are assumed to remain
unchanged, as we expect that any challenges posed by the harsher conditions of
the HL-LHC will be offset by improvements in analysis techniques.

###  3.2 Higgs boson couplings to SM particles

An extrapolation to the HL-LHC luminosity of the measurements in the main SM
Higgs boson decay channels (ATLAS and CMS: $\gamma\gamma$, $ZZ$, $W^{+}W^{-}$,
$\tau^{+}\tau^{-}$, $b\bar{b}$, and $\mu^{+}\mu^{-}$; ATLAS only: $Z\gamma$)
was performed per experiment and then combined using a simplified treatment
[23]. The interpretation of Higgs boson physics can be done in terms of
multiplicative coupling modifiers [24], denoted as $\kappa$, which represent
potential deviations from the SM predictions. These modifiers introduce a
linear change in the couplings of the Higgs boson to SM bosons and fermions.
In this implementation, six coupling modifiers corresponding to the tree-level
Higgs boson couplings are defined: $\kappa_{W}$, $\kappa_{Z}$, $\kappa_{t}$ ,
$\kappa_{b}$, $\kappa_{\tau}$, and $\kappa_{\mu}$. Additionally, three
effective coupling modifiers, $\kappa_{g}$, $\kappa_{\gamma}$, and
$\kappa_{Z\gamma}$, are introduced to describe the loop-induced processes
$gg\to H$, $H\to gg$, $gg\to ZH$, $H\to\gamma\gamma$ and $H\to Z\gamma$.
Figure 1 summarizes the expected uncertainties for the ATLAS+CMS HL-LHC
sensitivity to $\kappa$ parameters. The total expected uncertainty ranges from
a few percent to around 10%. Specifically, the uncertainties range from 1.5%
for $\kappa_{Z}$ to 4.3% for $\kappa_{\mu}$, with $\kappa_{Z\gamma}$
approaching 10%. It is important to note that theoretical uncertainties,
particularly for $\kappa_{t}$, $\kappa_{b}$, and $\kappa_{g}$, play a
significant role in these measurements. The expected precision on the coupling
modifiers assume that decays to SM particles only are allowed.

Figure 1: The total expected uncertainties on the coupling modifier parameters
($\kappa$), for the combination of ATLAS and CMS extrapolations [23]. For each
measurement, the total uncertainty is indicated by a grey box while the
statistical, experimental and theory uncertainties are indicated by a blue,
green and red line respectively. In addition, the numerical values are also
reported.

###  3.3 Higgs boson mass and width

The CMS HL-LHC projection for the measurement of the Higgs boson mass and
width [25] for the on-shell H production in $ZZ\to 4\ell$ decay mode, where
$\ell=e,\mu$, is based on the full Run-2 analysis. The latest public observed
results for Higgs boson mass and on-shell width measurements in the four
leptons final state were extracted from the analysis of 36 fb-1 data collected
during LHC Run 2, as follows: $m_{H}=125.26\pm 0.21[0.20(\mathrm{stat})\pm
0.08(\mathrm{syst})]$ GeV and the width is constrained to be
$\Gamma_{H}<0.41(1.10)$ GeV at 68(95)% confidence level [26]. The HL-LHC
projection workflow follows the one described in Ref. [26, 27] with several
improvements, such as a new approach implemented in the final likelihood
building. The projected expected result, for mass measurement, is
$m_{H}=125.38\pm 0.03[0.022(\mathrm{stat})\pm 0.020(\mathrm{syst})]$ GeV and
for width is $\Gamma_{H}<0.09(0.18)$ GeV at 68(95)% confidence level. Left
plot in Figure 2 shows the one dimensional likelihood scan comparing the
different contribution of each final state at the final results (at
statistical level only). The impact of the detector upgrades has been studied
as well using $gg\to H$ samples and DELPHES simulation [28]. The resolution of
the four muon invariant mass is expected to improve by 25% thanks to the new
tracker, providing an improvement of 17% in the $4e$ final state yield and a
similar yield improvement is expected in $4\mu$ final state thanks to the new
muon station. If detector upgrades are taken into account, further mass
measurement improvements are foreseeable. Increased acceptance for muons and
electrons and reduced systematic uncertainties (Optimistic scenario), lead to
a precision of 20 MeV on the mass measuremen including both statistical and
systematic uncertainties.

###  3.4 $CP$ structure of the Yukawa coupling between the Higgs boson and
$\tau$ leptons

Figure 2: Left: 1D likelihood scan comparing the contribution of each final
state in the Higgs boson mass measurement in $ZZ\to 4\ell$ decay mode, where
$\ell=e,\mu$ [25]. Right: Projections of the expected negative log-likelihood
scans as a function of the $CP$ mixing angle for the $H\tau\tau$ coupling,
different assumptions about the systematic uncertainties are compared [23].

Projections for the measurement of the $CP$ properties of the Higgs boson
coupling to $\tau$ leptons [23] at HL-LHC have been derived, based on the
Run-2 CMS analysis [21]. The $CP$ nature of the interaction is described in
terms of an effective mixing angle $\alpha^{H\tau\tau}$. A mixing angle of
$\alpha^{H\tau\tau}=0^{\circ}(90^{\circ})$ corresponds to a pure scalar
(pseudoscalar) coupling. Any other value of $\alpha^{H\tau\tau}$ indicates a
mixed coupling with both $CP$-even and $CP$-odd components, implying $CP$
violation. The projected negative log-likelihood scans are shown in the right
plot of Figure 2, comparing different assumptions about the systematic
uncertainties. For all systematic uncertainty schemes considered, the total
expected uncertainty on $\alpha^{H\tau\tau}$ is $5^{\circ}$, representing a
substantial reduction from the previous measurement of $4\pm 17^{\circ}$
achieved during Run 2. However, the sensitivities to larger values of
$\alpha^{H\tau\tau}$ differ depending on the assumptions made about the
systematic uncertainties.

###  3.5 $H\to\tau\tau$ cross-section measurement

Figure 3: Run-2 (red) and projected HL-LHC (blue) expected precision of the
$H\to\tau^{+}\tau^{-}$ measurement scaled to the cross-section expectation
value ($\Delta\sigma_{i}/\sigma_{i}$) in various STXS bins labelled $i$. The
uncertainty on the predicted signal cross-section in each bin is also shown,
illustrating the current (light grey) and projected HL-LHC (hashed) precision
of theory calculations [29].

An extrapolation is performed to assess the expected sensitivity of $pp\to
H\to\tau^{+}\tau^{-}$ cross-section measurements [29] at the HL-LHC and based
on the ATLAS Run-2 results obtained with 139 fb-1 of $pp$ collisions at
$\sqrt{s}=13$ TeV. The projected precision for the inclusive $pp\to
H\to\tau^{+}\tau^{-}$ cross-section measurement is 5%, corresponding to a
value of $\sigma(pp\to H\to\tau^{+}\tau^{-})_{\mathrm{exp}}/\sigma(pp\to
H\to\tau^{+}\tau^{-})_{\mathrm{SM}}=1.00\pm 0.05=1.00\pm
0.01(\mathrm{stat.})\pm 0.04(\mathrm{sig.th.})\pm 0.02(\mathrm{syst.})$. For
the four dominant production modes, the projected precision is 11%, 7%, 14%
and 24% for $gg$F, VBF, $VH$ and $t\bar{t}H$ respectively. This corresponds to
the observation of all production modes except for $t\bar{t}H$, where the
significance reaches $4.4\sigma$. Projected measurements in the Simplified
Template Cross Sections [30] (STXS) framework are shown in Figure 3, which
also shows the precision of the Run-2 measurements for comparison. The most
sensitive projected measurements are the $\mathrm{VBF}+V(\to qq)H$ cross-
section in events with at least two jets and a di-jet invariant mass of at
least 350 GeV (VBF topology), with an uncertainty of 7%. Additionally, the
$\mathrm{ggF}+gg\to Z(\to qq)H$ cross-section in events with Higgs boson
transverse momentum between 200 and 300 GeV, is projected to have an
uncertainty of 10%, while for transverse momenta above 300 GeV, the
uncertainty is 11%. These two measurements are expected to be among the most
sensitive ones for Higgs boson production in that momentum range at the HL-
LHC.

###  3.6 Higgs boson pair production

Figure 4: Left: Projected significance for SM $HH$ production combining the
$b\bar{b}b\bar{b}$, $bb\tau\tau$ and $bb\gamma\gamma$ channels from 1000 fb-1
to 3000 fb-1 at $\sqrt{s}=14$ TeV at the HL-LHC assuming the four different
uncertainty scenarios described in reference [31]. The significance is
evaluated using a signal plus background Asimov dataset generated under the SM
hypothesis. Right: Negative log-profile-likelihood as a function of
$\kappa_{\lambda}$ evaluated on an Asimov dataset constructed under the SM
hypothesis of $\kappa_{\lambda}=1$, for $b\bar{b}b\bar{b}$, $bb\tau\tau$ and
$bb\gamma\gamma$ projections [31], and their combination assuming no
systematic uncertainties. The intersections of the dashed horizontal lines
with the profile likelihood curve define the 68% and 95% confidence intervals,
respectively.

Prospects for the Higgs boson pair production at the HL-LHC are performed
using the $b\bar{b}b\bar{b}$, $bb\tau\tau$ and $bb\gamma\gamma$ final states
[31]. For the $HH\to b\bar{b}b\bar{b}$ analysis, a projection is made using
the latest results based on 126 fb-1 of data collected during Run 2 of the LHC
with the ATLAS detector. For the $HH\to bb\tau\tau$ and $HH\to bb\gamma\gamma$
analyses, projections are based on extrapolations of the Run-2 analysis
conducted with 139 fb-1. Figure 4 shows expected significance for the di-Higgs
signal and the exclusion limit on the Higgs boson self-coupling modifier
$\kappa_{\lambda}=\lambda_{HHH}/\lambda_{HHH}^{\mathrm{SM}}$, where
$\lambda_{HHH}$ is the tri-linear Higgs boson self-coupling. The statistical
combination of the $b\bar{b}b\bar{b}$, $bb\tau\tau$ and $bb\gamma\gamma$
search channels results in a projected significance of $HH$ signal of
$3.4\sigma$ ($4.9\sigma$) assuming the baseline (no systematic uncertainties)
scenario. The measurement of the $HH$ signal strength relative to the SM
prediction is expected to have an accuracy of ${}^{+33}_{-30}$% under the
baseline systematic uncertainty scenario. Finally, the combined result
constrains the $1\sigma$ confidence interval on $\kappa_{\lambda}$ to be
within the range of [0.5, 1.6].

##  4 Conclusion

The estimates indicate that the HL-LHC will offer a great opportunity for the
Higgs boson precision measurements, enabling us to measure its couplings with
a few percent precision and opening up new possibilities for exploring rare
processes. Notably, the observation of decays such as $H\to\mu\mu$ and $H\to
Z\gamma$, as well as the evidence of HH production, will become possible.
Significant progress is also anticipated through enhancements in object
reconstruction and identification performance, the reduction of systematic
uncertainties (both experimental and theoretical), and advancements in
analysis techniques. These improvements will yield further enhancements beyond
the anticipated gains from increased luminosity, as considered in the
prospective studies.

## References

  * [1] ATLAS Collaboration.  The ATLAS Experiment at the CERN Large Hadron Collider.  JINST, 3:S08003, 2008.
  * [2] CMS Collaboration.  The CMS Experiment at the CERN LHC.  JINST, 3:S08004, 2008.
  * [3] ATLAS Collaboration.  Observation of a new particle in the search for the Standard Model Higgs boson with the ATLAS detector at the LHC.  Phys. Lett. B, 716:1, 2012.
  * [4] CMS Collaboration.  Observation of a new boson at a mass of 125 GeV with the CMS experiment at the LHC.  Phys. Lett. B, 716:30, 2012.
  * [5] S. L. Glashow.  Partial-symmetries of weak interactions.  Nuclear Physics, 22(4):579–588, 1961.
  * [6] S. Weinberg.  A model of leptons.  Phys. Rev. Lett., 19:1264–1266, 1967.
  * [7] A. Salam.  Weak and Electromagnetic Interactions.  Conf. Proc. C, 680519:367–377, 1968.
  * [8] P. W. Higgs.  Broken Symmetries and the Masses of Gauge Bosons.  Phys. Rev. Lett., 13:508–509, 1964.
  * [9] P. W. Higgs.  Spontaneous Symmetry Breakdown without Massless Bosons.  Phys. Rev., 145:1156–1163, 1966.
  * [10] F. Englert and R. Brout.  Broken Symmetry and the Mass of Gauge Vector Mesons.  Phys. Rev. Lett., 13:321–323, 1964.
  * [11] ATLAS Collaboration.  Measurement of the Higgs boson mass in the $H\to ZZ^{*}\to 4\ell$ decay channel using $139\,\text{fb}^{-1}$ of $\sqrt{s}=13\,\text{TeV}$ $pp$ collisions recorded by the ATLAS detector at the LHC.  2022\.
  * [12] CMS Collaboration.  A measurement of the Higgs boson mass in the diphoton decay channel.  Phys. Lett. B, 805:135425, 2020.
  * [13] ATLAS Collaboration.  Study of the spin and parity of the Higgs boson in diboson decays with the ATLAS detector.  Eur. Phys. J. C, 75:476, 2015.
  * [14] CMS Collaboration.  Constraints on the spin-parity and anomalous HVV couplings of the Higgs boson in proton collisions at $7$ and $8\,\text{TeV}$.  Phys. Rev. D, 92:012004, 2015.
  * [15] ATLAS Collaboration.  A detailed map of Higgs boson interactions by the ATLAS experiment ten years after the discovery.  Nature, 607:52–59, 2022.
  * [16] CMS Collaboration.  Combined higgs boson production and decay measurements with up to 137 fb-1 of proton-proton collision data at $\sqrt{s}$ = 13 tev.  Technical Report CMS-PAS-HIG-19-00, CERN, 2020.
  * [17] ATLAS Collaboration.  Test of cp-invariance of the higgs boson in vector-boson fusion production and its decay into four leptons, 2023.
  * [18] CMS Collaboration.  Measurement of the properties of a Higgs boson in the four-lepton final state.  Phys. Rev. D, 89:092007, 2014.
  * [19] ATLAS Collaboration.  Measurement of the $CP$ properties of Higgs boson interactions with $\tau$-leptons with the ATLAS detector.  2022\.
  * [20] CMS Collaboration.  Measurements of $t\bar{t}H$ Production and the $CP$ Structure of the Yukawa Interaction between the Higgs Boson and Top Quark in the Diphoton Decay Channel.  Phys. Rev. Lett., 125:061801, 2020.
  * [21] CMS Collaboration.  Analysis of the CP structure of the Yukawa coupling between the Higgs boson and $\tau$ leptons in proton–proton collisions at $\sqrt{s}=13\,\text{TeV}$.  JHEP, 06:012, 2022.
  * [22] ATLAS Collaboration.  Report on the Physics at the HL-LHC and Perspectives for the HE-LHC.  ATL-PHYS-PUB-2019-006, 2019.
  * [23] ATLAS and CMS Collaborations.  Snowmass White Paper Contribution: Physics with the Phase-2 ATLAS and CMS Detectors.  ATL-PHYS-PUB-2022-018, 2022.
  * [24] J. R. Andersen et al.  Handbook of LHC Higgs Cross Sections: 3. Higgs Properties.  7 2013.
  * [25] CMS Collaboration.  Projection of the Higgs boson mass and on-shell width measurements in $H\rightarrow ZZ\rightarrow 4\ell$ decay channel at the HL-LHC,.  Technical Report CMS-PAS-FTR-21-007, CERN, 2022.
  * [26] CMS Collaboration.  Measurements of properties of the Higgs boson decaying into the four-lepton final state in $pp$ collisions at $\sqrt{s}=13\,\text{TeV}$.  JHEP, 11:047, 2017.
  * [27] CMS Collaboration.  Measurements of production cross sections of the Higgs boson in the four-lepton final state in proton–proton collisions at $\sqrt{s}=13\,\text{TeV}$.  Eur. Phys. J. C, 81:488, 2021.
  * [28] J. de Favereau et al.  DELPHES 3: a modular framework for fast simulation of a generic collider experiment.  Journal of High Energy Physics, 2014(2), 2014.
  * [29] ATLAS Collaboration.  Projection of $H\to\tau\tau$ cross-section measurement results to the HL-LHC.  ATL-PHYS-PUB-2022-003, 2022.
  * [30] N. Berger et al.  Simplified template cross sections - stage 1.1.  arXiv: 1906.02754, 2019.
  * [31] ATLAS Collaboration.  HL-LHC prospects for the measurement of Higgs boson pair production in the $b\bar{b}b\bar{b}$ final state and combination with the $b\bar{b}\gamma\gamma$ and $b\bar{b}\tau^{+}\tau^{-}$ final states at the ATLAS experiment.  ATL-PHYS-PUB-2022-053, 2022.


# Exploring the Potential of Polynomial Basis Functions in Kolmogorov-Arnold
Networks: A Comparative Study of Different Groups of Polynomials

Seyd Teymoor Seydi
University of Tehran
Tehran, Iran
<EMAIL_ADDRESS>

###### Abstract

This paper presents a comprehensive survey of 18 distinct polynomials and
their potential applications in Kolmogorov-Arnold Network (KAN) models as an
alternative to traditional spline-based methods. The polynomials are
classified into various groups based on their mathematical properties, such as
orthogonal polynomials, hypergeometric polynomials, q-polynomials, Fibonacci-
related polynomials, combinatorial polynomials, and number-theoretic
polynomials. The study aims to investigate the suitability of these
polynomials as basis functions in KAN models for complex tasks like
handwritten digit classification on the MNIST dataset. The performance metrics
of the KAN models, including overall accuracy, Kappa, and F1 score, are
evaluated and compared. The Gottlieb-KAN model achieves the highest
performance across all metrics, suggesting its potential as a suitable choice
for the given task. However, further analysis and tuning of these polynomials
on more complex datasets are necessary to fully understand their capabilities
in KAN models. The source code for the implementation of these KAN models is
available at https://github.com/seydi1370/Basis_Functions.

##  1 Introduction

Neural networks have revolutionized various fields, including computer vision,
natural language processing, and more. Kolmogorov-Arnold Networks (KANs) [1]
have shown promise in approximating complex functions using a combination of
simple functions. KANs are inspired by the Kolmogorov-Arnold representation
theorem [1], which states that any continuous function can be represented as a
composition of simple functions.

Recent works have explored the connection between KANs and neural networks.
Liu et al. [2] proposed a generalized version of KANs that can have arbitrary
widths and depths, showing their potential as a promising alternative to
multi-layer perceptrons (MLPs). Bozorgasl et al. [3] introduced Wav-KAN, which
incorporates wavelet functions into the KAN structure to enhance
interpretability and performance. Samadi et al. [4] discussed the importance
of smoothness in KANs and proposed smooth, structurally informed KANs that can
achieve equivalence to MLPs in specific function classes. SS et al. [5]
introduced the Chebyshev Kolmogorov-Arnold Network (Chebyshev KAN), which
combines the theoretical foundations of the Kolmogorov-Arnold Theorem with the
powerful approximation capabilities of Chebyshev polynomials to efficiently
approximate nonlinear functions.

KANs offer several advantages over traditional neural network architectures.
They provide a more interpretable and efficient representation of complex
functions by breaking them down into simpler components [2]. KANs can also
achieve better accuracy and generalization with fewer parameters compared to
MLPs [3]. Furthermore, by leveraging inherent structural knowledge, KANs may
reduce the data required for training and mitigate the risk of generating
hallucinated predictions [4].

In this study, we aim to explore the potential of using various polynomials as
basis functions in KAN models. We conduct a comprehensive survey of 18
distinct polynomials, classifying them into groups based on their mathematical
properties, such as orthogonal polynomials, hypergeometric polynomials,
q-polynomials, Fibonacci-related polynomials, combinatorial polynomials, and
number-theoretic polynomials. By evaluating the performance of KAN models
incorporating these polynomials on the MNIST dataset for handwritten digit
classification, we investigate their suitability and effectiveness in
approximating complex functions.

The main contributions of this study are as follows:

  1. 1.

A structured overview of 18 polynomials and their characteristics, providing
insights into their potential applications in KAN models.

  2. 2.

An evaluation of the performance of KAN models incorporating these polynomials
on the MNIST dataset.

  3. 3.

An analysis of the relationships between model complexity, number of
parameters, training time, and overall performance, offering insights into the
factors influencing KAN model performance.

  4. 4.

Identification of future research directions, emphasizing the need for further
investigation and tuning of these polynomials on more complex datasets and the
development of advanced analytical techniques for informed model selection and
optimization.

##  2 Background

###  2.1 Kolmogorov-Arnold Networks

Kolmogorov-Arnold Networks (KANs) are based on the Kolmogorov-Arnold
representation theorem [1], which states that any continuous multivariate
function can be represented as a composition of univariate continuous
functions and sums. Specifically, for a function $f(y_{1},y_{2},\ldots,y_{m})$
with $m$ input variables, the Kolmogorov-Arnold representation theorem states
that it can be expressed as:

$f(y)=f(y_{1},y_{2},\ldots,y_{m})=\sum_{r=1}^{2m+1}\Psi_{r}\left(\sum_{s=1}^{m}\psi_{r,s}(y_{s})\right),$
(1)

where $\psi_{r,s}:[0,1]\to\mathbb{R}$ and $\Psi_{r}:\mathbb{R}\to\mathbb{R}$.

where $\varphi_{r,s}:[0,1]\rightarrow\mathbb{R}$ are inner univariate
functions, and $\Phi_{r}:\mathbb{R}\rightarrow\mathbb{R}$ are outer univariate
functions. KANs are designed to approximate functions by parametrizing and
learning these inner and outer univariate functions using a combination of
spline coefficients and a set of simple functions.

###  2.2 A Survey of Different Groups of Polynomials

This study investigates 18 polynomials and their mathematical properties,
focusing on their potential use in Kolmogorov-Arnold Network (KAN) models as
an alternative to traditional spline-based methods. The polynomials are
classified into several distinct groups based on their characteristics and
areas of application.

Table 1 presents a classification of the 18 polynomials surveyed in the study,
along with their respective parameter sets. The polynomials are grouped into
several categories based on their mathematical properties and areas of
application, such as orthogonal polynomials, hypergeometric polynomials,
q-polynomials, Fibonacci-related polynomials, combinatorial polynomials, and
number-theoretic polynomials.

The Al-Salam-Carlitz polynomials are orthogonal polynomials that generalize
the Chebyshev polynomials. They have applications in quantum algebra and the
theory of basic hypergeometric series [6].

$U_{n+1}^{(a)}(x;q)=(x-aq^{n})U_{n}^{(a)}(x;q)-q^{n-1}(1-q^{n})U_{n-1}^{(a)}(x;q)$
(2)

Bannai-Ito polynomials are orthogonal polynomials that generalize the Racah
and Wilson polynomials. They have connections to the Askey scheme of
hypergeometric orthogonal polynomials [7].

$B_{n+1}(x)=(x-\rho_{n})B_{n}(x)-\tau_{n}B_{n-1}(x)$ (3)

Askey-Wilson polynomials are a four-parameter family of orthogonal polynomials
that generalize the classical orthogonal polynomials. They have applications
in quantum groups and the theory of special functions [8].

$p_{n}(x;a,b,c,d|q)=\frac{(ab,ac,ad;q)_{n}}{a^{n}}\,_{4}\phi_{3}\left(\begin{array}[]{c}q^{-n},abcdq^{n-1},ae^{i\theta},ae^{-i\theta}\\\
ab,ac,ad\end{array};q,q\right)$ (4)

Boas-Buck polynomials are a generalization of the Hermite, Laguerre, and other
classical orthogonal polynomials. They have applications in the theory of
special functions and generating functions [9].

$g(t)P_{n}(x)=\sum_{k=0}^{n}\binom{n}{k}f(t)^{k}P_{k}(x)$ (5)

Boubaker polynomials are a family of orthogonal polynomials that have
applications in mathematical physics, particularly in the solution of heat
transfer problems [10].

$B_{n}(x)=\frac{1}{n!}\frac{d^{n}}{dx^{n}}[(1-x^{2})^{n}]$ (6)

Fermat polynomials are a family of polynomials that arise in number theory and
have connections to Fermat’s Last Theorem. They are related to the cyclotomic
polynomials [11].

$F_{n}(x)=\prod_{k=0}^{n-1}(x-e^{2\pi ik/n})$ (7)

Gottlieb polynomials are a family of polynomials that arise in the study of
the Bernoulli numbers and have applications in combinatorics and number theory
[12].

$G_{n}(x)=\sum_{k=0}^{n}\binom{n}{k}B_{k}x^{n-k}$ (8)

Heptanacci polynomials are a generalization of the Fibonacci polynomials and
the heptanacci numbers. They have applications in combinatorics and number
theory [13].

$H_{n}(x)=xH_{n-1}(x)+H_{n-2}(x)+H_{n-3}(x)+H_{n-4}(x)+H_{n-5}(x)+H_{n-6}(x)+H_{n-7}(x)$
(9)

Hexanacci polynomials are a generalization of the Fibonacci polynomials and
the hexanacci numbers. They have applications in combinatorics and number
theory [14].

$H_{n}(x)=xH_{n-1}(x)+H_{n-2}(x)+H_{n-3}(x)+H_{n-4}(x)+H_{n-5}(x)+H_{n-6}(x)$
(10)

Meixner-Pollaczek polynomials are a family of orthogonal polynomials that
generalize the Laguerre polynomials. They have applications in quantum
mechanics and the theory of special functions [15].

$P_{n}^{(\lambda)}(x;\phi)=\frac{(2\lambda)_{n}}{n!}e^{in\phi}{{}_{2}F_{1}}\left(-n,\lambda+ix;\,2\lambda;\,1-e^{-2i\phi}\right)$
(11)

Narayana polynomials are a family of polynomials that have combinatorial
interpretations and are related to the Catalan numbers. They have applications
in lattice path enumeration and the theory of Dyck words [16].

$N_{n}(x)=\sum_{k=0}^{n}\frac{1}{n}\binom{n}{k}\binom{n}{k+1}x^{k}$ (12)

Octanacci polynomials are a generalization of the Fibonacci polynomials and
the octanacci numbers. They have applications in combinatorics and number
theory [17].

$O_{n}(x)=xO_{n-1}(x)+O_{n-2}(x)+O_{n-3}(x)+O_{n-4}(x)+O_{n-5}(x)+O_{n-6}(x)+O_{n-7}(x)+O_{n-8}(x)$
(13)

Padé polynomials are a family of rational approximations to a given function.
They have applications in numerical analysis, particularly in the
approximation of special functions and the solution of differential equations
[18].

$\sum_{k=0}^{m}a_{k}f_{n+k}=\sum_{k=0}^{n}b_{k}f_{k}$ (14)

Pentanacci polynomials are a generalization of the Fibonacci polynomials and
the pentanacci numbers. They have applications in combinatorics and number
theory [19].

$P_{n}(x)=xP_{n-1}(x)+P_{n-2}(x)+P_{n-3}(x)+P_{n-4}(x)+P_{n-5}(x)$ (15)

Tetranacci polynomials are a generalization of the Fibonacci polynomials and
the tetranacci numbers. They have applications in combinatorics and number
theory [20].

$T_{n}(x)=xT_{n-1}(x)+T_{n-2}(x)+T_{n-3}(x)+T_{n-4}(x)$ (16)

Tribo polynomials are a generalization of the Fibonacci polynomials and the
tribonacci numbers. They have applications in combinatorics and number theory
[21].

$T_{n}(x)=xT_{n-1}(x)+T_{n-2}(x)+T_{n-3}(x)$ (17)

VietaPell polynomials are a generalization of the Vieta polynomials and the
Pell numbers. They have applications in combinatorics and number theory [22].

$VP_{n}(x)=2xVP_{n-1}(x)+VP_{n-2}(x)$ (18)

Charlier polynomials, denoted by $C_{n}^{(a)}(x)$, are a family of discrete
orthogonal polynomials that are solutions to the Charlier differential
equation [23]. They are named after Carl Charlier, a Swedish astronomer and
mathematician who introduced them in 1905.

The Charlier polynomials are defined by the generating function:

$\sum_{n=0}^{\infty}C_{n}^{(a)}(x)\frac{t^{n}}{n!}=e^{t}\left(1-\frac{t}{a}\right)^{x}$
(19)

where $a>0$ is a real parameter and $x$ is a non-negative integer.

The explicit formula for Charlier polynomials is given by:

$C_{n}^{(a)}(x)={}_{2}F_{0}\left(-n,-x;-;-\frac{1}{a}\right)=\sum_{k=0}^{n}\binom{n}{k}\frac{(-a)^{n-k}}{k!}x^{\underline{k}}$
(20)

where ${}_{2}F_{0}$ is a generalized hypergeometric function and
$x^{\underline{k}}$ denotes the falling factorial.

Charlier polynomials satisfy the following orthogonality relation:

$\sum_{x=0}^{\infty}C_{n}^{(a)}(x)C_{m}^{(a)}(x)\frac{a^{x}e^{-a}}{x!}=\frac{n!}{a^{n}}\delta_{mn}$
(21)

where $\delta_{mn}$ is the Kronecker delta.

Table 1 presents a classification of the 18 polynomials surveyed in the study,
along with their respective parameter sets. The polynomials are grouped into
several categories based on their mathematical properties and areas of
application, such as orthogonal polynomials, hypergeometric polynomials,
q-polynomials, Fibonacci-related polynomials, combinatorial polynomials, and
number-theoretic polynomials. The parameter column lists the variables or
functions that define each polynomial, with some polynomials having no
additional parameters beyond their degree.

Table 1: Classification and parameters of the polynomials surveyed in the study Polynomial | Group | Parameters
---|---|---
Al-Salam-Carlitz | Orthogonal, Hypergeometric, q- |  $a$, $q$
Bannai-Ito | Orthogonal |  $\rho_{n}$, $\tau_{n}$
Askey-Wilson | Orthogonal, Hypergeometric, q- |  $a$, $b$, $c$, $d$, $q$
Boas-Buck | Orthogonal |  $f(t)$, $g(t)$
Boubaker | Orthogonal | -
Charlier | Orthogonal | $a$
Fermat | Number-theoretic | $n$
Gottlieb | Number-theoretic | -
Heptanacci | Fibonacci-related | -
Hexanacci | Fibonacci-related | -
Meixner-Pollaczek | Orthogonal, Hypergeometric |  $\lambda$, $\phi$
Narayana | Combinatorial | -
Octanacci | Fibonacci-related | -
Padé | Approximation |  $m$, $n$, $f_{k}$
Pentanacci | Fibonacci-related | -
Tetranacci | Fibonacci-related | -
Tribo | Fibonacci-related | -
Vieta-Pell | Combinatorial, Number-theoretic | -

##  3 Results

The table presented in Table 2 summarizes the performance metrics of various
KAN based models, each employing a different type of polynomial as the basis
function. These KAN models have been evaluated on the MNIST [24], for
handwritten digit classification, and the reported metrics (Overall Accuracy,
Kappa, and F1 Score) provide insights into their respective predictive
performances.

By examining the values in the table, the Gottlieb-KAN model exhibits the
highest overall accuracy of 0.9759, the highest Kappa of 0.9732, and the
highest F1 score of 0.9759, suggesting that it may be the most suitable choice
among the listed models for the given task.

Table 2: Performance Metrics of KAN Models Model | Overall Accuracy | Kappa | F1 Score
---|---|---|---
Fermat-KAN | 0.9619 | 0.9577 | 0.9619
AlSalam-Carlitz-KAN | 0.9675 | 0.9639 | 0.9675
BannaiIto-KAN | 0.9670 | 0.9633 | 0.9670
Boas-Buck-KAN | 0.9663 | 0.9625 | 0.9663
Boubaker-KAN | 0.9731 | 0.9701 | 0.9731
Charlier-KAN | 0.9726 | 0.9695 | 0.9726
Gottlieb-KAN | 0.9759 | 0.9732 | 0.9759
Heptanacci-KAN | 0.9426 | 0.9362 | 0.9426
Hexanacci-KAN | 0.9641 | 0.9601 | 0.9641
Meixner-Pollacze-KAN | 0.9703 | 0.9670 | 0.9703
Narayana-KAN | 0.9723 | 0.9692 | 0.9723
Octanacci-KAN | 0.9688 | 0.9653 | 0.9688
Pado-KAN | 0.9653 | 0.9614 | 0.9653
Pentanacci-KAN | 0.9542 | 0.9491 | 0.9542
Tetranacci-KAN | 0.9675 | 0.9639 | 0.9675
Tribo-KAN | 0.9569 | 0.9521 | 0.9569
Vieta-Pell-KAN | 0.9749 | 0.9721 | 0.9749
AskeyWilson-KAN | 0.9693 | 0.9659 | 0.9693

An in-depth analysis of the provided data reveals insights into the
relationships between model parameters, training time, and overall accuracy.
The Gottlieb-KAN model stands out with the highest number of parameters
(219,907) and the highest overall accuracy (0.9759), suggesting a potential
correlation between model complexity and performance. However, this
relationship is not consistent across all models, as some models with similar
parameter counts exhibit significant variations in training time and accuracy.
The Askey-KAN model, despite having the lowest number of parameters (105,871),
exhibits the longest training time (2,418.00 seconds), while the Heptanacci-
KAN model has the shortest training time (729.22 seconds). Factors such as
model architecture, optimization techniques, data characteristics, and
hardware resources likely contribute to these differences. Furthermore, the
overall accuracy values fall within a narrow range (0.9542 to 0.9759),
indicating comparable performance across models. While the analysis provides
valuable insights, a comprehensive understanding requires further examination
of model architectures, training data, optimization strategies, and relevant
performance metrics for the specific problem domain. Advanced analytical
techniques can quantify the relative importance of various factors and aid in
informed model selection and optimization decisions.

Table 3: Comparison of the complexity of the model Model | Number of Parameters | Training Time (s)
---|---|---
AlSalam | 105,862 | 927.30
BannaiIto | 105,865 | 1,261.58
Boas-Buck | 105,880 | 1,220.49
Boubaker | 105,856 | 807.36
Charlier | 105,856 | 853.90
Gottlieb | 219,907 | 923.96
Heptanacci | 105,856 | 729.22
Hexanacci | 105,856 | 760.49
Meixner | 105,862 | 1,095.85
Narayana | 105,856 | 773.70
Octanacci | 105,856 | 770.62
Pado | 105,856 | 764.13
Pentanacci | 105,856 | 762.46
Tetranacci | 105,856 | 767.92
Tribo | 105,856 | 751.97
Vieta-Pell | 105,856 | 792.53
Fermat | 105,856 | 757.21
Askey | 105,871 | 2,418.00

##  4 Conclusion

In this study, we conducted a comprehensive survey of 18 polynomials and their
potential applications as basis functions in Kolmogorov-Arnold Network (KAN)
models. The polynomials were classified into various groups based on their
mathematical properties, providing a structured overview of their
characteristics and areas of application. The performance of the KAN models
incorporating these polynomials was evaluated on the MNIST dataset for
handwritten digit classification, with the Gottlieb-KAN model achieving the
highest overall accuracy, Kappa, and F1 score. The analysis of the model
complexity, number of parameters, and training time revealed insights into the
relationships between these factors and the overall performance. However, the
narrow range of accuracy values suggests that further investigation and tuning
of these polynomials on more complex datasets are necessary to fully
understand their capabilities and potential in KAN models. This study serves
as an initial exploration of the use of various polynomials in KAN models and
highlights the need for further research in this area. Future work should
focus on applying these polynomials to a wider range of datasets with varying
complexity levels and investigating the impact of different model
architectures, optimization techniques, and hyperparameter settings on their
performance. Additionally, the development of more advanced analytical
techniques to quantify the relative importance of various factors in KAN model
performance could aid in informed model selection and optimization decisions.
By leveraging the insights gained from this study and conducting further
research, we can unlock the full potential of these polynomials in KAN models
and advance the field of machine learning.

## References

  * [1] Andrey Nikolaevich Kolmogorov and Vladimir Igorevich Arnold.  On the representation of continuous functions of several variables by superposition of continuous functions of a smaller number of variables.  Doklady Akademii Nauk SSSR, 114(5):953–956, 1957.
  * [2] Ziming Liu, Yixuan Wang, Sachin Vaidya, Fabian Ruehle, James Halverson, Marin Soljačić, Thomas Y. Hou, and Max Tegmark.  Kan: Kolmogorov-arnold networks.  arXiv preprint arXiv:2404.19756, 2024.
  * [3] Zavareh Bozorgasl and Hao Chen.  Wav-kan: Wavelet kolmogorov-arnold networks.  arXiv preprint arXiv:2405.12832, 2024.
  * [4] Moein E. Samadi, Younes M’́uller, and Andreas Schuppert.  Smooth kolmogorov arnold networks enabling structural knowledge representation.  arXiv preprint arXiv:2405.11318, 2024.
  * [5] Sidharth SS.  Chebyshev polynomial-based kolmogorov-arnold networks: An efficient architecture for nonlinear function approximation.  arXiv preprint arXiv:2405.07200, 2024.
  * [6] Waleed A Al-Salam and Leonard Carlitz.  Some orthogonal q-polynomials.  Mathematics of Computation, 19(92):618–627, 1965.
  * [7] Eiichi Bannai and Tatsuro Ito.  Algebraic combinatorics I: Association schemes.  Benjamin/Cummings, 1984.
  * [8] Richard Askey and James Wilson.  Some basic hypergeometric orthogonal polynomials that generalize Jacobi polynomials, volume 54.  American Mathematical Society, 1985.
  * [9] Ralph P Boas and Robert Creighton Buck.  Polynomial expansions of analytic functions.  Springer, 1958.
  * [10] Karem Boubaker.  The boubaker polynomials: A new polynomial family for solving bi-varied second-order differential equations.  Far East Journal of Mathematical Sciences, 31(2):299–320, 2007.
  * [11] Paulo Ribenboim.  Fermat’s Last Theorem for Amateurs.  Springer, 1995.
  * [12] Morris J Gottlieb.  Concerning some polynomials orthogonal on a finite or enumerable set of points.  American Journal of Mathematics, 60(2):453–458, 1938.
  * [13] Emrah Kiliç.  The heptanacci sequence and a generalization of the quintinomial coefficients.  Journal of Integer Sequences, 11(1):3, 2008.
  * [14] Emrah Kiliç.  The hexanacci sequence and some properties of the hexanacci polynomials.  Ars Combinatoria, 84:117–127, 2007.
  * [15] Felix Pollaczek.  Sur une famille de polynomes orthogonaux qui contient les polynomes d’hermite et de laguerre comme cas limites.  Comptes Rendus de l’Académie des Sciences Paris, 230:1563–1565, 1950.
  * [16] Tadepalli Venkata Narayana.  Lattice path combinatorics with statistical applications.  University of Toronto Press, 1979.
  * [17] Emrah Kiliç.  The octanacci sequence and some properties of the octanacci polynomials.  Ars Combinatoria, 93:131–141, 2009.
  * [18] George A Baker and Peter R Graves-Morris.  Padé approximants.  Cambridge University Press, 2 edition, 1996.
  * [19] Emrah Kiliç.  The pentanacci sequence and some properties of the pentanacci polynomials.  Ars Combinatoria, 88:153–162, 2008.
  * [20] Emrah Kiliç.  The tetranacci sequence and some properties of the tetranacci polynomials.  Ars Combinatoria, 85:13–22, 2007.
  * [21] Emrah Kiliç.  Combinatorial properties and recurrences for tribo polynomials.  Ars Combinatoria, 87:191–200, 2008.
  * [22] Roman Witula and Damian Slota.  On the vietapell polynomials.  Journal of Integer Sequences, 9(2):3, 2006.
  * [23] Carl Vilhelm Ludvig Charlier.  Über die darstellung willkürlicher funktionen.  Arkiv för Matematik, Astronomi och Fysik, 2(20):1–35, 1905.
  * [24] Yann LeCun, Corinna Cortes, and Christopher J.C. Burges.  The mnist database of handwritten digits, 1998.