11institutetext: School of Biomedical Engineering, ShanghaiTech University,
Shanghai 201210, China22institutetext: Shanghai Artificial Intelligence
Laboratory, Shanghai 200232, China33institutetext: Shanghai United Imaging
Intelligence Co., Ltd., Shanghai 200232, China44institutetext: Shanghai
Clinical Research and Trial Center, Shanghai, 201210, China

# A Progressive Single-Modality to Multi-Modality Classification Framework for
Alzheimer’s Disease Sub-type Diagnosis

Yuxiao Liu Co-first authors.11 Mianxin Liu††footnotemark:  22 Yuanwang Zhang
11 Kaicong Sun 11 Dinggang Shen 113344

###### Abstract

The current clinical diagnosis framework of Alzheimer’s disease (AD) involves
multiple modalities acquired from multiple diagnosis stages, each with
distinct usage and cost. Previous AD diagnosis research has predominantly
focused on how to directly fuse multiple modalities for an end-to-end one-
stage diagnosis, which practically requires a high cost in data acquisition.
Moreover, a significant part of these methods diagnose AD without considering
clinical guideline and cannot offer accurate sub-type diagnosis. In this
paper, by exploring inter-correlation among multiple modalities, we propose a
novel progressive AD sub-type diagnosis framework, aiming to give diagnosis
results based on easier-to-access modalities in earlier low-cost stages,
instead of modalities from all stages. Specifically, first, we design 1) a
text disentanglement network for better processing tabular data collected in
the initial stage, and 2) a modality fusion module for fusing multi-modality
features separately. Second, we align features from modalities acquired in
earlier low-cost stage(s) with later high-cost stage(s) to give accurate
diagnosis without actual modality acquisition in later-stage(s) for saving
cost. Furthermore, we follow the clinical guideline to align features at each
stage for achieving sub-type diagnosis. Third, we leverage a progressive
classifier that can progressively include additional acquired modalities (if
needed) for diagnosis, to achieve the balance between diagnosis cost and
diagnosis performance. We evaluate our proposed framework on large diverse
public and in-home datasets (8280 in total) and achieve superior performance
over state-of-the-art methods. Our codes will be released after the
acceptance.

###### Keywords:

Alzheimer’s disease Multi-modality fusion Contrastive learningExplanation
analysisMulti-stage framework

##  1 Introduction

Alzheimer’s disease (AD) poses a growing global health challenge, with an
escalating number of subjects affected by cognitive impairment. The total
healthcare costs for the treatment of AD are estimated as $305 billion in
2023, which brings a heavy burden to society [18]. Accurate diagnosis allows
effective treatment and hence alleviates AD-related burden. For AD diagnosis,
key modalities encompass cognitive assessment and plasma biomarkers (recorded
as tabular data), Magnetic Resonance Imaging (MRI), and Positron Emission
Tomography (PET). The tabular data is relatively easier to access and thus can
serve as initial screening data. For further diagnosis, MRI may be used to
assess essential information regarding brain atrophy. When needed, PET (that
can monitor the deposition of disease-related proteins in the brain) may be
used to provide more specific evidence for AD diagnosis. Based on these
modalities, deep learning (DL)-based methods usually can achieve higher
diagnosis performance than the traditional ones. However, there still exist
three major limitations in the existing DL-based methods.

First, many DL-based studies build models for an end-to-end one-stage
diagnosis using multiple modalities [14, 4]. This can significantly increase
the diagnosis cost because a considerable number of subjects can be firmly
diagnosed in the early stages without additional modalities acquired in later
stages. This practical challenge limits the application of these studies in
clinic. Second, many DL-based works use only imaging data. Although there has
emerged a series of works [14, 13] using relatively easier-to-access tabular
data for diagnosis, they simply input tabular data into the model without
considering semantic information of these tabular values. Some works [10, 12]
try to use pre-trained language encoder to encode textualized tabular data,
but they need to use detailed templates to textualize tabular data, which may
inadvertently introduce more common template information, instead of specific
tabular data information. Finally, previous methods directly classify the
state of health and general AD, rather than align modality data with AD sub-
types suggested by the clinical guideline [4, 6, 9]. Consequently, these
methods have difficulty in giving AD sub-type diagnosis results, and cannot
assist in precise AD treatment in clinic.

Therefore, in this paper, we propose a progressive AD diagnosis framework
(shown in Fig. 1) for cost-performance balance AD sub-type classification,
which can effectively address the above-mentioned limitations. We summarize
our contributions as follows:

  * •

We propose a multi-modality and multi-stage contrastive framework for AD sub-
type diagnosis, which can effectively align different modalities. This
alignment ensures feature(s) acquired in earlier-stage(s) to also carry
information from later stage(s), resulting in fewer diagnosis stages, reducing
diagnosis stages and costs.

  * •

To better exploit clinical tabular data, we employ a text disentanglement
network. This network extracts both common (from template) and specific
features (from tabular data) from textualized tabular data. It minimizes the
similarity of specific features to let them carry different aspects of subject
information. Moreover, we use a modality fusion module to fuse and enhance
feature representation across different stages (if needed) for better multi-
modality diagnosis performance.

  * •

We align features in each stage with AD sub-type criteria in the clinical
guideline. Moreover, based on the alignments, we develop a confidence-driven
multi-stage progressive diagnosis framework. The framework automatically
determines whether to give a final diagnosis based on already collected data
(without further acquisition to save cost) or to further acquire a more
advanced modality in the next stage.

  * •

Experimental results on large diverse public and in-home datasets (8280 in
total) show that our framework outperforms the other representative methods
significantly.

##  2 Methods

###  2.1 Notation & Problem Formulation

In alignment with real clinical procedure (often starting with easy-to-access
data and later adding more difficult/expensive-to-access data if needed), our
diagnostic process thus comprises $three\ distinct\ stages$, i.e., (Stage 1)
using only tabular data, (Stage 2) using both tabular data and MRI data, and
(Stage 3) using all the tabular data, MRI data, and PET data. Note that the
early stage has a larger number of subjects for training, while the later
stages always have less number of subjects for training (since most subjects
have been firmly diagnosed in the early stage(s)).

For each of the stages, a classifier for AD diagnosis is required. This
classifier generates two options based on the diagnosis confidence, $either$ a
confirmed decision $or$ a demand for next-stage modality acquisition. The
options rely solely on how these acquired modalities are matched with the
corresponding sub-type criteria in the guideline. In the final stage, the
model is compelled to generate the diagnosis decision, because there will be
no additional stages. Our comprehensive framework comprises the following
parts. Multi-modality Encoder: This part elucidates the details on multi-
modality feature extraction using the corresponding encoder across stages and
multi-modality fusion for diagnosis (Detailed in Section 2.2). Multi-modality
Alignment and Clinical Guideline Alignment: We show modality feature alignment
among different stages and also the alignment between the fused feature with
the corresponding sub-type criteria in the clinical guideline (Detailed in
Section 2.3). Progressive Classifier: This part describes progressive
diagnosis by referring to the diagnosis confidence (Detailed in Section 2.4).

Figure 1:  We propose a progressive multi-modality and multi-stage contrastive
framework for AD sub-type diagnosis. Starting from Stage 1, we acquire the
features from the textualized tabular data, and align them with the
corresponding sub-type criteria in the guideline. In cases where the diagnosis
confidence is low, we progressively utilize MR and PET images in Stages 2 and
3, respectively. To acquire modality information of the later stage(s) without
actual data acquisition, we further align the features from early stages with
those from the later stages.

###  2.2 Multi-modality Encoder

We have three encoders $\Phi^{1}$, $\Phi^{2}$, and $\Phi^{3}$ in three
corresponding stages for processing tabular data, MR images, and PET images,
respectively. At Stage $k$, we acquire the feature vector $f^{k}$ from the
corresponding modality. Specifically, in Stage 1, we use the BioMedCLIP [20]
text encoder as $\Phi^{1}$. For Stages 2 and 3, we train 3D image encoders
$\Phi^{2}$ and $\Phi^{3}$ from scratch, because the BioMedCLIP utilizes the 2D
image encoder which performs poorly for 3D brain image classification.

To better accommodate $\Phi^{1}$ to our task, we add an Adapter as shown in
Fig. 1. We first textualize different tabular data into corresponding texts,
including personal text $t_{p}$, healthy text $t_{h}$, and dementia text
$t_{d}$, following the same text template (with more details are given in Sec.
3.3.1). Then, we separately input them into $\Phi^{1}$. Each output from the
$\Phi^{1}$ is forwarded into the Adapter, which is a 4-layer multi-layer
perceptron (MLP), and mapped to the corresponding text feature vector
$f_{*}\in${$f_{p}$, $f_{h}$, $f_{d}$}.

Previous research [10] has demonstrated that a detailed text template can make
the language encoder perform better on downstream tasks. However, a highly
detailed template may inadvertently introduce more generic information from
the template itself rather than specific information from diverse tabular
data. This can make the language encoder only acquire similar features for
different textualized tabular data and thus hard to distinguish these features
[15]. To address this issue, we construct a text disentanglement network
$\mathcal{D}$ as formulated below.

####  2.2.1 Text disentanglement network.

The text feature vectors ($f_{p}$, $f_{h}$, $f_{d}$) output by $\Phi_{1}$ are
jointly forwarded into the disentanglement network $\mathcal{D}$, as shown in
Fig. 1. It aims to disentangle each feature vector
$f_{*}\in\\{f_{p},f_{h},f_{d}\\}$ into common a feature vector $f^{c}_{*}$
(from template) and specific feature vector $f^{s}_{*}$ (from tabular data)
[19, 2]. Specifically, we employ a common encoder $\Phi^{c}$ and a specific
encoder $\Phi^{s}$ to map $f_{*}$ into $f_{*}^{c}$ and $f_{*}^{s}$,
respectively. We minimize their mutual information (MI) with loss
$\mathcal{L}_{\mathcal{D}}^{MI}=\sum_{f_{*}\in\left\\{f_{p},f_{h},f_{d}\right\\}}MI\left(f^{c}_{*},f^{s}_{*}\right)$.
Besides, it is crucial to ensure features to exhibit similarity across
different text features. Conversely, the specific features carry different
aspects of subject information and should be orthogonal with each other.
Therefore, we maximize the similarity of common features and minimize the
similarity of specific features in all paired text features. Herein, we adopt
the orthogonal loss $\mathcal{L}_{\mathcal{D}}^{O}$ as formulated below,

$\mathcal{L}_{\mathcal{D}}^{O}=\frac{1}{2}\times\sum_{\begin{subarray}{c}f_{*}\in\\\
\left\\{f_{p},f_{h},f_{d}\right\\}\end{subarray}}\sum_{\begin{subarray}{c}f_{**}\in\\\
\\{f_{p},f_{h},f_{d}\\}\end{subarray}}-\cos\left(f_{*}^{c},f_{**}^{c}\right)+\left|\cos\left(f_{*}^{s},f_{**}^{s}\right)\right|.$
(1)

Finally, we concatenate disentangled features of each text feature, namely
$[f_{*}^{c},f_{*}^{s}]$, $f_{*}\in\\{f_{p},f_{h},f_{d}\\}$ as each output of
$\mathcal{D}$ to maximize the utilization of information from both the
template and tabular data. Subsequently, all the output features from
$\mathcal{D}$ are further concatenated to obtain features of the textualized
tabular data, feature $f^{1}$, at Stage 1.

####  2.2.2 Multi-modality fusion.

To adaptively fuse multi-modality features from multiple stages ($f^{1}$,
$f^{2}$, and $f^{3}$) for final diagnosis, we apply a transformer-based multi-
modality fusion module $\mathcal{F}$ as denoted in Fig. 1. The fusion module
$\mathcal{F}$ takes a transformer architecture to adaptively fuse $f^{1}$,
$f^{2}$, and $f^{3}$ into a fixed-length vector for the diagnosis.
Specifically, $\mathcal{F}$ first concatenates $k$ available feature(s) at
Stage $k$ into a vector $f^{1:k}=[f^{1},...f^{k}],k\leq 3$. Then, $f^{1:k}$ is
further concatenated with learnable query $Q$ as key and value, namely
$K=V=[f^{1:k},Q]$. Finally, the vectors $Q$, $K$, and $V$ are used to
calculate the attention $ATT$ and feed forward block $FFW$ [17] (an MLP with a
ReLU) to finally acquire $f_{r}^{k}$ as below:

$f_{r}^{k}=Q+FFW\left(Q+ATT(Q,K,V)\right).$ (2)

Based on the $ATT$ and the $FFW$, $f_{r}^{k}$ is mapped to the same dimension
as $Q$. Meanwhile, since our fusion module can process various lengths of
vectors, we can train our $\mathcal{F}$ using all the subjects we have.

###  2.3 Multi-modality Alignment and Clinical Guideline Alignment

####  2.3.1 Multi-modality alignment.

Since multiple modalities are acquired from the same subject, there often
exists underlying relevance among these modalities [11]. To enhance diagnosis
performance in earlier stages, we propose to let the early stage encoder(s)
entail modality information of the later stage(s) without actual acquisition.
Therefore, we attempt to align the modality features of all the stages. The
alignment loss $\mathcal{L}_{alig}$ is formulated below:

$\mathcal{L}_{alig}=\sum_{k=1}^{2}||f^{k}-f^{k+1}||_{2}.$ (3)

Based on the alignment of features in previous stages with the later stage
ones, the features of previous stages can embody particular information that
only the later stages can provide. This can improve the diagnosis performance
in the previous stages, and hence save diagnosis cost.

####  2.3.2 Clinical guideline alignment.

To achieve sub-type diagnosis with $f_{r}^{k}$ at Stage $k$, we align
$f_{r}^{k}$ with the sub-type criteria in the clinical guideline. To be
specific, we encode both the corresponding and non-corresponding sub-type
criteria of the subject, to make $f_{r}^{k}$ not only like its corresponding
criteria $Y$ but also different from the non-corresponding criteria $-^{Y}$.
We define the contrastive loss $\mathcal{L}_{con}^{k}$ as the negative
logarithm of the corresponding criteria contrastive score as expressed below:

$\mathcal{L}_{con}^{k}=-\log\frac{\exp((f_{r}^{k})\Phi^{1}(Y))}{\sum_{y\in^{-}Y}\exp((f_{r}^{k})\Phi^{1}(y)))+\exp((f_{r}^{k})\Phi^{1}(Y))}.$
(4)

During inference, we compare the contrastive score of all the AD sub-types and
choose the largest one as the diagnosis decision.

###  2.4 Progressive Classifier

In clinical practice, it is important to consider the cost of accurate
diagnosis. If the previous stage(s) can make confident diagnosis decisions
already, there will be no need for later stage(s). Therefore, our goal is to
minimize the total costs to the maximum extent without compromising
classification performance. To this end, inspired by [21], we define diagnosis
confidence $C^{k}$ in each Stage $k$ by subtracting the second highest
contrastive score from the highest one in clinical guideline alignment. This
subtracted value indicates how confident is the network about the diagnosis
decision. Then, we set a threshold $\theta$ on $C^{k}$. As we previously
mentioned in Section 1, there are two options for the classifier in Stage $k$.
If diagnosis confidence $C^{k}$ is larger than the threshold $\theta$, it
leads to the first option, i.e., to give the diagnosis result and stop.
Otherwise, a further stage is required. Inspired by [16], we encourage the
model to give confident diagnosis results in earlier stages. For each stage,
we set the loss $\mathcal{L}^{k}$ to avoid postponement by multiplying the
negative of the confidence by a penalty $\delta^{k}>0$.

$\mathcal{L}^{k}=\left\\{\begin{aligned}
&\mathcal{L}_{con}^{k}&C^{k}\geq\theta^{k}\text{ or
}k=3\cr&-C^{k}\delta^{k}&C^{k}<\theta^{k}\end{aligned}\right.$ (5)

##  3 Experiments

###  3.1 Dataset

We use data from four datasets including Alzheimer’s Disease Neuroimaging
Initiative (ADNI) [7, 1], Open Access Series of Imaging Studies (OASIS) [8],
National Alzheimer’s Coordinating Center (NACC) [3], and an in-house dataset
from Huashan Hospital. We follow the IWG-2 diagnosis guideline [5] and group
the subjects into four sub-types, including typical, atypical, pre-clinical
AD, and normal control. The corresponding sub-type criteria are detailed in
the supplementary material. We have totally 8280 subjects with tabular data,
4842 subjects with both tabular and MRI data, and 2336 subjects with all the
tabular, MRI, and PET data. We trained all models on the NACC and ADNI
datasets because these two datasets have the most complete data. Datasets are
randomly divided into 5 equal folds. 3 folds are used for training and
remaining 2 folds for validation and testing. Model is tested on subjects with
full modalities. We do not divide the dataset with overlapping subjects to
avoid the leakage. Note that although we use demetia information, this will
not cause the data leakage as IWG-2 [5] considers both dementia information
and in-vivo evidence for AD diagnosis instead of dementia information only.

###  3.2 Implementation Details

To ensure a fair comparison, we use 3D CNN as image encoders (as detailed in
supplementary material) for all comparison methods. We use the SGD optimizer
with a learning rate of $10^{-4}$. We train our model for 100 epochs. We
choose the MSE as the $\mathcal{L}_{alig}$. We set $\theta$ as 0.3, and
weights for all the loss functions are 1. $\delta^{1}$ and $\delta^{2}$ are
set to be 1 and 1.5, respectively. For text input $t_{p}$, we consider age,
education, and gender. For $t_{h}$, we consider healthy information, including
heart attack, hypertension, stroke, alcohol abuse, psychiatric disorder, and
the blood test. For $t_{d}$, we employ dementia level. As some subjects may
lack specific tabular information, we only input available textualized tabular
data to the text encoder.

###  3.3 Results and Discussions

We employ six distinct evaluation metrics to assess both diagnostic
performance and cost-effectiveness for subjects with full modalities (totally
837 subjects). The first four metrics are for sub-type diagnosis and the other
two are for cost-effectiveness. For estimating costs, we use the mean stage
number as a proxy. Finally, we compute the ratio of AUC to the estimated cost
to measure the cost-effectiveness.

Table 1: Quantitative comparison with other methods on our test dataset. Models | Acc | Spe | Sens | AUC | Cost | AUC/Cost Ratio
---|---|---|---|---|---|---
CoCoOp | 74.1 | 83.3 | 74.6 | 72.9 | 2.80 | 26.03
Xplainer | 75.3 | 85.6 | 73.2 | 75.5 | 2.53 | 29.84
Ours $w/o$ progressive classifier | 80.8 | 87.1 | 80.6 | 81.7 | 3 | 27.23
Ours $w/o$ multi-modality alignment | 76.5 | 86.8 | 76.6 | 75.3 | 2.45 | 30.73
Ours $w/o$ $\mathcal{D}$ | 74.5 | 83.6 | 71.6 | 73.8 | 2.42 | 30.50
Ours $w/o$ $\mathcal{F}$ | 75.8 | 84.9 | 76.2 | 73.7 | 2.52 | 29.24
Ours | 78.2 | 88.3 | 77.8 | 77.4 | 2.21 | 35.02

From Tab. 1, we can observe that our framework demonstrates a superior
classification performance over the representative methods (CoCoOp [22] and
Xplainer [12]), which are also built on multi-stage and multi-modality fusion.
These two comparison methods use different text encoders, but the same image
encoder as ours. Our performance improvement mainly arises from two reasons.
$First$, we use text disentanglement network $\mathcal{D}$ to extract more
information from different textualized tabular data. $Second$, we further
propose multi-modality alignment to let early-stage features entail later-
stage information, allowing the model to have improved diagnosis performance
in the early stages for reducing acquisition cost. The ablation studies on
$w/o$ $\mathcal{D}$ and multi-modality alignment further validate our
conclusion. Besides, we also show effectiveness of the progressive classifier.
We can see that, although there exists a certain performance drop in diagnosis
accuracy compared to the one without using the progressive classifier, it is
expected, since in earlier stages, we use fewer modalities for diagnosis.
However, the use of the progressive classifier achieves a substantial
reduction in diagnosis cost, and also improves the AUC/Cost ratio
significantly, implying that our model has great potential for practical
applications in clinical routine.

####  3.3.1 Text templates design.

In our evaluation, we test three types of text templates as outlined below (by
using age information in $t_{p}$ as an example). The first template (template
1) solely focuses on basic tabular information of the subject. For templates 2
and 3, we introduce more detailed descriptions. The corresponding performance
is depicted in Table 2. We can observe that, employing a more detailed text
template can enhance performance. Hence, we choose to utilize template 3 in
our method. Besides, we can see that the more detailed the template is, the
more contribution $\mathcal{D}$ can make to diagnosis performance. This
demonstrates that $\mathcal{D}$ can more effectively extract features from the
detailed text templates.

  * •

Template 1: {75 years old}

  * •

Template 2: {75 years old} subject

  * •

Template 3: {75 years old} subject for Alzheimer’s Disease diagnosis

Table 2: Evaluation on different text templates and text disentanglement
network $\mathcal{D}$.

| w/ $\mathcal{D}$ | w/o $\mathcal{D}$
---|---|---
Template | Acc | Spe | Sens | AUC | Cost | AUC/Cost Ratio | Acc | Spe | Sens | AUC | Cost | AUC/Cost Ratio
Template 1 | 70.7 | 78.3 | 71.9 | 71.5 | 2.39 | 29.91 | 69.8 | 77.7 | 70.5 | 71.2 | 2.46 | 28.94
Template 2 | 73.1 | 80.9 | 73.8 | 73.5 | 2.35 | 31.27 | 71.2 | 78.7 | 70.9 | 71.5 | 2.48 | 28.83
Template 3 | 78.2 | 88.3 | 77.8 | 77.4 | 2.21 | 35.02 | 74.5 | 83.6 | 71.6 | 73.8 | 2.42 | 30.50

####  3.3.2 Effect of threshold on confidence.

We delve deeper into exploring the impact of using varying confidence
thresholds in the progressive classifier. A relatively lower confidence
threshold encourages the model to make decisions in earlier stage(s) and
substantially decreases acquisition costs. However, it may also lead to
incorrect diagnosis results that cannot be corrected by subsequent stage(s).
On the contrary, opting for a higher confidence threshold can lead to more
accurate diagnosis results but cause increased diagnosis costs. In Table 3, we
present performance and cost ratio using different confidence thresholds. We
set the same threshold value in stages 1 and 2. As we can see, as the
threshold increases, diagnosis performance is also improved, but the
improvement ratio slows down. With regard to the AUC/Cost ratio, it increases
first and then decreases dramatically. This effect is reasonable since, when
the threshold becomes too large, most of the subjects are required to finish
later stages to obtain diagnosis decision. This will benefit diagnosis
correctness, but with sacrifice of the AUC/cost ratio since many subjects can
be firmly diagnosed in early stages without going to next expensive stage(s).

Table 3: Evaluation on different confidence thresholds. Threshold | Acc | Spe | Sens | AUC | Cost | AUC/Cost ratio
---|---|---|---|---|---|---
0.1 | 71.8 | 70.9 | 72.3 | 72.2 | 2.09 | 34.54
0.3 | 78.2 | 88.3 | 77.8 | 77.4 | 2.21 | 35.02
0.5 | 78.9 | 87.7 | 79.3 | 79.6 | 2.63 | 20.26

##  4 Conclusion

In this work, we present a multi-modal and multi-stage framework for cost-
effective AD sub-type diagnosis. Different from the existing works, our
framework is to progressively give diagnosis results from a single modality to
multiple modalities. Specifically, in the first stage, we utilize only tabular
data. To effectively extract textualized tabular information, we propose a
text disentanglement network to disentangle common and specific features of
different tabular data. If diagnosis confidence of the first stage is lower
than a pre-defined threshold, the respective subject requires later stages,
which integrates the MRI and PET data. To effectively fuse multiple
modalities, we propose a transformer-based multi-modality fusion module. To
improve diagnosis performance of early stages and reduce overall diagnosis
costs, we propose to align features of different stages to allow early stages
to contain particular information only available in later stage(s).
Furthermore, we match modality features and the AD diagnosis guideline to
achieve improved sub-type diagnosis. Extensive experiments demonstrate the
superiority of our framework, showing promise in real AD diagnosis.

## References

  * [1] Paul S. Aisen, Ronald C. Petersen, Michael Donohue, and Michael W. Weiner.  Alzheimer’s Disease Neuroimaging Initiative 2 Clinical Core: Progress and Plans, 2015.
  * [2] Vikash Balasubramanian, Ivan Kobyzev, Hareesh Bahuleyan, Ilya Shapiro, and Olga Vechtomova.  Polarized-vae: Proximity based disentangled representation learning for text generation.  arXiv preprint arXiv:2004.10809, 2020.
  * [3] Duane L Beekly, Erin M Ramos, William W Lee, Woodrow D Deitrich, Mary E Jacka, Joylee Wu, Janene L Hubbard, Thomas D Koepsell, John C Morris, Walter A Kukull, et al.  The national alzheimer’s coordinating center (nacc) database: the uniform data set.  Alzheimer Disease & Associated Disorders, 21(3):249–258, 2007.
  * [4] Danni Cheng and Manhua Liu.  Cnns based multi-modality classification for ad diagnosis.  In 2017 10th international congress on image and signal processing, biomedical engineering and informatics (CISP-BMEI), pages 1–5. IEEE, 2017.
  * [5] Bruno Dubois, Howard H Feldman, Claudia Jacova, Harald Hampel, José Luis Molinuevo, Kaj Blennow, Steven T DeKosky, Serge Gauthier, Dennis Selkoe, Randall Bateman, et al.  Advancing research diagnostic criteria for alzheimer’s disease: the iwg-2 criteria.  The Lancet Neurology, 13(6):614–629, 2014.
  * [6] Yechong Huang, Jiahang Xu, Yuncheng Zhou, Tong Tong, Xiahai Zhuang, and Alzheimer’s Disease Neuroimaging Initiative (ADNI).  Diagnosis of alzheimer’s disease via multi-modality 3d convolutional neural network.  Frontiers in neuroscience, 13:509, 2019.
  * [7] Clifford R. Jack.  Magnetic Resonance Imaging in Alzheimer’s Disease Neuroimaging Initiative 2, 2015.
  * [8] Pamela J LaMontagne, Tammie LS Benzinger, John C Morris, Sarah Keefe, Russ Hornbeck, Chengjie Xiong, Elizabeth Grant, Jason Hassenstab, Krista Moulder, Andrei G Vlassenko, et al.  Oasis-3: longitudinal neuroimaging, clinical, and cognitive dataset for normal aging and alzheimer disease.  MedRxiv, pages 2019–12, 2019.
  * [9] Feng Liu, Chong-Yaw Wee, Huafu Chen, and Dinggang Shen.  Inter-modality relationship constrained multi-modality multi-task feature selection for alzheimer’s disease and mild cognitive impairment identification.  NeuroImage, 84:466–475, 2014.
  * [10] Jie Liu, Yixiao Zhang, Jie-Neng Chen, Junfei Xiao, Yongyi Lu, Bennett A Landman, Yixuan Yuan, Alan Yuille, Yucheng Tang, and Zongwei Zhou.  Clip-driven universal model for organ segmentation and tumor detection.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 21152–21164, 2023.
  * [11] Yongsheng Pan, Mingxia Liu, Chunfeng Lian, Tao Zhou, Yong Xia, and Dinggang Shen.  Synthesizing missing pet from mri with cycle-consistent generative adversarial networks for alzheimer’s disease diagnosis.  In Medical Image Computing and Computer Assisted Intervention–MICCAI 2018: 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part III 11, pages 455–463. Springer, 2018.
  * [12] Chantal Pellegrini, Matthias Keicher, Ege Özsoy, Petra Jiraskova, Rickmer Braren, and Nassir Navab.  Xplainer: From x-ray observations to explainable zero-shot diagnosis.  arXiv preprint arXiv:2303.13391, 2023.
  * [13] Sebastian Pölsterl, Tom Nuno Wolf, and Christian Wachinger.  Combining 3d image and tabular data via the dynamic affine feature map transform.  In Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part V 24, pages 688–698. Springer, 2021.
  * [14] Shangran Qiu, Matthew I Miller, Prajakta S Joshi, Joyce C Lee, Chonghua Xue, Yunruo Ni, Yuwei Wang, Ileana De Anda-Duran, Phillip H Hwang, Justin A Cramer, et al.  Multimodal deep learning for alzheimer’s disease dementia assessment.  Nature communications, 13(1):3404, 2022.
  * [15] Constantin Seibold, Simon Reiß, M Saquib Sarfraz, Rainer Stiefelhagen, and Jens Kleesiek.  Breaking with fixed set pathology recognition through report-guided contrastive training.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 690–700. Springer, 2022.
  * [16] Kirill Trapeznikov, Venkatesh Saligrama, and David Castañón.  Multi-stage classifier design.  In Asian conference on machine learning, pages 459–474. PMLR, 2012.
  * [17] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [18] Winston Wong.  Economic burden of alzheimer disease and managed care considerations.  The American journal of managed care, 26(8 Suppl):S177–S183, 2020.
  * [19] Jingfeng Zhang, Haiwen Hong, Yin Zhang, Yao Wan, Ye Liu, and Yulei Sui.  Disentangled code representation learning for multiple programming languages.  Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, 2021.
  * [20] Sheng Zhang, Yanbo Xu, Naoto Usuyama, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, Cliff Wong, et al.  Large-scale domain-specific pretraining for biomedical vision-language processing.  arXiv preprint arXiv:2303.00915, 2023.
  * [21] Zefeng Zheng, Shaohua Teng, Naiqi Wu, Luyao Teng, Wei Zhang, and Lunke Fei.  Selected confidence sample labeling for domain adaptation.  Neurocomputing, 555:126624, 2023.
  * [22] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Conditional prompt learning for vision-language models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16816–16825, 2022.


# PoxVerifi: An Information Verification System to Combat Monkeypox
Misinformation

Akaash Kolluri
Computational Media Lab
The University of Texas at Austin
<EMAIL_ADDRESS>
Kami Vinton
School of Journalism and Media
The University of Texas at Austin
<EMAIL_ADDRESS>
Dhiraj Murthy
Computational Media Lab
School of Journalism and Media
The University of Texas at Austin
<EMAIL_ADDRESS>

###### Abstract

Following recent outbreaks, monkeypox-related misinformation continues to
rapidly spread online. This negatively impacts response strategies and
disproportionately harms LGBTQ+ communities in the short-term, and ultimately
undermines the overall effectiveness of public health responses. In an attempt
to combat monkeypox-related misinformation, we present PoxVerifi, an open-
source, extensible tool that provides a comprehensive approach to assessing
the accuracy of monkeypox related claims. Leveraging information from existing
fact checking sources and published World Health Organization (WHO)
information, we created an open-sourced corpus of 225 rated monkeypox claims.
Additionally, we trained an open-sourced BERT-based machine learning model for
specifically classifying monkeypox information, which achieved 96% cross-
validation accuracy. PoxVerifi is a Google Chrome browser extension designed
to empower users to navigate through monkeypox-related misinformation.
Specifically, PoxVerifi provides users with a comprehensive toolkit to assess
the veracity of headlines on any webpage across the Internet without having to
visit an external site. Users can view an automated accuracy review from our
trained machine learning model, a user-generated accuracy review based on
community-member votes, and have the ability to see similar, vetted, claims.
Besides PoxVerifi’s comprehensive approach to claim-testing, our platform
provides an efficient and accessible method to crowdsource accuracy ratings on
monkeypox related-claims, which can be aggregated to create new labeled
misinformation datasets.

_K_ eywords Monkeypox $\cdot$ Misinformation $\cdot$ Misinformation Detection
$\cdot$ Machine Learning $\cdot$ Chrome Extension $\cdot$ Crowdsourcing
$\cdot$ News Verification

##  1 Introduction

Monkeypox is a zoonotic virus related to the virus that causes smallpox. It
was first described in humans in 1970 [1, 2]. Though it has been traditionally
restricted to particular regional outbreaks, on July 23, 2022, the World
Health Organization (WHO) declared monkeypox a Public Health Emergency of
International Concern [3]. As of August 5, 2022, the US Centers for Disease
Control and Prevention (CDC) reported monkeypox had spread to 88 total
countries with 28,220 confirmed cases [4].

Shortly after monkeypox cases appeared across the globe [5], misinformation
related to monkeypox began circulating widely on the Internet and contributing
to inaccurate health information [6, 7]. Misinformation surrounding previous
monkeypox outbreaks has led to negative health outcomes (e.g., stigma which
increased delays in seeking treatment and suppressed reporting to public
health authorities which undermined education efforts and increased infection
risk [8]). The ability of misinformation to cause stigma [7, 8, 9, 10], as
occurred with COVID-19 misinformation[11], is also of particular concern
during monkeypox [12], which further undermines effective response strategies
[13]. Widely disseminated disinformation that claims monkeypox only affects
gay and bisexual men could further marginalize LGBTQ+ communities [14, 15].
Given the unprecedented speed and spread of health misinformation on social
media more broadly [16] and considering the harms caused by xenophobic,
inaccurate characterizations during the HIV/AIDS epidemic 1980s [17, 18, 19,
20], experts quickly recognized and warned the public about the dangers of
falsely believing that monkeypox only threatens sexually-active gay men [21].

Due to the continued dearth of available monkeypox misinformation tools
currently available, we developed PoxVerifi, a Google Chrome extension used to
help users verify the accuracy of monkeypox-related articles by 1) providing
automated accuracy reviews 2) displaying similar vetted claims to compare with
and 3) enabling users to vote on the accuracy of claims and see what others in
the community have voted. PoxVerifi is publicly available to download at no
cost on the Chrome
Store111chrome.google.com/webstore/detail/poxverifi/ddjpfhchdnofmaecejpmemaldpofmgoi.
Our code is available as open-source on
GitHub222github.com/computationalmedialab/PoxVerifi to enable future work to
utilize and expand the tool. A video demonstration of PoxVerifi, including
installation instructions and usage is available on
YouTube333youtu.be/cCu-4Divdbg.

###  1.1 Other Monkeypox-related Misinformation Studies and Tools

The emerging literature on monkeypox-related misinformation reveals worrying
trends that are reminiscent of both the COVID-19 pandemic and HIV epidemic.
For example, nearly half of the US public report that they have limited
knowledge about monkeypox [22], which exposes a collective uncertainty about
the disease. Uncertainty is generally an aversive state which motivates people
to seek information that reduces unknowns [23]. COVID-19-related uncertainty
rendered people more vulnerable to misinformation [24]. Indeed, the COVID-19
pandemic illustrated the harms of spreading disinformation at a time of high
collective public uncertainty. Many public health officials are alarmed by
xenophobic undertones which are driving current misinformation surrounding
monkeypox and undermining public health remedies [21].

Research has started to map the scale and impact of monkeypox misinformation
across various platforms. Most notably, Ortiz-Martínez et al. found that 52%
of monkeypox-related tweets contained misinformation or unverifiable
information and that these misleading tweets were more likely to receive more
replies, retweets, and likes than truthful tweets [25]. Other work found that
11.9% of highly viewed YouTube monkeypox-related videos were also
misinformative [26].

Informed by the potential of misinformation to adversely affect public health,
researchers have begun collecting publicly available monkeypox-related social
media content in an effort to preemptively counter the misinformation. For
example, Thakur [27] and Movahedi Nia et al. [28] built datasets of over
255,000 and 800,000 tweets related to 2022 monkeypox outbreaks respectively.
There are also emerging development projects on the software repository GitHub
with tools to specifically target monkeypox misinformation such as a
transformer model444huggingface.co/smcrone/monkeypox-misinformation  trained
on a labeled Twitter dataset555kaggle.com/datasets/stephencrone/monkeypox to
classify monkeypox misinformation [29]. This work represents a vanguard. As
such, there remains a dearth and further work in these areas is needed.

###  1.2 PoxVerifi Features and Data Availability

We combined strategies of past misinformation detection work [30, 31, 32] to
rapidly build a scalable tool to identify and curtail monkeypox misinformation
in circulation to mitigate harms to public health. Some of the key aspects and
features of PoxVerifi are detailed in sections 1.2.1-1.2.5.

####  1.2.1 PoxVerifi Misinformation Dataset

We created a dataset of labeled monkeypox information consisting of 170
factual claims and 55 false claims collected from existing fact-checking
repositories and World Health Organization articles on monkeypox. These data
are available as open-source on
Github.666github.com/computationalmedialab/PoxVerifi/tree/master/Datasets

####  1.2.2 Crowdsourced Voting

We extend Kolluri’s & Murthy’s [31] system of crowdsourced voting to
PoxVerifi. Specifically, PoxVerifi allows users to vote on whether or not they
think a claim is credible and can later see what other users have voted. We
further improve upon this feature by 1) allowing the ability to vote on claims
across multiple platforms on the Internet (e.g., news sources, blogs, etc.)
without leaving the page and 2) showing users similar, vetted claims to make
the most informed decisions possible.

####  1.2.3 Holistic User Experience

PoxVerifi does not require users to visit an external website in order to
verify claims. PoxVerifi combines user generated accuracy reviews, machine-
generated accuracy reviews, and the ability to view similar claims. This is
all done on a single pop-up that leads to no disturbances of a news viewing
experience.

####  1.2.4 Labeled, Scalable Monkeypox Claims Data

Because PoxVerifi allows users to provide an accuracy rating on articles,
PoxVerifi can be successfully scaled to crowdsource larger datasets of labeled
data, thus providing an open-sourced database (with claims labeled as true or
false), available to researchers facilitating textual or content analyses,
etc. on misinformation around monkeypox.

####  1.2.5 Bias Reduction

While individuals (including experts) routinely make errors in judgments [33,
p.83], PoxVerifi employed two approaches to reduce bias and increase the
accuracy of the pooled judgments of voters. First, users can access similarly
vetted claims, which can help users overcome heuristic biases by activating
deliberate cognitive processing of information. Deliberate cognitive
processing is better for making complex judgments because it is more
analytical and effortful [34, p.187]. Second, users can only see how other
individuals voted after they cast their ballot to avoid the bias of peer
pressure and voting as others did.

##  2 Software Tools

###  2.1 Frameworks and Hosting Tools

####  2.1.1 Front-end

The front-end of PoxVerifi was developed in React.js777reactjs.org, a
JavaScript library used traditionally to create websites. The front end does
not require a hosting platform because Chrome extensions are run locally on
users’ devices.

####  2.1.2 Back-end

The back-end of PoxVerifi is composed of three parts: a proprietary machine
learning model to detect misinformation, a fuzzy search algorithm for claim
matching, and a database to store user generated votes. The machine learning
model and fuzzy search algorithm were created with
Flask888flask.palletsprojects.com/en/2.2.x/—a Python-based framework to
develop API endpoints. The Flask API is hosted on a Oracle Cloud Compute
Instance. The database to store user votes is a Cloud Firestore hosted on
Google Firebase and allows for up to 1GiB of storage, 50,000 reads per day,
and 20,000 writes per day on a free plan.

###  2.2  Vetted Monkeypox-Related Information

When a user visits a website, PoxVerifi shows the user a set of vetted claims
which are textually similar to the headline of the website being viewed. To do
this, we created a database of accuracy-rated claims related specifically to
monkeypox. We used the Google Fact Check
API999toolbox.google.com/factcheck/apis, which aggregates claims from many
reliable fact-checking sources, including Politifact and FactCheck.org. Using
this API, we queried the term, “monkeypox,” and filtered out any non-English
claims. We limited our search to “monkeypox” only to prevent potential noise
that could come from more general terms such as “pox.” Additionally, the
Google Fact Check API employs a topic tagging function, which automatically
tags claims related to monkeypox, and, thus, our query returned all of the
tagged claims. Claims which had sentence-length ratings which were collapsed
to a single phrase representing the ratings so they could be succinctly
represented on the website (e.g., “There is no evidence this is true.
Monkeypox is caused by a virus.” was changed to “no evidence”). At the time of
creation, the resulting corpus contained 56 accuracy-rated claims that we were
able to extract from this process, likely due to a recent flurry of monkeypox
outbreaks. This dataset, however, had only a single claim labeled as true.

To expand our corpus to more true claims we scraped claims directly from
articles published by the WHO about monkeypox. We queried every article
published by the WHO using the term
“monkeypox,”101010who.int/home/search?indexCatalogue=genericsearchindex1&searchQuery=monkeypox
and then scraped the contents of each of the 90 articles returned. Each
article was then parsed into individual sentences by splitting the article
body by sentence-ending punctuation (e.g., “.”,”?”,”!”). Because not every
sentence contains a claim with a verifiable fact, we used the ClaimBuster API,
which assigns a score between 0 and 1 indicating the likelihood that the
sentence contains a verifiable claim [35]. All articles’ contents were fed
into ClaimBuster’s Claim Detection API. We only retained sentences with a
score over 0.8; Hassan et al. reported a threshold of 0.5 for “highly check-
worthy factual claims,” [35] but we raised our threshold to 0.8 to avoid
potential noise. Though any monkeypox-related text can contain errors, claims
published by the WHO are highly authoritative and the organization has an
established track record for authoritative health information on many diseases
(e.g, a previous monkeypox outbreak [36] and dengue fever [37]). As a result,
we added the resulting 169 claims to our corpus and labeled them as true. This
dataset is freely available as open-source for
others111111github.com/computationalmedialab/PoxVerifi/tree/master/Datasets.

###  2.3 Machine Learning Model

Various approaches to machine learning has been applied by others to identify
aspects of monkeypox-related information circulating on the Internet. For
example, Mohbey et al. [38] created a proprietary hybrid deep learning
approach in order to classify the sentiment of tweets as positive, negative,
or neutral. Sv and Ittamall [39] used a pre-trained sentiment analysis library
to find sentiment distribution of monkeypox-related tweets and used Latent
Dirichlet Allocation (LDA), an unsupervised generative probabilistic topic
modeling method first introduced by Blei et al. [40]. LDA is among the most
common topic modeling methods [41].

However, as the scope of these studies suggests, machine learning has not been
extensively applied to the specific task of identifying monkeypox-related
misinformation. Rather, machine learning has been more focused on the
diagnosis of the disease [42]. To address this gap in the literature, we
created our own approach to detect monkeypox-related misinformation.
Specifically, we trained a Bidirectional Encoder Representations from
Transformers (BERT)-based model on the data we collected to classify
monkeypox-related information. First, our vetted data were collapsed into two
categories: true and false. We converted the nominal labels of “no evidence”,
“false”, “inaccurate”, “mostly false”, “misleading”, “incorrect”, “half true”,
“not required”, “unsupported”, “needs context” were converted to just false,
and the remaining nominal label of “true” remained as true. Our dataset was
made up of claims where 170 were true and 55 were false. We trained a model
using the package DistilBERT, a smaller, faster version of BERT [43], and
extended and developed Alammar’s [44] Python notebook.

Since the size of our dataset was limited, we used 10-fold cross-validation—a
method that trains and tests 10 independent models on different splits of
train and test data and averages the accuracy together—to test our models’
accuracy to ensure that our accuracy was representative of our data. Our
trained BERT-based model achieved a high level of accuracy on the data set we
created: 96% 10-fold cross-validation accuracy. The codebook used to train the
model as well as the saved model parameters are available as open-source on
GitHub121212github.com/computationalmedialab/PoxVerifi/tree/master/BERT-Model-
Training.

##  3 PoxVerifi Extension Design

In order to rapidly create an accessible tool that could be used at scale to
combat monkeypox-related information, we opted to create a compact,
computationally light browser extension made widely available on the Google
Chrome Store as a no-cost download. PoxVerifi’s software is made up of three
components: 1) a front-end Google Chrome extension 2) a back-end database for
live vote storage and 3) a back-end Python framework to parse news sources,
compare headlines to previously vetted claims, and provide machine-learned
ratings.

Figure 1: PoxVerifi being used on a sample news article.

Figure 2: PoxVerifi display of a claim for review.

Figure 3: Sample accuracy display after expanding the “automated review”
section for a claim determined to be false.

Figure 4: Sample pop up after selecting community based-review (only available
after casting a vote).

Figure 5: Sample claim display after user expands the “similar vetted claims”
section.

###  3.1 Chrome Extension Interface

At any point during a browsing session, users can click on the PoxVerifi
extension icon in the top right corner of their browser. Once a user
encounters a claim on the Internet that they wish to verify, they click the
extension and a loading screen is displayed. While the loading screen is
visible, the URL of the current website, containing the monkeypox claim, is
sent to the backend headline_detection API method. If there is no headline
detected (e.g., there is only an image or it is the homepage of a browser),
the extension will instruct the user to view a news source. If a headline is
detected, it will be displayed along with the author of the article. PoxVerifi
then analyzes the accuracy of the displayed headline.

Below the headline section (see figure 2), is a section titled “Accuracy
Assessment" with three drop downs. When clicked, they display: 1) ML Model
Accuracy Estimates (see figure 3) 2) Other Users Crowdsourced votes (see
figure 4) and 3) Similar Vetted Claims (see figure 5). Finally, at the bottom
is a section enabling users to cast a vote.

####  3.1.1 ML Accuracy Estimate

The news headline is sent to the backend ml_classification API method which
returns a parameter representing the accuracy determined by our machine
learning model, which is displayed for the user to see.

####  3.1.2 Crowdsourced User Votes

A call is made to the Firebase database using the news URL as a key, and past
users’ votes are retrieved. This is displayed in a circular pie graph,
displaying red, yellow, and green sections corresponding to fake, mixed, and
truthful reviews (which labels appear on hover). Crowdsourced votes can only
be viewed after the user casts their own vote, to avoid the pull of conformity
influencing a user’s vote [45].

####  3.1.3 Similar Vetted Claims

The news headline is sent to the backend get_similiar_claims API method, which
returns a list of all vetted claims similar to the headline and their
corresponding review. Pagination allows users to click through to continue
viewing claims until they have enough context/have found a similar claim to
make an informed decision.

####  3.1.4 Vote Casting

Users can cast a vote to review the truth of the claim by clicking one of two
buttons: inaccurate or misleading news (red, frowning emoji) or accurate news
(green, happy face). After voting, a confirmation message is displayed
affirming their vote, with the opportunity to revoke the vote if the user made
a mistake. In order to prevent abuse of the vote casting system, each
installation of the Google Chrome extension is limited to only voting one time
at each unique URL (see the bottom of Figure 2 under “Give a Truth Review” for
a visual depiction).

###  3.2 Backend

Our Flask-based backend has three API endpoints, all accessible using an HTTP
GET request.

####  3.2.1 headline_detection

This function takes a parameter of a website URL. This website URL is used to
query the content of the webpage article. This is then parsed, and the
headline of the webpage is returned.

####  3.2.2 ml_classification

This function takes a parameter of a headline. The news headline first goes
through a preprocessing step to make sure that it has a claim for our model to
check. New headlines are fed into a machine learning model as described in
section 2.3, which produces a probability that can be coded as a binary
prediction. A nominal value of 0 or 1 is returned, where 0 represents a fake
determination and 1 represents a true determination.

####  3.2.3 get_similar_claims

This function takes a parameter of a headline. The Python package
fuzzysearch131313pypi.org/project/fuzzysearch/ is used to generate a similar
score (based on the number of approximately the same words) between the
parameter headline, and the headlines the other etted scraped claims scraped
as described in section 2.2. All claims which have a similarity score with the
parameter claim of over 50 (which we observed was a sufficient threshold to
yield topically similar sentences) are returned.

####  3.2.4 Database

PoxVerifi’s database, hosted on Cloud Firestore, is made up of documents each
identified by the URL of a news article. Each document record contains a fake,
neutral/mixed, and true vote count.

##  4 Discussion

PoxVerifi is built with an extensible framework to accurately and conveniently
verify the veracity of monkeypox-related claims. By integrating features of
other misinformation detection tools [30, 31, 32], PoxVerifi provides a
reliable methodology for testing claims. We successfully trained a deep
machine learning model on monkeypox-related information that achieves 96%
cross-validation accuracy on classifying news as likely true or likely false.

Our tool also allows users to access previously vetted claims to compare the
claim. This provides a reference point and the act of deliberating and
comparing claims engages active, deliberative processing. Only after voting on
the truth of a claim can users see how others voted. This reduces systemic
bias in the data by ensuring that each vote is made independent of the
existing group results. This is important in reducing the human tendency to
follow the crowd, which can render one’s judgment more vulnerable to errors
based on overweighting of partial or false evidence [46, p.89]. This will help
ensure that each vote is cast independently of others and is crucial to ensure
the accuracy of crowdsourced voting. The judgment errors by individuals that
contribute to group data tend to average to zero as long as there is no hidden
systemic biases (e.g., seeing how the majority voted before choosing and that
observations are independent and their errors are uncorrelated [33, p.84]).
PoxVerifi meets these conditions.

As a Google Chrome extension, PoxVerifi can be used by almost
anyone141414Anyone who has a computer and software that can run Chrome and
access the Chrome Extension Store.  with Internet access to rate claims and
view the ratings of claims. By using a variety of sources, PoxVerifi can help
mitigate monkeypox-related misinformation by providing a highly accurate,
easy-to-use, accessible tool to assess the veracity of almost any claim.

PoxVerifi also has the potential to rapidly and inexpensively yield
crowdsourced labeled datasets if it is well adopted. While user votes are not
always aligned with automatic models, cases where they do align could indicate
high confidence in labeling. As more people install PoxVerifi and rate
articles, these crowdsourced votes can be used to automatically create highly
accurate labels for new sources. This labeled data can be leveraged in future
work to study monkeypox-related misinformation.

PoxVerifi addresses a key concern about fact-checking aids. Namely, by giving
users access to previously-vetted claims, we reduced the potential for users
to rely solely on heuristic cognitive strategies tied to an implicit trust of
automatically-generated credibility labels. In a study of detecting
misinformative images [47, p.39], providing credibility labels was found to
reduce the amount of human deliberation. PoxVerifi is designed to engage
deeper processing of information in general, activating cognitive processes
that enable users to become more cognizant of misinformation. By showing users
similar claims that have been previously vetted and labeled, users can more
easily access the cognitive processes required to make evidence-based
comparisons when determining the accuracy of claims they encounter in news
stories. Previous work found that truth nudging resulted in a 3-fold increase
in users’ ability to discern truthful headlines relative to false headlines
[48]. PoxVerifi provides a framework to facilitate similar studies in the
context of monkeypox claims.

##  5 Limitations and Future Work

###  5.1 Limitations

A limitation of our work is partially due to the lack of accessible monkeypox-
related claims at the time we developed the PoxVerifi tool. Indeed, the dearth
of high-quality, labeled monkeypox-related information poses some inescapable
limitations. After exhaustively aggregating multiple sources, we were only
able to achieve a total of 225 pieces of content. Although we were still able
to develop a model achieving high accuracy (e.g., 96%), the representativeness
of the training data is somewhat limited. Moreover, to err on the side of
accuracy, we only used WHO content to ensure the highest level of veracity of
claims. However, future models could be improved by incorporating a variety of
highly credible sources to increase data representativeness. Yet, given the
rapidly changing nature of public health emergencies, this is a common
obstacle. However, as PoxVerifi accrues more data through crowdsourcing, it
will generate new data points that can be used in the future to train more
accurate machine learning models.

###  5.2 Future Work

Future work includes scaling PoxVerifi to be able to automatically collect new
information from existing fact-checking repositories in order to keep all
information up-to-date. Furthermore, while PoxVerifi’s machine learning model
does not currently continuously train as users label more data, future work
can be done to implement a continuous learning feature for our ML model based
on user-generated votes. Future work can also extend our PoxVerifi framework
to other use cases such as the seasonal flu, childhood diseases, and the
COVID-19 pandemic. These could be quickly adapted and scaled to mitigate
misinformation harms.

##  6 Conclusion

The circulation of monkeypox-related misinformation is perpetuating harmful
ideas that negatively impact response strategies. Misinformation about
monkeypox can marginalize LGBTQ+ communities and could exacerbate the severity
of the outbreak [13, 15, 17, 21]. It is therefore imperative to quickly
develop tools that can combat monkeypox-related misinformation and
disinformation. To help address this need, we introduced PoxVerifi, an open-
source Google Chrome extension that enables people to develop more accurate
monkeypox-related news consumption habits. PoxVerifi uses a comprehensive set
of checks, including automated reviews, community reviews, and the ability to
view similar claims to help people better understand the credibility of news
they are reading. As a Google Chrome extension, PoxVerifi does all of this
without requiring a user to visit external websites, which facilitates
relatively uninterrupted news reading experiences. PoxVerifi also provides an
open source framework to quickly obtain labeled misinformation data through
crowdsourcing. This enables future work to build upon the PoxVerifi codebase
to create more advanced crowdsourced information tools to tackle other types
of misinformation, health-related or otherwise.

## Acknowledgments

This work was supported by Good Systems, a research Grand Challenge at the
University of Texas at Austin. This work was supported in part by Oracle Cloud
credits and related resources provided by the Oracle for Research program. We
thank Nikhil Kolluri for providing suggestions of features to include in
PoxVerifi and River Terrell for providing manuscript feedback.

## References

  * [1] I.. Ladnyj, P. Ziegler and E. Kima  “A human infection caused by monkeypox virus in Basankusu Territory, Democratic Republic of the Congo”  In _Bulletin of the World Health Organization_ 46.5, 1972, pp. 593–597  URL: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2480792/
  * [2] John P. Thornhill et al.  “Monkeypox virus infection in humans across 16 Countries — April–June 2022”  In _New England Journal of Medicine_ 387.8, 2022  DOI: 10.1056/NEJMoa2207323
  * [3] World Health Organization  “WHO Director-General declares the ongoing monkeypox outbreak a Public Health Emergency of International Concern”, 2022  URL: https://www.who.int/europe/news/item/23-07-2022-who-director-general-declares-the-ongoing-monkeypox-outbreak-a-public-health-event-of-international-concern
  * [4] Centers Disease Control  “2022 Monkeypox Outbreak Global Map”  In _Centers for Disease Control and Prevention_ , 2022  URL: https://www.cdc.gov/poxvirus/monkeypox/response/2022/world-map.html
  * [5] Hans Kluge and Andrea Ammon  “Monkeypox in Europe and beyond – tackling a neglected disease together”  In _Eurosurveillance_ 27.24, 2022  DOI: 10.2807/1560-7917.ES.2022.27.24.2200482
  * [6] Benjamin Ryan  “Monkeypox misinformation spreading faster than the virus, experts say” NBC News, 2022  URL: https://www.nbcnews.com/health/health-news/monkeypox-spreads-experts-clear-misinformation-rcna42526
  * [7] Farah Ennab et al.  “Monkeypox outbreaks in 2022: Battling another “Pandemic” of misinformation”  In _International Journal of Public Health_ 67, 2022, pp. 1–2  DOI: 10.3389/ijph.2022.1605149
  * [8] Julii Brainard and Paul R. Hunter  “Misinformation making a disease outbreak worse: Outcomes compared for influenza, monkeypox, and norovirus”  In _Simulation_ 96.4, 2020, pp. 365–374  DOI: 10.1177/0037549719885021
  * [9] Mark L. Hatzenbuehler et al.  “Prospective associations between HIV-related stigma, transmission risk behaviors, and adverse mental health outcomes in men who have sex with men”  In _Annals of Behavioral Medicine_ 42.2, 2011, pp. 227–234  DOI: 10.1007/s12160-011-9275-z
  * [10] Gregory M. Herek, John P. Capitanio and Keith F. Widaman  “HIV-related stigma and knowledge in the United States: Prevalence and trends, 1991-1999”  In _American Journal of Public Health_ 92.3, 2002  URL: http://www.proquest.com/docview/215108121/abstract/CDDAE6DBD564D1FPQ/1
  * [11] Md Saiful Islam et al.  “COVID-19–related Infodemic and its impact on public health: A global social media analysis”  In _The American Journal of Tropical Medicine and Hygiene_ 103.4, 2020, pp. 1621–1629  DOI: 10.4269/ajtmh.20-0812
  * [12] Rhiannon Williams  “Homophobic misinformation is making it harder to contain the spread of monkeypox” MIT Technology Review  In _MIT Technology Review_ , 2022  URL: https://www.technologyreview.com/2022/06/17/1054408/homophobic-misinformation-spread-monkeypox-social-media/
  * [13] UNAIDS  “UNAIDS warns that stigmatizing language on Monkeypox jeopardises public health”, 2022  URL: https://www.unaids.org/en/resources/presscentre/pressreleaseandstatementarchive/2022/may/20220522_PR_Monkeypox
  * [14] Robert Hart  “Monkeypox misinformation is spreading, experts warn—Here’s what you need to know about the disease”, 2022  URL: https://www.forbes.com/sites/roberthart/2022/08/25/monkeypox-misinformation-is-spreading-experts-warn-heres-what-you-need-to-know-about-the-disease/
  * [15] Brooke Migdon  “Experts worry monkeypox disinformation will harm LGBTQ+ community” The Hill, 2022  URL: https://thehill.com/changing-america/respect/equality/3590636-experts-worry-monkeypox-disinformation-will-harm-lgbtq-community/
  * [16] Soroush Vosoughi, Deb Roy and Sinan Aral  “The spread of true and false news online”  In _Science_ 359.6380, 2018, pp. 1146–1151  DOI: 10.1126/science.aap9559
  * [17] Mark L. Hatzenbuehler, Jo C. Phelan and Bruce G. Link  “Stigma as a fundamental cause of population health inequalities”  In _American Journal of Public Health_ 103.5, 2013, pp. 813–821  DOI: 10.2105/AJPH.2012.301069
  * [18] World Health Organization Feature Stories  “Immunizing the public against misinformation”  In _WHO/Newsroom_ , 2020  URL: https://www.who.int/news-room/feature-stories/detail/immunizing-the-public-against-misinformation
  * [19] Stephan Lewandowsky et al.  “Misinformation and its correction: Continued influence and successful debiasing”  In _Psychological Science in the Public Interest_ 13.3, 2012, pp. 106–131  URL: http://www.jstor.org/stable/23484653
  * [20] Stephan Lewandowsky and Sander van der Linden  “Countering misinformation and fake news through inoculation and prebunking”  In _European Review of Social Psychology_ 0.0, 2021, pp. 1–38  DOI: 10.1080/10463283.2021.1876983
  * [21] Titanji Boghuma and Keletso Makofane  “Monkeypox is not a gay disease”  In _Speaking of Medicine and Health_ , 2022  URL: https://speakingofmedicine.plos.org/2022/05/19/monkeypox-is-not-a-gay-disease/
  * [22] Amyn A. Malik, Maike S. Winters and Saad B. Omer  “Attitudes of the US general public towards monkeypox”  medRxiv, 2022  DOI: 10.1101/2022.06.20.22276527
  * [23] Donna M Webster and Arie W Kruglanski  “Individual differences in need for cognitive closure”  In _Journal of Personality and Social Psychology_ 67.6, 1994, pp. 1049–1062  DOI: 10.1037//0022-3514.67.6.1049
  * [24] Emmett M. Larsen, Kayla R. Donaldson, Megan Liew and Aprajita Mohanty  “Conspiratorial thinking during COVID-19: The roles of paranoia, delusion-proneness, and intolerance of uncertainty”  In _Frontiers in Psychiatry_ 12, 2021  DOI: 10.3389/fpsyt.2021.698147
  * [25] Yeimer Ortiz-Martínez, Jheinner Sarmiento, D Katterine Bonilla-Aldana and Alfonso J Rodríguez-Morales  “Monkeypox goes viral: Measuring the misinformation outbreak on Twitter”  In _The Journal of Infection in Developing Countries_ 16.07, 2022, pp. 1218–1220  DOI: 10.3855/jidc.16907
  * [26] Yeimer Ortiz-Martínez et al.  “YouTube and 2022 monkeypox outbreak: Opportunities for awareness and infection control”  In _Journal of Hospital Infection_ 0.0, 2022  DOI: 10.1016/j.jhin.2022.07.007
  * [27] Nirmalya Thakur  “Monkeypox2022tweets: The first public Twitter dataset on the 2022 monkeypox outbreak”, 2022  DOI: 10.20944/preprints202206.0172.v3
  * [28] Zahra Movahedi Nia, Nicola Bragazzi, Jude Kong and Jianhong Wu  “A Twitter dataset for monkeypox outbreak in 2022”, 2022  DOI: 10.5061/DRYAD.ZPC866TBH
  * [29] Stephen Crone  “Monkeypox misinformation detector project” GitHub, 2022  URL: https://github.com/stephen-crone/monkeypox
  * [30] Bjarte Botnevik, Eirik Sakariassen and Vinay Setty  “BRENDA: Browser extension for fake news detection”  In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_ arXiv, 2020, pp. 2117–2120  DOI: 10.1145/3397271.3401396
  * [31] Nikhil L. Kolluri and Dhiraj Murthy  “CoVerifi: A COVID-19 news verification system”  In _Online Social Networks and Media_ 22, 2021  DOI: 10.1016/j.osnem.2021.100123
  * [32] Rohan Pandey et al.  “A machine learning application for raising WASH awareness in the times of COVID-19 pandemic”  arXiv, 2020  URL: http://arxiv.org/abs/2003.07074
  * [33] Daniel Kahneman  “Thinking, fast and slow”  New York: Farrar, StrausGiroux, 2013
  * [34] Jeanne Ellis Ormrod  “Human learning”  New York, NY: Pearson Education, Inc., 2018
  * [35] Naeemul Hassan, Fatma Arslan, Chengkai Li and Mark Tremayne  “Toward automated fact-checking: Detecting check-worthy factual claims by ClaimBuster”  In _Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , KDD ’17  New York, NY, USA: Association for Computing Machinery, 2017, pp. 1803–1812  DOI: 10.1145/3097983.3098131
  * [36] World Health Organization  “The World Health Organization’s monkeypox surveillance programme, 1980-86”  In _Wellcome History_ , 2011, pp. 1–24  URL: https://eprints.whiterose.ac.uk/64235/
  * [37] World Health Organization  “Dengue guidelines, for diagnosis, treatment, prevention and control”  World Health Organization, 2009
  * [38] Krishna Kumar Mohbey, Gaurav Meena, Sunil Kumar and K. Lokesh  “A CNN-LSTM-based hybrid deep learning approach to detect sentiment polarities on monkeypox tweets”  arXiv, 2022  DOI: 10.48550/arXiv.2208.12019
  * [39] Praveen Sv and Rajesh Ittamalla  “What concerns the general public the most about monkeypox virus? – A text analytics study based on Natural Language Processing (NLP)”  In _Travel Medicine and Infectious Disease_ 49, 2022, pp. 102404  DOI: 10.1016/j.tmaid.2022.102404
  * [40] David M Blei, Andrew Y. Ng and Michael I. Jordan  “Latent Dirichlet Allocation”  In _Journal of Machine Learning Research_ 3.Jan, 2003, pp. 993–1022
  * [41] Hamed Jelodar et al.  “Latent Dirichlet Allocation (LDA) and topic modeling: Models, applications, a survey”  In _Multimedia Tools and Applications_ 78.11, 2019, pp. 15169–15211  DOI: 10.1007/s11042-018-6894-4
  * [42] Joshua J. Tom and Nlerum P. Anebo  “A neuro-fussy based model for diagnosis of monkeypox diseases”, 2018  URL: http://repository.elizadeuniversity.edu.ng/jspui/handle/20.500.12398/733
  * [43] Victor Sanh, Lysandre Debut, Julien Chaumond and Thomas Wolf  “DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter”  arXiv, 2020  DOI: 10.48550/arXiv.1910.01108
  * [44] Jay Alammar  “A visual guide to using BERT for the first time”, 2019  URL: https://jalammar.github.io/a-visual-guide-to-using-bert-for-the-first-time/
  * [45] Daomeng Guo et al.  “Conformity feedback in an online review helpfulness evaluation task leads to less negative feedback-related negativity amplitudes and more positive P300 amplitudes.”  In _Journal of Neuroscience, Psychology, and Economics_ 12, 2019  DOI: 10.1037/npe0000102
  * [46] Robert B. Cialdini  “Influence: The psychology of persuasion”  New York, NY: Harper Business, 2006
  * [47] Gina Giotta  “Ways of seeing … what you want: Flexible visuality and image politics in the post-truth era”  In _Fake News: Understanding Media and Misinformation in the Digital Age_ Cambridge: The MIT Press, 2020, pp. 29–44  URL: https://direct.mit.edu/books/book/4625/chapter/212082/Ways-of-Seeing-What-You-Want-Flexible-Visuality
  * [48] Gordon Pennycook et al.  “Fighting COVID-19 misinformation on social media: Experimental evidence for a scalable accuracy-nudge intervention”  In _Psychological Science_ 31.7, 2020, pp. 770–780  DOI: 10.1177/0956797620939054