# Practical Statistical Considerations for the Clinical Validation of AI/ML-
enabled Medical Diagnostic Devices

Feiming Chen  Center for Devices and Radiological Health, Food and Drug
Administration, Silver Spring, MD, USA.  Corresponding author, E-mail:
<EMAIL_ADDRESS>Hong Laura Lu  Center for Devices and Radiological
Health, Food and Drug Administration, Silver Spring, MD, USA.  Arianna
Simonetti  Center for Devices and Radiological Health, Food and Drug
Administration, Silver Spring, MD, USA.

###### Abstract

Artificial Intelligence (AI) and Machine-Learning (ML) models have been
increasingly used in medical products, such as medical device software.
General considerations on the statistical aspects for the evaluation of AI/ML-
enabled medical diagnostic devices are discussed in this paper. We also
provide relevant academic references and note good practices in addressing
various statistical challenges in the clinical validation of AI/ML-enabled
medical devices in the context of their intended use.

Keywords: artificial intelligence, machine learning, AI model, medical device
software, diagnostic device, clinical validation.

##  1 Introduction

Latest advances in biomedical research have made available a new generation of
medical diagnostic devices based on the promising technology of Artificial
Intelligence/Machine Learning (AI/ML) that can be used to design advanced
modelling systems. Such models are trained on data relevant to specific tasks
by optimizing certain performance metrics, and are often embodied in software
or apps intended for medical purposes. Sometimes a software application
functions without being part of a hardware medical device (Cen, 2017; Int,
2017).

In the biomedical diagnostic field, AI/ML may be used to develop a software
algorithm for a medical intended use with the intent to assist medical experts
in making a more accurate and rapid diagnostic decision regarding the
classification, detection, and prediction of diseases. As part of the current
trend of automating traditional labor-intensive tasks, AI/ML technology as
applied in the biomedical field is exciting and has the potential to provide a
significant contribution to medical diagnostic systems, but it also presents
some limitations and new challenges in the statistical aspect of clinical
validation. The term "clinical validation" means the demonstration of
objective evidence that clinically meaningful outcomes of the medical device
software can be achieved predictively and reliably in the target population
according to the intended use of the device (Cen, 2017; Int, 2017).

While acknowledging that the choice of clinical validation methodology for a
particular technology is generally based on factors including its specific
intended use and technological characteristics, in this paper, we outline some
general statistical considerations in the clinical validation of AI/ML-enabled
medical devices. Section 2 briefly describes the development of an AI model.
Section 3 illustrates four basic aspects in the AI model validation. Section 4
includes additional considerations for model validation. Finally, a discussion
is provided in section 5.

##  2 Development of an AI Model

Statistical Learning Theory is a framework for AI/ML that borrows from the
field of statistics and functional analysis with the goal to generate insight
from data by using “supervised” or “unsupervised” modeling (Hastie et al.,
2009; James et al., 2013), where the term supervised/unsupervised indicates
the presence/absence of a true outcome during model development. A simple
example of a supervised learning model is a classic linear or logistic
regression. In the following section, we focus on supervised learning and use
"AI Model" as a shorthand umbrella term for modern predictive models or
algorithms using AI/ML technologies. An AI model is characterized by its need
to _learn_ from training data before it can be implemented to make
"intelligent" predictions on new, future data.

Clinical need, coupled with the availability of relevant training data, may
drive the development of an AI model $f(X)$, that aims to predict a target
(e.g., a true clinical outcome) $Y$ using a function $f$ on input $X$. The
form of $f$ (e.g., random forest, gradient boosting machine, deep learning
neural network) is often determined by model developers through experience or
tools like cross-validation and may incorporate important physiological
insights. A typical AI model, as applied to the diagnostic medical field,
often includes three parts:

  1. 1.

A quality control algorithm that drops cases deemed as low-quality

  2. 2.

A suite of models that extracts the data relationship

  3. 3.

A risk classification/score algorithm with either a diagnostic or prognostic
goal, often designed to satisfy significant clinical considerations

Once the form of an AI model has been determined and its parameters have been
estimated from a set of training data, algorithm developers may estimate a
_generalized_ performance (e.g., the model performance on out-of-sample data)
using statistical techniques such as cross-validation. This is called
_internal validation_. Since the performance is estimated from the same
training data, there is a tendency to arrive at an over-optimistic performance
estimate (Simon et al., 2003) that may not hold when the AI model is used in a
real-world situation. Two factors may account for such phenomenon:

  1. 1.

The training data informs the model development in many ways (e.g., cross-
validation or Bootstrap adjustment (Harrell Jr et al., 1996; Efron and
Tibshirani, 1997) are often used to tune hyperparameters or select models),
and it can be hard to get an unbiased performance estimation when the model is
already “richly” informed by the training data.

  2. 2.

The _generalizability_ of the model performance, i.e., the ability of the
model to produce accurate predictions in a new sample of patients, can be
difficult to account for during performance estimation.

Given the above reasons, the internal validation provides only a starting
point on performance estimation; and the _external validation_ (Moons et al.,
2015; Justice, 1999), as explained later in section 3.3, will provide a more
objective, unbiased estimate of the model performance.

##  3 Basic Statistical Considerations for the Validation of a Diagnostic
Device AI Model

At basic level, the validation for a diagnostic device AI model (simply
referred as an AI model in the following) is no different from how a
theoretical model is validated. In math and physics, scientists extract models
from studying the real-world data (e.g., hypothesize a law of gravity from
astronomic observations) and test their predictive powers in _new_ settings
(e.g., use it to discover a new planet). Similarly, in machine learning for
medical applications, we rely heavily on computers to extract models (e.g., a
black box deep learning model) from some real-world data and test their
predictive powers in _new_ datasets - ideally in a different geographical and
clinical setting. In this scenario, a key validation concern is the _strength
of the external evidence_ that is used to support the _generalizability_
(a.k.a. "external validity", an ability to provide accurate predictions in a
new sample of patients (Justice, 1999)) of an AI model performance. This is
discussed in section 3.3 below.

###  3.1 Four Aspects for the Validation of an AI Model

Four basic aspects for the validation of an AI model are as follows:

  1. 1.

Quality of the Training Data

  2. 2.

Quality of the Validation Data

  3. 3.

Model Precision (i.e., repeatability and reproducibility)

  4. 4.

Model Accuracy

For general reference, the readers can find some basic recommendations for AI-
enabled medical products in (Liu et al., 2020; Rivera et al., 2020), a
guideline for reporting diagnostic accuracy studies (STARD) (Cohen et al.,
2016), a checklist for predictive model validation (TRIPOD) (Moons et al.,
2015), and a helpful list of fundamental statistical considerations in the
context of biomarker test in (De et al., 2013). In section 4, additional
statistical considerations are provided.

###  3.2 Quality of the Training Data

One reason to assess the adequacy of the training data is to guard against the
_AI bias_ , a potential tendency to miss prediction target in systematic, but
often unforeseeable way, which has been well-documented (Hao, 2019). Although
the AI bias may be hard to eliminate completely, a good practice is to ensure
that the training data sufficiently represents the intended use (target)
population of a medical diagnostic device, often in terms of relevant disease
conditions (e.g., the full spectrum of disease severity) and demographics
(e.g., age, sex, race, ethnicity, geography), to avoid building bias into a
model. A descriptive statistical analysis on the various distributions of the
training data, as well as a subgroup analysis (Rosenkranz, 2020) on the
homogeneity of the model performance, can help check the alignment of the
training data with the target population and possibly prevent or reduce AI
bias.

Some common sense checks may also help improve the quality of the training
data. For example, in image recognition tasks, fatal confounding may occur
when all the diseased cases are imaged with the same instrument that is
identified by a label on the image. Such confounding in the training data, if
not identified and mitigated, may be opportunistically exploited by an AI
model. Thus, it is important that meaningful, unbiased, representative
training data is used so that the trained model is less likely to be
overfitted to the potential biases and idiosyncrasies of the training data and
is more likely to generalize and maintain performance with a new real-world
data.

###  3.3 Quality of the Validation Data

A common pitfall in the selection of validation/test data for validating an AI
model is that the test data resembles the training data "too well" (Altman and
Royston, 2000). For example, this can happen when a random splitting of a
procured database is used to generate the training and test data. Another
example is when both the training and test data are coming from the same
clinical site or health care system. In both cases, the performance of the AI
model could be overestimated since the model was trained on data similar to
the test data. In other words, the test data in both cases is lacking the
_uncontrollable variability_ that is expected in a real-world setting but
_unknown_ to the trained model.

A significant challenge is that both the training and test data are expected
to represent the intended use population of a device, with minimal overlap (in
terms of location and time) between test data and training data. This is
because the real validation goal is to check the _generalizability_ , also
called _external validity_ , of the model performance: whether a model trained
using a particular data source can maintain its claimed predictive performance
in a different real-world situation that the model has not seen before. A good
reference on generalizability is (Justice, 1999).

Given the above consideration and the validation goal, a good practice is to
perform an _external validation_ (Justice, 1999) where the test data are
_independent validation data_. Such test data could come from diverse health
care systems with diverse clinical practices in diverse geographies (multi-
site), an expectation that could also apply to the training data used in the
development phase of the model. Although ideally the training data would come
from a similarly diverse but _different_ source, an "overlap" may often exist
due to feasibility or cost constraints. In such case, alternative forms of
"transportability" (Justice, 1999) for validation may be considered.

###  3.4 Model Precision: Repeatability and Reproducibility

A precision study gauges the variability of a device output when making
repeated measurements on the same patient, either with the same operator and
product (_repeatability_), or with different operator and product
(_reproducibility_). More generally, repeatability is the closeness of
agreement of repeated measurements taken under the same conditions; and
reproducibility is the closeness of agreement of repeated measurements taken
under different, pre-specified conditions (Cli, 2022).

Key statistics to summarize the repeatability/reproducibility, based on a
variance component analysis using a model’s continuous metric (e.g., a
probability score), are the standard deviation (SD) and the percent
coefficient of variation (%CV). If the SD or %CV is too large, then additional
engineering improvement to the AI model may be appropriate. Improving the AI
model to reduce SD or %CV provides a low-cost way to improve product quality
and the success likelihood of a future pivotal clinical study. This is, in
part, because the clinical reference standard (i.e., the best available method
for establishing the presence of absence of the target condition (Cohen et
al., 2016)) does not need to be measured in a precision study. Depending on
the product, additional factors may be considered in the precision study. In
image classification tasks, an AI model may be sensitive to data perturbation
(e.g., image translation/rotation, light intensity change, random noise). Such
sensitivity could be abundant for an AI-enabled medical device software
running on a generic smartphone using its camera to capture measurement data
(e.g., skin lesion analyzers). Recently, an interesting type of data
perturbation that may flip the AI model result is documented in (Finlayson et
al., 2019), where the basic idea is that imperceptible data/image manipulation
may be designed to completely fail an AI model.

To ensure an AI model prediction is robust, algorithm developers often use
_data augmentation_ techniques to help increase the model robustness by adding
multiple variations of the same data to the training set during the model
development stage, in the expectation that a model so trained would withstand
"stress testing" during the model validation stage. In a sense, a basic type
of "stress test" is the precision study mentioned above, which can check a
model’s robustness with repetition of the operating procedure on the same
patient when feasible.

###  3.5 Model Accuracy

The key performance assessment of an AI model is its diagnostic accuracy,
which is evaluated in a pivotal performance study. Due to sampling variation,
the uncertainties of the accuracy estimates are typically quantified, usually
in the form of 95% confidence intervals. The study acceptance criteria can be
based on statistical inferences using hypothesis testing methods (e.g.,
comparing a lower/upper confidence limit to a pre-specified performance goal).
It is often very useful to have a comparator that can be tested and evaluated
on the same patient/data as the device. This comparator can be the clinician,
another device, or standard of care. The evaluation on the same patient/data
is key to mitigate differences in the task difficulty levels and disease
spectrum due to sampling variation.

Depending on the nature of the diagnostic output (i.e., binary, polychotomous,
or continuous), different evaluation metrics are possible.

  * •

For binary diagnostic output, sensitivity, specificity, positive/negative
predictive values (PPV/NPV), and positive/negative diagnostic likelihood
ratios (LR+/LR-) may be the preferred performance measures to be evaluated
(Cen, 2007).

  * •

For risk stratification output that classifies a patient into one of multiple
risk groups and that may often be found in prognostic models (Altman and
Royston, 2000; Altman et al., 2009), pre/post-test risks and diagnostic
likelihood ratios may be used.

  * •

For risk score output that evaluates a patient’s disease risk with a
continuous probability, calibration plot, ROC curve, and decision curve
analysis may be used (Steyerberg and Vergouwe, 2014) (also see section 4.3
below). In the context of biomarker evaluation, the predictiveness curve
analysis (Pepe et al., 2008a; Huang and Pepe, 2009; Huang et al., 2007) may be
used.

  * •

For continuous score, agreement study methods using scatter plot, Deming
regression (Deming, 1943), and Bland-Altman analysis (Bland and Altman, 1999)
are often used.

##  4 Other Statistical Considerations for the Validation of a Diagnostic
Device AI Model

###  4.1 Study Design Considerations

The objective evaluation of a diagnostic device typically involves an external
validation study (a.k.a. diagnostic performance study) to be performed with
the locked-down, fixed, market-ready version of the device, using subjects
completely different and separate from those used in the training of the AI
model and in all prior clinical investigations (De et al., 2013; Cen, 2013).
In the context of AI models, guiding principles have also been developed to
promote Good Machine Learning Practice (GMLP) (FDA, 2021).

A typical diagnostic performance study has a non-randomized single-arm
comparative study design that compares the subject device with either a
clinical reference standard or a comparator device. Note some SaMD’s are aids
to the clinicians (e.g., in radiology applications). In such case, instead of
a standalone performance study, a reader study (where the reader performs a
task with and without a SaMD) is often used to compare the performance with
the SaMD and the performance without the SaMD.

An appropriate study design and study conduct (Cen, 2013; Pepe et al., 2008b)
for an external validation study reduces _bias_ and builds confidence in the
study results. Here are some examples of good study design and study conduct
practices:

  * •

Study Design (Cen, 2013):

    * –

The test data (independent validation data) is representative of the intended
use population of the device.

    * –

Pre-specified clinical study protocol and statistical analysis plan: pre-
specification is used to avoid post-hoc analysis that may bias the performance
results.

    * –

A _prospective_ design, or if not feasible, a _prospective-retrospective_
design (De et al., 2013). Clinical data is best collected prospectively in the
intended use population according to the intended use of the device. A
prospective-retrospective design allows an existing data source to be sampled
as if in a prospective manner (e.g., sampling design is determined before
unmasking the data source).

    * –

An adequate _masking_ protocol, so that knowledge of one result (e.g.,
clinical reference standard) does not influence the other paired result (e.g.,
device result).

    * –

Demonstration of control of type I and type II errors: the former ensures an
observation of study success is not due to an outsized (e.g., more than 5%)
false positive error during statistical hypothesis testing; the latter ensures
reasonable study success likelihood (e.g., more than 80%) with adequate sample
size, assuming effectiveness of the device.

  * •

Study Conduct:

    * –

Algorithm developers are masked from the test data. This ensures no
information from the independent validation data would bias the algorithm
development. In the external validation study, interpreters of the device/test
outputs are masked from the reference results (ground truth) and vice versa.

    * –

"Freeze/lock" the algorithm before test data collection or unmasking. This
ensures the algorithm will not be tweaked opportunistically in light of the
independent validation data to bias the performance result.

###  4.2 Influence of Signal Quality Control Algorithm

An AI model is often preceded by a quality control algorithm that discards
"low" quality cases (e.g., images that are blurred or botched) from further
processing. However, such low-quality cases may sometimes _confound_ with
truly hard/difficult ones — an example of _missing not at random_ (MNAR),
which may lead to diagnostic performance that is improved in accuracy metrics
(e.g., sensitivity and specificity) but may also be more biased (e.g., in the
sense that more patients than warranted may not get any results due to
declared low-quality events).

For example, compare two hypothetical AI-enabled diagnostic devices (A and B)
using cellphone cameras for certain skin disease detection. Assume they use
the same diagnostic algorithms, except that A has a more aggressive quality
control (QC) algorithm than B in declaring low-quality cases. After excluding
those cases that fail the QC algorithm, it may not be surprising to observe
that A would have a better diagnostic performance than B, because many low-
quality images dropped by A but not by B may actually be good quality but
difficult cases which are not included in the performance assessment for A.

Thus, a good practice is to examine the influence of a QC algorithm by
checking the proportion of low-quality dropouts and the result of a
sensitivity analysis assuming a worst-case scenario (i.e., assuming the QC
failure cases are all difficult ones that the AI model fails to classify
successfully). Table 1 is a mock-up confusion table that illustrates a
potential evaluation of the impact of QC failures (labeled as “Ungradable”) on
diagnostic performance.

Table 1:  Evaluation of QC Failures in a Confusion Table Device | True | True | Post-test Risk | Likelihood
---|---|---|---|---
Result | Positive | Negative |  | Ratio
Positive | A | D | $\frac{A}{A+D}$ | $\frac{A/(A+B+C)}{D/(D+E+F)}$
Negative | B | E | $\frac{B}{B+E}$ | $\frac{B/(A+B+C)}{E/(D+E+F)}$
Ungradable | C | F | $\frac{C}{C+F}$ | $\frac{C/(A+B+C)}{F/(D+E+F)}$
Worst Case | Sensitivity | Specificity | Pre-test Risk | 1
Scenario | $=\frac{A}{A+B+C}$ | $=\frac{E}{D+E+F}$ | $=\frac{A+B+C}{A+B+C+D+E+F}$ |

###  4.3 Validation of AI Model with Risk Score Output

Certain clinical considerations (e.g., long-term trend monitoring) may justify
an AI model producing a continuous risk score that reflects the likelihood of
a physiological event (e.g., a cardiac or hemodynamic event in a time window)
and that may be continuously updated over time. The validation of such risk
score may call for a larger sample size than usual to reduce uncertainty in
the evaluation.

In general, two aspects of the accuracy evaluation are _calibration_ (i.e.,
how well does a predicted probability match the observed frequency in the
study population) and _discrimination_ (i.e., how well do the rankings of
predicted scores match those of observed frequencies in the study population)
(Justice, 1999). In addition, some evaluations for clinical usefulness such as
the decision curve analysis (Vickers and Elkin, 2006; Vickers et al., 2019;
Fitzgerald et al., 2015), relative utility curves (Baker et al., 2009),
standardized net benefits (Pepe et al., 2016; Marsh et al., 2020) may be
helpful.

In the model development stage, it is a good practice to check and improve the
model calibration (Chen et al., 2018). A calibration evaluation (Steyerberg
and Vergouwe, 2014; Van Calster et al., 2016) may involve the estimation of
calibration-in-the-large (i.e., the intercept that compares the mean of all
predicted risks with the mean observed risk), calibration-slope (reflecting
the extent of shrinkage), and calibration plot. A basic calibration analysis
can be done with a logistic regression of disease status on the logit of the
risk prediction. It is called the logistic recalibration method or Cox
calibration analysis (Miller et al., 1993; Calster and Steyerberg, 2014; Cox,
1958).

An AI model may be built using training data with _high_ disease prevalence
(e.g., 50/50 split between positive and negative cases). Thus, the prevalence
of the training data may be quite different from that of the validation data
(which would mirror the target population and may have very low disease
prevalence). This would cause a problem of calibration-in-the-large and could
be fixed with _prevalence scaling_ (Horsch et al., 2008), which uses Bayes
theorem to adjust the prediction probability in light of the realistic
prevalence expectation in a target population. Such issue illustrates the
challenge of validating risk scores, a validation that may call for a lot more
data to train and test.

A discrimination evaluation may involve the estimation of ROC curve, as well
as a _c_ index (Harrell Jr et al., 1984), which reflects the concordance
between prediction probabilities and actual outcomes and is equivalent to the
area under the curve for a ROC curve (ROC-AUC). The discriminatory power of
the model can be evaluated by comparing the $c$ index/ROC-AUC to a pre-
specified performance goal or the performance of a comparator (e.g., standard
of care).

Since a risk score may be subject to thresholding by clinicians at different
levels, it may be helpful to examine the diagnostic performance (e.g.,
sensitivity/specificity and their 95% confidence intervals) at potential
clinically relevant thresholds (e.g., every decile or a finer grid).

A clinical usefulness evaluation may involve a decision curve analysis or
other tools. A decision curve (Vickers and Elkin, 2006) visualizes a type of
risk-adjusted benefit (a.k.a. net benefit) at varying levels of risk
tolerance, where the risk tolerance (a.k.a. threshold probability) is between
zero (no tolerance for false negative) and one (no tolerance for false
positive), and the risk-adjusted benefit is quantified by the number of true
positives subtracted by the relative cost of false positives. This type of
analysis could facilitate comparison among different diagnostic alternatives
and inform population-level health policies. Note that sometimes an
appropriate net benefit analysis depends on whether a rule-in test (e.g., when
a disease needs to be confirmed and the treatment is harmful) or a rule-out
test (e.g., when a disease is not to be missed and the treatment is safe) may
be clinically needed.

To further understand how a risk score would influence the clinical decision,
it may be possible to compute a kind of diagnostic likelihood ratio based on
the continuous risk score, which can then be used to update a patient’s pre-
test disease probability to get a post-test disease probability using Bayes’
theorem (Sox et al., 2013). More research in this area (Ost, 2022) may be
helpful.

Since the validation of a risk score may call for a large sample size and
involves extensive work in evaluating calibration, discrimination, and
clinical decision making, a practical alternative is to predict the risk with
a few risk strata (i.e., risk stratification) instead of a continuous risk
score. A risk stratification output may often be derived by thresholding a
risk score at different levels during the model development. Risk thresholds
that determine the clinical action (or inaction) are sometimes available from
clinical guidelines (Pepe et al., 2016). Upon a satisfactory statistical
evaluation using post-test risks and diagnostic likelihood ratios across all
strata under an external validation study, risk stratification can often
provide better clarity to the clinical utility of an AI model than a risk
score.

###  4.4 Time-to-Event Outcomes in AI-enabled Diagnostic Device Validation
Data

An AI model may have a prognostic output that predicts whether a future event
may happen (e.g., likelihood of cardiac mortality in 10 years). Such a model
(e.g., Cox regression) may be derived by adding new innovative biomarkers to
baseline covariates (e.g., age, sex) and common clinical risk factors (e.g.,
smoking status, diabetes, hypertension). Validating the AI model may involve
collecting time-to-event outcomes along with missing data status (e.g., due to
patient withdrawal or loss to follow-up). Calibration and discrimination
analysis may be done via risk stratification (Royston and Altman, 2013). In
addition, the contribution of new biomarkers, in terms of the fraction of
prognostic information that is new information, may be quantified (Harrell,
2013). An example analysis is described in (Royston and Altman, 2013) and may
be applicable beyond the Cox model. In general, we may use the survival
analysis techniques in the following way:

  * •

Form risk groups from the prognostic output and compare their Kaplan-Meier
curves. Post-test risks across risk groups at any specific time (e.g., one
year after start) can be compared using corresponding Kaplan-Meier estimates
and their 95% confidence intervals.

  * •

Check calibration by comparing Kaplan-Meier estimates to apparent risk levels.
For example, a risk group with an average apparent risk level of 80% based on
the prognostic outputs therein is expected to roughly match its corresponding
Kaplan-Meier estimate.

  * •

Check the contribution of new biomarkers (e.g., in a Cox model or Andersen-
Gill model, the latter of which is an extension of the Cox proportional hazard
model suitable for analysis of recurrent events) with likelihood ratio test
(LRT) and visualizations (e.g., compare histograms of predicted probabilities
between the baseline and the full model) (Harrell, 2013).

###  4.5 Alternative to AI Black Box Models

An AI model may display a degree of opacity or may be a black box that defies
interpretation. An opaque or black box model typically involves a _higher_
level of validation evidence than _interpretable_ models, in terms of the
representativeness and the size of the training/validation data, partly
because a modern AI model such as a deep learning neural network can _shatter_
(i.e., memorize) any (small) training data, even if they are completely random
labels/noises (Zhang et al., 2021).

It is desirable that a simple and interpretable model could be used to
approximate or replace a black box model, and there is research suggesting
that an interpretable model built with advanced methods may have similar
predictive performance as a black box model (Rudin, 2019).

###  4.6 Improving Internal Validation Quality of AI Models

Algorithm developers often update estimated performance throughout iterative
improvement of an AI model before they decide to go into a new clinical trial.
A risk during the model development stage is that a model may be over-trained,
leading to an unrealistic performance expectation. It is possible to avoid
potential performance over-optimism with a high-quality internal validation
test data that would resemble an external validation test data and inform a
more realistic performance expectation. In addition, it may be possible to
avoid over-training a model, due to repeated use of the test data, by
considering a _privacy-preserving_ data analysis (Dwork et al., 2015; Gossmann
et al., 2021), which perturbs the test data with small random noise whenever
it is accessed for performance assessment. This would prevent a potentially
over-trained model from performing well on each new version of the randomly
perturbed test data. More research is needed here to inform better practice in
diagnostic device area.

##  5 Future Challenges

The field of AI-enabled diagnostic applications is expanding rapidly, which
also brings more statistical challenges regarding their clinical validations
(Friedrich et al., 2021). As an example, in the future, the intended use of an
AI-enabled diagnostic device may include a capability for continuous learning
(i.e., ongoing model training) from real-world data, which would presumably
improve its diagnostic accuracy over time to benefit users (Lee and Lee,
2020). However, the validation of such device may be very challenging. Another
challenging example is that the clinical reference standard may not be
available during the external validation study for an innovative AI model
(e.g., a prognostic biomarker for impending critical care need, intended for
continuous monitoring of patients in critical conditions).

Despite all the potential challenges for validating an AI/ML-enabled medical
diagnostic device, we hope readers of this article can get a better
understanding of various validation aspects, follow sound statistical
principles, apply reasonable statistical methods, and gain confidence as
practitioners over time.

##  6 Acknowledgments

The authors wish to acknowledge the following people whose comments led to a
substantially improved paper: Rajesh Nair, Felipe Aguel, Lilly Yue, Yunling
Xu, Nicholas Petrick, Gene Pennello, Matthew Diamond, Kathryn Drzewiecki.

##  7 Disclosure statement

The authors report there are no competing interests to declare.

##  8 Funding

The authors report there is no funding associated with the work featured in
this article.

## References

  * Altman and Royston (2000) Douglas G Altman and Patrick Royston.  What do we mean by validating a prognostic model?  _Statistics in medicine_ , 19(4):453–473, 2000\.
  * Altman et al. (2009) Douglas G Altman, Yvonne Vergouwe, Patrick Royston, and Karel GM Moons.  Prognosis and prognostic research: validating a prognostic model.  _Bmj_ , 338, 2009.
  * Baker et al. (2009) Stuart G Baker, Nancy R Cook, Andrew Vickers, and Barnett S Kramer.  Using relative utility curves to evaluate risk prediction.  _Journal of the Royal Statistical Society: Series A (Statistics in Society)_ , 172(4):729–748, 2009.
  * Bland and Altman (1999) J Martin Bland and Douglas G Altman.  Measuring agreement in method comparison studies.  _Statistical methods in medical research_ , 8(2):135–160, 1999.
  * Calster and Steyerberg (2014) Ben Van Calster and Ewout W Steyerberg.  Calibration of prognostic risk scores.  _Wiley StatsRef: Statistics Reference Online_ , pages 1–10, 2014\.
  * Cen (2007) _Statistical Guidance on Reporting Results from Studies Evaluating Diagnostic Tests_.  Center for Devices and Radiological Health, US Food and Drug Administration, final edition, March 2007.
  * Cen (2013) _Design Considerations for Pivotal Clinical Investigations for Medical Devices_.  Center for Devices and Radiological Health, US Food and Drug Administration, final edition, November 2013.
  * Cen (2017) _Software as a Medical Device (SAMD): Clinical Evaluation_.  Center for Devices and Radiological Health, US Food and Drug Administration, December 2017.
  * Chen et al. (2018) Weijie Chen, Berkman Sahiner, Frank Samuelson, Aria Pezeshk, and Nicholas Petrick.  Calibration of medical diagnostic classifier scores to the probability of disease.  _Statistical methods in medical research_ , 27(5):1394–1409, 2018.
  * Cli (2022) _Harmonized Terminology Database_.  Clinical and Laboratory Standards Institute (CLSI), 2022.  https://htd.clsi.org.
  * Cohen et al. (2016) Jérémie F Cohen, Daniël A Korevaar, Douglas G Altman, David E Bruns, Constantine A Gatsonis, Lotty Hooft, Les Irwig, Deborah Levine, Johannes B Reitsma, Henrica CW De Vet, et al.  Stard 2015 guidelines for reporting diagnostic accuracy studies: explanation and elaboration.  _BMJ open_ , 6(11):e012799, 2016.
  * Cox (1958) David R Cox.  Two further applications of a model for binary regression.  _Biometrika_ , 45(3/4):562–565, 1958.
  * De et al. (2013) Arkendra De, Kristen Meier, Rong Tang, Meijuan Li, Thomas Gwise, Shanti Gomatam, and Gene Pennello.  Evaluation of heart failure biomarker tests: a survey of statistical considerations.  _Journal of cardiovascular translational research_ , 6(4):449–457, 2013.
  * Deming (1943) William Edwards Deming.  _Statistical Adjustment of Data_.  wiley, 1943.
  * Dwork et al. (2015) Cynthia Dwork, Vitaly Feldman, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Aaron Roth.  The reusable holdout: Preserving validity in adaptive data analysis.  _Science_ , 349(6248):636–638, 2015.
  * Efron and Tibshirani (1997) Bradley Efron and Robert Tibshirani.  Improvements on cross-validation: the 632+ bootstrap method.  _Journal of the American Statistical Association_ , 92(438):548–560, 1997.
  * FDA (2021) _Good Machine Learning Practice for Medical Device Development:Guiding Principles_.  FDA, Health Canada, MHRA, October 2021.
  * Finlayson et al. (2019) Samuel G Finlayson, John D Bowers, Joichi Ito, Jonathan L Zittrain, Andrew L Beam, and Isaac S Kohane.  Adversarial attacks on medical machine learning.  _Science_ , 363(6433):1287–1289, 2019.
  * Fitzgerald et al. (2015) Mark Fitzgerald, Benjamin R Saville, and Roger J Lewis.  Decision curve analysis.  _Jama_ , 313(4):409–410, 2015.
  * Friedrich et al. (2021) Sarah Friedrich, Gerd Antes, Sigrid Behr, Harald Binder, Werner Brannath, Florian Dumpert, Katja Ickstadt, Hans A Kestler, Johannes Lederer, Heinz Leitgöb, et al.  Is there a role for statistics in artificial intelligence?  _Advances in Data Analysis and Classification_ , pages 1–24, 2021\.
  * Gossmann et al. (2021) Alexej Gossmann, Aria Pezeshk, Yu-Ping Wang, and Berkman Sahiner.  Test data reuse for the evaluation of continuously evolving classification algorithms using the area under the receiver operating characteristic curve.  _SIAM Journal on Mathematics of Data Science_ , 3(2):692–714, 2021.
  * Hao (2019) Karen Hao.  This is how ai bias really happens–and why it’s so hard to fix.  _MIT Technology Review_ , 2019.
  * Harrell (2013) Frank Harrell.  _Statistically Efficient Ways to Quantify Added Predictive Value of New Measurements_ , 2013.  https://www.fharrell.com/post/addvalue.
  * Harrell Jr et al. (1984) Frank E Harrell Jr, Kerry L Lee, Robert M Califf, David B Pryor, and Robert A Rosati.  Regression modelling strategies for improved prognostic prediction.  _Statistics in medicine_ , 3(2):143–152, 1984\.
  * Harrell Jr et al. (1996) Frank E Harrell Jr, Kerry L Lee, and Daniel B Mark.  Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors.  _Statistics in medicine_ , 15(4):361–387, 1996\.
  * Hastie et al. (2009) Trevor Hastie, Robert Tibshirani, and Jerome Friedman.  The elements of statistical learning.  _Springer Series in Statistics_ , 2009.  ISSN 2197-568X.  doi: 10.1007/978-0-387-84858-7.  URL http://dx.doi.org/10.1007/978-0-387-84858-7.
  * Horsch et al. (2008) Karla Horsch, Maryellen L Giger, and Charles E Metz.  Prevalence scaling: Applications to an intelligent workstation for the diagnosis of breast cancer.  _Academic radiology_ , 15(11):1446–1457, 2008\.
  * Huang and Pepe (2009) Ying Huang and MS Pepe.  A parametric roc model-based approach for evaluating the predictiveness of continuous markers in case–control studies.  _Biometrics_ , 65(4):1133–1144, 2009.
  * Huang et al. (2007) Ying Huang, Margaret Sullivan Pepe, and Ziding Feng.  Evaluating the predictiveness of a continuous marker.  _Biometrics_ , 63(4):1181–1188, 2007.
  * Int (2017) _Software as a Medical Device (SaMD): Clinical Evaluation_.  International Medical Device Regulators Forum, September 2017.
  * James et al. (2013) Gareth James, Daniela Witten, Trevor Hastie, and Robert Tibshirani.  An introduction to statistical learning.  _Springer Texts in Statistics_ , 2013.  ISSN 2197-4136.  doi: 10.1007/978-1-4614-7138-7.  URL http://dx.doi.org/10.1007/978-1-4614-7138-7.
  * Justice (1999) Amy C. Justice.  Assessing the generalizability of prognostic information.  _Annals of Internal Medicine_ , 130(6):515, 1999\.  doi: 10.7326/0003-4819-130-6-199903160-00016.  URL https://doi.org/10.7326/0003-4819-130-6-199903160-00016.
  * Lee and Lee (2020) Cecilia S Lee and Aaron Y Lee.  Clinical applications of continual learning machine learning.  _The Lancet Digital Health_ , 2(6):e279–e281, 2020.
  * Liu et al. (2020) Xiaoxuan Liu, Samantha Cruz Rivera, David Moher, Melanie J Calvert, and Alastair K Denniston.  Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the consort-ai extension.  _bmj_ , 370, 2020.
  * Marsh et al. (2020) Tracey L Marsh, Holly Janes, and Margaret S Pepe.  Statistical inference for net benefit measures in biomarker validation studies.  _Biometrics_ , 76(3):843–852, 2020.
  * Miller et al. (1993) Michael E Miller, Carl D Langefeld, William M Tierney, Siu L Hui, and Clement J McDonald.  Validation of probabilistic predictions.  _Medical Decision Making_ , 13(1):49–57, 1993\.
  * Moons et al. (2015) Karel G.M. Moons, Douglas G. Altman, Johannes B. Reitsma, John P.A. Ioannidis, Petra Macaskill, Ewout W. Steyerberg, Andrew J. Vickers, David F. Ransohoff, and Gary S. Collins.  Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (tripod): Explanation and elaboration.  _Annals of Internal Medicine_ , 162(1):W1–W73, 2015.  doi: 10.7326/m14-0698.  URL https://doi.org/10.7326/m14-0698.
  * Ost (2022) David E Ost.  Interpretation and application of the likelihood ratio to clinical practice in thoracic oncology.  _Journal of Bronchology & Interventional Pulmonology_, 29(1):62–70, 2022.
  * Pepe et al. (2008a) Margaret S Pepe, Ziding Feng, Ying Huang, Gary Longton, Ross Prentice, Ian M Thompson, and Yingye Zheng.  Integrating the predictiveness of a marker with its performance as a classifier.  _American journal of epidemiology_ , 167(3):362–368, 2008a.
  * Pepe et al. (2008b) Margaret S Pepe, Ziding Feng, Holly Janes, Patrick M Bossuyt, and John D Potter.  Pivotal evaluation of the accuracy of a biomarker used for classification or prediction: standards for study design.  _Journal of the National Cancer Institute_ , 100(20):1432–1438, 2008b.
  * Pepe et al. (2016) Margaret S Pepe, Holly Janes, Christopher I Li, Patrick M Bossuyt, Ziding Feng, and Jørgen Hilden.  Early-phase studies of biomarkers: what target sensitivity and specificity values might confer clinical utility?  _Clinical chemistry_ , 62(5):737–742, 2016.
  * Rivera et al. (2020) Samantha Cruz Rivera, Xiaoxuan Liu, An-Wen Chan, Alastair K Denniston, and Melanie J Calvert.  Guidelines for clinical trial protocols for interventions involving artificial intelligence: the spirit-ai extension.  _bmj_ , 370, 2020.
  * Rosenkranz (2020) Gerd Rosenkranz.  _Exploratory subgroup analyses in clinical research_.  John Wiley & Sons, 2020.
  * Royston and Altman (2013) Patrick Royston and Douglas G Altman.  External validation of a cox prognostic model: principles and methods.  _BMC medical research methodology_ , 13(1):1–15, 2013.
  * Rudin (2019) Cynthia Rudin.  Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead.  _Nature Machine Intelligence_ , 1(5):206–215, 2019.
  * Simon et al. (2003) Richard Simon, Michael D Radmacher, Kevin Dobbin, and Lisa M McShane.  Pitfalls in the use of dna microarray data for diagnostic and prognostic classification.  _Journal of the National Cancer Institute_ , 95(1):14–18, 2003.
  * Sox et al. (2013) H.C. Sox, M.C. Higgins, and D.K. Owens.  _Medical Decision Making_.  Wiley, 2013.  ISBN 9780470658666.  URL https://books.google.com/books?id=WRBszgAACAAJ.
  * Steyerberg and Vergouwe (2014) Ewout W Steyerberg and Yvonne Vergouwe.  Towards better clinical prediction models: seven steps for development and an abcd for validation.  _European heart journal_ , 35(29):1925–1931, 2014\.
  * Van Calster et al. (2016) Ben Van Calster, Daan Nieboer, Yvonne Vergouwe, Bavo De Cock, Michael J Pencina, and Ewout W Steyerberg.  A calibration hierarchy for risk models was defined: from utopia to empirical data.  _Journal of clinical epidemiology_ , 74:167–176, 2016.
  * Vickers and Elkin (2006) Andrew J Vickers and Elena B Elkin.  Decision curve analysis: a novel method for evaluating prediction models.  _Medical Decision Making_ , 26(6):565–574, 2006\.
  * Vickers et al. (2019) Andrew J Vickers, Ben van Calster, and Ewout W Steyerberg.  A simple, step-by-step guide to interpreting decision curve analysis.  _Diagnostic and prognostic research_ , 3(1):1–8, 2019.
  * Zhang et al. (2021) Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals.  Understanding deep learning (still) requires rethinking generalization.  _Communications of the ACM_ , 64(3):107–115, 2021\.


# Personalized Jargon Identification for Enhanced Interdisciplinary
Communication

Yue Guo1,∗ Joseph Chee Chang2 Maria Antoniak2
Erin Bransom2 Trevor Cohen1 Lucy Lu Wang1,2 Tal August2
1University of Washington 2Allen Institute for AI
{yguo50, cohenta<EMAIL_ADDRESS>
{josephc, mariaa, erinbransom<EMAIL_ADDRESS>

###### Abstract

Scientific jargon can impede researchers when they read materials from other
domains. Current methods of jargon identification mainly use corpus-level
familiarity indicators (e.g., Simple Wikipedia represents plain language).
However, researchers’ familiarity of a term can vary greatly based on their
own background. We collect a dataset of over 10K term familiarity annotations
from 11 computer science researchers for terms drawn from 100 paper abstracts.
Analysis of this data reveals that jargon familiarity and information needs
vary widely across annotators, even within the same sub-domain (e.g., NLP). We
investigate features representing individual, sub-domain, and domain knowledge
to predict individual jargon familiarity. We compare supervised and prompt-
based approaches, finding that prompt-based methods including personal
publications yields the highest accuracy, though zero-shot prompting provides
a strong baseline. This research offers insight into features and methods to
integrate personal data into scientific jargon identification.

**footnotetext: Work performed during internship at AI2.

##  1 Introduction

An important challenge to communicating knowledge across scientific domains is
aligning on a shared vocabulary (Strober, 2006). Each scientific domain has
unique terminology that optimizes communication within the field but can pose
a barrier to researchers in other domains (Lucy et al., 2022; Choi and Pak,
2007). As science becomes more specialized, so too does its terminology
(Barnett and Doubleday, 2020; Plaven-Sigray et al., 2017), raising the barrier
of learning and collaborating across disciplines. We envision systems that can
identify whether specialized terminology will be unfamiliar to an individual
scholar.

Figure 1: An annotated term from our dataset, with annotations by computer
science researchers. Despite sharing a common domain, these researchers
exhibit variation in their familiarity and additional information needs about
the term within the abstract. Abstract from Liu et al. (2014).

NLP techniques have been developed to identify and simplify scholarly jargon
(Gardner and Davies, 2013; Tanaka-Ishii and Terada, 2011; Guo et al., 2022,
2021), a first step in our envisioned setting. The majority of these
techniques use a corpus of documents as a proxy for what a reader knows (e.g.,
Wikipedia contains words known to a general audience). However, an
individual’s specific background knowledge also plays a role in determining
their familiarity with a word (Gooding and Tragut, 2022). For example, a
theoretical computer science (CS) researcher might struggle more with jargon
in a chemistry paper than in a mathematics paper, but the opposite may be true
for a CS researcher in computational biology. Information on researcher’s
background could help determines what they know, and what they need explained.

In this paper, we investigate techniques for estimating jargon familiarity for
individual researchers. We ground our investigation in the real-world setting
of interdisciplinary reading: researchers reading papers in less familiar
domains. We first validate our setting with an initial study around
interdisciplinary reading. The results reveal a clear preference for
supplementary information beyond what is provided in the abstract, especially
in less familiar domains. Building on these initial findings, we propose the
task of personalized scholarly jargon identification: predicting the
familiarity and any associated information needs of a term for an individual
researcher.

Prior work has explored providing interactive systems to augment scientific
abstracts (Fok et al., 2023) and provide term definitions (Head et al., 2020;
August et al., 2022). We investigate a complementary task: predicting which
terms and what information a researcher needs. Models that can achieve
accurate, individualized predictions in this context could greatly improve
interactive reading systems by focusing aids on only the most important,
unfamiliar words for an individual reader, avoiding information overload
(Ridder, 2002; Head et al., 2020). Further, having effective predictions of
term familiarity can improve writing aids by identifying terms most likely to
be unfamiliar to a defined audience.

We collect a dataset of over 10k individual familiarity ratings and
information needs from 11 CS researchers about terms drawn from 100 out-of-
domain abstracts (example in Figure 1). We enumerate features representing an
individual’s knowledge based on papers they have written and read within their
domain. Using these features and our dataset, we investigate baselines for
estimating term familiarity, including regression models and prompt-based
approaches using large language models (LLMs). Our analysis reveals that
incorporating individual-level information improves the accuracy of predicting
term familiarity, though a zero-shot baseline presents a strong alternative.
Our project contributes the following:

  1. 1.

We define the novel task of predicting personalized jargon familiarity. We
motivate our task based on initial experiments with interdisciplinary computer
science researchers.

  2. 2.

We collect a dataset of over 10k term familiarity ratings and individual
information needs. Eleven CS researchers provided familiarity ratings and
additional information needs for terms drawn from 100 out-of-domain abstracts.

  3. 3.

We enumerate features approximating an individual researcher’s knowledge (at
the domain, subdomain, and personal levels) and investigate integrating these
features into supervised and prompt-based methods for term familiarity
prediction. Our results show the value of domain, subdomain, and personal
level features for both methods.

##  2 Related Work

Relevant to the current paper is research on interdisciplinary communication,
scientific text simplification, and user modeling.

###  2.1 Interdisciplinary communication

Interdisciplinary research integrates knowledge from multiple disciplines to
address a shared question (Daniel et al., 2022). Choi and Pak (2007) surveyed
interdisciplinary researchers in the health sciences, finding that a mismatch
in terminology complicates efforts in communicating between disciplines.
Similar findings have been reported when interviewing researchers in
psychology and neuroscience (Wudarczyk et al., 2021). Lucy et al. (2022) found
that papers that used more discipline-specific terminology (i.e., jargon), had
fewer citations across disciplines, and Martínez and Mammola (2021) found that
papers who use more jargon are generally cited less.

###  2.2 Scientific text simplification

Detecting scholarly jargon is commonly done using corpus-based approaches
(Tanaka-Ishii and Terada, 2011). Gardner and Davies (2013) identified
scholarly jargon in English by studying the frequency of words within
scientific papers compared to a background corpus of general English writing.
Similar methods have identified jargon in specific fields of science,
including medical studies (August et al., 2022), and computer science papers
(Salatino et al., 2018). Taking into account individual knowledge can also
improve jargon identification (Lee and Yeung, 2018) and definitions (Murthy et
al., 2021). Gooding and Tragut (2022) found that training models at the
individual level improves complex word identification, and Lin et al. (2012)
found that using social media posts written by an individual can help predict
word familiarity. In work most similar to our own, Murthy et al. (2021)
introduced the task of defining scientific terms based on words a scientist
already knows. In contrast to prior work, our focus on scientists provides
unique opportunities to model individual knowledge. Scientists develop deep
knowledge of their field by reading and writing scientific papers. We
investigate techniques to leverage this information to predict personal term
familiarity.

###  2.3 User Modeling

Work has sought to model the knowledge of individuals using cognitive
approaches (Desmarais and Baker, 2012; Wu et al., 2021; Amith et al., 2020).
Amith et al. (2017) compared expert and consumer health knowledge on vaccines
and general medical concepts with estimates of semantic distance between terms
inferred from expert- and consumer-authored corpora using methods of
distributional semantics. Work has also defined differences in concept
knowledge between experts and lay readers for general medical concepts (Zhang,
2002). User modeling approaches require either an existing set of concepts or
information about how concepts are related to one another. We take a more
lightweight approach by leveraging existing text representing the knowledge of
an individual scientist to estimate their knowledge of new terms.

##  3 Task Description

We conduct an initial study with 10 computer science researchers to (1)
validate our intuition that term familiarity is important for
interdisciplinary reading, and (2) identify what information needs scholars
have for unfamiliar terms when reading across domains.

We recruited participants from two subdomains of computer science: Natural-
Language Processing (NLP) and Human-Computer Interaction (HCI). Participants
were asked to read two paper abstracts, one from a closer domain: Linguistics
or Psychology, and one from a more distant domain: Medicine. For each paper,
we provided two abstract variants: the original author-written abstract and a
generated abstract personalized to the participant’s background (details can
be found in App. Table 1). After reading each abstract pair, participants
identified what modifications they liked/disliked in the personalized
abstract, and provided a free-text response on whether they preferred the
generated abstract and why. The study was considered exempt upon University of
Washington IRB [MOD00015554].

###  3.1 Initial Findings

Of 20 responses collected from 10 researchers, 14 expressed a preference for
the personalized abstract over the original, with 9 preferring the
personalized abstract for the medical paper (90%) and 5 preferring the
personalized abstract for the linguistics or psychology paper (50%). Full
results can be found in App. A.1.

We categorize the participants’ preferred transformations as satisfying the
following information needs:

  * •

Definition: provides key information on the term independent of any context
(e.g., a specific scientific abstract). A definition answers the question,
“What is/are [term]?”

  * •

Background: introduces information that is important for understanding the
term in the context of the abstract, e.g., how the term relates to the overall
problem, significance, and motivation of the paper.

  * •

Example: offers specific instances that help illustrate the practical
application or usage of the term within the abstract.

  * •

Method/Result Details: details on the methodology and results of the paper,
e.g., how models were applied or data collected.

  * •

Relevant Downstream Connections: provides insights about how the current
paper’s findings relate to the reader’s own research.

Participants generally indicated a preference for receiving additional
information only when they were unfamiliar with a term. This suggests that an
effective approach to personalizing scientific abstracts would be to first
identify unfamiliar terms, then determine the applicable information need for
each. In this work, we focus on the first three needs—definitions, background,
and examples—as these were typically associated with researchers being
unfamiliar with the abstract domain (24 out of 30 of the preferred changes
were in the medical abstract). We omit method/result details from further
study because these needs usually surfaced when a researcher was more familiar
with a domain or particular method. While participants generally reacted
positively to relevant downstream connections, these were usually hallucinated
by the model; we consequently avoid targeting these as well.

###  3.2 Task Definition

Based on these initial findings, we identify the tasks of individual term
familiarity prediction and information need prediction as important steps for
assisting interdisciplinary reading of scientific abstracts. We formalize the
first task as: given an individual researcher defined by their authored
publications $R=\\{r_{1},r_{2},...r_{m}\\}$ and an abstract to personalize
$A$, which includes a set of terms $T=\\{t_{1},t_{2},...t_{n}\\}$, our goal is
to predict the subset of terms unfamiliar to $R$.

##  4 Dataset

As no pre-existing datasets exist for personalized scientific jargon
identification, we construct a new dataset of terms from abstracts with human
annotations of familiarity and additional information needs. We direct our
focus to abstracts that are outside the individual’s domain, with CS
researchers as the annotators.

###  4.1 Data Source

To ensure that the out-of-domain abstracts could realistically be read by our
annotators, we compile a corpus of non-CS papers often viewed by CS
researchers, published after 2010, using the Semantic Scholar API (Kinney et
al., 2023). We define CS researchers as anyone who has co-authored a paper
categorized as ‘Computer Science’ using the API. We take the top 500 viewed
papers not categorized as CS. To construct a representative corpus covering
diverse domains, we randomly draw from each domain such that the resulting
numbers of abstracts across all domains are proportionally the same as those
from the most-viewed non-CS papers. This method was employed to select 100
abstracts. For each abstract, the top-10 significant terms are identified
using the OpenAI model text-davinci-003 (details and prompt in App. Table 1).
We manually review 10 abstracts and confirm that these top 10 terms align with
our notion of salient terms in each abstract. We define salient terms as terms
that could be provided as keyword descriptors of the paper.

###  4.2 Annotation

We recruit annotators that are: (1) earning at least a master’s degree in CS,
and (2) an author on at least one published work. Annotators were asked to
annotate each term with the following information:

  * •

Familiarity on a scale of 1 (not at all familiar) to 5 (extremely familiar).
Not at all familiar was defined as ‘you have never heard of this term.’
Extremely familiar was defined as ‘you have a deep, comprehensive
understanding of this term.’

  * •

Additional information needs that could help annotators understand the
abstract. These included definitions, background, and examples (defined in
§3.1). Annotators could select more than one information need.

We recruited 11 annotators with Master’s (N=4) or Doctorate (N=7) degrees in
CS via UpWork, and paid each annotator $20-30 hourly based on their degree.
Each annotator reviewed terms from all sampled abstracts and answered all
questions. The 11 annotators are from 17 self-defined subfields, including
computer vision, machine learning, NLP, cognitive science, education, computer
networks, etc. Their numbers of publications varied from 1 to 60, with a
median publication count of 10.

###  4.3 Outcomes

We define a binary term familiarity outcome measure by grouping the collected
5-point familiarity ratings into the binary classes of “familiar” (ratings
$\geq$ 3) and “unfamiliar” (ratings $\leq$ 2). For terms annotated as
requiring further information, we also treat the need for additional
definitions, background, and/or examples as labels for a secondary
classification task.

###  4.4 Analysis

Our dataset includes 956 terms sourced from 100 abstracts from 22
domains.111GPT 3.5 identified ¡10 terms from some abstracts. Upon inspecting
these abstracts, the authors agreed that there were fewer than 10 important
terms to list. We collected a total of 10,571 familiarity ratings and
responses for additional information needs from the 11 annotators.

Figure 2: Mean familiarity and additional information needs (definition,
background, and example) across abstract domains. The ratio of terms shows how
many terms in the abstract domain are familiar, and require definitions,
background, and examples.

#### Data check

To ensure that annotator familiarity ratings were consistent, we conducted a
data check for all annotators. We selected 10 entities from each annotator, 5
rated as familiar and 5 rated as unfamiliar. For each entity, we asked
annotators to provide a definition of the entity without looking up any
information. If they could not define the term, we instructed them to write
‘N/A’. Annotators were generally consistent with their initial scores, with
81% of their responses matching their initial ratings (i.e., if they were
familiar, they wrote a correct definition). In cases where initial scores did
not align with the data check, annotators generally wrote definitions given by
context in the abstract.

#### Domain-specific variations

Figure 2 illustrates the differences in familiarity and additional information
needs across abstract domains. Generally, we see that annotators most often
select high familiarity for Art while Chemistry receives the lowest
familiarity ratings. This represents a general trend of annotators selecting
lower familiarity for technical sciences than for humanities or social
sciences. The same trend holds for additional information needs; annotators
request the most definitions and background for terms in Chemistry, and the
least in Art. Interestingly, though, annotators in general prefer a baseline
rate of examples that does not vary greatly across domains.

#### Individual-specific variations

19% of the terms receive uniform familiarity ratings from all annotators.
Consensus in familiarity is most frequently found within terms related to
Philosophy and Business. Conversely, terms associated with Chemistry and
Biology are consistently marked as unfamiliar. Notably, there is an even split
in familiarity for 15% of the terms—half deemed familiar and half not—with
Mathematics and Engineering being the predominant fields for such terms. This
diversity highlights the nuances of individual knowledge backgrounds even
within the common discipline of CS.

##  5 Predicting Term Familiarity

Given our dataset of term familiarity ratings and annotator backgrounds, we
investigate the effectiveness of a set of features and methods for predicting
individual term familiarity.

###  5.1 Features

Past work has explored using readability measures, frequency statistics, and
embeddings to predict term familiarity at the population-level (Rakedzon et
al., 2017; August et al., 2022; Lucy et al., 2022). We adapt the following
features for predicting individual-level term familiarity:

  * •

Frequency: The number of times a term appears in researcher’s publications
$R$.

  * •

Specificity: Quantifies the term’s uniqueness to a corpus (Zhang et al.,
2017), computed as the log probability ratio:

$S_{c}(t)=log\frac{P_{c}(t)}{P_{C}(t)}$

In our case, $c$ corresponds to the target abstract $A$ and $C$ to the
researcher’s publications $R$.

  * •

Embedding similarity: Leverages SPECTER Cohan et al. (2020), a citation-based
transformer model encoding semantic document similarity. We calculate the
minimum Euclidean distance between the target abstract $A$’s embedding and any
of the author’s publications $R=\\{r_{1},r_{2},...\\}$’s embeddings as its
SPECTER similarity.

For each feature, we start by defining different granularities of researcher’s
publications $R$, representing domain, subdomain, and individual-level
information. The following data is extracted from the Semantic Scholar API
(Kinney et al., 2023).

  * •

Domain: 10k randomly sampled CS papers from 2015-2022.

  * •

Subdomain: 10k randomly sampled papers from each annotator’s self-defined
subdomain corpus from CS papers from 2015-2022. Subdomain corpora are defined
based on venues associated with a given subdomain.

  * •

Individual: Represented by an individual’s publications. If the individual’s
number of publications is less than the necessary number for training in §5.2,
the remaining quantity is supplemented by a random selection from the cited
references within those publications.

In addition to these granular features, we include the following general-
purpose measures of readability and metadata.

  * •

Readability: We utilize the Flesch-Kincaid (F-K) Flesch (2007) readability
score and the GPT-2 perplexity score Martinc et al. (2021). F-K score is
computed at the passage level; all terms from an abstract are assigned the
same F-K score.

  * •

Metadata: We include the target paper domain, as well as the individual’s
attributes, including year of their first published paper, total number of
published papers, and their average citation count for published papers.

###  5.2 Models

We explore two modeling approaches: supervised and prompt-based:

#### Lasso regression

We adopt a Logistic regression model with L1 regulation to integrate features
across various levels of granularity and determine feature importance. A
binary label determination is made using a threshold value of 0.5. We train
one model per annotator. The model training is divided into two distinct
settings:

  * •

Individual model: the model is trained using data derived from terms annotated
by the specific annotator alone.

  * •

Mixed model: the model is trained on familiarity ratings from all other
annotators (i.e., leave-one-annotator-out testing). To maintain the same
sample size of training data as each individual Lasso model, for each
annotator we randomly select the same number of training data points as was
used to train their individual model.

#### Prompt-based LLMs

We design prompts to complete a binary classification task of familiarity
based on identified terms and the target abstract, using the GPT-4 model from
OpenAI. To explore different strategies, we use:

  * •

Baseline: providing only the term and abstract containing the term.

  * •

Metadata: providing annotator metadata along with the baseline prompt.

  * •

Context-enhanced learning: providing publications from the annotator’s domain,
subdomain, and individual level separately. This configuration requires less
data collection efforts as no labeled data is needed.

  * •

Few-shot learning: providing examples of term, abstract, and familiarity
ratings. The ratings are drawn from the three levels of granularity: ratings
from other annotators with no overlap in subdomain (domain), ratings from
other annotators within the same subdomain (subdomain), and ratings from the
individual annotator (individual)

Five examples are given for context-enhanced learning (e.g., abstracts from
five publications authored by the annotator) and few-shot learning (e.g., five
term, abstract, and familiarity labels from the annotator at the abstract
level.

###  5.3 Evaluation

We split the entities randomly into an 80/20 train/test set for each
annotator. All the models are then evaluated on the same test set for each
annotator. The F1 score is reported to measure the model’s performance on the
classification task. To determine the critical features in a Lasso model, we
count the features with non-zero value coefficients. A higher frequency
denotes greater and more consistent influence of the feature on prediction.

##  6 Results

#### RQ1. How do supervised and prompt-based methods perform?

Figure 3: Model performance on term familiarity prediction for mixed and
individual Lasso regression (blue) and GPT-4 with four prompt settings
(orange).

In Figure 3, we present a comparison of F1 score across the highest performing
predictive models. In the Lasso models, features from multiple levels of
granularity are integrated. As expected, the individual Lasso model shows
superior performance compared to the mixed Lasso, due to the fact that the
training data for individual Lasso was from the same annotator. Despite being
provided less data, GPT-4 outperforms Lasso in all settings. Metadata and
context-enhanced learning perform better than few-shot learning, suggesting
that the LM can infer some background knowledge of annotators from metadata or
free text.

#### RQ2. What features and granularity level influence performance?

Figure 4: Frequency of non-zero Coefficients in individual Lasso models across
researchers. The Lasso penalty minimizes less critical coefficients to zero.
Features with higher frequencies of non-zero values in individual models are
consistently identified as important. Figure 5: GPT-4 model performance: (a)
with metadata, (b) at diverse granularity levels with a fixed example number
of five, and (c) with varying example numbers at the personal granularity
level.

Figure 4 reveals the non-zero coefficients of the Lasso models. The domain of
the target abstract is consistently identified as a significant feature by all
individual models across annotators, indicating that an abstract’s content is
important for determining term familiarity, independent of individual
features. Key aspects like word specificity and embedding similarity at the
domain level underscore the relevance of a researcher’s domain in their
familiarity with jargon, aligning with previous population-level familiarity
research (Li et al., 2020). However, the importance of individual-level
features such as frequency, specificity, and embedding similarity also emerge,
highlighting the dual influence of both domain-specific and individual-
specific factors in accurately predicting term familiarity. Regarding GPT-4
model performance (shown in Figure 5), prompting with researcher subdomain
information seems to be the most effective strategy, suggesting that subdomain
information (e.g., that a researcher is in NLP) is useful beyond a
researcher’s broad field (e.g., CS) .

#### RQ3. How does the amount of labeled data affect task performance?

Figure 6: Performance of the Lasso model with varying training sizes.

To explore this, we incrementally increase the training data for Lasso models.
As indicated in Figure 6, both mixed and individual models reach a performance
plateau after the inclusion of 200 terms. Figure 5(c) shows that for GPT
models, performance deteriorates with more examples in few-shot learning,
potentially due to cross-domain data confusing the LLM’s ability to predict
individual level familiarity. For context-enhanced learning, five examples is
more effective than ten, indicating that for LLMs, having too many examples is
not necessarily beneficial. The performance across all granularity levels with
a range of example numbers can be seen in Appendix Figure 8.

##  7 Discussion

This paper introduces the novel task of scholarly jargon detection for the
individual researcher. This task is motivated by the needs of
interdisciplinary research, where researchers may need to read abstracts in
domains outside their own. By focusing on domain- and personal-level data in
term familiarity prediction, our research provides an analysis of the type and
granularity of features, methods, and modeling strategies useful in the task
of personalized jargon identification.

In pursuit of this task, we collect a dataset of over 10K term familiarity and
information need annotations from 11 CS researchers for terms from 100 out-of-
domain abstracts. Our dataset reveals significant variations in term
familiarity among CS researchers, underscoring the diversity of individual
knowledge. Additionally, the data shows that term familiarity is not always
associated with the information needs (e.g., a researcher might desire an
example even if they are familiar with a term), highlighting that information
needs might need to be addressed differently than term familiarity.

Our findings show that unlabeled individual-level data can improve the
prediction of term familiarity. This is particularly evident in prompt-based
methodologies, where the use of an individual’s publications as unlabeled data
proved more beneficial than term familiarity labels from that individual. This
might be because labeled examples did not provide enough relevant information
to a model (e.g., a single term) compared to the potentially rich information
found in a researcher’s publications.

Domain and subdomain level data (e.g., the subdomain of the researcher) also
benefited prompt-based and regression models. This suggests that aggregated
data can be effective substitutes for individual labels, a promising direction
given the cost of collecting researcher-labeled term familiarity. Furthermore,
for researchers with a small publication history (and therefore a small amount
of unlabeled individual-level data), domain and subdomain data can provide
valuable information for predicting jargon familiarity.

Our methods provide an exciting first step to support researchers reading
abstracts outside of their domain (Wudarczyk et al., 2021). Our dataset and
findings show that models can identify unfamiliar terms for individual
researchers with some accuracy. Combined with text simplification techniques
(Srikanth and Li, 2020), models that can predict an individual researcher’s
familiarity with a term and any subsequent information needs could assist
researchers by rewriting an abstract tuned to that particular researcher,
thereby minimizing the barriers of interdisciplinary communication.

##  8 Conclusion

This paper introduces the novel task of scientific jargon detection for the
individual researcher. We collect a dataset of over 10k term familiarity
annotations from computer science researchers and investigate supervised and
prompt-based methods to predict term familiarity. We find that leveraging a
researcher’s publication history, self-reported subdomain, and general domain
information can improve term familiarity prediction. Our results provide
insight on integrating an individual’s knowledge into scientific jargon
detection.

## Limitations

We focus on computer science researchers for both selecting annotators and
relevant abstracts in other fields. This might limit our ability to generalize
to other domains or researchers. Because of the cost of collecting
annotations, our dataset is also relatively small, which might further limit
its generalizability. Our goal with the dataset and analysis is to show the
potential of modeling individual term familiairty and information needs. We
are excited about future work expanding this goal into new domains.

## Ethics Statement

Some of the methods in the paper include personal data (e.g., publication
record, labeled terms), which might pose a privacy risk for some researchers.
Systems identifying term familiarity and information needs must keep any
personal data stored locally and allow researchers to remove or view their
data at any time. Focusing on computer science researchers as a first step for
predicting term familiarity might also allow computer science researchers to
more effectively read outside their discipline, but not researchers in other
domains reading within computer science. While encouraging interdisciplinary
reading can improve two-way communication, it is also important to consider
the voices of researchers in other domains.

## References

  * Amith et al. (2020) Muhammad Tuan Amith, Trevor A. Cohen, Rachel Cunningham, Lara S. Savas, Nina Smith, Paula M. Cuccaro, Efrat K. Gabay, Julie A. Boom, Roger W. Schvaneveldt, and Cui Tao. 2020.  Mining hpv vaccine knowledge structures of young adults from reddit using distributional semantics and pathfinder networks.  _Cancer Control : Journal of the Moffitt Cancer Center_ , 27.
  * Amith et al. (2017) Muhammad Tuan Amith, Rachel Cunningham, Lara S. Savas, Julie A. Boom, Roger W. Schvaneveldt, Cui Tao, and Trevor A. Cohen. 2017.  Using pathfinder networks to discover alignment between expert and consumer conceptual knowledge from online vaccine content.  _Journal of biomedical informatics_ , 74:33–45.
  * August et al. (2022) Tal August, Lucy Lu Wang, Jonathan Bragg, Marti A. Hearst, Andrew Head, and Kyle Lo. 2022.  Paper plain: Making medical research papers approachable to healthcare consumers with natural language processing.  _ACM Transactions on Computer-Human Interaction_ , 30:1 – 38.
  * Barnett and Doubleday (2020) Adrian P. A. Barnett and Zoe Doubleday. 2020.  The growth of acronyms in the scientific literature.  _eLife_ , 9.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  _ArXiv_ , abs/2005.14165.
  * Choi and Pak (2007) Bernard C. K. Choi and Anita W. P. Pak. 2007.  Multidisciplinarity, interdisciplinarity, and transdisciplinarity in health research, services, education and policy: 2. promotors, barriers, and strategies of enhancement. _Clinical and investigative medicine. Medecine clinique et experimentale_ , 30 6:E224–32.
  * Cohan et al. (2020) Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel S. Weld. 2020.  SPECTER: Document-level Representation Learning using Citation-informed Transformers.  In _ACL_.
  * Daniel et al. (2022) Kristy L Daniel, Myra McConnell, Anita Schuchardt, and Melanie E. Peffer. 2022.  Challenges facing interdisciplinary researchers: Findings from a professional development workshop.  _PLoS ONE_ , 17.
  * Desmarais and Baker (2012) Michel C. Desmarais and R. Baker. 2012.  A review of recent advances in learner and skill modeling in intelligent learning environments.  _User Modeling and User-Adapted Interaction_ , 22:9–38.
  * Flesch (2007) Rudolf Flesch. 2007.  Flesch-kincaid readability test.  _Retrieved October_ , 26(3):2007.
  * Fok et al. (2023) Raymond Fok, Joseph Chee Chang, Tal August, Amy X. Zhang, and Daniel S. Weld. 2023\.  Qlarify: Bridging scholarly abstracts and papers with recursively expandable summaries.  _ArXiv_ , abs/2310.07581.
  * Gardner and Davies (2013) Dee Gardner and Mark Davies. 2013.  A new academic vocabulary list.  _Applied Linguistics_ , 35(3):305–327.
  * Gooding and Tragut (2022) Sian Gooding and Manuel Tragut. 2022.  One size does not fit all: The case for personalised word complexity models.  _ArXiv_ , abs/2205.02564.
  * Guo et al. (2022) Yue Guo, Wei Qiu, Gondy Leroy, Sheng Wang, and Trevor A. Cohen. 2022.  Cells: A parallel corpus for biomedical lay language generation.  _ArXiv_ , abs/2211.03818.
  * Guo et al. (2021) Yue Guo, Wei Qiu, Yizhong Wang, and Trevor Cohen. 2021.  Automated lay language summarization of biomedical scientific reviews.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, pages 160–168.
  * Head et al. (2020) Andrew Head, Kyle Lo, Dongyeop Kang, Raymond Fok, Sam Skjonsberg, Daniel S. Weld, and Marti A. Hearst. 2020.  Augmenting scientific papers with just-in-time, position-sensitive definitions of terms and symbols.  _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_.
  * Kinney et al. (2023) Rodney Michael Kinney, Chloe Anastasiades, Russell Authur, Iz Beltagy, Jonathan Bragg, Alexandra Buraczynski, Isabel Cachola, Stefan Candra, Yoganand Chandrasekhar, Arman Cohan, Miles Crawford, Doug Downey, Jason Dunkelberger, Oren Etzioni, Rob Evans, Sergey Feldman, Joseph Gorney, David W. Graham, F.Q. Hu, Regan Huff, Daniel King, Sebastian Kohlmeier, Bailey Kuehl, Michael Langan, Daniel Lin, Haokun Liu, Kyle Lo, Jaron Lochner, Kelsey MacMillan, Tyler Murray, Christopher Newell, Smita R Rao, Shaurya Rohatgi, Paul L Sayre, Zejiang Shen, Amanpreet Singh, Luca Soldaini, Shivashankar Subramanian, A. Tanaka, Alex D Wade, Linda M. Wagner, Lucy Lu Wang, Christopher Wilhelm, Caroline Wu, Jiangjiang Yang, Angele Zamarron, Madeleine van Zuylen, and Daniel S. Weld. 2023.  The semantic scholar open data platform.  _ArXiv_ , abs/2301.10140.
  * Lee and Yeung (2018) John Sie Yuen Lee and Chak Yan Yeung. 2018.  Personalizing lexical simplification.  In _International Conference on Computational Linguistics_.
  * Li et al. (2020) Haoran Li, Junnan Zhu, Jiajun Zhang, Chengqing Zong, and Xiaodong He. 2020.  Keywords-guided abstractive sentence summarization.  In _AAAI Conference on Artificial Intelligence_.
  * Lin et al. (2012) Hsin-Ni Lin, Shu-Kai Hsieh, and Shiao-Hui Chan. 2012.  Measuring individual differences in word recognition: The role of individual lexical behaviors.  In _ROCLING/IJCLCLP_.
  * Liu et al. (2014) Hongjun Liu, Sheng Dai, and De-en Jiang. 2014.  Solubility of gases in a common ionic liquid from molecular dynamics based free energy calculations.  _The Journal of Physical Chemistry B_ , 118(10):2719–2725.
  * Lucy et al. (2022) Li Lucy, Jesse Dodge, David Bamman, and Katherine A. Keith. 2022.  Words as gatekeepers: Measuring discipline-specific terms and meanings in scholarly publications.  In _Annual Meeting of the Association for Computational Linguistics_.
  * Martinc et al. (2021) Matej Martinc, Senja Pollak, and Marko Robnik-Šikonja. 2021.  Supervised and unsupervised neural approaches to text readability.  _Computational Linguistics_ , 47(1):141–179.
  * Martínez and Mammola (2021) Alejandro Martínez and Stefano Mammola. 2021.  Specialized terminology reduces the number of citations of scientific papers.  _Proceedings of the Royal Society B_ , 288.
  * Murthy et al. (2021) Sonia K. Murthy, Daniel King, Tom Hope, Daniel S. Weld, and Doug Downey. 2021.  Towards personalized descriptions of scientific concepts.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.  _ArXiv_ , abs/2303.08774.
  * Plaven-Sigray et al. (2017) Pontus Plaven-Sigray, Granville James Matheson, Björn Christian Schiffler, and William Hedley Thompson. 2017.  The readability of scientific texts is decreasing over time.  _eLife_ , 6.
  * Rakedzon et al. (2017) Tzipora Rakedzon, Elad Segev, Noam Chapnik, Roy Yosef, and Ayelet Baram-Tsabari. 2017.  Automatic jargon identifier for scientists engaging with the public and science communication educators.  _PLoS ONE_ , 12.
  * Ridder (2002) Isabelle De Ridder. 2002.  Visible or invisible links? In _CHI Extended Abstracts_.
  * Salatino et al. (2018) Angelo Salatino, Thiviyan Thanapalasingam, Andrea Mannocci, Francesco Osborne, and Enrico Motta. 2018.  The computer science ontology: A large-scale taxonomy of research areas.  In _International Workshop on the Semantic Web_.
  * Srikanth and Li (2020) Neha Srikanth and Junyi Jessy Li. 2020.  Elaborative simplification: Content addition and explanation generation in text simplification.  In _Findings_.
  * Strober (2006) Myra H. Strober. 2006.  Habits of the mind: Challenges for multidisciplinary engagement.  _Social Epistemology_ , 20:315 – 331.
  * Tanaka-Ishii and Terada (2011) Kumiko Tanaka-Ishii and Hiroshi Terada. 2011.  Word familiarity and frequency.  _ArXiv_ , abs/1806.03431.
  * Wu et al. (2021) Chuhan Wu, Fangzhao Wu, Yongfeng Huang, and Xing Xie. 2021.  User-as-graph: User modeling with heterogeneous graph pooling for news recommendation.  In _International Joint Conference on Artificial Intelligence_.
  * Wudarczyk et al. (2021) Olga A. Wudarczyk, Murat Kirtay, Anna K. Kuhlen, Rasha Abdel Rahman, John-Dylan Haynes, Verena V. Hafner, and Doris Pischedda. 2021.  Bringing together robotics, neuroscience, and psychology: Lessons learned from an interdisciplinary project.  _Frontiers in Human Neuroscience_ , 15.
  * Zhang (2002) Jiajie Zhang. 2002.  Representations of health concepts: a cognitive perspective.  _Journal of biomedical informatics_ , 35 1:17–24.
  * Zhang et al. (2017) Justine Zhang, William Hamilton, Cristian Danescu-Niculescu-Mizil, Dan Jurafsky, and Jure Leskovec. 2017.  Community identity and user engagement in a multi-community landscape.  In _Proceedings of the international AAAI conference on web and social media_ , volume 11, pages 377–386.

##  Appendix A Appendix

###  A.1 Initial Task Definition

For our initial study, we focus on reading full scientific abstracts because
we wanted to understand if term familiarity was an important part of our
broader envisioned setting of researchers reading abstracts outside of their
domain. Further, our goal in providing a naive personalized abstract was to
probe what transformations are feasible with current models that researchers
respond positively to.

One abstract was from the medical domain, a domain not within any
participant’s dominant expertise, and one abstract was from a domain related
to but distinct from a participant’s specialization (e.g., psychology,
linguistics). Personalized abstracts were generated using text-davinci-003
model from OpenAI (OpenAI, 2023; Brown et al., 2020). As a first step in
including a individual background knowledge, the participant’s publications
were added in the prompt and the model was instructed to personalize the
abstract to the given reader.

Figure 7: Formative study results: domain-specific attitudes towards (a)
personalized abstracts and (b) transformations of personalized abstracts.
Medical abstracts are perceived as distant from the annotators’ background,
whereas linguistic or psychology abstracts are perceived as closer.

###  A.2 GPT Prompts

Task |  Prompt |  Model |  Max length |  Temp.
---|---|---|---|---
Abstract personalization |  You are tasked with the role as a scientific writer to generate personalized abstract for individual reader. To do this effectively, consider including relevant background information or motivations related to the subject matter, provide necessary definitions or explanations to elucidate complex concepts, and incorporate significant methodological or result-oriented details. However, ensure that any additional information included is directly relevant and can be traced back to the provided content. The paper needed to be personalized is:{ }; The reader’s publications are: { }; The reader’s references are:{}; The personalized abstract is: . |  text-davinci-003 |  500 |  0
Top-10 significant terms |  Please review the following scientific paper abstract. Your task is to identify all scientific-related word/phrases within the text and then rank these word/phrases in descending order based on their significance within the abstract itself. Retain the first 10 word/phrases:. |  text-davinci-003 |  100 |  0
Familiarity classification |  Your job is to estimate how much the reader knows about an entity. You will be provided with the entity, the abstract where the entity come from, and related data about either the reader or the abstract. Entity: {} Abstract:{} Related Data:{} Here’s how to gauge the reader’s familiarity: - 0: The reader knows this subject well and can describe it to others. - 1: The reader has either encountered this subject before but knows little about it, or has never come across it at all. Based on the information provied, determine the familiarity score, either 0 or 1: |  gpt-4 |  100 |  0
Definition needs classification |  Your job is to estimate whether the reader might need additional definition to fully grasp the entities mentioned in a given abstract. You will be provided with the entity, the abstract where the entity come from, and related data about either the reader or the abstract. Definition of definition/explanation: provides key information on the term independent of any context (e.g., a specific scientific abstract). A definition answers the question, Ẅhat is/are [term]?.̈ Entity: {} Abstract:{} Rel:{}. Provide the estimation whether additional information is needed in a list in the order of the entity. The estimation should be either 0(no) or 1(yes). No need to mention the entity: |  gpt-4 |  100 |  0
Background needs classification |  Your job is to estimate whether the reader might need additional background to fully grasp the entities mentioned in a given abstract. You will be provided with the entity, the abstract where the entity come from, and related data about either the reader or the abstract. Definition of background/motivation: introduces information that is important for understanding the term in the context of the abstract. Background can provide information about how the term relates to overall problem, significance, and motivation of the abstract. Entity: {} Abstract:{} Rel:{}. Provide the estimation whether additional information is needed in a list in the order of the entity. The estimation should be either 0(no) or 1(yes). No need to mention the entity: |  gpt-4 |  100 |  0
Example needs classification |  Your job is to estimate whether the reader might need additional example to fully grasp the entities mentioned in a given abstract. You will be provided with the entity, the abstract where the entity come from, and related data about either the reader or the abstract. Definition of example: offers specific instances that help illustrate the practical application or usage of the term within the abstract. Entity: {} Abstract:{} Rel:{}. Provide the estimation whether additional information is needed in a list in the order of the entity. The estimation should be either 0(no) or 1(yes). No need to mention the entity: |  gpt-4 |  100 |  0
Table 1: GPT-4 prompts and configurations.

All prompts used for GPT-4 experiments are shown in Table 1.

###  A.3 GPT-4 Results

GPT-4 modeling results are shown in Figure 8. We vary context-enhanced and ICL
example numbers at N=1, 5, and 10, and the granularity of input information by
domain, subdomain, and person/individual levels.

Figure 8: GPT-4 performance across all granularity levels with a range of
example numbers.