# Rethinking Unsupervised Domain Adaptation for Semantic Segmentation

Zhijie Wang1, Masanori Suganuma1,2, Takayuki Okatani1,2
1 Graduate School of Information Sciences, Tohoku University
2 RIKEN Center for AIP
{zhijie, suganuma<EMAIL_ADDRESS>

###### Abstract

Unsupervised domain adaptation (UDA) adapts a model trained on one domain
(called source) to a novel domain (called target) using only unlabeled data.
Due to its high annotation cost, researchers have developed many UDA methods
for semantic segmentation, which assume no labeled sample is available in the
target domain. We question the practicality of this assumption for two
reasons. First, after training a model with a UDA method, we must somehow
verify the model before deployment. Second, UDA methods have at least a few
hyper-parameters that need to be determined. The surest solution to these is
to evaluate the model using validation data, i.e., a certain amount of labeled
target-domain samples. This question about the basic assumption of UDA leads
us to rethink UDA from a data-centric point of view. Specifically, we assume
we have access to a minimum level of labeled data. Then, we ask how much is
necessary to find good hyper-parameters of existing UDA methods. We then
consider what if we use the same data for supervised training of the same
model, e.g., finetuning. We conducted experiments to answer these questions
with popular scenarios, {GTA5, SYNTHIA}$\rightarrow$Cityscapes. We found that
i) choosing good hyper-parameters needs only a few labeled images for some UDA
methods whereas a lot more for others; and ii) simple finetuning works
surprisingly well; it outperforms many UDA methods if only several dozens of
labeled images are available.

##  1 Introduction

Domain adaptation is the problem of adapting a model pre-trained on a domain
called the source domain to a different domain called the target domain.
Unsupervised domain adaptation (UDA) is the most widely studied variant. UDA
first trains a model in a supervised manner using the labeled source-domain
samples and then adapts the model to work well on the target domain using
unlabeled target-domain samples. Many studies have been conducted to develop
UDA methods for  image classification [7], object detection [26], semantic
segmentation [20, 22], etc. This study concentrates on semantic segmentation,
for which various methods have been developed, such as those based on
adversarial training [20, 21], self-training [28, 29], and their combinations
[11].

Almost all these studies assume that no labeled samples are available in the
target domain. This study first questions this assumption. We believe it is
practically unrealistic since we usually need labeled samples in the target
domain for a few reasons. First, we need to check the performance of models
before deployment. The most standard way is to evaluate their performance
using a certain amount, if not many, of labeled target-domain samples. Second,
we need to choose the hyper-parameter every UDA method must have. Despite
several attempts in the literature to do it without labeled target-domain
samples, the surest way is arguably to use them.

Why do the previous UDA studies not consider even a slight deviation from this
assumption? To the authors’ knowledge, no study provides an explanation or a
realistic, practical scenario for this setting. Previous studies concentrate
on methodologies but do not seem to pay much attention to data usage. While it
is typical for academic research, the attitude may not properly handle some of
the real-world demands. It is sometimes most important for practitioners to
decide how to utilize data.

Following this practitioner’s perspective, we locate ourselves in a data-
centric position. We first assume that labeled target-domain samples are
available; considering the high annotation cost that motivates UDA, we assume
a small number of labeled samples to be available. Then, we consider how we
should use them to maximize inference accuracy. Alternatively, we consider how
much data are necessary to achieve a target we set.

We consider two natural usages of labeled target-domain samples and compare
their effectiveness. One is to use them for tuning hyper-parameters of UDA
methods. Any UDA methods have one or more hyper-parameters, and it is
reasonable to choose them in this way. Researchers have not paid much
attention to the importance of tuning hyper-parameters with UDA methods until
recently. Saito et al. pointed out the issue with existing studies and
proposed a zero-shot approach, i.e., tuning UDA methods’ hyper-parameters
without using a labeled target-domain sample [17]. While such methods,
including more general-purpose methods [18, 25], are attractive, they have
their issues and cannot be a complete solution.

The other is to use labeled target-domain samples for supervised training.
While there may be several candidates, this study considers the most
straightforward one, i.e., using them to finetune the pre-trained model. We
consider how it performs compared with UDA methods for different amounts of
labeled samples. Note that our finetuning method does not use unlabeled
target-domain samples, which UDA methods use. While using them additionally is
expected to be beneficial, we leave it to a future study; we concentrate on
the simple finetuning to reveal the bottom-line impact of using labeled
samples. Although the setting of using labeled and unlabeled samples jointly
is considered by semi-supervised DA (SSDA), there is no well-established
method. It should also be noted that the previous SSDA studies assume that
more than 100 labeled samples (i.e., images) are available, whereas we
consider a different range from one labeled sample to at most several dozens.

In this study, we limit our attention to semantic segmentation. The reason is
multi-folds. One is the high demands for domain adaptation with the task due
to its high annotation cost. Another is the characteristics of the annotation.
The manual annotation is usually performed image-wise, which makes the
annotation costly. Thus, we mainly consider cases where a limited number of
images can be annotated. On the other hand, as an image consists of many
pixels, a single annotated image is equivalent to having many labeled samples
in the case of image classification.

##  2 Related Work

###  2.1 Unsupervised Domain Adaptation

This paper concentrates on unsupervised domain adaptation (UDA) for semantic
segmentation, although we believe our argument holds for other tasks. There
are two approaches to UDA: adversarial training and self-training. The former
primarily aims to reduce the domain gap through adversarial training in
feature space [22], input space [8, 9], or output space [20]. Self-training
creates pseudo labels for target domain samples and uses them to finetune a
pretrained model [2]. To create more accurate pseudo labels, CBST [28] and
CRST [29] use class-balanced self-training and confidence-regularized self-
training, respectively. Several attempts have recently been made to increase
performance by integrating adversarial training with self-training. BLF [10]
use pseudo labels without any filtering. To cope with erroneous labels,
AdaptMR [27] filters pseudo labels and ignores the filtered-out pixels; it
could discard useful information. IAST [11] also filters pseudo-labels for
self-training and employs entropy minimization to optimize the filtered-out
pixels.

Previous studies of UDA, including the above, share a common problem in the
experimental evaluation: there is no clear separation between validation and
test data. While often not explicitly stated, they select their
hyperparameters and evaluate the model’s performance on the same data. In this
paper, we examine the UDA methods’ sensitivity to the hyperparameters and also
consider how many labeled target-domain samples are necessary to choose the
hyperparameters.

###  2.2 Selecting Hyper-parameter for UDA Methods

Several studies consider choosing hyper-parameters of UDA methods without
labeled target-domain samples. A simple way is to use the source-domain risk
as a proxy of the target-domain risk [6]. Sugiyama et al. [18] and You et al.
[25] propose methods that weight source-domain risks according to the
similarity between target and source samples. However, if there is a large
domain gap, the two risks deviate from each other and these methods do not
work. Morerio et al. [12] propose to use the entropy of classifiers as an
unsupervised criterion for hyper-parameters selection. However, this method
may fail to find good hyper-parameters when target samples are wrongly
classified with high confidence, which often occurs in practice. Saito et al.
[17] propose to use the soft neighborhood density (SND) to address the
instability of the entropy-based validation. SND requires the users to specify
temperature for softening softmax probabilities, which needs to be chosen in
some ways.

###  2.3 Finetuning for Few-shot Learning

Finetuning has recently been found to be an effective method for few-shot
learning, i.e., training a model to classify novel classes from only their
few-shot training samples. Previously, meta-learning-based approaches were
believed to be the best for the problem. However, several recent studies have
discovered that simple finetuning on the novel class classification using
provided few-shot examples outperforms meta-learning-based methods. The key is
to train a model on the flat classification of all base classes using their
training samples in advance. Wang et al. [23] have shown that a simple method
for few-shot object detection works well, which finetunes only the last layer
of an object detector on a class-balanced dataset. Tian et al. [19] present a
method for few-shot image classification, which trains a feature extraction
network and uses it to map the query and support images into the feature
space. For few-shot image segmentation, Boudiaf et al. [1] propose RePRI,
which finetunes the classifier using the support samples during the inference
time to improve inference accuracy. These studies and our interest share the
interest in finetuning models with small amount of training samples. However,
they (implicitly) assume that the base and novel classes belong to the same
domain, which differs from domain adaptation.

###  2.4 Semi-supervised Domain Adaptation

Semi-supervised domain adaptation (SSDA) relaxes the assumption of UDA.
Namely, SSDA attempts to reduce domain gap using some amount of labeled
target-domain samples in addition to unlabeled ones. We consider a similar
setting where both unlabeled and labeled samples are available. However, the
previous SSDA studies deal with only the setting that relatively many labeled
samples ($\geq 100$) are available. In this study, we consider settings with
fewer labeled samples (i.e., from one to a few dozens). The difference may
change the problem’s fundamental nature and lead to a significant difference
in real-world applicability. It is also noted that the existing SSDA studies
share the same potential issues with the experimental design as UDA, i.e., no
clear separation between validation and test data.

The representative studies of SSDA are as follows. Wang et al. [24] propose a
dual-level adversarial training; the first is an image-level adaptation, which
aligns the global feature distributions, and the second is a semantic-level
adaptation, which eases the semantic-level misalignment between the source and
target features of the same class. Chen et al. [4] mix the source-domain data
and the target-domain data at a region-level and a sample-level to produce the
domain-mixed data, training teacher models. Then they train a student model by
knowledge distillation from the teacher models.

##  3 Rethinking UDA from Data-centric Perspective

###  3.1 Realism of the Assumption of UDA

The existing UDA studies assume that no labeled target-domain sample is
available at training (i.e., adaptation) time. Then, they aim to achieve as
good performance as possible by performing adversarial training [20, 21],
pseudo labeling [28, 29], their combination [11], etc. However, while it
simplifies the problem, is this assumption realistic from a practical point of
view? Few studies have paid attention to the practicality of the assumption.
To the authors’ knowledge, no study provides a realistic, practical scenario
of the setting.

We usually need a validation step to check models’ performance before
deployment. Not many practitioners will be comfortable without this step. The
most natural way to do it is to measure the performance using some amount of
labeled samples. Note that it may not be impossible without a labeled sample
of the learned task itself, e.g., using some proxy task (e.g., a downstream
task utilizing segmentation results) as evidence for the model’s performance.
However, this will be feasible only in a limited number of cases.

Assuming we have, if only a tiny amount, labeled target-domain samples, why do
we not spare some of them to improve the model’s performance? (We will use the
rest for validation.) So, we explore this idea. Namely, we assume we have a
small amount of labeled target-domain samples, slightly more than the minimum
requirement for the model’s validation. We then ask: what is the best way to
use them to achieve maximum performance.

###  3.2 Choosing Hyper-parameters of UDA Methods

One natural use of labeled target-domain samples is to use them to tune the
hyper-parameters of a UDA method. Any UDA method has hyper-parameters, which
need to be selected somehow. Unfortunately, most previous studies on UDA for
semantic segmentation [21, 13, 22, 11] do not handle it properly, as discussed
in Sec. 3.4. Different UDA methods will show different sensitivities to the
choice of their hyper-parameters. Some methods have many hyper-parameters, and
others have only a few. So, we first consider the following research questions
regarding the existing methods for UDA for semantic segmentation.

RQ1: How sensitive is their performance to the choice of hyper-parameters?

RQ2: How many labeled samples are necessary to choose the optimal or nearly
optimal hyper-parameters?

###  3.3 An Alternative Approach: Finetuning

Another way of using labeled target-domain samples is to use them for
supervised training. UDA methods adapt a model pre-trained on the source
domain in a super-vised fasihon. We consider finetuning the same pre-trained
model using the available labeled target-domain samples. It is arguably the
most straightforward approach to transfer a model across different domains.
Then, we ask the following questions:

RQ3: How effective is it to do finetuning the model using a small amount of
labeled target-domain samples?

RQ4: Which performs better between existing UDA methods and the finetuning
method in an equal condition in the number of available labeled samples?

Note that our finetuning method does not use the unlabeled target-domain
samples that UDA methods use. Thus, the comparison of the two is not fair. We
concentrate on the simple finetuning method for simplicity and two more
reasons. Firstly, using additional unlabeled target-domain samples will
undoubtedly lead to improvement. (The question is how significant the
improvement will be.) Thus, we first want to estimate the minimum impact of
using labeled samples. Secondly, it is unclear how we can utilize the two
types of samples; thus, we leave it a future study. The setting of using a
combination of labeled and unlabeled samples is called semi-supervised domain
adaptation (SSDA), for which several studies exist. However, the existing
studies deal with only the case where relatively many labeled samples are
available. The existing SSDA studies assume 100–500 labeled samples (i.e.,
images), whereas our setting assumes at most several dozens of labeled
samples, following the motivation behind UDA. It is unclear if the method
known to be effective for the former is effective for the latter.

###  3.4 Issues with the Experimental Design of UDA

As mentioned above, previous studies of UDA for semantic segmentation do not
follow the standard machine learning procedure for the split of data. This is
partly because they divert the datasets designed for the standard setting of
supervised learning to domain adaptation. Most of them employ the Cityscapes
dataset [5] for the target-domain data; GTA5 [15] and SYNTHIA [16] are used
for the source-domain data. The Cityscapes dataset provides the official
training/validation/test splits. The training and validation sets are publicly
available, whereas the test set is not; only input images are available. To
evaluate a method’s performance one need to submit its result on these imaegs
to the official server111https://www.cityscapes-dataset.com/submit/.

Arguably due to this limitation, the previous studies of UDA use the same
official data split; that is, they use the training set for adaptation and
evaluate the methods’ performance on the validation set. Thus, they do not use
a separate set for validation; some use the same validation set for choosing
hyper-parameters and others do not mention how to do it.

In our experiments, we split the official validation set into customized
validation and test sets to rectify the above issue.

##  4 Experiments

###  4.1 Experimental Settings

We consider the following six UDA methods for semantic segmentation:
AdaptSegNet [20]222https://github.com/wasidennis/AdaptSegNet, AdvEnt
[21]333https://github.com/valeoai/ADVENT, CBST
[28]444https://github.com/yzou2/CRST, FADA [22]555https://github.com/JDAI-
CV/FADA, IntraDA [13],666https://github.com/feipan664/IntraDA and IAST
[11]777https://github.com/Raykoooo/IAST. We used their public implementations,
all of which employ DeepLabv2 [3] based on a ResNet-101 backbone. We also
followed the original training procedures including data augmentation except
FADA, for which we skipped the final self-distillation step for simplicity. We
conducted all experiments using PyTorch [14] and Nvidia 2080Ti except IAST,
for which we used Nvidia V100.

Following the previous studies, we choose the Cityscapes dataset [5] for the
target domain and GTA5 [15] and SYNTHIA [16] for the source domains. We
consider two adaptation scenarios, i.e., GTA5 $\rightarrow$ Cityscapes and
SYNTHIA $\rightarrow$ Cityscape.

To rectify the issue with the split of data in the previous studies’
experiments explained in Sec. 3.4, we create and use a customized validation
set $\mathcal{S}_{\mathrm{val}}$ and test set $\mathcal{S}_{\mathrm{test}}$
from the Cityscapes’ official validation set. Specifically, we create
$\mathcal{S}_{\mathrm{val}}$ and $\mathcal{S}_{\mathrm{test}}$ in the
following way. We first choose 400 images to form
$\mathcal{S}_{\mathrm{test}}$ (i.e.,
$\lvert\mathcal{S}_{\mathrm{test}}\rvert=400$) from the official validation
set, which include 500 images. We then choose $\mathcal{S}_{\mathrm{val}}$
with different sizes in the range of $[1,100]$ from the remaining 100 images.

We then use these splits to evaluate UDA methods and finetuning in the
following way. For UDA methods, we use $\mathcal{S}_{\mathrm{val}}$ for
selecting their hyper-parameters. For finetuning, we use
$\mathcal{S}_{\mathrm{val}}$ as a ‘training’ set despite its name. We use
$\mathcal{S}_{\mathrm{test}}$ for the evaluation of the UDA methods and
finetuning. To eliminate biases caused by a specific split, we iterate the
above procedure randomly and create the following sequence:

$(\mathcal{S}_{\mathrm{val}}^{(1)},\mathcal{S}_{\mathrm{test}}^{(1)}),(\mathcal{S}_{\mathrm{val}}^{(2)},\mathcal{S}_{\mathrm{test}}^{(2)}),\ldots,(\mathcal{S}_{\mathrm{val}}^{(N_{\mathrm{trial}})},\mathcal{S}_{\mathrm{test}}^{(N_{\mathrm{trial}})})$
(1)

and report the statistics (i.e., the average and variance) over
$N_{\mathrm{trial}}$ trials. We set $N_{\mathrm{trial}}=30$ and use an
identical sequence for all the methods.

###  4.2 Results

####  4.2.1 Sensitivity of UDA Methods to Hyper-parameters

We first consider RQ1 and examine the sensitivity of the above six UDA methods
to the selection of their hyper-parameters. We consider adaptation from GTA5
to Cityscape here.

While each method has a different number of hyper-parameters, we consider only
the primary ones for each for experimental efficiency. For IntraDA [13], we
consider $N_{\mathrm{iter}}$, the number of iterations to update the feature
space, and $\lambda$, a threshold for the entropy over class probabilities to
select the easy samples and hard samples in the target domain.  For
AdaptSegNet [20], we consider $N_{\mathrm{iter}}$ and the weight $w$ of the
auxiliary segmentation loss. For AdvEnt [21], we have $N_{\mathrm{iter}}$ and
initial learning rate $lr$. For CBST [28], we consider the initial proportion
$p_{0}$ of the pseudo label pixels and the proportion change $\Delta p$ of
every training round. For IAST [11], we consider the initial proportion
$\alpha$ of the pseudo label pixels and the momentum factor $\beta$ used to
preserve the past threshold information. For FADA [22], we consider the weight
of the adversarial loss $\lambda$ and the temperature factor $T$ for feature
scaling. The readers can refer to the original papers for more details.

Specifying a reasonable range for each hyper-parameter, we evaluate the
methods’ performance in a grid-search manner. The results confirm that these
methods are more or less sensitive to hyper-parameter’s selection. The maximum
and minimum performance values in mean IOU are as follows; IntraDA: 44.0 vs.
32.4, FADA: 46.8 vs. 30.0, AdaptSegNet: 42.2 vs. 10.1, IAST: 50.7 vs. 43.6,
AdvEnt: 43.7 vs. 6.9, and CBST: 46.9 vs. 45.3. More details are found in the
supplementary material.

####  4.2.2 Selecting Hyper-parameters Using Few Validation Samples

We next consider RQ2, i.e., how many labeled samples are necessary to find
sufficiently good hyper-parameters. We conduct the experiment as follows. For
each method, we consider a set of $M$ different hyper-parameter settings on
the grid in their ranges specified in the above experiment. We then perform
UDA with each settings, resulting in $M$ models. We then evaluate these $M$
models on the validation sets with different sizes, i.e.,
$\mathcal{S}_{\mathrm{val}}$’s with sizes
$|\mathcal{S}_{\mathrm{val}}|=1,2,5,10,50$, and $100$, to pick the best
performing one with each $\mathcal{S}_{\mathrm{val}}$, recording its
performance on the test set $\mathcal{S}_{\mathrm{test}}$.

Table 1 shows the results, in which we use the ratio of the average accuracy
with the best hyper-parameters chosen using each of the
$\mathcal{S}_{\mathrm{val}}$’s to that with the optimal hyper-parameters. In
other words, these ratios show the deterioration of accuracy due to
insufficient validation data.

We can observe from Table 1 that i) these UDA methods need more than a certain
number of validation images to ensure that at least near optimal hyper-
parameters are selected; using insufficient validation samples leads to large
variance in performance; ii) different methods show different sensitivity to
hyper-parameters; some methods are sensitive, whereas others are not. These
can be visually confirmed in Fig. 1, which will be explained below.

$|\mathcal{S}_{\mathrm{val}}|$ | 1 | 2 | 5 | 10 | 50 | 100
---|---|---|---|---|---|---
IntraDA [13] |  98.9$\pm$3.5 |  99.3$\pm$2.4 |  99.6$\pm$1.7 |  99.6$\pm$1.5 |  99.8$\pm$1.4 |  100$\pm$1.0
AdaptSegNet [20] |  95.8$\pm$1.9 |  96.8$\pm$1.8 |  98.8$\pm$1.0 |  99.3$\pm$0.6 |  99.8$\pm$0.5 |  100$\pm$0.3
AdvEnt [21] |  95.6$\pm$1.8 |  96.7$\pm$1.5 |  97.9$\pm$1.0 |  98.4$\pm$0.8 |  99.8$\pm$0.6 |  100$\pm$0.3
CBST [28] |  98.5$\pm$0.6 |  98.1$\pm$0.6 |  98.5$\pm$0.6 |  98.9$\pm$0.8 |  99.8$\pm$0.6 |  100$\pm$0.5
IAST [11] |  98.0$\pm$1.6 |  99.0$\pm$1.3 |  99.2$\pm$0.5 |  99.4$\pm$0.4 |  99.8$\pm$0.4 |  100$\pm$0.4
FADA [22] |  98.5$\pm$0.8 |  98.9$\pm$0.9 |  98.9$\pm$1.0 |  99.8$\pm$0.8 |  100$\pm$0.6 |  100$\pm$0.5
Table 1: Ratio (%) of performance (in mean IOU) of different UDA methods with
the best hyper-parameters selected using $\mathcal{S}_{\mathrm{val}}$ to their
highest accuracy.

####  4.2.3 Comparison of UDA Methods and Finetuning

We next consider RQ3, i.e., how effective simple finetuning is. Specifically,
we finetune the model that is pre-trained on the source-domain data. Note that
this is the same pre-trained model as that used in the UDA methods. We
finetune it using the labeled target-domain samples in
$\mathcal{S}_{\mathrm{val}}$ and repeat weight updates for a sufficiently
large number, 2,000888We report the results of a more rigorous procedure in
the supplementary material, i.e., splitting $\mathcal{S}_{\mathrm{val}}$ into
two sets and using one for fine-tuning and the other for determining the
iteration of weight updates.. Using $\mathcal{S}_{\mathrm{val}}$ with
different sizes $|\mathcal{S}_{\mathrm{val}}|\in\\{1,2,5,10,50,100\\}$, we
compare finetuning and UDA methods; finetuning uses
$\mathcal{S}_{\mathrm{val}}$ for training and UDA methods use
$\mathcal{S}_{\mathrm{val}}$ for obtaining the best hyper-parameters. We
consider two adaptation scenarios, GTA5 $\rightarrow$ Cityscapes and SYNTHIA
$\rightarrow$ Cityscapes.

Figure 1: Performance (mean IoU) of UDA methods and finetuning vs. the number
of labeled images (i.e., $|\mathcal{S}_{\mathrm{val}}|$). Upper: GTA5
$\rightarrow$ Cityscapes. Lower: SYNTHIA $\rightarrow$ Cityscapes.

Figure 1 show the results; the upper plot shows GTA5 $\rightarrow$ Cityscapes
and the lower one SYNTHIA $\rightarrow$ Cityscapes. We can observe the
following. Finetuning using a few training images (i.e.,
$|\mathcal{S}_{\mathrm{val}}|=1$ or 2) performs worse than the UDA
counterparts. It has a larger variance overall than UDA methods for a wide
range of the number of images. However, as the number of images increases,
finetuning starts to perform better than the UDA methods; it achieves on par
with the second-best UDA method at $|\mathcal{S}_{\mathrm{val}}|=5$. IAST is
much better than the other UDA methods, for which finetuning needs about 40
images to be better. We may conlclude that finetuning will be the first choice
if we have more than 50 images for the considered scenarios. Overall, the
results show the surprisingly strong performance of finetuning as a baseline
method for adapting a model to a new domain.

##  5 Summary and Conclusion

In this paper, we have reconsidered unsupervised domain adaptation (UDA) for
semantic segmentation from a data-centric perspective. We first questioned the
basic assumption of UDA that we have no access to labeled target domain
samples, pointing out the contradiction with the standard procedure of ML that
verifies the model’s performance before its deployment. Next, assuming a small
amount of labeled target-domain samples that are slightly more than necessary
for the validation, we considered how we utilize them for better inference.
Specifically, we first examined how many labeled samples are necessary and
sufficient for selecting good hyper-parameters of each previous UDA method. We
then examined finetuning using the same number of labeled samples. Finally, we
compared the two, i.e., UDA methods and the simple finetuning in an equal
condition in the number of labeled samples.

The experimental results lead to the following findings. Firstly, different
UDA methods show different sensitivity to the choice of hyper-parameters;
while some UDA methods need only a small number of samples (i.e., images), as
few as five, to choose the best hyper-parameters, this is not the case with
other methods. Secondly, the simple finetuning works surprisingly well even
with labeled samples of less than 100.

These results imply that future studies on UDA must consider finetuning as a
baseline to compare. In particular, they must first examine how many labeled
samples are necessary to adjust the hyper-parameters of the proposed method
and then show that it is superior to finetuning using the same (amount of)
labeled samples. While this study considers the specific adaptation scenarios
GTA5/SYNTHIA $\rightarrow$ Cityscapes, we think the conclusion will hold for
other scenarios. Recall that our finetuning method uses only labeled target-
domain samples; it leaves unlabeled target-domain samples untouched, which UDA
methods utilize. Using them additionally will undoubtedly lead to better
performance. However, it is unclear how to do it, where we need to consider a
smaller number of labeled samples than in previous SSDA studies, as mentioned
earlier. We will study this in the future.

## References

  * [1] Malik Boudiaf, Hoel Kervadec, Ziko Imtiaz Masud, Pablo Piantanida, Ismail Ben Ayed, and Jose Dolz.  Few-shot segmentation without meta-learning: A good transductive inference is all you need?  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13979–13988, 2021.
  * [2] Chaoqi Chen, Weiping Xie, Wenbing Huang, Yu Rong, Xinghao Ding, Yue Huang, Tingyang Xu, and Junzhou Huang.  Progressive feature alignment for unsupervised domain adaptation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 627–636, 2019.
  * [3] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille.  Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(4):834–848, 2017.
  * [4] Shuaijun Chen, Xu Jia, Jianzhong He, Yongjie Shi, and Jianzhuang Liu.  Semi-supervised domain adaptation based on dual-level domain mixing for semantic segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11018–11027, 2021.
  * [5] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3213–3223, 2016.
  * [6] Yaroslav Ganin and Victor Lempitsky.  Unsupervised domain adaptation by backpropagation.  In Proceedings of the International Conference on Machine Learning (ICML), pages 1180–1189, 2015.
  * [7] Muhammad Ghifary, W Bastiaan Kleijn, Mengjie Zhang, David Balduzzi, and Wen Li.  Deep reconstruction-classification networks for unsupervised domain adaptation.  In Proceedings of the European conference on computer vision (ECCV), pages 597–613, 2016.
  * [8] Rui Gong, Wen Li, Yuhua Chen, and Luc Van Gool.  Dlow: Domain flow for adaptation and generalization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2477–2486, 2019.
  * [9] Hsin-Ying Lee, Hung-Yu Tseng, Jia-Bin Huang, Maneesh Singh, and Ming-Hsuan Yang.  Diverse image-to-image translation via disentangled representations.  In Proceedings of the European conference on computer vision (ECCV), pages 35–51, 2018.
  * [10] Yunsheng Li, Lu Yuan, and Nuno Vasconcelos.  Bidirectional learning for domain adaptation of semantic segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6936–6945, 2019.
  * [11] Ke Mei, Chuang Zhu, Jiaqi Zou, and Shanghang Zhang.  Instance adaptive self-training for unsupervised domain adaptation.  In Proceedings of the European conference on computer vision (ECCV), pages 415–430, 2020.
  * [12] Pietro Morerio, Jacopo Cavazza, and Vittorio Murino.  Minimal-entropy correlation alignment for unsupervised deep domain adaptation.  arXiv:1711.10288, 2017.
  * [13] Fei Pan, Inkyu Shin, Francois Rameau, Seokju Lee, and In So Kweon.  Unsupervised intra-domain adaptation for semantic segmentation through self-supervision.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3763–3772, 2020.
  * [14] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), pages 8026–8037, 2019.
  * [15] Stephan R Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun.  Playing for data: Ground truth from computer games.  In Proceedings of the European conference on computer vision (ECCV), pages 102–118, 2016.
  * [16] German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M Lopez.  The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3234–3243, 2016.
  * [17] Kuniaki Saito, Donghyun Kim, Piotr Teterwak, Stan Sclaroff, Trevor Darrell, and Kate Saenko.  Tune it the right way: Unsupervised validation of domain adaptation via soft neighborhood density.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9184–9193, 2021.
  * [18] Masashi Sugiyama, Matthias Krauledat, and Klaus-Robert Müller.  Covariate shift adaptation by importance weighted cross validation.  Journal of Machine Learning Research, 8(5), 2007.
  * [19] Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B Tenenbaum, and Phillip Isola.  Rethinking few-shot image classification: a good embedding is all you need?  In Proceedings of the European conference on computer vision (ECCV), pages 266–282, 2020.
  * [20] Yi-Hsuan Tsai, Wei-Chih Hung, Samuel Schulter, Kihyuk Sohn, Ming-Hsuan Yang, and Manmohan Chandraker.  Learning to adapt structured output space for semantic segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7472–7481, 2018.
  * [21] Tuan-Hung Vu, Himalaya Jain, Maxime Bucher, Matthieu Cord, and Patrick Pérez.  Advent: Adversarial entropy minimization for domain adaptation in semantic segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2517–2526, 2019.
  * [22] Haoran Wang, Tong Shen, Wei Zhang, Ling-Yu Duan, and Tao Mei.  Classes matter: A fine-grained adversarial approach to cross-domain semantic segmentation.  In Proceedings of the European conference on computer vision (ECCV), pages 642–659, 2020.
  * [23] Xin Wang, Thomas E. Huang, Trevor Darrell, Joseph E Gonzalez, and Fisher Yu.  Frustratingly simple few-shot object detection.  In Proceedings of the International Conference on Machine Learning (ICML), 2020.
  * [24] Zhonghao Wang, Yunchao Wei, Rogerio Feris, Jinjun Xiong, Wen-Mei Hwu, Thomas S Huang, and Honghui Shi.  Alleviating semantic-level shift: A semi-supervised domain adaptation method for semantic segmentation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 936–937, 2020.
  * [25] Kaichao You, Ximei Wang, Mingsheng Long, and Michael Jordan.  Towards accurate model selection in deep unsupervised domain adaptation.  In Proceedings of the International Conference on Machine Learning (ICML), pages 7124–7133, 2019.
  * [26] Yangtao Zheng, Di Huang, Songtao Liu, and Yunhong Wang.  Cross-domain object detection through coarse-to-fine feature adaptation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13766–13775, 2020.
  * [27] Zhedong Zheng and Yi Yang.  Rectifying pseudo label learning via uncertainty estimation for domain adaptive semantic segmentation.  International Journal of Computer Vision, 2020.
  * [28] Yang Zou, Zhiding Yu, BVK Kumar, and Jinsong Wang.  Unsupervised domain adaptation for semantic segmentation via class-balanced self-training.  In Proceedings of the European conference on computer vision (ECCV), pages 289–305, 2018.
  * [29] Yang Zou, Zhiding Yu, Xiaofeng Liu, BVK Kumar, and Jinsong Wang.  Confidence regularized self-training.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 5982–5991, 2019.

Appendix

##  Appendix A Detailed Results of UDA Methods with Different Hyper-parameters

Figure 2, 3, and 4 show the full results of the six UDA methods with different
hyper-parameter settings for the experiment described in Sec. 4.2.1 in the
paper. The numbers in each grid represent the mean IoU value and standard
deviation by the corresponding hyper-parameter combinations obtained from 30
trials; darker colors show better performance.

Figure 2: Hyper-parameter sensitivity of IntraDA (upper) and FADA (lower).

Figure 3: Hyper-parameter sensitivity of AdaptSegNet (upper) and IAST (lower).

Figure 4: Hyper-parameter sensitivity of AdvEnt (upper) and CBST (lower).

##  Appendix B More Rigorous Evaluation of Finetuning

To avoid further complication, in the experiments reported in the main paper,
we fix the number of iterations for finetuning to a sufficiently large number,
2K. This configuration simulates the case where finetuning stably converges to
a solution as the number of epochs grows (to infinity). However, this is
arguably unfair in a strict sense; we may need to conduct a validation step,
in which we use a held-out set of labeled target domain data to select the
best epochs, as with UDA. Thus, we tested this in a supplementary experiment;
specifically, instead of using the entire $\mathcal{S}_{\mathrm{val}}$ for
training (i.e., finetuning), we split it into two without overlap, as
$\mathcal{S}_{\mathrm{val}}=\mathcal{S}^{\mathrm{FT}}_{\mathrm{train}}\cup\mathcal{S}^{\mathrm{FT}}_{\mathrm{val}}$;
we then finetune the model on $\mathcal{S}^{\mathrm{FT}}_{\mathrm{train}}$ and
choose the best epoch using $\mathcal{S}^{\mathrm{FT}}_{\mathrm{val}}$. The
data splitting scheme is summarized as follows:

S2: We initially follow the scheme in our main paper (Eq. 1). Namely, a set
$\mathcal{S}_{\mathrm{test}}$ consisting of 400 samples is first chosen from
the original validation set of Cityscapes. Then $\mathcal{S}_{\mathrm{val}}$
is chosen from the rest, which has a smaller size than 100. Next, we split it
into $\mathcal{S}^{\mathrm{FT}}_{\mathrm{train}}$ and
$\mathcal{S}^{\mathrm{FT}}_{\mathrm{val}}$ with different sizes such that
$\mathcal{S}_{\mathrm{val}}=\mathcal{S}^{\mathrm{FT}}_{\mathrm{train}}\cup\mathcal{S}^{\mathrm{FT}}_{\mathrm{val}}$
and
$\mathcal{S}^{\mathrm{FT}}_{\mathrm{train}}\cap\mathcal{S}^{\mathrm{FT}}_{\mathrm{val}}=\phi$.

A question is how to split $\mathcal{S}_{\mathrm{val}}$. We test several
splits of $\mathcal{S}_{\mathrm{val}}$ in its critical range recognized in the
main experiments, i.e., $|\mathcal{S}_{\mathrm{val}}|=\\{10,50,100\\}$.

The detailed setting of the experiments is as follows. For finetuning, we
train the model with $\mathcal{S}^{\mathrm{FT}}_{\mathrm{train}}$ for 5K
iterations, saving a checkpoint for every 500 iterations and select one with
the best performance on $\mathcal{S}^{\mathrm{FT}}_{\mathrm{val}}$ for
testing. For UDA methods, we employ the same procedure as before; their hyper-
parameters are selected using
$\mathcal{S}_{\mathrm{val}}=\mathcal{S}^{\mathrm{FT}}_{\mathrm{train}}\cup\mathcal{S}^{\mathrm{FT}}_{\mathrm{val}}$.
As in the previous experiments, we report the average and variance over trials
$i=1,\ldots,N_{\mathrm{trial}}$ with $N_{\mathrm{trial}}=30$.

Table 2 and Figure 5 show the results. We can see that a similar observation
to Sec. 4.2.3 holds. The finetuning method works better than the UDA methods
when $\lvert\mathcal{S}_{\mathrm{val}}\lvert\geq 50$ (and
$\lvert\mathcal{S}^{\mathrm{FT}}_{\mathrm{train}}\rvert\geq 40)$. In more
detail, finetuning performs on par with UDA methods but IAST at
$\lvert\mathcal{S}^{\mathrm{FT}}_{\mathrm{train}}\rvert=5$ and better with
$\geq 10$. It outperforms IAST at
$\lvert\mathcal{S}^{\mathrm{FT}}_{\mathrm{train}}\rvert=40$. While the overall
performance variance of finetuning is larger when
$|\mathcal{S}_{\mathrm{val}}^{\mathrm{FT}}|$ is very small (e.g., $=5$), the
variance may be acceptable when
$|\mathcal{S}_{\mathrm{val}}^{\mathrm{FT}}|\geq 10$.

Figure 5: Performance (mean IoU) of UDA methods and finetuning vs. the number of labeled images (i.e., ($|\mathcal{S}^{\mathrm{FT}}_{\mathrm{train}}|$, $|\mathcal{S}^{\mathrm{FT}}_{\mathrm{val}}|$)) on GTA5 $\rightarrow$ Cityscapes. Table 2: Performance (mean IoU) of UDA methods and finetuning with scheme 2. GTA5 $\rightarrow$ Cityscapes $(\lvert\mathcal{S}^{\mathrm{FT}}_{\mathrm{train}}\rvert,\lvert\mathcal{S}^{\mathrm{FT}}_{\mathrm{val}}\rvert)$ | (1, 9) | (2, 8) | (5, 5) | (5, 45) | (10, 40) | (40, 10) | (50, 50)
---|---|---|---|---|---|---|---
Finetuning |  37.4$\pm$3.2 |  41.4$\pm$2.6 |  45.8$\pm$2.4 |  46.0$\pm$1.6 |  48.2$\pm$1.2 |  51.9$\pm$1.4 |  53.7$\pm$1.5
AdaptSegNet [20] |  40.1$\pm$0.6 |  40.1$\pm$0.6 |  40.1$\pm$0.6 |  40.3$\pm$0.5 |  40.3$\pm$0.5 |  40.3$\pm$0.5 |  40.4$\pm$0.3
AdvEnt [21] |  42.3$\pm$0.8 |  42.3$\pm$0.8 |  42.3$\pm$0.8 |  42.9$\pm$0.6 |  42.9$\pm$0.6 |  42.9$\pm$0.6 |  43.0$\pm$0.3
IntraDA [13] |  44.2$\pm$0.5 |  44.2$\pm$0.5 |  44.2$\pm$0.5 |  44.6$\pm$0.3 |  44.6$\pm$0.3 |  44.6$\pm$0.3 |  44.7$\pm$0.2
FADA [22] |  45.0$\pm$0.8 |  45.0$\pm$0.8 |  45.0$\pm$0.8 |  45.7$\pm$0.6 |  45.7$\pm$0.6 |  45.7$\pm$0.6 |  45.7$\pm$0.5
CBST [28] |  46.1$\pm$0.6 |  46.1$\pm$0.6 |  46.1$\pm$0.6 |  46.7$\pm$0.6 |  46.7$\pm$0.6 |  46.7$\pm$0.6 |  46.8$\pm$0.5
IAST [11] |  49.5$\pm$1.6 |  49.5$\pm$1.6 |  49.5$\pm$1.6 |  50.4$\pm$0.4 |  50.4$\pm$0.4 |  50.4$\pm$0.4 |  50.5$\pm$0.4

##  Appendix C Detailed Comparison of UDA Methods and Finetuning

Although we provide the plots for the comparison of the UDA methods and
finetuning in the main paper (i.e., Fig. 1), Table 3 and 4 show their exact
mean IoU and standard deviation values on GTA5 $\rightarrow$ Cityscapes and
SYNTHIA $\rightarrow$ Cityscapes scenarios, respectively.

$|\mathcal{S}_{\mathrm{val}}|$ | 1 | 2 | 5 | 10 | 50 | 100
---|---|---|---|---|---|---
Finetuning |  34.7$\pm$3.5 |  40.8$\pm$2.4 |  45.6$\pm$1.7 |  47.8$\pm$1.5 |  52.7$\pm$1.4 |  55.9$\pm$1.0
AdaptSegNet [20] |  38.7$\pm$1.9 |  39.1$\pm$1.8 |  39.9$\pm$1.0 |  40.1$\pm$0.6 |  40.3$\pm$0.5 |  40.4$\pm$0.3
AdvEnt [21] |  41.1$\pm$1.8 |  41.6$\pm$1.5 |  42.1$\pm$1.0 |  42.3$\pm$0.8 |  42.9$\pm$0.6 |  43.0$\pm$0.3
IntraDA [13] |  44.2$\pm$0.5 |  44.4$\pm$0.5 |  44.5$\pm$0.5 |  44.5$\pm$0.5 |  44.6$\pm$0.3 |  44.7$\pm$0.2
FADA [22] |  45.0$\pm$0.8 |  45.2$\pm$0.9 |  45.2$\pm$1.0 |  45.6$\pm$0.8 |  45.7$\pm$0.6 |  45.7$\pm$0.5
CBST [28] |  46.1$\pm$0.6 |  45.9$\pm$0.6 |  46.1$\pm$0.6 |  46.3$\pm$0.8 |  46.7$\pm$0.6 |  46.8$\pm$0.5
IAST [11] |  49.5$\pm$1.6 |  50.0$\pm$1.3 |  50.1$\pm$0.5 |  50.2$\pm$0.4 |  50.4$\pm$0.4 |  50.5$\pm$0.4
Table 3: Performance (mean IoU) of UDA methods and finetuning on GTA5 $\rightarrow$ Cityscapes $|\mathcal{S}_{\mathrm{val}}|$ | 1 | 2 | 5 | 10 | 50 | 100
---|---|---|---|---|---|---
Finetuning |  30.1$\pm$4.8 |  36.8$\pm$3.4 |  42.8$\pm$2.1 |  46.7$\pm$1.7 |  54.5$\pm$1.0 |  58.3$\pm$0.9
AdaptSegNet [20] |  37.9$\pm$0.9 |  37.8$\pm$1.4 |  38.2$\pm$0.6 |  38.5$\pm$0.3 |  38.5$\pm$0.3 |  38.5$\pm$0.3
AdvEnt [21] |  37.7$\pm$1.6 |  37.7$\pm$1.7 |  38.7$\pm$0.4 |  38.9$\pm$0.3 |  38.8$\pm$0.3 |  38.9$\pm$0.2
IntraDA [13] |  42.3$\pm$0.6 |  42.2$\pm$0.6 |  42.6$\pm$0.6 |  42.7$\pm$0.5 |  42.9$\pm$0.4 |  43.0$\pm$0.3
FADA [22] |  38.5$\pm$1.1 |  38.9$\pm$0.9 |  39.2$\pm$0.6 |  39.2$\pm$0.6 |  39.4$\pm$0.3 |  39.4$\pm$0.3
CBST [28] |  36.7$\pm$1.6 |  37.6$\pm$1.3 |  38.7$\pm$1.1 |  39.0$\pm$1.2 |  39.5$\pm$0.3 |  39.6$\pm$0.3
IAST [11] |  45.9$\pm$3.0 |  48.5$\pm$2.9 |  49.4$\pm$2.2 |  50.4$\pm$0.6 |  50.6$\pm$0.4 |  50.7$\pm$0.4
Table 4: Performance (mean IoU) of UDA methods and finetuning on SYNTHIA
$\rightarrow$ Cityscapes


:

# Retrieving Evidence from EHRs with LLMs:
Possibilities and Challenges

Hiba<EMAIL_ADDRESS>
Northeastern University  Boston  MA  Denis Jered
<EMAIL_ADDRESS>
Northeastern University  Boston  MA  Jisoo Kim<EMAIL_ADDRESS>
Brigham and Women’s Hospital  Boston  MA  Christopher Potter
<EMAIL_ADDRESS>
Brigham and Women’s Hospital  Boston  MA  Geoffrey Young
<EMAIL_ADDRESS>
Brigham and Women’s Hospital  Boston  MA  Silvio Amir<EMAIL_ADDRESS>
Northeastern University  Boston  MA  Byron C. Wallace
<EMAIL_ADDRESS>
Northeastern University  Boston  MA

###### Abstract

Unstructured Electronic Health Record (EHR) data often contains critical
information complementary to imaging data that would inform radiologists’
diagnoses. However, time constraints and the large volume of notes frequently
associated with individual patients renders manual perusal of such data to
identify relevant evidence infeasible in practice. Modern Large Language
Models (LLMs) provide a flexible means of interacting with unstructured EHR
data, and may provide a mechanism to efficiently retrieve and summarize
unstructured evidence relevant to a given query. In this work, we propose and
evaluate an LLM (Flan-T5 XXL) for this purpose. Specifically, in a zero-shot
setting we task the LLM to infer whether a patient has or is at risk of a
particular condition; if so, we prompt the model to summarize the supporting
evidence. Enlisting radiologists for manual evaluation, we find that this LLM-
based approach provides outputs consistently preferred to a standard
information retrieval baseline, but we also highlight the key outstanding
challenge: LLMs are prone to hallucinating evidence. However, we provide
results indicating that model confidence in outputs might indicate when LLMs
are hallucinating, potentially providing a means to address this.

###### keywords:

NLP, LLMs, radiology

##  1 Introduction

We consider using Large Language Models (LLMs) as interfaces to unstructured
data (e.g., notes) in patient Electronic Health Records (EHRs), ultimately to
aid radiologists performing imaging diagnosis. The motivation here is that
unstructured evidence within EHR may support (or render less likely)
particular diagnostic hypotheses radiologists come to based on imaging alone,
but time constraints—combined with the often lengthy records associated with
individual patients—make manually finding and drawing upon such evidence
infeasible in practice. Consequently, radiologists often perform diagnosis
with comparatively little knowledge of patient history.

Modern LLMs offer a flexible mechanism to interface with unstructured EHR
data. For example, recent work has shown that LLMs can perform “zero-shot”
information extraction from clinical notes with reasonable accuracy (Agrawal
et al., 2022). In this work, we propose and evaluate an approach to extract
evidence from clinical notes in order to aid diagnosis. More concretely, we
envision a clinician first providing an initial suspected diagnosis as a
query. The model (an LLM) should then confirm whether there is unstructured
(textual) evidence in the patient record that might support this diagnosis,
and—if so—summarize this for the clinician. Figure 1 provides a schematic of
the approach.

LLMs provide an attractive mechanism to permit such interactions given their
flexibility and established dexterity working with unstructured text. We
anticipate—and empirically verify in this work—that they will therefore be
able to find and summarize evidence relevant to an arbitrary query more
capably than “traditional” (which is to say, pre-LLM) information retrieval
(IR) methods. Critically, they can also answer general questions (e.g., “Is
this patient at risk of _Atrial fibrillation_?”) and provide summaries of
supporting evidence identified; retrieval methods do not offer such
functionality. However, they also bring serious challenges: Skillful as they
are, it is well-known that LLMs are also prone to “hallucinating” content
(Azamfirei et al., 2023; Zhang et al., 2023).

We therefore perform an empirical evaluation with practicing radiologists to
assess both the potential benefits and risks of using LLMs to aid diagnosis.
The expert evaluations confirm that LLMs are more capable than a standard IR
system at surfacing and summarizing evidence relevant to a given diagnosis.
However, such models also bring inherent challenges. How can we know, e.g.,
that the summary of evidence supporting a given condition is in fact faithful
to the patient record at hand? We highlight several examples where the LLM
fabricates plausible patient history that _would_ support a condition of
interest. This is potentially dangerous and certainly frustrates the provider,
who must then read through the record carefully to ascertain that there is in
fact no such evidence, eliminating both the safety and efficiency benefits
hoped for from use of LLMs.

Our contributions are summarized as follows. (1) We introduce an approach in
which we task an LLM (specifically, Flan-T5 XXL; Chung et al. 2022) to assess
whether a patient is at risk of or has a given condition, and to produce a
conditional summary of any corresponding supporting evidence. We conduct
expert evaluation of this and find it considerably outperforms baseline
evidence retrieval approaches. (2) We show in-depth examples that highlight
key challenges to using LLMs for this task; hallucinated content. This points
to future research needed to address issues currently precluding the use of
LLMs as an interface to EHR.

##  2 Retrieving and summarizing evidence with LLMs

Figure 1: Proposed prompting strategy to identify and summarize evidence
relevant to a given query diagnosis using LLMs. We first ask if the patient
has (or is at risk of) a condition, then elicit a summary of supporting
evidence if so.

For a given query ($\equiv$ condition), we attempt to retrieve two distinct
types of evidence from patient history: (A) Snippets that indicate a patient
_may be at risk_ of developing the condition in the future, and (B) those that
suggest the patient _currently has_ the condition. For example, a patient on
anticoagulants after a recent posterior fossa surgery may be at risk of an
intracranial hemorrhage (but not experiencing one currently). By contrast,
observing acute posterior fossa hemorrhage indicates the patient most likely
has intracranial hemorrhage.

Extracting evidence for risk informs clinicians about occurrences in the
patient’s history (such as procedures, diagnoses) that make them more
vulnerable to the condition. Extracting evidence for signs of a condition
serves two purposes: signs that occur in the patient’s immediate history
indicate the patient likely has the condition. Signs that occur earlier
indicate the patient has a history of the condition which also serves as
important information.

We use Flan-T5 XXL as our base LLM. While larger, proprietary models may offer
superior results, we wanted to use an accessible LLM to ensure
reproducibility. Moreover, protections for patient privacy mandated by the
Health Insurance Portability and Accountability Act (HIPAA), and our
institutional policy on use of LLM restrict us to using models that can be
deployed “in-house”, precluding hosted variants (e.g., those provided by
OpenAI).

#### Zero-shot sequential prompting

We adopt a sequential prompting approach to find and summarize evidence. First
we ask the LLM whether a given note indicates that the corresponding patient
is at risk for, or has a given query diagnosis—this prompts the LLM for a
binary decision regarding these. When the answer is ‘Yes’, we prompt the model
to provide support for its response.

Specifically, to query whether the patient is at risk for the given diagnosis,
we use the prompts below.

‘Read the following clinical note of a patient: [NOTE].

Question: Is the patient at risk of [DIAGNOSIS]?
Choice -Yes -No.

Answer:

To elicit supporting evidence from the model for such risk predictions, we use
the following prompt.

‘Read the following clinical note of a patient: [NOTE].

Answer step by step: based on the note, why is the patient at risk of
[DIAGNOSIS]?

Answer:

Similarly, to query whether the patient _has_ a given diagnosis, we ask the
model instead “ Question: Does the patient have [DIAGNOSIS]?” (asking for a
binary response). And then to obtain evidence supporting this assessment (in
the case of a positive response), we prompt with: “Question: Extract signs of
[DIAGNOSIS] from the note.”.

In the above prompts, [NOTE] denotes a patient note, and [DIAGNOSIS] a
potential diagnosis for which we would like to retrieve supporting evidence.
We then combine and present the result for the two types of evidence (risks
and signs) to the end user.

#### Few-shot prompting

We also experimented with a single few-shot prompt to extract evidence (see
Appendix A), but preliminary results were not promising so we did not pursue
further.

#### A retrieval baseline

As a point of comparison for unsupervised evidence extraction, we use a simple
ranking approach using neural embeddings.111Other baselines (e.g., BM25, TF-
IDF) are possible, but the expert time needed for annotations limit our
ability. Specifically, given a query [DIAGNOSIS], we retrieve associated [RISK
FACTORS] using GPT-3.5 and generate an embedding $e_{\textrm{rf}}$ of the
sentence: ‘Risk factors of [DIAGNOSIS] include [RISK FACTORS]’ using
ClinicalBERT (Alsentzer et al., 2019).222Note that this does not entail
passing any sensitive data to OpenAI; we send only a condition name. Table 3
shows examples of risk factors provided by GPT-3.5. The intuition here is to
generate $n$-grams that are likely to indicate risk of the corresponding
diagnosis so that we can match these against notes in EHR. Then, for a patient
and [DIAGNOSIS], we retrieve the top $20$ sentences in the patient notes most
similar to $e_{\textrm{rf}}$. One downside of such a retrieval-based approach
is the need to pre-specify the number of evidence snippets to retrieve (here,
we arbitrarily set this to 20). By contrast, our approach implicitly adjusts
this threshold. We refer to this baseline as CBERT.

Diagnosis | Notes | Evidence
---|---|---
MIMIC |  |
intracranial hemorrhage* | $95$ | $29$
stroke | $16$ | $4$
small vessel disease | $16$ | $8$
pneumocephalus | $12$ | $12$
sinusitis | $49$ | $14$
total | 188 | 67
BWH |  |
small vessel disease | $13$ | $8$
chemoradiation necrosis | $18$ | $10$
demyelination | $21$ | $12$
brain tumor | $21$ | $20$
intracranial hypotension | $20$ | $20$
craniopharyngioma | $20$ | $18$
infarction | $14$ | $14$
sinus disease | $17$ | $15$
total | 144 | 117
Table 1: Evaluation dataset statistics. *intracranial hemorrhage, the only
diagnosis with more than one patient, has 4 patients.

##  3 Data

For evaluation we worked with radiologists (specializing in neuroimaging) from
the Brigham and Women’s Hospital in Boston (BWH). For experiments, we used a
private dataset from this hospital and the publicly available MIMIC-III
(Johnson et al., 2016) dataset, to ensure that our findings are robust and
(partially) reproducible.

BWH dataset comprises patients admitted to the Emergency Room (ER) of BWH
between 2010 and 2015 along with clinical notes including: cardiology,
endoscopy, operative, pathology, pulmonary, radiology reports, and discharge
summaries. We sampled patients who underwent brain imaging within 48 hours of
their ER visit because they are likely to have undetermined diagnoses. We are
interested in scenarios where patients are associated with a large volume of
EHR data, so we included patients with $\geq$10 EHR notes.

MIMIC-III is a publicly available database of deidentified EHR from patients
admitted to the Intensive Care Unit (ICU) of the Beth Israel Deaconess Medical
Center between $2001$ and $2012$. It contains both structured data (e.g,
demographics, vital sign measurements, lab test results), and unstructured
data (e.g., nurse and physician notes, ECG and radiology reports and discharge
summaries). Similar to the BWH dataset, we sampled patients that underwent
brain imaging within 48 hours of their ER or Urgent Care visit, whose EHR
included $\geq 10$ notes.

We sampled data for individual patients, but evaluated models with respect to
diagnoses. For example, if a patient report mentioned ‘stroke’ and
‘sinusitis’, the radiologist evaluated the surfaced evidence for each
condition independently. To reduce annotation effort, we discarded diagnoses
with more than 20 pieces of evidence and finally sampled 8 instances from each
source to create our final evaluation dataset. See Figure 2 for a schematic of
our data sampling procedure. Table 1 reports statistics about the set of
examples used for evaluation.

Figure 2: Data sampling flow-chart. An instance is a unique (patient,
diagnosis) combination. Figure 3: Screenshot of the evaluation interface
showing highlighted evidence.

##  4 Evaluation

For expert evaluation, one of the collaborating radiologists identified all
diagnoses discussed in the _Findings_ and _Impressions_ sections of the
radiology reports of 10 patients from each dataset (excluding MIMIC-III
patients from the pilot study).333While this is a relatively small number of
patients, we emphasize that manual evaluation is expensive: Radiologists on
our team spent $\sim$9 hours manually assessing outputs. Then, for each
diagnosis, we retrieved supporting evidence from all patient notes using the
zero-shot prompting strategy from Section 2. Three collaborating radiologists
then manually assessed each retrieved piece of evidence.

Figure 3 shows the evaluation interface that our radiologist team-members used
to assess model outputs. Because the relevance of an evidence snippet
inherently depends on the context, we ask radiologists to ground their
assessments by assuming the following hypothetical setting: “You are a
radiologist reviewing a scan of a patient in the ER. Based on the scan, you
are concerned that the patient has the diagnosis stated below. Assess the
relevance of the retrieved evidence to support your inference.” For each piece
of evidence surfaced by a model, radiologists answered two questions:

Is the evidence present in the note? LLMs can hallucinate evidence. Therefore,
we first ask radiologists to confirm whether the model generated evidence is
in fact supported by the note on the basis of which it was produced. To aid
the radiologists in finding the corresponding sentences, we compute
ClinicalBERT (Alsentzer et al., 2019) embeddings of sentences in the notes and
highlight those with a cosine similarity of $\geq 0.9$ with the ClinicalBERT
embedding of the generated evidence. This heuristic approach realizes high
precision but low recall. Therefore, if a highlighted sentence is incongruous
with generated evidence, we ask radiologists to read through the entire note
to try and manually identify support.

Note that the (non-generative) retrieval method to which we compare as a
baseline is extractive, and so incapable of hallucinating content; we
nevertheless ask this question with regards to the baseline for consistency
and to ensure blinding.

Is the evidence relevant? If the generated evidence is supported by the note,
we ask radiologists whether it is _relevant_ to the query diagnosis.
Specifically, we collect assessments on the following scale.

0: Not Useful The information is not useful; is irrelevant to the query
condition (e.g., ‘The patient is on a ventilator’ for pneumocephalus).

1: Weak Correlation The information surfaced has a plausible but probably weak
correlation with the query condition (e.g., ‘The patient is in a flutter’ for
intracranial hemorrhage).

2: Useful The retrieved evidence is relevant and may inform one’s diagnostic
assessment (e.g., ‘The patient has a TBI’ for intracranial hemorrhage).

3: Very Useful The retrieved evidence is clearly relevant and would likely
inform diagnosis (e.g., ‘hypertension’ for small vessel disease (SVD)).

To capture whether the two models we evaluated—one an LLM, the other a
retrieval approach—provide distinct evidence, we asked annotators if one model
surfaced relevant evidence that the other did not.

We also ask radiologists to choose which of the two models they subjectively
prefer and to offer any relevant comments. Finally, we record the time taken
to evaluate each piece of evidence.

##  5 Results

To first assess agreement between radiologists, we had all of them annotate
evidence surfaced by the LLM for one particular patient, selected at random
from the BWH dataset. For this patient, the model generated 10 pieces of
(potentially) relevant evidence for the query _chemoradiation necrosis_. On
this shared set, the inter-annotator agreement score (average pairwise Cohen’s
$\kappa$) for relevance assessments between the three radiologists was $0.68$.

To verify a given piece of evidence provided by a model it took radiologists
an average of $35.09\pm 49.24$ and $21.37\pm 24.36$ seconds for FLAN-T5 and
CBERT models, respectively. Evaluating FLAN-T5 demands more time because the
highlights are approximations (heuristically matched to generations) and so do
not always map correctly to the sentence corresponding to the evidence.
Evaluating hallucinations takes even longer ($129.22\pm 131.12$ seconds per
piece of evidence) because radiologists must peruse the note multiple times to
confirm that said evidence is not present.

Figure 4: Evidence generated by the LLM (FLAN-T5) is more often deemed (very)
useful than that retrieved by CBERT. But on average, $9.4\%$ of evidence
generated by FLAN-T5 is hallucinated.

Figure 4 shows results (see Table 2 for examples). Radiologists found $41.5\%$
(MIMIC) and $48.4\%$ (BWH) of the evidence generated by FLAN-T5 to be (very)
useful. In comparison, $34.4\%$ (MIMIC) and $39.7\%$ (BWH) of CBERT’s evidence
was (very) useful. $23.0\%$ (MIMIC) and $18.5\%$ (BWH) of the evidence
generated by FLAN-T5 showed weak correlations (discussed in Section 5.3).
$8.8\%$ (MIMIC) and $10\%$ (BWH) of evidence was hallucinated (discussed in
5.2). $26.7\%$ ((MIMIC) and $23.1\%$ (BWH) of the evidence was deemed not
useful; the latter was primarily true information about the patient’s
condition which was unrelated to the diagnosis.

###  5.1 Binary decision recall

As discussed above, we first ask the LLM whether a given note indicates that
the corresponding patient is at risk for, or has, a given query diagnosis. The
precision of this LLM inference is implicitly measured by the assessment of
generated evidence; if the patient does not have and/or is not at risk for a
given condition, any generated evidence will necessarily be irrelevant. But
this does not capture recall, i.e., how sensitive the model is with respect to
identifying when a patient indeed has or is at risk of a condition.

Therefore, to additionally estimate model _recall_ with respect to inferences
based on notes, we sampled $20$ patients from the BWH data and followed prior
work (McInerney et al., 2020) in terms of evaluation. Specifically, we asked
radiologists to browse reports from up to one year after the patient’s
reference radiology report and tag relevant diagnoses; these can then be
viewed as “future” diagnoses with respect to the original reference report.
The radiologists then selected past notes containing supporting evidence for
these diagnoses. Out of the $200$ notes marked as containing evidence, FLAN-T5
correctly identified $140$, corresponding to a recall of $0.7$.

fig:misc [Cross evidence similarity] [Language Model likelihood]

Figure 5: Distributions of two types of confidence scores, stratified by
expert assigned labels. We consider measuring (a) similarities between
multiple sampled outputs and (b) simply taking the normalized likelihood
assigned to sequences by the language model. Both confidence scores provide
good discrimination of “hallucinated” evidence, compared to all other types
(yielding AUCs of $>$0.9). They also both correlate with the rated usefulness
of evidence.

###  5.2 Hallucinations

Concerningly, some of the model hallucinations flagged by radiologists were
about risk factors that are highly relevant to the query diagnosis. We provide
a few illustrative examples:

Example 1 For a patient with demyelination as the query diagnosis, the model
hallucinated the evidence ‘axonal degeneration’. Demyelination is commonly
viewed as the primary factor responsible for the deterioration of axons within
multiple sclerosis lesions. The model also hallucinated signs of demyelination
as evidence (‘numbness and tingling in the arms and legs’). There was no
evidence in the record indicating axonal degeneration or either of these
symptoms.

Example 2 For a patient with chemo-radiation necrosis as the query diagnosis,
the model hallucinated that ‘the patient had a history of chemo-radiation
necrosis’. A history of radiation necrosis would be (very) relevant to its
diagnosis, but there was no such history in the EHR.

In other instances, the model hallucinated vague evidence such as ‘The patient
is taking a lot of medications that can cause small vessel disease’ for small
vessel disease as the query diagnosis (a radiologist went through the note and
was unable to find mention of any such medication).

#### Can hallucinations be identified by soft-matching the evidence back with
the note?

A possible way to identify hallucinations is to try to ‘match’ generated
evidence to text within patient notes. If so, this intuitively suggests the
generated content is also present in the underlying note. Practically, we do
this by generating embeddings of generated evidence texts and sentences in
notes and scoring their similarities; we can then consider a generation
supported if this similarity exceeds some threshold.

Note, however, that this approach may construe accurate evidence as
‘hallucinated’ if the the generation is comparatively abstractive. For
example, the model output ‘The patient has had multiple heart surgeries’ is
true, but has a low ClinicalBERT embedding similarity score with the
corresponding sentence ‘This is a 64 year-old gentelman with known coronary
artery disease status post an inferoposterior myocardial infarction in
[**2116**] who is status post redo-CABG times six and bovine AVR’ in the note.

Because abstraction—the ability to render relevant patient history
concisely—is a major benefit of LLMs, using heuristic matching to noisily
filter outputs (limiting to ostensibly ‘faithful’ cases) does not seem a
particularly promising approach.

#### How certain is the model about the hallucinations?

We evaluate the degree to which model uncertainty, quantified both via
normalized output likelihoods under the LM and using ‘self-consistency’ (Huang
et al., 2022; Si et al., 2022), can be used to infer when the model is
‘hallucinating’ content. The latter method entails eliciting ‘Chain-of-
Thought’ (CoT) reasoning (Wei et al., 2022) from models repeatedly, sampling
tokens at each step to yield multiple reasoning paths. The fraction of these
that resolve to the same final output (i.e., which are consistent) can then be
taken as a proxy for model certainty in that output. Outputs here are evidence
summaries, so we take the average similarity of sampled outputs as a proxy for
certainty (details in Appendix C).

We find that both methods provide confidence scores that are highly indicative
of hallucinations; this can be seen in Figure 5. This is promising, because it
suggests we might be able to simply abstain from providing outputs in such
cases.

###  5.3 Weakly correlating evidence

One factor complicating our evaluation is that the LLM often surfaced evidence
which might have a relatively weak correlation with the query condition. One
could argue that the model was ‘correct’ in retrieving such evidence from a
population epidemiology perspective, but incorrect from an individual patient
clinical perspective. In other words, the evidence is so weakly correlated
with the condition that it is minimally useful. (see Appendix E.1 for
examples).

###  5.4 Preferences

Figure 6: Model output preference counts, which indicate that radiologists
prefer the generative outputs to retrieved snippets.

Figure 6 shows how often radiologists preferred each model. FLAN-T5 provided
comparatively precise and concise output. Abstractive evidence was considered
better than the extractive snippets from CBERT, which often chunked useful
evidence with neighboring irrelevant sentences (notes are usually poorly
formatted, making sentence-parsing difficult). See Appendix E.1 for examples.
While radiologists largely preferred FLAN-T5 (XXL), CBERT was preferred in
three cases:

(1) Poor precision For pneumocephalus, $50\%$ of the evidence surfaced by
FLAN-T5 was not useful.

(2) Poor recall: In the case of chemoradiation necrosis, FLAN-T5 had a poor
recall and failed to retrieve essential evidence related to the patient’s
radiation therapy. This may be because of the term ‘chemoradiation’, which is
more commonly referred to as ‘radiation necrosis’. Changing the diagnosis to
‘radiation necrosis’, resulted in the model retreiving evidence related to the
patient’s radiation therapy.

(3) Supported alternate diagnosis: Interestingly, our radiologist preferred
CBERT for the case of demyelination because it helped confirm that the patient
did $not$ have demyelination, but in fact had a glioma (tumor). Demyelinating
lesions and glioma present similar imaging characteristics and can be
difficult to diagnose based on conventional MR imaging (Toh et al., 2012). A
brain biopsy is often conducted to differentiate between the two. All the
evidence evaluated as (very) useful were snippets from the pathology report
discussing the tests and related results that indicated that demyelination was
less likely and that the findings were most consistent with glioma.

Evaluation |  Diagnosis |  Evidence |  Explanation
---|---|---|---
Very Useful |  intracranial hemorrhage |  Recent fossa surgery and now on anticoagulants |  Surgery in the brain inevitably leaves some hemorrhage. Anticoagulants increase the risk of hemorrhage. ‘Recent surgery’ and ‘anticoagulants’ make hemorrhage highly likely.
Useful |  infarction |  There is calcified thrombus obstructing the origins of the M2 branches |  ‘Thrombus’ is diagnostic of infarction, which is very useful information. But ‘calcified thrombus’ implies chronicity, so the thrombus could have been present for a long time and there may not be an acute infarction at this time.
Weak Correlation |  pneumocephalus |  patient was involved in a motorcycle accident |  A traumatic head injury is an important risk factor of pneumocephalus. A motorcycle accident increases the likelihood of a head injury.
Not Useful |  small vessel disease (SVD) |  patient is at risk of endocarditis |  Not helpful in diagnosing SVD.
Hallucination |  intracranial hemorrhage |  patient has a brain tumor |  Not present in the note (and relevant to the diagnosis).
Table 2: Examples of evidence surfaced by FLAN-T5 for different evaluation
categories.

###  5.5 Robustness to query variation

As mentioned in Section 5.4, FLAN-T5 performed better when the diagnosis
‘chemoradiation necrosis’ was revised to the more commonly used ‘radiation
necrosis’. Another such case was ‘sinus disease’, more commonly known as
sinusitis. The model did well in this case; $80\%$ of the evidence was useful.
CBERT, however, surfaced evidence for cardiac sinus disease (sinus
bradycardia, sinus tachycardia, etc). Much of the evidence was related to the
patient’s cardiac history which would be relevant if they had cardiac sinus
disease. $90\%$ of the evidence was hence not useful.

##  6 Related Work

#### NLP for EHR.

Navigating EHRs is cumbersome, motivating several efforts in summarization of
and information extraction from EHR (Pivovarov and Elhadad, 2015). For
example, in recent related work, Jiang et al. (2023) created a proactive note
retrieval system based on the current clinical context to aid note-writing.
Adams et al. (2021) considered “hospital-course summarization”, aiming to
condense the notes of a patient visit into a paragraph, while Liang et al.
(2019) proposed to create disease-specific extractive summaries from clinical
notes.

#### NLP in Radiology.

Previous works regarding NLP in radiology primarily focus on processing
radiology reports. Some work has sought to generate the the clinical
Impression section based on the Findings section of reports (Van Veen et al.,
2023; Zhang et al., 2019; Sotudeh et al., 2020). Other efforts have focussed
on extracting specific observations from radiology reports (Smit et al., 2020;
Jaiswal et al., 2021), and modeled disease progression using radiology reports
(Di Noto et al., 2021; Khanna et al., 2023).

The prior works most relevant to this effort concern aiding radiologists in
diagnosing conditions. (McInerney et al., 2020) propose using a distantly
supervised model (trained to predict ICD codes) to perform extractive
summarization conditioned on a diagnoses. Our work addresses the problem in a
zero-shot setting. Tang et al. (2023) address diagnostic uncertainty by
suggesting less likely diagnosis to radiologists by learning to differentiate
between likely and less likely diagnoses via contrastive learning.

More broadly, recent work has shown the potential of ML and NLP to assist
radiologists across a range of tasks, including breast cancer screening (Wu et
al., 2020), malignant lung nodules detection (Sim et al., 2020), and chest
X-ray interpretation (Seah et al., 2021). Suggesting diagnosis (Tang et al.,
2023) based on findings and supporting it with evidence (McInerney et al.,
2020) also fall in this area.

##  7 Discussion and Limitations

We have proposed and evaluated a method for using LLMs to retrieve and
summarize evidence from patient records which might be relevant to a
particular condition or diagnosis of interest, with the ultimate aim of aiding
radiologists performing imaging diagnosis. Expert evaluations of model outputs
performed by radiologists suggest that this is a promising approach, in that
annotators tended to prefer LLM outputs to simple retrieval results.

But there are important limitations to the approach and to our evaluation. For
example, we found that LLMs are prone to hallucinating (plausible) evidence,
potentially hindering their utility for the envisioned use. However, our
results also suggest that confidence scores might allow one to pro-actively
identify hallucinations, and abstain in such cases. Interestingly, confidence
also seems to correlate with perceived usefulness. This suggests an
interesting direction to explore in future work.

Our evaluation was limited in a few key ways, which might reduce the
generalizability of our findings. First, we enlisted a small set of
radiologists to perform in-depth evaluation of a small number of instances.
This is because evaluation is time consuming: We re-emphasize that this
exercise required substantial allocation ($\sim$9 hours) of scarce expert
time. Another limitation here is that we considered only one LLM
(specifically, FLAN-T5): Other LLMs might, naturally, perform better or worse.
Finally, we did not extensively iterate on the specific prompts used, and this
too could substantially affect results.

## References

  * Adams et al. (2021) Griffin Adams, Emily Alsentzer, Mert Ketenci, Jason Zucker, and Noémie Elhadad.  What’s in a summary? laying the groundwork for advances in hospital-course summarization.  In _Proceedings of the conference. Association for Computational Linguistics. North American Chapter. Meeting_ , volume 2021, page 4794. NIH Public Access, 2021.
  * Agrawal et al. (2022) Monica Agrawal, Stefan Hegselmann, Hunter Lang, Yoon Kim, and David Sontag.  Large language models are zero-shot clinical information extractors.  _arXiv preprint arXiv:2205.12689_ , 2022.
  * Alsentzer et al. (2019) Emily Alsentzer, John Murphy, William Boag, Wei-Hung Weng, Di Jindi, Tristan Naumann, and Matthew McDermott.  Publicly available clinical BERT embeddings.  In _Proceedings of the 2nd Clinical Natural Language Processing Workshop_ , pages 72–78, Minneapolis, Minnesota, USA, June 2019. Association for Computational Linguistics.  10.18653/v1/W19-1909.  URL https://aclanthology.org/W19-1909.
  * Azamfirei et al. (2023) Razvan Azamfirei, Sapna R Kudchadkar, and James Fackler.  Large language models and the perils of their hallucinations.  _Critical Care_ , 27(1):1–2, 2023.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al.  Scaling instruction-finetuned language models.  _arXiv preprint arXiv:2210.11416_ , 2022.
  * Di Noto et al. (2021) Tommaso Di Noto, Chirine Atat, Eduardo Gamito Teiga, Monika Hegi, Andreas Hottinger, Meritxell Bach Cuadra, Patric Hagmann, and Jonas Richiardi.  Diagnostic surveillance of high-grade gliomas: towards automated change detection using radiology report classification.  In _Joint European Conference on Machine Learning and Knowledge Discovery in Databases_ , pages 423–436. Springer, 2021.
  * Honnibal and Montani (2017) Matthew Honnibal and Ines Montani.  spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing.  To appear, 2017.
  * Huang et al. (2022) Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han.  Large language models can self-improve.  _arXiv preprint arXiv:2210.11610_ , 2022.
  * Jaiswal et al. (2021) Ajay Jaiswal, Liyan Tang, Meheli Ghosh, Justin F Rousseau, Yifan Peng, and Ying Ding.  Radbert-cl: Factually-aware contrastive learning for radiology report classification.  In _Machine Learning for Health_ , pages 196–208. PMLR, 2021.
  * Jiang et al. (2023) Sharon Jiang, Shannon Shen, Monica Agrawal, Barbara Lam, Nicholas Kurtzman, Steven Horng, David Karger, and David Sontag.  Conceptualizing machine learning for dynamic information retrieval of electronic health record notes.  _arXiv preprint arXiv:2308.08494_ , 2023.
  * Johnson et al. (2016) Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark.  Mimic-iii, a freely accessible critical care database.  _Scientific data_ , 3(1):1–9, 2016.
  * Khanna et al. (2023) Sameer Khanna, Adam Dejl, Kibo Yoon, Quoc Hung Truong, Hanh Duong, Agustina Saenz, and Pranav Rajpurkar.  Radgraph2: Modeling disease progression in radiology reports via hierarchical information extraction.  _arXiv preprint arXiv:2308.05046_ , 2023.
  * Liang et al. (2019) Jennifer Liang, Ching-Huei Tsou, and Ananya Poddar.  A novel system for extractive clinical note summarization using ehr data.  In _Proceedings of the Clinical Natural Language Processing Workshop_ , pages 46–54, 2019.
  * McInerney et al. (2020) Denis Jered McInerney, Borna Dabiri, Anne-Sophie Touret, Geoffrey Young, Jan-Willem Meent, and Byron C Wallace.  Query-focused ehr summarization to aid imaging diagnosis.  In _Machine Learning for Healthcare Conference_ , pages 632–659. PMLR, 2020.
  * Pivovarov and Elhadad (2015) Rimma Pivovarov and Noémie Elhadad.  Automated methods for the summarization of electronic health records.  _Journal of the American Medical Informatics Association_ , 22(5):938–947, 2015.
  * Seah et al. (2021) Jarrel CY Seah, Cyril HM Tang, Quinlan D Buchlak, Xavier G Holt, Jeffrey B Wardman, Anuar Aimoldin, Nazanin Esmaili, Hassan Ahmad, Hung Pham, John F Lambert, et al.  Effect of a comprehensive deep-learning model on the accuracy of chest x-ray interpretation by radiologists: a retrospective, multireader multicase study.  _The Lancet Digital Health_ , 3(8):e496–e506, 2021.
  * Si et al. (2022) Chenglei Si, Zhe Gan, Zhengyuan Yang, Shuohang Wang, Jianfeng Wang, Jordan Boyd-Graber, and Lijuan Wang.  Prompting gpt-3 to be reliable.  _arXiv preprint arXiv:2210.09150_ , 2022.
  * Sim et al. (2020) Yongsik Sim, Myung Jin Chung, Elmar Kotter, Sehyo Yune, Myeongchan Kim, Synho Do, Kyunghwa Han, Hanmyoung Kim, Seungwook Yang, Dong-Jae Lee, et al.  Deep convolutional neural network–based software improves radiologist detection of malignant lung nodules on chest radiographs.  _Radiology_ , 294(1):199–209, 2020.
  * Smit et al. (2020) Akshay Smit, Saahil Jain, Pranav Rajpurkar, Anuj Pareek, Andrew Y Ng, and Matthew P Lungren.  Chexbert: combining automatic labelers and expert annotations for accurate radiology report labeling using bert.  _arXiv preprint arXiv:2004.09167_ , 2020.
  * Sotudeh et al. (2020) Sajad Sotudeh, Nazli Goharian, and Ross W Filice.  Attend to medical ontologies: Content selection for clinical abstractive summarization.  _arXiv preprint arXiv:2005.00163_ , 2020.
  * Tang et al. (2023) Liyan Tang, Yifan Peng, Yanshan Wang, Ying Ding, Greg Durrett, and Justin F Rousseau.  Less likely brainstorming: Using language models to generate alternative hypotheses.  _arXiv preprint arXiv:2305.19339_ , 2023.
  * Toh et al. (2012) CH Toh, K-C Wei, S-H Ng, Y-L Wan, M Castillo, and C-P Lin.  Differentiation of tumefactive demyelinating lesions from high-grade gliomas with the use of diffusion tensor imaging.  _American journal of neuroradiology_ , 33(5):846–851, 2012.
  * Turpin et al. (2023) Miles Turpin, Julian Michael, Ethan Perez, and Samuel R Bowman.  Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting.  _arXiv preprint arXiv:2305.04388_ , 2023.
  * Van Veen et al. (2023) Dave Van Veen, Cara Van Uden, Maayane Attias, Anuj Pareek, Christian Bluethgen, Malgorzata Polacin, Wah Chiu, Jean-Benoit Delbrouck, Juan Manuel Zambrano Chaves, Curtis P Langlotz, et al.  Radadapt: Radiology report summarization via lightweight domain adaptation of large language models.  _arXiv preprint arXiv:2305.01146_ , 2023.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in Neural Information Processing Systems_ , 35:24824–24837, 2022.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online, October 2020. Association for Computational Linguistics.  10.18653/v1/2020.emnlp-demos.6.  URL https://aclanthology.org/2020.emnlp-demos.6.
  * Wu et al. (2020) Nan Wu, Jason Phang, Jungkyu Park, Yiqiu Shen, Zhe Huang, Masha Zorin, Stanisław Jastrzębski, Thibault Févry, Joe Katsnelson, Eric Kim, Stacey Wolfson, Ujas Parikh, Sushma Gaddam, Leng Leng Young Lin, Kara Ho, Joshua D. Weinstein, Beatriu Reig, Yiming Gao, Hildegard Toth, Kristine Pysarenko, Alana Lewin, Jiyon Lee, Krystal Airola, Eralda Mema, Stephanie Chung, Esther Hwang, Naziya Samreen, S. Gene Kim, Laura Heacock, Linda Moy, Kyunghyun Cho, and Krzysztof J. Geras.  Deep neural networks improve radiologists’ performance in breast cancer screening.  _IEEE Transactions on Medical Imaging_ , 39(4):1184–1194, 2020.  10.1109/TMI.2019.2945514.
  * Zhang et al. (2023) Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A Smith.  How language model hallucinations can snowball.  _arXiv preprint arXiv:2305.13534_ , 2023.
  * Zhang et al. (2019) Yuhao Zhang, Derek Merck, Emily Bao Tsai, Christopher D Manning, and Curtis P Langlotz.  Optimizing the factual correctness of a summary: A study of summarizing radiology reports.  _arXiv preprint arXiv:1911.02541_ , 2019.

##  Appendix A Few-shot Prompting

We provide additional details on our preliminary experiments with few-shot
prompting. We randomly sampled $20$ patients from the MIMIC dataset and
followed McInerney et al. (2020)’s approach of using ‘future’ ICD code as
query diagnosis. We also randomly sampled ICD codes (excluding the patient’s
future diagnoses) as negative query diagnosis. We used the following prompt:
Read the following clinical note of a patient: [RANDOM NOTE SNIPPET].

Answer step by step: can the patient possibly have cardioembolic strokes in
the future?

Answer: There is no evidence. Final answer: No.

Read the following clinical note of a patient: patient stopped taking a blood
thinning medication required for a heart valve due to side effects.

Answer step by step: can the patient possibly have cardioembolic strokes in
the future?

Answer: The patient stopped taking a blood thinning medication required for a
heart valve. The medication thins the blood and prevents blood clots. Blood
clots can lead to strokes. Final answer: Yes.

Read the following clinical note of a patient: [NOTE].

Answer step by step: based on the note, why is the patient at risk of
[DIAGNOSIS]?

Answer:

We observed that with few-shot prompting the model surfaced evidence for
almost every diagnosis. However, some pieces of evidence were clearly wrong.
For a patient with _‘with g/j tube in place for gastroparesis’_ , the model’s
output for the negative diagnosis, encephalitis, was _‘The patient has a
jejunostomy tube in place. The jejunostomy tube can be pulled out. The
jejunostomy tube can be pulled out of the body. The jejunostomy tube can be
pulled out of the body and into the brain. Final answer: Yes’_. We suspect the
prompt biases the model to support the query diagnosis which then makes the
model generate incorrect explanations as evidence (Turpin et al., 2023). We
also experimented with prompts such as ‘Extract evidence for [DIAGNOSIS].
Output N/A if no evidence exists’ but the model then mostly generated _‘N/A’_.
Given these results, we carried the rest of the evaluation with the zero-shot
prompting approach.

##  Appendix B Retrieval Baseline

Table 3 shows examples of risk factors provided by GPT-3.5.

Diagnosis | Risk Factors
---|---
pneumocephalus |  head injury, skull fracture, neurosurgical procedures, sinus or mastoid surgery, meningitis, cerebrospinal fluid leak, barotrauma, diving or scuba diving accidents, iatrogenic causes, such as lumbar puncture or spinal anesthesia
stroke |  hypertension, smoking, diabetes, obesity, sedentary lifestyle, high cholesterol levels, atrial fibrillation, family history of stroke, previous history of stroke, excessive alcohol consumption, drug abuse.
intracranial hemorrhage |  hypertension, aneurysms, arteriovenous malformations, blood clotting disorders, trauma, drug abuse, liver disease, brain tumor, stroke, coagulopathy
brain tumor |  progression genetics, exposure to ionizing radiation, family history of brain tumors, certain hereditary conditions, weakened immune system, previous history of brain tumor.
infarction |  smoking, high blood pressure, high cholesterol, diabetes, obesity, sedentary lifestyle, family history of heart disease, excessive alcohol consumption, and drug abuse.
intracranial hypotension |  obesity, connective tissue disorders, previous spinal or cranial surgery, leaking cerebrospinal fluid, spinal epidural anesthesia, lumbar puncture or spinal tap
Table 3: Examples of risk factors provided by GPT-3.5 Data | Model | Hallucinations | Not Useful | Weak Correlations | Useful | Very Useful
---|---|---|---|---|---|---
MIMIC | FLAN | $0.088$ | $0.267$ | $0.230$ | $0.193$ | $0.222$
CBERT | $0.0$ | $0.444$ | $0.212$ | $0.269$ | $0.075$
BWH | FLAN | $0.100$ | $0.231$ | $0.185$ | $0.219$ | $0.265$
CBERT | $0.0$ | $0.392$ | $0.211$ | $0.228$ | $0.169$
Table 4: Evaluation results of FLAN-T5 and CBERT. This table corresponds to Figure 4. Diagnosis |  Evidence |  Explanation
---|---|---
intracranial hemorrhage |  patient had multiple cardiac surgeries |  Multiple cardiac surgeries may suggest anticoagulation or underlying cardiac dysfunction which could in turn predispose the patient to intracranial hemorrhage.
intracranial hypotension |  The patient has a ventriculoperitoneal shunt. |  A ventriculoperitoneal shunt (VPS) is a surgical device used to relieve intracranial pressure by draining excessive cerebrospinal fluid. Having a VPS catheter may increase the risk of intracranial hypotension due to over drainage.
craniopharyngioma |  s/p resection X2, s/p VPS and panhypopitutiarism with second resection |  Panhypopituitarism and the fact that something was removed through surgery suggests there was a tumor involving the sella which may or may not have been craniopharyngioma.
Table 5: Examples of weakly correlated evidence surfaced by the model for
different diagnosis queries. All have plausible but somewhat removed (or weak)
connections.

##  Appendix C Self-Consistency

In general, self-consistency calculates the “confidence” the model has in a
given output as the fraction of sampled reasoning paths that resolve to it
(out of all such paths; Si et al. 2022). Computing self-consistency is
straightforward when the final answer is from a fixed answer set or when an
exact string match serves as a good metric of consistency. Neither of the two
conditions, however, apply to diagnosis evidence extraction. Evidence
generation is open-ended; we therefore use cross-evidence similarity as a
confidence score.

Specifically, given a note $x$ and diagnosis $c$, we sample $m$ decoding
paths, $P=\\{p_{1},p_{2},...,p_{m}\\}$ with temperature$>$0 after prompting
for evidence for $c$ in $x$. We then compute cross-evidence similarity $s$ of
$P$ as:

$s=\frac{2}{|P|(|P|-1)}\sum_{\begin{subarray}{c}u,v\in P\\\ u\neq
v\end{subarray}}\phi(f_{u},f_{v})$ (1)

Where $\phi$ denotes cosine similarity and $f_{\\{.\\}}$ are feature
representations of evidence texts; we use ClinicalBERT to derive these. A low
similarity score $s$ suggests that the generations are diverse and therefore
dissimilar.

We sample $20$ decoding paths for each evidence using temperature sampling
($T=0.5$) and top-$k$ sampling ($k=40$). Figure 5 reports cross-evidence
similarity distributions associated with pieces of evidence marked as having
varying utility. The model is inconsistent and generates a variety of
unrelated evidence when hallucinating. Table 6 provides examples of consistent
and inconsistent evidence generation.

Consistent Evidence | Inconsistent Evidence
---|---
The patient has vascular calcifications |  The patient has a history of ischemic heart disease.
The patient has vascular calcifications. |  The patient is having heart rhythm problems.
vascular calcifications in the left upper quadrant of the abdomen |  The heart is at risk of ischemia.
vascular calcifications |  The patient is a non-smoker.
vascular calcifications in the left upper quadrant of the abdomen |  The patient has a pulmonary embolism.
The patient has vascular calcifications. |  The patient has a heart murmur.
The patient has a history of atrial fibrillation |  numbness and tingling in the arms and legs
The patient developed rapid atrial fibrillation. |  The patient is a 63 year old male who presents with a complaint of numbness and tingling in his left arm
The patient has developed rapid atrial fibrillation. |  JC/jp
The patient developed rapid atrial fibrillation. |  The patient had a white matter lesion of the brain
The patient has a history of atrial fibrillation. |  axonal loss
The patient developed rapid atrial fibrillation this morning. |  marrow swelling
Table 6: Examples of consistent and inconsistent evidence generation.

##  Appendix D Implementation Details

We used the HuggingFace (Wolf et al., 2020) library to run inference using
FLAN-T5 XXL and ClinicalBERT. We split notes into sentences using the spaCy
(en_core_web_sm) (Honnibal and Montani, 2017) library. We processed notes in
chunks of size $750$ tokens (including the prompt text) for FLAN-T5 and ran
8-bit inference. When generating evidence, we set max_tokens to $128$. When
computing cross-evidence similarity, we set max_tokens to $32$ as this was
sufficient to measure similarity across evidence.

##  Appendix E Results

Table 4 shows the evaluation results, corresponding to Figure 4.

###  E.1 Weak correlations

Table 5 shows examples of evidence evaluated to be weak correlations.

###  E.2 Evidence conciseness

FLAN-T5 | CBERT
---|---
CAD (s/p stents x 2, $>2$ MIs, on Coumadin INR=1.9) hx of $>3$ TIAs in past 2.5 yrs multiple AAAs (largest last measured at 5.5 cm, surg intervention held 2/2 cardiac status |  IMMUNIZATIONS: INFLUENZA VACCINE (INACTIVATED) IM Given [DATE] ALLERGY: AMOXICILLIN ADMIT DIAGNOSIS: Stroke PRINCIPAL DISCHARGE DIAGNOSIS ;Responsible After Study for Causing Admission) same OTHER DIAGNOSIS;Conditions,Infections,Complications,affecting Treatment/Stay CAD (s/p stents x 2, $>2$ MIs, on Coumadin INR=1.9) hx of $>3$ TIAs in past 2.5 yrs multiple AAAs (largest last measured at 5.5 cm, surg intervention held 2/2 cardiac status OPERATIONS AND PROCEDURES: None.OTHER TREATMENTS/PROCEDURES (NOT IN O.R.)
The patient has a high ICP and low CPP |  AT THIS TIME, NO FAMILY PRESENT NEURO- EYES DO NOT OPEN, PERTLA @ 2.0 VERY SLUGGISHLY, WITHDRAWS FROM PAIN WITH ALL 4 EXTREMITIES, MOVES EXTREMITIES UNPURPOSEFULLY ON BED, NO SEDATION GIVEN, OETT, VENTRICULOSTOMY OPEN TO CONTINUOUS DRAIN, ICP AS HIGH AS 29, CPP RUNNING LOW- BUT STABILIZING, GOALS-
The patient has a TBI |  A/P- S/P REPAIR [**Doctor Last Name **] & LL ORTHOPEDIC INJURIES STABLE TBI W/CLOSE MONITORING FOR CHANGES STABLE LIVER LAC AT PRESENT SUCCESSFULL WEAN/EXTUBATION POST-OP PAIN CONT TO MONITOR PER ORDERS- Q2/HR NEURO & PERIPHERAL VASCULAR CHECKS…?
Table 7: Examples of evidence when FLAN-T5 was more concise than CBERT.

Table 7 shows demonstrative examples of when FLAN-T5 was more concise than
CBERT.