#  Trade-offs in Fine-tuned Diffusion Models Between Accuracy and
Interpretability

Mischa Dombrowski1, Hadrien Reynaud2, Johanna P Müller1, Matthew Baugh2,
Bernhard Kainz1,2

###### Abstract

Recent advancements in diffusion models have significantly impacted the
trajectory of generative machine learning research, with many adopting the
strategy of fine-tuning pre-trained models using domain-specific text-to-image
datasets. Notably, this method has been readily employed for medical
applications, such as X-ray image synthesis, leveraging the plethora of
associated radiology reports. Yet, a prevailing concern is the lack of
assurance on whether these models genuinely comprehend their generated
content. With the evolution of text-conditional image generation, these models
have grown potent enough to facilitate object localization scrutiny. Our
research underscores this advancement in the critical realm of medical
imaging, emphasizing the crucial role of interpretability. We further unravel
a consequential trade-off between image fidelity – as gauged by conventional
metrics – and model interpretability in generative diffusion models.
Specifically, the adoption of learnable text encoders when fine-tuning results
in diminished interpretability. Our in-depth exploration uncovers the
underlying factors responsible for this divergence. Consequently, we present a
set of design principles for the development of truly _interpretable_
generative models. Code is available at https://github.com/MischaD/chest-
distillation.

## Introduction

Automatic detection and localisation of diseases in medical images have great
potential to be applied at scale because of the considerable amount of
available data that connects images to radiology reports. Recently, multi-
modal models have come into focus because of their ability to capture both,
textual and visual information and combine them to improve the performance of
the model (Li et al. 2022) and its interpretability (Huang et al. 2021).
Additionally, (Boecking et al. 2022) showed how vision-language models can
greatly benefit from prompt engineering because of the meaningful
representations learned by large language models such as (Devlin et al. 2018;
Yan et al. 2022; Liu et al. 2020). Furthermore, these cross-modality abilities
can also provide interpretable outputs, for example, by using contrastive
learning approaches (Zhang et al. 2022; Huang et al. 2021; Gupta et al. 2020)
to perform phrase grounding, which associates certain tokens or words of the
input prompt with regions in the image. Due to recent advances in diffusion
models, there has been an increased focus on generative approaches to solve
common problems, such as categorical data imbalances and counterfactual image
generation. Additionally, since their recent popularization by Ho, Jain, and
Abbeel (2020), diffusion models have propelled the performance of generative
models (Dhariwal and Nichol 2021; Saharia et al. 2021, 2022). These advances
have also been adopted in the medical domain, where most studies thus far have
focused on improving generative capabilities, such as generating large corpora
of MRI scans (Pinaya et al. 2022b) or 4D data (Kim and Ye 2022). Current
approaches have reached a level of performance where the models can generate
videos of such high quality that clinicians cannot visually distinguish them
from real videos (Reynaud et al. 2023). These successes also led to advances
in discriminative tasks such as anomaly detection (Pinaya et al. 2022a; Wolleb
et al. 2022). One common approach is to use pre-trained diffusion models
(Rombach et al. 2022) and fine-tune them to generate, for example, Chest X-ray
(CXR) images. The resulting image quality is better compared to training
diffusion models from scratch, as demonstrated in (Chambon et al. 2022). These
approaches have in common that they do not interpret the results given by the
diffusion models, which have recently demonstrated, particularly,
interpretable latent spaces (Dombrowski et al. 2022; Tang et al. 2022; Hertz
et al. 2022). Interpreting the latent space of diffusion models is of great
importance because generative models have to properly represent what the
tokens mean in order to generate correctly matching images from them. Even if
we validate the generated samples on pre-trained classifiers (Chambon et al.
2022), the results can be deceiving. Many models have not been evaluated on
robustness towards unrealistic artefacts introduced by generative models.
Furthermore, if classifiers are already able to clearly label images as
belonging to a class, it is unclear whether this sample is useful for
applications such as data augmentation.

In this paper, we show that the state-of-the-art methods for fine-tuning
diffusion models on radiology reports produce models which are no longer
interpretable. We do this by analyzing the influence of jointly learning
language embedding and image generation. Our experiments show that diffusion
models trained for high image quality have no semantic understanding of their
conditional input, but instead, learn to produce images based on confounders.
Our hypothesis is that jointly learning the text and image embedding is too
difficult and that adapting language models to medical tasks should be
approached with more care, including a thorough analysis of the
interpretability capabilities of the diffusion models. To show this, we
evaluate the phrase grounding potential of the diffusion model on the state-
of-the-art approach to fine-tune diffusion models. Then we empirically analyze
the effect of keeping the textual encoder frozen. This results in
statistically inferior generative performance, but retains the
interpretability of the model to such an extent that we beat discriminative
phrase grounding approaches on two out of eight diseases in a selected
clinical downstream scenario. Furthermore, we statistically evaluate the
significance of our findings. By demonstrating the effect of losing accuracy
to gain interpretability, this paper is the first work that unveils this
important trade-off within the domain of generative medical imaging models.

## Method

Figure 1: Visual summary of our findings. We investigate different ways of
finetuning multi-modal diffusion models. We show that by keeping the language
encoder frozen, visualized by the blue snowflake, we can train diffusion
models that are interpretable by evaluating the localization abilities of the
attention map.

Pretrained foundation model We begin with Stable Diffusion v2 (SDv2) (Rombach
et al. 2022) as a pre-trained foundation model. Using pre-trained models as a
starting point is a common approach to improve training time and, furthermore,
(Chambon et al. 2022) have shown that it produces better results in terms of
image quality compared to starting from a randomly initialized model. SDv2 is
a latent diffusion model, which means that the diffusion is learned on a
reduced image size $Z=64$. Following (Chambon et al. 2022), we keep the model
used to compute the latent dimension fixed. Consequently, we can precompute
the latent representation of our input images to speed up the learning time
and to fit the entire dataset into system memory. Since SDv2 is a text-to-
image diffusion model, it was trained to learn the function $f(y)$ that
generates images conditioned on textual input $y$. At training time, this
prompt could be “a photo of a swiss mountain dog”. In order to inject the
conditional input into the diffusion model, the text is first tokenized using
a tokenizer $T(y)$ and then embedded using a clip-like model (Radford et al.
2021), in this case, a “ViT-H-14” (Dosovitskiy et al. 2020). The tokenizer
maps words to integers as input for the textual encoder and has a fixed
vocabulary size determined at training time. Short input prompts are padded to
a fixed length of $T_{max}$, which is typically set to 77 at training and
inference time. Since the language embedder was trained on normal images and
captions, many words from the medical domain do not appear in this vocabulary.
The tokenizer solves this issue by splitting unknown words into multiple
tokens. “Consolidation” for example exists, whereas the word “Cardiomegaly” is
split into four tokens. To formalize this, we define
$\tau_{i}\coloneqq{\forall\tau:\tau\in T(y_{i})}$ as the set of tokens
corresponding to input word $i$. In the next step, all tokens are encoded
using a pre-trained language encoder. It is common to use a learnable language
encoder to simultaneously train the language encoder and the diffusion model,
as this results in higher image quality (Chambon et al. 2022).

Classifier-free guidance For image synthesis, we use classifier-free guidance,
which has been shown to produce better results in terms of image fidelity in
the medical domain (Chambon et al. 2022). Intuitively, this means that the
reverse diffusion step is performed twice, once for unconditional denoising
and once for conditional denoising. Then the difference is computed and
amplified to push the denoising step further into the direction of the
conditioning and improve the quality of the results. To improve the
performance of conditional denoising, we do unconditional guidance training, a
common technique that drops the conditional input for healthy samples. We
executed this in 30% of the “No Finding” cases.

Interpretability Following Zhang and Zhu (2018) we claim that interpreting
mid-layer representations of deep neural networks helps to interpret the
predictions of the model. In the case of text-conditioned diffusion models
this means that we analyze the interpretability by looking at their attention
layers following the approach proposed by Dombrowski et al. (2022).
Internally, every input token is converted to a learned latent embedding. This
embedding is inserted into the model at every level of the denoising U-Net.
The denoising U-Net itself consists of multiple channels and pixels with a
reduced width and height. Attention is computed at every layer as the dot-
product of the latent representation of the image with the embedding of the
token:

$\text{Attention}(Q,K,V)=\text{softmax}(\frac{QK^{T}}{\sqrt{d}})\cdot V$ (1)

where K and V are the learned projection representations of the input token, Q
is the learned projection of the input image, and d is a normalization
constant. This means that the layers produce output conditioned on input to
localize the relevant pixel-wise features. In the next step we can save the
attention maps of multiple reverse diffusion steps of all cross-attention
layers of the U-Net architecture and compute the mean over it. Attention maps
in downsampled layers are resized to the size $Z\times Z$. The resulting
attention map has the shape $\mathbb{R}^{T_{max}\times Z\times Z}$. Due to the
softmax operation, attention is defined as a probability and the values over
Tmax sum up to one. To get a class prediction, all we have to do, is to
compute the location of all tokens $\tau_{i}$ corresponding to this class and
compute the average. For in-domain data, this is straightforward as all the
words have a single token representation due to them being part of the
training dataset of the model. For unknown words, the tokenizer splits the
words into several known tokens. Therefore, we aggregate the attention over
all tokens corresponding to this word. The pipeline to analyze this type of
interpretability is summarized in Fig. 1.

## Datasets

In-domain data To fine-tune a diffusion model on in-domain data we take MS-
COCO (Lin et al. 2014), a giant dataset comprised of natural images in their
usual environment, and fine-tune on the captions of these images. We split the
official training set into 98169 train and 10702 validation images. We use the
official validation set as our testset as we do not have ground truth
segmentation labels available for the test set. If multiple captions are
available for a single image, we use all captions for training but randomly
sample one of them for the training steps. During validation we first filter
the dataset such that only images remain where the object class is part of the
caption to make the task of token-based localization easier. For example, for
the class of birds “two seagulls standing on a handrail” would be removed but
“two birds standing on a hand rail” would be left in. Since these captions are
short descriptions of the content of the image this step introduces a bias
towards only evaluating our model on images where the object is prominent.

Out-of-domain data To fine-tune the models on data from a different domain, we
use posterior-anterior (PA) and anterior-posterior (AP) views of MIMIC-CXR
(Johnson et al. 2019), a large corpus of paired text and image CXR data. For
evaluating localization results we use the phrases and bounding boxes of MS-
CXR, a subset of MIMIC-CXR with revised bounding boxes and impressions curated
by board-certified radiologists based on the original reports (Boecking et al.
2022) and therefore serve as a higher quality textual description of the image
content. Evaluation of multi-label images is done by splitting the image into
two instances, one for each disease class. To evaluate segmentation results,
we do the following preprocessing step: If the image has multiple bounding
boxes describing the same disease, we keep them together and provide the
results for the union of multiple bounding boxes. For phrase grounding
benchmarks, these are kept separate if the phrases for the bounding boxes are
different so that we can compare our method to other methods. To evaluate the
statistical significance of our findings we report the average results over
five runs. Following (Chambon et al. 2022), we only considered samples with
impressions that have over 7 characters for training. MS-CXR and the p19
subset of MIMIC are left out as a test set which results in a total of 162651
samples for the training set. The p19 subset is used to test the performance
of our proposed image generation method. We sample 5000 images from this
subset, reducing the number of “No Finding” samples to avoid reporting our
generative metrics on a subset consisting of predominantly healthy images.

### Evaluation metrics

To evaluate feature localization accuracy we use AUC-ROC, Top-1 accuracy and
contrast to noise ratio (CNR) (Boecking et al. 2022) as they allow quantifying
the performance without tuning thresholds. Top-1 in this context means the
accuracy of the highest predicted pixel being within the boundary of the
ground truth annotation. Intuitively this score is supposed to caption the
attention of the human observer who looks at the output and focuses on the
highest prediction value first. In practice, this value is weak against
outliers, _e.g._ if annotations in the corner of the image are detected as
outliers. To assess the generative performance we report the Fréchet inception
distance (FID) of 5000 images using the standard InceptionV3 and additionally
a domain-specific DenseNet121 from (Cohen et al. 2022) (FIDXRV) for the CXR
data. The images used for comparison are first resampled from the p19 test set
to limit the number of “No Finding” input conditions (For details see
Supplements). The sampling settings for the diffusion model follow (Chambon et
al. 2022). To assess the diversity of our samples, we use MS-SSIM (Wang,
Simoncelli, and Bovik 2003) on pairs of 4 images created with the same prompt,
averaged over a set of 100 prompts.

### Baseline Method

As the baseline method, we use Stable Diffusion v2, a text-to-image model
trained for image generation of natural images of size $512\times 512$
(Rombach et al. 2022). We perform all our fine-tuning experiments closely
following the recommendations from (Chambon et al. 2022) (FinetuneL), by
choosing a learning rate of $5\times 10^{-5}$, a batch size of 256 and
training for 60000 steps. When sampling, we use a classifier-free guidance
scale of 4 over 75 sampling steps using the PNDM sampler (Liu et al. 2022).
When keeping the language encoder frozen (FinetuneF), we observe that the
models perform better after only 30000 training steps, due to the simplified
training objective and fewer parameters. We put ablations on this in the
supplementary material. Our approach differs from (Chambon et al. 2022)
because we do not exclude AP views from our evaluation. The reason behind this
is that the MS-CXR test set contains both AP and PA views. Furthermore, we
refrain from limiting the number of healthy samples in our training set. All
results are calculated after applying resizing and center-cropping to get to
an image size of $512\times 512$. We split our fine-tuning over 16 80GB A100
GPUs which takes roughly 240 GPU hours with the frozen, and 580 with the
learnable language encoder.

## Results

Category | SDv2 |  | FinetuneL |  | FinetuneF (Ours)
---|---|---|---|---|---
AUC $\uparrow$ | Top-1 $\uparrow$ | CNR $\uparrow$ |  | AUC $\uparrow$ | Top-1 $\uparrow$ | CNR $\uparrow$ |  | AUC $\uparrow$ | Top-1$\uparrow$ | CNR $\uparrow$
Atelectasis | 58.5 | 0.0 | 0.23 |  | 45$\pm$4.9 | 2$\pm$1.6 | -0.15$\pm$.13 |  | 86$\pm$0.9 | 80$\pm$3.3 | 1.25 $\pm$.06
Cardiomegaly | 53.1 | 8.4 | 0.14 |  | 45$\pm$1.6 | 2$\pm$0.8 | -0.16$\pm$.05 |  | 73$\pm$2.3 | 40$\pm$6.8 | 0.67 $\pm$.07
Consolidation | 59.3 | 2.6 | 0.21 |  | 53$\pm$4.8 | 14$\pm$4.3 | 0.05 $\pm$.11 |  | 86$\pm$1.6 | 48$\pm$7.5 | 1.22 $\pm$.10
Edema | 80.2 | 15.2 | 0.37 |  | 54$\pm$1.6 | 29$\pm$8.2 | 0.09 $\pm$.02 |  | 89$\pm$0.1 | 70$\pm$3.8 | 1.36 $\pm$.01
Lung Op. | 64.7 | 1.2 | 0.87 |  | 53$\pm$3.8 | 7$\pm$4.6 | 0.05 $\pm$.09 |  | 83$\pm$0.2 | 38$\pm$1.9 | 1.06 $\pm$.01
Pl. Effusion | 48.8 | 1.0 | 0.34 |  | 40$\pm$4.7 | 3$\pm$1.6 | -0.27$\pm$.12 |  | 83$\pm$1.1 | 66$\pm$3.8 | 1.05 $\pm$.06
Pneumonia | 61.3 | 0.5 | 0.64 |  | 49$\pm$2.4 | 3$\pm$1.9 | -0.05$\pm$.05 |  | 85$\pm$0.9 | 59$\pm$4.5 | 1.16 $\pm$.04
Pneumothorax | 71.0 | 10.2 | 0.01 |  | 81$\pm$6.9 | 27$\pm$26.5 | 0.97 $\pm$.33 |  | 74$\pm$2.7 | 20$\pm$3.4 | 0.68 $\pm$.08
Average | 60.6 | 5.7 | 0.321 |  | 54$\pm$3.7 | 10$\pm$7.4 | 0.133$\pm$.12 |  | 79$\pm$0.5 | 45$\pm$1.5 | 0.920$\pm$0.03
Table 1: AUC-ROC and Top-1 accuracy for the phrase grounding benchmark of
different chest diseases on MS-CXR. We compare our approach compared to SDv2
(Rombach et al. 2022) and FinetuneL (Chambon et al. 2022). Mean and standard
deviation over three different trainings are given. The results show that
training with a frozen language encoder significantly improves the
localization performance throughout all metrics for seven out of eight disease
clasess.

### In-domain Fine-tuning

First, we evaluate our method with respect to segmentation results on MS-COCO
data. We argue that this data is much closer to the training dataset of SDv2
as this model was trained to be a general-purpose text-to-image model.
Therefore, we first analyze whether the finetuning process itself destroys the
ability of the generative model to be interpretable, especially after
finetuning them on smaller compute clusters. The original model (Rombach et
al. 2022) was trained on a batch size of 2048, which enables the model to
learn a joint representation of the text and the image and therefore enables
interpretation of the latent space (Dombrowski et al. 2022). In Tab. 2 we
show, that finetuning on in-domain data remains possible even on a single
Nvidia A100 GPU and a batch size of 16.

Trade-off of learnable vs frozen encoder Interestingly, we can see that the
FID improves over time for both the learnable and the frozen encoder. However,
in the case of the learnable encoder, the task of jointly learning language
embedding and conditional image generation was too hard and therefore, the FID
got worse first. We can also see this in terms of interpretability measured in
AUC-ROC. The initial value is way worse than the pretrained baseline and the
frozen model. However, eventually the interpretability and synthetic quality
get better. The Frozen finetuning works better, with an improved FID score
compared to the baseline even for a low number of steps. Further finetuning
the model leads to the best FID score and a small improvement in terms of AUC-
ROC. Interestingly, in terms of segmentation, the pretrained SDv2 model is the
best out of all of them. Partially, this can be explained by the training
dataset. SDv2 was trained on a subset of the LAION-5B dataset (Schuhmann et
al. 2022), which is a huge agglomeration of natural images with text
descriptions scraped from the internet. It is likely that a huge chunk of MS-
COCO is already part of the training data. Additionally, the huge variability
in the training dataset of SDv2 helps the model to understand concepts better
and the result is marginally higher interpretability, even if we compare it to
a model finetuned to a more specific dataset.

Method |  Steps | Frozen | FID $\downarrow$ | AUC-ROC $\uparrow$
---|---|---|---|---
SDv2 |  0 | - | 26.1 | 91.8
Finetune |  10000 | ✗ | 29.5 | 79.0
150000 | ✗ | 25.0 | 90.1
10000 | ✓ | 25.8 | 91.3
150000 | ✓ | 19.4 | 91.5
Table 2: MS-COCO finetuning experiment: We compare the generative and object
segmentation performance of the two suggested finetuning models. Keeping the
language encoder frozen leads to better interpretability and faster
convergence.

### Localization Results

Next, we want to assess the interpretability of these models when training
them on data of new domains. We do this by looking at localization metrics
computed on the impressions from MS-CXR (Boecking et al. 2022). Choosing
tokens $\tau_{i}$ to attend to is not straightforward for textual conditioning
on medical data. Hence, in order to compare with phrase-grounding approaches,
if the name of the disease, or some altered version of it appears in the
impression, we use the attention maps of this token as the prediction. If this
does not occur, we compute the attention for the tokens of all the words,
excluding tokens indicating the start of a string, the end of a string, and
padding. Manually choosing tokens in this case, _e.g._ , the token for “heart”
to localize Cardiomegaly could potentially boost the performance even further,
but we avoid this for the sake of generalizability. Since we observed limited
localization results for the learnable approach, we do not compute the
absolute value of CNR in order to not wrongly overestimate the performance.

Quantitative results are shown in Tab. 1. SDv2 generally has a bad
localization performance, although some classes had better scores than
expected. One possible explanation is that some diseases have correlations
with certain regions, like the regions with medical equipment, lungs, or text
labels, that SDv2 often puts higher attention to.

FinetuneL is not able to perform localization consistently. Its localization
performance only reaches 54% AUC-ROC which is even worse than the SDv2
baseline and 10% Top-1 accuracy, which is slightly better. Overall, this
indicates that this model has not acquired any understanding of the disease-
defining features. Interestingly, the localization for “Edema”,
“Consolidation” and “Pneumothorax” has a decent Top-1 Accuracy and the CNR
results for “Pneumothorax” shows the best AUC-ROC out of all methods. However,
the large standard deviation indicates that the understanding of this disease
is not robust and might be related to random artefacts.

Trade-off of learnable vs frozen encoder Keeping the language encoder frozen
during fine-tuning, on the other hand, shows excellent results in terms of
localization. It achieves 79% AUC-ROC, 45% Top-1 accuracy, and 0.920 CNR
across all diseases. The attention maps are therefore satisfactory indicators
for the location of the disease, and we infer that the model has learned to
localize those conditions. Fig. 2 confirms our qualitative observations by
showing a considerable gap in the interpretability of the two different
diffusion models. FinetuneL mistakenly relates $\tau_{i}$ to image features,
such as rips, spine, or bones, that are unrelated to the disease class. Our
proposed approach, on the other hand, shows a good localization performance,
including multi-instance input samples.

Figure 2: Phrase grounding examples from MS-CXR. White labels show which bounding boxes are ground truth. Pixels with higher importance are shown in red. Weakly |  Supervised |  Method | Atel. | Card. | Cons. | L-Op. | Edem. | Pnem. | Pnth. | P-Ef. | Avg.
---|---|---|---|---|---|---|---|---|---|---|---
ConVIRT (Zhang et al. 2022) | 0.86 | 0.64 | 1.25 | 0.78 | 0.68 | 1.03 | 0.28 | 1.02 | 0.818
GLoRIA (Huang et al. 2021) | 0.98 | 0.53 | 1.38 | 1.05 | 0.66 | 1.18 | 0.47 | 1.20 | 0.930
BioViL-L (Boecking et al. 2022) | 1.17 | 0.95 | 1.45 | 1.19 | 0.96 | 1.19 | 0.74 | 1.50 | 1.142
Zero- |  shot |  SDv2 (Rombach et al. 2022) | 0.23 | 0.14 | 0.21 | 0.37 | 0.87 | 0.34 | 0.64 | 0.01 | 0.321
FinetuneL (Chambon et al. 2022) | -0.15 | -0.16 | 0.05 | 0.05 | 0.09 | -0.05 | 0.97 | -0.27 | 0.133
FinetuneF (Ours) | 1.25 | 0.67 | 1.22 | 1.06 | 1.36 | 1.16 | 0.68 | 1.05 | 0.920
Table 3: Phrase grounding results for contrast-to-noise ratio (CNR) on MS-CXR
of our approach compared to state-of-the-art weakly supervised phrase
grounding results. Values for ConVIRT, GLoRIA, and BioViL-L are taken from
(Boecking et al. 2022). Despite being generative in nature, interpreting the
attention layers of the diffusion model and using them for phrase grounding
reaches a similar performance as weakly supervised approaches.

Phrase grounding benchmarks To substantiate the performance of our method, we
can compare it with phrase grounding benchmarks. These methods were trained
using contrastive methods and were specifically designed with the task of
localization in mind. Our method, on the other hand, provides these
localization maps without additional effort, and they can therefore be used
for localization in a zero-shot manner. As it can be seen in Tab. 3, our best
method outperforms the discriminative approaches in terms of localization in
two out of eight disease classes, which is remarkable, given that our model is
_only_ generative by nature. The learnable method, once again, is the worst
method out of all generative approaches, with slightly worse results than our
SDv2 baseline.

Method | FID $\downarrow$ |  | FIDXRV $\downarrow$ |  | MS-SSIM $\downarrow$
---|---|---|---|---|---
Mimic | MS-CXR |  | Mimic | MS-CXR |  | Mimic | MS-CXR
SDv2 (Rombach et al. 2022) |  237.6 | 236.2 |  | 104.9 | 109.4 |  | 12.9 | 11.1
FinetuneL (Chambon et al. 2022) |  61.9 | 60.5 |  | 7.7 | 7.3 |  | 10.3 | 11.4
FinetuneF (Ours) |  75.5 | 75.7 |  | 10.1 | 10.0 |  | 10.1 | 8.2
Table 4: Quantitative comparison of fidelity in terms of general FID and
domain-specific FIDXRV score and diversity of image generation in terms of MS-
SSIM (in %). We compare the results obtained from using impressions from Mimic
or from MS-CXR as our prompts for generation.

### Generative Results

Next, we evaluate the generative quality of the different models. Table 4
shows the quantitative results of the fine-tuning. The fidelity scores of the
models follow the observations reported in (Chambon et al. 2022) that the
results are drastically improved by making the textual encoder learnable
during fine-tuning, despite the evidence that the model has a worse
understanding of what individual tokens mean. To investigate the correctness
of the conditional generation, we evaluate the prediction AUC-ROC of a pre-
trained classifier, similar to the approach chosen by (Chambon et al. 2022).
Even though no ground-truth labels are available for the generated images, it
indicates whether the generated artefacts resemble the ones used by trained
classifiers. For that, we use the label predictions of the previously used
DenseNet121 on our generated images and compare them with the predictions of
the same label for the generated healthy images. As a baseline, we take the
classification accuracy on 5000 real images from the p19 subset. Table 5 shows
the results. We can conclude that the class conditional generation follows the
same trend as the unconditional generation. The best model is the one that
jointly trains language embedding and image generation. However, the
difference between both models is very small compared to the FID scores. In
fact, FinetuneF even outperforms the other method in six out of eight classes.
The only exception are “Pneumothorax”, which was also the only disease class
that had good AUC-ROC performance, and “Edema”, which was the only class with
good Top-1 accuracy, as shown Tab. 1.

Method |  Source | Atel. | Card. | Cons. | L-Op. | Edem. | Pnem. | Pnth. | Effu. | Avg.
---|---|---|---|---|---|---|---|---|---|---
p19 Test |  Real | 83.0 | 85.5 | 87.7 | 80.2 | 91.4 | 75.6 | 84.4 | 88.7 | 84.6
FinetuneL (Chambon et al. 2022) |  Synthetic | 75.8 | 78.7 | 77.8 | 75.1 | 85.9 | 66.2 | 79.6 | 79.6 | 77.3
FinetuneF | 76.8 | 79.0 | 77.9 | 77.5 | 84.1 | 66.7 | 70.1 | 84.5 | 77.1
Table 5: AUC-ROC of a pre-trained classifier (Cohen et al. 2022) on real
samples (p19) compared to the accuracy on synthetic images. The metrics are
computed as the average over the scores from all diseases. Results for Pleural
Effusion are reported using the prediction for Effusion since the classifier
does not distinguish between different types of effusions.

Lack of interpretability To investigate where this lack of interpretability
comes from we look for every word that occurs more than 25 times in the test
set, how good the localization is and report the three best words and values
for all three models in Tab. 6. Interestingly, we can see that FinetuneL has
good values for some of the words in the dataset but these words are never
consistent with the diseases. The only exception is the end of string token
which has a good localization performance throughout all diseases. Due to the
language encoding, which is causal, tokens in the future are conditioned on
tokens before that. Therefore, the end of string token has learned to relate
the text to the location in which the important information is located.
Overall the end-of-string (EOS) token of FinetuneL has a CNR of 0.68 which is
still a bit worse than EOS for FinetuneF with 0.72 and far from the
performance of FinetuneL conditioned on the disease describing words. For
phrase grounding this method of localization is not feasible as it fails if
parts of the phrase describe different regions of the image.

Category | SDv2 (Rombach et al. 2022) |  | FinetuneL (Chambon et al. 2022) |  | FinetuneF
---|---|---|---|---|---
Word | Occ. | CNR |  | Word | Occ. | CNR |  | Word | Occ. | CNR
Atelactasis | EOS | 61 | 0.37 |  | EOS | 61 | 0.80 |  | atelectasis | 50 | 1.11
atelectasis | 50 | 0.25 |  | SOS | 61 | -0.17 |  | EOS | 61 | 0.85
SOS | 61 | -0.17 |  | atelectasis | 50 | -0.23 |  | SOS | 61 | -0.85
Cardiomegaly | EOS | 333 | 0.24 |  | EOS | 333 | 0.37 |  | enlarged | 222 | 1.16
cardiac | 220 | 0.21 |  | SOS | 333 | 0.19 |  | silhouette | 219 | 0.92
silhouette | 219 | 0.16 |  | moderate | 30 | -0.09 |  | moderate | 30 | 0.84
Consolidation | lung | 56 | 0.82 |  | EOS | 117 | 0.99 |  | lung | 56 | 1.41
lobe | 32 | 0.61 |  | patchy | 60 | 0.11 |  | consolidation | 93 | 1.34
patchy | 60 | 0.58 |  | left | 44 | 0.06 |  | left | 44 | 1.26
Edema | pulmonary | 39 | 1.20 |  | EOS | 46 | 0.90 |  | pulmonary | 39 | 1.25
EOS | 46 | 0.73 |  | pulmonary | 39 | 0.18 |  | edema | 42 | 1.20
edema | 42 | 0.53 |  | edema | 42 | 0.13 |  | EOS | 46 | 0.96
Lung opacity | lung | 35 | 0.75 |  | EOS | 82 | 0.97 |  | lung | 35 | 1.33
ground-glass | 32 | 0.60 |  | the | 43 | 0.10 |  | EOS | 82 | 1.06
patchy | 33 | 0.55 |  | right | 32 | 0.09 |  | opacity | 28 | 1.00
Effusion | SOS | 96 | 0.11 |  | EOS | 96 | 0.52 |  | left | 31 | 0.84
EOS | 96 | 0.03 |  | SOS | 96 | 0.03 |  | pleural | 88 | 0.74
left | 31 | -0.12 |  | Small | 39 | -0.20 |  | effusion | 51 | 0.71
Pneumonia | lung | 46 | 0.72 |  | EOS | 182 | 0.89 |  | lung | 46 | 1.33
lobe | 49 | 0.68 |  | right | 58 | -0.01 |  | consolidation | 31 | 1.26
EOS | 182 | 0.45 |  | opacities | 30 | -0.01 |  | lobe | 49 | 1.26
Pneumothorax | small | 70 | 0.73 |  | small | 70 | 0.95 |  | small | 70 | 0.70
pneumothorax | 226 | 0.58 |  | left | 85 | 0.70 |  | apical | 119 | 0.66
apical | 119 | 0.54 |  | apical | 119 | 0.69 |  | pneumothorax | 226 | 0.46
Table 6: Number of occurrences and CNR for the three words with the highest
localization accuracy of each model on MS-CXR. SOS and EOS are start and end
of string respectively. The EOS string token shows consistently high
localization performance for both fine-tuned models. However, the frozen model
is the only model that learns the connection between the token describing the
disease and its localization.

### Influence of Complexity of Text Input

Finally, we want to understand the influence of the text data on the
interpretability. Given the results of Tab. 1 we hypothesize that the
learnable models struggle with the large variability and the complexity of the
impressions of the radiology report in the training datasets. Therefore, we
experiment with the variability of the textual input phrases. Our goal is to
maximize the interpretability of the model by directly injecting the class
condition into the text prompts assuming that we can boost phrase grounding
performance. The highest variability is obtained by taking the impressions
directly (compare Tab. 1. The lowest variability is obtained by taking the
class, converting it to a string, and taking this as input. Additionally, to
get a text dataset with medium variability, we take ChatGPT version 3.5 and
ask it to generate 100 radiography impressions of CXR images with a certain
disease (We show a few examples in the supplementary material). Then we
randomly sample from this set during training. To enable a fair comparison we
only report the results on the subset of the MS-CXR test dataset that has the
label of the image in its impression. The results are given in Tab. 7.

Trade-off of learnable vs frozen encoder The learnable models once again
completely fail in terms of interpretability. We achieve the highest phrase
grounding performance when training the model on the impressions directly.
Training on labels surprisingly does not improve the localization results. We
assume this is partial because of the domain gap in the text input when we try
our localization method on impressions that not only consist of the class
label itself. The synthetic impressions achieved the lowest result validating
our observation that a high variability of the impressions is key for the
model to be interpretable.

Textual input | Variability | CNR
---|---|---
Learnable | Frozen
Impressions | high | 0.15 | 0.95
ChatGPT | middle | 0.22 | 0.42
Class labels | low | 0.29 | 0.80
Table 7: CNR results for different models in relation to the variability of
the textual input prompts. Maximizing the variability of the prompts also
maximizes the interpretability.

## Discussion

Our experiments reveal that there are three factors that heavily impact the
interpretability of diffusion models. First is the domain shift. Keeping this
shift low helps in producing interpretable models. Secondly, keeping the
variability high or using strong input signals helps. Finally, we revealed
that keeping the language encoder frozen heavily impacts the interpretability
of diffusion models, which in turn results in worse generative quality in
terms of FID. This gap is smaller for conditional image generation with
FinetuneF being slightly better at generation for six out of eight classes.
The only two exceptions are “Edema” and “Pneumothorax”, the only two classes
of FinetuneL that showed the slightest signs of interpretability. This is
evidence that FinetuneL did not properly learn to align textual and spatial
information for the majority of input prompts. Furthermore, we believe this is
evidence that focusing on designing interpretable diffusion models could also
boost their generative ability and that their interpretability is a reasonable
indicator of generative performance. Explaining performance differences in
medical imaging is of utmost importance as the assessment requires trained
experts, whose time is often limited. We worked around this by using FID,
which has its own limitation due to it being a trained metric and requiring
too many samples to be accurate, and conditional generation, which is limited
as there is no ground-truth data available. Localization of descriptive
features on the other hand is more straightforward, as there is a ground truth
available. Given that the results shown in our study match with the results of
conditional image generation, we believe that it is a valuable metric that
should be considered for text-to-image models. It would be interesting to
perform a large-scale study that analyses the quality of images judged by
domain experts in relation to FID and CNR to see which metric better captures
generative quality.

## Conclusion

In this paper, we show evidence that the state-of-the-art way of fine-tuning
diffusion models to medical tasks results in models that have extraordinary
image quality but completely loose interpretability. Although machine learning
models with limited interpretability may be suitable for certain industrial
applications and entertainment purposes, representation models intended for
deployment in medical environments are likely to face significant scrutiny
regarding their interpretability in the future. To alleviate this issue, we
perform rigorous experiments that analyze and unveil important relations in
terms of training and interpretability of the model.

## Acknowledgements

This work was supported by HPC resources provided by the Erlangen National
High Performance Computing Center (NHR@FAU) of FAU Erlangen-Nürnberg under the
NHR projects b143dc and b180dc. NHR funding is provided by federal and
Bavarian state authorities. NHR@FAU hardware is partially funded by the German
Research Foundation (DFG) – 440719683. Key support was received by the ERC -
project MIA-NORMAL 101083647, DFG DC-AIDE 512819079, and by the state of
Bavaria. H. Reynaud was supported by Ultromics Ltd. and the UKRI Centre or
Doctoral Training in Artificial Intelligence for Healthcare (EP/S023283/1).

## References

  * Boecking et al. (2022) Boecking, B.; Usuyama, N.; Bannur, S.; Castro, D. C.; Schwaighofer, A.; Hyland, S.; Wetscherek, M.; Naumann, T.; Nori, A.; Alvarez-Valle, J.; et al. 2022.  Making the most of text semantics to improve biomedical vision–language processing.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXVI_ , 1–21. Springer.
  * Chambon et al. (2022) Chambon, P.; Bluethgen, C.; Delbrouck, J.-B.; Van der Sluijs, R.; Połacin, M.; Chaves, J. M. Z.; Abraham, T. M.; Purohit, S.; Langlotz, C. P.; and Chaudhari, A. 2022.  RoentGen: Vision-Language Foundation Model for Chest X-ray Generation.  _arXiv preprint arXiv:2211.12737_.
  * Cohen et al. (2022) Cohen, J. P.; Viviano, J. D.; Bertin, P.; Morrison, P.; Torabian, P.; Guarrera, M.; Lungren, M. P.; Chaudhari, A.; Brooks, R.; Hashir, M.; and Bertrand, H. 2022.  TorchXRayVision: A library of chest X-ray datasets and models.  In _Medical Imaging with Deep Learning_.
  * Devlin et al. (2018) Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Dhariwal and Nichol (2021) Dhariwal, P.; and Nichol, A. 2021.  Diffusion models beat gans on image synthesis.  _Advances in Neural Information Processing Systems_ , 34: 8780–8794.
  * Dombrowski et al. (2022) Dombrowski, M.; Reynaud, H.; Baugh, M.; and Kainz, B. 2022.  Zero-Shot Object Segmentation through Concept Distillation from Generative Image Foundation Models.  _arXiv preprint arXiv:2212.14306_.
  * Dosovitskiy et al. (2020) Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. 2020.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_.
  * Gupta et al. (2020) Gupta, T.; Vahdat, A.; Chechik, G.; Yang, X.; Kautz, J.; and Hoiem, D. 2020.  Contrastive learning for weakly supervised phrase grounding.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III_ , 752–768. Springer.
  * Hertz et al. (2022) Hertz, A.; Mokady, R.; Tenenbaum, J.; Aberman, K.; Pritch, Y.; and Cohen-Or, D. 2022.  Prompt-to-prompt image editing with cross attention control.  _arXiv preprint arXiv:2208.01626_.
  * Ho, Jain, and Abbeel (2020) Ho, J.; Jain, A.; and Abbeel, P. 2020.  Denoising diffusion probabilistic models.  _Advances in Neural Information Processing Systems_ , 33: 6840–6851.
  * Huang et al. (2021) Huang, S.-C.; Shen, L.; Lungren, M. P.; and Yeung, S. 2021.  Gloria: A multimodal global-local representation learning framework for label-efficient medical image recognition.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 3942–3951.
  * Johnson et al. (2019) Johnson, A. E.; Pollard, T. J.; Berkowitz, S. J.; Greenbaum, N. R.; Lungren, M. P.; Deng, C.-y.; Mark, R. G.; and Horng, S. 2019.  MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports.  _Scientific data_ , 6(1): 317.
  * Kim and Ye (2022) Kim, B.; and Ye, J. C. 2022.  Diffusion deformable model for 4D temporal medical image generation.  In _Medical Image Computing and Computer Assisted Intervention–MICCAI 2022: 25th International Conference, Singapore, September 18–22, 2022, Proceedings, Part I_ , 539–548. Springer.
  * Li et al. (2022) Li, L. H.; Zhang, P.; Zhang, H.; Yang, J.; Li, C.; Zhong, Y.; Wang, L.; Yuan, L.; Zhang, L.; Hwang, J.-N.; et al. 2022.  Grounded language-image pre-training.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 10965–10975.
  * Lin et al. (2014) Lin, T.-Y.; Maire, M.; Belongie, S.; Hays, J.; Perona, P.; Ramanan, D.; Dollár, P.; and Zitnick, C. L. 2014.  Microsoft coco: Common objects in context.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_ , 740–755. Springer.
  * Liu et al. (2020) Liu, F.; Shareghi, E.; Meng, Z.; Basaldella, M.; and Collier, N. 2020.  Self-alignment pretraining for biomedical entity representations.  _arXiv preprint arXiv:2010.11784_.
  * Liu et al. (2022) Liu, L.; Ren, Y.; Lin, Z.; and Zhao, Z. 2022.  Pseudo numerical methods for diffusion models on manifolds.  _arXiv preprint arXiv:2202.09778_.
  * Pinaya et al. (2022a) Pinaya, W. H.; Graham, M. S.; Gray, R.; Da Costa, P. F.; Tudosiu, P.-D.; Wright, P.; Mah, Y. H.; MacKinnon, A. D.; Teo, J. T.; Jager, R.; et al. 2022a.  Fast unsupervised brain anomaly detection and segmentation with diffusion models.  In _Medical Image Computing and Computer Assisted Intervention–MICCAI 2022: 25th International Conference, Singapore, September 18–22, 2022, Proceedings, Part VIII_ , 705–714. Springer.
  * Pinaya et al. (2022b) Pinaya, W. H.; Tudosiu, P.-D.; Dafflon, J.; Da Costa, P. F.; Fernandez, V.; Nachev, P.; Ourselin, S.; and Cardoso, M. J. 2022b.  Brain imaging generation with latent diffusion models.  In _Deep Generative Models: Second MICCAI Workshop, DGM4MICCAI 2022, Held in Conjunction with MICCAI 2022, Singapore, September 22, 2022, Proceedings_ , 117–126. Springer.
  * Radford et al. (2021) Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , 8748–8763. PMLR.
  * Reynaud et al. (2023) Reynaud, H.; Qiao, M.; Dombrowski, M.; Day, T.; Razavi, R.; Gomez, A.; Leeson, P.; and Kainz, B. 2023.  Feature-Conditioned Cascaded Video Diffusion Models for Precise Echocardiogram Synthesis.  arXiv:2303.12644.
  * Rombach et al. (2022) Rombach, R.; Blattmann, A.; Lorenz, D.; Esser, P.; and Ommer, B. 2022.  High-Resolution Image Synthesis With Latent Diffusion Models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 10684–10695.
  * Saharia et al. (2022) Saharia, C.; Chan, W.; Saxena, S.; Li, L.; Whang, J.; Denton, E.; Ghasemipour, S. K. S.; Ayan, B. K.; Mahdavi, S. S.; Lopes, R. G.; et al. 2022.  Photorealistic text-to-image diffusion models with deep language understanding.  _arXiv preprint arXiv:2205.11487_.
  * Saharia et al. (2021) Saharia, C.; Ho, J.; Chan, W.; Salimans, T.; Fleet, D. J.; and Norouzi, M. 2021.  Image super-resolution via iterative refinement.  _arXiv:2104.07636_.
  * Schuhmann et al. (2022) Schuhmann, C.; Beaumont, R.; Vencu, R.; Gordon, C.; Wightman, R.; Cherti, M.; Coombes, T.; Katta, A.; Mullis, C.; Wortsman, M.; et al. 2022.  Laion-5b: An open large-scale dataset for training next generation image-text models.  _Advances in Neural Information Processing Systems_ , 35: 25278–25294.
  * Tang et al. (2022) Tang, R.; Pandey, A.; Jiang, Z.; Yang, G.; Kumar, K.; Lin, J.; and Ture, F. 2022.  What the daam: Interpreting stable diffusion using cross attention.  _arXiv preprint arXiv:2210.04885_.
  * Wang, Simoncelli, and Bovik (2003) Wang, Z.; Simoncelli, E. P.; and Bovik, A. C. 2003.  Multiscale structural similarity for image quality assessment.  In _The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003_, volume 2, 1398–1402. Ieee.
  * Wolleb et al. (2022) Wolleb, J.; Bieder, F.; Sandkühler, R.; and Cattin, P. C. 2022.  Diffusion models for medical anomaly detection.  In _Medical Image Computing and Computer Assisted Intervention–MICCAI 2022: 25th International Conference, Singapore, September 18–22, 2022, Proceedings, Part VIII_ , 35–45. Springer.
  * Yan et al. (2022) Yan, A.; McAuley, J.; Lu, X.; Du, J.; Chang, E. Y.; Gentili, A.; and Hsu, C.-N. 2022.  RadBERT: Adapting transformer-based language models to radiology.  _Radiology: Artificial Intelligence_ , 4(4): e210258.
  * Zhang and Zhu (2018) Zhang, Q.-s.; and Zhu, S.-C. 2018.  Visual interpretability for deep learning: a survey.  _Frontiers of Information Technology & Electronic Engineering_, 19(1): 27–39.
  * Zhang et al. (2022) Zhang, Y.; Jiang, H.; Miura, Y.; Manning, C. D.; and Langlotz, C. P. 2022.  Contrastive learning of medical visual representations from paired images and text.  In _Machine Learning for Healthcare Conference_ , 2–25. PMLR.

##  Appendix A Appendix

Technical details The architecture of our model is based on the $512\times
512$ pixel model from Stable Diffusion v2.0, as described by Rombach et al.
(2022). In fine-tuning this model, we automatically fix the latent
representation extracted for the localization. As detailed in the Methods
section, we extract the attention maps of all cross-attention layers of the
U-Net without manually selecting more important layers. We note that
selectively picking layers might enhance the localization results but we
believe that only picking certain layers hurts the overall interpretability.
The resulting latent dimensions are $S\times D\times T_{max}\times Z\times Z$
where $S$ is defined as the number of reverse diffusion steps and $D$ is
defined as the number of cross-attention layers in the encoder and decoder of
the U-Net. In our case $D$ is equal to 16.

Additional experiments on interpretability First we want to investigate
whether the difference in interpretability comes from the domain gap induced
by the different impressions. MS-CXR impressions are carefully aggregated as
opposed to MIMIC impressions which come from a radiology report. The result is
that parts of the expressions are tokens that were used to anonymize the data
and hide private information. Also, they are generally longer and less
concise. Therefore, we repeat our experiment which computes the tokens with
the highest localization accuracy for impressions extracted from MIMIC on
bounding boxes from MS-CXR. The results shown in Tab. 8 indicate that the
impressions of the MIMIC dataset are not polished enough to be useful for
localization. Looking at the results of using MS-CXR impressions in Tab. 6 we
can see that for some disease categories, it would be beneficial for
localization performance to include other keywords. One example would be the
keyword “enlarged” in the case of Cardiomegaly. However, we believe this post
hoc analysis would hurt interpretability, which we believe is more important
than localization performance.

Cat. | SDv2 |  | FinetuneL |  | FinetuneF
---|---|---|---|---|---
Word | Occ. | CNR |  | Word | Occ. | CNR |  | Word | Occ. | CNR
Atelactasis | EOS | 34 | 0.48 |  | EOS | 34 | 0.77 |  | EOS | 34 | 0.83
in | 28 | -0.03 |  | in | 28 | -0.03 |  | right | 42 | 0.78
right | 42 | -0.06 |  | right | 42 | -0.12 |  | in | 28 | 0.20
Cardiomegaly | lung | 71 | 0.68 |  | EOS | 214 | 0.32 |  | moderate | 30 | 0.99
chest | 36 | 0.44 |  | SOS | 264 | 0.15 |  | bilateral | 44 | 0.99
pulmonary | 121 | 0.41 |  | vascular | 30 | -0.16 |  | mild | 45 | 0.97
Consolidation | pulmonary | 33 | 0.82 |  | EOS | 58 | 0.95 |  | lung | 39 | 1.31
lung | 39 | 0.73 |  | a | 29 | 0.04 |  | left | 53 | 1.25
EOS | 58 | 0.46 |  | to | 50 | 0.02 |  | pulmonary | 33 | 1.09
Edema | pulmonary | 32 | 1.15 |  | EOS | 28 | 0.91 |  | pulmonary | 32 | 1.27
EOS | 28 | 0.62 |  | of | 42 | 0.23 |  | EOS | 28 | 0.80
of | 42 | 0.12 |  | the | 64 | 0.20 |  | and | 34 | 0.42
Lung Opacity | EOS | 46 | 0.50 |  | EOS | 46 | 0.95 |  | left | 28 | 1.09
left | 28 | 0.27 |  | to | 35 | 0.06 |  | right | 45 | 0.89
with | 33 | 0.15 |  | is | 67 | -0.02 |  | EOS | 46 | 0.86
Effusion | SOS | 91 | 0.10 |  | EOS | 61 | 0.41 |  | left | 56 | 0.88
pulmonary | 38 | 0.09 |  | SOS | 91 | 0.07 |  | effusion | 26 | 0.77
EOS | 61 | 0.00 |  | right | 67 | -0.12 |  | pleural | 86 | 0.61
Pneumonia | lung | 57 | 0.69 |  | EOS | 98 | 0.88 |  | pneumonia. | 52 | 1.34
lobe | 49 | 0.65 |  | are | 36 | -0.01 |  | lung | 57 | 1.33
pulmonary | 46 | 0.62 |  | No | 27 | -0.09 |  | left | 72 | 1.28
Pneumothorax | Small | 40 | 0.60 |  | 1. | 29 | 0.96 |  | ___. | 27 | 0.75
catheter | 29 | 0.60 |  | Small | 40 | 0.79 |  | with | 97 | 0.72
small | 69 | 0.60 |  | Right | 31 | 0.77 |  | chest | 86 | 0.72
Table 8: Number of occurrences and CNR for the three words with the highest
localization accuracy of each model on MIMIC. SOS and EOS are start and end of
string respectively. Only words with more than 25 occurrences were considered.
Radiology reports exceeding the length of the language encoder were ignored.

Keywords used for localization The pipeline described in the method section of
the main paper requires that we define the words $y_{i}$ that we compute the
attention over. For MS-COCO we use the object class directly. For the results
reported on MS-CXR we define a list of words that we consider as descriptive
of the disease class. We summarize the set of words in Tab. 9. This also
validates that we could add additional keywords to boost localization
performance. FinetuneF only achieved a CNR of 0.67 in our main experiments for
Cardiomegaly, but keywords like “enlarged” achieve a CNR of 1.16 for this
particular disease class. Our initial assumption was that this is based on
naturally occurring correlations between certain words in the training dataset
with the disease class. Therefore, we analyze whether the number of
occurrences of a word in impressions of certain diseases correlates with the
localization ability. The results shown in Fig. 3 indicate that there is no
real correlation visible, which could again be due to the difference between
training and validation tokens.

Figure 3: Token experiment: We show the percentage of occurrences of words in impressions of the training data against their ability to localize diseases. We only show words that appear more than 25 times in the validation dataset. Category | $\textbf{N}_{D}$ | $\textbf{N}_{G}$ | Keyword | Example
---|---|---|---|---
Atelectasis | 61 | 694 | atelectasis, atelectatic | the right lung is atelectatic
Cardiomegaly | 333 | 684 | cardiomegaly, cardiac | enlarged cardiac silhouette
Consolidation | 117 | 155 | consolidation | right basilar consolidation
Edema | 46 | 408 | edema | pulmonary edema
Lung Opacity | 82 | 903 | opacity, opacities, opacification | patchy left basilar opacity
Pleural Effusion | 96 | 805 | pleural, effusion, effusions | Small left pleural effusion
Pneumonia | 182 | 271 | pneumonia | Right lower lung pneumonia
Pneumothorax | 245 | 182 | pneumothorax | trace left pneumothorax
No Finding | 0 | 903 | - | -
Table 9: Instaces of each disease class in the generative $\textbf{N}_{G}$ and
in the discriminative $\textbf{N}_{D}$ test dataset. Keywords denote the words
that are considered as tokens $\tau_{i}$ for the attention extraction.

Ablation studies We perform multiple ablation studies on the design choices of
our model fine-tuning. First, we investigate the impact of disabling
unconditional training as described in the method section for the frozen
(FinetuneFnU) and the learnable (FinetuneLnU) approach. We also fine-tune
FinetuneL on an additional 10000 steps with frozen language encoder to see if
this works as a quick solution to reduce the negative effect of learnable
fine-tuning (Finetune60+10). Finally, we train FinetuneF exclusively on PA
images for 30000 steps and a batch size of 128 FinetunePA. Generative results
on these ablation studies are summarized in Tab. 10. Looking at the models
without unconditional guidance (FinetuneFnU and FinetuneFnL) we see that the
generative results are consistently worse. The FID and FIDXRV values drop in
all cases indicating the unconditional training is helpful to improve the
performance. Fine-tuning the learnable model on 10000 additional steps with a
frozen encoder has a worse performance than not fine-tuning (Finetune60+10 has
a FIDXRV of 9.3 while FinetuneF has a FID of 7.3). Finally, we achieve state-
of-the-art generative results with FinetunePA which indicates that it might be
favourable for the generative model to condition on the view as well as this
might improved the peformance.

Method | FinetuneLnU |  | FinetuneFnU |  | Finetune60+10 |  | FinetunePA
---|---|---|---|---|---|---|---
Mimic | CXR |  | Mimic | CXR |  | Mimic | CXR |  | Mimic | CXR
FID $\downarrow$ | 64.8 | 64.6 |  | 88.0 | 85.7 |  | 64.8 | 64.7 |  | 48.0 | 47.9
FIDXRV $\downarrow$ | 10.5 | 10.8 |  | 16.4 | 16.0 |  | 9.5 | 9.3 |  | 3.5 | 3.6
Table 10: Generative results on our ablation studies of using no unconditional
guidance (FinetuneFnU, FinetuneLnU), fine-tuning the frozen model
(Finetune60+10) and training only on PA images (FinetunePA).

Next, we evaluate the interpretability of these models by looking at the AUC
and Top-1 accuracy in Tab. 11. The experiments show that unconditional
training is beneficial for localization performance. Both training runs with
unconditional training enabled (FinetuneLnU, FinetuneFnU) are better at
localization than the models that did not use it. Finetuning the learnable
model on 10000 additional steps with a learnable language encoder did not
improve the model noticeably. The interpretability of the FinetunePA model is
worse as the test set also contains AP images.

Method | FinetuneLnU |  | FinetuneFnU |  | Finetune60+10 |  | FinetunePA
---|---|---|---|---|---|---|---
AUC $\uparrow$ | Top-1 $\uparrow$ |  | AUC $\uparrow$ | Top-1 $\uparrow$ |  | AUC $\uparrow$ | Top-1 $\uparrow$ |  | AUC $\uparrow$ | Top-1 $\uparrow$
Atelectasis | 47.3 | 0.0 |  | 84.4 | 70.5 |  | 44.5 | 0.0 |  | 82.5 | 55.7
Cardiomegaly | 39.1 | 0.3 |  | 70.0 | 26.7 |  | 42.2 | 0.1 |  | 71.4 | 26.7
Consolidation | 9.4 | 9.4 |  | 83.7 | 42.7 |  | 51.3 | 6.0 |  | 80.9 | 38.5
Edema | 55.9 | 41.3 |  | 87.4 | 67.4 |  | 54.9 | 30.4 |  | 86.9 | 86.9
Lung Opacity | 50.1 | 3.6 |  | 78.5 | 28.0 |  | 51.0 | 4.9 |  | 77.1 | 17.1
Pl. Effusion | 39.2 | 2.1 |  | 81.5 | 60.4 |  | 37.2 | 2.1 |  | 79.9 | 64.6
Pneumonia | 47.5 | 1.1 |  | 84.1 | 61.5 |  | 47.0 | 2.2 |  | 81.1 | 35.7
Pneumothorax | 80.2 | 15.1 |  | 79.3 | 39.2 |  | 77.2 | 4.9 |  | 69.9 | 16.3
Average | 52.0 | 5.3 |  | 78.6 | 43.2 |  | 52.1 | 4.3 |  | 76.9 | 32.5
Table 11: Ablation study on localization performance.

Contrast-to-noise ratio ablation study Finally, we provide some motivation on
why we use CNR, instead of the more commonly used CNRabs. We compute CNR using
the following formula:

$CNR=\frac{(\mu_{A}-\mu_{\overline{A}})}{(\sigma_{A}^{2}+\sigma^{2}_{\overline{A}})^{\frac{1}{2}}}$
(2)

where $A$ describes the ground-truth segmentation or bounding box and
$\overline{A}$ its complement. However, in literature it is more common to use

$CNR_{abs}=\frac{|\mu_{A}-\mu_{\overline{A}}|}{(\sigma_{A}^{2}+\sigma^{2}_{\overline{A}})^{\frac{1}{2}}}$
(3)

However, this means that if the localization is terrible, and the mean value
of the attention outside the region of interest is higher than inside, then
the numerator would be positive and the localization would count towards
positive results. In Tab. 12 we compare both approaches for our three main
models. It is clear that there is no meaningful difference between both
approaches when the localization results are generally good (FunituneF).
However, for SDv2 and FinetuneL CNRabs overestimates the performance visibly.

|  | Method | Atel. | Card. | Cons. | L-Op. | Edem. | Pnem. | Pnth. | P-Ef. | Avg.
---|---|---|---|---|---|---|---|---|---|---|---
CNR |  | SDv2 | 0.23 | 0.14 | 0.21 | 0.37 | 0.87 | 0.34 | 0.64 | 0.01 | 0.321
| FinetuneL | -0.20 | -0.20 | -0.01 | 0.00 | 0.11 | -0.08 | 0.75 | -0.35 | 0.075
| FinetuneF | 1.30 | 0.74 | 1.29 | 1.07 | 1.34 | 1.20 | 0.59 | 1.09 | 0.942
CNRabs |  | SDv2 | 0.37 | 0.37 | 0.31 | 0.41 | 0.87 | 0.41 | 0.65 | 0.23 | 0.445
| FinetuneL | 0.30 | 0.27 | 0.26 | 0.30 | 0.20 | 0.25 | 0.78 | 0.39 | 0.386
| FinetuneF | 1.30 | 0.74 | 1.29 | 1.07 | 1.34 | 1.20 | 0.59 | 1.09 | 0.944
Table 12:  Localization results using $\text{CNR}_{abs}$ and CNR on a single
run.

ChatGPT impressions To get impressions from ChatGPT we first prompt the model
to “Give me three radiography reports of a chest x-ray of a lung with [disease
class]”. The results are close to the structure in MIMIC, which leads to the
assumption that some reports of MIMIC were part of the training dataset. Next,
we ask the model only to return the impression section of the reports, as we
did as a preprocessing step for the training on the real reports. Finally, we
ask the model to generate 100 of those with emphasis on variability because we
observed that if we do not ask for that, the model will simply return the same
3-5 impressions. The following shows five examples:

  1. 1.

Right middle lobe atelectasis with associated volume loss and mediastinal
shift. Clinical correlation and further evaluation recommended.

  2. 2.

Left lower lobe atelectasis with associated volume loss. Clinical correlation
and further evaluation recommended to determine the underlying cause.

  3. 3.

Complete atelectasis of the right upper lobe with associated volume loss and
mediastinal shift. Clinical correlation and further evaluation recommended to
determine the underlying cause and appropriate management.

  4. 4.

Atelectasis involving the left upper lobe with associated crowding of adjacent
lung parenchyma. Clinical correlation advised.

  5. 5.

Right lower lobe atelectasis with associated volume loss. Further evaluation
recommended for proper management.