MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Models
Under review as a conference paper at ICLR 2025

MMED-RAG: VERSATILE MULTIMODAL RAG SYS-
TEM FOR MEDICAL VISION LANGUAGE MODELS

Anonymous authors
Paper under double-blind review

ABSTRACT

Artificial Intelligence (AI) has demonstrated significant potential in healthcare,
particularly in disease diagnosis and treatment planning. Recent progress in Med-
ical Large Vision-Language Models (Med-LVLMs) has opened up new possibil-
ities for interactive diagnostic tools. However, these models often suffer from
factual hallucination, which can lead to incorrect diagnoses. Fine-tuning and
retrieval-augmented generation (RAG) have emerged as methods to address these
issues. However, the amount of high-quality data and distribution shifts between
training data and deployment data limit the application of fine-tuning methods. Al-
though RAG is lightweight and effective, existing RAG-based approaches are not
sufficiently general to different medical domains and can potentially cause mis-
alignment issues, both between modalities and between the model and the ground
truth. In this paper, we propose a versatile multimodal RAG system, MMed-RAG,
designed to enhance the factuality of Med-LVLMs. Our approach introduces a
domain-aware retrieval mechanism, an adaptive retrieved contexts selection, and
a provable RAG-based preference fine-tuning strategy. These innovations make
the RAG process sufficiently general and reliable, significantly improving align-
ment when introducing retrieved contexts. Experimental results across five med-
ical datasets (involving radiology, ophthalmology, pathology) on medical VQA
and report generation demonstrate that MMed-RAG can achieve an average im-
provement of 43.8% in the factual accuracy of Med-LVLMs.

1

INTRODUCTION

Artificial Intelligence (AI) has already transformed healthcare and still has a lot of potential for
further advancements (T˘aut¸an et al., 2021; Wang et al., 2019; Ye et al., 2021; Tu et al., 2024).
Recently, Medical Large Vision-Language Models (Med-LVLMs) have shown great promise for
advancing interactive and intelligent diagnosis (Li et al., 2023a; Moor et al., 2023; Zhang et al.,
2023b; Wu et al., 2023b). Despite this potential (Li et al., 2023b; Wu et al., 2023a; Shi et al., 2024),
current Med-LVLMs still face significant reliability issues, particularly their tendency to generate
non-factual medical responses (Xia et al., 2024a; Royer et al., 2024; Chen et al., 2024a; Jiang et al.,
2024), making them unreliable in critical medical applications. These factuality issues raise serious
concerns when deploying such models in clinical settings, where even small diagnostic errors could
lead to severe consequences for patient care.

Recently, researchers have begun to focus on improving the factuality of Med-LVLMs through var-
ious techniques, including fine-tuning (Li et al., 2023a; Moor et al., 2023; Thawkar et al., 2023;
Zhang et al., 2023b; Chen et al., 2024b) and retrieval-augmented generation (RAG) (Xia et al.,
2024b; He et al., 2024; Sun et al., 2024b). Fine-tuning is a direct method to improve model per-
formance, but it faces several limitations in the medical field. First, there is a lack of sufficient
high-quality labeled data for fine-tuning in the medical domain. Additionally, a distribution gap
often exists between the training data and the real-world deployment data (Schrouff et al., 2022),
leading to significantly worse model performance during deployment. Hence, RAG has emerged
as a viable alternative by providing external references during the inference stage, enhancing the
factuality of Med-LVLMs (Wu et al., 2023c; Gao et al., 2023). However, despite its advantages, cur-
rent RAG implementations in Med-LVLMs have significant limitations. First, these methods tend to
be dataset-specific, reducing their generalizability across various medical domains. Second, these
models are still facing misalignment issues that lead to factuality problems. This misalignment may

1

000
001

002
003
004
005
006
007

008
009
010
011
012
013

014
015
016
017
018

019
020
021
022
023
024

025
026
027
028
029
030

031
032
033
034
035
036

037
038
039
040
041

042
043
044
045
046
047

048
049
050
051
052
053

Under review as a conference paper at ICLR 2025

arise from the impact of adding RAG on the original Med-LVLMs’ cross-modality alignment, as
well as on the overall alignment between the model and ground truth.

To address these challenges, we propose a versatile factual Multimodal Medical RAG system called
MMed-RAG. Specifically, MMed-RAG first introduces a domain-aware retrieval mechanism, de-
signed to handle different domains of medical images more effectively. Here, we design a domain
identification module to adaptively select a corresponding retrieval model given the input medical
image. Secondly, we include a adaptive calibration approach for selecting the number of retrieved
contexts. Lastly, MMed-RAG incorporates RAG-based preference fine-tuning to enhance cross-
modality alignment and overall alignment with ground truth. The preference pairs are designed to
achieve two goals: first, to improve cross-modality alignment by encouraging the model to avoid
generating responses without utilizing input medical images, even the responses are correct; sec-
ond, to improve overall alignment by encouraging the model to understand retrieved contexts when
unsure, while avoiding interference from irrelevant retrieved information.

The primary contribution of this paper is MMed-RAG, a versatile multimodal RAG system designed
specifically for Med-LVLMs to generate more factual responses. Under mild assumptions, our the-
oretical analysis demonstrates that MMed-RAG mitigates both cross-modality misalignment and
overall misalignment with ground truth. Furthermore, empirical results on five medical multimodal
datasets, covering three medical image modalities (radiology, pathology, and ophthalmology), show
that MMed-RAG significantly improves the factual accuracy of Med-LVLMs, achieving improve-
ments of 18.5% and 69.1% on Medical VQA and report generation tasks, respectively, compared
to the original Med-LVLM. These empirical findings further demonstrate the effectiveness of our
proposed components and support the theoretical analysis in addressing misalignment issues.

2 PRELIMINARIES

In this section, we will provide a brief overview of Med-LVLMs and preference optimization.

Medical Large Vision Language Models. Med-LVLMs bridge LLMs with medical visual mod-
ules, allowing the model to take medical image xv and clinical query xt as input x, and autoregres-
sively predict the probability distribution of the next token. The text output is denoted as y.

Preference Optimization. Preference optimization has achieved remarkable results in LLM align-
ment. Give an input x, a language model policy πθ can produce a conditional distribution πθ(y | x)
with y as the output text response. The recently popular DPO (Rafailov et al., 2023) utilizes
preference data achieve objective alignment in LLMs. The preference data is defined as D =
{x(i), y(i)
represent preferred and dispreferred responses given an in-
put prompt x. The probably of obtaining each preference pair is p(yw ≻ yl) = σ(r(x, yw) − r(x, yl)),
where σ(·) is the sigmoid function. In DPO, the optimization can be formulated as classification
loss over the preference data as:

i=1, where y(i)

w and y(i)

w , y(i)

l }N

l

LDPO(πθ; πref) = −E(x,yw ,yl)∼D

(cid:104)

log σ

(cid:16)

α log πθ (yw |x)

πref(yw |x) − α log πθ (yl|x)

πref(yl|x)

(cid:17)(cid:105)

.

(1)

where πθ represents the reference policy, which is the LLM fine-tuned through supervised learning.

3 MMED-RAG: A VERSATILE MEDICAL RAG SYSTEM

In this section, as illustrated in Figure 1, we will propose MMed-RAG, a versatile RAG system for
improving the factuality of Med-LVLMs. Specifically, MMed-RAG consists of three complemen-
tary modules. First, we design a domain-aware retrieval mechanism to select the optimal retriever
by feeding each given medical image to the domain identification module. Second, to select an
optimal number of retrieved contexts and filter out low-quality information, MMed-RAG adopts a
adaptive method by filtering out low-quality information using the similarity scores during the RAG
phase. Lastly, we use a RAG-based preference fine-tuning approach to improve the cross-modality
alignment and the overall alignment between groundtruth. We detail these steps as follows:

3.1 DOMAIN-AWARE RETRIEVAL MECHANISM

In MMed-RAG, we introduce a domain-aware retrieval mechanism to efficiently handle medical
images from different sources (e.g., radiology, pathology, ophthalmology). Specifically, we first

2

054
055

056
057
058
059
060
061

062
063
064
065
066
067

068
069
070
071
072

073
074
075
076
077
078

079
080
081
082
083
084

085
086
087
088
089
090

091
092
093
094
095

096
097
098
099
100
101

102
103
104
105
106
107

Under review as a conference paper at ICLR 2025

Figure 1: Overview of MMed-RAG, a versatile factual multimodal RAG system designed to enhance
the reliability of Med-LVLMs. It introduces a domain-aware retrieval mechanism that effectively
handles different domains of medical images by selecting suitable retrieval models. Additionally, it
uses an adaptive context selection approach to determine the optimal number of retrieved contexts
and employs preference fine-tuning to improve both cross-modality and overall alignment.

employ a domain identification module that assigns a domain label to each input medical image.
To achieve this, we create a small dataset with medical images as inputs and their corresponding
domain labels as outputs, using this dataset to fine-tune the BiomedCLIP model (Zhang et al., 2023a)
to improve its domain awareness. Formally, for a given medical image xv, we predict its domain
d = F(xv). Based on the assigned domain label d, the image xv is fed into the corresponding
multimodal retriever Rd(·) for knowledge retrieval.

Here, each multimodal retriever Rd(·) for each domain d is trained through contrastive learn-
ing (Radford et al., 2021). Specifically, the visual and textual information Ximg, Xtxt are pro-
cessed by their corresponding encoders Eimg(·), Etxt(·) to generate textual and visual embeddings
Vtxt = Etxt(Xtxt), Vimg = Eimg(Ximg). Contrastive learning loss is then applied to maximize the
similarity between text and image embeddings representing the same example, while minimizing
the similarity between embeddings representing different examples, as defined below:

L =

Limg + Ltxt
2

, where Limg = −

1
N

N
(cid:88)

log

i=1

exp(Si,i)
j=1 exp(Si,j)

(cid:80)N

, Ltxt = −

1
N

N
(cid:88)

log

i=1

exp(Si,i)
j=1 exp(Sj,i)

(cid:80)N

,

(2)
where S ∈ RN ×N represents the similarity matrix between image and text modalities, calculated
as: S = Vimg
|Vtxt| )T , where each element Si,j represents the similarity between the image
representation of example i and the text representation of example j.

|Vimg| · ( Vtxt

Finally, for the input image xt, after feeding into the corresponding multimodal retriever Rd(·), the
multimodal retriever will retrieves the top-k most similar reports for the image. These retrieved re-
ports xr = Rd(xv) are then provided to the Med-LVLM M(·) as references to guide the generation.

3.2 ADAPTIVE RETRIEVED CONTEXT SELECTION

Following the domain-aware retrieval mechanism, the next
step is to determine the optimal amount of context to retrieve.
Retrieving too much or too little information can result in hal-
lucinations (Xia et al., 2024b). Current RAG methods applied
to Med-LVLMs generally rely on empirical results or fixed
values based on validation sets to select the optimal value of
the number of retrieved contexts k (Xia et al., 2024b; He et al.,
2024; Sun et al., 2024b). However, the distribution of simi-
larity scores varies depending on the complexity of the image

3

Figure 2: Relations between se-
lected contexts and similarity score.

108
109

110
111
112
113
114
115

116
117
118
119
120
121

122
123
124
125
126

127
128
129
130
131
132

133
134
135
136
137
138

139
140
141
142
143
144

145
146
147
148
149

150
151
152
153
154
155

156
157
158
159
160
161

IU-XrayMIMICQuiltDomain IdentiﬁcationRadiologyRadiologyPathologyRetriever (Radiology)Retriever (Pathology)Domain-Aware Retrieval MechanismAdaptive Retrieved Context SelectionMedical ImageRetrieverDomain LabelMed-LVLMTop-k Reports…Is there any focal airspace opacity present?QuestionSimilarity Scores…Adaptive-k ReportsRAG-Based Preference Fine-TuningDirect Copy Homework from OthersThink it by Self ✏Unrelated ImageMed-LVLMUnrelated ImageRAGMed-LVLMOriginal ImageRAG1⃣2⃣Med-LVLMOriginal ImageRAGMed-LVLMOriginal ImageCannot Solve Problems by SelfLearn How to Copy ✏3⃣Med-LVLMOriginal ImageRAGMed-LVLMOriginal ImageCopied homework is WrongAvoid Interference from Incorrect Homework ✏Preference DataPreference Fine-TuningStronger Med-LVLMConstructed Preference PairsOriginal ImageGround-TruthUnder review as a conference paper at ICLR 2025

and its alignment with the textual information from the data source. These fixed-k methods do not
guarantee optimal performance on target data, as they overlook the similarity scores generated dur-
ing the retrieval process. To address this, we propose an adaptive method that dynamically selects
k based on the similarity scores of the retrieved contexts. Specifically, during the domain-aware
retrieval mechanism phase, the retrieved information is denoted as xr(k) = Rd(xv; k), where k
represents the number of retrieved contexts, and the corresponding similarity scores are denoted as
Sk. For simplicity, when there is no ambiguity, we will refer to xr(k) as xr.

As illustrated in Figure 2, our method is based on a key observation: the similarity scores (CLIP
score in this case) between retrieved contexts often exhibit a sharp decline after a certain number of
results (nearly top-9 in this case). This suggests that lower-quality information can still be included
among the top-k retrieved contexts when using a fixed-k strategy, especially in cases where the fixed
value of k is too large. These lower-quality retrievals introduce noise and irrelevant information,
which can significantly impair the model’s ability to generate factual and coherent responses. To
mitigate this issue, we draw inspiration from the Gap statistic method used in clustering (Tibshirani
et al., 2001) and extend this concept to RAG for Med-LVLMs. Specifically, after retrieving the
top-k contexts, we perform an additional round of k optimization by analyzing the similarity ratios
between consecutive retrievals. These similarity ratios are denoted as ui = log(Si/Si+1) for 0 <
i ≤ k, where Si represents the similarity score of the i-th retrieved context. When ui exceeds a
predefined threshold γ, this indicates a substantial drop in relevance, suggesting that the remaining
retrievals are less likely to contribute preferredly to the model’s output. At this point i, we truncate
k, effectively discarding the less relevant retrievals that follow. This adaptive truncation mechanism
ensures that only the most relevant contexts are retained for generating the final response, reducing
the risk of hallucination and improving the factual accuracy of the outputs.

Although the threshold γ is fixed, this approach provides a adaptive way to balance the bias and
variance in retrieved contexts. By adapting to the characteristics of each input xv, our method
enhances the robustness of the retrieval process and ensures that the selection of k is tailored to the
specific data at hand, thereby improving overall performance across diverse contexts and tasks.

3.3 RAG-BASED PREFERENCE FINE-TUNING

After context selection, MMed-RAG supplies Med-LVLM with reliable retrieved information as
external knowledge to aid in generating factual responses. However, incorporating this retrieved
knowledge may potentially disrupt the original alignment within the existing Med-LVLM, a concern
we will elaborate on below:

Alignment Analysis. In the alignment analysis, we aim to explore how incorporating retrieved con-
text impacts the original alignment in Med-LVLMs, focusing on two key aspects: (1) cross-modality
alignment and (2) overall alignment with the ground truth. To evaluate cross-modality alignment,
we conduct two tests on LLaVA-Med-1.5 (Li et al., 2023a) using the Harvard-FairVLMed (Luo
et al., 2024) dataset. First, when replacing the original image with a highly noisy image associated
with a different ground truth, the original model gives incorrect answers (the ground truth being
the response for the original image). After incorporating RAG, where context is retrieved based
on the original image, 55.08% of these cases return correct answers. This indicates that the model
directly references the retrieved knowledge without considering the input image, highlighting signif-
icant cross-modal misalignment issues. Furthermore, 43.31% of the questions that were originally
answered correctly are answered incorrectly after incorporating RAG, suggesting interference from
incorrect retrieval information, which leads to overall misalignment with the ground truth.

To address cross-modality misalignment and the overall misalignment introduced by incorporating
retrieved knowledge, as shown in Algorithm 1, we propose a RAG-based preference fine-tuning
(RAG-PT) approach to fine-tune the target Med-LVLM M(·). Specifically, RAG-PT constructs two
types of preference pairs designed to mitigate both categories of misalignment.

Preference Pairs for Cross-Modality Alignment. We first construct preference pairs aimed at
improving cross-modality alignment. In this dataset, we select samples from D = {x(i)
i=1,
where xv, xt, and y represent the input medical image, clinical query, and ground-truth answer,
respectively. For simplicity, we omit the sample index (i) in the following sections. A model’s
correct response using retrieved knowledge, i.e., M(xv, xt + xr) = y, is considered a preferred
response pi, where xr is the retrieved information. A dispreferred response ni is selected from cases

v , x(i)
t

, y(i)}N

4

162
163

164
165
166
167
168
169

170
171
172
173
174
175

176
177
178
179
180

181
182
183
184
185
186

187
188
189
190
191
192

193
194
195
196
197
198

199
200
201
202
203

204
205
206
207
208
209

210
211
212
213
214
215

Under review as a conference paper at ICLR 2025

Algorithm 1: Versatile Multimodal RAG System (MMed-RAG)
Input: D = {x(i)

, y(i)}N

v , x(i)
t

i=1: Dataset; πθ: Parameters of the Med-LVLM; Med-LVLM: M(·, ·);

Domain Identification: F(·); Retriever: R(·); Noisy Function: I(·).

Output: πref: Parameters of the reference model.

1 ▷ Training Stage
2 Initialize Dcm with an empty set
3 foreach (xv, xt, y) ∈ D do
4

5

6

7

8

9

10

11

12

13

14

15

16

Generate retrieved contexts with an assigned domain label xr ← RF (xv )(xv)
Generate the noisy image x∗
▷ Cross-Modality Alignment
if M(xv, (xt, xr)) = y and M(x∗

v, (xt, xr)) = y then
Select the preferred response yw,o1 ← y, dispreferred response yl,o1 ← M(x∗
Put {(xv, xt), yw,o1, yl,o1} into Dcm

v ← I(xv)

▷ Overall Alignment
Initialize D1
oa and D2
if M(xv, (xt, xr)) = y and M(xv, xt) ̸= y then

oa with empty set

Select the preferred response yw,o2 ← y, dispreferred response yl,o2 ← M(xv, xt)
Put {(xv, xt), yw,o2, yl,o2} into D1
oa

if M(xv, xt) = y and M(xv, (xt, xr)) ̸= y then

v, (xt, xr))

Select the preferred response yw,o3 ← y, dispreferred response yl,o3 ← M(xv, (xt, xr))
Put {(xv, xt), yw,o3, yl,o3} into D2
oa

17
18 Dpt = Dcm ∪ Doa, Doa = D1
oa ∪ D2
oa
19 foreach ((xv, xt), yw,o, yl,o) ∈ Dpt do
20
21 ▷ Inference Stage
22 foreach test sample (xv, xt) do
23

Compute the losses Lpt following equation 4 and update πref

Select top-k retrieved contexts with an assigned domain label xr ← RF (xv )(xv)
Get the predictions of the model w/ RAG-PT p ← M(xv, (xt, xr))

24

where the model makes a correct inference based on an unrelated image, i.e., M(x∗
v, xt) ̸= y, but
M(x∗
v, xt + xr) = y, reflecting the model’s reliance on the retrieved knowledge. The unrelated
v are generated through a two-step process: first, we use the retriever to select an image x′
images x∗
v
with the lowest similarity to the target image; then, we introduce diffusion noise into the selected
unrelated image. We define the noise step as s, and the noised image at step s is expressed as:

v + (cid:112)1 − ξs · ϵ,
where ¯ξs = (cid:81)s
i=0 ξi and ξs ∈ (0, 1) is a hyperparameter. The preference pairs constructed in
this stage are denoted as Dcm. By comparing the preferred and dispreferred responses in Dcm, we
encourage the model to prioritize the input medical image when generating responses.

v = (cid:112)ξs · x′
x∗

(3)

Preference Pairs for Overall Alignment. Second, we construct preference pairs to improve overall
alignment, focusing on enhancing the model’s ability to effectively leverage retrieved knowledge
when generating responses. The preference pairs in this stage are constructed from two subsets.
The first subset, D1
oa, is designed to strengthen the model’s comprehension and reasoning abilities
regarding the retrieved knowledge. Preferred responses are selected where the model correctly an-
swers based on both the original image and the retrieved information, i.e., M(xv, xt + xr) = y,
while dispreferred responses represent cases where the model answers incorrectly based on the im-
age without using retrieval, i.e., M(xv, xt) ̸= y. Comparing these preferred and dispreferred re-
sponses enhances the model’s understanding of the retrieved information and improves the overall
effectiveness of RAG. In the second subset, D2
oa, the goal is to mitigate interference from the re-
trieved knowledge. Preferred responses are selected where the model correctly answers based solely
on the original image without using retrieved knowledge, i.e., M(xv, xt) = y, while dispreferred
responses occur when the model answers incorrectly using both the image and retrieved informa-
tion, i.e., M(xv, xt + xr) ̸= y. This helps the model learn when to rely on its internal knowledge
versus retrieved knowledge. Finally, we combine the first and second subsets to form the second set
of preference pairs, Doa = D1
Finally, we merge the first and second preference set and denote the preference dataset as Dpt =
Dcm ∪ Doa = {x(i), y(i)
l,o are represented as preferred and dispreferred

i=1, where y(i)

oa ∪ D2

w,o, y(i)

w,o, y(i)

l,o }N

oa.

5

216
217
218
219

220
221
222
223
224
225

226
227
228
229
230

231
232
233
234
235
236

237
238
239
240
241
242

243
244
245
246
247
248

249
250
251
252
253

254
255
256
257
258
259

260
261
262
263
264
265

266
267
268
269

Under review as a conference paper at ICLR 2025

responses, respectively. Based on the curated preferences, we fine-tune Med-LVLM using direct
preference optimization (Rafailov et al., 2023) with the following loss:

Lpt = −E(x,yw,o,yl,o)∼D

(cid:104)

log σ

(cid:16)

4 THEORETICAL ANALYSIS

α log πθ (yw,o|x)

πo(yw,o|x) − α log πθ (yl,o|x)

πo(yl,o|x)

(cid:17)(cid:105)

.

(4)

In this section, we provide a theoretical analysis of the model obtained from equation 4 and examine
how the image input and retrieved context influences the model. Recall that xv, y, xt, xr denotes
input medical image, groundtruth answer, question, and retrieved information, respectively.

4.1 THE IMPROVEMENT ON CROSS-MODALITY ALIGNMENT

We first consider the loss for cross-modality alignment,

Lcm = −E(x,yw,o,yl,o)∼Dcm

(cid:104)
log σ

(cid:16)

α log πθ (yw,o|x)

πo(yw,o|x) − α log πθ (yl,o|x)

πo(yl,o|x)

(cid:17)(cid:105)

.

(5)

where (xw, yw,o) ∼ qw(xw, yw,o|xt, xr) and (xl, yl,o) ∼ ql(xl, yl,o|xt, xr) represent distributions
of the preferred responses and dispreferred responses on Dcm, respectively. Let x denote (xv, xr, xt)

Definition 4.1 Define the weight of xv with respect to log πθ(y|x) as

wt(xv, πθ) := Ey∼πθ (·|x)

(cid:20) ∂
∂xv

(cid:21)2

log πθ(y|x)

(6)

Definition 4.1 describes how log πθ(y|x) changes with respect to xv, and the weight is always non-
dispreferred. We demonstrate that this is a reasonable definition through Lemma 4.1.

Lemma 4.1 For linear model y = θ1xv + θ2xt + ϵ such that ϵ ∼ N (0, 1), wt(xv, πθ) = θ2
1

Assumption 4.1 Let h(x, y), abbreviate as h, be

h :=

(cid:34)

(cid:88)

y

πo(y|x)

(cid:19) 1
α

(cid:18) qw(y|x)
ql(y|x)

(cid:35)−1 (cid:18) qw(y|x)
ql(y|x)

(cid:19) 1
α

Assume that wt(xv, πo) < c2, where

(cid:115)(cid:13)
(cid:13)
(cid:13)
(cid:13)

c =

(cid:112)πo(y|x) ·

∂
∂xv

h

(cid:13)
(cid:13)
(cid:13)
(cid:13)

2

2

+

(cid:90) (cid:18) ∂
∂xv

h

(cid:19)2 πo(y|x)

h

dy −

(cid:13)
(cid:13)
(cid:13)
(cid:13)

(cid:112)πo(y|x) ·

∂
∂xv

(cid:13)
(cid:13)
h
(cid:13)
(cid:13)2

(7)

(8)

Assumption 4.1 requires that xv has a small weight in log πo(y|x). A model πo(y|x) independent of
xv could satisfy Assumption 4.1. In this case, the reference model generates answers without using
information from the image.

Theorem 4.1 Suppose that Assumption 4.1 holds, cross-modality loss increase the weight of xv.

wt(xv, πθ) > wt(xv, πo)

(9)

Theorem 4.1 indicates that when the weight of xv is too small in the initial model πo(y|x), the
cross-modality loss function adjusts the model to place greater emphasis on images, informed by
the retrieved data. Intuitively, for any sample (x, y), generating unrelated images causes the policy
to rely less on images. By using samples from this distribution as negative samples, the new model
diverges from the initial model, increasing its reliance on images.

4.2 THE IMPROVEMENT ON OVERALL ALIGNMENT

In this section, we analyze the improvement on overall alignment. Let q1
w(xv, yw,o|xt, xr) and
q1
l (xv, yl,o|xt) represent distributions of the preferred responses and dispreferred responses on
D1
oa, respectively; q2
w(xv, yw,o|xt) and q2
l (xv, yl,o|xt, xr) represent distributions of the preferred
responses and dispreferred responses on D2
oa, respectively. Overall loss is defined by
(cid:17)(cid:105)

(cid:16)

(cid:104)

Loa = −E(x,yw,o,yl,o)∼Doa

log σ

α log πθ(yw,o|x)

πo(yw,o|x) − α log πθ(yl,o|x)

πo(yl,o|x)

.

(10)

6

270
271

272
273
274
275
276
277

278
279
280
281
282
283

284
285
286
287
288

289
290
291
292
293
294

295
296
297
298
299
300

301
302
303
304
305
306

307
308
309
310
311

312
313
314
315
316
317

318
319
320
321
322
323

Under review as a conference paper at ICLR 2025

Consider π as the generative distribution underlying M, construction of D1
oa indicate that
there is a significant gap between π(y|xv, xt, xr) and π(y|xv, xt, ˜xr) for xr generates true answer
while ˜xr generate a false one.

oa and D2

Assumption 4.2 Assume that π(y|xx, xr, xt) : x → y is L-lipschitz continuous on xr for all
(xv, xt, y) such that |π(y|xv, xt, xr) − π(y|xv, xt, ˜xr)| ≤ L · dx(xr, ˜xr), where dx is any distance
metric on the text space.

Based on Assumption 4.2, ˜xr can be viewed as being far from the meaningful retrieved information
xr, resulting in different weight in the model. Then, we claim in the following theorem that the
overall loss in equation 10 can effectively leverage retrieved knowledge while training.

Assumption 4.3 Let h1(xv, xt, xr, y), abbreviate as h1, be

h1 :=

(cid:34)

(cid:88)

y

πo(y|x)

(cid:18) q1
w(y|xv, xt, xr) + q2
l (y|xv, xt) + q2
q1

w(y|xv, xt)
l (y|xv, xt, xr)

(cid:19) 1

α (cid:35)−1 (cid:18) q1
w(y|xv, xt, xr) + q2
l (y|xv, xt) + q2
q1

w(y|xv, xt)
l (y|xv, xt, xr)

Assume that wt(xr, πo) < c2

1 and wt(˜xr, πo) > c2
(cid:19)2 πo
h1

(cid:90) (cid:18) ∂h1
∂xr

2, where
(cid:13)
(cid:13)
(cid:13)
(cid:13)

∂h1
∂xr

dy −

(cid:13)
(cid:13)
(cid:13)
(cid:13)

+

2

2

√

πo ·

√

πo ·

∂h1
∂xr

(cid:13)
(cid:13)
(cid:13)
(cid:13)2

c1 =

c2 =

(cid:115)(cid:13)
(cid:13)
(cid:13)
(cid:13)
(cid:115)(cid:13)
(cid:13)
(cid:13)
(cid:13)

√

πo ·

∂h1
∂ ˜xr

(cid:13)
(cid:13)
(cid:13)
(cid:13)

2

2

+

(cid:90) (cid:18) ∂h1
∂ ˜xr

(cid:19)2 πo
h1

+

(cid:18) ∂πo
∂ ˜xr

(cid:19)2 h1
πo

dy +

√

πo ·

(cid:13)
(cid:13)
(cid:13)
(cid:13)

∂h1
∂ ˜xr

(cid:13)
(cid:13)
(cid:13)
(cid:13)2

(cid:19) 1

α

(11)

(12)

Theorem 4.2 Suppose that Assumption 4.3 holds, then overall loss 10 increase the weight of xr and
decrease the weight of ˜xr.

wt(xr, πθ) > wt(xr, πo), wt(˜xr, πθ) < wt(˜xr, πo)

(13)

Theorem 4.2 suggests that the model tend to improve the overall alignment. When ˜xr generates a
false answer, the training procedure tends to reduce the reliance on ˜xr, resulting in a decrease in the
weight assigned to ˜xr. Conversely, if xr is helpful for generating the true answer, πθ(y|x) tend to
enhance its use of xr.

5 EXPERIMENT

In this section, we evaluate the performance of MMed-RAG, aiming to answer the following
questions: (1) Can MMed-RAG effectively improve the factuality of Med-LVLMs compared to
decoding-based and RAG-based baselines? (2) How effective is each proposed component on per-
formance? (3) What is the effect of preference data for different alignment goals? and (4) Does
MMed-RAG actually improve cross-modality alignment and overall alignment?

5.1 EXPERIMENTAL SETUPS

Implementation Details. We use LLaVA-Med-1.5 7B (Li et al., 2023a) as the backbone model.
During the preference fine-tuning process, we adapt LoRA fine-tuning (Hu et al., 2021). For the
training of retriever, the vision encoder is a ResNet-50 (He et al., 2016), and the text encoder is a
bio-BioClinicalBERT (Alsentzer et al., 2019). We use the AdamW optimizer with a learning rate of
10−3, weight decay of 10−2 and a batch size of 32. The model is trained for 360 epochs. For more
detailed information on training hyperparameters and training data, please see Appendix A.1.1.

Baseline Methods. We compare MMed-RAG with two types of LVLM hallucination mitigation
methods that show promising results in natural image understanding. 1) Decoding-based methods,
including Greedy Decoding, Beam Search (Sutskever et al., 2014), DoLa (Chuang et al., 2023),
OPERA (Huang et al., 2023), VCD (Leng et al., 2023). These methods manipulate the logits of the
model’s output tokens to enhance factual accuracy. 2) Multimodal RAG-based methods, including
MedDr (He et al., 2024), FactMM-RAG (Sun et al., 2024b), RULE (Xia et al., 2024b). Furthermore,
we compare the performance with other open-source Med-LVLMs, including Med-Flamingo (Moor
et al., 2023), MedVInT (Zhang et al., 2023b), RadFM (Wu et al., 2023b).

7

324
325

326
327
328
329
330
331

332
333
334
335
336
337

338
339
340
341
342

343
344
345
346
347
348

349
350
351
352
353
354

355
356
357
358
359
360

361
362
363
364
365

366
367
368
369
370
371

372
373
374
375
376
377

Under review as a conference paper at ICLR 2025

378
379

380
381
382
383
384
385

386
387
388
389
390
391

392
393
394
395
396

397
398
399
400
401
402

403
404
405
406
407
408

409
410
411
412
413
414

415
416
417
418
419

420
421
422
423
424
425

426
427
428
429
430
431

Table 1: Model performance (%) of different methods based on LLaVA-Med-1.5 on medical VQA
task. Notably, we report the accuracy, F1 score and AUROC. The best results and second best results
are highlighted in red and blue , respectively.

Models

Radiology

Ophthalmology

Pathology

IU-Xray

MIMIC-CXR

Harvard-FairVLMed

Quilt-1M

PMC-OA (Pathology)

Acc

F1

AUC

Acc

F1

AUC

Acc

F1

AUC

Acc

F1

AUC

Acc

F1

AUC

LLaVA-Med-1.5

75.47

64.04

67.46

75.79

80.49

68.84

63.03

74.11

63.05

62.80

72.90

60.03

59.28

71.98

54.19

+ Greedy
+ Beam Search
+ DoLa
+ OPERA
+ VCD

76.88
76.91
78.00
70.59
68.99

+ MedDr
83.33
+ FactMM-RAG 84.51
87.84
+ RULE

65.59
66.06
66.75
61.54
54.35

67.80
68.51
78.00

68.74
68.77
72.19
63.22
61.08

77.15
77.07
85.78

78.32
81.56
81.35
69.34
70.89

55.16
77.58
83.92

86.75
86.36
85.73
76.66
75.57

56.18
81.86
87.49

71.13
73.79
72.73
62.46
64.61

58.47
70.09
83.44

82.54
80.93
76.87
71.41
65.88

70.17
83.67
87.12

85.98
88.08
85.53
81.37
77.20

80.72
87.21
92.89

70.09
68.94
67.10
65.59
64.16

64.15
72.20
77.08

64.72
63.52
63.47
60.51
61.43

68.15
69.25
68.97

70.12
69.33
69.10
66.32
67.39

73.23
73.62
73.80

58.75
57.65
57.58
54.79
55.72

67.01
68.15
68.13

58.61
56.29
57.71
55.32
55.10

59.97
60.49
61.41

70.42
69.84
70.27
68.30
67.94

69.19
69.38
70.36

53.10
52.89
52.95
51.86
51.62

57.01
57.31
58.91

MMed-RAG

89.54

80.72

87.13

83.57

88.49

85.08

87.94

92.78

80.81

72.95

76.35

72.25

64.54

73.09

61.42

Table 2: Model performance (%) of different methods on report generation task. Notably, we report
the average BLEU score, ROUGE-L, METEOR. For detailed BLEU score, see Appendix A.6.8.

Models

Radiology

IU-Xray

MIMIC-CXR

Ophthalmology

Harvard-FairVLMed

BLEU ROUGE-L METEOR BLEU ROUGE-L METEOR

BLEU ROUGE-L METEOR

LLaVA-Med-1.5

9.64

+ Greedy
+ Beam Search
+ DoLa
+ OPERA
+ VCD

+ MedDr
+ FactMM-RAG
+ RULE

MMed-RAG

11.47
12.10
11.79
10.66
10.42

12.37
14.70
27.53

31.38

12.26

15.38
16.21
15.82
14.70
14.14

16.45
18.05
23.16

25.59

8.21

12.69
13.17
12.72
12.01
11.59

13.50
15.92
27.99

32.43

12.11

16.63
16.97
17.11
15.40
15.18

18.59
18.71
18.61

23.25

13.05

14.26
14.74
14.89
12.52
12.30

15.72
15.84
15.96

12.34

11.16

14.19
14.43
14.81
13.72
13.38

16.77
16.82
17.42

20.47

18.11

17.98
18.37
18.26
16.59
16.73

19.82
20.82
22.35

24.82

11.36

11.49
12.62
12.51
11.47
11.38

13.72
14.17
14.93

16.59

10.75

13.77
14.50
14.51
13.63
13.89

15.40
15.31
17.74

19.85

Evaluation Datasets. We utilize five medical vision-language datasets for medical VQA and report
generation tasks, i.e., MIMIC-CXR (Johnson et al., 2019), IU-Xray (Demner-Fushman et al., 2016),
Harvard-FairVLMed (Luo et al., 2024), PMC-OA (Lin et al., 2023a) (we only select the pathology
part) and Quilt-1M (Ikezogwo et al., 2024). These datasets cover radiology, ophthalmology, and
pathology. To construct the VQA benchmarks, following (Xia et al., 2024a), we generate question-
answer pairs from medical reports using GPT-4 (OpenAI, 2023), with answers formatted as yes or
no. Pathology images are excluded from the report generation task due to their brief and insufficient
descriptions. The detailed dataset descriptions are provided in the Appendix A.2.

Evaluation Metrics. Following (Jing et al., 2017; Lin et al., 2023b), we use Accuracy, F1 Score and
AUROC for evaluating medical VQA task, and BLEU Score (Papineni et al., 2002), ROUGE-L (Lin,
2004) and METEOR (Banerjee & Lavie, 2005) for evaluating report generation task.

5.2 MAIN RESULTS

In this section, we provide a comprehensive comparison with various baseline methods and other
open-source Med-LVLMs on medical VQA and report generation tasks.

Comparison with Baselines. We compare MMed-RAG with baseline methods on medical VQA
and report generation tasks, with the results presented in Table 1 and Table 2, respectively. Overall,
MMed-RAG outperforms all baselines across nearly all metrics and datasets. Specifically, MMed-
RAG demonstrates a significant performance boost, improving by 18.5% and 69.1% over the orig-
inal Med-LVLM in medical VQA and report generation tasks, respectively. When compared to
baseline methods, MMed-RAG surpasses decoding-based approaches, achieving improvements of
11.5% and 44.2% in the two tasks. Furthermore, recent RAG-based methods show substantial im-
provements over earlier techniques, yet our approach still outperforms RAG-based baselines by
2.8% and 16.1% in the medical VQA and report generation tasks, respectively. This indicates that
MMed-RAG effectively mitigates misalignment issues introduced by RAG. Notably, MMed-RAG
achieves more pronounced gains in report generation, likely due to the higher complexity of the task
and the greater influence of retrieved contexts in guiding open-ended generation.

8

Under review as a conference paper at ICLR 2025

Comparison with Other Med-LVLMs. To provide a com-
prehensive comparison, we evaluate MMed-RAG against other
open-source Med-LVLMs to demonstrate the superiority of our
approach. We assess the performance of these models across
different medical image modalities, reporting the average re-
sults for medical VQA and report generation tasks in Table 3
(see Appendix A.6 for detailed results). Our findings show that
MMed-RAG significantly outperforms Med-LVLMs pre-trained
on large-scale datasets across various domains. This reinforces
the generalizability and effectiveness of our approach across di-
verse image domains and medical multimodal tasks.

Table 3: Performance compar-
ison with several Med-LVLMs.
Rad: Radiology, Opt: Ophthalo-
mology, Pat: Pathology.

Model

Rad

Opt

Pat

Med-Flamingo
MedVInT
RadFM
miniGPT-Med
MMed-RAG

27.42
33.17
35.82
36.66
56.94

22.50
29.40
27.07
25.28
56.38

29.11
25.33
24.82
23.16
54.10

5.3 ANALYSIS

In this section, we provide a detailed analysis of each module’s performance, along with a series of
analytical experiments, to better understand the performance gains of MMed-RAG. Additionally, we
demonstrate the compatibility of our method in Appendix A.6, including its application to generalist
and domain-specific Med-LVLMs. In Appendix A.6.7 and A.6.4, we show the strong performance
of our method on external validation datasets and in the environmental systems domain.

Table 4: Ablation results on two datasets
covering different domains. RG: report gen-
eration, FairVLMed: Harvard-FairVLMed.

Ablation Studies. We conduct a series of ablation
experiments to evaluate the impact of each compo-
nent in MMed-RAG. The results for both medical
VQA and report generation tasks on the IU-Xray and
Harvard-FairVLMed datasets are summarized in Ta-
ble 4. According to the results, we can see that: (1)
The domain-aware retrieval mechanism (DR) sig-
nificantly improves the factuality of Med-LVLM,
with an average performance increase of 17.9% and
16.1% on the IU-Xray and FairVLMed datasets, re-
spectively. Here, the retrieved knowledge aids the
model in generating more factual responses. (2) Building on this, the introduction of adaptive re-
trieval context selection (RCS) further filters out unreliable retrieved contexts, yielding an additional
performance boost of 19.3% and 6.3% on the IU-Xray and FairVLMed datasets. (3) The inclusion of
RAG-based preference fine-tuning (RAG-PT) enhances the model’s understanding of the retrieved
knowledge, leading to substantial performance gains of 37.1% and 16.9% on the respective datasets.
This demonstrates that RAG-PT effectively addresses misalignment issues.

LLaVA-Med-1.5
+DR
+RCS
+RAG-PT (Ours)

FairVLMed
RG

68.99
77.12
79.56
85.80

13.41
15.89
17.22
20.42

10.04
13.23
17.92
29.80

66.63
72.69
75.74
87.18

IU-Xray

Model

VQA

VQA

RG

Table 5: Performance using RAG-PT based
on subsets of preference data.

Impact of the Preference Data in RAG-PT. To
better understand how RAG-PT mitigates the mis-
alignment issue and improves performance, we con-
ducted a detailed study on the training preference
data composition of RAG-PT. As described in Sec-
tion 3.3, the RAG-PT data is designed to address
both cross-modality alignment and overall align-
ment objectives, with the latter focusing on en-
hanced understanding of retrieved knowledge and
minimizing retrieval interference. The detailed experimental results in Table 5 demonstrate that
the preference data tailored for different alignment objectives positively impacts the model’s perfor-
mance, showing the effectiveness of RAG-PT. We present additional ablation results on preference
data in Appendix A.6.6.

LLaVA-Med-1.5
+RAG-PT 1
+RAG-PT 2
+RAG-PT 3

FairVLMed
RG

10.04
19.38
20.16
19.43

13.41
18.37
18.66
18.92

66.63
79.42
79.35
80.07

68.99
80.19
80.27
81.30

IU-Xray

Model

VQA

VQA

RG

How Effective is MMed-RAG in Mitigating Misalignment Issues? To gain a more intuitive un-
derstanding of the effectiveness of MMed-RAG in addressing misalignment issues: 1) we calculate
the proportion of errors caused by RAG and compare it to the proportion after incorporating MMed-
RAG. 2) We visualize the attention maps of image and text tokens with and without RAG-PT. First,
as mentioned in Section 3.3, the model may directly copy reference information, referred to as
Copy-Reference (CR) rate. After applying MMed-RAG, as shown in Figure 3, the CR rate drops to
28.19%. Additionally, the proportion of errors affected by RAG interference, referred to as Over-
Reliance (OR) rate, which is initially 43.31%, decreased to 8.38% after incorporating MMed-RAG.
Furthermore, as shown in Figure 4, the original Med-LVLM tends to rely more heavily on text while

9

432
433

434
435
436
437
438
439

440
441
442
443
444
445

446
447
448
449
450

451
452
453
454
455
456

457
458
459
460
461
462

463
464
465
466
467
468

469
470
471
472
473

474
475
476
477
478
479

480
481
482
483
484
485

Under review as a conference paper at ICLR 2025

ignoring visual information. When retrieval information is introduced, the original Med-LVLM fo-
cused more on the retrieved answers, even if the content is incorrect. After RAG-PT, the model
significantly increases its attention to visual information and reduces the interference of RAG, thus
better aligning the model’s knowledge with the fundamental facts.

Figure 3: Alignment analysis
with and without RAG. OR:
Over-Reliance; CR: Copy-
Reference.

6 RELATED WORK

Figure 4: Visualization of attention map. The red box region is
labeled with the attentions that can be enhanced by MMed-RAG.

Factuality in Med-LVLMs.
The rapid advancements in Large Vision-Language Models
(LVLMs) (Liu et al., 2024a;b) are beginning to influence the field of medical image analysis. Sev-
eral Med-LVLMs (Li et al., 2023a; Moor et al., 2023; Zhang et al., 2023b; Wu et al., 2023b), have
emerged, showing remarkable performance across different medical imaging modalities. Despite
these advances, Med-LVLMs continue to present notable factual hallucination (Xia et al., 2024a;
Royer et al., 2024), generating textual outputs that contradict medical visual information. This
raises concerns about potential misdiagnoses or overlooked conditions. Recently, benchmarks have
been developed to assess the accuracy of Med-LVLMs in tasks such as visual question answering
(VQA) and report generation (Xia et al., 2024a; Royer et al., 2024). However, research aimed at
enhancing the factual accuracy of Med-LVLMs remains relatively unexplored.

Retrieval Augmented Generation in Med-LVLMs. Retrieval-Augmented Generation (RAG) has
proven to be a powerful technique for enhancing factual accuracy in language modeling (Gao et al.,
2023; Wu et al., 2023c; Chen et al., 2024c; Qu et al., 2024; Sun et al., 2024a). In the biomedi-
cal domain, RAG leverages external knowledge to guide the generation of Med-LVLMs, offering
clear advantages in tasks such as medical VQA and report generation (Yuan et al., 2023; Kumar
& Marttinen, 2024; Tao et al., 2024; He et al., 2024; Sun et al., 2024b). However, these works
mainly focus on enhancing the relevance of the retrieved contexts without considering the model’s
understanding of retrieved knowledge. There are several recent work on RAG fine-tuning in LLMs.
DPA-RAG (Dong et al., 2024) addresses the alignment issues between the external reranker and the
internal LLM through supervised fine-tuning. Then RAG-DDR (Li et al., 2024b) leverages a rolling
method to generate perturbed responses, further mitigating conflicts between parameter memory
and external knowledge. In the biomedical domain, RULE (Xia et al., 2024b) is proposed to use
preference fine-tuning to reduce the model’s over-reliance on retrieved contexts. However, it still
overlooks misalignment issues caused by RAG, as well as the generalizability of the retriever given
the diverse domains of input images. In response, we propose MMed-RAG to mitigate these risks,
enhancing the factuality of Med-LVLMs by addressing these overlooked factors. This can lead to
a better cross-modality and overall alignment to enhance the understanding of retrieved knowledge
and visual information, ensuring more consistent and reliable performance across tasks.

7 CONCLUSION

This paper introduces MMed-RAG, a versatile multimodal RAG system designed to address the
critical issue of factual hallucination in Med-LVLMs. MMed-RAG employs a domain-aware re-
trieval mechanism, adaptive calibration for selecting the optimal number of retrieved contexts, and
RAG-based preference fine-tuning to improve both cross-modality alignment and overall alignment
with the ground truth. These enhancements significantly boost the factual accuracy of Med-LVLMs.
Experimental results demonstrate MMed-RAG’ effectiveness in enhancing factual accuracy across
various imaging domains, underscoring its potential for reliable use in healthcare. Our findings
underscore the importance of incorporating robust multimodal RAG mechanism to ensure that Med-
LVLMs can serve as dependable tools in clinical settings.

10

486
487

488
489
490
491
492
493

494
495
496
497
498
499

500
501
502
503
504

505
506
507
508
509
510

511
512
513
514
515
516

517
518
519
520
521
522

523
524
525
526
527

528
529
530
531
532
533

534
535
536
537
538
539

LLaVA-Med-1.5MMed-RAG (Ours)Text TokensImage TokensCan focal airspace consolidation be seen on the image?QuestionMedical ImageReferenceNo, focal airspace consolidation cannot be seen on the image.OursLLaVA-Med-1.5Yes, there seems to be a focal airspace consolidation.The heart is normal in size…There appears to be a focal airspace consolidation on the right side of the lung…Under review as a conference paper at ICLR 2025

ETHICS STATEMENT

This paper presents a novel RAG-based approach to enhancing the factuality of Med-LVLMs. We
have followed best practices in data collection, model design, and evaluation, ensuring adherence
to privacy and ethical standards. All datasets used are sourced from publicly available medical
datasets or collected with appropriate ethical considerations, including patient data anonymization.
We adhere to principles of research integrity and transparency, and comply with all relevant regula-
tions. We hope that our research will contribute to safer, more reliable AI-assisted medical tools and
advance healthcare technology responsibly.

REPRODUCIBILITY STATEMENT

We have taken significant steps to ensure that our work is reproducible. All details regarding our pro-
posed multimodal RAG system, including the domain-aware retrieval mechanism, adaptive retrieved
context selection, and RAG-based preference fine-tuning strategy, are described comprehensively in
Section 3. We provide the hyperparameter settings and experimental configurations used in our eval-
uations in Section 5.1 and Appendix A.1.2. Additionally, we have included detailed pseudocode for
the proposed algorithms in Algorithm 1 and an in-depth explanation of the data processing steps for
each medical dataset used in Appendix A.1.1 and Appendix A.2.

REFERENCES

Asma Alkhaldi, Raneem Alnajim, Layan Alabdullatef, Rawan Alyahya, Jun Chen, Deyao Zhu,
Ahmed Alsinan, and Mohamed Elhoseiny. Minigpt-med: Large language model as a general
interface for radiology diagnosis. arXiv preprint arXiv:2407.04106, 2024.

Emily Alsentzer, John R Murphy, Willie Boag, Wei-Hung Weng, Di Jin, Tristan Naumann,
arXiv preprint

Publicly available clinical bert embeddings.

and Matthew McDermott.
arXiv:1904.03323, 2019.

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang
Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, local-
ization, text reading, and beyond. 2023.

Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved
correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic
evaluation measures for machine translation and/or summarization, pp. 65–72, 2005.

Jiawei Chen, Dingkang Yang, Tong Wu, Yue Jiang, Xiaolu Hou, Mingcheng Li, Shunli Wang,
Dongling Xiao, Ke Li, and Lihua Zhang. Detecting and evaluating medical hallucinations in
large vision language models. arXiv preprint arXiv:2406.10185, 2024a.

Junying Chen, Ruyi Ouyang, Anningzhe Gao, Shunian Chen, Guiming Hardy Chen, Xidong Wang,
Ruifei Zhang, Zhenyang Cai, Ke Ji, Guangjun Yu, et al. Huatuogpt-vision, towards injecting
medical visual knowledge into multimodal llms at scale. arXiv preprint arXiv:2406.19280, 2024b.

Zhanpeng Chen, Chengjin Xu, Yiyan Qi, and Jian Guo. Mllm is a strong reranker: Advancing
multimodal retrieval-augmented generation via knowledge-enhanced reranking and noise-injected
training. arXiv preprint arXiv:2407.21439, 2024c.

Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong,
Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to com-
mercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821, 2024d.

Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning
converts weak language models to strong language models, 2024e. URL https://arxiv.org/
abs/2401.01335.

Cl´ement Christophe, Praveen K Kanithi, Tathagata Raha, Shadab Khan, and Marco AF Pimentel.

Med42-v2: A suite of clinical llms. arXiv preprint arXiv:2408.06142, 2024.

11

540
541

542
543
544
545
546
547

548
549
550
551
552
553

554
555
556
557
558

559
560
561
562
563
564

565
566
567
568
569
570

571
572
573
574
575
576

577
578
579
580
581

582
583
584
585
586
587

588
589
590
591
592
593

Under review as a conference paper at ICLR 2025

594
595

596
597
598
599
600
601

602
603
604
605
606
607

608
609
610
611
612

613
614
615
616
617
618

619
620
621
622
623
624

625
626
627
628
629
630

631
632
633
634
635

636
637
638
639
640
641

642
643
644
645
646
647

Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. Dola:
Decoding by contrasting layers improves factuality in large language models. arXiv preprint
arXiv:2309.03883, 2023.

Dina Demner-Fushman, Marc D Kohli, Marc B Rosenman, Sonya E Shooshan, Laritza Rodriguez,
Sameer Antani, George R Thoma, and Clement J McDonald. Preparing a collection of radiol-
ogy examinations for distribution and retrieval. Journal of the American Medical Informatics
Association, 23(2):304–310, 2016.

Guanting Dong, Yutao Zhu, Chenghao Zhang, Zechen Wang, Zhicheng Dou, and Ji-Rong Wen.
Understand what llm needs: Dual preference alignment for retrieval-augmented generation. arXiv
preprint arXiv:2406.18676, 2024.

Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and
Haofen Wang. Retrieval-augmented generation for large language models: A survey. arXiv
preprint arXiv:2312.10997, 2023.

Pierre Gravel, Gilles Beaudoin, and Jacques A De Guise. A method for modeling noise in medical

images. IEEE Transactions on medical imaging, 23(10):1221–1232, 2004.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog-
nition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.
770–778, 2016.

Sunan He, Yuxiang Nie, Zhixuan Chen, Zhiyuan Cai, Hongmei Wang, Shu Yang, and Hao Chen.
Meddr: Diagnosis-guided bootstrapping for large-scale medical vision-language learning. arXiv
preprint arXiv:2404.15127, 2024.

Robbie Holland, Thomas RP Taylor, Christopher Holmes, Sophie Riedl, Julia Mai, Maria Patsia-
manidi, Dimitra Mitsopoulou, Paul Hager, Philip M¨uller, Hendrik PN Scholl, et al. Specialist
vision-language models for clinical ophthalmology. arXiv preprint arXiv:2407.08410, 2024.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang,
arXiv preprint

and Weizhu Chen. Lora: Low-rank adaptation of large language models.
arXiv:2106.09685, 2021.

Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming
Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi-modal large language models
via over-trust penalty and retrospection-allocation. arXiv preprint arXiv:2311.17911, 2023.

Wisdom Ikezogwo, Saygin Seyfioglu, Fatemeh Ghezloo, Dylan Geva, Fatwir Sheikh Mohammed,
Pavan Kumar Anand, Ranjay Krishna, and Linda Shapiro. Quilt-1m: One million image-text
pairs for histopathology. Advances in neural information processing systems, 36, 2024.

Yue Jiang, Jiawei Chen, Dingkang Yang, Mingcheng Li, Shunli Wang, Tong Wu, Ke Li, and Lihua
Zhang. Medthink: Inducing medical large-scale visual language models to hallucinate less by
thinking more. arXiv preprint arXiv:2406.11451, 2024.

Baoyu Jing, Pengtao Xie, and Eric Xing. On the automatic generation of medical imaging reports.

arXiv preprint arXiv:1711.08195, 2017.

Alistair EW Johnson, Tom J Pollard, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng,
Yifan Peng, Zhiyong Lu, Roger G Mark, Seth J Berkowitz, and Steven Horng. Mimic-cxr-jpg, a
large publicly available database of labeled chest radiographs. arXiv preprint arXiv:1901.07042,
2019.

Yogesh Kumar and Pekka Marttinen.

Improving medical multi-modal contrastive learning with

expert annotations. arXiv preprint arXiv:2403.10153, 2024.

Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing.
Mitigating object hallucinations in large vision-language models through visual contrastive de-
coding. arXiv preprint arXiv:2311.16922, 2023.

12

Under review as a conference paper at ICLR 2025

648
649

650
651
652
653
654
655

656
657
658
659
660
661

662
663
664
665
666

667
668
669
670
671
672

673
674
675
676
677
678

679
680
681
682
683
684

685
686
687
688
689

690
691
692
693
694
695

696
697
698
699
700
701

Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Nau-
mann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language-and-vision assis-
tant for biomedicine in one day. In Thirty-seventh Conference on Neural Information Processing
Systems Datasets and Benchmarks Track, 2023a.

Haoran Li, Junqi Liu, Zexian Wang, Shiyuan Luo, Xiaowei Jia, and Huaxiu Yao.

Lite:
Modeling environmental ecosystems with multimodal large language models. arXiv preprint
arXiv:2404.01165, 2024a.

Xinze Li, Sen Mei, Zhenghao Liu, Yukun Yan, Shuo Wang, Shi Yu, Zheni Zeng, Hao Chen, Ge Yu,
Zhiyuan Liu, et al. Rag-ddr: Optimizing retrieval-augmented generation using differentiable data
rewards. arXiv preprint arXiv:2410.13509, 2024b.

Yingshu Li, Yunyi Liu, Zhanyu Wang, Xinyu Liang, Lingqiao Liu, Lei Wang, Leyang Cui, Zhaopeng
Tu, Longyue Wang, and Luping Zhou. A comprehensive study of gpt-4v’s multimodal capabilities
in medical imaging. arXiv preprint arXiv:2310.20381, 2023b.

Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization

branches out, pp. 74–81, 2004.

Weixiong Lin, Ziheng Zhao, Xiaoman Zhang, Chaoyi Wu, Ya Zhang, Yanfeng Wang, and Weidi Xie.
Pmc-clip: Contrastive language-image pre-training using biomedical documents. In International
Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 525–536.
Springer, 2023a.

Zhihong Lin, Donghao Zhang, Qingyi Tao, Danli Shi, Gholamreza Haffari, Qi Wu, Mingguang
He, and Zongyuan Ge. Medical visual question answering: A survey. Artificial Intelligence in
Medicine, 143:102611, 2023b.

Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee.

Improved baselines with visual in-
struction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern
Recognition, pp. 26296–26306, 2024a.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances

in neural information processing systems, 36, 2024b.

Yan Luo, Min Shi, Muhammad Osama Khan, Muhammad Muneeb Afzal, Hao Huang, Shuaihang
Yuan, Yu Tian, Luo Song, Ava Kouhana, Tobias Elze, et al. Fairclip: Harnessing fairness in
vision-language learning. arXiv preprint arXiv:2403.19949, 2024.

Fanqing Meng, Jin Wang, Chuanhao Li, Quanfeng Lu, Hao Tian, Jiaqi Liao, Xizhou Zhu, Jifeng
Dai, Yu Qiao, Ping Luo, et al. Mmiu: Multimodal multi-image understanding for evaluating large
vision-language models. arXiv preprint arXiv:2408.02718, 2024.

Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Yash Dalmia, Jure Leskovec, Cyril
Zakka, Eduardo Pontes Reis, and Pranav Rajpurkar. Med-flamingo: a multimodal medical few-
shot learner. In Machine Learning for Health (ML4H), pp. 353–367. PMLR, 2023.

OpenAI. Gpt-4 technical report, 2023. https://arxiv.org/abs/2303.08774.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic
evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association
for Computational Linguistics, pp. 311–318, 2002.

Xiaoye Qu, Qiyuan Chen, Wei Wei, Jishuo Sun, and Jianfeng Dong. Alleviating halluci-
arXiv preprint

nation in large vision-language models with active retrieval augmentation.
arXiv:2408.00555, 2024.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar-
wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya
Sutskever. Learning transferable visual models from natural language supervision, 2021.

13

Under review as a conference paper at ICLR 2025

702
703

704
705
706
707
708
709

710
711
712
713
714
715

716
717
718
719
720

721
722
723
724
725
726

727
728
729
730
731
732

733
734
735
736
737
738

739
740
741
742
743

744
745
746
747
748
749

750
751
752
753
754
755

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea
Finn. Direct preference optimization: Your language model is secretly a reward model.
In
Thirty-seventh Conference on Neural Information Processing Systems, 2023.

Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-
baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gem-
ini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint
arXiv:2403.05530, 2024.

Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, Andr´e
Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts.
Advances in Neural Information Processing Systems, 34:8583–8595, 2021.

Corentin Royer, Bjoern Menze, and Anjany Sekuboyina. Multimedeval: A benchmark and a toolkit

for evaluating medical vision-language models. arXiv preprint arXiv:2402.09262, 2024.

Ma Guadalupe Sanchez, Ma Guadalupe S´anchez, Vicente Vidal, Gumersindo Verdu, Gumersindo
Verd´u, Patricia Mayo, and Francisco Rodenas. Medical image restoration with different types
In 2012 Annual International Conference of the IEEE Engineering in Medicine and
of noise.
Biology Society, pp. 4382–4385. IEEE, 2012.

Jessica Schrouff, Natalie Harris, Sanmi Koyejo, Ibrahim M Alabdulmohsin, Eva Schnider, Krista
Opsahl-Ong, Alexander Brown, Subhrajit Roy, Diana Mincu, Christina Chen, et al. Diagnosing
failures of fairness transfer across distribution shift in real-world medical settings. Advances in
Neural Information Processing Systems, 35:19304–19318, 2022.

Mehmet Saygin Seyfioglu, Wisdom O Ikezogwo, Fatemeh Ghezloo, Ranjay Krishna, and Linda
Shapiro. Quilt-llava: Visual instruction tuning by extracting localized narratives from open-source
histopathology videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pp. 13183–13192, 2024.

Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton,
and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.
arXiv preprint arXiv:1701.06538, 2017.

Congzhen Shi, Ryan Rezai, Jiaxi Yang, Qi Dou, and Xiaoxiao Li. A survey on trustworthiness in

foundation models for medical image analysis. arXiv preprint arXiv:2407.15851, 2024.

Jiashuo Sun, Jihai Zhang, Yucheng Zhou, Zhaochen Su, Xiaoye Qu, and Yu Cheng. Surf: Teach-
arXiv preprint

ing large vision-language models to selectively utilize retrieved information.
arXiv:2409.14083, 2024a.

Liwen Sun, James Zhao, Megan Han, and Chenyan Xiong. Fact-aware multimodal retrieval aug-
mentation for accurate medical radiology report generation. arXiv preprint arXiv:2407.15268,
2024b.

Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks.

In Advances in neural information processing systems, pp. 3104–3112, 2014.

Yitian Tao, Liyan Ma, Jing Yu, and Han Zhang. Memory-based cross-modal semantic alignment
network for radiology report generation. IEEE Journal of Biomedical and Health Informatics,
2024.

Alexandra-Maria T˘aut¸an, Bogdan Ionescu, and Emiliano Santarnecchi. Artificial intelligence in neu-
rodegenerative diseases: A review of available tools with a focus on machine learning techniques.
Artificial Intelligence in Medicine, 117:102081, 2021.

Omkar Thawkar, Abdelrahman Shaker, Sahal Shaji Mullappilly, Hisham Cholakkal, Rao Muham-
mad Anwer, Salman Khan, Jorma Laaksonen, and Fahad Shahbaz Khan. Xraygpt: Chest radio-
graphs summarization using medical vision-language models. arXiv preprint arXiv:2306.07971,
2023.

14

Under review as a conference paper at ICLR 2025

756
757

758
759
760
761
762
763

764
765
766
767
768
769

770
771
772
773
774

775
776
777
778
779
780

781
782
783
784
785
786

787
788
789
790
791
792

793
794
795
796
797

798
799
800
801
802
803

804
805
806
807
808
809

Robert Tibshirani, Guenther Walther, and Trevor Hastie. Estimating the number of clusters in
Journal of the Royal Statistical Society: Series B (Statistical

a data set via the gap statistic.
Methodology), 63(2):411–423, 2001.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko-
lay Bashlykov, Soumya Batra, Prajjwal Bhargava, and Shruti Bhosale. Llama 2: Open foundation
and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.

Tao Tu, Shekoofeh Azizi, Danny Driess, Mike Schaekermann, Mohamed Amin, Pi-Chuan Chang,
Andrew Carroll, Charles Lau, Ryutaro Tanno, Ira Ktena, et al. Towards generalist biomedical ai.
NEJM AI, 1(3):AIoa2300138, 2024.

Chunhao Wang, Xiaofeng Zhu, Julian C Hong, and Dandan Zheng. Artificial intelligence in radio-
therapy treatment planning: present and future. Technology in cancer research & treatment, 18:
1533033819873922, 2019.

Xiyao Wang, Yuhang Zhou, Xiaoyu Liu, Hongjin Lu, Yuancheng Xu, Feihong He, Jaehong Yoon,
Taixi Lu, Gedas Bertasius, Mohit Bansal, et al. Mementos: A comprehensive benchmark for mul-
timodal large language model reasoning over image sequences. arXiv preprint arXiv:2401.10529,
2024.

Zhepei Wei, Wei-Lin Chen, and Yu Meng. Instructrag: Instructing retrieval-augmented generation

with explicit denoising. arXiv preprint arXiv:2406.13629, 2024.

Chaoyi Wu, Jiayu Lei, Qiaoyu Zheng, Weike Zhao, Weixiong Lin, Xiaoman Zhang, Xiao Zhou,
Ziheng Zhao, Ya Zhang, Yanfeng Wang, et al. Can gpt-4v (ision) serve medical applications? case
studies on gpt-4v for multimodal medical diagnosis. arXiv preprint arXiv:2310.09909, 2023a.

Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Towards generalist foun-

dation model for radiology. arXiv preprint arXiv:2308.02463, 2023b.

Yuanhao Wu, Juno Zhu, Siliang Xu, Kashun Shum, Cheng Niu, Randy Zhong, Juntong Song, and
Tong Zhang. Ragtruth: A hallucination corpus for developing trustworthy retrieval-augmented
language models. arXiv preprint arXiv:2401.00396, 2023c.

Peng Xia, Ze Chen, Juanxi Tian, Yangrui Gong, Ruibo Hou, Yue Xu, Zhenbang Wu, Zhiyuan Fan,
Yiyang Zhou, Kangyu Zhu, et al. Cares: A comprehensive benchmark of trustworthiness in
medical vision language models. arXiv preprint arXiv:2406.06007, 2024a.

Peng Xia, Kangyu Zhu, Haoran Li, Hongtu Zhu, Yun Li, Gang Li, Linjun Zhang, and Huaxiu Yao.
Rule: Reliable multimodal rag for factuality in medical vision language models. arXiv preprint
arXiv:2407.05131, 2024b.

Qing Ye, Chang-Yu Hsieh, Ziyi Yang, Yu Kang, Jiming Chen, Dongsheng Cao, Shibo He, and
Tingjun Hou. A unified drug–target interaction prediction framework based on knowledge graph
and recommendation system. Nature communications, 12(1):6775, 2021.

Zheng Yuan, Qiao Jin, Chuanqi Tan, Zhengyun Zhao, Hongyi Yuan, Fei Huang, and Songfang
Huang. Ramm: Retrieval-augmented biomedical visual question answering with multi-modal
pre-training. In Proceedings of the 31st ACM International Conference on Multimedia, pp. 547–
556, 2023.

Jihai Zhang, Xiaoye Qu, Tong Zhu, and Yu Cheng. Clip-moe: Towards building mixture of experts

for clip with diversified multiplet upcycling. arXiv preprint arXiv:2409.19291, 2024.

Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Pre-
ston, Rajesh Rao, Mu Wei, Naveen Valluri, et al. Biomedclip: a multimodal biomedical
foundation model pretrained from fifteen million scientific image-text pairs.
arXiv preprint
arXiv:2303.00915, 2023a.

Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi
Xie. Pmc-vqa: Visual instruction tuning for medical visual question answering. arXiv preprint
arXiv:2305.10415, 2023b.

15

Under review as a conference paper at ICLR 2025

A EXPERIMENT

A.1 EXPERIMENTAL SETUP

A.1.1 DATA STATISTICS

The data quantities used in this study are presented in Table 6, Table 7 and Table 8. We clarify that
for training the retriever, the data refers to the number of image-text pairs, while for fine-tuning,
it refers to the number of QA items. The “All” category represents the total amount of data used
to construct the preference dataset for RAG-PT. The training of RAG-PT includes three types of
samples: (a) clean samples with originally correct answers that remain correct even after adding
noise to the images, (b) clean image samples with originally incorrect answers that become correct,
and (c) clean image samples with originally correct answers that become incorrect.

Table 6: Data statistics for medical VQA task. ”Train (DR)” refers to the number of image-text pairs
for retriever training, ”All (RAG-PT)” refers to the total data for RAG-PT, and ”Train (RAG-PT)-
a/b/c” refer to the respective subsets for RAG-PT training.

Dataset

Train (DR) All (RAG-PT)

Train (RAG-PT)-a

Train (RAG-PT)-b

Train (RAG-PT)-c

Ophthalomology
Radiology
Pathology

7000
4034
5000

3247
4836
1990

1082
1612
663

1030
1989
523

1135
1235
804

Table 7: Data statistics for report generation. ”Train (DR)” refers to the number of image-text pairs
for retriever training, ”All (RAG-PT)” refers to the total data for RAG-PT, and ”Train (RAG-PT)-
a/b/c” refer to the respective sample categories for RAG-PT training.

Dataset

Train (R) All (RAG-PT)

Train (RAG-PT)-a

Train (RAG-PT)-b

Train (RAG-PT)-c

Ophthalmology
Radiology

7000
4034

3247
4836

142
233

78
126

207
342

Table 8: Data statistics for various datasets. The rows represent the number of images and QA pairs
for each dataset.

Harvard-FairVLMed

IU-Xray MIMIC-CXR PMC-OA Quilt-1M

# Images
# QA Items

713
4285

589
2573

700
3470

530
3124

559
1994

A.1.2 HYPERPARAMETER SETTINGS

Following the settings of CLIP (Radford et al., 2021), we adopt the same architecture and hyperpa-
rameters for the vision and text encoders. The vision encoder is a ResNet-50 (He et al., 2016), and
the text encoder is a bio-bert-based model (Alsentzer et al., 2019). We use the AdamW optimizer
with a learning rate of 10−4 and a batch size of 512. The model is trained for 360 epochs. For the
first phase, we trained for 3 epochs, and for the second phase, the training was conducted for 1 epoch.
Training for 20 hours on one A100 80G GPU. For the RAG-PT phase, we adjust the diffusion noise
level, symbolized by ξ through a specific formula: ξ = Sigmoid(lt) × (0.5 × 10−2 − 10−5) + 10−5,
where ϵ is drawn from a normal distribution. The reports available for retrieval are from the training
set of the corresponding dataset. In our experiments, we apply cross-validation to tune all hyper-
parameters with grid search. All the experiments are implemented on PyTorch 2.1.2 using four
NVIDIA RTX A6000 GPUs. It takes roughly 3 and 4 hours for fine-tuning CLIP and LLaVA-Med-
1.5 7B, respectively.

A.2 EVALUATED DATASETS

We utilize five open-source medical vision-language datasets, i.e., MIMIC-CXR (Johnson et al.,
2019), IU-Xray (Demner-Fushman et al., 2016), Harvard-FairVLMed (Luo et al., 2024), PMC-
OA (Lin et al., 2023a) and Quilt-1M (Ikezogwo et al., 2024).

16

810
811

812
813
814
815
816
817

818
819
820
821
822
823

824
825
826
827
828

829
830
831
832
833
834

835
836
837
838
839
840

841
842
843
844
845
846

847
848
849
850
851

852
853
854
855
856
857

858
859
860
861
862
863

Under review as a conference paper at ICLR 2025

864
865

866
867
868
869
870
871

872
873
874
875
876
877

878
879
880
881
882

883
884
885
886
887
888

889
890
891
892
893
894

895
896
897
898
899
900

901
902
903
904
905

906
907
908
909
910
911

912
913
914
915
916
917

• MIMIC-CXR (Johnson et al., 2019) is a large publicly available dataset of chest X-ray images in

DICOM format with associated radiology reports.

• IU-Xray (Demner-Fushman et al., 2016) is a dataset that includes chest X-ray images and corre-

sponding diagnostic reports.

• Harvard-FairVLMed (Luo et al., 2024) focuses on fairness in multimodal fundus images, con-
taining image and text data from various sources. It aims to evaluate bias in AI models on this
multimodal data comprising different demographics.

• PMC-OA (Lin et al., 2023a) is a large-scale dataset comprising figure-caption pairs extracted from
PubMed Central. It covers 2,478,267 papers and includes a total of 12,211,907 figure-caption
pairs. We only use the pathology subset filtered by GPT-4 based on the captions.

• Quilt-1M (Ikezogwo et al., 2024) is the largest vision-language dataset in histopathology, contain-
ing 1 million image-text pairs sourced from platforms such as YouTube, Twitter, research papers,
and other parts of the internet.

A.3 EVALUATED MODELS

i.e., LLaVA-Med (Li et al., 2023a), Med-
We evaluate five open-source Med-LVLMs,
Flamingo (Moor et al., 2023), MedVInT (Zhang et al., 2023b), RadFM (Wu et al., 2023b), miniGPT-
Med (Alkhaldi et al., 2024). The selected models are all at the 7B level.

• LLaVA-Med (Li et al., 2023a) is a vision-language conversational assistant, adapting the general-
domain LLaVA (Liu et al., 2024b) model for the biomedical field. The model is fine-tuned using
a novel curriculum learning method, which includes two stages: aligning biomedical vocabulary
with figure-caption pairs and mastering open-ended conversational semantics.
It demonstrates
excellent multimodal conversational capabilities.

• Med-Flamingo (Moor et al., 2023) is a multimodal few-shot learner designed for the medical
domain. It builds upon the OpenFlamingo, continuing pre-training with medical image-text data
from publications and textbooks. This model aims to facilitate few-shot generative medical visual
question answering, enhancing clinical applications by generating relevant responses and ratio-
nales from minimal data inputs.

• RadFM (Wu et al., 2023b) serve as a versatile generalist model in radiology, distinguished by
its capability to adeptly process both 2D and 3D medical scans for a wide array of clinical tasks.
It integrates ViT as visual encoder and a perceiver module, alongside the MedLLaMA language
model, to generate sophisticated medical insights for a variety of tasks. This design allows RadFM
to not just recognize images but also to understand and generate human-like explanations.

• MedVInT (Zhang et al., 2023b), which stands for Medical Visual Instruction Tuning, is designed
to interpret medical images by answering clinically relevant questions. This model features two
variants to align visual and language understanding: MedVInT-TE and MedVInT-TD. Both Med-
VInT variants connect a pre-trained vision encoder ResNet-50 adopted from PMC-CLIP (Lin
et al., 2023a), which processes visual information from images.
It is an advanced model that
leverages a novel approach to align visual and language understanding.

• miniGPT-Med (Alkhaldi et al., 2024) is a vision-language model derived from large-scale lan-
guage models and tailored for radiology diagnosis applications. It handles various medical vision-
language task using distinct task identifiers, demonstrating advanced performance in disease
grounding, medical report generation, and medical VQA.

A.4 OVERVIEW OF THE BASELINES

We compare MMed-RAG with two types of LVLM hallucination mitigation methods that show
promising results in natural image understanding. 1) Decoding-based methods, including Greedy
Decoding, Beam Search (Sutskever et al., 2014), DoLa (Chuang et al., 2023), OPERA (Huang et al.,
2023), VCD (Leng et al., 2023). These methods manipulate the logits of the model’s output tokens to
enhance factual accuracy. 2) Multimodal RAG-based methods, including MedDr (He et al., 2024),
FactMM-RAG (Sun et al., 2024b), RULE (Xia et al., 2024b).

17

Under review as a conference paper at ICLR 2025

918
919

920
921
922
923
924
925

926
927
928
929
930
931

932
933
934
935
936

937
938
939
940
941
942

943
944
945
946
947
948

949
950
951
952
953
954

955
956
957
958
959

960
961
962
963
964
965

966
967
968
969
970
971

Instruction [Round1]
You are a professional medical expert. I will provide you with some medical reports. Please gen-
erate some questions with answers (the answer should be yes or no) based on the provided report.
The subject of the questions should be the medical image or patient, not the report.
Below are the given report:
[REPORT]
Instruction [Round2]
Please double-check the questions and answers, including how the questions are asked and whether
the answers are correct. You should only generate the questions with answers and no other unnec-
essary information.
Below are the given report and QA pairs in round1:
[REPORT]
[QA PAIRS R1]

Table 9: The instruction to GPT-4 for generating QA pairs.

• Greedy decoding involves selecting the most probable next token at each step of generation.
While it is efficient and straightforward, it can lead to suboptimal outcomes by getting stuck in
repetitive or less creative patterns.

• Beam search (Sutskever et al., 2014) expands on greedy decoding by maintaining multiple candi-
date sequences (or ”beams”) at each step, allowing for a broader exploration of possible outputs.
This approach balances quality and diversity by selecting the top-k sequences based on their prob-
abilities, resulting in more coherent and creative text generation compared to greedy decoding.

• DoLa (Chuang et al., 2023) derives the next-token distribution by contrasting the logits projected
from later layers against those from earlier layers, leveraging the fact that factual knowledge in
LLMs is typically localized within specific transformer layers.

• OPERA (Huang et al., 2023) is a LVLMs decoding method based on an Over-trust Penalty and a
Retrospection-Allocation strategy The key insight is that hallucinations are closely tied to knowl-
edge aggregation patterns in the self-attention matrix, where MLLMs tend to focus on summary
tokens, neglecting image tokens and resulting in content hallucination.

• VCD (Leng et al., 2023) is a decoding method that tackles the object hallucination issue in
LVLMs. It contrasts output distributions derived from original and distorted visual inputs to cal-
ibrate the model’s output without the usage of external tools, reducing the the over-reliance on
statistical bias and unimodal priors.

• MedDr (He et al., 2024) is a healthcare foundation model built upon generated diagnosis-based
datasets, demonstrating advanced capabilities in various data modalities. Meddr also integrates a
retrieval-augmented medical diagnosis strategy during inferencing to enhance factual accuracy.

• FactMM-RAG (Sun et al., 2024b) is a fact-aware multimodal retrieval-augmented pipeline for
radiology report generation. It utilize RadGraph to annotate chest radiograph reports and mine
clinically relevant pairs to train a universal multimodal retriever.

• RULE (Xia et al., 2024b) is an advanced medical retrieval-augmented generation strategy de-
signed to enhance the factuality of Med-LVLMs. First, it introduces a robust strategy for control-
ling factuality risk through the calibrated selection of retrieved contexts. Second, RULE develops
a preference optimization strategy to balance Med-LVLMs’ intrinsic knowledge and the retrieved
information.

A.5 PROMPTS

We convert the medical reports into a series of closed-ended questions with yes or no answers. To
ensure the quality of the VQA data, we perform a round of self-checks using GPT-4 (OpenAI, 2023).
Finally, we conduct an round of manual filtering to remove questions with obvious issues or those
related to multiple images or patient histories. The prompt templates used are shown in Table 9.

18

Under review as a conference paper at ICLR 2025

A.6 ADDITIONAL RESULTS

A.6.1 COMPATIBILITY ANALYSIS

To demonstrate the compatibility of our approach across different backbone models, we apply it to
LLaVA-Med-1.0. As shown in Table 10, our method delivers an average improvement of 40.3% over
the original LLaVA-Med-1.0, further highlighting its effectiveness in enhancing RAG performance
and its adaptability to various backbones. MMed-RAG can be transferred to different Med-LVLMs,
yielding consistent improvements across various domains, demonstrating the compatibility of our
method.

Table 10: Performance on different backbones.

Model

LLaVA-Med-1.0
+MMed-RAG

IU-Xray

VQA

61.73
80.32

RG

8.74
22.63

FairVLMed
RG
VQA

59.54
78.49

10.59
15.88

A.6.2 DETAILED RESULTS OF OTHER LVLMS

As shown in Table 11, we conduct a comparison of several general LVLMs and other Med-LVLMs,
including GPT-4o (OpenAI, 2023), Gemini-1.5 (Reid et al., 2024), QwenVL (Bai et al., 2023),
LLaVA-1.6 (Liu et al., 2024b), and InternVL-2 (Chen et al., 2024d). Our findings show that MMed-
RAG consistently outperforms these models, further demonstrating its effectiveness.

Table 11: Accuracy (%) of different Med-LVLMs based on LLaVA-Med-1.5 on medical VQA task.

Models

LLaVA-Med-1.5
MMed-RAG

Med-Flamingo
MedVInT
RadFM
miniGPT-Med

GPT-4o
Gemini-1.5
LLaVA-v1.6
Qwen-VL-Chat
InternVL-2

Radiology

Ophthalmology

Pathology

IU-Xray MIMIC-CXR Harvard-FairVLMed Quilt-1M PMC-OA (Pathology)

75.47
89.54

26.74
73.34
26.67
54.87

63.25
59.73
58.05
59.43
54.06

75.79
83.57

61.27
66.06
69.30
53.92

60.61
61.02
63.70
60.43
59.47

63.03
87.94

42.06
35.92
52.47
66.73

61.50
58.53
48.52
38.06
44.38

62.80
72.95

27.11
26.81
27.02
26.82

53.56
56.88
35.73
28.74
37.82

59.28
64.54

32.62
27.77
25.12
27.03

49.70
52.17
38.54
29.53
34.40

A.6.3 COMPARISON WITH DOMAIN-SPECIFIC MED-LVLMS AND THEM WITH RAG-PT

We conduct experiments to compare our method with domain-specific Med-LVLMs as follows:
Radiology: RadFM (Wu et al., 2023b), Pathology: Quilt-LLaVA (Seyfioglu et al., 2024), Ophthal-
mology: RetinaVLM (Holland et al., 2024). For radiology, we use the IU-Xray dataset to evaluate
VQA. For pathology, we use the PMC-OA pathology subset to evaluate VQA. For ophthalmology,
since the domain-specific Med-LVLM, i.e., RetinaVLM, is only trained on report-generation tasks,
we use the Harvard-FairVLMed dataset to evaluate report generation. As shown in Table 12, our
method significantly outperforms each domain-specific Med-LVLM. Additionally, we apply RAG-
PT to each domain-specific Med-LVLM. As shown in Table 12, after incorporating RAG-PT, the
performance of these models improve significantly, demonstrating the compatibility of our method.
Furthermore, domain-specific Med-LVLMs could outperform generalist Med-LVLMs in their spe-
cialized domains, as they are fine-tuned using specialized medical domain data. While this signifi-
cantly enhances their medical understanding in specific domains, it may reduce their generalization
ability, such as their capacity to comprehend retrieved information. Consequently, even after incor-
porating RAG-PT, the performance of several domain-specific Med-LVLMs (e.g., RetinaVLM and
RadFM) is inferior to MMed-RAG.

19

972
973

974
975
976
977
978
979

980
981
982
983
984
985

986
987
988
989
990

991
992
993
994
995
996

997
998
999
1000
1001
1002

1003
1004
1005
1006
1007
1008

1009
1010
1011
1012
1013

1014
1015
1016
1017
1018
1019

1020
1021
1022
1023
1024
1025

Under review as a conference paper at ICLR 2025

Table 12: Model performance comparison with domain-specific Med-LVLMs.

Radiology

Pathology

Ophthalmology

Acc

F1

AUC

BLEU ROUGE-L METEOR

Model

RadFM
+ RAG-PT

Quilt-LLaVA
+ RAG-PT

RetinaVLM
+ RAG-PT

Acc

26.67
48.39

F1

30.36
39.40

AUC

55.31
59.70

-
-

-
-

-
-

-
-

-
-

-
-

-
-

-
-

-
-

62.59
64.72

72.30
73.36

56.96
61.39

-
-

-
-

-
-

LLaVA-Med-1.5
MMed-RAG

75.47
84.10

64.04
71.92

67.46
86.40

59.28
64.54

71.98
73.09

54.19
61.42

-
-

-
-

19.96
22.26

18.11
24.82

-
-

-
-

12.73
14.64

11.36
16.59

-
-

-
-

13.52
16.87

10.75
19.85

A.6.4 RESULTS ON OTHER DOMAIN

We apply RAG-PT to one additional domain (i.e., environmental ecosystems modeling) to further
validate the effectiveness of RAG-PT. We conduct experiments on two environmental system model-
ing datasets (Li et al., 2024a). The CRW-Temp dataset is a river water temperature prediction dataset
aimed at forecasting the daily average water temperature of a specific day based on observed phys-
ical variables. The CRW-Flow dataset focuses on predicting river segment flow based on observed
physical variables. The model used is LITE (Li et al., 2024a), an environmental system large model
based on LLaMA2 (Touvron et al., 2023). We train a semantic time-series encoder using time-series
information-text pairs, which works in conjunction with a text encoder as the retriever. Then we re-
trieve the most similar environmental descriptions based on the current environmental descriptions.
As shown in Table 13, our approach demonstrates significant performance improvements on tasks
in this domain.

Table 13: Performance comparison of different models on CRW-Temp and CRW-Flow datasets.

Model

CRW-Temp

CRW-Flow

RMSE MAE RMSE MAE

LITE (Li et al., 2024a)
+RAG
+RAG-PT

2.02
1.93
1.74

1.70
1.62
1.46

2.39
2.27
2.11

1.02
0.96
0.90

A.6.5 STATISTICS OF COPY-REFERENCE RATE AND OVER-RELIANCE RATE FOR MORE

LVLMS.

Following the alignment analysis method we apply to LLaVA-Med-1.5 in Section 3.3, we conduct
two alignment analysis tests on multiple open-source Med-LVLMs and commercial LVLMs using
the Harvard-FairVLMed dataset with the incorporation of retrieved information. These tests respec-
tively evaluate (1) cross-modality alignment and (2) overall alignment with the ground truth. As
shown in Table 14, the results indicate that both existing open-source Med-LVLMs and commer-
cial LVLMs exhibit misalignment issues with retrieved information. In addition, it is worthwhile
to mention that GPT-4o demonstrates the best alignment performance compared with other models
when incorporating RAG, especially in cross-modal alignment. This is likely because GPT-4o has
been well-trained in visual perception and may also have utilized some post-training methods (like
preference optimization) to optimize modal alignment.

A.6.6 DETAILED ABLATION ANALYSIS

Preference data designed for different alignment objectives can indeed produce varying effects.
Therefore, conducting ablation experiments on combinations of different types of preference data
is necessary. We perform comprehensive ablation experiments on RAG-PT 1/2/3 as well as their
combinations (RAG-PT 1+2, 2+3, 1+3) to analyze the effectiveness of each type of data and their
combinations. We find that the combination of 1+3 produced the most significant results, indicat-
ing that the two misalignment issues (i.e., cross-modality and over-reliance issues) are the most

20

1026
1027

1028
1029
1030
1031
1032
1033

1034
1035
1036
1037
1038
1039

1040
1041
1042
1043
1044

1045
1046
1047
1048
1049
1050

1051
1052
1053
1054
1055
1056

1057
1058
1059
1060
1061
1062

1063
1064
1065
1066
1067

1068
1069
1070
1071
1072
1073

1074
1075
1076
1077
1078
1079

Under review as a conference paper at ICLR 2025

Table 14: Comparison of Copy-Reference Rate and Over-Reliance Rate across different models.

Model

Copy-Reference Rate Over-Reliance Rate

LLaVA-Med-1.5
Med-Flamingo
miniGPT-Med
GPT-4o

55.08
60.17
56.75
12.54

43.31
33.74
46.06
24.80

prominent. Targeted mitigation of these two issues yielded the greatest improvement. However,
incorporating data for all three alignment objectives yields the best performance, demonstrating the
importance of each alignment component.

Table 15: Ablation results using RAG-PT based on subsets of preference data.

Harvard-FairVLMed
VQA

RG

Model

IU-Xray

LLaVA-Med-1.5
+RAG-PT 1
+RAG-PT 2
+RAG-PT 3

+RAG-PT 1+2
+RAG-PT 1+3
+RAG-PT 2+3

VQA

68.99
80.19
80.27
81.30

82.58
82.96
83.61

RG

10.04
19.38
20.16
19.43

22.74
24.50
25.77

66.63
79.42
79.35
80.07

82.08
82.87
83.89

+RAG-PT 1+2+3

85.58

29.69

87.02

13.41
18.37
18.66
18.92

18.97
19.22
19.30

20.31

A.6.7 EXTERNAL VALIDATION

Considering the risk of overfitting, we use external validation datasets from the same domain to
evaluate the generalizability of MMed-RAG. We select two domain-specific subsets from PubMed-
Vision (Chen et al., 2024b), i.e., fundus digital photography and microscopy image, for ophthalmol-
ogy and pathology, respectively. The results show that MMed-RAG still significantly outperforms
other Med-LVLMs on the external validation datasets, indicating MMed-RAG performs well when
generalized to external datasets, demonstrating its strong generalization capability.

Table 16: Performance comparison of models on external validation datasets.

Model

Ophthalmology

Pathology

BLEU ROUGE-L METEOR

Acc

F1

LLAVA-Med-1.5
MMed-RAG

17.11
22.64

20.05
14.98

17.09
17.85

59.65
62.88

71.90
72.24

AUC

54.87
59.69

A.6.8 DETAILED BLEU SCORE

We report the average BLEU score above. Detailed results are provided in Table 17.

A.6.9 DEEPER ANALYSIS OF RETRIEVER

We have tried training a general retriever by mixing images from all modalities together, instead of
using a domain-specific retriever. We conduct experiments based on BiomedCLIP and MedCLIP,
but the results are unsatisfactory. Then we adopt an MoE (Mixture of Experts) architecture (Shazeer
et al., 2017; Riquelme et al., 2021). Based on CLIP-MoE, we fine-tune CLIP-MoE (Zhang et al.,
2024) with mixing images from all medical imaging modalities, but the performance is still subop-
timal. This might be because CLIP-MoE is not pretrained on large-scale biomedical data. All the
results are reported in Table 18. Considering model performance, we ultimately adopt a domain-
specific retriever architecture.
In fact, this approach is both flexible and scalable. Similar to a
general retriever, encountering a completely new modality may still require retraining the retriever

21

1080
1081

1082
1083
1084
1085
1086
1087

1088
1089
1090
1091
1092
1093

1094
1095
1096
1097
1098

1099
1100
1101
1102
1103
1104

1105
1106
1107
1108
1109
1110

1111
1112
1113
1114
1115
1116

1117
1118
1119
1120
1121

1122
1123
1124
1125
1126
1127

1128
1129
1130
1131
1132
1133

Under review as a conference paper at ICLR 2025

Table 17: BLEU Score (%) of different methods based on LLaVA-Med-1.5 on report generation
task.

Models

Radiology

IU-Xray

MIMIC-CXR

Ophthalmology

Harvard-FairVLMed

BLEU-1 BLEU-2 BLEU-3 BLEU-4 BLEU-1

BLEU-2 BLEU-3 BLEU-4

BLEU-1 BLEU-2

BLEU-3 BLEU-4

LLaVA-Med-1.5

+ Greedy
+ Beam Search
+ DoLa
+ OPERA
+ VCD

+ MedDr
+ FactMM-RAG
+ RULE

MMed-RAG

17.69

21.04
21.78
21.22
19.79
19.35

22.27
26.45
49.56

56.48

10.55

12.57
12.71
12.39
11.19
10.94

12.99
15.25
28.61

32.67

6.47

5.75
6.05
5.90
5.33
5.21

6.19
7.26
13.62

15.56

3.83

3.35
3.63
3.54
3.20
3.13

3.71
4.36
8.17

9.34

21.82

29.91
30.55
30.80
27.72
27.27

33.43
33.64
33.47

41.81

13.35

18.26
17.79
17.97
16.05
15.76

19.33
19.44
19.36

24.18

6.11

8.27
8.49
8.58
7.65
7.51

9.22
9.27
9.23

11.52

3.64

5.03
5.09
5.15
4.59
4.51

5.53
5.56
5.54

6.92

32.57

32.40
33.07
32.87
29.90
30.14

35.64
37.47
40.21

44.65

19.86

19.82
19.14
19.02
17.45
17.61

20.61
21.64
23.26

25.79

9.11

9.04
9.14
9.08
8.32
8.39

9.82
10.30
11.08

12.29

5.38

5.37
5.48
5.45
4.99
5.04

5.89
6.18
6.66

7.38

to achieve good retrieval performance, which incurs additional costs. For mixed datasets, as the
number of modalities increases, training a general retriever becomes increasingly challenging, mak-
ing it difficult to achieve reliable retrieval within a single domain. We address this by using a domain
identification module to classify the input image by modality and select the corresponding retriever.
In the future, a potential solution could involve pretraining a general retriever on large-scale biomed-
ical data using a Mixture of Experts (MoE) architecture to explore whether it is possible to develop
a general retriever.

Table 18: Performance comparison based on different retrievers.

Model

LLaVA-Med
+ RAG (BiomedCLIP-FT)
+ RAG (MedCLIP-FT)
+ RAG (CLIP-MoE-FT)
+ RAG (Ours)

IU-Xray

Acc

75.47
79.09
75.13
72.13
84.82

F1

64.04
65.87
63.88
62.72
68.85

AUC

67.46
69.52
67.16
65.11
77.54

A.6.10 COMPARISON UNDER FEW-SHOT SETTING

All our experiments are conducted under a zero-shot setting. We conduct experiments on LLaVA-
Med-1.5 using the same few-shot strategy as in Med-Flamingo. The results show that compared
to the zero-shot setting, the model’s performance significantly decreases, even with RAG applied.
Our analysis of this phenomenon reveals that, unlike Med-Flamingo, LLaVA-Med does not use
interleaved multimodal data for pretraining. As a result, it lacks the capability for few-shot learn-
ing. This point has been mentioned in some discussion forums and GitHub issues. In addition,
LLaVA-1.5’s unsatisfactory performance on multi-image understanding benchmarks also supports
this observation (Wang et al., 2024; Meng et al., 2024).

Table 19: Performance comparison under zero-shot and few-shot settings.

Model

LLaVA-Med (zero-shot)
+MMed-RAG
LLAVA-Med (few-shot)
+MMed-RAG

IU-Xray

Acc

75.47
89.54
66.77
84.10

F1

64.04
80.72
51.56
71.92

AUC

67.46
87.13
66.60
86.40

A.6.11 PERFORMANCE COMPARISON OF THE RETRIEVER

Regarding the retriever’s performance, as shown in Table 20, we compared the performance of our
retriever with several CLIP-based models on radiology datasets for image-to-text retrieval. The

22

1134
1135

1136
1137
1138
1139
1140
1141

1142
1143
1144
1145
1146
1147

1148
1149
1150
1151
1152

1153
1154
1155
1156
1157
1158

1159
1160
1161
1162
1163
1164

1165
1166
1167
1168
1169
1170

1171
1172
1173
1174
1175

1176
1177
1178
1179
1180
1181

1182
1183
1184
1185
1186
1187

Under review as a conference paper at ICLR 2025

results demonstrate that our retriever significantly outperforms the other models in retrieval perfor-
mance.

Table 20: Performance comparison of different retrievers on Recall@1 (R@1) and Recall@5 (R@5)
metrics.

Model

R@1

R@5

CLIP
PubMedCLIP
MedCLIP
BiomedCLIP
PMC-CLIP
Ours

3.91
1.47
6.74
15.7
12.3
45.6

7.88
1.64
12.69
23.8
21.2
71.8

A.6.12 RATIONALE-GUIDED RAG

For retrieved information, we minimize noise by optimizing the number of retrieved contexts k
(e.g., Adaptive Retrieved Context Selection in Section 3.2). Following this, we introduce RAG-PT
to specifically address the misalignment issues that arise after incorporating RAG, thereby strength-
ening Med-LVLM to balance its internal knowledge and external retrieval information. We employ
a rationale-guided approach (Wei et al., 2024) that uses LLM to explicitly learn denoising of re-
trieved content through self-synthesized rationales. First, given a question, the retrieved documents,
and the ground truth from the training set, we prompt a powerful Med-LLM (i.e., LLaMA3-Med42-
70B (Christophe et al., 2024)) to generate a rationale. This rationale explains how to derive the
answer from potentially noisy inputs. Next, we use the synthesized rationale from the previous step
to guide another smaller Med-LLM (i.e., LLaMA3-Med42-7B (Christophe et al., 2024)) to explic-
itly learn denoising of the retrieved documents through in-context learning and supervised learning.
By employing this rationale-guided Med-LLM to filter noisy retrieval information, the reliability of
our retrieved data improves. Experimental results show that after rationale-guided RAG, the model’s
performance further improved.

Table 21: Performance comparison on IU-Xray dataset, including RAG and Rationale-Guided RAG
variants.

Model

IU-Xray

Acc

F1

LLaVA-Med
+ RAG

75.47
84.82
89.54
+ Rationale-Guided RAG 85.38
89.91

+ RAG-PT

+ RAG-PT

64.04
68.85
80.72
69.23
80.86

AUC

67.46
77.54
87.13
77.90
87.32

A.7 THE CONTRIBUTION OF DOMAIN-SPECIFIC RETRIEVERS

We design a domain-specific retriever leveraging a generalist Med-LVLM to retrieve information
from a dedicated database based on the identified modality of the input medical image. Here, the
domain identification models used are capable of reliably recognizing modalities with high accuracy
( 99.83% accuracy in our experiments). For radiology VQA tasks, input radiology images are classi-
fied as “radiology” by the model, enabling the retrieval of knowledge exclusively from the radiology
database to enhance generation. All retrieved documents are specific to radiology and exclude other
modalities.

A.8 EXPLANATION OF CROSS-MODALITY ALIGNMENT

To construct preference pairs for cross-modality alignment, we first select a preferred response by
having the model generate an answer using the correct medical image, clinical query, and retrieved

23

1188
1189

1190
1191
1192
1193
1194
1195

1196
1197
1198
1199
1200
1201

1202
1203
1204
1205
1206

1207
1208
1209
1210
1211
1212

1213
1214
1215
1216
1217
1218

1219
1220
1221
1222
1223
1224

1225
1226
1227
1228
1229

1230
1231
1232
1233
1234
1235

1236
1237
1238
1239
1240
1241

Under review as a conference paper at ICLR 2025

Figure 5: Illustration Examples for noisy images in cross-modality alignment.

knowledge, ensuring the response matches the ground-truth answer. Then, we select a dispreferred
response by introducing an unrelated input image. This unrelated image is selected by finding
the one with the lowest similarity to the target image and adding noise to distort it further. The
dispreferred response is generated when the model uses this noisy, unrelated image along with the
query and retrieved knowledge to still produce the correct answer. By comparing these pairs during
training, the model learns to prioritize relevant and accurate inputs (e.g., the correct medical image)
over noisy or irrelevant ones, improving cross-modality alignment.

A.9 ANALYSIS OF NOISY IMAGE IN CROSS-MODALITY ALIGNMENT

In medical imaging, noise refers to random variations in image signals caused by hardware limita-
tions or environmental factors (Gravel et al., 2004; Sanchez et al., 2012). However, the noise we
refer to here pertains to images unrelated to the original image, generated through a two-step pro-
cess: 1. We use a retriever to select images with the lowest similarity to the target image. 2. We
introduce strong diffusion noise to these images. As a result, the noisy images in our case are almost
entirely random noise and are not merely examples of domain shifts, such as changes in lighting
conditions. Refer to the third section of Figure 1 for examples, and additional examples are included
in the Figure 5 for reference.

The motivation behind our design is that replacing the original image with a highly noisy image
while adding retrieved information corresponding to the original image reveals a significant issue
of cross-modal misalignment in the Med-LVLM—namely, it ignores the image information and
directly copies the retrieved contexts. To mitigate this issue, we construct such preference pairs to
specifically strengthen the model’s cross-modal alignment capability.

A.10 EXPLANATION OF OVER RELIANCE RATE

The overall alignment issue arises from the conflict between retrieved information and the model’s
internal knowledge. For retrieved information, we cannot guarantee 100% accuracy, so some noise
is inevitable. The Over-Reliance (OR) rate shown in Figure 3 refers to the proportion of initially
correct responses that become incorrect after adding the retrieved context, calculated relative to
the total number of incorrect samples, not the total number of all samples. This rate represents the
proportion of errors caused by over-reliance, rather than indicating poor performance of the retriever.
Through RAG-PT, we can effectively mitigate this issue, significantly reducing the OR rate.

1242
1243

1244
1245
1246
1247
1248
1249

1250
1251
1252
1253
1254
1255

1256
1257
1258
1259
1260

1261
1262
1263
1264
1265
1266

1267
1268
1269
1270
1271
1272

1273
1274
1275
1276
1277
1278

1279
1280
1281
1282
1283

1284
1285
1286
1287
1288
1289

1290
1291
1292
1293
1294
1295

24

Under review as a conference paper at ICLR 2025

B PROOFS FOR THEORETICAL RESULTS IN SECTION 4

Here we provide proofs for the results in Section 4.

B.1 NOTATIONS

Let xv, y, xt, xr be input medical image, ground-truth answer, question, and retrieved information,
respectively. Denote (xw, yw,o) ∼ qp(xw, yw,o|xt, xr) and (xl, yl,o) ∼ ql(xl, yl,o|xt, xr) as distri-
butions of the preferred responses and dispreferred responses. Let x denote (xv, xr, xt). We aim to
a fine-tune a generative model πθ(y|x, xt) through DPO loss (Rafailov et al., 2023):
(cid:19)

(cid:18)

arg min
πθ

E(xw,xl,yw,o,yl,o)∼DU

α log

πθ(yw,o|x)
πo(yw,o|x)

− α log

πθ(yl,o|x)
πo(yl,o|x)

.

(14)

where U (t) = log(1 + exp(−t)). Define the weight of xv with respect to log πθ(y|x) as

wt(xv, πθ) := Ey∼πθ(·|x)

(cid:20) ∂
∂xv

(cid:21)2

log πθ(y|x)

(15)

B.2 ASSUMPTIONS

Assumption B.1 (Large parameter space) Assume that π(xv, y|xt, xr) lies in the optimization

space {πθ, θ ∈ Θ} such that π(xv, y|xt, xr) ∝ πo(xv, y|xt, xr)

(cid:16) qw(xv,y|xt,xr)
ql(xv,y|xt,xr)

(cid:17) 1

α

Assumption B.1 requires that the parameter space sufficiently large to ensure that πθ can achieve its
global optimum, allowing us to represent the optimizer with a closed form.

Assumption B.2 Let h(x, y), abbreviate as h, be

h :=

(cid:34)

(cid:88)

y

πo(y|x)

(cid:19) 1
α

(cid:18) qw(y|x)
ql(y|x)

(cid:35)−1 (cid:18) qw(y|x)
ql(y|x)

(cid:19) 1
α

Assume that wt(xv, πo) < c2, where

(cid:115)(cid:13)
(cid:13)
(cid:13)
(cid:13)

c =

(cid:112)πo(y|x) ·

∂
∂xv

h

(cid:13)
(cid:13)
(cid:13)
(cid:13)

2

2

+

(cid:90) (cid:18) ∂
∂xv

h

(cid:19)2 πo(y|x)

h

dy −

(cid:13)
(cid:13)
(cid:13)
(cid:13)

(cid:112)πo(y|x) ·

∂
∂xv

(cid:13)
(cid:13)
h
(cid:13)
(cid:13)2

Assumption B.3 Let h1(xv, xt, xr, y), abbreviate as h1, be

h1 :=

(cid:34)

(cid:88)

y

πo(y|x)

(cid:18) q1
w(y|xv, xt, xr) + q2
l (y|xv, xt) + q2
q1

w(y|xv, xt)
l (y|xv, xt, xr)

(cid:19) 1

α (cid:35)−1 (cid:18) q1
w(y|xv, xt, xr) + q2
l (y|xv, xt) + q2
q1

w(y|xv, xt)
l (y|xv, xt, xr)

Assume that wt(xr, πo) < c2

1 and wt(˜xr, πo) > c2

2, where

c1 =

c2 =

(cid:115)(cid:13)
(cid:13)
(cid:13)
(cid:13)
(cid:115)(cid:13)
(cid:13)
(cid:13)
(cid:13)

√

πo ·

√

πo ·

∂h1
∂xr

∂h1
∂ ˜xr

(cid:13)
(cid:13)
(cid:13)
(cid:13)

(cid:13)
(cid:13)
(cid:13)
(cid:13)

2

2

2

2

+

+

(cid:90) (cid:18) ∂h1
∂xr

(cid:19)2 πo
h1

dy −

√

πo ·

(cid:13)
(cid:13)
(cid:13)
(cid:13)

∂h1
∂xr

(cid:13)
(cid:13)
(cid:13)
(cid:13)2

(cid:90) (cid:18) ∂h1
∂ ˜xr

(cid:19)2 πo
h1

+

(cid:18) ∂πo
∂ ˜xr

(cid:19)2 h1
πo

dy +

√

πo ·

(cid:13)
(cid:13)
(cid:13)
(cid:13)

∂h1
∂ ˜xr

(cid:13)
(cid:13)
(cid:13)
(cid:13)2

B.3 PROOFS

Lemma B.1 Suppose that Assumption B.1 hold, optimizing equation 14 gives

πθ(y|x) ∝ πo(y|x)

(cid:19) 1

α

(cid:18) qw(y|x)
ql(y|x)

25

(16)

(17)

(cid:19) 1

α

(18)

(19)

(20)

1296

1297
1298
1299
1300
1301

1302
1303
1304
1305
1306
1307

1308
1309
1310
1311
1312
1313

1314
1315
1316
1317
1318
1319

1320
1321
1322
1323
1324

1325
1326
1327
1328
1329
1330

1331
1332
1333
1334
1335
1336

1337
1338
1339
1340
1341
1342

1343
1344
1345
1346
1347

1348
1349

Under review as a conference paper at ICLR 2025

Lemma B.1 indicates that the model tends to increase πo(y|x) if qw(y|x) > ql(y|x), which is more
likely to occur when (xv, y) represents a preferred sample given xt and xr. Below, we provide an
application of Lemma B.1 using a linear regression example. Lemma B.1 is proved with Lemma
B.2 and Lemma B.3.

Lemma B.2 (Lemma C.1 in Chen et al. (2024e)) For a, b > 0, the following inequality holds

a · U (t) + b · U (−t) ≥ a log(1 + b/a) + b log(1 + a/b)

and equality holds if and only if t = log(a/b)

Lemma B.3 Denote

(cid:26) p1(xw, yw,o, xl, yl,o|xt, xr) = qw(xw, yw,o|xt, xr) · ql(xl, yl,o|xt, xr)
p2(xw, yw,o, xl, yl,o|xt, xr) = ql(xw, yw,o|xt, xr) · qw(xl, yl,o|xt, xr)

and abbreviated as p1 and p2 for notational convenience. Then,

2ED [U (f (xw, yw,o, xt, xr) − f (xl, yl,o, xt, xr))]
p1 + p2
2

≥2 log 2 − DKL

− DKL

p1 + p2
2

p2

p1

(cid:13)
(cid:13)

(cid:13)
(cid:13)

(cid:18)

(cid:18)

(cid:19)

(cid:19)

Equality holds if and only if

f (x, y) = g(x) + log

qw(xv, y|xt, xr)
ql(xv, y|xt, xr)

where g(x) is any function that is possibly dependent on xv, xt and xr.

Proof B.1

2ED [U (f (xw, yw,o, xt, xr) − f (xl, yl,o, xt, xr))]
(cid:90)

q(xt, xr) · p1 · U (f (xw, yw,o, xt, xr) − f (xl, yl,o, xt, xr)) dxdy

+

(cid:90)

(cid:90)

q(xt, xr) · p2 · U (f (xl, yl,o, xt, xr) − f (xw, yw,o, xt, xr)) dxdy

q(xt, xr)

(cid:20)
p1 · log

(cid:18)

1 +

(cid:18)

+ p2 · log

1 +

(cid:19)

p2
p1

(cid:19)(cid:21)

p1
p2

dxdy

=

≥

(21)

(22)

(23)

(cid:90)

=2 log 2 +

(cid:20)

q(xt, xr)

p1 · log

=2 log 2 − KL

(cid:18)

(cid:13)
(cid:13)

p1

p1 + p2
2

(cid:19)

(cid:19)

(cid:18) p1 + p2
2p1
(cid:18)

− KL

p2

+ p2 · log

(cid:19)

(cid:13)
(cid:13)

p1 + p2
2

(cid:19)(cid:21)

(cid:18) p1 + p2
2p2

dxdy

where the first inequality follows from Lemma B.2. For equivalence,

f (x, yw,o, xt, xr) − f (xl, yl,o, xt, xr) = log

qw(xw, yw,o|xt, xr) · ql(xl, yl,o|xt, xr)
ql(xw, yw,o|xt, xr) · qw(xl, yl,o|xt, xr)

(24)

Thus, for any xw, yw,o, xl, yl,o, xt, xr,

f (xw, yw,o, xt, xr) − log

qw(xw, yw,o|xt, xr)
ql(xw, yw,o|xt, xr)

= f (xl, yl,o, xt, xr) − log

qw(xl, yl,o|xt, xr)
ql(xl, yl,o|xt, xr)

(25)

Therefore, equation 25 holds if and only if there exists some g(xv, xt, xr) such that

f (xv, xt, xr, y) = g(xt, xr) + log

qw(xv, y|xt, xr)
ql(xv, y|xt, xr)

(26)

Lemma B.3 provides a closed-form solution to equation 14 if the parameter space is sufficiently
large. This lemma is crucial for the proof Lemma B.1, which follows below

26

1350

1351
1352
1353
1354
1355

1356
1357
1358
1359
1360
1361

1362
1363
1364
1365
1366
1367

1368
1369
1370
1371
1372
1373

1374
1375
1376
1377
1378

1379
1380
1381
1382
1383
1384

1385
1386
1387
1388
1389
1390

1391
1392
1393
1394
1395
1396

1397
1398
1399
1400
1401

1402
1403

Under review as a conference paper at ICLR 2025

Proof B.2 According to the Assumption B.1, we have

π(xv, y|xt, xr) = ˆg(xt, xr)πo(xv, y|xt, xr)

(cid:18) qw(xv, y|xt, xr)
ql(xv, y|xt, xr)

(cid:19) 1

α

After reparameterization,

α log

(cid:18) π(xv, y|xt, xr)
πo(xv, y|xt, xr)

(cid:19)

= α log[ˆg(xt, xr)] + log

qw(xv, y|xt, xr)
ql(xv, y|xt, xr)

which is the global minimum of

arg min
f

ED [U (f (xw, yw,o, xt, xr) − f (xl, yl,o, xt, xr))]

by Lemma B.3. Since π(xv, y|xt, xr) ∈ {πθ, θ ∈ Θ} lies in the optimization space, we have
EDU (f (xw, yw,o, xt, xr) − f (xl, yl,o, xt, xr))

min
f

= min
πθ

EDU

(cid:18)

α log

πθ(yw,o|xw, xt, xr)
πo(yw,o|xw, xt, xr)

− α log

πθ(yl,o|xl, xt, xr)
πo(yl,o|xl, xt, xr)

(cid:19)

and πθ(xv, y|xt, xr) is the optimizer of equation 30, which gives

(27)

(28)

(29)

(30)

α log

(cid:18) πθ(xv, y|xt, xr)
πo(xv, y|xt, xr)

(cid:19)

= g(xt, xr) + log

qw(xv, y|xt, xr)
ql(xv, y|xt, xr)

=⇒πθ(xv, y|xt, xr) = πo(xv, y|xt, xr)

(cid:18) qw(xv, y|xt, xr)
ql(xv, y|xt, xr)

(cid:19) 1

α

exp

(cid:18) 1
α

(31)

(cid:19)

g(xt, xr)

Then

πθ(y|x) =

πθ(xv, y|xt, xr)
πθ(x|xt, xr)

=

πo(xv, y|xt, xr)

(cid:80)

y πo(xv, y|xt, xr)

(cid:17) 1

α

(cid:16) qw(xv,y|xt,xr)
ql(xv,y|xt,xr)
(cid:16) qw(xv,y|xt,xr)
ql(xv,y|xt,xr)

exp (cid:0) 1
(cid:17) 1

α

α (g(xt, xr)(cid:1)

exp (cid:0) 1

α (g(xt, xr)(cid:1)

πo(y|x)

=

(cid:80)

y πo(y|x)

(cid:17) 1

α

(cid:16) qw(xv,y|xt,xr)
ql(xv,y|xt,xr)
(cid:16) qw(xv,y|xt,xr)
ql(xv,y|xt,xr)

=

(cid:17) 1

α

πo(y|x)

(cid:80)

y πo(y|x)

(cid:17) 1

α

(cid:16) qw(y|xv,xt,xr)
ql(y|xv,xt,xr)
(cid:16) qw(y|xv,xt,xr)
ql(y|xv,xt,xr)

(cid:17) 1

α

(32)

Corollary B.1 Suppose that preferred responses (xw, yw) and dispreferred responses (xl, yl) sat-
isfy yw = βxw + ϵ1 and yl = ˜βxl + ϵ2 respectively. DPO for y = θxv + ϵ3 is based on reference
model y = θoxv + ϵ4, where ϵi’s are independent and follow standard normal distribution. Then,

θ = θo +

(β − ˜β)

1
α

(33)

Corollary B.1 is a direct application of Lemma B.1, indicating that the model updates coefficient θo
towards the direction of β for preferred responses and away from ˜β for dispreferred responses.

Proof B.3 Let ϕ(·) denote the probability density function of standard normal, by Lemma B.1,

ϕ(y − θx) ∝ ϕ(y − θox)

=⇒ exp

(cid:19)

y2 − θ1xy

(cid:18) 1
2

(cid:19) 1

α

(cid:18) ϕ(y − βx)
ϕ(y − ˜βx)
(cid:18) 1
2

∝ exp

y2 − θoxy

(cid:19)

(cid:18)

· exp

−

1
α

(cid:19)

(β − ˜β)xy

=⇒ exp (θ1xy) ∝ exp (θoxy) · exp

(cid:19)

(β − ˜β)xy

(cid:18) 1
α

=⇒θ = θo +

(β − ˜β)

1
α

27

(34)

1404

1405
1406
1407
1408
1409

1410
1411
1412
1413
1414
1415

1416
1417
1418
1419
1420
1421

1422
1423
1424
1425
1426
1427

1428
1429
1430
1431
1432

1433
1434
1435
1436
1437
1438

1439
1440
1441
1442
1443
1444

1445
1446
1447
1448
1449
1450

1451
1452
1453
1454
1455

1456
1457

Under review as a conference paper at ICLR 2025

Lemma B.4 For linear model y = θ1xv + θ2xt + ϵ such that ϵ ∼ N (0, 1), wt(xv, πθ) = θ2
1

Proof B.4 Let ϕ(·) denote the probability density function of standard normal,

(cid:90) (cid:18)

wt(xv, πθ) =

(cid:90)

(cid:90)

(cid:90)

= θ2
1

= θ2
1

= θ2
1

−

1
2

∂
∂xv

(y − θ1xv − θ2xt)2

(cid:19)2

ϕ(y − θ1xv − θ2xt)dy

(y − θ1xv − θ2xt)2 ϕ(y − θ1xv − θ2xt)dy

(θ1xv + θ2xt − y)

dϕ(y − θ1xv − θ2xt)
dy

dy

ϕ(y − θ1xv − θ2xt)dy = θ2
1

(35)

Theorem B.2 Suppose that Assumption B.2 holds, then cross-modality increase the weight of xv.

wt(xv, πθ) > wt(xv, πo)

Proof B.5 By Lemma B.1, we have

πθ(y|x) = πo(y|x) · h(x, y),

(cid:90)

πo(y|x) · h(x, y)dy = 1

Abbreviate h(x, y) and πo(y|xv, xt) as h and πo respectively, we have

wt(xv, πθ) − wt(xv, πo) ≥

(cid:90) (cid:32) ∂
∂xv
πo

πo

+

∂
∂xv
h

(cid:33)2

h

πoh dy − wt(xv, πo)

≥

(cid:90) (cid:20) ∂
∂xv

(cid:21)2 πo
h
h

dy − 2(cid:112)wt(xv, πo) ·

√

(cid:13)
(cid:13)
(cid:13)
(cid:13)

πo ·

∂
∂xv

(cid:13)
(cid:13)
h
(cid:13)
(cid:13)2

(36)

(37)

− wt(xv, πo)

the second inequality follows from Cauchy–Schwarz inequality

(cid:90)

∂
∂xv

πo ·

∂
∂xv

h dy =

(cid:90)

∂
∂xv

πo ·

√
√

πo
πo

·

∂
∂xv

h dy ≤ (cid:112)wt(xv, πo) ·

√

(cid:13)
(cid:13)
(cid:13)
(cid:13)

πo ·

∂
∂xv

(cid:13)
(cid:13)
h
(cid:13)
(cid:13)2

Denote c as

√

(cid:115)(cid:13)
(cid:13)
(cid:13)
(cid:13)

πo ·

c :=

∂
∂xv

h

(cid:13)
2
(cid:13)
(cid:13)
(cid:13)
2

+

(cid:90) (cid:18) ∂
∂xv

(cid:19)2 πo
h
h

dy −

√

(cid:13)
(cid:13)
(cid:13)
(cid:13)

πo ·

∂
∂xv

(cid:13)
(cid:13)
h
(cid:13)
(cid:13)2

the last term in equation 38 is equivalent to

(cid:16)

(cid:17)
c − (cid:112)wt(xv, πo)

·

(cid:18)

(cid:112)wt(xv, πo) + c + 2

√

(cid:13)
(cid:13)
(cid:13)
(cid:13)

πo ·

(cid:19)

∂
∂xv

(cid:13)
(cid:13)
h
(cid:13)
(cid:13)2

Thus, wt(xv, πθ) > wt(xv, πo) if (cid:112)wt(xv, πo) < c.

(38)

(39)

(40)

(41)

Theorem B.3 Suppose that Assumption B.3 holds, the overall loss increase the weight of xr and
decrease the weight of ˜xr.

wt(xr, πθ) > wt(xr, πo), wt(˜xr, πθ) < wt(˜xr, πo)

(42)

Proof B.6 The distribution of preferred responses can be considered as a mixture distribution:
w(xv, yw,o|xt, xr) + q2
q1
w(xv, yw,o|xt). Similarly, for dispreferred responses, the distribution is rep-
resented as q1

l (xv, yl,o|xt, xr). By Lemma B.1,

l (xv, yl,o|xt) + q2

πθ(y|x) = πo(y|x) · h1(x, y),

(cid:90)

πo(y|x) · h1(x, y)dy = 1

(43)

28

1458

1459
1460
1461
1462
1463

1464
1465
1466
1467
1468
1469

1470
1471
1472
1473
1474
1475

1476
1477
1478
1479
1480
1481

1482
1483
1484
1485
1486

1487
1488
1489
1490
1491
1492

1493
1494
1495
1496
1497
1498

1499
1500
1501
1502
1503
1504

1505
1506
1507
1508
1509

1510
1511

Under review as a conference paper at ICLR 2025

1512
1513

1514
1515
1516
1517
1518
1519

1520
1521
1522
1523
1524
1525

1526
1527
1528
1529
1530

1531
1532
1533
1534
1535
1536

1537
1538
1539
1540
1541
1542

1543
1544
1545
1546
1547
1548

1549
1550
1551
1552
1553

1554
1555
1556
1557
1558
1559

1560
1561
1562
1563
1564
1565

(cid:19)

(44)

(45)

(46)

Abbreviate h1(x, y) as h1. Follow the same procedure in the proof of Theorem B.2,
(cid:21)2 πo
h1

dy − 2(cid:112)wt(xr, πo) ·

wt(xr, πθ) − wt(xr, πo) ≥

(cid:90) (cid:20) ∂
∂xr

πo ·

(cid:13)
(cid:13)
(cid:13)
(cid:13)

h1

h1

√

(cid:13)
(cid:13)
(cid:13)
(cid:13)2

(cid:16)

(cid:17)
c1 − (cid:112)wt(xr, πo)

·

=

(cid:18)

(cid:112)wt(xr, πo) + c1 + 2

∂
∂xr
(cid:13)
√
(cid:13)
(cid:13)
(cid:13)

− wt(xr, πo)

πo ·

∂
∂xr

h1

(cid:13)
(cid:13)
(cid:13)
(cid:13)2

where we apply Cauchy–Schwarz inequality in equation 44.

c1 =

(cid:115)(cid:13)
(cid:112)πo(y|x) ·
(cid:13)
(cid:13)
(cid:13)

(cid:13)
(cid:90) (cid:18) ∂
(cid:13)
(cid:13)
∂xr
(cid:13)2
Thus, wt(xr, πθ) > wt(xr, πo) if (cid:112)wt(xr, πo) < c1. Again, by Cauchy–Schwarz inequality

(cid:13)
(cid:112)πo(y|x) ·
(cid:13)
(cid:13)
(cid:13)

(cid:19)2 πo(y|x)

∂
∂xr

∂
∂xr

(cid:13)
2
(cid:13)
(cid:13)
(cid:13)
2

dy −

h1

h1

h1

h1

+

≤

wt(˜xr, πθ) − wt(˜xr, πo)
(cid:90) (cid:18) ∂h1
(cid:18) ∂πo
(cid:19)2 πo
h1
∂ ˜xr
∂ ˜xr
(cid:16)(cid:112)wt( ˜xr, πo) − c2
(cid:17)

+

·

= −

where

(cid:19)2 h1
πo
(cid:112)wt( ˜xr, πo) − c2 + 2

dy + 2(cid:112)wt( ˜xr, πo) ·
(cid:13)
(cid:13)
(cid:13)
(cid:13)

(cid:18)

(cid:13)
(cid:13)
(cid:13)
(cid:13)
√

√

πo ·

πo ·

∂
∂ ˜xr

(cid:13)
∂h1
(cid:13)
(cid:13)
∂ ˜xr
(cid:13)2
(cid:13)
(cid:19)
(cid:13)
(cid:13)
(cid:13)2

h1

− wt(˜xr, πo)

(cid:115)(cid:13)
(cid:13)
(cid:13)
(cid:13)

√

c2 =

(cid:19)2 πo
∂
∂ ˜xr
h1
Thus, wt(xr, πθ) < wt(xr, πo) if (cid:112)wt(xr, πo) > c2.

(cid:90) (cid:18) ∂
∂ ˜xr

πo ·

h1

h1

+

(cid:13)
(cid:13)
(cid:13)
(cid:13)

2

2

+

(cid:18) ∂
∂ ˜xr

πo

(cid:19)2 h1
πo

dy +

√

πo ·

(cid:13)
(cid:13)
(cid:13)
(cid:13)

∂
∂ ˜xr

h1

(cid:13)
(cid:13)
(cid:13)
(cid:13)2

(47)

29

