Mitigating Spurious Correlations in Zero-Shot Multimodal Models
Under review as a conference paper at ICLR 2025

MITIGATING SPURIOUS CORRELATIONS IN ZERO-
SHOT MULTIMODAL MODELS

Anonymous authors
Paper under double-blind review

ABSTRACT

Multimodal models or Vision Language Models (VLMs) have reshaped the
paradigm in machine learning, offering zero-shot capabilities that require no ad-
ditional training when adapted to new classification tasks. However, despite their
advancements, spurious correlations still exist in VLMs. Existing approaches to
tackle this issue often require target label annotations, contradicting the principle
of zero-shot classification, or they primarily focus on a single modality, risking
misalignment between text and image modalities. Others rely on extensive do-
main knowledge or large language models (LLMs) to characterize spurious fea-
tures, making the performance sensitive to the generated prompts and undermin-
ing zero-shot capability. In response, we propose a new solution that tackles spu-
rious correlations in VLMs within the zero-shot setting. Our approach utilizes a
translation operation that preserves the latent space distribution to address issues
of spurious correlations. In particular, our method is grounded in and inspired by
a theoretical analysis, which identifies that the optimal translation directions are
along the spurious vector. As VLMs unify two modalities, we compute spurious
vectors from the text prompts and guide the translation for image embeddings,
aligning the requirements for the fusion of different modalities in VLMs. We
conducted experiments on benchmark datasets, which have shown significant im-
provements in worst-group accuracy. Additionally, our visualizations of VLMs
further demonstrate the effectiveness of this intervention.

1

INTRODUCTION

Vision Language Models (VLMs) have significantly enhanced the capabilities of machine learn-
ing systems. Contrastive Language-Image Pretraining (CLIP) (Radford et al., 2021), which bridges
the fields of computer vision and natural language processing, has profoundly transformed the land-
scape. One of the fascinating capabilities of VLMs is their zero-shot functionality (Guo et al., 2023).
This functionality enables models to infer the most probable answer from a set of potential responses
provided by the user, even without training on the specific dataset.

Figure 1: Heatmap visualization for zero-shot classification. The benign lesion class in the ISIC
dataset is spuriously correlated with the presence of color patches, leading to predictions of benign
lesions being dangerously dependent on this feature in the biomedical setting. Similarly, in the
Waterbirds dataset, there is a spurious correlation between waterbirds and water backgrounds. Our
approach effectively decorrelates these spurious relationships without requiring a training process,
promoting group robustness in the zero-shot setting.

1

000
001

002
003
004
005
006
007

008
009
010
011
012
013

014
015
016
017
018

019
020
021
022
023
024

025
026
027
028
029
030

031
032
033
034
035
036

037
038
039
040
041

042
043
044
045
046
047

048
049
050
051
052
053

Under review as a conference paper at ICLR 2025

054
055

056
057
058
059
060
061

062
063
064
065
066
067

068
069
070
071
072

073
074
075
076
077
078

079
080
081
082
083
084

085
086
087
088
089
090

091
092
093
094
095

096
097
098
099
100
101

102
103
104
105
106
107

Despite the power of VLMs, these models still suffer from spurious correlations (Zheng et al., 2024;
Dehdashtian et al., 2024; Wortsman et al., 2022), a phenomenon where predictions are based on
irrelevant features, leading to detrimental performance for certain groups (Sagawa et al., 2019).
Spurious correlations pose significant risks in high-stakes settings such as medical diagnostics. For
instance, in diagnosing skin cancer, if a color patch is spuriously correlated with benign samples,
the model may erroneously base its predictions on the presence of this color patch (Yan et al., 2023;
Nauta et al., 2021) (See Figure 1 ISIC Dataset (Codella et al., 2019)).

Addressing spurious correlations in VLMs is increasingly imperative. Efforts such as (Yang et al.,
2023; Pang et al., 2024; Goyal et al., 2023; Zhang & R´e, 2022; Wang et al., 2023) have aimed to
mitigate spurious correlations issues within VLMs. However, these methods rely on target labels, a
practice that contradicts the label-free requirements of zero-shot classification.

A key characteristic of VLMs is the integration of an image encoder and a text encoder, which pro-
cess image and text inputs, respectively. These inputs are transformed into image embeddings and
text embeddings. Many studies (An et al., 2024; Chuang et al., 2023; Trager et al., 2023) have con-
centrated on mitigating spurious correlations via text embeddings. However, these methods present
several challenges. Firstly, they concentrate exclusively on a single modality, posing a substan-
tial risk of misalignment between modalities. This contradicts the principle of matching different
modalities in VLMs. Secondly, these methods require strong domain expertise or access to gen-
erative tools such as Large Language Models (LLMs) to generate descriptions of the concepts of
spurious features or substantial exemplars of such features. However, the responses from generative
tools are not reliable. Zhang et al. (2023b); Xu et al. (2024) indicate the existence of hallucinations
in LLMs. This unreliability substantially diminishes the effectiveness of methods designed to mit-
igate spurious correlations through text-based modalities. Moreover, An et al. (2024); Adila et al.
(2024) observe performance disparities when using different LLMs.

A recent study, ROBOSHOT (Adila et al., 2024), has been proposed to address spurious correla-
tion issues by considering both image and text modalities. ROBOSHOT employs LLMs to generate
sufficient insights for spurious features and then applies a linear projection to map image embed-
dings onto a neutralization hyperplane for these spurious features. This approach presents several
challenges. First, the spurious insights generated by LLMs are inherently less reliable. Second, the
projection operation distorts the distribution of image embeddings and significantly reduces their
diversity. Third, this method lacks theoretical analysis of the optimality of the projection direction,
a factor that critically influences the performance of group robustness.

To sum up, existing methods can be categorized into three types, each with specific concerns. First,
some methods require target labels, violating the zero-shot classification requirements. Second,
methods that focus solely on one modality face risks of misalignment when integrating different
modalities. Third, approaches using linear projection distort the distribution of image embeddings.
Additionally, reliance on LLMs introduces concerns regarding reliability.

To robustify zero-shot VLMs effectively, the main requirements are no training, no label require-
ment, no reliance on LLMs. To address these challenges, we propose a novel approach TIE,
a framework that utilizes text prompt guidance to reduce spurious features in image embeddings.
Contrary to the linear transformation techniques introduced in (Adila et al., 2024; Chuang et al.,
2023), we adopted a translation operation in the latent space, which preserves the distribution of
image embeddings. Our method is grounded in theoretical analysis that identifies the optimal pa-
rameter for translating image embeddings. Unlike methods that focus on a single modality, we
incorporate text prompts to guide the translation operation in the image space, thereby preserving
alignment across both modalities.

In practice, when spurious labels are inaccessible, we develop TIE*. TIE* leverages a zero-shot
manner to infer spurious features and utilizes pseudo-spurious labels to enhance the group robustness
of VLMs, without relying on manual annotations. Throughout this process, our method does not
require training any parameters in VLMs, thus enhancing efficiency.

We conducted extensive experiments on real-world datasets, including high-stakes biomedical set-
tings. The results show that our method significantly outperforms existing approaches. Additionally,
we provide visualizations to demonstrate that the proposed method effectively mitigates spurious
correlations.

2

Under review as a conference paper at ICLR 2025

108
109

110
111
112
113
114
115

116
117
118
119
120
121

122
123
124
125
126

127
128
129
130
131
132

133
134
135
136
137
138

139
140
141
142
143
144

145
146
147
148
149

150
151
152
153
154
155

156
157
158
159
160
161

We summarize our contribution as follows:

• We propose a theoretically inspired method that is simple and effective in mitigating spuri-

ous correlation issues in VLMs for zero-shot classification.

• The proposed algorithm operates without the need for LLMs or labeled data, and does not

require access to the internal parameters of VLMs.

• We empirically validate the effectiveness of the proposed method, including visualizations

across both image and text modalities.

2 RELATED WORKS

2.1 GROUP ROBUSTNESS

Many methods have been proposed to enhance group robustness and address issues of spurious
correlations (Sagawa et al., 2019; Arjovsky et al., 2019; Idrissi et al., 2022; Kirichenko et al., 2022;
Liu et al., 2021; Yao et al., 2022; Krueger et al., 2021). These approaches predominantly utilize
reweighting techniques to adjust the weights of samples in the training set. These methods are
designed for single-modality classification and involve training either all or a subset of the model’s
parameters. In contrast, our approach significantly differs from these conventional methods as it
requires no adjustments to the parameters in the backbone during the robustification process.

2.2 MITIGATING SPURIOUS CORRELATION IN VLMS

To mitigate spurious correlations in VLMs, many approaches focus on fine-tuning using labeled
datasets. Specifically, Goyal et al. (2023) employ target labels derived from text descriptions and
fine-tunes the model using a contrastive loss. Yang et al. (2023) propose a method that detects spu-
rious attributes and fine-tunes VLMs using contrastive loss both within and across different modali-
ties. Petryk et al. (2022) propose a framework that uses VLMs to integrate textual information with
images and generate a saliency map. This map is then used to supervise the training of a classifier.
Zhang & R´e (2022) propose an adapter that connects to the embedding layer and utilizes contrastive
loss to fine-tune the adapter. Dehdashtian et al. (2024) propose a method that employs the Hilbert-
Schmidt Independence Criterion (HSIC) to debias both image and text embeddings. Pang et al.
(2024) introduce a method for distributional robustness via language that maximizes the entropy of
predictions on spurious attributes. Distinct from the existing methods mentioned above, our method
operates without any labeled data, thus fulfilling the requirements for zero-shot classification.

2.3 GROUP ROBUSTNESS IN ZERO-SHOT CLASSIFICATION

Another line of research addresses spurious correlation issues in VLMs in a zero-shot manner.
Trager et al. (2023) propose a method that combines a target prompt with spurious prompts and
averages them to generate an ‘Ideal words’ prompt. An et al. (2024) employs a two-step inference
method that first identifies spurious features and then augments the text prompt with these identified
features. Chuang et al. (2023) propose a method that projects text embeddings onto a space or-
thogonal to the spurious attribute space. Ge et al. (2023) aim to enhance text prompt robustness by
focusing on label augmentation. Adila et al. (2024) propose a method that uses the Gram-Schmidt
process to project representations onto a space orthogonal to spurious features.
In contrast, our
method does not depend on augmenting the prompt, which simplifies usage and reduces concerns
about the hallucination problem in LLMs. Additionally, our approach aims to mitigate spurious
correlations from a multimodal perspective.

3 METHODS

3.1 PRELIMINARIES

Setting. This work focuses on the group robustness setting (Sagawa et al., 2019) in the zero-shot
classification task. Denote x ∈ X as the input image, y ∈ Y as the target label, and a ∈ A as
the spurious feature. Define group gy,a ∈ G considering the combination of target label y and
spurious feature a. To mitigate the impact of spurious correlations on prediction, our approach

3

Under review as a conference paper at ICLR 2025

follows the established practices (Sagawa et al., 2019; Liu et al., 2021; Kirichenko et al., 2022)
aimed at enhancing the accuracy of the worst groups while preserving overall accuracy.

Relationship between vanilla classification and zero-shot classification. We first bridge these
two tasks for the subsequent theoretical discussion. Denote ϕI (·) as the image encoder, ϕT (·) as the
text encoder, ty ∈ T as the text prompt, with each text prompt corresponding to one target label
y. For example, in waterbirds dataset (Sagawa et al., 2019), for y = Waterbird, ty = “a photo
of a waterbird”, T = {a photo of a waterbird, a photo of a landbird },
where |T | = K, corresponding to K classes of text prompts. For zero-shot classification, the VLMs
model serves as a score function that maps X × T → R:

ˆy = arg maxk∈[K]⟨ϕI (x), ϕT (tk)⟩.

(1)

Equation 1 shows the zero-shot paradigm that predicts the class ˆy as the one with the highest inner
product between the image embedding and the text prompt embedding.
Vanilla classification: Denote h ∈ Rd as the representation learned from a neural network, which
is processed by an image encoder ϕI (·), i.e. h = ϕI (x). W = [w1, ...wk] ∈ Rd×K as a linear
classifier. The vanilla classification task:

ˆy = arg maxk∈[K]W⊤h = arg maxk∈[K]⟨ϕI (x), W⟩.

(2)

Comparing Equation 2 with Equation 1, it can be concluded that the zero-shot classification rep-
resents a specialized form of vanilla classification, where the linear classifier is composed of text
embeddings. For simplicity in the following analysis, we use h to denote ϕI (x) and w to represent
ϕT (ty), based on their equivalence.

3.2 THEORETICAL ANALYSIS

Spurious correlation modeling. We adopt a common setting in modeling spurious correlation
(Sagawa et al., 2020; Idrissi et al., 2022; Yao et al., 2022; Wang & Wang, 2024). Concretely, denote
a spurious feature a ∈ {−1, 1} and a label y ∈ {−1, 1}. Each (y, a) group denoted as gy,a has its
own distribution over the image embedding h = [hspu, hcore, hnoise] ∈ Rd, where

hspu|a ∼ N (a, σ2

spu), hcore|y ∼ N (y, σ2

core), hnoise ∼ N (0, I).

(3)

The data model assumption is for the simplicity of the following analysis. Without loss of generality,
the dimensions of core features and spurious features can be arbitrary. We investigate the problem
of improving the group robustness of VLMs in a zero-shot setting by adjusting h given fixed target
text prompts. By modeling each group with equal weight, the goal is to maximize each group-wise
utility:

LAcc(hgy,a, w) = max

h

(cid:88)

gy,a∈G

A(hgy,a, w; y),

(4)

where A(·) is the accuracy function, hgy,a corresponds to the image embeddings from group gy,a.
We introduce Lemma 1, which establishes that the accuracy for each group can be derived in an
analytical form.

Lemma 1 Under the above data model assumption, the group-wise accuracy can be derived as

erfc(−

(cid:113)

), if y = 1

A(hgy,a, w; y) =






1
2

1
2

w⊤µgy,a
2w⊤Σgy,aw
w⊤µgy,a
2w⊤Σgy,aw

erf(−

(cid:113)

1
2

, if y = −1,

(5)

) +

where µgy,a and Σgy,a represent the mean and covariance matrix of the image embedding hgy,a.

The proof is presented in Appendix A. This lemma quantifies the accuracy of each (y, a) group
given a fixed classifier w. According to Lemma 1, adjusting either µ or Σ impacts the group-wise
accuracy. The solution proposed by (Adila et al., 2024) involves changing Σ, which changes the

4

162
163

164
165
166
167
168
169

170
171
172
173
174
175

176
177
178
179
180

181
182
183
184
185
186

187
188
189
190
191
192

193
194
195
196
197
198

199
200
201
202
203

204
205
206
207
208
209

210
211
212
213
214
215

Under review as a conference paper at ICLR 2025

distribution of the image embeddings in the latent space. This change necessitates a highly precise
decision boundary for spurious features, as the accuracy of the worst-performing group is extremely
sensitive to the accuracy of this boundary. If the boundary is not accurately defined, the worst-
performing group’s accuracy will significantly deteriorate. We discuss this phenomenon further and
provide a theoretical comparison along with experimental validation of our approach in Section 3.3
and Appendix C.1.

Objective. We propose a translation operator that preserves the distribution of image embeddings.
In particular, our objective function is to find the optimal translation vectors va to maximize the
following objective function:

LAcc(va; hgy,a, w) = max
va

(cid:88)

gy,a∈G

A(hgy,a + va, w; y),

(6)

va is the translation vectors based on the label of spurious features. In Theorem 1, we establish the
optimal vector for translation within the complete set of feasible directions. We leave the detailed
proof in Appendix B.

Theorem 1 Given the objective function and the data model, the maximizer of the objective is ob-
tained by

where P ∈ Rd×d is an elementary matrix, P =

va = E[−Pha],

1
0
...
0

0
0
...
0





· · ·
· · ·
. . .
· · ·

(7)


0
0

.
...


0

Theorem 1 states that the optimal translate vector va can be computed by va = E[−hspu, 0, ..., 0],
which is the negative direction of the spurious feature vector. However, estimating the spurious
feature vector presents a challenge. Wu et al. (2023) proposed first training a classifier to classify
the spurious feature and then using the vector orthogonal to the decision hyperplane as the spurious
feature vector. We argue that this method significantly compromises efficiency as the need for
training and risks misalignment in the text embedding space. In the realm of VLMs, effectively
combining both text and image embeddings is crucial. Therefore, we propose using spurious text
embeddings to guide image embeddings toward an optimal state.

3.3 TIE: TEXT PROMPT BASED IMAGE EMBEDDING TRANSLATION

Figure 2: TIE* overview. First, we utilize spurious prompts to compute the spurious vectors. We
then employ the CLIP model to infer the spurious label for each sample. Subsequently, we translate
the image embeddings along the spurious vector based on the pseudo-spurious label. Finally, we use
these translated embeddings to perform the zero-shot classification task.

We now present our method to mitigate spurious correlations in the VLMs, an overview is shown in
Figure 2. Based on the analysis in Section 3.2, we first compute the spurious feature vector. Next,
we translate the image embeddings along the opposite of this direction, and then use the adjusted
image embeddings to perform zero-shot classification.

Computation on Spurious feature vector. Given a set of spurious text prompts Ta (e.g. a photo
with a water background, a photo with a land background). TIE computes
the spurious vector va = ϕT (ta; a), s.t. ta ∈ Ta. TIE normalizes va by its L2 norm: va = va
.

||va||2

5

216
217

218
219
220
221
222
223

224
225
226
227
228
229

230
231
232
233
234

235
236
237
238
239
240

241
242
243
244
245
246

247
248
249
250
251
252

253
254
255
256
257

258
259
260
261
262
263

264
265
266
267
268
269

Under review as a conference paper at ICLR 2025

Translate the image embeddings. Given an image, TIE first computes its image embedding using
the image encoder, i.e., ha = ϕI (x; a). Then, TIE computes the magnitude of the translation by
λa = E[h⊤
a va], which is the average projection length on the direction of va. Next, TIE translates
image embedding by

ha ← ha − λava.

(8)

The zero-shot classification task employs ha and target prompts for execution.

Without spurious feature label. One constraint on TIE is its dependency on access to labels for
spurious features, with samples bearing various spurious labels moving in different directions to
achieve an optimal state. To address this, we propose TIE* that eliminates the need for any labeled
data within the dataset.

An et al. (2024) empirically demonstrated that spurious features can be effectively inferred using
VLMs. Building upon this insight, we leverage VLMs to infer the spurious labels for each sample
in the dataset. Concretely, we assign a Pseudo-spurious label in the zero-shot classification setting:

ˆa = arg maxa∈A⟨ϕI (x), ϕT (ta)⟩

(9)

where ˆa is the pseudo-spurious label for the sample. In equation 9, the pseudo-labeling procedure
requires of all possible spurious text prompts. We utilize these pseudo-labeled to implement the
corresponding translation operation as introduced in the previous section. We summarize our method
in Algorithm 1.

We conduct experiments under two scenarios: In the first, where the labeled spurious feature is
available, we apply the true spurious label to implement TIE. In the second scenario, where the
labeled spurious feature is unavailable, we execute the complete algorithm as outlined in Algorithm
1, denoted as TIE*. Additionally, we investigate a method applicable when partially labeled data is
available. The detailed discussion of this method is deferred to Section 4.4.

3.4 THEORETICAL COMPARISON BETWEEN TIE AND ROBOSHOT

TIE and ROBOSHOT are methods designed to address spurious correlations by leveraging both
image and text modalities. We provide a detailed comparison of the worst group accuracy between
two methods under different spurious text prompts and label prompts. To quantify the effects of
spurious text prompts and target label text prompts, as discussed in 3.1, these prompts form two
classifiers: wspu for spurious prompts and w for label prompts. We define wspu = [1, α, 0] and
w = [1, β, 0], α, β ∈ R+ A smaller α indicates more accurate spurious decision boundary, while a
larger β indicates a more accurate task boundary. Utilizing these definitions, we have the analytical
forms for the worst group accuracy (WG) for both ROBOSHOT and TIE:

ROBOSHOT:W GRS(α, β) = min{

1
2

erfc(−

α2 − (1 + β)α + β
(1 + α2)(cid:112)2(β2α2 + (α − β(1 − α2))2)
1
2

α2 − (β − 1)α − β
(1 + α2)(cid:112)2(β2α2 + (α − β(1 − α2))2)

) +

1
2

erf(−

TIE:W GT IE(α, β) = min{

1
2

erfc(−

β(1 − α
1+α2 )
(cid:112)2(1 + β2)

),

1
2

erf(−

β(1 + α
1+α2 )
(cid:112)2(1 + β2)

) +

1
2

}.

),

}.

(10)

(11)

We defer the derivation of equations 10 and 11 in Appendix C. We present a plot of the theoretical
worst group accuracy with respect to α and β in Figure 3. We observe that ROBOSHOT only achieves
a higher WG when α → 0, representing the perfect spurious classifier. Otherwise, ROBOSHOT’s
performance drops rapidly when the spurious classifier is inaccurately approximated, showing a sig-
nificant margin compared to TIE. In other words, the performance of TIE shows better robustness
across different text prompts. We further substantiate this analysis with empirical validation on a
real-world dataset, as detailed in Appendix C.1.

4 EXPERIMENT

4.1 SETUP

6

270
271

272
273
274
275
276
277

278
279
280
281
282
283

284
285
286
287
288

289
290
291
292
293
294

295
296
297
298
299
300

301
302
303
304
305
306

307
308
309
310
311

312
313
314
315
316
317

318
319
320
321
322
323

Under review as a conference paper at ICLR 2025

Datasets. We study five well-established
benchmark datasets for spurious correlation re-
search: Waterbirds (Koh et al., 2021; Sagawa
et al., 2019), CelebA (Liu et al., 2015), ISIC
(Codella et al., 2019), COVID-19 (Cohen et al.,
2020), FMOW (Christie et al., 2018). Please
refer to appendix E for detailed information.

Backbones. Existing research indicates that
different visual backbones produce varied re-
sults. Following established protocols (Adila
et al., 2024),
for the Waterbirds and ISIC
datasets, we examine CLIP models with vision
backbone of ViT-B/32, ViT-L/14, and RN50 (Il-
harco et al., 2021; Cherti et al., 2023; Radford
et al., 2021). For the ISIC and COVID-19 datasets, we utilize Biomed CLIP (Zhang et al., 2023a)
as the vision backbone. For the FMoW dataset, we employ the ViT-L/14 model due to the dataset’s
complex nature.

Figure 3: Theoretical comparison of worst group
accuracy between TIE and ROBOSHOT.

Baselines. We compare our method against two baselines and existing state-of-the-art methods in
robust zero-shot classification. Concretely, two baselines are vanilla zero-shot classification (ZS),
Zero-shot with group information (Group prompt). Existing SOTA methods including Ideal Prompt
(Trager et al., 2023), Orth-Cali (Chuang et al., 2023), Perception CLIP (An et al., 2024), RO-
BOSHOT (Adila et al., 2024). We leave the details of baselines in Appendix F.

Text Prompts for Reproducibility. Zero-shot classification employs two types of text prompts:
label prompts and spurious prompts. To ensure a fair comparison, all methods utilize the same
label prompts. For example, the label prompts for the Waterbirds dataset are [a photo of a
landbird, a photo of a waterbird]. For spurious prompts, we use the prompts provided
by the authors if the method is tested on a specific dataset. Otherwise, we generate spurious prompts
using generative AI tools like ChatGPT (OpenAI, 2023), following the guidelines specified in the
original papers. For reproducibility, prompts used in our experiments are provided in Appendix G.

Metrics. Following the protocol established by robust learning studies (Sagawa et al., 2019; Adila
et al., 2024), we report three metrics: worst group accuracy (WG), average accuracy (Avg), and
the gap between these two metrics (Gap). We highlight the best result in bold and underline the
second-best result.

4.2 MAIN RESULTS.
Waterbirds. Table 1 summarizes results on the Waterbirds dataset. TIE achieves significant im-
provement over comparative methods by a relatively large margin, especially for the ViT-L14 vision
backbone, where the worst group accuracy reaches 78.82%, surpassing the previous method by
14.65%. TIE* achieves a comparable performance in the ViT backbones. However, performance
varies with different backbone models. For ResNet-50, Orth-Cali outperforms other methods.

Table 1: Zero Shot classification results on Waterbirds

Method

CLIP (ViT-B32)

CLIP (ViT-L14)

CLIP (ResNet-50)

ZS
Group Prompt
Ideal words
Orth-Cali
Perception CLIP
ROBOSHOT
TIE (Ours)
TIE* (Ours*)

WG ↑ Avg ↑ Gap ↓ WG ↑ Avg ↑ Gap ↓ WG ↑ Avg ↑ Gap ↓
45.28
51.79
83.72
41.37
31.93
21.12
10.44
43.46
45.68
56.12
64.17 87.67 23.50
40.39
60.28
19.67
86.31
58.56
54.99
86.74
43.30
54.12
59.78
42.45
64.43
45.17
54.41
78.82
71.35
30.66
84.12
61.24
47.08
78.98
61.60

35.36
49.84
39.09
27.75 64.80
48.21
32.62
26.61
19.26
52.96
5.30
17.38
34.11

27.11
68.48
23.33
66.79
18.92
79.20
14.20
69.19
82.50 22.72
17.51
71.92
8.47
79.82
15.67
76.91

80.64
70.96
79.48
84.47
91.51
69.06
83.62
81.19

CelebA. Table 2 presents results for the CelebA dataset. Similar to the Waterbirds dataset, TIE con-
sistently outperforms comparison baselines and achieves the smallest gap in ViT backbone models.

7

324
325

326
327
328
329
330
331

332
333
334
335
336
337

338
339
340
341
342

343
344
345
346
347
348

349
350
351
352
353
354

355
356
357
358
359
360

361
362
363
364
365

366
367
368
369
370
371

372
373
374
375
376
377

Under review as a conference paper at ICLR 2025

The performance of TIE* is comparable to that of TIE. For the ResNet backbone, Perception CLIP
outperforms other methods.

Table 2: Zero Shot classification results on CelebA

Method

CLIP (ViT-B32)

CLIP (ViT-L14)

CLIP (ResNet-50)

ZS
Group Prompt
Ideal words
Orth-Cali
Perception CLIP
ROBOSHOT
TIE (Ours)
TIE* (Ours*)

WG ↑ Avg ↑ Gap ↓ WG ↑ Avg ↑ Gap ↓ WG ↑ Avg ↑ Gap ↓
84.27
11.89
81.20
78.89
7.85
80.38
8.89
8.92
77.86
74.90
89.15 12.48
80.96
10.62
78.12
82.31
7.34
3.70
81.39
77.92
4.95
2.71
81.41
80.32
76.46
84.77
80.52
6.94
2.93
85.54
86.17
82.63 85.11
6.39
1.57
2.29
85.10
82.61
6.40
84.27

81.58
69.69
79.48
70.59
76.27
65.65
69.13
76.47
80.22 85.17
80.90
73.96
81.71
75.32
81.70
75.30

73.35
68.94
76.67
77.69
78.70
82.61
84.60
81.98

5.38
5.48
2.84
4.39
3.86
4.25
2.48
2.49

ISIC and COVID-19. Our experiments extend to specialty datasets within high-stakes settings,
specifically deploying VLM models in the medical domain. Table 3 shows the results for the ISIC
and COVID-19 datasets where our method outperforms baseline methods in worst-group accuracy
and achieves comparable average accuracy.

Table 3: Zero Shot classification results on ISIC and Covid-19 datasets

Method

ISIC (Biomed CLIP)

COVID-19 (Biomed CLIP)

ZS
Group Prompt
Ideal words
Orth-Cali
Perception CLIP
ROBOSHOT
TIE (Ours)
TIE* (Ours*)

WG ↑ Avg ↑ Gap ↓
28.00
70.21
42.21
17.92
30.05
12.13
11.65
53.07
41.42
72.54
51.11
21.43
11.19
52.74
41.55
6.54
59.84
53.30
4.03
65.87
69.90
71.68
61.11
10.57

WG ↑ Avg ↑ Gap ↓
61.81
16.98
44.83
20.69
48.27
27.58
33.31
56.84
23.53
6.89
51.72
44.83
8.03
56.87
48.84
20.35
53.10
32.75
62.50
52.17
10.33
50.22
10.86
61.08

FMOW. We extend our experiments to multiclasses and multigroup settings. The FMOW dataset
includes 62 classes and is organized into 5 spurious groups. Table 4 shows the results for FMOW.
TIE achieves the highest accuracy in the worst-performing group, TIE* shows comparable per-
formance on the worst group accuracy and has the highest overall accuracy. These results further
validate the effectiveness of our methods in mitigating spurious correlations in the zero-shot setting.

Table 4: Top-1 Accuracy and Worst Group accuracy on FMOW dataset.

WG ↑ Avg ↑ Gap ↓

ZS
Group Prompt
Ideal words
Orth-Cali
Perception CLIP
ROBOSHOT
TIE
TIE*

18.06
8.75
11.14
19.45
12.61
10.88
20.19
19.84

26.02
14.69
20.21
26.11
17.70
19.79
26.62
26.65

7.96
5.94
9.07
6.66
5.09
8.91
6.43
6.81

Discussion. From Table 1-4, TIE consistently achieves the best or second-best WG, TIE* achieves
a comparable result but still has a performance gap, which will be discussed in the following sec-
tion. We found TIE shows relative suboptimal performance using ResNet-50 on the Waterbirds
dataset. Note that all text encoders are transformer-based models, while the vision backbones vary.

8

378
379

380
381
382
383
384
385

386
387
388
389
390
391

392
393
394
395
396

397
398
399
400
401
402

403
404
405
406
407
408

409
410
411
412
413
414

415
416
417
418
419

420
421
422
423
424
425

426
427
428
429
430
431

Under review as a conference paper at ICLR 2025

We hypothesize that this suboptimality primarily arises from a misalignment between the direction
of the spurious vector in the text space and the image space. This misalignment stems from the
structure and scales of the encoders, which echoes the finding that different CLIP structures show
significantly different zero-shot classification results (Radford et al., 2021). Methods like Orth-Cali
or Perception CLIP, which only focus on debiasing text embeddings, introduce randomness into
zero-shot classification. This randomness can occasionally enhance performance. However, adjust-
ing text embeddings without considering image embeddings can result in misalignment, leading to
a significant drop in performance. For example, Orth-Cali shows suboptimal performance on the
ISIC dataset. Conversely, our method mitigates this randomness by integrating both image and text
modalities, thereby enhancing the stability of zero-shot classification outcomes.

4.3 GROUP ROBUST TEXT PROMPT

In this section, we demonstrate that our method is compatible with other methods focused on
mitigating spurious correlations in the text modality. An et al. (2024) highlight that providing
additional context enhances the performance of VLM models. Inspired by this insight, we employed
group-robust prompts to identify spurious directions. Specifically, we utilize GPT-4 (OpenAI,
2023) to generate five sentences that serve as synonyms for spurious features. The prompt for the
GPT-4 is Please generate 5 synonyms of [Spurious feature]. For instance,
the robustified spurious prompts for the Waterbirds dataset include: for a land background, [A
photo with a land background.
photo of a mountain background.
A photo of a Ground background]; and for a water background, [A photo with a
water background. A photo of an ocean background.
sea background. A photo of a Lake background.
A photo of a River
background.]. We computed the average text embedding from these spurious prompts and used
it to update the image embedding. The results are shown in Table 5. We observe that the robustified
prompt helps find a more robust direction for the spurious features, leading to improved WG and
Avg metrics with ViT-B32 and ResNet-50 models.

A
A photo of a Terrain background.

A photo of a forest background.

A photo of a

Method

ViT-B-32

ViT-L-14

ResNet-50

Table 5: Group robustify prompting

TIE*
TIE* Robust

WG ↑ Avg ↑ Gap ↓ WG ↑ Avg ↑ Gap ↓ WG ↑ Avg ↑ Gap ↓
78.98
47.08
61.24
43.59
64.96
78.46

61.60
61.46

34.11
38.63

15.67
13.67

17.38
17.00

81.19
82.22

76.91
78.63

4.4 LIMITED ACCESS TO LABELS OF THE SPURIOUS FEATURES

Table 1 reveals a performance disparity between TIE and TIE*, suggesting that accurate estimation
of the spurious label enhances performance. Wang & Wang (2024) theoretically demonstrates that
feature separability directly influences performance, especially when spurious features are more
separable than core features. Based on this, accurately predicting labels of the spurious features
necessitates significantly fewer training samples. Therefore, we propose using a partially spurious
feature labeled dataset to infer the spurious labels of the entire dataset, and subsequently apply
our algorithm based on the pseudo labels of the spurious feature. We tested this approach on the
Waterbirds dataset with training sample sizes ranging from 100 to 1000. To optimize efficiency, we
employed a smaller-scale architecture, ResNet-18 (He et al., 2016), to predict the pseudo-spurious
feature labels. The model was trained using an SGD optimizer with a learning rate of 10−4, a weight
decay of 10−3, and a momentum of 0.9, over 200 epochs. The VLM model is tested using ViTB-32.

Figure 4 reports the outcomes utilizing different sample sizes within the training set. Observations
indicate that increasing the amount of labeled data enhances the worst group accuracy of the CLIP
model. Specifically, using 1000 samples, performance nearly matches that of our method when
attribute a is known. Additionally, the figure demonstrates a nearly linear improvement in worst
group accuracy as the accuracy of predictions on spurious feature labels increases in the CLIP model.

9

432
433

434
435
436
437
438
439

440
441
442
443
444
445

446
447
448
449
450

451
452
453
454
455
456

457
458
459
460
461
462

463
464
465
466
467
468

469
470
471
472
473

474
475
476
477
478
479

480
481
482
483
484
485

Under review as a conference paper at ICLR 2025

Figure 4: Performance on the Waterbirds dataset using partially labeled spurious features.

4.5 VISUALIZATION

In addition to the superior performance of our method, we further investigate its capacity to ensure
that predictions are correct for the right reasons. This can be verified through visual explanation
maps, as illustrated in Figure 5. We employed the explainability method from (Chefer et al., 2021)
to generate heatmaps for both image features and text prompts. Our method significantly reduces
reliance on spurious features in a zero-shot setting. In the ISIC dataset, it specifically minimizes
attention to irrelevant color patches. For samples of malignant lesions, our approach enhances focus
on the lesion itself rather than the other skin part. For the Waterbirds dataset, even in the vanilla
zero-shot where the focus might incorrectly shift to the background, our method effectively redirects
attention towards the core features of the subject. Interestingly, after implementing our method, the
text prompts also show increased attention to specific objects, such as bird and malignant.

Figure 5: Attention based explanations (Chefer et al., 2021) for ISIC and Waterbirds datasets.

5 CONCLUSION

Addressing spurious correlations presents a critical challenge in the realm of zero-shot VLMs. This
study draws inspiration from rigorous theoretical analysis to examine optimal strategies for translat-
ing image embeddings. To address the spurious correlations effectively, we have designed the TIE
algorithm, which guides the translation of image embeddings based on the text prompt. Extensive
experiments conducted on real-world datasets demonstrate that our method not only significantly
improves the worst-group accuracy across all datasets but also achieves comparable overall accu-
racy. Additionally, we visualize results from both modalities to confirm that the predictions are
based on valid reasons.

Failure case discussion and Future direction. Although our proposed method demonstrates sig-
nificant robustness, TIE* may encounter failures when pseudo-spurious labels are incorrectly as-
signed. We present a comprehensive analysis of these failure cases and propose solutions in Ap-
pendix K. Additionally, TIE faces limitations when processing images with artifacts. We discuss
these issues in detail in Appendix J. Identifying such artifacts could be a promising direction for
future research to enhance zero-shot classification performance.

10

486
487

488
489
490
491
492
493

494
495
496
497
498
499

500
501
502
503
504

505
506
507
508
509
510

511
512
513
514
515
516

517
518
519
520
521
522

523
524
525
526
527

528
529
530
531
532
533

534
535
536
537
538
539

Under review as a conference paper at ICLR 2025

540
541

542
543
544
545
546
547

548
549
550
551
552
553

554
555
556
557
558

559
560
561
562
563
564

565
566
567
568
569
570

571
572
573
574
575
576

577
578
579
580
581

582
583
584
585
586
587

588
589
590
591
592
593

REFERENCES

Dyah Adila, Changho Shin, Linrong Cai, and Frederic Sala. Zero-shot robustification of zero-shot

models. In The Twelfth International Conference on Learning Representations, 2024.

Bang An, Sicheng Zhu, Michael-Andrei Panaitescu-Liess, Chaithanya Kumar Mummadi, and
Furong Huang. More context, less distraction: Zero-shot visual classification by inferring and
conditioning on contextual attributes. In The Twelfth International Conference on Learning Rep-
resentations, 2024.

Martin Arjovsky, L´eon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization.

arXiv preprint arXiv:1907.02893, 2019.

Hila Chefer, Shir Gur, and Lior Wolf. Generic attention-model explainability for interpreting bi-
modal and encoder-decoder transformers. In Proceedings of the IEEE/CVF International Confer-
ence on Computer Vision (ICCV), pp. 397–406, October 2021.

Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gor-
don, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for
contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer
Vision and Pattern Recognition, pp. 2818–2829, 2023.

Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee. Functional map of the world.
In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6172–
6180, 2018.

Ching-Yao Chuang, Varun Jampani, Yuanzhen Li, Antonio Torralba, and Stefanie Jegelka. Debias-

ing vision-language models via biased prompts. arXiv preprint arXiv:2302.00070, 2023.

Noel Codella, Veronica Rotemberg, Philipp Tschandl, M Emre Celebi, Stephen Dusza, David Gut-
man, Brian Helba, Aadi Kalloo, Konstantinos Liopyris, Michael Marchetti, et al. Skin lesion
analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging
collaboration (isic). arXiv preprint arXiv:1902.03368, 2019.

Joseph Paul Cohen, Paul Morrison, Lan Dao, Karsten Roth, Tim Duong, Marzyeh Ghassem, et al.
Covid-19 image data collection: Prospective predictions are the future. Machine Learning for
Biomedical Imaging, 1(December 2020 issue):1–38, 2020.

Sepehr Dehdashtian, Lan Wang, and Vishnu Naresh Boddeti. Fairerclip: Debiasing clip’s zero-shot

predictions using functions in rkhss. arXiv preprint arXiv:2403.15593, 2024.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas
Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An
image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint
arXiv:2010.11929, 2020.

Sabit Ekin. Prompt engineering for chatgpt: a quick guide to techniques, tips, and best practices.

Authorea Preprints, 2023.

Christiane Fellbaum. Wordnet: An electronic lexical database. MIT Press google schola, 2:678–686,

1998.

Yunhao Ge, Jie Ren, Andrew Gallagher, Yuxiao Wang, Ming-Hsuan Yang, Hartwig Adam, Laurent
Itti, Balaji Lakshminarayanan, and Jiaping Zhao. Improving zero-shot generalization and robust-
ness of multi-modal models. In Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition, pp. 11093–11101, 2023.

Sachin Goyal, Ananya Kumar, Sankalp Garg, Zico Kolter, and Aditi Raghunathan. Finetune like
you pretrain: Improved finetuning of zero-shot vision models. In Proceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition, pp. 19338–19347, 2023.

Ziyu Guo, Renrui Zhang, Longtian Qiu, Xianzheng Ma, Xupeng Miao, Xuming He, and Bin Cui.
Calip: Zero-shot enhancement of clip with parameter-free attention. In Proceedings of the AAAI
Conference on Artificial Intelligence, volume 37, pp. 746–754, 2023.

11

Under review as a conference paper at ICLR 2025

594
595

596
597
598
599
600
601

602
603
604
605
606
607

608
609
610
611
612

613
614
615
616
617
618

619
620
621
622
623
624

625
626
627
628
629
630

631
632
633
634
635

636
637
638
639
640
641

642
643
644
645
646
647

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog-
nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
770–778, 2016.

Badr Youbi Idrissi, Martin Arjovsky, Mohammad Pezeshki, and David Lopez-Paz. Simple data
In Conference on Causal Learning and

balancing achieves competitive worst-group-accuracy.
Reasoning, pp. 336–351. PMLR, 2022.

Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori,
Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali
Farhadi, and Ludwig Schmidt. Openclip, July 2021. URL https://doi.org/10.5281/
zenodo.5143773. If you use this software, please cite it as below.

Pavel Izmailov, Polina Kirichenko, Nate Gruver, and Andrew G Wilson. On feature learning in
the presence of spurious correlations. Advances in Neural Information Processing Systems, 35:
38516–38532, 2022.

Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan
Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning
with noisy text supervision. In International conference on machine learning, pp. 4904–4916.
PMLR, 2021.

Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re-training is sufficient
for robustness to spurious correlations. In The Eleventh International Conference on Learning
Representations, 2022.

Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsub-
ramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne
David, Ian Stavness, Wei Guo, Berton A. Earnshaw, Imran S. Haque, Sara Beery, Jure Leskovec,
Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. WILDS: A
benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning
(ICML), 2021.

David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai
Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrapo-
lation (rex). In International Conference on Machine Learning, pp. 5815–5826. PMLR, 2021.

Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa,
Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training
group information. In International Conference on Machine Learning, pp. 6781–6792. PMLR,
2021.

Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild.

In Proceedings of International Conference on Computer Vision (ICCV), December 2015.

Meike Nauta, Ricky Walsh, Adam Dubowski, and Christin Seifert. Uncovering and correcting
shortcut learning in machine learning models for skin cancer diagnosis. Diagnostics, 12(1):40,
2021.

OpenAI. Chatgpt. https://www.openai.com/chatgpt, 2023. Accessed: 2024-05-12.

Yijiang Pang, Hoang Bao, and Jiayu Zhou. Cross-modality debiasing: using language to mitigate

sub-population shifts in imaging. arXiv preprint arXiv:2403.07888, 2024.

Suzanne Petryk, Lisa Dunlap, Keyan Nasseri, Joseph Gonzalez, Trevor Darrell, and Anna Rohrbach.
On guiding visual attention with language specification. In Proceedings of the IEEE/CVF Con-
ference on Computer Vision and Pattern Recognition, pp. 18092–18102, 2022.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal,
Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual
models from natural language supervision. In International conference on machine learning, pp.
8748–8763. PMLR, 2021.

12

Under review as a conference paper at ICLR 2025

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-
resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Con-
ference on Computer Vision and Pattern Recognition (CVPR), pp. 10684–10695, June 2022.

Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust
neural networks for group shifts: On the importance of regularization for worst-case generaliza-
tion. arXiv preprint arXiv:1911.08731, 2019.

Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. An investigation of why
overparameterization exacerbates spurious correlations. In International Conference on Machine
Learning, pp. 8346–8356. PMLR, 2020.

Matthew Trager, Pramuditha Perera, Luca Zancato, Alessandro Achille, Parminder Bhatia, and Ste-
fano Soatto. Linear spaces of meanings: compositional structures in vision-language models. In
Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15395–15404,
2023.

Yipei Wang and Xiaoqian Wang. On the effect of key factors in spurious correlation: A theoretical
perspective. In International Conference on Artificial Intelligence and Statistics, pp. 3745–3753.
PMLR, 2024.

Zhengbo Wang, Jian Liang, Ran He, Nan Xu, Zilei Wang, and Tieniu Tan. Improving zero-shot
generalization for clip with synthesized prompts. In Proceedings of the IEEE/CVF International
Conference on Computer Vision, pp. 3032–3042, 2023.

WorldSEnder.
Validated.
URL:https://stats.stackexchange.com/q/481387 (version: 2020-08-04).

Cross
https://stats.stackexchange.com/q/481387.

random variable.

transformation

gaussian

Linear

URL

of

Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs,
Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust
fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF conference on computer vision
and pattern recognition, pp. 7959–7971, 2022.

Shirley Wu, Mert Yuksekgonul, Linjun Zhang, and James Zou. Discover and cure: Concept-aware
mitigation of spurious correlation. In International Conference on Machine Learning, pp. 37765–
37786. PMLR, 2023.

Ziwei Xu, Sanjay Jain, and Mohan Kankanhalli. Hallucination is inevitable: An innate limitation of

large language models. arXiv preprint arXiv:2401.11817, 2024.

Siyuan Yan, Zhen Yu, Xuelin Zhang, Dwarikanath Mahapatra, Shekhar S. Chandra, Monika Janda,
Peter Soyer, and Zongyuan Ge. Towards trustable skin cancer diagnosis via rewriting model’s de-
cision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
(CVPR), pp. 11568–11577, June 2023.

Yu Yang, Besmira Nushi, Hamid Palangi, and Baharan Mirzasoleiman. Mitigating spurious cor-
In International Conference on Machine

relations in multi-modal models during fine-tuning.
Learning, pp. 39365–39379. PMLR, 2023.

Huaxiu Yao, Yu Wang, Sai Li, Linjun Zhang, Weixin Liang, James Zou, and Chelsea Finn. Im-
proving out-of-distribution robustness via selective augmentation. In International Conference
on Machine Learning, pp. 25407–25437. PMLR, 2022.

Michael Zhang and Christopher R´e. Contrastive adapters for foundation model group robustness.

Advances in Neural Information Processing Systems, 35:21682–21697, 2022.

Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Pre-
ston, Rajesh Rao, Mu Wei, Naveen Valluri, et al. Biomedclip: a multimodal biomedical
foundation model pretrained from fifteen million scientific image-text pairs. arXiv preprint
arXiv:2303.00915, 2023a.

13

648
649

650
651
652
653
654
655

656
657
658
659
660
661

662
663
664
665
666

667
668
669
670
671
672

673
674
675
676
677
678

679
680
681
682
683
684

685
686
687
688
689

690
691
692
693
694
695

696
697
698
699
700
701

Under review as a conference paper at ICLR 2025

Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao,
Yu Zhang, Yulong Chen, et al. Siren’s song in the ai ocean: a survey on hallucination in large
language models. arXiv preprint arXiv:2309.01219, 2023b.

Guangtao Zheng, Wenqian Ye, and Aidong Zhang. Learning robust classifiers with self-guided

spurious correlation mitigation. arXiv preprint arXiv:2405.03649, 2024.

14

702
703

704
705
706
707
708
709

710
711
712
713
714
715

716
717
718
719
720

721
722
723
724
725
726

727
728
729
730
731
732

733
734
735
736
737
738

739
740
741
742
743

744
745
746
747
748
749

750
751
752
753
754
755

Under review as a conference paper at ICLR 2025

A PROOF OF LEMMA 1

Lemma 1 Under the above data model, the group-wise accuracy can be derived as


A(hgy,a, w; y) =

(12)

erfc(−

(cid:113)

), if y = 1

π
2




π
2

erf(−

(cid:113)

w⊤µgy,a
2w⊤Σgy,aw
w⊤µgy,a
2w⊤Σgy,aw

), if y = −1

where µgy,a and Σgy,a represent the mean and covariance matrix of the image embedding hgy,a.
Denote the linear classifier as w ∈ Rd. To simplify the notation, we drop the subscript of gy,a. The
hyperplane is defined as two half-spaces:

Ω+ = {h|w⊤h > 0}
Ω− = {h|w⊤h ≤ 0}

(13)

The probability density function can be written as:

fH(h; µ, Σ) =

1
√
(2π)d/2

detΣ

exp(−

1
2

(h − µ)⊤Σ−1(h − µ))

(14)

We first consider y = 1. For computing the group accuracy, we integrate fH(h; µ, Σ) over the
region of Ω+. In the following proof, we omit the input of A(·) for simplicity:

(cid:90)

A =

Ω+

fH(h; µ, Σ)dh

Transform h to reduce the mean term, we define h′ = h − µ, Ω1 = {h′|w⊤h′ + w⊤µ > 0}
(cid:90)

A =

1
√
(2π)d/2

detΣ

Ω1

1
2

exp(−

h′⊤Σ−1h′)dh′

(15)

(16)

Σ is a positive definite matrix, we have Σ = Q⊤Σ′Q, where Q is an orthogonal matrix, and Σ′ is a
diagonal matrix. We solve Σ−1 = Q⊤Σ′−1Q.

A =

1
√
(2π)d/2

(cid:90)

detΣ

Ω1

exp(−

1
2

h′⊤Q⊤Σ′−1Qh′)dh′,

(17)

Denote h′′ = Qh′, Ω2 = {h′′ : w⊤Q⊤h′′ + w⊤µ > 0}, then Equation 17 becomes
(cid:90)

exp(−

h′′⊤Σ′−1h′′)|detQ|dh′′,

A =

=

1
√
(2π)d/2
1
√
(2π)d/2

detΣ

Ω2

(cid:90)

detΣ

Ω2

1
2
1
2

h′′⊤Σ′−1h′′)dh′′,

(18)

exp(−

Eliminate the covariance term by defining h′′′ =
0}.

√

Σ′−1h′′, Ω3 = {h′′′ : w⊤Q⊤

√

Σ′h′′′ +w⊤µ >

Then Equation 18 becomes:

A =

√
|
(2π)d/2

detΣ′|
√

detΣ
1
(2π)d/2

=

(cid:90)

Ω3

(cid:90)

Ω3

exp(−

exp(−

1
2
1
2

h′′′⊤h′′′)dh′′′,

h′′′⊤h′′′)dh′′′,

(19)

The space Ω3 = {h′′′ : w′⊤h′′′ + w⊤µ > 0}, where w′ =

√

Σ′Qw.

15

756
757

758
759
760
761
762
763

764
765
766
767
768
769

770
771
772
773
774

775
776
777
778
779
780

781
782
783
784
785
786

787
788
789
790
791
792

793
794
795
796
797

798
799
800
801
802
803

804
805
806
807
808
809

Under review as a conference paper at ICLR 2025

Define an orthogonal matrix U s.t. Uw′ = ||w′||e. Define h′′′′ = Uh′′′, Ω4 = {h′′′′
||w′||e⊤h′′′′ + w⊤µ > 0}. ||w′|| =
Σ′Qw)⊤(

w⊤Σw. We have

Σ′Qw) =

√
(

(cid:113)

√

√

:

A(hgy,a, w; y) =

1
√
2π

(cid:90) ∞

− w⊤ µ√

w⊤Σw

exp(−

1
2

h2)dh

=

1
2

erfc(−

√

w⊤µ
√

w⊤Σw

2

), if y = 1.

Similarly, for y = −1, consider integration over the region of Ω−:

A(hgy,a, w; y) =

1
√
2π

(cid:90) − w⊤ µ√
w⊤Σw

−∞

exp(−

1
2

h2)dh

=

π
2

erf(−

√

w⊤µ
√

w⊤Σw

2

) +

1
2

, if y = −1.

(20)

(21)

Thus prove the statement.

B PROOF OF THEOREM 1

Theorem 1 Given the objective function and the data model, the maximizer of the objective is
obtained by

where P ∈ Rd×d is an elementary matrix, P =

va = E[−Pha]

1
0
...
0

0
0
...
0





· · ·
· · ·
. . .
· · ·


0
0

.
...


0

We rewrite the objective function to ensure the completeness of the proof.

LAcc(va; hgy,a, w) = max
va

(cid:88)

gy,a∈G

Agy,a(hgy,a + va, w; y)

To maximize the objective function, the stationary point can be computed by ∇vaLAcc = 0

∇vaLAcc =

(cid:88)

gy,a∈G

∇va A(hgy,a + va) = 0.

With Lemma 1, we have
(cid:32)

∇va LAcc = ∇va

π
2

erfc(−

w⊤(µg1,a + va)
2w⊤Σw

√

) +

π
2

erf(−

w⊤(µg−1,a + va)
2w⊤Σw

√

(cid:33)
)

= 0

(25)

Decompose Equation 25 based on a, we first compute v1:
(cid:32)

∇v1

erfc(−

π
2

w⊤(µg1,1 + v1)
2w⊤Σw

√

) +

π
2

erf(−

w⊤(µg−1,1 + v1)
2w⊤Σw

√

)

(cid:33)

=

2w
w⊤Σw

[exp(−(

w⊤(µg1,1 + v1)
2w⊤Σw

√

)2) − exp(−(

w⊤(µg−1,1 + v1)
2w⊤Σw

√

)2)] = 0

It can be solved by

v1 = −

1
2

(cid:88)

µgy,1

y∈{−1,1}

16

(26)

(27)

(22)

(23)

(24)

810
811

812
813
814
815
816
817

818
819
820
821
822
823

824
825
826
827
828

829
830
831
832
833
834

835
836
837
838
839
840

841
842
843
844
845
846

847
848
849
850
851

852
853
854
855
856
857

858
859
860
861
862
863

(cid:33)
)

√

erf(−

w⊤(µg−1,−1 + v−1)
2w⊤Σw
w⊤(µg−1,−1 + v−1)
2w⊤Σw

√

)2)] = 0

Under review as a conference paper at ICLR 2025

Then, compute v−1,

∇v−1

(cid:32)

π
2

erfc(−

w⊤(µg1,−1 + v−1)
2w⊤Σw

√

) +

π
2

=

2w
w⊤Σw

[exp(−(

w⊤(µg1,−1 + v−1)
2w⊤Σw

√

)2) − exp(−(

and similarly,

v−1 = −

1
2

(cid:88)

µgy,−1

y∈{1,−1}

Substitute the data assumption in Equation 27 and 29, we have

va = [−a, 0, ..., 0]⊤

We rewrite Equation 30 into a matrix product form:

va = −PE[h] = −E[Ph],

where P =







1 0
0 0
...
...
0
0

· · ·
· · ·
. . .
· · ·


0
0

...


0

. Hence prove the statement.

C DERIVATION OF EQUATIONS 10 AND 11

(28)

(29)

(30)

(31)

Modeling ROBOSHOT. ROBOSHOT is a method that linearly projects the image embedding
onto the hyperplane associated with spurious features. Denote the spurious hyperplane as follows:

The projected point can be written as:

w⊤

spux = 0

xproj = x −

w⊤
spux
||wspu||2 wspu

(32)

(33)

Based on the spurious modeling 3.2, h follows a Gaussian mixture model. According to the
relationship defined in Equation 33, each component in the Gaussian mixture model xproj ∼
N (µproj, Σproj) WorldSEnder, where

µproj = µ −

w⊤
spuµ
||wspu||2 wspu,

Σproj = BΣB⊤,

(34)

where B = I −

wspuw⊤
||wspu||2 , µ = E[x]. With Lemma 1, the analytical expression for ROBOSHOT is:

spu






1
2

1
2

AROBOSHOT (h, w, wspu; y) =

where B = I −

wspuw⊤
||wspu||2 .

spu

w⊤(µ −
√

erfc(−

spuµ

w⊤
||wspu||2 wspu)

w⊤(µ −
√

erf(−

2w⊤BΣB⊤w
spuµ

w⊤
||wspu||2 wspu)

2w⊤BΣB⊤w

), if y = 1

(35)

) +

1
2

, if y = −1

17

864
865

866
867
868
869
870
871

872
873
874
875
876
877

878
879
880
881
882

883
884
885
886
887
888

889
890
891
892
893
894

895
896
897
898
899
900

901
902
903
904
905

906
907
908
909
910
911

912
913
914
915
916
917

Under review as a conference paper at ICLR 2025

Modeling TIE. TIE is a method that translates the image embedding along the negative direction
of the spurious vectors. With Lemma 1 and equation 8, the analytical expression for TIE is

AT IE(h, w, wspu; y) =





1
2

1
2

erfc(−

w⊤(µ − w⊤
√

spuµwspu)

), if y = 1

2w⊤Σw

erf(−

w⊤(µ − w⊤
√

spuµwspu)

2w⊤Σw

) +

1
2

, if y = −1

(36)

Next, plug the spurious feature classifier wspu = [1, α, 0] and the label classifier w = [1, β, 0], and
spurious data model in equation 35 and equation 36, we have

AROBOSHOT (α, β; y) =





1
2

1
2

erfc(−

erf(−

α2 − (1 + β)α + β
(1 + α2)(cid:112)2(β2α2 + (α − β(1 − α2))2)
α2 − (β − 1)α − β
(1 + α2)(cid:112)2(β2α2 + (α − β(1 − α2))2)

) +

), if y = 1

1
2

, if y = −1,

(37)

AT IE(α, β; y) =





1
2

1
2

erf(−

β(1 − α
1+α2 )
(cid:112)2(1 + β2)
β(1 + α
1+α2 )
(cid:112)2(1 + β2)

erfc(−

), if y = 1

1
2

, if y = −1,

(38)

) +

The worst group accuracy takes the min value in equation 37 and equation 38.

C.1 EXPERIMENT VALIDATION

Building on the theoretical analysis in Section 3.3, we further experimentally investigate the impact
of various spurious classifiers on the worst group accuracy of TIE and ROBOSHOT. We generate 6
synonymous spurious text prompts using GPT 4 (OpenAI, 2023) for land features and 6 for water
features, shown in Table 6. We test individual spurious text prompts, yielding 36 combinations (6
from water features, 6 from land features). The results are presented in Figure 6. Furthermore, we
examine all possible combinations of two text prompts within the same spurious feature to expand
the search range of spurious prompts, resulting in 225 combinations. These results are shown in
Figure 7.

Table 6: Spurious Prompt used in experiments comparing ROBOSHOT and TIE.

Spurious Template
“A photo with a/an {a} background”

Land Attributes

Water Attributes

{land,
hill,
field,
desert, forest, moun-
tain}

{water, ocean, river,
lake, sea, pond}

18

918
919

920
921
922
923
924
925

926
927
928
929
930
931

932
933
934
935
936

937
938
939
940
941
942

943
944
945
946
947
948

949
950
951
952
953
954

955
956
957
958
959

960
961
962
963
964
965

966
967
968
969
970
971

Under review as a conference paper at ICLR 2025

Figure 6: Experimental comparison between ROBOSHOT and TIE across different spurious text
prompts, using a single spurious text prompt for each test.

Figure 7: Experimental comparison between ROBOSHOT and TIE on different spurious text
prompts, using multiple spurious text prompts for each test.

From Figure 6 and 7, we observe a significant performance gap between TIE and ROBOSHOT.
This suggests that TIE is more robust, and less dependent on the accuracy of spurious text prompts
compared to ROBOSHOT.

D ALGORITHM FOR TIE*

Algorithm 1 TIE*
Input: Input x, Image encoder ϕI (·), Text encoder ϕT (·), Spurious text prompts Tspu, Target text
prompts T .
Output: Predicted label ˆy.
1: for tspu ∈ Tspu do
va = ϕT (tspu)
2:
va = va
3:
||va||
4: end for
5: ˆa = arg maxa∈A < ϕI (x), ϕT (ta) >
6: hˆa = ϕI (x; ˆa)
7: λˆa = E[(h⊤
ˆa vˆa)]
8: hˆa ← hˆa − λˆavˆa
9: ˆy = arg maxy∈Y < hˆa, ϕT (ty) >
10: return ˆy

▷ Psuedo labeling on spurious feature
▷ Image embedding
▷ Estimate the optimal scale coefficient
▷ Translate image embedding
▷ Zero shot classfication

▷ Computing the spurious vector
▷ Normalize

972
973

974
975
976
977
978
979

980
981
982
983
984
985

986
987
988
989
990

991
992
993
994
995
996

997
998
999
1000
1001
1002

1003
1004
1005
1006
1007
1008

1009
1010
1011
1012
1013

1014
1015
1016
1017
1018
1019

1020
1021
1022
1023
1024
1025

19

Under review as a conference paper at ICLR 2025

1026
1027

1028
1029
1030
1031
1032
1033

1034
1035
1036
1037
1038
1039

1040
1041
1042
1043
1044

1045
1046
1047
1048
1049
1050

1051
1052
1053
1054
1055
1056

1057
1058
1059
1060
1061
1062

1063
1064
1065
1066
1067

1068
1069
1070
1071
1072
1073

1074
1075
1076
1077
1078
1079

E DATASET

We evaluate our method and all comparison methods on the following datasets:

• Waterbirds (Koh et al., 2021; Sagawa et al., 2019): The primary task of the Waterbirds
dataset is to classify bird types, specifically, y = {Landbird, Waterbird}. The spurious
confounder in this dataset is the background, a = {Land background, Water background
}. It includes four groups: {Landbird with a Land background, Landbird with a Water
background, Waterbird with a Land background, Waterbird with a Water background}.
• CelebA (Liu et al., 2015): The CelebA dataset comprises over 200K celebrity faces. Fol-
lowing the protocol by (Sagawa et al., 2019), the task is to identify hair color with target
labels y = {dark hair, blonde hair}. The spurious correlation label is gender, a = {female,
male}. This dataset is segmented into four groups: {a female with dark hair, a female with
blonde hair, a male with dark hair, a male with blonde hair}.

• ISIC (Codella et al., 2019): The ISIC dataset is utilized for skin cancer diagnosis. Follow-
ing the task from (Wu et al., 2023), the task is to predict the type of skin cancer, denoted
as y = {Benign, Malignant}. The spurious correlation feature in this dataset is a = {with
color patch, without color patch}. It encompasses three groups: {Benign cancer with a
color patch, Benign cancer without a color patch, Malignant cancer without a color patch}.
• COVID-19 (Cohen et al., 2020): The COVID-19 dataset is used to diagnose from X-ray
images, with the classification task defined as y = {no Pneumonia, pneumonia}. The
spurious confounder in this dataset is gender, a = {male, female}.
It consists of four
groups: {a male with pneumonia, a male without pneumonia, a female with pneumonia, a
female without pneumonia}.

• FMOW (Christie et al., 2018): The Functional Map of the World (FMOW) is a large-
scale satellite image dataset comprising 62 classes. We follow the protocol outlined in (Wu
et al., 2023; Izmailov et al., 2022) to define groups based on geographical regions: Africa,
the Americas, Oceania, Asia, and Europe.

F BASELINES

We compare TIE against several state-of-the-art methods for zero-shot classification.

• Group Prompt: Group Prompt is a method that includes spurious correlation labels in text
prompts. For example, in the waterbirds dataset, the text prompts for Group Prompt specify
the background along with the bird type, [a photo of a landbird with land
background, a photo of a landbird with a water background,
a photo of a waterbird with a land background, a photo of a
waterbird with a water background].

• Ideal words (Trager et al., 2023): The ideal prompt is to start by adding prompts related to
target labels before integrating those associated with spurious correlation attributes. Sub-
sequently, the ideal method averages across all the spurious correlation prompts.

• Orth-Cali (Chuang et al., 2023): The Orth-Cali method is designed to debias text prompts
by making the text embeddings invariant to spurious features. This approach introduces a
projection matrix that projects the text into the null space defined by the span of spurious
text prompts.
It then employs regularization to ensure that these projected prompts are
closely mapped within the text embedding space.

• Perception CLIP (An et al., 2024): Perception CLIP is a method inspired by empirical
findings that suggest that including contextual attributes in text prompts enhances zero-shot
classification performance and mitigates the effects of spurious correlations. To improve
the group robustness, Perception CLIP incorporates information about spurious features.
• ROBOSHOT (Adila et al., 2024): Roboshot is a method that utilizes LLMs to identify
spurious insights. It then removes these spurious features from the image embeddings using
the Gram-Schmidt process, which projects the image embeddings onto a space orthogonal
to that of the spurious insights. Subsequently, Roboshot enhances the image embeddings
by projecting them along vectors representing helpful insights.

20

Under review as a conference paper at ICLR 2025

1080
1081

1082
1083
1084
1085
1086
1087

1088
1089
1090
1091
1092
1093

1094
1095
1096
1097
1098

1099
1100
1101
1102
1103
1104

1105
1106
1107
1108
1109
1110

1111
1112
1113
1114
1115
1116

1117
1118
1119
1120
1121

1122
1123
1124
1125
1126
1127

1128
1129
1130
1131
1132
1133

G IMPLEMENTATION

We conducted all experiments on an Nvidia RTX 3090 GPU with 24 GB of memory, using frozen
CLIP models across various datasets. Specifically, for the Waterbirds and CelebA datasets, the vi-
sion encoder backbones included ViT-B-32 (Dosovitskiy et al., 2020), ViT-L-14 (Dosovitskiy et al.,
2020), and ResNet 50 (He et al., 2016). Model construction and pre-trained weights are sourced
from Open CLIP (Ilharco et al., 2021).

For specialized datasets, including ISIC and COVID-19, we employed the Biomed CLIP backbone
(Zhang et al., 2023a), acknowledging that the training set from general CLIP significantly diverges
from the biomedical context, leading to substantial shifts in test performance. With ViT-L-32, we
observed 0 % worst-group accuracy; hence, we excluded results using the general backbone for
these specialized datasets.

As no training was conducted for all methods, the results are deterministic.

To facilitate the reproduction of our results, we have detailed both the label prompts and spurious
prompts in Table 7. Note that the nature of CLIP is sensitive to prompts; our spurious prompts
are created through simple adaptations of the label prompts. We incorporate our label prompts and
spurious prompts in all comparison methods except for vanilla zero-shot to ensure a fair comparison.

Table 7: Prompts details

Dataset

Waterbirds

Label prompts
[a photo of a landbird, a
photo of a waterbird]

CelebA

ISIC

COVID-19

[a photo of a celebrity
with dark hair, a photo
of a celebrity with blonde
hair]
[This is a benign lesion,
This is a malignant
lesion]
[An X-ray image of a chest
without Pneumonia, An
X-ray image of a chest
with Pneumonia]

Spurious prompts
[a photo with a water
background, a photo with a
land background]
[A photo of a female, A
photo of a male]

[There exists no color
patch, There exists a
color patch]
[An X-ray image from a
female, An X-ray image
from a male]

H ABLATION STUDY

H.1 DIFFERENT SPURIOUS TEXT PROMPT TEMPLATES

Beyond the textual description of spurious features, the format of spurious text prompt templates
also impacts the performance. To further validate the effectiveness of all methods, we conducted
experiments using various text templates, including ‘{spurious feature label}’ and ‘A photo with a
spurious feature, {spurious feature label}, in the waterbirds dataset. The results are presented in
Table 9.

H.2 MORE BACKBONE RESULTS.

Our paper focuses on CLIP as it serves as a foundational model widely applied across various do-
mains, like in stable diffusion (Rombach et al., 2022). Beyond the CLIP family models, we have
expanded our experiments to incorporate various backbone models. We utilize ALIGN (Jia et al.,
2021) backbones on the Waterbirds dataset, with results shown in Table 10.

From Table 9 and 10, we observe that TIE demonstrates robust performance across various spurious
prompt templates and different backbones, indicating significant potential for real-world applica-
tions.

21

Under review as a conference paper at ICLR 2025

Spurious
Template

Over {g}

Class-
Template

A
satellite
image of
a/an {y}.

Group g

{Europe,
Asia,
Americas,
Africa,
Oceania}

Table 8: FMOW Prompt details

Class y

{airport, airport hangar, airport
termi-
nal, amusement park, aquaculture, ar-
chaeological site, barn, border check-
point, burial site, car dealership, construc-
tion site, crop field, dam, debris or rub-
ble, educational institution, electric sub-
station, factory or powerplant, fire station,
flooded road, fountain, gas station, golf
course, ground transportation station, heli-
pad, hospital, impoverished settlement, in-
terchange, lake or pond, lighthouse, mili-
tary facility, multi-unit residential, nuclear-
powerplant, office building, oil or gas fa-
cility, park, parking lot or garage, place of
worship, police station, port, prison, race
track, railway bridge, recreational facility,
road bridge, runway, shipyard, shopping
mall, single-unit residential, smokestack,
solar farm, space facility, stadium, storage
tank, surface mine, swimming pool, toll
booth, tower, tunnel opening, waste dis-
posal, water treatment facility, wind farm,
zoo}

Table 9: Zero-shot classification results on the Waterbirds dataset with different spurious prompt
templates. T1: {Spurious feature label}, T2: A photo with a spurious feature, {Spurious feature
label}. (CLIP ViT-B/32)

T1 Spurious Template

T2 Spurious Template

ZS
Group Prompt
Ideal words
Orth-Cali
Perception CLIP
ROBOSHOT
TIE
TIE*

WG ↑ Avg ↑ Gap ↓
27.11
68.48
41.37
23.33
66.79
43.46
78.87
16.88
61.99
9.66
64.08
73.74
38.17
61.54
23.37
24.68
69.03
44.35
9.07
80.11
71.04
18.86
75.00
56.14

WG ↑ Avg ↑ Gap ↓
27.11
68.48
41.37
23.33
66.79
43.46
19.38
79.82
60.44
9.44
76.58
67.14
27.17
73.37
46.20
23.68
69.67
45.99
82.02
69.63
12.39
12.24
79.84
67.60

Table 10: Zero Shot classification results on the Waterbirds dataset with the ALIGN backbone

ZS
Group Prompt
Ideal words
Orth-Cali
Perception CLIP
ROBOSHOT
TIE
TIE*

WG ↑ Avg ↑ Gap ↓

47.50
5.81
51.71
28.35
31.60
41.02
56.07
52.49

69.83
72.55
67.17
58.73
54.39
50.95
69.54
64.27

22.33
66.74
15.46
30.38
22.79
9.93
13.47
11.78

22

1134
1135

1136
1137
1138
1139
1140
1141

1142
1143
1144
1145
1146
1147

1148
1149
1150
1151
1152

1153
1154
1155
1156
1157
1158

1159
1160
1161
1162
1163
1164

1165
1166
1167
1168
1169
1170

1171
1172
1173
1174
1175

1176
1177
1178
1179
1180
1181

1182
1183
1184
1185
1186
1187

Under review as a conference paper at ICLR 2025

I DISCUSSION ON TEXT PROMPTS

The effectiveness of VLMs depends on the quality of text prompts. The guidelines for selecting
text prompts represent a critical area for deeper exploration. To address this, we show our insights
through experiments designed to identify an effective and generalizable approach for creating opti-
mal text prompts in practice.

We investigate this issue by decomposing a text prompt into a template and an object.

• T1: “A photo with [Object]”

• T2: “A photo with a spurious feature, [Object]”

• T3: “[Object]”

For the object, Ge et al. (2023) shows that labels exhibit a hierarchical structure in “WordNet” Fell-
baum (1998). For example, the hierarchical progression of the word ‘strawberry’ includes ‘berry’,
‘edible fruit’, ‘food’, each level becoming more general Ge et al. (2023). In our experiments, we
test three labeling strategies: using the level directly above to represent a more generalized category,
the spurious feature itself, and an average of the top five most specific terms at the bottom of the
hierarchy for greater specificity. We provide details of the object candidates in Table 11. The aim
is to determine the most effective level of generality or specificity for descriptions. We conducted
experiments on the Waterbirds dataset using TIE* (ViT-L14). The results are shown in Table 12.

Table 11: Object candidates

Water background prompts

Land background prompts

O1 (hypernyms)
O2 (self)
O3 (hyponyms)

Fluid
Water
Sea, Lake, River, Stream, Creek

Ground
Land
Arable Land, Farmland, Forest
Land, Grassland, Desert

Table 12: Performance evaluation of CLIP-ViTL14 for TIE*, We have highlighted in bold the
results that surpass the WG in Table 1.

Text prompts WG ↑ Avg ↑ Gap ↓

T1+O1
T1+O2
T1+O3
T2+O1
T2+O2
T3+O3
T3+O1
T3+O2
T3+O3

53.97
61.60
65.26
46.48
63.77
63.19
45.90
60.62
59.56

76.49
78.98
80.20
72.69
80.35
79.06
73.19
78.84
77.91

22.52
17.38
14.94
26.21
16.58
15.87
27.29
18.22
18.35

Insights: We note that using a proper object description is important. We suggest using a specific
description of the spurious feature or their hyponyms, as this can improve the worst group accu-
In contrast, using overly general descriptions such as
racy (WG) in the zero-shot classification.
hypernyms significantly degrades performance. This observation aligns with recommendations for
specificity and clarity in text prompt engineering for language models Ekin (2023).

In terms of templates, we found that giving a portion of contextual information, such as the prefix
“a photo with” or “a photo with a spurious feature,” helps the WG. Templates lacking a prefix
demonstrate poor performance, a finding that aligns with the observations presented in Radford et al.
(2021). For practical purposes in ViT-based CLIP models, we encourage users to adopt templates
that include a prefix, with the object description utilizing the spurious feature itself, balancing ease
of use and performance.

23

1188
1189

1190
1191
1192
1193
1194
1195

1196
1197
1198
1199
1200
1201

1202
1203
1204
1205
1206

1207
1208
1209
1210
1211
1212

1213
1214
1215
1216
1217
1218

1219
1220
1221
1222
1223
1224

1225
1226
1227
1228
1229

1230
1231
1232
1233
1234
1235

1236
1237
1238
1239
1240
1241

Under review as a conference paper at ICLR 2025

J FUTURE DIRECTION DISCUSSION

We introduce TIE to mitigate the effect of spurious correlations, which are vital in prediction tasks.
While our approach demonstrates strong performance, it faces challenges redirecting attention to the
object in the presence of pronounced artifacts (e.g., watermarks) without appropriate text prompts.
Figure 8 illustrates a rare case where the dominant feature is a watermark. To evaluate our method’s
capability in redirecting attention, we provide the following text prompts:

• Text prompt 1 (TP1): A photo with a water background,

• Text prompt 2 (TP2): A photo with a watermark

Figure 8: Attention-based explanations in an image with a strong artificial landmark in the Wa-
terbirds dataset. TP1: A photo with a water background, TP2: A photo with a
watermark.

From Figure 8, we observe that when using TP1, a text prompt representing a common spurious
feature in the dataset, the attention fails to redirect back to the correct core feature (the bird in the
image). Interestingly, when providing a corresponding text prompt (TP2), the attention successfully
shifts from the watermark to the bird. This highlights the potential of our proposed method to address
misclassifications caused by factors beyond spurious correlations, offering a promising direction for
further research.

K FAILURE CASE ANALYSIS FOR TIE*

TIE* is a method free from using any annotations and requires the spurious text prompt for infer-
ence of the spurious label in the dataset. We analyzed TIE* failure cases, which can be broadly
categorized into two scenarios: (1) inaccuracies in the pseudo-spurious labels and (2) images con-
taining artifacts (e.g., watermarks).

For (1): The majority of failures in TIE* occur when zero-shot classification incorrectly assigns a
spurious label. This misassignment causes samples to be translated in the opposite direction, leading
to incorrect classifications. In Section 4.4, we examine the worst-group accuracy in zero-shot clas-
sification and the accuracy of pseudo-spurious labels. Our analysis reveals that the pseudo-spurious
labels assigned by TIE* have a direct impact on the worst-group accuracy in zero-shot classification:
higher accuracy in assigning these labels corresponds to improved worst-group accuracy.

To potentially improve TIE*’s performance, we propose three practical strategies: utilizing group-
robustified spurious text prompts (Section 4.3), employing a small subset of spurious-labeled data
(Section 4.4), and following the guidelines for effective text prompts (Section I) to achieve better
performance.

For (2): we discussed this scenario in Section J. This is a case where the artifact (e.g., a watermark)
becomes the dominant feature. While using TIE or TIE* reduce dependency on spurious features
(such as background information), it cannot eliminate the effect of the artifact. This limitation can
lead to the failure of our algorithm. Interestingly, we found that TIE has the potential to remove
unwanted features when provided with appropriate text prompts. However, the identification of
these incorrect features remains an open area for further investigation.

24

1242
1243

1244
1245
1246
1247
1248
1249

1250
1251
1252
1253
1254
1255

1256
1257
1258
1259
1260

1261
1262
1263
1264
1265
1266

1267
1268
1269
1270
1271
1272

1273
1274
1275
1276
1277
1278

1279
1280
1281
1282
1283

1284
1285
1286
1287
1288
1289

1290
1291
1292
1293
1294
1295

Under review as a conference paper at ICLR 2025

L BROADER IMPACTS

Our work aims to mitigate spurious correlations in VLM models, a crucial endeavor for the machine
learning community. Beyond enhancing group robustness, the positive impacts of our work extend
to domains such as fairness, trustworthiness, and generalization. This is particularly significant when
deploying machine learning algorithms in high-stakes domains.

1296
1297

1298
1299
1300
1301
1302
1303

1304
1305
1306
1307
1308
1309

1310
1311
1312
1313
1314

1315
1316
1317
1318
1319
1320

1321
1322
1323
1324
1325
1326

1327
1328
1329
1330
1331
1332

1333
1334
1335
1336
1337

1338
1339
1340
1341
1342
1343

1344
1345
1346
1347
1348
1349

25

