Improving neural network representations using human similarity
  judgments
Deep neural networks have reached human-level performance on many computer
vision tasks. However, the objectives used to train these networks enforce only
that similar images are embedded at similar locations in the representation
space, and do not directly constrain the global structure of the resulting
space. Here, we explore the impact of supervising this global structure by
linearly aligning it with human similarity judgments. We find that a naive
approach leads to large changes in local representational structure that harm
downstream performance. Thus, we propose a novel method that aligns the global
structure of representations while preserving their local structure. This
global-local transform considerably improves accuracy across a variety of
few-shot learning and anomaly detection tasks. Our results indicate that human
visual representations are globally organized in a way that facilitates
learning from few examples, and incorporating this global structure into neural
network representations improves performance on downstream tasks.
Lukas Muttenthaler, Lorenz Linhardt, Jonas Dippel, Robert A. Vandermeulen, Katherine Hermann, Andrew K. Lampinen, Simon Kornblith
# Improving neural network representations using human similarity judgments

###### Abstract

Deep neural networks have reached human-level performance on many computer vision tasks. However, the objectives used to train these networks enforce only that similar images are embedded at similar locations in the representation space, and do not directly constrain the global structure of the resulting space. Here, we explore the impact of supervising this global structure by linearly aligning it with human similarity judgments. We find that a naive approach leads to large changes in local representational structure that harm downstream performance. Thus, we propose a novel method that aligns the global structure of representations while preserving their local structure. This global-local transform considerably improves accuracy across a variety of few-shot learning and anomaly detection tasks. Our results indicate that human visual representations are globally organized in a way that facilitates learning from few examples, and incorporating this global structure into neural network representations improves performance on downstream tasks.

## 1 Introduction

Representation learning usually involves pretraining a network on a large, diverse dataset using one of a handful of different types of supervision. In computer vision, early successes came from training networks to predict class labels [47; 85; 28]. More recent work has demonstrated the power of contrastive representation learning [9; 29; 70; 8]. Self-supervised contrastive models learn a space where images lie close to other augmentations of the same image [9], whereas image/text contrastive models learn a space where images lie close to embeddings of corresponding text [70; 39; 68].

Although these representation learning strategies are effective, their pretraining objectives do not directly constrain the global structure of the learned representations. To classify ImageNet images
into 1000 classes, networks' penultimate layers must learn representations that allow examples of a given class to be linearly separated from representations of other classes, but the classes themselves could be arranged in any fashion. The objective itself does not directly encourage images of tabby cats to be closer to images of other breeds of cats than to images of raccoons. Similarly, contrastive objectives force related embeddings to be close, and unrelated embeddings to be far, but if a cluster of data points is sufficiently far from all other data points, then the location of this cluster in the embedding space has minimal impact on the value of the loss [62; 35; 9].

Even without explicit global supervision, however, networks implicitly learn to organize high-level concepts somewhat coherently. For example, ImageNet models' representations roughly cluster according to superclasses [19], and the similarity structure of neural network representation spaces is non-trivially similar to similarity structures inferred from brain data or human judgments [40; 92; 66; 80; 49]. The structure that is learned likely reflects a combination of visual similarity between images from related classes and networks' inductive biases. However, there is little reason to believe that this implicitly-learned global structure should be optimal.

Human and machine vision differ in many ways. Some differences relate to sensitivity to distortions and out-of-distribution generalization [21; 22; 25]. For example, ImageNet models are more biased by texture than humans in their decision process [22; 34]. However, interventions that reduce neural networks' sensitivity to image distortions are not enough to make their representations as robust and transferable as those of humans [23]. Experiments inferring humans' object representations from similarity judgments suggest that humans use a complex combination of semantics, texture, shape, and color-related concepts for performing similarity judgments [31; 60].

Various strategies have been proposed to improve _representational alignment_ between neural nets and humans [e.g., 66; 67; 2; 61], to improve robustness against image distortions [e.g., 34; 21], or for obtaining models that make more human-like errors [e.g., 22; 26]. Muttenthaler et al. [61] previously found that a linear transformation learned to maximize alignment on one dataset of human similarity judgments generalized to different datasets of similarity judgments. However, it remains unclear whether better representational alignment can improve networks' generalization on vision tasks.

Here, we study the impact of aligning representations' global structure with human similarity judgments on downstream tasks. These similarity judgments, collected by asking subjects to choose the _odd-one-out_ in a triplet of images [32], provide explicit supervision for relationships among disparate concepts. Although the number of images we use for alignment is three to six orders of magnitude smaller than the number of images in pretraining, we observe significant improvements. Specifically, our contributions are as follows:

Figure 1: Global-local (gLocal) transforms yield a _best-of-both-worlds_ representation space, which improves overall performance. (a) The original representations capture local structure, such as that different trees are similar, but have poor global structure. The gLocal transform preserves local structure, while integrating global information from human knowledge; e.g., unifying superordinate categories, organizing by “animacy”, or connecting semantically-related categories like “food” and “drink”. (b) The gLocal transforms improve both human alignment and downstream task performance compared to original and naively aligned representations for image/text models. We report mean accuracies on anomaly detection and (5-shot) few-shot learning tasks.
* We introduce the _gLocal transform_, a linear transformation that minimizes a combination of a _global alignment loss_, which aligns the representation with human similarity judgments, and a _local contrastive loss_ that maintains the local structure of the original representation space.
* We show that the gLocal transform preserves the local structure of the original space but captures the same global structure as a _naive transform_ that minimizes only the global alignment loss.
* We demonstrate that the gLocal transform substantially increases performance on a variety of few-shot learning and anomaly detection tasks. By contrast, the naive transform impairs performance.
* We compare the human alignment of gLocal and naively transformed representations on four human similarity judgment datasets, finding that the gLocal transform yields only marginally worse alignment than the naive transform.

## 2 Related work

How can we build models which learn representations that support performance on variable downstream tasks? This question has been a core theme of computer vision research [9; 27; 58], but the impact of pretraining on feature representations is complex, and better performance does not necessarily yield more transferable representations [e.g., 44; 45]. For example, some pretraining leads to shortcut learning [23; 50; 1; 57; 22; 34; 3; 91]. Because the relationship between training methods and representations is complicated, it is useful to study how datasets and training shape representations [33]. Standard training objectives do not explicitly constrain the global structure of representations; nevertheless, these objectives yield representations that capture some aspects of the higher-order category structure [e.g., 40] and neural unit activity [e.g., 92; 80] of human and animal representations of the same images. Some models progressively differentiate hierarchical structure over the course of learning [5] in a similar way to how humans learn semantic features [73; 18; 78; 5; 79]. Even so, learned representations still fail to capture important aspects of the structure that humans learn [6]. Human representations capture both perceptual and semantic features [7; 69], including many levels of semantic hierarchy (e.g. higher-level: "animate" [11], superordinate: "mammal", basic: "dog", subordinate: "dashund"), with a bias towards the basic level [74; 55; 38], as well as cross-cutting semantic features [84; 56].

While models may implicitly learn to represent some of this structure, these implicit representations may have shortcomings. For example, Huh et al. [37] suggest that ImageNet models capture only higher-level categories where the sub-categories are visually similar. Similarly, Peterson et al. [66] show that model representations do not natively fully capture the structure of human similarity judgments, though they can be transformed to align better [66; 2]. In some cases, language provides more accurate estimates of human similarity judgments than image representations [54], and image-text models can have more human-aligned representations [61]. How does human alignment affect downstream performance? Sucholutsky & Griffiths [87] show that models which are more human-aligned (but not specifically optimized for alignment) are more robust on few-shot learning tasks. Other work shows benefits of incorporating higher-level category structure [51; 83]. Here, we ask whether transforming model representations to align with human knowledge can improve transfer.

## 3 Methods

**Data.** For measuring the degree of alignment between human and neural network similarity spaces, we use the things dataset, which is a large behavioral dataset of \(4.70\) million unique triplet responses crowdsourced from \(12\),\(340\) human participants for \(1854\) natural object images [32]. Images used for collecting human responses in the triplet odd-one-out task were taken from the things object concept and image database [30], which is a collection of natural object images.

**Odd-one-out accuracy.** The triplet odd-one-out task is a commonly used task in the cognitive sciences to measure human notions of object similarity without biasing a participant into a specific direction [20; 72; 31; 60]. To measure the degree of alignment between human and neural network similarity judgments in the things triplet task, we examine the extent to which the odd-one-out can be identified directly from the similarities between images in models' representation spaces. Given representations \(\mathbf{x}_{1}\), \(\mathbf{x}_{2}\), and \(\mathbf{x}_{3}\) of the three images in a triplet, we first construct a similarity matrix \(\mathbf{S}\in\mathbb{R}^{3\times 3}\) where \(S_{i,j}\coloneqq\mathbf{x}_{i}^{\top}\mathbf{x}_{j}/(\|\mathbf{x}_{i}\|_{2}\|\mathbf{x}_{j}\|_ {2})\), the cosine similarity between a pair of representations.3 We identify the closest pair of images in the triplet as \(\arg\max_{i,j>i}S_{i,j}\) with the
remaining image being the odd-one-out. We define odd-one-out accuracy as the proportion of triplets where the odd-one-out matches the human odd-one-out choice.

**Alignment loss.** Given an image similarity matrix \(\mathbf{S}\) and a triplet \(\{i,j,k\}\) (here, images are indexed by natural numbers), the likelihood of a particular pair, \(\{a,b\}\subset\{i,j,k\}\), being most similar, and hence the remaining image being the odd-one-out, is modeled by the softmax of the object similarities,

\[p(\{a,b\}|\{i,j,k\},\mathbf{S})\coloneqq\exp(S_{a,b})/\left(\exp(S_{i,j})+\exp(S_{ i,k})+\exp(S_{j,k})\right). \tag{1}\]

For \(n\) triplet responses we use the following negative log-likelihood, precisely defined in [60],

\[\mathcal{L}_{\text{global}}(\mathbf{S})\coloneqq-\frac{1}{n}\sum_{s=1}^{n}\log \underbrace{p\left(\{a_{s},b_{s}\}|\{i_{s},j_{s},k_{s}\},\mathbf{S}\right)}_{ \text{odd-one-out prediction}}. \tag{2}\]

Since the triplets in [32] consist of randomly selected images, the concepts that humans use for their similarity judgments in the things triplet odd-one-out task primarily reflect superordinate categories rather than fine-grained object features [31, 60], the above alignment loss can be viewed as a loss function whose objective is to transform the representations into a globally-restructured human similarity space where superordinate categories are emphasized over subordinate categories.

**Naive transform.** We first investigate a linear transformation that naively maximizes alignment between neural network representations and human similarity judgments with \(L_{2}\) regularization. This transformation consists of a square matrix \(\mathbf{W}\) obtained as the solution to

\[\operatorname*{arg\,min}_{\mathbf{W},\mathbf{b}}\ \mathcal{L}_{\text{global}}(\mathbf{S})+ \lambda||\mathbf{W}||_{\text{F}}^{2}, \tag{3}\]

where \(S_{ij}=\left(\mathbf{W}\mathbf{x}_{i}+\mathbf{b}\right)^{\top}\left(\mathbf{W}\mathbf{x}_{j}+\mathbf{ b}\right)\). We call this transformation the _naive transform_ because the regularization term helps prevent overfitting to the training set, but does not encourage the transformed representation space to resemble the original space. Mutenthaler et al. [61] previously investigated a similar transformation. We determine \(\lambda\) via grid-search using \(k\)-fold cross-validation (CV). To obtain a minimally biased estimate of the odd-one-out accuracy of the transform, we partition the \(1854\) objects in things into two disjoint sets, following the procedure of Mutenthaler et al. [61].

**Global transform.** The naive transform does not preserve representational structure that is irrelevant to the odd-one-out task. The global transform instead shrinks \(\mathbf{W}\) toward a scaled identity matrix by penalizing \(\min_{\alpha}\left\|\mathbf{W}-\alpha I\right\|_{\text{F}}^{2}\), thus regularizing the transformed representation toward the original. The global transform solves the following minimization problem,

\[\operatorname*{arg\,min}_{\mathbf{W},\mathbf{b}}\mathcal{L}_{\text{global}}(\mathbf{S})+ \lambda\min_{\alpha}\left\|\mathbf{W}-\alpha I\right\|_{\text{F}}^{2}=\operatorname* {arg\,min}_{\mathbf{W},\mathbf{b}}\mathcal{L}_{\text{global}}(\mathbf{S})+\lambda\left\| \mathbf{W}-\left(\sum_{j=1}^{p}\mathbf{W}_{jj}/p\right)I\right\|_{\text{F}}^{2}. \tag{4}\]

We derive the above equality in Appx. F. Again, we select \(\lambda\) via grid-search with \(k\)-fold CV.

**gLocal transform.** In preliminary experiments, we observed a trade-off between alignment and the transferability of a neural network's human-aligned representation space to downstream tasks. Maximizing alignment appears to slightly worsen downstream task performance, whereas using a large value of \(\lambda\) in Eq. 4 leads to a representation that closely resembles the original (since \(\lim_{\lambda\to\infty}\mathbf{W}=\sigma I\)). Thus, we add an additional regularization term to the objective with the goal of preserving the local structure of the network's original representation space. This loss term can be seen as an additional constraint on the transformation matrix \(\mathbf{W}\).

We call this loss term _local loss_ and the transform that optimizes this full objective the _gLocal transform_, where global and local representations structures are jointly optimized. For this loss function, we embed all images of the ImageNet train and validation sets [17] in a neural network's \(p\)-dimensional penultimate layer space or image encoder space of image/text models. Let \(\mathbf{Y}\in\mathbb{R}^{m\times p}\) be a neural network's feature matrix for all \(m\) images in the ImageNet train set. Let \(\mathbf{S}^{*}\) be the cosine similarity matrix using untransformed representations where \(S^{*}_{ij}=\left(\mathbf{y}_{i}^{\top}\mathbf{y}_{j}\right)/\big{(}||\mathbf{y}_{i}||_{2} ||\mathbf{y}_{j}||_{2}\big{)}\) and let \(\mathbf{S}^{\dagger}\) be the cosine similarity matrix of the transformed representations where

\[S^{\dagger}_{ij}=\left(\left(\mathbf{W}\mathbf{y}_{i}+\mathbf{b}\right)^{\top}\left(\mathbf{W} \mathbf{y}_{j}+\mathbf{b}\right)\right)/\big{(}||\mathbf{W}\mathbf{y}_{i}+\mathbf{b}||_{2}||\mathbf{W} \mathbf{y}_{j}+\mathbf{b}||_{2}\big{)}.\]

Let \(\sigma\) be a softmax function that transforms a similarity matrix into a probability distribution,

\[\sigma(\mathbf{S},\tau)_{ij}\coloneqq\frac{\exp(\mathbf{S}_{ij}/\tau)}{\sum_{k}^{m} \mathbbm{1}_{[k\neq j]}\exp(\mathbf{S}_{ik}/\tau)},\]
where \(\tau\) is a temperature and \(\sigma(\mathbf{S},\tau)_{ij}\in(0,1)\). We can then define the local loss as the following contrastive objective between untransformed and transformed neural network similarity spaces,

\[\mathcal{L}_{\text{local}}(\mathbf{W},\mathbf{b},\tau)\coloneqq-\frac{1}{m^{2}-m}\sum_{i }^{m}\sum_{j}^{m}\mathbbm{1}_{[i\neq j]}\sigma(\mathbf{S}^{*},\tau)_{ij}\log\left[ \sigma(\mathbf{S}^{\dagger},\tau)_{ij}\right]. \tag{5}\]

To avoid distributions that excessively emphasize the self-similarity of objects for small \(\tau\), we exclude elements on the diagonal of the similarity matrices. The final _gLocal transform_ is then,

\[\operatorname*{arg\,min}_{\mathbf{W},\mathbf{b}}\ \underbrace{(1-\alpha)\,\mathcal{L}_{ \text{global}}(\mathbf{W},\mathbf{b})}_{\text{alignment}}+\underbrace{\alpha\mathcal{L }_{\text{local}}(\mathbf{W},\mathbf{b},\tau)}_{\text{locality-preserving}}+\lambda \left\|\mathbf{W}-\left(\sum_{j=1}^{p}\mathbf{W}_{jj}/p\right)I\right\|_{\text{F}}^{2}, \tag{6}\]

where \(\alpha\) is a hyperparameter that balances the trade-off between human alignment and preserving the local structure of a neural network's original representation space. We select values of \(\alpha\) and \(\lambda\) that give the lowest alignment loss via grid search; see details in Appx. A.2.

### Downstream tasks

**Few-shot learning.** In general, few-shot learning (FS) is used to measure the transferability of neural network representations to different downstream tasks [86]. Here, we use FS to investigate whether the gLocal transform, as defined in SS3, can improve downstream task performance and, hence, a network's transferability, compared to the original representation spaces. Specifically, we perform FS with and without applying the gLocal transforms. For all few-shot experiments, we use multinomial logistic regression, which has previously been shown to achieve near-optimal performance when paired with a good representation [88]. The regularization parameter is selected by \(n_{s}\)-fold cross-validation, with \(n_{s}\) being the number of shots per class (more details in Appx. A.3).

**Anomaly detection.** Anomaly detection (AD) is a task where one has a collection of data considered "nominal" and would like to detect if a test sample is different from nominal data. For semantic AD tasks, e.g. nominal images contain a cat, it has been observed that simple AD methods using a pretrained neural network perform best [53; 4; 16]. In this work, we apply \(k\)-nearest neighbor AD to representations from a neural network, a method which has been found to work well [4; 53; 71]. We use the standard one-vs.-rest AD benchmark on classification datasets, where a model is trained using "one" training class as nominal data and performs inference with the full test set with the "rest" classes being anomalous [75].

## 4 Experimental results

In this section, we report experimental results for different FS and AD tasks. In addition, we analyze the effect of the gLocal transforms on both local and global similarity structures and report changes in representational alignment of image/text models for four human similarity judgment datasets. We start this section by introducing the different tasks and datasets and continue with the analyses.

### Experimental setup

**CIFAR-100 coarse** The 100 classes in CIFAR-100 can be grouped into 20 semantically meaningful superclasses. Here, we use these superclasses as targets for which there exist 100 test images each.

**CIFAR-100 shift** simulates a distribution shift between the normal distribution of the training and testing images in CIFAR-100. For each of the 20 superclasses, there exist five subclasses. We use the first three subclasses for training and the last two subclasses for testing.

**Entity-13 and Entity-30** are datasets derived from ImageNet [17; 76]. They have been defined as part of the BREEDS dataset for subpopulation shift [77]. In BREEDS, ImageNet classes are grouped into superclasses based on a modified version of the WordNet hierarchy. Specifically, one starts at the _Entity_ node of that hierarchy and traverses the tree in a breadth-first manner until the desired level of granularity (three steps from the root for Entity-13 and four for Entity 30). The classes residing at that level are considered the new coarse class labels and a fixed number of subclasses are sampled from the trees rooted at these superclass nodes -- twenty for Entity-13 and eight for Entity-30. Through this procedure, Entity-13 results in more coarse-grained labels than Entity-30. The subclasses of each superclass are partitioned into source (training) and target (test) classes, introducing a subpopulation shift. For testing, we use all 50 validation images for each subclass.
### Impact of transforms on global and local structure

Our goal is to use human similarity judgments to correct networks' global representational structure without affecting local representational structure. Here, we study the extent to which our method succeeds at this goal. To quantify distortion of local structure, we first find the nearest neighbor of each ImageNet validation image among the remaining validation 49,999 images in the original representation space. We then measure the proportion of images for which the nearest neighbor in the untransformed representation space is among the closest \(k\) images in the transformed representation space. As shown in Fig. 2, both the gLocal and global transforms generally preserve nearest neighbors, although the gLocal transform is slightly more effective, preserving the closest neighbor of 76.3% of images vs. 73.7% for the global transform. By contrast, the naive, unregularized transform preserves the closest neighbor in only 12.2% of images. For further intuition, we show the neighbors of anchor images in the untransformed, gLocal, and naively transformed representation spaces in Appx. C.

Whereas the local structure of gLocal and global representations closely resembles that of the original representation, the global structure instead more closely resembles the naive transformed representation. We quantify global representational similarity using linear centered kernel alignment (LCKA) [43; 14]. LCKA can be thought of as measuring the similarity between principal components (PCs) of two representations weighted by the amount of variance they explain; see further discussion in Appx. G. It thus primarily reflects similarity between the large PCs that define global representational structure. As shown in Fig. 3 (left), LCKA indicates that the gLocal/global representations are more similar to the naive transformed representation than to the untransformed representation, suggesting that the gLocal, global, and naive transforms induce similar changes in global structure. We further measure LCKA between representations obtained by setting all singular values to zero except for those corresponding to either largest 10 PCs or the remaining 758 PCs. LCKA between representations retaining the largest 10 PCs resembles LCKA between the full representations (Fig. 3 middle). However, when retaining only the remaining PCs, the gLocal/global representations are more similar to the untransformed representation than to the naive-transformed representation (Fig. 3 right).

Figure 4: The gLocal transformed representation captures global structure of the naive transformed representation, as shown by PCA, but local structure of the untransformed representation, as shown by t-SNE with perplexity 10. Visualizations reflect embeddings of 10 images from each of the 260 Entity-13 ImageNet subclasses obtained from CLIP ViT-L and are colored by the superclasses. 2D embeddings are rotated to align with the untransformed representation using orthogonal Procrustes. The t-SNE fitting process is initialized using PCA; we pick the embedding with the lowest loss from 10 runs.

Figure 3: The top principal components (PCs) of gLocal and global representations of ViT-L on the ImageNet validation set resemble those of the naive transformed representation, indicating that they share similar global structure. The remaining PCs more closely match the untransformed representation. **Left:** CKA between full representations. **Middle:** CKA after setting singular values of each representation to zero for all but the largest 10 PCs. **Right:** CKA after setting singular values to zero for the largest 10 PCs, but retaining smaller PCs.

Figure 2: gLocal and global but not naive transforms preserve nearest neighbors in CLIP ViT-L representations. y-axis indicates the percentage of ImageNet validation images for which the closest image in the untransformed space is among the \(k\) closest after transformation.
In Fig. 4, we visualize the effects of different transformations using PCA, which preserves global structure, and t-SNE, which preserves local structure. As described by Van der Maaten & Hinton [89], PCA "focus[es] on keeping the low-dimensional representations of dissimilar datapoints far apart," whereas t-SNE tries to "keep the low-dimensional representations of very similar datapoints close together." In line with the results above, the global structure of the gLocal representation revealed by PCA closely resembles that of the naive transformed representation, whereas the local structure of the gLocal representation revealed by t-SNE closely resembles that of the untransformed representation.

To complement these analyses, in Appendix B we explore what the alignment transforms alter about the global category structure of the model representations. Generally, categories within a single superordinate category move more closely together, while different superordinate categories move apart, but with sensible exceptions -- e.g. food and drink move closer together.

### Few-shot learning

In this section, we examine the impact of the gLocal transform on few-shot classification accuracy on downstream datasets. We consider a standard _fine-grained_ few-shot learning setup on CIFAR-100 [48], SUN397 [90], and the Describable Textures Dataset [DTD, 12], as well as a _coarse-grained_ setup on Entity-[13, 30] of BREEDS [77].

In coarse-grained FS, classes are grouped into semantically meaningful superclasses. This is a more challenging setting than the standard fine-grained scenario, for which there does not exist a superordinate grouping. In fine-grained FS, training examples are uniformly drawn from all (sub-)classes. For coarse-grained FS, we classify according to superclasses rather than fine-grained classes, and choose \(k\) training examples uniformly at random for each superclass. This implies that not every subclass is contained in the train set if the number of training samples is smaller than the number of subclasses. On Entity-[13, 30], superclasses between train and test sets are guaranteed to be disjoint due to a subpopulation shift. To achieve high accuracy, models must consider examples from unseen members of a superclass similar to the few examples it has seen. Task performance is dependent on how well the partial information contained in the few examples of a superclass can be exploited by a model to characterize the entire superclass with its subclasses. Hence, global similarity structure is more crucial than local similarity structure to perform well on this task. We calculate classification accuracy across all available test images of all subclasses, using coarse labels.

We find that transforming the representations via gLocal transforms substantially improves performance over the untransformed representations across both coarse-grained and fine-grained FS tasks for all image/text models considered (Tab. 1). For CLIP models trained on LAION, however, we do not observe improvements for CIFAR-100 and DTD, which are the most fine-grained datasets. For ImageNet models, the gLocal transform improves performance on Entity-{13,30}, but has almost no impact on the performance for the remaining datasets (see Appx. D.1).

### Anomaly detection

Here, we evaluate the performance of representations on \(k\)-nearest neighbor AD with and without using the gLocal transform. AD methods typically return an anomaly score for which a detection threshold has to be chosen. Our method is evaluated using each training class as the nominal class and we report the average AUROC. Following SS3.1, we compute the representations for each normal sample of the training set and then evaluate the model with representations from the test set. We set \(k=5\) for our experiments but found that performance is fairly insensitive to the choice of \(k\) (see Appx. D.2). For measuring the distance between representations, we use cosine similarity.

In Tab. 2, we show that the gLocal transform substantially improves AD performance across all datasets considered in our analyses, for almost every image/text model. However, as in the few-shot setting, we observe no improvements over the untransformed representation space for ImageNet models (see Appx. D.2). In Tab. 3, we further investigate performance on distribution shift datasets,

\begin{table}
\begin{tabular}{l|c c|c c|c c|c c|c c|c c} \hline \multirow{2}{*}{Model \(\backslash\) Transform} & Entity-13 & \multicolumn{2}{c|}{Entity-30} & \multicolumn{2}{c|}{CIFAR100-Course} & \multicolumn{2}{c|}{CIFAR100} & \multicolumn{2}{c}{SUN397} & \multicolumn{2}{c}{DTD} \\  & original & gLocal & original & gLocal & original & gLocal & original & gLocal & original & gLocal & original & gLocal \\ \hline CLIP-RN50 & 63.98 & **67.96** & 57.87 & **59.8** & 44.09 & **47.43** & 38.78 & **39.99** & 57.22 & **58.79** & 56.32 & **56.44** \\ CLIP-ViViT-14 (AVIT) & 65.35 & **71.94** & 66.91 & **69.97** & 66.96 & **66.88** & 72.31 & **73.03** & 69.95 & **71.13** & 64.76 & **64.58** \\ CLIP-ViT-14 (LAION-400M) & 65.32 & **69.02** & 62.78 & **65.99** & 68.98 & **69.58** & **73.59** & 72.98 & 70.25 & **71.08** & **68.92** & 67.74 \\ CLIP-ViT-14 (LAION-28H) & 67.00 & **71.24** & 66.99 & **67.33** & 72.44 & **73.48\({}^{\dagger}\)** & **79.05\({}^{\dagger}\)** & 78.48 & 71.62 & **72.62\({}^{\dagger}\)** & **70.82** & 69.84 \\ \hline \end{tabular}
\end{table}
Table 1: 5-shot FS results using the original or transformed representations. \(\dagger\) indicates the highest accuracy for each dataset. Results are averaged over 5 runs.
where improvements are particularly striking. Here, global similarity structure appears to be crucial for generalizing between the superclasses. Tab. 2 additionally reports current state-of-the-art (SOTA) results on the standard benchmarks; SOTA results are not available for the distribution shift datasets. SOTA approaches generally use additional data relevant to the AD task, such as outlier exposure data or textual supervision for the normal class [53; 13; 36], whereas we use only human similarity judgments. Our transformation also works well in non-standard AD settings (see Appx. D.2).

### Representational alignment

We have shown above that the gLocal transform provides performance gains on FS and AD tasks. Here, we examine whether these performance gains come at the cost of alignment with human similarity judgments as compared to the naive transform, which does not preserve local structure. Thus, we examine the impact of the gLocal transform on human alignment, using the same human similarity judgment datasets evaluated in Mutenthaler et al. [61] plus an additional dataset.4 Specifically, we perform representational similarity analysis [RSA; 46] between representations of two image/text models -- CLIP RN50 and CLIP ViT-L/14 -- and human behavior for four different human similarity judgment datasets [31; 65; 66; 10; 41]. RSA is a method for comparing neural network representations to representations obtained from human behavior [46]. In RSA, one first obtains representational similarity matrices (RSMs) for the human behavioral judgments and for the neural network representations (more details in Appx. E). These RSMs measure the similarity between pairs of examples according to each source. As in previous work [10; 41; 61], we use the Spearman rank correlation coefficient to quantify the similarity of these RSMs. We find that there is almost no trade-off in representational alignment for the gLocally transformed representation space compared to the naively transformed representations (see Tab. 4). Hence, the gLocal transform is able to improve representational alignment while preserving local similarity structure.

Footnote 4: Human similarity judgments were collected by either letting participants arrange natural object images from different categories on a computer screen [66; 10; 41] or in the form of triplet odd-one-out choices [31].

In Fig. 5, we further demonstrate how the representational changes introduced by the linear and gLocal transforms lead to greater alignment with human similarity judgments by visualizing the RSMs on each dataset. The global similarity structure captured by the RSMs is qualitatively identical between naive and gLocal transforms, and both of these transforms lead to RSMs that closely resemble human RSMs (see Fig. 5). Human similarity judgments for data from Hebart et al. [31] were collected in the form of triplet odd-one-out choices. Therefore, we used VICE [60] -- an approximate Bayesian method for inferring mental representations of object concepts from human behavior -- to obtain an RSM for those judgments. Human RSMs are sorted into five different concept clusters using \(k\)-means for datasets from King et al. [41] and Cichy et al. [10] and using the things concept hierarchy [30] for RSMs from Hebart et al. [31]. A visualization for RSMs obtained from CLIP RN50 representations and a more detailed discussion on RSA can be found in Appx. E.

\begin{table}
\begin{tabular}{l|c|c|c|c|c|c|c|c} \hline \multirow{2}{*}{Model \(\backslash\) Transform} & \multicolumn{2}{c|}{CIFAR10} & \multicolumn{2}{c|}{CIFAR100} & \multicolumn{2}{c|}{CIFAR100-Came} & \multicolumn{2}{c}{ImageNet30} & \multicolumn{2}{c}{DTD} \\  & original & gLocal & original & gLocal & original & gLocal & original & gLocal & original & gLocal \\ \hline CLIP-RN50 & 89.44 & **91.9** & 90.83 & **92.92** & 86.47 & **89.29** & 98.89 & **90.85** & 90.67 & **92.29** \\ CLIP-ViT-L/14 (JANI) & 95.14 & **98.16** & 91.41 & **97.19** & 88.5 & **95.83** & 98.91 & **97.85** & 92.02 & **94.9** \\ CLIP-ViT-L/14 (JANI-400M) & **98.8** & 98.39 & **96.66** & 98.53 & **97.86** & 97.77 & 99.51 & **99.69** & 95.54 & **96.44** \\ CLIP-ViT-L/14 (JANI-2B) & 98.97 & **99.11** & 98.76 & **98.97** & 98.65 & 98.51 & 99.29 & **99.74** & 94.87 & **96.78\({}^{\dagger}\)** \\ \hline SOTA & 99.6 [53] & - & - & 97.34 [13] & 99.9 [53] & 94.6 [53] \\ \hline \end{tabular}
\end{table}
Table 2: One-vs-rest nearest neighbor based AD results; with and without transformation. \(\dagger\) indicates the highest accuracy for each dataset.

\begin{table}
\begin{tabular}{l|c c|c c|c c|c c|c} \hline \multirow{2}{*}{Model \(\backslash\) Transform} & \multicolumn{2}{c|}{Entity-13} & \multicolumn{2}{c|}{Entity-30} & \multicolumn{2}{c}{Linking-17} & \multicolumn{2}{c}{Nonliving-26} & \multicolumn{2}{c}{Clift00-ahith} \\  & original & gLocal & original & gLocal & original & gLocal & original & gLocal & original & gLocal \\ \hline CLIP-RN50 & 90.22 & **92.03** & 91.64 & **93.71** & **94.62** & 92.38 & 87.49 & **91.09** & 76.27 & **80.33** \\ CLIP-ViT-L/14 (JANI-400M) & 88.54 & **93.23\({}^{\dagger}\)** & 91.31 & **95.63** & 97.31 & 97.71 & 87.43 & **92.53** & 73.69 & **87.11** \\ CLIP-ViT-L/14 (JANI-2B0M) & 90.79 & **92.71** & 92.49 & **95.04** & **96.56** & 96.32 & 90.09 & **93.18** & 91.09 & **92.7** \\ CLIP-ViT-L/14 (JANI-2B) & 90.33 & **93.08** & 92.12 & **95.84\({}^{\dagger}\)** & 96.96 & **97.37** & 88.82 & **93.54\({}^{\dagger}\)** & 89.73 & **93.94\({}^{\dagger}\)** \\ \hline \end{tabular}
\end{table}
Table 3: One-vs-rest AD with a class distribution shift between train and test sets; with and without transformation. \(\dagger\) indicates the highest accuracy for each dataset.
## 5 Discussion

Although neural networks achieve near-human-level performance on a variety of computer vision tasks, they may not optimally capture _global_ object similarities. By contrast, humans represent concepts using rich semantic features -- including superordinate categories and other global constraints -- for performing object similarity judgments [66; 41; 10; 31; 60]. These representations may contribute to humans' strong generalization capabilities [21; 24]. Here, we investigated the impact of aligning neural network representations with human similarity judgments on different FS and AD tasks.

We find that naively aligning neural network representations, without regularizing the learned transformations to preserve structure in the original representations, can impair downstream task performance. However, our _gLocal transform_, which combines an _alignment loss_ that optimizes for representational alignment with a _local loss_ that preserves the nearest neighbor structure from the original representation, can substantially improve downstream task performance while increasing representational alignment. The transformed representation space transfers surprisingly well across different human similarity judgment datasets and achieves almost equally strong alignment as the naive approach [61], indicating that it captures human notions of image similarity. In addition, it substantially improves downstream task performance compared to both original and naively-aligned representations across a variety of FS and AD tasks. The gLocal transform yields state-of-the-art (SOTA) performance on the CIFAR-100 coarse AD task, and approaches SOTA on other AD benchmarks.

Our work has some limitations. First, as we show in Appx. D, the gLocal transform fails to consistently improve downstream task performance on ImageNet. We conjecture that the gLocal transform can succeed only if representations capture the concepts by which human representations are organized, and ImageNet representations may not. Second, human similarity judgments are more expensive

\begin{table}
\begin{tabular}{l c c c|c c c|c c c} \hline \hline  & \multicolumn{3}{c|}{CLIP-FN50} & \multicolumn{3}{c|}{CLIP-ViT-L/14 (WIT)} & \multicolumn{3}{c}{CLIP-ViT-L/14 (LAION-2B)} \\ Dataset \textbackslash{} Transform & original & naive & gLocal & original & naive & gLocal & original & naive & gLocal \\ \hline \multicolumn{10}{l}{_Human similarity datasets:_} \\ Hebart et al. [31] & 52.78\% & 59.92\% & 59.89\% & 46.71\% & 60.13\% & 60.05\% & 47.50\% & 60.47\% & 60.38\% \\ King et al. [41] & 0.386 & 0.650 & 0.645 & 0.355 & 0.638 & 0.637 & 0.292 & 0.620 & 0.613 \\ Cichy et al. [10] & 0.557 & 0.721 & 0.716 & 0.363 & 0.732 & 0.732 & 0.395 & 0.735 & 0.718 \\ Peterson et al. [65; 66] & 0.364 & 0.701 & 0.705 & 0.260 & 0.688 & 0.688 & 0.314 & 0.689 & 0.660 \\ \hline \multicolumn{10}{l}{_Dowstream tasks:_} \\ \(5\)-shot FS (avg.) & 53.05\% & 41.20\% & **55.01\%** & 67.71\% & 47.62\% & **69.91\%** & 71.10\% & 50.71\% & **72.27\%** \\ Anomaly detection (avg.) & 89.65\% & 90.75\% & **91.52\%** & 90.16\% & 92.79\% & **95.19\%** & 94.79\% & 93.09\% & **96.65\%** \\ \hline \hline \end{tabular}
\end{table}
Table 4: The gLocal transform yields both a high degree of alignment with datasets of human similarity judgments and good performance on FS/AD tasks. Performance on human similarity datasets is measured as odd-one-out accuracy on a held-out test set for things or Spearman’s \(\rho\) for the other three datasets, using either the original representations, the naively transformed representations, or representations transformed via the gLocal transform. For 5-shot FS and AD, we report the average performance across all tasks in Tab. (1, 2, 3).

Figure 5: RSMs for human behavior and CLIP ViT-L/14 [WIT; 70] for four different human similarity judgment datasets [31; 66; 10; 41]. We contrast RSMs obtained from the network’s original representations (second column), the naively aligned representations [61] (third column), and the gLocally-transformed representations (rightmost column) against RDMs directly constructed from human similarity judgments (leftmost column). Yellower colors indicate greater similarity; bluer colors indicate greater dissimilarity.
to acquire than other forms of supervision, and there may be important human concepts that are captured neither by the 1854 images we use for alignment nor by the pretrained representations.

Our results imply that even with hundreds of millions of image/text pairs, image/text contrastive learning does not learn a representation space with human-like global organization. However, since the gLocal transform successfully aligns contrastive representations' global structure using only a small number of images, these representations do seem to have a pre-existing representation of the concepts by which human representations are globally organized. Why does this happen? One possibility is that image/text pairs do not provide adequate global supervision, and thus contrastive representation learning is (near-)optimal given the data. Alternatively, contrastive learning may not incorporate signals that exist in the data into the learned representation because it imposes only local constraints. Previous work has shown that t-SNE and UMAP visualizations reflect global structure only if they are carefully initialized [42]. Given the similarity between contrastive representation learning and t-SNE/UMAP [15] and the known sensitivity of contrastive representations to initialization [52], it is plausible contrastive representations also inherit their global structure from their initialization. Although our gLocal transform provides a way to perform post-hoc alignment of representations from image/text models using human similarity judgments, there may be alternative initialization strategies or objectives that can provide the same benefits during training, using only image/text pairs.

## Acknowledgements

LM, LL, JD, and RV acknowledge funding from the German Federal Ministry of Education and Research (BMBF) for the grants BIFOLD22B and BIFOLD23B. LM acknowledges support through the Google Research Collabs Programme.
