Deep Neural Networks Can Learn Generalizable Same-Different Visual
  Relations
Although deep neural networks can achieve human-level performance on many
object recognition benchmarks, prior work suggests that these same models fail
to learn simple abstract relations, such as determining whether two objects are
the same or different. Much of this prior work focuses on training
convolutional neural networks to classify images of two same or two different
abstract shapes, testing generalization on within-distribution stimuli. In this
article, we comprehensively study whether deep neural networks can acquire and
generalize same-different relations both within and out-of-distribution using a
variety of architectures, forms of pretraining, and fine-tuning datasets. We
find that certain pretrained transformers can learn a same-different relation
that generalizes with near perfect accuracy to out-of-distribution stimuli.
Furthermore, we find that fine-tuning on abstract shapes that lack texture or
color provides the strongest out-of-distribution generalization. Our results
suggest that, with the right approach, deep neural networks can learn
generalizable same-different visual relations.
Alexa R. Tartaglini, Sheridan Feucht, Michael A. Lepori, Wai Keen Vong, Charles Lovering, Brenden M. Lake, Ellie Pavlick
# Deep Neural Networks Can Learn Generalizable Same-Different Visual Relations

###### Abstract

Although deep neural networks can achieve human-level performance on many object recognition benchmarks, prior work suggests that these same models fail to learn simple abstract relations, such as determining whether two objects are the _same_ or _different_. Much of this prior work focuses on training convolutional neural networks to classify images of two same or two different abstract shapes, testing generalization on within-distribution stimuli. In this article, we comprehensively study whether deep neural networks can acquire and generalize same-different relations both within and out-of-distribution using a variety of architectures, forms of pretraining, and fine-tuning datasets. We find that certain pretrained transformers can learn a same-different relation that generalizes with near perfect accuracy to out-of-distribution stimuli. Furthermore, we find that fine-tuning on abstract shapes that lack texture or color provides the strongest out-of-distribution generalization. Our results suggest that, with the right approach, deep neural networks can learn generalizable same-different visual relations.

## 1 Introduction

Humans and a wide variety of non-human animals can easily recognize whether two objects are the same as each other or whether they are different (see Figure 1; Martinho III & Kacelnik, 2016; Christie, 2021; Gentner et al., 2021; Hespos et al., 2021). The abstract concept of equality is simple--even 3-month-old infants (Anderson et al., 2018) and honeybees (Giurfa, 2021) can learn to distinguish between displays of two same or two different objects. Some researchers have even argued that it serves amongst a number of other basic logical operations as a foundation for higher-order cognition and reasoning (Gentner & Goldin-Meadow, 2003; Gentner & Hoyos, 2017). However, in contrast to humans and animals, recent work has argued that deep neural networks struggle to learn this simple relation (Elis et al., 2015; Gulcehre & Bengio, 2016; Stabinger et al., 2016; Kim et al., 2018; Webb et al., 2020; Puebla & Bowers, 2022). This difficulty is surprising given that deep neural networks achieve human or superhuman performance on a wide range of seemingly more complex visual tasks, such as image classification (Krizhevsky et al., 2012; He et al., 2016), segmentation (Long et al., 2015), and generation (Ramesh et al., 2022).

Figure 1: **Same or different?** For humans and a number of animal species, it is trivial to recognize that the image on the left contains two of the same objects, while the image on the right contains two different objects. Surprisingly, prior research has suggested that deep neural networks struggle to learn to discriminate between these images.
Past attempts to evaluate same-different relations in neural networks have generally used the following methodology. Models are trained to classify images containing either two of the same or two different abstract objects, such as those in Figure 1. A model is considered successful if it is then able to generalize the same-different relation to unseen shapes after training. Convolutional neural networks (CNNs) trained from scratch fail to learn a generalizable relation, and tend to memorize training examples (Kim et al., 2018; Webb et al., 2020). However, deep neural networks have been shown to successfully generalize the same-different relation in certain contexts. This generalization is either limited to in-domain test stimuli (Funke et al., 2021; Puebla and Bowers, 2022) or requires architectural modifications that build in an inductive bias towards relational tasks at the expense of other visual tasks (Kim et al., 2018; Webb et al., 2020; 2023a,b; Kerg et al., 2022; Geiger et al., 2023; Altabaa et al., 2023). Given these limited successes, an open question remains: without architectural modifications that restrict model expressivity in general, can standard neural networks learn an abstract same-different relation that generalizes to both in- and out-of-distribution stimuli?

Addressing this question requires going beyond past work in a number of ways. First, most previous studies test for in-distribution generalization--that is, they use test stimuli that are visually similar to the training stimuli. We believe that out-of-distribution generalization provides much stronger evidence that a model has learned a genuine abstract relation without relying on spurious features. Second, the existing literature uses training stimuli that demonstrate the same-different relation with either closed curves (as in Figure 1) or simple geometric shapes. It is unclear whether training on these types of objects is the most helpful for learning the relation versus more naturalistic objects that more closely resemble data seen during pretraining. Finally, most prior work focuses on convolutional architectures, but Vision Transformers (ViTs) (Dosovitskiy et al., 2020) adapted from the language domain (Vaswani et al., 2017) have recently emerged as a competitive alternative to CNNs on visual tasks. Self-attention, a key feature of ViTs, may provide an advantage when learning abstract visual relations--indeed, the ability to attend to and relate any part of a stimulus to any other part may be crucial for relational abstraction.

In this article, we address these limitations and comprehensively investigate how neural networks learn and generalize the same-different relation from image data. Our main findings are as follows:

* Fine-tuning pretrained ResNet and ViT models on the same-different relation enables both architectures to generalize the relation to unseen objects in the same distribution as the fine-tuning set. In particular, CLIP pretraining results in nearly \(100\%\) in-distribution test accuracy for ViT models, and close to that for ResNet models. (Section 3.1)
* Under certain conditions, CLIP-pretrained ViTs can reliably generalize the same-different relation to out-of-distribution stimuli with nearly \(100\%\) accuracy. This suggests that these models may acquire a genuinely abstract concept of equality. (Section 3.2)
* Different fine-tuning datasets lead to qualitatively different patterns of generalization--fine-tuning on more visually abstract objects (which do not contain color or texture) results in stronger out-of-distribution generalization, whereas fine-tuning on more naturalistic objects fails to generalize. (Section 3.2)
* ViTs generally prefer to determine equality between objects by comparing their color or texture, only learning to compare shape when the fine-tuning dataset lacks color and texture information. However, we find that CLIP pretraining helps to mitigate this preference for color and texture. (Section 4)

## 2 Methods

We operationalize the same-different task consistently with prior work e.g. Fleuret et al. (2011). Models are asked to perform a binary classification task on images containing either two of the same objects or two different objects (see the second and third rows of Figure 2). Models are either trained from scratch or fine-tuned on a version of this task with a particular type of stimuli (see Section 2.1 below). After training or fine-tuning, model weights are frozen, and validation and test accuracy scores are computed on sets of same-versus-different stimuli containing unfamiliar objects. These can be either be the same type of objects that they were trained or fine-tuned on (in-distribution generalization) or different types of objects (out-of-distribution generalization). Thus, in order to attain high validation and test accuracy scores, the model must successfully generalize the learned same-different relation to novel objects. This type of generalization is more challenging than the
standard image classification setting because of the abstract nature of what defines the classes--models must learn to attend to the relationship between two objects rather than learn to attend to any particular visual features of those objects in the training data.

### Training and Evaluation Datasets

We construct four same-versus-different datasets using four different types of objects (see Figure 2) ranging from abstract shapes to naturalistic images that are more familiar to pretrained models. We use the following objects to create these four datasets:

1. **Squiggles (SQU).** Randomly generated closed shapes following Fleuret et al. (2011).1 Most studies in the machine learning literature on the same-different relation uses this dataset (Kim et al., 2018; Funke et al., 2021; Puebla and Bowers, 2022; Messina et al., 2022).
2. **Alphanumeric (ALPH).** Sampled handwritten characters from the Omniglot dataset (Lake et al., 2015).
3. **Shapes (SHA).** Textured and colored shapes from Tartaglini et al. (2022). Objects that match in shape, texture, and color are considered the same, whereas objects that differ along all three dimensions are considered different.
4. **Naturalistic (NAT).** Photographs of real objects on white backgrounds from Brady et al. (2008). These stimuli are the most similar to the data that the pretrained models see before fine-tuning on the same-different task.

Footnote 1: The original method from Fleuret et al. (2011) produces closed contours with lines that are only one pixel thick. For our chosen image and object size, these shapes become very difficult to see. We correct this by using a dilation algorithm to darken and thicken the lines to a width of three pixels.

Each stimulus is an image that contains two objects that are either the same or different. We select a total of 1,600 unique objects for each dataset. These objects are split into disjoint sets of 1,200, 300, and 100 to form the training, validation, and test sets respectively. Unless otherwise specified, the training, validation, and test sets each contain 6,400 stimuli: 3,200 same and 3,200 different. To construct a given dataset, we first generate all possible pairs of same or different objects--we consider two objects to be the same if they are the same on a pixel level.2 Next, we randomly select

Figure 2: **Example stimuli from all four datasets. Each column represents one of the four same-versus-different datasets as indicated by the label beneath the stimuli. The top row shows an example object that is used to form the stimuli that comprise each dataset, while the second and third rows show an example “same” vs. “different” stimulus, respectively.**
a subset of the possible object pairs to create the stimuli such that each unique object is in at least one pair. Each object is resized to 64x64 pixels, and then a pair of these objects is placed over a 224x224 pixel white background in randomly selected, non-overlapping positions. We consider two objects in a specific placement as one unique stimulus--in other words, a given pair of objects may appear in multiple images but in different positions (but with all placements of the same two objects being confined to either the training, validation, or test set). All object pairs appear the same number of times to ensure that each unique object is equally represented.

### Models and Training Details

We evaluate one convolutional architecture, **ResNet-50**(He et al., 2016), and one Transformer architecture, **ViT-B/16**(Dosovitskiy et al., 2020). We also evaluate three pretraining procedures: (1) **randomly initialized**, in which all model parameters are randomly initialized (Kaiming normal for ResNet-50 and truncated normal for ViT-B/16) and models are trained from scratch, (2) **ImageNet,** in which models are pretrained in a supervised fashion on a large corpus of images (ImageNet-1k for ResNet-50 and ImageNet-21k for ViT-B/16; Deng et al., 2009) with category labels such as "barn owl" or "airplane," and (3) **CLIP**(Radford et al., 2021), in which models learn an image-text contrastive objective where the cosine similarity between an image embedding and its matching natural language caption embedding is maximized. Unlike ImageNet labels, CLIP captions contain additional linguistic information beyond category information (e.g. "a photo of a barn owl in flight"). To adapt these models for the same-different task, we add a randomly initialized linear classifier on top of the visual backbone of the original architecture.

Each model is trained from scratch or fine-tuned for 70 epochs with a batch size of 128, updating all parameters. We use a binary cross-entropy loss. For each architecture and pretraining combination, we perform hyperparameter tuning via grid search over the initial learning rate (1e-4, 1e-5, 1e-6, 1e-7, 1e-8), learning rate scheduler (exponential, ReduceLROnPlateau), and optimizer (SGD, Adam, AdamW). We select the best performing training configuration from the grid search according to in-distribution validation accuracy, and then train a model with those hyperparameters five times with different random seeds. We report the median test results across those five seeds.

## 3 Generalization to Unseen Objects

### In-Distribution Generalization

We first measure the performance of each model on test data containing the same types of objects used to train or fine-tune the model; e.g. models fine-tuned on pairs of handwritten characters are then tested on handwritten characters that were not seen during training. We refer to this as the in-distribution performance of the model. The starred (\(*\)) result in Figure 3 shows the in-distribution median test accuracy of randomly-initialized ResNet-50 models trained on the Squiggles dataset, which contains the same type of closed contours used by much of the prior work on the same-different relation (Fleuret et al., 2011; Kim et al., 2018; Funke et al., 2021; Puebla and Bowers, 2022; Messina et al., 2022). Confirming the primary findings from prior work, these models do not attain above chance level test accuracy. The same pattern holds for randomly initialized ViT-B/16 models.

However, as the rest of Figure 3 shows, pretrained models exhibit substantially improved in-distribution accuracy compared to randomly initialized models across all four datasets. In particular, models pretrained with CLIP demonstrate the largest improvements, attaining nearly \(100\%\) test accuracy irrespective of fine-tuning dataset. Even without any fine-tuning, CLIP features appear to be highly useful for the same-different task--linear probes trained to do the same-different task using CLIP ViT-B/16 embeddings of stimuli without any fine-tuning achieve between \(80\%\) and \(100\%\) median in-distribution test accuracy depending on the dataset (Table 10, Appendix A.5). Differences in performance can also be observed between architectures, with ViT-B/16 models consistently outperforming ResNet-50 after pretraining.3
Another main finding is that the two visually abstract, shape-based datasets (Squiggles and Alphanumeric) appear to pose more of a challenge to models than the Shapes and Naturalistic datasets--models attain noticeably higher in-distribution accuracy on the latter two across architectures and pretraining methods (although the effect is small for CLIP-pretrained models). This difference may be due to the color and texture information that is available in these datasets, which provides additional dimensions over which objects can be compared. We explore the possibility that some models find it easier to evaluate equality using color and texture information in addition to or instead of shape information in Section 4.

### Out-of-Distribution Generalization

The previous section showed that pretrained models can generalize to unseen, in-distribution objects. However, if a model learns a truly abstract notion of same-different, it should generalize the same-different relation to any two objects regardless of their particular visual features. Thus, model performance on stimuli that are substantially different from training stimuli is a stronger measure of abstraction. We therefore measure test accuracy for each model across all four datasets, yielding one in-distribution score and three out-of-distribution (OOD) scores per model. Table 1 shows median test accuracy over five seeds for CLIP-pretrained models; full generalization tables for all pretraining styles and architectures can be found in Appendix A.2.

Overall, CLIP ViT-B/16 models fine-tuned on the Squiggles task exhibit the strongest OOD generalization, achieving \(>\)95% median test accuracy on the three out-of-distribution datasets. It is worth noting that both this model and CLIP ResNet-50 fine-tuned on the Alphanumeric task (the model with the second best OOD generalization performance) exhibit some degree of sensitivity to the random seed used during fine-tuning: most random seeds result in nearly \(100\%\) OOD generalization for ViT or \(>\)80% for ResNet across all datasets, while some seeds result in substantially lower performance (1/5 seeds for ViT and 2/5 for ResNet). No other model configurations exhibit this bimodal behavior. See Appendix A.8 for details.

As in the previous section, models fine-tuned on objects with visually abstract shape features only (Squiggles and Alphanumeric) behave differently than those fine-tuned on datasets containing objects with shape, color, and texture features (Shapes and Naturalistic). The Squiggles and Alphanumeric models generally attain high OOD test accuracy. On the other hand, models fine-tuned on

Figure 3: **In-distribution test accuracy by architecture and pretraining method. Bars show median accuracy over 5 runs (individual points also shown), with the bar color denoting pretraining type and the x-axis denoting the dataset used for fine-tuning. See Section 2.1 for dataset descriptions, Section 2.2 for model details, and Figure 2 for visual examples. The starred (\(*\)) result is a replication of findings from prior work showing that CNNs trained from scratch on stimuli like the images in Figure 1 attain chance-level test accuracy. The double-starred (\(**\)) result mirrors Funke et al. (2021) and Puebla & Bowers (2022), who show that ImageNet-pretrained CNNs attain substantially higher in-distribution test accuracy relative to the same architectures trained from scratch.**
the Shapes or Naturalistic datasets generalize well to each other but struggle to generalize to the Squiggles and Alphanumeric tasks. Note that some of this effect can be attributed to miscalibrated bias, but not the entire effect--see Appendix A.3 for details.

Another way to understand the generalization pattern in Table 1 is that the more "challenging" a dataset is to generalize the same-different relation to, the more effective it is as a fine-tuning dataset for inducing out-of-distribution generalization. For example, CLIP ViT-B/16 models fine-tuned on datasets other than Squiggles attain a median test accuracy of only \(51.8\%\) on the Squiggles task on average, whereas CLIP ViT-B/16 fine-tuned on Squiggles attains an average OOD test accuracy of \(97.8\%\). On the other hand, the Shapes dataset is easy for models fine-tuned on other datasets to generalize to (\(99.5\%\) accuracy on average), but CLIP ViT fine-tuned on that "easier" dataset attains an average OOD test accuracy of only \(68.5\%\). This pattern of Squiggles being more "difficult" to generalize to persists across architectures and pretraining methods (Appendix A.2).

We further study why fine-tuning on different datasets results in different generalization behaviors by computing the average cosine similarity between objects in a given dataset using pretrained CLIP embeddings (Table 2). This value provides information about the visual variation in each dataset through the lens of a specific model: a higher number means that stimuli in that dataset are generally embedded more closely together by that model. Before fine-tuning, pretrained CLIP models embed Squiggles stimuli more closely together than stimuli from other datasets, potentially explaining the difficulty of that dataset as a test of OOD generalization. We also note what seems to be a correlation between "closeness" of stimuli in a model's embedding space and ability of models fine-tuned on that dataset to generalize OOD. Given that random noise is embedded even more closely together than Squiggles stimuli, we fine-tune models on the same-different task comparing patches of random noise and measure their OOD generalization in Appendix A.7. We find that models fine-tuned on noise exhibit weaker OOD generalization than models fine-tuned on Squiggles, indicating that "closeness" of stimuli is not a perfect correlate of OOD generalization and is only part of the story.

\begin{table}
\begin{tabular}{l c c c c c c c c c c} \hline \hline \multicolumn{1}{c}{} & \multicolumn{4}{c}{**CLIP ResNet-50**} & \multicolumn{4}{c}{**CLIP ViT-B/16**} \\ \hline \multicolumn{1}{c}{} & \multicolumn{4}{c}{\(\leftarrow\)**Test \(\rightarrow\)**} \\ \multicolumn{1}{c}{**Train \(\downarrow\)**} & \multicolumn{1}{c}{SOU} & \multicolumn{1}{c}{ALPH} & SHA & NAT & Avg. & \multicolumn{1}{c}{**Train \(\downarrow\)**} & \multicolumn{1}{c}{SOU} & \multicolumn{1}{c}{ALPH} & SHA & NAT & Avg. \\ \hline SQU & **97.7** & 82.9 & 86.9 & 82.0 & 83.9 & SQU & **99.6** & 97.7 & **99.1** & 96.7 & 97.8 \\ ALPH & **82.1** & **97.4** & 92.8 & 91.8 & 88.9 & ALPH & 55.3 & **99.4** & 99.6 & 91.2 & 82.0 \\ SHA & 56.0 & 78.1 & **98.1** & 96.1 & 76.7 & SHA & 50.0 & 55.4 & **100** & 100 & 68.5 \\ NAT & 50.1 & 59.3 & 93.4 & **97.3** & 67.6 & NAT & 50.0 & 68.0 & 99.8 & **100** & 72.6 \\ \hline Avg. & 62.7 & 73.4 & 91.1 & 90.0 & Avg. & 51.8 & 73.7 & 99.5 & 95.9 & \\ \hline \hline \end{tabular}
\end{table}
Table 1: **Out-of-distribution (OOD) test accuracy for CLIP models fine-tuned on each dataset.** Rows indicate the dataset that models are fine-tuned on, while columns indicate the test dataset. Each cell is the median performance over five random seeds. The rightmost column labeled “Avg.” is the row-wise average of accuracy scores across OOD test sets (i.e. off-diagonal values), which indicates how well a model fine-tuned on a given dataset is able to generalize to other datasets. The bottom row labeled “Avg.” is the column-wise average across off-diagonal values, indicating how difficult it is for models fine-tuned on other datasets to generalize to the given dataset. Note that the **bolded** diagonals are the red bars in Figure 3. OOD generalization results for all model architectures and pretraining styles can be found in Appendix A.2; Appendix A.3 shows median AUC-ROC scores.

\begin{table}
\begin{tabular}{l c c} \hline \hline Dataset \(\downarrow\) & **ResNet-50** & **ViT-B/16** \\ \hline noise & 0.992 & 0.993 \\ SQU & 0.929 & 0.940 \\ ALPH & 0.881 & 0.889 \\ SHA & 0.855 & 0.861 \\ NAT & 0.788 & 0.805 \\ \hline \hline \end{tabular}
\end{table}
Table 2: **Average pairwise cosine similarity between CLIP embeddings of training stimuli within each dataset.** Because \(n=6,400\) for each dataset, averages are computed over approximately 200 pairs. We extract CLIP embeddings _before_ fine-tuning on the same-different task. For similarities afterwards, see Appendix A.6.
## 4 Examination of Inductive Biases

### Grayscale and Graysmasked Objects

What features are most important for learning a genuinely abstract same-different relation, and do vision models have inductive biases towards these useful features? Previous work has claimed that CNN models trained on ImageNet are often biased towards texture over shape (Geirhos et al., 2019; Hermann et al., 2020), which seems consistent with results from Kim et al. (2018) showing low performance for CNNs trained from scratch on abstract shapes. Based on the results in the previous section, shape information appears to be the most important for effective OOD generalization, which motivates us to investigate whether models have an inductive bias towards it for the visual same-different task. One way of examining this is by training models on one of three variants of the Shapes dataset: objects are either kept the same (Figure 4a, "Colored"), grayscaled to preserve texture but remove color (Figure 4a, "Grayscale"), or completely masked over in gray to remove both texture and color (Figure 4a, "Masked"). If a model is biased towards color, we would expect performance to drop on the Grayscale and Masked datasets, and if it is biased towards texture, we would expect performance to suffer on the Masked dataset. Only a model biased towards shape would generalize effectively to all three settings.

Figure 5 shows the results of this experiment for randomly initialized and CLIP-pretrained models (ImageNet results in Appendix B.1). Figure 5A shows the test performance of three versions of randomly initialized ViT-B/16, trained on either Color, Grayscale, or Masked versions of the Shapes dataset (Figure 4a) and then tested on novel objects from each of those distributions. From this subplot, we can see that ViT-B/16 trained from scratch can only achieve high in-distribution accuracy (shown as hatched bars) for Color Shapes (\(92.9\%\)); the hatched gray and dark gray bars representing in-distribution accuracy for Grayscale and Masked Shapes are much lower (\(78.8\%\) and \(53.5\%\) respectively). Although ViTs trained from scratch on the Color Shapes dataset attain high in-distribution accuracy, performance drops to \(66.2\%\) and \(62.6\%\) when generalizing out-of-distribution to Grayscale and Masked Shapes, indicated by the two lower gray bars beside the hatched green bar. This gap suggests that ViT-B/16 may only learn to compare object _color_ when it is trained from scratch on the Color Shapes dataset, leading to greater errors when tested on datasets that do not contain color. Figure 5B shows that CLIP pretraining weakens this inductive bias towards color, allowing for high in-distribution accuracy and near-perfect out-of-distribution generalization when trained on any of the three modified Shapes datasets.

When ResNet-50 is trained from scratch, we do not see evidence of an inductive bias towards color or texture, since bars of all colors in Figure 5C are at roughly equal heights. However, this bias reappears after pretraining (albeit to a lesser extent than for randomly-initialized ViT), with a \(7.3\%\) gap between in-distribution and Masked OOD accuracy for CLIP ResNet-50 fine-tuned on Color Shapes (Figure 5D).

Figure 4: **Examples of stimuli used to test inductive biases. Figure (a) shows examples of objects from the three versions of the Shapes dataset used to produce results in Figure 5. Figures (b) and (c) are examples of stimuli with conflicting signals used in Section 4.2, where either color is the same while texture and shape are different, or color and texture are the same while shape is different.**
### Dissociating Color, Texture, and Shape

Results from Figure 5 suggest that some models learn to rely on certain features more than others to differentiate between objects in an image. We run an additional experiment to verify these claims by creating new stimuli for which color, texture, and shape send conflicting signals: for example, two objects in an image might be the same color but have different textures and shapes (Figure 4b). We then evaluate each model from Figure 5 on every possible combination of conflicting signals according to experiment details in Appendix B.2 to identify which features each model might be more biased towards. Median results are reported across five random seeds for the same set of hyperparameters. First, our results confirm that randomly-initialized ViT-B/16 is heavily biased towards color when trained on the Color Shapes dataset. When trained completely from scratch on Color Shapes, ViT-B/16 will predict "same" for any stimuli in which the colors of the two objects are the same, even if color is the only similarity between the two objects (classifying \(85.6\%\) of stimuli like Figure 4b as "same"). In contrast, CLIP ViT-B/16 fine-tuned on the same Color Shapes data classifies only \(12.3\%\) of stimuli like Figure 4b as "same." However, our results also show that even CLIP ViT-B/16, which appears to be unbiased, exhibits different patterns of behavior depending on the dataset it is fine-tuned on. When tested on stimuli where color and texture are the same and shape is different (Figure 4c), CLIP ViT-B/16 fine-tuned on Color Shapes classifies \(88.8\%\) of these stimuli as "same." The same model fine-tuned on Masked Shapes classifies \(98.2\%\) of these images as "different." All results can be found in Appendix B.2.

A similar phenomenon can be observed in pretrained ResNet-50 models. For example, CLIP ResNet-50 fine-tuned on Masked Shapes exhibits a strong bias towards shape, classifying \(\geq 89.8\%\) of objects with matching shapes as "same" while classifying \(\geq 93.3\%\) of objects with mismatching shapes as "different." Thus, even though pretraining can mitigate the strength of ViT inductive biases, all pretrained architectures still exhibit specific, understandable biases that are determined by the features available in their training data. Moreover, these distinct patterns of generalization provide a glimpse into how these networks are computing same-versus-different judgments.

Figure 5: **Test accuracy for ViT-B/16 fine-tuned on one version of the Shapes dataset – Color, Grayscale, or Masked – and tested on all three datasets.** See Figure 4a for example objects from each of the three datasets. Bars are grouped by fine-tuning dataset as indicated along the x-axis. Hatched bars indicate in-distribution accuracy. Median accuracy over five seeds is reported with individual runs plotted as points over each bar. See Appendix B.1 for ImageNet and ResNet results.
## 5 Related Work

**Prior work on the same-different relation.** Learning the same-different relation when the two objects being compared occupy the same image appears to be the most challenging setting for deep neural networks. Most closely related to our work is Puebla & Bowers (2022). Their setup is very similar to ours in a few respects: they define the same-different task identically to us, fine-tuning ImageNet pretrained ResNet-50 models on the task using stimuli from Fleuret et al. (2011) (which are nearly identical to our Squiggles stimuli), and testing OOD generalization on nine evaluation sets. They find that their models fail to generalize out-of-distribution and draw the conclusion that current CNNs are unable to learn the relation. Replicating their setup, we find that the difference in our results is due to the differing architectures and pretraining methods we investigated. Our ImageNet ResNet-50 model fine-tuned on Squigles also struggles to generalize to the evaluation sets used in Puebla & Bowers (2022), but our CLIP ViT-B/16 model fine-tuned on Squigles generalizes perfectly or nearly perfectly to seven out of nine of these sets. See Appendix A.4 for details. Additionally, Funke et al. (2021) report that ImageNet pretrained ResNet-50 models fine-tuned on stimuli from Fleuret et al. (2011) can generalize the relation to in-distribution test stimuli in this setting, but OOD generalization is not tested. The double-starred (\(**\)) bar in Figure 3 replicates their results. Messina et al. (2022) show that a recurrent, hybrid CNN+ViT can attain high in-distribution test accuracy when objects occupy the same image, while (Webb et al., 2023b) demonstrate success using slot attention to segment the objects. Otherwise, successful generalization of the same-different relation with deep neural networks has been limited to a setting where objects are segmented into two separate inputs by humans and separately passed into a neural network (Kim et al., 2018; Webb et al., 2020; Kerg et al., 2022; Altabaa et al., 2023; Geiger et al., 2023).

**Abstract relation learning.** More generally, our work relates to a larger body of work concerned with the abilities of deep neural networks to learn abstract relations. The ability to abstract from sparse sensory data is theorized to be fundamental to human intelligence (Tenenbaum et al., 2011; Ho, 2019) and is strengthened by the acquisition of language (Gentner & Hoyos, 2017). In contrast, standard deep neural networks often struggle to learn relational reasoning from sensory data alone, even when the training corpus is very large (Mitchell, 2021; Davidson et al., 2023). This is often pointed to as a key discrepancy between human and machine visual systems.

## 6 Discussion and Conclusion

Previous work has argued that deep neural networks struggle to learn the same-different relation between two objects in the same image (Kim et al., 2018; Puebla & Bowers, 2022), but the scope and nature of these difficulties are not fully understood. In this article, we explored a range of architectures, pretraining styles, and fine-tuning datasets in order to thoroughly investigate the ability of neural networks to learn and generalize the same-different relation. Some of our model configurations are able to generalize the relation across all of our out-of-distribution evaluation datasets; the best model is CLIP-pretrained ViT fine-tuned on the Squiggles same-different task. Across five random seeds, this model yields a median test accuracy of nearly \(100\%\) on every evaluation dataset we use. The existence of such a model suggests that deep neural networks can learn generalizable representations of the same-different relation, at least for the tests we examined.

There are a number of possible reasons why CLIP-pretrained Vision Transformers exhibit the strongest out-of-distribution generalization. CLIP pretraining may be helpful because of the diversity of the dataset, which Fang et al. (2022) argue is key in the robust generalization of CLIP models in other settings. Another hypothesis is that linguistic supervision from captions containing phrases like "same," "different," or "two of" (which ImageNet models would have no exposure to) helps models to separate same and different objects in their visual embedding spaces, an idea supported by the results of our linear probe experiments (Appendix A.5). ViTs may perform the best on the same-different task because of their larger receptive field size; CNNs can only compare distant image patches in deeper layers, whereas ViTs can compare any image patch to any other as early as the first self-attention layer. Thus, ViTs may be able to integrate complex shape information and compare individual objects to each other more efficiently than CNNs.

What mechanisms enable certain models to generalize the same-different relation? It is possible that models learn an internal circuit (e.g. Nanda et al. (2023)) in which they segment two objects
and compute their equality in embedding space, implicitly implementing the components of relational architectures from earlier works that explicitly separate objects. This would amount to true abstraction and would theoretically enable models to generalize to any distribution of same-different stimuli. Indeed, CLIP ViT-B/16 also generalizes with near perfect accuracy to multiple evaluation sets used in other work (see Appendix A.4). Since our results strongly suggest that neural networks _can_ learn generalizable same-different relations, the next step for future work is to investigate the internal workings of successful models. We believe that further progress in understanding abstraction ought to come from circuit-level investigations into the same-different relation, as well as other abstract visual relations. Such investigations may reveal additional relational reasoning capabilities that have long been considered out of reach for standard deep neural networks.
