Evaluating Visual Number Discrimination in Deep Neural Networks
The ability to discriminate between large and small quantities is a core
aspect of basic numerical competence in both humans and animals. In this work,
we examine the extent to which the state-of-the-art neural networks designed
for vision exhibit this basic ability. Motivated by studies in animal and
infant numerical cognition, we use the numerical bisection procedure to test
number discrimination in different families of neural architectures. Our
results suggest that vision-specific inductive biases are helpful in numerosity
discrimination, as models with such biases have lowest test errors on the task,
and often have psychometric curves that qualitatively resemble those of humans
and animals performing the task. However, even the strongest models, as
measured on standard metrics of performance, fail to discriminate quantities in
transfer experiments with differing training and testing conditions, indicating
that such inductive biases might not be sufficient.
Ivana Kajić, Aida Nematzadeh
# Evaluating Visual Number Discrimination in Deep Neural Networks

###### Abstract

The ability to discriminate between large and small quantities is a core aspect of basic numerical competence in both humans and animals. In this work, we examine the extent to which the state-of-the-art neural networks designed for vision exhibit this basic ability. Motivated by studies in animal and infant numerical cognition, we use the numerical bisection procedure to test number discrimination in different families of neural architectures. Our results suggest that vision-specific inductive biases are helpful in numerosity discrimination, as models with such biases have lowest test errors on the task, and often have psychometric curves that qualitatively resemble those of humans and animals performing the task. However, even the strongest models, as measured on standard metrics of performance, fail to discriminate quantities in transfer experiments with differing training and testing conditions, indicating that such inductive biases might not be sufficient.

## Basic Numerical Competence

The ability to represent abstract numbers and compare numerical quantities is a basic numerical competence observed in both animals and humans (Dehaene, Dehaene-Lambertz, & Cohen, 1998). It helps animals in foraging, navigation, hunting, and reproduction (Nieder, 2020), and is also correlated with the later mathematical ability in prelinguistic infants (Gilmore, McCarthy, & Spelke, 2007; Halberda, Mazzocco, & Feigenson, 2008). While such a skill is shared across species and is independent of explicit feedback or formal education (Dehaene, 1997; Gallistel & Gelman, 1992), the degree to which more advanced numerical skills, such as counting and symbolic representation of number, are present across species remains a debated topic (O'Shaughnessy, Gibson, & Piantadosi, 2021; Revkin, Piazza, Izard, Cohen, & Dehaene, 2008; Anobile, Cicchini, & Burr, 2016; Gallistel & Gelman, 1992).

To investigate number representation and processing, different neural networks have been used as cognitive models of various numerical skills such as magnitude comparison (Vergus & Fias, 2004; Dehaene & Changeux, 1993; Zorzi & Butterworth, 1999), subitizing (Peterson & Simon, 2000) and counting (Rodriguez, Wiles, & Elman, 1999; Fang, Zhou, Chen, & McClelland, 2018). Neural networks are able to encode exact magnitudes (Creatore, Sabathiel, & Solstad, 2021) and develop basic numerical abilities such as numerosity comparison (Testolin, Dolfi, Rochus, & Zorzi, 2020).

While such networks have been used successfully to explain different phenomena in numerical cognition, their architecture is often designed for a task targeting specific cognitive function. In contrast to such specialized networks, in recent years we have witnessed a radical improvement in both the performance, and the quality of representations learned by deep neural networks that are trained end-to-end across vision (Simonyan & Zisserman, 2014; He, Zhang, Ren, & Sun, 2016), language (Vaswani et al., 2017; Devlin, Chang, Lee, & Toutanova, 2018; Brown et al., 2020), and multimodal (Lu, Batra, Parikh, & Lee, 2019; Radford et al., 2021; Alayrac et al., 2022) domains.

Here, we investigate whether state-of-the-art models designed for visual processing, also referred to as vision encoders, can exhibit basic numerical competence as observed in humans and animals. Specifically, we evaluate _number discrimination_ in vision encoders, defined as the ability to make broad relative numerical judgements such as many versus few, which is imprecise and not as advanced as counting, but within the normal ability of many animals (Davis & Memmott, 1982). We draw inspiration from studies in animal and child cognition and use a simple discrimination paradigm known as the _bisection task_ to examine if recent vision encoders can learn to discriminate stimuli on the basis of number.

We consider three vision encoders with varying degrees of explicit inductive biases: ResNet(He et al., 2016), ViT (Dosovitskiy et al., 2020), and Swin(Liu et al., 2021), as well a simple, comparatively small, multi-layer perception (MLP) not designed for vision tasks as a baseline. Across all conditions, Swin and ResNet with image-specific inductive biases are the most successful models in number discrimination; moreover, Swin matches the empirical data from humans and animals in more conditions than ResNet suggesting that its additional hierarchical bias results in a better abstract number representation. Even the strongest models, however, often fail in conditions that test for the transfer of numerical skill to a new condition; for example, when models are trained on a stimulus with solid shapes but tested on a stimulus where shapes are not filled. Although models fail in such transfer conditions, we find that they do learn structured number representations, forming clusters that are ordered based on the number identity. This suggests that, unlike humans and animals whose numerical skills generalize across different ecological contexts, vision encoders might require additional modeling innovations or a greater quantity and va
riety of data to use their learned knowledge in new situations.

## 2 The Numerical Bisection Task

The numerical bisection task is used to assess perception of numerical quantitites in both animals and humans. First, a participant is trained to discriminate small and large sample numerosities by associating them with different responses (labels such as _few_ and _many_). For example, Emmerton, Lohmann, and Niemann (1997) train pigeons to respond to images with 1 or 2 shapes by packing to the left (corresponding to _few_), and to the right for images with 6 or 7 shapes (corresponding to _many_). In Almeida, Arantes, and Machado (2007), children learn to pick a green cup for 2 drumbeats, or a blue cup for 8 drumeats in one experiment, and raise a red glove on their left hand after 2 drumbeats and a yellow glove on their right hand after 8 drumbeats in another experiment. The numerosities used for training (_i.e.,_ 1, 2, or 8) are often referred to as _anchor numerosities_.

Then, to probe number discrimination, participants are subsequently tested on intermediate numbers that are _not_ seen during training (_e.g.,_ 3 in the previous experiment). A participant is more likely to select the response associated with the larger anchor value (_e.g., many_), resulting in an s-shaped psychometric curve. Such s-shaped psychometric curves have been used to characterize basic numerical competence in rats (Meck & Church, 1983), pigeons (Honig & Stewart, 1989; Emmerton et al., 1997), rhesus macaques (Jordan & Brannon, 2006), as well as children and adults (Droit-Volet, Clement, & Fayol, 2003; Almeida et al., 2007; Jordan & Brannon, 2006). Qualitatively, psychometric curves documented in the literature have the following characteristics: (1) the initial segment with smaller numerosities is mostly labeled with _few_, (2) intermediate segment with a gradually increasing slope reflecting an increase in _many_ responses, (3) final segment with the largest numerosities mostly labeled with _many_. Although these properties characterize the majority of psychometric responses documented in the literature, between- and within-subject variability has been observed depending on the task and numerosity ranges (Almeida et al., 2007).

### Experimental Stimuli

We automatically generate images with black background and white circles varying the number of circles from 1 to 7. Similar to Emmerton et al. (1997), we use images with 1, 2, 6, or 7 circles as anchor numerosities for training. When designing stimuli, previous work has identified and controlled for potential perceptual confounds such as the size of the constituent elements (_i.e.,_ circles in our case), total white area, or total perimeter (Honig & Stewart, 1989; Testolin et al., 2020; Emmerton et al., 1997); processing these non-numerical features--which may be a confound in the observed numerical discrimination behavior--can develop independently of number processing, as has indeed been observed in children's developmental trajectory (Odic, 2018). To control for such potential confounds, we generate six different stimulus categories shown in Figure 1:

1. _Vary Size_. This is our most general setting, where for each image, we draw circles with radii drawn randomly from a set of 3 values (\(r=\{10,35,55\}\)).
2. _Constant Size_. We control for the size of circles--all circles have the same radius (\(r=20\)); this enables us to examine whether models can discriminate numbers better when circles are identical compared to varied in size.
3. _Constant Area_. In the previous condition (_Constant Size_), the white area (covered by circles) increases as the number of circles increases. We control for this potential confound by fixing the total white area to be constant across stimuli. This results in smaller circles in images that depicts larger numbers.
4. _Constant Area (contour_). We also examine if solid shape background has an impact on models' behavior; we consider a condition the same as _Constant Area_, but using contours instead of shapes with white background.
5. _Constant Circumference_. While the total area is controlled for in the _Constant Size_ condition, the total circumference of circles increases with numerosity. Here, we control for the total circumference by keeping it constant across stimuli.

Figure 1: Sample visual stimuli used in the numerical bisection task. Rows are different numerosities and columns different stimuli types. ”(C)” denotes ”contours”, as opposed to shapes with solid white background.
6. _Constant Circumference (contour)._ It is the same as _Constant Circumference_, but using contours instead of shapes with white background.

We generate the stimuli on-the-fly for both the training and testing and store them in-memory to be used during training and testing, with 100 images generated for each numerosity category of one stimulus type, resulting in overall 400 images for training, and 1,100 images for testing for each stimulus type. The images are of dimensionality expected by models, _i.e.,_\(224\times 224\times 3\).

## 5 Experimental Setup

In this section, we examine a few recent families of deep neural networks designed for computer vision (henceforth, vision encoders); all these models have achieved impressive results on computer vision tasks (such as image classification), but differ with respect to inductive biases that their architecture encode. We first describe these models briefly, and then discuss the details of our experimental setup.

Models.We consider three types of vision encoders: ResNet [14], ViT [15], and Swin [17]. The ResNet model includes a stack of convolutional neural network (CNN) blocks that process images using convolution kernels. These kernels introduce an explicit _locality_ bias--pixels (or features depending on the layer) that are close spatially are combined; as a result, a model with CNN blocks typically learns to encode low-level features (such as edges) in its first layers, and more high-level ones (such as parts) in its last layers.

Both ViT and Swin use Transformer blocks [21] consisting of feed-forward layers and a _self-attention_ mechanism; self-attention introduces a weaker and less explicit _locality_ bias (compared to CNNs) as a model can learn to group neighboring image patches.1 Swin builds on ViT and introduces an explicit _hierarchical_ bias by modifying how self-attention is applied across different layers; more specifically, local image patches are merged at at various stages as the depth of the model increases, resulting in a hierarchical representation.

Footnote 1: Self-attention is designed for sequential data such as language; thus, it is less suitable for modelling the two-dimensional spatial relations among image patches.

We use specific variants of ResNet, ViT, and Swin encoders: the ResNet-50 variant with 25.6M parameters [14], ViT-B [15] with 86M parameters, and "tiny" Swin, Swin-T [16], with 29M parameters. We picked the smallest ViT and Swin variants, and a ResNet model that has a similar number of parameters to Swin.

Finally, as a simple baseline, we consider a generic feed-forward multi-layer perceptron (MLP) that does not include any inductive biases such as convolutions or attention which are known to be helpful for processing of real-world images. We use an MLP consisting of 2 hidden layers with 256 units each, separated by ReLU non-linearities, and a final linear layer with 2 units. With 0.13M parameters and no "bells and whistles", this makes it a substantially smaller, yet less computationally inexpensive baseline model.

Training.For each stimulus type (_e.g., Constant Area_), we train ResNet, ViT, Swin and MLP models on data generated for that stimulus, _i.e.,_ images and their labels (_few_ and _many_). More specifically, we add a classification head to these models, to predict the label _few_ for images with 1 and 2 circles, and _many_ for images with 6 and 7 circles, where labels are encoded as one-hot vectors. All models are trained with a cross-entropy loss and L2 regularization. To get an estimate of variability in model responses for each stimulus category, we train 10 networks by choosing a different seed that randomly initializes network weights.

We perform a hyper-parameter search on the batch size, number of steps, learning rate, and optimizer type to find combinations where training loss has converged on the validation set, and where a network is achieving close to 100% accuracy on the training set. Accuracy is defined as a percentage of correctly classified labels.2

Footnote 2: We find that the batch size of 16, and 5,000 steps worked well for all models, although losses in some models (_e.g.,_ Swin and ResNet) converged much faster. We use the Adam optimizer [15] MLP, ViT, and Swin models, with learning rates of 1e-04, 5e-04, and 5e-05, respectively. We use the SGD optimizer with a learning rate of 1e-2 for ResNet. The models are trained using a NVidia Tesla V100 GPU.

Testing.We test the models on new images of anchor numerosities (_i.e., not_ seen during training), as well as images of novel interpolated numerosities: 3, 4 and 5. We use 100 images for each numerosity category and each stimulus type.

## 6 Experimental Results

In Experiment 1, we investigate models' behavior when trained and tested on the same stimulus type. In Experiment 2, we investigate transfer of the number discrimination skill by testing models on images from a stimulus category that is not used in training (_i.e.,_ train on _Constant Size_, and test on _Constant Area_).

### Experiment 1: Number Discrimination

In this experiment, we test number discrimination using images from the same stimulus category that is used during training (_e.g.,_ train on _Constant Size_, and test on _Constant Size_). We evaluate models based on the accuracy of the stimulus test set, and the quantitative and qualitative characteristics of psychometric curves in relation to the empirical data.

Performance on seen numbers.We first examine the performance of the four architectures when tested on novel images of the anchor numerosities (seen during training): 1, 2, 6, 7. An error occurs when an image with a small numerosity
(_i.e.,_ 1 or 2 circles) is classified as _many_, or when an image with a large numerosity (_i.e.,_ 6 or 7) is classified as _few_. Average error rates for each network and each stimulus type are shown in Table 1. Overall, we observe that ResNet and Swin, with image-specific inductive biases, have smallest mean error rates of less than 1% in 6/12 and 11/12 conditions, respectively. ViT has mean error rates that are in some cases comparable to or even exceed errors of the MLP baseline. When averaged across all 4 numerosities, we find that highest error rates are consistently observed with the _Constant Circumference (contour)_ stimulus category (See Table 1, column "Total error"); suggesting that this combination of visual features represented the most challenging dataset for number abstraction. Meanwhile, no such consistent pattern exists for datasets resulting in smallest errors--the smallest error for ResNet is observed with _Constant Size_, for ViT and Swin with _Constant Circumference_, and for the MLP with _Constant Area_. This observation is not surprising given that these models encode different inductive biases.

Performance on new numbers.Next, we examine how different models perform on numbers not seen at the training time (_i.e.,_ 3, 4, and 5). We plot the psychometric curves for selected stimuli, showing percentages of _many_ responses across numerosities for models trained on that stimuli in Figure 2. We selected these stimulus categories as representative of the easiest (_Constant Size_, _Constant Circumference_) and hardest (_Constant Circumference (contour)_) conditions based on the average error rates in Table 1. Different from Table 1, each value on the y-axis represents a proportion of _many_ responses for a certain numerosity (x-axis).

Overall, some curves in Fig. 2 exhibit characteristics of typical psychometric functions as discussed in Sec. Experimental Setup--specifically, for small numerosities 1, 2, and sometimes 3, we observe a slowly accelerating initial segments, followed by gradual increase with intermediate numbers, and a slowly decelerating final segment for larger numerosities (6, 7). Examples of such curve profiles are Swin and ResNet responses to _Constant Size_, and _Constant Circumference_ stimulus categories. However, there are also curves that have atypical flat shapes indicative of failure to learn this task, _i.e.,_ those not found in the literature. Out of all 24 curves we analyzed, 4 in total exhibit such a shape with 3 shown in Fig. 2 (_i.e.,_ MLP with _Constant Circumference (contour)_, ViT with _Constant Circumference (contour)_). Such curves either do not start at, or do not end at expected values indicating lack of sensitivity to number categories, some of which is also evident based on large error values in Table 1.

### Experiment 2: Transfer to Novel Stimuli

The previous experiment demonstrates that Swin and ResNet architectures to a great degree appear to be able to differentiate number of items in an image when trained and tested on the same stimulus type (_e.g.,_ constant total area or circumference). To understand whether our models have indeed developed a notion of a number category as opposed to learning a given stimuli, we draw a parallel with research in animal cognition and examine if the models "_base their behaviour on the numerosity of a set, independent of its other attributes_" (Gallistel & Gelman, 1992). In other words, if models learn an abstract representation of a number category, we would expect this representation to be agnostic to perceptual features of the stimulus. To test this, we examine models in a cross-stimulus transfer setting: we train a model on one set of stimuli, but test it on other types of stimuli (_i.e.,_ train on _Constant Size_, test on _Constant Circumference_). The test stimuli are _out of distibution (OOD)_ with respect to the model's training distribution. Compared to the _in distribution_ setting where training and test are drawn from the same distribution, the OOD setting is known to be challenging for neural networks (Geirhos et al., 2020).3

\begin{table}
\begin{tabular}{l|r r r r|r r r|r r r r}  & \multicolumn{3}{c}{Few (1, 2)} & \multicolumn{3}{c}{Many (6, 7)} & \multicolumn{3}{c}{Total Error (Few+Many)} \\  & \multicolumn{1}{c}{ResNet} & \multicolumn{1}{c}{ViT} & \multicolumn{1}{c}{Swin} & \multicolumn{1}{c|}{MLP} & \multicolumn{1}{c}{ResNet} & \multicolumn{1}{c}{ViT} & \multicolumn{1}{c}{Swin} & \multicolumn{1}{c}{MLP} & \multicolumn{1}{c}{ResNet} & \multicolumn{1}{c}{ViT} & \multicolumn{1}{c}{Swin} & \multicolumn{1}{c}{MLP} \\ \hline Vary Size & 3.0 & 11.8 & 0.2 & **17.4** & 3.6 & **11.9** & 0.6 & 7.8 & 3.3 & 11.8 & 0.4 & **12.6** \\ Const. Size & 0.0 & **35.5** & 0.0 & 32.1 & 0.0 & 0.3 & 0.1 & **5.0** & 0.0 & 17.9 & 0.0 & **18.6** \\ Const. Area & 1.1 & **33.0** & 0.0 & 7.5 & 0.8 & **7.2** & 0.1 & 2.8 & 0.9 & **20.1** & 0.1 & 5.1 \\ Const. Area (C) & 0.4 & 8.0 & 0.0 & **63.2** & 1.5 & **12.8** & 0.3 & 10.1 & 0.9 & 10.4 & 0.1 & **36.6** \\ Const. Circ. & 0.6 & 0.4 & 0.0 & 3.5 & 0.0 & 3.5 & 0.0 & **51.8** & 0.3 & 1.9 & 0.0 & **27.6** \\ Const. Circ. (C) & 8.7 & **40.8** & 0.7 & 31.1 & 9.9 & 35.0 & 12.8 & **44.6** & 9.3 & **37.9** & 6.8 & **37.9** \\ \hline \end{tabular}
\end{table}
Table 1: Error rates (%) in classifying anchor numerosities as either “few” or “many” on respective test sets. Highest error rates for each stimulus type and each anchor numerosity are highlighted. Highest total error rates across stimuli for each model are underlined.

Figure 2: Psychometric functions for each model trained and tested on one stimulus type on the numerical bisection task. Vertical bars are 95% bootstrapped CIs.
Figures 3A) and 3B) show a selected subset of psychometric curves evaluated using such a cross-stimulus protocol, for models trained on _Vary Size_ and _Constant Circumference_. Orange curves denote cases where a model is trained and tested on the same stimulus category (_i.e.,_ the protocol from Exp. 1), and are included for reference, while blue curves are obtained when models are tested on datasets different from the ones they are trained on. We report results on _Vary Size_ and _Constant Circumference_ since they exhibit the most successful (_Vary Size_) and least successful (_Constant Circumference_) transfer cases, as defined by the expected qualititative characteristics of psychometric curves in Sec. The Numerical Bisection Task. Even for the best matching condition _Vary Size_ (Fig. 3A), we observe a number of _transfer failures_, where a trained model shows poor transfer of number discrimination ability to a novel stimulus category, revealing a failure to abstract the number category.

An interesting case of transfer failures is evident in _all-or-none_ responses, where models unanimously assign either _few_ or _many_ response to all numerosities. This has been observed across all models, and is particularly prominent with models trained on _Constant Circumference_ (Fig. 3 B). In some cases, most notably with MLP and to a smaller extent with ViT, we also observe flatter curves with smaller slopes, resulting from frequent misclassification of _many_ responses as _few_ and vice versa. Finally, we also observe a new response pattern, an _inverted_ psychometric curve, where small numerosities are overwhelmingly assigned label _many_, and the opposite for large numerosities. Fig. 3B) showcases that this pattern is consistent across models trained on _Constant Circumference_. We conjecture that this is due to models latching onto total white area during training, which is inversely correlated with numerosity in _Constant Circumference_.

Next, we consider an easier case of transfer, and examine if exposing models to more a diverse set of stimuli (as opposed to one type of stimulus) can help in learning a better representation of number categories; we train models on all but one stimulus type, and evaluate them on the hold-out stimulus type. Instead of 100 images per number category, a model is seeing 500 images per number category (_i.e.,_ 100 images for a number for each of the 5 training stimulus types). As shown in Fig. 3C), we see that increasing data variability results in more curves that resemble the expected s-shaped curve, especially for ResNet and Swin. However, even then, models failed to generalize to _Constant Circumference (contour)_, confirming again the difficulty of this stimulus category. Overall, we find that Swin and ResNet produce representations that better match observed empirical data even in the more challenging transfer setting. We also observe that the training models on a variety of stimulus types help in generalising to new stimulus.

Finally, we examine if learned number representations form meaningful clusters; to answer this question, we do a forward pass on images from a given stimulus category for two models with lowest error rates (_i.e.,_ ResNet and Swin). For each image, we extract embeddings from the last dense layer of the model, prior to the 2-unit classification head. We use PCA followed by the t-SNE [22] dimensionality reduction method to project high-dimensional embedding vectors (2,048 for ResNet and 768 for Swin) into 2D space. In Fig. 4A) we show one selected example of such a projection, where individual points have been color-coded based on the numerosity of the stimulus image. First, we observe that embeddings cluster in groups based on number, with a greater cluster overlap for subsequent numbers. Second, we observe an ordering of clusters based on numerosities. This type of pattern is observed more often with embeddings from Swin, compared to ResNet embeddings which generally result in less discernible clusters (with the exception of clusters for numerosities 1 and 2). Interestingly, based on visual inspection of the data, we do not find that more distinct projections suggest better performance on the task. For example, while clusters in Fig. 4A) seem to be discernible based on number, the model performs poorly when tested on _Constant Size_, possibly because the classifier does not discriminate based on the dimensions that are discriminable in the embeddings.

Figure 3: Psychometric curves in transfer experiments. Bars are standard errors of the mean. Models are represented across rows, and selected training datasets in columns. Shades of blue indicate the test stimulus, orange curves denote train and test on the same stimulus type.
## 4 Discriminability of Small vs. Large Numbers

Empirical data from humans and animals shows that in the numerical bisection task, it is consistently more difficult to distinguish larger numerosities from each other, compared to smaller ones (Almeida et al., 2007; Emmerton et al., 1997). This observation is likely to be related to a more general finding in numerical cognition that small numbers are processed differently than larger numbers (Dehaene, 1997; Revkin et al., 2008).

We examine whether similar observations can be made for our models' responses, using a measure of discriminability for different pairs of numbers. As an example, for a given model and a pair of numbers (such as 5 and 6), we statistically test if the mean percentage of _many_ responses for images of one number (5) is the same as that of images of the other number (6). Intuitively, the two numbers are harder to discriminate if their mean percentage of _many_ responses are the same. We consider models with smallest test error rates in Experiment 1 (_i.e.,_ResNet and Swin). For a given model and each stimulus type, we consider all possible number pairs (_i.e.,_ all points on average psychometric curves) and perform the Tukey's HSD test for multiple pair-wise comparison of means (with family-wise error rate FWER=.05). This approach is based on the similar statistical tests used with pigeon responses in Emmerton et al. (1997).

In Fig. 4B), we show the breakdown of 18 cases (out of total 228 comparisons) where we failed to reject the null hypothesis across number pairs; intuitively, the models find it difficult to discriminate between these number pairs.4 The figure shows that pairs at the higher end of the numerical range, such as (5, 7) and (5, 6) are frequently indistinguishable which is in contrast to the pairs on the lower end of the numerical scale. We conclude that similar to the empirical data, Swin and ResNet better distinguish numerosities at the lower end of the number range compared to those of the higher end of number ranges. Moreover, this effect is stronger among Swin responses compared to ResNet, suggesting that number representations learned by Swin are more discernible.

Footnote 4: From this plot we excluded 24 comparisons for number pairs (1, 2) and (6, 7) since means within these pairs are the same by the design of networks’ training objective.

## 5 Discussion

Number discrimination is a core aspect of basic numerical competence in humans and animals. We investigate if recent, state-of-the-art neural networks used in computer vision exhibit the capacity of discriminating between small and large quantities. We evaluate these models on the numerical bisection task where models learn to categorize numerosity of sets of items, and we investigate their performance on novel stimuli and novel numerosities.

We find that ResNet and Swin, the two models with vision-specific inductive biases, achieve the smallest errors when categorizing novel stimuli as _few_ or _many_. Psychometric curves of models trained on a wide range of stimuli, as well as those of models trained and tested on the same type of stimuli, often resemble the response curves of animals and humans on the same task. In addition, Swin responses are more discernible for smaller numbers compared to larger numbers, and its internal representations are structured in a way that reflects number category and order. Swin's predecessor, ViT, which is also a transformer-based model, albeit with a weak image-specific inductive bias, has errors on the task that are comparable to or even higher than the basic, substantially smaller MLP baseline. This is surprising considering that performance of ViT is within a few percentage points of Swin performance on different computer vision benchmarks (Liu et al., 2021).

Finally, when controlled for perceptual attributes (_e.g.,_ keeping the total white area constant during training, but varying area during testing), most of these models show poor transfer of the number discrimination skill. This might mean that models latch onto features that, while correlated with number, are considered non-numerical in the literature (Honig & Stewart, 1989; Testolin et al., 2020). When analyzing the internal representations of number in Swin, we find that often, despite poor transfer, number representations are structured in an interpretable way. In other words, although these representations could in theory support number discrimination, we do not observe this in practice. One possible reason for poor transfer might be that the models are trained in a limited data regime, in contrast to humans and animals whose numerical cognition develops gradually in a rich environmental context, and who might be biologically predisposed to represent and process numerical quantities (Dehaene et al., 1998). Future work should explore whether pretraining models on larger and more diverse sets of images would result in a more transferable skill. Finally, we only investigate one specific task--the numerical bisection, and it remains to be explored whether our findings generalize across other perceptual domains.

## 6 Acknowledgments

The authors would like to thank Stephanie Chan for detailed feedback on this manuscript as well as other colleagues at

Figure 4: A) t-SNE projections of _Constant Size_ image embeddings for Swin trained on _Constant Area_. B) Numerosities of image pairs for which there was no significant difference in the percentage of “many” responses.
DeepMind for feedback and discussions that helped improve this work.
