Performance-optimized deep neural networks are evolving into worse
  models of inferotemporal visual cortex
One of the most impactful findings in computational neuroscience over the
past decade is that the object recognition accuracy of deep neural networks
(DNNs) correlates with their ability to predict neural responses to natural
images in the inferotemporal (IT) cortex. This discovery supported the
long-held theory that object recognition is a core objective of the visual
cortex, and suggested that more accurate DNNs would serve as better models of
IT neuron responses to images. Since then, deep learning has undergone a
revolution of scale: billion parameter-scale DNNs trained on billions of images
are rivaling or outperforming humans at visual tasks including object
recognition. Have today's DNNs become more accurate at predicting IT neuron
responses to images as they have grown more accurate at object recognition?
  Surprisingly, across three independent experiments, we find this is not the
case. DNNs have become progressively worse models of IT as their accuracy has
increased on ImageNet. To understand why DNNs experience this trade-off and
evaluate if they are still an appropriate paradigm for modeling the visual
system, we turn to recordings of IT that capture spatially resolved maps of
neuronal activity elicited by natural images. These neuronal activity maps
reveal that DNNs trained on ImageNet learn to rely on different visual features
than those encoded by IT and that this problem worsens as their accuracy
increases. We successfully resolved this issue with the neural harmonizer, a
plug-and-play training routine for DNNs that aligns their learned
representations with humans. Our results suggest that harmonized DNNs break the
trade-off between ImageNet accuracy and neural prediction accuracy that assails
current DNNs and offer a path to more accurate models of biological vision.
Drew Linsley, Ivan F. Rodriguez, Thomas Fel, Michael Arcaro, Saloni Sharma, Margaret Livingstone, Thomas Serre
Performance-optimized deep neural networks are evolving into worse models of inferotemporal visual cortex

###### Abstract

One of the most impactful findings in computational neuroscience over the past decade is that the object recognition accuracy of deep neural networks (DNNs) correlates with their ability to predict neural responses to natural images in the inferotemporal (IT) cortex [1; 2]. This discovery supported the long-held theory that object recognition is a core objective of the visual cortex, and suggested that more accurate DNNs would serve as better models of IT neuron responses to images [3; 4; 5]. Since then, deep learning has undergone a revolution of scale: billion parameter-scale DNNs trained on billions of images are rivaling or outperforming humans at visual tasks including object recognition. Have today's DNNs become more accurate at predicting IT neuron responses to images as they have grown more accurate at object recognition?

Surprisingly, across three independent experiments, we find that this is not the case. DNNs have become progressively worse models of IT as their accuracy has increased on ImageNet. To understand why DNNs experience this trade-off and evaluate if they are still an appropriate paradigm for modeling the visual system, we turn to recordings of IT that capture spatially resolved maps of neuronal activity elicited by natural images [6]. These neuronal activity maps reveal that DNNs trained on ImageNet learn to rely on different visual features than those encoded by IT and that this problem worsens as their accuracy increases. We successfully resolved this issue with the _neural harmonizer_, a plug-and-play training routine for DNNs that aligns their learned representations with humans [7]. Our results suggest that harmonized DNNs break the trade-off between ImageNet accuracy and neural prediction accuracy that assails current DNNs and offer a path to more accurate models of biological vision. Our work indicates that the standard approach for modeling IT with task-optimized DNNs needs revision, and other biological constraints, including human psychophysics data, are needed to accurately reverse-engineer the visual cortex.

## 1 Introduction

The release of AlexNet [8] was significant not only for shifting the paradigm of computer vision into the era of deep learning, it also heralded a new approach for "systems identification," and
approximating the transformations used by neurons in the visual cortex to build robust and invariant object representations. Over the past decade, deep neural networks (DNNs) like AlexNet, which are trained for object recognition on ImageNet [9], have been found to contain units that significantly better fit neural activity in inferior temporal (IT) cortex in non-human primates compared to classic hand-tuned models from computational neuroscience [1]. It was later found that DNN fits with neural data improved as these models grew more accurate at object recognition and began to rival humans on the task [10; 11]. This surprising similarity between such task-optimized DNNs and brains supported the extant theory that object recognition is a core principle shaping the organization of the visual cortex [12], and raised the question of how important the many biological details amassed by visual neuroscience actually are for predicting neural responses in visual cortex. In the years since those findings, deep learning has undergone a revolution of scale, and current DNNs which rival or exceed human accuracy in vision and language are significantly larger and trained with orders of magnitude more data than ever before [13]. Does the object recognition accuracy of a DNN still correlate with its ability to predict IT responses to natural objects?

To answer this question, we turned to Brain-Score [3], the standard approach for benchmarking the accuracy of models at predicting neural activity in visual cortex of non-human primates. In brief, the Brain-Score evaluation method involves linearly mapping model unit responses to neural activity and then evaluating model unit predictions on held-out images. With this approach, we found a consistent trend across three different IT datasets hosted on the official Brain-Score website (Brain-score.org), which reflect neural responses to gray-scale versions of realistic rendered objects and natural images [14; 15; 16]. As DNNs have improved on ImageNet [9] over recent years, they have become progressively less accurate at predicting IT neuron responses to images (Fig. 1) 1

Footnote 1: In [11] it was noted that while there was an overall correlation of 0.92 between DNN accuracy on ImageNet and predicting IT responses, the correlation weakened for state-of-the-art models.

In this study, we investigate two potential explanations for why task-optimized DNNs are turning into poor models of IT. (_i_) The internet data diets of DNNs and the routines used to train them to high accuracy on ImageNet leads them to learn the wrong visual features for explaining primate vision and IT responses to objects [17; 18; 7; 19]. (_ii_) Newer DNNs have become less brain-like, and this problem has been magnified as DNNs have grown larger and ultimately deeper than the visual cortex [20].

Contributions.To understand if the trade-off between ImageNet accuracy and IT predictions tjat we observed (Fig. 1) is due to the training routines and data diets of DNNs or their architectures, we turned to a new set of experimental recordings of IT neuronal responses to high-resolution color images [6]. The experimental images were significantly closer to the statistical distribution of images in ImageNet (Fig. 1), unlike prior studies of IT [14]. The recordings also provided a coarse estimate of which image features drove neuronal activity (Fig. 2), which helped characterize DNN errors in explaining neural responses. As we will show, compared to the recordings used in the official Brain-Score benchmark, such spatially-resolved neural data provide far greater insight into why DNNs are becoming worse models of IT. We adopted the Brain-Score evaluation method to measure the prediction accuracy of 135 DNNs on recordings from medial (ML) and posterior (PL) lateral IT in two different Monkeys. To summarize our findings:

Figure 1: **Deep neural networks (DNNs) are becoming progressively worse models of inferior temporal (IT) cortex as they grow more accurate on ImageNet.** Experimental data shown here is taken from Brain-score.org, and each experiment utilized different stimuli. The 104 dots in each panel depict the ImageNet accuracy and neural prediction accuracy of DNNs, and the grey-shaded region denotes the pareto-front governing the trade-off between these variables.
* We observed the same trade-off we found on Brain-Score.org data (Fig. 1) in each of our recordings: DNNs are becoming less accurate at predicting IT responses as they improve on ImageNet (Fig. 3).
* DNNs trained on ImageNet learn different features than those encoded by IT (Fig. 4), and this mismatch is not helped by training on more internet data, using newer DNN architectures like Transformers, relying on self-supervision, or optimizing for adversarial robustness.
* We successfully broke this trade-off by training DNNs with the _neural harmonizer_[7] and aligning the representations they learn for object recognition with those that are diagnostic for humans.
* We further demonstrate that harmonized DNNs (hDNNs) are not only significantly better at explaining IT responses than any other DNN available, they also generate interpretable hypotheses on the features driving IT neuron responses.

## 2 Methods

Neural recordings.We leveraged recordings of IT neuronal responses from two monkeys [6], which were designed to reveal feature preferences of putative face-selective neurons. The recorded neurons also exhibited selectivity for non-face object features [6]. Recordings were made using chronically implanted 32-channel multi-electrode arrays within the fMRI-defined middle lateral (ML) and posterior lateral (PL) face patches of one monkey (Monkey 1), and ML of another monkey (Monkey 2, Fig. 2a). The activating regions in these areas were mapped (1-3\({}^{o}\)), then each image was shown to the animals after they were cued to fixate at a specific position in space (Fig. 2b). The same images were shown multiple times while the monkeys fixated at a red dot in the center of the screen, which made it possible to derive spatial activity maps of neuronal responses (Fig. 2c). Fixation positions fell in a 16 \(\times\) 16 grid for Monkey 1 and 7 \(\times\) 7 grid for Monkey 2, and the average neuronal activity at each position was taken to generate spatial activity maps. A total of 14 images were shown to Monkey 1, and a different set of 14 images were shown to Monkey 2.

Figure 2: **IT recordings that reveal spatial maps of neuronal responses to complex natural images offer unprecedented insights into their feature selectivity [6].****(a)** Neurons in posterior (PL) and/or medial (ML) lateral IT in two animals were localized using functional magnetic resonance imaging (fMRI), and neural responses to images were recorded using chronically implanted 32-channel multi-electrode arrays. **(b)** The monkeys were rapidly shown each image multiple times for 200ms each time (Monkey 1: \(n=256\), Monkey 2: \(n=49\)). Images were positioned differently each time to measure neural responses to every part of the image. **(c)** This procedure yielded spatially resolved maps of neural activity for an image, revealing the relative importance of different features for the recorded neurons. **(d)** DNNs were fit to these recordings following the Brain-Score evaluation method [11], in which partial least squares decoders were used to find the units in a DNN that provided the best match for neuronal responses to images.
The recordings for Monkey 1 resulted in responses from 32 neurons in ML and 31 neurons in PL. For Monkey 2, we obtained responses from 32 neurons (see Appendix SS2 for more details). Following [6], we binned neuronal responses every 40ms of the recording, from 50ms to 250ms. Within each bin, we calculated the noise ceiling for every neuron, which represents the maximum correlation achievable between any two neurons within that time interval. Noise ceilings for each recording were similar to those reported for standard IT Brain-Score datasets (Appendix SS2). In the main text, we present modeling results from the time bins that exhibited the highest average noise ceiling for each Monkey and recording site. Additional results from other time bins, which are consistent with these findings, can be found in Appendix SS2.

DNNs.We investigated the neural fits of 135 different DNNs representing a variety of approaches used in computer vision today: 62 convolutional neural networks trained on ImageNet [21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47] (CNNs), 23 DNNs trained on other datasets in addition to ImageNet (which we refer to as "DNN extra data") [32, 36, 48], 25 vision transformers [49, 50, 51, 52, 53, 54] (ViTs), 10 DNNs trained with self-supervision [55, 56], and 15 DNNs trained to be robust to noise or adversarial examples [57, 58]. Each model was implemented in PyTorch with the TIMM toolbox ([https://github.com/huggingface/pytorch-image-models](https://github.com/huggingface/pytorch-image-models)) and pre-trained weights. Inference was executed on one NVIDIA TITAN X GPU. Additional model details, including the licenses used for each, are detailed in Appendix SS3.

Neural Harmonizer.It was recently found that DNN representations are becoming less aligned with human perception as they evolve and improve on ImageNet [59, 7, 60]. A partial solution to this problem is the _neural harmonizer_, a training routine that can be combined with any DNN to align its representations with humans without sacrificing performance. Here, we test the hypothesis that aligning DNNs with human visual representations can similarly significantly improve their accuracy in predicting neural responses to images.

Training DNNs with the _neural harmonizer_ on ImageNet involves an additional loss to standard cross-entropy for object recognition. This extra loss forces a model's gradients to appear as similar as possible to feature importance maps derived from human behavioral experiments (see [7] for details). To implement this loss, let \(\mathcal{P}_{i}(.)\) be a function that a multi-scale Gaussian pyramid of a human feature importance map \(\mathbf{\phi}\) to \(N\), with \(i\in\{1,...,N\}\). During training, we seek to minimize \(\sum_{i}^{N}||\mathcal{P}_{i}(\mathbf{g}(\mathbf{f}_{\theta},\mathbf{x}))-\mathcal{P}_{i} (\mathbf{\phi})||^{2}\) in order to align feature importance maps between humans and DNNs at every scale of the pyramid. Before they are compared, feature importance maps from humans and DNNs are normalized and rectified using \(\mathbf{z}(.)\), a function that transforms a feature importance map \(\mathbf{\phi}\) from either source to have 0 mean and unit standard deviation. This procedure yields the complete neural harmonization loss:

\[\mathcal{L}_{\text{Harmonization}}= \lambda_{1}\sum_{i}^{N}||(\mathbf{z}\circ\mathcal{P}_{i}\circ\mathbf{g}( \mathbf{f}_{\theta},\mathbf{x}))^{+}-(\mathbf{z}\circ\mathcal{P}_{i}(\mathbf{\phi}))^{+}||_{2} \tag{1}\] \[+\mathcal{L}_{CCE}(\mathbf{f}_{\theta},\mathbf{x},\mathbf{y})+\lambda_{2}\sum _{i}\theta_{i}^{2} \tag{2}\]

Following the original _neural harmonizer_ implementation [7] and training recipe, we trained six DNNs on ImageNet and human feature importance maps from the _ClickMe_ game [61]: VGG16 [25], LeViT [62], ResNetV2-50 [63], EfficientNet_b0 [22], ConvNext [32], and MaxViT [64]. Each model was trained with Tensorflow 2.0 on 8 V4 TPU cores with all of the images in the ImageNet training set, and _ClickMe_ human feature importance maps for the 200,000 images which were annotated. Images and feature importance maps were augmented with mixup [65], random left-right flips, and random crops during training. Only the object recognition loss was computed for images without human feature importance maps. Model weights were optimized with stochastic gradient descent and momentum, batches of 512 images, label smoothing [66], a learning rate of \(0.3\), and a learning rate schedule that began with a 5-epoch warm-up period followed by a cosine decay over 90 epochs at steps 30, 50 and 80.

Neural fitting.We evaluated the neural fit of each model by computing their fit separately for each IT recording, using the Brain-Score evaluation method [11]. This method involved fitting image-evoked activities from a layer of a deep neural network (DNN) to the corresponding neural responses using the partial least squares regression algorithm from Scikit-Learn.
To implement the Brain-Score evaluation method on the spatially-resolved recordings we investigate here, we first pass the image through the network and split the resulting feature map into an equal number of patches as there were fixation locations in the image shown to the animal. Each image patch captured the receptive field of recorded neurons, and in total, there were \(4,046\) image patches for ML and \(4,046\) image patches for PL in Monkey 1, and \(1,134\) image patches for ML in Monkey 2. We measured DNN neuronal fits as the Spearman correlations between model predictions and true neural responses for each patch of an image held out of training divided by each neuron's noise ceiling. We then stored the median Spearman correlation over neurons and repeated the training/testing procedure to get the mean correlation across all images viewed by a monkey. Separate fitting procedures were performed for every layer of activities in a model, and we report a DNN's Brain-Score as its best possible fit across layers.

## 3 Results

Task Optimization is insufficient for reverse-engineering IT.Ever since it was found that DNNs optimized for object recognition produce accurate predictions of IT neural responses to images, it was suggested that prediction accuracy would continue improve alongside DNN performance on ImageNet [10; 11; 12]. Is this the case with today's DNNs that rival or exceed the performance of human object recognition?

To answer this question, we leverage recordings of neuronal responses to high-resolution natural images in medial (ML) and posterior lateral (PL) IT (see Methods and Fig. 2). These images fall within the same statistical distribution as ImageNet images (Appendix SS1), ensuring that our findings are not influenced by distributional shifts that are a well-known problem faced by computer vision models [67]. Image-evoked neural responses were spatially mapped, enabling insights into the visual features driving the responses of IT neurons and DNNs. Moreover, while these regions were localized according to their selectivity to face stimuli, they were also noted to respond to non-face stimuli [6].

Across 135 different DNNs, representing the variety of approaches used today in computer vision, we found that DNNs pretrained on ImageNet have become progressively less accurate at predicting ML and PL responses in two separate Monkeys (Fig. 3). For instance, ConvNext tiny, which achieved 74.3% accuracy on ImageNet, is as accurate in predicting neural responses to images as the ResNetv2-152x4, which reached 84.9% accuracy on ImageNet. Moreover, training routines that have been suggested to be more biologically plausible or yield representations that are closer to biological vision, such as training with self-supervision [68] or for adversarial robustness [69], made no difference. All DNNs trained on internet data faced a pareto-front that bounded their accuracy in predicting IT responses as a function of their ImageNet accuracy.

DNNs need biologically-aligned training routinesThere are at least two potential reasons why ImageNet-trained DNNs face a trade-off between object recognition and neural prediction accuracy:

Figure 3: **DNNs trained on ImageNet face a trade-off between achieving high object recognition accuracy and predicting responses to natural images in IT..** We measured the accuracy of 135 DNNs at predicting image-evoked responses from neurons in posterior lateral (PL) and medial-lateral (ML) areas of IT [6] by computing the neural predicity of each model with the Brain-Score evaluation method [11]. DNN neural predictivity is progressively worsening as models improve on ImageNet. This problem can be partially fixed by training DNNs with the _neural harmonizer_, which aligns their learned object representations with humans. Error bars denote 95% bootstrapped confidence intervals.
(_i_) DNN data diets and training routines work well for ImageNet but lead them to learn features that are misaligned with biological vision [7, 17, 18, 19] or (_ii_) their architectures are a poor match to visual cortex [20]. To test this first possibility, we turned to the _neural harmonizer_. Given prior work demonstrating that aligning DNNs with human perception using the _neural harmonizer_ significantly improved their ability to predict human behavior, we reasoned that harmonization might similarly improve DNN explanations of neural data by forcing them to rely on human-like visual features. Indeed, we found that harmonized DNNs (hDNNs) were significantly more accurate at predicting image-evoked responses in ML (\(T(5)=5.30\), \(p<0.01\)) and PL (\(T(5)=6.11\), \(p<0.001\)) neurons of Monkey 1 and ML (\(T(5)=6.89\), \(p<0.001\)) neurons of Monkey 2 (reported as the average of \(T\) score from independent samples \(t-\)tests comparing the predictivity of each hDNN to the DNN with the highest neural-predictivty). The success of hDNNs indicates that the architectural mismatch between DNNs and the visual cortex is far less of a problem for modeling the visual cortex than the training routines and data diets that are being used today to achieve high accuracy on ImageNet. Behavior - even from humans - is an important constraint on DNNs that is needed to build more accurate models of primate visual cortex.

Spatially-mapped neural responses reveal a feature mismatch between IT and DNNsThe IT recordings used in the Brain-Score benchmark all utilized the same paradigm, in which hundreds or thousands of images were shown to multiple animals during passive viewing [14]. In contrast, the recordings we rely on involve many fewer unique images, but neural responses for each image are spatially mapped. This spatial mapping reveals what types of features are driving neural responses and makes it easier to understand the successes and failures of DNNs in explaining these data [6].

Although the neurons in ML and PL were located based on their selectivity to faces, the spatial maps of their responses revealed a much more complex response profile. IT neurons in both animals were strongly driven by faces, non-face objects, and contextual cues associated with faces [6]. In contrast, the best-fitting units of DNNs with state-of-the-art accuracy on ImageNet, like the ResNetv2-152x4

Figure 4: **DNNs optimized for object recognition on ImageNet rely on different features than those encoded by neurons in primate inferior temporal (IT) cortex..** The activity of PL IT neurons is plotted next to the predicted activity of a model representing each class of DNNs: harmonized DNNs (hDNNs), visual transformers, self-supervised DNNs, convolutional neural networks, DNNs trained on more data than ImageNet, and adversarially robust DNNs.
(Fig. 4), responded strongly to background features. This problem was shared by DNNs trained on internet data but with routines that have been considered more biologically plausible, like self-supervised learning (e.g., SSL ResNet-18), or routines that yield perceptual robustness that is closer to humans, like adversarial training (e.g., Robust ResNetv2-50). hDNNs like the harmonized MaxViT, on the other hand, were, in general, reliably predictive of what parts of images elicited the most activity from IT neurons.

hDNNs generate testable predictions for visual neuroscienceThe surprising effectiveness of DNNs pre-trained on object recognition for predicting IT responses to images was a significant finding in computational neuroscience, which raised the tantalizing possibility that these models could form the basis of neuroprosthetics, and reduce the need for animal experiments. However, a persistent problem with DNNs since their inception is their lack of interpretability, meaning that using DNNs for systems identification effectively swaps one black box (the visual system) with another (a DNN) without getting the field any closer to understanding how vision works. Recent strides in the field of explainable artificial intelligence (XAI) have begun to alleviate this problem. For instance, it is now possible to reliably locate and characterize the features in datasets, like ImageNet, that drive patterns of model behavior using concept recursive activation factorization (CRAFT [70]). When paired with accurate models of IT and recordings that reveal locations in images that elicit neuronal responses, CRAFT can generate testable predictions of _what_ those features are.

We used CRAFT to extract features from harmonized and regular ResNet-50s trained on ImageNet that explain their predictions of neural activity in PL of Monkey 1 (Fig. 5). To do this, we decomposed a DNNs predicted activities for image patches into the most important and distinct features using non-negative matrix factorization, then scaned over every single image patch to find those which most strongly explained each discovered feature. While the harmonized ResNet-50 predicted that parts

Figure 5: **Predictions of the visual features drive neuronal responses in PL of Monkey 1 from a harmonized ResNet-50 and a standard ResNet-50.** Image patches for each model depict what the important features are across all images shown to the animal. The relative importance of each feature for predicting the recordings is color-coded in the bar chart on the right.
of faces, arms, and heads explained the majority of variance in IT, the less accurate and unharmonized ResNet-50 predicted that the background, oriented-edges, and a feature combining head, body, arm, and hand parts were most important for IT. Thus, hDNNs not only break the trade-off between ImageNet and neural prediction accuracy of ImageNet-trained DNNs, they can also generate testable hypotheses on the relative importance of different features for visual system neurons.

## 4 Related work

The scaling laws of deep learningThe immense gains of DNNs in computer vision [13] and natural language processing [71] over recent years share a common theme: they have relied on unprecedented computational resources to train models with billions of parameters on internet-scale datasets. These benefits of scale have been studied over the past several years across a number of domains and often yield predictable improvements on standard internet benchmarks [72, 71]. Surprisingly, scale has also caused models to exhibit human-like behavior in psychophysics experiments [73, 74, 13, 57]. In other words, many aspects of primate behavior _are_ captured progressively better by DNNs as they scale-up and improve in accuracy on benchmarks like ImageNet.

Scale is sufficient for modeling internet benchmarks, but not primate intelligenceIn parallel, there's a growing body of research suggesting that large-scale DNNs are becoming less aligned with human perception in multiple ways. For instance, the representations and visual decision-making behavior of DNNs are inconsistent with humans, and this problem has worsened as they have improved on ImageNet [7]. DNNs are also becoming less capable at predicting perceptual similarity ratings of humans, and they remain vulnerable [75] to so-called "adversarial attacks." Our work adds to this body of research, indicating that current scaling laws are incompatible with modeling primate behavior _and_ poorly suited for explaining the neural computations that shape it.

Aligning DNNs with primate visionThere have been a number of methods proposed for aligning DNNs with primate vision beyond the _neural harmonizer_ that we leverage here [7]. It was found that co-training DNNs for object recognition and minimizing representational dissimilarity improved the adversarial robustness of a recurrent DNN [76]. Others have shown that similar forms of representational alignment improve few-shot learning and predictions of human semantic judgments in DNNs [77, 78]. The successes of these behavioral and representational alignment methods highlight the real limitations of current DNN training routines and data diets for generating artificially intelligent systems that act like biological ones.

Biological learning routinesThere have been many efforts to align the object functions, learning rules, and data diets of DNNs more closely with biological systems. It was found that DNNs trained with self-supervision on ImageNet instead of supervised recognition achieve similar accuracy in predicting V1, V4, and IT responses to images [68]. DNNs trained with self-supervised learning on head-mounted camera video from infants instead of ImageNet were approximately as accurate at predicting neural data as DNNs trained on Imagenet but needed far less data to do this [68]. Others have found that adversarial training of DNNs yields models that are more accurate at predicting neural responses to images [79] and have similar tolerance to adversarial attacks as neurons in IT [69].

## 5 Discussion

A revised approach for reverse engineering visual cortexBiological data is expensive to gather and often noisy. This makes the prospect of accurately modeling the responses of IT to complex stimuli especially daunting. DNNs optimized for object recognition represented a potential solution for this problem: pretraining on internet datasets alleviated training issues associated with small-scale neural data, and the architectures and training routines of accurate DNNs could offer insights into the circuits that underlie visual perception as well as the developmental principles that shape those circuits. Our findings suggest that while DNNs still hold great potential for predicting image-evoked responses from IT neurons, new training routines and data diets are necessary for continued improvement.

The _neural harmonizer_ is a partial solution to the problems that DNNs face in modeling primate IT. The success of hDNNs in breaking the pareto-front faced by 135 different DNNs indicates that significant aspects of primate perception and the neural circuits that shape it cannot be divined from
internet data alone. We believe that the _neural harmonizer_ and learning constraints provided by large-scale human behavior data is only a short-term solution to this problem, and that if DNNs were able to learn about the visual world more like primates do, they would be even better at predicting neural data. In support of this goal, we release our code and data at [https://serre-lab.github.io/neural_harmonizer](https://serre-lab.github.io/neural_harmonizer).

Limitations.One limitation of our work is that we use the responses of neurons with face-selectivity to replicate and understand the trade-off between ImageNet accuracy and neural predictivity faced by DNNs. As faces (and even humans) are not a category in ImageNet, it is possible that this dataset [6] could bias our results in ways that are difficult to predict2. However, we find the same ImageNet accuracy and neural predictivity trade-off on this dataset (Fig. 3) as we did on the three recordings of object selective neurons hosted on the Brain-Score website (Fig. 1), indicating that DNNs face similar issues in predicting each set of recordings. Moreover, the neural harmonizer was successful in breaking this trade-off, even though it involves ImageNet training using human feature importance maps for objects. Finally, it was noted in the original paper where our recordings came from that the neurons, while localized based on their face-selectivity, responded to non-face stimuli as well [6]. In summary, our work reliably demonstrates that new paradigms are needed to advance DNNs as models of IT, and spatially-resolved recordings such as those used in this work support this goal.

Footnote 2: Note, however, that there are many animal and human faces in Imagenet images, giving DNNs ample opportunity to learn about them.

Another limitation of our work is that while we found that hDNNs are significantly more predictive of IT neuron responses than any other DNN, they still explain only 50-60% of the variance in neuronal activity. One straightforward way of doing better is by expanding the dataset of human feature importance maps we used for harmonization from annotations for approximately 200,000 images to the entire 1.2M ImageNet dataset.

Broader impacts.By building better models of primate IT, we are taking significant steps toward the reducing the reliance of visual neuroscience on animal models for experimentation, supporting the development of neuroprosthetic devices that resolve visual dysfunctions, and providing vision scientists with a richer understanding of how IT works. Our findings also highlight a main limitation of the scaling laws that are guiding progress throughout artificial intelligence today: scale is not sufficient for explaining biological intelligence.

## Acknowledgments and Disclosure of Funding

This work was supported by ONR (N00014-19-1-2029), NSF (IIS-1912280 and EAR-1925481), DARPA (D19AC00015), NIH/NINDS (R21 NS 112743), and the ANR-3IA Artificial and Natural Intelligence Toulouse Institute (ANR-19-PI3A-0004). Additional support provided by the Carney Institute for Brain Science and the Center for Computation and Visualization (CCV). We acknowledge the Cloud TPU hardware resources that Google made available via the TensorFlow Research Cloud (TFRC) program as well as computing hardware supported by NIH Office of the Director grant S10OD025181.
