Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

These results reflect first steps in modeling not just primate adult visual processing during inference, but also how the ventral visual stream might be "wired up" by evolution (a model's "birth" state) and by developmental learning (a model's updates based on visual experience).

The evidence to be evaluated is:We evaluate all models on a suite of ventral stream benchmarks in Brain-Score (Schrimpf et al., 2018; Schrimpf et al., 2020), and we base the new models presented here on the CORnet-S architecture, one of the most accurate models of adult primate visual processing (Kubilius et al., 2019).

All benchmarks feed the same images to a candidate model that were used for primate experiments while "recording" activations or measuring behavioral outputs. Specifically, the V1 and V2 benchmarks present 315 images of naturalistic textures and compare model representations to primate single-unit recordings from Freeman et al. (2013) (102 V1 and 103 V2 neurons); the V4 and IT benchmarks present 2,560 naturalistic images and compare models to primate Utah array recordings from Majaj et al. (2015) (88 V4 and 168 IT electrodes).

One of the current best model architectures on the Brain-Score benchmarks is CORnet-S (Kubilius et al., 2019), a shallow recurrent model which anatomically commits to ventral stream regions. CORnet-S has four computational areas, analogous to the ventral visual areas V1, V2, V4, and IT, and a linear decoder that maps from neurons in the model's last visual area to its behavioral choices. The recurrent circuitry (Figure 3B) uses up- and down-sampling convolutions to process features and is identical in each of the models visual areas (except for V1\({}_{COR}\)), but varies by the total number of neurons in each area.

Concretely relating supervised updates to primate ventral stream development, Seibert (2018) establishes that no more than \(\sim\)4 months - or 10 million seconds - of waking visual experience is needed to reach adult-level primate IT cortex (as assessed by its capability to support adult level object recognition). From this estimate, we can compute how many supervised updates per second different models in Figure 1A would require (assuming those updates are evenly distributed over the 10 million seconds). For instance, the fully trained model's 55 million supervised updates translate to 5.5 updates every second, whereas the model with 1 million updates and 76% relative brain predictivity translates to one labeled image update every 10 seconds which appears more plausible given the upper limit of 2-3 saccades per second in humans (Yarbus, 1967; Gibaldi & Sabatini, 2020).

In relation to biology and following the introduced framework of treating all consecutive training as developmental learning, these "at-birth" synaptic connections would result from information encoded in the genome as a product of evolution. Due to the genome's capacity bottleneck, it is infeasible to precisely encode every synapse. Primary visual cortex alone contains \(\sim\)1.4E8 neurons per hemisphere (Leuba & Krafts, 1994), \(\sim\)1E3 synapses per neuron, each requiring \(\sim\)37 bits per synapse (Zador, 2019). Thus, without any clever rules, specifying the connections in only one hemisphere of V1 could require up to \(\sim\)5.2E12 bits - orders of magnitude more than the entire genome's 1GB = 8E9 bits (Zador, 2019).

To test the hypothesis that the genome might already encode more powerful initial representations with synaptic wiring sampled from distributions specified by only few bits, we explored multi-dimensional distributions as a more expressive alternative. These distributions only require a small number of parameters, but unlike current generic initializers, we explicitly specify them for each layer. To determine the right parameterization, we compress a trained model's weights into clusters which we then sample from ("_Weight Compression, WC_"). More specifically, for all convolutional layers except the first, we cluster the kernel weights and later sample from the clusters.

For the first convolutional layer only, we employ a Gabor prior on the weights following studies in V1 (Hubel and Wiesel, 1962; Jones and Palmer, 1987) (Appendix B).

Applying _WC_ to CORnet-S, we first obtain a compressed and clustered set of parameters, from which we sample entirely new weights to yield a new model CORnet-S\({}_{\text{WC}}\). This model is _not trained at all_ and we only evaluate the goodness of its initial wiring on the suite of Brain-Score benchmarks.

_Weight Compression_ explores the hypothesis that evolution may have discovered an initialization strategy with improved at-birth representations (relative to current initializations). _WC_ is most likely not how evolution found the at-birth synaptic connections, but shows that with nearly identical capacity, an alternative initialization distribution leads to networks that are more brain-like in their adult state - revealing a new space of possibilities (hypotheses) that should be considered (see Appendix B.4 for more details on biological plausibility). These findings further suggests that matching representations in higher visual regions might be especially dependent on visual experience whereas early visual regions might already be reasonably well specified without experience.

While improved "at-birth" connectivity can reach 54% of a fully-trained model's score, additional experience dependent updates appear necessary to reach higher predictivities.

As a proof-of-principle that training a reduced set of layers can retain high performance, we propose a novel thin training technique, which we term _Critical Training (CT: Figure 3A). CT_ updates only the weights in critical layers, instead of updating every single model synapse.

We explore successive variants of applying _CT_ up to a block in the architecture and then training the following blocks, e.g. freezing V1, V2, V4 with critical training of the respective down-sampling layers and additional IT training. The final _CT_ ventral stream model is almost completely frozen and only the synapses generating each cortical area's output are trained.

However, while the average score with _DT_ (gray) already drops below 65% with over a quarter of trained parameters remaining and _BN_ drops to 62% with very few parameters, _CT_ (blue) maintains over 75% with only 1.4 out of 52.8 million parameters trained.

By reducing the number of trained parameters, _Critical Training_ also yields engineering benefits with more than 40% of the ImageNet score maintained at \(<3\%\) of parameters trained - a significant improvement over 22% accuracy with _BatchNormNorm_(Frankle et al., 2021) while adding only a small number of additional weights.

High brain predictivity can be achieved with a relatively small number of supervised synaptic updates

Testing synergies between these strategies, we combined all three methods to build novel models that only require a small number of supervised synaptic updates to reasonably capture the mechanisms of adult ventral visual stream processing and object recognition behavior.

_WC+CT_ yield strong benefits (green numbers) in a regime with few supervised updates, improving by up to 27 percent points when training for only 1 epoch on 1,000 images.

_WC+CT_ therefore most positively interact with a small budget of supervised updates (which is the focus of this work).

We developed a range of models with neural and behavioral scores approaching those of the current leading model of the adult ventral visual stream as quantified in Brain-Score, while requiring only a fraction of supervised synaptic updates. These models were built by complementarily 1) reducing the number of supervised updates, i.e. training epochs and labeled images; 2) improving the "at birth" distribution of synaptic connectivity; and 3) training only critical synapses at the end of each model area. The techniques and resulting models proposed here are first steps to more closely modeling not just adult primate visual processing, but also exploring the underlying mechanisms of evolution and developmental learning. These proof-of-principle demonstrations are far from accounting for the rich information encoded in the genome or the developmental learning that together result in adult mechanisms of visual processing, and require further experimental validation.

The fact that early visual areas converge earlier during training (Figure 1) and are better predicted than higher areas by _WC_ initialization is consistent with developmental studies of the primate ventral stream. In humans, behaviors that rely on low-level spatial and temporal processing of visual inputs reach adult-like performance considerably earlier than complex visual behaviors that rely on higher cortical regions, such as face perception (Ellenberg et al., 1999; Grill-Spector et al., 2008). A critical component in more closely modeling primate development is to reduce the dependence on labels altogether.

The changes to model initialization and training presented here serve as a proof-of-principle that models can be changed to more closely align with primate development by reducing training steps with labeled images and improving initialization.

We expect future work in this direction to further close the gap with improved evolutionarily encoded wiring mechanisms and developmental learning rules.

For the very first layer of size \(7\times 7\) we found a Gabor filter most effective following studies in V1 (Hubel and Wiesel, 1962; Jones and Palmer, 1987). To generate the Gabor kernels we fit trained channel weights to a Gabor function

Weight Compression (WC)'s cluster-based initialization generates an "at-birth" network that already captures some useful aspects of the visual inputs in a compressed manner and allows the system to learn faster.

However, independent of _how_ these clusters are determined, WC shows that it is possible to encode certain priors about a system's wiring diagram in a very compressed manner. Evolution also acquired aspects of the visual inputs (with a different strategy) and encoded them in the genome in a lower information regime. For example, we know that there are multiple neuro-developmental mechanisms, such as spontaneous retinal waves and axon guidance cues, that significantly shape the architecture and function of the visual system requiring no (or very little) visual experience ((Huberman et al., 2008)). These mechanisms depend on a relatively small number of proteins encoded in a genome and give rise to a highly complex pattern of at-birth synaptic connectivity representing a very large compression of information.

A model initialized with weight compression achieves (only _WC_) 54% brain predictivity score with 0 supervised synaptic updates. Figure 8B and C show detailed brain predictivity scores, relative to a fully trained model, for models initialized and trained with _WC+CT_ (B) and models initialized with standard Kaiming Normal and training all weights (C) when trained with a range of epochs and labeled images.

Notably, ImageNet performance of these networks seems to not be predictive of their brain predictivity, since even untrained networks with at-chance ImageNet performance correspond reasonably well to e.g. V1 (Figures 1 and 2).

For both transfer methods we initialize the first layer using the Gabor method based on CORnet-S's mixture-of-Gaussian distribution. Since the Gabor function is scalable we can produce Gabor kernels of varying size.

* **KN** A model initialized by standard _Kaiming Normal_ initialization without training.

* WC A model initialized by our _Weight Compression_ initialization, described in Section 4, without training.

* KN+CT The _KN_-initialized model trained with _Critical Training_ until convergence, i.e. three downstream layers and the decoder are trained and all other layers remain unchanged.

* **WC+CT** The _WC_-initialized model with _Critical Training_.

Input: Figure 1: High scores in brain predictivity can be achieved with few supervised updates (log xaxes). A Average brain predictivity of models trained with a range of supervised updates (different combinations of epochs × images). Each dot is a different hypothesis of how the ventral visual stream might have developed and shows the adult brain-likeness score that is achieved by that model. Fairly brain-like representations are already realized with few supervised updates, relative to a fully trained model (black dot; see also Figure 6). Standardly trained CORnet-S is set to be 100% score in brain predictivity on the benchmarks. A “pixels” baseline achieves 7% of this brain predictivity score. B Individual brain predictivity scores over epochs. Models start to approximate the primate ventral stream with few training epochs. Lower visual areas (V1, V2) are approximated earlier in training. C Like B, but number of training images instead of epochs. Few images are sufficient to approximate especially early visual areas.

Input: Figure 7: Alternative weight compression methods Comparison of different initializations that compress weights, “at birth” i.e. without any training (gray) and after training critical layers (shades of blue) for 6 epochs. Our best clustering-based approach WC achieved similar results as the Mixture Gaussian approach (∼3 percent points mean difference) but leads to more diverse clusters. Performance drops when solely sampling weights from kernel based normal distributions (Kernel normal) and additionally disabling the Gabor prior (No Gabor prior)

Input: Figure 4: Benefits of combining training reductions for high brain predictivity with few supervised synaptic updates and limitations of techniques in high-update regimes. A (log x-axis) By reducing updates with a combination of fewer supervised updates (Figure 1), improved initialization WC (Figure 2), and training only down-sampling layers CT (Figure 3), the resulting models (dark blue dots; fewer supervised updates alone in light blue) maintain high brain predictivity scores while requiring only a fraction of supervised synaptic updates compared to standard CORnet-S (black dot, top right). B Comparison between WC-initialized models trained with CT versus standardly initialized models training all weights, when varying training epochs and labeled images. Colors represent their percent point difference in brain predictivity scores. WC+CT improve performance in regimes with few epochs and images, but fall behind in regimes with many labeled images and epochs.

Input: Figure 9: Individual brain benchmark scores for WC+CT model A Individual brain predictivitiy scores of WC+CT models trained with a range of epochs on all images. These models score especially high on V1, V2 and V4 already after one epoch in comparison to a model with standard initialization training all weights. IT and Behavior benchmarks continuously improve over later epochs as well but fall short of a fully trained model. B Like A, but with models trained until convergence on different numbers of labeled images, up to the full dataset of 1.28M images (rightmost points). As in A we see > 80% V1, V2 and V4 scores with only 100,000 images. For comparable IT and behavioral scores, more images are required.

Input: Figure 2: “At-birth” synaptic connectivity yields reasonable scores in brain predictivity. A Specifying the initial weight distribution: Kaiming Normal (KN, He et al., 2015) samples from a generic Gaussian. Weight Compression (WC) compresses trained weights into low-parameter clustered distributions that weights can be initialized from. B Visualization of WC compressed parameters: Gabor filters for first layer and cluster centers for following layers with kernel size > 3. The cluster centers capture an intuitive variety of kernel types. C “At-birth” representations with WC achieve 54% score in brain predictivity of a fully trained model, with no training at all. Scores after training remain virtually unchanged (shaded bars).

Input: Figure 8: Detailed analysis of WC+CT. A When reducing the number of supervised synaptic updates, adding critical training (dark grey) and adding weight compression initialization (dark blue) both improve scores in brain predictivity at the same number of supervised synaptic updates, in comparison to a model with standard initialization and training all weights (bright blue). B Brain predictivities for the WC+CT model when trained with a range of epochs and labeled images. C Same as B, but for a standardly initialized (KN) model training all weights.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find this to be a relatively well-stated claim, though with some minor overstatement. The evidence strongly supports that these are "first steps" in modeling adult visual processing, with detailed benchmarking against primate data and clear improvements in initialization and training efficiency. The "wiring up by evolution" aspect is supported by the Weight Compression method showing improved initialization can approximate early visual areas without training. However, the authors themselves acknowledge these are "proof-of-principle demonstrations far from accounting for the rich information encoded in the genome," making this aspect appropriately qualified as "first steps."

<score>2</score>

Justification:
1. The phrase "first steps" appropriately frames this as early work rather than definitive modeling
2. Evidence demonstrates concrete advances in modeling adult visual processing:
- Detailed benchmarking against primate neural/behavioral data
- Weight Compression showing 54% brain predictivity without training
- Critical Training maintaining 75% performance with minimal parameters
3. The evolutionary/developmental claims are appropriately qualified:
- Authors explicitly state these are "proof-of-principle demonstrations"
- Acknowledge limitations in accounting for genomic information
- Support claims with relevant biological constraints (e.g., genome capacity)
4. Only minor overstatement in implying direct modeling of evolutionary processes, but this is tempered by careful language about "first steps"


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

These results reflect first steps in modeling not just primate adult visual processing during inference, but also how the ventral visual stream might be "wired up" by evolution (a model's "birth" state) and by developmental learning (a model's updates based on visual experience).

The review comment to be evaluated is:

Strengths:
- The paper is clear and well written
- The analysis is thorough and supports the authors' arguments
- The method of initializing weights via clusters of trained kernels is interesting
- The "critical training" method of training only selected layers is also interesting

Weaknesses:
- "Recently, such models have been criticized due to how their learning departs from brain development (Marcus, 2004; Grossberg, 2020; Zador, 2019)." This seems to be the motivation for the work, but those authors don't raise the issues that are addressed in this paper. The Marcus book predates deep learning. Grossberg focuses on catastrophic forgetting. Zador focuses on the sparsity of labelled data in life. I am not sure whether the number of weight updates needed to train deep networks presents a substantial independent concern about their biological plausibility, particularly given the possibility of constant unsupervised learning. If the issue has been raised in the literature, more specific references would help here. If not, I think the authors should argue more specifically that it is a problem, before going on to solve it.
- The cluster-based initialization method seems to be presented as a way of bringing some learning results through a genetic bottleneck, but the biological plausibility isn't obvious. Further discussion of how this relates to and departs from biology would be helpful.
- It's interesting that critical training worked so well, but in the context of the paper's goals, the biological relevance of this method should also be laid out more clearly.

Minor:
- I don't think it was shown how classification performance is affected by the cluster-based initialization method, but it would be nice to see that.
- "37 bits per synapse (Zador, 2019)" This refers to the binary connectivity pattern (an issue of architecture) rather than the weights (which are addressed in this paper).

The evidence to be evaluated is:We evaluate all models on a suite of ventral stream benchmarks in Brain-Score (Schrimpf et al., 2018; Schrimpf et al., 2020), and we base the new models presented here on the CORnet-S architecture, one of the most accurate models of adult primate visual processing (Kubilius et al., 2019).

All benchmarks feed the same images to a candidate model that were used for primate experiments while "recording" activations or measuring behavioral outputs. Specifically, the V1 and V2 benchmarks present 315 images of naturalistic textures and compare model representations to primate single-unit recordings from Freeman et al. (2013) (102 V1 and 103 V2 neurons); the V4 and IT benchmarks present 2,560 naturalistic images and compare models to primate Utah array recordings from Majaj et al. (2015) (88 V4 and 168 IT electrodes).

One of the current best model architectures on the Brain-Score benchmarks is CORnet-S (Kubilius et al., 2019), a shallow recurrent model which anatomically commits to ventral stream regions. CORnet-S has four computational areas, analogous to the ventral visual areas V1, V2, V4, and IT, and a linear decoder that maps from neurons in the model's last visual area to its behavioral choices. The recurrent circuitry (Figure 3B) uses up- and down-sampling convolutions to process features and is identical in each of the models visual areas (except for V1\({}_{COR}\)), but varies by the total number of neurons in each area.

Concretely relating supervised updates to primate ventral stream development, Seibert (2018) establishes that no more than \(\sim\)4 months - or 10 million seconds - of waking visual experience is needed to reach adult-level primate IT cortex (as assessed by its capability to support adult level object recognition). From this estimate, we can compute how many supervised updates per second different models in Figure 1A would require (assuming those updates are evenly distributed over the 10 million seconds). For instance, the fully trained model's 55 million supervised updates translate to 5.5 updates every second, whereas the model with 1 million updates and 76% relative brain predictivity translates to one labeled image update every 10 seconds which appears more plausible given the upper limit of 2-3 saccades per second in humans (Yarbus, 1967; Gibaldi & Sabatini, 2020).

In relation to biology and following the introduced framework of treating all consecutive training as developmental learning, these "at-birth" synaptic connections would result from information encoded in the genome as a product of evolution. Due to the genome's capacity bottleneck, it is infeasible to precisely encode every synapse. Primary visual cortex alone contains \(\sim\)1.4E8 neurons per hemisphere (Leuba & Krafts, 1994), \(\sim\)1E3 synapses per neuron, each requiring \(\sim\)37 bits per synapse (Zador, 2019). Thus, without any clever rules, specifying the connections in only one hemisphere of V1 could require up to \(\sim\)5.2E12 bits - orders of magnitude more than the entire genome's 1GB = 8E9 bits (Zador, 2019).

To test the hypothesis that the genome might already encode more powerful initial representations with synaptic wiring sampled from distributions specified by only few bits, we explored multi-dimensional distributions as a more expressive alternative. These distributions only require a small number of parameters, but unlike current generic initializers, we explicitly specify them for each layer. To determine the right parameterization, we compress a trained model's weights into clusters which we then sample from ("_Weight Compression, WC_"). More specifically, for all convolutional layers except the first, we cluster the kernel weights and later sample from the clusters.

For the first convolutional layer only, we employ a Gabor prior on the weights following studies in V1 (Hubel and Wiesel, 1962; Jones and Palmer, 1987) (Appendix B).

Applying _WC_ to CORnet-S, we first obtain a compressed and clustered set of parameters, from which we sample entirely new weights to yield a new model CORnet-S\({}_{\text{WC}}\). This model is _not trained at all_ and we only evaluate the goodness of its initial wiring on the suite of Brain-Score benchmarks.

_Weight Compression_ explores the hypothesis that evolution may have discovered an initialization strategy with improved at-birth representations (relative to current initializations). _WC_ is most likely not how evolution found the at-birth synaptic connections, but shows that with nearly identical capacity, an alternative initialization distribution leads to networks that are more brain-like in their adult state - revealing a new space of possibilities (hypotheses) that should be considered (see Appendix B.4 for more details on biological plausibility). These findings further suggests that matching representations in higher visual regions might be especially dependent on visual experience whereas early visual regions might already be reasonably well specified without experience.

While improved "at-birth" connectivity can reach 54% of a fully-trained model's score, additional experience dependent updates appear necessary to reach higher predictivities.

As a proof-of-principle that training a reduced set of layers can retain high performance, we propose a novel thin training technique, which we term _Critical Training (CT: Figure 3A). CT_ updates only the weights in critical layers, instead of updating every single model synapse.

We explore successive variants of applying _CT_ up to a block in the architecture and then training the following blocks, e.g. freezing V1, V2, V4 with critical training of the respective down-sampling layers and additional IT training. The final _CT_ ventral stream model is almost completely frozen and only the synapses generating each cortical area's output are trained.

However, while the average score with _DT_ (gray) already drops below 65% with over a quarter of trained parameters remaining and _BN_ drops to 62% with very few parameters, _CT_ (blue) maintains over 75% with only 1.4 out of 52.8 million parameters trained.

By reducing the number of trained parameters, _Critical Training_ also yields engineering benefits with more than 40% of the ImageNet score maintained at \(<3\%\) of parameters trained - a significant improvement over 22% accuracy with _BatchNormNorm_(Frankle et al., 2021) while adding only a small number of additional weights.

High brain predictivity can be achieved with a relatively small number of supervised synaptic updates

Testing synergies between these strategies, we combined all three methods to build novel models that only require a small number of supervised synaptic updates to reasonably capture the mechanisms of adult ventral visual stream processing and object recognition behavior.

_WC+CT_ yield strong benefits (green numbers) in a regime with few supervised updates, improving by up to 27 percent points when training for only 1 epoch on 1,000 images.

_WC+CT_ therefore most positively interact with a small budget of supervised updates (which is the focus of this work).

We developed a range of models with neural and behavioral scores approaching those of the current leading model of the adult ventral visual stream as quantified in Brain-Score, while requiring only a fraction of supervised synaptic updates. These models were built by complementarily 1) reducing the number of supervised updates, i.e. training epochs and labeled images; 2) improving the "at birth" distribution of synaptic connectivity; and 3) training only critical synapses at the end of each model area. The techniques and resulting models proposed here are first steps to more closely modeling not just adult primate visual processing, but also exploring the underlying mechanisms of evolution and developmental learning. These proof-of-principle demonstrations are far from accounting for the rich information encoded in the genome or the developmental learning that together result in adult mechanisms of visual processing, and require further experimental validation.

The fact that early visual areas converge earlier during training (Figure 1) and are better predicted than higher areas by _WC_ initialization is consistent with developmental studies of the primate ventral stream. In humans, behaviors that rely on low-level spatial and temporal processing of visual inputs reach adult-like performance considerably earlier than complex visual behaviors that rely on higher cortical regions, such as face perception (Ellenberg et al., 1999; Grill-Spector et al., 2008). A critical component in more closely modeling primate development is to reduce the dependence on labels altogether.

The changes to model initialization and training presented here serve as a proof-of-principle that models can be changed to more closely align with primate development by reducing training steps with labeled images and improving initialization.

We expect future work in this direction to further close the gap with improved evolutionarily encoded wiring mechanisms and developmental learning rules.

For the very first layer of size \(7\times 7\) we found a Gabor filter most effective following studies in V1 (Hubel and Wiesel, 1962; Jones and Palmer, 1987). To generate the Gabor kernels we fit trained channel weights to a Gabor function

Weight Compression (WC)'s cluster-based initialization generates an "at-birth" network that already captures some useful aspects of the visual inputs in a compressed manner and allows the system to learn faster.

However, independent of _how_ these clusters are determined, WC shows that it is possible to encode certain priors about a system's wiring diagram in a very compressed manner. Evolution also acquired aspects of the visual inputs (with a different strategy) and encoded them in the genome in a lower information regime. For example, we know that there are multiple neuro-developmental mechanisms, such as spontaneous retinal waves and axon guidance cues, that significantly shape the architecture and function of the visual system requiring no (or very little) visual experience ((Huberman et al., 2008)). These mechanisms depend on a relatively small number of proteins encoded in a genome and give rise to a highly complex pattern of at-birth synaptic connectivity representing a very large compression of information.

A model initialized with weight compression achieves (only _WC_) 54% brain predictivity score with 0 supervised synaptic updates. Figure 8B and C show detailed brain predictivity scores, relative to a fully trained model, for models initialized and trained with _WC+CT_ (B) and models initialized with standard Kaiming Normal and training all weights (C) when trained with a range of epochs and labeled images.

Notably, ImageNet performance of these networks seems to not be predictive of their brain predictivity, since even untrained networks with at-chance ImageNet performance correspond reasonably well to e.g. V1 (Figures 1 and 2).

For both transfer methods we initialize the first layer using the Gabor method based on CORnet-S's mixture-of-Gaussian distribution. Since the Gabor function is scalable we can produce Gabor kernels of varying size.

* **KN** A model initialized by standard _Kaiming Normal_ initialization without training.

* WC A model initialized by our _Weight Compression_ initialization, described in Section 4, without training.

* KN+CT The _KN_-initialized model trained with _Critical Training_ until convergence, i.e. three downstream layers and the decoder are trained and all other layers remain unchanged.

* **WC+CT** The _WC_-initialized model with _Critical Training_.

Input: Figure 1: High scores in brain predictivity can be achieved with few supervised updates (log xaxes). A Average brain predictivity of models trained with a range of supervised updates (different combinations of epochs × images). Each dot is a different hypothesis of how the ventral visual stream might have developed and shows the adult brain-likeness score that is achieved by that model. Fairly brain-like representations are already realized with few supervised updates, relative to a fully trained model (black dot; see also Figure 6). Standardly trained CORnet-S is set to be 100% score in brain predictivity on the benchmarks. A “pixels” baseline achieves 7% of this brain predictivity score. B Individual brain predictivity scores over epochs. Models start to approximate the primate ventral stream with few training epochs. Lower visual areas (V1, V2) are approximated earlier in training. C Like B, but number of training images instead of epochs. Few images are sufficient to approximate especially early visual areas.

Input: Figure 7: Alternative weight compression methods Comparison of different initializations that compress weights, “at birth” i.e. without any training (gray) and after training critical layers (shades of blue) for 6 epochs. Our best clustering-based approach WC achieved similar results as the Mixture Gaussian approach (∼3 percent points mean difference) but leads to more diverse clusters. Performance drops when solely sampling weights from kernel based normal distributions (Kernel normal) and additionally disabling the Gabor prior (No Gabor prior)

Input: Figure 4: Benefits of combining training reductions for high brain predictivity with few supervised synaptic updates and limitations of techniques in high-update regimes. A (log x-axis) By reducing updates with a combination of fewer supervised updates (Figure 1), improved initialization WC (Figure 2), and training only down-sampling layers CT (Figure 3), the resulting models (dark blue dots; fewer supervised updates alone in light blue) maintain high brain predictivity scores while requiring only a fraction of supervised synaptic updates compared to standard CORnet-S (black dot, top right). B Comparison between WC-initialized models trained with CT versus standardly initialized models training all weights, when varying training epochs and labeled images. Colors represent their percent point difference in brain predictivity scores. WC+CT improve performance in regimes with few epochs and images, but fall behind in regimes with many labeled images and epochs.

Input: Figure 9: Individual brain benchmark scores for WC+CT model A Individual brain predictivitiy scores of WC+CT models trained with a range of epochs on all images. These models score especially high on V1, V2 and V4 already after one epoch in comparison to a model with standard initialization training all weights. IT and Behavior benchmarks continuously improve over later epochs as well but fall short of a fully trained model. B Like A, but with models trained until convergence on different numbers of labeled images, up to the full dataset of 1.28M images (rightmost points). As in A we see > 80% V1, V2 and V4 scores with only 100,000 images. For comparable IT and behavioral scores, more images are required.

Input: Figure 2: “At-birth” synaptic connectivity yields reasonable scores in brain predictivity. A Specifying the initial weight distribution: Kaiming Normal (KN, He et al., 2015) samples from a generic Gaussian. Weight Compression (WC) compresses trained weights into low-parameter clustered distributions that weights can be initialized from. B Visualization of WC compressed parameters: Gabor filters for first layer and cluster centers for following layers with kernel size > 3. The cluster centers capture an intuitive variety of kernel types. C “At-birth” representations with WC achieve 54% score in brain predictivity of a fully trained model, with no training at all. Scores after training remain virtually unchanged (shaded bars).

Input: Figure 8: Detailed analysis of WC+CT. A When reducing the number of supervised synaptic updates, adding critical training (dark grey) and adding weight compression initialization (dark blue) both improve scores in brain predictivity at the same number of supervised synaptic updates, in comparison to a model with standard initialization and training all weights (bright blue). B Brain predictivities for the WC+CT model when trained with a range of epochs and labeled images. C Same as B, but for a standardly initialized (KN) model training all weights.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
The claim appears to be moderately overstated based on the available evidence and reviewer comments.

While the evidence shows meaningful progress in modeling visual processing with fewer supervised updates and improved initialization methods, several key limitations exist:

1. The reviewer notes that the biological plausibility of the cluster-based initialization and critical training methods needs more justification and clearer connections to actual brain development.

2. The authors acknowledge these are just "first steps" and "proof-of-principle demonstrations" that are "far from accounting for" actual visual processing development.

3. The evidence demonstrates technical achievements in reducing training requirements while maintaining performance, but does not fully validate the evolutionary and developmental learning implications suggested in the claim.

The core findings are valuable but the broader interpretations about modeling evolution and development appear to reach beyond what the current evidence firmly supports.

<score>4</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

These results reflect first steps in modeling not just primate adult visual processing during inference, but also how the ventral visual stream might be "wired up" by evolution (a model's "birth" state) and by developmental learning (a model's updates based on visual experience).

The review comment to be evaluated is:

### Strength:

First of all, the question addressed in the paper is essential for both the machine learning and neuroscience community and the author aim at addressing in a well-written piece of work that also clearly contextualizes and motivates the issue at hand. This work uses a state-of-the-art neural architecture, instead of a reduced version of models as is often the case in this type of work. The authors also propose a good analysis and clearly lay out their results, although their interpretation is one of the weaknesses of the paper.

### Weakness:

Although it is true that this work has many strengths it is hard to overcome the following weaknesses, which question the motivation of the approach undertaken. First, there exists a set of more principled approaches that have been developed over the last 4-5 years for training ANNs in a biologically plausible way.  Second, the results obtained are very subjective to the authors as the result of the improvement of the score is minor.

Although it is true that we don’t expect the authors to solve how the visual system develops and that this work corresponds to a proof of principle, it is still hard to navigate the plausibility of the assumptions made among all possibilities. On the other hand, there is a large set of work that has been proposed to build biologically plausible deep neural networks starting from clear objective function minimization and enforcing locality of learning rules. Compared to this set of works this one seems more adhoc than principled. This represents an important weakness when addressing the more neuroscience-friendly community.

Indeed the approach proposed is more of a combination of existing approaches rather than a novel idea: the use of training of a subset of layers, different weights initialization based on the distribution of trained weights. It is not clear how this provides new insight to the statistical learning community.

The evidence to be evaluated is:We evaluate all models on a suite of ventral stream benchmarks in Brain-Score (Schrimpf et al., 2018; Schrimpf et al., 2020), and we base the new models presented here on the CORnet-S architecture, one of the most accurate models of adult primate visual processing (Kubilius et al., 2019).

All benchmarks feed the same images to a candidate model that were used for primate experiments while "recording" activations or measuring behavioral outputs. Specifically, the V1 and V2 benchmarks present 315 images of naturalistic textures and compare model representations to primate single-unit recordings from Freeman et al. (2013) (102 V1 and 103 V2 neurons); the V4 and IT benchmarks present 2,560 naturalistic images and compare models to primate Utah array recordings from Majaj et al. (2015) (88 V4 and 168 IT electrodes).

One of the current best model architectures on the Brain-Score benchmarks is CORnet-S (Kubilius et al., 2019), a shallow recurrent model which anatomically commits to ventral stream regions. CORnet-S has four computational areas, analogous to the ventral visual areas V1, V2, V4, and IT, and a linear decoder that maps from neurons in the model's last visual area to its behavioral choices. The recurrent circuitry (Figure 3B) uses up- and down-sampling convolutions to process features and is identical in each of the models visual areas (except for V1\({}_{COR}\)), but varies by the total number of neurons in each area.

Concretely relating supervised updates to primate ventral stream development, Seibert (2018) establishes that no more than \(\sim\)4 months - or 10 million seconds - of waking visual experience is needed to reach adult-level primate IT cortex (as assessed by its capability to support adult level object recognition). From this estimate, we can compute how many supervised updates per second different models in Figure 1A would require (assuming those updates are evenly distributed over the 10 million seconds). For instance, the fully trained model's 55 million supervised updates translate to 5.5 updates every second, whereas the model with 1 million updates and 76% relative brain predictivity translates to one labeled image update every 10 seconds which appears more plausible given the upper limit of 2-3 saccades per second in humans (Yarbus, 1967; Gibaldi & Sabatini, 2020).

In relation to biology and following the introduced framework of treating all consecutive training as developmental learning, these "at-birth" synaptic connections would result from information encoded in the genome as a product of evolution. Due to the genome's capacity bottleneck, it is infeasible to precisely encode every synapse. Primary visual cortex alone contains \(\sim\)1.4E8 neurons per hemisphere (Leuba & Krafts, 1994), \(\sim\)1E3 synapses per neuron, each requiring \(\sim\)37 bits per synapse (Zador, 2019). Thus, without any clever rules, specifying the connections in only one hemisphere of V1 could require up to \(\sim\)5.2E12 bits - orders of magnitude more than the entire genome's 1GB = 8E9 bits (Zador, 2019).

To test the hypothesis that the genome might already encode more powerful initial representations with synaptic wiring sampled from distributions specified by only few bits, we explored multi-dimensional distributions as a more expressive alternative. These distributions only require a small number of parameters, but unlike current generic initializers, we explicitly specify them for each layer. To determine the right parameterization, we compress a trained model's weights into clusters which we then sample from ("_Weight Compression, WC_"). More specifically, for all convolutional layers except the first, we cluster the kernel weights and later sample from the clusters.

For the first convolutional layer only, we employ a Gabor prior on the weights following studies in V1 (Hubel and Wiesel, 1962; Jones and Palmer, 1987) (Appendix B).

Applying _WC_ to CORnet-S, we first obtain a compressed and clustered set of parameters, from which we sample entirely new weights to yield a new model CORnet-S\({}_{\text{WC}}\). This model is _not trained at all_ and we only evaluate the goodness of its initial wiring on the suite of Brain-Score benchmarks.

_Weight Compression_ explores the hypothesis that evolution may have discovered an initialization strategy with improved at-birth representations (relative to current initializations). _WC_ is most likely not how evolution found the at-birth synaptic connections, but shows that with nearly identical capacity, an alternative initialization distribution leads to networks that are more brain-like in their adult state - revealing a new space of possibilities (hypotheses) that should be considered (see Appendix B.4 for more details on biological plausibility). These findings further suggests that matching representations in higher visual regions might be especially dependent on visual experience whereas early visual regions might already be reasonably well specified without experience.

While improved "at-birth" connectivity can reach 54% of a fully-trained model's score, additional experience dependent updates appear necessary to reach higher predictivities.

As a proof-of-principle that training a reduced set of layers can retain high performance, we propose a novel thin training technique, which we term _Critical Training (CT: Figure 3A). CT_ updates only the weights in critical layers, instead of updating every single model synapse.

We explore successive variants of applying _CT_ up to a block in the architecture and then training the following blocks, e.g. freezing V1, V2, V4 with critical training of the respective down-sampling layers and additional IT training. The final _CT_ ventral stream model is almost completely frozen and only the synapses generating each cortical area's output are trained.

However, while the average score with _DT_ (gray) already drops below 65% with over a quarter of trained parameters remaining and _BN_ drops to 62% with very few parameters, _CT_ (blue) maintains over 75% with only 1.4 out of 52.8 million parameters trained.

By reducing the number of trained parameters, _Critical Training_ also yields engineering benefits with more than 40% of the ImageNet score maintained at \(<3\%\) of parameters trained - a significant improvement over 22% accuracy with _BatchNormNorm_(Frankle et al., 2021) while adding only a small number of additional weights.

High brain predictivity can be achieved with a relatively small number of supervised synaptic updates

Testing synergies between these strategies, we combined all three methods to build novel models that only require a small number of supervised synaptic updates to reasonably capture the mechanisms of adult ventral visual stream processing and object recognition behavior.

_WC+CT_ yield strong benefits (green numbers) in a regime with few supervised updates, improving by up to 27 percent points when training for only 1 epoch on 1,000 images.

_WC+CT_ therefore most positively interact with a small budget of supervised updates (which is the focus of this work).

We developed a range of models with neural and behavioral scores approaching those of the current leading model of the adult ventral visual stream as quantified in Brain-Score, while requiring only a fraction of supervised synaptic updates. These models were built by complementarily 1) reducing the number of supervised updates, i.e. training epochs and labeled images; 2) improving the "at birth" distribution of synaptic connectivity; and 3) training only critical synapses at the end of each model area. The techniques and resulting models proposed here are first steps to more closely modeling not just adult primate visual processing, but also exploring the underlying mechanisms of evolution and developmental learning. These proof-of-principle demonstrations are far from accounting for the rich information encoded in the genome or the developmental learning that together result in adult mechanisms of visual processing, and require further experimental validation.

The fact that early visual areas converge earlier during training (Figure 1) and are better predicted than higher areas by _WC_ initialization is consistent with developmental studies of the primate ventral stream. In humans, behaviors that rely on low-level spatial and temporal processing of visual inputs reach adult-like performance considerably earlier than complex visual behaviors that rely on higher cortical regions, such as face perception (Ellenberg et al., 1999; Grill-Spector et al., 2008). A critical component in more closely modeling primate development is to reduce the dependence on labels altogether.

The changes to model initialization and training presented here serve as a proof-of-principle that models can be changed to more closely align with primate development by reducing training steps with labeled images and improving initialization.

We expect future work in this direction to further close the gap with improved evolutionarily encoded wiring mechanisms and developmental learning rules.

For the very first layer of size \(7\times 7\) we found a Gabor filter most effective following studies in V1 (Hubel and Wiesel, 1962; Jones and Palmer, 1987). To generate the Gabor kernels we fit trained channel weights to a Gabor function

Weight Compression (WC)'s cluster-based initialization generates an "at-birth" network that already captures some useful aspects of the visual inputs in a compressed manner and allows the system to learn faster.

However, independent of _how_ these clusters are determined, WC shows that it is possible to encode certain priors about a system's wiring diagram in a very compressed manner. Evolution also acquired aspects of the visual inputs (with a different strategy) and encoded them in the genome in a lower information regime. For example, we know that there are multiple neuro-developmental mechanisms, such as spontaneous retinal waves and axon guidance cues, that significantly shape the architecture and function of the visual system requiring no (or very little) visual experience ((Huberman et al., 2008)). These mechanisms depend on a relatively small number of proteins encoded in a genome and give rise to a highly complex pattern of at-birth synaptic connectivity representing a very large compression of information.

A model initialized with weight compression achieves (only _WC_) 54% brain predictivity score with 0 supervised synaptic updates. Figure 8B and C show detailed brain predictivity scores, relative to a fully trained model, for models initialized and trained with _WC+CT_ (B) and models initialized with standard Kaiming Normal and training all weights (C) when trained with a range of epochs and labeled images.

Notably, ImageNet performance of these networks seems to not be predictive of their brain predictivity, since even untrained networks with at-chance ImageNet performance correspond reasonably well to e.g. V1 (Figures 1 and 2).

For both transfer methods we initialize the first layer using the Gabor method based on CORnet-S's mixture-of-Gaussian distribution. Since the Gabor function is scalable we can produce Gabor kernels of varying size.

* **KN** A model initialized by standard _Kaiming Normal_ initialization without training.

* WC A model initialized by our _Weight Compression_ initialization, described in Section 4, without training.

* KN+CT The _KN_-initialized model trained with _Critical Training_ until convergence, i.e. three downstream layers and the decoder are trained and all other layers remain unchanged.

* **WC+CT** The _WC_-initialized model with _Critical Training_.

Input: Figure 1: High scores in brain predictivity can be achieved with few supervised updates (log xaxes). A Average brain predictivity of models trained with a range of supervised updates (different combinations of epochs × images). Each dot is a different hypothesis of how the ventral visual stream might have developed and shows the adult brain-likeness score that is achieved by that model. Fairly brain-like representations are already realized with few supervised updates, relative to a fully trained model (black dot; see also Figure 6). Standardly trained CORnet-S is set to be 100% score in brain predictivity on the benchmarks. A “pixels” baseline achieves 7% of this brain predictivity score. B Individual brain predictivity scores over epochs. Models start to approximate the primate ventral stream with few training epochs. Lower visual areas (V1, V2) are approximated earlier in training. C Like B, but number of training images instead of epochs. Few images are sufficient to approximate especially early visual areas.

Input: Figure 7: Alternative weight compression methods Comparison of different initializations that compress weights, “at birth” i.e. without any training (gray) and after training critical layers (shades of blue) for 6 epochs. Our best clustering-based approach WC achieved similar results as the Mixture Gaussian approach (∼3 percent points mean difference) but leads to more diverse clusters. Performance drops when solely sampling weights from kernel based normal distributions (Kernel normal) and additionally disabling the Gabor prior (No Gabor prior)

Input: Figure 4: Benefits of combining training reductions for high brain predictivity with few supervised synaptic updates and limitations of techniques in high-update regimes. A (log x-axis) By reducing updates with a combination of fewer supervised updates (Figure 1), improved initialization WC (Figure 2), and training only down-sampling layers CT (Figure 3), the resulting models (dark blue dots; fewer supervised updates alone in light blue) maintain high brain predictivity scores while requiring only a fraction of supervised synaptic updates compared to standard CORnet-S (black dot, top right). B Comparison between WC-initialized models trained with CT versus standardly initialized models training all weights, when varying training epochs and labeled images. Colors represent their percent point difference in brain predictivity scores. WC+CT improve performance in regimes with few epochs and images, but fall behind in regimes with many labeled images and epochs.

Input: Figure 9: Individual brain benchmark scores for WC+CT model A Individual brain predictivitiy scores of WC+CT models trained with a range of epochs on all images. These models score especially high on V1, V2 and V4 already after one epoch in comparison to a model with standard initialization training all weights. IT and Behavior benchmarks continuously improve over later epochs as well but fall short of a fully trained model. B Like A, but with models trained until convergence on different numbers of labeled images, up to the full dataset of 1.28M images (rightmost points). As in A we see > 80% V1, V2 and V4 scores with only 100,000 images. For comparable IT and behavioral scores, more images are required.

Input: Figure 2: “At-birth” synaptic connectivity yields reasonable scores in brain predictivity. A Specifying the initial weight distribution: Kaiming Normal (KN, He et al., 2015) samples from a generic Gaussian. Weight Compression (WC) compresses trained weights into low-parameter clustered distributions that weights can be initialized from. B Visualization of WC compressed parameters: Gabor filters for first layer and cluster centers for following layers with kernel size > 3. The cluster centers capture an intuitive variety of kernel types. C “At-birth” representations with WC achieve 54% score in brain predictivity of a fully trained model, with no training at all. Scores after training remain virtually unchanged (shaded bars).

Input: Figure 8: Detailed analysis of WC+CT. A When reducing the number of supervised synaptic updates, adding critical training (dark grey) and adding weight compression initialization (dark blue) both improve scores in brain predictivity at the same number of supervised synaptic updates, in comparison to a model with standard initialization and training all weights (bright blue). B Brain predictivities for the WC+CT model when trained with a range of epochs and labeled images. C Same as B, but for a standardly initialized (KN) model training all weights.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence and reviewer comments, I'll evaluate the claim's validity:

The claim that these are "first steps in modeling" primate visual processing development appears well-calibrated. The evidence shows concrete progress through:
- Demonstrating high brain predictivity scores with fewer supervised updates
- Showing biologically plausible initialization strategies
- Achieving 54% brain predictivity without training
- Matching developmental patterns where early visual areas converge faster

However, the authors appropriately acknowledge limitations, noting these are "proof-of-principle demonstrations far from accounting for" full developmental processes. The reviewer also points out that while innovative, the approach combines existing methods rather than introducing entirely new principles.

<score>1</score>

The claim is well-stated (score of 1) because it:
1. Uses measured language ("first steps")
2. Specifies the scope clearly (modeling visual processing and development)
3. Acknowledges the preliminary nature of the work
4. Is supported by substantial empirical evidence
5. Avoids overgeneralizing beyond what the results demonstrate


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

These results reflect first steps in modeling not just primate adult visual processing during inference, but also how the ventral visual stream might be "wired up" by evolution (a model's "birth" state) and by developmental learning (a model's updates based on visual experience).

The review comment to be evaluated is:

## Strengths
The paper addresses an important problem in theoretical neuroscience: while trained deep networks appear to be good models of the visual stream, the amount of training they need to get there is incompatible with biology.

Two of the proposed approaches to reduce the amount of training are somewhat straightforward: training for fewer epochs and freezing the amount of trained layers. However, I think the provided results are important as they show good fit to brain data can be achieved without closely following the training procedures of deep learning.

The last approach, weight clustering, is perhaps more interesting to the theoretical neuro community, as it works in the same way as genomically encoding the network (the “at birth” state).

Importantly, the proposed ways to reduce the amount of training are tested not only using ImageNet performance, but also ventral stream data, which is arguably more important for neuroscience.

Overall, the paper presents a novel set of results for an important problem, and supports its claims with a significant number of experiments.

## Weaknesses
As mentioned before, I think the weight compression part is the most valuable one, and therefore it should receive more attention in the paper.
1. Clustering should be better explained in the main text. Especially how the number of clusters was chosen.
2. In the beginning of Sec. 4, you discuss how many bits it would take to encode different networks. I think the same should be done for your experiments. E.g. what is the size of CORnet-S vs. the size of a compressed network?
3. I think similar weight compression techniques have been used before, see http://proceedings.mlr.press/v80/wu18h.html for example
4. In general, it’d be interesting to see how at-birth performance changes with the amount of “genetically encoded” information, i.e. the number of clusters passed through compression. E.g., would doubling the amount of clusters you use drastically improve performance? I’m not expecting this experiment to appear in the paper, but perhaps it’s worth discussing.

The majority of results is limited to one architecture, CORnet-S, which is not a very standard deep learning model. However, I think it is OK as repeating the experiments on something like ResNet50 would be very resource-consuming, and CORnet-S appears to be a better model of the visual stream anyway.

## Recommendation
As the paper addresses an important problem with a novel approach and extensive experiments, I recommend to accept the paper.

## Minor points
In Fig.1, were learning rates tuned individually for each length of training, or were short runs just snapshots of the longer ones? If it’s the latter, I’d imagine you can get better results by cranking up the learning rate and later decreasing it before you hit a plateau (e.g., by spacing out the same three LR decreases within a shorter run). I guess doing it for each point in Fig.1A is too much training, but repeating all points in Fig.1B-C should be possible?

> For instance, the fully trained model’s 55 million supervised updates translate to 5.5 updates every second, whereas the model with 1 million updates and 76% relative brain predictivity translates to one labeled image update every 10 seconds which appears more plausible given the upper limit of 2-3 saccades per second in humans

I would say that a more correct conclusion is that supervised learning is a bad concept for development, but I also think your results are applicable beyond supervised learning. Basically, you show that reducing the amount of updates in backprop retains good Brain-Score performance. But that probably tells more about SGD dynamics in deep networks rather than a specific loss function, and might easily translate to, say, more realistic self-supervised learning. (I guess you briefly mention this point in the discussion.)

> Abstract

> First, we find that only 2% of supervised updates (epochs and images) are needed to achieve ∼80% of the match to adult ventral stream.

80% of the best performing network -- currently it reads as 80% of absolute performance, which would presumably be 1.0 Brain-Score?

The caption of Fig.3C-D should better explain what exactly is trained in each setting

In Sec.6, you should specify that you used MobileNet V1 and ResNet-50.

The evidence to be evaluated is:We evaluate all models on a suite of ventral stream benchmarks in Brain-Score (Schrimpf et al., 2018; Schrimpf et al., 2020), and we base the new models presented here on the CORnet-S architecture, one of the most accurate models of adult primate visual processing (Kubilius et al., 2019).

All benchmarks feed the same images to a candidate model that were used for primate experiments while "recording" activations or measuring behavioral outputs. Specifically, the V1 and V2 benchmarks present 315 images of naturalistic textures and compare model representations to primate single-unit recordings from Freeman et al. (2013) (102 V1 and 103 V2 neurons); the V4 and IT benchmarks present 2,560 naturalistic images and compare models to primate Utah array recordings from Majaj et al. (2015) (88 V4 and 168 IT electrodes).

One of the current best model architectures on the Brain-Score benchmarks is CORnet-S (Kubilius et al., 2019), a shallow recurrent model which anatomically commits to ventral stream regions. CORnet-S has four computational areas, analogous to the ventral visual areas V1, V2, V4, and IT, and a linear decoder that maps from neurons in the model's last visual area to its behavioral choices. The recurrent circuitry (Figure 3B) uses up- and down-sampling convolutions to process features and is identical in each of the models visual areas (except for V1\({}_{COR}\)), but varies by the total number of neurons in each area.

Concretely relating supervised updates to primate ventral stream development, Seibert (2018) establishes that no more than \(\sim\)4 months - or 10 million seconds - of waking visual experience is needed to reach adult-level primate IT cortex (as assessed by its capability to support adult level object recognition). From this estimate, we can compute how many supervised updates per second different models in Figure 1A would require (assuming those updates are evenly distributed over the 10 million seconds). For instance, the fully trained model's 55 million supervised updates translate to 5.5 updates every second, whereas the model with 1 million updates and 76% relative brain predictivity translates to one labeled image update every 10 seconds which appears more plausible given the upper limit of 2-3 saccades per second in humans (Yarbus, 1967; Gibaldi & Sabatini, 2020).

In relation to biology and following the introduced framework of treating all consecutive training as developmental learning, these "at-birth" synaptic connections would result from information encoded in the genome as a product of evolution. Due to the genome's capacity bottleneck, it is infeasible to precisely encode every synapse. Primary visual cortex alone contains \(\sim\)1.4E8 neurons per hemisphere (Leuba & Krafts, 1994), \(\sim\)1E3 synapses per neuron, each requiring \(\sim\)37 bits per synapse (Zador, 2019). Thus, without any clever rules, specifying the connections in only one hemisphere of V1 could require up to \(\sim\)5.2E12 bits - orders of magnitude more than the entire genome's 1GB = 8E9 bits (Zador, 2019).

To test the hypothesis that the genome might already encode more powerful initial representations with synaptic wiring sampled from distributions specified by only few bits, we explored multi-dimensional distributions as a more expressive alternative. These distributions only require a small number of parameters, but unlike current generic initializers, we explicitly specify them for each layer. To determine the right parameterization, we compress a trained model's weights into clusters which we then sample from ("_Weight Compression, WC_"). More specifically, for all convolutional layers except the first, we cluster the kernel weights and later sample from the clusters.

For the first convolutional layer only, we employ a Gabor prior on the weights following studies in V1 (Hubel and Wiesel, 1962; Jones and Palmer, 1987) (Appendix B).

Applying _WC_ to CORnet-S, we first obtain a compressed and clustered set of parameters, from which we sample entirely new weights to yield a new model CORnet-S\({}_{\text{WC}}\). This model is _not trained at all_ and we only evaluate the goodness of its initial wiring on the suite of Brain-Score benchmarks.

_Weight Compression_ explores the hypothesis that evolution may have discovered an initialization strategy with improved at-birth representations (relative to current initializations). _WC_ is most likely not how evolution found the at-birth synaptic connections, but shows that with nearly identical capacity, an alternative initialization distribution leads to networks that are more brain-like in their adult state - revealing a new space of possibilities (hypotheses) that should be considered (see Appendix B.4 for more details on biological plausibility). These findings further suggests that matching representations in higher visual regions might be especially dependent on visual experience whereas early visual regions might already be reasonably well specified without experience.

While improved "at-birth" connectivity can reach 54% of a fully-trained model's score, additional experience dependent updates appear necessary to reach higher predictivities.

As a proof-of-principle that training a reduced set of layers can retain high performance, we propose a novel thin training technique, which we term _Critical Training (CT: Figure 3A). CT_ updates only the weights in critical layers, instead of updating every single model synapse.

We explore successive variants of applying _CT_ up to a block in the architecture and then training the following blocks, e.g. freezing V1, V2, V4 with critical training of the respective down-sampling layers and additional IT training. The final _CT_ ventral stream model is almost completely frozen and only the synapses generating each cortical area's output are trained.

However, while the average score with _DT_ (gray) already drops below 65% with over a quarter of trained parameters remaining and _BN_ drops to 62% with very few parameters, _CT_ (blue) maintains over 75% with only 1.4 out of 52.8 million parameters trained.

By reducing the number of trained parameters, _Critical Training_ also yields engineering benefits with more than 40% of the ImageNet score maintained at \(<3\%\) of parameters trained - a significant improvement over 22% accuracy with _BatchNormNorm_(Frankle et al., 2021) while adding only a small number of additional weights.

High brain predictivity can be achieved with a relatively small number of supervised synaptic updates

Testing synergies between these strategies, we combined all three methods to build novel models that only require a small number of supervised synaptic updates to reasonably capture the mechanisms of adult ventral visual stream processing and object recognition behavior.

_WC+CT_ yield strong benefits (green numbers) in a regime with few supervised updates, improving by up to 27 percent points when training for only 1 epoch on 1,000 images.

_WC+CT_ therefore most positively interact with a small budget of supervised updates (which is the focus of this work).

We developed a range of models with neural and behavioral scores approaching those of the current leading model of the adult ventral visual stream as quantified in Brain-Score, while requiring only a fraction of supervised synaptic updates. These models were built by complementarily 1) reducing the number of supervised updates, i.e. training epochs and labeled images; 2) improving the "at birth" distribution of synaptic connectivity; and 3) training only critical synapses at the end of each model area. The techniques and resulting models proposed here are first steps to more closely modeling not just adult primate visual processing, but also exploring the underlying mechanisms of evolution and developmental learning. These proof-of-principle demonstrations are far from accounting for the rich information encoded in the genome or the developmental learning that together result in adult mechanisms of visual processing, and require further experimental validation.

The fact that early visual areas converge earlier during training (Figure 1) and are better predicted than higher areas by _WC_ initialization is consistent with developmental studies of the primate ventral stream. In humans, behaviors that rely on low-level spatial and temporal processing of visual inputs reach adult-like performance considerably earlier than complex visual behaviors that rely on higher cortical regions, such as face perception (Ellenberg et al., 1999; Grill-Spector et al., 2008). A critical component in more closely modeling primate development is to reduce the dependence on labels altogether.

The changes to model initialization and training presented here serve as a proof-of-principle that models can be changed to more closely align with primate development by reducing training steps with labeled images and improving initialization.

We expect future work in this direction to further close the gap with improved evolutionarily encoded wiring mechanisms and developmental learning rules.

For the very first layer of size \(7\times 7\) we found a Gabor filter most effective following studies in V1 (Hubel and Wiesel, 1962; Jones and Palmer, 1987). To generate the Gabor kernels we fit trained channel weights to a Gabor function

Weight Compression (WC)'s cluster-based initialization generates an "at-birth" network that already captures some useful aspects of the visual inputs in a compressed manner and allows the system to learn faster.

However, independent of _how_ these clusters are determined, WC shows that it is possible to encode certain priors about a system's wiring diagram in a very compressed manner. Evolution also acquired aspects of the visual inputs (with a different strategy) and encoded them in the genome in a lower information regime. For example, we know that there are multiple neuro-developmental mechanisms, such as spontaneous retinal waves and axon guidance cues, that significantly shape the architecture and function of the visual system requiring no (or very little) visual experience ((Huberman et al., 2008)). These mechanisms depend on a relatively small number of proteins encoded in a genome and give rise to a highly complex pattern of at-birth synaptic connectivity representing a very large compression of information.

A model initialized with weight compression achieves (only _WC_) 54% brain predictivity score with 0 supervised synaptic updates. Figure 8B and C show detailed brain predictivity scores, relative to a fully trained model, for models initialized and trained with _WC+CT_ (B) and models initialized with standard Kaiming Normal and training all weights (C) when trained with a range of epochs and labeled images.

Notably, ImageNet performance of these networks seems to not be predictive of their brain predictivity, since even untrained networks with at-chance ImageNet performance correspond reasonably well to e.g. V1 (Figures 1 and 2).

For both transfer methods we initialize the first layer using the Gabor method based on CORnet-S's mixture-of-Gaussian distribution. Since the Gabor function is scalable we can produce Gabor kernels of varying size.

* **KN** A model initialized by standard _Kaiming Normal_ initialization without training.

* WC A model initialized by our _Weight Compression_ initialization, described in Section 4, without training.

* KN+CT The _KN_-initialized model trained with _Critical Training_ until convergence, i.e. three downstream layers and the decoder are trained and all other layers remain unchanged.

* **WC+CT** The _WC_-initialized model with _Critical Training_.

Input: Figure 1: High scores in brain predictivity can be achieved with few supervised updates (log xaxes). A Average brain predictivity of models trained with a range of supervised updates (different combinations of epochs × images). Each dot is a different hypothesis of how the ventral visual stream might have developed and shows the adult brain-likeness score that is achieved by that model. Fairly brain-like representations are already realized with few supervised updates, relative to a fully trained model (black dot; see also Figure 6). Standardly trained CORnet-S is set to be 100% score in brain predictivity on the benchmarks. A “pixels” baseline achieves 7% of this brain predictivity score. B Individual brain predictivity scores over epochs. Models start to approximate the primate ventral stream with few training epochs. Lower visual areas (V1, V2) are approximated earlier in training. C Like B, but number of training images instead of epochs. Few images are sufficient to approximate especially early visual areas.

Input: Figure 7: Alternative weight compression methods Comparison of different initializations that compress weights, “at birth” i.e. without any training (gray) and after training critical layers (shades of blue) for 6 epochs. Our best clustering-based approach WC achieved similar results as the Mixture Gaussian approach (∼3 percent points mean difference) but leads to more diverse clusters. Performance drops when solely sampling weights from kernel based normal distributions (Kernel normal) and additionally disabling the Gabor prior (No Gabor prior)

Input: Figure 4: Benefits of combining training reductions for high brain predictivity with few supervised synaptic updates and limitations of techniques in high-update regimes. A (log x-axis) By reducing updates with a combination of fewer supervised updates (Figure 1), improved initialization WC (Figure 2), and training only down-sampling layers CT (Figure 3), the resulting models (dark blue dots; fewer supervised updates alone in light blue) maintain high brain predictivity scores while requiring only a fraction of supervised synaptic updates compared to standard CORnet-S (black dot, top right). B Comparison between WC-initialized models trained with CT versus standardly initialized models training all weights, when varying training epochs and labeled images. Colors represent their percent point difference in brain predictivity scores. WC+CT improve performance in regimes with few epochs and images, but fall behind in regimes with many labeled images and epochs.

Input: Figure 9: Individual brain benchmark scores for WC+CT model A Individual brain predictivitiy scores of WC+CT models trained with a range of epochs on all images. These models score especially high on V1, V2 and V4 already after one epoch in comparison to a model with standard initialization training all weights. IT and Behavior benchmarks continuously improve over later epochs as well but fall short of a fully trained model. B Like A, but with models trained until convergence on different numbers of labeled images, up to the full dataset of 1.28M images (rightmost points). As in A we see > 80% V1, V2 and V4 scores with only 100,000 images. For comparable IT and behavioral scores, more images are required.

Input: Figure 2: “At-birth” synaptic connectivity yields reasonable scores in brain predictivity. A Specifying the initial weight distribution: Kaiming Normal (KN, He et al., 2015) samples from a generic Gaussian. Weight Compression (WC) compresses trained weights into low-parameter clustered distributions that weights can be initialized from. B Visualization of WC compressed parameters: Gabor filters for first layer and cluster centers for following layers with kernel size > 3. The cluster centers capture an intuitive variety of kernel types. C “At-birth” representations with WC achieve 54% score in brain predictivity of a fully trained model, with no training at all. Scores after training remain virtually unchanged (shaded bars).

Input: Figure 8: Detailed analysis of WC+CT. A When reducing the number of supervised synaptic updates, adding critical training (dark grey) and adding weight compression initialization (dark blue) both improve scores in brain predictivity at the same number of supervised synaptic updates, in comparison to a model with standard initialization and training all weights (bright blue). B Brain predictivities for the WC+CT model when trained with a range of epochs and labeled images. C Same as B, but for a standardly initialized (KN) model training all weights.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, I find that this claim appropriately characterizes the study's findings as "first steps" and avoids overstating conclusions. The evidence demonstrates:

1. Clear progress in modeling "birth state" through Weight Compression (WC) initialization, achieving 54% brain predictivity without training
2. Development modeling through reduced training requirements (only 2% of updates needed for 80% performance)
3. Appropriate acknowledgment of limitations ("proof-of-principle demonstrations are far from accounting for...")
4. Support from reviewer comments highlighting novelty while noting limitations

The conservative framing of results as "first steps" and explicit recognition of what remains to be done makes this a well-stated claim that accurately reflects the evidence without exaggeration.

<score>1</score>