Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Further investigating the degree to which this network uses sparse coding, we find that it learns receptive fields similar to those of mammalian V1 with Gabor filters and retinal ganglion cells with on/off center surround (Sterling and Laughlin, 2015; Olshausen and Field, 1997).

The review comment to be evaluated is:

Strengths:
* The paper tackles the important problem of bridging the gap between biological plausible learning and current artificial neural networks.

Weaknesses:
* The paper mostly focuses on 1-hidden layer networks. It is not too clear why and how these results are relevant for practice, where typically much deeper networks are considered. For classification tasks, the observed sparsity seems to be much less pronounced.
* Unlike claimed in the paper, using the L1-norm can lead to biologically plausible and sparse features even for classification tasks, see e.g. Figure 4 in "Towards Learning Convolutions from Scratch" (https://papers.nips.cc/paper/2020/file/5c528e25e1fdeaf9d8160dc24dbf4d60-Paper.pdf)
* There are some issues regarding novelty/clarity (see next section)

The evidence to be evaluated is:However, we are unaware of any existing work which finds that noisy training with a ReLU activation function not only results in sparsity but also converges to a sparse coding, Top-K network. Learning reconstructions of training data with sparse activations is the mainstay of sparse coding (Olshausen & Field, 1997). Sparse coding uses an overcomplete basis to learn a reconstruction of data with an \(L_{1}\) regularization penalty on neuron activity. The image reconstruction can be written as \(\hat{\mathbf{x}}=\sum_{i}^{N}a_{i}\phi_{i}\) where \(\phi_{i}\) is a basis vector from the matrix or "dictionary" \(\Phi\); \(a_{i}\) is the activity of this basis vector, and the activation weighted sum of basis vectors produces the output \(\hat{\mathbf{x}}\). The objective function aims to minimize the mean squared-error between the reconstructed image and the original image while having sparse neuron activity: \(\operatorname*{arg\,min}_{\Phi,\mathbf{n}}(\mathbf{x}-\hat{\mathbf{x}})^{2}+ \lambda\sum_{i}^{N}|a_{i}|\). This sparse coding task is an example of a statistical model converging with neuroscience because, in addition to producing sparse representations, the basis functions learn to become Gabor filters like those found in area V1 of the visual system (Olshausen & Field, 1997).

To ensure our results are robust, we vary a number of settings, finding that in every case the relationship between noise and network sparsity holds.

Figure 2 shows that a single hidden layer ReLU network with 10,000 neurons learns to become a Top-K network with smaller \(k\) values as the amount of noise increases. This network was trained on the CIFAR10 embeddings to mimick training within a deeper network layer and where the manifold is smoother than when training on raw image pixels. While Fig. 2 only shows the mean number of neurons that are active across all 50,000 CIFAR10 inputs per epoch, Figure 3 shows the fraction of active neurons for every input.

Looking more closely at the sparse solutions found by our ReLU networks, we discovered that for \(\sigma\geq 0.3\), neurons begin to form receptive fields reminiscent of V1 Gabor filters and on/off center surround retinal ganglion cells (Sterling & Laughlin, 2015; Olshausen & Field, 1997).

This is a similar result to that of sparse coding (Olshausen & Field, 1997), however, here we use noise with the ReLU activation function instead of a sparsity penalty and train on full images instead of 8x8 patches.

The longer training progressed, the more similar the receptive fields appeared to those found in biology.

When we tried training the network without any noise but instead with an \(L_{1}\) activation penalty motivated by the original sparse coding setup (Olshausen & Field, 1997), no biological receptive fields were observed.

While still neither activation function becomes as sparse as ReLU, the GELU networks implement the same Top-K convergence as ReLU shown in Fig. 3 and form biologically plausible receptive fields like those of Fig. 5.

In natural images, for example, certain projections (e.g., with Gabor filters) have high kurtosis which allows them to be easily distinguished from noise via thresholding or "coring" (Simoncelli and Adelson, 1996). Similarly, we hypothesize that the encoding layer in our network is learning such projections, so that thresholding performs noise rejection while also producing sparse activations.

This result is particularly striking because of the way the networks converge with sparse coding, producing biological receptive fields and approximating an inhibitory interneuron that implements a Top-K operation.

More broadly, this work builds a new bridge between artificial and biological neural networks by showing how they can be more similar than they may otherwise appear.

Here we show additional images of the receptive fields for networks trained with different amounts of noise (Fig. 8), the queries used to activate these neurons (Fig. 9) and how the receptive fields look more biological as the network is trained for longer (Fig. 10).

Input: Figure 2: There is a positive relationship between activation sparsity and noise. The average fraction of neurons active during each CIFAR10 latent input over the course of training for 800 epochs. Each line corresponds to training randomly initialized networks with different noise levels σ denoted by different colors. We show the average of three different random seeds and their standard error of the mean (not visible as the variance is so low). Interestingly, the higher noise levels take longer to sparsify but upon convergence become the sparsest.

Input: Figure 3: Noise induces the formation of Top-K networks. Shown column-wise are the fraction of neurons that are active for each training input (Left), the values of each of the 10,000 neurons bias terms (Middle) and the L2 norms of their key vectors (Right), as a function of number of training epochs. Each is shown for three different representative noise levels (rows). These plots show each input to the network and use the density of blue to represent how many times a particular value occurs.

Input: Figure 5: Biological receptive fields form with noise. Networks trained on CIFAR10 pixels with σ = 0.8 Gaussian noise. The most active 125 neuron receptive fields are reshaped into 32x32x3 and re-scaled so their values span the range of pixel values [0, 255]. Neurons are sorted by their activity levels (top left is the most active, going across rows then down columns) for a random noisy input of a car. Starting at σ ≥ 0.1 and particularly for σ = 0.8 (shown here) we observe the formation of many Gabor filters and on/off center surround receptive fields that are unique to noisy training.

Input: Figure 8: Receptive fields across noise levels. We show for all noise levels the 625 most active neurons for the truck query (middle of Fig. 9). For 0.8 ≥ σ which activate fewer neurons, we also show the cat query below it. Biological like receptive fields start to appear for σ = 0.1 and can be seen most clearly for σ ∈ {0.3, 0.8, 1.5}.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim appears to be slightly overstated. While the evidence does show that receptive fields similar to biological Gabor filters and on/off center surround patterns emerge under specific noise conditions (σ ≥ 0.3), these patterns are most clearly visible at σ = 0.8 and within certain noise ranges (0.3-1.5). The claim doesn't acknowledge these specific conditions and limitations. Additionally, as noted in the review comments, these results are limited to single-layer networks, and the sparsity is less pronounced for classification tasks.

<score>3</score>

Justification:
1. The evidence does support the emergence of biologically-plausible receptive fields, as shown in Figures 5 and 8, particularly at noise levels σ ≥ 0.3.

2. However, the claim overstates by:
- Not specifying the noise conditions required (σ ≥ 0.3, optimal at σ = 0.8)
- Not mentioning these patterns emerge primarily in single-layer networks
- Not acknowledging the limited applicability to classification tasks (from review comments)
- Not mentioning that these effects are most pronounced under specific training conditions

3. Direct evidence supporting this evaluation:
- "Starting at σ ≥ 0.1 and particularly for σ = 0.8 (shown here) we observe the formation of many Gabor filters and on/off center surround receptive fields"
- Figure 8 shows the receptive fields are most biological-like for "σ ∈ {0.3, 0.8, 1.5}"
- Review comment notes: "The paper mostly focuses on 1-hidden layer networks" and "For classification tasks, the observed sparsity seems to be much less pronounced"


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Further investigating the degree to which this network uses sparse coding, we find that it learns receptive fields similar to those of mammalian V1 with Gabor filters and retinal ganglion cells with on/off center surround (Sterling and Laughlin, 2015; Olshausen and Field, 1997).

The review comment to be evaluated is:

In this empirical paper, the authors observe the correlation between the sparsity of the learned representations and the level of noise injected into the input of Neural Networks with the ReLU activation function. Experiments have been carried out on CIFAR10 both on pixel data and on a neural representation, MNIST, and on the WikiText-103 dataset. They report a positive correlation between the magnitude of the injected noise and the sparsity of the representation. They observe Gabor-like filters when reconstructing with noise on pixel data. For the classification task, the performance of the model did not decrease with sparsity. Also, compared with using only the top-K activations of a model learned without noise, the models learned with noise gave better validation accuracies.

The evidence to be evaluated is:However, we are unaware of any existing work which finds that noisy training with a ReLU activation function not only results in sparsity but also converges to a sparse coding, Top-K network. Learning reconstructions of training data with sparse activations is the mainstay of sparse coding (Olshausen & Field, 1997). Sparse coding uses an overcomplete basis to learn a reconstruction of data with an \(L_{1}\) regularization penalty on neuron activity. The image reconstruction can be written as \(\hat{\mathbf{x}}=\sum_{i}^{N}a_{i}\phi_{i}\) where \(\phi_{i}\) is a basis vector from the matrix or "dictionary" \(\Phi\); \(a_{i}\) is the activity of this basis vector, and the activation weighted sum of basis vectors produces the output \(\hat{\mathbf{x}}\). The objective function aims to minimize the mean squared-error between the reconstructed image and the original image while having sparse neuron activity: \(\operatorname*{arg\,min}_{\Phi,\mathbf{n}}(\mathbf{x}-\hat{\mathbf{x}})^{2}+ \lambda\sum_{i}^{N}|a_{i}|\). This sparse coding task is an example of a statistical model converging with neuroscience because, in addition to producing sparse representations, the basis functions learn to become Gabor filters like those found in area V1 of the visual system (Olshausen & Field, 1997).

To ensure our results are robust, we vary a number of settings, finding that in every case the relationship between noise and network sparsity holds.

Figure 2 shows that a single hidden layer ReLU network with 10,000 neurons learns to become a Top-K network with smaller \(k\) values as the amount of noise increases. This network was trained on the CIFAR10 embeddings to mimick training within a deeper network layer and where the manifold is smoother than when training on raw image pixels. While Fig. 2 only shows the mean number of neurons that are active across all 50,000 CIFAR10 inputs per epoch, Figure 3 shows the fraction of active neurons for every input.

Looking more closely at the sparse solutions found by our ReLU networks, we discovered that for \(\sigma\geq 0.3\), neurons begin to form receptive fields reminiscent of V1 Gabor filters and on/off center surround retinal ganglion cells (Sterling & Laughlin, 2015; Olshausen & Field, 1997).

This is a similar result to that of sparse coding (Olshausen & Field, 1997), however, here we use noise with the ReLU activation function instead of a sparsity penalty and train on full images instead of 8x8 patches.

The longer training progressed, the more similar the receptive fields appeared to those found in biology.

When we tried training the network without any noise but instead with an \(L_{1}\) activation penalty motivated by the original sparse coding setup (Olshausen & Field, 1997), no biological receptive fields were observed.

While still neither activation function becomes as sparse as ReLU, the GELU networks implement the same Top-K convergence as ReLU shown in Fig. 3 and form biologically plausible receptive fields like those of Fig. 5.

In natural images, for example, certain projections (e.g., with Gabor filters) have high kurtosis which allows them to be easily distinguished from noise via thresholding or "coring" (Simoncelli and Adelson, 1996). Similarly, we hypothesize that the encoding layer in our network is learning such projections, so that thresholding performs noise rejection while also producing sparse activations.

This result is particularly striking because of the way the networks converge with sparse coding, producing biological receptive fields and approximating an inhibitory interneuron that implements a Top-K operation.

More broadly, this work builds a new bridge between artificial and biological neural networks by showing how they can be more similar than they may otherwise appear.

Here we show additional images of the receptive fields for networks trained with different amounts of noise (Fig. 8), the queries used to activate these neurons (Fig. 9) and how the receptive fields look more biological as the network is trained for longer (Fig. 10).

Input: Figure 2: There is a positive relationship between activation sparsity and noise. The average fraction of neurons active during each CIFAR10 latent input over the course of training for 800 epochs. Each line corresponds to training randomly initialized networks with different noise levels σ denoted by different colors. We show the average of three different random seeds and their standard error of the mean (not visible as the variance is so low). Interestingly, the higher noise levels take longer to sparsify but upon convergence become the sparsest.

Input: Figure 3: Noise induces the formation of Top-K networks. Shown column-wise are the fraction of neurons that are active for each training input (Left), the values of each of the 10,000 neurons bias terms (Middle) and the L2 norms of their key vectors (Right), as a function of number of training epochs. Each is shown for three different representative noise levels (rows). These plots show each input to the network and use the density of blue to represent how many times a particular value occurs.

Input: Figure 5: Biological receptive fields form with noise. Networks trained on CIFAR10 pixels with σ = 0.8 Gaussian noise. The most active 125 neuron receptive fields are reshaped into 32x32x3 and re-scaled so their values span the range of pixel values [0, 255]. Neurons are sorted by their activity levels (top left is the most active, going across rows then down columns) for a random noisy input of a car. Starting at σ ≥ 0.1 and particularly for σ = 0.8 (shown here) we observe the formation of many Gabor filters and on/off center surround receptive fields that are unique to noisy training.

Input: Figure 8: Receptive fields across noise levels. We show for all noise levels the 625 most active neurons for the truck query (middle of Fig. 9). For 0.8 ≥ σ which activate fewer neurons, we also show the cat query below it. Biological like receptive fields start to appear for σ = 0.1 and can be seen most clearly for σ ∈ {0.3, 0.8, 1.5}.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find that it is relatively well-stated but has some minor overstatement issues. The evidence does show that the network learns receptive fields similar to biological ones, particularly at noise levels σ ≥ 0.3 (Fig. 5, Fig. 8). However, the claim could be more precise by specifying that this occurs only with sufficient noise levels and takes time to develop. The figures demonstrate Gabor-like filters and center-surround patterns, but the similarity to biological receptive fields varies across noise levels and isn't universal across all conditions.

<score>2</score>

Justification:
1. The evidence supports the basic claim through:
- Clear visualization of Gabor-like filters and center-surround patterns in Fig. 5 and Fig. 8
- Specific documentation that these patterns emerge at σ ≥ 0.3
- Demonstration that patterns become more biological-like with longer training

2. Minor overstatement issues:
- Doesn't mention the dependency on noise levels
- Doesn't acknowledge variability in the similarity to biological receptive fields
- Could be more specific about the conditions under which these similarities emerge

3. Supporting citations:
- "Looking more closely at the sparse solutions found by our ReLU networks, we discovered that for σ ≥ 0.3, neurons begin to form receptive fields reminiscent of V1 Gabor filters"
- "The longer training progressed, the more similar the receptive fields appeared to those found in biology"


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We find this Top-K network formation is particularly evident for reconstruction tasks but also exists for classification tasks and the intermediate MLP layers of a Transformer architecture.

The evidence to be evaluated is:Top-K activation functions inspired by inhibitory interneurons have been implemented in a number of deep learning models for greater computational efficiency (Ahmad & Scheinkman, 2019), adversarial robustness (Xiao et al., 2020; Tramer et al., 2020), interpretability (Nelson et al., 2022), and continual learning (Shen et al., 2021; Iyer et al., 2022).

SDM in particular can be viewed as a de-noising autoencoder and has been developed into an MLP that uses a Top-K activation function (Keeler, 1988). Finally, de-noising autoencoders are single step versions of diffusion models (Song & Ermon, 2019).

By default, we train a single hidden layer ReLU network with different amounts of random Gaussian noise applied to the input. The objective is to remove the noise and reconstruct the original input. Formally, noise is applied from an isotropic Gaussian with variance \(\sigma\) to the input \(\mathbf{x}\). The corrupted image is then put through our encoder and decoder.

Figure 2 shows that a single hidden layer ReLU network with 10,000 neurons learns to become a Top-K network with smaller \(k\) values as the amount of noise increases. This network was trained on the CIFAR10 embeddings to mimick training within a deeper network layer and where the manifold is smoother than when training on raw image pixels. While Fig. 2 only shows the mean number of neurons that are active across all 50,000 CIFAR10 inputs per epoch, Figure 3 shows the fraction of active neurons for every input.

We compared this network with explicitly creating sparse Top-K networks. These Top-K networks sort the neuron activations and keep only the \(k\) most active. They were either given fixed \(k\) values or could learn the \(k\) values using either a continuous relaxation that is differentiable or the REINFORCE algorithm (see Appendix A for details) (Williams, 2004). These approaches all performed worse than the shared bias (and default ReLU) networks, with the best networks getting similar validation accuracies as the much smaller 100 neuron network and only when they were not sparse, using \(k=3,000\) neurons. Even the approaches learning \(k\) were insufficiently dynamic and depended too much upon the \(k\) initialization value.

We believe this is due to the fact that classification only needs to cluster the data into the ten CIFAR10 labels, creating less interference between neural representations. There is also no reduction in reconstruction or classification performance. In fact, for classification \(\sigma\in\{0.1,0.3\}\) both get 94.3% validation accuracy instead of the 93.4% that the baseline gets, showing slightly better generalization while also being more sparse (66% for baseline versus 55% for \(\sigma=0.1\) and 45% for \(\sigma=0.3\)).

**Model Ablations -** Figure 7 shows the mean neuron sparsity at network convergence across a number of different experimental conditions: (i) three datasets: CIFAR10 pixels, CIFAR10 latent embeddings, MNIST; (ii) two noise distributions: Gaussian and Laplace; (iii) four numbers of neurons: 100, 1,000, 10,000 and 100,000; (iv) three training tasks: reconstruction, classification, and Transformer next token prediction.

While still neither activation function becomes as sparse as ReLU, the GELU networks implement the same Top-K convergence as ReLU shown in Fig. 3 and form biologically plausible receptive fields like those of Fig. 5.

While we just show the mean active neurons here, both solutions look like those of Fig. 3 with the neurons for all inputs converging to a Top-K solution where the bias terms become negative and synchronize along with the key vector \(L_{2}\) norms.

Input: Figure 7: Training ablations. The positive correlation between noise and sparsity is robust to (a) neuron counts, (b) datasets, (c) noise distributions, and (d) training tasks.

Input: Figure 2: There is a positive relationship between activation sparsity and noise. The average fraction of neurons active during each CIFAR10 latent input over the course of training for 800 epochs. Each line corresponds to training randomly initialized networks with different noise levels σ denoted by different colors. We show the average of three different random seeds and their standard error of the mean (not visible as the variance is so low). Interestingly, the higher noise levels take longer to sparsify but upon convergence become the sparsest.

Input: Figure 3: Noise induces the formation of Top-K networks. Shown column-wise are the fraction of neurons that are active for each training input (Left), the values of each of the 10,000 neurons bias terms (Middle) and the L2 norms of their key vectors (Right), as a function of number of training epochs. Each is shown for three different representative noise levels (rows). These plots show each input to the network and use the density of blue to represent how many times a particular value occurs.

Input: Figure 6: Noisy pretraining retains network sparsity. The vertical purple lines denote the start of noise annealing where each noise level is linearly decayed down to σ = 0. For both reconstruction (left) and classification (right) tasks the networks remain highly sparse even after all noise is removed. Moreover, there is no noticeable performance difference between any of the networks while some of those trained with moderate noise perform slightly better than the noise free baseline. We truncate the x-axis to start at epoch 600 for the sake of clarity. The classification networks continue to sparsify even when noise annealing has started because unlike in the reconstruction task, the sparsity level did not converge within the first 800 epochs.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate the claim: "We find this Top-K network formation is particularly evident for reconstruction tasks but also exists for classification tasks and the intermediate MLP layers of a Transformer architecture."

Summary: The evidence strongly supports this claim. Figures 2, 3, 6, and 7 demonstrate Top-K network formation across different tasks. Figure 7(d) specifically shows the sparsity patterns for reconstruction, classification, and language modeling (Transformer) tasks. The data indicates reconstruction tasks achieve higher sparsity levels compared to classification and language modeling, aligning with the claim's emphasis on reconstruction tasks. The evidence shows comprehensive testing across various conditions (datasets, noise types, neuron counts) that validate the observation without overstating the findings.

<score>1</score>

Justification:
1. Direct evidence from Fig 7(d) shows comparative sparsity patterns across all three task types
2. Multiple ablation studies (Fig 7a-c) demonstrate robustness of the finding
3. Figure 6 provides detailed comparison between reconstruction and classification tasks
4. The claim acknowledges the stronger effect in reconstruction tasks while accurately noting the existence in other architectures
5. The wording is appropriately measured, using "particularly evident" rather than absolute terms
6. No exaggeration of significance or unsupported generalizations


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We find this Top-K network formation is particularly evident for reconstruction tasks but also exists for classification tasks and the intermediate MLP layers of a Transformer architecture.

The review comment to be evaluated is:

Strengths

The paper is simple and easy to follow.

The sparsity effects shown appear relatively large and robust.

Adding noise is a simple manipulation that can have benefits beyond sparsity, and so there is some appeal to using this manipulation as a way to generate sparsity.

Weaknesses

It is odd that the authors focus so much on a one-layer model. For this finding to be broadly relevant, it is important to test whether this effect occurs in large-scale DNNs and whether networks can generate sparsity while maintaining good performance on real-world tasks. The only thing that is done along these lines is with a single, small GPT2 model.

It appears you need fairly high levels of noise to generate substantial sparsity (e.g., sigma=0.3 to get below 20% activation in Figure 2, which based on Figure 4 is quite a lot of noise). It seems this might limit the practical utility of the method, though it is hard to say in part because experiments are mostly limited to the single-layer training regime.

The link with biological receptive fields is tenuous. There are multiple ways of generating center-surround and oriented/bandpass receptive fields (e.g., whitening for center-surround RFs, sparsity for oriented RFs, CNNs often exhibit similar structure at the first layer). It is not clear this approach provides a better fit to biological data than these alternatives and there is no quantification of the match to biology or model comparison. Moreover, there are so many differences between the models trained here and biological networks, that it is hard to know the relevance of this finding, particularly given that the results can vary with the activation function (e.g., the sigmoid activation function does not generate sparsity) and the optimization algorithm (e.g., SGD with a low learning rate apparently does not generate sparsity).

The evidence to be evaluated is:Top-K activation functions inspired by inhibitory interneurons have been implemented in a number of deep learning models for greater computational efficiency (Ahmad & Scheinkman, 2019), adversarial robustness (Xiao et al., 2020; Tramer et al., 2020), interpretability (Nelson et al., 2022), and continual learning (Shen et al., 2021; Iyer et al., 2022).

SDM in particular can be viewed as a de-noising autoencoder and has been developed into an MLP that uses a Top-K activation function (Keeler, 1988). Finally, de-noising autoencoders are single step versions of diffusion models (Song & Ermon, 2019).

By default, we train a single hidden layer ReLU network with different amounts of random Gaussian noise applied to the input. The objective is to remove the noise and reconstruct the original input. Formally, noise is applied from an isotropic Gaussian with variance \(\sigma\) to the input \(\mathbf{x}\). The corrupted image is then put through our encoder and decoder.

Figure 2 shows that a single hidden layer ReLU network with 10,000 neurons learns to become a Top-K network with smaller \(k\) values as the amount of noise increases. This network was trained on the CIFAR10 embeddings to mimick training within a deeper network layer and where the manifold is smoother than when training on raw image pixels. While Fig. 2 only shows the mean number of neurons that are active across all 50,000 CIFAR10 inputs per epoch, Figure 3 shows the fraction of active neurons for every input.

We compared this network with explicitly creating sparse Top-K networks. These Top-K networks sort the neuron activations and keep only the \(k\) most active. They were either given fixed \(k\) values or could learn the \(k\) values using either a continuous relaxation that is differentiable or the REINFORCE algorithm (see Appendix A for details) (Williams, 2004). These approaches all performed worse than the shared bias (and default ReLU) networks, with the best networks getting similar validation accuracies as the much smaller 100 neuron network and only when they were not sparse, using \(k=3,000\) neurons. Even the approaches learning \(k\) were insufficiently dynamic and depended too much upon the \(k\) initialization value.

We believe this is due to the fact that classification only needs to cluster the data into the ten CIFAR10 labels, creating less interference between neural representations. There is also no reduction in reconstruction or classification performance. In fact, for classification \(\sigma\in\{0.1,0.3\}\) both get 94.3% validation accuracy instead of the 93.4% that the baseline gets, showing slightly better generalization while also being more sparse (66% for baseline versus 55% for \(\sigma=0.1\) and 45% for \(\sigma=0.3\)).

**Model Ablations -** Figure 7 shows the mean neuron sparsity at network convergence across a number of different experimental conditions: (i) three datasets: CIFAR10 pixels, CIFAR10 latent embeddings, MNIST; (ii) two noise distributions: Gaussian and Laplace; (iii) four numbers of neurons: 100, 1,000, 10,000 and 100,000; (iv) three training tasks: reconstruction, classification, and Transformer next token prediction.

While still neither activation function becomes as sparse as ReLU, the GELU networks implement the same Top-K convergence as ReLU shown in Fig. 3 and form biologically plausible receptive fields like those of Fig. 5.

While we just show the mean active neurons here, both solutions look like those of Fig. 3 with the neurons for all inputs converging to a Top-K solution where the bias terms become negative and synchronize along with the key vector \(L_{2}\) norms.

Input: Figure 7: Training ablations. The positive correlation between noise and sparsity is robust to (a) neuron counts, (b) datasets, (c) noise distributions, and (d) training tasks.

Input: Figure 2: There is a positive relationship between activation sparsity and noise. The average fraction of neurons active during each CIFAR10 latent input over the course of training for 800 epochs. Each line corresponds to training randomly initialized networks with different noise levels σ denoted by different colors. We show the average of three different random seeds and their standard error of the mean (not visible as the variance is so low). Interestingly, the higher noise levels take longer to sparsify but upon convergence become the sparsest.

Input: Figure 3: Noise induces the formation of Top-K networks. Shown column-wise are the fraction of neurons that are active for each training input (Left), the values of each of the 10,000 neurons bias terms (Middle) and the L2 norms of their key vectors (Right), as a function of number of training epochs. Each is shown for three different representative noise levels (rows). These plots show each input to the network and use the density of blue to represent how many times a particular value occurs.

Input: Figure 6: Noisy pretraining retains network sparsity. The vertical purple lines denote the start of noise annealing where each noise level is linearly decayed down to σ = 0. For both reconstruction (left) and classification (right) tasks the networks remain highly sparse even after all noise is removed. Moreover, there is no noticeable performance difference between any of the networks while some of those trained with moderate noise perform slightly better than the noise free baseline. We truncate the x-axis to start at epoch 600 for the sake of clarity. The classification networks continue to sparsify even when noise annealing has started because unlike in the reconstruction task, the sparsity level did not converge within the first 800 epochs.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find that it is moderately overstated. While the evidence clearly demonstrates Top-K network formation for reconstruction tasks (Figures 2-3), the evidence for classification tasks is less definitive, showing different sparsity patterns (Figure 6). For Transformer architectures, the claim is particularly tenuous, as Figure 7(d) only shows limited language modeling results without detailed analysis of intermediate MLP layers. The review comment correctly points out that experiments are mostly limited to single-layer models, with only brief testing on a small GPT2 model.

<score>3</score>

Justification:
1. For reconstruction tasks:
- Strong evidence from Figures 2-3 showing clear Top-K formation with increasing noise levels
- Multiple ablation studies (Figure 7) demonstrate robustness across datasets and conditions

2. For classification tasks:
- More limited evidence
- Different sparsification patterns observed (Figure 6)
- Still shows some Top-K behavior but with different dynamics
- Maintains performance (94.3% vs 93.4% baseline)

3. For Transformer architecture:
- Very limited evidence presented
- Only brief mention of language modeling in Figure 7(d)
- No detailed analysis of intermediate MLP layers as claimed
- Reviewer highlights limitation of testing only on small GPT2 model

The claim would be better stated by being more specific about the strength of evidence for each task type and acknowledging the limitations in the Transformer architecture findings.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We find this Top-K network formation is particularly evident for reconstruction tasks but also exists for classification tasks and the intermediate MLP layers of a Transformer architecture.

The review comment to be evaluated is:

Strengths

1. The authors find a simple, implicit manner to create top-k networks that compares favorably to explicit methods

2. The method is tested in several different circumstances, with different tasks, numbers of neurons, and noise levels. Simple characteristics of the network, such as the fraction of active neurons and their biases, are well described.

Weaknesses

Major

1. As the authors observe, the results are recapitulated using a global negative bias term, acting as common inhibition that requires greater input values to activate neurons and resulting in top-k networks.  Thus, while such an observation may indeed be useful for those interested in such networks, the advance it comprises is a minor one.

Minor

1. While the presented results may bear on biological considerations, as suggested therein, caution is advised, as noise in biological networks is dynamic and thus not perfectly analogous to simply adding noise to inputs of a feedforward network.

The evidence to be evaluated is:Top-K activation functions inspired by inhibitory interneurons have been implemented in a number of deep learning models for greater computational efficiency (Ahmad & Scheinkman, 2019), adversarial robustness (Xiao et al., 2020; Tramer et al., 2020), interpretability (Nelson et al., 2022), and continual learning (Shen et al., 2021; Iyer et al., 2022).

SDM in particular can be viewed as a de-noising autoencoder and has been developed into an MLP that uses a Top-K activation function (Keeler, 1988). Finally, de-noising autoencoders are single step versions of diffusion models (Song & Ermon, 2019).

By default, we train a single hidden layer ReLU network with different amounts of random Gaussian noise applied to the input. The objective is to remove the noise and reconstruct the original input. Formally, noise is applied from an isotropic Gaussian with variance \(\sigma\) to the input \(\mathbf{x}\). The corrupted image is then put through our encoder and decoder.

Figure 2 shows that a single hidden layer ReLU network with 10,000 neurons learns to become a Top-K network with smaller \(k\) values as the amount of noise increases. This network was trained on the CIFAR10 embeddings to mimick training within a deeper network layer and where the manifold is smoother than when training on raw image pixels. While Fig. 2 only shows the mean number of neurons that are active across all 50,000 CIFAR10 inputs per epoch, Figure 3 shows the fraction of active neurons for every input.

We compared this network with explicitly creating sparse Top-K networks. These Top-K networks sort the neuron activations and keep only the \(k\) most active. They were either given fixed \(k\) values or could learn the \(k\) values using either a continuous relaxation that is differentiable or the REINFORCE algorithm (see Appendix A for details) (Williams, 2004). These approaches all performed worse than the shared bias (and default ReLU) networks, with the best networks getting similar validation accuracies as the much smaller 100 neuron network and only when they were not sparse, using \(k=3,000\) neurons. Even the approaches learning \(k\) were insufficiently dynamic and depended too much upon the \(k\) initialization value.

We believe this is due to the fact that classification only needs to cluster the data into the ten CIFAR10 labels, creating less interference between neural representations. There is also no reduction in reconstruction or classification performance. In fact, for classification \(\sigma\in\{0.1,0.3\}\) both get 94.3% validation accuracy instead of the 93.4% that the baseline gets, showing slightly better generalization while also being more sparse (66% for baseline versus 55% for \(\sigma=0.1\) and 45% for \(\sigma=0.3\)).

**Model Ablations -** Figure 7 shows the mean neuron sparsity at network convergence across a number of different experimental conditions: (i) three datasets: CIFAR10 pixels, CIFAR10 latent embeddings, MNIST; (ii) two noise distributions: Gaussian and Laplace; (iii) four numbers of neurons: 100, 1,000, 10,000 and 100,000; (iv) three training tasks: reconstruction, classification, and Transformer next token prediction.

While still neither activation function becomes as sparse as ReLU, the GELU networks implement the same Top-K convergence as ReLU shown in Fig. 3 and form biologically plausible receptive fields like those of Fig. 5.

While we just show the mean active neurons here, both solutions look like those of Fig. 3 with the neurons for all inputs converging to a Top-K solution where the bias terms become negative and synchronize along with the key vector \(L_{2}\) norms.

Input: Figure 7: Training ablations. The positive correlation between noise and sparsity is robust to (a) neuron counts, (b) datasets, (c) noise distributions, and (d) training tasks.

Input: Figure 2: There is a positive relationship between activation sparsity and noise. The average fraction of neurons active during each CIFAR10 latent input over the course of training for 800 epochs. Each line corresponds to training randomly initialized networks with different noise levels σ denoted by different colors. We show the average of three different random seeds and their standard error of the mean (not visible as the variance is so low). Interestingly, the higher noise levels take longer to sparsify but upon convergence become the sparsest.

Input: Figure 3: Noise induces the formation of Top-K networks. Shown column-wise are the fraction of neurons that are active for each training input (Left), the values of each of the 10,000 neurons bias terms (Middle) and the L2 norms of their key vectors (Right), as a function of number of training epochs. Each is shown for three different representative noise levels (rows). These plots show each input to the network and use the density of blue to represent how many times a particular value occurs.

Input: Figure 6: Noisy pretraining retains network sparsity. The vertical purple lines denote the start of noise annealing where each noise level is linearly decayed down to σ = 0. For both reconstruction (left) and classification (right) tasks the networks remain highly sparse even after all noise is removed. Moreover, there is no noticeable performance difference between any of the networks while some of those trained with moderate noise perform slightly better than the noise free baseline. We truncate the x-axis to start at epoch 600 for the sake of clarity. The classification networks continue to sparsify even when noise annealing has started because unlike in the reconstruction task, the sparsity level did not converge within the first 800 epochs.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find the claim is well-supported by comprehensive experimental data. The evidence shows Top-K network formation across:

1. Reconstruction tasks: Clearly demonstrated in Figures 2-3 showing sparsity patterns
2. Classification tasks: Shown in Figure 6 (right panel) and validation accuracy data
3. Transformer architecture: Evidenced in Figure 7(d) showing language modeling results

The claim is carefully worded, noting it's "particularly evident" for reconstruction (which matches the stronger effects shown in figures) while acknowledging it "also exists" for other tasks. The reviewer comments don't dispute these findings, only suggesting caution about biological interpretations which isn't relevant to this specific claim.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We find this Top-K network formation is particularly evident for reconstruction tasks but also exists for classification tasks and the intermediate MLP layers of a Transformer architecture.

The review comment to be evaluated is:

Strengths:
* The paper tackles the important problem of bridging the gap between biological plausible learning and current artificial neural networks.

Weaknesses:
* The paper mostly focuses on 1-hidden layer networks. It is not too clear why and how these results are relevant for practice, where typically much deeper networks are considered. For classification tasks, the observed sparsity seems to be much less pronounced.
* Unlike claimed in the paper, using the L1-norm can lead to biologically plausible and sparse features even for classification tasks, see e.g. Figure 4 in "Towards Learning Convolutions from Scratch" (https://papers.nips.cc/paper/2020/file/5c528e25e1fdeaf9d8160dc24dbf4d60-Paper.pdf)
* There are some issues regarding novelty/clarity (see next section)

The evidence to be evaluated is:Top-K activation functions inspired by inhibitory interneurons have been implemented in a number of deep learning models for greater computational efficiency (Ahmad & Scheinkman, 2019), adversarial robustness (Xiao et al., 2020; Tramer et al., 2020), interpretability (Nelson et al., 2022), and continual learning (Shen et al., 2021; Iyer et al., 2022).

SDM in particular can be viewed as a de-noising autoencoder and has been developed into an MLP that uses a Top-K activation function (Keeler, 1988). Finally, de-noising autoencoders are single step versions of diffusion models (Song & Ermon, 2019).

By default, we train a single hidden layer ReLU network with different amounts of random Gaussian noise applied to the input. The objective is to remove the noise and reconstruct the original input. Formally, noise is applied from an isotropic Gaussian with variance \(\sigma\) to the input \(\mathbf{x}\). The corrupted image is then put through our encoder and decoder.

Figure 2 shows that a single hidden layer ReLU network with 10,000 neurons learns to become a Top-K network with smaller \(k\) values as the amount of noise increases. This network was trained on the CIFAR10 embeddings to mimick training within a deeper network layer and where the manifold is smoother than when training on raw image pixels. While Fig. 2 only shows the mean number of neurons that are active across all 50,000 CIFAR10 inputs per epoch, Figure 3 shows the fraction of active neurons for every input.

We compared this network with explicitly creating sparse Top-K networks. These Top-K networks sort the neuron activations and keep only the \(k\) most active. They were either given fixed \(k\) values or could learn the \(k\) values using either a continuous relaxation that is differentiable or the REINFORCE algorithm (see Appendix A for details) (Williams, 2004). These approaches all performed worse than the shared bias (and default ReLU) networks, with the best networks getting similar validation accuracies as the much smaller 100 neuron network and only when they were not sparse, using \(k=3,000\) neurons. Even the approaches learning \(k\) were insufficiently dynamic and depended too much upon the \(k\) initialization value.

We believe this is due to the fact that classification only needs to cluster the data into the ten CIFAR10 labels, creating less interference between neural representations. There is also no reduction in reconstruction or classification performance. In fact, for classification \(\sigma\in\{0.1,0.3\}\) both get 94.3% validation accuracy instead of the 93.4% that the baseline gets, showing slightly better generalization while also being more sparse (66% for baseline versus 55% for \(\sigma=0.1\) and 45% for \(\sigma=0.3\)).

**Model Ablations -** Figure 7 shows the mean neuron sparsity at network convergence across a number of different experimental conditions: (i) three datasets: CIFAR10 pixels, CIFAR10 latent embeddings, MNIST; (ii) two noise distributions: Gaussian and Laplace; (iii) four numbers of neurons: 100, 1,000, 10,000 and 100,000; (iv) three training tasks: reconstruction, classification, and Transformer next token prediction.

While still neither activation function becomes as sparse as ReLU, the GELU networks implement the same Top-K convergence as ReLU shown in Fig. 3 and form biologically plausible receptive fields like those of Fig. 5.

While we just show the mean active neurons here, both solutions look like those of Fig. 3 with the neurons for all inputs converging to a Top-K solution where the bias terms become negative and synchronize along with the key vector \(L_{2}\) norms.

Input: Figure 7: Training ablations. The positive correlation between noise and sparsity is robust to (a) neuron counts, (b) datasets, (c) noise distributions, and (d) training tasks.

Input: Figure 2: There is a positive relationship between activation sparsity and noise. The average fraction of neurons active during each CIFAR10 latent input over the course of training for 800 epochs. Each line corresponds to training randomly initialized networks with different noise levels σ denoted by different colors. We show the average of three different random seeds and their standard error of the mean (not visible as the variance is so low). Interestingly, the higher noise levels take longer to sparsify but upon convergence become the sparsest.

Input: Figure 3: Noise induces the formation of Top-K networks. Shown column-wise are the fraction of neurons that are active for each training input (Left), the values of each of the 10,000 neurons bias terms (Middle) and the L2 norms of their key vectors (Right), as a function of number of training epochs. Each is shown for three different representative noise levels (rows). These plots show each input to the network and use the density of blue to represent how many times a particular value occurs.

Input: Figure 6: Noisy pretraining retains network sparsity. The vertical purple lines denote the start of noise annealing where each noise level is linearly decayed down to σ = 0. For both reconstruction (left) and classification (right) tasks the networks remain highly sparse even after all noise is removed. Moreover, there is no noticeable performance difference between any of the networks while some of those trained with moderate noise perform slightly better than the noise free baseline. We truncate the x-axis to start at epoch 600 for the sake of clarity. The classification networks continue to sparsify even when noise annealing has started because unlike in the reconstruction task, the sparsity level did not converge within the first 800 epochs.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim "We find this Top-K network formation is particularly evident for reconstruction tasks but also exists for classification tasks and the intermediate MLP layers of a Transformer architecture" appears to be well-supported but slightly overstated.

The evidence shows:
- Figure 7(d) clearly demonstrates sparsification occurs across all tasks, with reconstruction showing strongest effects
- Figure 6 shows sparsification in both reconstruction and classification tasks
- The reviewer correctly points out that sparsity is "much less pronounced" for classification tasks
- While transformer results are mentioned, the evidence presented focuses primarily on single-layer networks
- The language "particularly evident" accurately reflects the stronger effect in reconstruction tasks

The claim would be more precise if it acknowledged the limited scope of transformer evidence and the weaker effect in classification tasks, but overall represents the findings reasonably well.

<score>2</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We find this Top-K network formation is particularly evident for reconstruction tasks but also exists for classification tasks and the intermediate MLP layers of a Transformer architecture.

The review comment to be evaluated is:

In this empirical paper, the authors observe the correlation between the sparsity of the learned representations and the level of noise injected into the input of Neural Networks with the ReLU activation function. Experiments have been carried out on CIFAR10 both on pixel data and on a neural representation, MNIST, and on the WikiText-103 dataset. They report a positive correlation between the magnitude of the injected noise and the sparsity of the representation. They observe Gabor-like filters when reconstructing with noise on pixel data. For the classification task, the performance of the model did not decrease with sparsity. Also, compared with using only the top-K activations of a model learned without noise, the models learned with noise gave better validation accuracies.

The evidence to be evaluated is:Top-K activation functions inspired by inhibitory interneurons have been implemented in a number of deep learning models for greater computational efficiency (Ahmad & Scheinkman, 2019), adversarial robustness (Xiao et al., 2020; Tramer et al., 2020), interpretability (Nelson et al., 2022), and continual learning (Shen et al., 2021; Iyer et al., 2022).

SDM in particular can be viewed as a de-noising autoencoder and has been developed into an MLP that uses a Top-K activation function (Keeler, 1988). Finally, de-noising autoencoders are single step versions of diffusion models (Song & Ermon, 2019).

By default, we train a single hidden layer ReLU network with different amounts of random Gaussian noise applied to the input. The objective is to remove the noise and reconstruct the original input. Formally, noise is applied from an isotropic Gaussian with variance \(\sigma\) to the input \(\mathbf{x}\). The corrupted image is then put through our encoder and decoder.

Figure 2 shows that a single hidden layer ReLU network with 10,000 neurons learns to become a Top-K network with smaller \(k\) values as the amount of noise increases. This network was trained on the CIFAR10 embeddings to mimick training within a deeper network layer and where the manifold is smoother than when training on raw image pixels. While Fig. 2 only shows the mean number of neurons that are active across all 50,000 CIFAR10 inputs per epoch, Figure 3 shows the fraction of active neurons for every input.

We compared this network with explicitly creating sparse Top-K networks. These Top-K networks sort the neuron activations and keep only the \(k\) most active. They were either given fixed \(k\) values or could learn the \(k\) values using either a continuous relaxation that is differentiable or the REINFORCE algorithm (see Appendix A for details) (Williams, 2004). These approaches all performed worse than the shared bias (and default ReLU) networks, with the best networks getting similar validation accuracies as the much smaller 100 neuron network and only when they were not sparse, using \(k=3,000\) neurons. Even the approaches learning \(k\) were insufficiently dynamic and depended too much upon the \(k\) initialization value.

We believe this is due to the fact that classification only needs to cluster the data into the ten CIFAR10 labels, creating less interference between neural representations. There is also no reduction in reconstruction or classification performance. In fact, for classification \(\sigma\in\{0.1,0.3\}\) both get 94.3% validation accuracy instead of the 93.4% that the baseline gets, showing slightly better generalization while also being more sparse (66% for baseline versus 55% for \(\sigma=0.1\) and 45% for \(\sigma=0.3\)).

**Model Ablations -** Figure 7 shows the mean neuron sparsity at network convergence across a number of different experimental conditions: (i) three datasets: CIFAR10 pixels, CIFAR10 latent embeddings, MNIST; (ii) two noise distributions: Gaussian and Laplace; (iii) four numbers of neurons: 100, 1,000, 10,000 and 100,000; (iv) three training tasks: reconstruction, classification, and Transformer next token prediction.

While still neither activation function becomes as sparse as ReLU, the GELU networks implement the same Top-K convergence as ReLU shown in Fig. 3 and form biologically plausible receptive fields like those of Fig. 5.

While we just show the mean active neurons here, both solutions look like those of Fig. 3 with the neurons for all inputs converging to a Top-K solution where the bias terms become negative and synchronize along with the key vector \(L_{2}\) norms.

Input: Figure 7: Training ablations. The positive correlation between noise and sparsity is robust to (a) neuron counts, (b) datasets, (c) noise distributions, and (d) training tasks.

Input: Figure 2: There is a positive relationship between activation sparsity and noise. The average fraction of neurons active during each CIFAR10 latent input over the course of training for 800 epochs. Each line corresponds to training randomly initialized networks with different noise levels σ denoted by different colors. We show the average of three different random seeds and their standard error of the mean (not visible as the variance is so low). Interestingly, the higher noise levels take longer to sparsify but upon convergence become the sparsest.

Input: Figure 3: Noise induces the formation of Top-K networks. Shown column-wise are the fraction of neurons that are active for each training input (Left), the values of each of the 10,000 neurons bias terms (Middle) and the L2 norms of their key vectors (Right), as a function of number of training epochs. Each is shown for three different representative noise levels (rows). These plots show each input to the network and use the density of blue to represent how many times a particular value occurs.

Input: Figure 6: Noisy pretraining retains network sparsity. The vertical purple lines denote the start of noise annealing where each noise level is linearly decayed down to σ = 0. For both reconstruction (left) and classification (right) tasks the networks remain highly sparse even after all noise is removed. Moreover, there is no noticeable performance difference between any of the networks while some of those trained with moderate noise perform slightly better than the noise free baseline. We truncate the x-axis to start at epoch 600 for the sake of clarity. The classification networks continue to sparsify even when noise annealing has started because unlike in the reconstruction task, the sparsity level did not converge within the first 800 epochs.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim that "Top-K network formation is particularly evident for reconstruction tasks but also exists for classification tasks and intermediate MLP layers of a Transformer architecture" is well-supported by the data. Figure 7(d) clearly shows the sparsity patterns across different tasks, with reconstruction showing stronger Top-K formation (lower fraction of active neurons) compared to classification and language modeling tasks. The experimental evidence is comprehensive, covering multiple datasets, neuron counts, noise distributions, and tasks. The authors acknowledge task-specific differences and provide detailed empirical support without overstating the findings.

<score>1</score>

Justification:
1. Direct evidence: Figure 7(d) demonstrates Top-K formation across all tasks, with reconstruction showing the strongest effect
2. Comprehensive validation: Results are robust across multiple experimental conditions (Fig 7a-d)
3. Performance validation: For classification, they show maintained or improved accuracy (94.3% vs 93.4% baseline) despite increased sparsity
4. Measured language: The claim uses appropriate qualifiers ("particularly evident" vs "also exists") that accurately reflect the relative strength of the effect across tasks
5. Thorough methodology: Results are validated across multiple architectures, datasets, and conditions
6. Conservative interpretation: The authors don't overextend beyond what the data shows


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We also observe that the network retains its Top-K approximation even after the noise is removed, making it an effective pre-training task for sparsifying activations.

The evidence to be evaluated is:From this perspective, it makes sense that sparsity can be used to turn off neurons and minimize this Jacobian, providing a potential explanation for our results.

Approaches to introduce activation sparsity have included explicitly using Top-K activation functions (Ranzato et al., 2007; Makhzani and Frey, 2014; Ahmad and Scheinkman, 2019), novel regularization terms (Kurtz et al., 2020; Yang et al., 2020) and other approaches (Schwarz et al., 2021; Molchanov et al., 2017). However, we are unaware of any existing work which finds that noisy training with a ReLU activation function not only results in sparsity but also converges to a sparse coding, Top-K network.

Top-K activation functions inspired by inhibitory interneurons have been implemented in a number of deep learning models for greater computational efficiency (Ahmad & Scheinkman, 2019), adversarial robustness (Xiao et al., 2020; Tramer et al., 2020), interpretability (Nelson et al., 2022), and continual learning (Shen et al., 2021; Iyer et al., 2022).

These Top-K networks, when trained with noise, are also closely related to associative memory models such as Sparse Distributed Memory (SDM) and Modern Hopfield Networks that activate a sparse subset of neurons and were developed to handle noisy queries (Kanerva, 1988; Krotov & Hopfield, 2016). SDM in particular can be viewed as a de-noising autoencoder and has been developed into an MLP that uses a Top-K activation function (Keeler, 1988).

By default, we train a single hidden layer ReLU network with different amounts of random Gaussian noise applied to the input. The objective is to remove the noise and reconstruct the original input. Formally, noise is applied from an isotropic Gaussian with variance \(\sigma\) to the input \(\mathbf{x}\). The corrupted image is then put through our encoder and decoder.

To ensure our results are robust, we vary a number of settings, finding that in every case the relationship between noise and network sparsity holds.

Figure 2 shows that a single hidden layer ReLU network with 10,000 neurons learns to become a Top-K network with smaller \(k\) values as the amount of noise increases. This network was trained on the CIFAR10 embeddings to mimick training within a deeper network layer and where the manifold is smoother than when training on raw image pixels. While Fig. 2 only shows the mean number of neurons that are active across all 50,000 CIFAR10 inputs per epoch, Figure 3 shows the fraction of active neurons for every input.

It is clear that these values all synchronize to a value decided on by the network and are responsible for the sparse \(k\) neurons remaining active after the ReLU nonlinearity is applied. As noise increases, the bias terms become more negative to raise the activation threshold and increase sparsity. The \(L_{2}\) norms of the key vectors also become smaller. We hypothesize this is a counter-response to the \(L_{2}\) norms of the data increasing with noise and explains why in high noise settings (\(\sigma\geq 0.8\)) neuron activations linger around \(\sim\)50% sparsity before rapidly becoming very sparse (see Fig. 2).

This means the network must wait to first shrink its key vector \(L_{2}\) norms before the bias terms can affect activation sparsity.

We compared this network with explicitly creating sparse Top-K networks. These Top-K networks sort the neuron activations and keep only the \(k\) most active.

These approaches all performed worse than the shared bias (and default ReLU) networks, with the best networks getting similar validation accuracies as the much smaller 100 neuron network and only when they were not sparse, using \(k=3,000\) neurons. Even the approaches learning \(k\) were insufficiently dynamic and depended too much upon the \(k\) initialization value.

This is a similar result to that of sparse coding (Olshausen & Field, 1997), however, here we use noise with the ReLU activation function instead of a sparsity penalty and train on full images instead of 8x8 patches.

When we tried training the network without any noise but instead with an \(L_{1}\) activation penalty motivated by the original sparse coding setup (Olshausen & Field, 1997), no biological receptive fields were observed. In addition, this approach resulted in unstable training, more dead neurons, less sparsity, and no synchronization of key vector norms or bias terms.1 Footnote 1: We did not implement the full sparse coding framework, only the \(L_{1}\) penalty, however, this shows that it is insufficient to implement sparse coding while noisy training is.

With these models we find that our networks become even more sparse and do so more quickly, as we would expect due to the explaining away of redundant neurons (see Appendix C).

Investigating our noise trained sparse networks, we find that even after the noise is removed, they remain sparse.

Fig. 6 shows that the sparsity levels remain largely unperturbed.

There is also no reduction in reconstruction or classification performance. In fact, for classification \(\sigma\in\{0.1,0.3\}\) both get 94.3% validation accuracy instead of the 93.4% that the baseline gets, showing slightly better generalization while also being more sparse (66% for baseline versus 55% for \(\sigma=0.1\) and 45% for \(\sigma=0.3\)).

We found that every optimizer with a high enough learning rate converged to a sparse Top-K solution without creating dead neurons.2

While still neither activation function becomes as sparse as ReLU, the GELU networks implement the same Top-K convergence as ReLU shown in Fig. 3 and form biologically plausible receptive fields like those of Fig. 5.

The correlation between noise and activation sparsity remains and is again retained even after noise is removed.

Figure 7(d) shows the final sparsity amounts of the first model layer with each of the other layers giving similar results with \(\sim\)3x to \(\sim\)9x more sparsity when \(\sigma=3.0\), compared to \(\sigma=0.0\).

An increase in noise will create more erroneous low-level activations in neurons that must be ignored to maximize signal to noise ratio. Therefore, during noisy training, the model must learn weight vectors to encode information about the data distribution that can be separated from the noise using only ReLU functions.

Similarly, we hypothesize that the encoding layer in our network is learning such projections, so that thresholding performs noise rejection while also producing sparse activations.

Moreover, the fact that noise alone results in sparse coding rather than having a sparsity penalty in the loss function suggests that the primary driver behind sparse coding may be the handling of noise with metabolic efficiency as an added benefit, rather than the other way around.

Meanwhile, the advantages of noisy training for generalization, continual learning, and adversarial robustness have already been documented.

**Conclusion -** We have shown that the realistic assumption of noisy training data is sufficient for feed-forward layers to organically become highly sparse and retain their sparsity, even after noise is removed. This result is particularly striking because of the way the networks converge with sparse coding, producing biological receptive fields and approximating an inhibitory interneuron that implements a Top-K operation.

There are two different approaches taken to learn the \(k\) values in the explicit Top-K networks. The first is to use a continuous relaxation of the \(k\) value that is otherwise a non-differentiable integer. We create a window of size \(w\) and apply a sigmoid weighting to all values between \(k\pm w\).

Training the same networks as before but using Eq. 2 with \(t^{*}=1\) and \(\gamma=1\) we present our results in Table 1. We show the sparsity amounts of the networks in Fig. 11.

Removing the bias terms and relying just on the inhibitory interneuron, we found that it performs well (Table 1) and becomes sparse as a function of noise but results in a number of dead neurons and is not as sparse as the baseline ReLU or Explaining Away models.

While we just show the mean active neurons here, both solutions look like those of Fig. 3 with the neurons for all inputs converging to a Top-K solution where the bias terms become negative and synchronize along with the key vector \(L_{2}\) norms.

Input: Table 1: Reconstruction Validation Accuracy - The baseline ReLU model varying the number of neurons, compared to the shared bias, explaining away, and inhibitory interneuron networks. We only show noise up to σ = 0.8 because the higher noise levels present trivial solutions that all networks perform equally well on. Due to very high convergence between random seeds, we just present a single run for each model.

Input: Figure 7: Training ablations. The positive correlation between noise and sparsity is robust to (a) neuron counts, (b) datasets, (c) noise distributions, and (d) training tasks.

Input: Figure 2: There is a positive relationship between activation sparsity and noise. The average fraction of neurons active during each CIFAR10 latent input over the course of training for 800 epochs. Each line corresponds to training randomly initialized networks with different noise levels σ denoted by different colors. We show the average of three different random seeds and their standard error of the mean (not visible as the variance is so low). Interestingly, the higher noise levels take longer to sparsify but upon convergence become the sparsest.

Input: Figure 3: Noise induces the formation of Top-K networks. Shown column-wise are the fraction of neurons that are active for each training input (Left), the values of each of the 10,000 neurons bias terms (Middle) and the L2 norms of their key vectors (Right), as a function of number of training epochs. Each is shown for three different representative noise levels (rows). These plots show each input to the network and use the density of blue to represent how many times a particular value occurs.

Input: Figure 6: Noisy pretraining retains network sparsity. The vertical purple lines denote the start of noise annealing where each noise level is linearly decayed down to σ = 0. For both reconstruction (left) and classification (right) tasks the networks remain highly sparse even after all noise is removed. Moreover, there is no noticeable performance difference between any of the networks while some of those trained with moderate noise perform slightly better than the noise free baseline. We truncate the x-axis to start at epoch 600 for the sake of clarity. The classification networks continue to sparsify even when noise annealing has started because unlike in the reconstruction task, the sparsity level did not converge within the first 800 epochs.

Input: Figure 14: The GELU network sparsifies with noise when trained on CIFAR10. The GELU network closely resembles the ReLU network by sparsifying (but not to the same degree). It also forms biological receptive fields. We use the arbitrary 0.0001 absolute activation threshold to label if a neuron is on or off.

Input: Figure 12: Dead Neurons during the reconstruction task. During noise annealing, the higher noise levels σ ≥ 0.8 see a number of dead neurons. However, this does not full explain their sparsity levels and on this trivial task it does not hurt their reconstruction performance.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate the claim "We also observe that the network retains its Top-K approximation even after the noise is removed, making it an effective pre-training task for sparsifying activations."

Summary of evaluation:
The evidence strongly supports this claim through multiple figures and analyses. Figure 6 directly demonstrates that networks maintain their sparsity levels after noise is removed during both reconstruction and classification tasks. The retention of sparsity is shown across different noise levels, with some noise-trained networks even performing slightly better than baselines. The claim is measured in its language, acknowledging the "approximation" nature of the Top-K behavior, and is well-grounded in the empirical evidence presented.

<score>1</score>

Justification:
1. Direct evidence from Figure 6 shows sparsity retention after noise removal
2. Validation through multiple tasks (reconstruction and classification)
3. Performance is maintained or improved (94.3% vs 93.4% baseline accuracy)
4. Conservative language ("approximation") matches the evidence
5. Results are robust across different experimental conditions (neuron counts, datasets, noise distributions - Figure 7)
6. Tables and figures provide quantitative support for the claim
7. Dead neuron analysis (Figure 12) acknowledges and accounts for potential confounding factors

The claim is well-stated and accurately reflects the evidence without exaggeration.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We also observe that the network retains its Top-K approximation even after the noise is removed, making it an effective pre-training task for sparsifying activations.

The review comment to be evaluated is:

Strengths

The paper is simple and easy to follow.

The sparsity effects shown appear relatively large and robust.

Adding noise is a simple manipulation that can have benefits beyond sparsity, and so there is some appeal to using this manipulation as a way to generate sparsity.

Weaknesses

It is odd that the authors focus so much on a one-layer model. For this finding to be broadly relevant, it is important to test whether this effect occurs in large-scale DNNs and whether networks can generate sparsity while maintaining good performance on real-world tasks. The only thing that is done along these lines is with a single, small GPT2 model.

It appears you need fairly high levels of noise to generate substantial sparsity (e.g., sigma=0.3 to get below 20% activation in Figure 2, which based on Figure 4 is quite a lot of noise). It seems this might limit the practical utility of the method, though it is hard to say in part because experiments are mostly limited to the single-layer training regime.

The link with biological receptive fields is tenuous. There are multiple ways of generating center-surround and oriented/bandpass receptive fields (e.g., whitening for center-surround RFs, sparsity for oriented RFs, CNNs often exhibit similar structure at the first layer). It is not clear this approach provides a better fit to biological data than these alternatives and there is no quantification of the match to biology or model comparison. Moreover, there are so many differences between the models trained here and biological networks, that it is hard to know the relevance of this finding, particularly given that the results can vary with the activation function (e.g., the sigmoid activation function does not generate sparsity) and the optimization algorithm (e.g., SGD with a low learning rate apparently does not generate sparsity).

The evidence to be evaluated is:From this perspective, it makes sense that sparsity can be used to turn off neurons and minimize this Jacobian, providing a potential explanation for our results.

Approaches to introduce activation sparsity have included explicitly using Top-K activation functions (Ranzato et al., 2007; Makhzani and Frey, 2014; Ahmad and Scheinkman, 2019), novel regularization terms (Kurtz et al., 2020; Yang et al., 2020) and other approaches (Schwarz et al., 2021; Molchanov et al., 2017). However, we are unaware of any existing work which finds that noisy training with a ReLU activation function not only results in sparsity but also converges to a sparse coding, Top-K network.

Top-K activation functions inspired by inhibitory interneurons have been implemented in a number of deep learning models for greater computational efficiency (Ahmad & Scheinkman, 2019), adversarial robustness (Xiao et al., 2020; Tramer et al., 2020), interpretability (Nelson et al., 2022), and continual learning (Shen et al., 2021; Iyer et al., 2022).

These Top-K networks, when trained with noise, are also closely related to associative memory models such as Sparse Distributed Memory (SDM) and Modern Hopfield Networks that activate a sparse subset of neurons and were developed to handle noisy queries (Kanerva, 1988; Krotov & Hopfield, 2016). SDM in particular can be viewed as a de-noising autoencoder and has been developed into an MLP that uses a Top-K activation function (Keeler, 1988).

By default, we train a single hidden layer ReLU network with different amounts of random Gaussian noise applied to the input. The objective is to remove the noise and reconstruct the original input. Formally, noise is applied from an isotropic Gaussian with variance \(\sigma\) to the input \(\mathbf{x}\). The corrupted image is then put through our encoder and decoder.

To ensure our results are robust, we vary a number of settings, finding that in every case the relationship between noise and network sparsity holds.

Figure 2 shows that a single hidden layer ReLU network with 10,000 neurons learns to become a Top-K network with smaller \(k\) values as the amount of noise increases. This network was trained on the CIFAR10 embeddings to mimick training within a deeper network layer and where the manifold is smoother than when training on raw image pixels. While Fig. 2 only shows the mean number of neurons that are active across all 50,000 CIFAR10 inputs per epoch, Figure 3 shows the fraction of active neurons for every input.

It is clear that these values all synchronize to a value decided on by the network and are responsible for the sparse \(k\) neurons remaining active after the ReLU nonlinearity is applied. As noise increases, the bias terms become more negative to raise the activation threshold and increase sparsity. The \(L_{2}\) norms of the key vectors also become smaller. We hypothesize this is a counter-response to the \(L_{2}\) norms of the data increasing with noise and explains why in high noise settings (\(\sigma\geq 0.8\)) neuron activations linger around \(\sim\)50% sparsity before rapidly becoming very sparse (see Fig. 2).

This means the network must wait to first shrink its key vector \(L_{2}\) norms before the bias terms can affect activation sparsity.

We compared this network with explicitly creating sparse Top-K networks. These Top-K networks sort the neuron activations and keep only the \(k\) most active.

These approaches all performed worse than the shared bias (and default ReLU) networks, with the best networks getting similar validation accuracies as the much smaller 100 neuron network and only when they were not sparse, using \(k=3,000\) neurons. Even the approaches learning \(k\) were insufficiently dynamic and depended too much upon the \(k\) initialization value.

This is a similar result to that of sparse coding (Olshausen & Field, 1997), however, here we use noise with the ReLU activation function instead of a sparsity penalty and train on full images instead of 8x8 patches.

When we tried training the network without any noise but instead with an \(L_{1}\) activation penalty motivated by the original sparse coding setup (Olshausen & Field, 1997), no biological receptive fields were observed. In addition, this approach resulted in unstable training, more dead neurons, less sparsity, and no synchronization of key vector norms or bias terms.1 Footnote 1: We did not implement the full sparse coding framework, only the \(L_{1}\) penalty, however, this shows that it is insufficient to implement sparse coding while noisy training is.

With these models we find that our networks become even more sparse and do so more quickly, as we would expect due to the explaining away of redundant neurons (see Appendix C).

Investigating our noise trained sparse networks, we find that even after the noise is removed, they remain sparse.

Fig. 6 shows that the sparsity levels remain largely unperturbed.

There is also no reduction in reconstruction or classification performance. In fact, for classification \(\sigma\in\{0.1,0.3\}\) both get 94.3% validation accuracy instead of the 93.4% that the baseline gets, showing slightly better generalization while also being more sparse (66% for baseline versus 55% for \(\sigma=0.1\) and 45% for \(\sigma=0.3\)).

We found that every optimizer with a high enough learning rate converged to a sparse Top-K solution without creating dead neurons.2

While still neither activation function becomes as sparse as ReLU, the GELU networks implement the same Top-K convergence as ReLU shown in Fig. 3 and form biologically plausible receptive fields like those of Fig. 5.

The correlation between noise and activation sparsity remains and is again retained even after noise is removed.

Figure 7(d) shows the final sparsity amounts of the first model layer with each of the other layers giving similar results with \(\sim\)3x to \(\sim\)9x more sparsity when \(\sigma=3.0\), compared to \(\sigma=0.0\).

An increase in noise will create more erroneous low-level activations in neurons that must be ignored to maximize signal to noise ratio. Therefore, during noisy training, the model must learn weight vectors to encode information about the data distribution that can be separated from the noise using only ReLU functions.

Similarly, we hypothesize that the encoding layer in our network is learning such projections, so that thresholding performs noise rejection while also producing sparse activations.

Moreover, the fact that noise alone results in sparse coding rather than having a sparsity penalty in the loss function suggests that the primary driver behind sparse coding may be the handling of noise with metabolic efficiency as an added benefit, rather than the other way around.

Meanwhile, the advantages of noisy training for generalization, continual learning, and adversarial robustness have already been documented.

**Conclusion -** We have shown that the realistic assumption of noisy training data is sufficient for feed-forward layers to organically become highly sparse and retain their sparsity, even after noise is removed. This result is particularly striking because of the way the networks converge with sparse coding, producing biological receptive fields and approximating an inhibitory interneuron that implements a Top-K operation.

There are two different approaches taken to learn the \(k\) values in the explicit Top-K networks. The first is to use a continuous relaxation of the \(k\) value that is otherwise a non-differentiable integer. We create a window of size \(w\) and apply a sigmoid weighting to all values between \(k\pm w\).

Training the same networks as before but using Eq. 2 with \(t^{*}=1\) and \(\gamma=1\) we present our results in Table 1. We show the sparsity amounts of the networks in Fig. 11.

Removing the bias terms and relying just on the inhibitory interneuron, we found that it performs well (Table 1) and becomes sparse as a function of noise but results in a number of dead neurons and is not as sparse as the baseline ReLU or Explaining Away models.

While we just show the mean active neurons here, both solutions look like those of Fig. 3 with the neurons for all inputs converging to a Top-K solution where the bias terms become negative and synchronize along with the key vector \(L_{2}\) norms.

Input: Table 1: Reconstruction Validation Accuracy - The baseline ReLU model varying the number of neurons, compared to the shared bias, explaining away, and inhibitory interneuron networks. We only show noise up to σ = 0.8 because the higher noise levels present trivial solutions that all networks perform equally well on. Due to very high convergence between random seeds, we just present a single run for each model.

Input: Figure 7: Training ablations. The positive correlation between noise and sparsity is robust to (a) neuron counts, (b) datasets, (c) noise distributions, and (d) training tasks.

Input: Figure 2: There is a positive relationship between activation sparsity and noise. The average fraction of neurons active during each CIFAR10 latent input over the course of training for 800 epochs. Each line corresponds to training randomly initialized networks with different noise levels σ denoted by different colors. We show the average of three different random seeds and their standard error of the mean (not visible as the variance is so low). Interestingly, the higher noise levels take longer to sparsify but upon convergence become the sparsest.

Input: Figure 3: Noise induces the formation of Top-K networks. Shown column-wise are the fraction of neurons that are active for each training input (Left), the values of each of the 10,000 neurons bias terms (Middle) and the L2 norms of their key vectors (Right), as a function of number of training epochs. Each is shown for three different representative noise levels (rows). These plots show each input to the network and use the density of blue to represent how many times a particular value occurs.

Input: Figure 6: Noisy pretraining retains network sparsity. The vertical purple lines denote the start of noise annealing where each noise level is linearly decayed down to σ = 0. For both reconstruction (left) and classification (right) tasks the networks remain highly sparse even after all noise is removed. Moreover, there is no noticeable performance difference between any of the networks while some of those trained with moderate noise perform slightly better than the noise free baseline. We truncate the x-axis to start at epoch 600 for the sake of clarity. The classification networks continue to sparsify even when noise annealing has started because unlike in the reconstruction task, the sparsity level did not converge within the first 800 epochs.

Input: Figure 14: The GELU network sparsifies with noise when trained on CIFAR10. The GELU network closely resembles the ReLU network by sparsifying (but not to the same degree). It also forms biological receptive fields. We use the arbitrary 0.0001 absolute activation threshold to label if a neuron is on or off.

Input: Figure 12: Dead Neurons during the reconstruction task. During noise annealing, the higher noise levels σ ≥ 0.8 see a number of dead neurons. However, this does not full explain their sparsity levels and on this trivial task it does not hurt their reconstruction performance.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim "the network retains its Top-K approximation even after the noise is removed" appears to be well-supported. Figure 6 directly demonstrates that networks maintain their sparsity levels after noise is removed during both reconstruction and classification tasks. The evidence shows this retention across multiple experiments and settings. While there are some minor increases in active neurons after noise removal, the overall sparsity patterns remain largely stable. The review comment does not specifically challenge this particular claim about retention of sparsity after noise removal.

<score>1</score>

Justification:
1. Direct evidence from Figure 6 shows sparsity levels remain stable after noise removal (purple lines marking noise annealing)
2. Supporting evidence of retained sparsity found across:
- Different tasks (reconstruction and classification)
- Multiple noise levels
- Different network architectures (ReLU and GELU networks)
3. Quantitative results show maintained performance:
- Classification accuracy improved slightly (94.3% vs 93.4% baseline)
- No reduction in reconstruction performance
4. The claim is carefully worded to only state what is directly supported by the evidence without extrapolating beyond the findings
5. The reviewer comments, while critical of other aspects, do not dispute this specific claim about retention of sparsity

The claim matches the evidence without overstatement, earning a score of 1 (well-stated).