Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We also observe that the network retains its Top-K approximation even after the noise is removed, making it an effective pre-training task for sparsifying activations.

The review comment to be evaluated is:

Strengths

1. The authors find a simple, implicit manner to create top-k networks that compares favorably to explicit methods

2. The method is tested in several different circumstances, with different tasks, numbers of neurons, and noise levels. Simple characteristics of the network, such as the fraction of active neurons and their biases, are well described.

Weaknesses

Major

1. As the authors observe, the results are recapitulated using a global negative bias term, acting as common inhibition that requires greater input values to activate neurons and resulting in top-k networks.  Thus, while such an observation may indeed be useful for those interested in such networks, the advance it comprises is a minor one.

Minor

1. While the presented results may bear on biological considerations, as suggested therein, caution is advised, as noise in biological networks is dynamic and thus not perfectly analogous to simply adding noise to inputs of a feedforward network.

The evidence to be evaluated is:From this perspective, it makes sense that sparsity can be used to turn off neurons and minimize this Jacobian, providing a potential explanation for our results.

Approaches to introduce activation sparsity have included explicitly using Top-K activation functions (Ranzato et al., 2007; Makhzani and Frey, 2014; Ahmad and Scheinkman, 2019), novel regularization terms (Kurtz et al., 2020; Yang et al., 2020) and other approaches (Schwarz et al., 2021; Molchanov et al., 2017). However, we are unaware of any existing work which finds that noisy training with a ReLU activation function not only results in sparsity but also converges to a sparse coding, Top-K network.

Top-K activation functions inspired by inhibitory interneurons have been implemented in a number of deep learning models for greater computational efficiency (Ahmad & Scheinkman, 2019), adversarial robustness (Xiao et al., 2020; Tramer et al., 2020), interpretability (Nelson et al., 2022), and continual learning (Shen et al., 2021; Iyer et al., 2022).

These Top-K networks, when trained with noise, are also closely related to associative memory models such as Sparse Distributed Memory (SDM) and Modern Hopfield Networks that activate a sparse subset of neurons and were developed to handle noisy queries (Kanerva, 1988; Krotov & Hopfield, 2016). SDM in particular can be viewed as a de-noising autoencoder and has been developed into an MLP that uses a Top-K activation function (Keeler, 1988).

By default, we train a single hidden layer ReLU network with different amounts of random Gaussian noise applied to the input. The objective is to remove the noise and reconstruct the original input. Formally, noise is applied from an isotropic Gaussian with variance \(\sigma\) to the input \(\mathbf{x}\). The corrupted image is then put through our encoder and decoder.

To ensure our results are robust, we vary a number of settings, finding that in every case the relationship between noise and network sparsity holds.

Figure 2 shows that a single hidden layer ReLU network with 10,000 neurons learns to become a Top-K network with smaller \(k\) values as the amount of noise increases. This network was trained on the CIFAR10 embeddings to mimick training within a deeper network layer and where the manifold is smoother than when training on raw image pixels. While Fig. 2 only shows the mean number of neurons that are active across all 50,000 CIFAR10 inputs per epoch, Figure 3 shows the fraction of active neurons for every input.

It is clear that these values all synchronize to a value decided on by the network and are responsible for the sparse \(k\) neurons remaining active after the ReLU nonlinearity is applied. As noise increases, the bias terms become more negative to raise the activation threshold and increase sparsity. The \(L_{2}\) norms of the key vectors also become smaller. We hypothesize this is a counter-response to the \(L_{2}\) norms of the data increasing with noise and explains why in high noise settings (\(\sigma\geq 0.8\)) neuron activations linger around \(\sim\)50% sparsity before rapidly becoming very sparse (see Fig. 2).

This means the network must wait to first shrink its key vector \(L_{2}\) norms before the bias terms can affect activation sparsity.

We compared this network with explicitly creating sparse Top-K networks. These Top-K networks sort the neuron activations and keep only the \(k\) most active.

These approaches all performed worse than the shared bias (and default ReLU) networks, with the best networks getting similar validation accuracies as the much smaller 100 neuron network and only when they were not sparse, using \(k=3,000\) neurons. Even the approaches learning \(k\) were insufficiently dynamic and depended too much upon the \(k\) initialization value.

This is a similar result to that of sparse coding (Olshausen & Field, 1997), however, here we use noise with the ReLU activation function instead of a sparsity penalty and train on full images instead of 8x8 patches.

When we tried training the network without any noise but instead with an \(L_{1}\) activation penalty motivated by the original sparse coding setup (Olshausen & Field, 1997), no biological receptive fields were observed. In addition, this approach resulted in unstable training, more dead neurons, less sparsity, and no synchronization of key vector norms or bias terms.1 Footnote 1: We did not implement the full sparse coding framework, only the \(L_{1}\) penalty, however, this shows that it is insufficient to implement sparse coding while noisy training is.

With these models we find that our networks become even more sparse and do so more quickly, as we would expect due to the explaining away of redundant neurons (see Appendix C).

Investigating our noise trained sparse networks, we find that even after the noise is removed, they remain sparse.

Fig. 6 shows that the sparsity levels remain largely unperturbed.

There is also no reduction in reconstruction or classification performance. In fact, for classification \(\sigma\in\{0.1,0.3\}\) both get 94.3% validation accuracy instead of the 93.4% that the baseline gets, showing slightly better generalization while also being more sparse (66% for baseline versus 55% for \(\sigma=0.1\) and 45% for \(\sigma=0.3\)).

We found that every optimizer with a high enough learning rate converged to a sparse Top-K solution without creating dead neurons.2

While still neither activation function becomes as sparse as ReLU, the GELU networks implement the same Top-K convergence as ReLU shown in Fig. 3 and form biologically plausible receptive fields like those of Fig. 5.

The correlation between noise and activation sparsity remains and is again retained even after noise is removed.

Figure 7(d) shows the final sparsity amounts of the first model layer with each of the other layers giving similar results with \(\sim\)3x to \(\sim\)9x more sparsity when \(\sigma=3.0\), compared to \(\sigma=0.0\).

An increase in noise will create more erroneous low-level activations in neurons that must be ignored to maximize signal to noise ratio. Therefore, during noisy training, the model must learn weight vectors to encode information about the data distribution that can be separated from the noise using only ReLU functions.

Similarly, we hypothesize that the encoding layer in our network is learning such projections, so that thresholding performs noise rejection while also producing sparse activations.

Moreover, the fact that noise alone results in sparse coding rather than having a sparsity penalty in the loss function suggests that the primary driver behind sparse coding may be the handling of noise with metabolic efficiency as an added benefit, rather than the other way around.

Meanwhile, the advantages of noisy training for generalization, continual learning, and adversarial robustness have already been documented.

**Conclusion -** We have shown that the realistic assumption of noisy training data is sufficient for feed-forward layers to organically become highly sparse and retain their sparsity, even after noise is removed. This result is particularly striking because of the way the networks converge with sparse coding, producing biological receptive fields and approximating an inhibitory interneuron that implements a Top-K operation.

There are two different approaches taken to learn the \(k\) values in the explicit Top-K networks. The first is to use a continuous relaxation of the \(k\) value that is otherwise a non-differentiable integer. We create a window of size \(w\) and apply a sigmoid weighting to all values between \(k\pm w\).

Training the same networks as before but using Eq. 2 with \(t^{*}=1\) and \(\gamma=1\) we present our results in Table 1. We show the sparsity amounts of the networks in Fig. 11.

Removing the bias terms and relying just on the inhibitory interneuron, we found that it performs well (Table 1) and becomes sparse as a function of noise but results in a number of dead neurons and is not as sparse as the baseline ReLU or Explaining Away models.

While we just show the mean active neurons here, both solutions look like those of Fig. 3 with the neurons for all inputs converging to a Top-K solution where the bias terms become negative and synchronize along with the key vector \(L_{2}\) norms.

Input: Table 1: Reconstruction Validation Accuracy - The baseline ReLU model varying the number of neurons, compared to the shared bias, explaining away, and inhibitory interneuron networks. We only show noise up to σ = 0.8 because the higher noise levels present trivial solutions that all networks perform equally well on. Due to very high convergence between random seeds, we just present a single run for each model.

Input: Figure 7: Training ablations. The positive correlation between noise and sparsity is robust to (a) neuron counts, (b) datasets, (c) noise distributions, and (d) training tasks.

Input: Figure 2: There is a positive relationship between activation sparsity and noise. The average fraction of neurons active during each CIFAR10 latent input over the course of training for 800 epochs. Each line corresponds to training randomly initialized networks with different noise levels σ denoted by different colors. We show the average of three different random seeds and their standard error of the mean (not visible as the variance is so low). Interestingly, the higher noise levels take longer to sparsify but upon convergence become the sparsest.

Input: Figure 3: Noise induces the formation of Top-K networks. Shown column-wise are the fraction of neurons that are active for each training input (Left), the values of each of the 10,000 neurons bias terms (Middle) and the L2 norms of their key vectors (Right), as a function of number of training epochs. Each is shown for three different representative noise levels (rows). These plots show each input to the network and use the density of blue to represent how many times a particular value occurs.

Input: Figure 6: Noisy pretraining retains network sparsity. The vertical purple lines denote the start of noise annealing where each noise level is linearly decayed down to σ = 0. For both reconstruction (left) and classification (right) tasks the networks remain highly sparse even after all noise is removed. Moreover, there is no noticeable performance difference between any of the networks while some of those trained with moderate noise perform slightly better than the noise free baseline. We truncate the x-axis to start at epoch 600 for the sake of clarity. The classification networks continue to sparsify even when noise annealing has started because unlike in the reconstruction task, the sparsity level did not converge within the first 800 epochs.

Input: Figure 14: The GELU network sparsifies with noise when trained on CIFAR10. The GELU network closely resembles the ReLU network by sparsifying (but not to the same degree). It also forms biological receptive fields. We use the arbitrary 0.0001 absolute activation threshold to label if a neuron is on or off.

Input: Figure 12: Dead Neurons during the reconstruction task. During noise annealing, the higher noise levels σ ≥ 0.8 see a number of dead neurons. However, this does not full explain their sparsity levels and on this trivial task it does not hurt their reconstruction performance.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me evaluate the claim: "We also observe that the network retains its Top-K approximation even after the noise is removed, making it an effective pre-training task for sparsifying activations."

Based on the evidence:
- Figure 6 directly demonstrates that networks maintain their sparsity levels after noise is removed (during noise annealing) for both reconstruction and classification tasks
- Table 1 shows comparable performance between different approaches
- The evidence shows this works across multiple settings (Figure 7) and activation functions (Figure 14 for GELU)
- However, Figure 12 indicates that for high noise levels (σ ≥ 0.8), some neurons become "dead" during noise annealing, which slightly qualifies the claim

The claim appears well-supported by comprehensive experimental evidence showing retention of sparsity across multiple conditions, with appropriate caveats noted about very high noise levels.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We also observe that the network retains its Top-K approximation even after the noise is removed, making it an effective pre-training task for sparsifying activations.

The review comment to be evaluated is:

Strengths:
* The paper tackles the important problem of bridging the gap between biological plausible learning and current artificial neural networks.

Weaknesses:
* The paper mostly focuses on 1-hidden layer networks. It is not too clear why and how these results are relevant for practice, where typically much deeper networks are considered. For classification tasks, the observed sparsity seems to be much less pronounced.
* Unlike claimed in the paper, using the L1-norm can lead to biologically plausible and sparse features even for classification tasks, see e.g. Figure 4 in "Towards Learning Convolutions from Scratch" (https://papers.nips.cc/paper/2020/file/5c528e25e1fdeaf9d8160dc24dbf4d60-Paper.pdf)
* There are some issues regarding novelty/clarity (see next section)

The evidence to be evaluated is:From this perspective, it makes sense that sparsity can be used to turn off neurons and minimize this Jacobian, providing a potential explanation for our results.

Approaches to introduce activation sparsity have included explicitly using Top-K activation functions (Ranzato et al., 2007; Makhzani and Frey, 2014; Ahmad and Scheinkman, 2019), novel regularization terms (Kurtz et al., 2020; Yang et al., 2020) and other approaches (Schwarz et al., 2021; Molchanov et al., 2017). However, we are unaware of any existing work which finds that noisy training with a ReLU activation function not only results in sparsity but also converges to a sparse coding, Top-K network.

Top-K activation functions inspired by inhibitory interneurons have been implemented in a number of deep learning models for greater computational efficiency (Ahmad & Scheinkman, 2019), adversarial robustness (Xiao et al., 2020; Tramer et al., 2020), interpretability (Nelson et al., 2022), and continual learning (Shen et al., 2021; Iyer et al., 2022).

These Top-K networks, when trained with noise, are also closely related to associative memory models such as Sparse Distributed Memory (SDM) and Modern Hopfield Networks that activate a sparse subset of neurons and were developed to handle noisy queries (Kanerva, 1988; Krotov & Hopfield, 2016). SDM in particular can be viewed as a de-noising autoencoder and has been developed into an MLP that uses a Top-K activation function (Keeler, 1988).

By default, we train a single hidden layer ReLU network with different amounts of random Gaussian noise applied to the input. The objective is to remove the noise and reconstruct the original input. Formally, noise is applied from an isotropic Gaussian with variance \(\sigma\) to the input \(\mathbf{x}\). The corrupted image is then put through our encoder and decoder.

To ensure our results are robust, we vary a number of settings, finding that in every case the relationship between noise and network sparsity holds.

Figure 2 shows that a single hidden layer ReLU network with 10,000 neurons learns to become a Top-K network with smaller \(k\) values as the amount of noise increases. This network was trained on the CIFAR10 embeddings to mimick training within a deeper network layer and where the manifold is smoother than when training on raw image pixels. While Fig. 2 only shows the mean number of neurons that are active across all 50,000 CIFAR10 inputs per epoch, Figure 3 shows the fraction of active neurons for every input.

It is clear that these values all synchronize to a value decided on by the network and are responsible for the sparse \(k\) neurons remaining active after the ReLU nonlinearity is applied. As noise increases, the bias terms become more negative to raise the activation threshold and increase sparsity. The \(L_{2}\) norms of the key vectors also become smaller. We hypothesize this is a counter-response to the \(L_{2}\) norms of the data increasing with noise and explains why in high noise settings (\(\sigma\geq 0.8\)) neuron activations linger around \(\sim\)50% sparsity before rapidly becoming very sparse (see Fig. 2).

This means the network must wait to first shrink its key vector \(L_{2}\) norms before the bias terms can affect activation sparsity.

We compared this network with explicitly creating sparse Top-K networks. These Top-K networks sort the neuron activations and keep only the \(k\) most active.

These approaches all performed worse than the shared bias (and default ReLU) networks, with the best networks getting similar validation accuracies as the much smaller 100 neuron network and only when they were not sparse, using \(k=3,000\) neurons. Even the approaches learning \(k\) were insufficiently dynamic and depended too much upon the \(k\) initialization value.

This is a similar result to that of sparse coding (Olshausen & Field, 1997), however, here we use noise with the ReLU activation function instead of a sparsity penalty and train on full images instead of 8x8 patches.

When we tried training the network without any noise but instead with an \(L_{1}\) activation penalty motivated by the original sparse coding setup (Olshausen & Field, 1997), no biological receptive fields were observed. In addition, this approach resulted in unstable training, more dead neurons, less sparsity, and no synchronization of key vector norms or bias terms.1 Footnote 1: We did not implement the full sparse coding framework, only the \(L_{1}\) penalty, however, this shows that it is insufficient to implement sparse coding while noisy training is.

With these models we find that our networks become even more sparse and do so more quickly, as we would expect due to the explaining away of redundant neurons (see Appendix C).

Investigating our noise trained sparse networks, we find that even after the noise is removed, they remain sparse.

Fig. 6 shows that the sparsity levels remain largely unperturbed.

There is also no reduction in reconstruction or classification performance. In fact, for classification \(\sigma\in\{0.1,0.3\}\) both get 94.3% validation accuracy instead of the 93.4% that the baseline gets, showing slightly better generalization while also being more sparse (66% for baseline versus 55% for \(\sigma=0.1\) and 45% for \(\sigma=0.3\)).

We found that every optimizer with a high enough learning rate converged to a sparse Top-K solution without creating dead neurons.2

While still neither activation function becomes as sparse as ReLU, the GELU networks implement the same Top-K convergence as ReLU shown in Fig. 3 and form biologically plausible receptive fields like those of Fig. 5.

The correlation between noise and activation sparsity remains and is again retained even after noise is removed.

Figure 7(d) shows the final sparsity amounts of the first model layer with each of the other layers giving similar results with \(\sim\)3x to \(\sim\)9x more sparsity when \(\sigma=3.0\), compared to \(\sigma=0.0\).

An increase in noise will create more erroneous low-level activations in neurons that must be ignored to maximize signal to noise ratio. Therefore, during noisy training, the model must learn weight vectors to encode information about the data distribution that can be separated from the noise using only ReLU functions.

Similarly, we hypothesize that the encoding layer in our network is learning such projections, so that thresholding performs noise rejection while also producing sparse activations.

Moreover, the fact that noise alone results in sparse coding rather than having a sparsity penalty in the loss function suggests that the primary driver behind sparse coding may be the handling of noise with metabolic efficiency as an added benefit, rather than the other way around.

Meanwhile, the advantages of noisy training for generalization, continual learning, and adversarial robustness have already been documented.

**Conclusion -** We have shown that the realistic assumption of noisy training data is sufficient for feed-forward layers to organically become highly sparse and retain their sparsity, even after noise is removed. This result is particularly striking because of the way the networks converge with sparse coding, producing biological receptive fields and approximating an inhibitory interneuron that implements a Top-K operation.

There are two different approaches taken to learn the \(k\) values in the explicit Top-K networks. The first is to use a continuous relaxation of the \(k\) value that is otherwise a non-differentiable integer. We create a window of size \(w\) and apply a sigmoid weighting to all values between \(k\pm w\).

Training the same networks as before but using Eq. 2 with \(t^{*}=1\) and \(\gamma=1\) we present our results in Table 1. We show the sparsity amounts of the networks in Fig. 11.

Removing the bias terms and relying just on the inhibitory interneuron, we found that it performs well (Table 1) and becomes sparse as a function of noise but results in a number of dead neurons and is not as sparse as the baseline ReLU or Explaining Away models.

While we just show the mean active neurons here, both solutions look like those of Fig. 3 with the neurons for all inputs converging to a Top-K solution where the bias terms become negative and synchronize along with the key vector \(L_{2}\) norms.

Input: Table 1: Reconstruction Validation Accuracy - The baseline ReLU model varying the number of neurons, compared to the shared bias, explaining away, and inhibitory interneuron networks. We only show noise up to σ = 0.8 because the higher noise levels present trivial solutions that all networks perform equally well on. Due to very high convergence between random seeds, we just present a single run for each model.

Input: Figure 7: Training ablations. The positive correlation between noise and sparsity is robust to (a) neuron counts, (b) datasets, (c) noise distributions, and (d) training tasks.

Input: Figure 2: There is a positive relationship between activation sparsity and noise. The average fraction of neurons active during each CIFAR10 latent input over the course of training for 800 epochs. Each line corresponds to training randomly initialized networks with different noise levels σ denoted by different colors. We show the average of three different random seeds and their standard error of the mean (not visible as the variance is so low). Interestingly, the higher noise levels take longer to sparsify but upon convergence become the sparsest.

Input: Figure 3: Noise induces the formation of Top-K networks. Shown column-wise are the fraction of neurons that are active for each training input (Left), the values of each of the 10,000 neurons bias terms (Middle) and the L2 norms of their key vectors (Right), as a function of number of training epochs. Each is shown for three different representative noise levels (rows). These plots show each input to the network and use the density of blue to represent how many times a particular value occurs.

Input: Figure 6: Noisy pretraining retains network sparsity. The vertical purple lines denote the start of noise annealing where each noise level is linearly decayed down to σ = 0. For both reconstruction (left) and classification (right) tasks the networks remain highly sparse even after all noise is removed. Moreover, there is no noticeable performance difference between any of the networks while some of those trained with moderate noise perform slightly better than the noise free baseline. We truncate the x-axis to start at epoch 600 for the sake of clarity. The classification networks continue to sparsify even when noise annealing has started because unlike in the reconstruction task, the sparsity level did not converge within the first 800 epochs.

Input: Figure 14: The GELU network sparsifies with noise when trained on CIFAR10. The GELU network closely resembles the ReLU network by sparsifying (but not to the same degree). It also forms biological receptive fields. We use the arbitrary 0.0001 absolute activation threshold to label if a neuron is on or off.

Input: Figure 12: Dead Neurons during the reconstruction task. During noise annealing, the higher noise levels σ ≥ 0.8 see a number of dead neurons. However, this does not full explain their sparsity levels and on this trivial task it does not hurt their reconstruction performance.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim "the network retains its Top-K approximation even after the noise is removed" is well-supported but slightly overstated. Figure 6 demonstrates that networks maintain significant sparsity after noise removal during both reconstruction and classification tasks. However, there are some nuances - the sparsity levels do shift slightly after noise removal, and the classification networks continue to sparsify even during noise annealing. The evidence shows retention of sparsification but not necessarily exact preservation of the Top-K approximation pattern.

<score>2</score>

Justification:
1. Strong supporting evidence:
- Figure 6 clearly shows maintained sparsity after noise removal
- Networks maintain performance levels after noise removal
- Classification networks with moderate noise (σ=0.1, 0.3) achieve slightly better accuracy (94.3% vs 93.4%)

2. Slight overstatement concerns:
- Some minor changes in sparsity levels occur after noise removal (visible in Figure 6)
- Classification networks continue evolving sparsity patterns during noise annealing
- Reviewer notes sparsity is "much less pronounced" for classification tasks

The claim's core observation about retained sparsity is valid, but the phrasing implies a more perfect preservation than the evidence demonstrates, warranting a score of 2 (mostly well-stated with minor overstatement).


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We also observe that the network retains its Top-K approximation even after the noise is removed, making it an effective pre-training task for sparsifying activations.

The review comment to be evaluated is:

In this empirical paper, the authors observe the correlation between the sparsity of the learned representations and the level of noise injected into the input of Neural Networks with the ReLU activation function. Experiments have been carried out on CIFAR10 both on pixel data and on a neural representation, MNIST, and on the WikiText-103 dataset. They report a positive correlation between the magnitude of the injected noise and the sparsity of the representation. They observe Gabor-like filters when reconstructing with noise on pixel data. For the classification task, the performance of the model did not decrease with sparsity. Also, compared with using only the top-K activations of a model learned without noise, the models learned with noise gave better validation accuracies.

The evidence to be evaluated is:From this perspective, it makes sense that sparsity can be used to turn off neurons and minimize this Jacobian, providing a potential explanation for our results.

Approaches to introduce activation sparsity have included explicitly using Top-K activation functions (Ranzato et al., 2007; Makhzani and Frey, 2014; Ahmad and Scheinkman, 2019), novel regularization terms (Kurtz et al., 2020; Yang et al., 2020) and other approaches (Schwarz et al., 2021; Molchanov et al., 2017). However, we are unaware of any existing work which finds that noisy training with a ReLU activation function not only results in sparsity but also converges to a sparse coding, Top-K network.

Top-K activation functions inspired by inhibitory interneurons have been implemented in a number of deep learning models for greater computational efficiency (Ahmad & Scheinkman, 2019), adversarial robustness (Xiao et al., 2020; Tramer et al., 2020), interpretability (Nelson et al., 2022), and continual learning (Shen et al., 2021; Iyer et al., 2022).

These Top-K networks, when trained with noise, are also closely related to associative memory models such as Sparse Distributed Memory (SDM) and Modern Hopfield Networks that activate a sparse subset of neurons and were developed to handle noisy queries (Kanerva, 1988; Krotov & Hopfield, 2016). SDM in particular can be viewed as a de-noising autoencoder and has been developed into an MLP that uses a Top-K activation function (Keeler, 1988).

By default, we train a single hidden layer ReLU network with different amounts of random Gaussian noise applied to the input. The objective is to remove the noise and reconstruct the original input. Formally, noise is applied from an isotropic Gaussian with variance \(\sigma\) to the input \(\mathbf{x}\). The corrupted image is then put through our encoder and decoder.

To ensure our results are robust, we vary a number of settings, finding that in every case the relationship between noise and network sparsity holds.

Figure 2 shows that a single hidden layer ReLU network with 10,000 neurons learns to become a Top-K network with smaller \(k\) values as the amount of noise increases. This network was trained on the CIFAR10 embeddings to mimick training within a deeper network layer and where the manifold is smoother than when training on raw image pixels. While Fig. 2 only shows the mean number of neurons that are active across all 50,000 CIFAR10 inputs per epoch, Figure 3 shows the fraction of active neurons for every input.

It is clear that these values all synchronize to a value decided on by the network and are responsible for the sparse \(k\) neurons remaining active after the ReLU nonlinearity is applied. As noise increases, the bias terms become more negative to raise the activation threshold and increase sparsity. The \(L_{2}\) norms of the key vectors also become smaller. We hypothesize this is a counter-response to the \(L_{2}\) norms of the data increasing with noise and explains why in high noise settings (\(\sigma\geq 0.8\)) neuron activations linger around \(\sim\)50% sparsity before rapidly becoming very sparse (see Fig. 2).

This means the network must wait to first shrink its key vector \(L_{2}\) norms before the bias terms can affect activation sparsity.

We compared this network with explicitly creating sparse Top-K networks. These Top-K networks sort the neuron activations and keep only the \(k\) most active.

These approaches all performed worse than the shared bias (and default ReLU) networks, with the best networks getting similar validation accuracies as the much smaller 100 neuron network and only when they were not sparse, using \(k=3,000\) neurons. Even the approaches learning \(k\) were insufficiently dynamic and depended too much upon the \(k\) initialization value.

This is a similar result to that of sparse coding (Olshausen & Field, 1997), however, here we use noise with the ReLU activation function instead of a sparsity penalty and train on full images instead of 8x8 patches.

When we tried training the network without any noise but instead with an \(L_{1}\) activation penalty motivated by the original sparse coding setup (Olshausen & Field, 1997), no biological receptive fields were observed. In addition, this approach resulted in unstable training, more dead neurons, less sparsity, and no synchronization of key vector norms or bias terms.1 Footnote 1: We did not implement the full sparse coding framework, only the \(L_{1}\) penalty, however, this shows that it is insufficient to implement sparse coding while noisy training is.

With these models we find that our networks become even more sparse and do so more quickly, as we would expect due to the explaining away of redundant neurons (see Appendix C).

Investigating our noise trained sparse networks, we find that even after the noise is removed, they remain sparse.

Fig. 6 shows that the sparsity levels remain largely unperturbed.

There is also no reduction in reconstruction or classification performance. In fact, for classification \(\sigma\in\{0.1,0.3\}\) both get 94.3% validation accuracy instead of the 93.4% that the baseline gets, showing slightly better generalization while also being more sparse (66% for baseline versus 55% for \(\sigma=0.1\) and 45% for \(\sigma=0.3\)).

We found that every optimizer with a high enough learning rate converged to a sparse Top-K solution without creating dead neurons.2

While still neither activation function becomes as sparse as ReLU, the GELU networks implement the same Top-K convergence as ReLU shown in Fig. 3 and form biologically plausible receptive fields like those of Fig. 5.

The correlation between noise and activation sparsity remains and is again retained even after noise is removed.

Figure 7(d) shows the final sparsity amounts of the first model layer with each of the other layers giving similar results with \(\sim\)3x to \(\sim\)9x more sparsity when \(\sigma=3.0\), compared to \(\sigma=0.0\).

An increase in noise will create more erroneous low-level activations in neurons that must be ignored to maximize signal to noise ratio. Therefore, during noisy training, the model must learn weight vectors to encode information about the data distribution that can be separated from the noise using only ReLU functions.

Similarly, we hypothesize that the encoding layer in our network is learning such projections, so that thresholding performs noise rejection while also producing sparse activations.

Moreover, the fact that noise alone results in sparse coding rather than having a sparsity penalty in the loss function suggests that the primary driver behind sparse coding may be the handling of noise with metabolic efficiency as an added benefit, rather than the other way around.

Meanwhile, the advantages of noisy training for generalization, continual learning, and adversarial robustness have already been documented.

**Conclusion -** We have shown that the realistic assumption of noisy training data is sufficient for feed-forward layers to organically become highly sparse and retain their sparsity, even after noise is removed. This result is particularly striking because of the way the networks converge with sparse coding, producing biological receptive fields and approximating an inhibitory interneuron that implements a Top-K operation.

There are two different approaches taken to learn the \(k\) values in the explicit Top-K networks. The first is to use a continuous relaxation of the \(k\) value that is otherwise a non-differentiable integer. We create a window of size \(w\) and apply a sigmoid weighting to all values between \(k\pm w\).

Training the same networks as before but using Eq. 2 with \(t^{*}=1\) and \(\gamma=1\) we present our results in Table 1. We show the sparsity amounts of the networks in Fig. 11.

Removing the bias terms and relying just on the inhibitory interneuron, we found that it performs well (Table 1) and becomes sparse as a function of noise but results in a number of dead neurons and is not as sparse as the baseline ReLU or Explaining Away models.

While we just show the mean active neurons here, both solutions look like those of Fig. 3 with the neurons for all inputs converging to a Top-K solution where the bias terms become negative and synchronize along with the key vector \(L_{2}\) norms.

Input: Table 1: Reconstruction Validation Accuracy - The baseline ReLU model varying the number of neurons, compared to the shared bias, explaining away, and inhibitory interneuron networks. We only show noise up to σ = 0.8 because the higher noise levels present trivial solutions that all networks perform equally well on. Due to very high convergence between random seeds, we just present a single run for each model.

Input: Figure 7: Training ablations. The positive correlation between noise and sparsity is robust to (a) neuron counts, (b) datasets, (c) noise distributions, and (d) training tasks.

Input: Figure 2: There is a positive relationship between activation sparsity and noise. The average fraction of neurons active during each CIFAR10 latent input over the course of training for 800 epochs. Each line corresponds to training randomly initialized networks with different noise levels σ denoted by different colors. We show the average of three different random seeds and their standard error of the mean (not visible as the variance is so low). Interestingly, the higher noise levels take longer to sparsify but upon convergence become the sparsest.

Input: Figure 3: Noise induces the formation of Top-K networks. Shown column-wise are the fraction of neurons that are active for each training input (Left), the values of each of the 10,000 neurons bias terms (Middle) and the L2 norms of their key vectors (Right), as a function of number of training epochs. Each is shown for three different representative noise levels (rows). These plots show each input to the network and use the density of blue to represent how many times a particular value occurs.

Input: Figure 6: Noisy pretraining retains network sparsity. The vertical purple lines denote the start of noise annealing where each noise level is linearly decayed down to σ = 0. For both reconstruction (left) and classification (right) tasks the networks remain highly sparse even after all noise is removed. Moreover, there is no noticeable performance difference between any of the networks while some of those trained with moderate noise perform slightly better than the noise free baseline. We truncate the x-axis to start at epoch 600 for the sake of clarity. The classification networks continue to sparsify even when noise annealing has started because unlike in the reconstruction task, the sparsity level did not converge within the first 800 epochs.

Input: Figure 14: The GELU network sparsifies with noise when trained on CIFAR10. The GELU network closely resembles the ReLU network by sparsifying (but not to the same degree). It also forms biological receptive fields. We use the arbitrary 0.0001 absolute activation threshold to label if a neuron is on or off.

Input: Figure 12: Dead Neurons during the reconstruction task. During noise annealing, the higher noise levels σ ≥ 0.8 see a number of dead neurons. However, this does not full explain their sparsity levels and on this trivial task it does not hurt their reconstruction performance.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim that "the network retains its Top-K approximation even after noise is removed" is well-supported by multiple pieces of evidence. Figure 6 directly demonstrates that networks maintain their sparsity levels after noise annealing across both reconstruction and classification tasks. The quantitative results show no performance degradation and even slight improvements in some cases (94.3% vs 93.4% validation accuracy). The robustness is demonstrated across different architectures, tasks, and conditions as shown in Figure 7. The claim is measured and accurately reflects the experimental findings without exaggeration.

<score>1</score>

Justification:
1. Direct evidence from Figure 6 shows sparsity retention after noise removal
2. Quantitative performance metrics validate maintained or improved accuracy
3. Comprehensive ablation studies (Fig 7) demonstrate robustness
4. The claim is carefully worded, using "retains" rather than stronger terms
5. No overstatement of significance or broader implications
6. Results are presented with appropriate context and limitations discussed
7. The reviewer's comments align with and support this characterization


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

In this work, we decouple privacy analysis and experimental behavior of noisy training to explore the trade-off with minimal computational requirements.

The evidence to be evaluated is:We consider a randomized mechanism \(\mathcal{M}\) that takes as input a dataset \(D\) and outputs a machine learning model \(\theta\sim\mathcal{M}(D)\).

DP-SGD (Abadi et al., 2016) is the most popular DP algorithm to train DNNs. It selects samples uniformly at random with probability \(q=B/N\) (Poisson sampling), clips per-sample gradients to a norm \(C\) (\(\text{clip}_{C}\)), aggregates them and adds (Gaussian) noise. With \(\theta\) the parameters of the DNN: \[g_{\text{noisy}}=\frac{1}{B}\sum_{i\in B}\text{clip}_{C}\left(\nabla_{\theta} \ell(\theta,(x_{i},y_{i}))\right)+\mathcal{N}\left(0,\frac{C^{2}\sigma^{2}}{B ^{2}}\right). \tag{2}\] The traditional privacy analysis of DP-SGD is obtained through Renyi DP.

Renyi DP is a convenient notion to track privacy because composition is additive: a sequence of two algorithms satisfying \((\alpha,d_{\alpha})\) and \((\alpha,d_{\alpha}^{\prime})\) RDP satisfies \((\alpha,d_{\alpha}+d_{\alpha}^{\prime})\) RDP. In particular, the succession of \(S\) steps of a \((\alpha,d_{\alpha})\) RDP mechanism satisfies \((\alpha,Sd_{\alpha})\) RDP. Mironov et al. (2019) show that each step of DP-SGD satisfies \((\alpha,g_{\alpha}(\sigma,q))\)-RDP with \[g_{\alpha}(\sigma,q):=D_{\alpha}((1-q)\mathcal{N}(0,\sigma^{2})+q\mathcal{N}( 1,\sigma^{2})\parallel\mathcal{N}(0,\sigma^{2})).\] Finally, a mechanism satisfying \((\alpha,d_{\alpha})\)-Renyi-DP also satisfies \((\tilde{\varepsilon},\delta)\)-DP (Mironov, 2017) for \(\tilde{\varepsilon}=d_{\alpha}+\frac{\log(1/\delta)}{\alpha-1}\). Training for \(S\) steps with DP-SGD thus satisfies \((\varepsilon_{RDP},\delta)\)-DP with \[\varepsilon_{\mathrm{RDP}}:=\min_{\alpha}Sg_{\alpha}(\sigma,q)+\frac{\log(1/ \delta)}{\alpha-1}. \tag{3}\] RDP is the traditional tool used to analyse DP-SGD, but other accounting tools have been proposed to obtain tighter bounds (Gopi et al., 2021). In this work, we use the accountant due to Balle et al. (2020), whose output is referred to as \(\varepsilon\), which is slightly smaller than \(\varepsilon_{\mathrm{RDP}}\).

Training ML models with DP-SGD typically incurs a loss of model utility, and recent work have shown that using very large batch sizes improves the privacy/utility trade-off (Anil et al., 2021; Li et al., 2021).

Computing per-sample gradients with mega batch sizes for a large number of steps and AugMul makes DP-SGD much more computationally intensive than non-private training, typically dozens of times. For instance, reproducing the previous SOTA on ImageNet of De et al. (2022) under \(\varepsilon=8\) necessitates a \(4\)-day run using \(32\) A100 GPUs, while the non-private SOTA can be reproduced in a few hours with the same hardware (Goyal et al., 2017).

We introduce the notion of Total Amount of Noise (TAN) and discuss its connections to RDP accounting. We then demonstrate how training with reference privacy parameters \((q_{\mathrm{ref}},\sigma_{\mathrm{ref}},S)\) can be simulated with much lower computational resources using the same TAN with a smaller batch size. **Definition 4**.: _Let the individual signal-to-noise ratio \(\eta\) (and its inverse \(\Sigma\), the Total Amount of Noise or TAN) be as follows:_ \[\eta^{2}=\frac{1}{\Sigma^{2}}:=\frac{q^{2}S}{2\sigma^{2}}.\] We begin with a simple case to motivate our definition of TAN. We assume a one-dimensional model, where the gradients of all points are clipped to \(C\). Looking at Equation 2, in one batch, the expected signal from each sample is \(C/B\) with probability \(q=B/N\) and \(0\) otherwise. Therefore, the expected individual signal of each sample after \(S\) steps is \(SC/N\), and its squared norm is \(S^{2}C^{2}/N^{2}\).

The ratio \(\sigma/q\) is noted by Li et al. (2021) as the effective noise. The authors found that for a fixed budget \(\varepsilon\) and fixed \(S\), the effective noise decreases with \(B\). Our analysis goes further by analyzing how RDP accounting explains this dependency. Intuitively, we expect that the privacy budget \(\varepsilon\) only depends on the signal-to-noise ratio \(\eta\). In Figure 2, we plot \(\varepsilon\) as a function of \(\sigma\) and \(S\), at a _fixed_\(\eta\), and observe that \(\varepsilon\) is indeed constant, but only when \(\sigma>2\). On the contrary, when \(\sigma\) gets smaller, \(\varepsilon\) increases exponentially, creating a "Privacy Wall". We can shed light on this phenomenon by looking at the underlying RDP values.

Replacing that in the definition of \(\varepsilon_{\mathrm{RDP}}\) (Equation 3), we get \[\varepsilon_{\mathrm{RDP}}\approx\eta^{2}+\min_{\alpha}\left((\alpha-1)\eta^ {2}+\frac{\log{(1/\delta)}}{\alpha-1}\right)=\eta^{2}+2\eta\sqrt{\log{(1/ \delta)}}=:\varepsilon_{\mathrm{TAN}}(\eta). \tag{4}\] We verify this relationship empirically, and in particular choose \(\eta\) to get a desired \(\varepsilon_{\mathrm{TAN}}\) in Figure 2.

Having this simple approximation for \(\varepsilon\) is useful in practice because it allows for simple mental gymnastics:

Our RDP analysis in Section 3.2 suggests a simple scaling strategy. Starting from \((q,\sigma,S)\), while \(\sigma<2\), we can double both \(q\) and \(\sigma\). This drastically improves privacy accounting (Figure 2), and the per step signal-to-noise ratio \(\eta_{\mathrm{step}}\) remains constant. However, given that the number of steps \(S\) is fixed, each doubling of \(q\) doubles the computational cost.

We find that for triplets \((q,\sigma,S)\) for which \(q/\sigma=q_{\mathrm{ref}}/\sigma_{\mathrm{ref}}\) (keeping \(\eta_{\mathrm{step}}\) constant), the performance of the network (log) linearly decreases with the batch size. This is consistent with the (non-private) work of Smith et al. (2020), which shows that for a fixed number of steps, small batch sizes perform better in generalisation (i.e., test accuracy) than large batch sizes. The difference with our work is that we consider noisy updates and observe a (log) linear relationship between batch size and performance.

It also divides the computational cost by \(2\). This explains why state-of-the-art approaches heuristically find that mega-batches work well: a blind grid search on the batch size and the noise level at constant privacy budget is likely to discover batches large enough to have \(\sigma>2\). Our analysis gives a principled explanation for the sweet spot of \(\sigma\in[2,4]\) used by most state-of-the-art approaches (De et al., 2022; Li et al., 2021).

We go one step further and study training in the small batch size setting. We choose the optimal hyper-parameters (including architecture, optimizer, type of data augmentation) in this simulated setting, and finally launch one single run at the reference (large) batch size, with desired privacy guarantees.

We thus reduce the computational requirements by a factor of \(128\times\). Each hyper-parameter search in the ImageNet setup takes \(4\) days using \(32\) A-100 GPUs; we reduce it to less than a day on a single GPU.

We then study the impact of the dataset size on the privacy/utility trade-off. We also demonstrate how our low compute simulation framework can be used to detect performance bottlenecks when training with noisy updates: in our case, the importance of the order between activation and normalization in a WideResNet on CIFAR-10.

In Section 4.2.1, we use our simulated training with constant TAN to find optimal hyperparameters at low compute and improve performance for reference privacy parameters \((B_{\mathrm{ref}},\sigma_{\mathrm{ref}},S)=(16384,2.5,72\mathrm{k})\). In Section 4.2.2, we find optimal privacy parameters at constant TAN by changing the number of steps \(S\), leading to a new state of the art on ImageNet for \(\varepsilon=8\) (Table 1).

We run a large hyper-parameter search and report the best hyper-parameters in Table 2 as well as the corresponding improvement for various batch sizes (at constant \(\eta_{\text{step}}\) and \(S\)).

We search over learning rates \(lr\in[1,2,4,8,12,16]\), momentum parameters \(\mu\in[0,0.1,0.5,0.9,1]\) and dampening factors \(d\in[0,0.1,0.5,0.9,1]\).

We have experimented with different architectures (ViTs, ResNets, NFResnets) and optimizers (DP-Adam, DP-AdamW, DP-SGD) (see Appendix B for details).

While we kept \(S\) constant in previous experiments, we now explore constant TAN triplets \((q,\sigma,S)\) by varying \(S\). We keep \(\sigma\) fixed to \(2.5\) and vary \((B,S)\) starting from the reference \((16384,72\text{K})\) at constant \(\eta=q^{2}S/(2\sigma^{2})\). Given that \(\sigma>2\), we stay in the _almost_ constant privacy regime (see Figure 2): we indeed observe \(\varepsilon\approx\varepsilon_{\mathrm{TAN}}\) in Table 4.

Since performing this privacy parameter search is computationally intensive, we first simulate training using our scaling law at \(B=256\) (with the same \(\eta_{\mathrm{step}}\)) and display our results in Table 3.

Thus, we have shown how we can use TAN to perform optimal privacy parameter search while simulating each choice of optimal parameters at a much smaller cost.

We observe that the two simulated experiments with \(B=512\) represented by lighter colors in Figure 5 (\(2\) standard deviations around the means) have the same properties. However, each simulation is \(8\) times less computationally expensive. Therefore, using TAN through our scaling law can facilitate studying variants of the network architecture while reducing the computational costs.

We now look at how collecting more data affects the tradeoff between privacy and utility. We show that doubling the data (from the same distribution) allows better performance with half the privacy budget. To this end, we train on portions of the CIFAR-10 training set (\(N=50\)k) and always report accuracies on the same test set. If we multiply by \(\beta\) the quantity of data \(N_{0}\) and keep the same \((B,\sigma,S)\), \(q\) (and thus \(\eta\)), is divided by \(\beta\) as well.

We argue that the total amount of noise (TAN) is a simple but useful guiding principle to experiment with private training. In particular, we demonstrate that the privacy budget is either a direct function of TAN or can be reduced. We further show that scaling batch size with noise level using TAN allows for ultra-efficient hyper-parameter search and demonstrate the power of this paradigm by establishing a new state of the art for DP training on Imagenet.

If available, one can use a similar public dataset (such as ImageNet) to choose hyper-parameters, and then perform only limited runs on the private dataset.

We believe that the important increase in the privacy budget \(\varepsilon\) as the noise level \(\sigma\) decreases is a real phenomenon and not an artifact of the analysis. Indeed, DP assumes that the adversary has access to all model updates, as is the case for example in Federated Learning. In such cases, a noise level that is too low is insufficient to hide the presence of individual data points and makes it impossible to obtain reasonable privacy guarantees. In the centralized case however, the adversary does not see intermediate models but only the final result of training. Some works have successfully taken into account this "privacy amplification by iteration" idea (Feldman et al., 2018; Ye and Shokri, 2022) but results are so far limited to convex models.

We show in Table 6 that similarly to the experiments in CIFAR-10, doubling the training data on ImageNet improves the accuracy while diving \(\varepsilon\) by \(2\). We also demonstrate that our scaling strategy can accurately detect the gain of accuracy.

When training with DP-SGD, the goal is to find the best possible local minimum within a constrained number of steps \(S\), and with noisy gradients.

We used our scaling strategy (keeping \(\eta_{step}\) and \(S\) constant) to simulate the DP training with different architectures at low compute, studying noisy training without the burden of DP accounting. The best simulated results were obtained with a NFResNet-50 (Brock et al., 2021b) designed to be fast learners in terms of number of FLOPS. The worst results were obtained with ViTs, and intermediate results with classical ResNets. In Figure 6, we compare different training trajectories of a ViT and a NF-ResNet.

This highlights the fact that training with DP-SGD is a different paradigm that requires its own tools. Using TAN allowed us to explore and compare different architectures and optimizers, which would have been computationally impossible in the normal DP training setting at \(B=16384\).

Because we are training for a fixed number of steps, it is a waste of gradient steps (i.e. privacy budget). With AugMult, the network may still be able to learn from these examples. Figure 7 shows the histograms of the norms of the **average over all augmentations for each image** of the per-sample gradients, before clipping and adding noise in equation 5 at different times of training.

Input: Table 1: Top-1 test accuracy when training on ImageNet from scratch using DP-SGD. We train a NF-ResNet-50 with σ = 2.5, hyper-parameters of Table 2 and (B,S) = (32768, 18k) (Table 4).

Input: Table 2: Comparing optimal hyper-parameters. Keeping ηstep and S constant, we compare various changes in the training pipeline. We compare with the baseline of De et al. (2022) (blue line in Figure 1: NFResNet-50, learning rate at 4, EMA decay at 0.99999, 4 random augmentations averaged over 3 runs). Each gain is compared to the previous column.

Input: Figure 7: Histograms of the norms of the average across all augmentations for each image of the per-sample gradients, before clipping and adding noise. On the left, we see that without augmentation, an increasing number of examples have their gradients going to zero during training. On the right, we see that when using a strong augmentation technique (SimCLR, (Chen et al., 2020)), the gradients are more concentrated during all the training.

Input: Table 4: Privacy parameter search. We use the optimal parameters described in Section 4.2.1 with σ = 2.5 for one expensive run and compare it with our optimal result

Input: Table 3: Low compute simulation of privacy parameter search. We start from B = 256 = 16384/64 and S = 72K. We use σ = 2.5/64 for all runs and no data augmentation.

Input: Figure 6: Training a ViT from scratch on ImageNet with DP-SGD. We simulate training with our scaling strategy and B = 256. We observe that the accuracies are not as good as for NF-ResNets, and that Augmentation Multiplicity plays a more important role.

Input: Table 6: Impact of adding more data on ImageNet. The “Simulated Gain” column corresponds to the accuracy gain we observe when simulating at lower compute using our scaling strategy for B = 256. The “Gain” column corresponds to the real gain at B = 16384.

Input: Figure 5: Impact of data augmentation on the test accuracy using pre-activation normalization vs post-activation in a 16-4-WideResnet on CIFAR-10. We compare simulation at (B, σ) = (512, 3 8 ) and reference (Bref , σref) = (4096, 3), both trained for S = 2,500 steps. Confidence intervals are plotted with two standard deviations over 5 runs. Augmentation Multiplicity Order corresponds to the number of augmentations per image, or K in Appendix C.

Input: Figure 2: Privacy budget ε as a function of the noise level σ with η constant. On both figures, each curve corresponds to a different number of steps S, and each point on the curve is computed at a sampling rate q such that η is constant. On the left, we use η = 0.13 (resulting in εTAN = 1 in Equation 4). On the right, we use η = 0.95 (εTAN = 8). We observe a “privacy wall” imposing σ ≥ 0.5 for meaningful level of privacy budget ε, and σ ≥ 2 for constant ε ≈ εTAN.

Input: Figure 3: Approximation of gα(σ, q). All curves correspond to distinct couples (q, σ) such that ηstep = 3.9 × 10−3 (used for ImageNet). The right plot corresponds to an enlargement of the left plot: the ratio is very close to 1 for σ ≥ 2. The phase transitions on the left plot were also observed by Wang et al. (2018) and Abadi et al. (2016).

Input: Figure 4: Test accuracies at constant ηstep = Bref/( √ 2Nσref ) and S are (log) linearly decreasing with B. Dashed lines are computed using a linear regression on the crosses. Shaded areas correspond to 3 std confidence intervals. (Left) CIFAR-10 with 16-4-WideResNet for Sref steps. Each curve corresponds to a different value of ηstep. (Right) ImageNet with NF-ResNet-50 with various numbers of steps. The scaling law holds for various training configurations.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate the claim: "In this work, we decouple privacy analysis and experimental behavior of noisy training to explore the trade-off with minimal computational requirements."

The evidence strongly supports this claim through:
- Introduction of Total Amount of Noise (TAN) concept that allows decoupling privacy and training behavior
- Demonstration that training with reference privacy parameters can be simulated with lower computational resources while maintaining the same TAN
- Empirical validation showing 128x reduction in computational requirements (from 4 days on 32 A100 GPUs to less than a day on a single GPU)
- Clear experimental results in tables and figures showing the effectiveness of this decoupling approach

The claim appears well-stated and accurately reflects what was demonstrated in the paper without exaggeration. The evidence provides thorough theoretical and empirical support.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

In this work, we decouple privacy analysis and experimental behavior of noisy training to explore the trade-off with minimal computational requirements.

The review comment to be evaluated is:

Strength:

* The observations (batch scaling law, privacy budget computation) in the paper are new and interesting.

* The proposed hyper-parameter tuning process can be very useful in practice, as it saves the computation requirement and therefore speeds up the development process.

Weaknesses:

* Overall, the writing is clear, but there are still some issues that need to be addressed:
    - Equation 2: many of the notations here are not defined, e.g., clip_C, l, theta, C.
    - Figure 4: B_ref, S_ref, \sigma_ref are not defined. I can guess their meaning from the context, but all notations should be defined explicitly in the text.
    - Page 5: it says "Simultaneously doubling the batch size and \sigma has a negligible or small impact on accuracy (Figure 4)". However, in Figure 4(b), the impact on the accuracy is not small. Indeed, this is claimed in the next sentence "Reciprocally, if \sigma > 4, dividing it by 2 simultaneously with halving q is likely to improve performance." These two sentences contradict each other.
    - Table 1: I guess "8,8." should be "8,"?

The evidence to be evaluated is:We consider a randomized mechanism \(\mathcal{M}\) that takes as input a dataset \(D\) and outputs a machine learning model \(\theta\sim\mathcal{M}(D)\).

DP-SGD (Abadi et al., 2016) is the most popular DP algorithm to train DNNs. It selects samples uniformly at random with probability \(q=B/N\) (Poisson sampling), clips per-sample gradients to a norm \(C\) (\(\text{clip}_{C}\)), aggregates them and adds (Gaussian) noise. With \(\theta\) the parameters of the DNN: \[g_{\text{noisy}}=\frac{1}{B}\sum_{i\in B}\text{clip}_{C}\left(\nabla_{\theta} \ell(\theta,(x_{i},y_{i}))\right)+\mathcal{N}\left(0,\frac{C^{2}\sigma^{2}}{B ^{2}}\right). \tag{2}\] The traditional privacy analysis of DP-SGD is obtained through Renyi DP.

Renyi DP is a convenient notion to track privacy because composition is additive: a sequence of two algorithms satisfying \((\alpha,d_{\alpha})\) and \((\alpha,d_{\alpha}^{\prime})\) RDP satisfies \((\alpha,d_{\alpha}+d_{\alpha}^{\prime})\) RDP. In particular, the succession of \(S\) steps of a \((\alpha,d_{\alpha})\) RDP mechanism satisfies \((\alpha,Sd_{\alpha})\) RDP. Mironov et al. (2019) show that each step of DP-SGD satisfies \((\alpha,g_{\alpha}(\sigma,q))\)-RDP with \[g_{\alpha}(\sigma,q):=D_{\alpha}((1-q)\mathcal{N}(0,\sigma^{2})+q\mathcal{N}( 1,\sigma^{2})\parallel\mathcal{N}(0,\sigma^{2})).\] Finally, a mechanism satisfying \((\alpha,d_{\alpha})\)-Renyi-DP also satisfies \((\tilde{\varepsilon},\delta)\)-DP (Mironov, 2017) for \(\tilde{\varepsilon}=d_{\alpha}+\frac{\log(1/\delta)}{\alpha-1}\). Training for \(S\) steps with DP-SGD thus satisfies \((\varepsilon_{RDP},\delta)\)-DP with \[\varepsilon_{\mathrm{RDP}}:=\min_{\alpha}Sg_{\alpha}(\sigma,q)+\frac{\log(1/ \delta)}{\alpha-1}. \tag{3}\] RDP is the traditional tool used to analyse DP-SGD, but other accounting tools have been proposed to obtain tighter bounds (Gopi et al., 2021). In this work, we use the accountant due to Balle et al. (2020), whose output is referred to as \(\varepsilon\), which is slightly smaller than \(\varepsilon_{\mathrm{RDP}}\).

Training ML models with DP-SGD typically incurs a loss of model utility, and recent work have shown that using very large batch sizes improves the privacy/utility trade-off (Anil et al., 2021; Li et al., 2021).

Computing per-sample gradients with mega batch sizes for a large number of steps and AugMul makes DP-SGD much more computationally intensive than non-private training, typically dozens of times. For instance, reproducing the previous SOTA on ImageNet of De et al. (2022) under \(\varepsilon=8\) necessitates a \(4\)-day run using \(32\) A100 GPUs, while the non-private SOTA can be reproduced in a few hours with the same hardware (Goyal et al., 2017).

We introduce the notion of Total Amount of Noise (TAN) and discuss its connections to RDP accounting. We then demonstrate how training with reference privacy parameters \((q_{\mathrm{ref}},\sigma_{\mathrm{ref}},S)\) can be simulated with much lower computational resources using the same TAN with a smaller batch size. **Definition 4**.: _Let the individual signal-to-noise ratio \(\eta\) (and its inverse \(\Sigma\), the Total Amount of Noise or TAN) be as follows:_ \[\eta^{2}=\frac{1}{\Sigma^{2}}:=\frac{q^{2}S}{2\sigma^{2}}.\] We begin with a simple case to motivate our definition of TAN. We assume a one-dimensional model, where the gradients of all points are clipped to \(C\). Looking at Equation 2, in one batch, the expected signal from each sample is \(C/B\) with probability \(q=B/N\) and \(0\) otherwise. Therefore, the expected individual signal of each sample after \(S\) steps is \(SC/N\), and its squared norm is \(S^{2}C^{2}/N^{2}\).

The ratio \(\sigma/q\) is noted by Li et al. (2021) as the effective noise. The authors found that for a fixed budget \(\varepsilon\) and fixed \(S\), the effective noise decreases with \(B\). Our analysis goes further by analyzing how RDP accounting explains this dependency. Intuitively, we expect that the privacy budget \(\varepsilon\) only depends on the signal-to-noise ratio \(\eta\). In Figure 2, we plot \(\varepsilon\) as a function of \(\sigma\) and \(S\), at a _fixed_\(\eta\), and observe that \(\varepsilon\) is indeed constant, but only when \(\sigma>2\). On the contrary, when \(\sigma\) gets smaller, \(\varepsilon\) increases exponentially, creating a "Privacy Wall". We can shed light on this phenomenon by looking at the underlying RDP values.

Replacing that in the definition of \(\varepsilon_{\mathrm{RDP}}\) (Equation 3), we get \[\varepsilon_{\mathrm{RDP}}\approx\eta^{2}+\min_{\alpha}\left((\alpha-1)\eta^ {2}+\frac{\log{(1/\delta)}}{\alpha-1}\right)=\eta^{2}+2\eta\sqrt{\log{(1/ \delta)}}=:\varepsilon_{\mathrm{TAN}}(\eta). \tag{4}\] We verify this relationship empirically, and in particular choose \(\eta\) to get a desired \(\varepsilon_{\mathrm{TAN}}\) in Figure 2.

Having this simple approximation for \(\varepsilon\) is useful in practice because it allows for simple mental gymnastics:

Our RDP analysis in Section 3.2 suggests a simple scaling strategy. Starting from \((q,\sigma,S)\), while \(\sigma<2\), we can double both \(q\) and \(\sigma\). This drastically improves privacy accounting (Figure 2), and the per step signal-to-noise ratio \(\eta_{\mathrm{step}}\) remains constant. However, given that the number of steps \(S\) is fixed, each doubling of \(q\) doubles the computational cost.

We find that for triplets \((q,\sigma,S)\) for which \(q/\sigma=q_{\mathrm{ref}}/\sigma_{\mathrm{ref}}\) (keeping \(\eta_{\mathrm{step}}\) constant), the performance of the network (log) linearly decreases with the batch size. This is consistent with the (non-private) work of Smith et al. (2020), which shows that for a fixed number of steps, small batch sizes perform better in generalisation (i.e., test accuracy) than large batch sizes. The difference with our work is that we consider noisy updates and observe a (log) linear relationship between batch size and performance.

It also divides the computational cost by \(2\). This explains why state-of-the-art approaches heuristically find that mega-batches work well: a blind grid search on the batch size and the noise level at constant privacy budget is likely to discover batches large enough to have \(\sigma>2\). Our analysis gives a principled explanation for the sweet spot of \(\sigma\in[2,4]\) used by most state-of-the-art approaches (De et al., 2022; Li et al., 2021).

We go one step further and study training in the small batch size setting. We choose the optimal hyper-parameters (including architecture, optimizer, type of data augmentation) in this simulated setting, and finally launch one single run at the reference (large) batch size, with desired privacy guarantees.

We thus reduce the computational requirements by a factor of \(128\times\). Each hyper-parameter search in the ImageNet setup takes \(4\) days using \(32\) A-100 GPUs; we reduce it to less than a day on a single GPU.

We then study the impact of the dataset size on the privacy/utility trade-off. We also demonstrate how our low compute simulation framework can be used to detect performance bottlenecks when training with noisy updates: in our case, the importance of the order between activation and normalization in a WideResNet on CIFAR-10.

In Section 4.2.1, we use our simulated training with constant TAN to find optimal hyperparameters at low compute and improve performance for reference privacy parameters \((B_{\mathrm{ref}},\sigma_{\mathrm{ref}},S)=(16384,2.5,72\mathrm{k})\). In Section 4.2.2, we find optimal privacy parameters at constant TAN by changing the number of steps \(S\), leading to a new state of the art on ImageNet for \(\varepsilon=8\) (Table 1).

We run a large hyper-parameter search and report the best hyper-parameters in Table 2 as well as the corresponding improvement for various batch sizes (at constant \(\eta_{\text{step}}\) and \(S\)).

We search over learning rates \(lr\in[1,2,4,8,12,16]\), momentum parameters \(\mu\in[0,0.1,0.5,0.9,1]\) and dampening factors \(d\in[0,0.1,0.5,0.9,1]\).

We have experimented with different architectures (ViTs, ResNets, NFResnets) and optimizers (DP-Adam, DP-AdamW, DP-SGD) (see Appendix B for details).

While we kept \(S\) constant in previous experiments, we now explore constant TAN triplets \((q,\sigma,S)\) by varying \(S\). We keep \(\sigma\) fixed to \(2.5\) and vary \((B,S)\) starting from the reference \((16384,72\text{K})\) at constant \(\eta=q^{2}S/(2\sigma^{2})\). Given that \(\sigma>2\), we stay in the _almost_ constant privacy regime (see Figure 2): we indeed observe \(\varepsilon\approx\varepsilon_{\mathrm{TAN}}\) in Table 4.

Since performing this privacy parameter search is computationally intensive, we first simulate training using our scaling law at \(B=256\) (with the same \(\eta_{\mathrm{step}}\)) and display our results in Table 3.

Thus, we have shown how we can use TAN to perform optimal privacy parameter search while simulating each choice of optimal parameters at a much smaller cost.

We observe that the two simulated experiments with \(B=512\) represented by lighter colors in Figure 5 (\(2\) standard deviations around the means) have the same properties. However, each simulation is \(8\) times less computationally expensive. Therefore, using TAN through our scaling law can facilitate studying variants of the network architecture while reducing the computational costs.

We now look at how collecting more data affects the tradeoff between privacy and utility. We show that doubling the data (from the same distribution) allows better performance with half the privacy budget. To this end, we train on portions of the CIFAR-10 training set (\(N=50\)k) and always report accuracies on the same test set. If we multiply by \(\beta\) the quantity of data \(N_{0}\) and keep the same \((B,\sigma,S)\), \(q\) (and thus \(\eta\)), is divided by \(\beta\) as well.

We argue that the total amount of noise (TAN) is a simple but useful guiding principle to experiment with private training. In particular, we demonstrate that the privacy budget is either a direct function of TAN or can be reduced. We further show that scaling batch size with noise level using TAN allows for ultra-efficient hyper-parameter search and demonstrate the power of this paradigm by establishing a new state of the art for DP training on Imagenet.

If available, one can use a similar public dataset (such as ImageNet) to choose hyper-parameters, and then perform only limited runs on the private dataset.

We believe that the important increase in the privacy budget \(\varepsilon\) as the noise level \(\sigma\) decreases is a real phenomenon and not an artifact of the analysis. Indeed, DP assumes that the adversary has access to all model updates, as is the case for example in Federated Learning. In such cases, a noise level that is too low is insufficient to hide the presence of individual data points and makes it impossible to obtain reasonable privacy guarantees. In the centralized case however, the adversary does not see intermediate models but only the final result of training. Some works have successfully taken into account this "privacy amplification by iteration" idea (Feldman et al., 2018; Ye and Shokri, 2022) but results are so far limited to convex models.

We show in Table 6 that similarly to the experiments in CIFAR-10, doubling the training data on ImageNet improves the accuracy while diving \(\varepsilon\) by \(2\). We also demonstrate that our scaling strategy can accurately detect the gain of accuracy.

When training with DP-SGD, the goal is to find the best possible local minimum within a constrained number of steps \(S\), and with noisy gradients.

We used our scaling strategy (keeping \(\eta_{step}\) and \(S\) constant) to simulate the DP training with different architectures at low compute, studying noisy training without the burden of DP accounting. The best simulated results were obtained with a NFResNet-50 (Brock et al., 2021b) designed to be fast learners in terms of number of FLOPS. The worst results were obtained with ViTs, and intermediate results with classical ResNets. In Figure 6, we compare different training trajectories of a ViT and a NF-ResNet.

This highlights the fact that training with DP-SGD is a different paradigm that requires its own tools. Using TAN allowed us to explore and compare different architectures and optimizers, which would have been computationally impossible in the normal DP training setting at \(B=16384\).

Because we are training for a fixed number of steps, it is a waste of gradient steps (i.e. privacy budget). With AugMult, the network may still be able to learn from these examples. Figure 7 shows the histograms of the norms of the **average over all augmentations for each image** of the per-sample gradients, before clipping and adding noise in equation 5 at different times of training.

Input: Table 1: Top-1 test accuracy when training on ImageNet from scratch using DP-SGD. We train a NF-ResNet-50 with σ = 2.5, hyper-parameters of Table 2 and (B,S) = (32768, 18k) (Table 4).

Input: Table 2: Comparing optimal hyper-parameters. Keeping ηstep and S constant, we compare various changes in the training pipeline. We compare with the baseline of De et al. (2022) (blue line in Figure 1: NFResNet-50, learning rate at 4, EMA decay at 0.99999, 4 random augmentations averaged over 3 runs). Each gain is compared to the previous column.

Input: Figure 7: Histograms of the norms of the average across all augmentations for each image of the per-sample gradients, before clipping and adding noise. On the left, we see that without augmentation, an increasing number of examples have their gradients going to zero during training. On the right, we see that when using a strong augmentation technique (SimCLR, (Chen et al., 2020)), the gradients are more concentrated during all the training.

Input: Table 4: Privacy parameter search. We use the optimal parameters described in Section 4.2.1 with σ = 2.5 for one expensive run and compare it with our optimal result

Input: Table 3: Low compute simulation of privacy parameter search. We start from B = 256 = 16384/64 and S = 72K. We use σ = 2.5/64 for all runs and no data augmentation.

Input: Figure 6: Training a ViT from scratch on ImageNet with DP-SGD. We simulate training with our scaling strategy and B = 256. We observe that the accuracies are not as good as for NF-ResNets, and that Augmentation Multiplicity plays a more important role.

Input: Table 6: Impact of adding more data on ImageNet. The “Simulated Gain” column corresponds to the accuracy gain we observe when simulating at lower compute using our scaling strategy for B = 256. The “Gain” column corresponds to the real gain at B = 16384.

Input: Figure 5: Impact of data augmentation on the test accuracy using pre-activation normalization vs post-activation in a 16-4-WideResnet on CIFAR-10. We compare simulation at (B, σ) = (512, 3 8 ) and reference (Bref , σref) = (4096, 3), both trained for S = 2,500 steps. Confidence intervals are plotted with two standard deviations over 5 runs. Augmentation Multiplicity Order corresponds to the number of augmentations per image, or K in Appendix C.

Input: Figure 2: Privacy budget ε as a function of the noise level σ with η constant. On both figures, each curve corresponds to a different number of steps S, and each point on the curve is computed at a sampling rate q such that η is constant. On the left, we use η = 0.13 (resulting in εTAN = 1 in Equation 4). On the right, we use η = 0.95 (εTAN = 8). We observe a “privacy wall” imposing σ ≥ 0.5 for meaningful level of privacy budget ε, and σ ≥ 2 for constant ε ≈ εTAN.

Input: Figure 3: Approximation of gα(σ, q). All curves correspond to distinct couples (q, σ) such that ηstep = 3.9 × 10−3 (used for ImageNet). The right plot corresponds to an enlargement of the left plot: the ratio is very close to 1 for σ ≥ 2. The phase transitions on the left plot were also observed by Wang et al. (2018) and Abadi et al. (2016).

Input: Figure 4: Test accuracies at constant ηstep = Bref/( √ 2Nσref ) and S are (log) linearly decreasing with B. Dashed lines are computed using a linear regression on the crosses. Shaded areas correspond to 3 std confidence intervals. (Left) CIFAR-10 with 16-4-WideResNet for Sref steps. Each curve corresponds to a different value of ηstep. (Right) ImageNet with NF-ResNet-50 with various numbers of steps. The scaling law holds for various training configurations.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I've reviewed the claim "In this work, we decouple privacy analysis and experimental behavior of noisy training to explore the trade-off with minimal computational requirements."

The evidence strongly supports this claim through:
- Demonstration of Total Amount of Noise (TAN) concept that allows decoupling privacy analysis from training behavior
- Empirical validation showing successful simulation of large-batch training using smaller batches while maintaining privacy guarantees
- Concrete examples of computational savings (e.g., reducing 4-day/32 GPU training to 1-day/single GPU)
- Experimental results validating the approach across multiple datasets and architectures

The claim appears well-stated and accurately reflects the demonstrated capabilities without exaggeration. The evidence thoroughly substantiates both the decoupling concept and the computational benefits.

<score>1</score>