Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We first show that within a wide range of the privacy parameters, the privacy budget \(\varepsilon\) is a function only of the total amount of noise.

The review comment to be evaluated is:

Strengths:
1) Integrating the constraints of DP-SGD with optimization theory, in order to reliably predict good hyper-parameter choices, is one of the most important problems that needs to be solved in order to make differentially private deep learning practical. I'm pleased to see the authors tackle this difficult and important problem.
2) The metric proposed by the authors is intuitive and it appears to provide some helpful intuition.
3) The authors achieve strong performance when training from scratch on ImageNet with DP-SGD.

Weaknesses:
1) If TAN were an accurate measure of optimizability, then we should expect the train accuracy to be roughly constant at fixed TAN. However in practice, the authors find in figure 1 that the test accuracy decays log-linearly at fixed TAN as the batch size rises. I feel that the current paper is quite confusing because this discrepancy is not explicitly stated anywhere. I'd encourage the authors to make this explicit, and to explore in more detail why this discrepancy arises (eg comparing test vs train performance).

2) As discussed above, Figure 1 shows that TAN over-estimates the performance of large batch sizes. It would be nice to also see a figure at fixed batch size, sweeping the number of steps S, to see whether TAN under/over-estimates the performance of small/large step budgets.

3) As a minor point, note that on some truly sensitive data we cannot train non-private models (even internally). It would be good to acknowledge that the hyper-parameter transfer process described here cannot be used in these cases.

4) For many of the experiments, key hyper-parameters are lifted from prior work (eg step budget, target batch size). Is it not possible to directly infer good settings for these hyper-parameters using the TAN framework?

5) The paper essentially only considers ImageNet (with some very limited experiments on CIFAR-10). Since very few groups have run experiments at this scale it is not clear how challenging the baseline from De et al. is in practice.

The evidence to be evaluated is:DP is a convenient notion to track privacy because composition is additive: a sequence of two algorithms satisfying \((\alpha,d_{\alpha})\) and \((\alpha,d_{\alpha}^{\prime})\) RDP satisfies \((\alpha,d_{\alpha}+d_{\alpha}^{\prime})\) RDP. In particular, the succession of \(S\) steps of a \((\alpha,d_{\alpha})\) RDP mechanism satisfies \((\alpha,Sd_{\alpha})\) RDP. Mironov et al. (2019) show that each step of DP-SGD satisfies \((\alpha,g_{\alpha}(\sigma,q))\)-RDP with

Training for \(S\) steps with DP-SGD thus satisfies \((\varepsilon_{RDP},\delta)\)-DP with \[\varepsilon_{\mathrm{RDP}}:=\min_{\alpha}Sg_{\alpha}(\sigma,q)+\frac{\log(1/ \delta)}{\alpha-1}. \tag{3}\]

We introduce the notion of Total Amount of Noise (TAN) and discuss its connections to RDP accounting.

**Definition 4**.: _Let the individual signal-to-noise ratio \(\eta\) (and its inverse \(\Sigma\), the Total Amount of Noise or TAN) be as follows:_ \[\eta^{2}=\frac{1}{\Sigma^{2}}:=\frac{q^{2}S}{2\sigma^{2}}.\]

Intuitively, we expect that the privacy budget \(\varepsilon\) only depends on the signal-to-noise ratio \(\eta\). In Figure 2, we plot \(\varepsilon\) as a function of \(\sigma\) and \(S\), at a _fixed_\(\eta\), and observe that \(\varepsilon\) is indeed constant, but only when \(\sigma>2\).

We observe (see Figure 3) that when \(\sigma>2\), \(g_{\alpha}(\sigma,q)\) is close to \(\alpha q^{2}/(2\sigma^{2})=\alpha\eta_{\mathrm{step}}^{2}\). Replacing that in the definition of \(\varepsilon_{\mathrm{RDP}}\) (Equation 3), we get \[\varepsilon_{\mathrm{RDP}}\approx\eta^{2}+\min_{\alpha}\left((\alpha-1)\eta^ {2}+\frac{\log{(1/\delta)}}{\alpha-1}\right)=\eta^{2}+2\eta\sqrt{\log{(1/ \delta)}}=:\varepsilon_{\mathrm{TAN}}(\eta). \tag{4}\] We verify this relationship empirically, and in particular choose \(\eta\) to get a desired \(\varepsilon_{\mathrm{TAN}}\) in Figure 2.

Having this simple approximation for \(\varepsilon\) is useful in practice because it allows for simple mental gymnastics: for instance, doubling the sampling rate \(q\) while dividing the number of steps \(S\) by \(4\) should leave the privacy budget constant, which we observe empirically. Our RDP analysis in Section 3.2 suggests a simple scaling strategy. Starting from \((q,\sigma,S)\), while \(\sigma<2\), we can double both \(q\) and \(\sigma\).

This explains why state-of-the-art approaches heuristically find that mega-batches work well: a blind grid search on the batch size and the noise level at constant privacy budget is likely to discover batches large enough to have \(\sigma>2\). Our analysis gives a principled explanation for the sweet spot of \(\sigma\in[2,4]\) used by most state-of-the-art approaches (De et al., 2022; Li et al., 2021).

While we kept \(S\) constant in previous experiments, we now explore constant TAN triplets \((q,\sigma,S)\) by varying \(S\). We keep \(\sigma\) fixed to \(2.5\) and vary \((B,S)\) starting from the reference \((16384,72\text{K})\) at constant \(\eta=q^{2}S/(2\sigma^{2})\). Given that \(\sigma>2\), we stay in the _almost_ constant privacy regime (see Figure 2): we indeed observe \(\varepsilon\approx\varepsilon_{\mathrm{TAN}}\) in Table 4.

If we multiply by \(\beta\) the quantity of data \(N_{0}\) and keep the same \((B,\sigma,S)\), \(q\) (and thus \(\eta\)), is divided by \(\beta\) as well.

We argue that the total amount of noise (TAN) is a simple but useful guiding principle to experiment with private training. In particular, we demonstrate that the privacy budget is either a direct function of TAN or can be reduced.

Input: Table 4: Privacy parameter search. We use the optimal parameters described in Section 4.2.1 with σ = 2.5 for one expensive run and compare it with our optimal result

Input: Figure 2: Privacy budget ε as a function of the noise level σ with η constant. On both figures, each curve corresponds to a different number of steps S, and each point on the curve is computed at a sampling rate q such that η is constant. On the left, we use η = 0.13 (resulting in εTAN = 1 in Equation 4). On the right, we use η = 0.95 (εTAN = 8). We observe a “privacy wall” imposing σ ≥ 0.5 for meaningful level of privacy budget ε, and σ ≥ 2 for constant ε ≈ εTAN.

Input: Figure 3: Approximation of gα(σ, q). All curves correspond to distinct couples (q, σ) such that ηstep = 3.9 × 10−3 (used for ImageNet). The right plot corresponds to an enlargement of the left plot: the ratio is very close to 1 for σ ≥ 2. The phase transitions on the left plot were also observed by Wang et al. (2018) and Abadi et al. (2016).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, I find this claim needs careful qualification.

The evidence shows that the privacy budget ε is indeed a function of the total amount of noise (TAN), but crucially only under specific conditions - primarily when σ > 2. Figure 2 clearly demonstrates that ε becomes approximately constant (matching εTAN) only after this threshold. Before σ > 2, there is significant variation in ε even at constant η. The paper acknowledges this limitation implicitly but the claim as stated doesn't reflect this important constraint.

The review comments don't directly address this specific claim, so they weren't factored into this assessment.

The claim should specify this key condition to be well-stated, as the relationship only holds "when σ > 2" rather than in a "wide range" of parameters.

<score>3</score>

Justification:
1. The claim overstates the range where this relationship holds true by saying "within a wide range" when it's specifically valid only for σ > 2
2. Figure 2 and Figure 3 provide clear evidence of the σ > 2 threshold requirement
3. The mathematical derivation in Equation 4 relies on the approximation that only holds when σ > 2
4. The paper itself uses more precise language in later sections, noting "Given that σ > 2, we stay in the almost constant privacy regime"


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We then analyze the scaling laws of DNNs at constant TAN and show that performance at very large batch sizes (computationally intensive) is (linearly) predictable from performance at small batch sizes as illustrated in Figure 1.

The evidence to be evaluated is:We introduce the notion of Total Amount of Noise (TAN) and discuss its connections to RDP accounting. We then demonstrate how training with reference privacy parameters \((q_{\mathrm{ref}},\sigma_{\mathrm{ref}},S)\) can be simulated with much lower computational resources using the same TAN with a smaller batch size. **Definition 4**.: _Let the individual signal-to-noise ratio \(\eta\) (and its inverse \(\Sigma\), the Total Amount of Noise or TAN) be as follows:_ \[\eta^{2}=\frac{1}{\Sigma^{2}}:=\frac{q^{2}S}{2\sigma^{2}}.\] We begin with a simple case to motivate our definition of TAN. We assume a one-dimensional model, where the gradients of all points are clipped to \(C\). Looking at Equation 2, in one batch, the expected signal from each sample is \(C/B\) with probability \(q=B/N\) and \(0\) otherwise. Therefore, the expected individual signal of each sample after \(S\) steps is \(SC/N\), and its squared norm is \(S^{2}C^{2}/N^{2}\).

In Figure 4, we perform this analysis on CIFAR-10 and ImageNet. We find that for triplets \((q,\sigma,S)\) for which \(q/\sigma=q_{\mathrm{ref}}/\sigma_{\mathrm{ref}}\) (keeping \(\eta_{\mathrm{step}}\) constant), the performance of the network (log) linearly decreases with the batch size. This is consistent with the (non-private) work of Smith et al. (2020), which shows that for a fixed number of steps, small batch sizes perform better in generalisation (i.e., test accuracy) than large batch sizes. The difference with our work is that we consider noisy updates and observe a (log) linear relationship between batch size and performance.

In Section 4.2.1, we use our simulated training with constant TAN to find optimal hyperparameters at low compute and improve performance for reference privacy parameters \((B_{\mathrm{ref}},\sigma_{\mathrm{ref}},S)=(16384,2.5,72\mathrm{k})\). In Section 4.2.2, we find optimal privacy parameters at constant TAN by changing the number of steps \(S\), leading to a new state of the art on ImageNet for \(\varepsilon=8\) (Table 1).

We run a large hyper-parameter search and report the best hyper-parameters in Table 2 as well as the corresponding improvement for various batch sizes (at constant \(\eta_{\text{step}}\) and \(S\)).

While we kept \(S\) constant in previous experiments, we now explore constant TAN triplets \((q,\sigma,S)\) by varying \(S\). We keep \(\sigma\) fixed to \(2.5\) and vary \((B,S)\) starting from the reference \((16384,72\text{K})\) at constant \(\eta=q^{2}S/(2\sigma^{2})\).

Since performing this privacy parameter search is computationally intensive, we first simulate training using our scaling law at \(B=256\) (with the same \(\eta_{\mathrm{step}}\)) and display our results in Table 3.

Finally, we perform one computationally expensive run at \(S=18\)k and \(B=32768\), with other hyper-parameters from Section 4.2.1, and show the results in Table 4.

Thus, we have shown how we can use TAN to perform optimal privacy parameter search while simulating each choice of optimal parameters at a much smaller cost.

We observe that the two simulated experiments with \(B=512\) represented by lighter colors in Figure 5 (\(2\) standard deviations around the means) have the same properties.

Therefore, using TAN through our scaling law can facilitate studying variants of the network architecture while reducing the computational costs.

On the other hand, since the global signal-to-noise ratio \(N\eta\) is held constant in all experiments, we expect to extract the same amount of information in each setup; adding more data makes this information richer, which explains the gain in accuracy.

We argue that the total amount of noise (TAN) is a simple but useful guiding principle to experiment with private training.

We further show that scaling batch size with noise level using TAN allows for ultra-efficient hyper-parameter search and demonstrate the power of this paradigm by establishing a new state of the art for DP training on Imagenet.

We used our scaling strategy (keeping \(\eta_{step}\) and \(S\) constant) to simulate the DP training with different architectures at low compute, studying noisy training without the burden of DP accounting.

Using TAN allowed us to explore and compare different architectures and optimizers, which would have been computationally impossible in the normal DP training setting at \(B=16384\).

Input: Table 1: Top-1 test accuracy when training on ImageNet from scratch using DP-SGD. We train a NF-ResNet-50 with σ = 2.5, hyper-parameters of Table 2 and (B,S) = (32768, 18k) (Table 4).

Input: Table 2: Comparing optimal hyper-parameters. Keeping ηstep and S constant, we compare various changes in the training pipeline. We compare with the baseline of De et al. (2022) (blue line in Figure 1: NFResNet-50, learning rate at 4, EMA decay at 0.99999, 4 random augmentations averaged over 3 runs). Each gain is compared to the previous column.

Input: Table 4: Privacy parameter search. We use the optimal parameters described in Section 4.2.1 with σ = 2.5 for one expensive run and compare it with our optimal result

Input: Table 3: Low compute simulation of privacy parameter search. We start from B = 256 = 16384/64 and S = 72K. We use σ = 2.5/64 for all runs and no data augmentation.

Input: Figure 5: Impact of data augmentation on the test accuracy using pre-activation normalization vs post-activation in a 16-4-WideResnet on CIFAR-10. We compare simulation at (B, σ) = (512, 3 8 ) and reference (Bref , σref) = (4096, 3), both trained for S = 2,500 steps. Confidence intervals are plotted with two standard deviations over 5 runs. Augmentation Multiplicity Order corresponds to the number of augmentations per image, or K in Appendix C.

Input: Figure 4: Test accuracies at constant ηstep = Bref/( √ 2Nσref ) and S are (log) linearly decreasing with B. Dashed lines are computed using a linear regression on the crosses. Shaded areas correspond to 3 std confidence intervals. (Left) CIFAR-10 with 16-4-WideResNet for Sref steps. Each curve corresponds to a different value of ηstep. (Right) ImageNet with NF-ResNet-50 with various numbers of steps. The scaling law holds for various training configurations.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
1. Summary of evaluation:
The claim states that performance at large batch sizes is linearly predictable from small batch sizes. The evidence strongly supports this through multiple empirical demonstrations:

- Figure 4 shows clear log-linear relationships between batch size and accuracy for both CIFAR-10 and ImageNet datasets
- Tables 2-4 demonstrate successful prediction and optimization using smaller batch sizes to inform larger ones
- Multiple experiments validate the scaling law holds across different architectures and training configurations
- The predictions led to state-of-the-art results (Table 1), providing practical validation

The claim appears to be well-stated and appropriately reflects the evidence without exaggeration.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We then analyze the scaling laws of DNNs at constant TAN and show that performance at very large batch sizes (computationally intensive) is (linearly) predictable from performance at small batch sizes as illustrated in Figure 1.

The review comment to be evaluated is:

Strength:

* The observations (batch scaling law, privacy budget computation) in the paper are new and interesting.

* The proposed hyper-parameter tuning process can be very useful in practice, as it saves the computation requirement and therefore speeds up the development process.

Weaknesses:

* Overall, the writing is clear, but there are still some issues that need to be addressed:
    - Equation 2: many of the notations here are not defined, e.g., clip_C, l, theta, C.
    - Figure 4: B_ref, S_ref, \sigma_ref are not defined. I can guess their meaning from the context, but all notations should be defined explicitly in the text.
    - Page 5: it says "Simultaneously doubling the batch size and \sigma has a negligible or small impact on accuracy (Figure 4)". However, in Figure 4(b), the impact on the accuracy is not small. Indeed, this is claimed in the next sentence "Reciprocally, if \sigma > 4, dividing it by 2 simultaneously with halving q is likely to improve performance." These two sentences contradict each other.
    - Table 1: I guess "8,8." should be "8,"?

The evidence to be evaluated is:We introduce the notion of Total Amount of Noise (TAN) and discuss its connections to RDP accounting. We then demonstrate how training with reference privacy parameters \((q_{\mathrm{ref}},\sigma_{\mathrm{ref}},S)\) can be simulated with much lower computational resources using the same TAN with a smaller batch size. **Definition 4**.: _Let the individual signal-to-noise ratio \(\eta\) (and its inverse \(\Sigma\), the Total Amount of Noise or TAN) be as follows:_ \[\eta^{2}=\frac{1}{\Sigma^{2}}:=\frac{q^{2}S}{2\sigma^{2}}.\] We begin with a simple case to motivate our definition of TAN. We assume a one-dimensional model, where the gradients of all points are clipped to \(C\). Looking at Equation 2, in one batch, the expected signal from each sample is \(C/B\) with probability \(q=B/N\) and \(0\) otherwise. Therefore, the expected individual signal of each sample after \(S\) steps is \(SC/N\), and its squared norm is \(S^{2}C^{2}/N^{2}\).

In Figure 4, we perform this analysis on CIFAR-10 and ImageNet. We find that for triplets \((q,\sigma,S)\) for which \(q/\sigma=q_{\mathrm{ref}}/\sigma_{\mathrm{ref}}\) (keeping \(\eta_{\mathrm{step}}\) constant), the performance of the network (log) linearly decreases with the batch size. This is consistent with the (non-private) work of Smith et al. (2020), which shows that for a fixed number of steps, small batch sizes perform better in generalisation (i.e., test accuracy) than large batch sizes. The difference with our work is that we consider noisy updates and observe a (log) linear relationship between batch size and performance.

In Section 4.2.1, we use our simulated training with constant TAN to find optimal hyperparameters at low compute and improve performance for reference privacy parameters \((B_{\mathrm{ref}},\sigma_{\mathrm{ref}},S)=(16384,2.5,72\mathrm{k})\). In Section 4.2.2, we find optimal privacy parameters at constant TAN by changing the number of steps \(S\), leading to a new state of the art on ImageNet for \(\varepsilon=8\) (Table 1).

We run a large hyper-parameter search and report the best hyper-parameters in Table 2 as well as the corresponding improvement for various batch sizes (at constant \(\eta_{\text{step}}\) and \(S\)).

While we kept \(S\) constant in previous experiments, we now explore constant TAN triplets \((q,\sigma,S)\) by varying \(S\). We keep \(\sigma\) fixed to \(2.5\) and vary \((B,S)\) starting from the reference \((16384,72\text{K})\) at constant \(\eta=q^{2}S/(2\sigma^{2})\).

Since performing this privacy parameter search is computationally intensive, we first simulate training using our scaling law at \(B=256\) (with the same \(\eta_{\mathrm{step}}\)) and display our results in Table 3.

Finally, we perform one computationally expensive run at \(S=18\)k and \(B=32768\), with other hyper-parameters from Section 4.2.1, and show the results in Table 4.

Thus, we have shown how we can use TAN to perform optimal privacy parameter search while simulating each choice of optimal parameters at a much smaller cost.

We observe that the two simulated experiments with \(B=512\) represented by lighter colors in Figure 5 (\(2\) standard deviations around the means) have the same properties.

Therefore, using TAN through our scaling law can facilitate studying variants of the network architecture while reducing the computational costs.

On the other hand, since the global signal-to-noise ratio \(N\eta\) is held constant in all experiments, we expect to extract the same amount of information in each setup; adding more data makes this information richer, which explains the gain in accuracy.

We argue that the total amount of noise (TAN) is a simple but useful guiding principle to experiment with private training.

We further show that scaling batch size with noise level using TAN allows for ultra-efficient hyper-parameter search and demonstrate the power of this paradigm by establishing a new state of the art for DP training on Imagenet.

We used our scaling strategy (keeping \(\eta_{step}\) and \(S\) constant) to simulate the DP training with different architectures at low compute, studying noisy training without the burden of DP accounting.

Using TAN allowed us to explore and compare different architectures and optimizers, which would have been computationally impossible in the normal DP training setting at \(B=16384\).

Input: Table 1: Top-1 test accuracy when training on ImageNet from scratch using DP-SGD. We train a NF-ResNet-50 with σ = 2.5, hyper-parameters of Table 2 and (B,S) = (32768, 18k) (Table 4).

Input: Table 2: Comparing optimal hyper-parameters. Keeping ηstep and S constant, we compare various changes in the training pipeline. We compare with the baseline of De et al. (2022) (blue line in Figure 1: NFResNet-50, learning rate at 4, EMA decay at 0.99999, 4 random augmentations averaged over 3 runs). Each gain is compared to the previous column.

Input: Table 4: Privacy parameter search. We use the optimal parameters described in Section 4.2.1 with σ = 2.5 for one expensive run and compare it with our optimal result

Input: Table 3: Low compute simulation of privacy parameter search. We start from B = 256 = 16384/64 and S = 72K. We use σ = 2.5/64 for all runs and no data augmentation.

Input: Figure 5: Impact of data augmentation on the test accuracy using pre-activation normalization vs post-activation in a 16-4-WideResnet on CIFAR-10. We compare simulation at (B, σ) = (512, 3 8 ) and reference (Bref , σref) = (4096, 3), both trained for S = 2,500 steps. Confidence intervals are plotted with two standard deviations over 5 runs. Augmentation Multiplicity Order corresponds to the number of augmentations per image, or K in Appendix C.

Input: Figure 4: Test accuracies at constant ηstep = Bref/( √ 2Nσref ) and S are (log) linearly decreasing with B. Dashed lines are computed using a linear regression on the crosses. Shaded areas correspond to 3 std confidence intervals. (Left) CIFAR-10 with 16-4-WideResNet for Sref steps. Each curve corresponds to a different value of ηstep. (Right) ImageNet with NF-ResNet-50 with various numbers of steps. The scaling law holds for various training configurations.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me evaluate the claim: "We then analyze the scaling laws of DNNs at constant TAN and show that performance at very large batch sizes (computationally intensive) is (linearly) predictable from performance at small batch sizes as illustrated in Figure 1."

Brief Assessment:
The evidence strongly supports this claim through:
- Figure 4 demonstrates clear log-linear relationships between batch size and performance for both CIFAR-10 and ImageNet datasets
- Multiple experimental validations at different steps and configurations show consistent scaling behavior
- Tables 2-4 provide empirical validation of using smaller batch size predictions to optimize larger batch size performance
- The claim is carefully worded with "(linearly) predictable" which accurately reflects the log-linear relationship shown

The claim appears well-stated and accurately reflects the evidence without exaggeration.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We then analyze the scaling laws of DNNs at constant TAN and show that performance at very large batch sizes (computationally intensive) is (linearly) predictable from performance at small batch sizes as illustrated in Figure 1.

The review comment to be evaluated is:

Strength: It studies an important problem of how to scale DP-SGD with hyper-parameter search. The paper provides a neat and simple relationship between the privacy budget and the other parameters, i.e., the noise multiplier, steps and the sampling rate. This relationship motivates a way of scaling the batchsize and the noise multiplier without affecting the privacy accountant.

Weakness:
1. There is no application of using TAN relation in language task, either pretraining or fine-tuning.

2. The reference needs to cover more. Especially, Yu et al. 2021 starts the fine-tuning large language models with differential privacy and low-rank reparametrization.
Yu et al. 2021 Large Scale Private Learning via Low-rank Reparametrization.

The evidence to be evaluated is:We introduce the notion of Total Amount of Noise (TAN) and discuss its connections to RDP accounting. We then demonstrate how training with reference privacy parameters \((q_{\mathrm{ref}},\sigma_{\mathrm{ref}},S)\) can be simulated with much lower computational resources using the same TAN with a smaller batch size. **Definition 4**.: _Let the individual signal-to-noise ratio \(\eta\) (and its inverse \(\Sigma\), the Total Amount of Noise or TAN) be as follows:_ \[\eta^{2}=\frac{1}{\Sigma^{2}}:=\frac{q^{2}S}{2\sigma^{2}}.\] We begin with a simple case to motivate our definition of TAN. We assume a one-dimensional model, where the gradients of all points are clipped to \(C\). Looking at Equation 2, in one batch, the expected signal from each sample is \(C/B\) with probability \(q=B/N\) and \(0\) otherwise. Therefore, the expected individual signal of each sample after \(S\) steps is \(SC/N\), and its squared norm is \(S^{2}C^{2}/N^{2}\).

In Figure 4, we perform this analysis on CIFAR-10 and ImageNet. We find that for triplets \((q,\sigma,S)\) for which \(q/\sigma=q_{\mathrm{ref}}/\sigma_{\mathrm{ref}}\) (keeping \(\eta_{\mathrm{step}}\) constant), the performance of the network (log) linearly decreases with the batch size. This is consistent with the (non-private) work of Smith et al. (2020), which shows that for a fixed number of steps, small batch sizes perform better in generalisation (i.e., test accuracy) than large batch sizes. The difference with our work is that we consider noisy updates and observe a (log) linear relationship between batch size and performance.

In Section 4.2.1, we use our simulated training with constant TAN to find optimal hyperparameters at low compute and improve performance for reference privacy parameters \((B_{\mathrm{ref}},\sigma_{\mathrm{ref}},S)=(16384,2.5,72\mathrm{k})\). In Section 4.2.2, we find optimal privacy parameters at constant TAN by changing the number of steps \(S\), leading to a new state of the art on ImageNet for \(\varepsilon=8\) (Table 1).

We run a large hyper-parameter search and report the best hyper-parameters in Table 2 as well as the corresponding improvement for various batch sizes (at constant \(\eta_{\text{step}}\) and \(S\)).

While we kept \(S\) constant in previous experiments, we now explore constant TAN triplets \((q,\sigma,S)\) by varying \(S\). We keep \(\sigma\) fixed to \(2.5\) and vary \((B,S)\) starting from the reference \((16384,72\text{K})\) at constant \(\eta=q^{2}S/(2\sigma^{2})\).

Since performing this privacy parameter search is computationally intensive, we first simulate training using our scaling law at \(B=256\) (with the same \(\eta_{\mathrm{step}}\)) and display our results in Table 3.

Finally, we perform one computationally expensive run at \(S=18\)k and \(B=32768\), with other hyper-parameters from Section 4.2.1, and show the results in Table 4.

Thus, we have shown how we can use TAN to perform optimal privacy parameter search while simulating each choice of optimal parameters at a much smaller cost.

We observe that the two simulated experiments with \(B=512\) represented by lighter colors in Figure 5 (\(2\) standard deviations around the means) have the same properties.

Therefore, using TAN through our scaling law can facilitate studying variants of the network architecture while reducing the computational costs.

On the other hand, since the global signal-to-noise ratio \(N\eta\) is held constant in all experiments, we expect to extract the same amount of information in each setup; adding more data makes this information richer, which explains the gain in accuracy.

We argue that the total amount of noise (TAN) is a simple but useful guiding principle to experiment with private training.

We further show that scaling batch size with noise level using TAN allows for ultra-efficient hyper-parameter search and demonstrate the power of this paradigm by establishing a new state of the art for DP training on Imagenet.

We used our scaling strategy (keeping \(\eta_{step}\) and \(S\) constant) to simulate the DP training with different architectures at low compute, studying noisy training without the burden of DP accounting.

Using TAN allowed us to explore and compare different architectures and optimizers, which would have been computationally impossible in the normal DP training setting at \(B=16384\).

Input: Table 1: Top-1 test accuracy when training on ImageNet from scratch using DP-SGD. We train a NF-ResNet-50 with σ = 2.5, hyper-parameters of Table 2 and (B,S) = (32768, 18k) (Table 4).

Input: Table 2: Comparing optimal hyper-parameters. Keeping ηstep and S constant, we compare various changes in the training pipeline. We compare with the baseline of De et al. (2022) (blue line in Figure 1: NFResNet-50, learning rate at 4, EMA decay at 0.99999, 4 random augmentations averaged over 3 runs). Each gain is compared to the previous column.

Input: Table 4: Privacy parameter search. We use the optimal parameters described in Section 4.2.1 with σ = 2.5 for one expensive run and compare it with our optimal result

Input: Table 3: Low compute simulation of privacy parameter search. We start from B = 256 = 16384/64 and S = 72K. We use σ = 2.5/64 for all runs and no data augmentation.

Input: Figure 5: Impact of data augmentation on the test accuracy using pre-activation normalization vs post-activation in a 16-4-WideResnet on CIFAR-10. We compare simulation at (B, σ) = (512, 3 8 ) and reference (Bref , σref) = (4096, 3), both trained for S = 2,500 steps. Confidence intervals are plotted with two standard deviations over 5 runs. Augmentation Multiplicity Order corresponds to the number of augmentations per image, or K in Appendix C.

Input: Figure 4: Test accuracies at constant ηstep = Bref/( √ 2Nσref ) and S are (log) linearly decreasing with B. Dashed lines are computed using a linear regression on the crosses. Shaded areas correspond to 3 std confidence intervals. (Left) CIFAR-10 with 16-4-WideResNet for Sref steps. Each curve corresponds to a different value of ηstep. (Right) ImageNet with NF-ResNet-50 with various numbers of steps. The scaling law holds for various training configurations.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find that the claim about performance being linearly predictable from small to large batch sizes is well-supported by the evidence, particularly through:

1. Figure 4 demonstrates clear log-linear relationships between batch size and accuracy for both CIFAR-10 and ImageNet datasets across different configurations.

2. The evidence shows successful practical applications of this scaling relationship:
- Used for efficient hyperparameter search (Tables 2-4)
- Enabled architecture comparisons at lower compute costs
- Led to state-of-the-art results on ImageNet (39.2% accuracy)

The claim is carefully worded and appropriately scoped, acknowledging it's specifically about "performance at constant TAN." The review comments don't contradict the claim's validity regarding batch size scaling, though they note limitations in language tasks.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We then analyze the scaling laws of DNNs at constant TAN and show that performance at very large batch sizes (computationally intensive) is (linearly) predictable from performance at small batch sizes as illustrated in Figure 1.

The review comment to be evaluated is:

Major strengths:
1. The observation of the scaling law is new.
2. The results of hyperparameter tuning improve the state-of-the-art greatly.

Minor strengths:
1. This paper is easy to follow.
2. The visualization of the experimental results is good.
3. Figure 2 seems to deliver the same message as the linear part of Figure 3 (left) in (Li et al, 2021a), but the authors provide more explanations in Section 3.2.

Major weaknesses:
1. The privacy guarantee parameter $\varepsilon_{TAN}$ is an approximation of $\varepsilon_{RDP}$ and it is only a lower bound. For strict privacy guarantees, we need an upper bound.
2. There is no explanation for the reason for the scaling law.

Minor weaknesses:
Some of the statements in this paper are inconsistent with each other.
1. In the left figure of Figure 4, the legend shows that $\sigma_{ref}$ is constant for each line, but the caption says the $\eta_{step}$ is constant. I also wonder why the authors choose to fix different values in Figure 4 for CIFAR-10 and ImageNet. Is it because the results of CIFAR-10 do not look as linear as of ImageNet?
2. In Page 5, Choice of $\sigma$, the authors claimed that if $\sigma>4$, dividing it by 2 with halving q is likely to improve performance, but I cannot find experimental results supporting this.
3. In Page 7, the authors say that they decide not to use the gain from 'testing with augmentations' in Table 1 and Table 4. In Table 2, I guess the Total column is the sum of the other columns for B=128, ..., 1024. Therefore, the last row, B=16384, also has the AugTest (+0.8) in the Total result (6.7%). I guess in Table 4, the test ACC for B=16384, 36.9%, is derived by adding the total improvement 6.7% to the baseline 30.2%. Therefore, I am not sure if the authors forget to remove the improvement from AugTest.
4. In Page 9, in the last paragraph, the authors believe that the exponential increase in the privacy budget $\varepsilon$ as the noise level $\sigma$ decreases. I guess this is from Figure 2, but the x-axis for $\sigma$ is log-scale. Therefore, it is not easy to identify whether there is a linear or exponential relationship between $\varepsilon$ and $\sigma$.

By the way, there are duplicate references (Li et al, 2021a and 2021b).

The evidence to be evaluated is:We introduce the notion of Total Amount of Noise (TAN) and discuss its connections to RDP accounting. We then demonstrate how training with reference privacy parameters \((q_{\mathrm{ref}},\sigma_{\mathrm{ref}},S)\) can be simulated with much lower computational resources using the same TAN with a smaller batch size. **Definition 4**.: _Let the individual signal-to-noise ratio \(\eta\) (and its inverse \(\Sigma\), the Total Amount of Noise or TAN) be as follows:_ \[\eta^{2}=\frac{1}{\Sigma^{2}}:=\frac{q^{2}S}{2\sigma^{2}}.\] We begin with a simple case to motivate our definition of TAN. We assume a one-dimensional model, where the gradients of all points are clipped to \(C\). Looking at Equation 2, in one batch, the expected signal from each sample is \(C/B\) with probability \(q=B/N\) and \(0\) otherwise. Therefore, the expected individual signal of each sample after \(S\) steps is \(SC/N\), and its squared norm is \(S^{2}C^{2}/N^{2}\).

In Figure 4, we perform this analysis on CIFAR-10 and ImageNet. We find that for triplets \((q,\sigma,S)\) for which \(q/\sigma=q_{\mathrm{ref}}/\sigma_{\mathrm{ref}}\) (keeping \(\eta_{\mathrm{step}}\) constant), the performance of the network (log) linearly decreases with the batch size. This is consistent with the (non-private) work of Smith et al. (2020), which shows that for a fixed number of steps, small batch sizes perform better in generalisation (i.e., test accuracy) than large batch sizes. The difference with our work is that we consider noisy updates and observe a (log) linear relationship between batch size and performance.

In Section 4.2.1, we use our simulated training with constant TAN to find optimal hyperparameters at low compute and improve performance for reference privacy parameters \((B_{\mathrm{ref}},\sigma_{\mathrm{ref}},S)=(16384,2.5,72\mathrm{k})\). In Section 4.2.2, we find optimal privacy parameters at constant TAN by changing the number of steps \(S\), leading to a new state of the art on ImageNet for \(\varepsilon=8\) (Table 1).

We run a large hyper-parameter search and report the best hyper-parameters in Table 2 as well as the corresponding improvement for various batch sizes (at constant \(\eta_{\text{step}}\) and \(S\)).

While we kept \(S\) constant in previous experiments, we now explore constant TAN triplets \((q,\sigma,S)\) by varying \(S\). We keep \(\sigma\) fixed to \(2.5\) and vary \((B,S)\) starting from the reference \((16384,72\text{K})\) at constant \(\eta=q^{2}S/(2\sigma^{2})\).

Since performing this privacy parameter search is computationally intensive, we first simulate training using our scaling law at \(B=256\) (with the same \(\eta_{\mathrm{step}}\)) and display our results in Table 3.

Finally, we perform one computationally expensive run at \(S=18\)k and \(B=32768\), with other hyper-parameters from Section 4.2.1, and show the results in Table 4.

Thus, we have shown how we can use TAN to perform optimal privacy parameter search while simulating each choice of optimal parameters at a much smaller cost.

We observe that the two simulated experiments with \(B=512\) represented by lighter colors in Figure 5 (\(2\) standard deviations around the means) have the same properties.

Therefore, using TAN through our scaling law can facilitate studying variants of the network architecture while reducing the computational costs.

On the other hand, since the global signal-to-noise ratio \(N\eta\) is held constant in all experiments, we expect to extract the same amount of information in each setup; adding more data makes this information richer, which explains the gain in accuracy.

We argue that the total amount of noise (TAN) is a simple but useful guiding principle to experiment with private training.

We further show that scaling batch size with noise level using TAN allows for ultra-efficient hyper-parameter search and demonstrate the power of this paradigm by establishing a new state of the art for DP training on Imagenet.

We used our scaling strategy (keeping \(\eta_{step}\) and \(S\) constant) to simulate the DP training with different architectures at low compute, studying noisy training without the burden of DP accounting.

Using TAN allowed us to explore and compare different architectures and optimizers, which would have been computationally impossible in the normal DP training setting at \(B=16384\).

Input: Table 1: Top-1 test accuracy when training on ImageNet from scratch using DP-SGD. We train a NF-ResNet-50 with σ = 2.5, hyper-parameters of Table 2 and (B,S) = (32768, 18k) (Table 4).

Input: Table 2: Comparing optimal hyper-parameters. Keeping ηstep and S constant, we compare various changes in the training pipeline. We compare with the baseline of De et al. (2022) (blue line in Figure 1: NFResNet-50, learning rate at 4, EMA decay at 0.99999, 4 random augmentations averaged over 3 runs). Each gain is compared to the previous column.

Input: Table 4: Privacy parameter search. We use the optimal parameters described in Section 4.2.1 with σ = 2.5 for one expensive run and compare it with our optimal result

Input: Table 3: Low compute simulation of privacy parameter search. We start from B = 256 = 16384/64 and S = 72K. We use σ = 2.5/64 for all runs and no data augmentation.

Input: Figure 5: Impact of data augmentation on the test accuracy using pre-activation normalization vs post-activation in a 16-4-WideResnet on CIFAR-10. We compare simulation at (B, σ) = (512, 3 8 ) and reference (Bref , σref) = (4096, 3), both trained for S = 2,500 steps. Confidence intervals are plotted with two standard deviations over 5 runs. Augmentation Multiplicity Order corresponds to the number of augmentations per image, or K in Appendix C.

Input: Figure 4: Test accuracies at constant ηstep = Bref/( √ 2Nσref ) and S are (log) linearly decreasing with B. Dashed lines are computed using a linear regression on the crosses. Shaded areas correspond to 3 std confidence intervals. (Left) CIFAR-10 with 16-4-WideResNet for Sref steps. Each curve corresponds to a different value of ηstep. (Right) ImageNet with NF-ResNet-50 with various numbers of steps. The scaling law holds for various training configurations.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me assess the claim's validity based on the evidence:

Summary: The evidence strongly supports the claim about linear predictability of performance at large batch sizes from small batch sizes. Figure 4 demonstrates clear log-linear relationships between batch size and test accuracy for both CIFAR-10 and ImageNet datasets across different parameters. The authors validate this through extensive experiments, showing consistent scaling behavior. While there are some minor inconsistencies noted in the review comments regarding figure legends and test augmentations, these don't significantly impact the core claim about linear predictability. The evidence thoroughly validates their assertion with empirical data.

<score>1</score>

The claim is well-stated (score of 1) because:
1. Figure 4 directly demonstrates the linear relationship
2. Multiple datasets and configurations support the scaling law
3. The authors are careful in their wording, stating "performance...is (linearly) predictable" rather than making broader claims
4. The reviewer comments acknowledge the scaling law observation as a major strength
5. Tables 2-4 provide additional supporting evidence through systematic experiments validating the linear predictability


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We then analyze the scaling laws of DNNs at constant TAN and show that performance at very large batch sizes (computationally intensive) is (linearly) predictable from performance at small batch sizes as illustrated in Figure 1.

The review comment to be evaluated is:

Strengths:
1) Integrating the constraints of DP-SGD with optimization theory, in order to reliably predict good hyper-parameter choices, is one of the most important problems that needs to be solved in order to make differentially private deep learning practical. I'm pleased to see the authors tackle this difficult and important problem.
2) The metric proposed by the authors is intuitive and it appears to provide some helpful intuition.
3) The authors achieve strong performance when training from scratch on ImageNet with DP-SGD.

Weaknesses:
1) If TAN were an accurate measure of optimizability, then we should expect the train accuracy to be roughly constant at fixed TAN. However in practice, the authors find in figure 1 that the test accuracy decays log-linearly at fixed TAN as the batch size rises. I feel that the current paper is quite confusing because this discrepancy is not explicitly stated anywhere. I'd encourage the authors to make this explicit, and to explore in more detail why this discrepancy arises (eg comparing test vs train performance).

2) As discussed above, Figure 1 shows that TAN over-estimates the performance of large batch sizes. It would be nice to also see a figure at fixed batch size, sweeping the number of steps S, to see whether TAN under/over-estimates the performance of small/large step budgets.

3) As a minor point, note that on some truly sensitive data we cannot train non-private models (even internally). It would be good to acknowledge that the hyper-parameter transfer process described here cannot be used in these cases.

4) For many of the experiments, key hyper-parameters are lifted from prior work (eg step budget, target batch size). Is it not possible to directly infer good settings for these hyper-parameters using the TAN framework?

5) The paper essentially only considers ImageNet (with some very limited experiments on CIFAR-10). Since very few groups have run experiments at this scale it is not clear how challenging the baseline from De et al. is in practice.

The evidence to be evaluated is:We introduce the notion of Total Amount of Noise (TAN) and discuss its connections to RDP accounting. We then demonstrate how training with reference privacy parameters \((q_{\mathrm{ref}},\sigma_{\mathrm{ref}},S)\) can be simulated with much lower computational resources using the same TAN with a smaller batch size. **Definition 4**.: _Let the individual signal-to-noise ratio \(\eta\) (and its inverse \(\Sigma\), the Total Amount of Noise or TAN) be as follows:_ \[\eta^{2}=\frac{1}{\Sigma^{2}}:=\frac{q^{2}S}{2\sigma^{2}}.\] We begin with a simple case to motivate our definition of TAN. We assume a one-dimensional model, where the gradients of all points are clipped to \(C\). Looking at Equation 2, in one batch, the expected signal from each sample is \(C/B\) with probability \(q=B/N\) and \(0\) otherwise. Therefore, the expected individual signal of each sample after \(S\) steps is \(SC/N\), and its squared norm is \(S^{2}C^{2}/N^{2}\).

In Figure 4, we perform this analysis on CIFAR-10 and ImageNet. We find that for triplets \((q,\sigma,S)\) for which \(q/\sigma=q_{\mathrm{ref}}/\sigma_{\mathrm{ref}}\) (keeping \(\eta_{\mathrm{step}}\) constant), the performance of the network (log) linearly decreases with the batch size. This is consistent with the (non-private) work of Smith et al. (2020), which shows that for a fixed number of steps, small batch sizes perform better in generalisation (i.e., test accuracy) than large batch sizes. The difference with our work is that we consider noisy updates and observe a (log) linear relationship between batch size and performance.

In Section 4.2.1, we use our simulated training with constant TAN to find optimal hyperparameters at low compute and improve performance for reference privacy parameters \((B_{\mathrm{ref}},\sigma_{\mathrm{ref}},S)=(16384,2.5,72\mathrm{k})\). In Section 4.2.2, we find optimal privacy parameters at constant TAN by changing the number of steps \(S\), leading to a new state of the art on ImageNet for \(\varepsilon=8\) (Table 1).

We run a large hyper-parameter search and report the best hyper-parameters in Table 2 as well as the corresponding improvement for various batch sizes (at constant \(\eta_{\text{step}}\) and \(S\)).

While we kept \(S\) constant in previous experiments, we now explore constant TAN triplets \((q,\sigma,S)\) by varying \(S\). We keep \(\sigma\) fixed to \(2.5\) and vary \((B,S)\) starting from the reference \((16384,72\text{K})\) at constant \(\eta=q^{2}S/(2\sigma^{2})\).

Since performing this privacy parameter search is computationally intensive, we first simulate training using our scaling law at \(B=256\) (with the same \(\eta_{\mathrm{step}}\)) and display our results in Table 3.

Finally, we perform one computationally expensive run at \(S=18\)k and \(B=32768\), with other hyper-parameters from Section 4.2.1, and show the results in Table 4.

Thus, we have shown how we can use TAN to perform optimal privacy parameter search while simulating each choice of optimal parameters at a much smaller cost.

We observe that the two simulated experiments with \(B=512\) represented by lighter colors in Figure 5 (\(2\) standard deviations around the means) have the same properties.

Therefore, using TAN through our scaling law can facilitate studying variants of the network architecture while reducing the computational costs.

On the other hand, since the global signal-to-noise ratio \(N\eta\) is held constant in all experiments, we expect to extract the same amount of information in each setup; adding more data makes this information richer, which explains the gain in accuracy.

We argue that the total amount of noise (TAN) is a simple but useful guiding principle to experiment with private training.

We further show that scaling batch size with noise level using TAN allows for ultra-efficient hyper-parameter search and demonstrate the power of this paradigm by establishing a new state of the art for DP training on Imagenet.

We used our scaling strategy (keeping \(\eta_{step}\) and \(S\) constant) to simulate the DP training with different architectures at low compute, studying noisy training without the burden of DP accounting.

Using TAN allowed us to explore and compare different architectures and optimizers, which would have been computationally impossible in the normal DP training setting at \(B=16384\).

Input: Table 1: Top-1 test accuracy when training on ImageNet from scratch using DP-SGD. We train a NF-ResNet-50 with σ = 2.5, hyper-parameters of Table 2 and (B,S) = (32768, 18k) (Table 4).

Input: Table 2: Comparing optimal hyper-parameters. Keeping ηstep and S constant, we compare various changes in the training pipeline. We compare with the baseline of De et al. (2022) (blue line in Figure 1: NFResNet-50, learning rate at 4, EMA decay at 0.99999, 4 random augmentations averaged over 3 runs). Each gain is compared to the previous column.

Input: Table 4: Privacy parameter search. We use the optimal parameters described in Section 4.2.1 with σ = 2.5 for one expensive run and compare it with our optimal result

Input: Table 3: Low compute simulation of privacy parameter search. We start from B = 256 = 16384/64 and S = 72K. We use σ = 2.5/64 for all runs and no data augmentation.

Input: Figure 5: Impact of data augmentation on the test accuracy using pre-activation normalization vs post-activation in a 16-4-WideResnet on CIFAR-10. We compare simulation at (B, σ) = (512, 3 8 ) and reference (Bref , σref) = (4096, 3), both trained for S = 2,500 steps. Confidence intervals are plotted with two standard deviations over 5 runs. Augmentation Multiplicity Order corresponds to the number of augmentations per image, or K in Appendix C.

Input: Figure 4: Test accuracies at constant ηstep = Bref/( √ 2Nσref ) and S are (log) linearly decreasing with B. Dashed lines are computed using a linear regression on the crosses. Shaded areas correspond to 3 std confidence intervals. (Left) CIFAR-10 with 16-4-WideResNet for Sref steps. Each curve corresponds to a different value of ηstep. (Right) ImageNet with NF-ResNet-50 with various numbers of steps. The scaling law holds for various training configurations.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find this claim to be somewhat overstated. While Figure 4 does show a log-linear relationship between batch size and performance at constant TAN, the claim of "linear predictability" oversimplifies important nuances:

1. The reviewer correctly points out that TAN overestimates performance at large batch sizes, showing decay rather than constant performance.

2. The evidence shows log-linear (not strictly linear) relationships.

3. The relationship holds primarily for test accuracy, but the claim doesn't specify this limitation.

4. While the scaling behavior is demonstrated, calling it "predictable" suggests more precision than shown in the data.

<score>3</score>

The claim captures an important finding but overstates the strength and simplicity of the relationship. A more precise statement would acknowledge the log-linear nature of the relationship, specify that it applies to test accuracy, and note the performance decay at larger batch sizes.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

* We define the notion of Total Amount of Noise (TAN) and show that when the budget \(\varepsilon\) is not overcharged, it only depends on TAN;

The evidence to be evaluated is:We introduce the notion of Total Amount of Noise (TAN) and discuss its connections to RDP accounting.

**Definition 4**.: _Let the individual signal-to-noise ratio \(\eta\) (and its inverse \(\Sigma\), the Total Amount of Noise or TAN) be as follows:_ \[\eta^{2}=\frac{1}{\Sigma^{2}}:=\frac{q^{2}S}{2\sigma^{2}}.\]

\[\varepsilon_{\mathrm{RDP}}\approx\eta^{2}+\min_{\alpha}\left((\alpha-1)\eta^ {2}+\frac{\log{(1/\delta)}}{\alpha-1}\right)=\eta^{2}+2\eta\sqrt{\log{(1/ \delta)}}=:\varepsilon_{\mathrm{TAN}}(\eta). \tag{4}\] We verify this relationship empirically, and in particular choose \(\eta\) to get a desired \(\varepsilon_{\mathrm{TAN}}\) in Figure 2.

Having this simple approximation for \(\varepsilon\) is useful in practice because it allows for simple mental gymnastics: for instance, doubling the sampling rate \(q\) while dividing the number of steps \(S\) by \(4\) should leave the privacy budget constant, which we observe empirically.

In Section 4.2.1, we use our simulated training with constant TAN to find optimal hyperparameters at low compute and improve performance for reference privacy parameters \((B_{\mathrm{ref}},\sigma_{\mathrm{ref}},S)=(16384,2.5,72\mathrm{k})\). In Section 4.2.2, we find optimal privacy parameters at constant TAN by changing the number of steps \(S\), leading to a new state of the art on ImageNet for \(\varepsilon=8\) (Table 1).

While we kept \(S\) constant in previous experiments, we now explore constant TAN triplets \((q,\sigma,S)\) by varying \(S\). We keep \(\sigma\) fixed to \(2.5\) and vary \((B,S)\) starting from the reference \((16384,72\text{K})\) at constant \(\eta=q^{2}S/(2\sigma^{2})\). Given that \(\sigma>2\), we stay in the _almost_ constant privacy regime (see Figure 2): we indeed observe \(\varepsilon\approx\varepsilon_{\mathrm{TAN}}\) in Table 4.

Thus, we have shown how we can use TAN to perform optimal privacy parameter search while simulating each choice of optimal parameters at a much smaller cost.

Therefore, using TAN through our scaling law can facilitate studying variants of the network architecture while reducing the computational costs.

We argue that the total amount of noise (TAN) is a simple but useful guiding principle to experiment with private training. In particular, we demonstrate that the privacy budget is either a direct function of TAN or can be reduced. We further show that scaling batch size with noise level using TAN allows for ultra-efficient hyper-parameter search and demonstrate the power of this paradigm by establishing a new state of the art for DP training on Imagenet.

Using TAN allowed us to explore and compare different architectures and optimizers, which would have been computationally impossible in the normal DP training setting at \(B=16384\).

Input: Table 1: Top-1 test accuracy when training on ImageNet from scratch using DP-SGD. We train a NF-ResNet-50 with σ = 2.5, hyper-parameters of Table 2 and (B,S) = (32768, 18k) (Table 4).

Input: Table 4: Privacy parameter search. We use the optimal parameters described in Section 4.2.1 with σ = 2.5 for one expensive run and compare it with our optimal result

Input: Figure 2: Privacy budget ε as a function of the noise level σ with η constant. On both figures, each curve corresponds to a different number of steps S, and each point on the curve is computed at a sampling rate q such that η is constant. On the left, we use η = 0.13 (resulting in εTAN = 1 in Equation 4). On the right, we use η = 0.95 (εTAN = 8). We observe a “privacy wall” imposing σ ≥ 0.5 for meaningful level of privacy budget ε, and σ ≥ 2 for constant ε ≈ εTAN.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim that TAN determines the privacy budget ε when not overcharged is supported by several pieces of evidence:

1. Figure 2 demonstrates that for σ ≥ 2, ε remains approximately constant when TAN (η) is fixed, regardless of steps S
2. Table 4 shows empirical validation - similar ε values (8.00 vs 7.97) are achieved with different (B,S) combinations while maintaining constant TAN
3. The mathematical relationship between TAN and ε is explicitly defined in Equation 4
4. Multiple experimental validations demonstrate the practical utility of TAN for parameter search

The claim appears well-stated and accurately reflects the mathematical and empirical evidence without exaggeration. The paper acknowledges limitations (e.g., "almost constant privacy regime") and provides thorough experimental validation.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

* We define the notion of Total Amount of Noise (TAN) and show that when the budget \(\varepsilon\) is not overcharged, it only depends on TAN;

The review comment to be evaluated is:

Strength:

* The observations (batch scaling law, privacy budget computation) in the paper are new and interesting.

* The proposed hyper-parameter tuning process can be very useful in practice, as it saves the computation requirement and therefore speeds up the development process.

Weaknesses:

* Overall, the writing is clear, but there are still some issues that need to be addressed:
    - Equation 2: many of the notations here are not defined, e.g., clip_C, l, theta, C.
    - Figure 4: B_ref, S_ref, \sigma_ref are not defined. I can guess their meaning from the context, but all notations should be defined explicitly in the text.
    - Page 5: it says "Simultaneously doubling the batch size and \sigma has a negligible or small impact on accuracy (Figure 4)". However, in Figure 4(b), the impact on the accuracy is not small. Indeed, this is claimed in the next sentence "Reciprocally, if \sigma > 4, dividing it by 2 simultaneously with halving q is likely to improve performance." These two sentences contradict each other.
    - Table 1: I guess "8,8." should be "8,"?

The evidence to be evaluated is:We introduce the notion of Total Amount of Noise (TAN) and discuss its connections to RDP accounting.

**Definition 4**.: _Let the individual signal-to-noise ratio \(\eta\) (and its inverse \(\Sigma\), the Total Amount of Noise or TAN) be as follows:_ \[\eta^{2}=\frac{1}{\Sigma^{2}}:=\frac{q^{2}S}{2\sigma^{2}}.\]

\[\varepsilon_{\mathrm{RDP}}\approx\eta^{2}+\min_{\alpha}\left((\alpha-1)\eta^ {2}+\frac{\log{(1/\delta)}}{\alpha-1}\right)=\eta^{2}+2\eta\sqrt{\log{(1/ \delta)}}=:\varepsilon_{\mathrm{TAN}}(\eta). \tag{4}\] We verify this relationship empirically, and in particular choose \(\eta\) to get a desired \(\varepsilon_{\mathrm{TAN}}\) in Figure 2.

Having this simple approximation for \(\varepsilon\) is useful in practice because it allows for simple mental gymnastics: for instance, doubling the sampling rate \(q\) while dividing the number of steps \(S\) by \(4\) should leave the privacy budget constant, which we observe empirically.

In Section 4.2.1, we use our simulated training with constant TAN to find optimal hyperparameters at low compute and improve performance for reference privacy parameters \((B_{\mathrm{ref}},\sigma_{\mathrm{ref}},S)=(16384,2.5,72\mathrm{k})\). In Section 4.2.2, we find optimal privacy parameters at constant TAN by changing the number of steps \(S\), leading to a new state of the art on ImageNet for \(\varepsilon=8\) (Table 1).

While we kept \(S\) constant in previous experiments, we now explore constant TAN triplets \((q,\sigma,S)\) by varying \(S\). We keep \(\sigma\) fixed to \(2.5\) and vary \((B,S)\) starting from the reference \((16384,72\text{K})\) at constant \(\eta=q^{2}S/(2\sigma^{2})\). Given that \(\sigma>2\), we stay in the _almost_ constant privacy regime (see Figure 2): we indeed observe \(\varepsilon\approx\varepsilon_{\mathrm{TAN}}\) in Table 4.

Thus, we have shown how we can use TAN to perform optimal privacy parameter search while simulating each choice of optimal parameters at a much smaller cost.

Therefore, using TAN through our scaling law can facilitate studying variants of the network architecture while reducing the computational costs.

We argue that the total amount of noise (TAN) is a simple but useful guiding principle to experiment with private training. In particular, we demonstrate that the privacy budget is either a direct function of TAN or can be reduced. We further show that scaling batch size with noise level using TAN allows for ultra-efficient hyper-parameter search and demonstrate the power of this paradigm by establishing a new state of the art for DP training on Imagenet.

Using TAN allowed us to explore and compare different architectures and optimizers, which would have been computationally impossible in the normal DP training setting at \(B=16384\).

Input: Table 1: Top-1 test accuracy when training on ImageNet from scratch using DP-SGD. We train a NF-ResNet-50 with σ = 2.5, hyper-parameters of Table 2 and (B,S) = (32768, 18k) (Table 4).

Input: Table 4: Privacy parameter search. We use the optimal parameters described in Section 4.2.1 with σ = 2.5 for one expensive run and compare it with our optimal result

Input: Figure 2: Privacy budget ε as a function of the noise level σ with η constant. On both figures, each curve corresponds to a different number of steps S, and each point on the curve is computed at a sampling rate q such that η is constant. On the left, we use η = 0.13 (resulting in εTAN = 1 in Equation 4). On the right, we use η = 0.95 (εTAN = 8). We observe a “privacy wall” imposing σ ≥ 0.5 for meaningful level of privacy budget ε, and σ ≥ 2 for constant ε ≈ εTAN.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me assess the claim: "We define the notion of Total Amount of Noise (TAN) and show that when the budget ε is not overcharged, it only depends on TAN."

The evidence strongly supports this claim through:
1. Clear definition of TAN (Definition 4)
2. Mathematical relationship between TAN and privacy budget (Equation 4)
3. Empirical validation shown in Figure 2, demonstrating constant ε ≈ εTAN when σ ≥ 2
4. Practical demonstrations in Tables 1 and 4 showing consistent results when maintaining constant TAN

The claim is well-stated and carefully qualified with "when the budget ε is not overcharged." The evidence provides both theoretical foundation and experimental validation. The reviewer comments don't raise concerns about this specific claim.

<score>1</score>

Justification:
- The claim is precise and accurately reflects the mathematical definition and empirical evidence
- The limitation ("when the budget ε is not overcharged") is explicitly stated
- Figure 2 provides clear empirical support for the TAN-ε relationship
- The tables demonstrate practical applications maintaining constant TAN
- The theoretical foundation is well-established through Definition 4 and Equation 4
- The paper acknowledges boundary conditions (e.g., σ ≥ 2 for constant ε regime)
- The reviewer's concerns about notation clarity don't impact the fundamental validity of this claim


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

* We define the notion of Total Amount of Noise (TAN) and show that when the budget \(\varepsilon\) is not overcharged, it only depends on TAN;

The review comment to be evaluated is:

Strength: It studies an important problem of how to scale DP-SGD with hyper-parameter search. The paper provides a neat and simple relationship between the privacy budget and the other parameters, i.e., the noise multiplier, steps and the sampling rate. This relationship motivates a way of scaling the batchsize and the noise multiplier without affecting the privacy accountant.

Weakness:
1. There is no application of using TAN relation in language task, either pretraining or fine-tuning.

2. The reference needs to cover more. Especially, Yu et al. 2021 starts the fine-tuning large language models with differential privacy and low-rank reparametrization.
Yu et al. 2021 Large Scale Private Learning via Low-rank Reparametrization.

The evidence to be evaluated is:We introduce the notion of Total Amount of Noise (TAN) and discuss its connections to RDP accounting.

**Definition 4**.: _Let the individual signal-to-noise ratio \(\eta\) (and its inverse \(\Sigma\), the Total Amount of Noise or TAN) be as follows:_ \[\eta^{2}=\frac{1}{\Sigma^{2}}:=\frac{q^{2}S}{2\sigma^{2}}.\]

\[\varepsilon_{\mathrm{RDP}}\approx\eta^{2}+\min_{\alpha}\left((\alpha-1)\eta^ {2}+\frac{\log{(1/\delta)}}{\alpha-1}\right)=\eta^{2}+2\eta\sqrt{\log{(1/ \delta)}}=:\varepsilon_{\mathrm{TAN}}(\eta). \tag{4}\] We verify this relationship empirically, and in particular choose \(\eta\) to get a desired \(\varepsilon_{\mathrm{TAN}}\) in Figure 2.

Having this simple approximation for \(\varepsilon\) is useful in practice because it allows for simple mental gymnastics: for instance, doubling the sampling rate \(q\) while dividing the number of steps \(S\) by \(4\) should leave the privacy budget constant, which we observe empirically.

In Section 4.2.1, we use our simulated training with constant TAN to find optimal hyperparameters at low compute and improve performance for reference privacy parameters \((B_{\mathrm{ref}},\sigma_{\mathrm{ref}},S)=(16384,2.5,72\mathrm{k})\). In Section 4.2.2, we find optimal privacy parameters at constant TAN by changing the number of steps \(S\), leading to a new state of the art on ImageNet for \(\varepsilon=8\) (Table 1).

While we kept \(S\) constant in previous experiments, we now explore constant TAN triplets \((q,\sigma,S)\) by varying \(S\). We keep \(\sigma\) fixed to \(2.5\) and vary \((B,S)\) starting from the reference \((16384,72\text{K})\) at constant \(\eta=q^{2}S/(2\sigma^{2})\). Given that \(\sigma>2\), we stay in the _almost_ constant privacy regime (see Figure 2): we indeed observe \(\varepsilon\approx\varepsilon_{\mathrm{TAN}}\) in Table 4.

Thus, we have shown how we can use TAN to perform optimal privacy parameter search while simulating each choice of optimal parameters at a much smaller cost.

Therefore, using TAN through our scaling law can facilitate studying variants of the network architecture while reducing the computational costs.

We argue that the total amount of noise (TAN) is a simple but useful guiding principle to experiment with private training. In particular, we demonstrate that the privacy budget is either a direct function of TAN or can be reduced. We further show that scaling batch size with noise level using TAN allows for ultra-efficient hyper-parameter search and demonstrate the power of this paradigm by establishing a new state of the art for DP training on Imagenet.

Using TAN allowed us to explore and compare different architectures and optimizers, which would have been computationally impossible in the normal DP training setting at \(B=16384\).

Input: Table 1: Top-1 test accuracy when training on ImageNet from scratch using DP-SGD. We train a NF-ResNet-50 with σ = 2.5, hyper-parameters of Table 2 and (B,S) = (32768, 18k) (Table 4).

Input: Table 4: Privacy parameter search. We use the optimal parameters described in Section 4.2.1 with σ = 2.5 for one expensive run and compare it with our optimal result

Input: Figure 2: Privacy budget ε as a function of the noise level σ with η constant. On both figures, each curve corresponds to a different number of steps S, and each point on the curve is computed at a sampling rate q such that η is constant. On the left, we use η = 0.13 (resulting in εTAN = 1 in Equation 4). On the right, we use η = 0.95 (εTAN = 8). We observe a “privacy wall” imposing σ ≥ 0.5 for meaningful level of privacy budget ε, and σ ≥ 2 for constant ε ≈ εTAN.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me evaluate the claim based on the provided evidence:

Summary: The claim about TAN's relationship with privacy budget ε appears well-supported by the evidence. The paper:
- Formally defines TAN and its relationship to RDP accounting (Definition 4)
- Demonstrates empirically through Figure 2 that ε ≈ εTAN when σ ≥ 2
- Validates the relationship through experimental results in Table 4
- Shows practical utility by achieving SOTA results on ImageNet (Table 1)
- Demonstrates how TAN enables efficient hyperparameter search

The claim is carefully worded with the crucial qualifier "when the budget ε is not overcharged," which is supported by the evidence showing this holds for σ ≥ 2. The reviewer comments do not contradict the claim's validity within its stated scope, though they point out limitations in application domains.

<score>1</score>

Justification:
1. The claim is precisely stated and matches the mathematical and empirical evidence
2. The limitation ("when ε is not overcharged") is explicitly mentioned
3. Both theoretical foundation (Definition 4, Equation 4) and empirical validation (Figure 2, Tables 1 & 4) are provided
4. The reviewer's strengths comment specifically validates the "neat and simple relationship" described
5. While the reviewer notes limitations in language tasks, this doesn't affect the claim's validity since it doesn't make assertions about specific domains


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

* We define the notion of Total Amount of Noise (TAN) and show that when the budget \(\varepsilon\) is not overcharged, it only depends on TAN;

The review comment to be evaluated is:

Major strengths:
1. The observation of the scaling law is new.
2. The results of hyperparameter tuning improve the state-of-the-art greatly.

Minor strengths:
1. This paper is easy to follow.
2. The visualization of the experimental results is good.
3. Figure 2 seems to deliver the same message as the linear part of Figure 3 (left) in (Li et al, 2021a), but the authors provide more explanations in Section 3.2.

Major weaknesses:
1. The privacy guarantee parameter $\varepsilon_{TAN}$ is an approximation of $\varepsilon_{RDP}$ and it is only a lower bound. For strict privacy guarantees, we need an upper bound.
2. There is no explanation for the reason for the scaling law.

Minor weaknesses:
Some of the statements in this paper are inconsistent with each other.
1. In the left figure of Figure 4, the legend shows that $\sigma_{ref}$ is constant for each line, but the caption says the $\eta_{step}$ is constant. I also wonder why the authors choose to fix different values in Figure 4 for CIFAR-10 and ImageNet. Is it because the results of CIFAR-10 do not look as linear as of ImageNet?
2. In Page 5, Choice of $\sigma$, the authors claimed that if $\sigma>4$, dividing it by 2 with halving q is likely to improve performance, but I cannot find experimental results supporting this.
3. In Page 7, the authors say that they decide not to use the gain from 'testing with augmentations' in Table 1 and Table 4. In Table 2, I guess the Total column is the sum of the other columns for B=128, ..., 1024. Therefore, the last row, B=16384, also has the AugTest (+0.8) in the Total result (6.7%). I guess in Table 4, the test ACC for B=16384, 36.9%, is derived by adding the total improvement 6.7% to the baseline 30.2%. Therefore, I am not sure if the authors forget to remove the improvement from AugTest.
4. In Page 9, in the last paragraph, the authors believe that the exponential increase in the privacy budget $\varepsilon$ as the noise level $\sigma$ decreases. I guess this is from Figure 2, but the x-axis for $\sigma$ is log-scale. Therefore, it is not easy to identify whether there is a linear or exponential relationship between $\varepsilon$ and $\sigma$.

By the way, there are duplicate references (Li et al, 2021a and 2021b).

The evidence to be evaluated is:We introduce the notion of Total Amount of Noise (TAN) and discuss its connections to RDP accounting.

**Definition 4**.: _Let the individual signal-to-noise ratio \(\eta\) (and its inverse \(\Sigma\), the Total Amount of Noise or TAN) be as follows:_ \[\eta^{2}=\frac{1}{\Sigma^{2}}:=\frac{q^{2}S}{2\sigma^{2}}.\]

\[\varepsilon_{\mathrm{RDP}}\approx\eta^{2}+\min_{\alpha}\left((\alpha-1)\eta^ {2}+\frac{\log{(1/\delta)}}{\alpha-1}\right)=\eta^{2}+2\eta\sqrt{\log{(1/ \delta)}}=:\varepsilon_{\mathrm{TAN}}(\eta). \tag{4}\] We verify this relationship empirically, and in particular choose \(\eta\) to get a desired \(\varepsilon_{\mathrm{TAN}}\) in Figure 2.

Having this simple approximation for \(\varepsilon\) is useful in practice because it allows for simple mental gymnastics: for instance, doubling the sampling rate \(q\) while dividing the number of steps \(S\) by \(4\) should leave the privacy budget constant, which we observe empirically.

In Section 4.2.1, we use our simulated training with constant TAN to find optimal hyperparameters at low compute and improve performance for reference privacy parameters \((B_{\mathrm{ref}},\sigma_{\mathrm{ref}},S)=(16384,2.5,72\mathrm{k})\). In Section 4.2.2, we find optimal privacy parameters at constant TAN by changing the number of steps \(S\), leading to a new state of the art on ImageNet for \(\varepsilon=8\) (Table 1).

While we kept \(S\) constant in previous experiments, we now explore constant TAN triplets \((q,\sigma,S)\) by varying \(S\). We keep \(\sigma\) fixed to \(2.5\) and vary \((B,S)\) starting from the reference \((16384,72\text{K})\) at constant \(\eta=q^{2}S/(2\sigma^{2})\). Given that \(\sigma>2\), we stay in the _almost_ constant privacy regime (see Figure 2): we indeed observe \(\varepsilon\approx\varepsilon_{\mathrm{TAN}}\) in Table 4.

Thus, we have shown how we can use TAN to perform optimal privacy parameter search while simulating each choice of optimal parameters at a much smaller cost.

Therefore, using TAN through our scaling law can facilitate studying variants of the network architecture while reducing the computational costs.

We argue that the total amount of noise (TAN) is a simple but useful guiding principle to experiment with private training. In particular, we demonstrate that the privacy budget is either a direct function of TAN or can be reduced. We further show that scaling batch size with noise level using TAN allows for ultra-efficient hyper-parameter search and demonstrate the power of this paradigm by establishing a new state of the art for DP training on Imagenet.

Using TAN allowed us to explore and compare different architectures and optimizers, which would have been computationally impossible in the normal DP training setting at \(B=16384\).

Input: Table 1: Top-1 test accuracy when training on ImageNet from scratch using DP-SGD. We train a NF-ResNet-50 with σ = 2.5, hyper-parameters of Table 2 and (B,S) = (32768, 18k) (Table 4).

Input: Table 4: Privacy parameter search. We use the optimal parameters described in Section 4.2.1 with σ = 2.5 for one expensive run and compare it with our optimal result

Input: Figure 2: Privacy budget ε as a function of the noise level σ with η constant. On both figures, each curve corresponds to a different number of steps S, and each point on the curve is computed at a sampling rate q such that η is constant. On the left, we use η = 0.13 (resulting in εTAN = 1 in Equation 4). On the right, we use η = 0.95 (εTAN = 8). We observe a “privacy wall” imposing σ ≥ 0.5 for meaningful level of privacy budget ε, and σ ≥ 2 for constant ε ≈ εTAN.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate the claim: "We define the notion of Total Amount of Noise (TAN) and show that when the budget ε is not overcharged, it only depends on TAN."

Summary of evaluation:
The evidence strongly supports the first part of the claim about defining TAN (Definition 4 clearly shows this). However, the second part about ε only depending on TAN when "not overcharged" is somewhat overstated. The evidence shows that this relationship is approximate (εRDP≈η²...) and the reviewer notes it's only a lower bound. Figure 2 demonstrates this relationship holds mainly when σ≥2, showing important limitations not acknowledged in the claim.

<score>3</score>

Detailed justification:

1. Evidence supporting parts of the claim:
- Clear definition of TAN through Definition 4
- Empirical verification shown in Figure 2
- Practical demonstrations in Tables 1 and 4

2. Areas of overstatement:
- The reviewer points out εTAN is only a lower bound, not exact
- The relationship is approximate (uses ≈ not =)
- Figure 2 shows this only holds in specific conditions (σ≥2)
- Reviewer notes "no explanation for the reason for the scaling law"

3. The claim would be better stated by:
- Acknowledging the approximate nature of the relationship
- Specifying the conditions under which it holds
- Noting it's a lower bound rather than exact relationship

The claim isn't severely overstated but lacks important qualifiers and limitations that are evident in both the evidence and reviewer comments.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

* We define the notion of Total Amount of Noise (TAN) and show that when the budget \(\varepsilon\) is not overcharged, it only depends on TAN;

The review comment to be evaluated is:

Strengths:
1) Integrating the constraints of DP-SGD with optimization theory, in order to reliably predict good hyper-parameter choices, is one of the most important problems that needs to be solved in order to make differentially private deep learning practical. I'm pleased to see the authors tackle this difficult and important problem.
2) The metric proposed by the authors is intuitive and it appears to provide some helpful intuition.
3) The authors achieve strong performance when training from scratch on ImageNet with DP-SGD.

Weaknesses:
1) If TAN were an accurate measure of optimizability, then we should expect the train accuracy to be roughly constant at fixed TAN. However in practice, the authors find in figure 1 that the test accuracy decays log-linearly at fixed TAN as the batch size rises. I feel that the current paper is quite confusing because this discrepancy is not explicitly stated anywhere. I'd encourage the authors to make this explicit, and to explore in more detail why this discrepancy arises (eg comparing test vs train performance).

2) As discussed above, Figure 1 shows that TAN over-estimates the performance of large batch sizes. It would be nice to also see a figure at fixed batch size, sweeping the number of steps S, to see whether TAN under/over-estimates the performance of small/large step budgets.

3) As a minor point, note that on some truly sensitive data we cannot train non-private models (even internally). It would be good to acknowledge that the hyper-parameter transfer process described here cannot be used in these cases.

4) For many of the experiments, key hyper-parameters are lifted from prior work (eg step budget, target batch size). Is it not possible to directly infer good settings for these hyper-parameters using the TAN framework?

5) The paper essentially only considers ImageNet (with some very limited experiments on CIFAR-10). Since very few groups have run experiments at this scale it is not clear how challenging the baseline from De et al. is in practice.

The evidence to be evaluated is:We introduce the notion of Total Amount of Noise (TAN) and discuss its connections to RDP accounting.

**Definition 4**.: _Let the individual signal-to-noise ratio \(\eta\) (and its inverse \(\Sigma\), the Total Amount of Noise or TAN) be as follows:_ \[\eta^{2}=\frac{1}{\Sigma^{2}}:=\frac{q^{2}S}{2\sigma^{2}}.\]

\[\varepsilon_{\mathrm{RDP}}\approx\eta^{2}+\min_{\alpha}\left((\alpha-1)\eta^ {2}+\frac{\log{(1/\delta)}}{\alpha-1}\right)=\eta^{2}+2\eta\sqrt{\log{(1/ \delta)}}=:\varepsilon_{\mathrm{TAN}}(\eta). \tag{4}\] We verify this relationship empirically, and in particular choose \(\eta\) to get a desired \(\varepsilon_{\mathrm{TAN}}\) in Figure 2.

Having this simple approximation for \(\varepsilon\) is useful in practice because it allows for simple mental gymnastics: for instance, doubling the sampling rate \(q\) while dividing the number of steps \(S\) by \(4\) should leave the privacy budget constant, which we observe empirically.

In Section 4.2.1, we use our simulated training with constant TAN to find optimal hyperparameters at low compute and improve performance for reference privacy parameters \((B_{\mathrm{ref}},\sigma_{\mathrm{ref}},S)=(16384,2.5,72\mathrm{k})\). In Section 4.2.2, we find optimal privacy parameters at constant TAN by changing the number of steps \(S\), leading to a new state of the art on ImageNet for \(\varepsilon=8\) (Table 1).

While we kept \(S\) constant in previous experiments, we now explore constant TAN triplets \((q,\sigma,S)\) by varying \(S\). We keep \(\sigma\) fixed to \(2.5\) and vary \((B,S)\) starting from the reference \((16384,72\text{K})\) at constant \(\eta=q^{2}S/(2\sigma^{2})\). Given that \(\sigma>2\), we stay in the _almost_ constant privacy regime (see Figure 2): we indeed observe \(\varepsilon\approx\varepsilon_{\mathrm{TAN}}\) in Table 4.

Thus, we have shown how we can use TAN to perform optimal privacy parameter search while simulating each choice of optimal parameters at a much smaller cost.

Therefore, using TAN through our scaling law can facilitate studying variants of the network architecture while reducing the computational costs.

We argue that the total amount of noise (TAN) is a simple but useful guiding principle to experiment with private training. In particular, we demonstrate that the privacy budget is either a direct function of TAN or can be reduced. We further show that scaling batch size with noise level using TAN allows for ultra-efficient hyper-parameter search and demonstrate the power of this paradigm by establishing a new state of the art for DP training on Imagenet.

Using TAN allowed us to explore and compare different architectures and optimizers, which would have been computationally impossible in the normal DP training setting at \(B=16384\).

Input: Table 1: Top-1 test accuracy when training on ImageNet from scratch using DP-SGD. We train a NF-ResNet-50 with σ = 2.5, hyper-parameters of Table 2 and (B,S) = (32768, 18k) (Table 4).

Input: Table 4: Privacy parameter search. We use the optimal parameters described in Section 4.2.1 with σ = 2.5 for one expensive run and compare it with our optimal result

Input: Figure 2: Privacy budget ε as a function of the noise level σ with η constant. On both figures, each curve corresponds to a different number of steps S, and each point on the curve is computed at a sampling rate q such that η is constant. On the left, we use η = 0.13 (resulting in εTAN = 1 in Equation 4). On the right, we use η = 0.95 (εTAN = 8). We observe a “privacy wall” imposing σ ≥ 0.5 for meaningful level of privacy budget ε, and σ ≥ 2 for constant ε ≈ εTAN.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After careful review, the claim that TAN determines privacy budget ε when not overcharged appears partially supported but somewhat overstated. While Figure 2 and equations show TAN's relationship with ε, and Table 4 demonstrates some practical applications, the reviewer points out important limitations: test accuracy decays log-linearly at fixed TAN as batch size increases (Figure 1, not shown), suggesting TAN isn't a complete predictor. The paper also lacks comprehensive validation across different datasets beyond ImageNet. The relationship exists but seems more complex than claimed, with notable exceptions not fully acknowledged.

<score>3</score>

Justification:
1. The evidence supports basic TAN-ε relationship:
- Equation 4 shows mathematical relationship
- Figure 2 demonstrates empirical validation for σ ≥ 2
- Table 4 shows practical application

2. Key limitations/overstatements:
- Reviewer points out TAN overestimates performance at large batch sizes
- Limited validation (mainly ImageNet)
- Reviewer note: "If TAN were an accurate measure of optimizability, then we should expect the train accuracy to be roughly constant at fixed TAN. However in practice...test accuracy decays log-linearly"
- Paper doesn't fully acknowledge these limitations or explain discrepancies

3. The claim would be better stated with qualifiers about conditions where TAN-ε relationship holds and acknowledgment of limitations.