Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We first use the tools of Renyi Differential Privacy (RDP) to show that the privacy budget, when not overcharged, only depends on the total amount of noise (TAN) injected throughout training.

The review comment to be evaluated is:

Strength: It studies an important problem of how to scale DP-SGD with hyper-parameter search. The paper provides a neat and simple relationship between the privacy budget and the other parameters, i.e., the noise multiplier, steps and the sampling rate. This relationship motivates a way of scaling the batchsize and the noise multiplier without affecting the privacy accountant.

Weakness:
1. There is no application of using TAN relation in language task, either pretraining or fine-tuning.

2. The reference needs to cover more. Especially, Yu et al. 2021 starts the fine-tuning large language models with differential privacy and low-rank reparametrization.
Yu et al. 2021 Large Scale Private Learning via Low-rank Reparametrization.

The evidence to be evaluated is:DP-SGD (Abadi et al., 2016) is the most popular DP algorithm to train DNNs. It selects samples uniformly at random with probability \(q=B/N\) (Poisson sampling), clips per-sample gradients to a norm \(C\) (\(\text{clip}_{C}\)), aggregates them and adds (Gaussian) noise. With \(\theta\) the parameters of the DNN: \[g_{\text{noisy}}=\frac{1}{B}\sum_{i\in B}\text{clip}_{C}\left(\nabla_{\theta} \ell(\theta,(x_{i},y_{i}))\right)+\mathcal{N}\left(0,\frac{C^{2}\sigma^{2}}{B ^{2}}\right). \tag{2}\] The traditional privacy analysis of DP-SGD is obtained through Renyi DP.

**Definition 3** (Renyi DP).: _A randomized mechanism \(\mathcal{M}\colon\mathcal{D}\to\mathcal{R}\) satisfies \((\alpha,d_{\alpha})\)-Renyi differential privacy (RDP) if, for any \(D,D\in\mathcal{D}^{\prime}\)that differ by one sample, we have_ \[D_{\alpha}(\mathcal{M}(D)\parallel\mathcal{M}(D^{\prime}))\leq d_{\alpha}.\] Renyi DP is a convenient notion to track privacy because composition is additive: a sequence of two algorithms satisfying \((\alpha,d_{\alpha})\) and \((\alpha,d_{\alpha}^{\prime})\) RDP satisfies \((\alpha,d_{\alpha}+d_{\alpha}^{\prime})\) RDP. In particular, the succession of \(S\) steps of a \((\alpha,d_{\alpha})\) RDP mechanism satisfies \((\alpha,Sd_{\alpha})\) RDP. Mironov et al. (2019) show that each step of DP-SGD satisfies \((\alpha,g_{\alpha}(\sigma,q))\)-RDP with

Training for \(S\) steps with DP-SGD thus satisfies \((\varepsilon_{RDP},\delta)\)-DP with \[\varepsilon_{\mathrm{RDP}}:=\min_{\alpha}Sg_{\alpha}(\sigma,q)+\frac{\log(1/ \delta)}{\alpha-1}. \tag{3}\] RDP is the traditional tool used to analyse DP-SGD, but other accounting tools have been proposed to obtain tighter bounds (Gopi et al., 2021). In this work, we use the accountant due to Balle et al. (2020), whose output is referred to as \(\varepsilon\), which is slightly smaller than \(\varepsilon_{\mathrm{RDP}}\).

We introduce the notion of Total Amount of Noise (TAN) and discuss its connections to RDP accounting. We then demonstrate how training with reference privacy parameters \((q_{\mathrm{ref}},\sigma_{\mathrm{ref}},S)\) can be simulated with much lower computational resources using the same TAN with a smaller batch size. **Definition 4**.: _Let the individual signal-to-noise ratio \(\eta\) (and its inverse \(\Sigma\), the Total Amount of Noise or TAN) be as follows:_ \[\eta^{2}=\frac{1}{\Sigma^{2}}:=\frac{q^{2}S}{2\sigma^{2}}.\]

Our analysis goes further by analyzing how RDP accounting explains this dependency. Intuitively, we expect that the privacy budget \(\varepsilon\) only depends on the signal-to-noise ratio \(\eta\). In Figure 2, we plot \(\varepsilon\) as a function of \(\sigma\) and \(S\), at a _fixed_\(\eta\), and observe that \(\varepsilon\) is indeed constant, but only when \(\sigma>2\).

Replacing that in the definition of \(\varepsilon_{\mathrm{RDP}}\) (Equation 3), we get \[\varepsilon_{\mathrm{RDP}}\approx\eta^{2}+\min_{\alpha}\left((\alpha-1)\eta^ {2}+\frac{\log{(1/\delta)}}{\alpha-1}\right)=\eta^{2}+2\eta\sqrt{\log{(1/ \delta)}}=:\varepsilon_{\mathrm{TAN}}(\eta). \tag{4}\] We verify this relationship empirically, and in particular choose \(\eta\) to get a desired \(\varepsilon_{\mathrm{TAN}}\) in Figure 2.

Having this simple approximation for \(\varepsilon\) is useful in practice because it allows for simple mental gymnastics: for instance, doubling the sampling rate \(q\) while dividing the number of steps \(S\) by \(4\) should leave the privacy budget constant, which we observe empirically.

This drastically improves privacy accounting (Figure 2), and the per step signal-to-noise ratio \(\eta_{\mathrm{step}}\) remains constant.

If \(\sigma<2\), simultaneously doubling \(q\) and \(\sigma\) has a small or negligible negative impact on accuracy (Figure 4) but it can greatly reduce the privacy budget (Figure 2).

This explains why state-of-the-art approaches heuristically find that mega-batches work well: a blind grid search on the batch size and the noise level at constant privacy budget is likely to discover batches large enough to have \(\sigma>2\). Our analysis gives a principled explanation for the sweet spot of \(\sigma\in[2,4]\) used by most state-of-the-art approaches (De et al., 2022; Li et al., 2021).

In Section 4.2.1, we use our simulated training with constant TAN to find optimal hyperparameters at low compute and improve performance for reference privacy parameters \((B_{\mathrm{ref}},\sigma_{\mathrm{ref}},S)=(16384,2.5,72\mathrm{k})\). In Section 4.2.2, we find optimal privacy parameters at constant TAN by changing the number of steps \(S\), leading to a new state of the art on ImageNet for \(\varepsilon=8\) (Table 1).

While we kept \(S\) constant in previous experiments, we now explore constant TAN triplets \((q,\sigma,S)\) by varying \(S\). We keep \(\sigma\) fixed to \(2.5\) and vary \((B,S)\) starting from the reference \((16384,72\text{K})\) at constant \(\eta=q^{2}S/(2\sigma^{2})\). Given that \(\sigma>2\), we stay in the _almost_ constant privacy regime (see Figure 2): we indeed observe \(\varepsilon\approx\varepsilon_{\mathrm{TAN}}\) in Table 4.

Thus, we have shown how we can use TAN to perform optimal privacy parameter search while simulating each choice of optimal parameters at a much smaller cost. We now illustrate the benefit of TAN for ablation analysis.

Therefore, using TAN through our scaling law can facilitate studying variants of the network architecture while reducing the computational costs.

If we multiply by \(\beta\) the quantity of data \(N_{0}\) and keep the same \((B,\sigma,S)\), \(q\) (and thus \(\eta\)), is divided by \(\beta\) as well.

On the one hand, when using \(\varepsilon_{\mathrm{TAN}}\), we can predict the impact on the privacy budget.

We argue that the total amount of noise (TAN) is a simple but useful guiding principle to experiment with private training. In particular, we demonstrate that the privacy budget is either a direct function of TAN or can be reduced.

We believe that the important increase in the privacy budget \(\varepsilon\) as the noise level \(\sigma\) decreases is a real phenomenon and not an artifact of the analysis. Indeed, DP assumes that the adversary has access to all model updates, as is the case for example in Federated Learning. In such cases, a noise level that is too low is insufficient to hide the presence of individual data points and makes it impossible to obtain reasonable privacy guarantees.

Using TAN allowed us to explore and compare different architectures and optimizers, which would have been computationally impossible in the normal DP training setting at \(B=16384\).

Input: Table 1: Top-1 test accuracy when training on ImageNet from scratch using DP-SGD. We train a NF-ResNet-50 with σ = 2.5, hyper-parameters of Table 2 and (B,S) = (32768, 18k) (Table 4).

Input: Table 4: Privacy parameter search. We use the optimal parameters described in Section 4.2.1 with σ = 2.5 for one expensive run and compare it with our optimal result

Input: Figure 2: Privacy budget ε as a function of the noise level σ with η constant. On both figures, each curve corresponds to a different number of steps S, and each point on the curve is computed at a sampling rate q such that η is constant. On the left, we use η = 0.13 (resulting in εTAN = 1 in Equation 4). On the right, we use η = 0.95 (εTAN = 8). We observe a “privacy wall” imposing σ ≥ 0.5 for meaningful level of privacy budget ε, and σ ≥ 2 for constant ε ≈ εTAN.

Input: Figure 4: Test accuracies at constant ηstep = Bref/( √ 2Nσref ) and S are (log) linearly decreasing with B. Dashed lines are computed using a linear regression on the crosses. Shaded areas correspond to 3 std confidence intervals. (Left) CIFAR-10 with 16-4-WideResNet for Sref steps. Each curve corresponds to a different value of ηstep. (Right) ImageNet with NF-ResNet-50 with various numbers of steps. The scaling law holds for various training configurations.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
1. Summary of evaluation:
The evidence strongly supports the claim through both theoretical derivations and empirical validation. The paper demonstrates that the privacy budget ε is indeed primarily dependent on TAN (η) when σ > 2, as shown in Figure 2 where ε remains constant at fixed η. The theoretical relationship is derived through RDP analysis (Equation 4) and validated experimentally. While there is a limitation that this relationship only holds when σ > 2, the authors clearly acknowledge this constraint. The claim is precise and accurately reflects what the evidence demonstrates without overstatement.

<score>1</score>

Justification:
- The claim is well-stated and carefully scoped with "when not overcharged" (referring to σ > 2 condition)
- Strong theoretical foundation through RDP analysis (Equations 3-4)
- Clear empirical validation through Figure 2 showing constant ε at fixed η when σ > 2
- Practical validation through parameter scaling experiments in Tables 1 & 4
- The reviewer's comments are generally positive about this relationship
- Authors acknowledge limitations and constraints appropriately
- No exaggeration or overreaching conclusions beyond what evidence supports


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We first use the tools of Renyi Differential Privacy (RDP) to show that the privacy budget, when not overcharged, only depends on the total amount of noise (TAN) injected throughout training.

The review comment to be evaluated is:

Major strengths:
1. The observation of the scaling law is new.
2. The results of hyperparameter tuning improve the state-of-the-art greatly.

Minor strengths:
1. This paper is easy to follow.
2. The visualization of the experimental results is good.
3. Figure 2 seems to deliver the same message as the linear part of Figure 3 (left) in (Li et al, 2021a), but the authors provide more explanations in Section 3.2.

Major weaknesses:
1. The privacy guarantee parameter $\varepsilon_{TAN}$ is an approximation of $\varepsilon_{RDP}$ and it is only a lower bound. For strict privacy guarantees, we need an upper bound.
2. There is no explanation for the reason for the scaling law.

Minor weaknesses:
Some of the statements in this paper are inconsistent with each other.
1. In the left figure of Figure 4, the legend shows that $\sigma_{ref}$ is constant for each line, but the caption says the $\eta_{step}$ is constant. I also wonder why the authors choose to fix different values in Figure 4 for CIFAR-10 and ImageNet. Is it because the results of CIFAR-10 do not look as linear as of ImageNet?
2. In Page 5, Choice of $\sigma$, the authors claimed that if $\sigma>4$, dividing it by 2 with halving q is likely to improve performance, but I cannot find experimental results supporting this.
3. In Page 7, the authors say that they decide not to use the gain from 'testing with augmentations' in Table 1 and Table 4. In Table 2, I guess the Total column is the sum of the other columns for B=128, ..., 1024. Therefore, the last row, B=16384, also has the AugTest (+0.8) in the Total result (6.7%). I guess in Table 4, the test ACC for B=16384, 36.9%, is derived by adding the total improvement 6.7% to the baseline 30.2%. Therefore, I am not sure if the authors forget to remove the improvement from AugTest.
4. In Page 9, in the last paragraph, the authors believe that the exponential increase in the privacy budget $\varepsilon$ as the noise level $\sigma$ decreases. I guess this is from Figure 2, but the x-axis for $\sigma$ is log-scale. Therefore, it is not easy to identify whether there is a linear or exponential relationship between $\varepsilon$ and $\sigma$.

By the way, there are duplicate references (Li et al, 2021a and 2021b).

The evidence to be evaluated is:DP-SGD (Abadi et al., 2016) is the most popular DP algorithm to train DNNs. It selects samples uniformly at random with probability \(q=B/N\) (Poisson sampling), clips per-sample gradients to a norm \(C\) (\(\text{clip}_{C}\)), aggregates them and adds (Gaussian) noise. With \(\theta\) the parameters of the DNN: \[g_{\text{noisy}}=\frac{1}{B}\sum_{i\in B}\text{clip}_{C}\left(\nabla_{\theta} \ell(\theta,(x_{i},y_{i}))\right)+\mathcal{N}\left(0,\frac{C^{2}\sigma^{2}}{B ^{2}}\right). \tag{2}\] The traditional privacy analysis of DP-SGD is obtained through Renyi DP.

**Definition 3** (Renyi DP).: _A randomized mechanism \(\mathcal{M}\colon\mathcal{D}\to\mathcal{R}\) satisfies \((\alpha,d_{\alpha})\)-Renyi differential privacy (RDP) if, for any \(D,D\in\mathcal{D}^{\prime}\)that differ by one sample, we have_ \[D_{\alpha}(\mathcal{M}(D)\parallel\mathcal{M}(D^{\prime}))\leq d_{\alpha}.\] Renyi DP is a convenient notion to track privacy because composition is additive: a sequence of two algorithms satisfying \((\alpha,d_{\alpha})\) and \((\alpha,d_{\alpha}^{\prime})\) RDP satisfies \((\alpha,d_{\alpha}+d_{\alpha}^{\prime})\) RDP. In particular, the succession of \(S\) steps of a \((\alpha,d_{\alpha})\) RDP mechanism satisfies \((\alpha,Sd_{\alpha})\) RDP. Mironov et al. (2019) show that each step of DP-SGD satisfies \((\alpha,g_{\alpha}(\sigma,q))\)-RDP with

Training for \(S\) steps with DP-SGD thus satisfies \((\varepsilon_{RDP},\delta)\)-DP with \[\varepsilon_{\mathrm{RDP}}:=\min_{\alpha}Sg_{\alpha}(\sigma,q)+\frac{\log(1/ \delta)}{\alpha-1}. \tag{3}\] RDP is the traditional tool used to analyse DP-SGD, but other accounting tools have been proposed to obtain tighter bounds (Gopi et al., 2021). In this work, we use the accountant due to Balle et al. (2020), whose output is referred to as \(\varepsilon\), which is slightly smaller than \(\varepsilon_{\mathrm{RDP}}\).

We introduce the notion of Total Amount of Noise (TAN) and discuss its connections to RDP accounting. We then demonstrate how training with reference privacy parameters \((q_{\mathrm{ref}},\sigma_{\mathrm{ref}},S)\) can be simulated with much lower computational resources using the same TAN with a smaller batch size. **Definition 4**.: _Let the individual signal-to-noise ratio \(\eta\) (and its inverse \(\Sigma\), the Total Amount of Noise or TAN) be as follows:_ \[\eta^{2}=\frac{1}{\Sigma^{2}}:=\frac{q^{2}S}{2\sigma^{2}}.\]

Our analysis goes further by analyzing how RDP accounting explains this dependency. Intuitively, we expect that the privacy budget \(\varepsilon\) only depends on the signal-to-noise ratio \(\eta\). In Figure 2, we plot \(\varepsilon\) as a function of \(\sigma\) and \(S\), at a _fixed_\(\eta\), and observe that \(\varepsilon\) is indeed constant, but only when \(\sigma>2\).

Replacing that in the definition of \(\varepsilon_{\mathrm{RDP}}\) (Equation 3), we get \[\varepsilon_{\mathrm{RDP}}\approx\eta^{2}+\min_{\alpha}\left((\alpha-1)\eta^ {2}+\frac{\log{(1/\delta)}}{\alpha-1}\right)=\eta^{2}+2\eta\sqrt{\log{(1/ \delta)}}=:\varepsilon_{\mathrm{TAN}}(\eta). \tag{4}\] We verify this relationship empirically, and in particular choose \(\eta\) to get a desired \(\varepsilon_{\mathrm{TAN}}\) in Figure 2.

Having this simple approximation for \(\varepsilon\) is useful in practice because it allows for simple mental gymnastics: for instance, doubling the sampling rate \(q\) while dividing the number of steps \(S\) by \(4\) should leave the privacy budget constant, which we observe empirically.

This drastically improves privacy accounting (Figure 2), and the per step signal-to-noise ratio \(\eta_{\mathrm{step}}\) remains constant.

If \(\sigma<2\), simultaneously doubling \(q\) and \(\sigma\) has a small or negligible negative impact on accuracy (Figure 4) but it can greatly reduce the privacy budget (Figure 2).

This explains why state-of-the-art approaches heuristically find that mega-batches work well: a blind grid search on the batch size and the noise level at constant privacy budget is likely to discover batches large enough to have \(\sigma>2\). Our analysis gives a principled explanation for the sweet spot of \(\sigma\in[2,4]\) used by most state-of-the-art approaches (De et al., 2022; Li et al., 2021).

In Section 4.2.1, we use our simulated training with constant TAN to find optimal hyperparameters at low compute and improve performance for reference privacy parameters \((B_{\mathrm{ref}},\sigma_{\mathrm{ref}},S)=(16384,2.5,72\mathrm{k})\). In Section 4.2.2, we find optimal privacy parameters at constant TAN by changing the number of steps \(S\), leading to a new state of the art on ImageNet for \(\varepsilon=8\) (Table 1).

While we kept \(S\) constant in previous experiments, we now explore constant TAN triplets \((q,\sigma,S)\) by varying \(S\). We keep \(\sigma\) fixed to \(2.5\) and vary \((B,S)\) starting from the reference \((16384,72\text{K})\) at constant \(\eta=q^{2}S/(2\sigma^{2})\). Given that \(\sigma>2\), we stay in the _almost_ constant privacy regime (see Figure 2): we indeed observe \(\varepsilon\approx\varepsilon_{\mathrm{TAN}}\) in Table 4.

Thus, we have shown how we can use TAN to perform optimal privacy parameter search while simulating each choice of optimal parameters at a much smaller cost. We now illustrate the benefit of TAN for ablation analysis.

Therefore, using TAN through our scaling law can facilitate studying variants of the network architecture while reducing the computational costs.

If we multiply by \(\beta\) the quantity of data \(N_{0}\) and keep the same \((B,\sigma,S)\), \(q\) (and thus \(\eta\)), is divided by \(\beta\) as well.

On the one hand, when using \(\varepsilon_{\mathrm{TAN}}\), we can predict the impact on the privacy budget.

We argue that the total amount of noise (TAN) is a simple but useful guiding principle to experiment with private training. In particular, we demonstrate that the privacy budget is either a direct function of TAN or can be reduced.

We believe that the important increase in the privacy budget \(\varepsilon\) as the noise level \(\sigma\) decreases is a real phenomenon and not an artifact of the analysis. Indeed, DP assumes that the adversary has access to all model updates, as is the case for example in Federated Learning. In such cases, a noise level that is too low is insufficient to hide the presence of individual data points and makes it impossible to obtain reasonable privacy guarantees.

Using TAN allowed us to explore and compare different architectures and optimizers, which would have been computationally impossible in the normal DP training setting at \(B=16384\).

Input: Table 1: Top-1 test accuracy when training on ImageNet from scratch using DP-SGD. We train a NF-ResNet-50 with σ = 2.5, hyper-parameters of Table 2 and (B,S) = (32768, 18k) (Table 4).

Input: Table 4: Privacy parameter search. We use the optimal parameters described in Section 4.2.1 with σ = 2.5 for one expensive run and compare it with our optimal result

Input: Figure 2: Privacy budget ε as a function of the noise level σ with η constant. On both figures, each curve corresponds to a different number of steps S, and each point on the curve is computed at a sampling rate q such that η is constant. On the left, we use η = 0.13 (resulting in εTAN = 1 in Equation 4). On the right, we use η = 0.95 (εTAN = 8). We observe a “privacy wall” imposing σ ≥ 0.5 for meaningful level of privacy budget ε, and σ ≥ 2 for constant ε ≈ εTAN.

Input: Figure 4: Test accuracies at constant ηstep = Bref/( √ 2Nσref ) and S are (log) linearly decreasing with B. Dashed lines are computed using a linear regression on the crosses. Shaded areas correspond to 3 std confidence intervals. (Left) CIFAR-10 with 16-4-WideResNet for Sref steps. Each curve corresponds to a different value of ηstep. (Right) ImageNet with NF-ResNet-50 with various numbers of steps. The scaling law holds for various training configurations.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll assess the claim: "We first use the tools of Renyi Differential Privacy (RDP) to show that the privacy budget, when not overcharged, only depends on the total amount of noise (TAN) injected throughout training."

Summary:
The evidence partially supports this claim but with important caveats. While Figure 2 shows that the privacy budget ε remains approximately constant when σ > 2 for fixed η (TAN), the relationship breaks down when σ < 2. The review comment correctly points out that εTAN is only a lower bound approximation of εRDP. The paper's authors acknowledge this limitation later by using "almost constant privacy regime." The claim would be more accurate if it specified these conditions and limitations.

<score>3</score>

Justification:
1. The claim overstates the universality of the relationship between privacy budget and TAN by not mentioning crucial conditions (σ > 2)
2. The review highlights that εTAN is only an approximation and lower bound
3. Figure 2 clearly shows the relationship isn't universal but holds only under specific conditions
4. The authors themselves use more measured language later in the paper
5. The evidence shows it's a useful approximation but not an absolute dependency as suggested in the claim


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We first use the tools of Renyi Differential Privacy (RDP) to show that the privacy budget, when not overcharged, only depends on the total amount of noise (TAN) injected throughout training.

The review comment to be evaluated is:

Strengths:
1) Integrating the constraints of DP-SGD with optimization theory, in order to reliably predict good hyper-parameter choices, is one of the most important problems that needs to be solved in order to make differentially private deep learning practical. I'm pleased to see the authors tackle this difficult and important problem.
2) The metric proposed by the authors is intuitive and it appears to provide some helpful intuition.
3) The authors achieve strong performance when training from scratch on ImageNet with DP-SGD.

Weaknesses:
1) If TAN were an accurate measure of optimizability, then we should expect the train accuracy to be roughly constant at fixed TAN. However in practice, the authors find in figure 1 that the test accuracy decays log-linearly at fixed TAN as the batch size rises. I feel that the current paper is quite confusing because this discrepancy is not explicitly stated anywhere. I'd encourage the authors to make this explicit, and to explore in more detail why this discrepancy arises (eg comparing test vs train performance).

2) As discussed above, Figure 1 shows that TAN over-estimates the performance of large batch sizes. It would be nice to also see a figure at fixed batch size, sweeping the number of steps S, to see whether TAN under/over-estimates the performance of small/large step budgets.

3) As a minor point, note that on some truly sensitive data we cannot train non-private models (even internally). It would be good to acknowledge that the hyper-parameter transfer process described here cannot be used in these cases.

4) For many of the experiments, key hyper-parameters are lifted from prior work (eg step budget, target batch size). Is it not possible to directly infer good settings for these hyper-parameters using the TAN framework?

5) The paper essentially only considers ImageNet (with some very limited experiments on CIFAR-10). Since very few groups have run experiments at this scale it is not clear how challenging the baseline from De et al. is in practice.

The evidence to be evaluated is:DP-SGD (Abadi et al., 2016) is the most popular DP algorithm to train DNNs. It selects samples uniformly at random with probability \(q=B/N\) (Poisson sampling), clips per-sample gradients to a norm \(C\) (\(\text{clip}_{C}\)), aggregates them and adds (Gaussian) noise. With \(\theta\) the parameters of the DNN: \[g_{\text{noisy}}=\frac{1}{B}\sum_{i\in B}\text{clip}_{C}\left(\nabla_{\theta} \ell(\theta,(x_{i},y_{i}))\right)+\mathcal{N}\left(0,\frac{C^{2}\sigma^{2}}{B ^{2}}\right). \tag{2}\] The traditional privacy analysis of DP-SGD is obtained through Renyi DP.

**Definition 3** (Renyi DP).: _A randomized mechanism \(\mathcal{M}\colon\mathcal{D}\to\mathcal{R}\) satisfies \((\alpha,d_{\alpha})\)-Renyi differential privacy (RDP) if, for any \(D,D\in\mathcal{D}^{\prime}\)that differ by one sample, we have_ \[D_{\alpha}(\mathcal{M}(D)\parallel\mathcal{M}(D^{\prime}))\leq d_{\alpha}.\] Renyi DP is a convenient notion to track privacy because composition is additive: a sequence of two algorithms satisfying \((\alpha,d_{\alpha})\) and \((\alpha,d_{\alpha}^{\prime})\) RDP satisfies \((\alpha,d_{\alpha}+d_{\alpha}^{\prime})\) RDP. In particular, the succession of \(S\) steps of a \((\alpha,d_{\alpha})\) RDP mechanism satisfies \((\alpha,Sd_{\alpha})\) RDP. Mironov et al. (2019) show that each step of DP-SGD satisfies \((\alpha,g_{\alpha}(\sigma,q))\)-RDP with

Training for \(S\) steps with DP-SGD thus satisfies \((\varepsilon_{RDP},\delta)\)-DP with \[\varepsilon_{\mathrm{RDP}}:=\min_{\alpha}Sg_{\alpha}(\sigma,q)+\frac{\log(1/ \delta)}{\alpha-1}. \tag{3}\] RDP is the traditional tool used to analyse DP-SGD, but other accounting tools have been proposed to obtain tighter bounds (Gopi et al., 2021). In this work, we use the accountant due to Balle et al. (2020), whose output is referred to as \(\varepsilon\), which is slightly smaller than \(\varepsilon_{\mathrm{RDP}}\).

We introduce the notion of Total Amount of Noise (TAN) and discuss its connections to RDP accounting. We then demonstrate how training with reference privacy parameters \((q_{\mathrm{ref}},\sigma_{\mathrm{ref}},S)\) can be simulated with much lower computational resources using the same TAN with a smaller batch size. **Definition 4**.: _Let the individual signal-to-noise ratio \(\eta\) (and its inverse \(\Sigma\), the Total Amount of Noise or TAN) be as follows:_ \[\eta^{2}=\frac{1}{\Sigma^{2}}:=\frac{q^{2}S}{2\sigma^{2}}.\]

Our analysis goes further by analyzing how RDP accounting explains this dependency. Intuitively, we expect that the privacy budget \(\varepsilon\) only depends on the signal-to-noise ratio \(\eta\). In Figure 2, we plot \(\varepsilon\) as a function of \(\sigma\) and \(S\), at a _fixed_\(\eta\), and observe that \(\varepsilon\) is indeed constant, but only when \(\sigma>2\).

Replacing that in the definition of \(\varepsilon_{\mathrm{RDP}}\) (Equation 3), we get \[\varepsilon_{\mathrm{RDP}}\approx\eta^{2}+\min_{\alpha}\left((\alpha-1)\eta^ {2}+\frac{\log{(1/\delta)}}{\alpha-1}\right)=\eta^{2}+2\eta\sqrt{\log{(1/ \delta)}}=:\varepsilon_{\mathrm{TAN}}(\eta). \tag{4}\] We verify this relationship empirically, and in particular choose \(\eta\) to get a desired \(\varepsilon_{\mathrm{TAN}}\) in Figure 2.

Having this simple approximation for \(\varepsilon\) is useful in practice because it allows for simple mental gymnastics: for instance, doubling the sampling rate \(q\) while dividing the number of steps \(S\) by \(4\) should leave the privacy budget constant, which we observe empirically.

This drastically improves privacy accounting (Figure 2), and the per step signal-to-noise ratio \(\eta_{\mathrm{step}}\) remains constant.

If \(\sigma<2\), simultaneously doubling \(q\) and \(\sigma\) has a small or negligible negative impact on accuracy (Figure 4) but it can greatly reduce the privacy budget (Figure 2).

This explains why state-of-the-art approaches heuristically find that mega-batches work well: a blind grid search on the batch size and the noise level at constant privacy budget is likely to discover batches large enough to have \(\sigma>2\). Our analysis gives a principled explanation for the sweet spot of \(\sigma\in[2,4]\) used by most state-of-the-art approaches (De et al., 2022; Li et al., 2021).

In Section 4.2.1, we use our simulated training with constant TAN to find optimal hyperparameters at low compute and improve performance for reference privacy parameters \((B_{\mathrm{ref}},\sigma_{\mathrm{ref}},S)=(16384,2.5,72\mathrm{k})\). In Section 4.2.2, we find optimal privacy parameters at constant TAN by changing the number of steps \(S\), leading to a new state of the art on ImageNet for \(\varepsilon=8\) (Table 1).

While we kept \(S\) constant in previous experiments, we now explore constant TAN triplets \((q,\sigma,S)\) by varying \(S\). We keep \(\sigma\) fixed to \(2.5\) and vary \((B,S)\) starting from the reference \((16384,72\text{K})\) at constant \(\eta=q^{2}S/(2\sigma^{2})\). Given that \(\sigma>2\), we stay in the _almost_ constant privacy regime (see Figure 2): we indeed observe \(\varepsilon\approx\varepsilon_{\mathrm{TAN}}\) in Table 4.

Thus, we have shown how we can use TAN to perform optimal privacy parameter search while simulating each choice of optimal parameters at a much smaller cost. We now illustrate the benefit of TAN for ablation analysis.

Therefore, using TAN through our scaling law can facilitate studying variants of the network architecture while reducing the computational costs.

If we multiply by \(\beta\) the quantity of data \(N_{0}\) and keep the same \((B,\sigma,S)\), \(q\) (and thus \(\eta\)), is divided by \(\beta\) as well.

On the one hand, when using \(\varepsilon_{\mathrm{TAN}}\), we can predict the impact on the privacy budget.

We argue that the total amount of noise (TAN) is a simple but useful guiding principle to experiment with private training. In particular, we demonstrate that the privacy budget is either a direct function of TAN or can be reduced.

We believe that the important increase in the privacy budget \(\varepsilon\) as the noise level \(\sigma\) decreases is a real phenomenon and not an artifact of the analysis. Indeed, DP assumes that the adversary has access to all model updates, as is the case for example in Federated Learning. In such cases, a noise level that is too low is insufficient to hide the presence of individual data points and makes it impossible to obtain reasonable privacy guarantees.

Using TAN allowed us to explore and compare different architectures and optimizers, which would have been computationally impossible in the normal DP training setting at \(B=16384\).

Input: Table 1: Top-1 test accuracy when training on ImageNet from scratch using DP-SGD. We train a NF-ResNet-50 with σ = 2.5, hyper-parameters of Table 2 and (B,S) = (32768, 18k) (Table 4).

Input: Table 4: Privacy parameter search. We use the optimal parameters described in Section 4.2.1 with σ = 2.5 for one expensive run and compare it with our optimal result

Input: Figure 2: Privacy budget ε as a function of the noise level σ with η constant. On both figures, each curve corresponds to a different number of steps S, and each point on the curve is computed at a sampling rate q such that η is constant. On the left, we use η = 0.13 (resulting in εTAN = 1 in Equation 4). On the right, we use η = 0.95 (εTAN = 8). We observe a “privacy wall” imposing σ ≥ 0.5 for meaningful level of privacy budget ε, and σ ≥ 2 for constant ε ≈ εTAN.

Input: Figure 4: Test accuracies at constant ηstep = Bref/( √ 2Nσref ) and S are (log) linearly decreasing with B. Dashed lines are computed using a linear regression on the crosses. Shaded areas correspond to 3 std confidence intervals. (Left) CIFAR-10 with 16-4-WideResNet for Sref steps. Each curve corresponds to a different value of ηstep. (Right) ImageNet with NF-ResNet-50 with various numbers of steps. The scaling law holds for various training configurations.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on the evidence provided, I'll evaluate the claim: "We first use the tools of Renyi Differential Privacy (RDP) to show that the privacy budget, when not overcharged, only depends on the total amount of noise (TAN) injected throughout training."

The evidence partially supports this claim, but it appears to be overstated. While Figure 2 shows that ε (privacy budget) becomes approximately constant with TAN when σ > 2, this relationship doesn't hold for all conditions. The authors themselves acknowledge limitations, stating "we expect that the privacy budget ε only depends on the signal-to-noise ratio η" but then showing this is true only under specific conditions (σ > 2). The review comments also point out that TAN over-estimates performance with large batch sizes.

<score>4</score>

Justification:
1. The paper demonstrates the relationship between privacy budget and TAN, but only under specific conditions (σ > 2)
2. The claim doesn't acknowledge these important limitations
3. Figure 2 shows clear variation in privacy budget when σ < 2, contradicting the blanket statement
4. The reviewer's comments highlight that TAN's predictive power has limitations, particularly with large batch sizes
5. The paper itself uses more qualified language later, suggesting the initial claim is overstated


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We then use TAN to find optimal privacy parameters, leading to a gain of \(+9\) points under \(\varepsilon=8\) compared to the previous SOTA;

The evidence to be evaluated is:We introduce the notion of Total Amount of Noise (TAN) and discuss its connections to RDP accounting. We then demonstrate how training with reference privacy parameters \((q_{\mathrm{ref}},\sigma_{\mathrm{ref}},S)\) can be simulated with much lower computational resources using the same TAN with a smaller batch size. **Definition 4**.: _Let the individual signal-to-noise ratio \(\eta\) (and its inverse \(\Sigma\), the Total Amount of Noise or TAN) be as follows:_ \[\eta^{2}=\frac{1}{\Sigma^{2}}:=\frac{q^{2}S}{2\sigma^{2}}.\] We begin with a simple case to motivate our definition of TAN. We assume a one-dimensional model, where the gradients of all points are clipped to \(C\). Looking at Equation 2, in one batch, the expected signal from each sample is \(C/B\) with probability \(q=B/N\) and \(0\) otherwise. Therefore, the expected individual signal of each sample after \(S\) steps is \(SC/N\), and its squared norm is \(S^{2}C^{2}/N^{2}\).

\[\varepsilon_{\mathrm{RDP}}\approx\eta^{2}+\min_{\alpha}\left((\alpha-1)\eta^ {2}+\frac{\log{(1/\delta)}}{\alpha-1}\right)=\eta^{2}+2\eta\sqrt{\log{(1/ \delta)}}=:\varepsilon_{\mathrm{TAN}}(\eta). \tag{4}\] We verify this relationship empirically, and in particular choose \(\eta\) to get a desired \(\varepsilon_{\mathrm{TAN}}\) in Figure 2.

We leverage our efficient TAN training strategy and obtain new state-of-the-art results on ImageNet for \(\varepsilon=8\) (Table 1).

In Section 4.2.1, we use our simulated training with constant TAN to find optimal hyperparameters at low compute and improve performance for reference privacy parameters \((B_{\mathrm{ref}},\sigma_{\mathrm{ref}},S)=(16384,2.5,72\mathrm{k})\). In Section 4.2.2, we find optimal privacy parameters at constant TAN by changing the number of steps \(S\), leading to a new state of the art on ImageNet for \(\varepsilon=8\) (Table 1).

We perform one run with these optimal parameters at \(B=16384\) which satisfies a privacy budget of \(\varepsilon=8\).

While we kept \(S\) constant in previous experiments, we now explore constant TAN triplets \((q,\sigma,S)\) by varying \(S\). We keep \(\sigma\) fixed to \(2.5\) and vary \((B,S)\) starting from the reference \((16384,72\text{K})\) at constant \(\eta=q^{2}S/(2\sigma^{2})\). Given that \(\sigma>2\), we stay in the _almost_ constant privacy regime (see Figure 2): we indeed observe \(\varepsilon\approx\varepsilon_{\mathrm{TAN}}\) in Table 4.

Overall, we improved performance by \(9\%\) when training from scratch on ImageNet with DP-SGD under \(\varepsilon=8\). We compare to our reproduction of the previous SOTA of De et al. (2022) at \(30.2\%\) (compared to the results reported in the original paper (\(32.4\%\)), we still gain \(7\%\) of accuracy). Thus, we have shown how we can use TAN to perform optimal privacy parameter search while simulating each choice of optimal parameters at a much smaller cost.

Therefore, using TAN through our scaling law can facilitate studying variants of the network architecture while reducing the computational costs.

We now look at how collecting more data affects the tradeoff between privacy and utility. We show that doubling the data (from the same distribution) allows better performance with half the privacy budget. To this end, we train on portions of the CIFAR-10 training set (\(N=50\)k) and always report accuracies on the same test set. If we multiply by \(\beta\) the quantity of data \(N_{0}\) and keep the same \((B,\sigma,S)\), \(q\) (and thus \(\eta\)), is divided by \(\beta\) as well.

We argue that the total amount of noise (TAN) is a simple but useful guiding principle to experiment with private training. In particular, we demonstrate that the privacy budget is either a direct function of TAN or can be reduced. We further show that scaling batch size with noise level using TAN allows for ultra-efficient hyper-parameter search and demonstrate the power of this paradigm by establishing a new state of the art for DP training on Imagenet.

Using TAN allowed us to explore and compare different architectures and optimizers, which would have been computationally impossible in the normal DP training setting at \(B=16384\).

Input: Table 1: Top-1 test accuracy when training on ImageNet from scratch using DP-SGD. We train a NF-ResNet-50 with σ = 2.5, hyper-parameters of Table 2 and (B,S) = (32768, 18k) (Table 4).

Input: Table 4: Privacy parameter search. We use the optimal parameters described in Section 4.2.1 with σ = 2.5 for one expensive run and compare it with our optimal result

Input: Figure 2: Privacy budget ε as a function of the noise level σ with η constant. On both figures, each curve corresponds to a different number of steps S, and each point on the curve is computed at a sampling rate q such that η is constant. On the left, we use η = 0.13 (resulting in εTAN = 1 in Equation 4). On the right, we use η = 0.95 (εTAN = 8). We observe a “privacy wall” imposing σ ≥ 0.5 for meaningful level of privacy budget ε, and σ ≥ 2 for constant ε ≈ εTAN.

Input: Figure 4: Test accuracies at constant ηstep = Bref/( √ 2Nσref ) and S are (log) linearly decreasing with B. Dashed lines are computed using a linear regression on the crosses. Shaded areas correspond to 3 std confidence intervals. (Left) CIFAR-10 with 16-4-WideResNet for Sref steps. Each curve corresponds to a different value of ηstep. (Right) ImageNet with NF-ResNet-50 with various numbers of steps. The scaling law holds for various training configurations.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After careful review of the evidence, the claim "We then use TAN to find optimal privacy parameters, leading to a gain of +9 points under ε=8 compared to the previous SOTA" appears to be slightly overstated. While Table 1 shows their method achieved 39.2% accuracy compared to their reproduction of De et al.'s 30.2% (a 9% improvement), the original De et al. paper reported 32.4% (making it a 6.8% gain). The authors acknowledge this discrepancy in the evidence, stating "compared to the results reported in the original paper (32.4%), we still gain 7% of accuracy." The claim would be more precise if it clarified this distinction.

<score>2</score>

Justification:
1. The claim is largely supported by data in Table 1 showing their 39.2% vs 30.2% reproduction (9% gain)
2. However, the original SOTA was 32.4%, making it a 6.8-7% improvement
3. The evidence shows thorough experimental validation of the TAN method and its benefits
4. The claim could be more precise by acknowledging the difference between their reproduction and the original SOTA results
5. The underlying improvement is still significant and well-documented, making this only a minor overstatement


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We then use TAN to find optimal privacy parameters, leading to a gain of \(+9\) points under \(\varepsilon=8\) compared to the previous SOTA;

The review comment to be evaluated is:

Strength:

* The observations (batch scaling law, privacy budget computation) in the paper are new and interesting.

* The proposed hyper-parameter tuning process can be very useful in practice, as it saves the computation requirement and therefore speeds up the development process.

Weaknesses:

* Overall, the writing is clear, but there are still some issues that need to be addressed:
    - Equation 2: many of the notations here are not defined, e.g., clip_C, l, theta, C.
    - Figure 4: B_ref, S_ref, \sigma_ref are not defined. I can guess their meaning from the context, but all notations should be defined explicitly in the text.
    - Page 5: it says "Simultaneously doubling the batch size and \sigma has a negligible or small impact on accuracy (Figure 4)". However, in Figure 4(b), the impact on the accuracy is not small. Indeed, this is claimed in the next sentence "Reciprocally, if \sigma > 4, dividing it by 2 simultaneously with halving q is likely to improve performance." These two sentences contradict each other.
    - Table 1: I guess "8,8." should be "8,"?

The evidence to be evaluated is:We introduce the notion of Total Amount of Noise (TAN) and discuss its connections to RDP accounting. We then demonstrate how training with reference privacy parameters \((q_{\mathrm{ref}},\sigma_{\mathrm{ref}},S)\) can be simulated with much lower computational resources using the same TAN with a smaller batch size. **Definition 4**.: _Let the individual signal-to-noise ratio \(\eta\) (and its inverse \(\Sigma\), the Total Amount of Noise or TAN) be as follows:_ \[\eta^{2}=\frac{1}{\Sigma^{2}}:=\frac{q^{2}S}{2\sigma^{2}}.\] We begin with a simple case to motivate our definition of TAN. We assume a one-dimensional model, where the gradients of all points are clipped to \(C\). Looking at Equation 2, in one batch, the expected signal from each sample is \(C/B\) with probability \(q=B/N\) and \(0\) otherwise. Therefore, the expected individual signal of each sample after \(S\) steps is \(SC/N\), and its squared norm is \(S^{2}C^{2}/N^{2}\).

\[\varepsilon_{\mathrm{RDP}}\approx\eta^{2}+\min_{\alpha}\left((\alpha-1)\eta^ {2}+\frac{\log{(1/\delta)}}{\alpha-1}\right)=\eta^{2}+2\eta\sqrt{\log{(1/ \delta)}}=:\varepsilon_{\mathrm{TAN}}(\eta). \tag{4}\] We verify this relationship empirically, and in particular choose \(\eta\) to get a desired \(\varepsilon_{\mathrm{TAN}}\) in Figure 2.

We leverage our efficient TAN training strategy and obtain new state-of-the-art results on ImageNet for \(\varepsilon=8\) (Table 1).

In Section 4.2.1, we use our simulated training with constant TAN to find optimal hyperparameters at low compute and improve performance for reference privacy parameters \((B_{\mathrm{ref}},\sigma_{\mathrm{ref}},S)=(16384,2.5,72\mathrm{k})\). In Section 4.2.2, we find optimal privacy parameters at constant TAN by changing the number of steps \(S\), leading to a new state of the art on ImageNet for \(\varepsilon=8\) (Table 1).

We perform one run with these optimal parameters at \(B=16384\) which satisfies a privacy budget of \(\varepsilon=8\).

While we kept \(S\) constant in previous experiments, we now explore constant TAN triplets \((q,\sigma,S)\) by varying \(S\). We keep \(\sigma\) fixed to \(2.5\) and vary \((B,S)\) starting from the reference \((16384,72\text{K})\) at constant \(\eta=q^{2}S/(2\sigma^{2})\). Given that \(\sigma>2\), we stay in the _almost_ constant privacy regime (see Figure 2): we indeed observe \(\varepsilon\approx\varepsilon_{\mathrm{TAN}}\) in Table 4.

Overall, we improved performance by \(9\%\) when training from scratch on ImageNet with DP-SGD under \(\varepsilon=8\). We compare to our reproduction of the previous SOTA of De et al. (2022) at \(30.2\%\) (compared to the results reported in the original paper (\(32.4\%\)), we still gain \(7\%\) of accuracy). Thus, we have shown how we can use TAN to perform optimal privacy parameter search while simulating each choice of optimal parameters at a much smaller cost.

Therefore, using TAN through our scaling law can facilitate studying variants of the network architecture while reducing the computational costs.

We now look at how collecting more data affects the tradeoff between privacy and utility. We show that doubling the data (from the same distribution) allows better performance with half the privacy budget. To this end, we train on portions of the CIFAR-10 training set (\(N=50\)k) and always report accuracies on the same test set. If we multiply by \(\beta\) the quantity of data \(N_{0}\) and keep the same \((B,\sigma,S)\), \(q\) (and thus \(\eta\)), is divided by \(\beta\) as well.

We argue that the total amount of noise (TAN) is a simple but useful guiding principle to experiment with private training. In particular, we demonstrate that the privacy budget is either a direct function of TAN or can be reduced. We further show that scaling batch size with noise level using TAN allows for ultra-efficient hyper-parameter search and demonstrate the power of this paradigm by establishing a new state of the art for DP training on Imagenet.

Using TAN allowed us to explore and compare different architectures and optimizers, which would have been computationally impossible in the normal DP training setting at \(B=16384\).

Input: Table 1: Top-1 test accuracy when training on ImageNet from scratch using DP-SGD. We train a NF-ResNet-50 with σ = 2.5, hyper-parameters of Table 2 and (B,S) = (32768, 18k) (Table 4).

Input: Table 4: Privacy parameter search. We use the optimal parameters described in Section 4.2.1 with σ = 2.5 for one expensive run and compare it with our optimal result

Input: Figure 2: Privacy budget ε as a function of the noise level σ with η constant. On both figures, each curve corresponds to a different number of steps S, and each point on the curve is computed at a sampling rate q such that η is constant. On the left, we use η = 0.13 (resulting in εTAN = 1 in Equation 4). On the right, we use η = 0.95 (εTAN = 8). We observe a “privacy wall” imposing σ ≥ 0.5 for meaningful level of privacy budget ε, and σ ≥ 2 for constant ε ≈ εTAN.

Input: Figure 4: Test accuracies at constant ηstep = Bref/( √ 2Nσref ) and S are (log) linearly decreasing with B. Dashed lines are computed using a linear regression on the crosses. Shaded areas correspond to 3 std confidence intervals. (Left) CIFAR-10 with 16-4-WideResNet for Sref steps. Each curve corresponds to a different value of ηstep. (Right) ImageNet with NF-ResNet-50 with various numbers of steps. The scaling law holds for various training configurations.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After careful review, the claim "We then use TAN to find optimal privacy parameters, leading to a gain of +9 points under ε=8 compared to the previous SOTA" appears to be well-stated based on the evidence provided. Table 1 clearly shows their method achieved 39.2% accuracy compared to their reproduction of De et al.'s work at 30.2%, confirming a 9 percentage point improvement. The evidence demonstrates thorough experimental validation through multiple tables and figures, particularly Table 4 which shows the detailed privacy parameter search results. The gains are precisely quantified and properly contextualized against their reproduction of previous work.

<score>1</score>

Justification:
1. The claim is directly supported by Table 1, showing their method achieved 39.2% accuracy vs 30.2% for their reproduction of De et al.'s work
2. Table 4 provides detailed experimental validation showing how they achieved these results through privacy parameter optimization
3. The authors acknowledge and transparently report both the original paper's results (32.4%) and their reproduction (30.2%)
4. The claim avoids overgeneralization and specifically refers to their reproduction benchmark
5. The methodology and experimental process are well-documented through figures and tables that demonstrate how TAN was used to optimize the parameters

The reviewer comments don't dispute the accuracy claims or methodology, focusing instead on notation clarity issues that don't affect the validity of the performance improvements claimed.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We then use TAN to find optimal privacy parameters, leading to a gain of \(+9\) points under \(\varepsilon=8\) compared to the previous SOTA;

The review comment to be evaluated is:

Strength: It studies an important problem of how to scale DP-SGD with hyper-parameter search. The paper provides a neat and simple relationship between the privacy budget and the other parameters, i.e., the noise multiplier, steps and the sampling rate. This relationship motivates a way of scaling the batchsize and the noise multiplier without affecting the privacy accountant.

Weakness:
1. There is no application of using TAN relation in language task, either pretraining or fine-tuning.

2. The reference needs to cover more. Especially, Yu et al. 2021 starts the fine-tuning large language models with differential privacy and low-rank reparametrization.
Yu et al. 2021 Large Scale Private Learning via Low-rank Reparametrization.

The evidence to be evaluated is:We introduce the notion of Total Amount of Noise (TAN) and discuss its connections to RDP accounting. We then demonstrate how training with reference privacy parameters \((q_{\mathrm{ref}},\sigma_{\mathrm{ref}},S)\) can be simulated with much lower computational resources using the same TAN with a smaller batch size. **Definition 4**.: _Let the individual signal-to-noise ratio \(\eta\) (and its inverse \(\Sigma\), the Total Amount of Noise or TAN) be as follows:_ \[\eta^{2}=\frac{1}{\Sigma^{2}}:=\frac{q^{2}S}{2\sigma^{2}}.\] We begin with a simple case to motivate our definition of TAN. We assume a one-dimensional model, where the gradients of all points are clipped to \(C\). Looking at Equation 2, in one batch, the expected signal from each sample is \(C/B\) with probability \(q=B/N\) and \(0\) otherwise. Therefore, the expected individual signal of each sample after \(S\) steps is \(SC/N\), and its squared norm is \(S^{2}C^{2}/N^{2}\).

\[\varepsilon_{\mathrm{RDP}}\approx\eta^{2}+\min_{\alpha}\left((\alpha-1)\eta^ {2}+\frac{\log{(1/\delta)}}{\alpha-1}\right)=\eta^{2}+2\eta\sqrt{\log{(1/ \delta)}}=:\varepsilon_{\mathrm{TAN}}(\eta). \tag{4}\] We verify this relationship empirically, and in particular choose \(\eta\) to get a desired \(\varepsilon_{\mathrm{TAN}}\) in Figure 2.

We leverage our efficient TAN training strategy and obtain new state-of-the-art results on ImageNet for \(\varepsilon=8\) (Table 1).

In Section 4.2.1, we use our simulated training with constant TAN to find optimal hyperparameters at low compute and improve performance for reference privacy parameters \((B_{\mathrm{ref}},\sigma_{\mathrm{ref}},S)=(16384,2.5,72\mathrm{k})\). In Section 4.2.2, we find optimal privacy parameters at constant TAN by changing the number of steps \(S\), leading to a new state of the art on ImageNet for \(\varepsilon=8\) (Table 1).

We perform one run with these optimal parameters at \(B=16384\) which satisfies a privacy budget of \(\varepsilon=8\).

While we kept \(S\) constant in previous experiments, we now explore constant TAN triplets \((q,\sigma,S)\) by varying \(S\). We keep \(\sigma\) fixed to \(2.5\) and vary \((B,S)\) starting from the reference \((16384,72\text{K})\) at constant \(\eta=q^{2}S/(2\sigma^{2})\). Given that \(\sigma>2\), we stay in the _almost_ constant privacy regime (see Figure 2): we indeed observe \(\varepsilon\approx\varepsilon_{\mathrm{TAN}}\) in Table 4.

Overall, we improved performance by \(9\%\) when training from scratch on ImageNet with DP-SGD under \(\varepsilon=8\). We compare to our reproduction of the previous SOTA of De et al. (2022) at \(30.2\%\) (compared to the results reported in the original paper (\(32.4\%\)), we still gain \(7\%\) of accuracy). Thus, we have shown how we can use TAN to perform optimal privacy parameter search while simulating each choice of optimal parameters at a much smaller cost.

Therefore, using TAN through our scaling law can facilitate studying variants of the network architecture while reducing the computational costs.

We now look at how collecting more data affects the tradeoff between privacy and utility. We show that doubling the data (from the same distribution) allows better performance with half the privacy budget. To this end, we train on portions of the CIFAR-10 training set (\(N=50\)k) and always report accuracies on the same test set. If we multiply by \(\beta\) the quantity of data \(N_{0}\) and keep the same \((B,\sigma,S)\), \(q\) (and thus \(\eta\)), is divided by \(\beta\) as well.

We argue that the total amount of noise (TAN) is a simple but useful guiding principle to experiment with private training. In particular, we demonstrate that the privacy budget is either a direct function of TAN or can be reduced. We further show that scaling batch size with noise level using TAN allows for ultra-efficient hyper-parameter search and demonstrate the power of this paradigm by establishing a new state of the art for DP training on Imagenet.

Using TAN allowed us to explore and compare different architectures and optimizers, which would have been computationally impossible in the normal DP training setting at \(B=16384\).

Input: Table 1: Top-1 test accuracy when training on ImageNet from scratch using DP-SGD. We train a NF-ResNet-50 with σ = 2.5, hyper-parameters of Table 2 and (B,S) = (32768, 18k) (Table 4).

Input: Table 4: Privacy parameter search. We use the optimal parameters described in Section 4.2.1 with σ = 2.5 for one expensive run and compare it with our optimal result

Input: Figure 2: Privacy budget ε as a function of the noise level σ with η constant. On both figures, each curve corresponds to a different number of steps S, and each point on the curve is computed at a sampling rate q such that η is constant. On the left, we use η = 0.13 (resulting in εTAN = 1 in Equation 4). On the right, we use η = 0.95 (εTAN = 8). We observe a “privacy wall” imposing σ ≥ 0.5 for meaningful level of privacy budget ε, and σ ≥ 2 for constant ε ≈ εTAN.

Input: Figure 4: Test accuracies at constant ηstep = Bref/( √ 2Nσref ) and S are (log) linearly decreasing with B. Dashed lines are computed using a linear regression on the crosses. Shaded areas correspond to 3 std confidence intervals. (Left) CIFAR-10 with 16-4-WideResNet for Sref steps. Each curve corresponds to a different value of ηstep. (Right) ImageNet with NF-ResNet-50 with various numbers of steps. The scaling law holds for various training configurations.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Here's my assessment:

The claim about achieving a +9 point improvement under ε=8 compared to previous SOTA appears to be well-stated based on the evidence. Table 1 clearly shows their method achieving 39.2% accuracy compared to their reproduction of De et al. (2022) at 30.2%, representing a 9 percentage point improvement. The evidence includes detailed experimental results, privacy parameter analysis, and empirical validation through multiple experiments. While the review comment notes some limitations regarding language tasks, this doesn't affect the specific claim about ImageNet results. The claim is precise about the conditions (ε=8) and accurately represents the numerical improvement shown in the data.

<score>1</score>

Justification:
1. Table 1 directly supports the 9-point improvement (39.2% vs 30.2%)
2. The methodology is well-documented with privacy parameters and experimental details
3. The claim is specific to ImageNet and doesn't overgeneralize
4. Results are validated through multiple experimental configurations (Tables 1 and 4)
5. The authors acknowledge their comparison against their own reproduction of the previous SOTA
6. The review's criticism about language tasks doesn't impact this specific claim about ImageNet results


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We then use TAN to find optimal privacy parameters, leading to a gain of \(+9\) points under \(\varepsilon=8\) compared to the previous SOTA;

The review comment to be evaluated is:

Major strengths:
1. The observation of the scaling law is new.
2. The results of hyperparameter tuning improve the state-of-the-art greatly.

Minor strengths:
1. This paper is easy to follow.
2. The visualization of the experimental results is good.
3. Figure 2 seems to deliver the same message as the linear part of Figure 3 (left) in (Li et al, 2021a), but the authors provide more explanations in Section 3.2.

Major weaknesses:
1. The privacy guarantee parameter $\varepsilon_{TAN}$ is an approximation of $\varepsilon_{RDP}$ and it is only a lower bound. For strict privacy guarantees, we need an upper bound.
2. There is no explanation for the reason for the scaling law.

Minor weaknesses:
Some of the statements in this paper are inconsistent with each other.
1. In the left figure of Figure 4, the legend shows that $\sigma_{ref}$ is constant for each line, but the caption says the $\eta_{step}$ is constant. I also wonder why the authors choose to fix different values in Figure 4 for CIFAR-10 and ImageNet. Is it because the results of CIFAR-10 do not look as linear as of ImageNet?
2. In Page 5, Choice of $\sigma$, the authors claimed that if $\sigma>4$, dividing it by 2 with halving q is likely to improve performance, but I cannot find experimental results supporting this.
3. In Page 7, the authors say that they decide not to use the gain from 'testing with augmentations' in Table 1 and Table 4. In Table 2, I guess the Total column is the sum of the other columns for B=128, ..., 1024. Therefore, the last row, B=16384, also has the AugTest (+0.8) in the Total result (6.7%). I guess in Table 4, the test ACC for B=16384, 36.9%, is derived by adding the total improvement 6.7% to the baseline 30.2%. Therefore, I am not sure if the authors forget to remove the improvement from AugTest.
4. In Page 9, in the last paragraph, the authors believe that the exponential increase in the privacy budget $\varepsilon$ as the noise level $\sigma$ decreases. I guess this is from Figure 2, but the x-axis for $\sigma$ is log-scale. Therefore, it is not easy to identify whether there is a linear or exponential relationship between $\varepsilon$ and $\sigma$.

By the way, there are duplicate references (Li et al, 2021a and 2021b).

The evidence to be evaluated is:We introduce the notion of Total Amount of Noise (TAN) and discuss its connections to RDP accounting. We then demonstrate how training with reference privacy parameters \((q_{\mathrm{ref}},\sigma_{\mathrm{ref}},S)\) can be simulated with much lower computational resources using the same TAN with a smaller batch size. **Definition 4**.: _Let the individual signal-to-noise ratio \(\eta\) (and its inverse \(\Sigma\), the Total Amount of Noise or TAN) be as follows:_ \[\eta^{2}=\frac{1}{\Sigma^{2}}:=\frac{q^{2}S}{2\sigma^{2}}.\] We begin with a simple case to motivate our definition of TAN. We assume a one-dimensional model, where the gradients of all points are clipped to \(C\). Looking at Equation 2, in one batch, the expected signal from each sample is \(C/B\) with probability \(q=B/N\) and \(0\) otherwise. Therefore, the expected individual signal of each sample after \(S\) steps is \(SC/N\), and its squared norm is \(S^{2}C^{2}/N^{2}\).

\[\varepsilon_{\mathrm{RDP}}\approx\eta^{2}+\min_{\alpha}\left((\alpha-1)\eta^ {2}+\frac{\log{(1/\delta)}}{\alpha-1}\right)=\eta^{2}+2\eta\sqrt{\log{(1/ \delta)}}=:\varepsilon_{\mathrm{TAN}}(\eta). \tag{4}\] We verify this relationship empirically, and in particular choose \(\eta\) to get a desired \(\varepsilon_{\mathrm{TAN}}\) in Figure 2.

We leverage our efficient TAN training strategy and obtain new state-of-the-art results on ImageNet for \(\varepsilon=8\) (Table 1).

In Section 4.2.1, we use our simulated training with constant TAN to find optimal hyperparameters at low compute and improve performance for reference privacy parameters \((B_{\mathrm{ref}},\sigma_{\mathrm{ref}},S)=(16384,2.5,72\mathrm{k})\). In Section 4.2.2, we find optimal privacy parameters at constant TAN by changing the number of steps \(S\), leading to a new state of the art on ImageNet for \(\varepsilon=8\) (Table 1).

We perform one run with these optimal parameters at \(B=16384\) which satisfies a privacy budget of \(\varepsilon=8\).

While we kept \(S\) constant in previous experiments, we now explore constant TAN triplets \((q,\sigma,S)\) by varying \(S\). We keep \(\sigma\) fixed to \(2.5\) and vary \((B,S)\) starting from the reference \((16384,72\text{K})\) at constant \(\eta=q^{2}S/(2\sigma^{2})\). Given that \(\sigma>2\), we stay in the _almost_ constant privacy regime (see Figure 2): we indeed observe \(\varepsilon\approx\varepsilon_{\mathrm{TAN}}\) in Table 4.

Overall, we improved performance by \(9\%\) when training from scratch on ImageNet with DP-SGD under \(\varepsilon=8\). We compare to our reproduction of the previous SOTA of De et al. (2022) at \(30.2\%\) (compared to the results reported in the original paper (\(32.4\%\)), we still gain \(7\%\) of accuracy). Thus, we have shown how we can use TAN to perform optimal privacy parameter search while simulating each choice of optimal parameters at a much smaller cost.

Therefore, using TAN through our scaling law can facilitate studying variants of the network architecture while reducing the computational costs.

We now look at how collecting more data affects the tradeoff between privacy and utility. We show that doubling the data (from the same distribution) allows better performance with half the privacy budget. To this end, we train on portions of the CIFAR-10 training set (\(N=50\)k) and always report accuracies on the same test set. If we multiply by \(\beta\) the quantity of data \(N_{0}\) and keep the same \((B,\sigma,S)\), \(q\) (and thus \(\eta\)), is divided by \(\beta\) as well.

We argue that the total amount of noise (TAN) is a simple but useful guiding principle to experiment with private training. In particular, we demonstrate that the privacy budget is either a direct function of TAN or can be reduced. We further show that scaling batch size with noise level using TAN allows for ultra-efficient hyper-parameter search and demonstrate the power of this paradigm by establishing a new state of the art for DP training on Imagenet.

Using TAN allowed us to explore and compare different architectures and optimizers, which would have been computationally impossible in the normal DP training setting at \(B=16384\).

Input: Table 1: Top-1 test accuracy when training on ImageNet from scratch using DP-SGD. We train a NF-ResNet-50 with σ = 2.5, hyper-parameters of Table 2 and (B,S) = (32768, 18k) (Table 4).

Input: Table 4: Privacy parameter search. We use the optimal parameters described in Section 4.2.1 with σ = 2.5 for one expensive run and compare it with our optimal result

Input: Figure 2: Privacy budget ε as a function of the noise level σ with η constant. On both figures, each curve corresponds to a different number of steps S, and each point on the curve is computed at a sampling rate q such that η is constant. On the left, we use η = 0.13 (resulting in εTAN = 1 in Equation 4). On the right, we use η = 0.95 (εTAN = 8). We observe a “privacy wall” imposing σ ≥ 0.5 for meaningful level of privacy budget ε, and σ ≥ 2 for constant ε ≈ εTAN.

Input: Figure 4: Test accuracies at constant ηstep = Bref/( √ 2Nσref ) and S are (log) linearly decreasing with B. Dashed lines are computed using a linear regression on the crosses. Shaded areas correspond to 3 std confidence intervals. (Left) CIFAR-10 with 16-4-WideResNet for Sref steps. Each curve corresponds to a different value of ηstep. (Right) ImageNet with NF-ResNet-50 with various numbers of steps. The scaling law holds for various training configurations.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate the claim: "We then use TAN to find optimal privacy parameters, leading to a gain of +9 points under ε=8 compared to the previous SOTA"

Summary: Based on Table 1, the evidence shows an improvement from 30.2% (their reproduction of previous SOTA) to 39.2% accuracy, which is indeed a 9 percentage point improvement under ε=8. The claim is supported by detailed experimental evidence, including privacy parameter optimization methodology and clear comparative results. The review comments don't dispute this specific performance claim. While there are some minor inconsistencies noted in the paper, they don't impact this particular claim's validity. The statement accurately reflects the empirical results without exaggeration.

<score>1</score>

Here's why:
1. The claim matches the numerical evidence exactly (+9 points: from 30.2% to 39.2%)
2. The privacy budget (ε=8) is clearly documented in Table 1 and Table 4
3. The methodology using TAN for parameter optimization is well-explained
4. The reviewer concerns about εTAN being a lower bound don't invalidate the accuracy improvements
5. The claim stays focused on the empirical improvement without overstating broader implications


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We then use TAN to find optimal privacy parameters, leading to a gain of \(+9\) points under \(\varepsilon=8\) compared to the previous SOTA;

The review comment to be evaluated is:

Strengths:
1) Integrating the constraints of DP-SGD with optimization theory, in order to reliably predict good hyper-parameter choices, is one of the most important problems that needs to be solved in order to make differentially private deep learning practical. I'm pleased to see the authors tackle this difficult and important problem.
2) The metric proposed by the authors is intuitive and it appears to provide some helpful intuition.
3) The authors achieve strong performance when training from scratch on ImageNet with DP-SGD.

Weaknesses:
1) If TAN were an accurate measure of optimizability, then we should expect the train accuracy to be roughly constant at fixed TAN. However in practice, the authors find in figure 1 that the test accuracy decays log-linearly at fixed TAN as the batch size rises. I feel that the current paper is quite confusing because this discrepancy is not explicitly stated anywhere. I'd encourage the authors to make this explicit, and to explore in more detail why this discrepancy arises (eg comparing test vs train performance).

2) As discussed above, Figure 1 shows that TAN over-estimates the performance of large batch sizes. It would be nice to also see a figure at fixed batch size, sweeping the number of steps S, to see whether TAN under/over-estimates the performance of small/large step budgets.

3) As a minor point, note that on some truly sensitive data we cannot train non-private models (even internally). It would be good to acknowledge that the hyper-parameter transfer process described here cannot be used in these cases.

4) For many of the experiments, key hyper-parameters are lifted from prior work (eg step budget, target batch size). Is it not possible to directly infer good settings for these hyper-parameters using the TAN framework?

5) The paper essentially only considers ImageNet (with some very limited experiments on CIFAR-10). Since very few groups have run experiments at this scale it is not clear how challenging the baseline from De et al. is in practice.

The evidence to be evaluated is:We introduce the notion of Total Amount of Noise (TAN) and discuss its connections to RDP accounting. We then demonstrate how training with reference privacy parameters \((q_{\mathrm{ref}},\sigma_{\mathrm{ref}},S)\) can be simulated with much lower computational resources using the same TAN with a smaller batch size. **Definition 4**.: _Let the individual signal-to-noise ratio \(\eta\) (and its inverse \(\Sigma\), the Total Amount of Noise or TAN) be as follows:_ \[\eta^{2}=\frac{1}{\Sigma^{2}}:=\frac{q^{2}S}{2\sigma^{2}}.\] We begin with a simple case to motivate our definition of TAN. We assume a one-dimensional model, where the gradients of all points are clipped to \(C\). Looking at Equation 2, in one batch, the expected signal from each sample is \(C/B\) with probability \(q=B/N\) and \(0\) otherwise. Therefore, the expected individual signal of each sample after \(S\) steps is \(SC/N\), and its squared norm is \(S^{2}C^{2}/N^{2}\).

\[\varepsilon_{\mathrm{RDP}}\approx\eta^{2}+\min_{\alpha}\left((\alpha-1)\eta^ {2}+\frac{\log{(1/\delta)}}{\alpha-1}\right)=\eta^{2}+2\eta\sqrt{\log{(1/ \delta)}}=:\varepsilon_{\mathrm{TAN}}(\eta). \tag{4}\] We verify this relationship empirically, and in particular choose \(\eta\) to get a desired \(\varepsilon_{\mathrm{TAN}}\) in Figure 2.

We leverage our efficient TAN training strategy and obtain new state-of-the-art results on ImageNet for \(\varepsilon=8\) (Table 1).

In Section 4.2.1, we use our simulated training with constant TAN to find optimal hyperparameters at low compute and improve performance for reference privacy parameters \((B_{\mathrm{ref}},\sigma_{\mathrm{ref}},S)=(16384,2.5,72\mathrm{k})\). In Section 4.2.2, we find optimal privacy parameters at constant TAN by changing the number of steps \(S\), leading to a new state of the art on ImageNet for \(\varepsilon=8\) (Table 1).

We perform one run with these optimal parameters at \(B=16384\) which satisfies a privacy budget of \(\varepsilon=8\).

While we kept \(S\) constant in previous experiments, we now explore constant TAN triplets \((q,\sigma,S)\) by varying \(S\). We keep \(\sigma\) fixed to \(2.5\) and vary \((B,S)\) starting from the reference \((16384,72\text{K})\) at constant \(\eta=q^{2}S/(2\sigma^{2})\). Given that \(\sigma>2\), we stay in the _almost_ constant privacy regime (see Figure 2): we indeed observe \(\varepsilon\approx\varepsilon_{\mathrm{TAN}}\) in Table 4.

Overall, we improved performance by \(9\%\) when training from scratch on ImageNet with DP-SGD under \(\varepsilon=8\). We compare to our reproduction of the previous SOTA of De et al. (2022) at \(30.2\%\) (compared to the results reported in the original paper (\(32.4\%\)), we still gain \(7\%\) of accuracy). Thus, we have shown how we can use TAN to perform optimal privacy parameter search while simulating each choice of optimal parameters at a much smaller cost.

Therefore, using TAN through our scaling law can facilitate studying variants of the network architecture while reducing the computational costs.

We now look at how collecting more data affects the tradeoff between privacy and utility. We show that doubling the data (from the same distribution) allows better performance with half the privacy budget. To this end, we train on portions of the CIFAR-10 training set (\(N=50\)k) and always report accuracies on the same test set. If we multiply by \(\beta\) the quantity of data \(N_{0}\) and keep the same \((B,\sigma,S)\), \(q\) (and thus \(\eta\)), is divided by \(\beta\) as well.

We argue that the total amount of noise (TAN) is a simple but useful guiding principle to experiment with private training. In particular, we demonstrate that the privacy budget is either a direct function of TAN or can be reduced. We further show that scaling batch size with noise level using TAN allows for ultra-efficient hyper-parameter search and demonstrate the power of this paradigm by establishing a new state of the art for DP training on Imagenet.

Using TAN allowed us to explore and compare different architectures and optimizers, which would have been computationally impossible in the normal DP training setting at \(B=16384\).

Input: Table 1: Top-1 test accuracy when training on ImageNet from scratch using DP-SGD. We train a NF-ResNet-50 with σ = 2.5, hyper-parameters of Table 2 and (B,S) = (32768, 18k) (Table 4).

Input: Table 4: Privacy parameter search. We use the optimal parameters described in Section 4.2.1 with σ = 2.5 for one expensive run and compare it with our optimal result

Input: Figure 2: Privacy budget ε as a function of the noise level σ with η constant. On both figures, each curve corresponds to a different number of steps S, and each point on the curve is computed at a sampling rate q such that η is constant. On the left, we use η = 0.13 (resulting in εTAN = 1 in Equation 4). On the right, we use η = 0.95 (εTAN = 8). We observe a “privacy wall” imposing σ ≥ 0.5 for meaningful level of privacy budget ε, and σ ≥ 2 for constant ε ≈ εTAN.

Input: Figure 4: Test accuracies at constant ηstep = Bref/( √ 2Nσref ) and S are (log) linearly decreasing with B. Dashed lines are computed using a linear regression on the crosses. Shaded areas correspond to 3 std confidence intervals. (Left) CIFAR-10 with 16-4-WideResNet for Sref steps. Each curve corresponds to a different value of ηstep. (Right) ImageNet with NF-ResNet-50 with various numbers of steps. The scaling law holds for various training configurations.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find this claim needs careful evaluation:

The claim states they achieved "+9 points under ε=8 compared to previous SOTA." Looking at Table 1, their method achieved 39.2% accuracy compared to their reproduction of De et al. (30.2%), showing a 9% improvement. However, compared to De et al.'s original reported result of 32.4%, the improvement is 6.8% (rounded to 7%). The authors acknowledge this discrepancy in the evidence, stating "compared to the results reported in the original paper (32.4%), we still gain 7% of accuracy."

Additionally, reviewer comment #5 raises a valid concern about the limited experimental scope focusing mainly on ImageNet, questioning how challenging the baseline really is.

<score>2</score>

Justification:
1. The claim could be more precise by:
- Clarifying whether the 9-point improvement is relative to their reproduction or the original paper's results
- Acknowledging the limited scope of experiments (primarily ImageNet)
2. The numerical improvement is supported by evidence in Table 1, but presentation could be more transparent
3. The basic claim of improvement is valid, but the way it's presented could be more carefully qualified


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

* We leverage TAN to quantify the impact of the dataset size on the privacy/utility trade-off and demonstrate that doubling dataset size halves \(\varepsilon\) while providing better performance.

The evidence to be evaluated is:We introduce the notion of Total Amount of Noise (TAN) and discuss its connections to RDP accounting. We then demonstrate how training with reference privacy parameters \((q_{\mathrm{ref}},\sigma_{\mathrm{ref}},S)\) can be simulated with much lower computational resources using the same TAN with a smaller batch size. **Definition 4**.: _Let the individual signal-to-noise ratio \(\eta\) (and its inverse \(\Sigma\), the Total Amount of Noise or TAN) be as follows:_ \[\eta^{2}=\frac{1}{\Sigma^{2}}:=\frac{q^{2}S}{2\sigma^{2}}.\] We begin with a simple case to motivate our definition of TAN. We assume a one-dimensional model, where the gradients of all points are clipped to \(C\). Looking at Equation 2, in one batch, the expected signal from each sample is \(C/B\) with probability \(q=B/N\) and \(0\) otherwise. Therefore, the expected individual signal of each sample after \(S\) steps is \(SC/N\), and its squared norm is \(S^{2}C^{2}/N^{2}\).

Replacing that in the definition of \(\varepsilon_{\mathrm{RDP}}\) (Equation 3), we get \[\varepsilon_{\mathrm{RDP}}\approx\eta^{2}+\min_{\alpha}\left((\alpha-1)\eta^ {2}+\frac{\log{(1/\delta)}}{\alpha-1}\right)=\eta^{2}+2\eta\sqrt{\log{(1/ \delta)}}=:\varepsilon_{\mathrm{TAN}}(\eta). \tag{4}\] We verify this relationship empirically, and in particular choose \(\eta\) to get a desired \(\varepsilon_{\mathrm{TAN}}\) in Figure 2.

for instance, doubling the sampling rate \(q\) while dividing the number of steps \(S\) by \(4\) should leave the privacy budget constant, which we observe empirically.

Starting from \((q,\sigma,S)\), while \(\sigma<2\), we can double both \(q\) and \(\sigma\).

We find that for triplets \((q,\sigma,S)\) for which \(q/\sigma=q_{\mathrm{ref}}/\sigma_{\mathrm{ref}}\) (keeping \(\eta_{\mathrm{step}}\) constant), the performance of the network (log) linearly decreases with the batch size. This is consistent with the (non-private) work of Smith et al. (2020), which shows that for a fixed number of steps, small batch sizes perform better in generalisation (i.e., test accuracy) than large batch sizes. The difference with our work is that we consider noisy updates and observe a (log) linear relationship between batch size and performance.

If \(\sigma<2\), simultaneously doubling \(q\) and \(\sigma\) has a small or negligible negative impact on accuracy (Figure 4) but it can greatly reduce the privacy budget (Figure 2).

We then study the impact of the dataset size on the privacy/utility trade-off.

We use the CIFAR-10 dataset (Krizhevsky et al., 2009) which contains \(50\)K \(32\times 32\) images grouped in \(10\) classes. The ImageNet dataset (Deng et al., 2009; Russakovsky et al., 2014) contains \(1.2\) million images partitioned into 1000 categories.

We fix \(\delta=1/N\) where \(N\) is the number of samples for each experiment and report the corresponding value of \(\varepsilon\).

In Section 4.2.1, we use our simulated training with constant TAN to find optimal hyperparameters at low compute and improve performance for reference privacy parameters \((B_{\mathrm{ref}},\sigma_{\mathrm{ref}},S)=(16384,2.5,72\mathrm{k})\). In Section 4.2.2, we find optimal privacy parameters at constant TAN by changing the number of steps \(S\), leading to a new state of the art on ImageNet for \(\varepsilon=8\) (Table 1).

While we kept \(S\) constant in previous experiments, we now explore constant TAN triplets \((q,\sigma,S)\) by varying \(S\). We keep \(\sigma\) fixed to \(2.5\) and vary \((B,S)\) starting from the reference \((16384,72\text{K})\) at constant \(\eta=q^{2}S/(2\sigma^{2})\). Given that \(\sigma>2\), we stay in the _almost_ constant privacy regime (see Figure 2): we indeed observe \(\varepsilon\approx\varepsilon_{\mathrm{TAN}}\) in Table 4.

We also discuss how gathering more training data helps improve performance while decreasing the privacy budget.

We now look at how collecting more data affects the tradeoff between privacy and utility. We show that doubling the data (from the same distribution) allows better performance with half the privacy budget. To this end, we train on portions of the CIFAR-10 training set (\(N=50\)k) and always report accuracies on the same test set. If we multiply by \(\beta\) the quantity of data \(N_{0}\) and keep the same \((B,\sigma,S)\), \(q\) (and thus \(\eta\)), is divided by \(\beta\) as well.

We show in Table 5 the effects on \(\varepsilon\) and model accuracy. On the one hand, when using \(\varepsilon_{\mathrm{TAN}}\), we can predict the impact on the privacy budget. On the other hand, since the global signal-to-noise ratio \(N\eta\) is held constant in all experiments, we expect to extract the same amount of information in each setup; adding more data makes this information richer, which explains the gain in accuracy.

We argue that the total amount of noise (TAN) is a simple but useful guiding principle to experiment with private training. In particular, we demonstrate that the privacy budget is either a direct function of TAN or can be reduced.

We show in Table 6 that similarly to the experiments in CIFAR-10, doubling the training data on ImageNet improves the accuracy while diving \(\varepsilon\) by \(2\).

We compare training on half of the ImageNet training set (\(N=600k\)) and the entire training set (\(N=1.2M\)).

Using TAN allowed us to explore and compare different architectures and optimizers, which would have been computationally impossible in the normal DP training setting at \(B=16384\).

Input: Table 1: Top-1 test accuracy when training on ImageNet from scratch using DP-SGD. We train a NF-ResNet-50 with σ = 2.5, hyper-parameters of Table 2 and (B,S) = (32768, 18k) (Table 4).

Input: Table 4: Privacy parameter search. We use the optimal parameters described in Section 4.2.1 with σ = 2.5 for one expensive run and compare it with our optimal result

Input: Table 6: Impact of adding more data on ImageNet. The “Simulated Gain” column corresponds to the accuracy gain we observe when simulating at lower compute using our scaling strategy for B = 256. The “Gain” column corresponds to the real gain at B = 16384.

Input: Figure 2: Privacy budget ε as a function of the noise level σ with η constant. On both figures, each curve corresponds to a different number of steps S, and each point on the curve is computed at a sampling rate q such that η is constant. On the left, we use η = 0.13 (resulting in εTAN = 1 in Equation 4). On the right, we use η = 0.95 (εTAN = 8). We observe a “privacy wall” imposing σ ≥ 0.5 for meaningful level of privacy budget ε, and σ ≥ 2 for constant ε ≈ εTAN.

Input: Table 5: Impact of the training set size N on the privacy/utility trade-off. We start training on 10% of the data (N0 = 5K). We use B = 4,096, σ = 3 and S = 2,500, with post-activation normalization, and no augmentation. Standard deviations are computed over 3 runs.

Input: Figure 4: Test accuracies at constant ηstep = Bref/( √ 2Nσref ) and S are (log) linearly decreasing with B. Dashed lines are computed using a linear regression on the crosses. Shaded areas correspond to 3 std confidence intervals. (Left) CIFAR-10 with 16-4-WideResNet for Sref steps. Each curve corresponds to a different value of ηstep. (Right) ImageNet with NF-ResNet-50 with various numbers of steps. The scaling law holds for various training configurations.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After careful review, the claim that "doubling dataset size halves ε while providing better performance" is well-supported by the evidence. Tables 5 and 6 directly demonstrate this relationship on both CIFAR-10 and ImageNet datasets. For CIFAR-10, increasing N from 5K to 50K shows ε decreasing proportionally while accuracy improves from 59.9% to 74.0%. Similarly, Table 6 shows that doubling ImageNet data from 0.6M to 1.2M samples halves ε from 17.98 to 8.00 while gaining 1.3% accuracy. The relationship is consistently demonstrated across different datasets, making this a well-stated claim.

<score>1</score>

Justification:
1. Direct empirical evidence from Tables 5 and 6 shows the claimed relationship between dataset size, ε, and performance
2. The relationship is demonstrated on multiple datasets (CIFAR-10 and ImageNet)
3. The exact numerical results align with the claim (e.g., doubling N from 0.6M to 1.2M halves ε from ~18 to ~8)
4. The claim appropriately acknowledges the improvement in performance ("better performance") without overstating the magnitude
5. The theoretical foundation is explained through TAN and signal-to-noise ratio analysis