Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We then derive scaling laws for training models with DP-SGD to optimize hyper-parameters with more than a \(100\times\) reduction in computational budget.

The review comment to be evaluated is:

Strengths:
1) Integrating the constraints of DP-SGD with optimization theory, in order to reliably predict good hyper-parameter choices, is one of the most important problems that needs to be solved in order to make differentially private deep learning practical. I'm pleased to see the authors tackle this difficult and important problem.
2) The metric proposed by the authors is intuitive and it appears to provide some helpful intuition.
3) The authors achieve strong performance when training from scratch on ImageNet with DP-SGD.

Weaknesses:
1) If TAN were an accurate measure of optimizability, then we should expect the train accuracy to be roughly constant at fixed TAN. However in practice, the authors find in figure 1 that the test accuracy decays log-linearly at fixed TAN as the batch size rises. I feel that the current paper is quite confusing because this discrepancy is not explicitly stated anywhere. I'd encourage the authors to make this explicit, and to explore in more detail why this discrepancy arises (eg comparing test vs train performance).

2) As discussed above, Figure 1 shows that TAN over-estimates the performance of large batch sizes. It would be nice to also see a figure at fixed batch size, sweeping the number of steps S, to see whether TAN under/over-estimates the performance of small/large step budgets.

3) As a minor point, note that on some truly sensitive data we cannot train non-private models (even internally). It would be good to acknowledge that the hyper-parameter transfer process described here cannot be used in these cases.

4) For many of the experiments, key hyper-parameters are lifted from prior work (eg step budget, target batch size). Is it not possible to directly infer good settings for these hyper-parameters using the TAN framework?

5) The paper essentially only considers ImageNet (with some very limited experiments on CIFAR-10). Since very few groups have run experiments at this scale it is not clear how challenging the baseline from De et al. is in practice.

The evidence to be evaluated is:Computing per-sample gradients with mega batch sizes for a large number of steps and AugMul makes DP-SGD much more computationally intensive than non-private training, typically dozens of times. For instance, reproducing the previous SOTA on ImageNet of De et al. (2022) under \(\varepsilon=8\) necessitates a \(4\)-day run using \(32\) A100 GPUs, while the non-private SOTA can be reproduced in a few hours with the same hardware (Goyal et al., 2017). Yu et al. (2021b) propose to use low-rank reparametrization of the weight matrices and diminish the computational cost of accessing per-sample gradients.

We introduce the notion of Total Amount of Noise (TAN) and discuss its connections to RDP accounting. We then demonstrate how training with reference privacy parameters \((q_{\mathrm{ref}},\sigma_{\mathrm{ref}},S)\) can be simulated with much lower computational resources using the same TAN with a smaller batch size. **Definition 4**.: _Let the individual signal-to-noise ratio \(\eta\) (and its inverse \(\Sigma\), the Total Amount of Noise or TAN) be as follows:_ \[\eta^{2}=\frac{1}{\Sigma^{2}}:=\frac{q^{2}S}{2\sigma^{2}}.\] We begin with a simple case to motivate our definition of TAN. We assume a one-dimensional model, where the gradients of all points are clipped to \(C\). Looking at Equation 2, in one batch, the expected signal from each sample is \(C/B\) with probability \(q=B/N\) and \(0\) otherwise. Therefore, the expected individual signal of each sample after \(S\) steps is \(SC/N\), and its squared norm is \(S^{2}C^{2}/N^{2}\).

\[\frac{\frac{S^{2}C^{2}}{N^{2}}}{\frac{2SC^{2}\sigma^{2}}{B^{2}}}=\frac{q^{2}S} {2\sigma^{2}}=\eta^{2}.\] Denoting \(\eta_{\mathrm{step}}:=q/\sqrt{2}\sigma\), we have \(\eta^{2}=S\eta_{\mathrm{step}}^{2}\). The ratio \(\sigma/q\) is noted by Li et al. (2021) as the effective noise. The authors found that for a fixed budget \(\varepsilon\) and fixed \(S\), the effective noise decreases with \(B\). Our analysis goes further by analyzing how RDP accounting explains this dependency. Intuitively, we expect that the privacy budget \(\varepsilon\) only depends on the signal-to-noise ratio \(\eta\). In Figure 2, we plot \(\varepsilon\) as a function of \(\sigma\) and \(S\), at a _fixed_\(\eta\), and observe that \(\varepsilon\) is indeed constant, but only when \(\sigma>2\). On the contrary, when \(\sigma\) gets smaller, \(\varepsilon\) increases exponentially, creating a "Privacy Wall".

for instance, doubling the sampling rate \(q\) while dividing the number of steps \(S\) by \(4\) should leave the privacy budget constant, which we observe empirically. Our RDP analysis in Section 3.2 suggests a simple scaling strategy. Starting from \((q,\sigma,S)\), while \(\sigma<2\), we can double both \(q\) and \(\sigma\).

Batch Scaling Laws. We now analyse how this strategy affects the performance of the network. In Figure 4, we perform this analysis on CIFAR-10 and ImageNet. We find that for triplets \((q,\sigma,S)\) for which \(q/\sigma=q_{\mathrm{ref}}/\sigma_{\mathrm{ref}}\) (keeping \(\eta_{\mathrm{step}}\) constant), the performance of the network (log) linearly decreases with the batch size. This is consistent with the (non-private) work of Smith et al. (2020), which shows that for a fixed number of steps, small batch sizes perform better in generalisation (i.e., test accuracy) than large batch sizes. The difference with our work is that we consider noisy updates and observe a (log) linear relationship between batch size and performance.

If \(\sigma<2\), simultaneously doubling \(q\) and \(\sigma\) has a small or negligible negative impact on accuracy (Figure 4) but it can greatly reduce the privacy budget (Figure 2).

We choose the optimal hyper-parameters (including architecture, optimizer, type of data augmentation) in this simulated setting, and finally launch one single run at the reference (large) batch size, with desired privacy guarantees.

We thus reduce the computational requirements by a factor of \(128\times\). Each hyper-parameter search in the ImageNet setup takes \(4\) days using \(32\) A-100 GPUs; we reduce it to less than a day on a single GPU.

In Section 4.2.1, we use our simulated training with constant TAN to find optimal hyperparameters at low compute and improve performance for reference privacy parameters \((B_{\mathrm{ref}},\sigma_{\mathrm{ref}},S)=(16384,2.5,72\mathrm{k})\). In Section 4.2.2, we find optimal privacy parameters at constant TAN by changing the number of steps \(S\), leading to a new state of the art on ImageNet for \(\varepsilon=8\) (Table 1).

We run a large hyper-parameter search and report the best hyper-parameters in Table 2 as well as the corresponding improvement for various batch sizes (at constant \(\eta_{\text{step}}\) and \(S\)).

We search over learning rates \(lr\in[1,2,4,8,12,16]\), momentum parameters \(\mu\in[0,0.1,0.5,0.9,1]\) and dampening factors \(d\in[0,0.1,0.5,0.9,1]\). We use exponential moving average (EMA) on the weights (Tan & Le, 2019) with a decay parameter in \([0.9,0.99,0.999,0.9999,0.99999]\).

We have experimented with different architectures (ViTs, ResNets, NFResnets) and optimizers (DP-Adam, DP-AdamW, DP-SGD) (see Appendix B for details). Our best results are obtained using a NFResnet-50 and DP-SGD with constant learning rate, which differs from standard practice in non-private training. While we kept \(S\) constant in previous experiments, we now explore constant TAN triplets \((q,\sigma,S)\) by varying \(S\). We keep \(\sigma\) fixed to \(2.5\) and vary \((B,S)\) starting from the reference \((16384,72\text{K})\) at constant \(\eta=q^{2}S/(2\sigma^{2})\). Given that \(\sigma>2\), we stay in the _almost_ constant privacy regime (see Figure 2): we indeed observe \(\varepsilon\approx\varepsilon_{\mathrm{TAN}}\) in Table 4.

Since performing this privacy parameter search is computationally intensive, we first simulate training using our scaling law at \(B=256\) (with the same \(\eta_{\mathrm{step}}\)) and display our results in Table 3.

Thus, we have shown how we can use TAN to perform optimal privacy parameter search while simulating each choice of optimal parameters at a much smaller cost.

However, each simulation is \(8\) times less computationally expensive. Therefore, using TAN through our scaling law can facilitate studying variants of the network architecture while reducing the computational costs.

If we multiply by \(\beta\) the quantity of data \(N_{0}\) and keep the same \((B,\sigma,S)\), \(q\) (and thus \(\eta\)), is divided by \(\beta\) as well.

In particular, we demonstrate that the privacy budget is either a direct function of TAN or can be reduced. We further show that scaling batch size with noise level using TAN allows for ultra-efficient hyper-parameter search and demonstrate the power of this paradigm by establishing a new state of the art for DP training on Imagenet.

To illustrate this extremely, we train a Vision Transformer (ViT) (Dosovitskiy et al., 2020) from scratch on ImageNet using DP-SGD.

We used our scaling strategy (keeping \(\eta_{step}\) and \(S\) constant) to simulate the DP training with different architectures at low compute, studying noisy training without the burden of DP accounting. The best simulated results were obtained with a NFResNet-50 (Brock et al., 2021b) designed to be fast learners in terms of number of FLOPS. The worst results were obtained with ViTs, and intermediate results with classical ResNets.

This highlights the fact that training with DP-SGD is a different paradigm that requires its own tools. Using TAN allowed us to explore and compare different architectures and optimizers, which would have been computationally impossible in the normal DP training setting at \(B=16384\).

Input: Table 1: Top-1 test accuracy when training on ImageNet from scratch using DP-SGD. We train a NF-ResNet-50 with σ = 2.5, hyper-parameters of Table 2 and (B,S) = (32768, 18k) (Table 4).

Input: Table 2: Comparing optimal hyper-parameters. Keeping ηstep and S constant, we compare various changes in the training pipeline. We compare with the baseline of De et al. (2022) (blue line in Figure 1: NFResNet-50, learning rate at 4, EMA decay at 0.99999, 4 random augmentations averaged over 3 runs). Each gain is compared to the previous column.

Input: Table 4: Privacy parameter search. We use the optimal parameters described in Section 4.2.1 with σ = 2.5 for one expensive run and compare it with our optimal result

Input: Table 3: Low compute simulation of privacy parameter search. We start from B = 256 = 16384/64 and S = 72K. We use σ = 2.5/64 for all runs and no data augmentation.

Input: Figure 2: Privacy budget ε as a function of the noise level σ with η constant. On both figures, each curve corresponds to a different number of steps S, and each point on the curve is computed at a sampling rate q such that η is constant. On the left, we use η = 0.13 (resulting in εTAN = 1 in Equation 4). On the right, we use η = 0.95 (εTAN = 8). We observe a “privacy wall” imposing σ ≥ 0.5 for meaningful level of privacy budget ε, and σ ≥ 2 for constant ε ≈ εTAN.

Input: Figure 4: Test accuracies at constant ηstep = Bref/( √ 2Nσref ) and S are (log) linearly decreasing with B. Dashed lines are computed using a linear regression on the crosses. Shaded areas correspond to 3 std confidence intervals. (Left) CIFAR-10 with 16-4-WideResNet for Sref steps. Each curve corresponds to a different value of ηstep. (Right) ImageNet with NF-ResNet-50 with various numbers of steps. The scaling law holds for various training configurations.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim "We then derive scaling laws for training models with DP-SGD to optimize hyper-parameters with more than a 100× reduction in computational budget" appears to be well-supported but slightly overstated. The evidence shows they achieved significant computational savings (128× reduction mentioned), and demonstrated effective scaling laws through extensive experimentation. However, the reviewer points out some limitations - the TAN metric overestimates performance at large batch sizes, and experiments were limited primarily to ImageNet. The claim's core assertion about deriving useful scaling laws is valid, but the exact magnitude of computational savings may vary depending on the specific scenario.

<score>2</score>

Justification:
1. Strong evidence supports substantial computational reduction (e.g., "reduce the computational requirements by a factor of 128×" and "reduce it to less than a day on a single GPU")
2. Empirical validation shown in multiple figures/tables demonstrating scaling relationships
3. Reviewer concerns about:
   - TAN accuracy limitations at large batch sizes
   - Limited experimental scope (mostly ImageNet)
4. The claim is mostly accurate but slightly overstates the universality/consistency of the 100× reduction


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We apply the proposed method on CIFAR-10 and ImageNet and, in particular, strongly improve the state-of-the-art on ImageNet with a \(+9\) points gain in accuracy for a privacy budget \(\varepsilon=8\).

The evidence to be evaluated is:De et al. (2022) recently introduced Augmentation Multiplicity (AugMul), which averages the gradients from different augmented versions of every sample before clipping and leads to improved performance on CIFAR-10.

For instance, reproducing the previous SOTA on ImageNet of De et al. (2022) under \(\varepsilon=8\) necessitates a \(4\)-day run using \(32\) A100 GPUs, while the non-private SOTA can be reproduced in a few hours with the same hardware (Goyal et al., 2017).

In Figure 4, we perform this analysis on CIFAR-10 and ImageNet.

We leverage our efficient TAN training strategy and obtain new state-of-the-art results on ImageNet for \(\varepsilon=8\) (Table 1).

The ImageNet dataset (Deng et al., 2009; Russakovsky et al., 2014) contains \(1.2\) million images partitioned into 1000 categories.

On Imagenet, we compare Vision Transformers (ViTs) (Dosovitskiy et al., 2020), Residual Neural Networks (ResNets) (He et al., 2016) and Normalizer-Free ResNets (NF-ResNets) (Brock et al., 2021b).

In Section 4.2.1, we use our simulated training with constant TAN to find optimal hyperparameters at low compute and improve performance for reference privacy parameters \((B_{\mathrm{ref}},\sigma_{\mathrm{ref}},S)=(16384,2.5,72\mathrm{k})\). In Section 4.2.2, we find optimal privacy parameters at constant TAN by changing the number of steps \(S\), leading to a new state of the art on ImageNet for \(\varepsilon=8\) (Table 1).

We perform one run with these optimal parameters at \(B=16384\) which satisfies a privacy budget of \(\varepsilon=8\).

We have experimented with different architectures (ViTs, ResNets, NFResnets) and optimizers (DP-Adam, DP-AdamW, DP-SGD) (see Appendix B for details). Our best results are obtained using a NFResnet-50 and DP-SGD with constant learning rate, which differs from standard practice in non-private training.

Given that \(\sigma>2\), we stay in the _almost_ constant privacy regime (see Figure 2): we indeed observe \(\varepsilon\approx\varepsilon_{\mathrm{TAN}}\) in Table 4.

Overall, we improved performance by \(9\%\) when training from scratch on ImageNet with DP-SGD under \(\varepsilon=8\). We compare to our reproduction of the previous SOTA of De et al. (2022) at \(30.2\%\) (compared to the results reported in the original paper (\(32.4\%\)), we still gain \(7\%\) of accuracy).

We show similar results for ImageNet in Appendix A.

We further show that scaling batch size with noise level using TAN allows for ultra-efficient hyper-parameter search and demonstrate the power of this paradigm by establishing a new state of the art for DP training on Imagenet.

Unlocking high-accuracy differentially private image classification through scale, 2022.

We show in Table 6 that similarly to the experiments in CIFAR-10, doubling the training data on ImageNet improves the accuracy while diving \(\varepsilon\) by \(2\).

We compare training on half of the ImageNet training set (\(N=600k\)) and the entire training set (\(N=1.2M\)).

In particular, we noticed that DP-SGD without momentum is always optimal, even with ViTs, and that NF-ResNets-50 performed the best.

Input: Table 1: Top-1 test accuracy when training on ImageNet from scratch using DP-SGD. We train a NF-ResNet-50 with σ = 2.5, hyper-parameters of Table 2 and (B,S) = (32768, 18k) (Table 4).

Input: Table 4: Privacy parameter search. We use the optimal parameters described in Section 4.2.1 with σ = 2.5 for one expensive run and compare it with our optimal result

Input: Table 6: Impact of adding more data on ImageNet. The “Simulated Gain” column corresponds to the accuracy gain we observe when simulating at lower compute using our scaling strategy for B = 256. The “Gain” column corresponds to the real gain at B = 16384.

Input: Figure 2: Privacy budget ε as a function of the noise level σ with η constant. On both figures, each curve corresponds to a different number of steps S, and each point on the curve is computed at a sampling rate q such that η is constant. On the left, we use η = 0.13 (resulting in εTAN = 1 in Equation 4). On the right, we use η = 0.95 (εTAN = 8). We observe a “privacy wall” imposing σ ≥ 0.5 for meaningful level of privacy budget ε, and σ ≥ 2 for constant ε ≈ εTAN.

Input: Figure 4: Test accuracies at constant ηstep = Bref/( √ 2Nσref ) and S are (log) linearly decreasing with B. Dashed lines are computed using a linear regression on the crosses. Shaded areas correspond to 3 std confidence intervals. (Left) CIFAR-10 with 16-4-WideResNet for Sref steps. Each curve corresponds to a different value of ηstep. (Right) ImageNet with NF-ResNet-50 with various numbers of steps. The scaling law holds for various training configurations.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
1. The claim states a "+9 points gain in accuracy" for ImageNet at ε=8 privacy budget. According to Table 1, the method achieves 39.2% accuracy compared to De et al.'s 30.2% (their reproduction) or 32.4% (original paper) - supporting either a 9% or 6.8% improvement. The evidence thoroughly documents the experimental setup, architecture choices, and parameter settings. The results are demonstrated through multiple tables and figures with clear comparisons to previous work. The claim appears to be well-stated when comparing to their reproduction (9% gain) though slightly overstated if comparing to the original paper's reported results (6.8% gain). However, since they explicitly reference their reproduction as the baseline, the claim is generally accurate.

<score>1</score>

Justification:
- The claim is supported by concrete evidence in Table 1 showing the numerical improvements
- The methodology is well-documented with specific architectural choices (NF-ResNet-50), optimizer settings (DP-SGD), and privacy parameters (ε=8, σ=2.5)
- They transparently acknowledge both the original paper results and their reproduction results
- The experimental details are comprehensive, including hardware requirements, training duration, and parameter settings
- The improvement is consistently demonstrated across multiple analyses (Tables 1, 4, 6)
- They provide thorough ablation studies and scaling analyses to support their findings


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We apply the proposed method on CIFAR-10 and ImageNet and, in particular, strongly improve the state-of-the-art on ImageNet with a \(+9\) points gain in accuracy for a privacy budget \(\varepsilon=8\).

The review comment to be evaluated is:

Strength:

* The observations (batch scaling law, privacy budget computation) in the paper are new and interesting.

* The proposed hyper-parameter tuning process can be very useful in practice, as it saves the computation requirement and therefore speeds up the development process.

Weaknesses:

* Overall, the writing is clear, but there are still some issues that need to be addressed:
    - Equation 2: many of the notations here are not defined, e.g., clip_C, l, theta, C.
    - Figure 4: B_ref, S_ref, \sigma_ref are not defined. I can guess their meaning from the context, but all notations should be defined explicitly in the text.
    - Page 5: it says "Simultaneously doubling the batch size and \sigma has a negligible or small impact on accuracy (Figure 4)". However, in Figure 4(b), the impact on the accuracy is not small. Indeed, this is claimed in the next sentence "Reciprocally, if \sigma > 4, dividing it by 2 simultaneously with halving q is likely to improve performance." These two sentences contradict each other.
    - Table 1: I guess "8,8." should be "8,"?

The evidence to be evaluated is:De et al. (2022) recently introduced Augmentation Multiplicity (AugMul), which averages the gradients from different augmented versions of every sample before clipping and leads to improved performance on CIFAR-10.

For instance, reproducing the previous SOTA on ImageNet of De et al. (2022) under \(\varepsilon=8\) necessitates a \(4\)-day run using \(32\) A100 GPUs, while the non-private SOTA can be reproduced in a few hours with the same hardware (Goyal et al., 2017).

In Figure 4, we perform this analysis on CIFAR-10 and ImageNet.

We leverage our efficient TAN training strategy and obtain new state-of-the-art results on ImageNet for \(\varepsilon=8\) (Table 1).

The ImageNet dataset (Deng et al., 2009; Russakovsky et al., 2014) contains \(1.2\) million images partitioned into 1000 categories.

On Imagenet, we compare Vision Transformers (ViTs) (Dosovitskiy et al., 2020), Residual Neural Networks (ResNets) (He et al., 2016) and Normalizer-Free ResNets (NF-ResNets) (Brock et al., 2021b).

In Section 4.2.1, we use our simulated training with constant TAN to find optimal hyperparameters at low compute and improve performance for reference privacy parameters \((B_{\mathrm{ref}},\sigma_{\mathrm{ref}},S)=(16384,2.5,72\mathrm{k})\). In Section 4.2.2, we find optimal privacy parameters at constant TAN by changing the number of steps \(S\), leading to a new state of the art on ImageNet for \(\varepsilon=8\) (Table 1).

We perform one run with these optimal parameters at \(B=16384\) which satisfies a privacy budget of \(\varepsilon=8\).

We have experimented with different architectures (ViTs, ResNets, NFResnets) and optimizers (DP-Adam, DP-AdamW, DP-SGD) (see Appendix B for details). Our best results are obtained using a NFResnet-50 and DP-SGD with constant learning rate, which differs from standard practice in non-private training.

Given that \(\sigma>2\), we stay in the _almost_ constant privacy regime (see Figure 2): we indeed observe \(\varepsilon\approx\varepsilon_{\mathrm{TAN}}\) in Table 4.

Overall, we improved performance by \(9\%\) when training from scratch on ImageNet with DP-SGD under \(\varepsilon=8\). We compare to our reproduction of the previous SOTA of De et al. (2022) at \(30.2\%\) (compared to the results reported in the original paper (\(32.4\%\)), we still gain \(7\%\) of accuracy).

We show similar results for ImageNet in Appendix A.

We further show that scaling batch size with noise level using TAN allows for ultra-efficient hyper-parameter search and demonstrate the power of this paradigm by establishing a new state of the art for DP training on Imagenet.

Unlocking high-accuracy differentially private image classification through scale, 2022.

We show in Table 6 that similarly to the experiments in CIFAR-10, doubling the training data on ImageNet improves the accuracy while diving \(\varepsilon\) by \(2\).

We compare training on half of the ImageNet training set (\(N=600k\)) and the entire training set (\(N=1.2M\)).

In particular, we noticed that DP-SGD without momentum is always optimal, even with ViTs, and that NF-ResNets-50 performed the best.

Input: Table 1: Top-1 test accuracy when training on ImageNet from scratch using DP-SGD. We train a NF-ResNet-50 with σ = 2.5, hyper-parameters of Table 2 and (B,S) = (32768, 18k) (Table 4).

Input: Table 4: Privacy parameter search. We use the optimal parameters described in Section 4.2.1 with σ = 2.5 for one expensive run and compare it with our optimal result

Input: Table 6: Impact of adding more data on ImageNet. The “Simulated Gain” column corresponds to the accuracy gain we observe when simulating at lower compute using our scaling strategy for B = 256. The “Gain” column corresponds to the real gain at B = 16384.

Input: Figure 2: Privacy budget ε as a function of the noise level σ with η constant. On both figures, each curve corresponds to a different number of steps S, and each point on the curve is computed at a sampling rate q such that η is constant. On the left, we use η = 0.13 (resulting in εTAN = 1 in Equation 4). On the right, we use η = 0.95 (εTAN = 8). We observe a “privacy wall” imposing σ ≥ 0.5 for meaningful level of privacy budget ε, and σ ≥ 2 for constant ε ≈ εTAN.

Input: Figure 4: Test accuracies at constant ηstep = Bref/( √ 2Nσref ) and S are (log) linearly decreasing with B. Dashed lines are computed using a linear regression on the crosses. Shaded areas correspond to 3 std confidence intervals. (Left) CIFAR-10 with 16-4-WideResNet for Sref steps. Each curve corresponds to a different value of ηstep. (Right) ImageNet with NF-ResNet-50 with various numbers of steps. The scaling law holds for various training configurations.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me evaluate the claim "We apply the proposed method on CIFAR-10 and ImageNet and, in particular, strongly improve the state-of-the-art on ImageNet with a +9 points gain in accuracy for a privacy budget ε=8":

1. Summary:
The evidence substantially supports this claim. Table 1 shows their method achieved 39.2% accuracy compared to their reproduction of De et al.'s previous SOTA at 30.2% - a 9% improvement at ε=8. They validate this through extensive experimentation with different architectures and parameters, and provide detailed evidence of their methodology. While the gain is 7% compared to De et al.'s original reported result (32.4%), the claim specifically references their reproduction as the baseline. The experimental details, comparative results, and clear documentation of methods make this a well-stated claim that accurately reflects the evidence.

<score>1</score>

Justification:
- The claim is precise and directly supported by Table 1 and accompanying experimental details
- They acknowledge both their reproduction (30.2%) and the original paper's results (32.4%)
- Comprehensive evidence is provided through:
  - Detailed experimental setup and architecture comparisons
  - Privacy budget validation (Table 4, Figure 2)
  - Scaling analysis (Figure 4)
  - Data size impact studies (Table 6)
- The reviewer comments don't challenge the accuracy claims, focusing instead on notation clarity issues


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We apply the proposed method on CIFAR-10 and ImageNet and, in particular, strongly improve the state-of-the-art on ImageNet with a \(+9\) points gain in accuracy for a privacy budget \(\varepsilon=8\).

The review comment to be evaluated is:

Strength: It studies an important problem of how to scale DP-SGD with hyper-parameter search. The paper provides a neat and simple relationship between the privacy budget and the other parameters, i.e., the noise multiplier, steps and the sampling rate. This relationship motivates a way of scaling the batchsize and the noise multiplier without affecting the privacy accountant.

Weakness:
1. There is no application of using TAN relation in language task, either pretraining or fine-tuning.

2. The reference needs to cover more. Especially, Yu et al. 2021 starts the fine-tuning large language models with differential privacy and low-rank reparametrization.
Yu et al. 2021 Large Scale Private Learning via Low-rank Reparametrization.

The evidence to be evaluated is:De et al. (2022) recently introduced Augmentation Multiplicity (AugMul), which averages the gradients from different augmented versions of every sample before clipping and leads to improved performance on CIFAR-10.

For instance, reproducing the previous SOTA on ImageNet of De et al. (2022) under \(\varepsilon=8\) necessitates a \(4\)-day run using \(32\) A100 GPUs, while the non-private SOTA can be reproduced in a few hours with the same hardware (Goyal et al., 2017).

In Figure 4, we perform this analysis on CIFAR-10 and ImageNet.

We leverage our efficient TAN training strategy and obtain new state-of-the-art results on ImageNet for \(\varepsilon=8\) (Table 1).

The ImageNet dataset (Deng et al., 2009; Russakovsky et al., 2014) contains \(1.2\) million images partitioned into 1000 categories.

On Imagenet, we compare Vision Transformers (ViTs) (Dosovitskiy et al., 2020), Residual Neural Networks (ResNets) (He et al., 2016) and Normalizer-Free ResNets (NF-ResNets) (Brock et al., 2021b).

In Section 4.2.1, we use our simulated training with constant TAN to find optimal hyperparameters at low compute and improve performance for reference privacy parameters \((B_{\mathrm{ref}},\sigma_{\mathrm{ref}},S)=(16384,2.5,72\mathrm{k})\). In Section 4.2.2, we find optimal privacy parameters at constant TAN by changing the number of steps \(S\), leading to a new state of the art on ImageNet for \(\varepsilon=8\) (Table 1).

We perform one run with these optimal parameters at \(B=16384\) which satisfies a privacy budget of \(\varepsilon=8\).

We have experimented with different architectures (ViTs, ResNets, NFResnets) and optimizers (DP-Adam, DP-AdamW, DP-SGD) (see Appendix B for details). Our best results are obtained using a NFResnet-50 and DP-SGD with constant learning rate, which differs from standard practice in non-private training.

Given that \(\sigma>2\), we stay in the _almost_ constant privacy regime (see Figure 2): we indeed observe \(\varepsilon\approx\varepsilon_{\mathrm{TAN}}\) in Table 4.

Overall, we improved performance by \(9\%\) when training from scratch on ImageNet with DP-SGD under \(\varepsilon=8\). We compare to our reproduction of the previous SOTA of De et al. (2022) at \(30.2\%\) (compared to the results reported in the original paper (\(32.4\%\)), we still gain \(7\%\) of accuracy).

We show similar results for ImageNet in Appendix A.

We further show that scaling batch size with noise level using TAN allows for ultra-efficient hyper-parameter search and demonstrate the power of this paradigm by establishing a new state of the art for DP training on Imagenet.

Unlocking high-accuracy differentially private image classification through scale, 2022.

We show in Table 6 that similarly to the experiments in CIFAR-10, doubling the training data on ImageNet improves the accuracy while diving \(\varepsilon\) by \(2\).

We compare training on half of the ImageNet training set (\(N=600k\)) and the entire training set (\(N=1.2M\)).

In particular, we noticed that DP-SGD without momentum is always optimal, even with ViTs, and that NF-ResNets-50 performed the best.

Input: Table 1: Top-1 test accuracy when training on ImageNet from scratch using DP-SGD. We train a NF-ResNet-50 with σ = 2.5, hyper-parameters of Table 2 and (B,S) = (32768, 18k) (Table 4).

Input: Table 4: Privacy parameter search. We use the optimal parameters described in Section 4.2.1 with σ = 2.5 for one expensive run and compare it with our optimal result

Input: Table 6: Impact of adding more data on ImageNet. The “Simulated Gain” column corresponds to the accuracy gain we observe when simulating at lower compute using our scaling strategy for B = 256. The “Gain” column corresponds to the real gain at B = 16384.

Input: Figure 2: Privacy budget ε as a function of the noise level σ with η constant. On both figures, each curve corresponds to a different number of steps S, and each point on the curve is computed at a sampling rate q such that η is constant. On the left, we use η = 0.13 (resulting in εTAN = 1 in Equation 4). On the right, we use η = 0.95 (εTAN = 8). We observe a “privacy wall” imposing σ ≥ 0.5 for meaningful level of privacy budget ε, and σ ≥ 2 for constant ε ≈ εTAN.

Input: Figure 4: Test accuracies at constant ηstep = Bref/( √ 2Nσref ) and S are (log) linearly decreasing with B. Dashed lines are computed using a linear regression on the crosses. Shaded areas correspond to 3 std confidence intervals. (Left) CIFAR-10 with 16-4-WideResNet for Sref steps. Each curve corresponds to a different value of ηstep. (Right) ImageNet with NF-ResNet-50 with various numbers of steps. The scaling law holds for various training configurations.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
1. Summary of evaluation:
The claim of improving state-of-the-art on ImageNet with a +9 points gain in accuracy for ε=8 is well-supported by the evidence. Table 1 clearly shows their method achieving 39.2% accuracy compared to their reproduction of De et al.'s previous SOTA of 30.2% (a 9% improvement) at the same privacy budget (ε=8). The evidence includes comprehensive experimental details, architecture comparisons, and thorough ablation studies. The claim is precise about the privacy budget and accuracy gain, avoiding exaggeration. The reviewer comments are positive about the methodology but don't directly address this specific accuracy claim.

<score>1</score>

Justification:
- The 9-point accuracy improvement is directly demonstrated in Table 1 with precise numbers
- The privacy budget ε=8 is consistently referenced and verified across multiple experiments
- The paper provides extensive experimental validation including:
  - Architecture comparisons (ViTs, ResNets, NF-ResNets)
  - Optimizer evaluations
  - Scaling analysis (Figure 4)
  - Data size impact (Table 6)
  - Reproduction of previous SOTA for fair comparison
- Claims are measured and specific, avoiding overclaiming broader implications
- Limitations and technical details are thoroughly documented
- The methodology is supported by positive reviewer comments regarding the approach's theoretical foundation


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We apply the proposed method on CIFAR-10 and ImageNet and, in particular, strongly improve the state-of-the-art on ImageNet with a \(+9\) points gain in accuracy for a privacy budget \(\varepsilon=8\).

The review comment to be evaluated is:

Major strengths:
1. The observation of the scaling law is new.
2. The results of hyperparameter tuning improve the state-of-the-art greatly.

Minor strengths:
1. This paper is easy to follow.
2. The visualization of the experimental results is good.
3. Figure 2 seems to deliver the same message as the linear part of Figure 3 (left) in (Li et al, 2021a), but the authors provide more explanations in Section 3.2.

Major weaknesses:
1. The privacy guarantee parameter $\varepsilon_{TAN}$ is an approximation of $\varepsilon_{RDP}$ and it is only a lower bound. For strict privacy guarantees, we need an upper bound.
2. There is no explanation for the reason for the scaling law.

Minor weaknesses:
Some of the statements in this paper are inconsistent with each other.
1. In the left figure of Figure 4, the legend shows that $\sigma_{ref}$ is constant for each line, but the caption says the $\eta_{step}$ is constant. I also wonder why the authors choose to fix different values in Figure 4 for CIFAR-10 and ImageNet. Is it because the results of CIFAR-10 do not look as linear as of ImageNet?
2. In Page 5, Choice of $\sigma$, the authors claimed that if $\sigma>4$, dividing it by 2 with halving q is likely to improve performance, but I cannot find experimental results supporting this.
3. In Page 7, the authors say that they decide not to use the gain from 'testing with augmentations' in Table 1 and Table 4. In Table 2, I guess the Total column is the sum of the other columns for B=128, ..., 1024. Therefore, the last row, B=16384, also has the AugTest (+0.8) in the Total result (6.7%). I guess in Table 4, the test ACC for B=16384, 36.9%, is derived by adding the total improvement 6.7% to the baseline 30.2%. Therefore, I am not sure if the authors forget to remove the improvement from AugTest.
4. In Page 9, in the last paragraph, the authors believe that the exponential increase in the privacy budget $\varepsilon$ as the noise level $\sigma$ decreases. I guess this is from Figure 2, but the x-axis for $\sigma$ is log-scale. Therefore, it is not easy to identify whether there is a linear or exponential relationship between $\varepsilon$ and $\sigma$.

By the way, there are duplicate references (Li et al, 2021a and 2021b).

The evidence to be evaluated is:De et al. (2022) recently introduced Augmentation Multiplicity (AugMul), which averages the gradients from different augmented versions of every sample before clipping and leads to improved performance on CIFAR-10.

For instance, reproducing the previous SOTA on ImageNet of De et al. (2022) under \(\varepsilon=8\) necessitates a \(4\)-day run using \(32\) A100 GPUs, while the non-private SOTA can be reproduced in a few hours with the same hardware (Goyal et al., 2017).

In Figure 4, we perform this analysis on CIFAR-10 and ImageNet.

We leverage our efficient TAN training strategy and obtain new state-of-the-art results on ImageNet for \(\varepsilon=8\) (Table 1).

The ImageNet dataset (Deng et al., 2009; Russakovsky et al., 2014) contains \(1.2\) million images partitioned into 1000 categories.

On Imagenet, we compare Vision Transformers (ViTs) (Dosovitskiy et al., 2020), Residual Neural Networks (ResNets) (He et al., 2016) and Normalizer-Free ResNets (NF-ResNets) (Brock et al., 2021b).

In Section 4.2.1, we use our simulated training with constant TAN to find optimal hyperparameters at low compute and improve performance for reference privacy parameters \((B_{\mathrm{ref}},\sigma_{\mathrm{ref}},S)=(16384,2.5,72\mathrm{k})\). In Section 4.2.2, we find optimal privacy parameters at constant TAN by changing the number of steps \(S\), leading to a new state of the art on ImageNet for \(\varepsilon=8\) (Table 1).

We perform one run with these optimal parameters at \(B=16384\) which satisfies a privacy budget of \(\varepsilon=8\).

We have experimented with different architectures (ViTs, ResNets, NFResnets) and optimizers (DP-Adam, DP-AdamW, DP-SGD) (see Appendix B for details). Our best results are obtained using a NFResnet-50 and DP-SGD with constant learning rate, which differs from standard practice in non-private training.

Given that \(\sigma>2\), we stay in the _almost_ constant privacy regime (see Figure 2): we indeed observe \(\varepsilon\approx\varepsilon_{\mathrm{TAN}}\) in Table 4.

Overall, we improved performance by \(9\%\) when training from scratch on ImageNet with DP-SGD under \(\varepsilon=8\). We compare to our reproduction of the previous SOTA of De et al. (2022) at \(30.2\%\) (compared to the results reported in the original paper (\(32.4\%\)), we still gain \(7\%\) of accuracy).

We show similar results for ImageNet in Appendix A.

We further show that scaling batch size with noise level using TAN allows for ultra-efficient hyper-parameter search and demonstrate the power of this paradigm by establishing a new state of the art for DP training on Imagenet.

Unlocking high-accuracy differentially private image classification through scale, 2022.

We show in Table 6 that similarly to the experiments in CIFAR-10, doubling the training data on ImageNet improves the accuracy while diving \(\varepsilon\) by \(2\).

We compare training on half of the ImageNet training set (\(N=600k\)) and the entire training set (\(N=1.2M\)).

In particular, we noticed that DP-SGD without momentum is always optimal, even with ViTs, and that NF-ResNets-50 performed the best.

Input: Table 1: Top-1 test accuracy when training on ImageNet from scratch using DP-SGD. We train a NF-ResNet-50 with σ = 2.5, hyper-parameters of Table 2 and (B,S) = (32768, 18k) (Table 4).

Input: Table 4: Privacy parameter search. We use the optimal parameters described in Section 4.2.1 with σ = 2.5 for one expensive run and compare it with our optimal result

Input: Table 6: Impact of adding more data on ImageNet. The “Simulated Gain” column corresponds to the accuracy gain we observe when simulating at lower compute using our scaling strategy for B = 256. The “Gain” column corresponds to the real gain at B = 16384.

Input: Figure 2: Privacy budget ε as a function of the noise level σ with η constant. On both figures, each curve corresponds to a different number of steps S, and each point on the curve is computed at a sampling rate q such that η is constant. On the left, we use η = 0.13 (resulting in εTAN = 1 in Equation 4). On the right, we use η = 0.95 (εTAN = 8). We observe a “privacy wall” imposing σ ≥ 0.5 for meaningful level of privacy budget ε, and σ ≥ 2 for constant ε ≈ εTAN.

Input: Figure 4: Test accuracies at constant ηstep = Bref/( √ 2Nσref ) and S are (log) linearly decreasing with B. Dashed lines are computed using a linear regression on the crosses. Shaded areas correspond to 3 std confidence intervals. (Left) CIFAR-10 with 16-4-WideResNet for Sref steps. Each curve corresponds to a different value of ηstep. (Right) ImageNet with NF-ResNet-50 with various numbers of steps. The scaling law holds for various training configurations.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, I find that the claim of a "+9 points gain in accuracy for a privacy budget ε=8" is largely supported by the data but requires some clarification:

Table 1 shows their method achieved 39.2% accuracy compared to their reproduction of De et al.'s work at 30.2% (a 9% gain). However, compared to De et al.'s original reported result of 32.4%, the gain is 6.8%. The authors acknowledge this difference in their evidence, stating "compared to the results reported in the original paper (32.4%), we still gain 7% of accuracy."

While the claim is technically accurate based on their reproduction, it would be more precise to mention both comparisons for complete transparency.

<score>2</score>

Justification:
1. The claim is supported by empirical evidence in Table 1 and explicit statements in the text
2. The authors acknowledge the difference between their reproduction and original paper results
3. The methodology is well-documented with specific architectural choices and hyperparameters
4. While slightly imprecise by not mentioning both comparison points, it doesn't significantly overstate the results
5. The review comments don't dispute the accuracy claims, though they raise some methodological concerns about privacy guarantees


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We apply the proposed method on CIFAR-10 and ImageNet and, in particular, strongly improve the state-of-the-art on ImageNet with a \(+9\) points gain in accuracy for a privacy budget \(\varepsilon=8\).

The review comment to be evaluated is:

Strengths:
1) Integrating the constraints of DP-SGD with optimization theory, in order to reliably predict good hyper-parameter choices, is one of the most important problems that needs to be solved in order to make differentially private deep learning practical. I'm pleased to see the authors tackle this difficult and important problem.
2) The metric proposed by the authors is intuitive and it appears to provide some helpful intuition.
3) The authors achieve strong performance when training from scratch on ImageNet with DP-SGD.

Weaknesses:
1) If TAN were an accurate measure of optimizability, then we should expect the train accuracy to be roughly constant at fixed TAN. However in practice, the authors find in figure 1 that the test accuracy decays log-linearly at fixed TAN as the batch size rises. I feel that the current paper is quite confusing because this discrepancy is not explicitly stated anywhere. I'd encourage the authors to make this explicit, and to explore in more detail why this discrepancy arises (eg comparing test vs train performance).

2) As discussed above, Figure 1 shows that TAN over-estimates the performance of large batch sizes. It would be nice to also see a figure at fixed batch size, sweeping the number of steps S, to see whether TAN under/over-estimates the performance of small/large step budgets.

3) As a minor point, note that on some truly sensitive data we cannot train non-private models (even internally). It would be good to acknowledge that the hyper-parameter transfer process described here cannot be used in these cases.

4) For many of the experiments, key hyper-parameters are lifted from prior work (eg step budget, target batch size). Is it not possible to directly infer good settings for these hyper-parameters using the TAN framework?

5) The paper essentially only considers ImageNet (with some very limited experiments on CIFAR-10). Since very few groups have run experiments at this scale it is not clear how challenging the baseline from De et al. is in practice.

The evidence to be evaluated is:De et al. (2022) recently introduced Augmentation Multiplicity (AugMul), which averages the gradients from different augmented versions of every sample before clipping and leads to improved performance on CIFAR-10.

For instance, reproducing the previous SOTA on ImageNet of De et al. (2022) under \(\varepsilon=8\) necessitates a \(4\)-day run using \(32\) A100 GPUs, while the non-private SOTA can be reproduced in a few hours with the same hardware (Goyal et al., 2017).

In Figure 4, we perform this analysis on CIFAR-10 and ImageNet.

We leverage our efficient TAN training strategy and obtain new state-of-the-art results on ImageNet for \(\varepsilon=8\) (Table 1).

The ImageNet dataset (Deng et al., 2009; Russakovsky et al., 2014) contains \(1.2\) million images partitioned into 1000 categories.

On Imagenet, we compare Vision Transformers (ViTs) (Dosovitskiy et al., 2020), Residual Neural Networks (ResNets) (He et al., 2016) and Normalizer-Free ResNets (NF-ResNets) (Brock et al., 2021b).

In Section 4.2.1, we use our simulated training with constant TAN to find optimal hyperparameters at low compute and improve performance for reference privacy parameters \((B_{\mathrm{ref}},\sigma_{\mathrm{ref}},S)=(16384,2.5,72\mathrm{k})\). In Section 4.2.2, we find optimal privacy parameters at constant TAN by changing the number of steps \(S\), leading to a new state of the art on ImageNet for \(\varepsilon=8\) (Table 1).

We perform one run with these optimal parameters at \(B=16384\) which satisfies a privacy budget of \(\varepsilon=8\).

We have experimented with different architectures (ViTs, ResNets, NFResnets) and optimizers (DP-Adam, DP-AdamW, DP-SGD) (see Appendix B for details). Our best results are obtained using a NFResnet-50 and DP-SGD with constant learning rate, which differs from standard practice in non-private training.

Given that \(\sigma>2\), we stay in the _almost_ constant privacy regime (see Figure 2): we indeed observe \(\varepsilon\approx\varepsilon_{\mathrm{TAN}}\) in Table 4.

Overall, we improved performance by \(9\%\) when training from scratch on ImageNet with DP-SGD under \(\varepsilon=8\). We compare to our reproduction of the previous SOTA of De et al. (2022) at \(30.2\%\) (compared to the results reported in the original paper (\(32.4\%\)), we still gain \(7\%\) of accuracy).

We show similar results for ImageNet in Appendix A.

We further show that scaling batch size with noise level using TAN allows for ultra-efficient hyper-parameter search and demonstrate the power of this paradigm by establishing a new state of the art for DP training on Imagenet.

Unlocking high-accuracy differentially private image classification through scale, 2022.

We show in Table 6 that similarly to the experiments in CIFAR-10, doubling the training data on ImageNet improves the accuracy while diving \(\varepsilon\) by \(2\).

We compare training on half of the ImageNet training set (\(N=600k\)) and the entire training set (\(N=1.2M\)).

In particular, we noticed that DP-SGD without momentum is always optimal, even with ViTs, and that NF-ResNets-50 performed the best.

Input: Table 1: Top-1 test accuracy when training on ImageNet from scratch using DP-SGD. We train a NF-ResNet-50 with σ = 2.5, hyper-parameters of Table 2 and (B,S) = (32768, 18k) (Table 4).

Input: Table 4: Privacy parameter search. We use the optimal parameters described in Section 4.2.1 with σ = 2.5 for one expensive run and compare it with our optimal result

Input: Table 6: Impact of adding more data on ImageNet. The “Simulated Gain” column corresponds to the accuracy gain we observe when simulating at lower compute using our scaling strategy for B = 256. The “Gain” column corresponds to the real gain at B = 16384.

Input: Figure 2: Privacy budget ε as a function of the noise level σ with η constant. On both figures, each curve corresponds to a different number of steps S, and each point on the curve is computed at a sampling rate q such that η is constant. On the left, we use η = 0.13 (resulting in εTAN = 1 in Equation 4). On the right, we use η = 0.95 (εTAN = 8). We observe a “privacy wall” imposing σ ≥ 0.5 for meaningful level of privacy budget ε, and σ ≥ 2 for constant ε ≈ εTAN.

Input: Figure 4: Test accuracies at constant ηstep = Bref/( √ 2Nσref ) and S are (log) linearly decreasing with B. Dashed lines are computed using a linear regression on the crosses. Shaded areas correspond to 3 std confidence intervals. (Left) CIFAR-10 with 16-4-WideResNet for Sref steps. Each curve corresponds to a different value of ηstep. (Right) ImageNet with NF-ResNet-50 with various numbers of steps. The scaling law holds for various training configurations.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
1. The claim states they "strongly improve the state-of-the-art on ImageNet with a +9 points gain in accuracy" for ε=8. The evidence from Table 1 shows their method achieved 39.2% accuracy compared to their reproduction of De et al.'s work at 30.2% (a 9% improvement) and De et al.'s original reported result of 32.4% (a 6.8% improvement). While the numerical improvement is accurate when compared to their reproduction, the claim could be more precise by acknowledging the smaller gain relative to the original published baseline. The evidence supports a substantial improvement but the wording "strongly improve" may be slightly overstated given the full context.

<score>2</score>

Justification:
- The claim's core numerical statement is supported by empirical evidence in Table 1
- The evidence shows comprehensive experimentation with different architectures and parameters
- The improvement is significant but varies depending on which baseline is used (9% vs 6.8%)
- The review comments acknowledge the strong performance but raise some concerns about limited experimental scope and baseline challenge level
- The term "strongly improve" could be more measured, but overall the claim stays reasonably close to what the evidence demonstrates


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

These results are important because segmentation with DL makes photogrammetry for limb scanning feasible on a large clinical scale.

The evidence to be evaluated is:The development of digital cameras and the accompanying transition to digital photogrammetry has led to technologies for the reconstruction of 3D models using photogrammetric algorithms Linder (2016). Digital photogrammetry has been used successfully to reconstruct patient anatomy in many medical contexts such as cranial deformation scanning Barbero-Garcia et al. (2017, 2020, 2021), facial scanning Ross et al. (2018); Nightingale et al. (2020, 2021), and amputated limb scanning R. B. Taqriban et al. (2019); Cabrera et al. (2020); Ismail et al. (2020); De Vivo Nicoloso et al. (2021).

Close range photogrammetric approaches have been proven to have accuracy comparable to clinical gold standard technologies Ross et al. (2018).

For this reason, recent research has focused on various methods for automating photogrammetric workflows for anatomical scanning Barbero-Garcia et al. (2020); Cabrera et al. (2020).

Automated approaches have focused their attention on this acquisition stage to ensure completeness of the results Nocerino et al. (2017) with recent advances incorporating machine learning for landmark detection Barbero-Garcia et al. (2021). Still, automation of photogrammetric image processing (specifically image segmentation) remains a large problem Cabrera et al. (2021). Automating this image segmentation step could dramatically increase the speed of photogrammetric workflows for medical imaging, improving the clinical viability. Medical image segmentation plays an essential role in modern clinical practice by facilitating computer aided diagnoses and making patient anatomical structures clear Wang et al. (2022).

Recent research has shown great success in using transfer learning with the DeepLabv3 architecture for medical imaging segmentation Roy et al. (2020). We performed transfer learning on a pre-trained resnet-101 based DeepLabv3 model downloaded from the Pytorch Model Zoo Paszke et al. (2019). This model was pre-trained on the COCO train2017 data set comprising 200,000 images and 80 object categories Lin et al. (2015). Our fine-tuning dataset consisted of 806 manually labeled images of limbs with transtibial amputations.

We captured 24 limb scans to test the applicability of the neural network model for image segmentation. 12 scans were performed with a single transtibial amputee wearing a standard liner then 12 additional scans were performed with the transtibial amputee wearing a colored sock over their standard liner. Scans were performed utilizing the Lim(b)itless smartphone application Cabrera et al. (2020) on a Motorola Moto G7 (MSRP $299, Camera - 12 MP, f/1.8, 1/2.8") see Fig. 1. The scanning procedure involved making two revolutions (one clockwise, one counterclockwise) around the outside of the limb while keeping the limb in frame of the application. Scans were taken at a constant radius of 30 cm from the limb, similar to Barbero-Garcia et al. (2018), at a rotation rate of two revolutions per minute. This application automatically initiates a capture sequence at 10 degree intervals with each capture sequence including a burst of 3 photographs.

The remaining 22 scans, totaling 6312 images, were fed into the fine-tuned neural network for segmentation. Segmentation time using the neural network averaged 890 s \(\pm\) 111 s, Table 1. 3D limb models were rendered from these segmented images using Agisoft Metashape, Fig. 2.

After 10 epochs of training, the fine tuned neural network showed excellent performance in segmenting images of transtibial amputations, see Fig. 4.

However, the testing IoU remains largely constant, with a value of 79.9% after 10 epochs of training. This IoU is comparable with values reported by Chen et al. (2017b) which reached peak mIoU values of 81.3% on the Cityscapes test set and 85.7% on the PASCAL VOC 2012 test set with the DeepLabv3 architecture.

Increasing the number of images in the training dataset is feasible, but would require scanning additional persons with limb loss and manual segmentation. As with other forms of medical imaging, limited data quantities and expert annotations present hurdles for deep learning techniques Wang et al. (2018). On the other hand, reducing the image downsampling could improve the model performance without requiring additional annotations.

The average MAE values obtained are excellent (0.57 mm \(\pm\) 0.63 mm) and comparable with the sub-millimeter MAE accuracy reported by Barbero-Garcia et al. (2020).

However the use of the colored sock improved tie point detection and image alignment leading to a successful render rate of 90.9 % for the colored sock vs 72.7% for the bare liner.

Failure was primarily due to failed photo alignment which led to incomplete and/or distorted models.

This is important because poor surface finish can be directly tied to the quality of image segmentation, with better segmentation leading to smoother finishes.

Even without Laplacian smoothing, this model shows remarkable rendering accuracy in comparison with the ground truth. Since the RMSE errors are systematically impacted by limb flexure, MAE is the more appropriate measure of accuracy in these scans. For the purposes of rendering residual limb models, sub-millimeter accuracy presents little meaningful advantage to clinicians and practitioners. Unlike the skulls studied by Barbero-Garcia et al. (2017, 2020), amputated limbs undergo constant shape and deformation changes due to effects of donning-doffing, muscle contractions, interstitial fluid movements etc.

The advances in photogrammetric limb scanning resulting from automation of the segmentation step could have potentially significant consequences. Notably, automatic segmentation decreases the amount of human time and effort required to render photogrammetric models. As mentioned previously in the methods, a limb scan captured via the Lim(b)ittless smartphone application captures at least 216 images. Manual segmentation of those images would take (at minimum) 1 minute per image leading to a segmentation time of 12,960 seconds. The fine-tuned network trained in this paper would only take 675 seconds to perform this task based on the segmentation rate reported in Table 1.

Evaluating the end to end process using the rate values in Table 1 reveals that the model rendering workflow is 4.88x faster overall. Although the process bottleneck shifts from segmentation rate to rendering rate, the only step requiring significant human input is the scanning stage which takes less than 1 minute on average, Table 1. Models from the updated photogrammetric workflow could be used in a variety of contexts, since there was no significant geometrical difference between ground truth models and models rendered from images segmented by the fine-tuned neural network. Most notably, photogrammetric models built from smartphone scans can facilitate a host of low-cost telemedicine solutions. As one example, the limb models could be utilized as the first stage of digital prosthetic socket rectification. This would enable patients to be able to be scanned outside of a clinical environment (eg. at home) and have a prosthesis fabricated digitally and shipped to them.

Photogrammetric limb scanning could also facilitate long-term and large scale studies of limb shape and size fluctuations over time. Being able to capture data outside of a clinical environment could help clinicians to increase the number of data points and the frequency of measurements beyond what is currently possible.

This research sought to address limitations in photogrammetric workflows by automating image segmentation via a fine-tuned deep learning model.

After 10 epochs of transfer learning, the fine-tuned deep learning model was able to achieve an IoU of 79.9% on the testing training set.

Applying this fine-tuned model to segment a dataset of 22 scans containing 6312 images showed that there was no significant geometric difference compared to scans rendered from manually segmented images.

Using the fine-tuned model increased the rate of image segmentation by 19.2x as well as sped up the entire photogrammetric workflow by a factor of 4.88x.

This fine-tuned deep learning model can help facilitate large scale image processing for telemedicine applications. Using smartphones to acquire 3D scans of amputated limbs, clinicians could monitor patients, conduct large scale research studies, and even build prostheses remotely. This could potentially increase the standard of care for persons with limb loss who do not live near a clinic by matching patients with clinicians seeking to expand their geographic service area. \begin{table}
\begin{tabular}{l l} \hline
**Number of Images** & 287 \(\pm\) 29 \\
**Scan Time (s)** & 52.4 \(\pm\) 6.1 \\
**Segmentation Time (s)** & 890 \(\pm\) 111 \\
**Render Time (s)** & 3430 \(\pm\) 850 \\
**Scan Rate (images/s)** & 5.54 \(\pm\) 0.74 \\
**Segmentation Rate (images/s)** & 0.32 \(\pm\) 0.01 \\
**Render Rate (images/s)** & 0.088 \(\pm\) 0.022 \\ \hline \end{tabular}
\end{table}
Table 1: Image acquisition and rendering pipeline statistics for smartphone scans (n = 22) used in this study.

Input: Table 1: Image acquisition and rendering pipeline statistics for smartphone scans (n = 22) used in this study.

Input: Figure 4: Limbs with transtibial amputations used in the transfer learning process. A) These limb photographs were taken in a variety of lighting conditions and with different liners. B) Manually segmented binary masks representing the ground truth. C) Masks generated by the fine-tuned neural network. D) Applying the masks from the fine-tuned neural network to the original images.

Input: Figure 5: Selected comparison of limbs in CloudCompare. In these 3D scans, it is evident that the large RMSE is due to a change in geometry from limb flexure, and is not due to segmentation or render quality. A) Ground truth model rendered from manually segmented images. B) Scan 12 - Highest RMSE (5.54 mm) evident limb flexure C) Scan 13 - Low RMSE (1.26 mm) no limb flexure D) Scan 16 - Medium RMSE (3.31 mm) evident limb flexure E) Scan 17 - Medium RMSE (2.71 mm) evident surface noise and limb flexure.

Input: Figure 1: Scanning a Transtibial Amputee using the Lim(b)itless Cell Phone Application

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate the claim "These results are important because segmentation with DL makes photogrammetry for limb scanning feasible on a large clinical scale."

The evidence supports that DL segmentation significantly improves workflow efficiency (19.2x faster segmentation, 4.88x faster overall workflow) with comparable accuracy to manual methods (IoU of 79.9%, sub-millimeter MAE). However, several limitations are noted: success rates vary (72.7-90.9% depending on conditions), problems with photo alignment persist, and the model was only tested on one participant. While DL segmentation clearly improves efficiency, claiming it makes the technology "feasible on a large clinical scale" overstates the current evidence given the limited testing and existing technical challenges.

<score>4</score>

Justification:
1. The claim overstates clinical feasibility by not acknowledging:
- Limited testing (single participant, 24 scans)
- Remaining technical challenges (photo alignment failures)
- Need for larger training datasets
- Variable success rates with different conditions

2. Supporting evidence shows promising results but at prototype stage:
- Significant speed improvements (19.2x faster segmentation)
- Comparable accuracy to manual methods
- Sub-millimeter accuracy (0.57mm ± 0.63mm MAE)

The claim would be better stated by acknowledging these limitations while highlighting the promising efficiency gains that could enable broader clinical adoption after further validation.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

These results are important because segmentation with DL makes photogrammetry for limb scanning feasible on a large clinical scale.

The review comment to be evaluated is:

**Strengths**
- The paper is easy to follow and contains nice visualisation
- The pipeline is evaluated not only for the segmentation quality but also its downstream task: 3D rendering
- There are some interesting qualitative descriptions of the results

**Weaknesses**
- There is little to no technical novelty, all employed methods have been previously presented and are simply applied to a new task
- The description of the dataset and train/validation split is slightly confusing: it is stated that the "fine- tuning dataset consisted of 806 manually labeled images" and further "we randomly select 80% of the images as our training data set and reserve the remaining 20% for validation" does this mean the same subject can be part of training and validation?
- Later on the authors state: "Of the 24 limb scans acquired in the previous step, two scans were manually segmented via Adobe Photoshop and set aside as ground truth references. The remaining 22 scans, totaling 6312 images, were fed into the fine-tuned neural network for segmentation.". Are those different subjects to the ones used for fine-tuning? Furthermore, since 24 scans correspond to 12 subjects with or without additional sock, again the question of whether the 22/2 split is done on a subject-level?
- Overall the low number (2) of held-out ground truth cases makes it somewhat  hard to trust those results, since some cherry-picking might be unavoidable
- All models converge very quickly and no data augmentation seems to be used, this could indicate some overfitting
- an additional practical problem of the pipeline seems to be failure of image alignment that caused scans 2, 7, 8 and 14 to not render properly with the automated workflow. I wonder whether the authors have attempted to use the automatic segmentation information or some other features from the fine-tuned network to improve on this?
- the inference times for automatic segmentation seem awfully slow (0.32 images/s), even when being restricted to Google Colab infrastructure measuring the feed-forward path of a DeepLabV3 with mixed precision (AMP) on a Tesla T4 should result in at least two orders of magnitude faster throughput

The evidence to be evaluated is:The development of digital cameras and the accompanying transition to digital photogrammetry has led to technologies for the reconstruction of 3D models using photogrammetric algorithms Linder (2016). Digital photogrammetry has been used successfully to reconstruct patient anatomy in many medical contexts such as cranial deformation scanning Barbero-Garcia et al. (2017, 2020, 2021), facial scanning Ross et al. (2018); Nightingale et al. (2020, 2021), and amputated limb scanning R. B. Taqriban et al. (2019); Cabrera et al. (2020); Ismail et al. (2020); De Vivo Nicoloso et al. (2021).

Close range photogrammetric approaches have been proven to have accuracy comparable to clinical gold standard technologies Ross et al. (2018).

For this reason, recent research has focused on various methods for automating photogrammetric workflows for anatomical scanning Barbero-Garcia et al. (2020); Cabrera et al. (2020).

Automated approaches have focused their attention on this acquisition stage to ensure completeness of the results Nocerino et al. (2017) with recent advances incorporating machine learning for landmark detection Barbero-Garcia et al. (2021). Still, automation of photogrammetric image processing (specifically image segmentation) remains a large problem Cabrera et al. (2021). Automating this image segmentation step could dramatically increase the speed of photogrammetric workflows for medical imaging, improving the clinical viability. Medical image segmentation plays an essential role in modern clinical practice by facilitating computer aided diagnoses and making patient anatomical structures clear Wang et al. (2022).

Recent research has shown great success in using transfer learning with the DeepLabv3 architecture for medical imaging segmentation Roy et al. (2020). We performed transfer learning on a pre-trained resnet-101 based DeepLabv3 model downloaded from the Pytorch Model Zoo Paszke et al. (2019). This model was pre-trained on the COCO train2017 data set comprising 200,000 images and 80 object categories Lin et al. (2015). Our fine-tuning dataset consisted of 806 manually labeled images of limbs with transtibial amputations.

We captured 24 limb scans to test the applicability of the neural network model for image segmentation. 12 scans were performed with a single transtibial amputee wearing a standard liner then 12 additional scans were performed with the transtibial amputee wearing a colored sock over their standard liner. Scans were performed utilizing the Lim(b)itless smartphone application Cabrera et al. (2020) on a Motorola Moto G7 (MSRP $299, Camera - 12 MP, f/1.8, 1/2.8") see Fig. 1. The scanning procedure involved making two revolutions (one clockwise, one counterclockwise) around the outside of the limb while keeping the limb in frame of the application. Scans were taken at a constant radius of 30 cm from the limb, similar to Barbero-Garcia et al. (2018), at a rotation rate of two revolutions per minute. This application automatically initiates a capture sequence at 10 degree intervals with each capture sequence including a burst of 3 photographs.

The remaining 22 scans, totaling 6312 images, were fed into the fine-tuned neural network for segmentation. Segmentation time using the neural network averaged 890 s \(\pm\) 111 s, Table 1. 3D limb models were rendered from these segmented images using Agisoft Metashape, Fig. 2.

After 10 epochs of training, the fine tuned neural network showed excellent performance in segmenting images of transtibial amputations, see Fig. 4.

However, the testing IoU remains largely constant, with a value of 79.9% after 10 epochs of training. This IoU is comparable with values reported by Chen et al. (2017b) which reached peak mIoU values of 81.3% on the Cityscapes test set and 85.7% on the PASCAL VOC 2012 test set with the DeepLabv3 architecture.

Increasing the number of images in the training dataset is feasible, but would require scanning additional persons with limb loss and manual segmentation. As with other forms of medical imaging, limited data quantities and expert annotations present hurdles for deep learning techniques Wang et al. (2018). On the other hand, reducing the image downsampling could improve the model performance without requiring additional annotations.

The average MAE values obtained are excellent (0.57 mm \(\pm\) 0.63 mm) and comparable with the sub-millimeter MAE accuracy reported by Barbero-Garcia et al. (2020).

However the use of the colored sock improved tie point detection and image alignment leading to a successful render rate of 90.9 % for the colored sock vs 72.7% for the bare liner.

Failure was primarily due to failed photo alignment which led to incomplete and/or distorted models.

This is important because poor surface finish can be directly tied to the quality of image segmentation, with better segmentation leading to smoother finishes.

Even without Laplacian smoothing, this model shows remarkable rendering accuracy in comparison with the ground truth. Since the RMSE errors are systematically impacted by limb flexure, MAE is the more appropriate measure of accuracy in these scans. For the purposes of rendering residual limb models, sub-millimeter accuracy presents little meaningful advantage to clinicians and practitioners. Unlike the skulls studied by Barbero-Garcia et al. (2017, 2020), amputated limbs undergo constant shape and deformation changes due to effects of donning-doffing, muscle contractions, interstitial fluid movements etc.

The advances in photogrammetric limb scanning resulting from automation of the segmentation step could have potentially significant consequences. Notably, automatic segmentation decreases the amount of human time and effort required to render photogrammetric models. As mentioned previously in the methods, a limb scan captured via the Lim(b)ittless smartphone application captures at least 216 images. Manual segmentation of those images would take (at minimum) 1 minute per image leading to a segmentation time of 12,960 seconds. The fine-tuned network trained in this paper would only take 675 seconds to perform this task based on the segmentation rate reported in Table 1.

Evaluating the end to end process using the rate values in Table 1 reveals that the model rendering workflow is 4.88x faster overall. Although the process bottleneck shifts from segmentation rate to rendering rate, the only step requiring significant human input is the scanning stage which takes less than 1 minute on average, Table 1. Models from the updated photogrammetric workflow could be used in a variety of contexts, since there was no significant geometrical difference between ground truth models and models rendered from images segmented by the fine-tuned neural network. Most notably, photogrammetric models built from smartphone scans can facilitate a host of low-cost telemedicine solutions. As one example, the limb models could be utilized as the first stage of digital prosthetic socket rectification. This would enable patients to be able to be scanned outside of a clinical environment (eg. at home) and have a prosthesis fabricated digitally and shipped to them.

Photogrammetric limb scanning could also facilitate long-term and large scale studies of limb shape and size fluctuations over time. Being able to capture data outside of a clinical environment could help clinicians to increase the number of data points and the frequency of measurements beyond what is currently possible.

This research sought to address limitations in photogrammetric workflows by automating image segmentation via a fine-tuned deep learning model.

After 10 epochs of transfer learning, the fine-tuned deep learning model was able to achieve an IoU of 79.9% on the testing training set.

Applying this fine-tuned model to segment a dataset of 22 scans containing 6312 images showed that there was no significant geometric difference compared to scans rendered from manually segmented images.

Using the fine-tuned model increased the rate of image segmentation by 19.2x as well as sped up the entire photogrammetric workflow by a factor of 4.88x.

This fine-tuned deep learning model can help facilitate large scale image processing for telemedicine applications. Using smartphones to acquire 3D scans of amputated limbs, clinicians could monitor patients, conduct large scale research studies, and even build prostheses remotely. This could potentially increase the standard of care for persons with limb loss who do not live near a clinic by matching patients with clinicians seeking to expand their geographic service area. \begin{table}
\begin{tabular}{l l} \hline
**Number of Images** & 287 \(\pm\) 29 \\
**Scan Time (s)** & 52.4 \(\pm\) 6.1 \\
**Segmentation Time (s)** & 890 \(\pm\) 111 \\
**Render Time (s)** & 3430 \(\pm\) 850 \\
**Scan Rate (images/s)** & 5.54 \(\pm\) 0.74 \\
**Segmentation Rate (images/s)** & 0.32 \(\pm\) 0.01 \\
**Render Rate (images/s)** & 0.088 \(\pm\) 0.022 \\ \hline \end{tabular}
\end{table}
Table 1: Image acquisition and rendering pipeline statistics for smartphone scans (n = 22) used in this study.

Input: Table 1: Image acquisition and rendering pipeline statistics for smartphone scans (n = 22) used in this study.

Input: Figure 4: Limbs with transtibial amputations used in the transfer learning process. A) These limb photographs were taken in a variety of lighting conditions and with different liners. B) Manually segmented binary masks representing the ground truth. C) Masks generated by the fine-tuned neural network. D) Applying the masks from the fine-tuned neural network to the original images.

Input: Figure 5: Selected comparison of limbs in CloudCompare. In these 3D scans, it is evident that the large RMSE is due to a change in geometry from limb flexure, and is not due to segmentation or render quality. A) Ground truth model rendered from manually segmented images. B) Scan 12 - Highest RMSE (5.54 mm) evident limb flexure C) Scan 13 - Low RMSE (1.26 mm) no limb flexure D) Scan 16 - Medium RMSE (3.31 mm) evident limb flexure E) Scan 17 - Medium RMSE (2.71 mm) evident surface noise and limb flexure.

Input: Figure 1: Scanning a Transtibial Amputee using the Lim(b)itless Cell Phone Application

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate whether the claim "These results are important because segmentation with DL makes photogrammetry for limb scanning feasible on a large clinical scale" is well-stated or overstated.

The evidence shows mixed support for this claim. While the DL segmentation achieved:
- 19.2x faster segmentation rate than manual
- 4.88x faster overall workflow
- Comparable accuracy to manual segmentation
- Sub-millimeter accuracy (MAE 0.57mm ±0.63mm)

However, several limitations exist:
- Small validation set (only 2 ground truth cases)
- Slow inference times (0.32 images/s)
- Some alignment failures (72.7-90.9% success rate)
- Limited testing on only one subject
- No data augmentation suggesting possible overfitting

The claim modestly overstates the current evidence for "large clinical scale" feasibility given these technical limitations and limited validation.

<score>3</score>