Question:
You are a highly experienced, conscientious, and fair academic reviewer, please help me review this paper. The review should be organized into nine sections:
1. Summary: A summary of the paper in 100-150 words.
2. Strengths/Weaknesses/Questions: The Strengths/Weaknesses/Questions of paper, which should be listed in bullet points, with each point supported by specific examples from the article where possible.
3. Soundness/Contribution/Presentation: Rate the paper's Soundness/Contribution/Presentation, and match this score to the corresponding description from the list below and provide the result. The possible scores and their descriptions are:
    1 poor
    2 fair
    3 good
    4 excellent
4. Rating: Give this paper an appropriate rating, match this rating to the corresponding description from the list below and provide the result. The possible Ratings and their descriptions are:
    1 strong reject
    2 reject, significant issues present
    3 reject, not good enough
    4 possibly reject, but has redeeming facets
    5 marginally below the acceptance threshold
    6 marginally above the acceptance threshold
    7 accept, but needs minor improvements
    8 accept, good paper
    9 strong accept, excellent work
    10 strong accept, should be highlighted at the conference
5. Paper Decision: It must include the Decision itself(Accept or Reject) and the reasons for this decision, based on the criteria of originality, methodological soundness, significance of results, and clarity and logic of presentation.

Here is the template for a review format, you must follow this format to output your review result:
**Summary:**
Summary content

**Strengths:**
- Strength 1
- Strength 2
- ...

**Weaknesses:**
- Weakness 1
- Weakness 2
- ...

**Questions:**
- Question 1
- Question 2
- ...

**Soundness:**
Soundness result

**Presentation:**
Presentation result

**Contribution:**
Contribution result

**Rating:**
Rating result

**Paper Decision:**
- Decision: Accept/Reject
- Reasons: reasons content

Please ensure your feedback is objective and constructive. The paper is as follows:

# Permutation Decision Trees using Structural Impurity

Anonymous Author(s)

Affiliation

Address

email

###### Abstract

Decision Tree is a well understood Machine Learning model that is based on minimizing impurities in the internal nodes. The most common impurity measures are _Shannon entropy_ and _Gini impurity_. These impurity measures are insensitive to the order of training data and hence the final tree obtained is invariant to a permutation of the data. This leads to a serious limitation in modeling data instances that have order dependencies. In this work, we use _Effort-To-Compress_ (ETC) - a complexity measure, for the first time, as an impurity measure. Unlike Shannon entropy and Gini impurity, structural impurity based on ETC is able to capture order dependencies in the data, thus obtaining potentially different decision trees for different permutation of the same data instances (_Permutation Decision Trees_). We then introduce the notion of _Permutation Bagging_ achieved using permutation decision trees without the need for random feature selection and sub-sampling. We compare the performance of the proposed permutation bagged decision trees with Random Forest. Our model does not assume independent and identical distribution of data instances. Potential applications include scenarios where a temporal order is present in the data instances.

## 1 Introduction

The assumptions in Machine Learning (ML) models play a crucial role in interpretability, reproducibility, and generalizability. One common assumption is that the dataset is independent and identically distributed (iid). However, in reality, this assumption may not always hold true, as human learning often involves connecting new information with what was previously observed. Psychological theories such as Primacy and Recency Effects [1], Serial Position Effect, and Frame Effect suggest that the order in which data is presented can impact decision-making processes. In this work, we have devised a learning algorithm that exhibits sensitivity to the order in which data is shuffled. This unique characteristic imparts our proposed model with decision boundaries or decision functions that rely on the specific arrangement of training data.

In our research, we introduce the novel use of 'Effort to Compress' (ETC) as an impurity function for Decision Trees, marking the first instance of its application in Machine Learning. ETC effectively measures the effort required for lossless compression of an object through a predetermined lossless compression algorithm [2]. ETC was initially introduced in [3] as a measure of complexity for timeseries analysis, aiming to overcome the limitations of entropy-based complexity measures. It is worth noting that the concept of complexity lacks a singular, universally accepted definition. In [2], complexity was explored from different perspectives, including the effort-to-describe (Shanon entropy, Lempel-Ziv complexity), effort-to-compress (ETC complexity), and degree-of-order (Subsymmetry). The same paper highlighted the superior performance of ETC in distinguishing between periodic and chaotic timeseries. Moreover, ETC has played a pivotal role in the development of an interventional causality testing method called Compression-Complexity-Causality (CCC) [4]. The effectiveness CCC has been tested in various causality discovery applications [5; 6; 7; 8]. ETChas demonstrated good performance when applied to short and noisy time series data, leading to its utilization in diverse fields such as investigating cardiovascular dynamics [9], conducting cognitive research [10], and analysis of musical compositions [11]. The same is not the case with entropy based methods.

In this research, we present a new application of ETC in the field of Machine Learning, offering a fresh perspective on its ability to capture structural impurity. Leveraging this insight, we introduce a decision tree classifier that maximizes the ETC gain. It is crucial to highlight that Shannon entropy and Gini impurity fall short in capturing structural impurity, resulting in an impurity measure that disregards the data's underlying structure (in terms of order). The utilization of ETC as an impurity measure provides the distinct advantage of generating different decision trees for various permutations of data instances. Consequently, this approach frees us from the need to adhere strictly to the i.i.d. assumption commonly employed in Machine Learning. Thus, by simply permuting data instances, we can develop a Permutation Decision Forest.

The paper is structured as follows: Section 2 introduces the Proposed Method, Section 3 presents the Experiments and Results, Section 4 discusses the Limitations of the research, and Section 5 provides the concluding remarks and outlines the future work.

## 2 Proposed Method

In this section, we establish the concept of structural impurity and subsequently present an illustrative example to aid in comprehending the functionality of ETC.

_Definition:_ Structural impurity for a sequence \(S=s_{0},s_{1},\ldots,s_{n}\), where \(s_{i}\in\{0,1,\ldots,K\}\), and \(K\in\mathbf{Z}^{+}\) is the the extent of irregularity in the sequence \(S\).

We will now illustrate how ETC serves as a measure of structural impurity. The formal definition of ETC is the effort required for lossless compression of an object using a predefined lossless compression algorithm. The specific algorithm employed to compute ETC is known as Non-sequential Recursive Pair Substitution (NSRPS). NSRPS was initially proposed by Ebeling [12] in 1980 and has since undergone improvements [13], ultimately proving to be an optimal choice [14]. Notably, NSRPS has been extensively utilized to estimate the entropy of written English [15]. The algorithm is briefly discussed below: Let's consider the sequence \(S=00011\) to demonstrate the iterative steps of the algorithm. In each iteration, we identify the pair of symbols with the highest frequency and replace all non-overlapping instances of that pair with a new symbol. In the case of sequence \(S\), the pair with the maximum occurrence is \(00\). We substitute all occurrences of \(00\) with a new symbol, let's say \(2\), resulting in the transformed sequence \(2011\). We continue applying the algorithm iteratively. The sequence \(2011\) is further modified to become \(311\), where the pair \(20\) is replaced by \(3\). Then, the sequence \(311\) is transformed into \(41\) by replacing \(31\) with \(4\). Finally, the sequence \(41\) is substituted with \(5\). At this point, the algorithm terminates as the stopping criterion is achieved when the sequence becomes homogeneous. ETC, as defined in [3], represents the count of iterations needed for the NSRPS algorithm to attain a homogeneous sequence.

We consider the following three sequence and compute the ETC:

Referring to Table 1, we observe that for sequence A, the ETC, Shannon Entropy, and Gini impurity all have a value of zero. This outcome arises from the fact that the sequence is homogeneous, devoid of any impurity. Conversely, for sequences B, C, D, and E, the Shannon entropy and Gini impurity remain constant, while ETC varies based on the structural characteristics of each sequence. Having shown that the ETC captures the structural impurity of a sequence, we now define _ETC Gain_. ETC

\begin{table}
\begin{tabular}{|c|c|c|c|c|} \hline
**Sequence ID** & **Sequence** & **ETC** & **Entropy** & **Gini Impurity** \\ \hline A & 111111 & 0 & 0 & 0 \\ \hline B & 121212 & 1 & 1 & 0.5 \\ \hline C & 222111 & 5 & 1 & 0.5 \\ \hline D & 122112 & 4 & 1 & 0.5 \\ \hline E & 211122 & 5 & 1 & 0.5 \\ \hline \end{tabular}
\end{table}
Table 1: Comparison of ETC with Shannon entropy, and Gini impurity for various binary sequences.

gain is the reduction in ETC caused by partioning the data instances according to a particular attribute of the dataset. Consider the decision tree structure provided in Figure 1.

The ETC Gain for the chosen parent attribute of the tree is defined as follows:

\[ETC\_Gain=ETC(Parent)-[w_{Left\_Child}\_ETC(Left\_Child)+w_{Right\_Child}\_ETC( Right\_Child)], \tag{1}\]

where \(w_{Left\_Child}\) and \(w_{Right\_Child}\) are the weights associated to left child and right child respectively. The formula for ETC Gain, as given in equation 1, bears resemblance to information gain. The key distinction lies in the use of ETC instead of Shannon entropy in the calculation. We now provide the different steps in the _Permutation Decision Tree_ algorithm.

1. Step 1: Choose an attribute to be the root node and create branches corresponding to each possible value of the attribute.
2. Step 2: Evaluate the quality of the split using ETC gain.
3. Step 3: Repeat Step 1 and Step 2 for all other attributes, recording the quality of split based on ETC gain.
4. Step 4: Select the partial tree with the highest ETC gain as a measure of quality.
5. Step 5: Iterate Steps 1 to 4 for each child node of the selected partial tree.
6. Step 6: If all instances at a node share the same classification (homogeneous class), stop developing that part of the tree.

## 3 Experiments and Results

To showcase the effectiveness of the ETC impurity measure in capturing the underlying structural dependencies within the data and subsequently generating distinct decision trees for different permutations of input data, we utilize the following illustrative toy example.

The visual representation of the toy example provided in Table 2 is represented in Figure 2

Figure 1: Decision Tree structure with a parent node and two child node (Left Child and Right Child).

\begin{table}
\begin{tabular}{|c|c|c|c|} \hline
**Serial No.** & \(f_{1}\) & \(f_{2}\) & **label** \\ \hline
1 & 1 & 1 & 2 \\ \hline
2 & 1 & 2 & 2 \\ \hline
3 & 1 & 3 & 2 \\ \hline
4 & 2 & 1 & 2 \\ \hline
5 & 2 & 2 & 2 \\ \hline
6 & 2 & 3 & 2 \\ \hline
7 & 4 & 1 & 2 \\ \hline
8 & 4 & 2 & 2 \\ \hline
9 & 4 & 3 & 1 \\ \hline
10 & 4 & 4 & 1 \\ \hline
11 & 5 & 1 & 1 \\ \hline
12 & 5 & 2 & 1 \\ \hline
13 & 5 & 3 & 1 \\ \hline
14 & 5 & 4 & 1 \\ \hline \end{tabular}
\end{table}
Table 2: Toy example dataset to showcase the potential of a permuted decision tree generated with a novel impurity measure known as “Effort-To-Compress”.

We consider the following permtation of dataset, for each of the below permutation we get distinct decision tree.

* Serial No. Permutation A: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14. Figure 3 represents the corresponding decision tree.
* Serial No Permutation B: 14, 3, 10, 12, 2, 4, 5, 11, 9, 8, 7, 1, 6, 13. Figure 4 represents the corresponding decision tree.
* Serial No Permutation C: 13, 11, 8, 12, 7, 6, 4, 14, 10, 5, 2, 3, 1, 9. Figure 5 represents the corresponding decision tree.

Figure 4: Decision Tree using ETC for Serial No. Permutation B.

Figure 3: Decision using ETC for Serial No. Permutation A.

Figure 2: A visual representation of the toy example provided in Table 2.

* Serial No Permutation D: 3, 2, 13, 10, 11, 1, 4, 7, 6, 9, 8, 14, 5, 12. Figure 6 represents the corresponding decision tree.
The variability in decision trees obtained from different permutations of data instances (Figures 3, 4, 5, 6,and 7) can be attributed to the ETC impurity function's ability to capture the structural impurity of labels, which sets it apart from Shannon entropy and Gini impurity. Table 3 highlights the sensitivity of ETC to permutation, contrasting with the insensitivity of Shannon entropy and Gini impurity towards data instance permutations. In the given toy example, there are six class-1 data instances and eight class-2 data instances. Since Shannon entropy and Gini impurity are probability-based methods, they remain invariant to label permutation. This sensitivity of ETC to the structural pattern of the label motivates us to develop a bagging algorithm namely Permutation Decision Forest.

### Permutation Decision Forest

Permutation decision forest distinguishes itself from Random Forest by eliminating the need for random subsampling of data and feature selection in order to generate distinct decision trees. Instead, permutation decision forest achieves tree diversity through permutation of the data instances. The accompanying architecture diagram provided in Figure 8 illustrates the operational flow of permutation decision forest.

The architecture diagram depicted in Figure 8 showcases the workflow of the Permutation Decision Forest, illustrating its functioning. Consisting of individual permutation decision trees, each tree operates on a permuted dataset to construct a classification model, collectively forming a strong

Figure 8: Architecture diagram of Permutation Decision Forest. Permutation Decision Forest, which comprises multiple individual permutation decision trees. The results from each permutation decision tree are then fed into a voting scheme to determine the final predicted label.

\begin{table}
\begin{tabular}{|l|c|c|c|} \hline \multirow{2}{*}{Label Impurity} & Shannon Entropy (bits) & Gini Impurity & Effort-To-Compress \\ \hline Permutation A & 0.985 & 0.490 & 7 \\ \hline Permutation B & 0.985 & 0.490 & 8 \\ \hline Permutation C & 0.985 & 0.490 & 9 \\ \hline Permutation D & 0.985 & 0.490 & 9 \\ \hline Permutation E & 0.985 & 0.490 & 8 \\ \hline \end{tabular}
\end{table}
Table 3: Comparison between Shannon Entropy, Gini Impurity and Effort to Compress for the toy example.

classifier. The outcomes of the permutation decision trees are then fed into a voting scheme, where the final predicted label is determined by majority votes. Notably, the key distinction between the Permutation Decision Forest and Random Forest lies in their approaches to obtaining distinct decision trees. While Random Forest relies on random subsampling and feature selection, Permutation Decision Forest achieves diversity through permutation of the input data. This distinction is significant as random feature selection in Random Forest may result in information loss, which is avoided in Permutation Decision Forest.

### Performance comparison between Random Forest and Permutation Decision Forest

We evaluate the performance of the proposed method with the following datasets: _Iris_[16], _Breast Cancer Wisconsin_[17], _Haberman's Survival_[18], _Ionosphere_[19], _Seeds_[20], _Wine_[21]. For all datasets, we allocate 80% of the data for training and reserve the remaining 20% for testing. Table 4 provides a comparison of the hyperparameters used and the test data performance as measured by macro F1-score.

In our experimental evaluations, we observed that the proposed method surpasses Random Forest (F1-score = 0.56) solely for the Haberman's survival dataset (F1-score = 0.621). However, for the Seeds dataset, the permutation decision forest yields comparable performance to Random Forest (F1-score = 0.877). In the remaining cases, Random Forest outperforms the proposed method.

## 4 Limitations

The current framework demonstrates that the proposed method, permutation decision forest, achieves slightly lower classification scores compared to random forest. We acknowledge this limitation and aim to address it in our future work by conducting thorough testing on diverse publicly available datasets. It is important to note that permutation decision trees offer an advantage when dealing with datasets that possess a temporal order in the generation of data instances. In such scenarios, permutation decision trees can effectively capture the specific temporal ordering within the dataset. However, this use case has not been showcased in our present work. In our future endeavors, we intend to incorporate and explore this aspect more comprehensively.

## 5 Conclusion

In this research, we present a unique approach that unveils the interpretation of the _Effort-to-Compress_ (ETC) complexity measure as an impurity measure capable of capturing structural impurity in timeseries data. Building upon this insight, we incorporate ETC into Decision Trees, resulting in the introduction of the innovative _Permutation Decision Tree_. By leveraging permutation techniques, Permutation Decision Tree facilitates the generation of distinct decision trees for varying permutations of data instances. Inspired by this, we further develop a bagging method known as _Permutation Decision Forest_, which harnesses the power of permutation decision trees. Moving forward, we are committed to subjecting our proposed method to rigorous testing using diverse publicly available datasets. Additionally, we envision the application of our method in detecting adversarial attacks.

\begin{table}
\begin{tabular}{|l|l|l|l|l|l|l|} \hline \multicolumn{1}{|c|}{\multirow{2}{*}{**Dataset**}} & \multicolumn{3}{c|}{**Random Forest**} & \multicolumn{3}{c|}{**Permutation**} \\ \multicolumn{1}{|c|}{} & \multicolumn{3}{c|}{**Decision Forest**} \\ \hline  & **F1-score** & **n\_estimators** & **max\_depth** & **F1-score** & **n\_estimators** & **max\_depth** \\ \hline
**Iris** & 1.000 & 100 & 3 & 0.931 & 31 & 10 \\ \hline
**Breast Cancer Wisconsin** & 0.918 & 1000 & 9 & 0.893 & 5 & 10 \\ \hline
**Haberman’s Survival** & 0.560 & 1 & 3 & 0.621 & 5 & 10 \\ \hline
**Ionosphere** & 0.980 & 1000 & 4 & 0.910 & 5 & 5 \\ \hline
**Seeds** & 0.877 & 100 & 5 & 0.877 & 11 & 10 \\ \hline
**Wine** & 0.960 & 10 & 4 & 0.943 & 5 & 10 \\ \hline \end{tabular}
\end{table}
Table 4: Performance comparison of Permutation Decision Forest with Random Forest for various publicly available datasets

## References

* [1] Jamie Murphy, Charles Hofacker, and Richard Mizerski. Primacy and recency effects on clicking behavior. _Journal of computer-mediated communication_, 11(2):522-535, 2006.
* [2] Nithin Nagaraj and Karthi Balasubramanian. Three perspectives on complexity: entropy, compression, subsymmetry. _The European Physical Journal Special Topics_, 226:3251-3272, 2017.
* [3] Nithin Nagaraj, Karthi Balasubramanian, and Suitrth Dey. A new complexity measure for time series analysis and classification. _The European Physical Journal Special Topics_, 222(3-4):847-860, 2013.
* [4] Aditi Kathpalia and Nithin Nagaraj. Data-based intervention approach for complexity-causality measure. _PeerJ Computer Science_, 5:e196, 2019.
* [5] SY Pranay and Nithin Nagaraj. Causal discovery using compression-complexity measures. _Journal of Biomedical Informatics_, 117:103724, 2021.
* [6] Vikram Ramanan, Nikhil A Baraiya, and SR Chakravarthy. Detection and identification of nature of mutual synchronization for low-and high-frequency non-premixed syngas combustion dynamics. _Nonlinear Dynamics_, 108(2):1357-1370, 2022.
* [7] Aditi Kathpalia, Pouya Manshour, and Milan Palus. Compression complexity with ordinal patterns for robust causal inference in irregularly sampled time series. _Scientific Reports_, 12(1):1-14, 2022.
* [8] Harikrishnan NB, Aditi Kathpalia, and Nithin Nagaraj. Causality preserving chaotic transformation and classification using neurochaos learning. _Advances in Neural Information Processing Systems_, 35:2046-2058, 2022.
* [9] Karthi Balasubramanian, K Harikumar, Nithin Nagaraj, and Sandipan Pati. Vagus nerve stimulation modulates complexity of heart rate variability differently during sleep and wakefulness. _Annals of Indian Academy of Neurology_, 20(4):403, 2017.
* [10] Vasilios K Kimiskidis, Christos Koutlis, Alkiviadis Tsimpiris, Reetta Kalviainen, Philippe Ryvlin, and Dimitris Kugiumtzis. Transcranial magnetic stimulation combined with eeg reveals covert states of elevated excitability in the human epileptic brain. _International journal of neural systems_, 25(05):1550018, 2015.
* [11] Abhishek Nandekar, Preeth Khona, MB Rajani, Anindya Sinha, and Nithin Nagaraj. Causal analysis of carnatic music compositions. In _2021 IEEE International Conference on Electronics, Computing and Communication Technologies (CONECCT)_, pages 1-6. IEEE, 2021.
* [12] Werner Ebeling and Miguel A Jimenez-Montano. On grammars, complexity, and information measures of biological macromolecules. _Mathematical Biosciences_, 52(1-2):53-71, 1980.
* [13] Miguel A Jimenez-Montano, Werner Ebeling, Thomas Pohl, and Paul E Rapp. Entropy and complexity of finite sequences as fluctuating quantities. _Biosystems_, 64(1-3):23-32, 2002.
* [14] Dario Benedetto, Emanuele Caglioti, and Davide Gabrielli. Non-sequential recursive pair substitution: some rigorous results. _Journal of Statistical Mechanics: Theory and Experiment_, 2006(09):P09011, 2006.
* [15] Peter Grassberger. Data compression and entropy estimates by non-sequential recursive pair substitution. _arXiv preprint physics/0207023_, 2002.
* [16] R. A. FISHER. The use of multiple measurements in taxonomic problems. _Annals of Eugenics_, 7(2):179-188, 1936.
* [17] W Nick Street, William H Wolberg, and Olvi L Mangasarian. Nuclear feature extraction for breast tumor diagnosis. In _Biomedical image processing and biomedical visualization_, volume 1905, pages 861-870. SPIE, 1993.

* [18] Shelby J Haberman. The analysis of residuals in cross-classified tables. _Biometrics_, pages 205-220, 1973.
* [19] Vincent G Sigillito, Simon P Wing, Larrie V Hutton, and Kile B Baker. Classification of radar returns from the ionosphere using neural networks. _Johns Hopkins APL Technical Digest_, 10(3):262-266, 1989.
* [20] Dheeru Dua and Casey Graff. UCI machine learning repository, 2017.
* [21] Michele Forina, Riccardo Leardi, Armanino C, and Sergio Lanteri. _PARVUS: An Extendable Package of Programs for Data Exploration_. 01 1998.

Response:
## Review

### Summary

The paper proposes a novel splitting criterion for decision trees based on the Effort-To-Compress (ETC) complexity measure, aiming to address order-dependent data. While the idea of using ETC is innovative, the theoretical foundation is insufficiently defined, and the experimental validation lacks depth. The authors present a permutation voting forest that generates diverse decision trees from shuffled data, but results show it performs worse than traditional random forests on several datasets. The paper attempts to explore a new research direction in handling non-i.i.d. data; however, it does not provide adequate empirical evidence to support its claims.

### Strengths

- Presents a novel approach to decision trees that considers order dependencies in data.
- Introduces Effort-To-Compress as a new impurity measure for decision trees.
- The concept of permutation voting forests allows for the creation of diverse trees from shuffled datasets.

### Weaknesses

- The theoretical justification for the proposed method is underdeveloped.
- Experimental evaluation is limited, using only a small number of datasets without cross-validation.
- The performance of the proposed method does not meet or exceed that of traditional random forests.
- There are inconsistencies in notation and unfair comparisons in experimental settings.
- Lack of formal mathematical explanation for the ETC measure and related algorithms.

### Questions

- What specific types of datasets or real-world applications justify the use of order-sensitive methods?
- How do hyperparameters impact the performance of the proposed method compared to traditional methods?
- Should the authors consider comparing their method against other approaches beyond random forests?

### Soundness

**Score:** 1

**Description:** 1 = poor; the theoretical foundations are weak, and empirical validation is inadequate.

### Presentation

**Score:** 2

**Description:** 2 = fair; while the paper is generally well-structured, there are inconsistencies and presentation issues that detract from clarity.

### Contribution

**Score:** 2

**Description:** 2 = fair; the paper introduces a novel concept but lacks sufficient validation and clarity in its significance.

### Rating

**Score:** 2

**Description:** 2 = Strong Reject; the paper exhibits major technical flaws, poor evaluation, and limited impact.

### Paper Decision

**Decision:** Reject

**Reasons:** The paper presents an interesting but poorly substantiated approach to a novel problem in decision tree analysis. It lacks a solid theoretical foundation and fails to provide convincing experimental evidence to validate its claims. The limited scope of evaluation and issues with clarity further contribute to the decision to reject.


Question:
You are a highly experienced, conscientious, and fair academic reviewer, please help me review this paper. The review should be organized into nine sections:
1. Summary: A summary of the paper in 100-150 words.
2. Strengths/Weaknesses/Questions: The Strengths/Weaknesses/Questions of paper, which should be listed in bullet points, with each point supported by specific examples from the article where possible.
3. Soundness/Contribution/Presentation: Rate the paper's Soundness/Contribution/Presentation, and match this score to the corresponding description from the list below and provide the result. The possible scores and their descriptions are:
    1 poor
    2 fair
    3 good
    4 excellent
4. Rating: Give this paper an appropriate rating, match this rating to the corresponding description from the list below and provide the result. The possible Ratings and their descriptions are:
    1 strong reject
    2 reject, significant issues present
    3 reject, not good enough
    4 possibly reject, but has redeeming facets
    5 marginally below the acceptance threshold
    6 marginally above the acceptance threshold
    7 accept, but needs minor improvements
    8 accept, good paper
    9 strong accept, excellent work
    10 strong accept, should be highlighted at the conference
5. Paper Decision: It must include the Decision itself(Accept or Reject) and the reasons for this decision, based on the criteria of originality, methodological soundness, significance of results, and clarity and logic of presentation.

Here is the template for a review format, you must follow this format to output your review result:
**Summary:**
Summary content

**Strengths:**
- Strength 1
- Strength 2
- ...

**Weaknesses:**
- Weakness 1
- Weakness 2
- ...

**Questions:**
- Question 1
- Question 2
- ...

**Soundness:**
Soundness result

**Presentation:**
Presentation result

**Contribution:**
Contribution result

**Rating:**
Rating result

**Paper Decision:**
- Decision: Accept/Reject
- Reasons: reasons content

Please ensure your feedback is objective and constructive. The paper is as follows:

# Gaussian Membership Inference Privacy

Tobias Leemann

University of Tubingen

Technical University of Munich

&Martin Pawelczyk

Harvard University

Equal contribution. Corresponding authors: tobias.leemann@uni-tuebingen.de and martin.pawelczyk.1@gmail.com.

Gjergji Kasneci

Technical University of Munich

###### Abstract

We propose a novel and practical privacy notion called \(f\)-Membership Inference Privacy (\(f\)-MIP), which explicitly considers the capabilities of realistic adversaries under the membership inference attack threat model. Consequently, \(f\)-MIP offers interpretable privacy guarantees and improved utility (e.g., better classification accuracy). In particular, we derive a parametric family of \(f\)-MIP guarantees that we refer to as \(\mu\)-Gaussian Membership Inference Privacy (\(\mu\)-GMIP) by theoretically analyzing likelihood ratio-based membership inference attacks on stochastic gradient descent (SGD). Our analysis highlights that models trained with standard SGD already offer an elementary level of MIP. Additionally, we show how \(f\)-MIP can be amplified by adding noise to gradient updates. Our analysis further yields an analytical membership inference attack that offers two distinct advantages over previous approaches. First, unlike existing state-of-the-art attacks that require training hundreds of shadow models, our attack does not require _any_ shadow model. Second, our analytical attack enables straightforward auditing of our privacy notion \(f\)-MIP. Finally, we quantify how various hyperparameters (e.g., batch size, number of model parameters) and specific data characteristics determine an attacker's ability to accurately infer a point's membership in the training set. We demonstrate the effectiveness of our method on models trained on vision and tabular datasets.

## 1 Introduction

Machine learning (ML) has seen a surge in popularity and effectiveness, leading to its widespread application across various domains. However, some of these domains, such as finance and healthcare, deal with sensitive data that cannot be publicly shared due to ethical or regulatory concerns. Therefore, ensuring data privacy becomes crucial at every stage of the ML process, including model development and deployment. In particular, the trained model itself [5, 31] or explanations computed to make the model more interpretable [29, 32] may leak information about the training data if appropriate measures are not taken. For example, this is a problem for recent generative Diffusion Models [7] and Large Language models, where the data leakage seems to be amplified by model size [6].

Differential privacy (DP) [14] is widely acknowledged as the benchmark for ensuring provable privacy in academia and industry [10]. DP utilizes randomized algorithms during training and guarantees that the output of the algorithm will not be significantly influenced by the inclusion or exclusion of any individual sample in the dataset. This provides information-theoretic protection against the maximum amount of information that an attacker can extract about any specific sample in the dataset, even when an attacker has full access to and full knowledge about the predictive model.

While DP is an appealing technique for ensuring privacy, DP's broad theoretical guarantees often come at the expense of a significant loss in utility for many ML algorithms. This utility loss cannot be further reduced by applying savvier algorithms: Recent work [26, 27] confirms that an attacker canbe implemented whose empirical capacity to differentiate between neighboring datasets \(D\) and \(D^{\prime}\) when having access to privatized models matches the theoretical upper bound. This finding suggests that to improve a model's utility, we need to take a step back and inspect the premises underlying DP. For example, previous work has shown that privacy attacks are much weaker when one imposes additional realistic restrictions on the attacker's capabilities [26].

In light of these findings, we revisit the DP threat model and identify three characteristics of an attacker that might be overly restrictive in practice. First, DP grants the attacker full control over the dataset used in training including the capacity to poison all samples in the dataset. For instance, DP's protection includes pathological cases such as an empty dataset and a dataset with a single, adversarial instance [27]. Second, in many applications, it is more likely that the attacker only has access to an API to obtain model predictions [31, 13] or to model gradients [20]. Finally, one may want to protect typical samples from the data distribution. As argued by Triastcyn & Faltings [38], it may be over-constraining to protect images of dogs in a model that is conceived and trained with images of cars.

Such more realistic attackers have been studied in the extensive literature on Membership Inference (MI) attacks (e.g., [5, 41]), where the attacker attempts to determine whether a sample from the data distribution was part of the training dataset. Under the MI threat model, Carlini et al. [5] observe that ML models with very lax (\(\epsilon>5000\)) or no (\(\epsilon=\infty\)) DP-guarantees still provide some defense against membership inference attacks [5, 41]. Hence, we hypothesize that standard ML models trained with low or no noise injection may already offer some level of protection against realistic threats such as MI, despite resulting in very large provable DP bounds.

To build a solid groundwork for our analysis, we present a hypothesis testing interpretation of MI attacks. We then derive \(f\)-Membership Inference Privacy (\(f\)-MIP), which bounds the trade-off between an MI attacker's false positive rate (i.e., FPR, type I errors) and false negative rate (i.e., FNR, type II errors) in the hypothesis testing problem by some function \(f\). We then analyze the privacy leakage of a gradient update step in stochastic gradient descent (SGD) and derive the first analytically optimal attack based on a likelihood ratio test. However, for \(f\)-MIP to cover practical scenarios, post-processing and composition operations need to be equipped with tractable privacy guarantees as well. Using \(f\)-MIP's handy composition properties, we analyze full model training via SGD and derive explicit \(f\)-MIP guarantees. We further extend our analysis by adding carefully calibrated noise to the SGD updates to show that \(f\)-MIP may be guaranteed without any noise or with less noise than the same parametric level of \(f\)-DP [11], leading to a smaller loss of utility.

Our analysis comes with a variety of novel insights: We confirm our hypothesis that, unlike for DP, no noise (\(\tau^{2}=0\)) needs to be added during SGD to guarantee \(f\)-MIP. Specifically, we prove that the trade-off curves of a single SGD step converge to the family of Gaussian trade-offs identified by Dong et al. [11] and result in the more specific \(\mu\)-Gaussian Membership Inference Privacy (\(\mu\)-GMIP). The main contributions this research offers to the literature on privacy preserving ML include:

1. **Interpretable and practical privacy notion**: We suggest the novel privacy notion of \(f\)-MIP that addresses the realistic threat of MI attacks. \(f\)-MIP considers the MI attacker's full trade-off curve between false positives and false negatives. Unlike competing notions, \(f\)-MIP offers appealing composition and post-processing properties.
2. **Comprehensive theoretical analysis**: We provide (tight) upper bounds on any attacker's ability to run successful MI attacks, i.e., we bound any MI attacker's ability to identify whether points belong to the training set when ML models are trained via gradient updates.
3. **Verification and auditing through novel attacks**: As a side product of our theoretical analysis, which leverages the Neyman-Pearson lemma, we propose a novel set of attacks for auditing privacy leakages. An important advantage of our analytical Gradient Likelihood-Ratio (GLiR) attack is its computational efficiency. Unlike existing attacks that rely on training hundreds of shadow models to approximate the likelihood ratio, our attack does not require any additional training steps.
4. **Privacy amplification through noise addition**: Finally, our analysis shows how one can use noisy SGD (also known as Differentially Private SGD [1]) to reach \(f\)-MIP while maintaining worst-case DP guarantees. Thereby our work establishes a theoretical connection between \(f\)-MIP and \(f\)-DP [11], which allows to translate an \(f\)-DP guarantee into an \(f\)-MIP guarantee and vice versa.

Related Work

**Privacy notions.** DP and its variants provide robust, information-theoretic privacy guarantees by ensuring that the probability distribution of an algorithm's output remains stable even when one sample of the input dataset is changed [14]. For instance, a DP algorithm is \(\varepsilon\)-DP if the probability of the algorithm outputting a particular subset \(E\) for a dataset \(S\) is not much higher than the probability of outputting \(E\) for a dataset \(S_{0}\) that differs from \(S\) in only one element. DP has several appealing features, such as the ability to combine DP algorithms without sacrificing guarantees.

A few recent works have proposed to carefully relax the attacker's capabilities in order to achieve higher utility from private predictions [4; 13; 17; 38]. For example, Dwork & Feldman [13] suggest the notion of "privacy-preserving prediction" to make private model predictions through an API interface. Their work focuses on PAC learning guarantees of any class of Boolean functions. Similarly, Triastoyn & Faltings [38] suggest "Bayesian DP", which is primarily based on the definition of DP, but restricts the points in which the datasets \(S\) and \(S_{0}\) may differ to those sampled from the data distribution. In contrast, Izzo et al. [17] introduces a notion based on MI attacks, where their approach guarantees that an adversary \(\mathcal{A}\) does not gain a significant advantage in terms of accuracy when distinguishing whether an element \(\mathbf{x}\) was in the training data set compared to just guessing the most likely option. However, they only constrain the accuracy of the attacker, while we argue that it is essential to bound the entire trade-off curve, particularly in the low FPR regime, to prevent certain re-identification of a few individuals [5]. Our work leverages a hypothesis testing formulation that covers the entire trade-off curve thereby offering protection also to the most vulnerable individuals. Additionally, our privacy notion maintains desirable properties such as composition and privacy amplification through subsampling, which previous notions did not consider.

**Privacy Attacks on ML Models.** Our work is also related to auditing privacy leakages through a common class of attacks called MI attacks. These attacks determine if a given instance is present in the training data of a particular model [5; 7; 9; 15; 23; 29; 30; 31; 32; 35; 36; 40; 41]. Compared to these works, our work suggests a new much stronger class of MI attacks that is analytically derived and uses information from model gradients. An important advantage of our analytically derived attack is its computational efficiency, as it eliminates the need to train any additional shadow models.

## 3 Preliminaries

The classical notion of \((\varepsilon,\delta)\)-differential privacy [14] is the current workhorse of private ML and can be described as follows: An algorithm is DP if for any two neighboring datasets \(S,S^{\prime}\) (that differ by one instance) and any subset of possible outputs, the ratio of the probabilities that the algorithm's output lies in the subset for inputs \(S,S^{\prime}\) is bounded by a constant factor. DP is a rigid guarantee, that covers _every_ pair of datasets \(S\) and \(S^{\prime}\), including pathologically crafted datasets (for instance, Nasr et al. [27] use an empty dataset) that might be unrealistic in practice. For this reason, we consider a different attack model in this work: The MI game [41]. This attack mechanism on ML models follows the goal of inferring an individual's membership in the training set of a learned ML model. We will formulate this problem using the language of hypothesis testing and trade-off functions, a concept from hypothesis testing theory [11]. We will close this section by giving several useful properties of trade-off functions which we leverage in our main theoretical results presented in Sections 4 and 5.

### Membership Inference Attacks

The overarching goal of privacy-preserving machine learning lies in protecting personal data. To this end, we will show that an alternative notion of privacy can be defined through the success of a MI attack which attempts to infer whether a given instance was present in the training set or not. Following Yeom et al. [41] we define the standard MI experiment as follows:

**Definition 3.1** (Membership Inference Experiment [41]).: _Let \(\mathcal{A}\) be an attacker, \(A\) be a learning algorithm, \(N\) be a positive integer, and \(\mathcal{D}\) be a distribution over data points \(\mathbf{x}\in D\), where the vector \(\mathbf{x}\) may also be a tuple of data and labels. The MI experiment proceeds as follows: The model and data owner \(\mathcal{O}\) samples \(S\sim\mathcal{D}_{N}\) (i.e, sample n points i.i.d. from \(\mathcal{D}\)) and trains \(A_{S}=A(S)\). They choose \(b\in\{0,1\}\) uniformly at random and draw \(\mathbf{x}^{\prime}\sim\mathcal{D}\) if \(b=0\), or \(\mathbf{x}^{\prime}\sim S\) if \(b=1\). Finally, the attacker is successful if \(\mathcal{A}(\mathbf{x}^{\prime},A_{S},N,\mathcal{D})=b\). \(\mathcal{A}\) must output either 0 or 1._We note that the membership inference threat model features several key differences to the threat model underlying DP, which are listed in Table 1. Most notably, in MI attacks, the datasets are sampled from the distribution \(\mathcal{D}\), whereas DP protects all datasets. This corresponds to granting the attacker the capacity of full dataset manipulation. Therefore, the MI attack threat model is sensible in cases where the attacker cannot manipulate the dataset through injection of malicious samples also called "canaries". This may be realistic for financial and healthcare applications, where the data is often collected from actual events (e.g., past trades) or only a handful of people (trusted hospital staff) have access to the records. In such scenarios, it might be overly restrictive to protect against worst-case canary attacks as attackers cannot freely inject arbitrary records into the training datasets. Furthermore, MI attacks are handy as a fundamental ingredient in crafting data extraction attacks [6]. Hence we expect a privacy notion based on the MI threat model to offer protection against a broader class of reconstruction attacks. Finally, being an established threat in the literature [5; 9; 31; 40; 41], MI can be audited through a variety of existing attacks.

### Membership Inference Privacy as a Hypothesis Testing Problem

While DP has been studied through the perspective of a hypothesis testing formulation for a while [3; 11; 19; 39], we adapt this route to formulate membership inference attacks. To this end, consider the following hypothesis test:

\[\text{H}_{0}:\mathbf{x}^{\prime}\notin S\text{ vs. }\text{H}_{1}:\mathbf{x}^{\prime} \in S. \tag{1}\]

Rejecting the null hypothesis corresponds to detecting the presence of the individual \(\mathbf{x}^{\prime}\) in \(S\), whereas failing to reject the null hypothesis means inferring that \(\mathbf{x}^{\prime}\) was not part of the dataset \(S\). The formulation in (1) is a natural vehicle to think about any attacker's capabilities in detecting members of a train set in terms of false positive and true positive rates. The motivation behind these measures is that the attacker wants to reliably identify the subset of data points belonging to the training set (i.e., true positives) while incurring as few false positive errors as possible [5]. In other words, the attacker wants to maximize their true positive rate at any chosen and ideally low false positive rate (e.g., 0.001). From this perspective, the formulation in (1) allows to define membership inference privacy via trade-off functions \(f\) which exactly characterize the relation of false negative rates (i.e., 1-TPR) and false positive rates that an optimal attacker can achieve.

**Definition 3.2**.: _(Trade-off function [11]) For any two probability distributions \(P\) and \(Q\) on the same space, denote the trade-off function Test\((P;Q):[0;1]\rightarrow[0;1]\)_

\[\text{Test}(P;Q)(\alpha)=\inf\left\{\text{FNR}\mid\text{FPR}=\alpha\right\}, \tag{2}\]

_where the infimum is taken over all (measurable) rejection rules ("tests") which lead to a FPR of \(\alpha\) between distributions \(P,Q\)._

\begin{table}
\begin{tabular}{c l l} \hline \hline  & f-DP threat model & f-MIP threat model (this work) \\ \hline Goal & Distinguish between \(S\) and \(S^{\prime}\) for _any_\(S,S^{\prime}\) that differ in at most one instance. & Distinguish whether \(\mathbf{x}^{\prime}\in S\) (training data set) or not. \\ \hline Dataset access & Attacker has full data access. For example, the attacker can poison or adversarially combat attack on which ML models could be trained; e.g., \(S=\{\}\) and \(S^{\prime}=\{10^{6}\}\). & Attacker has no access to the training data set; i.e., the model owner privately trains their model free of adversarially poisoned samples. \\ \hline Protected Instances & The instance in which \(S\) and \(S^{\prime}\) differ is arbitrary. This includes OOD samples and extreme outliers. & The sample \(\mathbf{x}^{\prime}\) for which membership is to be inferred is drawn from the data distribution \(\mathcal{D}\). Therefore, MI is concerned with typical samples that can occur in practice. \\ \hline Best used & When the specific attack model is unknown. Offers a form of general protection. & When dataset access (e.g. canary injection) of an attacker can be ruled out and the main attack goal lies in revealing private training data (e.g., membership inference, data reconstruction). \\ \hline Model knowledge & The attacker knows the model architecture and has full access to the model in form of its parameters, hyperparameters and its model outputs. \\ \hline \hline \end{tabular}
\end{table}
Table 1: Comparing the threat models underlying \(f\)-DP and \(f\)-MIP.

Not every function makes for a valid trade-off function. Instead, trade-off functions possess certain characteristics that are handy in their analysis.

**Definition 3.3** (Characterization of trade-off functions [11]).: _A function \(f:[0,1]\rightarrow[0,1]\) is a trade-off function if \(f\) is convex, continuous at zero, non-increasing, and \(f(r)\leq 1-r\) for \(r\in[0,1]\)._

We additionally introduce a semi-order on the space of trade-off functions to make statements on the hardness of different trade-offs in relation to each other.

**Definition 3.4** (Comparing trade-offs).: _A trade-off function \(f\) is uniformly at least as hard as another trade-off function \(g\), if \(f(r)\geq g(r)\) for all \(0\leq r\leq 1\). We write \(f\geq g\)._

If \(\mathrm{Test}(P;Q)\geq\mathrm{Test}(P^{\prime};Q^{\prime})\), testing \(P\) vs \(Q\) is uniformly at least as hard as testing \(P^{\prime}\) vs \(Q^{\prime}\). Intuitively, this means that for a given FPR \(\alpha\), the best test possible test on \((P;Q)\) will result in an equal or higher FNR than the best test on \((P^{\prime};Q^{\prime})\).

### Noisy Stochastic Gradient Descent (Noisy SGD)

Most recent large-scale ML models are trained via stochastic gradient descent (SGD). Noisy SGD (also known as DP-SGD) is a variant of classical SGD that comes with privacy guarantees. We consider the algorithm as in the work by Abadi et al. [1], which we restate for convenience in Appendix A. While its characteristics with respect to DP have been extensively studied, we take a fundamentally different perspective in this work and study the capabilities of this algorithm to protect against membership inference attacks. In summary, the algorithm consists of three fundamental steps: _gradient clipping_ (i.e., \(\mathbf{\theta}_{i}\coloneqq\mathbf{g}(\mathbf{x}_{i},y_{i})\cdot\max(1,C/\|\mathbf{g}(\mathbf{x} _{i},y_{i})\|)\) where \(\mathbf{g}(\mathbf{x}_{i},y_{i})=\nabla\mathcal{L}(\mathbf{x}_{i},y_{i})\) is the gradient with respect to the loss function \(\mathcal{L}\)), _aggregation_ (i.e., \(\mathbf{m}=\frac{1}{n}\sum_{i=1}^{n}\mathbf{\theta}_{i}\)) and _adding Gaussian noise_ (i.e., \(\tilde{\mathbf{m}}=\mathbf{m}+Y\) where \(Y\sim\mathcal{N}(\mathbf{0},\tau^{2}\mathbf{I})\) with variance parameter \(\tau^{2}\)). To obtain privacy bounds for this algorithm, we study MI attacks for means of random variables. This allows us to bound the MI vulnerability of SGD.

## 4 Navigating Between Membership Inference Privacy and DP

In this section, we formally define our privacy notion \(f\)-MIP. To this end, it will be handy to view MI attacks as hypothesis tests.

### Membership Inference Attacks from a Hypothesis Testing Perspective

Initially, we define the following distributions of the algorithm's output

\[A_{0}=A(\mathbf{X}\cup\{\mathbf{x}\})\text{ with }\mathbf{X}\sim\mathcal{D}^{n-1},\mathbf{x} \sim\mathcal{D}\text{ and }A_{1}(\mathbf{x}^{\prime})=A(\mathbf{X}\cup\{\mathbf{x}^{\prime}\}) \text{ with }\mathbf{X}\sim\mathcal{D}^{n-1}, \tag{3}\]

where we denote other randomly sampled instances that go into the algorithm by \(\mathbf{X}=\{\mathbf{x}_{1},...\mathbf{x}_{n-1}\}\). Here \(A_{0}\) represents the output distribution under the null hypothesis (\(H_{0}\)) where the sample \(\mathbf{x}^{\prime}\) is not part of the training dataset. On the other hand, \(A_{1}\) is the output distribution under the alternative hypothesis (\(H_{1}\)) where \(\mathbf{x}^{\prime}\) was part of the training dataset. The output contains randomness due to the instances drawn from the distribution \(\mathcal{D}\) and due to potential inherent randomness in \(A\).

We observe that the distribution \(A_{1}\) depends on the sample \(\mathbf{x}^{\prime}\) which is known to the attacker. The attacker will have access to samples for which \(A_{0}\) and \(A_{1}(\mathbf{x}^{\prime})\) are simpler to distinguish and others where the distinction is harder. To reason about the characteristics of such a stochastically composed test, we define a composition operator that defines an optimal test in such a setup. To obtain a global FPR of \(\alpha\), an attacker can target different FPRs \(\bar{\alpha}(\mathbf{x}^{\prime})\) for each specific test. We need to consider the optimum over all possible ways of choosing \(\bar{\alpha}(\mathbf{x}^{\prime})\), which we refer to as _test-specific FPR function_, giving rise to the following definition.

**Definition 4.1** (Stochastic composition of trade-off functions).: _Let \(\mathcal{F}\) be a family of trade-off functions, \(h:D\subset\mathbb{R}^{d}\rightarrow\mathcal{F}\) be a function that maps an instance of the data domain to a corresponding trade-off function, and \(\mathcal{D}\) be a probability distribution on \(D\). The set of valid test-specific FPR functions \(\bar{\alpha}:D\rightarrow[0,1]\) that result in a global FPR of \(\alpha\in[0,1]\) given the distribution \(\mathcal{D}\) is defined through_

\[\mathcal{E}(\alpha,\mathcal{D})=\left\{\bar{\alpha}:D\rightarrow[0,1]\ \middle| \,\mathbb{E}_{\mathbf{x}^{\prime}\sim\mathcal{D}}\left[\bar{\alpha}(\mathbf{x}^{ \prime})\right]=\alpha\right\}. \tag{4}\]_For a given test-specific FPR function, \(\bar{\alpha}\) the global false negative rate (type II error) \(\beta\) is given by_

\[\beta_{h}(\bar{\alpha})=\mathbb{E}_{\mathbf{x}^{\prime}\sim\mathcal{D}} \left[h(\mathbf{x})(\bar{\alpha}(\mathbf{x}))\right], \tag{5}\]

_where \(\bar{\alpha}(\mathbf{x})\) is the argument to the trade-off function \(h(\mathbf{x})\in\mathcal{F}\). For a global \(\alpha\in[0,1]\) the stochastic composition of these trade-functions is defined as_

\[\left(\bigotimes_{\mathbf{x}\sim\mathcal{D}}h(\mathbf{x})\right)(\alpha) =\min_{\bar{\alpha}\in\mathcal{E}(\alpha,\mathcal{D})}\left\{\beta_{h}(\bar{ \alpha})\right\}, \tag{6}\]

_(supposing the minimum exists), the smallest global false negative rate possible at a global FPR of \(\alpha\)._

This definition specifies the trade-off function \(\bigotimes_{\mathbf{x}\sim\mathcal{D}}h(\mathbf{x}):[0,1]\rightarrow[0,1]\) of a stochastic composition of several trade-offs. While it is reminiscent of the "most powerful test" (MPT) [28], there are several differences to the MPT that are important in our work. Most prominently, a straightforward construction of the MPT to MI problems does not work since the adversary does not only run one hypothesis test to guess whether one sample belongs to the training data set or not; instead, the adversary draws multiple samples and runs sample-dependent and (potentially) different hypotheses tests for each drawn sample. This is necessary due to the form of the alternative hypotheses in the formulation of the test in (3), which depends on the sample \(\mathbf{x}^{\prime}\). We therefore require a tool to compose the results from different hypothesis tests. Finally, we prove that the trade-off of the stochastic composition has the properties of a trade-off function (see App. D.1):

**Theorem 4.1** (Stochastic composition of trade-off functions).: _The stochastic composition \(\bigotimes_{\mathbf{x}\sim\mathcal{D}}h(\mathbf{x})\) of trade-off functions \(h(\mathbf{x})\) maintains the characteristics of a trade-off function, i.e., it is convex, non-increasing, \(\left(\bigotimes_{\mathbf{x}\sim\mathcal{D}}h(\mathbf{x})\right)(r)\leq 1-r\) for all \(r\in[0,1]\), and it is continuous at zero._

### \(f\)-Membership Inference Privacy (\(f\)-Mip)

This rigorous definition of the stochastic composition operator allows us to define membership inference privacy from a hypothesis testing perspective.

**Definition 4.2** (\(f\)-Membership Inference Privacy).: _Let \(f\) be a trade-off function. An algorithm1\(A:D^{n}\rightarrow\mathbb{R}^{d}\) is said to be \(f\)-membership inference private (\(f\)-MIP) with respect to a data distribution \(\mathcal{D}\) if_

Footnote 1: When using the term “algorithm”, we also include randomized mappings.

\[\bigotimes_{\mathbf{x}^{\prime}\sim\mathcal{D}}\text{Test}\left(A_{0 };A_{1}(\mathbf{x}^{\prime})\right)\geq f, \tag{7}\]

_where \(\mathbf{x}^{\prime}\sim\mathcal{D}\) and \(\bigotimes\) denotes the stochastic composition built from individual trade-off functions of the MI hypotheses tests for random draws of \(\mathbf{x}^{\prime}\)._

In this definition, both sides are functions dependent on the false positive rate \(\alpha\). A prominent special case of a trade-off function is the Gaussian trade-off, which stems from testing one-dimensional normal distributions of unit variance that are spaced apart by \(\mu\in\mathbb{R}_{\geq 0}\). Therefore, defining the following special case of \(f\)-MIP will be useful.

**Definition 4.3** (\(\mu\)-Gaussian Membership Inference Privacy).: _Let \(\Phi\) be the cumulative distribution function (CDF) of a standard normal distribution. Define \(g_{\mu}(\alpha)\coloneqq\Phi(\Phi^{-1}(1-\alpha)-\mu)\) to be the trade-off function derived from testing two Gaussians; one with mean \(0\) and one with mean \(\mu\). An algorithm \(A\) is \(\mu\)-Gaussian Membership Inference private (\(\mu\)-GMP) with privacy parameter \(\mu\) if it is \(g_{\mu}\)-MIP, i.e., it is MI private with trade-off function \(g_{\mu}\)._

**Remark 4.1**.: _DP can also be defined via the Gaussian trade-off function, which results in \(\mu\)-Gaussian Differential Privacy (\(\mu\)-GDP, [11]). While the trade-off curves for both \(\mu\)-GDP and \(\mu\)-GMP have the same parametric form, they have different interpretations: \(\mu\)-GDP describes the trade-off function that an attacker with complete knowledge (left column in Table 1) could achieve while \(\mu\)-GMP describes the trade-off function that an attacker with MI attack capability can achieve (right column in Table 1). In the next section we will quantify their connection further._

### Relating \(f\)-MIP and \(f\)-Dp

We close this section by providing first results regarding the relation between \(f\)-DP and \(f\)-MIP. As expected, \(f\)-DP is strictly stronger than \(f\)-MIP, which can be condensed in the following result:

**Theorem 4.2** (\(f\)-Dp implies \(f\)-Mip).: _Let an algorithm \(A:D^{n}\rightarrow\mathbb{R}^{d}\) be \(f\)-differentially private [11]. Then, algorithm \(A\) will also be \(f\)-membership inference private._

We proof this result in Appendix F.1. This theorem suggests one intuitive, simple and yet actionable approach to guarantee Membership Inference Privacy. This approach involves the use of DP learning algorithms such as DP-SGD [1], which train models using noised gradients. However, as we will see in the next section, using noise levels to guarantee \(f\)-DP is usually suboptimal to guarantee \(f\)-MIP.

## 5 Implementing \(f\)-MIP through Noisy SGD

We would now like to obtain a practical learning algorithm that comes with \(f\)-MIP guarantees. As the dependency between the final model parameters and the input data is usually hard to characterize, we follow the common approach and trace the information flow from the data to the model parameters through the training process of stochastic gradient descent [1, 33]. Since the gradient updates are the only path where information flows from the data into the model, it suffices to privatize this step.

### \(f\)-MIP for One Step of Noisy SGD

We start by considering a single SGD step. Following prior work [1, 33], we make the standard assumption that only the mean over the individual gradients \(\mathbf{m}=\frac{1}{n}\sum_{i=1}^{n}\mathbf{\theta}_{i}\) is used to update the model (or is published directly) where \(\mathbf{\theta}_{i}\in\mathbb{R}^{d}\) is a sample gradient. Consistent with the definition of the membership inference game, the attacker tries to predict whether a specific gradient \(\mathbf{\theta}^{\prime}\) was part of the set \(\left\{\mathbf{\theta}_{i}\right\}_{i}\) that was used to compute the model's mean gradient \(\mathbf{m}\) or not. We are interested in determining the shape of the attacker's trade-off function. For the sake of conciseness, we directly consider one step of noisy SGD (i.e., one averaging operation with additional noising, see Algorithm 2 from the Appendix), which subsumes a result for the case without noise by setting \(\tau^{2}=0\). We establish the following theorem using the Central Limit Theorem (CLT) for means of adequately large batches of \(n\) sample gradients, which is proven in Appendix E.

**Theorem 5.1** (One-step noisy SGD is \(f\)-membership inference private).: _Denote the cumulative distribution function of the non-central chi-squared distribution with \(d\) degrees of freedom and non-centrality parameter \(\gamma\) by \(F_{\chi^{2}_{d}(\gamma)}\). Let the gradients \(\mathbf{\theta}^{\prime}\in\mathbb{R}^{d}\) of the test points follow a distribution with mean \(\mathbf{\mu}\) and covariance \(\mathbf{\Sigma}\), let \(K\geq\|\mathbf{\Sigma}^{-1/2}\mathbf{\theta}^{\prime}\|_{2}^{2}\) and define \(n_{\text{effective}}=n+\frac{\tau^{2}n^{2}}{G^{2}}\). For sufficiently large batch sizes \(n\), one step of noisy SGD is \(f\)-membership inference private with trade-off function given by:_

\[\beta(\alpha)\approx 1-F_{\chi^{2}_{d}((n_{\text{effective}}-1)K)}\left( \frac{n_{\text{effective}}}{n_{\text{effective}}-1}F_{\chi^{2}_{d}(n_{\text{ effective}},K)}^{-1}(\alpha)\right). \tag{8}\]

The larger the number of parameters \(d\) and the batch size \(n\) grow, the more the trade-off curve approaches the \(\mu\)-GMIP curve, which we show next (see Figure 1).

**Corollary 5.1** (One step noisy SGD is approx. \(\mu\)-GMIP).: _For large \(d,n\), noisy SGD is approximately \(g_{\text{nsep}}\)-GMIP. In particular, \(\beta(\alpha)\approx\Phi(\Phi^{-1}(1-\alpha)-\mu_{\text{Step}})\) with privacy parameter:_

\[\mu_{\text{step}}=\frac{d+(2n_{\text{effective}}-1)K}{n_{\text{ effective}}\sqrt{2d+4n_{\text{effective}}K}}. \tag{9}\]

This result is striking in its generality as it also covers models trained without additional noise or gradient cropping (\(n_{\text{effective}}=n\) in that case). Unlike for DP, even standard models trained with non-noisy SGD offer an elementary level of MIP. Our result further explicitly quantifies four factors

Figure 1: **Trade-off function convergence. The trade-off function from Theorem 5.1 converges to the one from Corollary 5.1 where \(\tau^{2}{=}0\) and \(K{=}d\).**that lead to attack success: the batch size \(n\), the number of parameters \(d\), the strength of the noise \(\tau^{2}\) and the worst-case data-dependent _gradient susceptibility_\(\|\mathbf{\Sigma}^{-1/2}\boldsymbol{\theta}^{\prime}\|_{2}^{2}\). The closeness of the trade-off function to the diagonal, which is equivalent to the attacker randomly guessing whether a gradient \(\boldsymbol{\theta}^{\prime}\) was part of the training data or not, is majorly determined by the ratio of \(d\) to \(n\). The higher the value of \(d\) relative to \(n\), the easier it becomes for the attacker to identify training data points. Furthermore, a higher gradient susceptibility \(K\), which measures the atypicality of a gradient with respect to the gradient distribution, increases the likelihood of MI attacks succeeding in identifying training data membership. It is worth noting that if we do not restrict the gradient distribution or its support, then there might always exist gradient samples that significantly distort the mean, revealing their membership in the training dataset. This phenomenon is akin to the \(\delta\) parameter in DP, which also allows exceptions for highly improbable events.

**Remark 5.1** (Magnitude of \(\mu_{\text{Step}}\)).: _When the dimensions of the uncorrelated components in \(\mathbf{\Sigma}^{-\frac{1}{2}}\boldsymbol{\theta}^{\prime}\) are also independent, we expect \(K\) to follow a \(\chi^{2}\)-distribution with \(d\) degrees of freedom and thus \(K\in\mathcal{O}(d)\). In the standard SGD-regime (\(\tau^{2}=0\)) with \(d,n\gg 1\), we obtain \(\mu\in\mathcal{O}\big{(}\sqrt{d/n}\big{)}\)._

**Remark 5.2** (On Optimality).: _The dependency on \(d\) when \(\tau^{2}>0\) is a consequence of our intentionally broad proving strategy. Our proof approach consists of two key steps: First, we establish the optimal LRT under general gradient distributions, without adding noise or imposing any cropping constraints (See Appendix E.1). This initial step serves as the foundation for our subsequent analysis and is (1) as general as possible covering all distributions with finite variance and is (2) optimal in the sense of the Neyman-Pearson Lemma, i.e., it cannot be improved. This means that our result covers all models trained with standard SGD (\(\tau^{2}=0\) and \(C=\infty\)) and is remarkable in its generality as it is the first to suggest clear conditions when adding noise is not required to reach \(f\)-MIP. Second, we specialize our findings to cropped random variables with added noise (See Appendix E.2). This analysis could potentially be improved by considering individual gradient dimensions independently._

### Composition and Subsampling

In the previous section, we have derived the trade-off function for a single step of SGD. Since SGD is run over multiple rounds, we require an understanding of how the individual trade-off functions can be composed when a sequence of \(f\)-MIP operations is conducted, and a random subset of the entire data distribution is used as an input for the privatized algorithm. The next lemma provides such a result for \(\mu\)-GMIP and follows from a result that holds for hypotheses tests between Gaussian random variables due to Dong et al. [11] (see Appendix D.3 for details and more results).

**Lemma 5.1** (Asymptotic convergence of infinite DP-SGD).: _Let \(n\) be the batch size in SGD, and \(N\) be the entire size of the dataset. If a single SGD-Step is at least as hard as \(\mu_{\text{step}}\)-GMP with respect to the samples that were part of the batch and \(\frac{n\sqrt{I}}{N}\to c\) as \(\lim_{t\rightarrow\infty}\) (the batch size is gradually decreased), then the noisy SGD algorithm will be \(\mu\)-GMIP with_

\[\mu=\sqrt{2}c\sqrt{\exp(\mu_{\text{step}}^{2})\Phi\left(1.5\mu_{\text{step}} \right)+3\Phi\left(-0.5\mu_{\text{step}}\right)-2}. \tag{10}\]

Note that this result also provides a (loose) bound for the case where exactly \(T\) iterations are run with a batch size of \(n^{\prime}\) with \(c=\frac{n^{\prime}\sqrt{T}}{N}\) (through using \(n(t)=n^{\prime}\) if \(t\leq T,\text{ else }n(t)=\frac{n^{\prime}\sqrt{T}}{\sqrt{t}}\)). With this result in place, we can defend against MI attacks using the standard noisy SGD algorithm.

## 6 Experimental Evaluation

**Datasets and Models.** We use three datasets that were previously used in works on privacy risks of ML models [32]: The CIFAR-10 dataset which consists of 60k small images [21], the Purchase tabular classification dataset [25] and the Adult income classification dataset from the UCI machine learning repository [12]. Following prior work by Abadi et al. [1], we use a model pretrained on CIFAR-100 and finetune the last layer on CIFAR-10 using a ResNet-56 model for this task [16] where the number of fine-tuned parameters equals \(d=650\). We follow a similar strategy on the Purchase dataset, where we use a three-layer neural network. For finetuning, we use the 20 most common classes and \(d=2580\) parameters while the model is pretrained on 80 classes. On the adult dataset, we use a two-layer network with 512 random features in the first layer trained from scratch on the dataset such that \(d=1026\). We refer to Appendix C.1 for additional training details. We release our code online.2

Footnote 2: [https://github.com/tleemann/gaussian_mip](https://github.com/tleemann/gaussian_mip)

### Gradient Attacks Based on the Analytical LRT

To confirm our theoretical analysis for one step of SGD and its composition, we implement the gradient attack based on the likelihood ratio test derived in the proof of Theorem 5.1. We provide a sketch of the implementation in Algorithm 1 and additional details in Appendix C.3. An essential requirement in the construction of the empirical test is the estimation of the true gradient mean \(\mathbf{\mu}\) and the true inverse covariance matrix \(\mathbf{\Sigma}^{-1}\) since these quantities are essential parts of both the test statistic \(\hat{S}\) and the true gradient susceptibility term \(\hat{K}\) needed for the analytical attack. The attacker uses their access to the gradient distribution (which is standard for membership inference attacks [5; 29] and realistic in federated learning scenarios [20]), to estimate the distribution parameters. In practice, however, the empirical estimates of \(\hat{\mathbf{\mu}}\), \(\hat{\mathbf{\Sigma}}^{-1}\) and thus \(\hat{K}\) will be noisy and therefore we do not expect that the empirical trade-off curves match the analytical curves exactly.

Using our novel Gradient Likelihood Ratio (GLiR) attack we can audit our derived guarantees and their utility. First, we audit our one-step guarantees from Theorem 5.1. To compare the models, we adapt the batch size \(n\) such that all models reach the same level of \(\mu\)-GMIP. In Figure 1(a), we use a simulated gradient distribution with known parameters \(\mathbf{\mu},\mathbf{\Sigma}^{-1}\) and \(d\). In this case, we can estimate \(K\) accurately and observe that our bounds are tight when the distribution parameters and thus the respective gradient susceptibilities can be computed accurately. We provide additional ablation studies that gauge the approximation quality of with small values for \(d\), \(n\) and different simulated distributions in Appendix C.2. When the parameters are unknown and we have to estimate the parameters, our attacks become weaker and do not match the analytical prediction (see Figure 1(b)).

We also audit our composition guarantees. We do five SGD-steps in Figure 1(c). While there is a small gain in attack performance on the CIFAR-10 dataset (e.g., at FPR=0.25), the attack performance on the other datasets remains largely unaffected. This mismatch occurs since the theoretical analysis is based on the premise that the attacker gains access to independently sampled gradient means for each step to separate training and non-training points, but in practice we do not gain much new information as the model updates are not statistically independent and too incremental to change the gradient means significantly between two subsequent steps. Therefore, a practical attacker does not gain much additional information through performing several steps instead of one. Future work is required to model these dependencies and potentially arrive at a tighter composition result under incremental parameter updates. We provide results for additional existing membership inference attacks, for instance the recent loss-based likelihood-ratio attack by Carlini et al. [5] in Appendix C.4, which all show weaker success rates than the gradient-based attack that proved most powerful in our setting.

### Comparing Model Utility under \(\mu\)-Gdp and \(\mu\)-Gmp

Here we compare the utility under our privacy notion to the utility under differential privacy. We sample 20 different privacy levels ranging from \(\mu\in[0.4,...,50]\) and calibrate the noise in the SGD iteration to reach the desired value of \(\mu\). We can do so both for \(\mu\)-GMIP using the result in Equation (10) and using the result by Dong et al. [11; Corollary 4] for \(\mu\)-GDP, which result in the same attack success rates while \(\mu\)-GDP allows for stronger privacy threat models. Due to Theorem 4.2, we never need to add more noise for \(\mu\)-GMIP than for \(\mu\)-DP. Further details are provided in Appendix C.1. Figure 3 shows a comparison of the accuracy that the models obtain. We observe that the model under GMIP results in significantly higher accuracy for most values of \(\mu\). As \(\mu\to 0\) both privacy notions require excessive amounts of noise such that the utility decreases towards the random guessing accuracy. On the other hand, for higher values of \(\mu\), there is no need to add any noise to the gradient to obtain \(\mu\)-GMIP, allowing to obtain the full utility of the unconstrained model. This indicates that useful GMIP-bounds do not necessarily require noise. For instance, on the CIFAR-10 model, no noise is required for \(\mu\geq 0.86\) which is a reasonable privacy level [11]. Overall, these results highlight that useful and interpretable privacy guarantees can often be obtained without sacrificing utility.

## 7 Conclusion and Future Work

In the present work, we derived the general notion of \(f\)-Membership Inference Privacy (\(f\)-MIP) by taking a hypothesis testing perspective on membership inference attacks. We then studied the noisy SGD algorithm as a model-agnostic tool to implement \(f\)-Membership Inference Privacy, while maintaining Differential Privacy (DP) as a worst-case guarantee. Our analysis revealed that significantly less noise may be required to obtain \(f\)-MIP compared to DP resulting in increased utility. Future work is required to better model the dependencies when composing subsequent SGD steps which could lead to improved bounds in practice. Furthermore, our analysis shows that when the capacity of the attacker is further restricted, e.g., to API access of predictions, there remains a gap between our theoretical bounds and loss-based membership inference attacks that can be implemented for real models. More work is required to either produce more sophisticated attacks or derive theoretical bounds for even less powerful attackers to close this gap.

Figure 3: **Utility of DP versus MIP.** Model performance on three datasets across different privacy levels \(\mu\) (small \(\mu\) denotes high privacy) using the notions of \(\mu\)-Gaussian Differential Privacy (parametric form of \(f\)-DP, [11]) and \(\mu\)-Gaussian Membership Inference Privacy (parametric form of \(f\)-MIP, ours) on three datasets. GMIP usually allows for substantially increased accuracy over the corresponding GDP guarantee with the same attack success rates controlled by \(\mu\). However, the attacker under GMIP runs membership inference (MI) attacks while GDP allows for a wider set of privacy threat models. For more details on differences in the underlying threat models see Table 1.

Figure 2: **Auditing \(f\)-MIP with our gradient attack (GLiR) when \(\tau^{2}=0\). We show trade-off curves when the gradient distribution is known (a) and when the gradients are obtained from a trained model that was finetuned on various data sets (b, c). The analytical solutions are computed with a value of \(K=d\) and using the composition result for \(k\) steps in Appendix D.3 for (c).**

## References

* Abadi et al. [2016] Abadi, M., Chu, A., Goodfellow, I., McMahan, H. B., Mironov, I., Talwar, K., and Zhang, L. Deep learning with differential privacy. In _Proceedings of the 2016 ACM SIGSAC conference on computer and communications security_, pp. 308-318, 2016.
* Andrew et al. [2023] Andrew, G., Kairouz, P., Oh, S., Oprea, A., McMahan, H. B., and Suriyakumar, V. One-shot empirical privacy estimation for federated learning. _arXiv preprint arXiv:2302.03098_, 2023.
* Balle et al. [2020] Balle, B., Barthe, G., Gaboardi, M., Hsu, J., and Sato, T. Hypothesis testing interpretations and renyi differential privacy. In _International Conference on Artificial Intelligence and Statistics_, pp. 2496-2506. PMLR, 2020.
* Bassily et al. [2018] Bassily, R., Thakkar, O., and Guha Thakurta, A. Model-agnostic private learning. _Advances in Neural Information Processing Systems (NeurIPS)_, 31, 2018.
* Carlini et al. [2021] Carlini, N., Chien, S., Nasr, M., Song, S., Terzis, A., and Tramer, F. Membership inference attacks from first principles. _arXiv preprint arXiv:2112.03570_, 2021.
* Carlini et al. [2021] Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T. B., Song, D., Erlingsson, U., et al. Extracting training data from large language models. In _USENIX Security Symposium_, volume 6, 2021.
* Carlini et al. [2023] Carlini, N., Hayes, J., Nasr, M., Jagielski, M., Sehwag, V., Tramer, F., Balle, B., Ippolito, D., and Wallace, E. Extracting training data from diffusion models. _arXiv preprint arXiv:2301.13188_, 2023.
* Chen et al. [2020] Chen, D., Yu, N., Zhang, Y., and Fritz, M. Gan-leaks: A taxonomy of membership inference attacks against generative models. In Ligatti, J., Ou, X., Katz, J., and Vigna, G. (eds.), _CCS '20: 2020 ACM SIGSAC Conference on Computer and Communications Security, Virtual Event, USA, November 9-13, 2020_, pp. 343-362. ACM, 2020. doi: 10.1145/3372297.3417238. URL [https://doi.org/10.1145/3372297.3417238](https://doi.org/10.1145/3372297.3417238).
* Choquette-Choo et al. [2020] Choquette-Choo, C. A., Tramer, F., Carlini, N., and Papernot, N. Label-only membership inference attacks. In _Proceedings of the 37th International Conference on Machine Learning (ICML)_, volume abs/2007.14321, 2020.
* Cummings et al. [2023] Cummings, R., Desfontaines, D., Evans, D., Geambasu, R., Jagielski, M., Huang, Y., Kairouz, P., Kamath, G., Oh, S., Ohrimenko, O., et al. Challenges towards the next frontier in privacy. _arXiv preprint arXiv:2304.06929_, 2023.
* Dong et al. [2022] Dong, J., Roth, A., and Su, W. J. Gaussian differential privacy. _Journal of the Royal Statistical Society Series B: Statistical Methodology_, 84(1):3-37, 2022.
* Dua and Graff [2017] Dua, D. and Graff, C. UCI machine learning repository, 2017. URL [http://archive.ics.uci.edu/ml](http://archive.ics.uci.edu/ml).
* Dwork and Feldman [2018] Dwork, C. and Feldman, V. Privacy-preserving prediction. In _Conference On Learning Theory_, pp. 1693-1702. PMLR, 2018.
* Dwork et al. [2006] Dwork, C., McSherry, F., Nissim, K., and Smith, A. Calibrating noise to sensitivity in private data analysis. In _Theory of cryptography conference_, pp. 265-284. Springer, 2006.
* Haim et al. [2022] Haim, N., Vardi, G., Yehudai, G., Shamir, O., et al. Reconstructing training data from trained neural networks. In _Advances in Neural Information Processing Systems (NeurIPS)_, 2022.
* He et al. [2016] He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In _Proceedings of the IEEE conference on computer vision and pattern recognition_, pp. 770-778, 2016.
* Izzo et al. [2022] Izzo, Z., Yoon, J., Arik, S. O., and Zou, J. Provable membership inference privacy. _arXiv preprint arXiv:2211.06582_, 2022.

* [18] Jagielski, M., Ullman, J., and Oprea, A. Auditing differentially private machine learning: How private is private sgd? _Advances in Neural Information Processing Systems (NeurIPS)_, 33:22205-22216, 2020.
* [19] Kairouz, P., Oh, S., and Viswanath, P. The composition theorem for differential privacy. In _Proceedings of the 32nd International Conference on International Conference on Machine Learning (ICML)_, pp. 1376-1385, 2015.
* [20] Kairouz, P., McMahan, H. B., Avent, B., Bellet, A., Bennis, M., Bhagoji, A. N., Bonawitz, K., Charles, Z., Cormode, G., Cummings, R., et al. Advances and open problems in federated learning. _Foundations and Trends(r) in Machine Learning_, 14(1-2):1-210, 2021.
* [21] Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
* [22] Lehmann, E. L., Romano, J. P., and Casella, G. _Testing statistical hypotheses_, volume 3. Springer, 2005.
* [23] Long, Y., Bindschaedler, V., Wang, L., Bu, D., Wang, X., Tang, H., Gunter, C. A., and Chen, K. Understanding membership inferences on well-generalized learning models. _arXiv preprint arXiv:1802.04889_, 2018.
* [24] Maddock, S., Sablayrolles, A., and Stock, P. Canife: Crafting canaries for empirical privacy measurement in federated learning. In _International Conference on Learning Representations (ICLR)_, 2023.
* [25] Nasr, M., Shokri, R., and Houmansadr, A. Machine learning with membership privacy using adversarial regularization. In _Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security_, pp. 634-646, 2018.
* [26] Nasr, M., Songi, S., Thakurta, A., Papernot, N., and Carlin, N. Adversary instantiation: Lower bounds for differentially private machine learning. In _2021 IEEE Symposium on security and privacy (SP)_, pp. 866-882. IEEE, 2021.
* [27] Nasr, M., Hayes, J., Steinke, T., Balle, B., Tramer, F., Jagielski, M., Carlini, N., and Terzis, A. Tight auditing of differentially private machine learning. In _32nd USENIX Security Symposium (USENIX Security 23)_, pp. 1631-1648. USENIX Association, 2023.
* [28] Neyman, J. and Pearson, E. S. Ix. on the problem of the most efficient tests of statistical hypotheses. _Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character_, 231(694-706):289-337, 1933.
* [29] Pawelczyk, M., Lakkaraju, H., and Neel, S. On the Privacy Risks of Algorithmic Recourse. In _International Conference on Artificial Intelligence and Statistics (AISTATS)_, 2023.
* [30] Sablayrolles, A., Douze, M., Schmid, C., Ollivier, Y., and Jegou, H. White-box vs black-box: Bayes optimal strategies for membership inference. In _Proceedings of the 36th International Conference on Machine Learning (ICML)_, 2019.
* [31] Shokri, R., Stronati, M., Song, C., and Shmatikov, V. Membership inference attacks against machine learning models. In _2017 IEEE symposium on security and privacy (SP)_, pp. 3-18. IEEE, 2017.
* [32] Shokri, R., Strobel, M., and Zick, Y. On the privacy risks of model explanations. In _Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society (AIES)_, pp. 231-241, 2021.
* [33] Song, S., Chaudhuri, K., and Sarwate, A. D. Stochastic gradient descent with differentially private updates. In _2013 IEEE global conference on signal and information processing_, pp. 245-248. IEEE, 2013.
* [34] Steinke, T., Nasr, M., and Jagielski, M. Privacy auditing with one (1) training run. In _Advances in Neural Information Processing Systems (NeurIPS)_, 2023.

* [35] Tan, J., Mason, B., Javadi, H., and Baraniuk, R. Parameters or privacy: A provable tradeoff between overparameterization and membership inference. _Advances in Neural Information Processing Systems (NeurIPS)_, 35:17488-17500, 2022.
* [36] Tan, J., LeJeune, D., Mason, B., Javadi, H., and Baraniuk, R. G. A blessing of dimensionality in membership inference through regularization. In _International Conference on Artificial Intelligence and Statistics (AISTATS)_, pp. 10968-10993. PMLR, 2023.
* [37] Thudi, A., Shumailov, I., Boenisch, F., and Papernot, N. Bounding membership inference. _arXiv preprint arXiv:2202.12232_, 2022.
* [38] Triastcyn, A. and Faltings, B. Bayesian differential privacy for machine learning. In _International Conference on Machine Learning (ICML)_, pp. 9583-9592. PMLR, 2020.
* [39] Wasserman, L. and Zhou, S. A statistical framework for differential privacy. _Journal of the American Statistical Association_, 105(489):375-389, 2010.
* [40] Ye, J., Maddi, A., Murakonda, S. K., Bindschaedler, V., and Shokri, R. Enhanced membership inference attacks against machine learning models. In _Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security (CCS)_, 2022.
* [41] Yeom, S., Giacomelli, I., Fredrikson, M., and Jha, S. Privacy risk in machine learning: Analyzing the connection to overfitting. In _2018 IEEE 31st computer security foundations symposium (CSF)_, pp. 268-282. IEEE, 2018.
* [42] Zanella-Beguelin, S., Wutschitz, L., Tople, S., Salem, A., Ruhle, V., Paverd, A., Naseri, M., Kopf, B., and Jones, D. Bayesian estimation of differential privacy. In _International Conference on Machine Learning (ICML)_. PMLR, 2023.

Response:
## Review

### Summary

The paper introduces a new privacy notion called Gaussian Membership Inference Privacy (GMIP), which aims to provide a more relaxed yet practical framework compared to Differential Privacy (DP) for protecting against data membership inference attacks. It analytically explores the information leakage from training algorithms, showing how GMIP captures the trade-off in membership inference through hypothesis testing. The authors validate their theoretical findings with empirical evaluations, demonstrating improved model utility over traditional DP methods. Despite its contributions, there are concerns regarding the assumptions made in the analysis and comparisons to existing privacy definitions.

### Strengths

- The paper addresses an important problem by providing a new analytical approach to measure information leakage in training algorithms against realistic adversaries.
- It introduces a novel analytical derivation of the likelihood ratio test, assuming aggregated noisy gradients follow a multivariate Gaussian distribution.
- The GMIP bound has interesting dependencies on model and data distribution, and the paper includes detailed empirical evaluations supporting its findings.
- The proposed f-MIP method offers interpretable privacy guarantees and enhances utility when attacker capabilities are realistically constrained.

### Weaknesses

- The GMIP bound relies on several assumptions about the gradient distribution, model dimension, and dataset size, which should be clarified to avoid misleading comparisons.
- The assumption that averaged gradients follow a Gaussian distribution is questionable due to gradient clipping.
- There are concerns about the tightness of the GMIP bound compared to the standard mu-GDP bound in high-dimensional cases.
- The paper lacks a theoretical exploration of the relationship between f-MIP and f-DP, limiting the understanding of its applicability.
- The iid assumption may not be realistic in practical scenarios, potentially lowering the actual privacy guarantees.

### Questions

- What are the specific approximations and assumptions used for the GMIP bound, especially regarding the comparison in Figure 1?
- How does the GMIP bound perform in high-dimensional settings compared to the standard mu-GDP bound?
- Could the authors discuss the implications of the iid assumption on the practical applicability of their privacy guarantees?
- Is the comparison across the entire range of TPR and FPR in Figure 4 appropriate, given the focus on low FPR scenarios?

### Soundness

**Score:** 3

**Description:** Good. The theoretical foundations and derivations are solid, but some assumptions and approximations require further scrutiny.

### Presentation

**Score:** 3

**Description:** Good. The paper is well-structured and easy to follow, but could benefit from clarifying certain assumptions and their implications.

### Contribution

**Score:** 3

**Description:** Good. The paper proposes a promising framework for privacy definitions that enhances model utility, though some contributions may not be entirely novel.

### Rating

**Score:** 5

**Description:** Borderline accept: The paper is technically solid, with justifiable reasons for acceptance despite some concerns about the analysis.

### Paper Decision

**Decision:** Accept (poster)

**Reasons:** The paper introduces a significant new approach to privacy in machine learning, addressing an important area of research. While there are some weaknesses related to assumptions and comparisons, the overall contributions, soundness, and clarity of presentation outweigh these concerns, justifying acceptance as a poster.