The paper makes a few claims about accuracy and best guesses which are understandably not so common. One of the main metrics it leans on in its abstract is the achievement of 37.59% accuracy for defense. While this is done in a scientific way, the authors do not consider other avenues for testing that make more sense such as random experiments based on other work.
034 - "Unfortunately, large 035 language models tend to memorize training data", cite or explain more... 042 - "LM-based...", how do you know this?
045 - This may be better attached to the previous sentence.
045 - Figure 1 does not seem to show that the attacks were successfully able to pick the persona consistently. It shows 5 out of 14, but the truth is not given in the figure.
083 - this paragraph is really good, the explanation helps understand what is done later in the experiments 089 - "KL loss", please cite == Problem Formulation = 158 - "Casual", describe what casual means with language instead of math please. 160 - The LM shown is the negative log of the probability, while that could be a typical training phase, how do you assume it is the "casual" one?
173 - "The goal", why did you choose this goal, is there evidence that by correctly choosing a persona you are able to get more personal information?
196 - "Machine Learning as a Service" --> machine learning as a service (MLaaS) 196 - "can be directly..", are you sure? Please cite the evidence where this has happened.
== Defense Learning Strategies == 204 - "simple LM training", what is this, cite?
206 -LM = LMs?
207 - "to avoid...", sentence no longer needed 213 - 216 - The statement makes sense but do you have something to show that?
220 - "intuition...", this is good == KL Loss == - The KL divergence is introduced here but mentioned before. Also, please cite.
== MI Loss == 281 - 299 The paper takes on the idea of a game gotten from other work. This is okay but it probably should have been mentioned as the main premise for the work instead of a finding.
== Experiment == --> Experiments?
319 - experimental setting --> experimental settings == Experimental Setting == --> Experimental Settings?
- Table 1 --> Good 363 - PPL, this is not defined anywhere.
371 - Table 2, from random 0 to 37.59 is a stretch. Also, the fact that there is "no" knowledge is probably not a fair comparison. Not sure whether to leave it or not but "best guesses" makes more sense.
384 - 52x --> 52 times? This number does not seem to be a valid measurement. The method of collection and distribution assumptions are questionable.
== Ablation Study == - Good comparisons
173 - "The goal", why did you choose this goal, is there evidence that by correctly choosing a persona you are able to get more personal information?