weakness Originality
+ Novel setting. As far as I am aware, the paper proposes a novel setting - Few-shot Hypothesis Adaptation (FHA) - a combination of existing problems - Hypothesis Transfer Learning and the Few-Shot Domain Adaptation.
+/- Somewhat novel method. As far as I am aware, the paper also proposes a novel method - TOHAN - which is a minor adaptation of FADA [23] into the setting. The method architecture seems to be heavily inspired by FADA [23]. Unlike FADA, TOHAN generates and uses the generated intermediate domain instead of the original source domain. However, apart from that, it is not entirely clear what the technical differences are between these methods. Which line in Algorithm 1 would be different for FADA / S-FADA / T-FADA / ST-FADA?
- The relation between FHA and FSL is poorly explained, and FSL is poorly cited. I found the sentence in lines 88-89 particularly vague and poorly written. In addition, line 90 that compares TSN [37] and ProtoNets [29] ("which is relatively weaker than the former") brings little value to the paper. In my humble opinion, these methods were designed for two different problems (i.e. video action classification [37] and single image classification [29]), and it is inappropriate to compare them in this way. The authors might find the following works from FSL literature more related to their work: (Antreas et al., 2018), (Hariharan & Girshick., 2017), (Wang et al., 2018). Generally, these methods also generate/hallucinate samples from limited target-domain data and should be cited. Quality
+ Some good experiments. The paper performs a solid number of comparisons with representative baselines from HTL and FDA literature ([19] and [23], respectively) and basic fine-tuning baseline proposed by the authors.
+/- Work seems grounded in some theoretical work. However, I did not attempt to verify the proof, so I cannot comment on its correctness.
- Only marginal improvements over baselines, mostly within the error bar range. Although the authors claim the method performs better than the baselines, the error range is rather high, suggesting that the performance differences between some methods are not very significant.
- There might be a possible fundamental flaw to the claim of full data privacy. The authors claim that the TOHAN method (and therefore also the FHA problem) "strictly" protects the privacy of the source domain by using the source hypothesis rather than the source data itself (lines 245-247). However, the claim that knowledge is "completely" inaccessible may be false. For instance, Ateniese et al. (2015) have shown that it is possible to extract certain types of information from pretrained classifiers and models. In a more recent privacy analysis of deep learning, Nasr et al. (2019) suggest that even well-trained models might leak a significant amount of information about the data they were trained on. Moreover, the proposed TOHAN relies on the leaked source-domain knowledge to generator appropriate source-domain data. Therefore, it seems to me that claim that TOHAN is privacy secure is completely false.
- Lack of sufficient evidence for no source domain features in intermediate data. The authors claim that "high-level, visual and useful features of source domain are rare in the generated intermediate data (Figure 6)" (lines 243-244); however, no empirical value is provided to support this claim (i.e. what do authors mean by "rare" in this context? how rare is "rare"?). Figure 6a/b does support this claim as it shows only 4 intermediate-domain images and no examples of the original source-domain data. This is insufficient evidence to draw the conclusion in lines 243-244, and in fact, it points to the contrary. Clarity
- The paper is poorly written. The paper is not particularly easy to read. It contains many spelling and grammatical errors (see below for proposed corrections). There is a large number of acronyms (e.g. FHA, SHOT, HTL, ST+F etc..) and some confusing mathematical notation (e.g. D , D , X and X
refer to different entities) which make this work more confusing to read. The notation in the figure is inconsistent with the main text (uses X
instead of D ). Significance
- The paper presents a novel and interesting problem but could be flawed. This novel problem and method could have important consequences in the context of data privacy - however, to me, the idea seems fundamentally flawed (see my comments in the "Quality" subsection, or "Limitations And Societal Impact").
- The TOHAN improvements over baselines are mostly marginal. Although the authors claim the method performs better than the baselines, the error range is rather high, suggesting that the performance differences between some methods are not very significant and the improvements are marginal.
Spelling, Grammer, and Other Possible Mistakes.
- Line 32, grammar: "face of generation" --> "face generation"
- Figure 2 caption, grammar: "which two data come from ... " --> "where the two data points come from ... "
- Line 128, wrong word: "dependents" --> "depends"
- Line 167, redundant word: "regarding to the ..." --> "regarding the ... "
- Line 210, redundant words: "which confuses D unable to distinguish between ..." --> "which confuses D between ... "
- Line 213, wrong word: "initial" --> "initialize" References
Ateniese et al., 2015, "Hacking smart machines with smarter ones: How to extract meaningful data from machine learning classifiers", International Journal of Security and Networks, Volume 10, Issue 3
Nasr et al., 2019, "Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-box Inference Attacks against Centralized and Federated Learning," 2019 IEEE Symposium on Security and Privacy
Antoniou et al., 2018, "Data Augmentation Generative Adversarial Networks", (ICLR 2018 Workshop)
Hariharan & Girshick., 2017, "Low-shot Visual Recognition by Shrinking and Hallucinating Features", (ICCV 2017)
Wang et al., 2018, "Low-shot learning from imaginary data" (CVPR 2018) POST-REBUTTAL
After a detailed discussion with the authors, I decided to increase my original rating from 3 to 7. The initial low rating was due to initially hidden assumptions and a poorly defined scope of data privacy which are central in the paper. These have been discussed and clarified by the authors during the rebuttal. The authors also addressed my concerns regarding the novelty and source-domain leakage into the intermediate domain. The authors have agreed to improve the clarity, literature review, dampen down on the privacy claims, and include additional experiments, and I am happy to increase the rating. Although the privacy claims are now not as strong as originally claimed (e.g. the method does not guard against source-information leakage, but rather shelters individual source data points from a possible data leakage), the paper still opens up an interesting area of research and presents a novel method that will likely attract attention in the community.
The authors claim that their method allows for full privacy of the source domain by relying on a well-trained source domain classifier. However, this seems to be based on the false assumption that the source domain classifier does not leak any private information. Recent studies (for example, Nasr et al. (2019) and Ateniese et al. (2015)) suggest that there might be a significant amount of source-domain information that pre-trained models leak. Moreover, TOHAN relies on leaking information from the source domain to generate realistic/compatible intermediate-domain data. This completely invalidates one of the main claims of the paper that private information is protected.
- Only marginal improvements over baselines, mostly within the error bar range. Although the authors claim the method performs better than the baselines, the error range is rather high, suggesting that the performance differences between some methods are not very significant.