, * Typos/Comments)
Overall, I like and value the research topic and motivation of this paper and lean positive. However, some details are not clear enough. I would update my rating depending on the authors' feedback. The details are as follows.
+ Interesting and important research problem. This paper focuses on how to obtain disentangle representations for feature-level augmentation. This topic is interesting and important, and will attract many interests of the NeurIPS community.
+ Good quality of writing and organization. Overall, the writing quality is good and the paper is well organized. It is comfortable to read this paper, although some details are not clear.
+ Comprehensive experiments. Experiments are conducted on two synthetic datasets Colored MNIST and Corrupted CIFAR-10) and two real-world datasets (BAR and Biased FFHQ).
- Relative difficulty score and generalized cross-entropy (GCE) loss. It is not clear how the relative difficulty score W ( x )
in Eq. (1) is used in the pipeline. W(x) is not mentioned again in both the overall objective functions Eq. (2) or Algorithm 1. Since readers may not be familiar with the generalized cross-entropy (GCE) loss, it is encouraged to briefly introduce the formulation and key points of the GCE loss to make this paper more self-contained.
- How bias-conflicting samples and bias-aligned samples are selected. This weakness follows the first one. It seems that the "bias-conflicting" is determined based on the relative difficulty score, but the details are missed. Also, the ablation study on how the "bias-conflicting" is determined, e.g., setting the threshold for the relative difficulty score, is encouraged to be considered and included.
- Disentanglement. It is not clear how disentanglement is guaranteed. Although "Broader Impacts and Limitations" stated that "Obtaining fully disentangled latent vectors ... a limitation", it is still important to highlight how the disentanglement is realized and guaranteed without certain bias types.
- Inference stage. It is not clear how the inference is conducted during testing. Which encoders/decoders are preserved during the test stage?
- Figure 1 is not clear. First, it seems that the two y towards L CE
are the outputs of C i
, but they are illustrated like labels rather than predictions. Second, the illustration of the re-weighting module is not clear. Does it represent Eq. (4)?
- Table 4 reported a much lower performance of "swapping" on BAR compared to the other three datasets. Is there any explanation for this, like the difference of datasets?
- Sensitivity to hyperparameters. The proposed framework consists of three important hyperparameters, ( λ dis , λ s w a p b , λ swap )
. It is not clear whether the framework is sensitive to these hyperparameters and how these hyperparameters are determined.
* (Suggestion) Illustration of backpropagation. As introduced in Line 167-168, the loss from C i
is not backpropagated to E b
. It would be clearer if this can be added in Figure 1.
* Line 280. Is "the first row and column ... respectively" a typo? It is a little confusing for me to understand this.
* Typos in Algorithm 1. Are λ dis and λ s w a p b
missed in L dis and L swap ?
* Typo in Line 209. Corrputed -> Corrupted.
============================= After rebuttal ===================================
After reading the authors' response to my questions and concerns, I would like to vote for acceptance.
The major strengths of this paper are:
The research problem, unbiased classification via learning debiased representation, is interesting and would attract the NeurIPS audience's attention.
The proposed method is simple but effective. The method is built on top of LfF [12] and further considers (1) intrinsic and bias feature disentanglement and (2) data augmentation by swapping the bias features among training samples.
The paper is clearly written and well organized.
These strengths and contributions are also pointed out by other colleague reviewers.
My main concerns were:
Unclear technical details of the GCE loss and the relative difficulty score. This concern was also shared with Reviewer 8Ai1 and iKKw. The authors' response clearly introduced the details and addressed my concern well.
Sensitivity to hyper-parameters. The authors' response provided adequate results to show the sensitivity to hyper-parameters. Other details of implementation and analysis of experimental results. The authors' responses clearly answered my questions.
Considering both strengths and the weakness, I am happy to accept this paper.
The authors have adequately addressed the limitations and potential negative societal impact of their work.
- Disentanglement. It is not clear how disentanglement is guaranteed. Although "Broader Impacts and Limitations" stated that "Obtaining fully disentangled latent vectors ... a limitation", it is still important to highlight how the disentanglement is realized and guaranteed without certain bias types.