weakness Originality
+ The paper presents a novel algorithm. The algorithms applies ideas from bilevel optimization [43, 46, 55, 26, 45] to fairness and class imbalance learning. Unlike previous work [41, 33, 16, 32, 8, 13, 50, 63], which employs fixed class balancing loss functions based on some training data statistics, the proposed method automatically guides the loss function design and learns a more optimal set of parameters for the balanced loss. Quality
+ Claims are well supported through experimental evidence.
+ The authors are careful and honest about reporting the paper's limitations.
+ The paper compares its method to recent state-of-the-art [8,50] and beats them.
- Lack of more baselines. The paper compares performance to only two baselines [8,50], and it is not clear to me why more baselines are not evaluated (e.g. [41, 33, 16, 32, 13, 63]).
- No error bars in the results. There are no error bars in the result tables, making it more challenging to assess the significance of the improvements. Clarity
+ The paper is written very well, and it is easy to follow. On the most part, the paper adequately informs the reader.
- Some clarifications about the experiments are needed. Table 1 does not have any P D A
component in Algo. 1, does this mean that no P D A
was applied in those experiments? (if P D A
was used, did the authors also apply it to the baselines?) Similarly, did experiments in Table 2 apply L A init
? And lastly, does Table 3 show results for "Algo 1. α ← Δ & l , L A init , P D A "?
- A possible inconsistency in the results but I may also be misunderstanding something (see above for my confusion). Specifically, if I am correct in understanding, the diagonal results in Table 3 show the performance for "Algo 1. α ← Δ & l , L A init , P D A
" which have a higher error rate than both "Algo 1. α ← P D A , Δ & l
" in Table 2 and "Algo 1. α ← Δ & l , L A init
" in Table 1. This suggests that, in fact, P D A and L A init
degrade performance when applied together. If my understanding is correct, this undermines the second contribution point (lines 73-77) and should be explicitly clarified in the paper. If my understanding is wrong, why are the results in the diagonal in Table 3 different from either the bottom row of Table 1 or Table 2? Significance
+ This work aims to address a timely problem of fairness-seeking optimization, and the paper will be relevant to the community.
Post rebuttal update
I have read other reviewers' comments and the authors' rebuttals, and I am inclined to keep my original score of 7 with increased confidence of 4, condioned on the additional baselines being included in Table 1 in the main paper. The rebuttal addresses the clarity issues, and the additional results show that the method is at least as good as other baselines [28][63][41] in the Cifar-10-LT setting. The performance advantage over [28][63] in the ImageNet-LT dataset is much more obvious. Although results for [41] in the ImageNet-LT setting were not provided, likely the method would also be beaten like in the Cifar-10-LT setting. I think this work is good, addressing a timely problem and it will be relevant for the NeurIPS conference.
The paper adequately and fairly outlines its limitations in a dedicated section.
- Some clarifications about the experiments are needed. Table 1 does not have any P D A component in Algo. 1, does this mean that no P D A was applied in those experiments? (if P D A was used, did the authors also apply it to the baselines?) Similarly, did experiments in Table 2 apply L A init ? And lastly, does Table 3 show results for "Algo 1. α ← Δ & l , L A init , P D A "?