Fully training group labels baselines could include more recent methods such as SGDRO (non-flat version of GDRO proposed in Goel et al.). There is a misleading sentence on page 3 when describing the Waterbirds dataset: “The bird images are then modified with either a water or land background.” There is no “modification” involved, in the dataset the birds are already placed either on water or land background (this is a minor note, but should be revisited). The main weaknesses are all in the “Experiments” section. 1) Although the hyperparameters are fixed, some ablation is still required (for example the regularisation in the second stage for the CivilComments dataset). 2) In Tables 1 and 2 it is not clear what the results are, is it correct to assume they are the mean and std. performances over three different initialization seeds? 3) The results in Tables 1 and 2 are not correct, CROIS is using the validation set for training in the second stage, not only for model selection. For the datasets considered, the validation set has the same distribution as the test set, it does not seem correct to talk about “group shift” anymore. 4) Table 2 should also include the average test accuracy to show the trade-off between average and worst-case performances when varying the size of the validation set. Also, why is the std. not reported here? 5) Table 3 shows promising results, but the best fraction “p” is a hyperparameter that should be tuned using the worst-group performance on the validation set, like the regularization term for GDRO. Its behaviour is not linear (e.g. the more the better) nor consistent across datasets. For example, in both text datasets, only one fraction produces better results than the GDRO baseline. Here, I disagree with the statement “In practice, p is not a parameter to choose (there’s no reason to throw away group labels) but rather is limited by the resources available to obtain group labels”: using p=0.5 is often worse than using p=0.3. 6) To obtain robust results, it would have been better to evaluate the methods across different splits of train-val-test, not simply different initialisation seeds.
6) To obtain robust results, it would have been better to evaluate the methods across different splits of train-val-test, not simply different initialisation seeds.