Limited novelty of some contributions:
SMA is mostly based on related models (SWA and SWAD); it “simply” removes unnecessary hyperparameters but the principle of averaging models is unchanged. Ensembling SMAs is novel but this contribution is mostly empirical as a theoretical explanation of its success over standard ensembling is missing.
The change in the model selection procedure of SMA vs SWAD is also minor although useful in practice; it would be interesting to see if better stability with MA applies to other learning objectives (e.g. domain-invariance criterias) to better highlight this contribution.
The theoretical section simply applies the well-known bias-variance trade-off to domain generalization with minor consideration of the experimental setting (c.f. questions below). A big part of the theoretical results (Taylor expansion l198) tries to validate empirically an existing result (Izmailov2018). The novelty of this section would be improved if an explanation why EoA outperforms standard ensembling is provided, which is currently missing.
Two other weaknesses that justify my score:
Question on applicability of theoretical results to practise and unjustified statements in the theoretical section (see questions below). Missing explanation of the sucess of EoA.
Fairness of the comparison to baseline models (see limitations below). In particular, EoA and the ensemble baseline see more training data than other baselines.
Izmailov2018: Averaging wights leads to wider optima and better generalization.
The limitations highlited by the authors are interesting. Yet, several important limitations are not enough discussed / missing:
How realistic is the theoretical analysis w.r.t. experimental setup? C.f. question above.
The SoTA results are obtained with two major limitations which are not discussed. This puts into question the fairness of the comparison.
high inference cost of EoA and ensembling: 6 times higher than the baselines; this important question is not discussed by the authors.
the ensembling and EoA models see more data than baselines as there are trained on different training/validation splits. It seems normal that the results are better than using a single training/validation split. A fairer evaluation would only consider a same training/validation split.
6 times higher than the baselines; this important question is not discussed by the authors. the ensembling and EoA models see more data than baselines as there are trained on different training/validation splits. It seems normal that the results are better than using a single training/validation split. A fairer evaluation would only consider a same training/validation split.