///
Most of my comments here are related to wanting more results/substantiation relative to what was claimed in the paper, not necessarily fatal flaws.
Sensitivity to noise levels: the paper claims their method (NResilient) “has better or similar RD than each baseline at all noise levels” (336-338) — unless I’m drastically misinterpreting figure 10b in the appendix, this seems to be untrue? -> leading me to also wonder at what noise levels do the fairness-utility curves (as shown in the main paper) start looking less nice.
Group membership: The model for group membership is not clear from the main body of the paper, specifically what it means for the model to handle “multiple” sensitive features, or the case where each individual item can belong to “one or more” groups. The experiments also only illustrate results where group membership is disjoint. This is addressed in the appendix (use marginals to model more complex relationships between different groups), but given that [the uncertainty of] group membership is the core motivation of the paper, it feels important for this model, and how to achieve specific instantiations of the model, to be clearer! I’d further be curious to see empirical results, though I could live without them.
Fairness metrics: Though the work as a whole claims to present a framework for a class of metrics/constraints (e.g. 158-160), empirical results are only shown (even in the appendix) for constraints/metrics that implement versions of equal representation, and thm. 3.2 is specific to equal representation (though 3.1 is more general to this class of constraints). Equal representation feels like an "easier" metric to achieve than proportional representation (similar to demographic parity being "easier" than equalized odds in classification), so I'm just curious as to what performance looks like for other constraints. Furthermore, I do wish there had been at least a nominal discussion, even in the appendix, of choice of constraint and what the consequences of different constraints would be, more than just the handwavy mention of appropriate choice being “context dependent.”
/// Typos ///
191 problem 2.5 format?
224 constraint 5 appl[ies]
249 “solve[d]”
311 “l[o]sing”
399 “works [for?] a general”
Yes - the authors are very clear about the specific scope of their work and the assumptions that their models rely on. As discussed above, I would have appreciated some discussion of these assumptions and choices (e.g. the existence of the utility matrix, choice of fairness constraint and associated values), but understand the need for a focused work.
224 constraint 5 appl[ies] 249 “solve[d]” 311 “l[o]sing” 399 “works [for?] a general” Yes - the authors are very clear about the specific scope of their work and the assumptions that their models rely on. As discussed above, I would have appreciated some discussion of these assumptions and choices (e.g. the existence of the utility matrix, choice of fairness constraint and associated values), but understand the need for a focused work.