1) The nature of the contribution with respect to ECE_sweep is not clearly described in the text. Concretely, this amounts to a way to choose the number of bins using data (i.e., autotuning a hyperparameter in the estimate). While this, of course, leads to a different estimator, this is not something fundamentally different. I would much rather that the paper was upfront about the contribution. (In fact, I was pretty confused about the point the paper was making until I realised this).
2) I don't think the baseline comparisons made in the experiments are appropriate. The proposal is a method to choose the appropriate number of bins in the estimate, and should be compared to other methods to do so instead of to an arbitrary choice of number of bins as is done in section 5.2. Without this comparison, I have no way to judge if this is a good autotuning method or not. Reasonable comparisons could be, e.g., choosing b
by cross validation, or, in equal mass binning, choosing b
so that each bin has a reasonable number of samples for the error y ― k
to not be too large.
3) While the focus of the paper is on bias, it should be noted that by searching over many different bin sizes, the variance of ECE_sweep may be inflated. If this is to such an extent that the gains in bias relative to other autotuning methods are washed out, then this estimator would not be good. To judge this requires at least that the variances for ECE_sweep are reported, but these are never mentioned in the main text.
4) Choice of law in simulation in section 3, which are used to illustrate the dependence of bias on the number of bins, not aligned with the laws/curves in figure 3. Taking the latter as representative of the sort of laws and calibration curves that arise in practice, there are two issues:
4a) The pdfs of f
tend to be a lot more peaked near the end than the one explored in section 3 - this is borne out by the values of α , β
in the fits in Table 1. Beta(1.1,1) is remarkably flat compared to the curves in Fig 3.
4b) There seem to be a few different qualitative properties of the calibration curves - monotone but with a large intercept at 0; those with an inflection point in the middle; and those with the bulk lying below the y = x
line. In particular, all of them tend to have at least some region above the y = x
line. The choice of curve c 2
in section 3 doesn't completely align with any of these cases, but even if we make the case that it aligns with the third type, this leaves two qualitative behaviours unexplored.
In fact, the choice of laws is such that the error of the hard classifier that thresholds f at 1 / 2 is 26 %
. I don't think we're usually interested in the calibration of a predictor as poor as this in practice.
All of this makes me question the relevance of this simulation. Is the dependence of the bias on the number of bins as strong for the estimated laws as it is for these? Seeing the the equivalents of figs 7 and 8 for the laws from section 5 would go a long way in sorting this out.
5) Experiments: As I previously mentioned, I don't think the correct baselines are compared to. Instead of posing the method against other autotuning schemes, just one choice of the number of bins is taken. This already makes it near impossible to judge the efficacy of this method.
Despite this, even the data presented does not make a clear case for ECE_sweep. In Fig. 4 we see that the bias of EW_sweep is even worse than EW. This already means that the sweep estimate doesn't fix the issues of ECE_bin in all contexts. It is the case that EM_sweep has better bias than EM, but again, for samples large enough for the variances to be in control, it seems like these numbers are both converging to the same, so I don't see any distinct advantage when it comes to estimation. (of course, this is moot because this isn't the right comparison anyway)
Also, Fig. 5 is flawed because it compares EW and EM_sweep. It should either compare EM and EM_sweep, or EW and EW_sweep, I don't see why EW and EM_sweep are directly comparable.
Minor issues:
a) Algorithm (1) and the formula for ECE_sweep in section 4 don't compute the same thing. In algorithm (1), you find the largest b
such that the resulting y ― k
is a monotone sequence, and return the ECE_bin for this number of bins. In the formula, you maximise the ECE_bin for all b that yield a monotone y ― k
. From the preceding text, I assumed that the quantity in Algorithm (1) is intended.
b) Why is the L p
norm definition of the ECEs introduced at all? In the paper only p = 2
is used throughout. I feel like the p
just complicates things without adding much - even if you only present the L 2
definition, the fact that a generic p
can be used instead should be obvious to the audience.
c) Design considerations for ECE_sweep - it is worth noting that accuracy is not all that we want in an estimate of calibration error. For instance, one might reasonably want to add this as a regulariser when training a model in order to obtain better calibrated solutions. One issue with ECE_sweep is that it introduces a problem in that how the number of bins in the ECE_sweep estimate changes with a small change in model parameters seems very difficult to handle, which makes this a nondifferentiable loss. Broader issues of this form, and a discussion of how they may be mitigated, could lead to a more well rounded paper. Comments:
a) Exact monotonicity in the ECE_sweep proposal - I find the argument stemming from the monotonicity of the true calibration curve, and the idea to use this to nail down a maximum binning size interesting. However, why should we demand exact monotonicity in the bin heights? Each y ― k
will have noise at the scale of roughly b / n ,
(for equal mass binning with b
bins), and in my opinion, violation of monotonicity at this scale should not be penalised. Also, what if a few y ― k
s decrease but most are increasing (i.e., the sequence has a few falling regions, but the bulk is increasing)? Perhaps instead of dealing with this crudely, the error of a shape constrained estimator may serve as a better proxy.
b) Isn't the procedure for parametrically fitting the pdf of f
, and E [ Y f ( X ) ] ,
and then integrating the bias a completely different estimator for TCE of a model? In fact, if the laws are a good fit, as is claimed in section 5.1, then this plug in estimator might do well simply because the integration is exact. In fact, since the fit is parametric, this can further be automatically differentiated (if, say, f
were a DNN), and thus used to train.
c) It would be interesting to see what number of bins are ultimately adopted in the ECE_sweep computations that are performed.
Overall opinion: The lack of comparison to appropriate baselines makes it near impossible for me to judge the validity of the proposed estimator. I feel like this is a deep methodological flaw when it comes to evaluating the main proposal of the paper. This is a real pity because I quite like some of the ideas in the paper.
Due to the inability to evaluate the main contribution of the paper, i am rating it a strong reject. I'd be completely open to re-rating it if appropriate comparisons are performed, and the case for the method is properly made.
1) The nature of the contribution with respect to ECE_sweep is not clearly described in the text. Concretely, this amounts to a way to choose the number of bins using data (i.e., autotuning a hyperparameter in the estimate). While this, of course, leads to a different estimator, this is not something fundamentally different. I would much rather that the paper was upfront about the contribution. (In fact, I was pretty confused about the point the paper was making until I realised this).