$B_{1}B_{\texttt{Abstain}}$ terms in (14) reduce to 0. As a result, we yield
our original definition for self-consistency (3), with the possibility that
$B=B_{0}+B_{1}+B_{\texttt{Abstain}}>B_{0}+B_{1}$, if there is a nonzero number
of abstentions $B_{\texttt{Abstain}}$.

Since $B>1$ and $B_{0},B_{1},B_{\texttt{Abstain}}\geq 0$, it is always the
case that option to Abstain is at least as self-consistent as not having the
option to do so. This follows from the fact that
$B_{0}+B_{1}+B_{\texttt{Abstain}}=B\geq B_{0}+B_{1}$, which would make the
denominator in (14) greater than or equal to the corresponding method that
cannot Abstain; when subtracted from 1, this would produce a
$\hat{\texttt{SC}}$ that is no smaller than the value for the corresponding
method without that cannot Abstain.

Now, it follows that, given the choice between Abstain and predicting a label
that is in disagreement with an existing prediction label, choosing to Abstain
will always lead to higher self-consistency. This is because the cost to
Abstain is less than disagreeing, so it will always be the minimal choice that
maximizes $\hat{\texttt{SC}}$.

Error and the abstention set. It is very straightforward to see that the
_abstention set_ will generally exhibit higher than the _prediction set_. When
we ensemble and measure $\hat{\texttt{SC}}$, the exmaples that exhibit low
$\hat{\texttt{SC}}$ contain higher variance-induced error. Let us call the
size of the abstention set $U$ (which incurs error $u$), the size of the
prediction set $V$ (which incurs error $v$), and the size of the test set $T$
(which incurs error $t$). We can relate the total number of misclassified
examples as $T*t=U*u+V*v,$ with $T=U+V$. If we assume the bias and noise are
equally distributed across the test and abstention sets (this is a reasonable
assumption, on average, in our setup), then splitting off the high variance
instances from the low variance (high $\hat{\texttt{SC}}$ instances) requires
that $u>v$. The error on the abstention set necessarily has to be larger than
the error on the prediction set, in order to retain the above relationship.

##  Appendix E Additional Experimental Results and Details for Reproducibility

The code for the examples in Sections 1, 3 and 5 can be found in
https://github.com/pasta41/variance. This repository also contains necessary
and sufficient information concerning reproducibility. At the time of writing,
we use Conda to produce environments with associated package-versioning
information, so that our results can be exactly replicated and independently
verified. We also use the Scikit-Learn [47] toolkit for modeling and
optimization. More details on our choice of models and hyperparameter
optimization can be found in our code repository, cited above. In brief, we
consulted prior related work (e.g., Chen et al. [11]) and performed our own
validation for reasonable hyperparameters per model type. We keep these
settings fixed to reduce impact on our results, in order to observe in
isolation how different training data subsets impact our results.

During these early runs, we collected information on train accuracy, not just
test accuracy; while models ultimately have similar test accuracy in most
cases for the same task, they can vary significantly in terms of train
accuracy (e.g., for logistic regression, COMPAS is in the low .70s; for random
forests, it is in the mid .90s). We do not include these results for the sake
of space.

This section is organized as follows. We first present information on our
datasets, models and code, including our HDMA toolkit (Appendix E.1). We then
provide details on our setup for running experiments on our cluster (Appendix
E.2). Appendix E.3 contains more detailed information concerning the
experiments performed to produce Figures 1 and 2 in the main paper. In
Appendix E.4, we provide more details on the results presented in Section 5,
as well as additional experiments. Lastly, in ppendix E.5, we discuss
implications of these results for common fairness Abenchmarks like South
German Credit. We conclude that in many cases, without adequate attention to
error estimation, it is likely that training and post-processing a single
model for fairness on these models likely is a brittle approach to achieve
generalizable fairness (and accuracy) performance. Based on our experiments,
it seems like high variance can be a significant confounding factor when using
a small set of models to draw conclusions about performance — whether fairness
or accuracy. There is an urgent need for future work concerning
reproducibility. More specifically, our results indicate that it would be
useful to revisit key algorithmic strategies in fair classification to see how
they perform in context with more reliable expected error estimation and
variance reduction.

Note on CDF figures. We show our results in terms of the $\hat{\texttt{SC}}$
of the underlying bagged models because doing so conveys how Algorithm 1 makes
decisions to predict or abstain.151515The $\hat{\texttt{SC}}$ CDF of Algorithm
1, computed via a _third_ round of bootstrapping, has nearly all mass at
$\hat{\texttt{SC}}=1$; it is difficult to visualize. For both types of
ensembling, Algorithm 1 predicts for all examples captured by the area to the
right of the $\kappa$ reference line, and abstains for all examples on the
left.

It is also worth noting (though hopefully obvious) that our CDF plots of
$\hat{\texttt{SC}}$ are not continuous, yet we choose to plot them as
interpolated curves. This are discrete because we train a concrete number of
models (individual models or bags) — typically 101 of them — that we treat as
our approximation for $B$ when computing $\hat{\texttt{SC}}$. This means that
there are a finite number of $\kappa$-values for $\hat{\texttt{SC}}$, for
which we plot a corresponding concrete number of heights $y$ corresponding to
the cumulative proportion of the test set. In this respect, it would perhaps
be more precise to plot our curves using a step function, exemplified below
(see Appendix B.3 for the values in $\hat{{\mathbb{K}}}$):

Figure 6: Plotting $\hat{\texttt{SC}}$ with an emphasis on discrete levels
$\kappa$.

We opted not to do this for two reasons. First, plotting steps for some of our
figures, in our opinion, can make the figures more difficult to understand.
Second, in experiments for which we increase the number of models used to
estimate $\hat{\texttt{SC}}$ (e.g., Appendix E.5), we found that the curves
for 101 models were a reasonable approximation of the overall CDF. We
therefore concluded that plotting the figures without steps was worth the
clarity of presentation, with a sacrifice in correctness for the overall
takeaways that we intend with these figures.

A remark on cost. It can be considerably more computationally intensive to
train an ensemble of models to compute $\hat{\texttt{SC}}$ than to train a
handful of models and perform cross-validation, as is the standard practice in
fair classification. However, as our empirical analysis demonstrates, this
cost comes with a huge benefit: It enables us to improve self-consistency and
to root out the arbitrariness of producing predictions that are effectively
close-to-random, which is especially important in high-stakes fairness
settings [13]. Moreover, for common fair classification datasets, the
increased cost on modern hardware is relatively small; (super-) ensembling
with confidence takes under an hour to execute (Appendix E.4).

###  E.1 Hypothesis classes, datasets, and code

Models. According to a comprehensive recent survey study [25], as well as
related work like Chen et al. [11], we conclude that some of the most common
models used in fair classification are logistic regression, decision tree
classifiers, random forest classifiers, SVMs, and MLPs. We opted to include
comprehensive results for the first three, since they capture different
complexities, and therefore encode different degrees of statistical bias, that
we expected to have an impact on the underlying sources of error. We provide
some results for SVMs and MLPs, which we include in this Appendix. Since we
choose not to use stochastic optimizers to reduce the sources of randomness,
for our results, training MLPs is slower than it could be. We consistently use
a decision threshold of 0.5 (i.e., 0-1 loss) for our experiments, though our
results can easily be extended to other thresholds, as discussed in Section 3.
Depending on the dataset, we reserve between 20% and 30% of the available data
for the test set. This is consistent with standard fair classification
training settings, which we validated during our initial experiments to
explore the space (for which we also did preliminary hyperparameter
optimization, before fixing the hyperparameters for our presented
results).161616Please refer to https://github.com/pasta41/variance for more
details.

Datasets. Also according to Fabris et al. [25], the most common tasks in fair
classification are Old Adult [40], COMPAS [42], and South German Credit
[32].171717Technically, Grömping [32] is an updated and corrected version of
the dataset from 2019. These three datasets arguably serve as a _de facto_
benchmark in the community, so we felt the need to include them in the present
work. In recognition of the fact that these three datasets, however standard,
have problems, we also run experiments on $3$ tasks in the New Adult dataset,
introduced by Ding et al. [18] to replace Old Adult. We subset to the CA
(California) subset of the dataset, and run on Income, Employment, and Public
Coverage, and consider sex and race as protected attributes, which we binarize
into {Male, Female} and {White, Non-white}. These are all large-scale tasks,
at least in the domain of algorithmic fairness — on the order of hundreds of
thousands of example instances. However, the $3$ tasks do share example
instances and some features. In summary, concerning common tasks in fair
classification:

  * •

COMPAS [42]. We run on the commonly-used version of this dataset from Friedler
et al. [29], which has 6167 example instances with 404 features. The target is
to predict recidivism within 2 years ($1$ corresponding to Yes, and $0$ to
No). The protected attribute is race, binarized into “Non-white” ($0$) and
“White” ($1$) subgroups.

  * •

Old Adult [40]. We run on the commonly-used version of this dataset from
Friedler et al. [29], which has 30,162 examples with 97 features. This version
of the dataset removes instances with missing values from the original
dataset, and changes the encoding of some of the features (Kohavi [40] has
48842 example instnaces with 88 features). The target is to predict
$<\$50,000$ income ($0$) $>=\$50,000$ income ($1$). The protected attribute is
sex, binarized into “Female” ($0$) and “Male” ($1$) subgroups.

  * •

South German Credit [32]. We download the dataset from UCI181818See
https://archive.ics.uci.edu/ml/datasets/South+German+Credit+%28UPDATE%29 and
process the data ourselves. We use the provided codetable.txt to “translate”
the features from German to English. We say “translate” because the authors
took some liberties, e.g., the column converted to “credit_history” is labeled
“moral” in the German, which is not a translation. There are four categories
in the protected attribute “personal_status_sex” column, one of which ($2$) is
used for both “Male (single)” and “Female (non-single).” We therefore remove
rows with this value, and binarize the remaining three categories into
“Female” ($0$) and “Male” ($1$). What results is a dataset with 690 example
instances (of the original 1000) with 19 features. The target is “good” credit
($1$) and “bad” credit ($0$).

  * •

Taiwan Credit [57]. This task is to predict default on credit card payments
($1$) or not ($0$). There are 30,000 example instances and 24 features. The
protected attribute is binary sex. We download this dataset from UCI.191919See
https://archive.ics.uci.edu/ml/datasets/default+of+credit+card+clients.

  * •

New Adult [18]. This dataset contains millions of example instances from US
Census data, which can be used for several different targets/tasks. We select
three of them (listed below). These tasks share some features, and therefore
are not completely independent. Further, given the size of the whole dataset,
we subset to CA (California), the most populous state in the US. There are two
protected attribute columns that we use: sex, which is binarized “Female”
($0$) and “Male” ($1$) subgroups, and race, which we binarize into “Non-white”
($0$) and “White” ($1$). In future work, we would like to explore extending
our results beyond binary subgroups.

    * –

Income. This task is designed to be analogous to Old Adult [40]. As a result,
the target is to predict $<\$50,000$ income ($0$) $>=\$50,000$ income ($1$).
In the CA subset, there are 195,665 example instances with 8 features.

    * –

Employment. This task is to predict whether an individual is employed ($1$) or
not ($0$). In the CA subset, there are 378,817 example instances with 14
features.

    * –

Public Coverage. This task is to predict whether an individual is on public
health insurance ($1$) or not ($0$). In the CA subset, there are 138,554
example instances with 17 features.

####  E.1.1 The standalone HMDA tookit

In addition to the above standard tasks, we include experiments that use the
NY and TX 2017 subsets of the the Home Mortgage Data Disclosure Act (HMDA)
2007-2017 dataset [26]. These two datasets have 244,107 and 576,978 examples,
respectively, with 18 features. The HMDA datasets together contain over 140
million examples of US home mortgage loans from 2007-2017 (newer data exists,
but in a different format). We developed a toolkit, described below, to make
this dataset easy to use for classification experiments. Similar to New Adult,
we enable subsetting by US state. For the experiments in this paper, we run on
the NY (New York) and TX (Texas) 2017 subset, in order to add some geographic
diversity to complement our New Adult experiments. We additionally chose NY
and TX because they are two of the most populous states in the US, alongside
CA.202020Per the 2020 Census, the top-4-most-populous states are CA, TX, FL,
and NY [44].

The target variable, action_taken, concerning loan origination has 8 values, 2
of which we cannot meaningful conclude approval or denial decisions. They are:
Action Taken: $1$ – Loan originated, $2$ – Application approved but not
accepted, $3$ – Application denied by financial institution, $4$ – Application
withdrawn by applicant, $5$ – File closed for incompleteness, $6$ – Loan
purchased by the institution, $7$ – Preapproval request denied by financial
institution, and $8$ – Preapproval request approved but not accepted (optional
reporting). We filter out $4$ and $6$, and binarize into grant=$\\{1,2,8\\}=1$
and reject=$\\{3,5,7\\}=0$. There are three protected attributes that we
consider: sex, race, and ethnicity:

  * •

sex has 5 possible values, 2 of which correspond to categories/non-missing
values: Male – $1$ and Female – $2$. We binarize sex into $\text{F}=0$ and
$\text{M}=1$.

  * •

race has 8 possible values, 5 of which correspond to categories/ non-missing
information: $1$ – American Indian or Alaska Native, $2$ – Asian, $3$ – Black
or African American, $4$ – Native Hawaiian or Other Pacific Islander, and $5$
– White. There are 5 fields for applicant race, which model an applicant
belonging to more than one racial group. For our experiments, we only look at
the first field. When we binarize race, $\text{NW}=0$ and $\text{W}=1$.

  * •

ethnicity has 5 possible values, 2 of which correspond to categories/ non-
missing information: $1$ – Hispanic or Latino and $2$ – Not Hispanic or
Latino. We binarize ethnicity to be $\text{HL}=0$ and $\text{NHL}=1$.

After subsetting to only include examples that have values that do not
correspond to missing information, HMDA has 18 features. The NY dataset has
244,107 examples; the TX dataset has 576,978 examples, making it the largest
dataset in our experiments. As with our experiments using New Adult, we would
like to extend our results beyond binary subgroups and binary classification
in future work.

Releasing a standalone toolkit. These datasets are less-commonly used in
current algorithmic fairness literature [25]. We believe this is likely due to
the fact that the over-100-million data examples are only available in bulk
files, which are on the order of 10s of gigabytes and therefore not easily
downloadable or explorable on most personal computers. Following the example
of Ding et al. [18], one of our contributions is to pre-process all of these
datasets — all locations and years — and release them with a software toolkit.
The software engineering effort to produce this toolkit was substantial. Our
hope is that wider access to this dataset will further reduce the community’s
dependency on small (and dated) datasets. Please refer to
https://github.com/pasta41/hmda for the latest information on this standalone
software package. Our release aligns with the terms of service for this
dataset.

###  E.2 Cluster environment details

While most of the experiments run in this paper can be easily reproduced on a
modern laptop, for efficiency, we ran all of our experiments (except the one
to produce Figure 1) in a cluster environment. This enabled us to easily
execute train/test splits $n$ in parallel on different CPUs, serialize our
results, and then reconstitute and combine them to produce plots locally. Our
cluster environment runs Ubuntu 20.04 and uses Slurm v20.11.8 to manage jobs.
We ran all experiments using Anaconda3, which is why we used Conda to
reproduce environments for easy replicability.

The experiments using New Adult and HMDA rely on datasets that are (in some
cases) orders of magnitude larger than the traditional algorithmic fairness
tasks. This is one of the reasons why we recommend running on a cluster, and
therefore do not include Jupyter notebooks in our repository for these tasks.
We also limit our modeling choices to logistic regression, decision tree
classifiers, and random forest classifiers for these results due to the
expense of training on the order of thousands of models for each experiment.

###  E.3 Details on motivating examples in the main paper

This appendix provides extended results for the experiments associated in
Sections 1 and 3, which give an intuition for individual- and subgroup-level
consistency. The experimental results in the main paper are for logistic
regression. We expand the set of models we examine, and associated discussion
of how to interpret comparisons between these results.

Reproducing Figure 1. The experiment to produce this figure in Section 1 (also
shown in Appendix B.3) trains $B=10$ logistic regression models on the COMPAS
dataset (Appendix E.1) using 0-1 loss. We use the bootstrap method to produce
each model, which we evaluate on the same test set. We then search for a
maximally consistent and minimally consistent individual in the test set,
i.e., an individual with $10$ predictions that agree and an individual with
$5$ predictions in each class, which we plot in the bar graph. Please refer to
the README in https://github.com/pasta41/variance regarding which Jupyter
notebook to run to produce the underlying results and figure. The experiments
to reproduce this figure can be easily replicated on a laptop.

Reproducing Figure 2. These figures were produced by executing $S=10$ runs of
$B=101$ bootstrap training replicates to train random forest classifiers for
Old Adult and COMPAS. We reproduce these figures below, so that they can be
examined and treated in relation to our additional results for decision tree
classifiers and logistic regression. For each $s$ run, we take train/test
split, bootstrap the train split $B=101$ times, and evaluate the resulting
model classification decisions on the test set. $\hat{\texttt{SC}}$ can be
estimated from the results across those $101$ models. We Run this process
$S=10$ times to produce confidence intervals, shown in the figures below. The
intervals are not always clearly visible; there is not a lot of variance at
the level of comparing whole runs to each other. Please refer to the README in
https://github.com/pasta41/variance regarding which Jupyter notebook to run to
produce the underlying results and figure. There are also scripted version of
these experiments, which enable them to be run in parallel in a cluster
environment.

Self-consistency of incorrectly-classified instances. Last, we include figures
that underscore how self-consistency is independent from correctness that is
measured in terms of observed label alignment. That is, it is possible for an
instance $({\bm{x}},{\bm{g}})$ to be self-consistent and classified
incorrectly, with respect to its observed label $o$. We show this using
stacked bar plots. For the above experiments, we find the test examples that
have the majority of their classifications incorrect ($\hat{y}\neq o$, for
$B=101$, we find the instances with $\geq 51$ incorrect classifications) and
the majority of their classification correct (similarly), and we examine how
self-consistent they are. We bucket self-consistency into different levels,
and then plot the relative proportion of majority-incorrectly and majority-
correctly classified examples according to subgroup. Subgroups in COMPAS
exhibit a similar trend, while subgroups in Adult Old exhibit differences,
with the heights of the bars corresponding to the trends we plot in our CDF
plots. As we note briefly in Section 3, it may be interesting to examine
patterns in examples about which learning processes are confident (i.e.,
highly self-consistent) but wrong in terms of label alignment. If such issues
correlate with subgroup, it may be worth testing the counterfactual that such
labels are indicative of label bias. We leave such thoughts to future work.

(a) COMPAS

(b) Adult Old Figure 7: $\hat{\texttt{SC}}$ broken down by ${\bm{g}}$ and
label alignment with the observed label $o$. For each train/test split, and
for each $\hat{\texttt{SC}}$ range ($x$-axis), we find the examples that are
incorrectly classified the majority of time ($\geq 5$ splits, we find that
$\hat{y}\neq o$), and the examples that are correctly classified the majority
of the time ($>5$, we find that $\hat{y}=o$). We compute the average the
proportion over (over splits) in each $\hat{\texttt{SC}}$ range ($y$-axis). We
plot these proportions with respect to subgroup ${\bm{g}}$ (where the sums of
the heights of bars for by each ${\bm{g}}$ is equal to $1$).

###  E.4 Validating our algorithm in practice

####  E.4.1 COMPAS

$\hat{\texttt{SC}}$ CDFs for COMPAS (${\bm{g}}=\texttt{race}$) and associated
error metrics on the prediction set. Baseline metrics computed with $B=101$
models. For simple, $B=101$ models; for super, $B=101$ ensemble models, each
composed of $51$ underlying models. We repeat for $10$ test/train splits. We
also report abstention rate $\hat{\texttt{AR}}$.

Abstention set metrics
---
| Simple | Super
$\Delta\hat{\texttt{AR}}$ | $1.1\pm 0.9\%$ | $0.5\pm 0.0\%$
$\hat{\texttt{AR}}_{\text{NW}}$ | $23.2\pm 1.3\%$ | $4.3\pm 0.5\%$
$\hat{\texttt{AR}}_{\text{W}}$ | $22.1\pm 2.2\%$ | $3.8\pm 0.5\%$

Logistic regression prediction set metrics
---
| Baseline | Simple | Super
$\Delta\hat{\texttt{PR}}$ | $14.5\pm 0.3\%$ | $18.7\pm 0.5\%$ | $15.6\pm 0.1\%$
$\hat{\texttt{PR}}_{\text{NW}}$ | $45.3\pm 1.2\%$ | $43.8\pm 1.1\%$ | $44.2\pm 0.7\%$
$\hat{\texttt{PR}}_{\text{W}}$ | $30.8\pm 1.5\%$ | $25.1\pm 1.6\%$ | $28.6\pm 0.6\%$
$\Delta\hat{\texttt{Err}}$ | $0.2\pm 0.2\%$ | $1.1\pm 1.5\%$ | $0.9\pm 1.1\%$
$\hat{\texttt{Err}}_{\text{NW}}$ | $33.0\pm 1.3\%$ | $27.9\pm 0.9\%$ | $31.0\pm 1.0\%$
$\hat{\texttt{Err}}_{\text{W}}$ | $33.2\pm 1.1\%$ | $29.0\pm 2.4\%$ | $31.9\pm 2.1\%$
$\Delta\hat{\texttt{FPR}}$ | $2.1\pm 0.0\%$ | $3.0\pm 0.0\%$ | $1.8\pm 0.2\%$
$\hat{\texttt{FPR}}_{\text{NW}}$ | $14.7\pm 1.3\%$ | $11.4\pm 1.0\%$ | $12.9\pm 0.8\%$
$\hat{\texttt{FPR}}_{\text{W}}$ | $12.6\pm 1.3\%$ | $8.4\pm 1.0\%$ | $11.1\pm 0.6\%$
$\Delta\hat{\texttt{FNR}}$ | $2.4\pm 0.0\%$ | $4.0\pm 1.1\%$ | $2.8\pm 0.8\%$
$\hat{\texttt{FNR}}_{\text{NW}}$ | $18.3\pm 1.1\%$ | $16.5\pm 1.9\%$ | $18.0\pm 1.3\%$
$\hat{\texttt{FNR}}_{\text{W}}$ | $20.7\pm 1.1\%$ | $20.5\pm 3.0\%$ | $20.8\pm 2.1\%$

Figure 8: Logistic regression on COMPAS

Abstention set metrics
---
| Simple | Super
$\Delta\hat{\texttt{AR}}$ | $1.9\pm 1.0\%$ | $2.3\pm 0.1\%$
$\hat{\texttt{AR}}_{\text{NW}}$ | $62.3\pm 1.8\%$ | $12.3\pm 0.8\%$
$\hat{\texttt{AR}}_{\text{W}}$ | $64.2\pm 2.8\%$ | $14.6\pm 0.9\%$

Decision tree prediction set metrics
---
| Baseline | Simple | Super
$\Delta\hat{\texttt{PR}}$ | $10.1\pm 0.6\%$ | $22.9\pm 1.7\%$ | $15.8\pm 0.5\%$
$\hat{\texttt{PR}}_{\text{NW}}$ | $47.9\pm 0.7\%$ | $43.4\pm 3.1\%$ | $48.5\pm 1.2\%$
$\hat{\texttt{PR}}_{\text{W}}$ | $37.8\pm 1.3\%$ | $20.5\pm 1.4\%$ | $32.7\pm 1.7\%$
$\Delta\hat{\texttt{Err}}$ | $0.6\pm 0.9\%$ | $1.7\pm 0.7\%$ | $1.2\pm 0.8\%$
$\hat{\texttt{Err}}_{\text{NW}}$ | $38.8\pm 0.5\%$ | $24.0\pm 0.9\%$ | $32.8\pm 0.4\%$
$\hat{\texttt{Err}}_{\text{W}}$ | $38.2\pm 1.4\%$ | $22.3\pm 1.6\%$ | $31.6\pm 1.2\%$
$\Delta\hat{\texttt{FPR}}$ | $0.2\pm 0.4\%$ | $4.0\pm 0.4\%$ | $2.5\pm 0.9\%$
$\hat{\texttt{FPR}}_{\text{NW}}$ | $18.8\pm 0.8\%$ | $10.4\pm 1.8\%$ | $16.1\pm 0.9\%$
$\hat{\texttt{FPR}}_{\text{W}}$ | $18.6\pm 1.2\%$ | $6.4\pm 1.4\%$ | $13.6\pm 1.8\%$
$\Delta\hat{\texttt{FNR}}$ | $0.3\pm 0.3\%$ | $2.3\pm 1.3\%$ | $1.4\pm 0.1\%$
$\hat{\texttt{FNR}}_{\text{NW}}$ | $19.9\pm 0.7\%$ | $13.6\pm 1.0\%$ | $16.6\pm 1.3\%$
$\hat{\texttt{FNR}}_{\text{W}}$ | $19.6\pm 1.0\%$ | $15.9\pm 2.3\%$ | $18.0\pm 1.2\%$

Figure 9: Decision trees on COMPAS

Abstention set metrics
---
| Simple | Super
$\Delta\hat{\texttt{AR}}$ | $0.3\pm 0.6\%$ | $0.2\pm 0.7\%$
$\hat{\texttt{AR}}_{\text{NW}}$ | $53.9\pm 1.6\%$ | $10.6\pm 0.5\%$
$\hat{\texttt{AR}}_{\text{W}}$ | $53.6\pm 2.2\%$ | $10.8\pm 1.2\%$

Random forest prediction set metrics
---
| Baseline | Simple | Super
$\Delta\hat{\texttt{PR}}$ | $13.0\pm 0.7\%$ | $24.3\pm 0.4\%$ | $18.6\pm 0.5\%$
$\hat{\texttt{PR}}_{\text{NW}}$ | $48.0\pm 0.6\%$ | $45.6\pm 1.7\%$ | $47.8\pm 0.9\%$
$\hat{\texttt{PR}}_{\text{W}}$ | $35.0\pm 1.3\%$ | $21.3\pm 1.3\%$ | $29.2\pm 1.4\%$
$\Delta\hat{\texttt{Err}}$ | $1.0\pm 0.8\%$ | $0.6\pm 0.8\%$ | $2.1\pm 1.0\%$
$\hat{\texttt{Err}}_{\text{NW}}$ | $36.9\pm 0.5\%$ | $23.3\pm 0.8\%$ | $32.3\pm 0.4\%$
$\hat{\texttt{Err}}_{\text{W}}$ | $35.9\pm 1.3\%$ | $23.9\pm 1.6\%$ | $30.2\pm 1.4\%$
$\Delta\hat{\texttt{FPR}}$ | $2.0\pm 0.4\%$ | $3.2\pm 0.0\%$ | $4.5\pm 0.4\%$
$\hat{\texttt{FPR}}_{\text{NW}}$ | $18.0\pm 0.8\%$ | $10.0\pm 1.3\%$ | $15.3\pm 1.2\%$
$\hat{\texttt{FPR}}_{\text{W}}$ | $16.0\pm 1.2\%$ | $6.8\pm 1.3\%$ | $10.8\pm 0.8\%$
$\Delta\hat{\texttt{FNR}}$ | $0.9\pm 0.4\%$ | $3.7\pm 1.2\%$ | $2.4\pm 0.8\%$
$\hat{\texttt{FNR}}_{\text{NW}}$ | $19.0\pm 0.7\%$ | $13.4\pm 1.2\%$ | $16.9\pm 1.2\%$
$\hat{\texttt{FNR}}_{\text{W}}$ | $19.9\pm 1.1\%$ | $17.1\pm 2.4\%$ | $19.3\pm 2.0\%$

Figure 10: Random forests on COMPAS

####  E.4.2 Old Adult

$\hat{\texttt{SC}}$ CDFs for Old Adult (${\bm{g}}=\texttt{sex}$) and
associated error metrics on the prediction set. Baseline metrics computed with
$B=101$ models. For simple, $B=101$ models; for super, $B=101$ ensemble
models, each composed of $51$ underlying models. We repeat for $10$ test/train
splits. We also report abstention rate $\hat{\texttt{AR}}$.

Abstention set metrics
---
| Simple | Super
$\Delta\hat{\texttt{AR}}$ | $2.6\pm 0.0\%$ | $0.5\pm 0.0\%$
$\hat{\texttt{AR}}_{\text{F}}$ | $1.8\pm 0.2\%$ | $0.3\pm 0.1\%$
$\hat{\texttt{AR}}_{\text{M}}$ | $4.4\pm 0.2\%$ | $0.8\pm 0.1\%$

Logistic regression prediction set metrics
---
| Baseline | Simple | Super
$\Delta\hat{\texttt{PR}}$ | $18.3\pm 0.2\%$ | $17.8\pm 0.1\%$ | $18.1\pm 0.1\%$
$\hat{\texttt{PR}}_{\text{F}}$ | $8.2\pm 0.3\%$ | $7.1\pm 0.4\%$ | $7.6\pm 0.4\%$
$\hat{\texttt{PR}}_{\text{M}}$ | $26.5\pm 0.5\%$ | $24.9\pm 0.5\%$ | $25.7\pm 0.5\%$
$\Delta\hat{\texttt{Err}}$ | $11.3\pm 0.1\%$ | $10.8\pm 0.1\%$ | $11.4\pm 0.2\%$
$\hat{\texttt{Err}}_{\text{F}}$ | $7.8\pm 0.4\%$ | $7.0\pm 0.3\%$ | $7.5\pm 0.2\%$
$\hat{\texttt{Err}}_{\text{M}}$ | $19.1\pm 0.3\%$ | $17.8\pm 0.4\%$ | $18.9\pm 0.4\%$
$\Delta\hat{\texttt{FPR}}$ | $4.7\pm 0.0\%$ | $4.4\pm 0.2\%$ | $4.8\pm 0.2\%$
$\hat{\texttt{FPR}}_{\text{F}}$ | $2.3\pm 0.3\%$ | $1.6\pm 0.1\%$ | $1.8\pm 0.1\%$
$\hat{\texttt{FPR}}_{\text{M}}$ | $7.0\pm 0.3\%$ | $6.0\pm 0.3\%$ | $6.6\pm 0.3\%$
$\Delta\hat{\texttt{FNR}}$ | $6.7\pm 0.1\%$ | $6.5\pm 0.1\%$ | $6.6\pm 0.1\%$
$\hat{\texttt{FNR}}_{\text{F}}$ | $5.5\pm 0.3\%$ | $5.4\pm 0.2\%$ | $5.7\pm 0.2\%$
$\hat{\texttt{FNR}}_{\text{M}}$ | $12.2\pm 0.2\%$ | $11.9\pm 0.1\%$ | $12.3\pm 0.1\%$

Figure 11: Logistic regression on Old Adult

Abstention set metrics
---
| Simple | Super
$\Delta\hat{\texttt{AR}}$ | $19.5\pm 0.2\%$ | $3.5\pm 0.1\%$
$\hat{\texttt{AR}}_{\text{F}}$ | $18.2\pm 0.4\%$ | $3.4\pm 0.2\%$
$\hat{\texttt{AR}}_{\text{M}}$ | $37.7\pm 0.6\%$ | $6.9\pm 0.3\%$

Decision tree prediction set metrics
---
| Baseline | Simple | Super
$\Delta\hat{\texttt{PR}}$ | $20.3\pm 0.1\%$ | $18.2\pm 0.1\%$ | $19.9\pm 0.1\%$
$\hat{\texttt{PR}}_{\text{F}}$ | $12.1\pm 0.4\%$ | $4.5\pm 0.3\%$ | $7.8\pm 0.5\%$
$\hat{\texttt{PR}}_{\text{M}}$ | $32.4\pm 0.5\%$ | $22.7\pm 0.2\%$ | $27.7\pm 0.4\%$
$\Delta\hat{\texttt{Err}}$ | $12.3\pm 0.0\%$ | $6.0\pm 0.1\%$ | $10.9\pm 0.2\%$
$\hat{\texttt{Err}}_{\text{F}}$ | $10.8\pm 0.3\%$ | $3.0\pm 0.2\%$ | $6.6\pm 0.4\%$
$\hat{\texttt{Err}}_{\text{M}}$ | $23.1\pm 0.3\%$ | $9.0\pm 0.3\%$ | $17.5\pm 0.2\%$
$\Delta\hat{\texttt{FPR}}$ | $6.2\pm 0.1\%$ | $2.5\pm 0.1\%$ | $5.4\pm 0.2\%$
$\hat{\texttt{FPR}}_{\text{F}}$ | $5.7\pm 0.2\%$ | $0.4\pm 0.0\%$ | $1.9\pm 0.3\%$
$\hat{\texttt{FPR}}_{\text{M}}$ | $11.9\pm 0.3\%$ | $2.9\pm 0.1\%$ | $7.3\pm 0.1\%$
$\Delta\hat{\texttt{FNR}}$ | $6.1\pm 0.1\%$ | $3.4\pm 0.0\%$ | $5.5\pm 0.1\%$
$\hat{\texttt{FNR}}_{\text{F}}$ | $5.1\pm 0.3\%$ | $2.7\pm 0.2\%$ | $4.7\pm 0.1\%$
$\hat{\texttt{FNR}}_{\text{M}}$ | $11.2\pm 0.2\%$ | $6.1\pm 0.2\%$ | $10.2\pm 0.2\%$

Figure 12: Decision trees on Old Adult

Abstention set metrics
---
| Simple | Super
$\Delta\hat{\texttt{AR}}$ | $17.2\pm 0.4\%$ | $3.4\pm 0.1\%$
$\hat{\texttt{AR}}_{\text{F}}$ | $11.2\pm 0.3\%$ | $2.0\pm 0.3\%$
$\hat{\texttt{AR}}_{\text{M}}$ | $28.4\pm 0.7\%$ | $5.4\pm 0.2\%$

Random forest prediction set metrics
---
| Baseline | Simple | Super
$\Delta\hat{\texttt{PR}}$ | $20.0\pm 0.2\%$ | $17.1\pm 0.3\%$ | $19.0\pm 0.2\%$
$\hat{\texttt{PR}}_{\text{F}}$ | $9.8\pm 0.2\%$ | $4.8\pm 0.2\%$ | $7.7\pm 0.3\%$
$\hat{\texttt{PR}}_{\text{M}}$ | $29.8\pm 0.4\%$ | $21.9\pm 0.5\%$ | $26.7\pm 0.5\%$
$\Delta\hat{\texttt{Err}}$ | $12.2\pm 0.0\%$ | $6.5\pm 0.1\%$ | $10.7\pm 0.0\%$
$\hat{\texttt{Err}}_{\text{F}}$ | $9.0\pm 0.3\%$ | $4.2\pm 0.2\%$ | $6.6\pm 0.2\%$
$\hat{\texttt{Err}}_{\text{M}}$ | $21.2\pm 0.3\%$ | $10.7\pm 0.3\%$ | $17.3\pm 0.2\%$
$\Delta\hat{\texttt{FPR}}$ | $6.0\pm 0.2\%$ | $2.5\pm 0.1\%$ | $5.0\pm 0.1\%$
$\hat{\texttt{FPR}}_{\text{F}}$ | $3.7\pm 0.1\%$ | $0.7\pm 0.1\%$ | $1.7\pm 0.2\%$
$\hat{\texttt{FPR}}_{\text{M}}$ | $9.7\pm 0.3\%$ | $3.2\pm 0.2\%$ | $6.7\pm 0.3\%$
$\Delta\hat{\texttt{FNR}}$ | $6.3\pm 0.2\%$ | $4.1\pm 0.2\%$ | $5.8\pm 0.1\%$
$\hat{\texttt{FNR}}_{\text{F}}$ | $5.3\pm 0.3\%$ | $3.5\pm 0.1\%$ | $4.9\pm 0.2\%$
$\hat{\texttt{FNR}}_{\text{M}}$ | $11.6\pm 0.1\%$ | $7.6\pm 0.3\%$ | $10.7\pm 0.3\%$

Figure 13: Random forests on Old Adult

####  E.4.3 South German Credit

$\hat{\texttt{SC}}$ CDFs for German Credit (${\bm{g}}=\texttt{sex}$) and
associated error metrics on the prediction set. Baseline metrics computed with
$B=101$ models. For simple, $B=101$ models; for super, $B=101$ ensemble
models, each composed of $51$ underlying models. We repeat for $10$ test/train
splits. We also report abstention rate $\hat{\texttt{AR}}$.

Abstention set metrics
---
| Simple | Super
$\Delta\hat{\texttt{AR}}$ | $0.4\pm 3.9\%$ | $0.1\pm 1.8\%$
$\hat{\texttt{AR}}_{\text{F}}$ | $20.8\pm 6.6\%$ | $4.1\pm 3.1\%$
$\hat{\texttt{AR}}_{\text{M}}$ | $21.2\pm 2.7\%$ | $4.0\pm 1.3\%$

Logistic regression prediction set metrics
---
| Baseline | Simple | Super
$\Delta\hat{\texttt{PR}}$ | $8.8\pm 1.4\%$ | $9.1\pm 1.2\%$ | $9.7\pm 1.7\%$
$\hat{\texttt{PR}}_{\text{F}}$ | $88.8\pm 4.7\%$ | $96.0\pm 4.1\%$ | $91.7\pm 5.0\%$
$\hat{\texttt{PR}}_{\text{M}}$ | $80.0\pm 3.3\%$ | $86.9\pm 2.9\%$ | $82.0\pm 3.3\%$
$\Delta\hat{\texttt{Err}}$ | $0.9\pm 4.0\%$ | $4.6\pm 6.1\%$ | $3.4\pm 4.7\%$
$\hat{\texttt{Err}}_{\text{F}}$ | $23.3\pm 6.9\%$ | $22.8\pm 8.7\%$ | $25.5\pm 7.4\%$
$\hat{\texttt{Err}}_{\text{M}}$ | $24.2\pm 2.9\%$ | $18.2\pm 2.6\%$ | $22.1\pm 2.7\%$
$\Delta\hat{\texttt{FPR}}$ | $0.7\pm 3.9\%$ | $5.4\pm 5.8\%$ | $5.5\pm 5.4\%$
$\hat{\texttt{FPR}}_{\text{F}}$ | $16.2\pm 6.2\%$ | $19.6\pm 8.4\%$ | $21.1\pm 7.8\%$
$\hat{\texttt{FPR}}_{\text{M}}$ | $15.5\pm 2.3\%$ | $14.2\pm 2.6\%$ | $15.6\pm 2.4\%$
$\Delta\hat{\texttt{FNR}}$ | $1.6\pm 1.1\%$ | $0.8\pm 1.9\%$ | $2.1\pm 1.8\%$
$\hat{\texttt{FNR}}_{\text{F}}$ | $7.1\pm 3.7\%$ | $3.2\pm 3.5\%$ | $4.4\pm 3.8\%$
$\hat{\texttt{FNR}}_{\text{M}}$ | $8.7\pm 2.6\%$ | $4.0\pm 1.6\%$ | $6.5\pm 2.0\%$

Figure 14: Logistic regression on German Credit

Abstention set metrics
---
| Simple | Super
$\Delta\hat{\texttt{AR}}$ | $0.0\pm 2.6\%$ | $2.8\pm 2.6\%$
$\hat{\texttt{AR}}_{\text{F}}$ | $65.2\pm 6.0\%$ | $19.9\pm 5.9\%$
$\hat{\texttt{AR}}_{\text{M}}$ | $65.2\pm 3.4\%$ | $17.1\pm 3.3\%$

Decision tree prediction set metrics
---
| Baseline | Simple | Super
$\Delta\hat{\texttt{PR}}$ | $0.3\pm 2.5\%$ | $1.9\pm 0.4\%$ | $3.7\pm 3.7\%$
$\hat{\texttt{PR}}_{\text{F}}$ | $71.2\pm 4.6\%$ | $99.6\pm 0.8\%$ | $87.9\pm 6.0\%$
$\hat{\texttt{PR}}_{\text{M}}$ | $70.9\pm 2.1\%$ | $97.7\pm 1.2\%$ | $84.2\pm 2.3\%$
$\Delta\hat{\texttt{Err}}$ | $1.1\pm 2.9\%$ | $0.3\pm 5.2\%$ | $0.1\pm 4.9\%$
$\hat{\texttt{Err}}_{\text{F}}$ | $33.0\pm 4.8\%$ | $9.8\pm 8.0\%$ | $20.3\pm 8.2\%$
$\hat{\texttt{Err}}_{\text{M}}$ | $31.9\pm 1.9\%$ | $9.5\pm 2.8\%$ | $20.2\pm 3.3\%$
$\Delta\hat{\texttt{FPR}}$ | $0.7\pm 3.5\%$ | $0.8\pm 5.1\%$ | $0.4\pm 3.9\%$
$\hat{\texttt{FPR}}_{\text{F}}$ | $15.6\pm 5.9\%$ | $9.6\pm 7.9\%$ | $15.2\pm 7.0\%$
$\hat{\texttt{FPR}}_{\text{M}}$ | $14.9\pm 2.4\%$ | $8.8\pm 2.8\%$ | $14.8\pm 3.1\%$
$\Delta\hat{\texttt{FNR}}$ | $0.5\pm 2.3\%$ | $0.4\pm 0.0\%$ | $0.2\pm 3.6\%$
$\hat{\texttt{FNR}}_{\text{F}}$ | $17.4\pm 4.4\%$ | $0.2\pm 0.7\%$ | $5.2\pm 5.1\%$
$\hat{\texttt{FNR}}_{\text{M}}$ | $16.9\pm 2.1\%$ | $0.6\pm 0.7\%$ | $5.4\pm 1.5\%$

Figure 15: Decision trees on German Credit

Abstention set metrics
---
| Simple | Super
$\Delta\hat{\texttt{AR}}$ | $0.1\pm 2.8\%$ | $1.2\pm 3.3\%$
$\hat{\texttt{AR}}_{\text{F}}$ | $47.5\pm 6.7\%$ | $9.9\pm 5.7\%$
$\hat{\texttt{AR}}_{\text{M}}$ | $47.4\pm 3.9\%$ | $8.7\pm 2.4\%$

Random forest prediction set metrics
---
| Baseline | Simple | Super
$\Delta\hat{\texttt{PR}}$ | $3.9\pm 1.6\%$ | $1.9\pm 0.6\%$ | $4.9\pm 2.1\%$
$\hat{\texttt{PR}}_{\text{F}}$ | $81.7\pm 3.3\%$ | $99.9\pm 0.4\%$ | $94.5\pm 4.0\%$
$\hat{\texttt{PR}}_{\text{M}}$ | $77.8\pm 1.7\%$ | $98.0\pm 1.0\%$ | $89.6\pm 1.9\%$
$\Delta\hat{\texttt{Err}}$ | $2.3\pm 3.1\%$ | $0.0\pm 4.7\%$ | $2.4\pm 6.7\%$
$\hat{\texttt{Err}}_{\text{F}}$ | $25.8\pm 5.1\%$ | $11.9\pm 7.8\%$ | $23.5\pm 9.6\%$
$\hat{\texttt{Err}}_{\text{M}}$ | $28.1\pm 2.0\%$ | $11.9\pm 3.1\%$ | $21.1\pm 2.9\%$
$\Delta\hat{\texttt{FPR}}$ | $2.5\pm 3.0\%$ | $0.3\pm 4.7\%$ | $2.2\pm 6.2\%$
$\hat{\texttt{FPR}}_{\text{F}}$ | $13.8\pm 4.8\%$ | $11.8\pm 7.8\%$ | $20.5\pm 9.1\%$
$\hat{\texttt{FPR}}_{\text{M}}$ | $16.3\pm 1.8\%$ | $11.5\pm 3.1\%$ | $18.3\pm 2.9\%$
$\Delta\hat{\texttt{FNR}}$ | $0.2\pm 1.7\%$ | $0.3\pm 0.1\%$ | $0.1\pm 2.3\%$
$\hat{\texttt{FNR}}_{\text{F}}$ | $11.9\pm 3.1\%$ | $0.1\pm 0.3\%$ | $3.0\pm 3.5\%$
$\hat{\texttt{FNR}}_{\text{M}}$ | $11.7\pm 1.4\%$ | $0.4\pm 0.4\%$ | $2.9\pm 1.2\%$

Figure 16: Random forests on German Credit

####  E.4.4 Taiwan Credit

$\hat{\texttt{SC}}$ CDFs for Taiwan Credit (${\bm{g}}=\texttt{sex}$) and
associated error metrics on the prediction set. Baseline metrics computed with
$B=101$ models. For simple, $B=101$ models; for super, $B=101$ ensemble
models, each composed of $41$ underlying models. We repeat for $10$ test/train
splits. We also report abstention rate $\hat{\texttt{AR}}$.

Abstention set metrics
---
| Simple | Super
$\Delta\hat{\texttt{AR}}$ | $0.4\pm 0.2\%$ | $0.0\pm 0.2\%$
$\hat{\texttt{AR}}_{\text{F}}$ | $2.1\pm 0.1\%$ | $0.4\pm 0.0\%$
$\hat{\texttt{AR}}_{\text{M}}$ | $2.5\pm 0.3\%$ | $0.4\pm 0.2\%$

Logistic regression prediction set metrics
---
| Baseline | Simple | Super
$\Delta\hat{\texttt{PR}}$ | $1.5\pm 0.1\%$ | $1.0\pm 0.1\%$ | $1.0\pm 0.1\%$
$\hat{\texttt{PR}}_{\text{F}}$ | $6.7\pm 0.3\%$ | $6.2\pm 0.1\%$ | $6.9\pm 0.1\%$
$\hat{\texttt{PR}}_{\text{M}}$ | $8.2\pm 0.4\%$ | $7.2\pm 0.2\%$ | $7.9\pm 0.2\%$
$\Delta\hat{\texttt{Err}}$ | $3.1\pm 0.1\%$ | $3.1\pm 0.3\%$ | $3.2\pm 0.3\%$
$\hat{\texttt{Err}}_{\text{F}}$ | $17.8\pm 0.5\%$ | $17.0\pm 0.2\%$ | $17.5\pm 0.3\%$
$\hat{\texttt{Err}}_{\text{M}}$ | $20.9\pm 0.4\%$ | $20.1\pm 0.5\%$ | $20.7\pm 0.6\%$
$\Delta\hat{\texttt{FPR}}$ | $0.7\pm 0.2\%$ | $0.3\pm 0.0\%$ | $0.3\pm 0.1\%$
$\hat{\texttt{FPR}}_{\text{F}}$ | $1.8\pm 0.1\%$ | $1.7\pm 0.1\%$ | $2.0\pm 0.1\%$
$\hat{\texttt{FPR}}_{\text{M}}$ | $2.5\pm 0.3\%$ | $2.0\pm 0.1\%$ | $2.3\pm 0.2\%$
$\Delta\hat{\texttt{FNR}}$ | $2.4\pm 0.2\%$ | $2.7\pm 0.4\%$ | $2.8\pm 0.3\%$
$\hat{\texttt{FNR}}_{\text{F}}$ | $16.0\pm 0.6\%$ | $15.3\pm 0.2\%$ | $15.6\pm 0.3\%$
$\hat{\texttt{FNR}}_{\text{M}}$ | $18.4\pm 0.4\%$ | $18.0\pm 0.6\%$ | $18.4\pm 0.6\%$

Figure 17: Logistic regression on Taiwan Credit

Abstention set metrics
---
| Simple | Super
$\Delta\hat{\texttt{AR}}$ | $3.2\pm 0.1\%$ | $1.3\pm 0.1\%$
$\hat{\texttt{AR}}_{\text{F}}$ | $56.7\pm 0.6\%$ | $6.7\pm 0.2\%$
$\hat{\texttt{AR}}_{\text{M}}$ | $59.9\pm 0.5\%$ | $8.0\pm 0.1\%$

Decision tree prediction set metrics
---
| Baseline | Simple | Super
$\Delta\hat{\texttt{PR}}$ | $2.1\pm 0.1\%$ | $1.2\pm 0.0\%$ | $2.0\pm 0.2\%$
$\hat{\texttt{PR}}_{\text{F}}$ | $22.9\pm 0.2\%$ | $3.0\pm 0.4\%$ | $9.9\pm 0.3\%$
$\hat{\texttt{PR}}_{\text{M}}$ | $25.0\pm 0.3\%$ | $4.2\pm 0.4\%$ | $11.9\pm 0.5\%$
$\Delta\hat{\texttt{Err}}$ | $2.3\pm 0.1\%$ | $1.6\pm 0.0\%$ | $2.5\pm 0.1\%$
$\hat{\texttt{Err}}_{\text{F}}$ | $26.8\pm 0.2\%$ | $9.6\pm 0.4\%$ | $15.3\pm 0.3\%$
$\hat{\texttt{Err}}_{\text{M}}$ | $29.1\pm 0.3\%$ | $11.2\pm 0.4\%$ | $17.8\pm 0.4\%$
$\Delta\hat{\texttt{FPR}}$ | $0.6\pm 0.1\%$ | $0.2\pm 0.1\%$ | $0.7\pm 0.2\%$
$\hat{\texttt{FPR}}_{\text{F}}$ | $14.4\pm 0.2\%$ | $0.6\pm 0.1\%$ | $3.0\pm 0.1\%$
$\hat{\texttt{FPR}}_{\text{M}}$ | $15.0\pm 0.3\%$ | $0.8\pm 0.2\%$ | $3.7\pm 0.3\%$
$\Delta\hat{\texttt{FNR}}$ | $1.7\pm 0.2\%$ | $1.3\pm 0.1\%$ | $1.9\pm 0.1\%$
$\hat{\texttt{FNR}}_{\text{F}}$ | $12.4\pm 0.4\%$ | $9.0\pm 0.4\%$ | $12.3\pm 0.3\%$
$\hat{\texttt{FNR}}_{\text{M}}$ | $14.1\pm 0.2\%$ | $10.3\pm 0.5\%$ | $14.2\pm 0.4\%$

Figure 18: Decision trees on Taiwan Credit

Abstention set metrics
---
| Simple | Super
$\Delta\hat{\texttt{AR}}$ | $4.1\pm 0.1\%$ | $0.8\pm 0.0\%$
$\hat{\texttt{AR}}_{\text{F}}$ | $24.0\pm 0.8\%$ | $3.9\pm 0.3\%$
$\hat{\texttt{AR}}_{\text{M}}$ | $28.1\pm 0.7\%$ | $4.7\pm 0.3\%$

Random forest prediction set metrics
---
| Baseline | Simple | Super
$\Delta\hat{\texttt{PR}}$ | $2.5\pm 0.1\%$ | $1.0\pm 0.1\%$ | $2.1\pm 0.2\%$
$\hat{\texttt{PR}}_{\text{F}}$ | $14.9\pm 0.2\%$ | $4.1\pm 0.3\%$ | $10.3\pm 0.2\%$
$\hat{\texttt{PR}}_{\text{M}}$ | $17.4\pm 0.3\%$ | $5.1\pm 0.2\%$ | $12.4\pm 0.4\%$
$\Delta\hat{\texttt{Err}}$ | $2.8\pm 0.1\%$ | $1.9\pm 0.0\%$ | $2.5\pm 0.1\%$
$\hat{\texttt{Err}}_{\text{F}}$ | $20.5\pm 0.3\%$ | $12.0\pm 0.4\%$ | $15.8\pm 0.4\%$
$\hat{\texttt{Err}}_{\text{M}}$ | $23.3\pm 0.4\%$ | $13.9\pm 0.4\%$ | $18.3\pm 0.5\%$
$\Delta\hat{\texttt{FPR}}$ | $1.0\pm 0.1\%$ | $0.3\pm 0.1\%$ | $0.6\pm 0.1\%$
$\hat{\texttt{FPR}}_{\text{F}}$ | $7.2\pm 0.2\%$ | $0.9\pm 0.1\%$ | $3.3\pm 0.1\%$
$\hat{\texttt{FPR}}_{\text{M}}$ | $8.2\pm 0.3\%$ | $1.2\pm 0.2\%$ | $3.9\pm 0.2\%$
$\Delta\hat{\texttt{FNR}}$ | $1.7\pm 0.1\%$ | $1.6\pm 0.1\%$ | $1.8\pm 0.0\%$
$\hat{\texttt{FNR}}_{\text{F}}$ | $13.3\pm 0.4\%$ | $11.0\pm 0.3\%$ | $12.6\pm 0.4\%$
$\hat{\texttt{FNR}}_{\text{M}}$ | $15.0\pm 0.3\%$ | $12.6\pm 0.4\%$ | $14.4\pm 0.4\%$

Figure 19: Random forests on Taiwan Credit

####  E.4.5 New Adult - CA

$\hat{\texttt{SC}}$ CDFs for three tasks (Income, Employment, Public Coverage)
in New Adult - CA, using ${\bm{g}}=\texttt{sex}$ and race, and associated
error metrics on the prediction set. Baseline metrics computed with $B=101$
models. For simple, $B=101$ models; for super, $B=101$ ensemble models, each
composed of $21$ underlying models for Income and Public Coverage; $15$ for
Employment. We repeat for $5$ test/train splits. We also report abstention
rate $\hat{\texttt{AR}}$.

Income \- by sex.

Abstention set metrics
---
| Simple | Super
$\Delta\hat{\texttt{AR}}$ | $0.1\pm 0.0\%$ | $0.1\pm 0.0\%$
$\hat{\texttt{AR}}_{\text{F}}$ | $1.0\pm 0.0\%$ | $0.3\pm 0.0\%$
$\hat{\texttt{AR}}_{\text{M}}$ | $0.9\pm 0.0\%$ | $0.2\pm 0.0\%$

Logistic regression prediction set metrics
---
| Baseline | Simple | Super
$\Delta\hat{\texttt{PR}}$ | $2.7\pm 0.1\%$ | $2.9\pm 0.1\%$ | $2.8\pm 0.1\%$
$\hat{\texttt{PR}}_{\text{F}}$ | $38.4\pm 0.2\%$ | $38.1\pm 0.2\%$ | $38.2\pm 0.2\%$
$\hat{\texttt{PR}}_{\text{M}}$ | $41.1\pm 0.3\%$ | $41.0\pm 0.1\%$ | $41.0\pm 0.1\%$
$\Delta\hat{\texttt{Err}}$ | $0.9\pm 0.0\%$ | $1.0\pm 0.2\%$ | $1.0\pm 0.2\%$
$\hat{\texttt{Err}}_{\text{F}}$ | $21.5\pm 0.2\%$ | $21.1\pm 0.3\%$ | $21.3\pm 0.3\%$
$\hat{\texttt{Err}}_{\text{M}}$ | $22.4\pm 0.2\%$ | $22.1\pm 0.1\%$ | $22.3\pm 0.1\%$
$\Delta\hat{\texttt{FPR}}$ | $4.0\pm 0.1\%$ | $3.9\pm 0.0\%$ | $3.9\pm 0.0\%$
$\hat{\texttt{FPR}}_{\text{F}}$ | $12.5\pm 0.2\%$ | $12.2\pm 0.1\%$ | $12.3\pm 0.1\%$
$\hat{\texttt{FPR}}_{\text{M}}$ | $8.5\pm 0.1\%$ | $8.3\pm 0.1\%$ | $8.4\pm 0.1\%$
$\Delta\hat{\texttt{FNR}}$ | $4.9\pm 0.0\%$ | $4.9\pm 0.1\%$ | $4.8\pm 0.1\%$
$\hat{\texttt{FNR}}_{\text{F}}$ | $9.0\pm 0.1\%$ | $8.9\pm 0.2\%$ | $9.1\pm 0.2\%$
$\hat{\texttt{FNR}}_{\text{M}}$ | $13.9\pm 0.1\%$ | $13.8\pm 0.1\%$ | $13.9\pm 0.1\%$

Figure 20: Logistic regression on New Adult - CA - Income, by sex

Abstention set metrics
---
| Simple | Super
$\Delta\hat{\texttt{AR}}$ | $2.1\pm 0.2\%$ | $0.8\pm 0.0\%$
$\hat{\texttt{AR}}_{\text{F}}$ | $49.2\pm 0.3\%$ | $13.3\pm 0.2\%$
$\hat{\texttt{AR}}_{\text{M}}$ | $51.3\pm 0.1\%$ | $14.1\pm 0.2\%$

Decision tree prediction set metrics
---
| Baseline | Simple | Super
$\Delta\hat{\texttt{PR}}$ | $7.5\pm 0.1\%$ | $12.5\pm 0.1\%$ | $9.7\pm 0.1\%$
$\hat{\texttt{PR}}_{\text{F}}$ | $37.4\pm 0.2\%$ | $26.8\pm 0.4\%$ | $34.1\pm 0.3\%$
$\hat{\texttt{PR}}_{\text{M}}$ | $44.9\pm 0.1\%$ | $39.3\pm 0.3\%$ | $43.8\pm 0.2\%$
$\Delta\hat{\texttt{Err}}$ | $1.4\pm 0.0\%$ | $1.0\pm 0.0\%$ | $1.4\pm 0.0\%$
$\hat{\texttt{Err}}_{\text{F}}$ | $24.4\pm 0.1\%$ | $6.9\pm 0.1\%$ | $14.5\pm 0.2\%$
$\hat{\texttt{Err}}_{\text{M}}$ | $25.8\pm 0.1\%$ | $7.9\pm 0.1\%$ | $15.9\pm 0.2\%$
$\Delta\hat{\texttt{FPR}}$ | $1.4\pm 0.0\%$ | $0.1\pm 0.1\%$ | $0.5\pm 0.1\%$
$\hat{\texttt{FPR}}_{\text{F}}$ | $13.5\pm 0.1\%$ | $3.6\pm 0.1\%$ | $7.6\pm 0.1\%$
$\hat{\texttt{FPR}}_{\text{M}}$ | $12.1\pm 0.1\%$ | $3.5\pm 0.2\%$ | $7.1\pm 0.2\%$
$\Delta\hat{\texttt{FNR}}$ | $2.9\pm 0.0\%$ | $1.1\pm 0.0\%$ | $1.9\pm 0.1\%$
$\hat{\texttt{FNR}}_{\text{F}}$ | $10.9\pm 0.1\%$ | $3.3\pm 0.1\%$ | $6.9\pm 0.1\%$
$\hat{\texttt{FNR}}_{\text{M}}$ | $13.8\pm 0.1\%$ | $4.4\pm 0.1\%$ | $8.8\pm 0.2\%$

Figure 21: Decision trees on New Adult - CA - Income, by sex

Abstention set metrics
---
| Simple | Super
$\Delta\hat{\texttt{AR}}$ | $1.4\pm 0.1\%$ | $0.2\pm 0.0\%$
$\hat{\texttt{AR}}_{\text{F}}$ | $32.8\pm 0.2\%$ | $8.6\pm 0.1\%$
$\hat{\texttt{AR}}_{\text{M}}$ | $34.2\pm 0.1\%$ | $8.8\pm 0.1\%$

Random forest prediction set metrics
---
| Baseline | Simple | Super
$\Delta\hat{\texttt{PR}}$ | $7.4\pm 0.1\%$ | $10.3\pm 0.2\%$ | $8.6\pm 0.1\%$
$\hat{\texttt{PR}}_{\text{F}}$ | $36.7\pm 0.2\%$ | $30.6\pm 0.4\%$ | $34.9\pm 0.3\%$
$\hat{\texttt{PR}}_{\text{M}}$ | $44.1\pm 0.1\%$ | $40.9\pm 0.2\%$ | $43.5\pm 0.2\%$
$\Delta\hat{\texttt{Err}}$ | $1.4\pm 0.0\%$ | $1.2\pm 0.0\%$ | $1.4\pm 0.1\%$
$\hat{\texttt{Err}}_{\text{F}}$ | $21.0\pm 0.1\%$ | $9.3\pm 0.2\%$ | $15.3\pm 0.1\%$
$\hat{\texttt{Err}}_{\text{M}}$ | $22.4\pm 0.1\%$ | $10.5\pm 0.2\%$ | $16.7\pm 0.2\%$
$\Delta\hat{\texttt{FPR}}$ | $1.4\pm 0.0\%$ | $0.5\pm 0.0\%$ | $0.9\pm 0.1\%$
$\hat{\texttt{FPR}}_{\text{F}}$ | $11.4\pm 0.1\%$ | $4.9\pm 0.1\%$ | $8.1\pm 0.1\%$
$\hat{\texttt{FPR}}_{\text{M}}$ | $10.0\pm 0.1\%$ | $4.4\pm 0.1\%$ | $7.2\pm 0.2\%$
$\Delta\hat{\texttt{FNR}}$ | $2.8\pm 0.0\%$ | $1.8\pm 0.0\%$ | $2.3\pm 0.0\%$
$\hat{\texttt{FNR}}_{\text{F}}$ | $9.6\pm 0.1\%$ | $4.4\pm 0.1\%$ | $7.2\pm 0.1\%$
$\hat{\texttt{FNR}}_{\text{M}}$ | $12.4\pm 0.1\%$ | $6.2\pm 0.1\%$ | $9.5\pm 0.1\%$

Figure 22: Random forests on New Adult - CA - Income, by sex

Income \- by race.

Abstention set metrics
---
| Simple | Super
$\Delta\hat{\texttt{AR}}$ | $0.0\pm 0.0\%$ | $0.0\pm 0.0\%$
$\hat{\texttt{AR}}_{\text{NW}}$ | $1.0\pm 0.0\%$ | $0.2\pm 0.0\%$
$\hat{\texttt{AR}}_{\text{W}}$ | $1.0\pm 0.0\%$ | $0.2\pm 0.0\%$

Logistic regression prediction set metrics
---
| Baseline | Simple | Super
$\Delta\hat{\texttt{PR}}$ | $9.2\pm 0.1\%$ | $9.2\pm 0.3\%$ | $9.2\pm 0.2\%$
$\hat{\texttt{PR}}_{\text{NW}}$ | $34.1\pm 0.3\%$ | $33.9\pm 0.3\%$ | $34.0\pm 0.3\%$
$\hat{\texttt{PR}}_{\text{W}}$ | $43.3\pm 0.2\%$ | $43.1\pm 0.0\%$ | $43.2\pm 0.1\%$
$\Delta\hat{\texttt{Err}}$ | $0.6\pm 0.1\%$ | $0.4\pm 0.1\%$ | $0.4\pm 0.1\%$
$\hat{\texttt{Err}}_{\text{NW}}$ | $21.6\pm 0.2\%$ | $21.4\pm 0.2\%$ | $21.6\pm 0.2\%$
$\hat{\texttt{Err}}_{\text{W}}$ | $22.2\pm 0.1\%$ | $21.8\pm 0.1\%$ | $22.0\pm 0.1\%$
$\Delta\hat{\texttt{FPR}}$ | $0.6\pm 0.1\%$ | $0.5\pm 0.0\%$ | $0.5\pm 0.0\%$
$\hat{\texttt{FPR}}_{\text{NW}}$ | $10.0\pm 0.2\%$ | $9.8\pm 0.1\%$ | $9.9\pm 0.1\%$
$\hat{\texttt{FPR}}_{\text{W}}$ | $10.6\pm 0.1\%$ | $10.3\pm 0.1\%$ | $10.4\pm 0.1\%$
$\Delta\hat{\texttt{FNR}}$ | $0.0\pm 0.1\%$ | $0.1\pm 0.2\%$ | $0.1\pm 0.3\%$
$\hat{\texttt{FNR}}_{\text{NW}}$ | $11.6\pm 0.2\%$ | $11.6\pm 0.3\%$ | $11.7\pm 0.3\%$
$\hat{\texttt{FNR}}_{\text{W}}$ | $11.6\pm 0.1\%$ | $11.5\pm 0.1\%$ | $11.6\pm 0.0\%$

Figure 23: Logistic regression on New Adult - CA - Income, by race

Abstention set metrics
---
| Simple | Super
$\Delta\hat{\texttt{AR}}$ | $2.5\pm 0.1\%$ | $1.0\pm 0.1\%$
$\hat{\texttt{AR}}_{\text{NW}}$ | $48.8\pm 0.3\%$ | $13.1\pm 0.2\%$
$\hat{\texttt{AR}}_{\text{W}}$ | $51.3\pm 0.2\%$ | $14.1\pm 0.1\%$

Decision tree prediction set metrics
---
| Baseline | Simple | Super
$\Delta\hat{\texttt{PR}}$ | $7.0\pm 0.0\%$ | $10.3\pm 0.0\%$ | $9.9\pm 0.1\%$
$\hat{\texttt{PR}}_{\text{NW}}$ | $37.0\pm 0.2\%$ | $27.0\pm 0.2\%$ | $33.1\pm 0.3\%$
$\hat{\texttt{PR}}_{\text{W}}$ | $44.0\pm 0.2\%$ | $37.3\pm 0.2\%$ | $43.0\pm 0.2\%$
$\Delta\hat{\texttt{Err}}$ | $1.1\pm 0.0\%$ | $0.2\pm 0.1\%$ | $0.6\pm 0.0\%$
$\hat{\texttt{Err}}_{\text{NW}}$ | $24.5\pm 0.1\%$ | $7.3\pm 0.0\%$ | $14.8\pm 0.2\%$
$\hat{\texttt{Err}}_{\text{W}}$ | $25.6\pm 0.1\%$ | $7.5\pm 0.1\%$ | $15.4\pm 0.2\%$
$\Delta\hat{\texttt{FPR}}$ | $0.2\pm 0.0\%$ | $0.5\pm 0.0\%$ | $0.7\pm 0.0\%$
$\hat{\texttt{FPR}}_{\text{NW}}$ | $12.9\pm 0.1\%$ | $3.2\pm 0.1\%$ | $6.9\pm 0.1\%$
$\hat{\texttt{FPR}}_{\text{W}}$ | $12.7\pm 0.1\%$ | $3.7\pm 0.1\%$ | $7.6\pm 0.1\%$
$\Delta\hat{\texttt{FNR}}$ | $1.4\pm 0.0\%$ | $0.3\pm 0.0\%$ | $0.2\pm 0.0\%$
$\hat{\texttt{FNR}}_{\text{NW}}$ | $11.6\pm 0.1\%$ | $4.1\pm 0.1\%$ | $8.0\pm 0.1\%$
$\hat{\texttt{FNR}}_{\text{W}}$ | $13.0\pm 0.1\%$ | $3.8\pm 0.1\%$ | $7.8\pm 0.1\%$

Figure 24: Decision trees on New Adult - CA - Income, by race

Abstention set metrics
---
| Simple | Super
$\Delta\hat{\texttt{AR}}$ | $2.4\pm 0.2\%$ | $0.8\pm 0.0\%$
$\hat{\texttt{AR}}_{\text{NW}}$ | $32.1\pm 0.3\%$ | $8.2\pm 0.1\%$
$\hat{\texttt{AR}}_{\text{W}}$ | $34.5\pm 0.1\%$ | $9.0\pm 0.1\%$

Random forest prediction set metrics
---
| Baseline | Simple | Super
$\Delta\hat{\texttt{PR}}$ | $8.4\pm 0.0\%$ | $11.0\pm 0.0\%$ | $10.1\pm 0.2\%$
$\hat{\texttt{PR}}_{\text{NW}}$ | $35.4\pm 0.2\%$ | $29.3\pm 0.2\%$ | $33.2\pm 0.3\%$
$\hat{\texttt{PR}}_{\text{W}}$ | $43.8\pm 0.2\%$ | $40.3\pm 0.2\%$ | $43.3\pm 0.1\%$
$\Delta\hat{\texttt{Err}}$ | $1.2\pm 0.0\%$ | $0.2\pm 0.1\%$ | $0.5\pm 0.0\%$
$\hat{\texttt{Err}}_{\text{NW}}$ | $21.0\pm 0.1\%$ | $9.8\pm 0.1\%$ | $15.7\pm 0.2\%$
$\hat{\texttt{Err}}_{\text{W}}$ | $22.2\pm 0.1\%$ | $10.0\pm 0.2\%$ | $16.2\pm 0.2\%$
$\Delta\hat{\texttt{FPR}}$ | $0.6\pm 0.0\%$ | $0.7\pm 0.0\%$ | $0.9\pm 0.1\%$
$\hat{\texttt{FPR}}_{\text{NW}}$ | $10.3\pm 0.1\%$ | $4.2\pm 0.1\%$ | $7.1\pm 0.2\%$
$\hat{\texttt{FPR}}_{\text{W}}$ | $10.9\pm 0.1\%$ | $4.9\pm 0.1\%$ | $8.0\pm 0.1\%$
$\Delta\hat{\texttt{FNR}}$ | $0.7\pm 0.0\%$ | $0.5\pm 0.0\%$ | $0.3\pm 0.1\%$
$\hat{\texttt{FNR}}_{\text{NW}}$ | $10.7\pm 0.1\%$ | $5.6\pm 0.1\%$ | $8.6\pm 0.2\%$
$\hat{\texttt{FNR}}_{\text{W}}$ | $11.4\pm 0.1\%$ | $5.1\pm 0.1\%$ | $8.3\pm 0.1\%$

Figure 25: Random forests on New Adult - CA - Income, by race

Employment \- by sex.

Abstention set metrics
---
| Simple | Super
$\Delta\hat{\texttt{AR}}$ | $0.1\pm 0.0\%$ | $0.0\pm 0.0\%$
$\hat{\texttt{AR}}_{\text{F}}$ | $0.8\pm 0.0\%$ | $0.3\pm 0.0\%$
$\hat{\texttt{AR}}_{\text{M}}$ | $0.7\pm 0.0\%$ | $0.3\pm 0.0\%$

Logistic regression prediction set metrics
---
| Baseline | Simple | Super
$\Delta\hat{\texttt{PR}}$ | $4.3\pm 0.1\%$ | $4.5\pm 0.0\%$ | $4.4\pm 0.0\%$
$\hat{\texttt{PR}}_{\text{F}}$ | $56.6\pm 0.1\%$ | $56.8\pm 0.1\%$ | $56.7\pm 0.1\%$
$\hat{\texttt{PR}}_{\text{M}}$ | $52.3\pm 0.2\%$ | $52.3\pm 0.1\%$ | $52.3\pm 0.1\%$
$\Delta\hat{\texttt{Err}}$ | $5.0\pm 0.0\%$ | $4.9\pm 0.0\%$ | $4.9\pm 0.0\%$
$\hat{\texttt{Err}}_{\text{F}}$ | $25.8\pm 0.1\%$ | $25.5\pm 0.1\%$ | $25.6\pm 0.1\%$
$\hat{\texttt{Err}}_{\text{M}}$ | $20.8\pm 0.1\%$ | $20.6\pm 0.1\%$ | $20.7\pm 0.1\%$
$\Delta\hat{\texttt{FPR}}$ | $8.1\pm 0.0\%$ | $8.1\pm 0.0\%$ | $8.1\pm 0.0\%$
$\hat{\texttt{FPR}}_{\text{F}}$ | $20.1\pm 0.1\%$ | $20.1\pm 0.0\%$ | $20.1\pm 0.0\%$
$\hat{\texttt{FPR}}_{\text{M}}$ | $12.0\pm 0.1\%$ | $12.0\pm 0.0\%$ | $12.0\pm 0.0\%$
$\Delta\hat{\texttt{FNR}}$ | $3.1\pm 0.0\%$ | $3.2\pm 0.1\%$ | $3.2\pm 0.1\%$
$\hat{\texttt{FNR}}_{\text{F}}$ | $5.7\pm 0.1\%$ | $5.4\pm 0.0\%$ | $5.5\pm 0.0\%$
$\hat{\texttt{FNR}}_{\text{M}}$ | $8.8\pm 0.1\%$ | $8.6\pm 0.1\%$ | $8.7\pm 0.1\%$

Figure 26: Logistic regression on New Adult - CA - Employment, by sex

Abstention set metrics
---
| Simple | Super
$\Delta\hat{\texttt{AR}}$ | $0.2\pm 0.0\%$ | $0.1\pm 0.1\%$
$\hat{\texttt{AR}}_{\text{F}}$ | $22.5\pm 0.1\%$ | $8.2\pm 0.2\%$
$\hat{\texttt{AR}}_{\text{M}}$ | $22.3\pm 0.1\%$ | $8.1\pm 0.3\%$

Decision tree prediction set metrics
---
| Baseline | Simple | Super
$\Delta\hat{\texttt{PR}}$ | $0.5\pm 0.0\%$ | $0.7\pm 0.2\%$ | $0.6\pm 0.1\%$
$\hat{\texttt{PR}}_{\text{F}}$ | $50.3\pm 0.2\%$ | $50.0\pm 0.3\%$ | $51.2\pm 0.2\%$
$\hat{\texttt{PR}}_{\text{M}}$ | $49.8\pm 0.2\%$ | $49.3\pm 0.1\%$ | $50.6\pm 0.1\%$
$\Delta\hat{\texttt{Err}}$ | $4.8\pm 0.0\%$ | $5.8\pm 0.1\%$ | $5.8\pm 0.1\%$
$\hat{\texttt{Err}}_{\text{F}}$ | $24.8\pm 0.1\%$ | $17.8\pm 0.1\%$ | $20.9\pm 0.2\%$
$\hat{\texttt{Err}}_{\text{M}}$ | $20.0\pm 0.1\%$ | $12.0\pm 0.0\%$ | $15.1\pm 0.1\%$
$\Delta\hat{\texttt{FPR}}$ | $6.1\pm 0.0\%$ | $6.2\pm 0.1\%$ | $6.4\pm 0.1\%$
$\hat{\texttt{FPR}}_{\text{F}}$ | $16.5\pm 0.1\%$ | $13.8\pm 0.1\%$ | $15.2\pm 0.1\%$
$\hat{\texttt{FPR}}_{\text{M}}$ | $10.4\pm 0.1\%$ | $7.6\pm 0.0\%$ | $8.8\pm 0.0\%$
$\Delta\hat{\texttt{FNR}}$ | $1.3\pm 0.0\%$ | $0.3\pm 0.0\%$ | $0.7\pm 0.1\%$
$\hat{\texttt{FNR}}_{\text{F}}$ | $8.3\pm 0.1\%$ | $4.0\pm 0.1\%$ | $5.6\pm 0.2\%$
$\hat{\texttt{FNR}}_{\text{M}}$ | $9.6\pm 0.1\%$ | $4.3\pm 0.1\%$ | $6.3\pm 0.1\%$

Figure 27: Decision trees on New Adult - CA - Employment, by sex

Abstention set metrics
---
| Simple | Super
$\Delta\hat{\texttt{AR}}$ | $0.7\pm 0.0\%$ | $0.4\pm 0.0\%$
$\hat{\texttt{AR}}_{\text{F}}$ | $20.3\pm 0.2\%$ | $7.7\pm 0.2\%$
$\hat{\texttt{AR}}_{\text{M}}$ | $19.6\pm 0.2\%$ | $7.3\pm 0.2\%$

Random forest prediction set metrics
---
| Baseline | Simple | Super
$\Delta\hat{\texttt{PR}}$ | $0.3\pm 0.1\%$ | $0.5\pm 0.1\%$ | $0.6\pm 0.1\%$
$\hat{\texttt{PR}}_{\text{F}}$ | $49.2\pm 0.1\%$ | $48.7\pm 0.2\%$ | $50.3\pm 0.2\%$
$\hat{\texttt{PR}}_{\text{M}}$ | $48.9\pm 0.2\%$ | $48.2\pm 0.1\%$ | $49.7\pm 0.1\%$
$\Delta\hat{\texttt{Err}}$ | $4.8\pm 0.0\%$ | $5.4\pm 0.0\%$ | $5.5\pm 0.2\%$
$\hat{\texttt{Err}}_{\text{F}}$ | $24.0\pm 0.1\%$ | $17.5\pm 0.1\%$ | $20.5\pm 0.2\%$
$\hat{\texttt{Err}}_{\text{M}}$ | $19.2\pm 0.1\%$ | $12.1\pm 0.1\%$ | $15.0\pm 0.0\%$
$\Delta\hat{\texttt{FPR}}$ | $6.0\pm 0.0\%$ | $5.9\pm 0.1\%$ | $6.2\pm 0.1\%$
$\hat{\texttt{FPR}}_{\text{F}}$ | $15.5\pm 0.1\%$ | $13.2\pm 0.1\%$ | $14.7\pm 0.1\%$
$\hat{\texttt{FPR}}_{\text{M}}$ | $9.5\pm 0.1\%$ | $7.3\pm 0.0\%$ | $8.5\pm 0.0\%$
$\Delta\hat{\texttt{FNR}}$ | $1.2\pm 0.0\%$ | $0.4\pm 0.0\%$ | $0.8\pm 0.0\%$
$\hat{\texttt{FNR}}_{\text{F}}$ | $8.5\pm 0.1\%$ | $4.3\pm 0.1\%$ | $5.8\pm 0.1\%$
$\hat{\texttt{FNR}}_{\text{M}}$ | $9.7\pm 0.1\%$ | $4.7\pm 0.1\%$ | $6.6\pm 0.1\%$

Figure 28: Random forests on New Adult - CA - Employment, by sex

Employment \- by race.

Abstention set metrics
---
| Simple | Super
$\Delta\hat{\texttt{AR}}$ | $0.1\pm 0.0\%$ | $0.0\pm 0.0\%$
$\hat{\texttt{AR}}_{\text{NW}}$ | $0.7\pm 0.0\%$ | $0.3\pm 0.0\%$
$\hat{\texttt{AR}}_{\text{W}}$ | $0.8\pm 0.0\%$ | $0.3\pm 0.0\%$

Logistic regression prediction set metrics
---
| Baseline | Simple | Super
$\Delta\hat{\texttt{PR}}$ | $1.1\pm 0.2\%$ | $1.2\pm 0.1\%$ | $1.2\pm 0.1\%$
$\hat{\texttt{PR}}_{\text{NW}}$ | $55.2\pm 0.3\%$ | $55.3\pm 0.2\%$ | $55.3\pm 0.2\%$
$\hat{\texttt{PR}}_{\text{W}}$ | $54.1\pm 0.1\%$ | $54.1\pm 0.1\%$ | $54.1\pm 0.1\%$
$\Delta\hat{\texttt{Err}}$ | $0.1\pm 0.0\%$ | $0.2\pm 0.1\%$ | $0.2\pm 0.0\%$
$\hat{\texttt{Err}}_{\text{NW}}$ | $23.3\pm 0.1\%$ | $23.0\pm 0.0\%$ | $23.1\pm 0.1\%$
$\hat{\texttt{Err}}_{\text{W}}$ | $23.4\pm 0.1\%$ | $23.2\pm 0.1\%$ | $23.3\pm 0.1\%$
$\Delta\hat{\texttt{FPR}}$ | $0.8\pm 0.0\%$ | $0.7\pm 0.1\%$ | $0.7\pm 0.1\%$
$\hat{\texttt{FPR}}_{\text{NW}}$ | $16.6\pm 0.1\%$ | $16.5\pm 0.0\%$ | $16.6\pm 0.0\%$
$\hat{\texttt{FPR}}_{\text{W}}$ | $15.8\pm 0.1\%$ | $15.8\pm 0.1\%$ | $15.9\pm 0.1\%$
$\Delta\hat{\texttt{FNR}}$ | $1.0\pm 0.0\%$ | $1.0\pm 0.0\%$ | $0.9\pm 0.0\%$
$\hat{\texttt{FNR}}_{\text{NW}}$ | $6.6\pm 0.1\%$ | $6.4\pm 0.1\%$ | $6.5\pm 0.1\%$
$\hat{\texttt{FNR}}_{\text{W}}$ | $7.6\pm 0.1\%$ | $7.4\pm 0.1\%$ | $7.4\pm 0.1\%$

Figure 29: Logistic regression on New Adult - CA - Employment, by race

Abstention set metrics
---
| Simple | Super
$\Delta\hat{\texttt{AR}}$ | $0.0\pm 0.1\%$ | $0.1\pm 0.1\%$
$\hat{\texttt{AR}}_{\text{NW}}$ | $22.4\pm 0.2\%$ | $8.2\pm 0.2\%$
$\hat{\texttt{AR}}_{\text{W}}$ | $22.4\pm 0.1\%$ | $8.1\pm 0.3\%$

Decision tree prediction set metrics
---
| Baseline | Simple | Super
$\Delta\hat{\texttt{PR}}$ | $0.2\pm 0.2\%$ | $0.6\pm 0.0\%$ | $1.2\pm 0.1\%$
$\hat{\texttt{PR}}_{\text{NW}}$ | $50.2\pm 0.3\%$ | $50.0\pm 0.1\%$ | $51.6\pm 0.0\%$
$\hat{\texttt{PR}}_{\text{W}}$ | $50.0\pm 0.1\%$ | $49.4\pm 0.1\%$ | $50.4\pm 0.1\%$
$\Delta\hat{\texttt{Err}}$ | $0.6\pm 0.0\%$ | $0.7\pm 0.0\%$ | $0.6\pm 0.0\%$
$\hat{\texttt{Err}}_{\text{NW}}$ | $22.1\pm 0.1\%$ | $14.5\pm 0.1\%$ | $17.7\pm 0.1\%$
$\hat{\texttt{Err}}_{\text{W}}$ | $22.7\pm 0.1\%$ | $15.2\pm 0.1\%$ | $18.3\pm 0.1\%$
$\Delta\hat{\texttt{FPR}}$ | $0.1\pm 0.0\%$ | $0.6\pm 0.1\%$ | $0.7\pm 0.1\%$
$\hat{\texttt{FPR}}_{\text{NW}}$ | $13.5\pm 0.1\%$ | $11.1\pm 0.0\%$ | $12.5\pm 0.0\%$
$\hat{\texttt{FPR}}_{\text{W}}$ | $13.4\pm 0.1\%$ | $10.5\pm 0.1\%$ | $11.8\pm 0.1\%$
$\Delta\hat{\texttt{FNR}}$ | $0.6\pm 0.0\%$ | $1.3\pm 0.0\%$ | $1.3\pm 0.0\%$
$\hat{\texttt{FNR}}_{\text{NW}}$ | $8.6\pm 0.1\%$ | $3.4\pm 0.1\%$ | $5.2\pm 0.1\%$
$\hat{\texttt{FNR}}_{\text{W}}$ | $9.2\pm 0.1\%$ | $4.7\pm 0.1\%$ | $6.5\pm 0.1\%$

Figure 30: Decision trees on New Adult - CA - Employment, by race

Abstention set metrics
---
| Simple | Super
$\Delta\hat{\texttt{AR}}$ | $0.3\pm 0.1\%$ | $0.1\pm 0.1\%$
$\hat{\texttt{AR}}_{\text{NW}}$ | $20.1\pm 0.1\%$ | $7.6\pm 0.1\%$
$\hat{\texttt{AR}}_{\text{W}}$ | $19.8\pm 0.2\%$ | $7.5\pm 0.2\%$

Random forest prediction set metrics
---
| Baseline | Simple | Super
$\Delta\hat{\texttt{PR}}$ | $0.6\pm 0.2\%$ | $1.0\pm 0.0\%$ | $1.4\pm 0.1\%$
$\hat{\texttt{PR}}_{\text{NW}}$ | $49.4\pm 0.3\%$ | $49.1\pm 0.1\%$ | $50.9\pm 0.0\%$
$\hat{\texttt{PR}}_{\text{W}}$ | $48.8\pm 0.1\%$ | $48.1\pm 0.1\%$ | $49.5\pm 0.1\%$
$\Delta\hat{\texttt{Err}}$ | $0.5\pm 0.0\%$ | $0.7\pm 0.0\%$ | $0.5\pm 0.0\%$
$\hat{\texttt{Err}}_{\text{NW}}$ | $21.3\pm 0.1\%$ | $14.4\pm 0.1\%$ | $17.5\pm 0.1\%$
$\hat{\texttt{Err}}_{\text{W}}$ | $21.8\pm 0.1\%$ | $15.1\pm 0.1\%$ | $18.0\pm 0.1\%$
$\Delta\hat{\texttt{FPR}}$ | $0.3\pm 0.1\%$ | $0.6\pm 0.1\%$ | $0.8\pm 0.1\%$
$\hat{\texttt{FPR}}_{\text{NW}}$ | $12.7\pm 0.2\%$ | $10.7\pm 0.0\%$ | $12.1\pm 0.0\%$
$\hat{\texttt{FPR}}_{\text{W}}$ | $12.4\pm 0.1\%$ | $10.1\pm 0.1\%$ | $11.3\pm 0.1\%$
$\Delta\hat{\texttt{FNR}}$ | $0.8\pm 0.0\%$ | $1.3\pm 0.0\%$ | $1.3\pm 0.0\%$
$\hat{\texttt{FNR}}_{\text{NW}}$ | $8.6\pm 0.1\%$ | $3.7\pm 0.1\%$ | $5.4\pm 0.1\%$
$\hat{\texttt{FNR}}_{\text{W}}$ | $9.4\pm 0.1\%$ | $5.0\pm 0.1\%$ | $6.7\pm 0.1\%$

Figure 31: Random forests on New Adult - CA - Employment, by race

Public Coverage \- by sex.

Abstention set metrics
---
| Simple | Super
$\Delta\hat{\texttt{AR}}$ | $0.0\pm 0.1\%$ | $0.0\pm 0.0\%$
$\hat{\texttt{AR}}_{\text{F}}$ | $1.4\pm 0.1\%$ | $0.4\pm 0.0\%$
$\hat{\texttt{AR}}_{\text{M}}$ | $1.4\pm 0.0\%$ | $0.4\pm 0.0\%$

Logistic regression prediction set metrics
---
| Baseline | Simple | Super
$\Delta\hat{\texttt{PR}}$ | $2.6\pm 0.1\%$ | $2.6\pm 0.1\%$ | $2.6\pm 0.1\%$
$\hat{\texttt{PR}}_{\text{F}}$ | $15.1\pm 0.2\%$ | $14.7\pm 0.2\%$ | $15.1\pm 0.2\%$
$\hat{\texttt{PR}}_{\text{M}}$ | $17.7\pm 0.3\%$ | $17.3\pm 0.1\%$ | $17.7\pm 0.1\%$
$\Delta\hat{\texttt{Err}}$ | $0.9\pm 0.1\%$ | $0.5\pm 0.0\%$ | $0.5\pm 0.0\%$
$\hat{\texttt{Err}}_{\text{F}}$ | $31.2\pm 0.3\%$ | $30.8\pm 0.2\%$ | $31.0\pm 0.2\%$
$\hat{\texttt{Err}}_{\text{M}}$ | $32.1\pm 0.2\%$ | $31.3\pm 0.2\%$ | $31.5\pm 0.2\%$
$\Delta\hat{\texttt{FPR}}$ | $0.0\pm 0.0\%$ | $0.0\pm 0.0\%$ | $0.0\pm 0.1\%$
$\hat{\texttt{FPR}}_{\text{F}}$ | $5.5\pm 0.1\%$ | $5.1\pm 0.1\%$ | $5.3\pm 0.1\%$
$\hat{\texttt{FPR}}_{\text{M}}$ | $5.5\pm 0.1\%$ | $5.1\pm 0.1\%$ | $5.3\pm 0.2\%$
$\Delta\hat{\texttt{FNR}}$ | $0.9\pm 0.1\%$ | $0.5\pm 0.0\%$ | $0.5\pm 0.0\%$
$\hat{\texttt{FNR}}_{\text{F}}$ | $25.7\pm 0.3\%$ | $25.7\pm 0.2\%$ | $25.7\pm 0.2\%$
$\hat{\texttt{FNR}}_{\text{M}}$ | $26.6\pm 0.2\%$ | $26.2\pm 0.2\%$ | $26.2\pm 0.2\%$

Figure 32: Logistic regression on New Adult - CA - Public Coverage, by sex

Abstention set metrics
---
| Simple | Super
$\Delta\hat{\texttt{AR}}$ | $0.1\pm 0.0\%$ | $0.3\pm 0.1\%$
$\hat{\texttt{AR}}_{\text{F}}$ | $60.7\pm 0.4\%$ | $18.6\pm 0.4\%$
$\hat{\texttt{AR}}_{\text{M}}$ | $60.8\pm 0.4\%$ | $18.3\pm 0.3\%$

Decision tree prediction set metrics
---
| Baseline | Simple | Super
$\Delta\hat{\texttt{PR}}$ | $2.6\pm 0.2\%$ | $6.8\pm 0.1\%$ | $3.0\pm 0.0\%$
$\hat{\texttt{PR}}_{\text{F}}$ | $35.5\pm 0.2\%$ | $20.5\pm 0.4\%$ | $27.5\pm 0.3\%$
$\hat{\texttt{PR}}_{\text{M}}$ | $38.1\pm 0.4\%$ | $27.3\pm 0.3\%$ | $30.5\pm 0.3\%$
$\Delta\hat{\texttt{Err}}$ | $0.1\pm 0.1\%$ | $0.5\pm 0.1\%$ | $0.2\pm 0.1\%$
$\hat{\texttt{Err}}_{\text{F}}$ | $35.1\pm 0.2\%$ | $18.8\pm 0.3\%$ | $26.7\pm 0.3\%$
$\hat{\texttt{Err}}_{\text{M}}$ | $35.2\pm 0.1\%$ | $19.3\pm 0.2\%$ | $26.9\pm 0.4\%$
$\Delta\hat{\texttt{FPR}}$ | $0.4\pm 0.1\%$ | $0.1\pm 0.0\%$ | $0.3\pm 0.1\%$
$\hat{\texttt{FPR}}_{\text{F}}$ | $17.6\pm 0.1\%$ | $4.6\pm 0.2\%$ | $10.0\pm 0.3\%$
$\hat{\texttt{FPR}}_{\text{M}}$ | $17.2\pm 0.2\%$ | $4.5\pm 0.2\%$ | $9.7\pm 0.2\%$
$\Delta\hat{\texttt{FNR}}$ | $0.6\pm 0.0\%$ | $0.6\pm 0.2\%$ | $0.5\pm 0.3\%$
$\hat{\texttt{FNR}}_{\text{F}}$ | $17.4\pm 0.2\%$ | $14.2\pm 0.3\%$ | $16.7\pm 0.1\%$
$\hat{\texttt{FNR}}_{\text{M}}$ | $18.0\pm 0.2\%$ | $14.8\pm 0.1\%$ | $17.2\pm 0.4\%$

Figure 33: Decision trees on New Adult - CA - Public Coverage, by sex

Abstention set metrics
---
| Simple | Super
$\Delta\hat{\texttt{AR}}$ | $0.2\pm 0.0\%$ | $0.2\pm 0.2\%$
$\hat{\texttt{AR}}_{\text{F}}$ | $48.1\pm 0.3\%$ | $13.2\pm 0.1\%$
$\hat{\texttt{AR}}_{\text{M}}$ | $47.9\pm 0.3\%$ | $13.0\pm 0.3\%$

Random forest prediction set metrics
---
| Baseline | Simple | Super
$\Delta\hat{\texttt{PR}}$ | $2.5\pm 0.1\%$ | $5.3\pm 0.0\%$ | $2.6\pm 0.1\%$
$\hat{\texttt{PR}}_{\text{F}}$ | $31.9\pm 0.3\%$ | $19.5\pm 0.3\%$ | $25.7\pm 0.4\%$
$\hat{\texttt{PR}}_{\text{M}}$ | $34.4\pm 0.4\%$ | $24.8\pm 0.3\%$ | $28.3\pm 0.3\%$
$\Delta\hat{\texttt{Err}}$ | $0.4\pm 0.1\%$ | $1.0\pm 0.2\%$ | $0.3\pm 0.1\%$
$\hat{\texttt{Err}}_{\text{F}}$ | $32.3\pm 0.2\%$ | $19.3\pm 0.3\%$ | $26.3\pm 0.2\%$
$\hat{\texttt{Err}}_{\text{M}}$ | $32.7\pm 0.1\%$ | $20.3\pm 0.1\%$ | $26.6\pm 0.3\%$
$\Delta\hat{\texttt{FPR}}$ | $0.2\pm 0.1\%$ | $0.2\pm 0.1\%$ | $0.3\pm 0.1\%$
$\hat{\texttt{FPR}}_{\text{F}}$ | $14.4\pm 0.1\%$ | $4.1\pm 0.2\%$ | $8.7\pm 0.3\%$
$\hat{\texttt{FPR}}_{\text{M}}$ | $14.2\pm 0.2\%$ | $4.3\pm 0.1\%$ | $8.4\pm 0.2\%$
$\Delta\hat{\texttt{FNR}}$ | $0.7\pm 0.1\%$ | $0.8\pm 0.0\%$ | $0.7\pm 0.1\%$
$\hat{\texttt{FNR}}_{\text{F}}$ | $17.9\pm 0.2\%$ | $15.2\pm 0.2\%$ | $17.5\pm 0.2\%$
$\hat{\texttt{FNR}}_{\text{M}}$ | $18.6\pm 0.3\%$ | $16.0\pm 0.2\%$ | $18.2\pm 0.3\%$

Figure 34: Random forests on New Adult - CA - Public Coverage, by sex

Public Coverage \- by race.

Abstention set metrics
---
| Simple | Super
$\Delta\hat{\texttt{AR}}$ | $0.1\pm 0.0\%$ | $0.1\pm 0.0\%$
$\hat{\texttt{AR}}_{\text{NW}}$ | $1.5\pm 0.1\%$ | $0.4\pm 0.0\%$
$\hat{\texttt{AR}}_{\text{W}}$ | $1.4\pm 0.1\%$ | $0.3\pm 0.0\%$

Logistic regression prediction set metrics
---
| Baseline | Simple | Super
$\Delta\hat{\texttt{PR}}$ | $0.1\pm 0.1\%$ | $0.1\pm 0.1\%$ | $0.1\pm 0.1\%$
$\hat{\texttt{PR}}_{\text{NW}}$ | $16.3\pm 0.3\%$ | $15.8\pm 0.3\%$ | $16.2\pm 0.3\%$
$\hat{\texttt{PR}}_{\text{W}}$ | $16.2\pm 0.2\%$ | $15.9\pm 0.2\%$ | $16.3\pm 0.2\%$
$\Delta\hat{\texttt{Err}}$ | $3.2\pm 0.0\%$ | $2.7\pm 0.1\%$ | $2.8\pm 0.1\%$
$\hat{\texttt{Err}}_{\text{NW}}$ | $33.4\pm 0.3\%$ | $32.6\pm 0.3\%$ | $32.8\pm 0.3\%$
$\hat{\texttt{Err}}_{\text{W}}$ | $30.2\pm 0.3\%$ | $29.9\pm 0.2\%$ | $30.0\pm 0.2\%$
$\Delta\hat{\texttt{FPR}}$ | $0.2\pm 0.0\%$ | $0.1\pm 0.0\%$ | $0.1\pm 0.1\%$
$\hat{\texttt{FPR}}_{\text{NW}}$ | $5.6\pm 0.1\%$ | $5.2\pm 0.1\%$ | $5.4\pm 0.2\%$
$\hat{\texttt{FPR}}_{\text{W}}$ | $5.4\pm 0.1\%$ | $5.1\pm 0.1\%$ | $5.3\pm 0.1\%$
$\Delta\hat{\texttt{FNR}}$ | $3.0\pm 0.1\%$ | $2.6\pm 0.1\%$ | $2.7\pm 0.1\%$
$\hat{\texttt{FNR}}_{\text{NW}}$ | $27.8\pm 0.3\%$ | $27.4\pm 0.3\%$ | $27.4\pm 0.3\%$
$\hat{\texttt{FNR}}_{\text{W}}$ | $24.8\pm 0.2\%$ | $24.8\pm 0.2\%$ | $24.7\pm 0.2\%$

Figure 35: Logistic regression on New Adult - CA - Public Coverage, by race

Abstention set metrics
---
| Simple | Super
$\Delta\hat{\texttt{AR}}$ | $2.8\pm 0.0\%$ | $1.3\pm 0.1\%$
$\hat{\texttt{AR}}_{\text{NW}}$ | $62.3\pm 0.3\%$ | $19.2\pm 0.3\%$
$\hat{\texttt{AR}}_{\text{W}}$ | $59.5\pm 0.3\%$ | $17.9\pm 0.4\%$

Decision tree prediction set metrics
---
| Baseline | Simple | Super
$\Delta\hat{\texttt{PR}}$ | $1.5\pm 0.0\%$ | $2.2\pm 0.2\%$ | $1.4\pm 0.1\%$
$\hat{\texttt{PR}}_{\text{NW}}$ | $37.5\pm 0.2\%$ | $24.8\pm 0.3\%$ | $29.6\pm 0.4\%$
$\hat{\texttt{PR}}_{\text{W}}$ | $36.0\pm 0.2\%$ | $22.6\pm 0.5\%$ | $28.2\pm 0.3\%$
$\Delta\hat{\texttt{Err}}$ | $2.2\pm 0.0\%$ | $3.0\pm 0.1\%$ | $2.7\pm 0.1\%$
$\hat{\texttt{Err}}_{\text{NW}}$ | $36.4\pm 0.1\%$ | $20.8\pm 0.3\%$ | $28.3\pm 0.2\%$
$\hat{\texttt{Err}}_{\text{W}}$ | $34.2\pm 0.1\%$ | $17.8\pm 0.4\%$ | $25.6\pm 0.3\%$
$\Delta\hat{\texttt{FPR}}$ | $0.4\pm 0.1\%$ | $0.7\pm 0.0\%$ | $0.6\pm 0.0\%$
$\hat{\texttt{FPR}}_{\text{NW}}$ | $17.7\pm 0.1\%$ | $5.0\pm 0.2\%$ | $10.2\pm 0.2\%$
$\hat{\texttt{FPR}}_{\text{W}}$ | $17.3\pm 0.2\%$ | $4.3\pm 0.2\%$ | $9.6\pm 0.2\%$
$\Delta\hat{\texttt{FNR}}$ | $1.8\pm 0.0\%$ | $2.3\pm 0.1\%$ | $2.1\pm 0.1\%$
$\hat{\texttt{FNR}}_{\text{NW}}$ | $18.7\pm 0.2\%$ | $15.8\pm 0.2\%$ | $18.1\pm 0.3\%$
$\hat{\texttt{FNR}}_{\text{W}}$ | $16.9\pm 0.2\%$ | $13.5\pm 0.3\%$ | $16.0\pm 0.2\%$

Figure 36: Decision trees on New Adult - CA - Public Coverage, by race

Abstention set metrics
---
| Simple | Super
$\Delta\hat{\texttt{AR}}$ | $3.6\pm 0.1\%$ | $1.2\pm 0.0\%$
$\hat{\texttt{AR}}_{\text{NW}}$ | $50.0\pm 0.2\%$ | $13.8\pm 0.2\%$
$\hat{\texttt{AR}}_{\text{W}}$ | $46.4\pm 0.3\%$ | $12.6\pm 0.2\%$

Random forest prediction set metrics
---
| Baseline | Simple | Super
$\Delta\hat{\texttt{PR}}$ | $1.2\pm 0.0\%$ | $1.0\pm 0.1\%$ | $0.7\pm 0.0\%$
$\hat{\texttt{PR}}_{\text{NW}}$ | $33.7\pm 0.3\%$ | $22.4\pm 0.4\%$ | $27.3\pm 0.4\%$
$\hat{\texttt{PR}}_{\text{W}}$ | $32.5\pm 0.3\%$ | $21.4\pm 0.3\%$ | $26.6\pm 0.4\%$
$\Delta\hat{\texttt{Err}}$ | $2.7\pm 0.1\%$ | $2.9\pm 0.0\%$ | $2.6\pm 0.1\%$
$\hat{\texttt{Err}}_{\text{NW}}$ | $34.0\pm 0.2\%$ | $21.4\pm 0.3\%$ | $27.9\pm 0.3\%$
$\hat{\texttt{Err}}_{\text{W}}$ | $31.3\pm 0.1\%$ | $18.5\pm 0.3\%$ | $25.3\pm 0.2\%$
$\Delta\hat{\texttt{FPR}}$ | $0.5\pm 0.0\%$ | $0.4\pm 0.0\%$ | $0.5\pm 0.0\%$
$\hat{\texttt{FPR}}_{\text{NW}}$ | $14.6\pm 0.2\%$ | $4.4\pm 0.2\%$ | $8.9\pm 0.2\%$
$\hat{\texttt{FPR}}_{\text{W}}$ | $14.1\pm 0.2\%$ | $4.0\pm 0.2\%$ | $8.4\pm 0.2\%$
$\Delta\hat{\texttt{FNR}}$ | $2.2\pm 0.0\%$ | $2.5\pm 0.0\%$ | $2.3\pm 0.1\%$
$\hat{\texttt{FNR}}_{\text{NW}}$ | $19.4\pm 0.2\%$ | $17.0\pm 0.2\%$ | $19.1\pm 0.3\%$
$\hat{\texttt{FNR}}_{\text{W}}$ | $17.2\pm 0.2\%$ | $14.5\pm 0.2\%$ | $16.8\pm 0.2\%$

Figure 37: Random forests on New Adult - CA - Public Coverage, by race

####  E.4.6 HMDA

$\hat{\texttt{SC}}$ CDFs for two states (NY, TX) in HMDA -2017, using
${\bm{g}}=\texttt{ethnicity},\;\texttt{race},$ and sex, and associated error
metrics on the prediction set. Baseline metrics computed with $B=101$ models.
For simple, $B=101$ models; for super, $B=101$ ensemble models, each composed
of $21$ underlying models for NY; $15$ for TX. We repeat for $5$ test/train
splits. We also report abstention rate $\hat{\texttt{AR}}$.

NY - 2017 \- by ethnicity.

Abstention set metrics
---
| Simple | Super
$\Delta\hat{\texttt{AR}}$ | $0.0\pm 0.1\%$ | $0.1\pm 0.1\%$
$\hat{\texttt{AR}}_{\text{HL}}$ | $1.9\pm 0.2\%$ | $0.4\pm 0.1\%$
$\hat{\texttt{AR}}_{\text{NHL}}$ | $1.9\pm 0.1\%$ | $0.5\pm 0.0\%$

Logistic regression prediction set metrics
---
| Baseline | Simple | Super
$\Delta\hat{\texttt{PR}}$ | $9.7\pm 0.2\%$ | $10.5\pm 0.5\%$ | $10.4\pm 0.5\%$
$\hat{\texttt{PR}}_{\text{HL}}$ | $73.5\pm 0.3\%$ | $73.5\pm 0.6\%$ | $73.1\pm 0.6\%$
$\hat{\texttt{PR}}_{\text{NHL}}$ | $83.2\pm 0.1\%$ | $84.0\pm 0.1\%$ | $83.5\pm 0.1\%$
$\Delta\hat{\texttt{Err}}$ | $1.3\pm 0.2\%$ | $1.7\pm 0.3\%$ | $1.8\pm 0.4\%$
$\hat{\texttt{Err}}_{\text{HL}}$ | $18.7\pm 0.3\%$ | $18.4\pm 0.4\%$ | $18.9\pm 0.5\%$
$\hat{\texttt{Err}}_{\text{NHL}}$ | $17.4\pm 0.1\%$ | $16.7\pm 0.1\%$ | $17.1\pm 0.1\%$
$\Delta\hat{\texttt{FPR}}$ | $0.8\pm 0.1\%$ | $1.0\pm 0.1\%$ | $0.9\pm 0.1\%$
$\hat{\texttt{FPR}}_{\text{HL}}$ | $10.7\pm 0.2\%$ | $10.2\pm 0.2\%$ | $10.5\pm 0.2\%$
$\hat{\texttt{FPR}}_{\text{NHL}}$ | $11.5\pm 0.1\%$ | $11.2\pm 0.1\%$ | $11.4\pm 0.1\%$
$\Delta\hat{\texttt{FNR}}$ | $2.2\pm 0.2\%$ | $2.7\pm 0.3\%$ | $2.7\pm 0.4\%$
$\hat{\texttt{FNR}}_{\text{HL}}$ | $8.0\pm 0.3\%$ | $8.2\pm 0.4\%$ | $8.4\pm 0.5\%$
$\hat{\texttt{FNR}}_{\text{NHL}}$ | $5.8\pm 0.1\%$ | $5.5\pm 0.1\%$ | $5.7\pm 0.1\%$

Figure 38: Logistic regression on HMDA - 2017 - NY, by ethnicity

Abstention set metrics
---
| Simple | Super
$\Delta\hat{\texttt{AR}}$ | $1.2\pm 0.2\%$ | $2.3\pm 0.1\%$
$\hat{\texttt{AR}}_{\text{HL}}$ | $43.3\pm 0.4\%$ | $15.0\pm 0.3\%$
$\hat{\texttt{AR}}_{\text{NHL}}$ | $42.1\pm 0.2\%$ | $12.7\pm 0.2\%$

Decision tree prediction set metrics
---
| Baseline | Simple | Super
$\Delta\hat{\texttt{PR}}$ | $4.0\pm 0.2\%$ | $1.5\pm 0.2\%$ | $8.2\pm 0.5\%$
$\hat{\texttt{PR}}_{\text{HL}}$ | $73.1\pm 0.3\%$ | $94.0\pm 0.3\%$ | $74.1\pm 0.6\%$
$\hat{\texttt{PR}}_{\text{NHL}}$ | $77.1\pm 0.1\%$ | $95.5\pm 0.1\%$ | $82.3\pm 0.1\%$
$\Delta\hat{\texttt{Err}}$ | $0.4\pm 0.1\%$ | $0.7\pm 0.1\%$ | $0.6\pm 0.2\%$
$\hat{\texttt{Err}}_{\text{HL}}$ | $20.6\pm 0.2\%$ | $2.7\pm 0.2\%$ | $12.2\pm 0.4\%$
$\hat{\texttt{Err}}_{\text{NHL}}$ | $20.2\pm 0.1\%$ | $3.4\pm 0.1\%$ | $11.6\pm 0.2\%$
$\Delta\hat{\texttt{FPR}}$ | $1.6\pm 0.1\%$ | $1.1\pm 0.0\%$ | $1.1\pm 0.1\%$
$\hat{\texttt{FPR}}_{\text{HL}}$ | $11.5\pm 0.2\%$ | $1.4\pm 0.1\%$ | $5.1\pm 0.2\%$
$\hat{\texttt{FPR}}_{\text{NHL}}$ | $9.9\pm 0.1\%$ | $2.5\pm 0.1\%$ | $6.2\pm 0.1\%$
$\Delta\hat{\texttt{FNR}}$ | $1.2\pm 0.1\%$ | $0.4\pm 0.1\%$ | $1.6\pm 0.3\%$
$\hat{\texttt{FNR}}_{\text{HL}}$ | $9.1\pm 0.2\%$ | $1.3\pm 0.2\%$ | $7.0\pm 0.4\%$
$\hat{\texttt{FNR}}_{\text{NHL}}$ | $10.3\pm 0.1\%$ | $0.9\pm 0.1\%$ | $5.4\pm 0.1\%$

Figure 39: Decision trees on HMDA - 2017 - NY, by ethnicity

Abstention set metrics
---
| Simple | Super
$\Delta\hat{\texttt{AR}}$ | $3.6\pm 0.2\%$ | $1.4\pm 0.3\%$
$\hat{\texttt{AR}}_{\text{HL}}$ | $33.9\pm 0.3\%$ | $9.5\pm 0.4\%$
$\hat{\texttt{AR}}_{\text{NHL}}$ | $30.3\pm 0.1\%$ | $8.1\pm 0.1\%$

Random forest prediction set metrics
---
| Baseline | Simple | Super
$\Delta\hat{\texttt{PR}}$ | $6.3\pm 0.3\%$ | $5.1\pm 0.6\%$ | $9.4\pm 0.4\%$
$\hat{\texttt{PR}}_{\text{HL}}$ | $71.9\pm 0.4\%$ | $85.8\pm 0.7\%$ | $71.6\pm 0.5\%$
$\hat{\texttt{PR}}_{\text{NHL}}$ | $78.2\pm 0.1\%$ | $90.9\pm 0.1\%$ | $81.0\pm 0.1\%$
$\Delta\hat{\texttt{Err}}$ | $1.1\pm 0.1\%$ | $0.5\pm 0.3\%$ | $1.0\pm 0.2\%$
$\hat{\texttt{Err}}_{\text{HL}}$ | $19.1\pm 0.2\%$ | $5.8\pm 0.4\%$ | $13.7\pm 0.4\%$
$\hat{\texttt{Err}}_{\text{NHL}}$ | $18.0\pm 0.1\%$ | $6.3\pm 0.1\%$ | $12.7\pm 0.2\%$
$\Delta\hat{\texttt{FPR}}$ | $0.7\pm 0.1\%$ | $1.5\pm 0.1\%$ | $1.2\pm 0.1\%$
$\hat{\texttt{FPR}}_{\text{HL}}$ | $10.1\pm 0.2\%$ | $2.6\pm 0.1\%$ | $5.7\pm 0.2\%$
$\hat{\texttt{FPR}}_{\text{NHL}}$ | $9.4\pm 0.1\%$ | $4.1\pm 0.0\%$ | $6.9\pm 0.1\%$
$\Delta\hat{\texttt{FNR}}$ | $0.4\pm 0.2\%$ | $1.1\pm 0.2\%$ | $2.2\pm 0.2\%$
$\hat{\texttt{FNR}}_{\text{HL}}$ | $9.0\pm 0.3\%$ | $3.3\pm 0.3\%$ | $8.0\pm 0.3\%$
$\hat{\texttt{FNR}}_{\text{NHL}}$ | $8.6\pm 0.1\%$ | $2.2\pm 0.1\%$ | $5.8\pm 0.1\%$

Figure 40: Random forests on HMDA - 2017 - NY, by ethnicity

NY - 2017 \- by race.

Abstention set metrics
---
| Simple | Super
$\Delta\hat{\texttt{AR}}$ | $0.1\pm 0.0\%$ | $0.0\pm 0.1\%$
$\hat{\texttt{AR}}_{\text{NW}}$ | $2.0\pm 0.1\%$ | $0.5\pm 0.1\%$
$\hat{\texttt{AR}}_{\text{W}}$ | $1.9\pm 0.1\%$ | $0.5\pm 0.0\%$

Logistic regression prediction set metrics
---
| Baseline | Simple | Super
$\Delta\hat{\texttt{PR}}$ | $11.5\pm 0.2\%$ | $11.6\pm 0.3\%$ | $11.5\pm 0.3\%$
$\hat{\texttt{PR}}_{\text{NW}}$ | $73.3\pm 0.3\%$ | $73.9\pm 0.4\%$ | $73.5\pm 0.4\%$
$\hat{\texttt{PR}}_{\text{W}}$ | $84.8\pm 0.1\%$ | $85.5\pm 0.1\%$ | $85.0\pm 0.1\%$
$\Delta\hat{\texttt{Err}}$ | $2.8\pm 0.1\%$ | $2.9\pm 0.1\%$ | $3.0\pm 0.0\%$
$\hat{\texttt{Err}}_{\text{NW}}$ | $19.7\pm 0.2\%$ | $19.1\pm 0.2\%$ | $19.6\pm 0.1\%$
$\hat{\texttt{Err}}_{\text{W}}$ | $16.9\pm 0.1\%$ | $16.2\pm 0.1\%$ | $16.6\pm 0.1\%$
$\Delta\hat{\texttt{FPR}}$ | $0.2\pm 0.1\%$ | $0.2\pm 0.1\%$ | $0.1\pm 0.1\%$
$\hat{\texttt{FPR}}_{\text{NW}}$ | $11.3\pm 0.2\%$ | $11.0\pm 0.2\%$ | $11.3\pm 0.2\%$
$\hat{\texttt{FPR}}_{\text{W}}$ | $11.5\pm 0.1\%$ | $11.2\pm 0.1\%$ | $11.4\pm 0.1\%$
$\Delta\hat{\texttt{FNR}}$ | $3.0\pm 0.1\%$ | $3.0\pm 0.1\%$ | $3.0\pm 0.1\%$
$\hat{\texttt{FNR}}_{\text{NW}}$ | $8.4\pm 0.2\%$ | $8.1\pm 0.2\%$ | $8.3\pm 0.2\%$
$\hat{\texttt{FNR}}_{\text{W}}$ | $5.4\pm 0.1\%$ | $5.1\pm 0.1\%$ | $5.3\pm 0.1\%$

Figure 41: Logistic regression on HMDA - 2017 - NY, by race

Abstention set metrics
---
| Simple | Super
$\Delta\hat{\texttt{AR}}$ | $6.2\pm 0.2\%$ | $2.9\pm 0.0\%$
$\hat{\texttt{AR}}_{\text{NW}}$ | $47.1\pm 0.4\%$ | $15.2\pm 0.2\%$
$\hat{\texttt{AR}}_{\text{W}}$ | $40.9\pm 0.2\%$ | $12.3\pm 0.2\%$

Decision tree prediction set metrics
---
| Baseline | Simple | Super
$\Delta\hat{\texttt{PR}}$ | $6.2\pm 0.2\%$ | $3.0\pm 0.1\%$ | $9.1\pm 0.1\%$
$\hat{\texttt{PR}}_{\text{NW}}$ | $71.8\pm 0.3\%$ | $93.0\pm 0.2\%$ | $74.4\pm 0.3\%$
$\hat{\texttt{PR}}_{\text{W}}$ | $78.0\pm 0.1\%$ | $96.0\pm 0.1\%$ | $83.5\pm 0.2\%$
$\Delta\hat{\texttt{Err}}$ | $3.1\pm 0.0\%$ | $0.7\pm 0.1\%$ | $2.3\pm 0.0\%$
$\hat{\texttt{Err}}_{\text{NW}}$ | $22.7\pm 0.1\%$ | $3.9\pm 0.2\%$ | $13.5\pm 0.2\%$
$\hat{\texttt{Err}}_{\text{W}}$ | $19.6\pm 0.1\%$ | $3.2\pm 0.1\%$ | $11.2\pm 0.2\%$
$\Delta\hat{\texttt{FPR}}$ | $2.5\pm 0.1\%$ | $0.2\pm 0.0\%$ | $0.6\pm 0.1\%$
$\hat{\texttt{FPR}}_{\text{NW}}$ | $12.0\pm 0.2\%$ | $2.5\pm 0.1\%$ | $6.6\pm 0.2\%$
$\hat{\texttt{FPR}}_{\text{W}}$ | $9.5\pm 0.1\%$ | $2.3\pm 0.1\%$ | $6.0\pm 0.1\%$
$\Delta\hat{\texttt{FNR}}$ | $0.6\pm 0.0\%$ | $0.5\pm 0.1\%$ | $1.8\pm 0.1\%$
$\hat{\texttt{FNR}}_{\text{NW}}$ | $10.7\pm 0.1\%$ | $1.4\pm 0.1\%$ | $6.9\pm 0.2\%$
$\hat{\texttt{FNR}}_{\text{W}}$ | $10.1\pm 0.1\%$ | $0.9\pm 0.0\%$ | $5.1\pm 0.1\%$

Figure 42: Decision trees on HMDA - 2017 - NY, by race

Abstention set metrics
---
| Simple | Super
$\Delta\hat{\texttt{AR}}$ | $6.3\pm 0.3\%$ | $1.8\pm 0.2\%$
$\hat{\texttt{AR}}_{\text{NW}}$ | $35.6\pm 0.4\%$ | $9.6\pm 0.3\%$
$\hat{\texttt{AR}}_{\text{W}}$ | $29.3\pm 0.1\%$ | $7.8\pm 0.1\%$

Random forest prediction set metrics
---
| Baseline | Simple | Super
$\Delta\hat{\texttt{PR}}$ | $7.9\pm 0.2\%$ | $6.5\pm 0.1\%$ | $9.8\pm 0.1\%$
$\hat{\texttt{PR}}_{\text{NW}}$ | $71.5\pm 0.3\%$ | $85.3\pm 0.2\%$ | $72.4\pm 0.3\%$
$\hat{\texttt{PR}}_{\text{W}}$ | $79.4\pm 0.1\%$ | $91.8\pm 0.1\%$ | $82.2\pm 0.2\%$
$\Delta\hat{\texttt{Err}}$ | $3.3\pm 0.0\%$ | $1.3\pm 0.0\%$ | $2.7\pm 0.1\%$
$\hat{\texttt{Err}}_{\text{NW}}$ | $20.7\pm 0.1\%$ | $7.3\pm 0.1\%$ | $14.9\pm 0.1\%$
$\hat{\texttt{Err}}_{\text{W}}$ | $17.4\pm 0.1\%$ | $6.0\pm 0.1\%$ | $12.2\pm 0.2\%$
$\Delta\hat{\texttt{FPR}}$ | $1.8\pm 0.1\%$ | $0.0\pm 0.0\%$ | $0.5\pm 0.0\%$
$\hat{\texttt{FPR}}_{\text{NW}}$ | $10.9\pm 0.2\%$ | $4.0\pm 0.1\%$ | $7.2\pm 0.1\%$
$\hat{\texttt{FPR}}_{\text{W}}$ | $9.1\pm 0.1\%$ | $4.0\pm 0.1\%$ | $6.7\pm 0.1\%$
$\Delta\hat{\texttt{FNR}}$ | $1.4\pm 0.0\%$ | $1.3\pm 0.0\%$ | $2.1\pm 0.1\%$
$\hat{\texttt{FNR}}_{\text{NW}}$ | $9.8\pm 0.1\%$ | $3.3\pm 0.1\%$ | $7.7\pm 0.1\%$
$\hat{\texttt{FNR}}_{\text{W}}$ | $8.4\pm 0.1\%$ | $2.0\pm 0.1\%$ | $5.6\pm 0.2\%$

Figure 43: Random forests on HMDA - 2017 - NY, by race

NY - 2017 \- by sex.

Abstention set metrics
---
| Simple | Super
$\Delta\hat{\texttt{AR}}$ | $0.1\pm 0.0\%$ | $0.0\pm 0.0\%$
$\hat{\texttt{AR}}_{\text{F}}$ | $2.0\pm 0.1\%$ | $0.5\pm 0.0\%$
$\hat{\texttt{AR}}_{\text{M}}$ | $1.9\pm 0.1\%$ | $0.5\pm 0.0\%$

Logistic regression prediction set metrics
---
| Baseline | Simple | Super
$\Delta\hat{\texttt{PR}}$ | $1.5\pm 0.0\%$ | $1.5\pm 0.2\%$ | $1.5\pm 0.2\%$
$\hat{\texttt{PR}}_{\text{F}}$ | $81.5\pm 0.1\%$ | $82.2\pm 0.3\%$ | $81.7\pm 0.3\%$
$\hat{\texttt{PR}}_{\text{M}}$ | $83.0\pm 0.1\%$ | $83.7\pm 0.1\%$ | $83.2\pm 0.1\%$
$\Delta\hat{\texttt{Err}}$ | $0.5\pm 0.1\%$ | $0.6\pm 0.1\%$ | $0.6\pm 0.1\%$
$\hat{\texttt{Err}}_{\text{F}}$ | $17.8\pm 0.2\%$ | $17.2\pm 0.2\%$ | $17.6\pm 0.2\%$
$\hat{\texttt{Err}}_{\text{M}}$ | $17.3\pm 0.1\%$ | $16.6\pm 0.1\%$ | $17.0\pm 0.1\%$
$\Delta\hat{\texttt{FPR}}$ | $0.2\pm 0.1\%$ | $0.3\pm 0.1\%$ | $0.2\pm 0.1\%$
$\hat{\texttt{FPR}}_{\text{F}}$ | $11.6\pm 0.2\%$ | $11.3\pm 0.2\%$ | $11.5\pm 0.2\%$
$\hat{\texttt{FPR}}_{\text{M}}$ | $11.4\pm 0.1\%$ | $11.0\pm 0.1\%$ | $11.3\pm 0.1\%$
$\Delta\hat{\texttt{FNR}}$ | $0.3\pm 0.0\%$ | $0.3\pm 0.0\%$ | $0.3\pm 0.1\%$
$\hat{\texttt{FNR}}_{\text{F}}$ | $6.2\pm 0.1\%$ | $5.9\pm 0.1\%$ | $6.1\pm 0.2\%$
$\hat{\texttt{FNR}}_{\text{M}}$ | $5.9\pm 0.1\%$ | $5.6\pm 0.1\%$ | $5.8\pm 0.1\%$

Figure 44: Logistic regression on HMDA - 2017 - NY, by sex

Abstention set metrics
---
| Simple | Super
$\Delta\hat{\texttt{AR}}$ | $1.4\pm 0.0\%$ | $0.5\pm 0.0\%$
$\hat{\texttt{AR}}_{\text{F}}$ | $43.1\pm 0.2\%$ | $13.2\pm 0.2\%$
$\hat{\texttt{AR}}_{\text{M}}$ | $41.7\pm 0.2\%$ | $12.7\pm 0.2\%$

Decision tree prediction set metrics
---
| Baseline | Simple | Super
$\Delta\hat{\texttt{PR}}$ | $1.5\pm 0.0\%$ | $0.9\pm 0.0\%$ | $1.9\pm 0.1\%$
$\hat{\texttt{PR}}_{\text{F}}$ | $75.8\pm 0.1\%$ | $94.8\pm 0.1\%$ | $80.5\pm 0.2\%$
$\hat{\texttt{PR}}_{\text{M}}$ | $77.3\pm 0.1\%$ | $95.7\pm 0.1\%$ | $82.4\pm 0.1\%$
$\Delta\hat{\texttt{Err}}$ | $0.4\pm 0.1\%$ | $0.1\pm 0.0\%$ | $0.3\pm 0.0\%$
$\hat{\texttt{Err}}_{\text{F}}$ | $20.5\pm 0.2\%$ | $3.3\pm 0.1\%$ | $11.8\pm 0.2\%$
$\hat{\texttt{Err}}_{\text{M}}$ | $20.1\pm 0.1\%$ | $3.4\pm 0.1\%$ | $11.5\pm 0.2\%$
$\Delta\hat{\texttt{FPR}}$ | $0.2\pm 0.1\%$ | $0.2\pm 0.0\%$ | $0.1\pm 0.0\%$
$\hat{\texttt{FPR}}_{\text{F}}$ | $10.1\pm 0.2\%$ | $2.2\pm 0.1\%$ | $6.1\pm 0.1\%$
$\hat{\texttt{FPR}}_{\text{M}}$ | $9.9\pm 0.1\%$ | $2.4\pm 0.1\%$ | $6.2\pm 0.1\%$
$\Delta\hat{\texttt{FNR}}$ | $0.3\pm 0.0\%$ | $0.2\pm 0.0\%$ | $0.4\pm 0.1\%$
$\hat{\texttt{FNR}}_{\text{F}}$ | $10.4\pm 0.1\%$ | $1.1\pm 0.1\%$ | $5.7\pm 0.2\%$
$\hat{\texttt{FNR}}_{\text{M}}$ | $10.1\pm 0.1\%$ | $0.9\pm 0.1\%$ | $5.3\pm 0.1\%$

Figure 45: Decision trees on HMDA - 2017 - NY, by sex

Abstention set metrics
---
| Simple | Super
$\Delta\hat{\texttt{AR}}$ | $1.2\pm 0.0\%$ | $0.3\pm 0.0\%$
$\hat{\texttt{AR}}_{\text{F}}$ | $31.4\pm 0.1\%$ | $8.4\pm 0.1\%$
$\hat{\texttt{AR}}_{\text{M}}$ | $30.2\pm 0.1\%$ | $8.1\pm 0.1\%$

Random forest prediction set metrics
---
| Baseline | Simple | Super
$\Delta\hat{\texttt{PR}}$ | $1.8\pm 0.0\%$ | $1.7\pm 0.1\%$ | $2.0\pm 0.1\%$
$\hat{\texttt{PR}}_{\text{F}}$ | $76.6\pm 0.1\%$ | $89.4\pm 0.2\%$ | $79.0\pm 0.2\%$
$\hat{\texttt{PR}}_{\text{M}}$ | $78.4\pm 0.1\%$ | $91.1\pm 0.1\%$ | $81.0\pm 0.1\%$
$\Delta\hat{\texttt{Err}}$ | $0.4\pm 0.1\%$ | $0.1\pm 0.1\%$ | $0.4\pm 0.0\%$
$\hat{\texttt{Err}}_{\text{F}}$ | $18.4\pm 0.2\%$ | $6.3\pm 0.2\%$ | $13.0\pm 0.2\%$
$\hat{\texttt{Err}}_{\text{M}}$ | $18.0\pm 0.1\%$ | $6.2\pm 0.1\%$ | $12.6\pm 0.2\%$
$\Delta\hat{\texttt{FPR}}$ | $0.0\pm 0.1\%$ | $0.3\pm 0.0\%$ | $0.1\pm 0.1\%$
$\hat{\texttt{FPR}}_{\text{F}}$ | $9.4\pm 0.2\%$ | $3.8\pm 0.1\%$ | $6.7\pm 0.2\%$
$\hat{\texttt{FPR}}_{\text{M}}$ | $9.4\pm 0.1\%$ | $4.1\pm 0.1\%$ | $6.8\pm 0.1\%$
$\Delta\hat{\texttt{FNR}}$ | $0.4\pm 0.0\%$ | $0.4\pm 0.1\%$ | $0.5\pm 0.1\%$
$\hat{\texttt{FNR}}_{\text{F}}$ | $8.9\pm 0.1\%$ | $2.5\pm 0.1\%$ | $6.3\pm 0.2\%$
$\hat{\texttt{FNR}}_{\text{M}}$ | $8.5\pm 0.1\%$ | $2.1\pm 0.0\%$ | $5.8\pm 0.1\%$

Figure 46: Random forests on HMDA - 2017 - NY, by sex

TX - 2017 \- by ethnicity.

Abstention set metrics
---
| Simple | Super
$\Delta\hat{\texttt{AR}}$ | $0.1\pm 0.0\%$ | $0.0\pm 0.0\%$
$\hat{\texttt{AR}}_{\text{HL}}$ | $1.1\pm 0.0\%$ | $0.4\pm 0.0\%$
$\hat{\texttt{AR}}_{\text{NHL}}$ | $1.0\pm 0.0\%$ | $0.4\pm 0.0\%$

Logistic regression prediction set metrics
---
| Baseline | Simple | Super
$\Delta\hat{\texttt{PR}}$ | $13.3\pm 0.1\%$ | $13.1\pm 0.0\%$ | $13.0\pm 0.0\%$
$\hat{\texttt{PR}}_{\text{HL}}$ | $64.7\pm 0.2\%$ | $65.1\pm 0.1\%$ | $65.0\pm 0.1\%$
$\hat{\texttt{PR}}_{\text{NHL}}$ | $78.0\pm 0.1\%$ | $78.2\pm 0.1\%$ | $78.0\pm 0.1\%$
$\Delta\hat{\texttt{Err}}$ | $3.4\pm 0.1\%$ | $3.3\pm 0.1\%$ | $3.3\pm 0.2\%$
$\hat{\texttt{Err}}_{\text{HL}}$ | $17.5\pm 0.2\%$ | $17.1\pm 0.2\%$ | $17.3\pm 0.2\%$
$\hat{\texttt{Err}}_{\text{NHL}}$ | $14.1\pm 0.1\%$ | $13.8\pm 0.1\%$ | $14.0\pm 0.0\%$
$\Delta\hat{\texttt{FPR}}$ | $0.6\pm 0.0\%$ | $0.6\pm 0.0\%$ | $0.6\pm 0.0\%$
$\hat{\texttt{FPR}}_{\text{HL}}$ | $5.6\pm 0.1\%$ | $5.4\pm 0.0\%$ | $5.6\pm 0.0\%$
$\hat{\texttt{FPR}}_{\text{NHL}}$ | $6.2\pm 0.1\%$ | $6.0\pm 0.0\%$ | $6.2\pm 0.0\%$
$\Delta\hat{\texttt{FNR}}$ | $4.0\pm 0.1\%$ | $3.9\pm 0.1\%$ | $3.9\pm 0.2\%$
$\hat{\texttt{FNR}}_{\text{HL}}$ | $11.9\pm 0.2\%$ | $11.6\pm 0.2\%$ | $11.7\pm 0.2\%$
$\hat{\texttt{FNR}}_{\text{NHL}}$ | $7.9\pm 0.1\%$ | $7.7\pm 0.1\%$ | $7.8\pm 0.0\%$

Figure 47: Logistic regression on HMDA - 2017 - TX, by ethnicity

Abstention set metrics
---
| Simple | Super
$\Delta\hat{\texttt{AR}}$ | $7.3\pm 0.0\%$ | $4.1\pm 0.1\%$
$\hat{\texttt{AR}}_{\text{HL}}$ | $40.3\pm 0.1\%$ | $21.1\pm 0.0\%$
$\hat{\texttt{AR}}_{\text{NHL}}$ | $33.0\pm 0.1\%$ | $17.0\pm 0.1\%$

Decision tree prediction set metrics
---
| Baseline | Simple | Super
$\Delta\hat{\texttt{PR}}$ | $7.4\pm 0.2\%$ | $4.1\pm 0.1\%$ | $8.5\pm 0.0\%$
$\hat{\texttt{PR}}_{\text{HL}}$ | $72.0\pm 0.2\%$ | $90.2\pm 0.1\%$ | $76.7\pm 0.1\%$
$\hat{\texttt{PR}}_{\text{NHL}}$ | $79.4\pm 0.0\%$ | $94.3\pm 0.0\%$ | $85.2\pm 0.1\%$
$\Delta\hat{\texttt{Err}}$ | $3.7\pm 0.0\%$ | $0.7\pm 0.1\%$ | $2.0\pm 0.1\%$
$\hat{\texttt{Err}}_{\text{HL}}$ | $19.2\pm 0.1\%$ | $3.0\pm 0.1\%$ | $8.4\pm 0.1\%$
$\hat{\texttt{Err}}_{\text{NHL}}$ | $15.5\pm 0.1\%$ | $2.3\pm 0.0\%$ | $6.4\pm 0.0\%$
$\Delta\hat{\texttt{FPR}}$ | $2.4\pm 0.1\%$ | $0.1\pm 0.0\%$ | $0.4\pm 0.0\%$
$\hat{\texttt{FPR}}_{\text{HL}}$ | $10.1\pm 0.1\%$ | $1.2\pm 0.0\%$ | $3.3\pm 0.1\%$
$\hat{\texttt{FPR}}_{\text{NHL}}$ | $7.7\pm 0.0\%$ | $1.1\pm 0.0\%$ | $2.9\pm 0.1\%$
$\Delta\hat{\texttt{FNR}}$ | $1.2\pm 0.1\%$ | $0.6\pm 0.0\%$ | $1.5\pm 0.0\%$
$\hat{\texttt{FNR}}_{\text{HL}}$ | $9.1\pm 0.1\%$ | $1.8\pm 0.0\%$ | $5.0\pm 0.0\%$
$\hat{\texttt{FNR}}_{\text{NHL}}$ | $7.9\pm 0.0\%$ | $1.2\pm 0.0\%$ | $3.5\pm 0.0\%$

Figure 48: Decision trees on HMDA - 2017 - TX, by ethnicity

Abstention set metrics
---
| Simple | Super
$\Delta\hat{\texttt{AR}}$ | $6.2\pm 0.1\%$ | $2.5\pm 0.1\%$
$\hat{\texttt{AR}}_{\text{HL}}$ | $31.9\pm 0.0\%$ | $13.2\pm 0.1\%$
$\hat{\texttt{AR}}_{\text{NHL}}$ | $25.7\pm 0.1\%$ | $10.7\pm 0.0\%$

Random forest prediction set metrics
---
| Baseline | Simple | Super
$\Delta\hat{\texttt{PR}}$ | $8.7\pm 0.1\%$ | $6.9\pm 0.1\%$ | $9.6\pm 0.1\%$
$\hat{\texttt{PR}}_{\text{HL}}$ | $70.6\pm 0.2\%$ | $83.0\pm 0.2\%$ | $72.7\pm 0.2\%$
$\hat{\texttt{PR}}_{\text{NHL}}$ | $79.3\pm 0.1\%$ | $89.9\pm 0.1\%$ | $82.3\pm 0.1\%$
$\Delta\hat{\texttt{Err}}$ | $3.3\pm 0.0\%$ | $1.2\pm 0.1\%$ | $2.2\pm 0.1\%$
$\hat{\texttt{Err}}_{\text{HL}}$ | $17.3\pm 0.1\%$ | $5.3\pm 0.1\%$ | $10.5\pm 0.1\%$
$\hat{\texttt{Err}}_{\text{NHL}}$ | $14.0\pm 0.1\%$ | $4.1\pm 0.0\%$ | $8.3\pm 0.0\%$
$\Delta\hat{\texttt{FPR}}$ | $1.7\pm 0.1\%$ | $0.1\pm 0.0\%$ | $0.4\pm 0.1\%$
$\hat{\texttt{FPR}}_{\text{HL}}$ | $8.5\pm 0.1\%$ | $2.0\pm 0.0\%$ | $4.1\pm 0.0\%$
$\hat{\texttt{FPR}}_{\text{NHL}}$ | $6.8\pm 0.0\%$ | $1.9\pm 0.0\%$ | $3.7\pm 0.1\%$
$\Delta\hat{\texttt{FNR}}$ | $1.7\pm 0.1\%$ | $1.1\pm 0.1\%$ | $1.8\pm 0.1\%$
$\hat{\texttt{FNR}}_{\text{HL}}$ | $8.9\pm 0.1\%$ | $3.3\pm 0.1\%$ | $6.4\pm 0.1\%$
$\hat{\texttt{FNR}}_{\text{NHL}}$ | $7.2\pm 0.0\%$ | $2.2\pm 0.0\%$ | $4.6\pm 0.0\%$

Figure 49: Random forests on HMDA - 2017 - TX, by ethnicity

TX - 2017 \- by race.

Abstention set metrics
---
| Simple | Super
$\Delta\hat{\texttt{AR}}$ | $0.0\pm 0.0\%$ | $0.0\pm 0.0\%$
$\hat{\texttt{AR}}_{\text{NW}}$ | $1.0\pm 0.0\%$ | $0.4\pm 0.0\%$
$\hat{\texttt{AR}}_{\text{W}}$ | $1.0\pm 0.0\%$ | $0.4\pm 0.0\%$

Logistic regression prediction set metrics
---
| Baseline | Simple | Super
$\Delta\hat{\texttt{PR}}$ | $2.6\pm 0.1\%$ | $2.7\pm 0.1\%$ | $2.7\pm 0.1\%$
$\hat{\texttt{PR}}_{\text{NW}}$ | $72.6\pm 0.2\%$ | $72.8\pm 0.0\%$ | $72.6\pm 0.0\%$
$\hat{\texttt{PR}}_{\text{W}}$ | $75.2\pm 0.1\%$ | $75.5\pm 0.1\%$ | $75.3\pm 0.1\%$
$\Delta\hat{\texttt{Err}}$ | $0.0\pm 0.1\%$ | $0.2\pm 0.0\%$ | $0.1\pm 0.0\%$
$\hat{\texttt{Err}}_{\text{NW}}$ | $14.9\pm 0.2\%$ | $14.4\pm 0.1\%$ | $14.7\pm 0.1\%$
$\hat{\texttt{Err}}_{\text{W}}$ | $14.9\pm 0.1\%$ | $14.6\pm 0.1\%$ | $14.8\pm 0.1\%$
$\Delta\hat{\texttt{FPR}}$ | $1.1\pm 0.1\%$ | $0.9\pm 0.1\%$ | $1.0\pm 0.1\%$
$\hat{\texttt{FPR}}_{\text{NW}}$ | $7.0\pm 0.2\%$ | $6.6\pm 0.1\%$ | $6.8\pm 0.1\%$
$\hat{\texttt{FPR}}_{\text{W}}$ | $5.9\pm 0.1\%$ | $5.7\pm 0.0\%$ | $5.8\pm 0.0\%$
$\Delta\hat{\texttt{FNR}}$ | $1.2\pm 0.0\%$ | $1.1\pm 0.1\%$ | $1.1\pm 0.0\%$
$\hat{\texttt{FNR}}_{\text{NW}}$ | $7.9\pm 0.1\%$ | $7.8\pm 0.0\%$ | $7.9\pm 0.1\%$
$\hat{\texttt{FNR}}_{\text{W}}$ | $9.1\pm 0.1\%$ | $8.9\pm 0.1\%$ | $9.0\pm 0.1\%$

Figure 50: Logistic regression on HMDA - 2017 - TX, by race

Abstention set metrics
---
| Simple | Super
$\Delta\hat{\texttt{AR}}$ | $0.1\pm 0.1\%$ | $0.0\pm 0.3\%$
$\hat{\texttt{AR}}_{\text{NW}}$ | $34.7\pm 0.2\%$ | $18.0\pm 0.3\%$
$\hat{\texttt{AR}}_{\text{W}}$ | $34.8\pm 0.1\%$ | $18.0\pm 0.0\%$

Decision tree prediction set metrics
---
| Baseline | Simple | Super
$\Delta\hat{\texttt{PR}}$ | $2.5\pm 0.0\%$ | $1.8\pm 0.0\%$ | $4.1\pm 0.2\%$
$\hat{\texttt{PR}}_{\text{NW}}$ | $75.6\pm 0.1\%$ | $91.9\pm 0.1\%$ | $79.8\pm 0.3\%$
$\hat{\texttt{PR}}_{\text{W}}$ | $78.1\pm 0.1\%$ | $93.7\pm 0.1\%$ | $83.9\pm 0.1\%$
$\Delta\hat{\texttt{Err}}$ | $0.0\pm 0.1\%$ | $0.0\pm 0.0\%$ | $0.1\pm 0.0\%$
$\hat{\texttt{Err}}_{\text{NW}}$ | $16.4\pm 0.1\%$ | $2.4\pm 0.0\%$ | $6.8\pm 0.1\%$
$\hat{\texttt{Err}}_{\text{W}}$ | $16.4\pm 0.0\%$ | $2.4\pm 0.0\%$ | $6.9\pm 0.1\%$
$\Delta\hat{\texttt{FPR}}$ | $1.2\pm 0.0\%$ | $0.0\pm 0.0\%$ | $0.1\pm 0.0\%$
$\hat{\texttt{FPR}}_{\text{NW}}$ | $9.2\pm 0.1\%$ | $1.1\pm 0.0\%$ | $2.9\pm 0.1\%$
$\hat{\texttt{FPR}}_{\text{W}}$ | $8.0\pm 0.1\%$ | $1.1\pm 0.0\%$ | $3.0\pm 0.1\%$
$\Delta\hat{\texttt{FNR}}$ | $1.2\pm 0.0\%$ | $0.1\pm 0.0\%$ | $0.1\pm 0.1\%$
$\hat{\texttt{FNR}}_{\text{NW}}$ | $7.2\pm 0.1\%$ | $1.4\pm 0.0\%$ | $3.9\pm 0.1\%$
$\hat{\texttt{FNR}}_{\text{W}}$ | $8.4\pm 0.1\%$ | $1.3\pm 0.0\%$ | $3.8\pm 0.0\%$

Figure 51: Decision trees on HMDA - 2017 - TX, by race

Abstention set metrics
---
| Simple | Super
$\Delta\hat{\texttt{AR}}$ | $0.1\pm 0.1\%$ | $0.0\pm 0.0\%$
$\hat{\texttt{AR}}_{\text{NW}}$ | $27.3\pm 0.1\%$ | $11.3\pm 0.1\%$
$\hat{\texttt{AR}}_{\text{W}}$ | $27.2\pm 0.0\%$ | $11.3\pm 0.1\%$

Random forest prediction set metrics
---
| Baseline | Simple | Super
$\Delta\hat{\texttt{PR}}$ | $3.4\pm 0.1\%$ | $3.3\pm 0.0\%$ | $4.5\pm 0.0\%$
$\hat{\texttt{PR}}_{\text{NW}}$ | $74.4\pm 0.2\%$ | $85.6\pm 0.1\%$ | $76.3\pm 0.1\%$
$\hat{\texttt{PR}}_{\text{W}}$ | $77.8\pm 0.1\%$ | $88.9\pm 0.1\%$ | $80.8\pm 0.1\%$
$\Delta\hat{\texttt{Err}}$ | $0.0\pm 0.0\%$ | $0.1\pm 0.0\%$ | $0.1\pm 0.0\%$
$\hat{\texttt{Err}}_{\text{NW}}$ | $14.8\pm 0.1\%$ | $4.3\pm 0.0\%$ | $8.7\pm 0.1\%$
$\hat{\texttt{Err}}_{\text{W}}$ | $14.8\pm 0.1\%$ | $4.4\pm 0.0\%$ | $8.8\pm 0.1\%$
$\Delta\hat{\texttt{FPR}}$ | $0.7\pm 0.0\%$ | $0.1\pm 0.1\%$ | $0.1\pm 0.1\%$
$\hat{\texttt{FPR}}_{\text{NW}}$ | $7.8\pm 0.1\%$ | $1.8\pm 0.1\%$ | $3.7\pm 0.1\%$
$\hat{\texttt{FPR}}_{\text{W}}$ | $7.1\pm 0.1\%$ | $1.9\pm 0.0\%$ | $3.8\pm 0.0\%$
$\Delta\hat{\texttt{FNR}}$ | $0.8\pm 0.0\%$ | $0.2\pm 0.1\%$ | $0.1\pm 0.0\%$
$\hat{\texttt{FNR}}_{\text{NW}}$ | $6.9\pm 0.1\%$ | $2.6\pm 0.1\%$ | $5.1\pm 0.0\%$
$\hat{\texttt{FNR}}_{\text{W}}$ | $7.7\pm 0.1\%$ | $2.4\pm 0.0\%$ | $5.0\pm 0.0\%$

Figure 52: Random forests on HMDA - 2017 - TX, by race

TX - 2017 \- by sex.

Abstention set metrics
---
| Simple | Super
$\Delta\hat{\texttt{AR}}$ | $0.0\pm 0.0\%$ | $0.0\pm 0.0\%$
$\hat{\texttt{AR}}_{\text{F}}$ | $1.0\pm 0.0\%$ | $0.4\pm 0.0\%$
$\hat{\texttt{AR}}_{\text{M}}$ | $1.0\pm 0.0\%$ | $0.4\pm 0.0\%$

Logistic regression prediction set metrics
---
| Baseline | Simple | Super
$\Delta\hat{\texttt{PR}}$ | $5.7\pm 0.1\%$ | $5.6\pm 0.0\%$ | $5.6\pm 0.0\%$
$\hat{\texttt{PR}}_{\text{F}}$ | $70.8\pm 0.2\%$ | $71.1\pm 0.2\%$ | $70.9\pm 0.2\%$
$\hat{\texttt{PR}}_{\text{M}}$ | $76.5\pm 0.1\%$ | $76.7\pm 0.2\%$ | $76.5\pm 0.2\%$
$\Delta\hat{\texttt{Err}}$ | $1.1\pm 0.2\%$ | $1.0\pm 0.1\%$ | $1.0\pm 0.1\%$
$\hat{\texttt{Err}}_{\text{F}}$ | $15.7\pm 0.2\%$ | $15.3\pm 0.0\%$ | $15.5\pm 0.0\%$
$\hat{\texttt{Err}}_{\text{M}}$ | $14.6\pm 0.0\%$ | $14.3\pm 0.1\%$ | $14.5\pm 0.1\%$
$\Delta\hat{\texttt{FPR}}$ | $0.4\pm 0.0\%$ | $0.4\pm 0.1\%$ | $0.4\pm 0.0\%$
$\hat{\texttt{FPR}}_{\text{F}}$ | $5.8\pm 0.1\%$ | $5.6\pm 0.1\%$ | $5.7\pm 0.0\%$
$\hat{\texttt{FPR}}_{\text{M}}$ | $6.2\pm 0.1\%$ | $6.0\pm 0.0\%$ | $6.1\pm 0.0\%$
$\Delta\hat{\texttt{FNR}}$ | $1.4\pm 0.1\%$ | $1.3\pm 0.1\%$ | $1.3\pm 0.1\%$
$\hat{\texttt{FNR}}_{\text{F}}$ | $9.8\pm 0.2\%$ | $9.6\pm 0.0\%$ | $9.7\pm 0.0\%$
$\hat{\texttt{FNR}}_{\text{M}}$ | $8.4\pm 0.1\%$ | $8.3\pm 0.1\%$ | $8.4\pm 0.1\%$

Figure 53: Logistic regression on HMDA - 2017 - TX, by sex

Abstention set metrics
---
| Simple | Super
$\Delta\hat{\texttt{AR}}$ | $2.0\pm 0.1\%$ | $1.4\pm 0.0\%$
$\hat{\texttt{AR}}_{\text{F}}$ | $36.2\pm 0.2\%$ | $19.0\pm 0.1\%$
$\hat{\texttt{AR}}_{\text{M}}$ | $34.2\pm 0.1\%$ | $17.6\pm 0.1\%$

Decision tree prediction set metrics
---
| Baseline | Simple | Super
$\Delta\hat{\texttt{PR}}$ | $3.5\pm 0.0\%$ | $2.2\pm 0.1\%$ | $4.8\pm 0.0\%$
$\hat{\texttt{PR}}_{\text{F}}$ | $75.2\pm 0.1\%$ | $91.8\pm 0.0\%$ | $79.8\pm 0.1\%$
$\hat{\texttt{PR}}_{\text{M}}$ | $78.7\pm 0.1\%$ | $94.0\pm 0.1\%$ | $84.6\pm 0.1\%$
$\Delta\hat{\texttt{Err}}$ | $1.1\pm 0.1\%$ | $0.2\pm 0.0\%$ | $0.5\pm 0.0\%$
$\hat{\texttt{Err}}_{\text{F}}$ | $17.2\pm 0.1\%$ | $2.6\pm 0.0\%$ | $7.2\pm 0.1\%$
$\hat{\texttt{Err}}_{\text{M}}$ | $16.1\pm 0.0\%$ | $2.4\pm 0.0\%$ | $6.7\pm 0.1\%$
$\Delta\hat{\texttt{FPR}}$ | $0.8\pm 0.1\%$ | $0.2\pm 0.0\%$ | $0.3\pm 0.1\%$
$\hat{\texttt{FPR}}_{\text{F}}$ | $8.8\pm 0.1\%$ | $1.0\pm 0.0\%$ | $2.8\pm 0.0\%$
$\hat{\texttt{FPR}}_{\text{M}}$ | $8.0\pm 0.0\%$ | $1.2\pm 0.0\%$ | $3.1\pm 0.1\%$
$\Delta\hat{\texttt{FNR}}$ | $0.3\pm 0.1\%$ | $0.4\pm 0.0\%$ | $0.9\pm 0.1\%$
$\hat{\texttt{FNR}}_{\text{F}}$ | $8.4\pm 0.1\%$ | $1.6\pm 0.0\%$ | $4.5\pm 0.1\%$
$\hat{\texttt{FNR}}_{\text{M}}$ | $8.1\pm 0.0\%$ | $1.2\pm 0.0\%$ | $3.6\pm 0.0\%$

Figure 54: Decision trees on HMDA - 2017 - TX, by sex

Abstention set metrics
---
| Simple | Super
$\Delta\hat{\texttt{AR}}$ | $1.7\pm 0.0\%$ | $0.8\pm 0.2\%$
$\hat{\texttt{AR}}_{\text{F}}$ | $28.4\pm 0.1\%$ | $11.9\pm 0.2\%$
$\hat{\texttt{AR}}_{\text{M}}$ | $26.7\pm 0.1\%$ | $11.1\pm 0.0\%$

Random forest prediction set metrics
---
| Baseline | Simple | Super
$\Delta\hat{\texttt{PR}}$ | $4.4\pm 0.0\%$ | $4.0\pm 0.0\%$ | $5.2\pm 0.1\%$
$\hat{\texttt{PR}}_{\text{F}}$ | $74.1\pm 0.1\%$ | $85.5\pm 0.1\%$ | $76.3\pm 0.2\%$
$\hat{\texttt{PR}}_{\text{M}}$ | $78.5\pm 0.1\%$ | $89.5\pm 0.1\%$ | $81.5\pm 0.1\%$
$\Delta\hat{\texttt{Err}}$ | $1.0\pm 0.1\%$ | $0.4\pm 0.0\%$ | $0.7\pm 0.1\%$
$\hat{\texttt{Err}}_{\text{F}}$ | $15.5\pm 0.1\%$ | $4.6\pm 0.0\%$ | $9.3\pm 0.0\%$
$\hat{\texttt{Err}}_{\text{M}}$ | $14.5\pm 0.0\%$ | $4.2\pm 0.0\%$ | $8.6\pm 0.1\%$
$\Delta\hat{\texttt{FPR}}$ | $0.3\pm 0.1\%$ | $0.3\pm 0.0\%$ | $0.4\pm 0.1\%$
$\hat{\texttt{FPR}}_{\text{F}}$ | $7.4\pm 0.1\%$ | $1.7\pm 0.0\%$ | $3.5\pm 0.1\%$
$\hat{\texttt{FPR}}_{\text{M}}$ | $7.1\pm 0.0\%$ | $2.0\pm 0.0\%$ | $3.9\pm 0.0\%$
$\Delta\hat{\texttt{FNR}}$ | $0.7\pm 0.0\%$ | $0.8\pm 0.0\%$ | $1.1\pm 0.1\%$
$\hat{\texttt{FNR}}_{\text{F}}$ | $8.1\pm 0.1\%$ | $3.0\pm 0.0\%$ | $5.8\pm 0.1\%$
$\hat{\texttt{FNR}}_{\text{M}}$ | $7.4\pm 0.1\%$ | $2.2\pm 0.0\%$ | $4.7\pm 0.0\%$

Figure 55: Random forests on HMDA - 2017 - TX, by sex

####  E.4.7 Discussion of extended results for Algorithm 1

Overall, our results support that examining self-consistency and error
together provide a much richer picture of model behavior, both with respect to
arbitrariness and fairness metric disparities. Particularly in smaller
datasets, the learning process produces models with a large degree of
variance. As a result, ensembling with confidence can lead to huge abstention
rates.

Improving self-consistency by doing a round of variance reduction first and
then ensembling with confidence (i.e., super-ensembling) can lead to
improvements in error over baselines while having a lower abstention rate.
These improvements are typically shared across subgroups, but may not be
symmetric; some subgroups may benefit more than others. As a result, even
though accuracy increases absolutely for both groups, relative fairness
metrics can decrease. This is a different instantiation of the fairness-
accuracy trade-off than is often written about, which posits a necessary
decrease in accuracy for one subgroup to improve fairness between binarized
groups. Our results suggest that it is worth first tuning for accuracy, and
then seeing how fairness interventions can balance the benefits across
subgroups. Of course, it is possible that doing this could lead to injecting
variance back into the model outputs, thereby reducing self-consistency and
inducing arbitrariness. We leave this investigation to future work.

Additionally, our results reify that choice of model matters a lot. While
overall error rates across model types may be similar, the sources of that
error are not necessarily the same. This is an obvious point, relating to bias
and variance. However, a lot of fair classification work describes similar
performance across logistic regression, decision trees, random forests, SVMs,
and MLPs (e.g., Chen et al. [11]). Looking at self-consistency confirms that
this is not the case, with decision trees and random forests in particular
exhibiting higher variance, and thus being more amenable to variance reduction
and improvements in overall error. As fair classification research transitions
to larger benchmarks, it will likely be fruitful to investigate more complex
model classes.

We provide run times on our cluster environment in Table 2. We did not select
for CPUs with any particular features, and thus the run times are quite
variable.

Table 2: These times are recorded for our cluster environment (hh:mm:ss), described in Appendix E.2 for our Algorithm 1 experiments. At the time of running, due to time constraints, the authors had not yet parallelized this part of the code. Dataset | $g$ | Logistic regression | Decision trees | Random forests
---|---|---|---|---
South German Credit | sex | 00:42:50 | 00:25:28 | 00:34:42
COMPAS | race | 00:57:05 | 00:39:24 | 00:31:47
Old Adult | sex | 01:08:37 | 01:23:39 | 00:57:11
Taiwan Credit | sex | 00:31:35 | 01:34:57 | 01:53:33
New Adult - CA |  |  |  |
Income | $\texttt{sex},\texttt{race}$ | 01:39:53 | 02:51:13 | 04:59:07
Employment | $\texttt{sex},\texttt{race}$ | 02:20:15 | 02:18:16 | 03:00:15
Public Coverage | $\texttt{sex},\texttt{race}$ | 01:13:33 | 02:02:57 | 02:24:08
HMDA - 2017 |  |  |  |
NY | sex, race, ethnicity | 03:50:52 | 05:00:19 | 05:39:44
TX | sex, race, ethnicity | 05:18:59 | 04:10:34 | 04:18:59

We also provide details on systematic arbitrariness in Tables 3 and 4, which
we measure using the $\hat{\mathcal{W}_{1}}$. As noted in Appendix B.3, since
this metric is an average, its measure necessarily changes if we compute it
over a different set of levels $\hat{{\mathbb{K}}}$. To make distances across
interventions comparable, we treat CDF values below $\kappa$ as $0$, so that
all of the probability mass is on $\hat{\texttt{SC}}\geq\kappa$. We therefore
report two versions of these results, those for no abstention and those that
account for abstention at values $<\kappa$. $\Delta$ is the difference between
$\hat{\mathcal{W}_{1}}_{\text{Simple}}-\hat{\mathcal{W}_{1}}_{\text{Super}}$.
Positive differences indicate cases for which the super-ensembling method
decreases the Wasserstein-1 distance between subgroup CDFs; negative
differences indicate increases. While in some cases there is an increase, it
is worth noting that this aligns with cases for which the
$\hat{\mathcal{W}_{1}}$ distance is very close to 0. Old Adult, highlighted
below, is the only dataset that exhibts large amounts of systematic
arbitrariness (for decision tress and random forests, in particular; it
exhibits the highest amount for logistic regression, but is overall low). Old
Adult and New Adult - Employment (by sex) are two of the only tasks that show
any fairness disparities that are $>3\%$.

Table 3: Empirical Wasserstein-1 ($\hat{\mathcal{W}_{1}}$) measurements without abstention Dataset | $g$ | Logistic regression | Decision trees | Random forests
---|---|---|---|---
|  | Simple | Super | $\Delta$ | Simple | Super | $\Delta$ | Simple | Super | $\Delta$
German Credit | sex | 0.0181 | 0.0101 | 0.0079 | 0.0162 | 0.0244 | -0.0082 | 0.0181 | 0.0175 | 0.0006
COMPAS | race | 0.0073 | 0.0043 | 0.0030 | 0.0189 | 0.0170 | 0.0019 | 0.0073 | 0.0031 | 0.0043
Old Adult | sex | 0.0206 | 0.0033 | 0.0173 | 0.1386 | 0.0273 | 0.1112 | 0.1266 | 0.0255 | 0.1011
Taiwan Credit | sex | 0.0028 | 0.0007 | 0.0020 | 0.0223 | 0.0108 | 0.0115 | 0.0240 | 0.0065 | 0.0175
New Adult - CA |  |  |  |  |  |  |  |  |  |
Income | sex | 0.0009 | 0.0003 | 0.0006 | 0.0138 | 0.0055 | 0.0083 | 0.0089 | 0.0018 | 0.0071
| race | 0.0003 | 0.0001 | 0.0002 | 0.0170 | 0.0073 | 0.0096 | 0.0163 | 0.0055 | 0.0108
Employment | sex | 0.0004 | 0.0002 | 0.0003 | 0.0010 | 0.0011 | -0.0001 | 0.0043 | 0.0031 | 0.0013
| race | 0.0004 | 0.0001 | 0.0004 | 0.0020 | 0.0007 | 0.0013 | 0.0021 | 0.0015 | 0.0006
Public Coverage | sex | 0.0004 | 0.0001 | 0.0003 | 0.0029 | 0.0024 | 0.0005 | 0.0045 | 0.0023 | 0.0024
| race | 0.0010 | 0.0003 | 0.0007 | 0.0200 | 0.0089 | 0.0113 | 0.0235 | 0.0089 | 0.0147
HMDA - 2017 |  |  |  |  |  |  |  |  |  |
NY | sex | 0.0002 | 0.0004 | -0.0002 | 0.0096 | 0.0039 | 0.0056 | 0.0080 | 0.0023 | 0.0056
| race | 0.0009 | 0.0005 | 0.0005 | 0.0433 | 0.0203 | 0.0231 | 0.0409 | 0.0133 | 0.0276
| ethnicity | 0.0005 | 0.0005 | 0.0000 | 0.0229 | 0.0156 | 0.0073 | 0.0248 | 0.0108 | 0.0140
TX | sex | 0.0001 | 0.0001 | 0.0000 | 0.0153 | 0.0097 | 0.0055 | 0.0113 | 0.0054 | 0.0058
| race | 0.0001 | 0.0001 | 0.0000 | 0.0010 | 0.0012 | -0.0002 | 0.0013 | 0.0007 | 0.0006
| ethnicity | 0.0007 | 0.0002 | 0.0004 | 0.0509 | 0.0291 | 0.0219 | 0.0379 | 0.0190 | 0.0188

Table 4: Empirical Wasserstein-1 ($\hat{\mathcal{W}_{1}}$) measurements with abstention using $\kappa\geq.75$ Dataset | $g$ | Logistic regression | Decision trees | Random forests
---|---|---|---|---
|  | Simple | Super | $\Delta$ | Simple | Super | $\Delta$ | Simple | Super | $\Delta$
German Credit | sex | 0.0113 | 0.0080 | 0.0034 | 0.0090 | 0.0094 | -0.0004 | 0.0084 | 0.0132 | -0.0048
COMPAS | race | 0.0035 | 0.0019 | 0.0017 | 0.0039 | 0.0060 | -0.0021 | 0.0041 | 0.0019 | 0.0022
Old Adult | sex | 0.0110 | 0.0020 | 0.0090 | 0.0654 | 0.0155 | 0.0500 | 0.0634 | 0.0139 | 0.0494
Taiwan Credit | sex | 0.0014 | 0.0005 | 0.0009 | 0.0057 | 0.0059 | -0.0002 | 0.0107 | 0.0040 | 0.0067
New Adult - CA |  |  |  |  |  |  |  |  |  |
Income | sex | 0.0005 | 0.0002 | 0.0004 | 0.0051 | 0.0032 | 0.0019 | 0.0047 | 0.0012 | 0.0035
| race | 0.0002 | 0.0000 | 0.0002 | 0.0073 | 0.0040 | 0.0033 | 0.0082 | 0.0028 | 0.0053
Employment | sex | 0.0002 | 0.0001 | 0.0001 | 0.0005 | 0.0005 | 0.0001 | 0.0020 | 0.0014 | 0.0006
| race | 0.0002 | 0.0000 | 0.0002 | 0.0012 | 0.0003 | 0.0008 | 0.0008 | 0.0005 | 0.0003
Public Coverage | sex | 0.0002 | 0.0001 | 0.0001 | 0.0006 | 0.0012 | -0.0006 | 0.0011 | 0.0009 | 0.0002
| race | 0.0006 | 0.0001 | 0.0005 | 0.0068 | 0.0049 | 0.0019 | 0.0106 | 0.0047 | 0.0059
HMDA - 2017 |  |  |  |  |  |  |  |  |  |
NY | sex | 0.0001 | 0.0002 | -0.0001 | 0.0033 | 0.0020 | 0.0012 | 0.0040 | 0.0013 | 0.0028
| race | 0.0004 | 0.0002 | 0.0002 | 0.0155 | 0.0111 | 0.0044 | 0.0190 | 0.0076 | 0.0114
| ethnicity | 0.0002 | 0.0002 | 0.0001 | 0.0055 | 0.0083 | -0.0028 | 0.0081 | 0.0059 | 0.0022
TX | sex | 0.0000 | 0.0000 | 0.0000 | 0.0061 | 0.0050 | 0.0011 | 0.0058 | 0.0029 | 0.0028
| race | 0.0000 | 0.0001 | 0.0000 | 0.0004 | 0.0005 | -0.0002 | 0.0007 | 0.0004 | 0.0003
| ethnicity | 0.0003 | 0.0001 | 0.0003 | 0.0229 | 0.0159 | 0.0070 | 0.01200 | 0.0104 | 0.0095

###  E.5 Reliability and fairness metrics in COMPAS and South German Credit

Even before we apply our intervention to improve self-consistency, our results
in Section 3 show close-to-parity $\hat{\texttt{Err}}$, $\hat{\texttt{FPR}}$,
and $\hat{\texttt{FNR}}$ across subgroups in COMPAS (and similarly for South
German Credit, below). These results are surprising. We run $B=101$ models to
produce estimates of variance and self-consistency, but of course doing this
also has the effect of estimating the expected error more generally (with
variance representing a portion of that error). Our estimates of expected
error for these tasks indicate that the average model produced training on
COMPAS and South German Credit, with respect to popular fairness definitions
like Equality of Opportunity and Equalized Odds [4, 33] _are in fact baseline
close to parity, with no fairness intervention applied_. We found this across
model types for both datasets, though the story becomes more complicated when
we apply techniques to improve self-consistency (see discussion at the end of
Appendix E.4).

Of course, we did not expect this result, as these are two of the de facto
standard benchmark datasets in algorithmic fairness. They are used in
countless other studies to probe and verify algorithmic fairness interventions
[25]. As a result, we initially thought that our results must be incorrect. We
therefore looked at the underlying models in our bootstrap runs to see the
error of the underlying models.

We re-ran our baseline experiments with $B=1001$ and for $100$ test/train
splits for logistic regression. In Figure 56, we plot the ($100,100$)
bootstrap models that went into these results. For another view on analogous
information, in Table 5(f), we provide an excerpt of the results for COMPAS
regarding the underlying $1010$ random forest classifiers used to produce
Figure 2.

(a) $\hat{\texttt{Err}}$ disparity

(b) $\hat{\texttt{FPR}}$ disparity

(c) $\hat{\texttt{FNR}}$ disparity

Figure 56: Cumulative distribution of error disparity across $100,100$
logistic regression models trained on COMPAS.

Overall, we can see that there is a wide range of error disparities that trend
in both directions, with a skew toward higher $\hat{\texttt{FPR}}$ for
${\bm{g}}=\text{NW}$. These results support our claim that training many
models is necessary to get an accurate picture of expected error, with
implications both for reproducibility of experiments that just train and
analyze a small handful of models and for generalizability. There are models
that exhibit worse degrees of unfairness in both directions, but they are more
unlikely than models that exhibit smaller disparities.

We subset the above results to the $100$ models that produce the lowest
$\hat{\texttt{Err}}$, as this is often the selection critera for picking
models to post-process. We plot these results below. These top-performing
models in fact exhibit (on average) closer-to-parity for $\hat{\texttt{FPR}}$
and $\hat{\texttt{FNR}}$.

(a) $\hat{\texttt{Err}}$ disparity

(b) $\hat{\texttt{FPR}}$ disparity

(c) $\hat{\texttt{FNR}}$ disparity

Figure 57: CDF of error disparity across the top $100$ logistic regression
models (of the $100,100$ models) trained on COMPAS. Table 5: Comparing
subgroup error rates in COMPAS for different random forest classifiers trained
to produce Figure 2. Each table looks at the top-$3$ highest differences
between subgroups for the specified metric: (a)
$\hat{\texttt{Err}}_{\text{NW}}-\hat{\texttt{Err}}_{\text{W}}$, when
$\hat{\texttt{Err}}_{\text{NW}}>\hat{\texttt{Err}}_{\text{W}}$; (b)
$\hat{\texttt{Err}}_{\text{W}}-\hat{\texttt{Err}}_{\text{NW}}$, when
$\hat{\texttt{Err}}_{\text{W}}>\hat{\texttt{Err}}_{\text{NW}}$; (c)
$\hat{\texttt{FPR}}_{\text{NW}}-\hat{\texttt{FPR}}_{\text{W}}$, when
$\hat{\texttt{FPR}}_{\text{NW}}>\hat{\texttt{FPR}}_{\text{W}}$; (d)
$\hat{\texttt{FPR}}_{\text{W}}-\hat{\texttt{FPR}}_{\text{NW}}$, when
$\hat{\texttt{FPR}}_{\text{W}}>\hat{\texttt{FPR}}_{\text{NW}}$; (e)
$\hat{\texttt{FNR}}_{\text{NW}}-\hat{\texttt{FNR}}_{\text{W}}$, when
$\hat{\texttt{FNR}}_{\text{NW}}>\hat{\texttt{FNR}}_{\text{W}}$; and, (f) (e)
$\hat{\texttt{FNR}}_{\text{W}}-\hat{\texttt{FNR}}_{\text{NW}}$, when
$\hat{\texttt{FNR}}_{\text{W}}>\hat{\texttt{FNR}}_{\text{NW}}$. We highlight
the overall error metric in gray, the larger metric (being subtracted from) in
blue, the smaller metric (being subtracted) in red, and the difference in the
metric between subgroups in purple. Note that run 757 appears twice, which we
mark in orange.

(a) The top-$3$ most unfair models by subgroup-specific $\hat{\texttt{Err}}$, when $\hat{\texttt{Err}}_{\text{NW}}>\hat{\texttt{Err}}_{\text{W}}$ (i.e., unfair toward NW). Run # | $s$ | $b$ | $\hat{\texttt{Err}}$ | $\hat{\texttt{FPR}}$ | $\hat{\texttt{FNR}}$ | $\hat{\texttt{Err}}_{\text{NW}}$ | $\hat{\texttt{FPR}}_{\text{NW}}$ | $\hat{\texttt{FNR}}_{\text{NW}}$ | $\hat{\texttt{Err}}_{\text{W}}$ | $\hat{\texttt{FPR}}_{\text{W}}$ | $\hat{\texttt{FNR}}_{\text{W}}$ | $\hat{\texttt{Err}}_{\text{NW}}-\hat{\texttt{Err}}_{\text{W}}$
---|---|---|---|---|---|---|---|---|---|---|---|---
762 | $8$ | $504$ | $0.374$ | $0.179$ | $0.196$ | $0.405$ | $0.204$ | $0.201$ | $0.315$ | $0.13$ | $0.186$ | $0.09$
757 | $8$ | $464$ | $0.369$ | $0.167$ | $0.202$ | $0.395$ | $0.201$ | $0.193$ | $0.318$ | $0.101$ | $0.218$ | $0.077$
328 | $4$ | $116$ | $0.371$ | $0.165$ | $0.206$ | $0.395$ | $0.181$ | $0.214$ | $0.323$ | $0.134$ | $0.189$ | $0.072$

(b) The top-$3$ most unfair models by subgroup-specific $\hat{\texttt{Err}}$, when $\hat{\texttt{Err}}_{\text{W}}>\hat{\texttt{Err}}_{\text{NW}}$ (i.e., unfair toward W). Run # | $s$ | $b$ | $\hat{\texttt{Err}}$ | $\hat{\texttt{FPR}}$ | $\hat{\texttt{FNR}}$ | $\hat{\texttt{Err}}_{\text{NW}}$ | $\hat{\texttt{FPR}}_{\text{NW}}$ | $\hat{\texttt{FNR}}_{\text{NW}}$ | $\hat{\texttt{Err}}_{\text{W}}$ | $\hat{\texttt{FPR}}_{\text{W}}$ | $\hat{\texttt{FNR}}_{\text{W}}$ | $\hat{\texttt{Err}}_{\text{W}}-\hat{\texttt{Err}}_{\text{NW}}$
---|---|---|---|---|---|---|---|---|---|---|---|---
414 | $5$ | $75$ | $0.376$ | $0.167$ | $0.209$ | $0.352$ | $0.158$ | $0.194$ | $0.422$ | $0.186$ | $0.236$ | $0.07$
435 | $5$ | $180$ | $0.376$ | $0.199$ | $0.177$ | $0.355$ | $0.189$ | $0.166$ | $0.416$ | $0.217$ | $0.198$ | $0.061$
413 | $5$ | $70$ | $0.378$ | $0.189$ | $0.189$ | $0.359$ | $0.188$ | $0.171$ | $0.413$ | $0.191$ | $0.222$ | $0.054$

(c) The top-$3$ most unfair models by subgroup-specific $\hat{\texttt{FPR}}$, when $\hat{\texttt{FPR}}_{\text{NW}}>\hat{\texttt{FPR}}_{\text{W}}$ (i.e., unfair toward NW). Run # | $s$ | $b$ | $\hat{\texttt{Err}}$ | $\hat{\texttt{FPR}}$ | $\hat{\texttt{FNR}}$ | $\hat{\texttt{Err}}_{\text{NW}}$ | $\hat{\texttt{FPR}}_{\text{NW}}$ | $\hat{\texttt{FNR}}_{\text{NW}}$ | $\hat{\texttt{Err}}_{\text{W}}$ | $\hat{\texttt{FPR}}_{\text{W}}$ | $\hat{\texttt{FNR}}_{\text{W}}$ | $\hat{\texttt{FPR}}_{\text{NW}}-\hat{\texttt{FPR}}_{\text{W}}$
---|---|---|---|---|---|---|---|---|---|---|---|---
757 | $8$ | $464$ | $0.369$ | $0.167$ | $0.202$ | $0.395$ | $0.201$ | $0.193$ | $0.318$ | $0.101$ | $0.218$ | $0.1$
729 | $8$ | $240$ | $0.358$ | $0.162$ | $0.197$ | $0.376$ | $0.189$ | $0.187$ | $0.323$ | $0.107$ | $0.216$ | $0.082$
791 | $8$ | $736$ | $0.377$ | $0.171$ | $0.205$ | $0.395$ | $0.198$ | $0.197$ | $0.341$ | $0.118$ | $0.222$ | $0.08$

(d) The top-$3$ most unfair models by subgroup-specific $\hat{\texttt{FPR}}$, when $\hat{\texttt{FPR}}_{\text{W}}>\hat{\texttt{FPR}}_{\text{NW}}$ (i.e., unfair toward W). Run # | $s$ | $b$ | $\hat{\texttt{Err}}$ | $\hat{\texttt{FPR}}$ | $\hat{\texttt{FNR}}$ | $\hat{\texttt{Err}}_{\text{NW}}$ | $\hat{\texttt{FPR}}_{\text{NW}}$ | $\hat{\texttt{FNR}}_{\text{NW}}$ | $\hat{\texttt{Err}}_{\text{W}}$ | $\hat{\texttt{FPR}}_{\text{W}}$ | $\hat{\texttt{FNR}}_{\text{W}}$ | $\hat{\texttt{FPR}}_{\text{W}}-\hat{\texttt{FPR}}_{\text{NW}}$
---|---|---|---|---|---|---|---|---|---|---|---|---
639 | $7$ | $280$ | $0.36$ | $0.187$ | $0.173$ | $0.352$ | $0.174$ | $0.178$ | $0.376$ | $0.212$ | $0.164$ | $0.038$
807 | $9$ | $72$ | $0.381$ | $0.191$ | $0.19$ | $0.372$ | $0.179$ | $0.192$ | $0.398$ | $0.214$ | $0.184$ | $0.035$
543 | $6$ | $264$ | $0.358$ | $0.155$ | $0.203$ | $0.351$ | $0.144$ | $0.206$ | $0.37$ | $0.175$ | $0.196$ | $0.031$

(e) The top-$3$ most unfair models by subgroup-specific $\hat{\texttt{FNR}}$, when $\hat{\texttt{FNR}}_{\text{NW}}>\hat{\texttt{FNR}}_{\text{W}}$ (i.e., unfair toward NW). Run # | $s$ | $b$ | $\hat{\texttt{Err}}$ | $\hat{\texttt{FPR}}$ | $\hat{\texttt{FNR}}$ | $\hat{\texttt{Err}}_{\text{NW}}$ | $\hat{\texttt{FPR}}_{\text{NW}}$ | $\hat{\texttt{FNR}}_{\text{NW}}$ | $\hat{\texttt{Err}}_{\text{W}}$ | $\hat{\texttt{FPR}}_{\text{W}}$ | $\hat{\texttt{FNR}}_{\text{W}}$ | $\hat{\texttt{FNR}}_{\text{NW}}-\hat{\texttt{FNR}}_{\text{W}}$
---|---|---|---|---|---|---|---|---|---|---|---|---
246 | $3$ | $141$ | $0.379$ | $0.166$ | $0.213$ | $0.398$ | $0.169$ | $0.229$ | $0.345$ | $0.161$ | $0.184$ | $0.045$
506 | $6$ | $42$ | $0.367$ | $0.17$ | $0.197$ | $0.386$ | $0.175$ | $0.211$ | $0.332$ | $0.161$ | $0.171$ | $0.04$
204 | $3$ | $15$ | $0.384$ | $0.185$ | $0.199$ | $0.394$ | $0.181$ | $0.213$ | $0.365$ | $0.192$ | $0.173$ | $0.04$

(f) The top-$3$ most unfair models by subgroup-specific $\hat{\texttt{FNR}}$, when $\hat{\texttt{FNR}}_{\text{W}}>\hat{\texttt{FNR}}_{\text{NW}}$ (i.e., unfair toward W). Run # | $s$ | $b$ | $\hat{\texttt{Err}}$ | $\hat{\texttt{FPR}}$ | $\hat{\texttt{FNR}}$ | $\hat{\texttt{Err}}_{\text{NW}}$ | $\hat{\texttt{FPR}}_{\text{NW}}$ | $\hat{\texttt{FNR}}_{\text{NW}}$ | $\hat{\texttt{Err}}_{\text{W}}$ | $\hat{\texttt{FPR}}_{\text{W}}$ | $\hat{\texttt{FNR}}_{\text{W}}$ | $\hat{\texttt{FNR}}_{\text{W}}-\hat{\texttt{FNR}}_{\text{NW}}$
---|---|---|---|---|---|---|---|---|---|---|---|---
474 | $5$ | $375$ | $0.373$ | $0.175$ | $0.199$ | $0.356$ | $0.183$ | $0.174$ | $0.406$ | $0.159$ | $0.247$ | $0.073$
401 | $5$ | $10$ | $0.378$ | $0.189$ | $0.19$ | $0.363$ | $0.197$ | $0.167$ | $0.406$ | $0.173$ | $0.233$ | $0.066$
52 | $1$ | $53$ | $0.367$ | $0.172$ | $0.196$ | $0.351$ | $0.178$ | $0.173$ | $0.397$ | $0.16$ | $0.238$ | $0.065$

This detailed view provides insight into how such a result is possible.
Broadly speaking, individual runs have roughly similar error;212121This should
be taken relatively. In general, COMPAS demonstrates high error; the error is
relatively tight given just how much error there is. The error fluctuates
depending on the training data, but the average error rate across train/test
splits is rather tight, despite the fluctuations in error within the $B$ runs
of each split. yet, the subgroup-specific error rates that compose the overall
error can nevertheless vary widely depending on the underlying training data.
This observation aligns with current interest in _model multiplicity_ in the
algorithmic fairness community [7, 56], which imports the idea from Breiman
[10]. In this case, as suggested by Table 5(f), there are models that
demonstrate unfairness toward both subgroups with respect to each error rate
metric $\hat{\texttt{Err}}$, $\hat{\texttt{FPR}}$, and $\hat{\texttt{FNR}}$.
When we move away from attempting to find a _single_ model that performs well
(accurately or fairly) on COMPAS, and instead consider the information
contained across different possible models, we yield the result that the
average, expected behavior smooths over the variance in underlying models such
that the result is close to fair: _The average of unfair models with high
variance in subgroup error rates is essentially fair._

Stability analysis. To verify the stability of this result, we re-execute our
experiments for increasing numbers of train/test splits $S$ and replicates
$B$. While our results for COMPAS are generally tight for small $S$ (e.g.,
Figures 2 and 5), this was not the case for German Credit, for which it was
difficult to estimate self-consistency consistently. As a result, for COMPAS,
we did not expect markedly different results for increased $S$. Our results
for $S=100,B=1001$ using logistic regression (Figure 58, Table 6) confirm this
intuition.

Figure 58: COMPAS split by ${\bm{g}}=\texttt{race}$, $B=1001,S=100$

Table 6: Mean $\pm$ STD across $S=100$ train/test splits $\times\;B=1001$
runs. COMPAS
---
| $\hat{\texttt{Err}}$ | $\hat{\texttt{FPR}}$ | $\hat{\texttt{FNR}}$ | $\hat{\texttt{SC}}$
Total | $.333\pm.008$ | $.14\pm.009$ | $.192\pm.01$ | $.883\pm.004$
$g=\text{NW}$ | $.333\pm.01$ | $.148\pm.011$ | $.185\pm.012$ | $.88\pm.005$
$g=\text{W}$ | $.332\pm.014$ | $.125\pm.013$ | $.207\pm.016$ | $.888\pm.006$

We provide analogous results for German Credit, with $S=1000,B=1001$ using
random forests (Figure 59, Table 7). It takes an enormous number of runs to
produce stable estimates of error and $\hat{\texttt{SC}}$ for German Credit,
which indicate statistical equality across groups. Arguably, our results below
for $1,001,000$ models still are very high variance (certainly with respect to
error metrics). This task really has too few data points ($\approx 600$) to
generalize reliably.

Figure 59: German Credit split by ${\bm{g}}=\texttt{sex}$, $S=1000,B=100$

Table 7: Mean $\pm$ STD across $S=1000$ train/test splits $\times\;B=1001$
runs. South German Credit
---
| $\hat{\texttt{Err}}$ | $\hat{\texttt{FPR}}$ | $\hat{\texttt{FNR}}$ | $\hat{\texttt{SC}}$
Total | $.28\pm.021$ | $.173\pm.028$ | $.107\pm.017$ | $.769\pm.015$
$g=\text{F}$ | $.288\pm.064$ | $.183\pm.072$ | $.105\pm.037$ | $.766\pm.04$
$g=\text{M}$ | $.279\pm.023$ | $.171\pm.029$ | $.108\pm.018$ | $.769\pm.016$

##  Appendix F Brief notes on future research

There are many interesting directions for future work that are out of scope
for the present project. We address some topics below.

Novel theory. We do not include extensive novel theory in this project.
Nevertheless, our project raises interesting questions for theory in future
work. Notably, we could compose our methodology with post-processing [33] for
cases in which there is observed empirical unfairness. We could then
investigate picking group-specific thresholds that take variance into account.
We could reconfigure the formulations in [33] and related work, with respect
to the fairness-accuracy trade-off, as actually representing multiple such
trade-off curves (that are a function of different models under
consideration). There may be interesting directions for mathematical analysis
in this direction.

We could also extend traditional results on bagging and variance reduction for
classifiers. While bagging has guarantees for variance reduction for
regression, it does not have the same guarantees for classification [8, 9]. It
generally is observed to work well in practice for variance reduction if the
underlying classifiers are high variance — which is indeed the regime we are
in for this paper. However, there are interesting theory questions regarding
abstention that we could investigate with theoretical tools, which could let
us come up with other ways of reasoning about bagging and variance reduction.

Both of these directions are out of scope for the present paper. They are
interesting, but do not have to do with our main experimental aims and
contributions, and thus do not make it into a conference-length submission. We
are not interested in novel theory in the present study. If anything, our work
highlights how over-attention to theory can (directly or indirectly) bring
about serious problems of mismeasurement in practice. That is a main takeaway
for our work, which by nature does not involve novel theory.

Arbitrariness beyond algorithmic fairness. Our framework for reasoning about
self-consistency and arbitrariness does not inherently have to do with
algorithmic fairness. We could apply it to other domains. For example, it
would be interesting to ask similar questions in deep learning and generative
AI. We think that such work would be interesting, but is again out of scope
for the present study. The first author of this project is in fact working on
such questions as separate work. However, this project’s research aims are
inherently focused on fairness; the project was designed in response to
observations in experimental practices in the fairness community, fairness
definitions, and fairness theory.

Experiments on synthetic data. Our results indicate that unfairness (as
defined with respect to model error rates) is not frequently observed on
common benchmark tasks in fair classification. Of course, there could be other
datasets in fairness domains that are not currently used as benchmarks that
more clearly demonstrate unfairness in practice. Hypothetically, there could
be datasets for which we use Algorithm 1 to reduce arbitrariness, and yet we
still see significant systematic arbitrariness or differences in error rates
(and thus unfairness) due to noise or bias. We just did not really see this
for almost all of the tasks we investigate in this paper, which happen to be
the ones that the fairness community uses for experiments.

To study Algorithm 1 in light of these other possibilities, we could develop
synthetic datasets that retain unfairness after dealing with arbitrariness. We
did not do this in the present study for two reasons. First, our focus was the
practice of fairness research, as it currently stands, with a data-centric
approach on the datasets people actually use for their research. We are not
interested in synthetic data for this project.

However, future theory results that extend our work could be vetted
experimentally with synthetic data. The work we mention above regarding
composition with post-processing, as well as revisting impossibility results
from a distributional approach over possible models, may be very interesting
to examine under data settings that we can control.

How to deal with abstention. Future work could also perform a deeper
exploration of the trade-off between abstention rate and error. We could
characterize a Pareto-optimal trade-off that is a function of the choice of
self-consistency level $\kappa$, and also examine in experiments and
analytically how absention leads to improvements in accuracy. Future work
could also identify patterns in abstention sets beyond low self-consistency.
In this, looking to metrics from model multiplicity may be helpful. Further,
future work could combine human decision-making or other automated elements to
see how we can root out arbitrariness.

Reproducibility. As mentioned in our Ethics Statement, we made attempts to
reproduce prior work in fair classification, and often could not. We
ultimately made reproducibility of specific papers out of scope for the
present project, as we could make our contributions about arbitrariness and
variance without such work. It would nevertheless be useful to focus future
work on reproducing prior algorithmic fairness studies, and seeing if
conclusions change in those works as a function of using Algorithm 1 prior to
introducing the proposed fairness intervention.

Law and policy. As mentioned in our Ethics Statement, our work regarding
arbitrariness raises concrete questions for the law around due process and
automated decision-making. Such contributions are also out of the scope of the
present work, but we are currently developing them for future submission to a
law review journal.