# (Un)fairness in Post-operative Complication Prediction Models

Sandhya Tripathi  ,  Bradley A. Fritz  ,  Mohamed Abdelhack  ,  Michael S.
Avidan  ,  Yixin Chen  and  Christopher R. King

###### Abstract.

With the current ongoing debate about fairness, explainability and
transparency of machine learning models, their application in high-impact
clinical decision-making systems must be scrutinized. We consider a real-life
example of risk estimation before surgery and investigate the potential for
bias or unfairness of a variety of algorithms. Our approach creates
transparent documentation of potential bias so that the users can apply the
model carefully. We augment a model-card like analysis using propensity scores
with a decision-tree based guide for clinicians that would identify
predictable shortcomings of the model. In addition to functioning as a guide
for users, we propose that it can guide the algorithm development and
informatics team to focus on data sources and structures that can address
these shortcomings.

Clinical prediction models, fairness, propensity score matching, transparent
decision making

††ccs: Computing methodologies Machine learning††ccs: Social and professional
topics Medical information policy††ccs: Applied computing Health care
information systems

##  1\. Introduction

Biases and irrationality in human decision making based on irrelevant or
socially unacceptable factors has been observed in essentially every context
(Levinson, 2006) with substantial variation based on personal background
(Hamberg, 2008). Although the reasons and mechanisms are different, decision-
making algorithms have shown similar biases. Potential explanations include
bias encoded in the choice of training data, choice of data collection, the
impact of miss-classified outcomes and missing input data, the effect of group
size, and the chosen objectives (Gianfrancesco et al., 2018). An instructive
example on the power of these biases came in 2016, (Angwin et al., 2016) when
a detailed analysis of COMPAS, a recidivism prediction algorithm that has been
used in United States to aid pre-trial holding decisions, found that it was
disadvantaging black defendants. Based on these observations, fairness and
transparency is a critical evaluation to make when deploying machine learning
for clinical decision making (Vollmer et al., 2020). For example, if the
system made unacceptable predictions selectively for racial minorities, its
application could increase health disparities and even worsen overall outcomes
compared to no system at all (Rajkomar et al., 2018; Goodman et al., 2018;
Ferryman and Pitcan, 2018; Obermeyer et al., 2019; Nordling, 2019; Vyas et
al., 2020). Suppose there is no true difference in outcome between the
majority and minority groups. Consider a system that accurately identifies
high or low risk in the majority group, but never flags members of the
minority as high risk. Implementation of such a system would shift healthcare
resources from the minority group to the majority group, potentially
increasing disparities. Empirical studies have demonstrated these issues in
actual application: gender imbalance leading to biased classifiers (Larrazabal
et al., 2020), in general medicine and mental health care (Chen et al., 2019),
in renal failure prediction models (Williams and Razavian, 2019), classifying
chest x-rays (Seyyed-Kalantari et al., 2020), and in clinical contextual word
embeddings (Zhang et al., 2020).

One such system for clinical decision assistance without explicit decision
support was described by (Fritz et al., 2019; Cui et al., 2019), which
developed machine-learning classifiers for postoperative mortality and other
complications based on preoperative risk factors and real-time intra-operative
data. The explicit goal of this program was directing attention from
telemedicine providers (King et al., 2019) to the “neediest” patients, with
similar systems used to stratify patients into risk-appropriate surgery during
preoperative clinic visits and to inform them of potential adverse events
(Bilimoria et al., 2013). A major obstacle to using this system is the need to
understand if it makes “fair predictions,” or if it has “bias” with respect to
protected or sensitive characteristics. For clarity of prose, a sensitive or
protected characteristic refers to the membership information of societal
groups like race or sex, that are protected by anti-discrimination law
(Barocas and Selbst, 2016; Act, 1964).

There are three major conceptual variations of fairness which have been
proposed for use with predictive algorithms: disparate treatment, disparate
impact and disparate mistreatment.(Zafar et al., 2019) An algorithm suffers
from:

  * •

Disparate treatment (Barocas and Selbst, 2016) if it provides different
outputs for groups of people with the same (or similar) values of non-
sensitive features but different values of sensitive features. This is also
referred as direct discrimination (Pedreshi et al., 2008).

  * •

Disparate impact (Barocas and Selbst, 2016) if it provides outputs that
benefit (or hurt) a group of people sharing a value of a sensitive feature
more frequently than other groups of people marginal on the values other
features take. Also referred as statistical parity (Corbett-Davies et al.,
2017) or demographic parity (Dwork et al., 2012).

  * •

Disparate mistreatment (Zafar et al., 2017) if the algorithm’s performance
differs for otherwise similar people at different levels of a sensitive
feature. Also referred as equality of opportunity (Hardt et al., 2016) and
predictive equality (Corbett-Davies et al., 2017). This approach may offer
equal utility in some contexts (Rajkomar et al., 2018).

We will consider covariates as a potential explanations, and do not consider
disparate impact as it requires proportionality in decision outcomes for
sensitive groups that could have arisen from justifiable reasons (e.g. if
women have lower rates of smoking). For the remainder of the paper, we will
refer to “bias” as disparate treatment and “unfairness” as disparate
mistreatment / differential benefit. We argue that unfairness is most relevant
at the preoperative clinic stage of our application, where the most important
use is to guide patients to appropriate preoperative procedures and inform
them of the risks of surgery. However, in the telemedicine stage of the
application, bias also becomes relevant. Even if a group were truly at
systematically lower risk, we may not want that group to receive a strongly
reduced amount of clinician attention, especially as clinicians may react in a
non-linear way to the risk estimates.

### Addressing and Preventing Bias

A review and comparison of existing fairness interventions can be found in
(Friedler et al., 2019; Parikh et al., 2019). However, reducing or removing
bias can come at the cost of decreased predictive performance. For example,
flexible classifiers ought optimize performance within and across groups;
imposing equal accuracy may simply force the algorithm to degrade its
predictions in the “easier” category (Chen et al., 2018). Examples of
enforcing minimal bias while optimizing the model performance have
nevertheless been attempted where context-specific justifications apply,
(Zafar et al., 2019; Cho et al., 2020; Ghassami et al., 2018; Adel et al.,
2019) including several clinical examples (Pfohl et al., 2019a, 2020; Pfohl et
al., 2019b). Without contextualizing to domain specific needs, blind
application of off the shelf approaches can worsen differential performance
(Corbett-Davies and Goel, 2018; Herington, 2020; Fazelpour and Lipton, 2020)
and create other problems (Veinot et al., 2018).

In contexts where eliminating bias is impossible, an alternative approach is
to transparently document the problem and educate users so that they can apply
the model with care. (Mitchell et al., 2019) Our approach is closely related
to (Mitchell et al., 2019); after initial empirical demonstration of
unfairness on ML models in a real example in Section 3, in Section 4 we extend
the tabular presentation of accuracy in different groups with a decision-tree
based guide which identifies for clinicians where the model has predictable
shortcomings. With this knowledge, clinicians can act accordingly on the
predicted risk for a particular patient, the algorithm development team can
focus innovation on data sources and structures likely to address present
shortcomings, and informatics and clinical partners can improve data
collection efforts among underserved populations.

##  2\. Dataset and prediction task

Postoperative complication prediction models rely on the information collected
during preoperative (pre-anesthesia clinic) assessments contained in the
electronic health record. Access to the data was approved by the Human
Research Protection Office at Washington University in St Louis, USA with a
waiver of informed consent (IRB number $201607122$). Data definitions and
inclusion criteria were selected to match prior reports. (Fritz et al., 2019;
Cui et al., 2019) Data for patients age at least 18 years who underwent
surgery with tracheal intubation at Barnes-Jewish Hospital (St Louis, MO, USA)
between June 2012 and August 2016 are included in the current study. The
preoperative assessments contain lists of past medical history items,
demographics, anthropometrics, laboratory values, vital signs, integrative
wellness assessments (ASA-PS values), and some lifestyle factors (smoking,
heavy drinking). For each laboratory test, the most recent value before
surgery (but no more than 30 days old) is utilised. For this example, we use
only preoperative features, mimicking a tool that could be used in
preoperative clinic or in day-of-surgery planning. A more complete description
of the predictors is including in (Fritz et al., 2019; Cui et al., 2019; King
et al., 2020). Continuous features were normalized to $[0,1]$ as a pre-
processing step. Values that fell outside the plausible ranges for continuous
features (provided by domain experts) were interpreted as input errors and set
to missing and were later imputed using the mean value over the dataset. For
categorical features, all values that did not exist in the data dictionary of
possible values were labelled as missing and combined into a new category.

The two prediction tasks we focus on are post operative 30 day mortality and
Acute Kidney Injury (AKI). Acute kidney injury is defined according to KDIGO
guidelines for changes in serum creatinine (KDIGO, 2012); an increase in the
serum creatinine value by $\geq 0.3$ mg/dL or $\geq$50% within 48 hours is
considered positive. Patients with preoperative renal failure were excluded
from AKI prediction models. Sensitive features for our example were taken to
be self-reported sex and race. Because of the very small number of individuals
reporting sex as other than female or male, only these two categories are
shown. Similarly, the dataset contains a relatively small number of
individuals reporting race other than black or white; we present results only
for the black/white contrast. The outcome rates and marginal distribution of
sensitive features are given in Table 1.

As can be seen in Table 1, the outcome rate is highly imbalanced for both the
targets, which could decrease the performance of the prediction models. To
reduce the effect of imbalance, we used an approach of cost sensitive learning
with domain expert given costs for class specific mis-classification and down
sampling (Elkan, 2001). We used negative and positive class mis-classification
cost pair value of $(1,25)$ and $(1,14)$ for 30 day mortality and AKI
respectively.

We checked for the existence of bias or unfairness across a range of
classifiers, namely, Decision Trees (DT), Random Forests (RF), Gradient Boost
(GB), Logistic Regression (LR) and a Deep Neural Network (DNN). All these
classifiers were implemented using the sklearn module available in Python.
Most of the arguments of these classifiers were implemented with default
values except for the following parameters: {DT: max_depth =20}, {RF:
n_estimators=200}, {GB: n_estimators=200, max_depth=1}, {DNN:
hidden_layer_sizes=(256,256), solver=’sgd’, alpha=8e-5, batch_size=64,
learning_rate_init=0.01, max_iter=10}. In main text, we only report the
difference between the performance measures for RF and LR classifiers. The
actual performance measure values for all classifiers is provided in the
figures in Supplementary Material (SM).

Performance measures examined for unfairness are False Negative Rate (FNR),
False Positive Rate (FPR), False Discovery Rate (FDR) and False Omission Rate
(FOR). We present bootstrap estimates and standard errors of performance
metrics using 150 bootstrap samples using the out-of-bag error as a smoothed
version of k-fold cross validation. (Efron, 1983) Because of the reduced
effective sample size within each bootstrap replicate, the CV error will tend
to overestimate that which would be obtained with training on the entire
dataset, but this is at least consistent within each experiment. We present
the results in mean and confidence interval format rather than a hypothesis
testing one. In the case of discrimination, the magnitude of differences is
critically important in addition to their statistical certainty; we would be
interested in large differences even if there was remaining uncertainty, and
we would be willing to de-prioritize very small differences even if highly
confident that they were non-zero.

Let $\mathcal{X}\subseteq\mathbb{R}^{n}$ be the feature space and
$\mathcal{Y}=\\{-1,1\\}$ be the label set. Let $\mathcal{D}$ be the joint
distribution over $\mathbf{X},\mathbf{Z},Y$ with
$\mathbf{X},\mathbf{Z}\in\mathcal{X}$ , $Y\in\mathcal{Y}$ and $\mathbf{Z}$ be
the set of sensitive features. $\mathbf{Z}$ generally consists of variables
taking ordinal or nominal values. In practice, however, we are only provided
with an i.i.d. sample of size $m$ from distribution $\mathcal{D}$, viz.,
$D=\\{(\mathbf{x}_{i},\mathbf{z}_{i},y_{i})\\}_{i=1}^{m}$.

| Outcome | Total | Female | Male | Other Sex |  | Black
---
| White
---
Other Race
|  $\\#$ of samples
---
(full dataset)
Both | 52499 | 26206 | 25897 | 396 | 10207 | 39838 | 2454
Outcome rate | 30 day mortality | 0.035 | 0.027 | 0.042 | 0.038 | 0.027 | 0.034 | 0.085
AKI | 0.063 | 0.048 | 0.078 | 0.040 | 0.065 | 0.061 | 0.082
Table 1. First row reports the marginal distribution for subgroups of the
sensitive variables. Last 2 rows report the outcome rates for subgroups of
sensitive variable.

##  3\. Empirical examination of bias

We first document whether our classifiers exhibit bias. As demonstrated in
Table 1, there are notable differences between men and women in marginal rates
of death and AKI, and smaller differences between white and black
participants. Participants with ”Other” or missing race had substantially
higher rates of death and AKI, but this is difficult to interpret because
“missing” can encode “declined to answer”, “no applicable category”, or
informative missingness during e.g. an emergency surgery in which the patient
is unable to answer.

An important point is whether the difference in outcomes is explained away by
differences in other variables or stems from direct use of the sensitive
feature. For a given patient $i$ from test set whose data is denoted by
$(\mathbf{x}_{i},\mathbf{z}_{i},y_{i})$ and for a trained predictor
$f(\cdot)$, we compare the predictions made for patient $i$ for the true value
of the sensitive feature and for the complementary value of the sensitive
feature, i.e., $f(\mathbf{x}_{i},\mathbf{z}_{i},y_{i})$ and
$f(\mathbf{x}_{i},1-\mathbf{z}_{i},y_{i})$. We also compute the predicted
values using classifiers trained not including the sensitive characteristic.
Finally, we evaluate the classifiers on a propensity score matched sample.
That is, in order to compare only individuals with similar covariates, we
calculate $e_{i}=P(\mathbf{Z}_{i}=1|\mathbf{X}_{i})$ and 1-to-1 match
individuals with $z=1$ to individuals with similar $e$ and $z=0$. Because the
propensity score, is a balancing score, after conditioning on it, the
distribution of measured baseline covariates (features other than sensitive
features) is similar between “treated” (sensitive feature $\mathbf{Z}_{i}=1$)
and “untreated” (sensitive feature $\mathbf{Z}_{i}=0$) subjects (patients)
(Austin, 2011).

We estimate the propensity score by using a logistic regression model where
the sensitive feature is regressed on observed baseline features
($\mathbf{X}$), although more flexible classifiers can also be used. While
there are many methods of using the propensity score to accomplish balance, we
chose propensity score matching (PSM) as several studies demonstrate that it
eliminates a greater proportion of the systematic differences in baseline
characteristics between the subgroups of sensitive feature (Austin, 2011), is
simple to implement, and can be applied to all classifiers easily. With a
caliper value of $0.05$ in PSM, our test set did not have any match for 2.8%
and 69% of the disadvantaged (Z=1) samples when conditioned on race and sex
respectively, reflecting major differences in distributions of procedures
performed and risk factors. As expected, matching across sex was more
difficult as there are a fair number of sex-specific surgeries (gynecologic
surgeries, prostate surgery) or nearly sex-specific surgeries (breast
surgery); patients undergoing these procedures essentially do not contribute
to the estimates in the propensity matched sample. Using a relatively
transparent classifier to generate propensity scores also helps to identify
any features which are essentially surrogates for the sensitive characteristic
(such as gynecologic surgery). In our example, we found that ideal body weight
also acted as a surrogate (because it is a deterministic function of sex and
height) and was excluded from propensity score calculation; height also
contributed strongly to group separation, but was retained in the model
because of its potential importance in other calculations.

In Figure 1 we examine the influence of the sensitive characteristic on
average outputs. We see that the differences with respect to sex and race
observed in Table 1 are maintained. These differences persist or are even
magnified when excluding sex as a feature or swapping the sex inputs. Sex
differences are greatly reduced in the propensity matched sample. These
suggest that observed characteristics (covariables) correlated with sex
largely explain the differences in average predicted values. The smaller
differences in predicted outcomes across race are similarly unchanged by
scrambling or omitting race, but are eliminated (or reversed in sign) in the
propensity matched group, suggesting that observable covariables explain some
of the differences in outputs, but that unobserved factors directly explain
some as well. Similar patterns were observed across classifiers.

Figure 1.  Classifier predicted outcome (30 day mortality and AKI) rates by
sensitive characteristic (Sex and Race). Inclusion of sensitive characteristic
in learning set (W), average outputs with sensitive characteristic inverted
(SWAP), sensitive characteristic omitted from learning set (WO), and
restricted to propensity score matched sample (PSM). Each panel depicts the
combination of an outcome and sensitive characteristic $(Z)$ with y-axis
values being the difference in predicted outcome rates between the subgroups
$Z=1$ and $Z=0$.

We next turn to unfairness in the form of differential predictive accuracy.
For each subgroup of a sensitive feature, we separately computed the test set
performance of the models trained on sample $D$ with criterion $\in$ {FNR,
FPR, FDR, FOR}. These measures are shown in Figure 2. No meaningful
differences are observed in AKI performance across race. However, all
classifiers have markedly higher FNR and lower FPR among women. While these
measures are natural inputs to a utility calculation, they are affected by the
baseline rate of the outcome. FDR shows no meaningful differences across sex,
but the FOR remains lower among women. For mortality prediction, RF and GB
show modestly higher FPR among black participants, and all classifiers show
higher FDR among black participants. No differences are observed across race
in FNR or FOR. Women have equivalent FNR and slightly better FOR across all
classifiers, and substantially lower FPR with essentially equivalent FDR.

Figure 2. Each panel in the figure corresponds to a performance measure $\in$
FNR, FOR, FPR, FDR for the classifiers $\in$ RF, LR, DT, GB, DNN marked by
color across subgroups (1 = solid bar or 0 = hash marked bar) of sensitive
characteristic Race (R) and Sex (S) for outcome 30 day mortality (Mort) and
AKI. For sensitive characteristic Sex, 1 denotes females and 0 denotes males;
for Race, 1 denotes Black and 0 denotes White. A high value of FNR or FOR (FPR
or FDR) implies discrimination against (benefits for) the population belonging
to that group of protected attribute.

###  3.1. Algorithmic sources of bias

To explain these differences, we first consider purely algorithmic effects
related to group size and information sharing. Although the total sample size
is reasonably large, the number of adverse events is modest, and we could
easily be in a regime where smaller groups (such as black race) have not
“saturated” the more flexible classifier’s ability to learn from the provided
features, leading to worse performance in minority groups. In a closely
related effect, classifiers may learn patterns in the larger majority group
which do not discriminate as well in the minority groups (interaction of other
features with the sensitive feature). We explore this possibility in three
ways. First, we downsample the larger group to an equal number of observations
as the smaller group. Second, we fit classifiers in stratified samples,
preventing any carry-over of pattern recognition. Third, we compare
classifiers with different degrees of freedom. We would expect carry-over
effects to be the worst in low-degree of freedom classifiers and group size
effects to be the worst in high degree of freedom classifiers.

In Figure 3 we show the effect of downsampling by race. Because sex is nearly
balanced, we do not show downsampling results. To improve the readability of
the figures, we show results only for two examplars: RF (because it was the
best overall performing classifier, is highly flexible, and is commonly used)
and LR (because it is the simplest model that is commonly used in practice,
strongly averages effects of covariables across levels of Z, and for $n\ll m$
is relatively resistant to over-fitting ). We observed that GB stumps and DNN
(shown in SM Figures 11 and 12) had similar results to RF, with GB showing
somewhat exaggerated differences and DNN somewhat smaller. We found that DT
was overall a high-variance classifier; this probably relates to the
complexity limitations related to our hyperparameter selections.

Interestingly, balancing the group sizes did not alleviate differences, and in
several cases made them worse. One might expect the downsampling to
selectively reduce performance in the more flexible classifiers prone to
overfitting, but logistic regression has the highest variability in
performance.

Figure 3. Figure comparing the difference in performance measure values $\in$
FNR, FPR across subgroups of sensitive characteristic Race (R) when the models
were trained on down-sampled majority subgroup data for outcomes 30 day
mortality (Mort) and AKI. Full and SS denotes training on full and down-
sampled data. Down-sampling did reduce the FNR for people belonging to black
race but increased it for white race which is not a good solution. Actual
values of FNR, FPR along with FOR and FDR for all classifiers can be found in
SM Figures 11 and 12.

Next, we tested the performance of combined learner on the subgroups from the
test datasets and compared it to the performance of the individual learners
for the respective subgroups. The intention behind this experiment is to see
if differential performance results from applying patterns across the
subgroup; if so the disparity will improve if the learner is trained only on
the subgroup dataset. We present the results of this experiment in Figure 4.
Disparities are somewhat improved by training separate classifiers. There is
no consistent difference between the model with interactions (RF) and the
model without (LR).

Figure 4. Each panel in the figure depicts the difference in performance
measure $\in$ FNR, FPR for the classifiers $\in$ RF, LR marked by color across
sensitive characteristic Race (R) or Sex (S) subgroups for the outcomes 30 day
mortality (Mort) and AKI. Markers $\Delta$ and $\circ$ denote whether the
classifiers were trained separately (S) on subgroups of a particular value of
sensitive characteristic or both subgroup data was combined together (T) for
training. Actual values of FNR, FPR along with FOR and FDR for all classifiers
can be found in SM Figure 13 and 14.

###  3.2. Group Differences and Performance

Next, we examine the role of correlates of the sensitive feature in explaining
performance differences. Similar to the case of bias, we examine if the
difference in performance is removed by scrambling or omitting the sensitive
feature (which would suggest that it relates to the value of the sensitive
feature tagging unobserved characteristics) and by evaluating in a propensity-
matched set (which would suggest it is the result of differences in observed
characteristics).

To further examine for potential differences due to features correlated with
the sensitive characteristic, we stratify on several major features. We also
address a commonly made argument for explaining the differences in performance
across subgroups: ”the differential performance is due to the inclusion of the
sensitive features while training”. Figure 5 displays the results with and
without the sensitive feature included in training. The differential
performance is not affected. Figure 6 displays when the value of sensitive
characteristic was inverted. In particular, for both the outcomes and sex as
the sensitive characteristic there is no change in differential performance.
However, the differences exist both for FNR and FPR. This suggests that there
is more to the differences than “direct discrimination”.

Figure 5. Comparison of differences in performance measures $\in${FNR, FPR}
for the cases when the sensitive variable was used (W) or not (WO) while
training across Race (R) and Sex (S) for the outcomes 30 day mortality (Mort)
and AKI. Similar pattern was observed for the performance measures FDR and
FOR. As can be seen, FNR differences decreased but the FPR differences
increased even though the magnitude is very small. Overall, removing the
sensitive characteristic doesn’t decrease the bias. Actual values of FNR, FPR
along with FOR and FDR for all classifiers can be found in SM Figure 15 and
16.

Figure 6. Comparison of difference in performance measures $\in$ {FNR, FPR} of
classifiers with true value of sensitive attribute (Orig) and when the value
of sensitive attribute is inverted/swapped (Swap) for Race (R) and Sex (S) for
the outcomes 30 day mortality (Mort) and AKI.

To detect differences in performance related to covariables, Figure 7 and 8
illustrates the results stratifying the accuracy measures on ASA-PS, a global
subjective assessment of health performed by the anesthesiologist. (ASo, 2014)
Our initial experiments suggested that ASA physical status is one of the most
important features for predicting the outcome 30 day mortality and AKI. Domain
experts also agreed that based on the existing literature this feature was
likely to be very influential and absorb much of the information of specific
comorbidities. A patient can be assigned to any of the six categories: ASA I
(healthy), ASA II (mild systematic disease), ASA III (severe systematic
disease), ASA IV (severe systematic disease that has constant threat to life),
ASA V (surgery is necessary for survival), ASA VI (brain dead organ donor)
(ASo, 2014). The ”E” modifier refers to the situation where the delay in
surgery would increase the threat to patient’s life. To simplify presentation,
we merged several categories using input from clinical domain experts. ASA1_NE
(ASA1_E) has patients that were classified as healthy and non-emergency
(emergency). ASA2_NE (ASA2_E) combines the patients that were classified as
ASA II or ASA III, i.e., have systematic disease and non-emergency
(emergency). ASA3_NE (ASA3_E) combines the patients that were classified as
ASA IV or ASA V, i.e., have a constant threat to life and non-emergency
(emergency).

As the true outcome rate for ASA category 1 (i.e., healthy patients) is 0 and
0.001 for 30 day mortality and AKI respectively, the comparison of FNR and FPR
across two groups of sensitive characteristics is not conclusive. For the
other two cases, the results are presented in Figure 7 and 8. For patients
belonging to ASA2_NE and ASA2_E category and when the outcome to be predicted
is 30 day Mortality, the FNR value is consistently higher for black race
people across all algorithms. FPR plot further supports this where the FPR
value is higher for white race people when the patients were operated as an
emergency case. For patients from ASA3_NE and ASA3_E category, there were
enough positive outcome examples for the classifier to learn and hence, the
FNR value is generally low. However, one can still observe some differences
between the FNR value for two subgroups of sensitive characteristic race.

Figure 7. Comparison of differences in performance measures $\in$ {FNR, FPR}
on the group ASA2_NE and ASA2_E (patients having systematic disease) across
the values taken by sensitive characteristics Race (R) and Sex (S) for the
outcomes 30 day mortality (Mort) and AKI. E and NE denotes group
categorization in ASA emergency case or not. Actual FNR, FPR values for all
classifiers can be found in SM Figure 17 and 18.

Figure 8. Comparison of differences in performance measures $\in$ {FNR, FPR}
on the group ASA3_NE and ASA3_E (patients having constant threat to life)
across the values taken by sensitive characteristics Race (R) and Sex (S) for
the outcomes 30 day mortality (Mort) and AKI. E and NE denotes group
categorization in ASA emergency case or not. Actual FNR, FPR values for all
classifiers can be found in SM Figure 19 and 20.

The results of various classifiers on the Propensity matched samples across
the subgroups of sensitive attributes are presented in Figure 9. Differential
performance was affected a great deal (especially across sex), but not in a
consistent way; occasionally the sign is even reversed. Collectively, these
suggest that differences in covariables explain only part of the disparity in
predictive accuracy.

Figure 9. Comparison of differences in performance measures $\in$ {FNR, FPR}
on complete test set (Full) and propensity matched test set (PSM) across the
subgroups of sensitive characteristics Race (R) and Sex (S) for the outcomes
30 day mortality (Mort) and AKI. Actual FNR, FPR along with FOR and FDR values
for all classifiers can be found in SM Figure 21 and 22.

##  4\. Passive approaches for bias reduction

So far we have stratified model performance on pre-specified sensitive
characteristics. However, this may obscure other features which “stand in” for
a sensitive characteristic in an unexpected way. Additionally, interactions
with other features may reflect important deficiencies in the model. Ideally,
an adaptive approach would answer the question “which patients benefit from
the use of the model?” in a manner transparent to the end user, similar to the
“model card” approach. (Mitchell et al., 2019) This would allow the decision
maker to decide whether to trust the model (and intervene accordingly) or to
take potential model deficiencies into account in the case under scrutiny. In
this section, we propose an approach that helps us understand how useful the
full model prediction have been in comparison to a very basic model. Here, the
“full model” is the one that uses all the pre-operative data and the “basic
model” is one that uses only type of surgery, age, race, and sex. The basic
model (rather than a null one) is employed to acknowledge that clinicians
often have a learned heuristic about the importance of these common, easily
observed factors. The utility for every patient is defined in a manner
analogous to the “equalized odds” criterion (Rajkomar et al., 2018). Unlike
the binary cross entropy loss, which is defined as
$-(y\log(\hat{(}p))+(1-y)\log(1-\hat{p}))$ where $y\in\\{0,1\\}$ is true label
and $\hat{p}\in(0,1)$ is the predicted in-class probability, we focus on the
improvement of the absolute probability of a correct prediction. We define the
weighted utility to be

(1) $IU=w_{1}y\hat{p}+w_{2}(1-y)(1-\hat{p}).$

The weights $(w_{1},w_{2})$ correspond to the relative importance of correct
positive class and correct negative class predictions, respectively. This is a
re-weighted version of the mean absolute error, which is an easily interpreted
quantity by users, in contrast to cross-entropy. Let $IU_{Mfull}$ and
$IU_{Mbasic}$ denote the performance of full model and basic model on a
particular patient. As we are interested in the improvement in utility
obtained by the use of full model, we define
$IU_{diff}=IU_{Mfull}-IU_{Mbasic}$. To identify which patients experience
increased or decreased utility, we consider a model that regresses $IU_{diff}$
on available features. Because it has a high level of transparency and forms
natural groups, we train a decision tree on $IU_{diff}$ in the test dataset.

To be consistent with our initial cost value for negative and positive class
mis-classification, we have taken $(w_{1},w_{2})$ to be $(25,1)$ and $(14,1)$
for 30 day mortality and AKI respectively. An example of such a decision tree
is provided in Figure 10, which uses LR as the “full model” and “base model”
classifier. Because LR tends to produce nearly calibrated outputs, this avoids
the need to include a calibration step mapping outputs to probabilities. These
trees demonstrate the transparency and interpetability of the scheme for the
users as they can easily read this for a patient and decide how much to trust
the risk predictions. For example, patients between 50 and 72 benefit more
than other age groups in mortality prediction, with several surgical
specialties benefiting less than others. In particular, a subset of thoracic
surgery patients experience decreased utility compared to the minimal model.
Interesting, although Figure 2 would suggest that sex should be an important
variable, when comparing to a base model including sex, it is not selected as
a node. In contrast, the AKI utility decision tree includes race as a (large)
predictor of utility and sex as a relatively smaller one; Figure 2 would have
suggested the opposite.

ST_0.0 | Gynecology | ST_5.0 |  | Orthopedic/
---
Orthopedic spine
ST_10.0 | Cardiovascular | Sex_1.0 | Male
ST_1.0 |  | Acute Critical care surgery, colorectal,
---
hepatobiliary, minimally invasive surgery,
pediatric, trauma, general surgery
ST_6.0 |  | Neurosurgery/
---
Neurosurgery spine
ST_11.0 |  | Anesthesiology, minor procedures, gastroenterology,
---
oral/maxillofacial, pain management, pulmonary,
dental, oncology, transplant, hepatology
Sex_2.0 | Female
ST_2.0 | Otolaryngology | ST_7.0 | Plastic | ST_12.0 | Opthalomology | Race_7.0 | Black
| ST_3.0
---
ST_14.0
Cardiothoracic | ST_8.0 | Transplant | ST_13.0 | Radiation oncology | Race_9.0 | White
ST_4.0 | Vascular | ST_9.0 | Urology |  |  |  |
Table 2. Data dictionary (for sex, race and surgery type) to be used while
reading the decision trees in Figure 10. ‘Surgery_Type_*’ has been shortened
to ‘ST_*’ in the table where $*$ denotes the number.

Figure 10. Left (Right) panel has 30 day Mortality (AKI) outcome with utility
computed for an LR classifier. ’value’ at each node denotes the improvement in
utility obtained by using a classifier trained on all features in comparison
to a classifier trained on only surgery type, age, sex and race. Minimal cost
complexity pruning with the complexity parameter $\alpha$ taking the $5th$
highest value from the $\alpha$ path. Minimum number of sample to split and
minimum number of samples for a leaf node were set to be 50. The mapping
between surgery type number, sex and race is provided in Table 2.

##  5\. Discussion

Detection of algorithmic bias has become a major concern in the application of
predictive analytics to clinical data. Importantly, we focus on the direct
results of the algorithm and not biased interpretation or selective use of the
outputs resulting from implicit bias within the user (van Ryn et al., 2011).
This concern is especially worrisome in the case of relatively opaque
classifiers such as DNN and GBDT and when covariables may be surrogates for
the sensitive characteristics. This setting can be difficult to distinguish
from the case that real differences in covariables explain important
differences in predictions.

We step through a recommended procedure of comparing classifier output and
performance across subgroups, testing for direct use of known sensitive
characteristics, pattern-transfer effects, and sample-size effects. We also
check for surrogates of the sensitive characteristic by use of propensity
scores, and evaluate performance differences among patients with otherwise
similar features. We sanity-check these results by using domain experts to
identify key covariables and stratify within those covariables. Finally, we
illustrate the use of individual level predicted utility as target for
grouping algorithms to identify patients who gain much or little from the use
of the algorithm. Although some classifiers (such as LR and RF) have in-sample
estimates of uncertainty, by using test sample predictions we can apply the
method generically.

In our example application, we find large differences in performance in AKI
prediction across sexes, and smaller differences across race. None of our
checks fully eliminate the sex differences, and our final utility-grouping
approach identifies race and age as major drivers of algorithmic utility. What
residual sources of unfairness could explain these differences? First,
unmeasured variation in other variables is always a potential source of
variance. For example, there may be more variation in the surgeries performed
on men than women above that is captured by the “surgery type” variable in the
dataset. Our dataset also contains very little social and economic
information; these sources of differences could be more variable among men.
This is consistent with the shared pattern across mortality and AKI in the
propensity matched set. It is also well appreciated that historical
epidemiology in the US struggled to adequately include participants other than
white men, and so risk factors sought out in clinic may be those most relevant
to this group. (Konkel, 2015; Murthy et al., 2004) We hope that by identifying
the differential value of the available information, targeted improvement to
data collection, algorithm development, and user education can be made to
avoid worsening disparities with these new tools.

## References

  * (1)
  * Act (1964) Civil Rights Act. 1964\.  Civil Rights Act of 1964, Title VII, Equal Employment Opportunities.  (1964).
  * Adel et al. (2019) Tameem Adel, Isabel Valera, Zoubin Ghahramani, and Adrian Weller. 2019.  One-network adversarial fairness. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 33. 2412–2420.
  * Angwin et al. (2016) Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. 2016.  Machine bias.  _ProPublica, May 23_ (2016).
  * ASo (2014) Anesthesiologists ASo. 2014\.  ASA physical status classification system.  _ASA House of Delegates_ (2014).
  * Austin (2011) Peter C Austin. 2011\.  An introduction to propensity score methods for reducing the effects of confounding in observational studies.  _Multivariate behavioral research_ 46, 3 (2011), 399–424.
  * Barocas and Selbst (2016) Solon Barocas and Andrew D Selbst. 2016.  Big data’s disparate impact.  _Calif. L. Rev._ 104 (2016), 671.
  * Bilimoria et al. (2013) Karl Y Bilimoria, Yaoming Liu, Jennifer L Paruch, Lynn Zhou, Thomas E Kmiecik, Clifford Y Ko, and Mark E Cohen. 2013.  Development and Evaluation of the Universal ACS NSQIP Surgical Risk Calculator: A Decision Aide and Informed Consent Tool for Patients and Surgeons.  _Journal of the American College of Surgeons_ 217, 5 (Nov. 2013), 833–842.e3.  https://doi.org/10.1016/j.jamcollsurg.2013.07.385
  * Chen et al. (2018) Irene Chen, Fredrik D Johansson, and David Sontag. 2018\.  Why is my classifier discriminatory?. In _Advances in Neural Information Processing Systems_. 3539–3550.
  * Chen et al. (2019) Irene Y Chen, Peter Szolovits, and Marzyeh Ghassemi. 2019\.  Can AI help reduce disparities in general medical and mental health care?  _AMA journal of ethics_ 21, 2 (2019), 167–179.
  * Cho et al. (2020) Jaewoong Cho, Gyeongjo Hwang, and Changho Suh. 2020\.  A Fair Classifier Using Mutual Information. In _Proceeding of IEEE International Symposium on Information Theory_.
  * Corbett-Davies and Goel (2018) Sam Corbett-Davies and Sharad Goel. 2018.  The measure and mismeasure of fairness: A critical review of fair machine learning.  _arXiv preprint arXiv:1808.00023_ (2018).
  * Corbett-Davies et al. (2017) Sam Corbett-Davies, Emma Pierson, Avi Feller, Sharad Goel, and Aziz Huq. 2017.  Algorithmic decision making and the cost of fairness. In _Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining_. 797–806.
  * Cui et al. (2019) Zhicheng Cui, Bradley A. Fritz, Christopher R. King, Michael S. Avidan, and Yixin Chen. 2019\.  A Factored Generalized Additive Model for Clinical Decision Support in the Operating Room.  _AMIA … Annual Symposium proceedings. AMIA Symposium_ 2019 (2019), 343–352.
  * Dwork et al. (2012) Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. 2012.  Fairness through awareness. In _Proceedings of the 3rd innovations in theoretical computer science conference_. 214–226.
  * Efron (1983) Bradley Efron. 1983\.  Estimating the error rate of a prediction rule: improvement on cross-validation.  _Journal of the American statistical association_ 78, 382 (1983), 316–331.
  * Elkan (2001) Charles Elkan. 2001\.  The foundations of cost-sensitive learning. In _International joint conference on artificial intelligence_ , Vol. 17. Lawrence Erlbaum Associates Ltd, 973–978.
  * Fazelpour and Lipton (2020) Sina Fazelpour and Zachary C Lipton. 2020.  Algorithmic Fairness from a Non-ideal Perspective. In _Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society_. 57–63.
  * Ferryman and Pitcan (2018) Kadija Ferryman and Mikaela Pitcan. 2018.  Fairness in precision medicine.  _Data & Society_ (2018).
  * Friedler et al. (2019) Sorelle A Friedler, Carlos Scheidegger, Suresh Venkatasubramanian, Sonam Choudhary, Evan P Hamilton, and Derek Roth. 2019.  A comparative study of fairness-enhancing interventions in machine learning. In _Proceedings of the conference on fairness, accountability, and transparency_. 329–338.
  * Fritz et al. (2019) Bradley A. Fritz, Zhicheng Cui, Muhan Zhang, Yujie He, Yixin Chen, Alex Kronzer, Arbi Ben Abdallah, Christopher R. King, and Michael S. Avidan. 2019.  Deep-learning model for predicting 30-day postoperative mortality.  _British Journal of Anaesthesia_ 123, 5 (2019), 688–695.  https://doi.org/10/ggchqr
  * Ghassami et al. (2018) AmirEmad Ghassami, Sajad Khodadadian, and Negar Kiyavash. 2018\.  Fairness in supervised learning: An information theoretic approach. In _2018 IEEE International Symposium on Information Theory (ISIT)_. IEEE, 176–180.
  * Gianfrancesco et al. (2018) Milena A Gianfrancesco, Suzanne Tamang, Jinoos Yazdany, and Gabriela Schmajuk. 2018.  Potential biases in machine learning algorithms using electronic health record data.  _JAMA internal medicine_ 178, 11 (2018), 1544–1547.
  * Goodman et al. (2018) Steven N Goodman, Sharad Goel, and Mark R Cullen. 2018\.  Machine learning, health disparities, and causal reasoning.  _Annals of internal medicine_ 169, 12 (2018), 883–884.
  * Hamberg (2008) Katarina Hamberg. 2008\.  Gender bias in medicine.  _Women’s Health (London, England)_ 4, 3 (May 2008), 237–243.  https://doi.org/10.2217/17455057.4.3.237
  * Hardt et al. (2016) Moritz Hardt, Eric Price, and Nati Srebro. 2016.  Equality of opportunity in supervised learning. In _Advances in neural information processing systems_. 3315–3323.
  * Herington (2020) Jonathan Herington. 2020\.  Measuring Fairness in an Unfair World. In _Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society_. 286–292.
  * KDIGO (2012) KDIGO. 2012.  Kidney Disease: Improving Global Outcomes (KDIGO) Clinical Practice Guideline for Acute Kidney Injury.  https://kdigo.org/guidelines/acute-kidney-injury/
  * King et al. (2019) Christopher R. King, Joanna Abraham, Thomas G. Kannampallil, Bradley A. Fritz, Arbi Ben Abdallah, Yixin Chen, Bernadette Henrichs, Mary Politi, Brian A. Torres, Angela Mickle, Thaddeus P. Budelier, Sherry McKinnon, Stephen Gregory, Sachin Kheterpal, Troy Wildes, Michael S. Avidan, and TECTONICS Research Group. 2019.  Protocol for the Effectiveness of an Anesthesiology Control Tower System in Improving Perioperative Quality Metrics and Clinical Outcomes: the TECTONICS randomized, pragmatic trial.  _F1000Research_ 8 (Nov. 2019), 2032.  https://doi.org/10/ggdzcg
  * King et al. (2020) Christopher R. King, Bradley A. Fritz, Krisztina Escallier, Yo-El S. Ju, Nan Lin, Sherry McKinnon, Michael S. Avidan, and Ben Julian Palanca. 2020.  Association Between Preoperative Obstructive Sleep Apnea and Preoperative Positive Airway Pressure With Postoperative Intensive Care Unit Delirium.  _JAMA network open_ 3, 4 (April 2020), e203125.  https://doi.org/10/gg5fxq
  * Konkel (2015) Lindsey Konkel. 2015\.  Racial and Ethnic Disparities in Research Studies: The Challenge of Creating More Diverse Cohorts.  _Environmental Health Perspectives_ 123, 12 (Dec. 2015), A297–A302.  https://doi.org/10.1289/ehp.123-A297
  * Larrazabal et al. (2020) Agostina J Larrazabal, Nicolás Nieto, Victoria Peterson, Diego H Milone, and Enzo Ferrante. 2020\.  Gender imbalance in medical imaging datasets produces biased classifiers for computer-aided diagnosis.  _Proceedings of the National Academy of Sciences_ (2020).
  * Levinson (2006) Justin D. Levinson. 2006\.  _Forgotten Racial Equality: Implicit Bias, Decision-Making and Misremembering_.  SSRN Scholarly Paper ID 927547. Social Science Research Network, Rochester, NY.  https://papers.ssrn.com/abstract=927547
  * Mitchell et al. (2019) Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019.  Model cards for model reporting. In _Proceedings of the conference on fairness, accountability, and transparency_. 220–229.
  * Murthy et al. (2004) Vivek H. Murthy, Harlan M. Krumholz, and Cary P. Gross. 2004\.  Participation in cancer clinical trials: race-, sex-, and age-based disparities.  _JAMA_ 291, 22 (June 2004), 2720–2726.  https://doi.org/10.1001/jama.291.22.2720
  * Nordling (2019) Linda Nordling. 2019\.  A fairer way forward for AI in health care.  _Natur_ 573, 7775 (2019), S103–S105.
  * Obermeyer et al. (2019) Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. 2019.  Dissecting racial bias in an algorithm used to manage the health of populations.  _Science_ 366, 6464 (2019), 447–453.
  * Parikh et al. (2019) Ravi B Parikh, Stephanie Teeple, and Amol S Navathe. 2019\.  Addressing Bias in Artificial Intelligence in Health Care.  _Jama_ 322, 24 (2019), 2377–2378.
  * Pedreshi et al. (2008) Dino Pedreshi, Salvatore Ruggieri, and Franco Turini. 2008\.  Discrimination-aware data mining. In _Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining_. 560–568.
  * Pfohl et al. (2019b) Stephen Pfohl, Ben Marafino, Adrien Coulet, Fatima Rodriguez, Latha Palaniappan, and Nigam H Shah. 2019b.  Creating fair models of atherosclerotic cardiovascular disease risk. In _Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society_. 271–278.
  * Pfohl et al. (2019a) Stephen R Pfohl, Tony Duan, Daisy Yi Ding, and Nigam H Shah. 2019a.  Counterfactual Reasoning for Fair Clinical Risk Prediction. In _Machine Learning for Healthcare Conference_. 325–358.
  * Pfohl et al. (2020) Stephen R Pfohl, Agata Foryciarz, and Nigam H Shah. 2020\.  An Empirical Characterization of Fair Machine Learning For Clinical Risk Prediction.  _arXiv preprint arXiv:2007.10306_ (2020).
  * Rajkomar et al. (2018) Alvin Rajkomar, Michaela Hardt, Michael D. Howell, Greg Corrado, and Marshall H. Chin. 2018.  Ensuring Fairness in Machine Learning to Advance Health Equity.  _Annals of internal medicine_ 169, 12 (Dec. 2018), 866–872.  https://doi.org/10.7326/M18-1990
  * Seyyed-Kalantari et al. (2020) Laleh Seyyed-Kalantari, Guanxiong Liu, Matthew McDermott, and Marzyeh Ghassemi. 2020.  CheXclusion: Fairness gaps in deep chest X-ray classifiers.  _arXiv preprint arXiv:2003.00827_ (2020).
  * van Ryn et al. (2011) Michelle van Ryn, Diana J. Burgess, John F. Dovidio, Sean M. Phelan, Somnath Saha, Jennifer Malat, Joan M. Griffin, Steven S. Fu, and Sylvia Perry. 2011.  THE IMPACT OF RACISM ON CLINICIAN COGNITION, BEHAVIOR, AND CLINICAL DECISION MAKING.  _Du Bois review : social science research on race_ 8, 1 (April 2011), 199–218.  https://doi.org/10.1017/S1742058X11000191
  * Veinot et al. (2018) Tiffany C Veinot, Hannah Mitchell, and Jessica S Ancker. 2018\.  Good intentions are not enough: how informatics interventions can worsen inequality.  _Journal of the American Medical Informatics Association_ 25, 8 (2018), 1080–1088.
  * Vollmer et al. (2020) Sebastian Vollmer, Bilal A Mateen, Gergo Bohner, Franz J Király, Rayid Ghani, Pall Jonsson, Sarah Cumbers, Adrian Jonas, Katherine SL McAllister, Puja Myles, et al. 2020\.  Machine learning and artificial intelligence research for patient benefit: 20 critical questions on transparency, replicability, ethics, and effectiveness.  _bmj_ 368 (2020).
  * Vyas et al. (2020) DA Vyas, LG Eisenstein, and DS Jones. 2020.  Hidden in Plain Sight-Reconsidering the Use of Race Correction in Clinical Algorithms.  _The New England Journal of Medicine_ 383, 9 (2020), 874–882.
  * Williams and Razavian (2019) Josie Williams and Narges Razavian. 2019.  Towards Quantification of Bias in Machine Learning for Healthcare: A Case Study of Renal Failure Prediction.  _arXiv preprint arXiv:1911.07679_ (2019).
  * Zafar et al. (2017) Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, and Krishna P Gummadi. 2017.  Fairness beyond disparate treatment & disparate impact: Learning classification without disparate mistreatment. In _Proceedings of the 26th international conference on world wide web_. 1171–1180.
  * Zafar et al. (2019) Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez-Rodriguez, and Krishna P Gummadi. 2019.  Fairness Constraints: A Flexible Approach for Fair Classification.  _Journal of Machine Learning Research_ 20, 75 (2019), 1–42.
  * Zhang et al. (2020) Haoran Zhang, Amy X Lu, Mohamed Abdalla, Matthew McDermott, and Marzyeh Ghassemi. 2020.  Hurtful words: quantifying biases in clinical contextual word embeddings. In _Proceedings of the ACM Conference on Health, Inference, and Learning_. 110–120.

Supplementary material for
(Un)fairness in Post-operative Complication Prediction Models

Figure 11. Figure comparing the performance measure $\in$ {FNR, FPR} values
across subgroups of sensitive characteristic race (R) when the models were
trained on down-sampled majority subgroup data for outcomes 30 day mortality
(Mort) and AKI. Full and SS denotes training on full and down-sampled data.

Figure 12. Figure comparing the performance measure $\in$ {FOR, FDR} values
across subgroups of sensitive characteristic race (R) when the models were
trained on down-sampled majority subgroup data for outcomes 30 day mortality
(Mort) and AKI. Full and SS denotes training on full and down-sampled data.

Figure 13. Comparison of performance measure $\in$ FNR, FPR values for the
different classifiers marked by color across sensitive characteristic race (R)
or sex (S) subgroups. Some classifiers were trained separately (D) on
subgroups of a particular value of sensitive characteristic and others on
combined subgroup dat (T).

Figure 14. Comparison of performance measure $\in$ FOR, FDR values for the
different classifiers marked by color across sensitive characteristic race (R)
or sex (S) subgroups. Some classifiers were trained separately (D) on
subgroups of a particular value of sensitive characteristic and others on
combined subgroup data (T).

Figure 15. Comparison of performance measures $\in${FNR, FPR} for the cases
when the sensitive variable was used or not (WO) while training across race
(R) and sex (S). As can be seen, FNR differences decreased but the FPR
differences increased even though the magnitude is very small. Overall,
removing the sensitive characteristic doesn’t decrease the bias.

Figure 16. Comparison of performance measures $\in${FOR, FDR} for the cases
when the sensitive variable was used or not (WO) while training across race
(R) and sex (S). As can be seen, there is almost no change in the differences
implying that removing the sensitive characteristic doesn’t decrease the bias.

Figure 17. Comparison of performance measure $\in$ {FNR, FPR} values on the
group ASA2_NE and ASA2_E (patients having systematic disease) across the
values taken by sensitive characteristics race (R) and sex (S). E and NE
denotes the group categorized in ASA emergency case or not.

Figure 18. Comparison of performance measure $\in$ {FOR, FDR} values on the
group ASA2_NE and ASA2_E (patients having systematic disease) across the
values taken by sensitive characteristics race (R) and sex (S). E and NE
denotes the group categorized in ASA emergency case or not.

Figure 19. Comparison of performance measures $\in$ {FNR, FPR} values on the
group ASA3_NE and ASA3_E (patients having constant threat to life) across the
values taken by sensitive characteristics race (R) and sex (S). E and NE
denotes the group categorized in ASA emergency case or not.

Figure 20. Comparison of performance measures $\in$ {FOR, FDR} values on the
group ASA3_NE and ASA3_E (patients having constant threat to life) across the
values taken by sensitive characteristics race (R) and sex (S). E and NE
denotes the group categorized in ASA emergency case or not.

Figure 21. Comparison of performance measures values $\in$ {FNR, FPR} on
complete test set and propensity matched test set (P) across the subgroups of
sensitive characteristics Race (R) and Sex (S).

Figure 22. Comparison of performance measures values $\in$ {FOR, FDR} on
complete test set and propensity matched test set (P) across the subgroups of
sensitive characteristics Race (R) and Sex (S).


Gravitational Uncertainty and Black Hole Remnants
Kourosh Nozari***<EMAIL_ADDRESS>and S. Hamid
<EMAIL_ADDRESS>
Department of Physics, Faculty of Basic Sciences, University of Mazandaran,
P. O. Box 47416-1467, Babolsar, IRAN
October 2005

###### Abstract

Possible existence of black holes remnants provides a suitable candidates for
dark matter. In this paper we study the possibility of existence for such
remnants. We consider quantum gravitational induced corrections of black
hole’s entropy and temperature to investigate the possibility of such relics.
Observational scheme for detection of these remnants and their cosmological
constraints are discussed.
PACS: 04.60.-m , 04.70.-s, 04.70.Dy, 95.35.+d
Key Words: Quantum Gravity, Generalized Uncertainty Principle, Varying Speed
of Light Models, Black Holes Thermodynamics, Dark Matter

##  1 Introduction

It is by now widely accepted that dark matter (DM) constitutes a substantial
fraction of the present critical energy density in the Universe. However, the
nature of DM remains an open problem. There exist many DM candidates, most of
them are non-baryonic weakly interacting massive particles (WIMPs), or WIMP-
like particles [1]. By far the DM candidates that have been more intensively
studied are the lightest supersymmetric (SUSY) particles such as neutralinos
or gravitinos, and the axions (as well as the axinos). There are additional
particle physics inspired dark matter candidates [1]. A candidate which is not
as closely related to particle physics is the relics of primordial black holes
(Micro Black Holes) [2,3]. Certain inflation models naturally induce a large
number of such a black holes. As a specific example, hybrid inflation can in
principle yield the necessary abundance of primordial black hole remnants for
them to be the primary source of dark matter [4,5]. In recent years it has
been suggested that measurements in quantum gravity should be governed by
generalized uncertainty principle (GUP). In fact, some evidences from string
theory, quantum geometry and black hole physics, have led some authors to re-
examine usual uncertainty principle of Heisenberg [6-13]. These evidences have
origin on the quantum fluctuations of the background space-time metric.
Existence of a minimal length scale on the order of Planck length is an
immediate consequence of GUP. Introduction of this idea has attract
considerable attention and many authors considered various problems in the
framework of generalized uncertainty principle [14-28]. The issue of black
holes remnants has been considered by some authors. Adler and his coworkers
have argued that contrary to standard viewpoint, GUP may prevent small black
holes total evaporation in exactly the same manner that the uncertainty
principle prevents the Hydrogen atom from total collapse [29]. Chen
considering inflation induced primordial black holes, have investigated the
issue of stability of such relics [30]. Recently, Varying Speed of Light
(VSL), as a new conjecture, which has been proposed to solve the problems of
standard cosmology, has attract some attentions. After introduction of this
conjecture, several alternative VSL theories have been proposed and some of
their novel implications have been examined extensively [31-35].
In this paper the issue of black hole remnants will be considered in the
framework of both GUP and VSL. The main consequence of this combination is
related to the stability problem of possible remnants. In forthcoming sections
first we show that GUP provides a reasonable framework for VSL. Then using a
simple VSL model we will show that if one consider both the effect of GUP and
VSL on the thermodynamics of black holes, the results of Bekenstein-Hawking
concerning total evaporation of black holes should be re-examined. In
Bekenstein-Hawking approach the total evaporation of micro black hole is
possible. Here we will see that it is possible to have relics of evaporating
black holes which can be considered as a possible candidate for dark matter.
The structure of the paper is as follows: in section 2 we show that VSL can be
considered as an immediate consequence of GUP. Section 3 is devoted to the
quantum gravitational corrected black hole thermodynamics. Some numerical
calculations have been down and their physical results are discussed. The
paper follows by conclusions and discussion regarding observational scheme for
detection of such relics and their cosmological constraints in section 4.

##  2 Preliminaries

As has been revealed in introduction, usual uncertainty principle of quantum
mechanics, the so-called Heisenberg uncertainty principle, should be re-
formulated due to noncommutative nature of spacetime at Planck scale. As a
consequence, it has been indicated that in quantum gravity there exists a
minimal observable distance on the order of the Planck length which governs on
all measurements in extreme quantum gravity limit. In the context of string
theories, this observable distance is referred to GUP. A generalized
uncertainty principle can be formulated as

$\Delta x\geq\frac{\hbar}{\Delta p}+const.G\Delta p,$ (1)

which, using the minimal nature of $l_{P}$ can be written as,

$\Delta x\geq\frac{\hbar}{\Delta p}+\frac{\alpha^{\prime}l_{p}^{2}\Delta
p}{\hbar}.$ (2)

The main consequence of GUP is that measurement of the position is possible
only up to Planck length, $l_{P}$. So one can not setup a measurement to find
more accurate particle position than Planck length, and this means that the
notion of locality breaks down. It is important to note that there are more
generalization which contain further terms in right hand side of equation (2)
(see [36]), but in some sense regarding dynamics, equation (2) has more
powerful physical grounds. Suppose that

$\Delta x\sim x,\quad\quad\Delta p\sim p,\quad\quad p=\hbar k,\quad\quad
x=\bar{\lambda}=\frac{\lambda}{2\pi}.$

Therefore one can write,

$\bar{\lambda}=\frac{1}{k}+\alpha^{\prime}l_{p}^{2}\,k\qquad
and\qquad\omega=\frac{c}{\bar{\lambda}}.$ (3)

In this situation the dispersion relation becomes,

$\omega=\omega(k)=\frac{kc}{1+\alpha^{\prime}l_{p}^{2}\,k^{2}}.$ (4)

This relation can be described in another viewpoint. By expansion of
$\Big{(}1+\alpha^{\prime}l_{p}^{2}\,k^{2}\Big{)}^{-1}$ and neglecting second
and higher order terms of $\alpha^{\prime}$, we find that
$\omega=kc\big{(}1-\alpha^{\prime}l_{p}^{2}\,k^{2}\big{)}$. This can be
considered as $\omega=kc^{\prime}$ where
$c^{\prime}=c\big{(}1-\alpha^{\prime}l_{p}^{2}\,k^{2}\big{)}$. This relation
indicates the possibility of variation in $c$. Accepting the possibility of
variation in $c$, one can consider its time variation also. So we consider the
time dependence of light speed.
Actually, there are some evidences indicating that fine structure constant,
$\alpha=\frac{e^{2}}{\hbar c}$ is not constant [11]. The question then arises
that which of the quantities: $e$, $c$ or $\hbar$ are variable? A possible
situation is the variation in $c$. This is referred as varying speed of light
(VSL) theories in literatures. After introduction of this idea several varying
speed of light models have been proposed to solve the problems of standard
cosmology. One of the simplest of these models is the model proposed by Barrow
[35]. Barrow has considered the speed of light as

$c(t)=c_{0}a^{n}(t),$ (5)

where $c_{0}$ and $n$ are constant. Using this form of $c(t)$ to solve
problems of standard cosmology, some constraint will be imposed on the value
of $n$, depending on the nature of the problems. For example if we consider
the equation of state for matter content of the Universe as $p=(\gamma-1)\rho
c^{2}(t)$, then exact solutions with varying $c(t)$ and $G(t)$, restrict $n$
to the following limit

$n\leq-1\,\,\,\,for\,\,\,\gamma=4/3\quad\quad\quad
Radiation\,\,Dominated\,\,Era$ (6) $n\leq-1/2\,\,\,for\,\,\,\gamma=1\quad\quad
Matter\,\,Dominated\,\,Era,\,Dust.$ (7)

##  3 Black Holes Thermodynamics

In the current standard viewpoint, small black holes emit black body radiation
at the Hawking temperature,

$T_{H}\approx\frac{\hbar c^{3}}{8\pi GM}=\frac{M^{2}_{P}c^{2}}{8\pi M},$ (8)

where $M_{P}=\sqrt{\frac{\hbar c}{G}}$ is the Planck mass and we have set
$k_{B}=1$. The related entropy is obtained by integration of
$dS=c^{2}T^{-1}dM$ which is the standard Bekenstein entropy,

$S_{B}=\frac{4\pi GM^{2}}{\hbar c}=4\pi\frac{M^{2}}{M^{2}_{P}}.$ (9)

If one consider the GUP as equation (2), the last two equations become
respectively,

$T_{GUP}=\frac{Mc^{2}}{4\pi}\Bigg{[}1\mp\sqrt{1-\frac{M^{2}_{P}}{M^{2}}}\,\Bigg{]},$
(10)

and

$S_{GUP}=2\pi\Bigg{[}\frac{M^{2}}{M^{2}_{P}}\Bigg{(}1-\frac{M^{2}_{P}}{M^{2}}+\sqrt{1-\frac{M^{2}_{P}}{M^{2}}}\,\Bigg{)}-\ln\Bigg{(}\frac{M+\sqrt{M^{2}-M^{2}_{P}}}{M_{P}}\,\Bigg{)}\,\Bigg{]}.$
(11)

In equation (10), to recover the corresponding result in the limit of large
mass ($T_{H}$), one should consider the minus sign. These equations strongly
suggest the existence of black holes remnants. As it is evident from figure 2,
in the framework of GUP black hole can evaporate until when it reachs the
Planck mass. In this view point black hole remnants are stable. Now consider
the case of VSL. For simplicity we assume that only $c$ is varying and $G$ and
$\hbar$ are constant which we set $G=\hbar=1$. In this situation
$M_{P}=\sqrt{c(t)}$. This is a novel concept: a time-varying Planck mass!. It
means that Planck scales are varying with time and are actually cosmological
models dependent via dependence of $c(t)$ to cosmological scale factor. The
corresponding equations both in Hawking-Bekenstein and GUP viewpoint will
become as follows respectively,

$T^{(VSL)}_{H}(t)=\frac{c^{3}(t)}{8\pi M},$ (12) $S^{(VSL)}_{B}(t)=\frac{4\pi
M^{2}}{c(t)},$ (13)
$T^{(VSL)}_{GUP}(t)=\frac{Mc^{2}(t)}{4\pi}\Bigg{[}1-\sqrt{1-\frac{c(t)}{M^{2}}}\,\Bigg{]},$
(14)

and

$S^{(VSL)}_{GUP}(t)=2\pi\Bigg{[}\frac{M^{2}}{c(t)}\Bigg{(}1-\frac{c(t)}{M^{2}}+\sqrt{1-\frac{c(t)}{M^{2}}}\,\Bigg{)}-\ln{\Bigg{(}\frac{M+\sqrt{M^{2}-c(t)}}{\sqrt{c(t)}}\,\Bigg{)}}\,\Bigg{]}.$
(15)

Now one should specify the time dependence of $c(t)$. Using equation (5),
since there exists several possibilities for $a(t)$ and
$c(t)=M^{2}_{P}(t)=a^{n}(t)$, we can consider de Sitter Universe as an
example,

$c(t)=\Big{[}a_{0}\cosh(\frac{t}{a_{0}})\Big{]}^{n}.$ (16)

Now, equations (12)-(15) for de Sitter Universe become respectively,

$T^{(VSL)}_{H}(t)=\frac{\cosh^{3n}(t)}{8\pi M},$ (17)
$S^{(VSL)}_{B}(t)=\frac{4\pi M^{2}}{\cosh^{n}(t)},$ (18)
$T^{(VSL)}_{GUP}(t)=\frac{M\cosh^{2n}(t)}{4\pi}\Bigg{[}1-\sqrt{1-\frac{\cosh^{n}(t)}{M^{2}}}\,\Bigg{]},$
(19)

and

$S^{(VSL)}_{GUP}(t)=2\pi\Bigg{[}\frac{M^{2}}{\cosh^{n}(t)}\Bigg{(}1-\frac{\cosh^{n}(t)}{M^{2}}+\sqrt{1-\frac{\cosh^{n}(t)}{M^{2}}}\,\Bigg{)}-\ln{\Bigg{(}\frac{M+\sqrt{M^{2}-\cosh^{n}(t)}}{\sqrt{\cosh^{n}(t)}}\,\Bigg{)}}\,\Bigg{]}.$
(20)

Where we have set $a_{0}=1$. The results of numerical calculations are shown
in figures. In these figures we have considered $n=-1$ and the results are
shown for de Sitter Universe. Note that for different $n$, the overall
behavior of the solutions do not change considerably and other model Universes
give similar results.

##  4 Conclusions and Discussion

Based on our model and numerical calculations, the following results are
obtained

  1. 1-

When one considers the time variation of speed of light alone, total
evaporation of black hole is possible in principle. Thus in the framework of
VSL micro black hole can evaporate completely. This is in agreement with the
Hawking-Bekenstein and in contrast with the results of GUP.

  2. 2-

Application of generalized uncertainty principle to black holes thermodynamics
strongly suggests the possible existence of black holes remnant (figure 1 and
2).

  3. 3-

When one considers thermodynamics of black holes in the framework of both GUP
and VSL, some novel results are obtained. The figure for temperature of black
hole versus the mass and time in a combination of GUP and VSL (figure 5) shows
that where the mass of black hole is zero, its temperature is zero also. This
is in contrast to Hawking result and seems completely reasonable since in the
absence of matter there is no meaning for temperature. When the mass
increases, the temperature is increases until the mass becomes equal to the
Planck mass. After that, increasing of mass is corresponding to decreasing of
temperature in complete agreement with the results of GUP (Adler et al [29]).
But the situation for mass less than Planck mass is completely different from
GUP results.

  4. 4-

The figure for entropy of black hole versus the mass and time in a combination
of GUP and VSL (figure 6) shows that when one approaches the Planck mass,
entropy do not vanishes. This is physically reasonable but rules out the
result of Adler et al since they have zero entropy for remnants. Increasing
the time will increase the entropy which is natural. The possibility of having
black hole remnant at Planck mass is evident from this figure. In our model
there is a remnant entropy for black remnant. This can be at least related to
background spacetime metric fluctuation.

  5. 5-

Adler and his coworkers have constructed their formulation based on analogy
between hydrogen atom and black holes. They have argued that since uncertainty
principle prevents the hydrogen atom from total collapse, generalized
uncertainty principle may prevent black holes total evaporation in the same
manner. In our opinion, the basic mistake of Adler et al is that they have not
considered hydrogen atom in GUP. Our calculation shows that in GUP hydrogen
atom is not stable. Since,

$\Delta r\Delta p\geq\frac{\hbar}{2}(1+\beta(\Delta p)^{2}).$ (21)

Suppose that $\Delta p\sim p$ and $\Delta r\sim r$, then one finds

$pr=\frac{\hbar}{2}(1+\beta p^{2})\Rightarrow\hbar\beta p^{2}-2pr+\hbar=0.$
(22)

So one obtains,

$p=\frac{r\pm\sqrt{r^{2}-\beta\hbar^{2}}}{\beta\hbar}.$ (23)

As has been argued we should consider the minus sign in (23). Since

$E=\frac{p^{2}}{2m}-\frac{e^{2}}{r},$ (24)

one find,

$E=\frac{1}{2m}\bigg{(}\frac{r-\sqrt{r^{2}-\beta\hbar^{2}}}{\beta\hbar}\,\bigg{)}^{2}-\frac{e^{2}}{r}\,.$
(25)

Now we set, $\frac{dE}{dr}=0$ and find $r_{min}=\hbar\sqrt{\frac{\beta}{2}}$.
The extremum value of energy becomes,

$E_{min}=-\bigg{[}\Big{(}\frac{e^{2}}{\hbar}\,\sqrt{\frac{2}{\beta}}\,\Big{)}+i\frac{1}{2m\beta}\bigg{]},$
(26)

where its real part is,

$E_{min}^{real}=-\Big{(}\frac{e^{2}}{\hbar}\,\sqrt{\frac{2}{\beta}}\,\Big{)}.$
(27)

Since $r_{min}$ is very small length, the radius of stability for hydrogen
atom is very small. Therefore in GUP scale, the hydrogen atom is not stable
and will collapse completely. If this is the case, one can not construct
analogy between hydrogen atom in Heisenberg uncertainty principle viewpoint
and black hole in GUP viewpoint. If we consider hydrogen atom in GUP, as we
have shown, this atom will collapse totally. But in the framework of GUP black
holes evaporate until they reach Planck mass. The issue of stability for
remnant can be considered in the framework of symmetry principle in the
system. In this regard supersymmetry, in particular supergravity, stands a
very good framework of providing such black hole remnants [30].
Note that our arguments for the existence of black hole remnants based on GUP
is heuristic. The search for its deeper theoretical foundation is currently
underway. As interactions with black hole remnants are purely gravitational,
the cross section is extremely small, and direct observation of these remnants
seems unlikely. One possible indirect signature may be associated with the
cosmic gravitational wave background. Unlike photons, the gravitons radiated
during evaporation would be instantly frozen. Since, according to our notion,
the black hole evaporation would terminate when it reduces to a remnants, the
graviton spectrum should have a cutoff at Planck mass. Such a cutoff would
have by now been red-shifted to $\sim 10^{14}GeV$. Another possible
gravitational wave-related signature may be the gravitational wave released
during the gravitational collapse. The frequencies of such gravitational waves
would by now be in the range of $\sim 10^{7}-10^{8}Hz$. It would be
interesting to investigate whether these signals are in principle observable.
Another possible signature may be some imprints on the CMB fluctuations due to
the thermodynamics of black hole remnants-CMB interactions. Possible
production of such remnants in LHC (Large Hadron Collider) and also in cosmic
ray showers are under investigation. If we consider hybrid inflation as our
primary cosmological model, there will be some observational constraints on
hybrid inflation parameters. For example a simple calculation based on hybrid
inflation suggests that the time it took for black holes to reduce to remnants
is about $10^{-10}Sec$. Thus primordial black holes have been produced before
baryogenesis and subsequent epochs in the standard cosmology [30].

Acknowledgment
This Work has been supported partially by Research Institute for Astronomy and
Astrophysics of Maragha, Iran. Also we would like to appreciate an unknown
referee of MPLA for his (her) valuable comments on original version of the
paper.

## References

  * [1] P. Gondolo, Lectures delivered at the NATO Advanced Study Institute ”Frontiers of the Universe”, 8-20 Sept 2003, Cargese, France; astro-ph/0403064.
  * [2] Ya. B. Zeldovich and I. D. Novikov, Sov. Astron. 10 (1966) 602.
  * [3] S. W. Hawking, Mon. Not. R. Astron. Soc. 152 (1971) 75.
  * [4] E. J. Copeland et al, Phys. Rev. D 49 (1994) 6410.
  * [5] D. H. Lyth and A. Riotto, Phys. Rep. 314 (1999) 1; A. Linde, Phys. Rep. 575 (2000) 333.
  * [6] G. Veneziano, Europhys. Lett. 2 (1986) 199; Proc. of Texas Superstring Workshop (1989).
  * [7] D. Amati, M. Ciafaloni and G. Veneziano, Phys. Lett. B 216 (1989) 41.
  * [8] D. Amati, M. Ciafaloni and G. Veneziano, Phys. Lett. B 197 (1987) 81; Int. J. Mod. Phys. A 3 (1988) 1615; Nucl. Phys. B 347 (1990) 530.
  * [9] D. J. Gross and P. F. Mende, Phys. Lett. B 197 (1987) 129; Nucl. Phys. B 303 (1988) 407.
  * [10] K. Konishi, G. Paffuti and P. Provero, Phys. Lett. B 234 (1990) 276; R. Guida, K. Konishi and P. Provero, Mod. Phys. Lett. A 6 (1991) 1487.
  * [11] M. Kato, Phys. Lett. B 245 (1990) 43.
  * [12] L. J. Garay, Int. J. Mod. Phys. A 10 (1995) 145.
  * [13] S. Capozziello, G. Lambiase and G. Scarpetta, Int. J. Theor. Phys. 39 (2000) 15.
  * [14] M. Maggiore, Phys. Lett. B 304 (1993) 65.
  * [15] C. Castro, Found. Phys. Lett. 10 (1997) 273.
  * [16] A. Camacho, Gen. Rel. Grav. 34 (2002) 1839.
  * [17] M. Maggiore, Phys. Rev. D 49 (1994) 5182.
  * [18] M. Maggiore, Phys. Lett. B 319 (1993) 83.
  * [19] S. Kalyana Rama, Phys. Lett. B 519 (2001) 103.
  * [20] A. Camacho, Gen. Rel. Grav. 35 (2003) 1153.
  * [21] F. Scardigli and R. Casadio, Class. Quant. Grav. 20 (2003) 3915.
  * [22] S. Hossenfelder, Mod. Phys. Lett. A 19 (2004) 2727.
  * [23] S. Hossenfelder, Phys. Rev. D 70 (2004) 105003.
  * [24] S. Hossenfelder, Mod. Phys. Lett. A 19 (2004) 2727.
  * [25] A. J. M. Medved and E. C. Vagenas, Phys. Rev. D 70 (2004) 124021\.
  * [26] A. J. M. Medved, Class. Quant. Grav. 22 (2005) 133.
  * [27] B. Bolen and M. Cavaglia, Gen. Rel. Grav. 37 (2005) 1255.
  * [28] M. R. Setare, Phys. Rev. D 70 (2004) 087501.
  * [29] P. Chen and R. J. Adler, Nucl. Phys. B (Proc.Suppl.) 124 (2003) 103; R. J. Adler, P. Chen and D. I. Santiago, Gen. Rel. Grav. 33 (2001) 2101.
  * [30] P. Chen, New Astron. Rev. 49 (2005) 233.
  * [31] A. Albrecht and J. Magueijo, Phys. Rev. D 59 (1999) 043516\.
  * [32] J. Magueijo, Phys. Rev. D 62 (2000) 103521.
  * [33] J. D. Barrow and J. Magueijo, Phys. Lett. B 443 (1998) 104; Phys. Lett. B 447 (1999) 246; Class. Quant. Grav. 16 (1999) 1435.
  * [34] J. Magueijo, Phys. Rev. D 63 (2001) 043502.
  * [35] J. D. Barrow, Phys. Rev. D 59 (1999) 043515.
  * [36] A. Kempf et al., Phys. Rev. D 52 (1995) 1108.

Figure 1: Temperature of a Black Hole Versus the Mass. Mass is in units of the
Planck mass and temperature is in units of the Planck energy. The lower curve
is the Hawking result, and the upper curve is the result of GUP.  Figure 2:
Entropy of a Black Hole Versus the Mass. Entropy is dimensionless and mass is
in units of the Planck mass. The upper curve is the Hawking result, and the
lower curve is the result of GUP.  Figure 3: Temperature of Black Hole Versus
the Mass and Time in VSL. The units are as previous and the result is shown
for De Sitter model. As figure shows, the result of Hawking is recovered.
Decreasing of temperature with increasing of time is natural.  Figure 4:
Entropy of Black Hole Versus the Mass and Time in VSL. The units are as
previous and the result is shown for De Sitter model. As figure shows, the
result of Hawking is recovered. Increasing of Entropy with increasing of time
is natural. Figure 5: Temperature of Black Hole Versus the Mass and Time in a
combination of GUP and VSL. The units are as previous and the result is shown
for De Sitter model. As figure shows, where the mass is zero the temperature
is zero also. This is in contrast to Hawking result. When the mass increases,
the temperature increases until the mass becomes equal to the Planck mass.
After that, increasing of mass is corresponding to decreasing of temperature.
This is a novel result of GUP+VSL Scenario. Figure 6: Entropy of Black Hole
Versus the Mass and Time in a Combination of GUP and VSL. The units are as
previous and the result is shown for De Sitter model. As figure shows, When
one approaches the Planck mass, entropy do not vanishes. Increasing entropy
with time is natural. This figure shows the possibility of having black holes
relics.