:

# Out-of-Distribution Detection for Medical Applications: Guidelines for
Practical Evaluation

Karina Zadorozhny<EMAIL_ADDRESS>
Pacmed BV - Amsterdam  The Netherlands  Patrick Thoral
<EMAIL_ADDRESS>
Paul Elbers<EMAIL_ADDRESS>
Department of Intensive Care Medicine  Laboratory for Critical Care
Computational Intelligence (LCCCI)  Amsterdam Medical Data Science (AMDS)
Amsterdam UMC  Vrije Universiteit  Amsterdam  The Netherlands
Giovanni Cinà<EMAIL_ADDRESS>
Pacmed BV - Amsterdam  The Netherlands

###### Abstract

Detection of Out-of-Distribution (OOD) samples in real time is a crucial
safety check for deployment of machine learning models in the medical field.
Despite a growing number of uncertainty quantification techniques, there is a
lack of evaluation guidelines on how to select OOD detection methods in
practice. This gap impedes implementation of OOD detection methods for real-
world applications. Here, we propose a series of practical considerations and
tests to choose the best OOD detector for a specific medical dataset. These
guidelines are illustrated on a real-life use case of Electronic Health
Records (EHR). Our results can serve as a guide for implementation of OOD
detection methods in clinical practice, mitigating risks associated with the
use of machine learning models in healthcare.

##  1 Introduction

When deploying a machine learning model in a high-risk environment such as
healthcare, it is crucial to reduce the risks of giving an incorrect
prediction. One of the most common causes of failures of a well-performing
model is dissimilarity between training (in-distribution) data and data that
the model is used on after deployment (Saria and Subbaswamy, 2019). This
problem is further exacerbated in a non-stationary medical environment: the
performance of predictive models can drop drastically when evaluated on
medical data collected just a couple of years removed from in-distribution
data (Nestor et al., 2018). The reason for this can be, for example, a change
in the demographics of the target population, changes in treatment protocols,
or changes in data collection procedures (Finlayson et al., 2021). These
examples constitute a covariate shift where feature distributions changed
compared to in-distribution data (Shimodaira, 2000; Moreno-Torres et al.,
2012). For reliable use of prediction models in healthcare it is necessary to
have the ability to detect these changes in real-time.

In the past five years, there has been a surge in methods for OOD detection
and uncertainty quantification. Despite the progress, OOD detection methods
often fail at flagging atypical inputs. While most studies have focus on image
data (Ovadia et al., 2019; Nalisnick et al., 2019), it has recently been shown
that many OOD detection methods fail to distinguish OOD inputs on medical
tabular data (Ulmer et al., 2020).

While predictive models are tailored and tested on specific datasets, there is
no universal way to evaluate OOD detection methods in practice. This gap
impedes wide-spread implementation of OOD detection methods. In this paper, we
provide a practical set of guidelines on how to select an OOD detector in a
given medical AI application. Our specific contributions are the following:

  * •

We describe how to take into account dataset-specific variables when
evaluating OOD detector models in practice.

  * •

We show how to test OOD detectors on distinct families of OOD groups that can
be created from available data using inclusion-exclusion criteria and
withholding samples during training.

  * •

We describe how to to use interpretability tools to assess whether influential
features that drive novelty predictions are clinically relevant.

  * •

We provide an open-source repository that can be used to evaluate OOD
detection methods for any mixed-type tabular
dataset.111https://github.com/Giovannicina/selecting_OOD_detector

We illustrate these general principles on real-world EHR data from the
Intensive Care Unit (ICU) of Amsterdam University Medical Centers
(AmsterdamUMC) (Thoral et al., 2021). We show that using the guidelines
proposed in this paper, we can detect underperforming OOD detectors and select
the best methods to be deployed in practice.

##  2 Related Work

Despite the impressive performance of machine learning models, recent studies
have highlighted the potential risks of undetected OOD data points in critical
areas such as healthcare. Besides deliberate threats such as data poisoning
and adversarial attacks (Papangelou et al., 2019; Finlayson et al., 2019;
Mozaffari-Kermani et al., 2015), deployed models can suffer from severe
malfunction due to dataset shifts. Finlayson et al. (2021) described different
scenarios that lead to dataset shifts in practice and highlighted the need for
more robust and reliable models in healthcare.

To mitigate risks of failure due to dataset shift caused by covariate shifts
(Bickel et al., 2009), recent studies proposed a plethora of uncertainty
quantification methods. These methods can be broadly divided into two groups
based on the type of uncertainty they express. The first group uses predictive
uncertainty (uncertainty about predictions) which is model-specific and
indicates how confident a model is in its prediction (Ovadia et al., 2019);
such uncertainty can be measured with a variety of metrics. Models in this
category include Bayesian Neural Networks (Blundell et al., 2015), Ensemble
Neural Networks (Lakshminarayanan et al., 2017), or Monte Carlo Dropout (Gal
and Ghahramani, 2016). The second group of models, density estimators,
expresses uncertainty about samples by learning the distribution of training
data. Models such as Variational Autoencoder (VAE; Kingma and Welling, 2014;
Ran et al., 2020) or Normalizing Flows (Rezende and Mohamed, 2015) can be used
to flag samples with low likelihood under an estimated in-distribution
function.

There has been only limited work published to date that focuses on the
practical evaluation of OOD detection methods in real-world scenarios.
Hendrycks et al. (2020) proposed a benchmark for OOD detection in the imaging
domain that could replace small-scale datasets containing only a few classes
with a more realistic high-scale multi-class anomaly segmentation task.
Techapanurak and Okatani (2021) described practical tests of OOD detection in
the imaging domain to aid the evaluation of models in real-world applications.
However, no guidelines on how to test OOD detectors are available for medical
tabular data. In this paper, we aim to close this gap between research studies
and the practical implementation of OOD detection methods in a medical
setting.

##  3 Methods

We first describe considerations that should be taken into account when
choosing an OOD detector for a specific application. In Section 3.2, we put
forward guidelines for creating tests to assess the performance of OOD
detection methods.

The proposed OOD tests are illustrated on a real-world use case of EHR data of
AmsterdamUMC. Details about the dataset, experimental set-up, implemented
models, and the way OOD groups were selected for this dataset are described in
Appendix A.

| Type of
---
OOD groups
| Examples | How To Create? |  | AmsterdamUMC
---
Example
Outside inclusion- exclusion criteria | Static | Age, race |  | Separate excluded groups
---
according to demographic-
related characteristics.
Dynamic |  | Treatment,
---
length of stay
| Split on one or more features
---
related to a clinical status that
is subject to change.
| Ventilation,
---
medications,
patients far
from discharge.
New disease |  |  | New infectious
---
disease, any diagnosis
underrepresented in
training data
| Artificially create
---
an OOD group by withholding
them during training.
| COVID-19 patients
---
and suspects.

Table 1: Clinically relevant OOD groups.

###  3.1 Considerations

####  3.1.1 Model Type

It has been shown that models that utilize predictive uncertainty in a
classification setting can be overconfident in their faulty predictions (Guo
et al., 2017; Ovadia et al., 2019). Specifically, a group of discriminator
models assign high predictive confidence to regions with no training data
(Hein et al., 2019; Ulmer and Cinà, 2021). Given these findings, we advise
using density estimators instead of models that express predictive
uncertainty. While there is a growing interest in the limitations of density
estimators for OOD detection (Nalisnick et al., 2019; Kirichenko et al., 2020;
Zhang et al., 2021), density models outperformed discriminator models on
several OOD detection tasks on tabular medical data (Ulmer et al., 2020).

####  3.1.2 Data Type

Many density estimators can perform well on continuous data but are not
directly suitable for discrete distributions. Research efforts of extending
models to discrete distributions most commonly focus on text inputs (Tran et
al., 2019; Miao et al., 2016). For some applications, such as learning density
of images, dequantization can be applied to map discrete pixel values data to
continuous distributions (Hoogeboom et al., 2020). However, this approach is
not applicable to categorical data with no intrinsic order.

Mixed type data that consist of categorical and continuous features pose
additional challenges to density modeling (Nazabal et al., 2020). Therefore,
the proportion of categorical features should be taken into account when
choosing an OOD detector.

####  3.1.3 Dimensionality and Size of Data

The dimensionality of data is a crucial factor that influences the performance
of density estimators. For example, standard kernel density estimators can
achieve very good results on low-dimensional inputs but their performance
degrades exponentially with an increased number of dimensions (Nagler and
Czado, 2016; Wang and Scott, 2019). Other models, such as Autoencoder (AE) and
Probabilistic Principal Component Analysis (PPCA) that perform dimensionality
reduction by default are less affected by the number of features.

Having a large number of samples from which to estimate data distribution can
unequally affect different density estimators. While most models benefit from
having access to more data, algorithms such as Local Outlier Factor (LOF) that
measure local density of samples using k-nearest-neighbors algorithm become
very time- and memory- inefficient with an increasing number of samples (de
Vries et al., 2010).

####  3.1.4 Class Balance

In the classification setting, having unbalanced classes, which is very common
for medical diagnostics, can negatively influence usefulness of learned data
density. If there is not a sufficient number of samples of one class present
in the training dataset, new samples of this minority class can be mistakenly
flagged as OOD despite being an essential component of the dataset. While
densities of two classes could be modeled separately or using a class-
conditional density estimator (Sohn et al., 2015), this approach is not
feasible as we would have to first predict a sample’s class and only then test
whether the sample is in-distribution. Therefore, likelihood scores for
underrepresented classes should be monitored when comparing different density
models.

###  3.2 Designing OOD Tests

Figure 1: OOD groups can be created from data that was excluded under
inclusion-exclusion criteria (panel A). If there are not enough samples in
excluded data, groups can be withheld during training (panel B).

While any data point outside the training distribution is considered as OOD,
it is not feasible to test against all possible inputs. Work by Zhang et al.
(2021) argues that it is not theoretically possible to design a method that
could detect all possible out-distribution inputs. Winkens et al. (2020)
distinguished far-OOD and near-OOD depending on how similar inputs are to in-
distribution data. Given that the latter group is more relevant and difficult
to detect, we focus on near-OOD scenarios, specifying families of OOD groups
that could be encountered in a medical setting in real life (Appendix Figure
9).

We suggest to create clinically relevant OOD tests to cover the following
categories (Table 1):

  * •

Detecting patients outside static inclusion-exclusion criteria (e.g.
demographics).

  * •

Detecting patients outside dynamic inclusion-exclusion criteria (e.g. a
treatment protocol).

  * •

Detecting patients with a new or underrepresented disease.

The OOD groups can be created either by using data eliminated under inclusion-
exclusion principle or by withholding specific groups from in-distribution
(Figure 1).

####  3.2.1 Patients Outside Static Inclusion-Exclusion Criteria

Processing medical data involves defining inclusion and exclusion criteria
that specify which samples are going to be part of the cohort to be analyzed.
Inclusion criteria, which are typically clinical or demographic
characteristics, must be present in order to include a sample. Exclusion
criteria eliminate samples that passed inclusion criteria but have other
characteristics that could invalidate downstream analysis (Patino and
Ferreira, 2018).

Inclusion-exclusion criteria can be used to evaluate the performance of OOD
detection methods. Provided that models are trained only on samples that
follow inclusion-exclusion criteria, we can treat ineligible samples as OOD
and compare how well these groups are flagged by OOD detectors. Such groups
can be made using demographic or clinical characteristics (Figure 1A).

####  3.2.2 Patients Outside Dynamic Inclusion-Exclusion Criteria

Another example of a clinically relevant OOD group is patients that should
currently be excluded from data, but this status can change over time. In a
medical setting, this is a common occurrence and can be caused by a change in
the treatment protocol.

If data about individual patients are available over a period of time, such
groups can be created by finding a criterion that is not static and is subject
to change. Alternatively, OOD groups can be created by splitting data on a
received treatment.

Figure 2: Assessing interpretability of OOD detectors. See Section 3.2.4 for
description.

####  3.2.3 Patients with New Diseases

A clinically relevant scenario that highlights the necessity of OOD detection
methods is an occurrence of novel diseases. COVID-19 provides an example of
the importance of being able to detect atypical symptoms in real-time instead
of giving a prediction and only assessing data retrospectively. A common
scenario occurs when a model is used for patients with a disease that was not
sufficiently represented in the in-distribution data. We advise creating an
OOD group of patients with underrepresented disease by either applying
inclusion-exclusion criteria (Figure 1A) or withholding samples of patients
with a specific disease from in-distribution training data (Figure 1B).

####  3.2.4 Interpretability of OOD Detection Methods

Another consideration when deploying OOD detection methods is how transparent
novelty predictions are. In high-stakes areas such as healthcare, it is not
only useful to be able to flag a sample as OOD but also to see what set of
features makes it different from in-distribution data.

Some density estimators are inherently interpretable. For example, for AE-
based models, it is possible to inspect features that received the highest
reconstruction error. To provide a comparative evaluation of interpretability
for all models, we suggest using Shapley Additive Explanations (SHAP; Lundberg
and Lee, 2017).

We describe two approaches of assessing interpretability of OOD detection
methods (Figure 2). The first, dataset-level approach, splits data on one
feature to divide in-distribution and OOD data. After scoring OOD samples,
SHAP is used to determine feature importance for the assigned novelty scores.
Models are then scored on the importance of the feature that was used for
splitting (Figure 2A).

In the second, qualitative approach, OOD detectors are trained on in-
distribution data and are used to score test data (Figure 2B). Samples that
receive the highest novelty scores (outliers) are then inspected individually.
Ranking the most influential features using SHAP and having medical
professionals compare values of these features can help validate whether
models are influenced by clinically relevant features and verify that outliers
are indeed meant to be in-distribution.

####  3.2.5 Inference Time for Real-Time Prediction

The ability to flag OOD samples in real-time, as compared to detecting batches
of atypical samples retrospectively, is an important consideration for the
deployment of OOD detection methods. While inference time for individual
predictions tends not be a limiting factor, when coupled with explainability
tools such as SHAP, can significantly slow down the prediction process.

###  3.3 Experimental setup

To illustrate the above-described principles in practice, we put them to test
on the AmsterdamUMC dataset. The downstream prediction task is a binary
classification of readmission to ICU after discharge or mortality. For this
task, any predictive model can be used while we aim to compare different OOD
detection on this dataset (see Appendix A for more details about the dataset
and the experimental set-up).

The density estimators compared in the following experiments include AE, VAE
(Kingma and Welling, 2014), Deterministic Uncertainty Estimator (DUE; van
Amersfoort et al., 2021) based on Spectral Normalized Gaussian Process (Liu et
al., 2020), Masked Autoregressive Flow (Flow Papamakarios et al., 2018), PPCA
(Tipping and Bishop, 1999), and LOF de Vries et al. (2010). Description of the
models and hyperparameters can be found in Appendix A.5.

##  4 Results

In the following section we describe the results obtained applying the
aforementioned methodology on the AmsterdamUMC data.

Dynamic Inclusion-Exclusion Criteria: Length of Stay. We followed the
guidelines described in Section 3.2.2 and selected patients outside dynamic
inclusion-exclusion criteria which are subject to change in time. We used the
fact that the AmsterdamUMC dataset contains retrospective time series data for
each patients. The in-distribution data for this experiment are patients in
the last day of their ICU admission and OOD groups are separated based on how
far patients are from being discharged (Figure 3). The further patients are
from discharge, the more dissimilar their features are compared to in-
distribution data. While all models except DUE showed the desired gradient of
increasing AUC-ROC score, Flow achieved a near-perfect score already for
patients 4-5 days before discharge.

Figure 3: Mean AUC scores ($n$=5) of detecting patients far from discharge.
Row labels indicate the number of days before discharge.

Dynamic Inclusion-Exclusion Criteria: Discharge Destination. Given that the
main predictive task is classification of readmission probability, patients
that died at the ICU or were transferred to another department or a different
hospital are excluded from in-distribution data.

Most models detected patients with discharge destination of mortuary more
easily than patients that were transferred to an ICU of a different hospital
(Appendix Figure 5). This result is reassuring as, arguably, features of
transferred patients more closely resemble in-distribution data.

Figure 4: Top panel: Mean AUC scores and standard deviations ($n$=5) of
detecting ventilated patients and patients with a renal kidney failure
receiving CVVH. Bottom panel: Novelty score distributions. Plotted values are
clipped to 5-95% of test novelty scores to prevent outlier scores from skewing
the graphs.

Dynamic Inclusion-Exclusion Criteria: Received Treatment. Next, we selected
patients outside dynamic inclusion-exclusion criteria based on the type of
received treatment. We compared the OOD detection methods on two categories of
treatment-related exclusion criteria: patients connected to a specific device
and patients receiving medication interventions.

We first tested whether novelty detectors are able to flag patients connected
to a ventilating machine and continuous veno-venous hemofiltration (CVVH)
device (Figure 4). Note that in our experiment, there are no features that
would directly indicate whether patients are ventilated or on CVVH, and
therefore, OOD detectors must infer this information from the rest of the
features. While most models are assigning greater novelty scores to the OOD
groups, Flow model has the least overlapping distributions for the groups and
in-distribution test data (Figure 4, bottom panel). We also compared how the
detection of the ventilated patients changes in time as they get closer to
discharge (Appendix Figure 6).

To test medication-related OOD detection, we evaluated the models in detection
patients that fall outside of inclusion-exclusion criteria due to
administration of inotropic or vassopressor medication (Appendix Figure 7).
Flow flagged these patients with AUC as high as 0.97-0.98%

Observing Patients with New Diseases. We use data from the same ICU containing
confirmed and suspected COVID-19 patients collected as part of the Dutch Data
Warehouse (Fleuren et al., 2021) to test OOD detectors in their ability to
flag patients with this disease (Appendix Figure 8). While AE, PPCA, and VAE
achieved AUC-ROC above 70% for COVID-19 patients, Flow was able to achieve
over 90% for both COVID-19 patients and suspects.

Inference Time for Real-Time Predictions. Given that predictions about the
novelty of a sample are most useful when given in real-time, we timed OOD
detectors in predicting a novelty score for a single sample and when coupled
with SHAP (See Appendix Table 2). AE and PPCA are almost 30 times faster in
inference and SHAP calculations than the other methods.

##  5 Discussion

While it is standard practice to evaluate predictive models on each dataset,
there is no established way to compare different OOD detection methods. In
this paper, we described a set of practical guidelines on how to create and
test OOD detection methods on medical tabular data.

A similar effort in bridging the gap between research studies and practical
implementation was shown for other critical areas such as AI explainability.
Mohseni et al. (2020) provided categorization of evaluation strategies of
explainable AI, while Gade et al. (2019) presented practical guidelines for
using explainability techniques which were illustrated using different case
studies. Other studies have proposed general considerations for deploying
machine learning models into clinical practice Chen et al. (2019).

There are many outstanding questions related to design and selection of OOD
detection methods. First, to our knowledge, there is no direct solution to
class imbalance of in-distribution data for OOD detection and current methods
have to rely on availability of sufficiently large sample size. Second,
current models are not well suited for dealing with mixed-type data containing
categorical and continuous distributions as they require different likelihood
functions. Extending density estimators to mixed-type data (Nazabal et al.,
2020; Ma et al., 2020) could greatly enhance OOD detection results. Third,
more efficient ways to deal with feature correlation can improve evaluation of
interpretability of OOD detectors to prevent feature-importance spread
(Lundberg and Lee, 2017; Aas et al., 2021).

The setup described here assumes some prior knowledge about the possible OOD
groups that should be detected as near-OOD (Winkens et al., 2020). In
addition, we acknowledge that apart from tests described in this paper, other
clinically relevant scenarios include detecting samples from different
hospitals, and testing for corrupted features (see for example Ulmer et al.
(2020)).

How to aggregate results of different OOD tests and select the most reliable
OOD detector across the results is another open-ended and application-specific
question. In our experiments on real-life mixed-type EHR data, Flow and VAE
performed consistently well across different OOD tests. Given that the
performance of density estimators depends on the type of data and
dimensionality of the dataset (see Section 3.1), the superior performance of
these two models can be explained by a relatively low number of categorical
features (less than 10%), potential non-linear interactions of different
features (which could be more problematic for PPCA), and large data size
(which can make LOF less efficient). Recently, failures of normalizing flows
in assigning higher likelihood to OOD inputs were discussed by several papers
(Kirichenko et al., 2020; Zhang et al., 2021; Nalisnick et al., 2019). We
hypothesize that structured data such as images that give rise to local pixel
correlations are more prone to such failures than tabular data.

Finally, the guidelines for evaluating OOD detectors for medical data
described in this paper can help OOD detectors bridge the gap from theoretical
possibility to deployed application, enhancing the safety of AI tools and
facilitating the uptake of this new technology in the medical field.

We would like to thank our colleagues at Pacmed and Amsterdam UMC for
providing us with the data and insights. We also thank Dennis Ulmer for
providing valuable feedback on the manuscript.

## References

  * Aas et al. (2021) Kjersti Aas, Martin Jullum, and Anders Løland.  Explaining individual predictions when features are dependent: More accurate approximations to Shapley values.  _Artificial Intelligence_ , 298:103502, September 2021.  ISSN 0004-3702.  10.1016/j.artint.2021.103502.  URL https://www.sciencedirect.com/science/article/pii/S0004370221000539.
  * Bickel et al. (2009) Steffen Bickel, Michael Brückner, and Tobias Scheffer.  Discriminative Learning Under Covariate Shift.  _Journal of Machine Learning Research_ , 10(75):2137–2155, 2009.  URL http://jmlr.org/papers/v10/bickel09a.html.
  * Blundell et al. (2015) Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra.  Weight Uncertainty in Neural Networks.  In Francis Bach and David Blei, editors, _Proceedings of the 32nd International Conference on Machine Learning_ , volume 37 of _Proceedings of Machine Learning Research_ , pages 1613–1622, 2015.
  * Chen et al. (2019) Po-Hsuan Cameron Chen, Yun Liu, and Lily Peng.  How to develop machine learning models for healthcare.  _Nature Materials_ , 18(5):410–414, May 2019\.  ISSN 1476-4660.  10.1038/s41563-019-0345-0.  URL https://www.nature.com/articles/s41563-019-0345-0.  Bandiera_abtest: a Cg_type: Nature Research Journals Number: 5 Primary_atype: Comments & Opinion Publisher: Nature Publishing Group.
  * de Vries et al. (2010) Timothy de Vries, Sanjay Chawla, and Michael E. Houle.  Finding Local Anomalies in Very High Dimensional Space.  In _2010 IEEE International Conference on Data Mining_ , pages 128–137, December 2010.  10.1109/ICDM.2010.151.  ISSN: 2374-8486.
  * Durkan et al. (2020) Conor Durkan, Artur Bekasov, Iain Murray, and George Papamakarios.  nflows: normalizing flows in PyTorch, November 2020.  URL https://doi.org/10.5281/zenodo.4296287.
  * Finlayson et al. (2019) Samuel G. Finlayson, John D. Bowers, Joichi Ito, Jonathan L. Zittrain, Andrew L. Beam, and Isaac S. Kohane.  Adversarial attacks on medical machine learning.  _Science_ , 363(6433):1287–1289, March 2019.  ISSN 0036-8075, 1095-9203.  10.1126/science.aaw4399.  URL https://science.sciencemag.org/content/363/6433/1287.  Publisher: American Association for the Advancement of Science Section: Policy Forum.
  * Finlayson et al. (2021) Samuel G. Finlayson, Adarsh Subbaswamy, Karandeep Singh, John Bowers, Annabel Kupke, Jonathan Zittrain, Isaac S. Kohane, and Suchi Saria.  The Clinician and Dataset Shift in Artificial Intelligence.  _New England Journal of Medicine_ , 385(3):283–286, July 2021.  ISSN 0028-4793.  10.1056/NEJMc2104626.  URL https://doi.org/10.1056/NEJMc2104626.  Publisher: Massachusetts Medical Society _eprint: https://doi.org/10.1056/NEJMc2104626.
  * Fleuren et al. (2021) Lucas M. Fleuren, Tariq A. Dam, Michele Tonutti, Daan P. de Bruin, Robbert C. A. Lalisang, Diederik Gommers, Olaf L. Cremer, Rob J. Bosman, Sander Rigter, Evert-Jan Wils, Tim Frenzel, Dave A. Dongelmans, Remko de Jong, Marco Peters, Marlijn J. A. Kamps, Dharmanand Ramnarain, Ralph Nowitzky, Fleur G. C. A. Nooteboom, Wouter de Ruijter, Louise C. Urlings-Strop, Ellen G. M. Smit, D. Jannet Mehagnoul-Schipper, Tom Dormans, Cornelis P. C. de Jager, Stefaan H. A. Hendriks, Sefanja Achterberg, Evelien Oostdijk, Auke C. Reidinga, Barbara Festen-Spanjer, Gert B. Brunnekreef, Alexander D. Cornet, Walter van den Tempel, Age D. Boelens, Peter Koetsier, Judith Lens, Harald J. Faber, A. Karakus, Robert Entjes, Paul de Jong, Thijs C. D. Rettig, Sesmu Arbous, Sebastiaan J. J. Vonk, Mattia Fornasa, Tomas Machado, Taco Houwert, Hidde Hovenkamp, Roberto Noorduijn-Londono, Davide Quintarelli, Martijn G. Scholtemeijer, Aletta A. de Beer, Giovanni Cina, Martijn Beudel, Willem E. Herter, Armand R. J. Girbes, Mark Hoogendoorn, Patrick J. Thoral, and Paul W. G. Elbers.  The Dutch Data Warehouse, a multicenter and full-admission electronic health records database for critically ill COVID-19 patients.  _Critical Care_ , 25(1):304, August 2021.  ISSN 1364-8535.  10.1186/s13054-021-03733-z.  URL https://doi.org/10.1186/s13054-021-03733-z.
  * Gade et al. (2019) Krishna Gade, Sahin Cem Geyik, Krishnaram Kenthapadi, Varun Mithal, and Ankur Taly.  Explainable AI in Industry.  In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, KDD ’19, pages 3203–3204, New York, NY, USA, July 2019. Association for Computing Machinery.  ISBN 978-1-4503-6201-6.  10.1145/3292500.3332281.  URL https://doi.org/10.1145/3292500.3332281.
  * Gal and Ghahramani (2016) Yarin Gal and Zoubin Ghahramani.  Dropout as a Bayesian approximation: Representing Model Uncertainty in Deep Learning.  In _International conference on Machine Learning_ , pages 1050–1059, 2016.
  * Guo et al. (2017) Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger.  On Calibration of Modern Neural Networks.  In Doina Precup and Yee Whye Teh, editors, _Proceedings of the 34th International Conference on Machine Learning_ , volume 70 of _Proceedings of Machine Learning Research_ , pages 1321–1330. PMLR, 2017\.
  * Hein et al. (2019) Matthias Hein, Maksym Andriushchenko, and Julian Bitterwolf.  Why ReLU Networks yield High-confidence Predictions far away from the Training Data and how to Mitigate the Problem.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 41–50, 2019.
  * Hendrycks et al. (2020) Dan Hendrycks, Steven Basart, Mantas Mazeika, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Song.  Scaling out-of-distribution detection for real-world settings, 2020.
  * Hoogeboom et al. (2020) Emiel Hoogeboom, Taco S. Cohen, and Jakub M. Tomczak.  Learning Discrete Distributions by Dequantization.  _arXiv:2001.11235 [cs, stat]_ , January 2020.  URL http://arxiv.org/abs/2001.11235.  arXiv: 2001.11235.
  * Huang et al. (2018) Chin-Wei Huang, Shawn Tan, Alexandre Lacoste, and Aaron C Courville.  Improving Explorability in Variational Inference with Annealed Variational Objectives.  In _Advances in Neural Information Processing Systems_ , volume 31. Curran Associates, Inc., 2018.
  * Kingma and Welling (2014) Diederik P Kingma and Max Welling.  Auto-encoding variational bayes, 2014.
  * Kirichenko et al. (2020) Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson.  Why Normalizing Flows Fail to Detect Out-of-Distribution Data.  _arXiv:2006.08545 [cs, stat]_ , June 2020.  URL http://arxiv.org/abs/2006.08545.  arXiv: 2006.08545.
  * Lakshminarayanan et al. (2017) Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell.  Simple and Scalable Predictive Uncertainty Estimation Using Deep Ensembles.  In _Advances in neural information processing systems_ , pages 6402–6413, 2017.
  * Liu et al. (2020) Jeremiah Z. Liu, Zi Lin, Shreyas Padhy, Dustin Tran, Tania Bedrax-Weiss, and Balaji Lakshminarayanan.  Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness.  In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_ , 2020.
  * Lundberg and Lee (2017) Scott Lundberg and Su-In Lee.  A unified approach to interpreting model predictions, 2017.
  * Ma et al. (2020) Chao Ma, Sebastian Tschiatschek, José Miguel Hernández-Lobato, Richard Turner, and Cheng Zhang.  Vaem: a deep generative model for heterogeneous mixed type data, 2020\.
  * Miao et al. (2016) Yishu Miao, Lei Yu, and Phil Blunsom.  Neural Variational Inference for Text Processing.  _arXiv:1511.06038 [cs, stat]_ , June 2016.  URL http://arxiv.org/abs/1511.06038.  arXiv: 1511.06038.
  * Mohseni et al. (2020) Sina Mohseni, Niloofar Zarei, and Eric D. Ragan.  A multidisciplinary survey and framework for design and evaluation of explainable ai systems, 2020.
  * Moreno-Torres et al. (2012) Jose G Moreno-Torres, Troy Raeder, Rocío Alaiz-Rodríguez, Nitesh V Chawla, and Francisco Herrera.  A Unifying View on Dataset Shift in Classification.  _Pattern recognition_ , 45(1):521–530, 2012.
  * Mozaffari-Kermani et al. (2015) Mehran Mozaffari-Kermani, Susmita Sur-Kolay, Anand Raghunathan, and Niraj K. Jha.  Systematic Poisoning Attacks on and Defenses for Machine Learning in Healthcare.  _IEEE journal of biomedical and health informatics_ , 19(6):1893–1905, November 2015.  ISSN 2168-2208.  10.1109/JBHI.2014.2344095.
  * Nagler and Czado (2016) Thomas Nagler and Claudia Czado.  Evading the curse of dimensionality in nonparametric density estimation with simplified vine copulas.  _Journal of Multivariate Analysis_ , 151:69–89, October 2016.  ISSN 0047259X.  10.1016/j.jmva.2016.07.003.  URL http://arxiv.org/abs/1503.03305.  arXiv: 1503.03305.
  * Nalisnick et al. (2019) Eric T. Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Görür, and Balaji Lakshminarayanan.  Do Deep Generative Models Know What They Don’t Know?  In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net, 2019.
  * Nazabal et al. (2020) Alfredo Nazabal, Pablo M. Olmos, Zoubin Ghahramani, and Isabel Valera.  Handling Incomplete Heterogeneous Data using VAEs.  _arXiv:1807.03653 [cs, stat]_ , May 2020.  URL http://arxiv.org/abs/1807.03653.  arXiv: 1807.03653.
  * Nestor et al. (2018) Bret Nestor, Matthew B. A. McDermott, Geeticka Chauhan, Tristan Naumann, Michael C. Hughes, Anna Goldenberg, and Marzyeh Ghassemi.  Rethinking clinical prediction: Why machine learning must consider year of care and feature aggregation, 2018.
  * Ovadia et al. (2019) Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek.  Can you Trust your Model’s Uncertainty? Evaluating Predictive Uncertainty under Dataset Shift.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Papamakarios et al. (2018) George Papamakarios, Theo Pavlakou, and Iain Murray.  Masked autoregressive flow for density estimation, 2018.
  * Papangelou et al. (2019) Konstantinos Papangelou, Konstantinos Sechidis, James Weatherall, and Gavin Brown.  Toward an Understanding of Adversarial Examples in Clinical Trials.  In Michele Berlingerio, Francesco Bonchi, Thomas Gärtner, Neil Hurley, and Georgiana Ifrim, editors, _Machine Learning and Knowledge Discovery in Databases_ , Lecture Notes in Computer Science, pages 35–51, Cham, 2019. Springer International Publishing.  ISBN 978-3-030-10925-7.  10.1007/978-3-030-10925-7_3.
  * Patino and Ferreira (2018) Cecilia Maria Patino and Juliana Carvalho Ferreira.  Inclusion and exclusion criteria in research studies: definitions and why they matter.  _Jornal Brasileiro de Pneumologia_ , 44(2):84, 2018.  ISSN 1806-3713.  10.1590/S1806-37562018000000088.  URL https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6044655/.
  * Ran et al. (2020) Xuming Ran, Mingkun Xu, Lingrui Mei, Qi Xu, and Quanying Liu.  Detecting out-of-distribution samples via variational auto-encoder with reliable uncertainty estimation, 2020.
  * Rezende and Mohamed (2015) Danilo Rezende and Shakir Mohamed.  Variational inference with normalizing flows.  In Francis Bach and David Blei, editors, _Proceedings of the 32nd International Conference on Machine Learning_ , volume 37 of _Proceedings of Machine Learning Research_ , pages 1530–1538, Lille, France, 07–09 Jul 2015. PMLR.  URL http://proceedings.mlr.press/v37/rezende15.html.
  * Saria and Subbaswamy (2019) Suchi Saria and Adarsh Subbaswamy.  Tutorial: Safe and reliable machine learning, 2019.
  * Shimodaira (2000) Hidetoshi Shimodaira.  Improving Predictive Inference under Covariate Shift by Weighting the Log-likelihood Function.  _Journal of statistical planning and inference_ , 90(2):227–244, 2000.
  * Sohn et al. (2015) Kihyuk Sohn, Honglak Lee, and Xinchen Yan.  Learning structured output representation using deep conditional generative models.  In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, _Advances in Neural Information Processing Systems_ , volume 28. Curran Associates, Inc., 2015.  URL https://proceedings.neurips.cc/paper/2015/file/8d55a249e6baa5c06772297520da2051-Paper.pdf.
  * Techapanurak and Okatani (2021) Engkarat Techapanurak and Takayuki Okatani.  Practical evaluation of out-of-distribution detection methods for image classification, 2021.
  * Thoral et al. (2021) Patrick J. Thoral, Jan M. Peppink, Ronald H. Driessen, Eric J. G. Sijbrands, Erwin J. O. Kompanje, Lewis Kaplan, Heatherlee Bailey, Jozef Kesecioglu, Maurizio Cecconi, Matthew Churpek, Gilles Clermont, Mihaela van der Schaar, Ari Ercole, Armand R. J. Girbes, Paul W. G. Elbers, and Amsterdam University Medical Centers Database (AmsterdamUMCdb) Collaborators and the SCCM/ESICM Joint Data Science Task Force.  Sharing ICU Patient Data Responsibly Under the Society of Critical Care Medicine/European Society of Intensive Care Medicine Joint Data Science Collaboration: The Amsterdam University Medical Centers Database (AmsterdamUMCdb) Example.  _Critical Care Medicine_ , 49(6):e563–e577, June 2021.  ISSN 1530-0293.  10.1097/CCM.0000000000004916.
  * Tipping and Bishop (1999) Michael E. Tipping and Christopher M. Bishop.  Probabilistic principal component analysis.  _Journal of the Royal Statistical Society. Series B (Statistical Methodology)_ , 61(3):611–622, 1999.  ISSN 13697412, 14679868.  URL http://www.jstor.org/stable/2680726.
  * Tran et al. (2019) Dustin Tran, Keyon Vafa, Kumar Krishna Agrawal, Laurent Dinh, and Ben Poole.  Discrete Flows: Invertible Generative Models of Discrete Data.  _arXiv:1905.10347 [cs, stat]_ , May 2019.  URL http://arxiv.org/abs/1905.10347.  arXiv: 1905.10347.
  * Ulmer and Cinà (2021) Dennis Ulmer and Giovanni Cinà.  Know your limits: Uncertainty estimation with relu classifiers fails at reliable ood detection, 2021.
  * Ulmer et al. (2020) Dennis Ulmer, Lotta Meijerink, and Giovanni Cinà.  Trust Issues: Uncertainty Estimation does not Enable Reliable OOD Detection on Medical Tabular Data.  In _Proceedings of the Machine Learning for Health NeurIPS Workshop_ , volume 136, pages 341–354, 2020.
  * van Amersfoort et al. (2021) Joost van Amersfoort, Lewis Smith, Andrew Jesson, Oscar Key, and Yarin Gal.  Improving Deterministic Uncertainty Estimation in Deep Learning for Classification and Regression, 2021.
  * Wang and Scott (2019) Zhipeng Wang and David W. Scott.  Nonparametric Density Estimation for High-Dimensional Data \- Algorithms and Applications.  _Wiley Interdisciplinary Reviews: Computational Statistics_ , 11(4):e1461, July 2019.  ISSN 1939-5108, 1939-0068.  10.1002/wics.1461.  URL http://arxiv.org/abs/1904.00176.  arXiv: 1904.00176.
  * Winkens et al. (2020) Jim Winkens, Rudy Bunel, Abhijit Guha Roy, Robert Stanforth, Vivek Natarajan, Joseph R. Ledsam, Patricia MacWilliams, Pushmeet Kohli, Alan Karthikesalingam, Simon Kohl, Taylan Cemgil, S. M. Ali Eslami, and Olaf Ronneberger.  Contrastive training for improved out-of-distribution detection, 2020\.
  * Zhang et al. (2021) Lily H. Zhang, Mark Goldstein, and Rajesh Ranganath.  Understanding Failures in Out-of-Distribution Detection with Deep Generative Models.  _arXiv:2107.06908 [cs]_ , July 2021.  URL http://arxiv.org/abs/2107.06908.  arXiv: 2107.06908.

##  Appendix A Extended Methods

###  A.1 Datasets

We used a dataset from Amsterdam University Medical Center (AmsterdamUMC).
This dataset was chosen to illustrate a real-life medical use case of EHR
data, which typically has a lot of features, including categorical, and an
unbalanced outcome.

AmsterdamUMC has released a freely accessible de-identified version of the
data from 2003-2016. Access to the data and/or the Dutch Data Warehouse
against COVID-19 can be requested from Amsterdam Medical Data Science
(https://amsterdammedicaldatascience.nl/).

AmsterdamUMC dataset contains retrospective data of patients admitted to
Intensive Care Unit (ICU) for each day of their admission stay. This results
in $58,142$ rows of data corresponding to $15,753$ patients. For some analyses
we sub-select rows of data for patients in their last day of admission at the
ICU. We refer to this part of the dataset as AmsterdamUMC-DIS and to the full
dataset as AmsterdamUMC-ROW. After eliminating ineligible patients using
inclusion-exclusion criteria, AmsterdamUMC-ROW dataset contained $n=40,923$
patients and AmsterdamUMC-DIS contained $n=15,695$ patients.

The total number of features in the dataset is 5269. Features were created by
analyzing time series and dividing data into windows of 24 hours. We selected
56 features of which 49 are continuous and 7 categorical. For model training
and hyperparameter search, we used a a 0.7, 0.15, 0.15, split for train,
validation, and test split, respectively. All feature names used are shown in
the list A1.1 at the end of the document

For the experiment in Figure 7, a second version of AmsterdamUMC dataset was
used with $n=18,064$ patients at their last day of ICU admission and $150$
features. This selection includes 84 categorical features and 66 continuous
features. We refer to this dataset as AmsterdamUMC-DIS-II. After applying
inclusion-exclusion criteria, the dataset contained $n=17,482$ patients.

###  A.2 Predictive Task

Our main goal is to find the best OOD detectors to flag samples that are not
well represented in the training data, and which should not receive a
prediction. We define this downstream prediction task to be a binary
classification of patients who died or were readmitted back to the ICU within
7 days after discharge. Given that the dataset provides labels of readmission
or mortality after discharge for each patient, any predictive model can be
trained for this task. The dataset contains 4.9% of patients who were
readmitted or died at the ICU. Samples of patients at the last day of
admission (AmsterdamUMC-DIS) are used for training predictive models. The
rationale for this choice is that the predictive model should be used when
clinicians are already considering discharge of a patient from the ICU.

###  A.3 AUC Score of OOD Detection

We use AUC-ROC scores to report the performance of OOD detection methods.
First, models are trained on in-distribution training data. Then, they are
used to predict novelty scores of in-distribution testing data and each
samples in each OOD group. The scores of test in-distribution data and an OOD
group are then used to calculate AUC-ROC score by assigning a label 0 to all
in-distribution samples and a label of 1 to all samples in an OOD group.

###  A.4 OOD Tests for AmsterdamUMC Dataset

Length of stay. For the AmsterdamUMC dataset, we created OOD groups based on
the admission day at the ICU. In-distribution data for this experiment are
eligible patients at their last day at the ICU (AmsterdamUMC-DIS). To create
OOD groups, we used AmsterdamUMC-ROW dataset. For each patient we computed the
total length of stay at the ICU and bracketed samples based on the number of
days before discharge of each patient. Given that patients stayed at the ICU
for different number of days, the groups of patients with the highest number
of days before discharge contained fewer patients (see patient counts listed
in Figure 3).

Discharge destination. We created four OOD groups by applying the inclusion-
exclusion criteria of discharge destination. In-distribution data for this
experiment are eligible patients in the AmsterdamUMC-DIS-II dataset. We split
excluded patients into four groups: patients transferred to Critical Care Unit
(CCU) or ICU of another hospital contained ($n=245$) patients, a group of
patients that were transferred to another hospital contained ($n=456$)
patients, and two groups of patients that died at the ICU and were sent to a
mortuary either with autopsy ($n=4$) or without ($n=6$).

Received treatment: devices. To split patients on a received device-related
treatment, we separated patients based on whether they are connect to a
ventilator device ($n=15,374$) or to CVVH ($n=1,610$). In-distribution data
for this experiment are eligible patients in the AmsterdamUMC-ROW dataset.

Received treatment: medication. To split patients on a received medication-
related treatment we separated patients who were administered Enoximone
($n=57$) or Norepinephrine ($n=774$) in the last 24 hours. In-distribution
data for this experiment are eligible patients in the AmsterdamUMC-ROW
dataset.

COVID-19 patients and suspects. To create OOD group with of patients with a
new disease on AmsterdamUMC, we used groups of COVID-19 patients ($n=130$) and
COVID-19 suspects ($n=140$) which fall outside exclusion criteria. In-
distribution data for this experiment are eligible patients in the
AmsterdamUMC-ROW dataset.

###  A.5 Models

Given that discriminator models were shown to underperform on OOD detection
for medical tabular data (Ulmer et al., 2020), we limited our analysis to
density estimators. We included models that do not model explicit density
function, such as Autoencoder and Local Outlier Factor (LOF).

Autoencoder. AE was used in combination with a reconstruction error metric.
This metric was chosen because it is expected that the model will learn to
encode training distribution faithfully whereas samples that are different
from training data will not be reconstructed properly and will receive a high
reconstruction error. For the AmsterdamUMC dataset, we used the following
hyperparameters: both encoder and the encoder had 1 layer with 75 units, the
latent space contained 20 dimensions, and the learning rate was 0.007.

Variational Autoencoder. We implemented a Variational Autoencoder (VAE), one
of the most popular density estimation techniques, (Kingma and Welling, 2014).
Similar to AE, the novelty metric was chosen to be reconstruction error. For
experiments on AmsterdamUMC, we used the following hyperparameters: encoder
and the encoder contained 3 layers each with 25 units, the latent space
contained 10 dimensions, and the learning rate was 0.001. Additionally, we
used beta-annealing (a deterministic warm-up; Huang et al., 2018).

Probabilistic Principal Component Analysis. We used Probabilistic Principal
Component Analysis (PPCA; Tipping and Bishop, 1999) as a latent variable model
with a log likelihood metric. The hyperparameter for PPCA model was only the
number of components which was selected as 19 for AmsterdamUMC.

Deep Neural Gaussian Process. We implemented a deep neural Gaussian Process
model called Deterministic Uncertainty Estimation (DUE) according to (van
Amersfoort et al., 2021). This approach builds on the idea of feature-distance
awareness which denotes ability of model to quantify distance of new samples
to training data. DUE uses a distance-preserving neural network for feature
extraction and models uncertainties using a Gaussian Process in the last
layer. Standard deviation was used a novelty metric. The hyperparameters that
were used for AmsterdamUMC data were the following: we used the Matern 12
kernel function with 50 inducing points, feature-extracting neural network
consisted of 4 layers with 256 units, we set the Lipschitz coefficient at 0.5,
and the learning rate at 0.002.

Normalizing Flows. Normalizing flow models are tractable explicit density
estimators that learn training data distribution by series of transformations
into a normal distribution. We used Masked Autoregressive Flow as described in
(Papamakarios et al., 2018) from an open-source implementation available at
(Durkan et al., 2020). The hyperparameters were selected to be the following:
20 layers of composite transformations and reverse permutation, the number of
hidden units was 256, we used batch normalization between layers, and the
learning rate was set to 0.001.

Local Outlier Factor. LOF (de Vries et al., 2010) does not model density
function of training data, it expresses local density of features of a sample
and compares this density to closest neighbors. The novelty score is then an
average local reachability density of the neighbors denoted as outlier score.
The only tunable hyperparameter is the number of closest neighbors which was
set to 5.

##  Appendix B Extended Results

Below, we show figures of the following experiments performed on the
AmsterdamUMC which were described in Section 4:

  * •

Dynamic Inclusion-Exclusion Criteria: Discharge Destination (Appendix Figure
5).

  * •

Dynamic Inclusion-Exclusion Criteria: Days of Ventilation (Appendix Figure 6).

  * •

Dynamic Inclusion-Exclusion Criteria: Medication (Appendix Figure 7).

  * •

New disease: COVID-19 patients (Appendix Figure 8).

Figure 5: AUC-ROC scores of detecting patients that died at the ICU (Mortuary
with or without autopsy) and patients that were transferred to another
hospital (ICU or other department). This experiment was performed on the
AmsterdamUMC-DIS-II dataset (see dataset description in Appendix A.1) which
contains more categorical features than AmsterdamUMC-DIS used for the above-
described experiments. This could explain the poor performance of Flow in
detecting Mortuary with autopsy patients as the model is not suitable for
categorical data.  Figure 6: Top panel: AUC-ROC of detecting patients
connected to a ventilation machine. First day indicates a group of patients in
their first day of being connected to the machine, while last day indicates
patients that are about to be disconnected from the machine. Middle day is a
day in between first and last day of ventilation. Patients with at least 3
days of ventilation were selected. Mean and standard deviation shown ($n$=5).
Bottom panel: Averaged novelty score distributions. Plotted values are clipped
to 5-95% of novelty scores to prevent outlier scores from skewing the
distributions. Figure 7: Top panel: AUC-ROC of detecting patients on inotropic
or vasopressor medication. Mean and standard deviation shown ($n$=5). Bottom
panel: Averaged novelty score distributions. Plotted values are clipped to
5-95% of novelty scores to prevent outlier scores from skewing the
distributions. Figure 8: Top panel: AUC scores and standard deviations ($n$=5)
of detecting COVID-19 patients and suspects. Bottom panel: Novelty score
distributions. Plotted values are clipped to 5-95% of test novelty scores.
Figure 9: Schematic representation of OOD groups in a medical setting and
groups that can be constructed using inclusion-exclusion criteria. Table 2:
Mean and std of user CPU times (in seconds) of inference and SHAP calculation
for a single sample.

| Inference (n=1000) | SHAP (n=5)
---|---|---
AE (reconstr_err) | 0.0006 ± 0.0001 | 1.7583 ± 0.0789
DUE (std) | 0.0293 ± 0.0006 | 45.7132 ± 2.4317
Flow (log_prob) | 0.0781 ± 0.0011 | 23.9759 ± 0.7971
LOF (outlier_score) | 0.0041 ± 0.0002 | 68.1364 ± 0.2217
PPCA (log_prob) | 0.0016 ± 0.0001 | 1.7503 ± 0.0488
VAE (reconstr_err) | 0.1367 ± 0.0005 | 34.1674 ± 1.3905

A1.1 AmsterdamUMC Feature List

        age
        alanine_transaminase__mean__last_24h
        albumin__mean__last_24h__padded
        alkaline_phosphatase__mean__last_24h
        amylase__mean__last_24h
        arterial_blood_pressure_diastolic__mean__last_24h
        arterial_blood_pressure_systolic__mean__last_24h
        aspartate_transaminase__mean__last_24h
        base_excess__mean__last_24h
        bicarbonate_arterial__mean__last_24h
        bilirubin_total__mean__last_24h
        c_reactive_protein__mean__last_24h
        calcium_ionised__mean__last_24h
        cardiac_output__is_measured__last_24h
        cardiac_output__maximum__last_24h
        cardiac_output__minimum__last_24h
        chloride__mean__last_24h
        cough_stimulus__mode__last_24h__cough_reflex_normal
        cough_stimulus__mode__last_24h__cough_reflex_reasonable
        cough_stimulus__mode__last_24h__cough_reflex_weak
        creatinine__last__overall
        fio2__mean__last_24h
        fluid_out_urine__total_dose__last_24h
        gender_M
        gender_V
        glasgow_coma_scale_total__maximum__last_24h
        glasgow_coma_scale_total__minimum__last_24h
        glucose__mean__last_24h
        heart_rate__mean__last_24h
        hemoglobin__mean__last_24h
        lactate_dehydrogenase__mean__last_24h
        lactate_unspecified__last__overall
        leukocytes__mean__last_24h__padded
        magnesium__mean__last_24h
        neutrophils__mean__last_24h
        o2_flow__mean__last_24h
        o2_saturation__mean__last_24h
        pao2_over_fio2__mean__last_24h
        pco2_arterial__mean__last_24h
        ph_arterial__mean__last_24h
        phosphate__mean__last_24h
        potassium__mean__last_24h
        respiratory_rate_measured__mean__last_24h

        richmond_agitation_sedation_scale_score__maximum__last_24h
        richmond_agitation_sedation_scale_score__minimum__last_24h
        sodium__mean__last_24h
        temperature_blood__mean__last_24h
        temperature_internal__maximum__last_24h
        temperature_internal__minimum__last_24h
        thrombocytes__mean__last_24h
        tidal_volume__mean__last_24h
        time_since_start
        troponin_t__is_measured__last_24h
        troponin_t__maximum__last_24h
        ureum__mean__last_24h
        ureum_over_creatinine__mean__last_24h