# The Concordance Index Decomposition: A Measure for a Deeper Understanding of
Survival Prediction Models

Abdallah Alabdallah<EMAIL_ADDRESS>Mattias Ohlsson  Sepideh
Pashami  Thorsteinn Rögnvaldsson  Center for Applied Intelligent Systems
Research (CAISR), Halmstad University, Sweden  Dept. Astronomy and Theoretical
Physics, Lund University, Sweden  RISE Research Institutes of Sweden,
Stockholm, Sweden

###### Abstract

The Concordance Index (C-index) is a commonly used metric in Survival Analysis
for evaluating the performance of a prediction model. In this paper, we
propose a decomposition of the C-index into a weighted harmonic mean of two
quantities: one for ranking observed events versus other observed events, and
the other for ranking observed events versus censored cases. This
decomposition enables a finer-grained analysis of the relative strengths and
weaknesses between different survival prediction methods. The usefulness of
this decomposition is demonstrated through benchmark comparisons against
classical models and state-of-the-art methods, together with the new
variational generative neural-network-based method (SurVED) proposed in this
paper. The performance of the models is assessed using four publicly available
datasets with varying levels of censoring. Using the C-index decomposition and
synthetic censoring, the analysis shows that deep learning models utilize the
observed events more effectively than other models. This allows them to keep a
stable C-index in different censoring levels. In contrast to such deep
learning methods, classical machine learning models deteriorate when the
censoring level decreases due to their inability to improve on ranking the
events versus other events.

###### keywords:

Survival Analysis , Evaluation Metric , Concordance Index , Variational
Encoder-Decoder

††journal: Journal of Artificial Intelligence In Medicine

##  1 Introduction

More and more data is being collected to improve the estimation of the
probability of survival and the expected remaining lifetime, for humans as
well as equipment. Making such estimates is the purpose of Survival Analysis.
This is an analysis of the time to an _event_ , e.g., an individual’s death or
the breakdown of a piece of equipment. While several statistical methods for
survival analysis have been developed [1], the availability of large
quantities of data has spurred the development of machine learning (ML) based
approaches that consider more intricate covariate effects [2].

An important aspect of survival analysis is handling _censored_ cases, e.g.,
hospitalized patients who do not experience a relapse before the end of a
study, equipment that is replaced before a breakdown, or equipment that has
not experienced a breakdown yet. Censoring is very common in clinical studies
and can occur for various reasons. It is possible for a patient not to
experience the event during the study’s timeframe (for example, death or
relapse). Also, a patient might experience a different event, making it
impossible to follow up on the event of interest.

Censoring also makes it more difficult to evaluate the goodness-of-fit when
the target variable is not fully observed. Several evaluation metrics have
been proposed to assess various aspects of a model’s performance [3]. However,
the Concordance Index (C-index) is one of the most used metrics as it
encompasses both observed events and censored cases. In doing so, it
quantifies the rank correlation between actual survival times and a model’s
predictions. Multiple C-index estimators have been proposed, like Harrel’s
C-index [4], Uno’s C-index [5] (a modified weighted version of Harrel’s
C-index), and Gonen and Heller’s measure [6]. The latter serves as an
alternative estimator based on the reversed definition of concordance.
Finally, a time-dependent version of the C-index was proposed in [7], which
takes the whole survival function into consideration.

Harrel’s C-index, the focus of this study, is perhaps the most often used
index and has an intuitive and straightforward interpretation. It measures the
ability of a predictor to order subjects by estimating the proportion of
correctly ordered pairs among all comparable pairs in the dataset. In the
presence of censoring, there are two types of times; event time and censoring
time. This results in two types of comparable pairs: event vs. event $(ee)$
and event vs. censored $(ec)$. A predictor may not perform equally well in
ranking both types of comparable pairs. Comparisons of given models’
performance using the C-index tend to show few significant differences in
those datasets with a high ratio of censored cases. More significant
differences however appear on datasets with low censoring ratios. This
phenomenon can be attributed to unseen differences in the models’ abilities to
rank the different types of pairs $(ee)$ and $(ec)$.

We therefore propose a decomposition of the C-index into a weighted harmonic
mean of two quantities: the C-index for ranking observed events ($CI_{ee}$),
and a C-index for ranking observed events versus censored cases ($CI_{ec}$),
weighted by $\alpha\in[0,1]$. This decomposition makes it easier to understand
an algorithm’s strengths and weaknesses under different censoring levels. As
such, the role of the weighting factor $\alpha$ in assessing the balance of a
predictor when dealing with the two categories of pairs, namely $(ee)$ and
$(ec)$ becomes clearer.

From a modeling perspective, the primary outcome of such survival analyses is
the Survival Function denoted as $S(t)=P(T>t)$, which represents the
probability of surviving beyond time $t$, where $T$ is the event time. Over
time a number of classical statistical and machine learning models have been
developed to estimate the survival function $S(t)$ in a non-parametric, semi-
parametric, or parametric way [8, 9, 10, 11, 12, 13]. More recently however,
deep learning models have been introduced for survival time modeling [14, 15,
16, 17, 18, 19, 20, 21, 22, 23]. DeepSurv [15], for example, is a direct
extension of the Cox Proportional Hazard (CPH) model [10] that employs a deep
neural network in place of the CPH linear predictor. As such, DeepSurv
maintains the constraint of the proportional hazards assumption. Unlike
DeepSurv however, some deep learning models discretize the survival timeline.
Most notably, DeepHit [16] estimates the probability mass function based on a
discrete output. Predictions from such discrete-time models, in contrast to
continuous-time models, are however constrained by the choice of the upper
limit of the output timeline.

Deep generative models facilitate the estimation of date distributions. In the
case of survival analysis, deep generative models can be utilized to estimate
the distribution of the event times in both parametric and non-parametric ways
[14, 18]. The Deep Adversarial Time-to-Event model (DATE) [17] for example, is
a survival model based on Generative Adversarial Networks (GAN) [24]. DATE
estimates the event distribution in a non-parametric manner using adversarial
training and is trained to generate $p(t|\mathbf{x})$ while penalizing fake
samples ($\mathbf{x},t$). However, such GAN models suffer from instability
issues, such as the Mode Collapse and the Non-Convergence problems, making
them challenging to train and potentially lead to a poor local equilibrium
[25, 26].

Recently, the Variational Survival Inference (VSI) model [20] was introduced,
adopting variational inference to approximate $p(t|\mathbf{x})$. VSI is a
discrete-time model that employs two encoders, $p(z|\mathbf{x})$ and
$q(z|\mathbf{x},t)$, and encourages these two distributions to be similar by
using Kullback-Leibler divergence which means the model can better account for
interactions between covariates and event times. In addition, the VSI model
discretized output constrains the prediction timeline to be limited by the
maximum time in the training data. To highlight the importance of the
interactions between the covariates and the event times captured by the $q$
branch, the authors of the VSI model developed a variant of VSI, labeled VSI-
NoQ which lacks the encoder’s q branch. It is worth noting that although the
VSI performs significantly better than VSI-NoQ, the role of the
$q(z|\mathbf{x},t)$ branch is unclear.

In this work, a new survival model is proposed: SurVED (Survival Variational
Encoder-Decoder). SurVED is essentially a translation of the Variational Auto
Encoder (VAE) [27] into the field of survival analysis. It is a conditional
generative model with a single encoder and a single decoder, which learns to
model the distribution of events conditioned on the covariates $\mathbf{x}$.

SurVED and VSI are both variational-inference-based models. However, SurVED
derives its objective function from the DATE model [17]. This adaptation
enables SurVED to deal with continuous time where, unlike the VSI model, no
discretization is required. Moreover, SurVED does not impose any upper-limit
constraint on the timeline of the model predictions. The loss function has
separate terms with different weights for censored and non-censored samples.
Additionally, SurVED and VSI differ in terms of architecture. Specifically,
while VSI comprises two encoders $p(\mathbf{z}|\mathbf{x})$ and
$q(\mathbf{z}|\mathbf{x},t)$, where $q$ is utilized to capture the
interactions between the covariates and the event times, SurVED uses only one
encoder. This makes SurVED more similar to the variant VSI-NoQ, albeit with
additional regularization on the latent space, continuous output, and a
different loss function.

In summary, this work presents two contributions. Firstly, it derives a
decomposition of the concordance index which provides insights into the
distinctions between seemingly similar-performing models. It also helps to
understand why there are larger-magnitude differences between classical and
deep learning models in the case of low censoring. Ultimately, by showing
areas of strengths and weaknesses, the C-index decomposition has the potential
to serve as a guide in the development of new survival models and offers
insights to enhance existing ones. Additionally, this work introduces a new
continuous-time variational-based model that overcomes the limitations of its
predecessors, DATE and VSI, and achieves a ranking performance comparable to
the state of the art.

##  2 Method

In this section, we introduce the Concordance Index Decomposition as a new
approach to highlight the difference between survival models. Additionally, we
present the SurVED model (Survival Variational Encoder-Decoder) and provide an
overview of the four datasets used for numerical tests and comparisons.

###  2.1 The Concordance Index Decomposition

The C-index is a measure of the probability that the predicted event times
($\hat{t}_{i}$, $\hat{t}_{j}$) of two randomly selected subjects maintain the
same relative order as their true event times ($t_{i}$, $t_{j}$), i.e.,
$P(\hat{t}_{i}>\hat{t}_{j}|t_{i}>t_{j})$. It’s important to note that not all
pairs can be compared when censoring is present; a pair
$(\mathbf{x}_{i},\mathbf{x}_{j})$ is comparable (usable) if the earliest time
represents an event, or both times are events. Conversely, a pair is deemed
not comparable if the earliest time is censored or if both are censored cases
[28].

The C-index can be decomposed into two parts; one to measure the relative
ordering of cases with observed events, and another to measure the ordering of
cases with observed events relative to censored cases. This decomposition is
useful when comparing how methods perform in situations with a high proportion
of censored cases, to situations with a low proportion of censored cases.

We define the random variable $o_{ij}=\hat{t}_{i}>\hat{t}_{j}|t_{i}>t_{j}$
that takes the value $1$ if the $ij$ pair is ordered (concordant) and $0$ if
it is discordant. We also define the random variable $k_{ij}$, which takes the
value ($1$) if the ($ij$) pair is an event-event $(ee)$ pair and ($0$) if the
($ij$) is an event-censored $(ec)$ pair. To simplify the notation, $P(o)$
represents $P(o_{ij}=1)$, $P(ee)$ represents $P(k_{ij}=1)$, and $P(ec)$
represents $P(k_{ij}=0)$. Note that $P(ee)+P(ec)=1$. With these definitions,
the C-index can be written as $CI=P(o)$, and hence:

$\displaystyle\frac{1}{CI}$ $\displaystyle=$ $\displaystyle\frac{1}{P(o)}$
$\displaystyle=$ $\displaystyle\frac{P(ee)+P(ec)}{P(o)}$ $\displaystyle=$
$\displaystyle\frac{P(ee)}{P(o)}+\frac{P(ec)}{P(o)}$ $\displaystyle=$
$\displaystyle\frac{P(o|ee)}{P(o|ee)}\frac{P(ee)}{P(o)}+\frac{P(o|ec)}{P(o|ec)}\frac{P(ec)}{P(o)}$
$\displaystyle=$
$\displaystyle\frac{P(o|ee)P(ee)}{P(o)}\frac{1}{P(o|ee)}+\frac{P(o|ec)P(ec)}{P(o)}\frac{1}{P(o|ec)}$
$\displaystyle=$ $\displaystyle
P(ee|o)\frac{1}{P(o|ee)}+P(ec|o)\frac{1}{P(o|ec)}$ $\displaystyle=$
$\displaystyle P(ee|o)\frac{1}{P(o|ee)}+(1-P(ee|o))\frac{1}{P(o|ec)}$

We define $CI_{ee}$ as a C-index for event-event cases, $CI_{ec}$ as a C-index
for events-censored cases, and we introduce the notation $\alpha$ for the
conditional probability that the pair is an event-event pair ($ee$) given that
it is a correctly ordered pair:

$\displaystyle CI_{ee}$ $\displaystyle\equiv$ $\displaystyle P(o|ee)$ (1)
$\displaystyle CI_{ec}$ $\displaystyle\equiv$ $\displaystyle P(o|ec)$ (2)
$\displaystyle\alpha$ $\displaystyle\equiv$ $\displaystyle P(ee|o)=1-P(ec|o)$
(3)

This yields the following relationship, which shows that the full C-index
($CI$) is a weighted harmonic mean of the C-indices defined for the subsets
$ee$ and $ec$:

$\frac{1}{CI}=\alpha\frac{1}{CI_{ee}}+(1-\alpha)\frac{1}{CI_{ec}}$ (4)

The C-index and its decomposed parts $CI_{ee}$, $CI_{ec}$, and $\alpha$ can be
estimated based on the number of correctly ordered pairs $N^{+}$, incorrectly
ordered pairs $N^{-}$, and the number of ties $N^{=}$. Since there are two
kinds of comparable (usable) pairs: event-event pairs ($ee$) and event-
censored pairs ($ec$), then:

$\displaystyle N^{+}=N_{ee}^{+}+N_{ec}^{+}$ $\displaystyle
N^{-}=N_{ee}^{-}+N_{ec}^{-}$ $\displaystyle N^{=}=N_{ee}^{=}+N_{ec}^{=}$ (5)

There are multiple ways to handle ties, and we use the Somers’ $d$ measure
[29], which considers the ties in the event cases to be incomparable pairs. It
also considers the ties in the predicted values to be binary random guesses;
hence, half of them are counted as correctly ordered.

$CI=\frac{N^{+}+\frac{1}{2}N^{=}}{N^{+}+N^{-}+N^{=}}=\frac{N_{ee}^{+}+N_{ec}^{+}+\frac{1}{2}N_{ee}^{=}+\frac{1}{2}N_{ec}^{=}}{N_{ee}^{+}+N_{ec}^{+}+N_{ee}^{-}+N_{ec}^{-}+N_{ee}^{=}+N_{ec}^{=}}$
(6)

From expressions (1), (2), and (3) we thus have:

$\displaystyle CI_{ee}$ $\displaystyle=$
$\displaystyle\frac{N_{ee}^{+}+\frac{1}{2}N_{ee}^{=}}{N_{ee}^{+}+N_{ee}^{-}+N_{ee}^{=}}$
(7) $\displaystyle CI_{ec}$ $\displaystyle=$
$\displaystyle\frac{N_{ec}^{+}+\frac{1}{2}N_{ec}^{=}}{N_{ec}^{+}+N_{ec}^{-}+N_{ec}^{=}}$
(8) $\displaystyle\alpha$ $\displaystyle=$
$\displaystyle\frac{N_{ee}^{+}+\frac{1}{2}N_{ee}^{=}}{N_{ee}^{+}+N_{ec}^{+}+\frac{1}{2}N_{ee}^{=}+\frac{1}{2}N_{ec}^{=}}$
(9)

The factor $\alpha$ is the conditional probability that the pair is event-
event ($ee$) given that it is a correctly ordered pair. This factor weights
the contribution of the correct ordering of event-event pairs relative to the
correct ordering of event-censored pairs in the C-index. Changes in $\alpha$
are directly associated with variations in the model’s performance in
accurately ordering pairs and indirectly related to the ratio of observed
events to censored cases in the dataset. A predictor that can order all events
and censored cases correctly will have an $\alpha$ value equal to the fraction
of event-event pairs within the comparable pairs, a value we can denote as
$\alpha^{*}$. However, even an imperfect predictor can have
$\alpha=\alpha^{*}$ as long as it scores equally on event-event pairs and
event-censored pairs in proportion to their percentages; such a predictor can
be denoted as a “balanced” predictor.

The $\alpha$-Deviation is defined as the difference between $\alpha$ and
$\alpha^{*}$. A predictor that excels at ordering event-event ($ee$) pairs
more than event-censored ($ec$) pairs will have $\alpha>\alpha^{*}$, resulting
in a positive $\alpha$-Deviation. On the other hand, a predictor that is
better at ordering event-censored ($ec$) pairs compared to event-event ($ee$)
pairs will have $\alpha<\alpha^{*}$, leading to a negative $\alpha$-Deviation.

$\alpha$-Deviation $\displaystyle\equiv$ $\displaystyle\alpha-\alpha^{*}$ (10)
$\displaystyle\alpha^{*}$ $\displaystyle\equiv$
$\displaystyle\frac{N_{ee}}{N_{ee}+N_{ec}}$ (11)

where $N_{ee}$ and $N_{ec}$ are the number of the comparable (ee) and (ec)
pairs in the dataset. In this paper, we study the absolute value of the
$\alpha$-Deviation. This is a measure of how unbalanced the predictor is when
making mistakes.

###  2.2 SurVED: Survival Variational Encoder-Decoder

Our model, SurVED, employs a conditional generator $G_{\theta}$ to estimate
$f(t|\mathbf{x})$, the distribution of death conditioned on the covariate
vector $\mathbf{x}$, with $\theta$ representing the parameters of the model.
This generative model can be sampled to produce the conditional death function
$f(t|\mathbf{x})$, from which the conditional cumulative death distribution
function ($F$) and the conditional survival function ($S$) can be computed:

$\displaystyle F(t|\mathbf{x})$ $\displaystyle=$
$\displaystyle\int_{0}^{t}f(\tau|\mathbf{x})\,d\tau$ (12) $\displaystyle
S(t|\mathbf{x})$ $\displaystyle=$ $\displaystyle 1-F(t|\mathbf{x})$ (13)

The model comprises two components: an Encoder
$E_{\theta_{1}}(\mathbf{z}|\mathbf{x})$ which encodes the input $\mathbf{x}$
into a multi-dimensional Gaussian latent space represented by
$(\mu_{z},\sigma_{z})$, and a Decoder $D_{\theta_{2}}(t|\mathbf{z})$
responsible for decoding a sample $\mathbf{z}$ from the latent space and
generating a sample $t$ from the conditional distribution $f(t|\mathbf{x})$.
Here $\theta_{1}$ and $\theta_{2}$ constitute $\theta$; the total parameters
of $G_{\theta}$. For each input $\mathbf{x}$, $n$ values $t_{i}$
($i=1,\ldots,n$) from $f(t|\mathbf{x})$ are sampled. The survival function can
be estimated using the Kaplan-Meier estimator considering the sampled times
$t_{i}$ as observed event times. These $n$ samples ($t_{i}$) are also utilized
to estimate the expected value $\mathbb{E}_{t\sim f(t|\mathbf{x})}[t]$ for the
purpose of model evaluation.

####  2.2.1 The Objective Function

The objective function of the generative model $G_{\theta}$ consists of four
parts: $L_{e}$, $L_{c}$, $L_{KL}$, and $C_{lb}$. The first two, $L_{e}$ and
$L_{c}$, represent construction losses that are separately evaluated for event
cases and censored cases. These losses are designed to optimize the balance
between events and censored cases. The third term, $L_{KL}$, originates from
the VAE formulation and is the Kullback-Leibler divergence, serving as a
regularization term. The first three terms are:

$\displaystyle L_{e}$ $\displaystyle=$
$\displaystyle\mathbb{E}_{\mathbf{x}\sim
P_{e}(\mathbf{x})}\left[|t-G_{\theta}(\mathbf{x})|\right]$ (14) $\displaystyle
L_{c}$ $\displaystyle=$ $\displaystyle\mathbb{E}_{\mathbf{x}\sim
P_{c}(\mathbf{x})}\left[\max(0,t-G_{\theta}(\mathbf{x}))\right]$ (15)
$\displaystyle L_{KL}$ $\displaystyle=$ $\displaystyle\mbox{\it
KL}\left(P(\mathbf{z}|\mathbf{x}),N(0,1)\right)$ (16)

where the subscripts, _e_ and _c_ , indicate that the terms exclusively
involve event cases or censored cases, respectively. The notation
$P_{e}(\mathbf{x})$ denotes that $\mathbf{x}$ was drawn from the event cases,
while $P_{c}(\mathbf{x})$ indicates that $\mathbf{x}$ was drawn from the
censored cases. Additionally, $\mbox{\it KL}(p,q)$ represents the Kullback-
Leibler divergence between the two distributions $p$ and $q$. The fourth term

$C_{lb}(\theta,\varepsilon)=\frac{1}{|\varepsilon|}\sum_{(\mathbf{x}_{i},\mathbf{x}_{j})\in\varepsilon}\left(1+\frac{\log\sigma(G_{\theta}(\mathbf{x}_{i})-G_{\theta}(\mathbf{x}_{j}))}{\log
2}\right)$ (17)

is a differentiable lower bound for the C-index [30]. Here, $\varepsilon$ is
the set of comparable pairs, the symbol $\sigma$ is the standard sigmoid
function, and $|\varepsilon|$ denotes the set $\varepsilon$ cardinality.
Adding the $C_{lb}$ term to the loss function enables the model to directly
optimize the C-index, encouraging concordance in the model predictions. The
SurVED model aims to minimize the total loss:

$L=\lambda_{e}L_{e}+\lambda_{c}L_{c}+\lambda_{KL}L_{KL}-\lambda_{lb}C_{lb}$
(18)

where the $\lambda_{e}$, $\lambda_{c}$, $\lambda_{KL}$, and $\lambda_{lb}$ are
tunable weights.

These objective terms have been used previously in the literature in different
settings. The $L_{e}$ and $L_{c}$ terms, eqs. (14) and (15), match the
$\ell_{2}$ and $\ell_{3}$ terms used in the DATE loss function [17]. However,
they can be traced back to earlier work by Van Belle et al. [12]. The fourth
objective term, eq. (17), was suggested for the DATE model [17] as well.

###  2.3 Description of Datasets

The SurVED method has been evaluated against the reference methods on four
publicly available medical datasets. The datasets are all fairly large, and
cover different censoring levels, number of samples, and number of features;
see Table 1. They have also been used in several previous benchmark studies.

FLCHAIN: A dataset used in a study [31] to determine whether the free light
chain (FLC) assay is a predictor of better/worse survival for the general
population. The study showed that a high FLC was significantly predictive of
worse overall survival.

METABRIC: The Molecular Taxonomy of Breast Cancer International Consortium
dataset [32]. This dataset is used to predict the survivability of breast
cancer patients using gene expression profiles and clinical data.

NWTCO: Data from the US National Wilm’s Tumor Study to predict survival based
on tumor histology [33]. This data is available in the package survival in R
[34].

SUPPORT: This data comes from the Study to Understand Prognoses and
Preferences for Outcomes and Risks of Treatment [35]. This study aimed to
understand the survival of seriously ill hospitalized patients and validate
the predictions of a new prognostic model against an existing prognostic model
and predictions by physicians. The SUPPORT data is sometimes split into
subsets since there is more than one diagnosis, but it is used as one dataset
here.

Table 1: Dataset statistics
Dataset |  Events ($\%$) | Samples | Features |  Missing Values ($\%$)
---|---|---|---|---
FLCHAIN |  27.55$\%$ | 7,874 | 25 |  0.6$\%$
METABRIC |  44.83$\%$ | 1,981 | 79 |  0.0$\%$
NWTCO |  14.18$\%$ | 4,028 | 9 |  0.0$\%$
SUPPORT |  68.11$\%$ | 9,105 | 59 |  6.5$\%$

###  2.4 Experimental Settings

Seven models were compared: Cox Proportional Hazard model (CPH), Random
Survival Forest (RSF), Deep Adversarial Time-to-Event model (DATE), DeepHit,
DeepSurv, Variational Learning of Individual Survival Distributions model
(VSI), and our model Survival Variational Encoder-Decoder (SurVED). The models
were first compared based on the C-index performance and then analyzed further
using the C-index Decomposition.

The same sampling scheme was applied to all the experiments: $30\%$ of the
data was used as a hold-out test set, and the remaining $70\%$ was used for
hyperparameter tuning and training. The models were tuned using five-fold
cross-validation, maximizing the C-index performance. At each fold, three sets
were used for training, one set for early stopping for deep learning models,
and the last set was used for validation. The early stopping set was not used
for optimizing RSF. In the final testing phase, a $100$-fold testing on the
hold-out test set was done, varying the training data. At each fold, $90\%$ of
the training data was used to train the models keeping $10\%$ as a validation
set for deep-learning models.

The categorical features were one-hot encoded, and the numerical features were
standardized with zero mean and unit variance. The target variable was scaled
by the maximum value of the training set, and power transformed. Moreover, the
missing values were filled with the training data median and mode for
numerical and categorical features, respectively. The deep learning models
were configured with a common architecture that included two hidden layers
with $32$ nodes, a hyperbolic tangent activation function, and a $0.5$ dropout
rate on the first hidden layer. For the models that have special types of
structure (DATE and VSI), we used the suggested structures in their
repositories. SurVED has a latent size of four nodes and a single-layer linear
perceptron as its decoder. Details about the network structures, data
standardization, and transformation are available on our Github
repository111https://github.com/abdoush/SurVED. DATE’s implementation from the
authors’ GitHub
repository222https://github.com/paidamoyo/adversarial_time_to_event was used,
while the Scikit-Survival library [36] was used for the CPH and the RSF
models. Moreover, the VSI model implementation provided by the authors on
Github333https://github.com/ZidiXiu/VSI was used. For DeepHit and DeepSurv,
the PyCox library [37] was used. A random search was performed to optimize the
weights of the loss functions for DeepHit and SurVED. The number of output
bins for the two discrete models, VSI and DeepHit, were optimized with choices
including $[100,200,400,1000]$. Additionally, a random search was conducted
for RSF to optimize parameters such as max_depth, min_samples_split,
min_samples_leaf, and max_features.

##  3 Results and Discussion

###  3.1 Comparison on the four data sets

Tables 2, 3, 4, and 5, present a comprehensive list of methods’ scores based
on the C-index ($CI$), C-index for event-event pairs ($CI_{ee}$), C-index for
event-censored pairs ($CI_{ec}$), and $\alpha$-Deviation on the four datasets.
In the context of the C-index, higher values indicate better performance.
Conversely, when considering the $\alpha$-Deviation, lower values reflect a
more “balanced” model, i.e., it performs more equally in ordering event-event
and event-censored pairs. The statistical significance level was set to $5\%$,
and hypothesis testing was carried out with $100$-fold test values using the
Wilcoxon rank-sum test.

We begin by comparing SurVED with DATE and VSI as it has close ties to both
models. SurVED shares the same loss function with the DATE model and employs a
variational inference approach similar to the VSI model. The results depicted
in Figure 1 demonstrate that SurVED, with its regression-based loss function,
outperformed the discrete-time-based VSI and the GAN-based DATE model across
all datasets.

It is worth noting that the VSI model exhibited unstable performance on
METABRIC datasets, depicted by the large variance of its results as shown in
Figure 1. This instability may be attributed to the fact that METABRIC is the
smallest dataset with the largest time horizons spanning over $9,200$ days. In
such cases, time discretization can lead to information loss.

Remarkably, although SurVED outperformed DATE in the C-index on NWTOC,
FLCHAIN, and METABRIC they demonstrated contrasting behaviors regarding
$CI_{ee}$, $CI_{ec}$, and $\alpha$-Deviation. While DATE showed better
performance in $CI_{ee}$ on these three datasets, SurVED was better in terms
of $CI_{ec}$. Additionally, due to its higher $\alpha$-Deviation, SurVED
placed higher weight on the $CI_{ec}$, resulting in a higher overall $CI$
performance.

(a) C-index

(b) Cee

(c) Cec

(d) $\alpha$-Deviation

Figure 1: The results of $CI$, $\alpha$, $CI_{ee}$, $CI_{ec}$, and
$|\alpha\text{-Deviation}|$ in eq. (4) of the SurVED, DATE, and VSI models on
the four datasets (Censoring level decreases from the highest censoring
(NWTCO) to the lowest censoring (SUPPORT)).

Looking at the full list of results in Tables 2, 3, 4, and 5 we see that in
the cases where there are no significant differences between the models in the
C-index, they show significant differences in the decomposition terms
$CI_{ee}$ and $CI_{ec}$.

For example, comparing RSF and DeepHit on the NWTCO dataset shows that RSF has
a significantly better $CI_{ee}$ with no significant difference observed on
the $CI_{ec}$. However, because DeepHit has a higher $\alpha$-Deviation, it
places more weight on the $CI_{ec}$, resulting in no significant difference in
the overall C-index. A similar scenario unfolds when comparing SurVED and CPH
on the FLCHAIN dataset.

More interesting cases show contrasting differences in the decomposition terms
leading to an insignificant difference in the C-index due to weighted
averaging. For instance, on the NWTCO dataset, DeepHit exhibits a higher
$CI_{ee}$ while CPH outperforms in $CI_{ec}$. Consequently, the total C-index
shows no significant difference. A similar phenomenon is observed on the
FLCHAIN dataset when comparing RSF with DeepHit and DeepSurv, where RSF excels
in $CI_{ee}$ while DeepHit and DeepSurv demonstrate better performance in
$CI_{ec}$, thereby diminishing the difference in the total C-index. This
pattern is also observed in the comparison between DeepHit and DeepSurv on the
FLCHAIN and the METABRIC datasets.

Contrasting differences in the decomposition terms do not always diminish the
difference in the total C-index. In some cases, a higher $\alpha$-Deviation
can outweigh one model over another. For example, consider the comparison of
SurVED and DeepSurv on NWTCO, where DeepSurv exhibits a higher $CI_{ee}$ while
SurVED has a higher $CI_{ec}$. Nevertheless, SurVED’s higher
$\alpha$-Deviation shifts the balance in favor of the $CI_{ec}$ term,
resulting in a higher C-index. Similar scenarios arise in various cases like
the comparison of CPH with RSF, DATE, VSI, and DeepSurv on NWTCO. In all these
cases CPH demonstrates a lower $CI_{ee}$ but a higher $CI_{ec}$ and a higher
$\alpha$-Deviation resulting in a higher C-index.

Occasionally, outweighing one term does not compensate for the differences in
the terms, especially when the difference is substantial. For example,
consider the case of CPH compared to DATE, DeepHit, and DeepSurv on the
METABRIC dataset. While CPH has a higher $CI_{ec}$ and a higher
$\alpha$-Deviation, it has a much lower $CI_{ee}$. In this scenario,
outweighing the $CI_{ec}$ term does not compensate for the considerable gap in
the $CI_{ee}$ term, resulting in CPH having a significantly lower total
C-index.

Poor performance on the METABRIC dataset was observed for the DeepHit model.
This is similar to the VSI model which shares the discrete-time property with
DeepHit. It is worth noting that this result cannot be compared to the result
reported in DeepHit paper [16] as they used a different version of the
METABRIC dataset, where they re-scaled the time step to a month instead of a
day as in our case. Additionally, they used the time-dependent C-index (Ctd)
as an evaluation measure.

Overall, the results indicate that classical models either outperformed or
performed equally well compared to deep learning models for the smaller
datasets with higher censoring levels. RSF was the best on METABRIC, while CPH
was the best on NWTCO. On FLCHAIN, RSF shares the best performance with
DeepSurv and DeepHit. However, deep learning models have a clear advantage on
SUPPORT, the largest dataset with the lowest censoring level.

To assess the models comprehensively, pair-wise comparisons were performed
between the seven models on the four datasets. Each model was compared against
the other six models on each dataset, resulting in $24$ comparisons for each
model. The results are summarized in Figure 2 as Win/Lose/Draw.

(a) CI

(b) CIee

(c) CIec

(d) $\alpha$-Deviation

Figure 2: The Win/Lose/Draw comparison based on $CI$, $CI_{ee}$, $CI_{ec}$,
and $\alpha$-Deviation in eq. (4) of the compared models on the four datasets.

The chi-square test was applied to the Win/Lose/Draw data in Figure 2,
treating draws as a $50\%$ chance of winning or losing. Regarding the C-index
performances, SurVED, DeepHit, RSF, and DeepSurv show a similar performance,
whereas CPH, DATE, and VSI lag behind. However, analyzing the other C-index
decomposition terms reveals more interesting insights. For example, DATE has
an excellent performance in terms of the $CI_{ee}$ but falls short in the
$CI_{ec}$ which impacts its overall C-index. In contrast, the VSI model shows
poor performance in both terms. The results also show that the main
differences between the models stem from the $CI_{ee}$ part, while all models,
except for DATE and VSI, exhibit similar overall $CI_{ec}$ performance.

Table 2: The C-index ($CI$) values (%) of the compared models on the four
datasets. Numbers show the median, the 2.5%, and the 97.5% quantiles of
100-folds. The highest numerical value in each dataset is boldfaced.

| NWTCO | FLCHAIN | METABRIC | SUPPORT
---|---|---|---|---
CPH | 72.91 (72.57, 73.25) | 78.37 (78.30, 78.46) | 63.90 (63.02, 64.68) | 84.29 (84.19, 84.57)
RSF | 72.84 (72.23, 73.40) | 78.43 (78.28, 78.62) | 67.80 (67.22, 68.49) | 84.17 (83.80, 84.55)
DATE | 70.06 (68.85, 71.32) | 76.84 (76.44, 77.38) | 65.09 (63.49, 66.87) | 84.38 (83.54, 84.96)
DeepSurv | 72.05 (70.40, 73.24) | 78.45 (77.99, 78.58) | 64.40 (61.96, 66.11) | 87.88 (87.55, 88.05)
DeepHit | 72.88 (70.43, 73.36) | 78.43 (78.27, 78.57) | 63.99 (63.26, 64.80) | 88.22 (88.01, 88.42)
VSI | 68.69 (53.68, 71.35) | 77.70 (75.09, 78.24) | 52.04 (45.61, 65.94) | 87.40 (84.71, 87.77)
SurVED | 72.75 (69.26, 73.37) | 78.40 (76.92, 78.53) | 66.63 (63.72, 67.58) | 88.13 (87.76, 88.27)

Table 3: The $CI_{ee}$ values (%) of the compared models on the four datasets.
Numbers show the median, the 2.5%, and the 97.5% quantiles of 100-folds. The
highest numerical value in each dataset is boldfaced.

| NWTCO | FLCHAIN | METABRIC | SUPPORT
---|---|---|---|---
CPH | 56.38 (56.08, 56.65) | 57.48 (57.37, 57.58) | 55.98 (54.84, 56.99) | 82.28 (82.14, 82.78)
RSF | 57.39 (56.60, 57.94) | 57.90 (57.73, 58.08) | 61.43 (60.62, 62.29) | 80.54 (79.83, 81.28)
DATE | 57.16 (56.04, 57.94) | 57.89 (57.45, 58.45) | 61.63 (59.34, 63.86) | 80.58 (79.24, 81.58)
DeepSurv | 56.67 (55.46, 57.64) | 57.43 (56.96, 57.60) | 57.78 (56.66, 59.12) | 85.94 (85.56, 86.19)
DeepHit | 57.28 (55.95, 57.72) | 57.59 (57.47, 57.71) | 59.17 (57.87, 60.40) | 86.82 (86.63, 86.98)
VSI | 56.90 (50.49, 58.52) | 57.15 (56.40, 57.97) | 51.03 (46.01, 60.15) | 85.63 (81.67, 86.21)
SurVED | 56.36 (55.36, 57.18) | 57.37 (56.84, 57.62) | 60.83 (57.74, 62.03) | 86.70 (86.08, 86.94)

Table 4: The $CI_{ec}$ values (%) of the compared models on the four datasets.
Numbers show the median, the 2.5%, and the 97.5% quantiles of 100-folds. The
highest numerical value in each dataset is boldfaced.

| NWTCO | FLCHAIN | METABRIC | SUPPORT
---|---|---|---|---
CPH | 74.34 (73.97, 74.71) | 82.79 (82.70, 82.89) | 68.70 (67.73, 69.75) | 86.56 (86.36, 86.71)
RSF | 74.18 (73.56, 74.76) | 82.76 (82.60, 82.99) | 71.75 (71.03, 72.58) | 88.30 (88.16, 88.47)
DATE | 71.19 (69.88, 72.57) | 80.86 (80.40, 81.53) | 67.31 (64.54, 69.63) | 88.79 (87.94, 89.37)
DeepSurv | 73.41 (71.61, 74.67) | 82.89 (82.40, 83.03) | 68.38 (64.36, 70.95) | 90.11 (89.55, 90.43)
DeepHit | 74.25 (71.68, 74.71) | 82.84 (82.63, 83.01) | 66.84 (65.90, 68.28) | 89.82 (89.44, 90.13)
VSI | 69.64 (54.03, 72.63) | 82.06 (78.92, 82.67) | 53.04 (45.10, 69.71) | 89.39 (88.27, 89.94)
SurVED | 74.17 (70.46, 74.80) | 82.84 (81.07, 83.01) | 70.06 (67.05, 71.65) | 89.79 (88.95, 90.11)

Table 5: The $\alpha$-Deviation values of the compared models on the four
datasets. Numbers show the median, the 2.5%, and the 97.5% quantiles of
100-folds. All values are scaled by a factor of $10^{2}$. The lowest numerical
value in each dataset is boldfaced.

| NWTCO | FLCHAIN | METABRIC | SUPPORT
---|---|---|---|---
CPH | 1.81 (1.77, 1.85) | 4.65 (4.63, 4.68) | 4.75 (4.26, 5.17) | 1.26 (1.11, 1.35)
RSF | 1.69 (1.64, 1.76) | 4.57 (4.52, 4.61) | 3.59 (3.20, 3.96) | 2.31 (2.06, 2.53)
DATE | 1.47 (1.33, 1.61) | 4.30 (4.21, 4.45) | 1.92 (1.07, 3.25) | 2.44 (2.00, 2.80)
DeepSurv | 1.72 (1.53, 1.84) | 4.67 (4.63, 4.72) | 3.89 (2.14, 4.70) | 1.18 (1.00, 1.35)
DeepHit | 1.71 (1.62, 1.78) | 4.64 (4.59, 4.68) | 2.87 (2.20, 3.55) | 0.85 (0.73, 0.93)
VSI | 1.32 (0.52, 1.67) | 4.62 (4.21, 4.77) | 1.73 (0.11, 4.14) | 1.16 (0.76, 1.93)
SurVED | 1.78 (1.53, 1.83) | 4.67 (4.47, 4.72) | 3.28 (2.31, 4.20) | 0.87 (0.61, 1.11)

The Deep learning models outperformed classical models by a substantial margin
on the SUPPORT dataset. To understand this notable difference and to explore
how the models behave under different levels of censoring and dataset sizes,
the following section employs the C-index decomposition to investigate the
models’ performances across various conditions simulated using the SUPPORT
dataset.

###  3.2 The Effect of Censoring and Size

Among the datasets utilized in this paper, the SUPPORT dataset is the largest
and has the highest proportion of events. This characteristic allowed us to
investigate the impact of varying the censoring and the dataset size across
three different dimensions. Originally, the dataset contained 9,105 examples,
with 6,201 observed events and 2,904 censored cases, resulting in $68\%$
events, and $32\%$ censored cases. In the first experiment (Size Only), we
varied the dataset size by randomly removing examples while keeping the
censoring level fixed. This resulted in four datasets with different sizes
(3,642, 4,462, 5,828, and 9,105) and approximately the same event percentage
of $68\%$. In the second (Censoring Only), we varied the censoring level by
randomly censoring observed events while maintaining the size. This resulted
in four datasets of the same size (9,105) with varying event percentages
($20\%$, $35\%$, $50\%$, and $68\%$). Lastly, in the third experiment (Size
and Censoring), we simultaneously varied both dataset size and censoring
level, by randomly dropping observed event examples. This resulted in four
datasets with different censoring levels (events percentages) ($20\%$, $35\%$,
$50\%$, and $68\%$) and different sizes (3,630, 4,467, 5,808, and 9,105)
respectively. The models were trained and tested on each of the four datasets
in each experiment, and Fig. 3 illustrates how the C-indices for the models
changed with varying dataset sizes and fractions of event cases (different
levels of censoring). It is worth noting that the right-hand side of the three
figures 3(a), 3(b), and 3(c) is the performance of the models on the original
SUPPORT dataset.

Two distinct types of behaviors can be observed in these experiments (see Fig.
3): One related to the group {SurVED, DeepSurv, DeepHit, VSI}, i.e., the deep
learning models except for DATE, and one related to the group {DATE, CPH,
RSF}, i.e., the classical models plus DATE. In the first experiment, Figure
3(a), where only the dataset size was changed, all the models improved in
C-index performance as the dataset size increased. However, they maintained
their relative differences between the two groups. In the second experiment,
Figure 3(b), where only the censoring level was varied, the models’
performances remained relatively constant, with DATE and the classical models
exhibiting a slight drop in the C-index performances.

The most intriguing result was obtained in the third experiment, Figure 3(c),
where classical models behaved unexpectedly when both the size and the
censoring level of the dataset were varied. The Deep learning models
maintained a constant C-index performance as the data set size and the
percentage of the observed events both decreased (reading Figure 3(c) from
right to left). In contrast, DATE and the classical models’ performance
improved eventually reaching a point where, in the extreme case of the
smallest dataset and the lowest event percentage (the left-hand side of Figure
3(c)), all models performed similarly.

(a) Size Only

(b) Censoring Only

(c) Size and Censoring

Figure 3: The change of $CI$ as the size of the dataset and the ratio of
events change. The x-axis shows the sizes of the datasets and percentages of
the events (for the SUPPORT dataset) in the three experiments.

To better understand these trends in the behavior concerning changes in
censoring levels and dataset size, the performance of the models was further
examined using the C-index Decomposition. The aim was to shed light on the
underlying reasons behind such differences in behavior.

Figure 4 shows the C-index decomposition of the seven models on SUPPORT
datasets in the three experiments (varying the dataset size only, varying the
censoring level only, and varying both the size and the censoring level). Two
distinct trends in behavior are observed: one corresponding to classical
models, CPH and RSF. The other one corresponds to the deep learning models
except for DATE, which followed the classical models’ behavior. Hence DATE
will be included with the classical models when referring to the classical
models’ behavior below.

In the first experiment (the leftmost column in Figure 4), increasing the size
of the dataset led to an increase in both the $CI_{ee}$ and $CI_{ec}$.
Furthermore, keeping the percentage of the events fixed maintained similar
values for the $\alpha$ term in the decomposition through the four datasets
(approximately $0.5$). This balance in the $\alpha$ gave equal weight to the
two terms in the C-index decomposition resulting in improvement in the total
C-index for all models with increased dataset size.

(a) SurVED-Size Only

(b) SurVED-Censoring Only

(c) SurVED-Censoring and Size

(d) DeepHit-Size Only

(e) DeepHit-Censoring Only

(f) DeepHit-Censoring and Size

(g) DeepSurv-Size Only

(h) DeepSurv-Censoring Only

(i) DeepSurv-Censoring and Size

(j) VSI-Size Only

(k) VSI-Censoring Only

(l) VSI-Censoring and Size

(m) DATE-Size Only

(n) DATE-Censoring Only

(o) DATE-Censoring and Size

(p) CPH-Size Only

(q) CPH-Censoring Only

(r) CPH-Censoring and Size

(s) RSF-Size Only

(t) RSF-Censoring Only

(u) RSF-Censoring and Size

Figure 4: The change of $CI$, $CI_{ee}$, $CI_{ec}$, and $\alpha$ in eq. (4) as
the ratio of events changes. The x-axis shows different percentages of events
(for the SUPPORT dataset).

In the second experiment (the middle column in Figure 4), keeping the size
fixed and decreasing the censoring level (increasing events %) slightly
increased the $CI_{ee}$ performance for deep learning models and, to a lesser
extent, for classical models. On the other hand, the $CI_{ec}$ stayed almost
constant for deep learning models, with a slight increase for classical
models. Nevertheless, changing the censoring level affected $\alpha$ changing
the weighting on the two decomposition terms across four datasets. As a
result, with smaller $\alpha$, the total C-index was mainly influenced by the
$CI_{ec}$ at the high censoring level (low events % to the left side of the
figure), whereas $\alpha$ increases (hence the weight on the $CI_{ee}$) as the
events percentage increase. This caused the total C-index to stay constant for
deep learning models but slightly decreased for classical models.

In the third experiment, when changing the dataset’s size and the censoring
level (the column to the right in Figure 4), the impact became more
pronounced. All the methods essentially achieved high C-indices at a high
censoring level (low % of events) and smaller dataset, resulting in very
similar performances with respect to $CI$, $CI_{ee}$, and $CI_{ec}$. However,
at such a high censoring level, the $\alpha$ term of the C-index is relatively
small, which makes the C-index primarily influenced by the $CI_{ec}$ term with
minimal contribution from the $CI_{ee}$ term. As the size increases and
censoring decreases, the $\alpha$ value increases, giving more weight to the
$CI_{ee}$ term. In this case, as the classical models did not exhibit
improvements on the $CI_{ee}$, which remained almost the same as more events
were added to the dataset, this caused the total C-index to decrease with the
increasing weight on this term. In contrast, the deep learning models
exhibited an increase in $CI_{ee}$, which kept the total C-index the same for
all levels of censoring.

The main difference between the second and the third experiments lies in their
approach to handling censoring. In the second experiment (Censoring Only), a
fraction of the observed event examples are censored, while in the third
experiment (Censoring and size) those observed event examples are entirely
removed from the dataset. To achieve the same censoring percentage in the two
scenarios, more event cases need to be removed in the third experiment
compared to the ones that need to be censored in the second experiment. This
results in that, for example, a dataset with $20\%$ events in the second
experiment has 1,821 event cases compared to 726 event cases in a dataset with
a similar event percentage in the third experiment. This explains the larger
drop in performance in the $CI_{ee}$ in the third experiment which has less
number of observed event cases.

##  4 Conclusion

In this work, we derived a decomposition of the C-index, separating it into
two terms: one for ranking observed events, and another for ranking observed
events versus censored cases. These terms are weighted by the parameter
$\alpha$. The $\alpha$ factor expresses the contribution of the two parts for
the total C-index and can be interpreted as a conditional probability for
event-event pairs given that it is correctly ordered $P((ee)\
\mbox{pair}|\mbox{ordered pair})$. A model that perfectly orders the two types
of pairs will have an optimal $\alpha$ factor ($\alpha^{*}$). Unbalanced
models, i.e., models that are not equally good at ranking event-event pairs
and event-censored pairs will deviate from this value. Based on this deviation
from the $\alpha^{*}$, the $\alpha$-Deviation measure can assess how balanced
a model is with respect to the ranking of the two groups of pairs.

SurVED is also proposed, a new approach for estimating the time-to-event
distribution using a variational encoder-decoder with a Gaussian latent layer.
In benchmark tests, SurVED performs significantly better than the two closely
related methods, DATE and VSI, and achieves a comparable overall performance
to DeepSurv and DeepHit.

Using the C-index decomposition, it was shown that in cases where models
perform differently in terms of the $CI_{ee}$ and $CI_{ec}$, such differences
often go unnoticed when evaluating the total C-index due to the averaging.
Furthermore, it was demonstrated, using the SUPPORT dataset with varying
censoring levels and dataset size, that all methods benefitted from increasing
the dataset size. It was also shown that all methods have comparable
performance in terms of the total C-index at a high censoring percentage and
smaller dataset size, but all methods do better at ranking event-censored
pairs compared to ranking event-event pairs. However, as the number of events
grows, SurVED and the other deep learning models VSI, DeepSurv, and DeepHit
are better than the other algorithms at improving their performance in ranking
event-event pairs. This helped deep learning models maintain a constant
C-index performance across different censoring levels in contrast to the
classical models which suffered from a drop in the C-index. This explains the
large magnitude of the difference between deep learning models and the
classical models on the SUPPORT dataset.

This work focuses on analyzing the ranking performance of survival models
using the C-index decomposition trying to get a better understanding of the
strengths and weaknesses of models with respect to the different types of
events and censored observations. Such understanding drawn from decomposition
can help to design better objective functions of survival models which we
leave for future work. Moreover, studying the relation between the
decomposition terms and other evaluation metrics can potentially give more
insights that help develop better survival models which we also leave for
future work.

##  5 Declarations of interest

The authors declare that they have no known competing financial interests or
personal relationships that could have influenced the work reported in this
paper.

##  6 Acknowledgement

This research was performed under the CAISR+ project funded by the Swedish
Knowledge Foundation.

## References

  * [1] D. G. Kleinbaum, M. Klein, Survival Analysis – A Self-Learning Text, 3rd Edition, Springer, New York, NY, 2010.
  * [2] P. Wang, Y. Li, C. K. Reddy, Machine learning for survival analysis: A survey, ACM Computing Surveys 10 (6) (2019) Article 110.  doi:10.1145/3214306.
URL https://doi.org/10.1145/3214306

  * [3] M. S. Rahman, G. Ambler, B. Choodari-Oskooei, R. Z. Omar, Review and evaluation of performance measures for survival prediction models in external validation settings, BMC Medical Research Methodology 17 (60).  doi:10.1186/s12874-017-0336-2.
URL https://doi.org/10.1186/s12874-017-0336-2

  * [4] F. E. Harrell Jr., R. M. Califf, D. B. Pryor, K. L. Lee, R. A. Rosati, Evaluating the Yield of Medical Tests, JAMA 247 (18) (1982) 2543–2546.
URL https://doi.org/10.1001/jama.1982.03320430047030

  * [5] H. Uno, T. Cai, M. Pencina, R. D’Agostino, L. Wei, On the c-statistics for evaluating overall adequacy of risk prediction procedures with censored survival data., Statistics in Medicine 30 (10) (2011) 1105–1117.
URL https://doi.org/10.1002/sim.4154

  * [6] M. Gönen, G. Heller, Concordance probability and discriminatory power in proportional hazards regression, Biometrika 92 (4) (2005) 965–970.
URL https://doi.org/10.1093/biomet/92.4.965

  * [7] L. Antolini, P. Boracchi, E. Biganzoli, A time-dependent discrimination index for survival data, Statistics in Medicine 24 (24) (2005) 3927–3944.
URL https://doi.org/10.1002/sim.2427

  * [8] E. L. Kaplan, P. Meier, Nonparametric estimation from incomplete observations, Journal of the American Statistical Association 53 (282) (1958) 457–481.
URL http://www.jstor.org/stable/2281868

  * [9] L. J. Wei, The accelerated failure time model: A useful alternative to the cox regression model in survival analysis, Statistics in Medicine 11 (14-15) (1992) 1871–1879.  arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1002/sim.4780111409, doi:https://doi.org/10.1002/sim.4780111409.
URL https://onlinelibrary.wiley.com/doi/abs/10.1002/sim.4780111409

  * [10] D. R. Cox, Regression models and life-tables, Journal of the Royal Statistical Society. Series B (Methodological) 34 (2) (1972) 187–220.
URL http://www.jstor.org/stable/2985181

  * [11] H. Ishwaran, U. B. Kogalur, E. H. Blackstone, M. S. Lauer, Random survival forests, Ann. Appl. Stat. 2 (3) (2008) 841–860.
URL https://doi.org/10.1214/08-AOAS169

  * [12] V. Van Belle, K. Pelckmans, J. Suykens, S. Van Huffel, Additive survival least-squares support vector machines., Statistics in Medicine 29 (2) (2010) 296–308.
URL https://doi.org/10.1002/sim.3743

  * [13] V. Van Belle, K. Pelckmans, S. Van Huffel, J. A. Suykens, Support vector methods for survival analysis: a comparison between ranking and regression approaches, Artificial Intelligence in Medicine 53 (2) (2011) 107–118.  doi:https://doi.org/10.1016/j.artmed.2011.06.006.
URL https://www.sciencedirect.com/science/article/pii/S0933365711000765

  * [14] R. Ranganath, A. Perotte, N. Elhadad, D. Blei, Deep survival analysis, in: F. Doshi-Velez, J. Fackler, D. Kale, B. Wallace, J. Wiens (Eds.), Proceedings of the 1st Machine Learning for Healthcare Conference, Vol. 56 of Proceedings of Machine Learning Research, PMLR, Northeastern University, Boston, MA, USA, 2016, pp. 101–114.
URL http://proceedings.mlr.press/v56/Ranganath16.html

  * [15] J. L. Katzman, U. Shaham, A. Cloninger, J. Bates, T. Jiang, Y. Kluger, Deepsurv: personalized treatment recommender system using a cox proportional hazards deep neural network., BMC medical research methodology 18 (1) (2018) 24.
URL https://doi.org/10.1186/s12874-018-0482-1

  * [16] C. Lee, W. Zame, J. Yoon, M. van der Schaar, Deephit: A deep learning approach to survival analysis with competing risks, Proceedings of the AAAI Conference on Artificial Intelligence 32 (1).
URL https://ojs.aaai.org/index.php/AAAI/article/view/11842

  * [17] P. Chapfuwa, C. Tao, C. Li, C. Page, B. Goldstein, L. C. Duke, R. Henao, Adversarial time-to-event modeling, in: J. Dy, A. Krause (Eds.), Proceedings of the 35th International Conference on Machine Learning, Vol. 80 of Proceedings of Machine Learning Research, PMLR, Stockholmsmä ssan, Stockholm Sweden, 2018, pp. 735–744.
URL http://proceedings.mlr.press/v80/chapfuwa18a.html

  * [18] X. Miscouridou, A. Perotte, N. Elhadad, R. Ranganath, Deep survival analysis: Nonparametrics and missingness, in: F. Doshi-Velez, J. Fackler, K. Jung, D. Kale, R. Ranganath, B. Wallace, J. Wiens (Eds.), Proceedings of the 3rd Machine Learning for Healthcare Conference, Vol. 85 of Proceedings of Machine Learning Research, PMLR, Palo Alto, California, 2018, pp. 244–256.
URL http://proceedings.mlr.press/v85/miscouridou18a.html

  * [19] B. Jing, T. Zhang, Z. Wang, Y. Jin, K. Liu, W. Qiu, L. Ke, Y. Sun, C. He, D. Hou, L. Tang, X. Lv, C. Li, A deep survival analysis method based on ranking, Artificial Intelligence in Medicine 98 (2019) 1–9.  doi:https://doi.org/10.1016/j.artmed.2019.06.001.
URL https://www.sciencedirect.com/science/article/pii/S0933365718305992

  * [20] Z. Xiu, C. Tao, R. Henao, Variational learning of individual survival distributions, in: CHIL ’20: Proceedings of the ACM Conference on Health, Inference, and Learning, ACM, 2020, pp. 10–18.
URL https://doi.org/10.1145/3368555.3384454

  * [21] C. Nagpal, X. Li, A. Dubrawski, Deep survival machines: Fully parametric survival regression and representation learning for censored data with competing risks, IEEE Journal of Biomedical and Health Informatics 25 (8) (2021) 3163–3175.  doi:10.1109/JBHI.2021.3052441.
  * [22] S. Hu, E. Fridgeirsson, G. v. Wingen, M. Welling, Transformer-based deep survival analysis, in: R. Greiner, N. Kumar, T. A. Gerds, M. van der Schaar (Eds.), Proceedings of AAAI Spring Symposium on Survival Prediction - Algorithms, Challenges, and Applications 2021, Vol. 146 of Proceedings of Machine Learning Research, PMLR, 2021, pp. 132–148.
URL https://proceedings.mlr.press/v146/hu21a.html

  * [23] L. Xu, C. Guo, Coxnam: An interpretable deep survival analysis model, Expert Systems with Applications 227 (2023) 120218.  doi:https://doi.org/10.1016/j.eswa.2023.120218.
URL https://www.sciencedirect.com/science/article/pii/S0957417423007200

  * [24] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio, Generative adversarial nets, in: Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, K. Q. Weinberger (Eds.), Advances in Neural Information Processing Systems, Vol. 27, Curran Associates, Inc., 2014, pp. 2672–2680.
URL https://doi.org/10.1145/3422622

  * [25] N. Kodali, J. Abernethy, J. Hays, Z. Kira, On convergence and stability of gans (2017).  arXiv:1705.07215.
URL https://arxiv.org/abs/1705.07215

  * [26] H. Chen, Challenges and corresponding solutions of generative adversarial networks (GANs): A survey study, Journal of Physics: Conference Series 1827 (1) (2021) 012066.  doi:10.1088/1742-6596/1827/1/012066.
URL https://doi.org/10.1088/1742-6596/1827/1/012066

  * [27] D. P. Kingma, M. Welling, Auto-encoding variational bayes, 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings.
URL http://arxiv.org/abs/1312.6114

  * [28] F. E. Harrel Jr., K. L. Lee, D. B. Mark, Multivariable prognostic models: Issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors, Statistics in Medicine 15 (4) (1996) 361–387.
URL https://doi.org/10.1002/(SICI)1097-0258(19960229)15:4<361::AID-
SIM168>3.0.CO;2-4

  * [29] R. H. Somers, A new asymmetric measure of association for ordinal variables, American Sociological Review 27 (6) (1962) 799–811.
URL http://www.jstor.org/stable/2090408

  * [30] H. Steck, B. Krishnapuram, C. Dehing-Oberije, P. Lambin, V. C. Raykar, On ranking in survival analysis: Bounds on the concordance index, in: J. Platt, D. Koller, Y. Singer, S. Roweis (Eds.), Advances in Neural Information Processing Systems, Vol. 20, Curran Associates, Inc., 2008, pp. 1209–1216.
URL https://dl.acm.org/doi/10.5555/2981562.2981714

  * [31] A. Dispenzieri, J. A. Katzmann, R. A. Kyle, D. R. Larson, T. M. Therneau, C. L. Colby, R. J. Clark, B. Graham P. Mead, S. Kumar, L. J. M. III, S. V. Rajkumar, Use of nonclonal serum immunoglobulin free light chains to predict overall survival in the general population., Mayo Clin Proc. 87 (6) (2012) 517–23.
URL https://doi.org/10.1016/j.mayocp.2012.03.009

  * [32] E. Bilal, J. Dutkowski, J. Guinney, I. S. Jang, B. A. Logsdon, G. Pandey, B. A. Sauerwine, Y. Shimoni, H. K. M. Vollan, B. H. Mecham3, O. M. Rueda, J. Tost, C. Curtis, M. J. Alvarez, V. N. Kristensen, S. Aparicio, A.-L. Børresen-Dale, C. Caldas, A. Califano, S. H. Friend, T. Ideker, E. E. Schadt, G. A. Stolovitzky, A. A. Margolin, Improving breast cancer survival analysis through competition-based multidimensional modeling, PLoS Computational Biology.
URL https://doi.org/10.1371/journal.pcbi.1003047

  * [33] N. Breslow, N. Chatterjee, Design and analysis of two-phase studies with binary outcome applied to Wilms tumour prognosis, Journal of the Royal Statistical Society. Series C: Applied Statistics 48 (4) (1999) 457–468.
URL https://doi.org/10.1111/1467-9876.00165

  * [34] T. M. Therneau, A Package for Survival Analysis in R, r package version 3.2-7 (2020).
URL https://CRAN.R-project.org/package=survival

  * [35] W. A. Knaus, F. E. Harrell, J. Lynn, L. Goldman, R. S. Phillips, A. F. Connors, N. V. Dawson, W. J. Fulkerson, R. M. Califf, N. Desbiens, P. Layde, R. K. Oye, P. E. Bellamy, R. B. Hakim, D. P. Wagner, The support prognostic model. objective estimates of survival for seriously ill hospitalized adults. study to understand prognoses and preferences for outcomes and risks of treatments, Annals of internal medicine 122 (3) (1995) 191–203.
URL https://doi.org/10.7326/0003-4819-122-3-199502010-00007

  * [36] S. Pölsterl, scikit-survival: A library for time-to-event analysis built on top of scikit-learn, Journal of Machine Learning Research 21 (212) (2020) 1–6.
URL http://jmlr.org/papers/v21/20-729.html

  * [37] H. Kvamme, Ørnulf Borgan, I. Scheel, Time-to-event prediction with neural networks and cox regression, Journal of Machine Learning Research 20 (129) (2019) 1–30.
URL http://jmlr.org/papers/v20/18-424.html